Package: log10-io
log10⚡ Unified LLM data management to drive accuracy at scale ⚡Quick Installpip install log10-io🤔 What is this?A one-line Python integration to manage your LLM data.importopenaifromlog10.loadimportlog10log10(openai)# all your openai calls are now logged - including 3rd party libs using openaiFor OpenAI v1, usefrom log10.load import OpenAIinstead offrom openai import OpenAIfromlog10.loadimportOpenAIclient=OpenAI()Access your LLM data atlog10.io🚀 What can this help with?📝📊 LoggingUse Log10 to log both closed and open-source LLM calls. It helps you:Compare and identify the best models and prompts (tryplaygroundandllmeval)Store feedback for fine-tuningCollect performance metrics such as latency and usagePerform analytics and monitor compliance for LLM powered applicationsLog10 offers various integration methods, including a python LLM library wrapper, the Log10 LLM abstraction, and callbacks, to facilitate its use in both existing production environments and new projects.
Pick the one that works best for you.OpenAIlog10 veropenai v0openai v10.4load(openai)✅❌0.5load(openai)✅from log10.load import OpenAI✅OpenAI v0- Use library wrapperlog10(openai). Check outexamples/loggingin log10 version0.4.6.importopenaifromlog10.loadimportlog10log10(openai)# openai calls are now logged - including 3rd party libs using openai such as magentic or langchainOpenAI v1NOTE: We added OpenAI v1 API support in log100.5.0release.load.log10(openai)still works for openai v1.fromlog10.loadimportOpenAI# from openai import OpenAIclient=OpenAI()completion=client.completions.create(model="gpt-3.5-turbo-instruct",prompt="Once upon a time")# All completions.create and chat.completions.create calls will be loggedFull scripthere.Use Log10 LLM abstractionfromlog10.openaiimportOpenAIllm=OpenAI({"model":"gpt-3.5-turbo"},log10_config=Log10Config())openai v1+ lib required. Full scripthere.AnthropicUse library wrapperlog10(anthropic).
Full scripthere.importanthropicfromlog10.loadimportlog10log10(anthropic)# anthropic calls are now loggedUse Log10 LLM abstraction.
Full scripthere.fromlog10.anthropicimportAnthropicllm=Anthropic({"model":"claude-2"},log10_config=Log10Config())Open-source LLMsLog open-source LLM calls, e.g. Llama-2, Mistral, etc from providers.
Currently we support inference endpoints on Together.AI and MosaicML (ranked on the top based on ourbenchmarkingon Llama-2 inference providers).
Adding other providers is on the roadmap.MosaicMLwith LLM abstraction. Full scripthere.fromlog10.mosaicmlimportMosaicMLllm=MosaicML({"model":"llama2-70b-chat/v1"},log10_config=Log10Config())Togetherwith LLM abstraction. Full scripthere.fromlog10.togetherimportTogetherllm=Together({"model":"togethercomputer/llama-2-70b-chat"},log10_config=Log10Config())Other LLM frameworksUse Log10 callbacks if you use LangChain's LLM abstraction. Full scripthere.fromlangchain.chat_modelsimportChatOpenAIfromlangchain.schemaimportHumanMessagefromlog10.langchainimportLog10Callbackfromlog10.llmimportLog10Configlog10_callback=Log10Callback(log10_config=Log10Config())messages=[HumanMessage(content="You are a ping pong machine"),HumanMessage(content="Ping?"),]llm=ChatOpenAI(model_name="gpt-3.5-turbo",callbacks=[log10_callback])Read more here for options for logging using library wrapper, langchain callback logger and how to apply log10 tagshere.🤖👷 Prompt engineering copilotOptimizing prompts requires a lot of manual effort. Log10 provides a copilot that can help you with suggestions on how tooptimize your prompt.👷🔢 FeedbackAdd feedback to your completions. Checkout the Pythonexampleor use CLIlog10 feedback-task createandlog10 feedback create. Please check ourdocfor more details.🔍🐞 Prompt chain debuggingPrompt chains such as those inLangchaincan be difficult to debug. Log10 provides prompt provenance, session tracking and call stack functionality to help debug chains.🧠🔁 Readiness for RLHF & self hostingUse your data and feedback from users to fine-tune custom models with RLHF with the option of building and deploying more reliable, accurate and efficient self-hosted models.👥🤝 CollaborationCreate flexible groups to share and collaborate over all of the above features⚙️ SetupCreate a free account atlog10.ioSet the following environment variables:LOG10_URL=https://log10.ioLOG10_TOKEN: From the Settings tab in log10.ioLOG10_ORG_ID: From the Organization tab in log10.ioOPENAI_API_KEY: OpenAI API keyANTHROPIC_API_KEY: Anthropic API key✅ Run examples and testsYou can find and run examples under folderexamples, e.g. run a logging example:python examples/logging/chatcompletion.pyAlso you can run some end-to-end tests withxdocttestinstalled (pip install xdoctest).# list all tests
python -m xdoctest log10 list

# run all tests
python -m xdoctest log10 all

# run a single test, e.g.
python -m xdoctest /Users/wenzhe/dev/log10/log10/load.py log10:0LoggingFew options to enable debug logging:set environment varibleexport LOG10_DEBUG=1setlog10.load.log10(DEBUG_=True)when usinglog10.loadsetlog10_config(DEBUG=True)when using llm abstraction classes or callback.💿🧩 Flexible data storelog10 provides a managed data store, but if you'd prefer to manage data in your own environment, you can use data stores like google big query.Install the big query client library with:pip install log10-io[bigquery]And provide the following configuration in either a.envfile, or as environment variables:NameDescriptionLOG10_DATA_STOREEitherlog10orbigqueryLOG10_BQ_PROJECT_IDYour google cloud project idLOG10_BQ_DATASET_IDThe big query dataset idLOG10_BQ_COMPLETIONS_TABLE_IDThe name of the table to store completions inNotethat your environment should have been setup with google cloud credentials. Read morehereabout authenticating.💬 CommunityWe welcome community participation and feedback. Please leave an issue, submit a PR or join ourDiscord. For enterprise use cases, pleasecontact usto set up a shared slack channel.