Package: comet-llm
CometLLMis a tool to log and visualize your LLM prompts and chains. Use CometLLM to identify effective prompt strategies, streamline your troubleshooting, and ensure reproducible workflows!⚡️ QuickstartInstallcomet_llmPython library with pip:pipinstallcomet_llmIf you don't have already,create your free Comet accountand grab your API Key from the account settings page.Now you are all set to log your first prompt and response:importcomet_llmcomet_llm.log_prompt(prompt="What is your name?",output=" My name is Alex.",api_key="<YOUR_COMET_API_KEY>",)🎯 FeaturesLog your prompts and responses, including prompt template, variables, timestamps and duration and any metadata that you need.Visualize your prompts and responses in the UI.Log your chain execution down to the level of granularity that you need.Visualize your chain execution in the UI.Automatically tracks your prompts when using the OpenAI chat models.Track and analyze user feedback.Diff your prompts and chain execution in the UI.👀 ExamplesTo log a single LLM call as an individual prompt, usecomet_llm.log_prompt. If you require more granularity, you can log a chain of executions that may include more than one LLM call, context retrieval, or data pre- or post-processing withcomet_llm.start_chain.Log a full prompt and responseimportcomet_llmcomet_llm.log_prompt(prompt="Answer the question and if the question can't be answered, say\"I don't know\"\n\n---\n\nQuestion: What is your name?\nAnswer:",prompt_template="Answer the question and if the question can't be answered, say\"I don't know\"\n\n---\n\nQuestion: {{question}}?\nAnswer:",prompt_template_variables={"question":"What is your name?"},metadata={"usage.prompt_tokens":7,"usage.completion_tokens":5,"usage.total_tokens":12,},output=" My name is Alex.",duration=16.598,)Read the full documentation for more details about logging a prompt.Log a LLM chainfromcomet_llmimportSpan,end_chain,start_chainimportdatetimefromtimeimportsleepdefretrieve_context(user_question):if"open"inuser_question:return"Opening hours: 08:00 to 17:00 all days"defllm_answering(user_question,current_time,context):prompt_template="""You are a helpful chatbot. You have access to the following context:{context}The current time is:{current_time}Analyze the following user question and decide if you can answer it, if the question can't be answered, say\"I don't know\":{user_question}"""prompt=prompt_template.format(user_question=user_question,current_time=current_time,context=context)withSpan(category="llm-call",inputs={"prompt_template":prompt_template,"prompt":prompt},)asspan:# Call your LLM model heresleep(0.1)result="Yes we are currently open"usage={"prompt_tokens":52,"completion_tokens":12,"total_tokens":64}span.set_outputs(outputs={"result":result},metadata={"usage":usage})returnresultdefmain(user_question,current_time):start_chain(inputs={"user_question":user_question,"current_time":current_time})withSpan(category="context-retrieval",name="Retrieve Context",inputs={"user_question":user_question},)asspan:context=retrieve_context(user_question)span.set_outputs(outputs={"context":context})withSpan(category="llm-reasoning",inputs={"user_question":user_question,"current_time":current_time,"context":context,},)asspan:result=llm_answering(user_question,current_time,context)span.set_outputs(outputs={"result":result})end_chain(outputs={"result":result})main("Are you open?",str(datetime.datetime.now().time()))Read the full documentation for more details about logging a chain.⚙️ ConfigurationYou can configure your Comet credentials and where you are logging data to:NamePython parameter nameEnvironment variable nameComet API KEYapi_keyCOMET_API_KEYComet Workspace nameworkspaceCOMET_WORKSPACEComet Project nameprojectCOMET_PROJECT_NAME📝 LicenseCopyright (c)Comet2023-present.cometLLMis free and open-source software licensed under theMIT License.