Package: honeyhive
HoneyHive Python SDKHoneyHive is a model observability and evaluation platform that helps you continuously improve your models in production. We help you evaluate, deploy, monitor and fine-tune both closed and open-source large language models for production use-cases, allowing you to optimize model performance & align your models with your users’ preferences.SDK InstallationpipinstallhoneyhiveAuthenticationAfter signing up on the app, you can find your API key in theSettings page.importhoneyhivehoneyhive.api_key="HONEYHIVE_API_KEY"honeyhive.openai_api_key="OPENAI_API_KEY"UsageHere we are assuming that you have a project setup in HoneyHive. If you don't, you can create onehereby clicking on theNew Projectbutton.Generating a text completion via HoneyHiveCustom PromptAdding HoneyHive is as simple as changingopenai.Completions.createtohoneyhive.Completions.create. Just make sure to specify which project associated with the generation.It's recommended to move the prompt to a template in our platform so that we can analyze user input distributions better.response=honeyhive.Completions.create(project="Sandbox - Email Writer",model="text-davinci-003",prompt="Write an email to a colleague named Jill congratulating her on her promotion. The tone should be warm yet professional. Mention how you admire the work she's been putting in.  Include a joke about how her pet lizard Max enjoys eating grasshoppers. Mention how you're looking forward to the team off-site next week.",temperature=0,max_tokens=256,top_p=1,frequency_penalty=0,presence_penalty=0)generation_id=response.generation_idDeployed PromptAssuming you have a prompt deployed for your project, you can generate it by callinghoneyhive.generations.generate(response=honeyhive.generations.generate(task="PROJECT_NAME",input={"EXAMPLE1":"input","EXAMPLE2":"input"})Specific PromptIf you want to generate a completion for a specific prompt, you can do so by callinghoneyhive.generations.generate(response=honeyhive.generations.generate(task="PROJECT_NAME",input={"EXAMPLE1":"input","EXAMPLE2":"input"},prompts=["PROMPT_NAME"])Evaluating a text completion via HoneyHiveFor providing feedback on a generation, you can callhoneyhive.feedback(response=honeyhive.feedback(task="PROJECT_NAME",generation_id=generation_id,feedback={"EXAMPLE1":"feedback","EXAMPLE2":True,})SDK CapabilitiesCompletionshoneyhive.Completions.create- calls OpenAI and logs the generation in HoneyHiveGenerationshoneyhive.generations.generate- generates a completion via HoneyHivehoneyhive.generations.get_generations- gets a pandas dataframe of all generations for a projectFeedbackhoneyhive.feedback- provides feedback on a generationProjectshoneyhive.tasks.get_tasks- gets a list of all projectshoneyhive.tasks.get_task- gets a specific projecthoneyhive.tasks.create_task- creates a new projecthoneyhive.tasks.update_task- updates a projecthoneyhive.tasks.delete_task- deletes a projectPromptshoneyhive.prompts.get_prompts- gets a list of all prompts for a projecthoneyhive.prompts.create_prompt- creates a new prompt for a projecthoneyhive.prompts.update_prompt- updates a prompt for a projecthoneyhive.prompts.delete_prompt- deletes a prompt for a projectMetricshoneyhive.metrics.get_metrics- gets a list of all metrics for a projecthoneyhive.metrics.create_metric- creates a new metric for a projecthoneyhive.metrics.delete_metric- deletes a metric for a projectDatasetshoneyhive.datasets.get_datasets- gets a list of all datasets for a projecthoneyhive.datasets.get_dataset- gets a specific dataset for a projecthoneyhive.datasets.create_dataset- creates a new dataset for a projecthoneyhive.datasets.delete_dataset- deletes a dataset for a project