Package: prompttools
PromptTools:wrench: Test and experiment with prompts, LLMs, and vector databases. :hammer:Welcome toprompttoolscreated byHegel AI! This repo offers a set of open-source, self-hostable tools for experimenting with, testing, and evaluating LLMs, vector databases, and prompts. The core idea is to enable developers to evaluate using familiar interfaces likecode,notebooks, and a localplayground.In just a few lines of codes, you can test your prompts and parameters across different models (whether you are using
OpenAI, Anthropic, or LLaMA models). You can even evaluate the retrieval accuracy of vector databases.fromprompttools.experimentimportOpenAIChatExperimentmessages=[[{"role":"user","content":"Tell me a joke."},],[{"role":"user","content":"Is 17077 a prime number?"},],]models=["gpt-3.5-turbo","gpt-4"]temperatures=[0.0]openai_experiment=OpenAIChatExperiment(models,messages,temperature=temperatures)openai_experiment.run()openai_experiment.visualize()To stay in touch with us about issues and future updates, join theDiscord.QuickstartTo installprompttools, you can usepip:pip install prompttoolsYou can run a simple example of aprompttoolslocally with the followinggit clone https://github.com/hegelai/prompttools.git
cd prompttools && jupyter notebook examples/notebooks/OpenAIChatExperiment.ipynbYou can also run the notebook inGoogle ColabPlaygroundIf you want to interact withprompttoolsusing our playground interface, you can launch it with the following commands.First, install prompttools:pip install prompttoolsThen, clone the git repo and launch the streamlit app:git clone https://github.com/hegelai/prompttools.git
cd prompttools && streamlit run prompttools/playground/playground.pyYou can also access a hosted version of the playground on theStreamlit Community Cloud.Note: The hosted version does not support LlamaCppDocumentationOurdocumentation websitecontains the full API reference
and more description of individual components. Check it out!Supported IntegrationsHere is a list of APIs that we support with our experiments:LLMsOpenAI (Completion, ChatCompletion, Fine-tuned models) -SupportedLLaMA.Cpp (LLaMA 1, LLaMA 2) -SupportedHuggingFace (Hub API, Inference Endpoints) -SupportedAnthropic -SupportedGoogle PaLM -SupportedGoogle Vertex AI -SupportedAzure OpenAI Service -SupportedReplicate -SupportedOllama -In ProgressVector Databases and Data UtilityChroma -SupportedWeaviate -SupportedQdrant -SupportedLanceDB -SupportedMilvus - ExploratoryPinecone -SupportedEpsilla -In ProgressFrameworksLangChain -SupportedMindsDB -SupportedLlamaIndex - ExploratoryComputer VisionStable Diffusion -SupportedReplicate's hosted Stable Diffusion -SupportedIf you have any API that you'd like to see being supported soon, please open an issue or
a PR to add it. Feel free to discuss in our Discord channel as well.Frequently Asked Questions (FAQs)Will this library forward my LLM calls to a server before sending it to OpenAI, Anthropic, and etc.?No, the source code will be executed on your machine. Any call to LLM APIs will be directly executed from your machine without any forwarding.Doesprompttoolsstore my API keys or LLM inputs and outputs to a server?No, all of those data stay on your local machine. We do not collect any PII (personally identifiable information).How do I persist my results?To persist the results of your tests and experiments, you can export yourExperimentwith the methodsto_csv,to_json,to_lora_json, orto_mongo_db. We are building more persistence features and we will be happy to further discuss your use cases, pain points, and what export
options may be useful for you.SentryUsage TrackingSince we are changing our API rapidly, there are some errors caused by our negligence or out of date documentation.
To improve user experience, we collect data from normal package usage that helps us understand the
errors that are raised. This data is collected and sent toSentry,
a third-party error tracking service, commonly used in open-source softwares. It only logs this library's own actions.You can easily opt-out by defining an environment variable calledSENTRY_OPT_OUT.ContributingWe welcome PRs and suggestions! Don't hesitate to open a PR/issue or to reach out to usvia email.
Please have a look at ourcontribution guideand"Help Wanted" issuesto get started!Usage and FeedbackWe will be delighted to work with early adopters to shape our designs. Please reach out to usvia emailif you're
interested in using this tooling for your project or have any feedback.LicenseWe will be gradually releasing more components to the open-source community. The current license can be found in theLICENSEfile. If there is any concern, pleasecontact usand we will be happy to work with you.