Package: quke
qukeCompare the answering capabilities of different LLMs - for example LlaMa, ChatGPT, Cohere, Falcon - against user provided document(s) and questions.Explore the docs »Report Bug·Request FeatureTable of ContentsAbout The ProjectBuilt WithGetting StartedPrerequisitesInstallationUsageBaseSpecify models and embeddingsExperimentsConfigurationSearch your own documentsLimitationsPrivacyLicenseContactAbout The ProjectCompare the answering capabilities of different LLMs - for example LlaMa, ChatGPT, Cohere, Falcon - against user provided document(s) and questions.Specify the different models, embedding tools and vector databases in configuration files.Maintain reproducable experiments reflecting combinations of these configurations.(back to top)Getting StartedPrerequisitesPoetryThe instructions assume a Python environment withPoetryinstalled. Development of the tool is done in Python 3.11. While Poetry is not actually needed for the tool to function, the examples assume Poetry is installed.API keysThe tool uses 3rd party hosted inference APIs. API keys need to be specified as environment variables.The services used:HuggingFaceOpenAICohereReplicateThe API keys can be specied in a.env file. Use the provided .env.example file as an example (enter your own API keys and rename it to '.env').At present, all services used in the example configuration have free tiers available.(back to top)InstallationNavigate to the directory that contains the pyproject.toml file, then execute thepoetryinstallcommand.(back to top)UsageFor the examples the project comes with a public financial document for a Canadian Bank (CIBC) as source pdf file.BaseIn order to run the first example, ensure to specify your HuggingFace API key.Use the commandpoetryrunquketo ask the default questions, using the default embedding and the default LLM.The answers provided by the LLM - in addition to various other logging messages - are saved in the ./output/ or ./multirun directories in separate date and time subdirectories, including in a file calledchat_session.md.The defaults are specified in the config.yaml file (in the ./quke/conf/ directory).(back to top)Specify models and embeddingsEnsure to specify your Cohere API key before running.As per the configuration files, the default LLM is Falcon and the default embedding uses HuggingFace embedding.To specify a different LLM - Cohere in this example - run the following:poetryrunqukeembedding=huggingfacellm=coherequestion=eps(back to top)ExperimentsEnsure to specify your OpenAI API key before running.The LLMs, embeddings, questions and other configurations can be captured in experiment config files. The commandpoetryrunquke+experiment=openaiuses an experiment file openai.yaml (see folder ./config/experiments) which specifies the LLM, embedding and questions to be used. It is equivalent to running:poetryrunqukeembedding=openaillm=gpt3-5question=epsMultiple experiments can be run at once as follows:Ensure to specify your Replicate API key before running.poetryrunquke--multirun+experiment=openai,llama2(back to top)ConfigurationLLMs, embeddings, questions, experiments and other items are specified in a set of configuration files. These are saved in the ./config directory.TheHydrapackage is used for configuration management. Their website explains more about the configuration system in general.Four different models are specified (ChatGPT, LlaMa2, Falcon, and Cohere); using 4 different APIs (OpenAI, HuggingFace, Cohere and Replicate).Additional LLMs (or embeddings, questions) can be set up by adding new configuration files.(back to top)Search your own documentsThe documents to be searched are stored in the ./docs/pdf directory. At present only pdf documents are considered.
Note to setvectorstore_write_modetoappendoroverwritein the embedding configuration file (or delete the folder with the existing vector database, in the ./idata folder).(back to top)LimitationsThe free tiers for the third party services generally come with fairly strict limitations. They differ between services; and may differ over time.To try out the tool with your own documents it is best to start with a single small source document, no more than two questions and only one combination of LLM/embedding.Error messages due to limitations of the APIs are not always clearly indicated as such.(back to top)PrivacyThe tool uses third party APIs (OpenAI, HuggingFace, Cohere, Replicate). These process your source documents and your questions, to the extent that you provide these. They track your usage of their APIs. They may do other things; I do not know.The tool uses theLangChainPython package to interact with the third party services. I do not know if the package 'leaks' any of the data in any way.In general I do not know to what extent any of the data is encrypted during transmission.The tool shares no information with me.(back to top)Built With(back to top)LicenseDistributed under the MIT License. SeeLICENSE.txtfor more information.(back to top)ContactProject Link:https://github.com/ejoosterop/quke(back to top)