Package: knowledge-base-guardian
Welcome to the KnowledgeBase Guardian, an LLM-powered solution tokeep your knowledge base consistent and free of contradictions! How, you ask? Well, every time you want to add new information to your knowledge base, the Guardian will check that it does not conflict with information that is already contained in there. To grasp the general idea, feel free to have a look at this notebook:.Regardless of the purpose of your knowledge base, maintaining consistency is highly desirable. However, in a large and constantly evolving knowledge base, this can prove to be a challenging task.At Dataroots, we're developing an LLM-powered Q&A system, with our internal documents serving as the knowledge base. To us, maintaining consistency brings a range of benefits:Enhanced user trust: With conflicting information eliminated, users can rely on the knowledge base with confidence, leading to a positive user experience.Improved answer quality: By eliminating conflicting information, we can enhance the accuracy and reliability of the generated answers.Simplified maintenance: By automating conflict detection as much as possible, we reduce the manual effort required to maintain the knowledge base.Off course, these benefits are not restricted to LLM-powered Q&A systems. If you're interested to keep your knowledge consistent as well, make sure to keep reading.Keep in mind that this repo currently acts as a Proof of Concept and not as a full-fledged knowledge base management system.💡 How it worksOur use case can be visualized as follows:We have an initial vector store, which contains embeddings of all documents in our knowledge base. To keep the example simple, we assume it is initially free of contradictions.We want to add new documents to this vector store, but are unsure if these documents are consistent with the information in the vector store.Before adding a document, we first retrieve the most semantically similar documents in the vector store. We then use an LLM to compare the documents and search for contradictions:If no contradiction is detected, the document is added to the vector store.If a contradiction is detected, the document is not added and we keep a log of the failed attempt.PrerequisitesOS: Linux or MacOSPython: 3.9 or higherOpenAI accountORAzure OpenAI with deployed embedding and LLM⚙️ SetupOpen a terminal and clone this repository by runninggitclonehttps://github.com/datarootsio/knowledgebase_guardian.gitGo to the cloned folder and create a virtual environment. Choose your favorite one or use venv:python-mvenvcontradiction_detectionsourcecontradiction_detection/bin/activateInstall the dependencies:python-mpipinstall-e.⚡️ QuickstartWe provide a small demo example to get started right away. If you prefer to play around with your own data, you can jump ahead to the next section.Indata/vectorstoreyou'll find an index and vector store file calledbelgium. It was created from three articles about Belgian cities, which we scraped from Wikipedia and which you can find indata/raw. In our example, we'll try to add three new documents to our vector store. You can find these indata/extension.To follow along with the example, follow the setup section above andexecute the following steps:Make sure you have an OpenAI account. Look up your OpenAI API key and write it down in.envIn your terminal, runpythonscripts/extend_vectorstore.pyThis will result in the following threeoutputs:A new index and vector store file calledbelgium_extended, located in thedata/vectorstorefolder.Acontradictions.logfile, indicating for which new files a contradiction was detected. If all went well, you should see that the documentLeuven_contradictionswas not added to the vector store and the output should look more or less like this:Aexecution.logfile providing information about the run. Here you'll also find logs for new documents that were added succesfully. Assuming all went well, you'll see that the documentsLeuven_alignedandLeuven_newwere succesfully added to the vector store. The first document contains only information that is already present in the vector store, while the second introduces new information that is not conflicting with any of the information contained in the vector store. The output should look similar to this:and this:⚒️ Setting up KnowledgeBase Guardian with your own dataMake sure to first execute the steps of the setup section above.Choose OpenAI or Azure OpenAIFor AzureOpenAI, complete the.env.cloudfile.For OpenAI, complete the.envfile.Set theazure_openaivariable inconfig.ymlto true if you use AzureOpenAI, else set it to false.Initializing your vector storeA) You already have a FAISS vector storePlace the index file and the actual vector store file in thedata/vectorstorefolder. Make sure that:Both files have the same nameThe index file has extension.indexThe vector store file has extension.pklNow head over toconfig.ymland change thevectorstore_nameparameter to the name of your vector store.B) You don't have a FAISS vector storePlace all your.txtdata in thedata/rawfolder.Head over toconfig.ymland change thevectorstore_nameparameter to the desired name for your vector store.Optional: change thechunk_sizeandchunk_overlapparametersCreate a vector store and index file with the chosen name in thedata/vectorstorefolder by runningpythonscripts/create_vectorstore.pyExtending your vector store and detecting contradictionsNow we want to add new documents to the vector store, but only if they are not contradicting with the information that is already contained in the vector store.Place the.txtfiles to be added in thedata/extensionfolder.Optional: change thechunk_size,chunk_overlap,nb_retrieval_docs,system_messageanduser_messageparameters in theconfig.ymlfile.Start the contradiction detection and vector store extension with the following command. To bypass the contradiction detection mechanism, add--disable-contradiction-detection.pythonscripts/extend_vectorstore.pyThe output is threefold:A new index and vector store file in thedata/vectorstorefolder, recognizable by the presence of_extendedin their name.Acontradictions.logfile, indicating for which new files a contradiction was detected. For debugging purposes, it also displays the output of the LLM and the content of the most similar documents that were retrieved.Aexecution.logfile indicating information about the run. Here you'll also find logs for new documents that were added succesfully.🧐 LimitationsThe performance of this technique is highly dependent on the prompt. You will likely need to fine-tune the prompt (i.e., thesystem_messageanduser_messageinconfig.yml) to your use-caseThere is no consistent handling of all chunks in a document. This means that if your document is split into multiple chunks and some of them contain contradictions while others don't, some chunks will be added to the vector store and others will not. Depending on your use case, you might want to change this behaviour.To keep the example as small as possible, we chose to support onlyone vector store type (FAISS)one file extension (.txt)Extending this code to other vector stores and file extensions is possible by leveraging Langchain or LlamaIndex.LicenseThis project is licensed under the terms of the MIT license.