Package: dewy
Dewy - The Knowledgebase for AIOpinionated knowledge extraction and semantic retrieval for Gen AI applications.Explore the docs »Report Bug·Request FeatureAbout The ProjectDewy helps you build AI agents and RAG applications by managing the extraction of knowledge from your documents and implementing semantic search over the extracted content.
Load your documents and Dewy takes care of parsing, chunking, summarizing, and indexing for retrieval.
Dewy builds on the lessons of putting real Gen AI applications into production so you can focus on getting 💩 done, rather than comparing vector databases and building data extraction infrastructure.Below is the typical architecture of an AI agent performing RAG.
Dewy handles all of the parts shown in brown so you can focus on your application -- the parts in green.(back to top)Getting StartedTo get a local copy up and running follow these steps.(Optional) Start apgvectorinstance to persist your dataDewy uses a vector database to store metadata about the documents you've loaded as well as embeddings used to provide semantic search results.dockerrun-d\-p5432:5432\-ePOSTGRES_DB=dewydb\-ePOSTGRES_USER=dewydbuser\-ePOSTGRES_PASSWORD=dewydbpwd\-ePOSTGRES_HOST_AUTH_METHOD=trust\ankane/pgvectorIf you already have an instance ofpgvectoryou can create a database for Dewy and configure Dewy use it using theDBenv var (see below).Install Dewypip install dewyThis will install Dewy in your local Python environment.Configure Dewy.
Dewy will read env vars from an.envfile if provided. You can also set these directly
in the environment, for example when configuring an instance running in docker / kubernetes.# ~/.envENVIRONMENT=LOCALDB=postgresql://...OPENAI_API_KEY=...Fire up DewydewyDewy includes an admin console you can use to create collections, load documents, and run test queries.openhttp://localhost:8000/adminUsing Dewy in Typescript / JavascriptInstall the API client librarynpminstalldewy-tsConnect to an instance of Dewyimport{Dewy}from'dewy_ts';constdewy=newDewy()Add documentsawaitdewy.kb.addDocument({collection_id:1,url:“https://arxiv.org/abs/2005.11401”,})Retrieve document chunks for LLM promptingconstcontext=awaitdewy.kb.retrieveChunks({collection_id:1,query:"tell me about RAG",n:10,});// Minimal prompt exampleconstprompt=[{role:'system',content:`You are a helpful assistant.You will take into account any CONTEXT BLOCK that is provided in a conversation.START CONTEXT BLOCK${context.results.map((c:any)=>c.chunk.text).join("\n")}END OF CONTEXT BLOCK`,},]// Using OpenAI to generate responsesconstresponse=awaitopenai.chat.completions.create({model:'gpt-3.5-turbo',stream:true,messages:[...prompt,[{role:'user':content:'Tell me about RAG'}]]})Using Dewy in PythonInstall the API client librarypipinstalldewy-clientConnect to an instance of Dewyfromdewy_clientimportClientdewy=Client(base_url="http://localhost:8000")Add documentsfromdewy_client.api.kbimportadd_documentfromdewy_client.modelsimportAddDocumentRequestawaitadd_document.asyncio(client=dewy,body=AddDocumentRequest(collection_id=1,url=“https://arxiv.org/abs/2005.11401”,))Retrieve document chunks for LLM promptingfromdewy_client.api.kbimportretrieve_chunksfromdewy_client.modlesimportRetrieveRequestchunks=awaitretrieve_chunks.asyncio(client=dewy,body=RetrieveRequest(collection_id=1,query="tell me about RAG",n=10,))# Minimal prompt exampleprompt=f"""You will take into account any CONTEXT BLOCK that is provided in a conversation.START CONTEXT BLOCK{"\n".join([chunk.textforchunkinchunks.text_results])}END OF CONTEXT BLOCK"""See [python-langchain.ipynb'](demos/python-langchain-notebook/python-langchain.ipynb) for an example using Dewy in LangChain, including an implementation of LangChain'sBaseRetriever` backed by Dewy.RoadmapDewy is under active development.
This is an overview of our current roadmap - please 👍 issues that are important to you.
Don't see a feature that would make Dewy better for your application -create a feature request!Support more document formats (ieMarkdown,DOCX,HTML)Support more types of chunk extractorsMulti-modal search over images, tables, audio, etc.Integrations with LangChain, LlamaIndex, Haystack, etc.Support flexible result ranking (ie rag-fusion, mmr, etc).Provide metrics around which chunks are used, relevance scores, etc.Query history and explorer in the UI.Multi-tenancyHybrid searchContributingContributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make aregreatly appreciated.If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement".
Don't forget to give the project a star! Thanks again!Fork the ProjectCreate your Feature Branch (git checkout -b feature/AmazingFeature)Commit your Changes (git commit -m 'Add some AmazingFeature')Push to the Branch (git push origin feature/AmazingFeature)Open a Pull RequestDevelopment InstallationClone the repogitclonehttps://github.com/DewyKB/dewy.gitInstall Python packagespoetryinstallConfigure Dewy.
Dewy will read env vars from an.envfile if provided. You can also set these directly
in the environment, for example when configuring an instance running in docker / kubernetes.cat>.env<< EOFENVIRONMENT=LOCALDB=postgresql://...OPENAI_API_KEY=...EOFBuild the frontendcdfrontend&&npminstall&&npmrunbuildBuild the clientcddewy-client&&poetryinstallRun the Dewy servicepoetryrundewyPracticesSome skeleton code based on best practices fromhttps://github.com/zhanymkanov/fastapi-best-practices.The following commands run tests and apply linting.
If you're in apoetry shell, you can omit thepoetry run:Running tests:poetry run pytestLinting (and formatting):poetry run ruff check --fixFormatting:poetry run ruff formatType Checking:poetry run mypy dewyTo regenerate the OpenAPI spec and client libraries:poetrypoeextract-openapi
poetrypoeupdate-client(back to top)ReleasingLook at thedraft releaseto determine the suggested next version.Create a PR updating the following locations to that version:
a.pyproject.tomlfordewyb.pyproject.tomlfordewy-clientc. API version inconfig.pyd.openapi.yamlanddewy-clientby runningpoe extract-openapiandpoe update-client.Once that PR is in, edit the draft release, make sure the version and tag match what you selected in step 1 (and used in the PR), check "Set as a pre-release" (will be updated by the release automation) and choose to publish the release.The release automation should kick in and work through the release steps. It will need approval for the pypi deployment environment to publish thedewyanddewy-clientpackages.(back to top)LicenseDistributed under the Apache 2 License. SeeLICENSE.txtfor more information.(back to top)