Package: chatdbt
chatdbtWhat is this?chatdbt is an openai-baseddbtdocumentation robot. You can use natural language to describe your data query requirements to the robot, and chatdbt will help you select the dbt model you need, or generate sql responses based on these dbt models to meet your needs. Of course, you need to set up yourdbt documentationfor chatdbt in advance.Quick Installpip install chatdbtpackage extras:nomic: use nomic/atlas as vector storage backendpgvector: use pgvector as vector storage backendInternalsChatdbt uses openai'stext-embedding-ada-002model interface to embed your dbt documentation and save the vectors to the vector storage you provide. When you make an inquiry to chatdbt, it retrieves themodelsandmetrics(todo😊) that are semantically similar to your question. Based on the returned content and your question, it uses openaigpt-3.5-turbomodel to provide appropriate answers. Similar tolangchainorllama_index.How does chatdbt integrate with my dbt doc, and where is my embedding data stored?There are several interfaces within chatdbt:VectorStorageis responsible for storing embedding vectors. Currently supporting:atlasSet up yourapi_keyandproject_nameto use Nomic Atlas for storing and retrieving the vector data.pgvectorSet up yourconnect_stringandtable_nameto use pgvector for storing and retrieving the vector data.DBTDocResolveris responsible for providing dbt manifest and catalog data. Currently supporting:localfsSet upmanifest_json_pathandmanifest_json_path, and chatdbt will read the dbt manifest and catalog from the local file system.TikTokenProvideris responsible for estimating the number of tokens consumed by OpenAI. Currently supporting:tiktoken_http_serverSet up atiktoken-http-serverapi_base(example:http://localhost:8080) to use tiktoken-http-server for estimating the number of tokens consumed by OpenAI.You can also implement the above interfaces yourself and integrate them into your own system.Quick StartYou can initialize a chatdbt instance manually:your_pgvector_connect_string="postgresql+psycopg://postgres:foobar@localhost:5432/chatdbt"your_pgvector_table_name="chatdbt"your_manifest_json_path="data/manifest.json"your_catalog_json_path="data/catalog.json"your_openai_key="sk-foobar"importosos.environ["OPENAI_API_KEY"]=your_openai_keyfromchatdbtimportChatBotfromchatdbt.vector_storage.pgvectorimportPGVectorStoragefromchatdbt.dbt_doc_resolver.localfsimportLocalfsDBTDocResolvervector_storage=PGVectorStorage(connect_string=your_pgvector_connect_string,table_name=your_pgvector_table_name)dbt_doc_resolver=LocalfsDBTDocResolver(manifest_json_path=your_manifest_json_path,catalog_json_path=your_catalog_json_path)bot=ChatBot(doc_resolver=dbt_doc_resolver,vector_storage=vector_storage,tiktoken_provider=None)bot.suggest_table("query the number of users who have purchased a product")bot.suggest_sql("query the number of users who have purchased a product")or initialize a chatdbt instance with environment variables:importosos.environ["CHATDBT_I18N"]="zh-cn"os.environ["CHATDBT_VECTOR_STORAGE_TYPE"]="pgvector"os.environ["CHATDBT_VECTOR_STORAGE_CONFIG_CONNECT_STRING"]=your_pgvector_connect_stringos.environ["CHATDBT_VECTOR_STORAGE_CONFIG_TABLE_NAME"]=your_pgvector_table_nameos.environ["CHATDBT_DBT_DOC_RESOLVER_TYPE"]="localfs"os.environ["CHATDBT_DBT_DOC_RESOLVER_CONFIG_MANIFEST_JSON_PATH"]=your_manifest_json_pathos.environ["CHATDBT_DBT_DOC_RESOLVER_CONFIG_CATALOG_JSON_PATH"]=your_catalog_json_pathos.environ["OPENAI_API_KEY"]=your_openai_keyimportchatdbtchatdbt.suggest_table("query the number of users who have purchased a product")chatdbt.suggest_sql("query the number of users who have purchased a product")