Package: semantic-ai
Semantic AI LibAn open-source framework for Retrieval-Augmented System (RAG) uses semantic search to retrieve the expected results and generate human-readable conversational responses with the help of LLM (Large Language Model).Semantic AI Library DocumentationDocs hereRequirementsPython 3.10+ asyncioInstallation# Using pip$python-mpipinstallsemantic-ai# Manual install$python-mpipinstall.Set the environment variableSet the credentials in .env file. Only give the credential for an one connector, an one indexer and an one llm model config. other fields put as empty# DefaultFILE_DOWNLOAD_DIR_PATH=# default directory name 'download_file_dir'EXTRACTED_DIR_PATH=# default directory name 'extracted_dir'# Connector (SharePoint, S3, GCP Bucket, GDrive, Confluence etc.,)CONNECTOR_TYPE="connector_name"# sharepointSHAREPOINT_CLIENT_ID="client_id"SHAREPOINT_CLIENT_SECRET="client_secret"SHAREPOINT_TENANT_ID="tenant_id"SHAREPOINT_HOST_NAME='<tenant_name>.sharepoint.com'SHAREPOINT_SCOPE='https://graph.microsoft.com/.default'SHAREPOINT_SITE_ID="site_id"SHAREPOINT_DRIVE_ID="drive_id"SHAREPOINT_FOLDER_URL="folder_url"# /My_folder/child_folder/# IndexerINDEXER_TYPE="vector_db_name"# elasticsearch, qdrantELASTICSEARCH_URL="elasticsearch_url"# give valid urlELASTICSEARCH_USER="elasticsearch_user"# give valid userELASTICSEARCH_PASSWORD="elasticsearch_password"# give valid passwordELASTICSEARCH_INDEX_NAME="index_name"ELASTICSEARCH_SSL_VERIFY="ssl_verify"# True or False# QdrantQDRANT_URL="<qdrant_url>"QDRANT_INDEX_NAME="<index_name>"QDRANT_API_KEY="<apikey>"# LLMLLM_MODEL="<llm_model>"# llama, openaiLLM_MODEL_NAME_OR_PATH=""# model nameOPENAI_API_KEY="<openai_api_key>"# if using openai# SQLSQLITE_SQL_PATH="<database_path>"# sqlit db path# MYSQLMYSQL_HOST="<host_name>"# localhost or Ip AddressMYSQL_USER="<user_name>"MYSQL_PASSWORD="<password>"MYSQL_DATABASE="<database_name>"MYSQL_PORT="<port>"# default port is 3306Method 1:
To load the .env file. Env file should have the credentials%load_extdotenv
%dotenv
%dotenvrelative/or/absolute/path/to/.env(or)dotenv-f.envrun--pythonMethod 2:fromsemantic_ai.configimportSettingssettings=Settings()Un-Structure1. Import the moduleimportasyncioimportsemantic_ai2. To download the files from a given source, extract the content from the downloaded files and index the extracted data in the given vector db.awaitsemantic_ai.download()awaitsemantic_ai.extract()awaitsemantic_ai.index()After completion of download, extract and index, we can generate the answer from indexed vector db. That code given below.3. To generate the answer from indexed vector db using retrieval LLM model.search_obj=awaitsemantic_ai.search()query=""search=awaitsearch_obj.generate(query)Suppose the job is running for a long time, we can watch the number of files processed, the number of files failed, and that filename stored in the text file that is processed and failed in the 'EXTRACTED_DIR_PATH/meta' directory.ExampleTo connect the source and get the connection object. We can see that in the examples folder.
Example: SharePoint connectorfromsemantic_ai.connectorsimportSharepointCLIENT_ID='<client_id>'# sharepoint client idCLIENT_SECRET='<client_secret>'# sharepoint client seceretTENANT_ID='<tenant_id>'# sharepoint tenant idSCOPE='https://graph.microsoft.com/.default'# scopeHOST_NAME="<tenant_name>.sharepoint.com"# for example 'contoso.sharepoint.com'# Sharepoint object creationconnection=Sharepoint(client_id=CLIENT_ID,client_secret=CLIENT_SECRET,tenant_id=TENANT_ID,host_name=HOST_NAME,scope=SCOPE)Structure1. Import the moduleimportasyncioimportsemantic_ai2. The database connectionSqlite:from semantic_ai.connectors import Sqlite

file_path= <database_file_path>

sql = Sqlite(sql_path=file_path)Mysql:from semantic_ai.connectors import Mysql

sql = Mysql(
    host=<host_name>,
    user=<user_name>,
    password=<password>,
    database=<database>,
    port=<port_number>  # 3306 is default port
)3. To generate the answer from db using retrieval LLM model.query = ""
search_obj = await semantic_ai.db_search(query=query)Run in the server$semantic_aiserve-f.env

INFO:Loadingenvironmentfrom'.env'INFO:Startedserverprocess[43973]INFO:Waitingforapplicationstartup.
INFO:Applicationstartupcomplete.
INFO:Uvicornrunningonhttp://127.0.0.1:8000(PressCTRL+Ctoquit)Open your browser athttp://127.0.0.1:8000/semantic-aiInteractive API docsNow go tohttp://127.0.0.1:8000/docs.
You will see the automatic interactive API documentation (provided by Swagger UI):