Package: map2gpt
MAP2GPTdescriptionThis project is a versatile and powerful search tool that leverages state-of-the-art natural language processing models to provide relevant and contextually rich results. The primary goal of this project is to build a semantic search engine for textual content from various sources such as PDF files and Wikipedia pages.The project utilizes the GPT-3.5-turbo model for generating responses and French Semantic model to create embeddings of textual data. Users can build an index of embeddings from a PDF file or a Wikipedia page, explore the index interactively, and deploy the search functionality on Telegram. The search results are presented as the top k relevant chunks of information, which are then used as context to generate an informative response from the GPT-3.5-turbo model.The project is implemented in Python, and it employs several open-source libraries such as Click, OpenAI, Wikipedia, PyTorch, Tiktoken, and Rich. The code is organized into modular functions and classes, making it easy to understand, maintain, and extend. The main script provides a command-line interface for users to interact with the project's functionalities.Table of ContentsInstallationUsageBuild Index from PDFBuild Index from WikipediaExplore IndexInstallationTo install the necessary dependencies, run the following command:python-mvenvenvsourceenv/bin/activate
pipinstall--upgradepippipinstallmap2gptSupported Transformer ModelsThis project supports a variety of transformer models, including models from the Hugging Face Model Hub and sentence-transformers. Below are some examples:
- Hugging Face Model: 'Sahajtomar/french_semantic'
- Sentence-Transformers Model: 'paraphrase-MiniLM-L6-v2', 'all-mpnet-base-v2', etc...Please ensure that the model you choose is compatible with the project requirements and adjust the--transformer_model_nameoption accordingly.CLI usageset env varsexportOPENAI_API_KEY=sk-TRANSFORMERS_CACHE=/path/to/cacheQDRANT_PERSISTENT_FOLDER=/path/to_persistentBuild Index from PDF filesTo build an index from a PDF file, run the following command:python-mmap2gpt.main--transformer_model_name'Sahajtomar/french_semantic'build-index-from-pdf-files--path2pdf_files/path/to/file-000.pdf\--path2pdf_files/path/to/file-001.pdf\--nameqdrant_collection_name\--chunk_size256\--batch_size128Build Index from Wikipedia pagesTo build an index from a Wikipedia page, run the following command:python-mmap2gpt.main--transformer_model_name'Sahajtomar/french_semantic'build-index-from-wikipedia-pages--urlshttps://...wikipedia\--urlshttps://...wikipedia\--nameqdrant_collection_name\--chunk_size256\--batch_size128Build Index from Youtube linksTo build an index from a Wikipedia page, run the following command:python-mmap2gpt.main--transformer_model_name'Sahajtomar/french_semantic'build-index-from-youtube-links--urlshttps://...youtube\--urlshttps://...youtube\--nameqdrant_collection_name\--chunk_size256\--batch_size128Build Index from textsTo build an index from a Wikipedia page, run the following command:python-mmap2gpt.main--transformer_model_name'Sahajtomar/french_semantic'build-index-from-wikipedia-pages--path2directory/path/to/corpus_text_files--nameqdrant_collection_name\--chunk_size256\--batch_size128Explore IndexTo explore the index, run the following command:query the indexpython-mmap2gpt.main--transformer_model_name'Sahajtomar/french_semantic'query-index--query"...."\--nameqdrant_collection_name\--top_k7--source_k3--description"service description"deploy on telegrampython-mmap2gpt.main--transformer_model_name'Sahajtomar/french_semantic'deploy-on-telegram--telegram_tokenXXXXXXXXX...XXXXXXXXXXX\--nameqdrant_collection_name\--top_k7--source_k3--description"service description"Module usage# create qdrant clientqdrant=QdrantClient(':memory:')# use path for persistence QdrantClient(path=path2persistent_dir)# initialize runnerrunner=GPTRunner(device='cuda:0',# cpuqdrant=qdrant,tokenizer='gpt-3.5-turbo',openai_api_key='sk-XXXXXXXXXXXXXXXXXXXXX',transformers_cache='/path/to/transformers_cache',transformer_model_name='Sahajtomar/french_semantic'# use all-mpnet-case-v2 for english)# build index from wikipedia pagesknowledge_base=runner.build_index_from_pdf_files(path2pdf_files=['https://www.youtube.com/watch?v=tH-i_FeagJc','https://www.youtube.com/watch?v=tH-i_FeagJc',],chunk_size=256,batch_size=128,name='collection_name',)# create qdrant indexrunner.create_qdrant_index(knowledge_base=knowledge_base)# deploy on telegramdeploy_on_telegram(telegram_token='XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX',runner=runner,name='collection_name',description="service name description",top_k=10,source_k=3)