Package: knowledgegpt
knowledgegptknowledgegptis designed to gather information from various sources, including the internet and local data, which
can be used to create prompts. These prompts can then be utilized by OpenAI's GPT-3 model to generate answers that are
subsequently stored in a database for future reference.To accomplish this, the text is first transformed into a fixed-size vector using either open source or OpenAI models.
When a query is submitted, the text is also transformed into a vector and compared to the stored knowledge embeddings.
The most relevant information is then selected and used to generate a prompt context.knowledgegptsupports various information sources including websites, PDFs, PowerPoint files (PPTX), and
documents (Docs). Additionally, it can extract text from YouTube subtitles and audio (using speech-to-text technology)
and use it as a source of information. This allows for a diverse range of information to be gathered and used for
generating prompts and answers.InstallationPyPI installation, run in terminal:pip install knowledgegptOr you can use the latest version from the repository:pip install -r requirements.txtand thenpip install .Download needed language model for parsing:python3 -m spacy download en_core_web_smSet Your API KeyGo toOpenAI > Account > Api KeysCreate new secret key and copyEnter the key toexample_config.pyHow to use the library# Import the libraryfromknowledgegpt.extractors.web_scrape_extractorimportWebScrapeExtractor# Import OpenAI and Set the API Keyimportopenaifromexample_configimportSECRET_KEYopenai.api_key=SECRET_KEY# Define target websiteurl="https://en.wikipedia.org/wiki/Bombard_(weapon)"# Initialize the WebScrapeExtractorscrape_website=WebScrapeExtractor(url=url,embedding_extractor="hf",model_lang="en")# Prompt the OpenAI Modelanswer,prompt,messages=scrape_website.extract(query="What is a bombard?",max_tokens=300,to_save=True,mongo_client=db)# See the answerprint(answer)# Output: 'A bombard is a type of large cannon used during the 14th to 15th centuries.'Other examples can be found in theexamplesfolder.
But to give a better idea of how to use the library, here is a simple example:# Basic Usagebasic_extractor=BaseExtractor(df)answer,prompt,messages=basic_extractor.extract("What is the title of this PDF?",max_tokens=300)# PDF Extractionpdf_extractor=PDFExtractor(pdf_file_path,extraction_type="page",embedding_extractor="hf",model_lang="en")answer,prompt,messages=pdf_extractor.extract(query,max_tokens=1500)# PPTX Extractionppt_extractor=PowerpointExtractor(file_path=ppt_file_path,embedding_extractor="hf",model_lang="en")answer,prompt,messages=ppt_extractor.extract(query,max_tokens=500)# DOCX Extractiondocs_extractor=DocsExtractor(file_path="../example.docx",embedding_extractor="hf",model_lang="en",is_turbo=False)answer,prompt,messages=\docs_extractor.extract(query="What is an object detection system?",max_tokens=300)# Extraction from Youtube video (audio)scrape_yt_audio=YoutubeAudioExtractor(video_id=url,model_lang='tr',embedding_extractor='hf')answer,prompt,messages=scrape_yt_audio.extract(query=query,max_tokens=1200)# Extraction from Youtube video (transcript)scrape_yt_subs=YTSubsExtractor(video_id=url,embedding_extractor='hf',model_lang='en')answer,prompt,messages=scrape_yt_subs.extract(query=query,max_tokens=1200)Docker Usagedockerbuild-tknowledgegptimage.
dockerrun-p8888:8888knowledgegptimage