Package: safe-store
safe_storesafe_storeis an open-source Python library that provides essential tools for text data management, vectorization, and document retrieval. It empowers users to work with text documents efficiently and effortlessly.safe_storeKey Features:1. Text Vectorizer2. Generic Data LoaderWhat Can You Usesafe_storeFor?Text VectorizerFeaturesInstallationGetting StartedInitializing the Text VectorizerAdding and Indexing DocumentsEmbedding a Query and Retrieving Similar DocumentsGeneric Data LoaderFeaturesUsageSupported File TypesAuthorLicenseKey Features:1. Text VectorizerVersatile Vectorization:Choose between TF-IDF vectorization, model-based embeddings to convert text documents into numerical representations or use BM25 ranking for text retreival.Document Similarity:Find documents similar to a given query text, making it ideal for document retrieval tasks.Interactive Visualization:Visualize document embeddings in a scatter plot to gain insights into document relationships.No Authentication Required:Use the library without the need for API keys or authentication, making it accessible for everyone.Commercially Usable:safe_storeis 100% open-source and free to use, even for commercial purposes, under the Apache 2.0 License.2. Generic Data LoaderMulti-format Support:Read various file formats, including PDF, DOCX, JSON, HTML, and more.Simplified Text Extraction:Convert file content to plain text or data structures with ease.Efficient and Time-Saving:Streamline data loading and processing tasks, reducing the need for manual extraction.What Can You Usesafe_storeFor?Text Document Analysis:Analyze and understand the content of text documents quickly and efficiently.Document Retrieval:Retrieve documents similar to a given query text, facilitating content recommendation and search tasks.Text Data Preprocessing:Prepare text data for natural language processing (NLP) tasks, such as sentiment analysis and text classification.Data Loading:Streamline the process of reading and extracting content from various file formats.safe_storeis designed to be accessible, versatile, and free for all users. It's an ideal choice for developers, data scientists, and researchers who want a user-friendly and open-source solution for working with text data.Explore the world of text data management and analysis withsafe_storetoday!Text VectorizerFeaturesVectorize and index text documents.Retrieve similar documents based on a query.Supports both TF-IDF vectorization and model-based embeddings.Interactive visualization of document embeddings.No authentication or API keys required.InstallationTo installsafe_store, you can usepip:pipinstallsafe_storeGetting StartedInitializing the Text Vectorizerfromsafe_storeimportTextVectorizer,VectorizationMethodfrompathlibimportPath# Create an instance of TextVectorizervectorizer=TextVectorizer(vectorization_method=VectorizationMethod.TFIDF_VECTORIZER,database_path="database.json",save_db=False)Adding and Indexing Documents# Add documents for vectorizationdocuments=["llm","space","submarines","new york"]fordocindocuments:document_name=Path(__file__).parent/f"{doc}.txt"withopen(document_name,'r',encoding='utf-8')asfile:text=file.read()vectorizer.add_document(document_name,text,chunk_size=100,overlap_size=20,force_vectorize=False,add_as_a_bloc=False)# Index the documents (perform vectorization)vectorizer.index()Embedding a Query and Retrieving Similar Documents# Embed a query and retrieve similar documentsquery_text="what is space"similar_texts,_=vectorizer.recover_text(query_text,top_k=3)# Show the interactive document visualizationvectorizer.show_document(show_interactive_form=True)print("Similar Documents:")fori,textinenumerate(similar_texts):print(f"{i+1}:{text}")Thevectorizer.show_document(show_interactive_form=True)should yield a plot like this where you can read the text by pointing on the dots. Each dot is a chunk of the text. We can clearly see that chunks that come from the same document tend to form a cluster.Generic Data LoaderFeaturesRead various file formats including PDF, DOCX, JSON, HTML, and more.Convert file content to text or data structures.UsageTo read a file usingGenericDataLoader, you can use theread_filemethod and provide the file path:fromsafe_storeimportGenericDataLoaderfrompathlibimportPathfile_path=Path("example.pdf")file_content=GenericDataLoader.read_file(file_path)Supported File TypesPDFDOCXJSONHTMLPPTXTXTRTFMDLOGCPPJavaJSPythonRubyShell ScriptSQLCSSPHPXMLYAMLINIINFMAPBATFeel free to replace"example.pdf"with the path to your specific file.AuthorParisNeoLicenseThis project is licensed under the Apache 2.0 License.