Package: pgml
Open Source Alternative for Building End-to-End Vector Search Applications without OpenAI & PineconeTable of ContentsOverviewQuickstartUpgradingDeveloper setupRoadmapDocumentationExamplesOverviewPython SDK is designed to facilitate the development of scalable vector search applications on PostgreSQL databases. With this SDK, you can seamlessly manage various database tables related to documents, text chunks, text splitters, LLM (Language Model) models, and embeddings. By leveraging the SDK's capabilities, you can efficiently index LLM embeddings using PgVector for fast and accurate queries.Documentation:PostgresML SDK DocsExamples Folder:ExamplesKey FeaturesAutomated Database Management: With the SDK, you can easily handle the management of database tables related to documents, text chunks, text splitters, LLM models, and embeddings. This automated management system simplifies the process of setting up and maintaining your vector search application's data structure.Embedding Generation from Open Source Models: The Python SDK provides the ability to generate embeddings using hundreds of open source models. These models, trained on vast amounts of data, capture the semantic meaning of text and enable powerful analysis and search capabilities.Flexible and Scalable Vector Search: The Python SDK empowers you to build flexible and scalable vector search applications. The Python SDK seamlessly integrates with PgVector, a PostgreSQL extension specifically designed for handling vector-based indexing and querying. By leveraging these indices, you can perform advanced searches, rank results by relevance, and retrieve accurate and meaningful information from your database.Use CasesEmbeddings, the core concept of the Python SDK, find applications in various scenarios, including:Search: Embeddings are commonly used for search functionalities, where results are ranked by relevance to a query string. By comparing the embeddings of query strings and documents, you can retrieve search results in order of their similarity or relevance.Clustering: With embeddings, you can group text strings by similarity, enabling clustering of related data. By measuring the similarity between embeddings, you can identify clusters or groups of text strings that share common characteristics.Recommendations: Embeddings play a crucial role in recommendation systems. By identifying items with related text strings based on their embeddings, you can provide personalized recommendations to users.Anomaly Detection: Anomaly detection involves identifying outliers or anomalies that have little relatedness to the rest of the data. Embeddings can aid in this process by quantifying the similarity between text strings and flagging outliers.Classification: Embeddings are utilized in classification tasks, where text strings are classified based on their most similar label. By comparing the embeddings of text strings and labels, you can classify new text strings into predefined categories.How the Python SDK WorksThe Python SDK streamlines the development of vector search applications by abstracting away the complexities of database management and indexing. Here's an overview of how the SDK works:Automatic Document and Text Chunk Management: The SDK provides a convenient interface to manage documents and pipelines, automatically handling chunking and embedding for you. You can easily organize and structure your text data within the PostgreSQL database.Open Source Model Integration: With the SDK, you can seamlessly incorporate a wide range of open source models to generate high-quality embeddings. These models capture the semantic meaning of text and enable powerful analysis and search capabilities.Embedding Indexing: The Python SDK utilizes the PgVector extension to efficiently index the embeddings generated by the open source models. This indexing process optimizes search performance and allows for fast and accurate retrieval of relevant results.Querying and Search: Once the embeddings are indexed, you can perform vector-based searches on the documents and text chunks stored in the PostgreSQL database. The SDK provides intuitive methods for executing queries and retrieving search results.QuickstartFollow the steps below to quickly get started with the Python SDK for building scalable vector search applications on PostgresML databases.PrerequisitesBefore you begin, make sure you have the following:PostgresML Database: Ensure you have a PostgresML database version >=2.7.7You can spin up a database usingDockerorsign up for a free GPU-powered database.Set theDATABASE_URLenvironment variable to the connection string of your PostgresML database.Python version >=3.8.1InstallationTo install the Python SDK, use pip:pip install pgmlSample CodeOnce you have the Python SDK installed, you can use the following sample code as a starting point for your vector search application:frompgmlimportCollection,Model,Splitter,Pipelinefromdatasetsimportload_datasetfromtimeimporttimefromdotenvimportload_dotenvfromrich.consoleimportConsoleimportasyncioasyncdefmain():load_dotenv()console=Console()# Initialize collectioncollection=Collection("quora_collection")Explanation:The code imports the necessary modules and packages, including pgml, datasets, time, and rich.It creates an instance of the Collection class which we will add pipelines and documents ontoContinuing withinasync def main():# Create a pipeline using the default model and splittermodel=Model()splitter=Splitter()pipeline=Pipeline("quorav1",model,splitter)awaitcollection.add_pipeline(pipeline)ExplanationThe code creates an instance ofModelandSplitterusing their default arguments.Finally, the code constructs a pipeline called"quroav1"and add it to the collection we Initialized above. This pipeline automatically generates chunks and embeddings for every upserted document.Continuing withasync def main():# Prep documents for upsertingdata=load_dataset("squad",split="train")data=data.to_pandas()data=data.drop_duplicates(subset=["context"])documents=[{"id":r["id"],"text":r["context"],"title":r["title"]}forrindata.to_dict(orient="records")]# Upsert documentsawaitcollection.upsert_documents(documents[:200])ExplanationThe code loads the "squad" dataset, converts it to a pandas DataFrame, and drops any duplicate context values.It creates a list of dictionaries representing the documents to be indexed, with each dictionary containing the document's id, text, and title.Finally, they are upserted. As mentioned above, the pipeline added earlier automatically runs and generates chunks and embeddings for each document.Continuing withinasync def main():# Queryquery="Who won 20 grammy awards?"results=awaitcollection.query().vector_recall(query,pipeline).limit(5).fetch_all()console.print(results)# Archive collectionawaitcollection.archive()Explanation:Thequerymethod is called to perform a vector-based search on the collection. The query string isWho won more than 20 grammy awards?, and the top 5 results are requested.The search results are printed.Finally, thearchivemethod is called to archive the collection and free up resources in the PostgresML database.Callmainfunction in an async loop.asyncio.run(main())Running the CodeOpen a terminal or command prompt and navigate to the directory where the file is saved.Execute the following command:python vector_search.pyYou should see the search results printed in the terminal. As you can see, our vector search engine found the right text chunk with the answer we are looking for.[(0.8423336495860181,'Beyoncé has won 20 Grammy Awards, both as a solo artist and member of Destiny\'s Child, making her the second most honored female artist by the Grammys, behind Alison Krauss and the most nominated woman in Grammy Award history with 52 nominations. "Single Ladies (Put a Ring on It)" won Song of the Year in 2010 while "Say My Name" and"Crazy in Love"hadpreviouslywonBestR&BSong.DangerouslyinLove,B\'Day and I Am... Sasha Fierce have all won Best Contemporary R&B Album. Beyoncé set the record for the most Grammy awards won by a female artist in one night in 2010 when she won six awards, breaking the tie she previously held with Alicia Keys, Norah Jones, Alison Krauss,andAmyWinehouse,withAdeleequalingthisin2012.FollowingherroleinDreamgirlsshewasnominatedforBestOriginalSongfor"Listen"andBestActressattheGoldenGlobeAwards,andOutstandingActressinaMotionPictureattheNAACPImageAwards.BeyoncéwontwoawardsattheBroadcastFilmCriticsAssociationAwards2006;BestSongfor"Listen"andBestOriginalSoundtrackforDreamgirls:MusicfromtheMotionPicture.',{'id':'56becc903aeaaa14008c949f','title':'Beyoncé'}),(0.8210567582713351,'A self-described "modern-day feminist", Beyoncé creates songs that are often characterized by themes of love, relationships, and monogamy, as well as female sexuality and empowerment. On stage, her dynamic, highly choreographed performances have led to critics hailing her as one of the best entertainers in contemporary popular music.Throughoutacareerspanning19years,shehassoldover118millionrecordsasasoloartist,andafurther60millionwithDestiny\'s Child, making her one of the best-selling music artists of all time. She has won 20 Grammy Awards and is the most nominated woman in the award\'s history. The Recording Industry Association of America recognizedherastheTopCertifiedArtistinAmericaduringthe2000sdecade.In2009,BillboardnamedhertheTopRadioSongsArtistoftheDecade,theTopFemaleArtistofthe2000sandtheirArtistoftheMillenniumin2011.Timelistedheramongthe100mostinfluentialpeopleintheworldin2013and2014.Forbesmagazinealsolistedherasthemostpowerfulfemalemusicianof2015.',{'id':'56be88473aeaaa14008c9080','title':'Beyoncé'})]UpgradingChanges between SDK versions are not necessarily backwards compatible. We provide a migrate function to help transition smoothly.frompgmlimportmigrateawaitmigrate()This will migrate all collections to be compatible with the latest SDK version.Developer SetupThis Python library is generated from our core rust-sdk. Please checkrust-sdk documentationfor developer setup.RoadmapEnable filters on document metadata invector_search.Issuetext_searchfunctionality on documents using Postgres text search.Issuehybrid_searchfunctionality that does a combination ofvector_searchandtext_search.IssueAbility to call and manage OpenAI embeddings for comparison purposes.IssuePerform chunking on the DB with multiple langchain splitters.IssueSavevector_searchhistory for downstream monitoring of model performance.Issue