Package: vembed
vembedLibrary for Generating Vector Embeddings, performing Similarity searches, and creating Visualizations from Data.pip3installvembedStrings to EmbeddingsConvertaStringto aVector Embedding.fromvembedimportstring_to_embeddinginput_string="This is a test sentence."embedding=string_to_embedding(input_string)#  [0.337, 0.143, 0.714 ...]UseBatchingto Convert Lists of Strings to theirVector FloatRepresentations.fromvembedimportlists_to_embeddingsembeddings=lists_to_embeddings(["Convert to a List[Float]","Another String","More Strings!"])# print(embeddings) [[0.123, 0.456, ...], [0.789, 0.012, ...]]SerializationFunctions for Embedding Serialization for Network Transfer.ProtobufSerialization for usage withgRPCServicesJSONSerialization for usage withRESTAPI'sfromvembedimportlists_to_embeddings,embeddings_to_proto_format,embeddings_to_json_formatembeddings=lists_to_embeddings(["CSV,Row,1,with,some,data","CSV,Row,2,with,other,cols"])# Convert to a Protobuf Serializable Format to send over a gRPC Serviceproto_embedding=embeddings_to_proto_format(embeddings)# Convert to a JSON String for usage with REST API'sjson_embedding=embeddings_to_json_format(embeddings)SimilaritySemantic Similarity Between EntitiesExtract Insights such as Patterns or Relevancy from your Data.Calculating Similarity for Entities.fromvembedimportcalculate_similarities,plot_similaritiescustomer_feedback=["Loved the recent update","The app is user-friendly","Facing issues after the update","The new interface is great"]themes=["positive feedback","negative feedback","app interface","app functionality"]cos_df,dot_df=calculate_similarities(customer_feedback,themes,print_results=True)# Prints and Returns Results# ResultsCosineSimilarities:

Query:'Loved the recent update'Data:'positive feedback'=>SimilarityScore:0.45Data:'app functionality'=>SimilarityScore:0.22Data:'app interface'=>SimilarityScore:0.19Data:'negative feedback'=>SimilarityScore:0.11

Query:'Facing issues after the update'Data:'negative feedback'=>SimilarityScore:0.31Data:'positive feedback'=>SimilarityScore:0.27Data:'app interface'=>SimilarityScore:0.24Data:'app functionality'=>SimilarityScore:0.21

DotProductSimilarities:

Query:'Loved the recent update'Data:'positive feedback'=>SimilarityScore:4.51Data:'app functionality'=>SimilarityScore:2.06Data:'negative feedback'=>SimilarityScore:1.91Data:'app interface'=>SimilarityScore:1.80

Query:'Facing issues after the update'Data:'negative feedback'=>SimilarityScore:2.92Data:'positive feedback'=>SimilarityScore:2.51Data:'app interface'=>SimilarityScore:2.07Data:'app functionality'=>SimilarityScore:1.82GeneratingClean,BeautifulVisualizations from Data.fromvembedimportplot_similarities# .... cos_df, dot_df = calculate_similarities(queries, data)# Create HeapMap for Visualizing Relationshipsplot_similarities(cos_df,dot_df,save_path="heatmaps/customer_feedback_similarity.png")# View and access the Heatmap at /heatmaps/customer_feedback_similarity.pngCosine and Dot Product Vector Similarity Measures@CoefficientLegend

Negative[-]-LowSimilarity
Zero[0]-Orthogonal,NoCommonality
Positive[+]-StrongSimilarityCosine SimilarityRanges between-1and1Recommended when the Context and Similarity is important - and Frequency is not important (Magnitude)Use Case for Cosine SimilarityIn the following example,Direction(thematic orienation)-climate change, agricultureis relevant.Cosine Similarityis useful here as we want to find the relevancy of documents discussing similar topics(direction)- irrespective of the length of frequency of specific words(Magnitude)@Usagequeries=["Climate change effects on agriculture"]data=["Effects of climate change on wheat production","Agriculture in developing countries","Climate change and its impact on global food security","Advances in agricultural technology"]# Calculate cosine similaritiescos_df,_=calculate_similarities(queries,data,sorted=True,print_results=True)Dot Product SimilarityRanges between any Real NumberWhen both theMagnitudeandDirectionof the vectors are important, and we are dealing with vectors in a similar scale.When theFrequency(Magnitude)as well as theDirection(Relevancy)is both important.Use Case for Dot ProductDirection(Types of Articles)andMagnitude(Frequency of Reading Habits)- both have relevancy.@Usageuser_reading_profile=["Read many articles on machine learning","Occasionally reads about space exploration"]article_options=["Latest trends in machine learning","Beginner's guide to space travel","In-depth analysis of neural networks","Recent discoveries in astronomy"]# Calculate dot product similarities_,dot_df=calculate_similarities(user_reading_profile,article_options,sorted=True,print_results=True)Calculating Similarity@Usagequeries=["What is the capital of France?","How is the weather today?"]data=["Paris is the capital of France.","The weather is sunny.","Berlin is the capital of Germany.","It is raining in Berlin."]# Calculate similarities and Print Resultscos_df,dot_df=calculate_similarities(queries,data,sorted=True,print_results=True)# Cosine Similarities:# ...# Dot Product Similarities:# ...VisualizationsGenerate Visualizations from Embeddings such as HeatMap DistributionsCreate aVisualizationto display theEntity Similaritiesusing aHeatmap.@Usagecustomer_feedback=["Loved the recent update","The app is user-friendly","Facing issues after the update","The new interface is great"]themes=["positive feedback","negative feedback","app interface","app functionality"]# Heatmap of Both Cosine and Dot Productcos_df,dot_df=calculate_similarities(customer_feedback,themes,sorted=True)plot_similarities(cos_df,dot_df,save_path="customer_feedback_similarity.png")# Heatmap of Only Cosine Similaritycos_df,_=calculate_similarities(customer_feedback,themes,sorted=True)plot_similarities(cos_df,None,save_path="customer_feedback_similarity.png")# Heatmap of Only Dot Product Similarity_,dot_df=calculate_similarities(customer_feedback,themes,sorted=True)plot_similarities(None,dot_df,save_path="customer_feedback_similarity.png")# View customer_feedback_similarity.png to see the HeatmapTest SuiteLatestTest RungRPC TestsJSON TestsNumpy TestsSerialization TestsEmbedding Generation TestsBatched Embedding Generation TestsCustom Model TestsCaching TestsTransformer Metadata TestsModule Resolution TestsBuild and Run Locally from Sourcegitclonegit@github.com:kuro337/vembed.git# Create Isolated Virtual Envpython3-mvenvvenvsourcevenv/bin/activate# Install Depspipinstall-e.# Run Testschmod+xRUN_TESTS.sh
./RUN_TESTS.sh# Create Distpip3installbuild&&python3-mbuild# Use Built Dist in any projectpip3install./vembed/dist/vembed-0.24-py3-none-any.whlDependenciessentence_transformerstorchtransformerspandasmatplotlibseabornNote: vembed GPU Usage can be enabled from usingNvidia CudaandTorchif a supported Nvidia Graphics Card is Available.RunCUDA Testsfrom thevembed Test Suiteto check Nvidia System GPU availability# Nvidia CUDA and PyTorch Testpython3tests/test_cuda.py# Run all Testspython3-munittestdiscover-stests-vChecking Virtual or System Environment Deps and Cache Size# Check Disk Allocation for Packagesdu-hvenv|sort-hr|head-n102.8Gvenv/lib/python3.11/site-packages/nvidia1.4Gvenv/lib/python3.11/site-packages/torch1.3Gvenv/lib/python3.11/site-packages/torch/lib1.2Gvenv/lib/python3.11/site-packages/nvidia/cudnn/lib1.2Gvenv/lib/python3.11/site-packages/nvidia/cudnn
596Mvenv/lib/python3.11/site-packages/nvidia/cublas# Checking System Cache# Show pip cache locationpipcachedir# /home/user/.cache/pip# Getting Top Folders from Cache by Sizedu-h/home/user/.cache/pip|sort-hr|head-n10# Remove Cached Filespipcachepurge# Cached Filespipcachelist# Installing Packages without Cachepipinstall--no-cache-dir<package_name>Author:kuro337