Package: fast-vector-similarity
Fast Vector Similarity LibraryIntroductionThe Fast Vector Similarity Library is designed to provide efficient computation of various similarity measures between vectors. It is suitable for tasks such as data analysis, machine learning, and statistics, where measuring the relationship between vectors is essential. Written in Rust, the library offers a high-performance solution and can be easily integrated with Python through provided bindings.FeaturesSimilarity MeasuresThe library implements several popular similarity measures, including:Spearman's Rank-Order Correlation (spearman_rho)Kendall's Tau Rank Correlation (kendall_tau)Approximate Distance Correlation (approximate_distance_correlation)Jensen-Shannon Similarity (jensen_shannon_similarity)Hoeffding's D Measure (hoeffding_d)Bootstrapping TechniqueThe library supports bootstrapping for robust similarity computation. This technique involves randomly resampling the dataset to estimate the distribution of the similarity measures, providing more confidence in the results.Performance OptimizationsTo achieve high efficiency, the library leverages:Parallel Computing: Using therayoncrate, computations are parallelized across available CPU cores.Vectorized Operations: The library leverages efficient vectorized operations provided by thendarraycrate.Python BindingsThe library includes Python bindings to enable seamless integration with Python code. Functionspy_compute_vector_similarity_statsandpy_compute_bootstrapped_similarity_statsare exposed, allowing you to compute similarity statistics and bootstrapped similarity statistics, respectively.InstallationRustInclude the library in your Rust project by adding it to your Cargo.toml file.PythonYou can install the Python bindings for the Fast Vector Similarity Library directly from PyPI using the following command:pipinstallfast_vector_similarityThis command will download and install the package, making it available for use in your Python projects.Use with Text Embedding Vectors from LLMsThe Python demo code below introduces additional functionality that allows the Fast Vector Similarity Library to easily work with text embedding vectors from Language Models like LLMs (e.g., Llama2), such as those generated by my other library, theLlama2 Embeddings FastAPI Service(particularly, the output of the/get_all_embedding_vectors_for_documentendpoint):1. Converting Text Embeddings to Pandas DataFrameThe functionconvert_embedding_json_to_pandas_dfreads text embeddings from a JSON file and converts them into a Pandas DataFrame. Each embedding is associated with a specific text (e.g., a sentence from Shakespeare's Sonnets), and the DataFrame structure facilitates manipulation and analysis of these embeddings.2. Applying Fast Vector Similarity to Text EmbeddingsThe functionapply_fvs_to_vectortakes a row of embeddings and a query embedding, applies the chosen similarity measures, and returns the results as a JSON object. This function leverages the core similarity computation functionality provided by the Fast Vector Similarity Library, allowing it to be used directly on text embeddings.3. Comparing Embeddings with Large DatasetsThe main section of the code includes an example where text embeddings from Llama2 are loaded and analyzed. This part of the code demonstrates how to:Select a random query embedding from the dataset.Compute similarity measures between the query embedding and the rest of the dataset.Create a DataFrame to hold the similarity results.Sort and display the top 10 most similar embeddings by Hoeffding's D.4. Compatibility with Higher-Dimensional EmbeddingsThe example works with 4096-dimensional vectors, showcasing the library's ability to handle high-dimensional data typical of modern language models. This compatibility ensures that the library can be used with a wide range of language models and text embedding techniques. In fact, the library can easily handle even larger dimensions.Example Python Codeimporttimeimportnumpyasnpimportjsonimportpandasaspdfromrandomimportchoiceimportfast_vector_similarityasfvsdefconvert_embedding_json_to_pandas_df(file_path):# Read the JSON filewithopen(file_path,'r')asfile:data=json.load(file)# Extract the text and embeddingstexts=[item['text']foritemindata]embeddings=[item['embedding']foritemindata]# Determine the total number of vectors and the dimensions of each vectortotal_vectors=len(embeddings)vector_dimensions=len(embeddings[0])iftotal_vectors>0else0# Print the total number of vectors and dimensionsprint(f"Total number of vectors:{total_vectors}")print(f"Dimensions of each vector:{vector_dimensions}")# Convert the embeddings into a DataFramedf=pd.DataFrame(embeddings,index=texts)returndfdefapply_fvs_to_vector(row_embedding,query_embedding):params={"vector_1":query_embedding.tolist(),"vector_2":row_embedding.tolist(),"similarity_measure":"all"}similarity_stats_str=fvs.py_compute_vector_similarity_stats(json.dumps(params))returnjson.loads(similarity_stats_str)defmain():length_of_test_vectors=15000print(f"Generating 2 test vectors of length{length_of_test_vectors}...")vector_1=np.linspace(0.,length_of_test_vectors-1,length_of_test_vectors)vector_2=vector_1**0.2+np.random.rand(length_of_test_vectors)print("Generated vector_1 using linear spacing and vector_2 using vector_1 with a power of 0.2 and some random noise.\n")similarity_measure="all"# Or specify a particular measureparams={"vector_1":vector_1.tolist(),"vector_2":vector_2.tolist(),"similarity_measure":similarity_measure}# Time the exact similarity calculationprint("Computing Exact Similarity Measures...")start_time_exact=time.time()similarity_stats_str=fvs.py_compute_vector_similarity_stats(json.dumps(params))similarity_stats_json=json.loads(similarity_stats_str)elapsed_time_exact=time.time()-start_time_exactprint(f"Time taken for exact calculation:{elapsed_time_exact:.5f}seconds")# Print resultsprint("_______________________________________________________________________________________________________________________________________________\n")print("Spearman's rho:",similarity_stats_json["spearman_rho"])print("Kendall's tau:",similarity_stats_json["kendall_tau"])print("Distance Correlation:",similarity_stats_json["approximate_distance_correlation"])print("Jensen-Shannon Similarity:",similarity_stats_json["jensen_shannon_similarity"])print("Hoeffding's D:",similarity_stats_json["hoeffding_d"])print("_______________________________________________________________________________________________________________________________________________\n")# Bootstrapped calculationsnumber_of_bootstraps=2000n=15sample_size=int(length_of_test_vectors/n)print(f"Computing Bootstrapped Similarity Measures with{number_of_bootstraps}bootstraps and a sample size of{sample_size}...")start_time_bootstrapped=time.time()params_bootstrapped={"x":vector_1.tolist(),"y":vector_2.tolist(),"sample_size":sample_size,"number_of_bootstraps":number_of_bootstraps,"similarity_measure":similarity_measure}bootstrapped_similarity_stats_str=fvs.py_compute_bootstrapped_similarity_stats(json.dumps(params_bootstrapped))bootstrapped_similarity_stats_json=json.loads(bootstrapped_similarity_stats_str)elapsed_time_bootstrapped=time.time()-start_time_bootstrappedprint(f"Time taken for bootstrapped calculation:{elapsed_time_bootstrapped:.5f}seconds")time_difference=abs(elapsed_time_exact-elapsed_time_bootstrapped)print(f"Time difference between exact and robust bootstrapped calculations:{time_difference:.5f}seconds")# Print bootstrapped resultsprint("_______________________________________________________________________________________________________________________________________________\n")print("Number of Bootstrap Iterations:",bootstrapped_similarity_stats_json["number_of_bootstraps"])print("Bootstrap Sample Size:",bootstrapped_similarity_stats_json["sample_size"])print("\nRobust Spearman's rho:",bootstrapped_similarity_stats_json["spearman_rho"])print("Robust Kendall's tau:",bootstrapped_similarity_stats_json["kendall_tau"])print("Robust Distance Correlation:",bootstrapped_similarity_stats_json["approximate_distance_correlation"])print("Robust Jensen-Shannon Similarity:",bootstrapped_similarity_stats_json["jensen_shannon_similarity"])print("Robust Hoeffding's D:",bootstrapped_similarity_stats_json["hoeffding_d"])print("_______________________________________________________________________________________________________________________________________________\n")# Compute the differences between exact and bootstrapped resultsmeasures=["spearman_rho","kendall_tau","approximate_distance_correlation","jensen_shannon_similarity","hoeffding_d"]formeasureinmeasures:exact_value=similarity_stats_json[measure]bootstrapped_value=bootstrapped_similarity_stats_json[measure]absolute_difference=abs(exact_value-bootstrapped_value)percentage_difference=(absolute_difference/exact_value)*100print(f"\nDifference between exact and bootstrapped{measure}:{absolute_difference}")print(f"Difference as % of the exact value:{percentage_difference:.2f}%")print("Now testing with a larger dataset, using sentence embedddings from Llama2 (4096-dimensional vectors) on some Shakespeare Sonnets...")# Load the embeddings into a DataFrameinput_file_path="sample_input_files/Shakespeare_Sonnets_small.json"embeddings_df=convert_embedding_json_to_pandas_df(input_file_path)# Select a random row for the query embeddingquery_embedding_index=choice(embeddings_df.index)query_embedding=embeddings_df.loc[query_embedding_index]print(f"Selected query embedding for sentence: `{query_embedding_index}`")# Remove the selected row from the DataFrameembeddings_df=embeddings_df.drop(index=query_embedding_index)# Apply the function to each row of embeddings_dfjson_outputs=embeddings_df.apply(lambdarow:apply_fvs_to_vector(row,query_embedding),axis=1)# Create a DataFrame from the list of JSON outputsvector_similarity_results_df=pd.DataFrame.from_records(json_outputs)vector_similarity_results_df.index=embeddings_df.index# Add the required columns to the DataFramecolumns=["spearman_rho","kendall_tau","approximate_distance_correlation","jensen_shannon_similarity","hoeffding_d"]vector_similarity_results_df=vector_similarity_results_df[columns]# Sort the DataFrame by the hoeffding_d column in descending ordervector_similarity_results_df=vector_similarity_results_df.sort_values(by="hoeffding_d",ascending=False)print("\nTop 10 most similar embedding results by Hoeffding's D:")print(vector_similarity_results_df.head(10))UsageIn RustYou can utilize the core functionality in Rust by calling functions likecompute_vector_similarity_statsandcompute_bootstrapped_similarity_statswith appropriate parameters.In PythonInstall the provided Python package and use the functions as demonstrated in the example above.About the Different Similarity MeasuresSpearman's Rank-Order Correlation (spearman_rho):
Spearman's rho assesses the strength and direction of the monotonic relationship between two ranked variables. Unlike Pearson's correlation, it does not assume linearity and is less sensitive to outliers, making it suitable for non-linear relationships.Kendall's Tau Rank Correlation (kendall_tau):
Kendall's tau measures the ordinal association between two variables. It's valuable for its ability to handle ties and its interpretability as a probability. Unlike other correlation measures, it's based on the difference between concordant and discordant pairs, making it robust and versatile.Approximate Distance Correlation (approximate_distance_correlation):
Distance correlation quantifies both linear and non-linear dependencies between variables. It has the powerful property of being zero if and only if the variables are independent. This makes it a more comprehensive measure of association than traditional correlation metrics.Jensen-Shannon Similarity (jensen_shannon_similarity):
Jensen-Shannon Similarity is derived from the Jensen-Shannon Divergence, a symmetrical and smoothed version of the Kullback-Leibler divergence. It quantifies the similarity between two probability distributions and is especially useful in comparing distributions that may have non-overlapping support.Hoeffding's D Measure (hoeffding_d):
Hoeffding's D is a non-parametric measure that detects complex non-linear relationships between variables. The D statistic is robust against various alternatives to independence, including non-monotonic relationships. It's particularly useful when the nature of the relationship between variables is unknown or unconventional.Each of these measures has unique properties and applicability, providing a comprehensive toolkit for understanding the relationships between variables in different contexts. By including both classical and more specialized measures, the library offers flexibility and depth in analyzing vector similarities.Bootstrapping Technique for Robust EstimatorsBootstrapping is a powerful statistical method that allows us to estimate the distribution of a statistic by repeatedly resampling with replacement from the observed data. In the context of the Fast Vector Similarity Library, bootstrapping is employed to obtain robust estimators of similarity measures between vectors. Here's how it works:ProcessRandom Subset Selection:
For each bootstrap iteration, a random subset of indices is selected from the original vectors. These indices are used to create resampled vectors that preserve the original data's structure and relationships.Compute Similarity:
The selected random subset of indices is used to compute the similarity between the resampled vectors according to the chosen similarity measure. This process is repeated many times, resulting in a distribution of similarity estimates.Robust Averaging:
To obtain a robust estimator that is less sensitive to outliers, the interquartile range (IQR) of the distribution of similarity estimates is considered. By keeping only the values within the IQR, the influence of extreme values is minimized. A robust average of the values within this range provides the final robust similarity estimate.Why It's UsefulThe bootstrapping technique offers several advantages:Robustness to Outliers: By focusing on the interquartile range and using a robust average, bootstrapping minimizes the influence of outliers. This makes the estimator more reliable, especially when the original data may contain anomalous values.Model-Free Estimation: Bootstrapping doesn't assume a specific underlying distribution, making it a non-parametric method. This flexibility allows it to be applied to various types of data and similarity measures.Confidence Intervals: Bootstrapping can also be used to construct confidence intervals for the similarity measures. These intervals provide insights into the uncertainty associated with the estimate, enhancing interpretability.Enhanced Understanding of Relationships: By assessing the distribution of similarity measures, bootstrapping provides a more comprehensive view of the relationships between vectors. This deeper understanding can be vital in contexts like data analysis, where subtle differences in relationships may have significant implications.The inclusion of the bootstrapping technique in the Fast Vector Similarity Library adds a layer of robustness and flexibility to the computation of similarity measures. It offers users a way to obtain more reliable and insightful estimates, accommodating various data characteristics and analytical needs. Whether dealing with potential outliers or seeking a more nuanced understanding of vector relationships, bootstrapping provides a valuable tool within the library's comprehensive suite of features.ConclusionThe Fast Vector Similarity Library offers an extensive set of tools for computing various similarity measures between vectors. With the inclusion of performance optimizations and Python bindings, it can be an essential part of any data analysis or machine learning pipeline.For further details and customization, please refer to the source code.