string2string: A Modern Python Library for String-to-String Algorithms (2024)

Abstract: We introduce string2string, an open-source library that offers a comprehensive suite of efficient algorithms for a broad range of stringto-string problems. It includes traditional algorithmic solutions as well as recent advanced neural approaches to tackle various problems in string alignment, distance measurement, lexical and semantic search, and similarity analysis—along with several helpful visualization tools and metrics to facilitate the interpretation and analysis of these methods. Notable algorithms featured in the library include the Smith-Waterman algorithm for pairwise local alignment, the Hirschberg algorithm for global alignment, the Wagner-Fischer algorithm for edit distance, BARTScore and BERTScore for similarity analysis, the Knuth-Morris-Pratt algorithm for lexical search, and Faiss for semantic search. In addition, it wraps existing efficient and widely-used implementations of certain frameworks and metrics, such as sacreBLEU and ROUGE. Overall, the library aims to provide extensive coverage and increased flexibility in comparison to existing libraries for strings. It can be used for many downstream applications, tasks, and problems in natural-language processing, bioinformatics, and computational social sciences. It is implemented in Python, easily installable via pip, and accessible through a simple API. Source code, documentation, and tutorials are all available on our GitHub page: https://github.com/ stanfordnlp/string2string.1

heading: 1 Introduction
text: String-to-string problems have a wide range of applications in various domains and fields, such as natural-language processing (e.g., information extraction, spell checking, and semantic search), computational molecular biology (e.g., DNA sequence alignment), programming languages and compilers
1Correspondence to: msuzgun@cs.stanford.edu.
(e.g., parsing and compiling), as well as computational social sciences and digital humanities (e.g., lexical and semantic analysis of literary texts).
The current state of string-to-string processing, alignment, distance, similarity, and search algorithms is marked by a multitude of implementations in many programming languages, such as C++, Java, and Python, but these implementations are not unified and lack flexibility, modularity, and comprehensive documentation, hindering their accessibility to users. Thus, there is a need for a unified platform that combines these functionalities into one accessible and comprehensive system.
In this work, we present an open-source library that offers a broad collection of algorithms and techniques for the alignment, manipulation, and evaluation of string-to-string mappings.2 These problems include measuring the lexical distance between two strings (e.g., under the Levenshtein edit distance metric), computing the local or global alignment between two DNA sequences (e.g., based on a substitution matrix such as BLOSUM), calculating the semantic similarity between two texts (e.g., using BART-embeddings), and performing efficient semantic search (e.g., via the Faiss library by FAIR (Johnson et al., 2019)).
The string2string library has been purposefully crafted to prioritize key design principles, including modularity, completeness, efficiency, flexibility, and clarity. As an open-source initiative, the library will continue to grow and adapt to meet the evolving of its user community in the future, and we are committed to ensuring that the library remains a flexible, accessible, and dynamic resource, capable of accommodating the changing landscape of string-to-string problems and tasks.
2We define a string as an ordered collection of characters— such as letters, numerals, symbols—which serves as a representation of a unit of information, text, or data. Strings can be used to represent anything, from simple sentences to complex nucleic acid sequences or elaborate computer programs.
278

heading: 2 Related Work
text: The fields of natural-language processing and machine learning have a long-standing and exemplary tradition of fostering a culture that values opensource tools and libraries. While designing our own string-to-string library, which includes both traditional algorithmic and neural approaches to various problems, we have drawn inspirations from Natural Language Toolkit (NLTK; Bird and Loper (2004)), Gensim (Řehůřek and Sojka, 2010), OpenGrm Ngram (Roark et al., 2012), Stanford CoreNLP Toolkit (Manning et al., 2014), OpenNMT (Klein et al., 2017), tensor2tensor (Vaswani et al., 2018), AllenNLP (Gardner et al., 2018), fairseq (Ott et al., 2019), spaCy (Neumann et al., 2019), Stanza (Qi et al., 2020), Transformers (Wolf et al., 2020), and Torch-Struct (Rush, 2020), among many others.

heading: 3 Overview of Algorithms
text: The string2string library offers a rich collection of algorithmic solutions to tackle a wide range of string-to-string problems and tasks. We have clustered these algorithms into four categories: pair-
wise alignment, distance measurement, similarity analysis, and search.3 Each category contains a suite of efficient algorithms that are tailored to address specific problems within their respective domain. In what follows, we provide a brief overview of these algorithms, along with the associated problems or tasks they are designed to solve.

heading: 3.1 Pairwise Alignment
text: Pairwise string alignment is the problem of identifying an optimal alignment between two strings, such as nucleotide sequences in DNA or paragraphs in a text. This task involves aligning them in a way that maximizes the number of matching symbols while allowing for gaps or mismatches where necessary. Pairwise string alignment is a widely-used technique that plays a crucial role in tasks such as DNA sequence alignment, database searching, and phylogenetic analysis.
As exhibited in Table 1, the library, in its current state, provides efficient solutions to local alignment, global alignment, longest common substring
3By duality, distance measurement methods can naturally be used for string similarity analysis, and vice versa.
(LCSubstring), longest common subsequence (LCSubsequence), and dynamic time warping (DTW) problems. It is worth noting that all of the problems and tasks covered in this suite can be solved using standard dynamic programming-based solutions. Alternative approaches to long sequence or string alignment problems, such as FASTA (Lipman and Pearson, 1985) and BLAST (Altschul et al., 1990), also exist; they offer improved efficiency through the use of probabilistic or heuristic methods, but they do not always guarantee optimal solutions and may sacrifice accuracy for speed. Due to their ability to handle large datasets quickly and provide reasonably accurate results, BLAST and FASTA are still widely used in bioinformatics, and for that reason, we plan on including them in the string2string library in the future.

heading: 3.2 Distance
text: String distance refers to the problem of quantifying the extent to which two given strings are dissimilar based on a distance function. The Levenshtein edit distance metric, for instance, corresponds to the minimum number of insertion, deletion, or substitution operations required to transform one string into another. It has a famous dynamic programming solution, which is often referred to as the Wagner-Fischer algorithm (Wagner and Fischer, 1974). In this library, we provide an implementation of the Wagner-Fischer algorithm, which has a quadratic time and space complexity, as well as an improved version of it, which reduces the overall space complexity to linear.4 We further cover and provide efficient solutions to the Hamming dis-
4Incidentally, we highlight an important discovery by Backurs and Indyk (2015) that the edit distance between two strings cannot be computed in strongly subquadratic time, unless the strong exponential time hypothesis is false.
tance, Damerau-Levenshtein distance, and Jaccard distance problems, as shown in Table 1.
One noteworthy feature of the library is that it allows the user to specify the weight of string operations (insertions, deletions, substitutions, and transpositions) depending on the distance function of choice. Furthermore, it can compute the distance between not only string pairs but also pairs of lists of strings, thereby not limiting the users to make comparisons only at the character or symbol level.

heading: 3.3 Similarity
text: String similarity refers to the problem of measuring the degree to which two given strings are similar to each other based on a similarity function—which can be defined on various criteria, such as character matching, longest common substring or subsequence comparison, or structural alignment. There is a natural duality between string similarity measures and string distance measures, which means
that it is possible to convert one into the other with ease; hence, it is often the case that one uses string similarity and distance measures interchangeably.
Jaccard similarity, Jaro similarity, Jaro-Winkler similarity, LCSubsequence similarity, cosine similarity, BERTScore, and BARTScore are among the similarity measures that are covered in the library. The first four can be seen as lexical similarity measures, as they assess surface or structural closeness, whereas the remaining three can be regarded as semantic similarity measures, as they take the implied meaning of the constituents of the given strings into account.
The present library provides users with the ability to calculate cosine similarity not only between individual words—via pre-trained GloVe (Pennington et al., 2014) or fastText (Joulin et al., 2016) word embeddings—but also for longer pieces of text such as sentences, paragraphs, or even documents—via averaged or last-token embed-
dings obtained from a neural language model such as BERT (Devlin et al., 2019). As we also mention in Section 4, this feature enables users to compare the semantic similarity of longer segments of text in only a few lines of code, providing greater flexibility in text analysis tasks.

heading: 3.4 Search
text: String search, also known as string matching, refers to the problem of determining whether a given pattern string exists inside a longer string. The brute-force approach to string search would involve examining each position of the longer string to determine if it matches the pattern string; however, this method can be inefficient, particularly when dealing with large strings. In the library, we therefore include the Rabin-Karp (Karp and Rabin, 1987), Boyer-Moore (Boyer and Moore, 1977), and Knuth-Morris-Pratt (Knuth et al., 1977) algorithms for identical string matching as well.
The library additionally provides support for semantic search via Facebook AI Research’s Faiss library (Johnson et al., 2019), which, in essence, allows efficient similarity search and clustering of dense vectors. In contrast to the previous setup for identical string matching, Faiss initially requires the user to provide a list of strings (texts) as a corpus and creates a fixed-vector representation of each string using a neural language model.5 Once the initialization of the corpus is done, one can perform “queries” on the corpus. Given a new query, one can automatically get the embedding of that query, map it onto the embedding space of the corpus, and return the nearest neighbours of the query on the embedding space, thereby finding the texts that are semantically closest to the query.
As one might imagine, string search algorithms are highly practical tools that have a wide range of applications across different fields. These algorithms allow users to locate and retrieve specific patterns within a long text or a large corpus. For instance, the string search algorithms covered in the string2string library—as shown in Table 2— can be used for pattern recognition, DNA matching, plagiarism detection, and data mining, among many other downstream applications and tasks.
5The user is provided with the flexibility to determine how to obtain a fixed embedding for each text. Specifically, the user has the option to choose between different embedding methods, such as averaging the token embeddings or selecting the embedding of the final token in the sequence.

heading: 4 Additional Features
text: One noteworthy feature of the library is its ability to simplify the use of the GloVe (Pennington et al., 2014) or fastText (Joulin et al., 2016) word embeddings by enabling users to download and use them with just one line of code. This streamlined process not only saves users time and effort but also eliminates the need for additional installations or complex configurations. By providing this feature, we have sought to make the library more accessible to users and encourage the use of pre-trained word embeddings in various string-to-string tasks and applications, such as measuring the cosine similarity between two words.6
Similarly, users can seamlessly get the averaged or last token embeddings of a piece of text from a pre-trained language model that is hosted on Hugging Face Models (Wolf et al., 2020) or on their local path in a few lines of code, and we provide both CPU and GPU support for these computations.
Finally, we note that the library offers various visualization capabilities that allow users to visually inspect the alignment between two strings or the score matrix of the distance or similarity between them. This functionality facilitates the understanding and interpretation of the output of various algorithms and can aid in the selection of the most suitable algorithm for a given task. By incorporating this feature into the library, we aim to enhance the user experience and provide intuitive means of interpreting the outputs.7 Figure 2 shows a simple alignment between two lists of strings, as generated by our library.
6Notably, fastText offers pre-trained word embeddings for 157 languages—trained on Common Crawl and Wikipedia— that one can easily download and use with our library.
7For instance, our library provides a practical, hands-on tutorial focused on the HUPD (Suzgun et al., 2022b), a large patent corpus. This tutorial showcases the efficient use of our library’s functionalities and features for performing semantic search and visualizing the textual content of patent documents.

heading: 5 Library Design Principles
text: We have endeavoured to build a comprehensive and easy-to-use platform for numerous string-to-string processing, comparison, manipulation, and search algorithms. We have purposefully structured and organized the library to allow easy customization, functional extension, and modular integration.
Completeness. The library offers a comprehensive set of classical algorithms as well as neural approaches to tackle a wide range of string-to-string problems. We have intentionally included both efficient and simpler solutions, such as brute-force and memoization-based approaches, where appropriate. By providing multiple solutions to the same problem, the library allows users to compare and contrast the performance of different algorithmic methods. This approach enhances users’ understanding of the trade-offs between different algorithmic solutions and helps them appreciate the strengths and limitations of each approach.
Modularity. To improve the efficiency and maintainability of the codebase, we have adopted a modular design approach that breaks down the code into smaller and self-contained modules. This approach simplifies the process of adding new features and functionalities and modifying existing ones for developers, while also enabling efficient testing, debugging, and overall maintenance of the library. The modular design has allowed us to quickly locate and fix any errors, without disrupting the entire codebase, during development. Moreover, this modular approach ensures that the library is scalable and adaptable to future updates and changes, which should enable us to easily improve the library’s functionality and expand its use in various tasks and applications moving forward.
Efficiency. We have taken great care to ensure that the algorithm implementations are efficient both computationally and memory-wise so that they could easily handle large datasets and complex tasks. We provide basic support for process-based parallelism via Python’s inherent multiprocessing package, as well as joblib. Additionally, we provide GPU support for neuralbased approaches, whenever applicable. While we strive to balance efficiency and clarity, we acknowledge that in some cases, trade-offs may exist between the two. In such cases, we have placed greater emphasis on clarity, ensuring that the algorithms are transparent and easy to understand, even at the cost of some efficiency. Nonetheless, we be-
lieve that the library’s overall efficiency, combined with its transparency and comprehensibility, makes it a valuable resource for the community.
Support for List of Strings. The library has been designed to support not only individual strings but also lists of strings—whenever possible, enabling users to align or compare strings at the subword or token level. This feature provides greater flexibility in the library’s use cases, as it allows users to analyze and compare more complex data structures beyond only individual strings. By supporting lists of strings, the library can handle a wider range of textual input types and structures.
Strong Typing. The use of strong typing requirements is an essential aspect of the library, as it ensures that the inputs are always consistent and accurate, which is crucial for generating reliable results. By carefully annotating all the arguments of the algorithms used in the library, we have sought to increase the robustness and reliability of the codebase. This approach has helped prevent input-related errors, such as incorrect data type or format, from occurring during execution.
Accessibility. The library has been implemented in Python, a programming language which has been the core of many natural-language processing tools and applications in academia and industry. The string2string library is “pip”-installable and can be integrated into common machine learning and natural-language processing frameworks such as PyTorch, TensorFlow, and scikit-learn.
Open-Source Effort. The library is—and will remain—free and accessible to all users. We hope that this approach will promote community-driven development and encourage collaboration among researchers and developers, enabling them to contribute to and improve the library.

heading: 6 Conclusion
text: We introduced string2string, an open-source library that offers a large collection of algorithms for a broad range of string-to-string problems. The library is implemented in Python, hosted on GitHub, and installable via pip. It contains extensive documentation along with several hands-on tutorials to aid users to explore and utilize the library effectively. With the help of the open-source community, we hope to grow and improve the library. We encourage users to feel free to provide us with feedback, report any issues, and propose new features to expand the functionality and scope of the library.

heading: Acknowledgements
text: We would like to express our deepest appreciation to Corinna Coupette for providing us with the inspiration to create this open-source library. Furthermore, we would like to thank Sarah DeMott, Sebastian Gehrmann, Elena Glassman, Tayfun Gür, Daniel E. Ho, Şule Kahraman, Deniz Keleş, Scott D. Kominers, Christopher Manning, Luke MelasKyriazi, Tolúlo.pé. Ògúnrè.mí, Alexander “Sasha” Rush, George Saussy, Kutay Serova, Holger Spamann, Kyle Swanson, and Garrett Tanzer for their valuable comments, useful suggestions, and support. We owe a special debt of gratitude to Federico Bianchi for inspecting our code, documentation, and tutorials many times and providing us with such helpful and constructive feedback.

heading: Future Work
text: Moreover, this modular approach ensures that the library is scalable and adaptable to future updates and changes, which should enable us to easily improve the library’s functionality and expand its use in various tasks and applications moving forward. Efficiency. We have taken great care to ensure that the algorithm implementations are efficient both computationally and memory-wise so that they could easily handle large datasets and complex tasks. We provide basic support for process-based parallelism via Python’s inherent multiprocessing package, as well as joblib. Additionally, we provide GPU support for neuralbased approaches, whenever applicable. While we strive to balance efficiency and clarity, we acknowledge that in some cases, trade-offs may exist between the two. In such cases, we have placed greater emphasis on clarity, ensuring that the algorithms are transparent and easy to understand, even at the cost of some efficiency. Nonetheless, we be-
lieve that the library’s overall efficiency, combined with its transparency and comprehensibility, makes it a valuable resource for the community. Support for List of Strings. The library has been designed to support not only individual strings but also lists of strings—whenever possible, enabling users to align or compare strings at the subword or token level. This feature provides greater flexibility in the library’s use cases, as it allows users to analyze and compare more complex data structures beyond only individual strings. By supporting lists of strings, the library can handle a wider range of textual input types and structures. Strong Typing. The use of strong typing requirements is an essential aspect of the library, as it ensures that the inputs are always consistent and accurate, which is crucial for generating reliable results. By carefully annotating all the arguments of the algorithms used in the library, we have sought to increase the robustness and reliability of the codebase. This approach has helped prevent input-related errors, such as incorrect data type or format, from occurring during execution. Accessibility. The library has been implemented in Python, a programming language which has been the core of many natural-language processing tools and applications in academia and industry. The string2string library is “pip”-installable and can be integrated into common machine learning and natural-language processing frameworks such as PyTorch, TensorFlow, and scikit-learn. Open-Source Effort. The library is—and will remain—free and accessible to all users. We hope that this approach will promote community-driven development and encourage collaboration among researchers and developers, enabling them to contribute to and improve the library