Package: pysdcxx
This library may be used to compare strings by a similarity measure,
defined as Sørensen–Dice coefficient calculated on multi-sets of the
strings' "bigrams" (all couples of adjacent characters).Bigram (multi-)set is relatively easy to generate (O(n)time
complexity lower bound in terms of the string length).Also, compared to just using a (multi-)set of characters, bigrams do
retain certain level of word structure. E.g. backwards spelled words are
not deemed similar (unlike when single character set is used).Just note that single-character words are all perfectly similar in this
metric, as their bigram sets are empty---and therefore the same. It may
be a good idea to augment words like that with an additional character
(like a whitespace) to mitigate that problem.Using implementation of bigram multisets above, a sequence matcher
implementation is also available. Given a sequence of text tokens, the
matcher then allows "fuzzy" matching of strings (using their bigrams
again) to sub-sequences of the text tokens.See detailed documentation indocdirectory. Rendered documentation:https://htmlpreview.github.io/?https://github.com/vencik/libsdcxx/blob/master/doc/sequence_matcher.htmlList of featuresBigram multi-set objects implemented as sorted lists of character
tuples with count (O(n log n)creation time complexity in terms of
the string length)Union operation hasO(m+n)time complexity (sum of multi-sets'
cardinalities at most)Intersection doesn’t produce objects; only its size is calculated inO(m+n)timeTemplate implementation, allowing for both ASCII/ANSI characters and
UNICODE charactersImplementations usingstd::multisetandstd::unordered_multisetalso availablePerformance tests show that, long story short, the "custom"
implementation is the best (notably faster unions, intersection size
computation in similar or better time)Sequence matcher (using the best performing bigrams) with several
optimisationsPython v3 binding is provided (aspysdcxxmodule, packaged)Pythonmultisetbased implementation also compared---and is
expectedly much slowerBuild and installationYou shall need C++ compiler with support for at least C++17. Recent
enough gcc (v8.3 or newer) is OK (older versions might do as well,
though). You shall also needcmakeandmake.Python v3.7 or newer is supported. For the Python package build, you
shall needpipand Pythondistutilsand thewheelpackage. If you
wish to run Python UTs (which is highly recommended), you shall also
needpytest.E.g. on Debian-based (or similar,aptusing) systems, the following
should get you the required prerequisites unless you wish to usepyenv.# apt-get install g++ cmake make git # apt-get install python3-pip
python3-distutils # unless you use pyenv $ pip install wheel pytest #
ditto, better do that in pyenv sandbox</programlisting>On Mac OS X, you’ll need Xcode tools and Homebrew. Then, install the
required prerequisites by$ brew install coreutils cmake make git</programlisting>If you do wish to usepyenvto create and manage project sandbox
(which is advisable), see short intro to that in the subsection below.Anyhow, with all the prerequisites installed, clone the project:$ git clonehttps://github.com/vencik/libsdcxx.git\</programlisting>Build the project, run UTs and build Python packages:$ cd libsdcxx $ ./build.sh -ugp</programlisting>Note that thebuild.shscript has options; run$ ./build.sh -hto
see them.Most importantly, run with-sor--enable-ptoptions to perform
performance tests. Performance tests compare computation times of object
construction, union operations and intersection size computations of the
implementations. If you installmultisetPython package (usingpip),
the perf. tests shall also produce results for pure-Python
implementation usingmultiset.Multiset(not necessary).The perf. tests are written on the Python level, so the measured times
also contain some Python code runtime (and not trivial). I’d expect
results in native code to be notably better; but the test code is
identical, so the measurements should be meaningful for comparison.If you wish, usepipto install the Python package:# pip install pysdcxx-*.whl</programlisting>Note that it’s recommended to usepyenv, especially for development
purposes.Managing project sandbox withpyenvFirst installpyenv. You may use either your OS package repo (Homebrew
on Mac OS X) or webpyenvinstaller. Setuppyenv(set environment
variables) as instructed.Then, createpysdcxxproject sandbox, thus:$ pyenv install 3.9.6 # your choice of the Python interpreter version,
>= 3.7 $ pyenv virtualenv 3.9.6 pysdcxx</programlisting>Now, you can always (de)activate the virtual environment (switch to the
sandbox) by# pyenv activate pysdcxx # pyenv deactivate</programlisting>In the sandbox, install Python packages usingpipas usual.$ pip install wheel pytest</programlisting>UsageC++Usingbigrams#include<libsdcxx/bigrams.hxx>usingbigrams=libsdcxx::bigrams;// wbigrams for UNICODEconstautobgrms_empty=bigrams();// empty bigrams setconstautobgrms1=bigrams("Hello world!");// construct from stringsize_tcnt=bgrms1.size();// number of bigramsstd::cout<<bgrms1;// serialisationfor(constauto&bigram_cnt:bgrms1){// tuple of (bigram, count)constauto&bigram=std::get(bigram_cnt);std::cout<<"Bigram : "<<std::get(bigram)<<std::get(bigram)<<std::endl;std::cout<<"Count  : "<<std::get(bigram_cnt)<<std::endl;}// (Const.) iterators are supported via cbegin, cend and begin, end method callsconstautobgrms2=bigrams("Hell or woes.");size_tisect_size=bigrams::intersect_size(bgrms1,bgrms2);// intersection cardinalitydoublesdc=bigrams.sorensen_dice_coef(bgrms1,bgrms2);// similarity, in [0,1]autouni0n=bgrms1+bgrms2;// 2 bigrams unionautouni0n=bigrams::unite(bgrms1,bgrms2/* , ... */);// variadic unionuni0n+=bigrams("more stuff");// objects are mutableUsingsequence_matcher#include<libsdcxx/sequence_matcher.hxx>#include<libsdcxx/bigrams.hxx>usingsequence_matcher=libsdcxx::sequence_matcher;// wsequence_matcher for UNICODEusingbigrams=libsdcxx::bigrams;// wbigrams for UNICODEautomatcher=sequence_matcher();matcher.reserve(10);// reserve space for bigrams matrix for text of 10 tokens// (reservation is not strictly necessary, but advisable)constautobgrms_hello=bigrams("Hello");constautobgrms_world=bigrams("world");matcher.emplace_back("Prologue");// create token bigrams in-placematcher.emplace_back(" .",true);// it's a good idea to pad single-char strings...matcher.emplace_back("  ");// to 2 characters (so that they produce a bigram)matcher.push_back(bigrams_hello);// push existing bigrams backmatcher.emplace_back("  ",true);// true here stands for "strip" token; matched...matcher.push_back(bigrams_world);// sub-sequences are restricted not to begin/end...matcher.emplace_back(" !",true);// with such tokensmatcher.emplace_back("  ",true);matcher.emplace_back("Epilogue");matcher.emplace_back(" .",true);constautobgrms_helo_wordl=bigrams::unite(// note thatbigrams of the wholebigrams("Helo"),bigrams("  "),bigrams("wordl"));// sentence would differautomatch=matcher.begin(bgrms_helo_wordl,0.7);// match with threshold 0.7for(;match!=matcher.end();++match){std::cout<<match<<std::endl;// simple string form of match info, try itstd::cout<<"Match bigrams: "<<*match<<std::endl// sub-sequence bigrams<<"Match at index: "<<match.begin()<<std::endl// beginning token index<<"Match end: "<<match.end()<<std::endl// index just past the end<<"Match size: "<<match.size()<<std::endl// number of tokens<<"Match score: "<<match.sorensen_dice_coef()<<std::endl;}// ... you may of course continue matching other sequences...Pyton v3UsingBigramsfrompysdcxximportBigrams# Python Bigrams are implemented by wbigrams, support UNICODEbgrms_empty=Bigrams()# empty bigrams setbgrms1=Bigrams("Hello world!")# construct from stringcnt=len(bgrms1)# number of bigramsprint(str(bgrms1),f"{bgrms1}")# string serialisationforbigram,cntinbgrms1:# Bigrams are tuple[str, int] generatorsassertlen(bigram)==2bgrms2=Bigrams("Hell or woes.")isect_size=Bigrams.intersect_size(bgrms1,bgrms2)# intersection cardinalitysdc=Bigrams.sorensen_dice_coef(bgrms1,bgrms2)# simiarity, in [0,1]union=bgrms1+bgrms2# 2 bigrams unionunion+=Bigrams("more stuff")# objects are mutableUsingSequenceMatcherfrompysdcxximportSentenceMatcher,Bigramsmatcher=SequenceMatcher()# empty matchermatcher=SequenceMatcher(reserve=4)# empty matcher, reserved space for 4 tokens# (not necessary, but speeds up token addition)matcher.append("Hello")# append token (Bigrams are constructed)matcher.append(Bigrams("  "))# append token bigrams directlymatcher.append("world",strip=False)# "strip" token means that no match may begin...matcher.append(" !",strip=True)# nor end by that token# Alternatively, you may pass `Iterable` of tokens directly to the constructor.# If the `Iterable` length can be taken, reservation of space is done; otherwise,# you may still use the `reserve` constructor parameter if you know how many# tokens there shall be...  Again, if you don't, the constructor will handle it anyway# (construction may just take a bit longer).strip=Truematcher=SequenceMatcher(["This",("  ",strip),"uses",("  ",strip),"Sørensen",(" -",strip),"Dice",("  ",strip),"coefficient",(" .",strip),])formatchinmatcher.match(["Sørenson","and","Dice"],0.65):# matchingprint(f"Match begin :{match.begin}")# 4 (index of the 1st match token)print(f"Match end   :{match.end}")# 7 (1 past the last match token)print(f"Match score :{match.score}")# >0.65, <1.0 as it's not a perfect match# You may continue matching other sequences# Note that this is only a quick summary; see `SequenceMatcher` docstrings for more...LicenseThe software is available open-source under the terms of 3-clause BSD
license.AuthorVáclav Krpec <vencik@razdva.cz>