Package: equate
EquateThis is a package with tools for matching things.Dirty things like language, files in your file system, socks and whistles.Installpip install equateMoving on...Little peepMerging/joining tables is very common instance, yet only a small part of what is
possible, and often needed. Consider the following use cases:Find the columns to match (join keys) by comparing how well the values of the column
match.Comparing the values of the columns with something more flexible than hard equality.
For example, correlation, similarity, etc.Find near duplicate columnsFind rows to align, based on flexible comparison of fuzzily matched cellsSimple caseSay you have two sets of strings, and all you do is want to is match each element of
the "keys" set to an element of the "values" set (never reusing the same value for a
different key), and say you know that a matching value string will differ from its
key only by a few characters. In that case just do this:keys=['apple','banana','carrot']values=['car','app','carob','cabana']dict(match_greedily(keys,values))# == {'apple': 'app', 'banana': 'cabana', 'carrot': 'carob'}Note that by default,match_greedilyuses the edit-distance as itsscore_func,
and you can specify your own customscore_func. But the matching is done "greedily".
This means that at every step, the highest-score value will be taken as the match.
Some times this is not ideal (typically, your matches validity will decline along with
the number of keys matched).More involved methodsHere's a more involved entry point:fromequateimportmatch_keys_to_valueskeys=['apple pie','apple crumble','banana split']values=['american pie','big apple','american girl','banana republic']dict(match_keys_to_values(keys,values))# == {'apple pie': 'american pie',# 'apple crumble': 'big apple',# 'banana split': 'banana republic'}The algorithm that gets you these matches is a bit more involved, and parametrizable.
This is how it works.First, it computes a similarity matrix, and then applies a "matcher" that will
search through this similarity matrix for an optimal matching, using any
optimal search function you want (and of course, we provide a few standard ones).The similarity matrix, if full, contains scores for every possible combination of
keys and values.
You can specify how to compute this similarity matrix, which means that if you
don't want to compute the similarity for every combination, you don't need to.
Usually, your similarity matrix will be a "sparse matrix", that is, only specify a
few non-zero entries.The default similarity matrix function uses aobj_to_vectfunction along with a
vector similarity functionsimilarty_functo compute what thescore_funcdid in our firstmatch_greedilyfunction.
By default here, though,similarity_matrixwill use methods that are more powerful
than just an edit-distance. It will learn and use a "TfIdf" vectorization
(a.k.a. embedding) and a cosine similarity function.
As a result, you should get finer matchings.fromequateimportsimilarity_matrixkeys=['apple pie','apple crumble','banana split']values=['american pie','big apple','american girl','banana republic']m=similarity_matrix(keys,values)m.round(2).tolist()# == [[0.54, 0.38, 0.0, 0.0], [0.0, 0.33, 0.0, 0.0], [0.0, 0.0, 0.0, 0.41]]Theequate.utilmodule has a few optimal matching function you can use from
there to extract the matching pairs from the matrix.
At the time of writing this, we've implemented:greedy_matching,hungarian_matching,maximal_matching,stable_marriage_matching, andkuhn_munkres_matching.At the time of writing this, the defaultmatcherused bymatch_keys_to_valuesishungarian_matching.An example: In search of a import-to-package name matcherHere, we'll go through an actual practical example of when you might want to match
things: "Guessing" the pip install name from a pip package name,
and other related analyses.The problemEver got an import error and wondered what the pip install package name was.Say...ImportError: No module named skimageBut it ain'tpip install skimageis it (well, it USED to not to, but you get the point...).
What you actually need to do to install (withpip) is:pip install scikit-imageI would have guessed that!So no, it's annoying. It shouldn't be allowed. And since it is, there should be an index out there to help out, right?pip install --just-find-it-for-me skimageInstead of just complaining, I thought I'd throw some code at it.
(I'll still complain though.)Here's a solution: Ask the world (of semantic clouds -- otherwise known as "Google") about it...A (fun) solutionimportrequestsimportrefromcollectionsimportCountersearch_re=re.compile('(?<=pip install\W)[-\w]+')defpkg_name_options(query):r=requests.get('https://www.google.com/search',params={'q':f'python "pip install"{query}'})ifr.status_code==200:returnCounter(filter(lambdax:x!=query,p.findall(r.content.decode('latin-1')))).most_common()defbest_guess(query):t=pkg_name_options(query)ift:returnt[0][0]>>>pkg_name_options('skimage')[('scikit-image',5),('-e',2),('virtualenv',1),('scikit',1),('scikit-',1),('pillow',1)]>>>best_guess('skimage')'scikit-image'Yay, it works!
With a sample of one!
Let's try two...>>>pkg_name_options('sklearn')[('numpy',3),('scikit-learn',2),('-U',2),('scikit-',1),('scipy',1)]Okay, so it already fails.Sure, I could parse more carefully. I could dig into the webpages and get more scope.That'd be fun.But that's not very nice too Google (and probably is illegal, if anyone cares).What you'll find next is an attempt to look at the man in the mirror instead. Looking locally, where the packages actually are: In the site-packages folders...Extract, analyze and compare site-packages info namesimportpandasaspdimportnumpyasnpfromequate.examples.site_namesimport(DFLT_SITE_PKG_DIR,site_packages_info_df,print_n_null_elements_in_each_column_containing_at_least_one,Lidx,)>>>DFLT_SITE_PKG_DIR'~/.virtualenvs/382/lib/python3.8/site-packages'>>>data=site_packages_info_df()>>>print(f"{data.shape}")(303,8)>>>data<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}</style>dist_info_dirnameinfo_kinddist_namemost_frequent_record_dirnamefirst_line_of_top_level_txtinstallermetadata_namepypi_url_name0xlrd-1.2.0.dist-infodist-infoxlrdxlrdxlrdpipxlrdNone1boltons-20.2.0.dist-infodist-infoboltonsboltonsboltonspipboltonsNone2appdirs-1.4.3.dist-infodist-infoappdirsappdirsappdirspipappdirsNone3yapf-0.29.0.dist-infodist-infoyapfyapftestsyapfpipyapfNone4cmudict-0.4.4.dist-infodist-infocmudictcmudictcmudictpipcmudictNone...........................298simplegeneric-0.8.1.dist-infodist-infosimplegenericsimplegenericsimplegenericpipsimplegenericNone299plotly-4.6.0.dist-infodist-infoplotlyplotly_plotly_future_pipplotlyNone300rsa-3.4.2.dist-infodist-inforsarsarsapiprsaNone301backcall-0.1.0.dist-infodist-infobackcallbackcallbackcallpipbackcallNone302cantools-33.1.1.dist-infodist-infocantoolscantoolscantoolspipcantoolsNone303 rows × 8 columns>>>print_n_null_elements_in_each_column_containing_at_least_one(data)most_frequent_record_dirname:1nullvaluesfirst_line_of_top_level_txt:6nullvaluesinstaller:32nullvaluesmetadata_name:1nullvaluespypi_url_name:255nullvalues>>>lidx=Lidx(data)>>>df=data[lidx.no_nans]>>>print(f"no nan df:{len(df)=}")no_nans:302equal:187dash_underscore_eq:220('equal','dash_underscore_eq'):186>>>lidx=Lidx(df)>>>lidx.print_diagnosis()no_nans:302equal:187dash_underscore_eq:220('equal','dash_underscore_eq'):186>>>lidx=Lidx(df,'first_line_of_top_level_txt')>>>lidx.print_diagnosis()no_nans:297equal:182dash_underscore_eq:214('equal','dash_underscore_eq'):181>>>t=Lidx(df,'most_frequent_record_dirname')>>>tt=Lidx(df,'first_line_of_top_level_txt')>>>sum(t.equal|tt.equal)199>>sum(t.dash_underscore_eq|tt.dash_underscore_eq)233