Package: correlate
correlateA clever brute-force correlator for kinda-messy dataCopyright 2019-2023 by Larry HastingsOverviewcorrelateis a data analysis library.  It's designed to find
matches between two datasets that conceptually represent the
same information, just in different formats.  Its goal is to
tell you "valueAin the first dataset is a good match for
valueBin the second dataset".To usecorrelate, you feed in the two datasets of (opaque) values
and their associated metadata information.correlateuses the
metadata to find matches between the two datasets, ranks these matches
using a unique scoring heuristic, and returns the matches.Quick StartThis code:importcorrelatec=correlate.Correlator()a,b=c.datasetsgreg,carol,tony,steve="greg carol tony steve".split()Greg,Carol,Tony,Steve="Greg Carol Tony Steve".split()a.set("this",greg)a.set("is",greg)a.set("Greg",greg,weight=5)a.set_keys("Carol over here".split(),carol)a.set_keys("My name is Tony".split(),tony)a.set_keys("Hi I'm Steve".split(),steve,weight=2)b.set_keys("gosh my name is Greg".split(),Greg)b.set_keys("Carol is my name".split(),Carol)b.set_keys("Pretty sure I'm still Tony".split(),Tony)b.set_keys("I'm Steve".split(),Steve)result=c.correlate()formatchinresult.matches:print(f"{match.score:1.3f}{match.value_a:>5}->{match.value_b}")produces this output:5.750  greg -> Greg
3.800 steve -> Steve
1.286 carol -> Carol
1.222  tony -> TonyA Real-Life ExampleThere's a podcast I like.  I download it as MP3 files
using an RSS feed, 1990s-style. But the metadata in the RSS
feed is junk--the episode titles are inconsistent,
and the episode numbers are almost wholly absent.This podcast also has a
list of episodes on its website. This data ismuchcleaner,
including clean, proper episode numbers.  And it's easily
scraped.  But it's still not perfect.
The two lists of episodes aren't exactly the same, and even the episodes
that are present in both are sometimes reordered.Obviously, I want to take the MP3s from the RSS feed,
and match them up with the nice clean metadata scraped from the website.
This gets me the best of both worlds.But there are more thanseven hundredepisodes of this
particular podcast!  Matching those by hand would be alotof work.
And we get a new episode every week.
And sometimes they actually add back in old episodes, or update the
metadata on old episodes--changes which would mess up any hand-built
ordering.  And I might want to listen to more than one podcast from
this same website someday!
So I really didn't want to do all of this by hand.Happily, after applying just a bit of intelligence to the two
datasets,correlatedid a perfect job.Why correlate Works So WellThe insight that inspiredcorrelateis this:
unique keys in the two datasets are probably very good matches.
Let's say the key"egyptian"maps to valueA1in one dataset
and valueB1in the other dataset--and itonlymaps to those two
values.  In that case,A1andB1are probably a match.This leads to a virtuous cycle.  Let's say the word"nickel"maps to two values in each of the two datasets:A1andA2,
andB1andB2.  We have two options to match those four values:A1->B1 and A2->B2
or
A1->B2 and A2->B1But the key"egyptian"already told usA1andB1is
a good match.  Since we've already matched those, that
eliminates the second option.  Our only remaining choice is matchingA2->B2.  Choosing a good match based on"egyptian"helped
us to eliminated extraneous possibilities and make a good choice
based on"nickel", too.In short,correlatecapitalizes on therelative uniquenessof keys.Getting Started With correlateRequirementscorrelaterequires Python 3.6 or newer.  It has no
other dependencies.If you want to run thecorrelatetest suite,
you'll need to install therapidfuzzpackage.
(rapidfuzzis a fuzzy string matching library.
It's a lot likefuzzywuzzy, exceptrapidfuzzis MIT licensed, wherasfuzzywuzzyis GPL
licensed.)The High-Level Conceptual ModelTo correlate two datasets withcorrelate,
you first create acorrelate.Correlatorobject.
This object contains two membersdataset_aanddataset_b; these represent
the two datasets you want to correlate.You fill each dataset withkeysandvalues.Avalueis (nearly) any Python object.
Each value should represent one value from your dataset.correlatedoesn't examine your values--they're completely
opaque tocorrelate.Akeyis a Python object that represents some metadata
about a value.  Keys "map" to values;correlateexamines
keys, using matching keys in the two datasets to
match up the values between them.Keys are usually strings--for example, the individual words
from the title of some movie, TV show, song, or book.
But keys don't have to be strings.  Instances of lots of
Python data types can be used as keys; they just need to
be hashable.Once you've filled in thecorrelate.Correlatorobject
with your data, you call itscorrelate()method.  This
computes the matches.  It returns acorrelate.CorrelateResultcontaining
those matches, and lists of any objects from
the two datasets that didn't get matched.The matches are returned as a list ofcorrelator.CorrelatorMatchobjects.  Each object contains three members:value_a, a reference to an object fromdataset_a,value_b, a reference to an object fromdataset_b,and a floating-pointscore.EachCorrelatorMatchobject tells you thatcorrelatorthinks thisvalue_amaps to thisvalue_b.
Thescoreis a sort of mathematical confidence level--it's
a direct result of the keys and other metadata you provide
tocorrelate.  The list of matches is sorted byscore--higher scores first, as higher scores represent
higher confidence in the match.That's the basics.  Butcorrelatesupports some very sophisticated
behavior:When mapping a key to a value, you may specify an optionalweight,
which represents the relative importance of this key.  The default
weight is 1.  Higher scores indicate a higher significance; a weight of
2 tellscorrelatethat this key mapped to this value is twice as
significant.  (Weight is an attribute of a particular mapping of a key
to a value--an "edge" in the graph of mapping keys to values.)A key can map to a value multiple times.  Each mapping can have its own weight.If both datasets are ordered, this ordering can optionally influence the match scores.correlatecalls thisranking.Ranking is an attribute of values, not keys.Keys can be "fuzzy", meaning two keys can be a partial match rather than a binary yes/no.
Fuzzy keys incorrelatemust inherit from a custom abstract base class calledcorrelate.FuzzyKey.Sample Code And infer_mvcorrelateships with some sample code for your reading pleasure.
The hope is it'll help you get a feel for what it's like to usecorrelate.
Take a look at the scripts in thetestsandutilitiesdirectories.In particular,utilitiescontains a script calledinfer_mv.infer_mvtakes a source directory and a list of files and directories
to rename, and produces a mapping from the former to the latter.
In other words, when you run it, you're saying"here's a source directory and a list of files and directories to rename.
For each file in the list of things to rename, find the
filename in the source directory that most closely resembles that file,
and rename the file so it's exactly like that other filename from the
source directory."(If you askinfer_mvto rename a directory,
it renames all the files and directories inside that directory, recursively.)This is useful if, for example, you have a directory where
you've already renamed the files the way you you like them, but then
you get a fresh copy from somewhere.  Simply runinfer_mvwith your
existing directory as the "source directory" and the fresh copy
as the "files".infer_mvwill figure out how to rename the
fresh files so they have the filenames how you like them.Note thatinfer_mvdoesn't actually do the work of renaming!
Instead,infer_mvprints out ashell scriptthat, if executed,
performs the renames.
Why?  It's always a good idea to check over the output ofcorrelatebefore you commit to it.I useinfer_mvlike so:% infer_mv ../old_path *
# look at output, if it's all fine run
% infer_mv ../old_path * | shOr you can direct the output ofinfer_mvinto a file, then
edit the file, then execute that.  Or something else!
Whatever works for you!Terminology And RequirementsValuesValues are Python objects that represent individual elements of your two
datasets.correlatedoesn't examine values, and it makes very few
demands on them.  Here are the rules for values:Values must support==.Value comparison must bereflexive,symmetric,transitive,andconsistent.
For all these examples,abandcrepresent values:reflexive:A value must always compare as equal to itself.a == amust evaluate toTrue.symmetric:Ifa == bisTrue, thenb == amust also beTrue.transitive:Ifa == bisTrue, andb == cisTrue, thena == cmust also beTrue.consistent:Ifa == bisTrue, it must always beTrue,
and ifa == bisFalseit must always beFalse.KeysKeys are Python objects thatcorrelateuses to find matches between
the two datasets.  If a key maps to a value indataset_aand also
a value indataset_b, those two values might be a good match.Keys must obey all the same rules as values.  In addition,
keys must behashable.This in turn requires that keys
must beimmutable.Exact KeysAn "exact" key is whatcorrelatecalls any key that isn't a "fuzzy" key.
Strings, integers, floats, complex,datetimeobjects--they're all fine to use
ascorrelatekeys, and instances of many more types too.When considering matches, exact keys are binary--either they're an exact match
or they don't match at all.  To work with non-exact matches you'll have
to use "fuzzy" keys.Fuzzy KeysA "fuzzy" key is a key that supports a special protocol for performing "fuzzy"
comparisons--comparisons where the result can represent imperfect or partial matches.Technically speaking, acorrelate"fuzzy" key
is an instance of a subclass ofcorrelate.FuzzyKey.  If a key is an instance of
a subclass of that base class, it's a "fuzzy" key, and if it isn't, it's an "exact" key.Fuzzy keys must follow the rules for keys above.
Also, the type of your fuzzy keys must also obey the same rules as keys;
they must be hashable, they must support==,
and their comparison must be reflexive, symmetric, transitive, and consistent.In addition, fuzzy keys must support a method calledcomparewith this signature:self.compare(other).otherwill be another fuzzy key of the same type.  Yourcomparefunction should return a number (eitherintorfloat) between (and including)0and1, indicating how close a matchselfis toother.  Ifcomparereturns1, it's saying this is a perfect match, that the two values are identical;
if it returns0, it's a perfect mismatch, tellingcorrelatethat the two keys
have nothing in common.correlaterequires thatcomparealso obey the four mathematical constraints required
of comparisons between keys.  In the following rules,aandbare fuzzy keys of the
same type.comparemust conform to these familiar four rules:reflexive:a.compare(a)must return1(or1.0).symmetric:Ifa.compare(b)returnsx, thenb.compare(a)must also returnx.transitive:Ifa.compare(b)returnsx, andb.compare(c)returnsx,
thena.compare(c)must also returnx.consistent:Ifa.compare(b)returnsx, it mustalwaysreturnx.It's important to note: fuzzy keys of twodifferenttypes are automatically
considered different to each other.correlatewon't even bother callingcompare()on them--it automatically assigns the comparison a fuzzy score of0.
This is true even for subclasses; if you declareclass MyFuzzyNumber(correlate.FuzzyKey)and alsoclass MyFuzzyInteger(MyFuzzyNumber),correlatewill never compare an instance ofMyFuzzyNumberandMyFuzzyKeyto each other--it automatically assumes they have nothing in common.(Internallycorrelatestores fuzzy keys of different types segregated from each other.
This is a vital optimization!)On a related note,correlatemay optionally neveractuallycalla.compare(a), either.  That is, if the exact same key
maps to a value in bothdataset_aanddataset_b,correlateis permitted to skip callingcompare()and instead automatically
assign the comparison a fuzzy score of1.  (Currently if this situation
arose itwouldcalla.compare(a), but that wasn't true at various
times during development.)Finally, it's important to note that fuzzy keys are dramatically slower
than exact keys.  If you can express your problem purely using exact keys,
you should do so!  It'll run faster as a result.  You can get a sense of
the speed difference by runningtests/ytjd.test.pywith verbose mode
on (-v). A test using the same corpus but switching everything to fuzzy
keys runs about12x sloweron my computer.APICorrelator(default_weight=1)The correlator class.default_weightis the weight used
when you map a key to a value without specifying an explicit weight.Correlator.dataset_aCorrelator.dataset_bInstances ofCorrelator.Datasetobjects representing the two sets of data you want to correlate.  Initially empty.Correlator.datasetsA list containing the two datasets:[dataset_a, dataset_b]Correlator.correlate(*, minimum_score=0, score_ratio_bonus=1, ranking=BestRanking, ranking_bonus=0, ranking_factor=0, reuse_a=False, reuse_b=False)Correlates the two datasets.  Returns acorrelate.CorrelatorResultobject.minimum_scoreis the minimum permissible score for a match.  It must be
greater than or equal to 0.score_ratio_bonusspecifies the weight of a bonus awarded to a match, based on the ratio of
the actual score computed between these two values divided by the maximum possible score.
For more information, consult theScore Ratio Bonussection.rankingspecifies which approch to computing rankingcorrelateshould use.
The default value ofBestRankingmeanscorrelatewill try all approaches
and choose the one with the highest cumulative score across all matches.
Other values includeAbsoluteRankingandRelativeRanking.ranking_bonusspecifies the weight of the bonus awarded to a match
based on the proximity of the two values in their respective datasets, as specified
by their rankings.  The closer the two values are to the same position in their
respective datasets, the higher a percentage of theranking_bonuswill be awarded.ranking_factorspecifies the ratio of the base score of a match that is multiplied
by the proximity of the two values in their respective datasets.  If you uesranking_factor=0.4,
then a match only automatically keeps 60% of its original score; some percentage
of the remaining 40% will be re-awarded based on the proximity of the two values.(You can't use both a nonzeroranking_bonusand a nonzeroranking_factorin the
same correlation.  Pick at most one!)For more information on all these ranking-related parameters,
consult theRankingsection of this document.reuse_apermits values indataset_ato be matched to more than one value indataset_b.reuse_bis the same but for values indataset_bmatchingdataset_a.
If you set both reuse flags to True, thecorrelate.CorrelatorResult.matcheslist returned will containeverypossible match.Correlator.print_datasets()Prints both datasets in a human-readable form.  Usesself.printto print,
which defaults toprint.Correlate.Dataset()The class for objects representing a dataset.  Behaves somewhat like
a write-only dict.Correlator.Dataset.set(key, value, weight=default_weight)Adds a new correlation.You can useDataset[key] = valueas a shortcut forDataset.set(key, value).Correlator.Dataset.set_keys(keys, value, weight=default_weight)Map multiple keys to a single value, all using the same weight.keysmust be an iterable containing keys.Correlator.Dataset.value(value, *, ranking=None)Annotates a value with extra metadata.  Currently only one metadatum
is supported:ranking.rankingrepresents the position of this value in the dataset,
if the dataset is ordered.rankingshould be an integer
representing the ranking; if this value is the 19th in the dataset,
you should supplyranking=19.Correlator.Dataset.clear(default_weight=1, *, id=None)Clears the Dataset.  Reinitializes it to its empty state.
You can optionally also change the dataset's default weight
and its id.CorrelatorResult()The class for objects returned byCorrelator.correlate().
Contains four members:matches, alistofCorrelatorMatch()objects, sorted with highest score firstunmatched_a, thesetof values fromdataset_athat were not matchedunmatched_b, thesetof values fromdataset_bthat were not matchedstatistics, adictcontaining human-readable statistics about the correlationCorrelatorResult.normalize(high=None, low=None)Normalizes the scores inmatches.
Whennormalize()is called with its default values, it adjusts every score
so that they fall in the range(0, 1].
Ifhighis not specified, it defaults to the highest score inmatches.
Iflowis not specified, it defaults to theminimum_scoreused for the correlation.CorrelatorMatch()The class for objects representing an individual match made
byCorrelator.correlate().
Contains three members:value_a, a value fromdataset_a.value_b, a value fromdataset_b.score, a number representing the confidence in this match.
The higher thescore, the higher the confidence.
Scores don't have a predefined intrinsic meaning; they're a result
of all the inputs tocorrelate.Correlator.str_to_keys(s)A convenience function.
Converts stringsinto a list of string keys using a reasonable approach.
Lowercases the string, converts some common punctuation into spaces, then splits
the string at whitespace boundaries.  Returns a list of strings.Getting Good Results Out Of CorrelateUnfortunately, you can't always expect perfect results withcorrelateevery time.  You'll usually have to play with it at least a little.
At its heart,correlateis a heuristic, not an exact technology.
It often requires a bit of tuning before it produces the results you want.RankingNaturally, the first step withcorrelateis to plug in your data.
I strongly encourage you to add ranking information if possible.If the two datasets are ordered, and equivalent items should appear in
roughly the same place in each of the two datasets, ranking information
can make asizeableimprovement in the quality of your matches.
To use ranking information, you set therankingfor each value in each dataset
that you can, and specify eitherranking_bonusorranking_factorwhen
runningcorrelate().
Which one you use kind of depends on how much confidence you have in the
ordering of your datasets.  If you think your ranking information is pretty
accurate, you should definitely useranking_factor; this exerts a much
stronger influence on the matches.
If you have a low confidence in the ordering of your datasets,
chooseranking_bonus, which only provides a little nudge.Ranking can also speed upcorrelatequite a bit.  If there are a
lot of matches that end up with the same score, this can create a lot
of work for the "match boiler" (see below), and that can get expensive
quick.  Even a gentle nudge from ranking information can help differentiate
scores enough to result in adramaticspeedup.Minimum ScoreOnce you've plugged in all your data, you should run the correlation,
print out the result in sorted order with the best
matches on top, then scroll to the bottom and see what theworst5% or 10% of matches look like.If literally all your matches are already perfect--congratulations!
You'realreadygetting good results out ofcorrelateand you
can stop reading here.  But if you're not that lucky, you've got more
work to do.The first step in cleaning upcorrelate'soutput is usually
to stop it from making bad matches by setting aminimum_score.When you have bad matches, it's usually because the two datasets don't map
perfectly to each other.  If there's a value indataset_athat has no good
match indataset_b, well,correlatedoesn't really have a way of
knowing that.   So it may match that value to something anyway.Look at it this way: the goal ofcorrelateis to find matches between
the two datasets.  If it's made all the good matches it can, and there's
only one item left in each of the the two datasets, and they haveanythingin common at all,correlatewill match those two values together
out of sheer desparation.However!  Bad matches like these tend to have a very low score.
So all those low-scoring bad matches clump together at the very
bottom.  There'll probably be an inflection point where the scores
drop off significantly and the matches go from good to bad.This is whatminimum_scoreis for.minimum_scoretellscorrelatethe minimum permissible score for a match.  When you have a clump of bad
matches at the bottom, you simply setminimum_scoreto be somewhere
between the highest bad match and the lowest good match--and
behold!  No more bad matches!  The values that were used in the bad
matches will move tounused_aandunused_b, which is almost
certainly the correct place for them.(Technically,minimum_scoreisn'tactuallythe minimum score.
It's ever-so-slightlylessthan the lowest
permitted score.  As in, for a match to be considered viable, its score
must begreater thanminimum_score.In Python 3.9+, you can
express this concept as:actual_minimum_score = math.nextafter(minimum.score, math.inf)The default value forminimum_scoreis 0, which meanscorrelatewill keep
any match with a positive score.)Unfortunately it's hard to predict what to setminimum_scoreto in advance.
Its value really depends on your data set--how many keys you have, how good
the matches are, what weights you're using, everything.  It's much more
straightforward to run the correlation, look over the output, find where the
correlations turn bad, and set a minimum score.  With large data sets there's
generally a sudden and obvious dropoff in score, associated withcorrelatemaking poor matches.  That makes it pretty easy: set the minimum score so it
keeps the last good match and forgets the rest.  But there's no predicting what
that score will be in advance--every data set is different, and it's really
an emergent property of your keys and weights--so
you'll have to calibrate it correctly for each correlation you run.(Sometimes there are good matches mixed in with the bad ones at the bottom.
When that happens, the first step is generally to fixthat,so that the
bad ones are all clumped together at the bottom.  I can't give any general-purpose
advice on what to do here.  All I can suggest is to start experimenting.  Change
your keys, adjust your weights, run the correlation again and see what happens.
Usually when I do this, I realize something I can do to improve the data I feed
in tocorrelate, and I can fix the problem externally.)WeightsIf you're still not getting the results you want, the next adjustment you
should consider is increasing the weight of
keys that provide a clear signal.  If the datasets you're comparing
have some sort of unique identifier associated with each value--like
an episode number, or release date--you should experiment with giving those
keys a heavier weight.  Heavily-weighted keys like this can helpcorrelatezero in on the best matches right away.It's up to you what that weight
should be; I sometimes use weights as heavy as 5 for super-important keys,
which means this one single key will have the same weight as 5 normal
keys.  Note that a weight of 5 on the mapping indataset_aanddataset_bmeans that, if those keys match, they'll have a base score
of 25!  If that key only appears once in each dataset, that will almostcertainlyresult in a match.But using weights can be a dual-edged sword.  If your data has mistakes
in it, heavily weighting the bad data can magnify those mistakes.  One bad
heavily-weighted key on the wrong value can inflate the score of a bad match
over the correct match.  And that can result in a vicious cycle--if valueA1should match valueB1, but it get mapped to valueB43instead,
that meansB1is probably going to get mismatched too.  Which deprives
another value of its correct match.  And so on and so on and so on.One final note on weights.  The weight of a key doesn't affect howdesirableit is in a match, it only affects the resulting score of that
match.  Consider this scenario involving weighted fuzzy keys:FA1 and FA2 are fuzzy keys in dataset_a
FB is a fuzzy key in dataset_b
VA1 and VA2 are values in dataset_a

dataset_a.set(FA1, VA1, weight=1)
dataset_a.set(FA2, VA2, weight=5)

FA1.compare(FB) == 0.4
FA2.compare(FB) == 0.2Ifcorrelatehad to choose between these two matches, which one
will it prefer?  It'll preferFA1->FB, becausecorrelatedoesn't consider weights when considering matches.  It always prefers
the match with the higherunweightedscore.  It's true,
matchingFA2toFBresults in a higher final score once you
factor in the weights.  But that doesn't make it a better match.The best way to conceptualize this: weights don't make matcheshigher quality,they just make matches moreinterestingwhen true.Too-Common KeysSimilarly, if there are super-common keys that aren't going to help with
the correlation, consider throwing them away and not even feeding them in as
data.  Keys that map to most or all of the values in a dataset add little
clarity, and will mainly serve just to makecorrelateslower.
I usually throw away the word "The", and the name of the podcast or show.
(When correlating filenames, I may throw away the file extension too.)Then again, often leaving them in won't hurt anything, and it can occasionally
be helpful!  The waycorrelateworks, it considers multiple maps of a
key to a value as different things--if you map the key"The"to a value
twice,correlateunderstands that those are two separate mappings.
And if there's only one value in each dataset that has two"The"mappings,
that can be a very strong signal indeed.  So it's really up to you.
Throwing away largely-redundant keys is a speed optimization, but it
shouldn't affect the quality of your matches.Note thatcorrelateis nowveryefficient when it comes to
matching with exact keys.  For most people, the additional runtime cost
for redundant or common keys is probably negligible, and not worth the
additional development time or engoing support cost to make it even worth
considering.  It's true they provide only a tiny amount of signal--but they
also have relatively little runtime cost, either in memory or CPU time.
At this point it'sprobablynot worth the bother to almost anybody.(But here's a theoretical best of both worlds to consider: for very
common keys, consider throwing away thefirstinstance.  I admit I
haven't tried this experiment myself.)Check Your InputsAs always, it's helpful to make sure your code is doing what you intend it to.
Several times I've goofed up the mechanism I use to feed data sets intocorrelate; for example, instead of feeding in words as keys, I've occasionally
fed in the individual characters in those words as keys.  (Like, instead of
the single key"booze", I accidentally fed in the five keys'b','o','o','z', and'e'.)
However, thecorrelatealgorithm works so well,
it still did a shockingly good job!  (Though it was alotslower.)I've learned to double-check that I'm inputting the mappings and weights I meant
to, with a debugger or withCorrelator.print_datasets().  Making sure you gavecorrelatethe right data can make it not only much more accurate,
it might make it faster too!Normalize StringsWhen using strings as keys from real-world sources, I recommend younormalizethe strings:
lowercase the strings,
remove most or all punctuation,
and
break the strings up into individual keys at word boundaries.
In the real world, punctuation
and capitalization can both be inconsistent, so throwing it away can help
dispel those sorts of inconsistencies.correlateprovides a utility function calledcorrelate.str_to_keys()that does this
for you.  But you can use any approach to string normalizing you like.You might also considerinterningyour strings.  In my limited experimentation
this provided a small but measurable speedup.Sharpen Your Fuzzy KeysIf you're using fuzzy keys, make sure yousharpenyour fuzzy keys.
Fuzzy string-matching libraries have a naughty habit of scoring
not-very-similar strings as notthatmuch less than almost-exactly-the-same
strings.  If you give that data unaltered tocorrelate,that "everything
looks roughly the same" outlook will be reflected in your results as
mediocre matches.In general, you want to force your fuzzy matches to extremes.
Two good techniques:Specify a minimum score for fuzzy matches, and replace any fuzzy score
below that minimum with0.Possibly remap the remaining range to the entire range.
For example, if your minimum score is0.6, should you simply
return values from0.6to1?  Or should you stretch the scores
over the entire range with(fuzzy_score - 0.6) / (1 - 0.6)?
You may need to experiment with both to find out what works well for you.Multiply your fuzzy score by itself.  Squaring or even cubing a fuzzy
score will preserve high scores and attenuate low scores.
However, note that the scoring algorithm for fuzzy key matches alreadycubesthe fuzzy score.  Additional multiplying of the score by itself is
probably unnecessary in most cases.What Do These Scores Mean?The scores you seee in the results are directly related to the data you
gave tocorrelate.  The scores really only have as much or as
little meaning as you assign to them.If you don't enjoy the unpredictable nature ofcorrelatescores,
consider callingnormalize()on your Correlate result object.
This normalizes the scores as follows: the highest score measured
will be adjusted to 1.0,minimum_scorewill be adjusted to 0.0,
and every other score will be adjusted linearly between those two.Mathematically:score = the original score for this match
highest_score = highest score of any match
minimum_score = the minimum_score passed in to correlate()
delta = highest_score - minimum_score
normalized_score = (score - minimum_score) / deltaImplementation Notes On The Algorithm And The CodeIf the implementation is hard to explain, it's a bad idea.
--The Zen Of Pythonby Tim PetersWhat follows is an exhaustive (and exhausting!) chapter
on the implementation ofcorrelate.  This is here
partially for posterity, partially because I like
reading this sort of thing in other people's projects,
but mostly to make it easier to reaquaint myself with
the code when I have to fix a bug three years from now.The High-Level OverviewAt the heart ofcorrelateis a brute-force algorithm.  It's what
computer scientists would call anO(n²) algorithm.correlatecomputes every possible "match"--every mapping of a value indataset_ato a value indataset_bwhere the two values have keys in common.
For exact keys, it uses set intersections to ignore pairs of values that have
nothing in common, discarding early matches it knows will have a score of 0.
Sadly, it can't do that for fuzzy keys, which is why fuzzy keys tend to
noticably slow downcorrelate.For each key that matches between the two values,correlatecomputes a score.  It then adds all those scores together,
computing the final cumulative score for the "match",
which it may modifiy based on the various bonuses and factors.
It then iterates over these scores in sorted order, highest score first.
For every match where neither of the two values have been used
in a match yet, it counts that as a "match" and adds it to the output.
(This assumesreuse_aandreuse_bare bothFalse.  Also, this
is a little bit of an oversimplification; see theMatch Boilersection below.)One important detail:correlatestrives to be 100%
deterministic.  Randomness can creep in around the edges in Python
programs; for example, if you ever iterate over a dictionary,
the order you will see the keys will vary from run to run.correlateeliminates every source of randomness it can.
As far as I can tell: given the exact same inputs, it performs
the same operations in the same order and produces the same result,
every time.There are a number of concepts involved with how thecorrelatealgorithm works, each of which I'll explain in exhausting detail
in the following sub-sections.correlate'sSix Passes And Big-O ComplexityA singlecorrelatecorrelation makes six passes over its data.
Here's a high-level overview of those passes, followed by
deep-dives into the new terms and technical details of those passes.Pass 1Iterate over both datasets and compute the "streamlined"
data.Complexity:O(n)Pass 2Iterate over all keys and compute a sorted list of all matches
that could possibly have a nonzero score.  (The list represents
a match with a pair of indices into the lists of values for each
dataset.)  This pass also performs all fuzzy key comparisons and
caches their results.Complexity:O(n²), for the fuzzy key comparisons step.
For acorrelaterun with a lot of fuzzy keys, this is often
the slowest part of the run.  If your correlate is mostly
exact keys, this part will be pretty quick, because all theO(n²) work is done in C code (set intersections, and sorting).Pass 3For every match with a nonzero score,
compute subtotals for matching all fuzzy keys.
We need to add some of these together to compute the final
scores for fuzzy key matches.Complexity:O(n²)Pass 4For every match with a nonzero score:compute the scores for matching all exact keys,finalize the scores for fuzzy key match scores,compute the bonuses (score_ratio_bonus, ranking),and store the result per-ranking.The score for each match is now finalized.Complexity:O(n²)Pass 5For every ranking approach being used,
compute the final list of successful matches,
using the "match boiler" and "greedy algorithm".Complexity:O(n log n) (approximate)Pass 6Choose the highest-scoring ranking approach,
compute unseen_a and unseen_b,
and back-substitute the "indexes" with their actual values
before returning.Complexity:O(n)Thus the big-O notation forcorrelateoverall isO(n²).
The slowest part ofcorrelateis processing lots of fuzzy
keys; if you can stick mostly to exact keys, yourcorrelateruns will be a lot quicker.You can see how longcorrelatespent in each of these
passes by examining thestatisticmember of theCorrelatorResultobject.  This is a dict mapping string descriptions of
passes to a floating-point number of seconds.  Pass 2's
sub-passes dealing with exact keys and fuzzy keys are
broken out separately, as is the "match boiler" phase
of Pass 5.RoundsIf you callcorrelateas follows:c = correlate.Correlator()
o = object()
c.dataset_a.set('a', o)
c.dataset_a.set('a', o)then key'a'reallyismapped to valueotwice,
and those two mappings can have different weights.
Technically, the correct way to think of this is as
having two edges from the same key to the same value
in the dataset graph.
Another way to think of it is to consider repeated keys as
being two different keys--identical, but somehow distinct.(If it helps, you can also think of it as being
like two files with the same filename in two different
directories.  They have the samefilename,but
they're not the samefile.)correlatecalls groups of these multiple mappings"rounds".
A "round" contains all the keys from the Nth time they were
repeated.  Round 0 contains every key, round 1 contains the second
instances of all the keys that were repeated twice, round 2 contains
all the third instances of all the keys that were repeated three times,
etc.
Rounds are per-value, and there are as
many rounds as the maximum number of redundant mappings of
any single key to any particular value in a dataset.Naturally, exact keys and fuzzy keys use a different method
to determine whether or not something is "the same key".
Technically both types of keys use==to determine equivalence.
However, fuzzy keys don't implement a custom__eq__, so Python
uses its default mechanism to determine equivalence, which is
really just theisoperator.  Therefore: exact keys are the
same if==says they're the same, and (in practice) fuzzy keys
are the same if and only if they're the same object.(Of course, you could implement your own__eq__when you write
your own fuzzy subclasses.  But I don't know why you would bother.)Consider this example:c = correlate.Correlator()
o = object()
c.dataset_a.set('a', o, weight=1)
c.dataset_a.set('a', o, weight=3)
c.dataset_a.set('a', o, weight=5)
c.dataset_a.set('b', o)
c.dataset_a.set('b', o)
c.dataset_a.set('c', o)

o2 = object()
c.dataset_b.set('d', o2)
c.dataset_b.set('d', o2)
c.dataset_b.set('e', o2)
c.dataset_b.set('f', o2)Here, the valueoindataset_awould have three rounds:Round 0 would contain the keys{'a', 'b', 'c'}.Round 1 would contain the keys{'a', 'b'}.Round 2 would contain only one key,{'a'}.Ando2indataset_bwould have only two rounds:Round 0 would contain the keys{'d', 'e', 'f'}.Round 1 would contain only one key,{'d'}.Again, conceptually, the"a"in round 0 is a different key
from the"a"in round 1, and so on.For exact keys, rounds are directly matched iteratively
to each other; the exact keys in round 0 for a value indataset_aare matched to the round 0 exact keys for a value indataset_b, round 1 indataset_ais matched to
round 1 indataset_b, and so on.
If one side runs out of rounds early, you stop; if you compute
the intersection of a round and they have nothing in common,
you stop.One invariant property: each subsequent round has a subset of
the keys before it.  The set of keys in roundN+1mustbe a subset of the keys in roundN.  (Though not necessarily
astrictsubset.)What about weights?  Higher weights are sorted to lower rounds.
The weight for a keykin roundN-1mustbe greater than
or equal to the weight ofkin roundN.
In the above example, the'a'in round 0 has weight 5, in round 1
it has weight 3, and in round 2 it has weight 1.
(It doesn't matter what order you insert them in,correlateautomatically sorts the weights as you add the redundant mappings.)Thus, round 0 always contains every exact key mapped to
a particular value, with the highest weights for each of those
mappings.Rounds can definitely help find the best matches.  If the
key"The"maps to most of your values once,
that's not particularly interesting, and it won't affect the scores
very much one way or another.  But if there's only one
value in each dataset that"The"maps totwice,that's a very strong signal indeed!correlatedoes anexcellentjob of noticing unique-ness like that and factoring
it into the scoring.Streamlined DataThecorrelatedatasets store data in a format designed
to eliminate redundancy and be easy to modify.  But this representation
is inconvenient for performing the actual correlate.  Therefore, the first
step ("Pass 1") is to reprocess the data into a "streamlined" format.
This is an internal-only implementation detail, and in fact the data
is thrown away at the end of each correlation.  As an end-user you'll
never have to deal with it.  It's only documented here just in case you
ever need to understand the implementation ofcorrelate.This streamlined data representation is an important optimization.
It greatly speeds up computing a match between two values.  And it
only costs a little overhead, compared to all that matching work.
Consider: if you have 600 values indataset_aand 600 values indataset_b,correlatewill recompute 1,200 streamlined datasets.
But it'll then use it in as many as 360,000 comparisons!   Spending
a little time precomputing the data in a convenient format saves a
lot of time in the long run.The format of the streamlined data changes as the implementation
changes.  And since it's an internal-only detail, it's largely undocumented
here.  If you need more information, you'll just have to read the code.
Search the code for the wordstreamlined.The Scoring Formula, And Conservation Of ScoreFor each match it considers,correlatecomputes the intersection of
the keys that map to each of those two values in the two datasets,
then computes a score based on each of those key matches.
This scoring formula is the heart ofcorrelate, and it was a key
insight--without itcorrelatewouldn't work nearly as well as it does.In the abstract, it looks like this:for value_a in dataset_a:
    for value_b in dataset_b:
        subtotal_score = 0
        for key_a, weight_a that maps to value_a:
            for key_b, weight_b that maps to value_b:
                score = value of key_a compared to key_b
                cumulative_a = the sum of all scores resulting from key_a mapping to any value in dataset_b
                cumulative_b = the sum of all scores resulting from key_b mapping to any value in dataset_a
                score_ratio_a = score / cumulative_a
                score_ratio_b = score / cumulative_b
                unweighted_score = score * score_ratio_a * score_ratio_b
                score_a = weight_a * unweighted_score_a
                score_b = weight_b * unweighted_score_b
                final_score = score_a * score_b
                subtotal_score += final_scoreTwo notes before we continue:key_aandkey_bmust always beper-round,for a number
of reasons, the least of which is because we use their
weights in computing thefinal_score.subtotal_scoreis possibly further adjusted byscore_ratio_bonusand ranking, if used.  We'll discuss that later.This formula is howcorrelatecomputes a mathematical representation
of "uniqueness".  The fewer values a key maps to in a dataset,
the higher it scores.  A key that's only mapped once in each
dataset scores 4x higher than a key mapped twice in each dataset.This scoring formula has a virtuous-feeling mathematical
property I call"conservation of score".Each key that
you add to a round in a dataset adds 1 to the total cumulative
score of all possible matches; when you map a key to multiple
values, you divide this score up evenly between those values.
For example, if the keyxis mapped to three values indataset_aand four values indataset_b, each of those possible matches
only gets 1/12 of that key's score, and the final cumulative
score for all matches only goes up by 1/12.  So a key always
adds 1 to the sum of all scores across allpossiblematches,
but only increases the actual final score by the amount of
signal itactuallycommunicates.Also, now you see why repeated keys can be so interesting.
They add 1 foreach roundthey're in, but that score is only
divided by the number of values they're mapped toin that round!Since there tend to be fewer and fewer uses of a key in
subsequent rounds, the few keys that make it to later rounds
can potentially score much higher than they did in earlier
rounds, making them a more noteworthy signal.Matching And Scoring Exact KeysThe "streamlined" data format for exact keys looks like this:exact_rounds[index][round] = (set(d), d)That is, it's indexed by "index" (which represents the value), then
by round number.  That gives you a tuple containing a dict mapping
keys to weights, and aset()of just the keys.correlateusesset.intersection()(which is super fast!)
to find the set of exact keys the two values have in common for that round.
Thelen()of this resulting set is the base cumulative score for that round,
although that number is only directly useful in computingscore_ratio_bonus.Althoughcorrelateuses the same scoring formula for both exact keys and
fuzzy keys in an abstract sense, scoring matches between exact keys is much
simpler in practice.  Let's tailor the "abstract" scoring algorithm above for
exact keys.  This lets us optimize the algorithm in a couple places, making
it much faster!First, with exact keys, naturally they're either an exact match or they aren't.
If they're an exact match, they're the same Python value.  Thereforekey_aandkey_bmust be identical.  Therefore, conceptually, we can swap them.  Let's
rewrite the "scoring formula" equation slightly:cumulative_a = the sum of all scores between key_b and all keys in dataset_b
cumulative_b = the sum of all scores between key_a and all keys in dataset_aAll we've changed is: we've swappedkey_aandkey_b.
(Why?  It'll help.  Hey, keep reading.)Now consider:scorefor exact keys is always either1or0.  It's1when
two keys are exactly the same, and0otherwise.  If the basescorefor the
match is0, then thefinal_scorewill be0and we can skip all of it.
So we only ever compute afinal_scorewhenscoreis1, when the keys
are identical.Sincescoreis only ever used as a multiplier, we can discard it.cumulative_aandcumulative_bare similarly easy to compute.
They're just the number of times that key is mapped to any value in
the relevant dataset,in that round.These counts are precomputed
and stored in the "streamlined" data.So, finally: if you do the substitutions, and drop out the constantscorefactors,final_scorefor exact keys is computed like this:final_score = (weight_a * weight_b) / (cumulative_a * cumulative_b)Which we can rearrange into:final_score = (weight_a / cumulative_b) * (weight_b / cumulative_a)At the point we precompute the streamlined data fordataset_a,
we knowweight_a, and we can computecumulative_bbecause it only
uses terms indataset_a.  So we can pre-compute those terms,
making the final math:# when computing the streamlined data
precomputed_a = weight_a / cumulative_b
precomputed_b = weight_b / cumulative_a

# ...

# when computing the score for a matching exact key
final_score = precomputed_a * precomputed_bThat's alotsimpler!  And these optimizations madecorrelatealotfaster.Fuzzy KeysLet me tell you a wonderful bed-time story.
Once upon a time,correlatewas small and beautiful.
But that version only supported exact keys.
By the time fuzzy keys were completely implemented, and feature-complete,
and working great,correlatewas much more complex and... "practical".
It's because fuzzy keys introduce a lot of complex behavior, resulting in
tricky scenarios that just don't arise with exact keys.Consider this example:Your two datasets represent lists of farms.  Both datasets list
animals, but might have generic information ("horse") or might
have specifics ("Clydesdale").  You create a fuzzy key subclass calledAnimalKeythat can handle matching these together;AnimalKey("Horse/Clydesdale")matchesAnimalKey("Horse"),
though with a score less than 1 because it isn't a perfect match.The same farm,Farm X, is present in both datasets:Indataset_a, the keyAnimalKey("Horse")maps to Farm Xtwice.Indataset_b, the keysAnimalKey("Horse/Clydesdale")andAnimalKey("Horse/Shetland Pony")map to Farm X.Question: should one of the"Horse"keys indataset_amatch"Horse/Clydesdale"indataset_b, and should the other"Horse"key match"Horse/Shetland Pony"?Of course they should!  But consider the ramifications: we just matched
a key fromround 2indataset_ato a key fromround 1indataset_b.
That's simplyimpossiblewith exact keys!The scoring used for fuzzy keys is conceptually the same as the scoring
for exact keys, including the concept of "rounds".  In practice, fuzzy key
scoring is much more complicated; there are some multipliers I elided
in the description for exact keys because they're always 1, and some other
things that are easy to compute for exact keys that we must do the hard way
for fuzzy keys.  (There's a whole section at the end of this document about
the history of fuzzy key scoring incorrelate, in case you're interested.)Also, it's reasonable for a single value in a dataset to have multiple fuzzy
keys of the same type, which means that now we could have multiple keys
in one dataset in contention for the same key in the other dataset.
In the above example with farms and horses,correlatewill need to
compare bothAnimalKey("Horse/Clydesdale")andAnimalKey("Horse/Shetland Pony")fromdataset_atoAnimalKey("Horse")indataset_b.Butcorrelatedoesn't add up every possible fuzzy score generated by a
key; when computing the final score, a fuzzy key is only matched against
one other fuzzy key.  If fuzzy keysFA1andFA2map to valueVAindataset_a, and fuzzy keyFBmaps to valueVBindataset_b,correlatewill considerFA1->FBand alsoFA2->FBand only keep the match with the highest score.  A match "consumes"
the two keys (one from each dataset) and they can't be matched again.
(Again: when I say "key" here, I mean "this key in this round".)
The flip side of this: a key that isn't matchedisn't"consumed".
What do we do with it?  The example above with horses and farms makes
it clear: unconsumed fuzzy keys should get recycled--reused in
subsequent rounds.So, where exact keys use very precise "rounds", fuzzy keys
require a more dynamic approach.  Precisely speaking, an unused key in
roundNconceptually "survives" to roundN+1.  That's what the above
example with farms and ponies shows us; in round 0, if"Horse/Clysedale"indataset_agets matched to"Horse"indataset_b,"Horse/Shetland Pony"indataset_agoes unmatched, and survives,
and advances on to round 1.  This also made scoring more complicated.
(For more on this, check out the test suite.  There's a regression test
that exercises this exact behavior.)After a bunch of rewrites, I found the fastest way to compute fuzzy matches
was: for each fuzzy type the two values have in common, computeall possible
matchesbetween all fuzzy keys mapping to the two values, even mixing
between rounds.  Then sort the matches, preferring higher scores to lower
scores, and preferring matches in lower rounds to matches in higher rounds.The streamlined data for fuzzy keys looks like this:fuzzy_types[index][type] = [
                           [(key1, weight, round#0),  (key1, weight, round#1), ...],
                           [(key2, weight, round#0),  (key2, weight, round#1), ...],
                           ]That is, they're indexed by index (a representation of the value),
then by fuzzy type.  That gets you a list of lists.  Each inner list
is a list of tuples of(key, weight, round_number)wherekeyis always the same in all entries in the list, andround_numberis always the same as that tuple's index in that list.When computing matches between fuzzy keys,correlatetakes the
two lists of lists and does nestedforloops over them.  Since the
keys don't change, it only needs to look up the fuzzy score once.
If the fuzzy score is greater than 0, it stores the match in an
array.Once it's done with the fuzzy key matching, it sorts this array of matches,
then use the "match boiler" to reduce it down so that every per-round key
is matched at most once.  (The "match boiler" is discussed later; for now
just assume it's a magic function that does the right thing.  Though I had
to ensure it was super-stable for this approach to work.)Sorting these fuzzy key matches was tricky.  They aren't merely sorted
by score; we also must ensure that fuzzy key matches from earlier rounds
arealwaysconsumed before matches using that key in later rounds.
So we use a specialsort_bytuple as the sorting key, computed as follows:key_a, weight_a, round_a = fuzzy_types_a[index_a][type]...
key_b, weight_b, round_b = fuzzy_types_b[index_b][type]...
fuzzy_score = key_a.compare(key_b)
lowest_round_number  = min(round_a, round_b)
highest_round_number = max(round_a, round_b)
sort_by = (fuzzy_score, -lowest_round_number, -highest_round_number)The-lowest_round_numbertrick is the very clever bit.  This lets
us sort with highest values last, which is what the "match boiler" wants.
But negating it means lower round numbers are nowhighernumbers, which
lets us prefer keys withlowerround numbers.In terms of the abstract scoring formula,scoreis the fuzzy score,
what's returned by calling thecompare()method. Andcumulative_ais the sum of all fuzzyscorescores for all matches usingkey_a.Score Ratio BonusThere's a "bonus" score calculated usingscore_ratio_bonus.  It's scored for the
overall mapping of a value indataset_ato a value indataset_b.
This bonus is one of the last things computed for a match, just before ranking.The bonus is calculated as follows:value_a = a value from dataset_a
value_b = a value from dataset_b
actual_a = total actual score for all keys that map to value_a in dataset_a
actual_b = total actual score for all keys that map to value_b in dataset_b
possible_a = total possible score for all keys that map to value_a in dataset_a
possible_b = total possible score for all keys that map to value_b in dataset_b
bonus_weight = score_ratio_bonus * (actual_a + actual_b) / (possible_a + possible_b)This bonus calculated withscore_ratio_bonusclears up the
ambiguity when the set of keys mapping to one value is a subset of the keys
mapping to a different value in the same dataset.  The higher percentage
of keys that match, the larger this bonus will be.Consider this example:c = correlate.Correlator()
c.dataset_a.set('breakin', X)

c.dataset_b.set('breakin', Y)
c.dataset_b.set_keys(['breakin', '2', 'electric', 'boogaloo'], Z)Which is the better match,X->YorX->Z?
In early versions ofcorrelate, both matches got the exact same score.
So it was the luck of the draw as to which matchcorrelatewould choose.score_ratio_bonusdisambiguates this scenario.  It awards a larger bonus
toX->Ythan it does toX->Z,  because a higher percentage of the keys
matched betweenXandYthan matched betweenXandZ.
That small nudge is generally all that's needed to letcorrelatedisambiguate these situations and pick the correct match.Two things to note.  First, when I say "keys", this is another situation
where the same key mapped twice to the same value is conceptually considered
to be two different keys.
In the example I gave in theRoundssubsection above, wherevalue_aisoandvalue_biso2,possible_awould be 6 andpossible_bwould be 4.Second, the scores used to computeactualandpossibleareunweighted.If a match between two fuzzy keys resulted in a fuzzy score of0.3,
that adds0.3to bothactual_aandactual_b, but each of those fuzzy
keys adds1.0topossible_aandpossible_brespectively.
Weights are always ignored when computingscore_ratio_bonus, just like
they're ignored when comparing matches.Choosing Which Matches To Keep: The "Greedy Algorithm" And The "Match Boiler"Here's a problem, presented in the abstract: if you're presented with
a list of match objects calledmatches,
where each match objectMhas three attributesvalue_a,value_b,
andscore, how would you compute an optimal subset ofmatchessuch that:every discrete value ofvalue_aandvalue_bappears only once, andthe sum of thescoreattributes is maximized?Finding the perfectly optimal solution would require computing every
possible set of matches, then computing the cumulative score of that
set, then choosing the set with the highest score.  Unfortuantely,
that algorithm isO(nⁿ),
which is so amazingly expensive that we can't even consider it.
(You probably want your results fromcorrelatebefore our sun
turns into a red giant.)Instead,correlateuses a comparatively cheap "greedy" algorithm
to compute the subset.  It's notguaranteedto produce the optimal
subset, but in practice it seems to produce optimal results on
real-world data.Here's a short description of thecorrelate"greedy" algorithm:Sortmatcheswith highest score first.For every matchMinmatches:ifvalue_ahasn't been matched yet,andvalue_bhasn't been matched yet,keepMas a good match,remember thatvalue_ahas been matched,and remember thatvalue_bhas been matched.The sorting uses Python's built-in sort (Timsort), so it'sO(n log n).  It's implemented in C so it's pretty quick.
Theforloop isO(n).However!  Late in development ofcorrelateI
realized there was a corner case where odds are good the
greedy algorithm wouldn't produce an optimal result.
Happily, this had a relatively easy fix, and the
fix didn't makecorrelateany slower in the general case.Let's start with the problem, the nasty corner case.  What if two
matches in the list are both viable, and they have thesamescore,
and they have eithervalue_aorvalue_bin common?
It's ambiguous as to which match the greedy algorithm
will choose.  But choosing the wrong onecouldresult
in less-than-optimal scoring in practice.Here's a specific example:dataset_acontains fuzzy keysfka1andfka2.dataset_bcontains fuzzy keysfkbHandfkbL.
Any match containingfkbHhas a higher score than any match containingfkbL.
(the H meanshighscoring,  the L meanslowscoring.)The matchesfka1->fkbHandfka2->fkbHhave the same score.The matchfka1->fkbLhas a lower score thanfka2->fkbL.The cumulative score over all matches would be higher ifcorrelatechosefka2->fkbLtofka1->fkbL.
And since scores incorrelateare an indicator of
the quality of a match, a higher cumulative score reflects
higher quality matches.  Therefore we should maximize the
cumulative score wherever possible.But the greedy algorithm can only pick the higher-scoring second
match if it previously pickedfka1->fkbH.  And there's no guarantee
that it would!  If two items in the list have the same score,
it's ambiguous which one the greedy algorithm would choose.To handle this properly it needs to look ahead and experiment.
So that's why I wrote what I call the "match boiler", or the
"boiler" for short.  The boiler uses a hybrid approach.  By default,
when the scores for matches are unique, it uses the "greedy"
algorithm.  But if it encounters a group of items with matching
scores, where any of those items havevalue_aorvalue_bin
common, it recursively runs an experiment where it chooses each
of those matches in turn.  It computes the score from each of these
recursive experiments and keeps the one with the highest score.(If two or more experiments have the same score, it keeps the first one
it encountered with that score--but, since the input to the "match boiler"
is a list, sorted with highest scores to the end, technically it's thelastentry in the list that produced the high-scoring experiment.)With the "match boiler" in place,correlateseems to produce optimal
results even in these rare ambigous situations.I'm honestly not sure what thebig-Onotation is for the "match boiler".
The pathological worst case
isprobablyon the order ofO(n log n), where thelog ncomponent
represents the recursions.
In this case, every match has the same score, and they're all connected to
each other via havingvalue_aandvalue_bin common.  I still don't
think the "match boiler" would be as bad asO(n²).
The thing is, sooner or later the recursive step would cut the
"group" of "connected items" in half (see next section).  It's guaranteednotto recurse on every single item.
So I assert that roughly cuts the number of recursive
steps down tolog n, in the pathological worst case that you would
never see in real-world data.Cheap Recursion And The "Grouper"But wait!  It gets even more complicated!Compared to the rest of the algorithm, the recursive step of the
"match boiler" is quite expensive.  It does reduce the domain of
the problem at every step, so it's guaranteed to complete...
someday.  But, if we're not careful, it'll perform a lot of
expensive and redundant calculations.  So there are a bunch
of optimizations to the match boiler's recursive step, mainly
to do with the group of matches that have the same score.The first step is to analyze these matches and boil them out
into "connected groups".  A "connected group" is a set of
match objects where either each object has avalue_aor avalue_bin common with another object in the group.  These
are relevant because choosing one of the matches from these
groups will remove at least one other value from consideration
in that group, because thatvalue_aorvalue_bis
now "used" and so all remaining match using those values
will be discarded.An example might help here.  Let's say you have these
six matches in a row all with the same score:match[1]: value_a = A1, value_b = B1
match[2]: value_a = A1, value_b = B2
match[3]: value_a = A2, value_b = B1
match[4]: value_a = A3, value_b = B2
match[5]: value_a = A10, value_b = B10
match[6]: value_a = A10, value_b = B11This would split into two "connected groups": matches 1-4
would be in the first group, and matches 5-6 would be in the
second.  Every match in the first group has one member
(value_aorvalue_b) in common with at least one other
match in the first group; every match in the second group
has one member in common with at least one other match in
the second group.  So every match in the first group is
"connected"; if you put them in a graph, every match would
be "reachable" from every other match in that group,
even if they aren't directly connected.  For example,match[4]doesn't have any members in common withmatch[1],
but both of them have a member in common withmatch[2].
But none of the matches in the first group have any member in
common with any of the matches in the second group (and
naturally vice-versa).There's a utility function calledgrouper()that computes
these connected groups.  (grouper()only handles the case
whenreuse_a == reuse_b == False; there are alternate
implementations to handle the other possible cases, e.g.grouper_reuse_a().)The second step is to take those "connected groups" and,
for every group containing only one match object,
"keep" it immediately.  We already know we're keeping
these and it's cheaper to do that first.Now that the only remaining connected groups are size 2 or
more, the third step is to recurse over each of the values of thesmallestof these connected groups.  Why the smallest?
It's cheaper.  Let's say there are 50 items left in the list
of matches.  At the top are 6 match objects with the same score.
There are two groups: one of length 2, the other of length 4.The important realization is that, when we perform the experiment
and recurse using each of these values, we're still going to
have to examine all the remaining matches we didn't throw away.
The number of operations we'll perform by looping and recursing is,
roughly,N•M, whereNislen(group)andMislen(matches - group).  So which one has fewer operations:2 x 48, or4 x 46?Obviously the first one!  By recursing into the smaller group,
we perform fewer overall operations.(There's a theoretical opportunity for further optimization here:
when recursing, if there's more than one connected group of length 2
or greater, pass in the list of groups wedidn'tconsume to the recursive step.  That would save the recursive
call from re-calculating the connected groups.  In practice I imagine this
happens rarely, so handling it would result in a couple ofifbranches that never get taken.  Also, it's a little more complicated
than it seems, because you'd have to re-use thegrouper()on
the group you're examining before passing it down, because it might
split it into two groups.  In practice it wouldn't speed up anybody's
correlations, and it'd make the code more complicated.  So let's skip it.
The code is already more-or-less correct in these rare circumstances
and that's good enough.)The Match Boiler Reused For Fuzzy Key ScoringOnce my first version of the "match boiler" was done, I realized I could reuse
it for boiling down all fuzzy key matches too.  Fuzzy key matches already
used basically the same "greedy" algorithm that were used for matches,
and it dawned on me that the same corner case existed here too.My first attempt was quite complicated, as the "match boiler" doesn't
itself understand rounds.  I added a callback which it'd call every time it
kept a match, which passed in the keys that got matched.  Since those keys
were now "consumed", I would inject new matches using those keys from
subsequent rounds (if any). This worked but the code was complicated.And it got even more complicated later when I added the recursive step!
I had to save and restore all the state of which fuzzy keys had been
consumed from which rounds.  I wound up building it into the subclass
ofMatchBoiler, which is part of whyMatchBoilerclones itself
when recursing.  This made the code cleaner but it was still clumsy
and a bit slow.The subsequent rewrite using thesort_bytuple was a big win all around:
it simplified the code, it let me remove the callback, it let the match
boiler implicitly handle all the rounds without really understanding them,andit was even slightly faster!But this is why it's so important that the boiler is super-stable.
Earlier versions of the match boiler assumed it could sort the
input array any time it wanted.  But the array passed in was sorted
by score,thenby round numbers--highest score is most important,
lowest round number is second-most important.
And the array is sorted with highest score, then lowest round number, last.
When recursing, the match boiler has to prefer thelastentry in its
input that produced the same score--otherwise, it might accidentally
consume a key from a later round before consuming that key from an
earlier round.I didn't want to teach the boiler to understand thissort_bytuple.
Happily, I didn't have to.  It wasn't much work to ensure that the boiler
was super-stable, and once that was true it always produced correct results.
(Not to mention... faster!)Theoretical Failings Of The Match BoilerEven with the "match boiler", you can still contrive scenarios
wherecorrelatewill produce arguably sub-optimal results.
The boiler only tries experiments where the matches have the
same score.  But it's possible that the greedy algorithm may
find a local maximum that causes it to miss the global maximum.IfAandBare values indataset_a, andXandYare
values indataset_b, and the matches have these scores:A->X == 10
A->Y == 9
B->X == 8
B->Y == 1In this scenario, the boiler will pickA->X, which means it's
left withB->Y.  Total score: 11.  But if it had pickedA->Y,
that means it would get to pickB->X, and the total score would
be 17!  Amazing!Is that better?  Your first reaction is probably "of course!".
But in an abstract, hypothetical scenario like this, it's
hard to say for sure.  I mean, yes it's a betterscore.But
is it a bettermatch?Is this the output the user would have
wanted?  Who knows--this scenario is completely hypothetical in
the first place.I doubt this is a real problem in practice.  Ensuringcorrelatehandles the ambiguous scenario where items had identical scores is
already "gilding the lily", considering how rare it happens with
real-world data.  And when would real data behave in this contrived
way?  Why wouldAscore so highly againstXandY, butBscores
high againstXbut low againstY?  IfBis a good match forX,
andXis a good match forA, andAis a good match forY, then,
with real-world data, transitivity would suggestBis a good match
forY.  This contrived scenario seems more and more contrived the more
we look at it, and unlikely to occur in the real world.I think pathological scenarios where the "match boiler" will fail like
this aren't realistic.  And the only way I can think of to fix it is
with the crushingly expensiveO(nⁿ) algorithm.
It's just not worth it.  So, relax!  As we say in Python: YAGNI.RankingRanking information can help a great deal.
If a value indataset_ais near the beginning, and the order
of values is significant, then we should prefer matching it to values
indataset_bnear the beginning too.  When the datasets are
ordered, matching the first value indataset_aagainst the last
value indataset_bis probably a bad match.Conceptually it works as follows: when scoring a match,
measure the distance between
the two values and let that distance influence the score.  The closer
the two values are to each other, the higher the resulting score.But how do you compute that distance?  What do the ranking numbers mean?correlatesupports three possible interpretations
of the rankings:Absoluteranking,Relativeranking, andReversed Absoluteranking.These three approaches differ in how they compare the ranking numbers,
as follows:Absoluteranking assumes the ranking numbers are the same
for both datasets.ranking=5indataset_ais a perfect
match toranking=5indataset_b.  This works well when
your datasets are both reasonably complete; if they're different
sizes, perhaps one or both are truncated at either the beginning
or end.Relativeranking assumes that the two datasets represent the
same range of data, and uses the ratio of the ranking of a value
divided by the highest ranking set in that dataset to compute
its relative position.  If the highest ranking we saw in a
particular dataset wasranking=150,
then a value that hasranking=12set is calculated to be 8%
of the way from the beginning to the end.  This percentage
is calculated similarly for both datasets, and the distance
between two values is the distance between these two percentages.
This works well if one or both of your datasets are sparse.Reversed Absoluteis likeAbsolute, but starts from theendrather than from thebeginning.Think aboutAbsoluteranking this way: it's comparing the distance from
beginning of the dataset to the particular value.  Well,Reversed Absoluteuses the distance from theendof the
dataset to the particular value.  Consider: ifdataset_acontains 100 values, anddataset_bonly contains 15 values,
but they're matches for thelast15 values ofdataset_a,
neitherAbsolutenorRelativeare a good fit.  What you'd
want there isReversed Absolute.Here's a more concrete example of how these approaches work.
Let's saydataset_ahas 101 items ranked 0 to 100,dataset_bhas 801 items ranked 0 to 800, and we have a value indataset_awithranking=50:WithAbsoluteranking, the closest value indataset_bwould haveranking=50.  The two values are both 50 elements after
the first (lowest-ranked) value.WithRelativeranking, the closest value indataset_bwould haveranking=400.  The two values are in the middle of the
rankings for their respective datasets.WithReversed Absoluteranking, the closest value indataset_bwould haveranking=750.  The two value are both 50 elements
in front of the last (highest-ranked) value.Which one doescorrelateuse?  It's configurable with therankingparameter tocorrelate().  By default it uses the "best" ranking.
"Best" ranking meanscorrelatecompute a score usingallmethods and chooses the one with the highest score.
You can override this by supplying a different value torankingbut this shouldn't be necessary.  (Theoretically it should be faster
to use only one ranking approach.  Unfortunately this hasn't been
optimized yet, so using only one ranking doesn't currently speed
things up.)Ranking is the last step in computing the score of a match.
As for how ranking affects the score, it depends on whether you
useranking_bonusorranking_factor.Both approaches start with these four calculations:semifinal_scores_sum = sum of all "semifinal" scores above
ranking_a = the ranking value computed for value_a
ranking_b = the ranking value computed for value_b
ranking_score = 1 - abs(ranking_a - ranking_b)ranking_bonusis then calculated per-match as follows:bonus = ranking_score * ranking_bonus
final_score = semifinal_scores_sum + bonusranking_factoris also calculated per-match, as follows:unranked_portion = (1 - ranking_factor) * semifinal_scores_sum
ranked_portion = ranking_factor * semifinal_scores_sum * ranking_score
final_score = unranked_portion + ranked_portion(If you don't use either, the final score for the match is effectivelysemifinal_scores_sum.)Obviously,rankingmust be set on both values in both datasets to
properly computeranking_score.  If it's not set onbothvalues
being considered for a match,correlatestill appliesranking_bonusorranking_factoras usual, but it skips the
initial four calculations and just uses aranking_scoreof 0.Final Random TopicsDebuggingWhen all else fails... what next?correlatecan optionally produce an enormous amount of debug output.
The main feature is showing every match it tests, and the score arrived
at for that match, including every step along the way.  This log output
quickly gets very large; even a comparison of 600x600 elements will produce
tens of megabytes of output.Unfortunately, producing this much debugging output incurred a measurable
performance penalty--even when you had logging turned off!  It was mostly
in computing the "f-strings" for the log, but the calls to the logging
functions definitely added overhead too.My solution: by default, each of the debug print statements
is commented out.correlateships with a custom script preprocessor calleddebug.pythat can toggle debugging on and off, by uncommenting and
re-commenting the debug code.How does it know which lines to uncomment?  Each line of the logging-only
code ends with the special marker "#debug".To turn on this logging, run thedebug.pyscript in the same directory
ascorrelate's__init__.pyscript.  Each time you run it, it'll
toggle (comment / uncomment) the debug print statements.
Note that the debug feature incorrelaterequires Python 3.8 or higher,
because it frequently uses 3.8's beloved "equals sign inside f-strings" syntax.By default the logging is sent to stdout.  If you want to override where
it's sent, write your ownprintfunction, and assign it to yourCorrelatorobject before callingcorrelate().The format of the log is undocumented and subject to change.  Good luck!
The main thing you'll want to do is figure out the "index" of the values
indataset_aanddataset_bthat you want to compare, then search for" (index_a) x (index_b) ".  For example, if the match you want to see
is between value index 35 indataset_aand value index 51 indataset_b,
search in the log for" 35 x 51 ". (The leading and trailing spaces
means your search will skip over, for example,235 x 514.)Alternate Fuzzy Scoring Approaches That Didn't WorkI find the math behind fuzzy scoring a bit surprising.
If you boil down the formula to its constituent factors,
you'll notice one of the factors isfuzzy_scorecubed.Why is itcubed?The simplest answer: that's the first approach that seemed to work
properly.  To really understand why, you'll need to understand the
history of fuzzy scoring incorrelate--all the approaches
I tried first thatdidn'twork.Initially, the score for a fuzzy match was simply the fuzzy score
multiplied by the weights and other modifiers.
This was always a dumb idea; it meant fuzzy matches hadwaymore
impact on the score than they should have.  This was particularly
true when you got down to the last 10% or 20% of your matches,
by which point the score contributed by exact keys had
fallen off a great deal.  This approach stayed in for what is,
in retrospect, an embarassingly long time; I'd convinced myself
that fuzzy keys were innately moreinterestingthan exact keys,
and so this comparative importance was fitting.Once I realized how dumb that was, the obvious approach was to score
them identically to exact keys--divide the fuzzy score by the product
of the number of keys thiscouldhave matched against in each
of the two datasets.  This was obviously wrong right away.
In the "YTJD" test, every value had one or more fuzzy keys, depending
on the test: every value always had a fuzzy date key, and depending
on the test it might have a fuzzy title key and/or a fuzzy episode number
key too.  So each of the 812 values in the first dataset had one
fuzzy key for each fuzzy type, and each of the 724 values in the second
dataset did too.  Even if we got a perfect fuzzy match, the maximum score
for a fuzzy match was now1.0 / (812 * 724)which is0.0000017.  So now
we had the opposite problem: instead of being super important, even a perfect
fuzzy match contributed practically nothing to the final score.After thinking about it for a while,
I realized that the exact key score wasn'treallybeing
divided by the number ofkeysin the two datasets, per se;
it was being divided by the total possiblescorecontributed
by that key in each of the two datasets.  So instead of
dividing fuzzy scores by the number of keys, they should be divided
by the cumulative fuzzy score of all matches involving those two keys.
That formula looks likefuzzy_score / (sum_of_fuzzy_scores_for_key_in_A * sum_of_fuzzy_scores_for_key_in_B).This was a lot closer to correct!  But this formula had a glaring new problem.
Let's say that in your entire correlation,dataset_aonly had one
fuzzy key that maps to a single value,
anddataset_aonly had one fuzzy key that also only maps to a single value.
And let's say the fuzzy score you get from matching those two keys
is0.000001--a really terrible match.
Let's plug those numbers into our formula, shall we!
We get0.000001 / (0.000001 * 0.000001), which is1000000.0.
A million!  That's crazy!  We've taken an absolutely
terrible fuzzy match and inflated its score to be nonsensically high.
Clearly that's not right either.This leads us to the formula that actually works.  The insight here
is that the same formula needs to work identically for exact keys.
If you take this formula and compute it where everyfuzzy_scoreis 1 (or 0), it produces the same result as the formula for exact keys.
So the final trick is that we can multiply byfuzzy_scorewherever we need
to, because multiplying by 1 doesn't change anything.  That means
the resulting formula will still be consistent with the exact keys
scoring formula.  And what worked was the formula where we multiply byfuzzy_scorethree times!Here again is the formula used to compute the score for a fuzzy match,
simplified to ignore weights and rounds:value_a = a value from dataset_a
value_b = a value from dataset_b
key_a = a fuzzy key in dataset_a that maps to value_a
key_b = a fuzzy key in dataset_b that maps to value_b
fuzzy_score = the result of key_a.compare(key_b)
cumulative_a = the cumulative score of all matches between key_a and every fuzzy key of the same type in dataset_b
cumulative_b = the cumulative score of all matches between key_b and every fuzzy key of the same type in dataset_a
score_ratio_a = fuzzy_score / cumulative_a
score_ratio_b = fuzzy_score / cumulative_b
unweighted_score = fuzzy_score * score_ratio_a * score_ratio_bThe final trick really was realizing whatscore_ratio_arepresents.
Really, it represents the ratio of how muchthisfuzzy match forkey_acontributed to the sum ofallfuzzy matches forkey_aacross all successful matches indataset_a.Why correlate Doesn't Use The Gale-Shapley AlgorithmA friend asked me if the problemcorrelatesolves is isomorphic to the Stable Matching Problem:https://en.wikipedia.org/wiki/Stable_matching_problemBecause, if it was, I could use the Gale-Shapley algorithm:https://en.wikipedia.org/wiki/Gale%E2%80%93Shapley_algorithmI thought about this for quite a while, and I don't think the problemcorrelatesolves maps perfectly onto the stable matching problem.correlatesolves a problem that is:simpler,different, andharder.For inputs that are valid for both Gale-Shapley andcorrelate, I assert that both
algorithms will return the same results, butcorrelatewill be faster.(Not a claim I make lightly!  Gale and Shapley were both brilliant
mathematicians--they each independently won thevon Neumannprize!--and the
Gale-Shapley algorithm is marvelous and elegant.  It's just thatcorrelatecan take shortcuts Gale-Shapley cannot, becausecorrelateis solving
a simpler problem.)In all the following examples,A,B, andCare values indataset_aandX,Y, andZare values
indataset_b.  The expressionA: XYmeans "AprefersXtoY". The expressionA:X=1.5means
"when matchingAandX, their score is1.5".  When I talk about Gale-Shapley,dataset_awill stand for the
"men" anddataset_bwill stand for the "women", which meansAis a man andXis a woman.  Where we need to talk
about the matches themselves, we'll call themPandQ.How is it simpler?The stable matching problem only requires a local ordering, where the preferences of any value in either dataset
are disjoint from the orderings of any other value.  Butcorrelateuses an absolute "score"--a number--to compute
these preferences, and this score is symmetric; ifA:X=1.5, thenX:A=1.5too.On this related Wikipedia page:https://en.wikipedia.org/wiki/Lattice_of_stable_matchingswe find a classic example of a tricky stable matching problem:A: YXZ
B: ZYX
C: XZY

X: BAC
Y: CBA
Z: ACBGale-Shapley handles this situation with aplomb.  Doescorrelate?
The answer is... this arrangement of constraints just can'thappenwithcorrelate, because it uses scores to establish its preferences,
and the scores are symmetric. There are nine possible pairings with
those six values. It's impossible to assign a unique score to each of
those nine pairings such that the preferences of each value match
those constraints.(And, yes, I'm pretty certain.  Not only did I work my way through it,
I also wrote a brute-force program that tried every possible combination.
362,880 attempts later, I declared that there was no possible solution.)How is it different?One minor difference: Gale-Shapley specifies that the two sets be of
equal size;correlatepermits the two sets to be different sizes.
But extending Gale-Shapley to handle this isn't a big deal.  Simply
extend the algorithm to say that, if the size of the two datasets are
inequal, swap the datasets if necessary so that the "men" are the
larger group. Then, if you have an unmatched "man" who iterates
through all the "women" and nobody traded up to him, he remains
unmatched.The second thing: Gale-Shapley requires that every value in each
dataset expresses a strictly ordered preference for every value
in the other dataset. But incorrelate, two matches can have
the same score.Consider this expression of acorrelateproblem:A:X=100
A:Y=1

B:X=100
B:Y=2Gale-Shapley as originally stated can't solve that problem, because
X doesn't prefer A or B--it likes them both equally.  It wouldn't
be hard to extend Gale-Shapley to handle this; in the case where it
prefers two equally, let it arbitrarily pick one.  For example,
if X prefers A and B equally, say "maybe" to the first one that asks,
and then let that decide for you that you prefer A to B.How is it harder?Here's the real problem.correlateuses a numerical score to weigh the merits of each match, and seeks to
maximize the cumulative score acrossallmatches. Gale-Shapley's goals are comparatively
modest--any match that's stable is fine. There may be multiple stable matchings; Gale-Shapley
considers them all equally good.In practice, I think if you apply the original Gale-Shapley algorithm to an input data set
where the matches have numerical scores, itwouldreturn the set of matches with the
highest cumulative score.  In thinking about it I haven't been able to propose a situation
where it wouldn't.  The problem lies in datasets where two matches have thesamescore--which
the original Gale-Shapley algorithm doesn't allow.Ensuring thatcorrelatereturns the highest cumulative score in this situation
required adding the sophisticated recursive step to the "match boiler".  We'd have to make
a similar modification to Gale-Shapley, giving it a recursive step.  Gale-Shapley is
alreadyO(n²); I think the modified version would beO(n² log n).
(But, likecorrelate, this worst-case shouldn't happen with real-world data.)
Anyway, a modified Gale-Shapley that works for allcorrelateinputs is definitely
more expensive than whatcorrelatehas--or what it needs.Mapping Gale-Shapley To The Correlate Greedy AlgorithmAgain, the set of valid inputs forcorrelateand Gale-Shapley aren't
exactly the same.  But there's a lot of overlap.  Both algorithms can
handle an input where:we can assign every match between a manAand a womanXa numerical score,every match involving any particular manAhas a unique score,and also for every womanX,and there are exactly as many men as there are women.I assert that, for these mutually acceptable inputs, both algorithms
produce the same result.  Here's an informal handwave-y proof.First, since Gale-Shapley doesn't handle preferring two matches equally,
we'll only consider datasets where all matches have a unique score.
This lets us dispense with the match boiler's "recursive step", so all we
need is the comparatively simple "greedy algorithm".  (Again: this
"greedy algorithm" is much cheaper than Gale-Shapley, but as I'll show,
it's sufficient for the simple problem domain we face here.)So let's run Gale-Shapley on our dataset.  And every time we perform
an operation, we write it down in a list--we write down
"manAasked womanX" and whether her reply wasmaybeorno.Observe two things about this list:First, order is significant in this list of operations.  If you change
the order in which particular men ask particular women, the pattern of
resultingmaybeandnoresponses will change.Second, thelast"maybe" said by each woman is always,
effectively, ayes.So let's iterate backwards through this list of matches, and the first
time we see any particular woman replymaybe, we change that answer
to ayes.Next: observe that we can swap any two adjacent operations--with one
important caveat.  We must maintain this invariant: for every womanX,
for every operation containingXthat happens afterXsaysyes,Xmust sayno.Thus, if there are two adjacent operationsPandQ, wherePis currently first, and we want to swap them soQis first,
and if the following conditions are all true:The same womanXis asked in bothPandQ.InQthe womanXsays eithermaybeoryes.InPthe womanXsaysmaybe.Then we can swapPandQif and only if we changeX's
response inPtono.Now that we can reorder all the operations, let's sort the
operations by score, with the highest score first.
Let's call the operation with the highest scoreP,
and say that it matches manAwith womanX.We now know the following are true:Xis the first choice ofA.  This must be true becausePis the match with the highest score.  ThereforeAwill
askXfirst.Ais the first choice ofX, again becausePis the
match with the highest score.  ThereforeXis
guaranteed to sayyes.Since the first operationPis guaranteed to be ayes,
that means that every subsequent operation involving eitherAorXmust be ano.We now iterate down the list to find an operationQinvolving manBand womanY. We defineQas the
first operation such that:Q != P, and thereforeQis afterPin our ordered list of operations,B != A, andY != X.By definitionQmust also be ayes, becauseBandYare each other's first choices now thatAandXare
unavailable for matching.  If there are any operations
betweenPandQ, these operations involve eitherAorX.
Therefore they must beno.  ThereforeQrepresents the
highest remaining preference for bothBandY.Observe that the whole list looks like this.  Everyyesis
the first operation for both that man and that woman in which
they weren't paired up with a woman or man (respectively) that
had already saidyesto someone else.This list of operations now more or less resembles the operations
performed by thecorrelate"greedy" algorithm. It sorts the
matches by score, then iterates down that sorted list.  For every
manAand womanX, if neitherAnorXhas been matched yet,
it matchesAandXand remembers that they've been matched.It's possible that there's minor variation in the list of
operations; any operation involving any man or any womanafterthey've been matched with ayesis extraneous.
So you can add or remove them all you like without affecting
the results.Version History1.1Added a new ranking approach!  The first two wereAbsoluteRankingandRelativeRanking, this new third one isReversedAbsoluteRanking.1.0No code changes.  Butcorrelatehas been stable and working
for a while now... it's time to mark it as 1.0.0.8.3A slight bugfix forprint_datasets().print_datasets()prints out the keys for each value in sorted order.  But that
meant sorting the keys, and if you have keys of disparate
types, attempting to compare them with<or>could throw
aValueError.  Soprint_datasets()now separates the keys
by type and sorts and prints each list of keys separately.The dataset API allows you to set values that don't actually have
any keys mapping to them.  (You can calldataset.value()with a value that you never pass in toset()orset_keys().)correlate()used to simply assert that every value had at
least one key; now it raises aValueErrorwith a string
that prints every value.  (This can be unreadable if there
are a lot!  But better safe than sorry.)0.8.2Fixed upinfer_mv.  It works the same, but the comments it
prints out are now much improved.  In particular, there was
a bug where it reported the same score for every match--the score
of the lowest-ranked match--instead of the correct score for each
match.There were no other changes; thecorrelatealgorithm is
unchanged from 0.8.1.0.8.1Fixed compatibility with Python 3.6.  All I needed to do was
remove someequals-sign-in-f-stringsusage in spots.0.8The result of loving hand-tuned optimization:correlateversion 0.8
is now an astonishing19.5%faster than version 0.7--and27.3%faster
than version 0.5!The statistics have been improved, including some useful timing information.
This really demonstrates how much slower fuzzy keys are.(To see for yourself, runpython3 tests/ytjd.test.py -vand compare
the slowest test to the fastest, using the same corpus.  On my computer
the test without fuzzy keys is12x fasterthan the one that uses
fuzzy keys for everything.)0.7Careful micro-optimizations for both exact and fuzzy key
code paths have madecorrelateup to 7.5% faster!TheMatchBoilerwas made even more ridiculously stable.
It should now always:returnresultsin the same order they appeared in inmatches, andprefer thelastequivalent item when two or more items
produce the same cumulative score.0.6.1Bugfix for major but rare bug: if there are multiple
groups oflen() > 1of "connected" match objects with
the same score, the match boiler would only keep the
smallest one--the rest were accidentally discarded.
(match_boiler_2_test()was added totests/regression_test.pyto check for this.)0.6Big performance boost in "fuzzy boiling"!  Clever sorting of fuzzy matches,
and improvements in the stability (as in "stable sort") ofMatchBoiler,
allowed using an unmodified boiler to process fuzzy matches.  This allowed
removal ofFuzzyMatchBoilerand theMatchBoiler.filter()callback mechanism.Minor performance improvement inMatchBoiler: when recursing, find the
smallest group of connected matches with the same score,
and only recursively check each of those,
rather than all possibly-connected matches with the same score.Removedkey_reuse_penalty_factor.
In the early days ofcorrelate, it didn't understand rounds; if you mapped the same
key to the same value twice, it only remembered one mapping, the one with the
highest weight.  Later I added rounds but they didn't seem to add much signal.
I thought redundant keys were uninteresting.  So I addedkey_reuse_penalty_factor.
That let you turn down the signal they provided, in case it
was adding more noise than actual useful signal.
It wasn't until the realization thatkey->valuein round 0 andkey->valuein round 1 were conceptuallytwo different keysthat I really understood
how redundant mappings of the same key to the same value should work.  And
once rounds maintained distinct counts ofkeys / scoresfor the scoring
formula, redundant keys in different rounds becamewaymore informative.
I now thinkkey_reuse_penalty_factoris dumb and worse than useless and
I've removed it.  If you thinkkey_reuse_penalty_factoris useful,
please contact me and tell me why!  Or, quietly just pre-multiply it into
your weights.The cumulative effect: a speedup of up to 30% in fuzzy match boiling,
and up to 5% on YTJD tests using a lot of fuzzy keys.  Match boiling got
slightly faster too.0.5.1Bugfix release.  In the original version, if a match didn't have any matches between
fuzzy keys (with a positive score), it ignored the weights of its exact keys and just
used the raw exact score.0.5Initial public release.