Package: red-string-grouper
Red String GrouperRecordEquivalenceDiscoverer based onString Grouper(Red String Grouper) is a python package that finds similarities between
rows/records of a table with multiple fields.It is an extension ofString Grouper, a library that
makes finding groups of similar strings within one or two lists of strings
easy — and fast.Installationpip install red_string_grouperUsageHow do we achieve matching for multiple fields and priorities?Import the functionrecord_linkage()fromred_string_grouperand specify
the fields of the table over which the comparisons will be made.importpandasaspdimportnumpyasnpfromred_string_grouperimportrecord_linkage,field,field_pairmatches=record_linkage(data_frames,fields_2b_matched_fuzzily,fields_2b_matched_exactly=None,hierarchical=True,**kwargs)This is a function that combines similarity-matching results of several fields of one
or two DataFrames (data_frames) and returns them in another DataFrame (matches).Examples are givenbelow.ParameterStatusDescriptiondata_framesRequiredEither apandas.DataFrameor a list of twopandas.DataFrames of strings which are the tables over which the comparisons will be made.fields_2b_matched_fuzzilyRequiredList of tuples.  Whendata_framesis aDataFrame, each tuple must be a triple:(〈field name〉, 〈weight〉, 〈field_kwargs〉)which is best input using the provided auxiliary function:field()in the following way:field(<field name>, weight=<weight>, **kwargs).〈field name〉 is a string denoting the name of a field indata_framewhose values are to be matched.Whendata_framesis a list of twoDataFrames, then each tuple must be a quadruple:(〈field name 1〉, 〈field name 2〉, 〈weight〉, 〈field_kwargs〉)which is best input using the provided auxiliary function:field_pair()in the following way:field_pair(<field name1>, <field name2>, weight=<weight>, **kwargs).〈field name1〉 is a string denoting the name of a field indata_frames[0].  Similarly,field name2〉 is a string denoting the name of a field indata_frames[1]whose values are to be matched with those of 〈field name1〉.〈weight〉 is a number (default: 1.0) that defines therelativeimportance of the field (or field-pair) to other fields (or field-pairs) -- the field's (or field-pair's) contribution to the mean similarity will be weighted by this number.〈weighted mean similarity score〉 =(Σfield〈weight〉field× 〈similarity〉field) / (Σfield〈weight〉field),whereΣfieldmeans "sum over fields".〈field_kwargs〉 is adictcapturing any keyword arguments to be passed toStringGrouperfor this field.  AnyStringGrouperkeyword argumentsspecified for the **kwargs of thefield()(orfield_pair()) function will be captured by 〈field_kwargs〉.fields_2b_matched_exactlyOptionalList of tuples.  Whendata_framesis aDataFrame, each tuple must be a pair:(〈field name〉, 〈weight〉).〈field name〉 is the name of a field indata_framewhich is to be matched exactly.  The auxiliary function:field()may be used to enter each tuple in the following way:field(<field name>, weight=<weight>, **kwargs).Whendata_framesis a list of twoDataFrames, then each tuple must be a triple:(〈field name 1〉, 〈field name 2〉, 〈weight〉)which is best input using the provided auxiliary function:field_pair()in the following way:field_pair(<field name1>, <field name2>, weight=<weight>).〈field name1〉 is a string denoting the name of a field indata_frames[0].  Similarly,field name2〉 is a string denoting the name of a field indata_frames[1]whose values are to be matched with those of 〈field name1〉.〈weight〉 has the same meaning as in parameterfields_2b_matched_fuzzily. Defaults toNone.hierarchicalOptionalbool.  Determines if the output DataFrame will have a hierarchical column-structure (True) or not (False). Defaults toTrue.**kwargsOptionalAnystring_grouperkeyword arguments may be specified here.  These will apply to all fields (or field-pairs) infields_2b_matched_fuzzily.  However, any keyword arguments already given infields_2b_matched_fuzzilytake precedence over those specified here.Examplesimportpandasaspdimportnumpyasnpfromred_string_grouperimportrecord_linkage,fieldPrepare the Input Data:Here's some sample data:file='data/us-cities-real-estate-sample-zenrows.csv'df=pd.read_csv(file,dtype=str,keep_default_na=False)# all stringsdfna=pd.read_csv(file,dtype=str,keep_default_na=True)# has null-valuesNote that the data has been read into memory as strings (dtype=str), sincered_string_grouperis based on string comparison.The dataset is a table of 10 000 records:len(df)10000Let us examine the data to determine which fields are good to compare
(that is, we will use only columns withoutnullorNaNdata).
At the same time, we will also check how many unique values each field has
and its maximum string-length (to inform our choice of its n-gram size):column_info=pd.concat([df.nunique().rename('#unique'),dfna.isna().any().rename('has null?'),df.applymap(len).max().rename('max strlen')],axis=1).rename_axis("column").reset_index()column_info[(column_info['#unique']>1)&(~column_info['has null?'])]column#uniquehas null?max strlen0zpid10000False101id10000False103imgSrc9940False2315detailUrl10000False1217statusText24False2611address10000False7214addressState51False215addressZipcode6446False516isUndisclosedAddress2False522isZillowOwned2False531has3DModel2False532hasVideo2False538isFeaturedListing2False539list2False5Thus we may set field'zpid'as the index, since it has exactly the same number
of unique values as the number of rows.zpidwill thus be used to identify
each row in the matching results.df.set_index('zpid',inplace=True)Callrecord_linkage():There is more than one way to achieve the same matching result.  But some ways are faster than others, depending on the data.Plot comparing run-times ofrecord_linkage()calls with and without grouping on a test field having a varying number of unique values in a 10 000-row DataFrame1. Grouping by fields that are to be matched exactlyNote that those fields that have very few unique values distributed among a
large number of rows, such as 'hasVideo' (2 unique values) and 'addressState'
(51 unique values), can be specified as "fields that are to be matched exactly"
(that is, in parameterfields_2b_matched_exactly) which can lead to a significant
performance boost.Behind the scenes, this allowsrecord_linkage()to avoid using cosine-
similarity matching on these fields (which is time-consuming, since many
matches are likely to be found), and instead group the data by these fields.In this way, cosine-similarity matching can be performed only on the other
fields (in parameterfields_2b_matched_fuzzily) for each group.On the other hand, grouping by 'addressZipcode' (which has 6446 unique values)
degrades performance, since the groups by this field are so many.To illustrate, the following call took
≈ 5 minutes to run:matches=record_linkage(df,fields_2b_matched_fuzzily=[field('statusText'),field('address')],fields_2b_matched_exactly=[field('addressZipcode',weight=2),field('addressState',weight=4),field('hasVideo')])whereas, the following call (which produces the same result) took ≈8 seconds to run:matches=record_linkage(df,fields_2b_matched_fuzzily=[field('statusText'),field('address'),field('addressZipcode',weight=2,min_similarity=0.9999)],fields_2b_matched_exactly=[field('addressState',weight=4),field('hasVideo')])Note that in the above calls, unspecified options, such asweightandstring_grouper keyword arguments) such asmin_similarityandngram_sizetake up their default values.  The default value forweightis1.0, while the default values for thestring_grouperkeyword arguments can be found atthis link.
Let's display the results:matchesExactly Matched FieldsFuzzily Matched FieldsaddressStatehasVideostatusTextaddressaddressZipcodeWeighted Mean Similarity Scoreleftsimilarityrightleftsimilarityrightleftsimilarityrightleft_zpidright_zpid207766780320776796431.000000WYfalseLot / Land for sale1.0Lot / Land for saleJlda Minor Sub Division LOT C, Buffalo, WY 828341.000000Jlda Minor Subdivision LOT C, Buffalo, WY 82834828341.082834207524405720753589430.997100OHfalseLot / Land for sale1.0Lot / Land for sale0 Township Road 118, Kimbolton, OH 437490.973904Township Road 118, Kimbolton, OH 43749437491.043749207767662220776768090.993867NDfalseLot / Land for sale1.0Lot / Land for sale4 55th St SE, Christine, ND 580150.9448022 55th St SE, Christine, ND 58015580151.058015207709306420788434980.993328SDfalseLot / Land for sale1.0Lot / Land for sale17 Sidney Park Rd, Custer, SD 577300.939948Sidney Park Rd, Custer, SD 57730577301.05773015069039220761236040.992909NJfalseLot / Land for sale1.0Lot / Land for sale5 Windsor Ln, Gladstone, NJ 079340.9361800 Windsor Ln, Gladstone, NJ 0793479341.07934..........................................207083751620720473180.978032HIfalseNew construction1.0New constructionD12C Plan, Kaikoi at Hoopili0.802290D12B Plan, Kaikoi at Hoopili967061.096706305578084900357580.977991MOfalseCondo for sale1.0Condo for sale210 N 17th St UNIT 203, Saint Louis, MO 631030.801920210 N 17th St UNIT 1202, Saint Louis, MO 63103631031.0631032071195670880865290.977983MIfalseCondo for sale1.0Condo for sale6533 E Jefferson Ave APT 426, Detroit, MI 482070.8018446533 E Jefferson Ave APT 102E, Detroit, MI 48207482071.0482072472630332472631360.977941IAfalseNew construction1.0New construction1 University Way #511, Iowa City, IA 522460.8014741 University Way #503, Iowa City, IA 52246522461.052246208365613820836561460.977873INfalseCondo for sale1.0Condo for sale3789 S Anderson Dr, Terre Haute, IN 478030.8008553776 S Anderson Dr, Terre Haute, IN 47803478031.04780394 rows × 12 columns2. No groupingThe results above can be obtained in yet another way.  However, as mentioned
above, it can take much longer to compute in cases where some fuzzily matched
fields have very few uniques values.The following call took ≈3 minutes 30 seconds to run:record_linkage(df,fields_2b_matched_fuzzily=[field('statusText'),field('address'),field('addressZipcode',weight=2,min_similarity=0.9999),field('addressState',weight=4,ngram_size=2,min_similarity=0.9999),field('hasVideo',min_similarity=0.9999)])statusTextaddressaddressZipcodehasVideoaddressStateWeighted Mean Similarity Scoreleftsimilarityrightleftsimilarityrightleftsimilarityrightleftsimilarityrightleftsimilarityrightleft_zpidright_zpid207766780320776796431.000000Lot / Land for sale1.0Lot / Land for saleJlda Minor Sub Division LOT C, Buffalo, WY 828341.000000Jlda Minor Subdivision LOT C, Buffalo, WY 82834828341.082834false1.0falseWY1.0WY207524405720753589430.997100Lot / Land for sale1.0Lot / Land for sale0 Township Road 118, Kimbolton, OH 437490.973904Township Road 118, Kimbolton, OH 43749437491.043749false1.0falseOH1.0OH207767662220776768090.993867Lot / Land for sale1.0Lot / Land for sale4 55th St SE, Christine, ND 580150.9448022 55th St SE, Christine, ND 58015580151.058015false1.0falseND1.0ND207709306420788434980.993328Lot / Land for sale1.0Lot / Land for sale17 Sidney Park Rd, Custer, SD 577300.939948Sidney Park Rd, Custer, SD 57730577301.057730false1.0falseSD1.0SD15069039220761236040.992909Lot / Land for sale1.0Lot / Land for sale5 Windsor Ln, Gladstone, NJ 079340.9361800 Windsor Ln, Gladstone, NJ 0793479341.07934false1.0falseNJ1.0NJ......................................................207083751620720473180.978032New construction1.0New constructionD12C Plan, Kaikoi at Hoopili0.802290D12B Plan, Kaikoi at Hoopili967061.096706false1.0falseHI1.0HI305578084900357580.977991Condo for sale1.0Condo for sale210 N 17th St UNIT 203, Saint Louis, MO 631030.801920210 N 17th St UNIT 1202, Saint Louis, MO 63103631031.063103false1.0falseMO1.0MO2071195670880865290.977983Condo for sale1.0Condo for sale6533 E Jefferson Ave APT 426, Detroit, MI 482070.8018446533 E Jefferson Ave APT 102E, Detroit, MI 48207482071.048207false1.0falseMI1.0MI2472630332472631360.977941New construction1.0New construction1 University Way #511, Iowa City, IA 522460.8014741 University Way #503, Iowa City, IA 52246522461.052246false1.0falseIA1.0IA208365613820836561460.977873Condo for sale1.0Condo for sale3789 S Anderson Dr, Terre Haute, IN 478030.8008553776 S Anderson Dr, Terre Haute, IN 47803478031.047803false1.0falseIN1.0IN94 rows × 16 columnsOne may choose to remove the field-values and output single-level column-
headings by setting hierarchical toFalse:record_linkage(df,fields_2b_matched_fuzzily=[field('statusText'),field('address'),field('addressZipcode',weight=2,min_similarity=0.9999),field('addressState',weight=4,ngram_size=2,min_similarity=0.9999),field('hasVideo',min_similarity=0.9999)],hierarchical=False)Weighted Mean Similarity ScorestatusTextaddressaddressZipcodehasVideoaddressStateleft_zpidright_zpid207766780320776796431.0000001.01.0000001.01.01.0207524405720753589430.9971001.00.9739041.01.01.0207767662220776768090.9938671.00.9448021.01.01.0207709306420788434980.9933281.00.9399481.01.01.015069039220761236040.9929091.00.9361801.01.01.0........................207083751620720473180.9780321.00.8022901.01.01.0305578084900357580.9779911.00.8019201.01.01.02071195670880865290.9779831.00.8018441.01.01.02472630332472631360.9779411.00.8014741.01.01.0208365613820836561460.9778731.00.8008551.01.01.094 rows × 6 columnsPerformancePlots of run-times ofrecord_linkage()vs the numbern_blocks[1]of blocks into which the right matrix-operand of the dataset (663 000 strings from sec__edgar_company_info.csv) was split before performing the string comparison.  As shown in the legend, each plot corresponds to the numbern_blocks[0]of blocks into which the left matrix-operand was split.String comparison, as implemented bystring_grouper, is essentially matrix
multiplication.  A DataFrame of strings is converted (tokenized) into a
matrix.  Then that matrix is multiplied by itself (or another) transposed.Here is an illustration of multiplication of two matricesDandMT:It turns out that when the matrix (or DataFrame) is very large, the computer
proceeds quite slowly with the multiplication (apparently due to the RAM being
too full).  Some computers give up with anOverflowError.To circumvent this issue,red_string_grouperallows to divide the DataFrame
into smaller chunks (or blocks) and multiply the chunks one pair at a time
instead to get the same result:But surprise ... the run-time of the process is sometimes drastically reduced
as a result.  For example, the speed-up of the following call is about 500%
(here, the DataFrame is divided into 200 blocks on the right operand, that is,
1 block on the left × 200 on the right) compared to the same call with no
splitting [n_blocks=(1, 1), the default, which is equivalent tomatch_stringscall ofstring_grouper(versions 0.5.0 and earlier)]:# 663000 records:companies=pd.read_csv('data/sec__edgar_company_info.csv')# the following call produces the same result as# string_grouper using# match_strings(companies['Company Name'])# but is more than 3 times faster!record_linkage(companies,fields_2b_matched_fuzzily=[field('Company Name')],n_blocks=(1,200))Further exploration of the block number space (see plot above) has revealed that for any fixed
number of right blocks, the run-time gets longer the larger the number of left
blocks specified.  For this reason, it is recommendednotto split the left matrix.So what are the optimum block number values for any given DataFrame? That is
anyone's guess, and the answer may vary from computer to computer.We however encourage the user to make judicious use of then_blocksparameter to boost performance ofrecord_linkage().