Package: cgvdiit-string-grouper
String GrouperClick to see imageThe image displayed above is a visualization of the graph-structure of one of the groups of strings found bystring_grouper.  Each circle (node) represents a string, and each connecting arc (edge) represents a match between a pair of strings with a similarity score above a given threshold score (here0.8).Thecentroidof the group, as determined bystring_grouper(seetutorials/group_representatives.mdfor an explanation), is the largest node, also with the most edges originating from it.  A thick line in the image denotes a strong similarity between the nodes at its ends, while a faint thin line denotes weak similarity.The power ofstring_grouperis discernible from this image: in large datasets,string_grouperis often able to resolve indirect associations between strings even when, say, due to memory-resource-limitations, direct matches between those strings cannot be computed using conventional methods with a lower threshold similarity score.———This image was designed using the graph-visualization software Gephi 0.9.2 with data generated bystring_grouperoperating on thesec__edgar_company_info.csvsample data file.string_grouperis a library that makes finding groups of similar strings within a single, or multiple, lists of strings easy â€” and fast.string_grouperusestf-idfto calculatecosine similaritieswithin a single list or between two lists of strings. The full process is described in the blogSuper Fast String Matching in Python.Installingpip install string-grouperUsageimportpandasaspdfromstring_grouperimportmatch_strings,match_most_similar,\group_similar_strings,compute_pairwise_similarities,\StringGrouperAs shown above, the library may be used together withpandas, and contains four high level functions (match_strings,match_most_similar,group_similar_strings, andcompute_pairwise_similarities) that can be used directly, and one class (StringGrouper) that allows for a more interactive approach.The permitted calling patterns of the four functions, and their return types, are:FunctionParameterspandasReturn Typematch_strings(master, **kwargs)DataFramematch_strings(master, duplicates, **kwargs)DataFramematch_strings(master, master_id=id_series, **kwargs)DataFramematch_strings(master, duplicates, master_id, duplicates_id, **kwargs)DataFramematch_most_similar(master, duplicates, **kwargs)Series(if kwargignore_index=True) otherwiseDataFrame(default)match_most_similar(master, duplicates, master_id, duplicates_id, **kwargs)DataFramegroup_similar_strings(strings_to_group, **kwargs)Series(if kwargignore_index=True) otherwiseDataFrame(default)group_similar_strings(strings_to_group, strings_id, **kwargs)DataFramecompute_pairwise_similarities(string_series_1, string_series_2, **kwargs)SeriesIn the rest of this document the names,SeriesandDataFrame, refer to the familiarpandasobject types.Parameters:NameDescriptionmasterASeriesof strings to be matched with themselves (or with those induplicates).duplicatesASeriesof strings to be matched with those ofmaster.master_id(orid_series)ASeriesof IDs corresponding to the strings inmaster.duplicates_idASeriesof IDs corresponding to the strings induplicates.strings_to_groupASeriesof strings to be grouped.strings_idASeriesof IDs corresponding to the strings instrings_to_group.string_series_1(_2)ASeriesof strings each of which is to be compared with its corresponding string instring_series_2(_1).**kwargsKeyword arguments (seebelow).New in version 0.6.0: each of the high-level functions listed above also has aStringGroupermethod counterpart of the same name and parameters.  Calling such a method of any instance ofStringGrouperwill not rebuild the instance's underlying corpus to make string-comparisons but rather use it to perform the string-comparisons.  The input Series to the method (master,duplicates, and so on) will thus be encoded, or transformed, into tf-idf matrices, using this corpus.  For example:# Build a corpus using strings in the pandas Series master:sg=StringGrouper(master)# The following method-calls will compare strings first in# pandas Series new_master_1 and next in new_master_2# using the corpus already built above without rebuilding or# changing it in any way:matches1=sg.match_strings(new_master_1)matches2=sg.match_strings(new_master_2)Functions:match_stringsReturns aDataFramecontaining similarity-scores of all matching pairs of highly similar strings frommaster(andduplicatesif given).  Each matching pair in the output appears in its own row/record consisting ofits "left" part: a string (with/without its index-label) frommaster,its similarity score, andits "right" part: a string (with/without its index-label) fromduplicates(ormasterifduplicatesis not given),in that order.  Thus the column-names of the output are a collection of three groups:The name ofmasterand the name(s) of its index (or index-levels) all prefixed by the string'left_','similarity'whose column has the similarity-scores as values, andThe name ofduplicates(ormasterifduplicatesis not given) and the name(s) of its index (or index-levels) prefixed by the string'right_'.Indexes (or their levels) only appear when the keyword argumentignore_index=False(the default). (Seetutorials/ignore_index_and_replace_na.mdfor a demonstration.)If eithermasterorduplicateshas no name, it assumes the name'side'which is then prefixed as described above.  Similarly, if any of the indexes (or index-levels) has no name it assumes itspandasdefault name ('index','level_0', and so on) and is then prefixed as described above.In other words, if only parametermasteris given, the function will return pairs of highly similar strings withinmaster.  This can be seen as a self-join where both'left_'and'right_'prefixed columns come frommaster. If both parametersmasterandduplicatesare given, it will return pairs of highly similar strings betweenmasterandduplicates. This can be seen as an inner-join where'left_'and'right_'prefixed columns come frommasterandduplicatesrespectively.The function also supports optionally inputting IDs (master_idandduplicates_id) corresponding to the strings being matched.  In which case, the output includes two additional columns whose names are the names of these optionalSeriesprefixed by'left_'and'right_'accordingly, and containing the IDs corresponding to the strings in the output.  If any of theseSerieshas no name, then it assumes the name'id'and is then prefixed as described above.match_most_similarIfignore_index=True, returns aSeriesof strings, where for each string induplicatesthe most similar string inmasteris returned.  If there are no similar strings inmasterfor a given string induplicates(because there is no potential match where the cosine similarity is above the threshold [default: 0.8]) then the original string induplicatesis returned.  The outputSeriesthus has the same length and index asduplicates.For example, if an inputSerieswith the values\['foooo', 'bar', 'baz'\]is passed as the argumentmaster, and\['foooob', 'bar', 'new'\]as the values of the argumentduplicates, the function will return aSerieswith values:\['foooo', 'bar', 'new'\].The name of the outputSeriesis the same as that ofmasterprefixed with the string'most_similar_'.  Ifmasterhas no name, it is assumed to have the name'master'before being prefixed.Ifignore_index=False(the default),match_most_similarreturns aDataFramecontaining the sameSeriesdescribed above as one of its columns.  So it inherits the same index and length asduplicates.  The rest of its columns correspond to the index (or index-levels) ofmasterand thus contain the index-labels of the most similar strings being output as values.  If there are no similar strings inmasterfor a given string induplicatesthen the value(s) assigned to this index-column(s) for that string isNaNby default.  However, if the keyword argumentreplace_na=True, then theseNaNvalues are replaced with the index-label(s) of that string induplicates.  Note that such replacements can only occur if the indexes ofmasterandduplicateshave the same number of levels.  (Seetutorials/ignore_index_and_replace_na.mdfor a demonstration.)Each column-name of the outputDataFramehas the same name as its corresponding column, index, or index-level ofmasterprefixed with the string'most_similar_'.If both parametersmaster_idandduplicates_idare also given, then aDataFrameis always returned with the same column(s) as described above, but with an additional column containing those IDs from these inputSeriescorresponding to the output strings.  This column's name is the same as that ofmaster_idprefixed in the same way as described above.  Ifmaster_idhas no name, it is assumed to have the name'master_id'before being prefixed.group_similar_stringsTakes a singleSeriesof strings (strings_to_group) and groups them by assigning to each string one string fromstrings_to_groupchosen as the group-representative for each group of similar strings found. (Seetutorials/group_representatives.mdfor details on how the the group-representatives are chosen.)Ifignore_index=True, the output is aSeries(with the same name asstrings_to_groupprefixed by the string'group_rep_') of the same length and index asstrings_to_groupcontaining the group-representative strings.  Ifstrings_to_grouphas no name then the name of the returnedSeriesis'group_rep'.For example, an input Series with values:\['foooo', 'foooob', 'bar'\]will return\['foooo', 'foooo', 'bar'\].  Here'foooo'and'foooob'are grouped together into group'foooo'because they are found to be similar.  Another example can be foundbelow.Ifignore_index=False, the output is aDataFramecontaining the above outputSeriesas one of its columns with the same name.  The remaining column(s) correspond to the index (or index-levels) ofstrings_to_groupand contain the index-labels of the group-representatives as values.  These columns have the same names as their counterparts prefixed by the string'group_rep_'.Ifstrings_idis also given, then the IDs fromstrings_idcorresponding to the group-representatives are also returned in an additional column (with the same name asstrings_idprefixed as described above).  Ifstrings_idhas no name, it is assumed to have the name'id'before being prefixed.compute_pairwise_similaritiesReturns aSeriesof cosine similarity scores the same length and index asstring_series_1.  Each score is the cosine similarity between the pair of strings in the same position (row) in the two inputSeries,string_series_1andstring_series_2, as the position of the score in the outputSeries.  This can be seen as an element-wise comparison between the two inputSeries.All functions are built using a classStringGrouper. This class can be used through pre-defined functions, for example the four high level functions above, as well as using a more interactive approach where matches can be added or removed if needed by calling theStringGrouperclass directly.Options:kwargsAll keyword arguments not mentioned in the function definitions above are used to update the default settings. The following optional arguments can be used:ngram_size: The amount of characters in each n-gram. Default is3.regex: The regex string used to clean-up the input string. Default isr"[,-./]|\s".ignore_case: Determines whether or not letter case in strings should be ignored. Defaults toTrue.tfidf_matrix_dtype: The datatype for the tf-idf values of the matrix components. Allowed values arenumpy.float32andnumpy.float64.  Default isnumpy.float32.  (Note:numpy.float32often leads to faster processing and a smaller memory footprint albeit less numerical precision thannumpy.float64.)max_n_matches: The maximum number of matching strings inmasterallowed per string induplicates. Default is the total number of strings inmaster.min_similarity: The minimum cosine similarity for two strings to be considered a match.
Defaults to0.8number_of_processes: The number of processes used by the cosine similarity calculation. Defaults tonumber of cores on a machine - 1.ignore_index: Determines whether indexes are ignored or not.  IfFalse(the default), index-columns will appear in the output, otherwise not.  (Seetutorials/ignore_index_and_replace_na.mdfor a demonstration.)replace_na: For functionmatch_most_similar, determines whetherNaNvalues in index-columns are replaced or not by index-labels fromduplicates. Defaults toFalse.  (Seetutorials/ignore_index_and_replace_na.mdfor a demonstration.)include_zeroes: Whenmin_similarity≤ 0, determines whether zero-similarity matches appear in the output.  Defaults toTrue.  (Seetutorials/zero_similarity.md.)Note:Ifinclude_zeroesisTrueand the kwargmax_n_matchesis set then it must be sufficiently high to captureallnonzero-similarity-matches, otherwise an error is raised andstring_groupersuggests an alternative value formax_n_matches.  To allowstring_grouperto automatically use the appropriate value formax_n_matchesthen do not set this kwarg at all.group_rep: For functiongroup_similar_strings, determines how group-representatives are chosen.  Allowed values are'centroid'(the default) and'first'.  Seetutorials/group_representatives.mdfor an explanation.force_symmetries: In cases whereduplicatesisNone, specifies whether corrections should be made to the results to account for symmetry, thus compensating for those losses of numerical significance which violate the symmetries. Defaults toTrue.n_blocks: This parameter is a tuple of twoints provided to help boost performance, if possible, of processing large DataFrames (seeSubsection Performance), by splitting the DataFrames inton_blocks[0]blocks for the left operand (of the underlying matrix multiplication) and inton_blocks[1]blocks for the right operand before performing the string-comparisons block-wise.  Defaults to'guess', in which case the numbers of blocks are estimated based on previous empirical results.  Ifn_blocks = 'auto', then splitting is done automatically in the event of anOverflowError.ExamplesIn this section we will cover a few use cases for which string_grouper may be used. We will use the same data set of company names as used in:Super Fast String Matching in Python.Find all matches within a single data setimportpandasaspdimportnumpyasnpfromstring_grouperimportmatch_strings,match_most_similar,\group_similar_strings,compute_pairwise_similarities,\StringGroupercompany_names='/media/chris/data/dev/name_matching/data/sec_edgar_company_info.csv'# We only look at the first 50k as an example:companies=pd.read_csv(company_names)[0:50000]# Create all matches:matches=match_strings(companies['Company Name'])# Look at only the non-exact matches:matches[matches['left_Company Name']!=matches['right_Company Name']].head()left_indexleft_Company Namesimilarityright_Company Nameright_index15140210, LLC0.87029190210 LLC42111671651 800 MUTUALS ADVISOR SERIES0.9316151 800 MUTUALS ADVISORS SERIES1661681661 800 MUTUALS ADVISORS SERIES0.9316151 800 MUTUALS ADVISOR SERIES1651721681 800 RADIATOR FRANCHISE INC1.0000001-800-RADIATOR FRANCHISE INC.2011781731 FINANCIAL MARKETPLACE SECURITIES LLC        ...0.9493641 FINANCIAL MARKETPLACE SECURITIES, LLC174Find all matches in between two data sets.Thematch_stringsfunction finds similar items between two data sets as well. This can be seen as an inner join between two data sets:# Create a small set of artificial company names:duplicates=pd.Series(['S MEDIA GROUP','012 SMILE.COMMUNICATIONS','foo bar','B4UTRADE COM CORP'])# Create all matches:matches=match_strings(companies['Company Name'],duplicates)matchesleft_indexleft_Company Namesimilarityright_sideright_index012012 SMILE.COMMUNICATIONS LTD0.944092012 SMILE.COMMUNICATIONS1149777B.A.S. MEDIA GROUP0.854383S MEDIA GROUP0249855B4UTRADE COM CORP1.000000B4UTRADE COM CORP3349856B4UTRADE COM INC0.810217B4UTRADE COM CORP3449857B4UTRADE CORP0.878276B4UTRADE COM CORP3Out of the four company names induplicates, three companies are found in the original company data set. One company is found three times.Finding duplicates from a (database extract to) DataFrame where IDs for rows are supplied.A very common scenario is the case where duplicate records for an entity have been entered into a database. That is, there are two or more records where a name field has slightly different spelling. For example, "A.B. Corporation" and "AB Corporation". Using the optional 'ID' parameter in thematch_stringsfunction duplicates can be found easily. Atutorialthat steps though the process with an example data set is available.For a second data set, find only the most similar matchIn the example above, it's possible that multiple matches are found for a single string. Sometimes we just want a string to match with a single most similar string. If there are no similar strings found, the original string should be returned:# Create a small set of artificial company names:new_companies=pd.Series(['S MEDIA GROUP','012 SMILE.COMMUNICATIONS','foo bar','B4UTRADE COM CORP'],\name='New Company')# Create all matches:matches=match_most_similar(companies['Company Name'],new_companies,ignore_index=True)# Display the results:pd.concat([new_companies,matches],axis=1)New Companymost_similar_Company Name0S MEDIA GROUPB.A.S. MEDIA GROUP1012 SMILE.COMMUNICATIONS012 SMILE.COMMUNICATIONS LTD2foo barfoo bar3B4UTRADE COM CORPB4UTRADE COM CORPDeduplicate a single data set and show items with most duplicatesThegroup_similar_stringsfunction groups strings that are similar using a single linkage clustering algorithm. That is, if item A and item B are similar; and item B and item C are similar; but the similarity between A and C is below the threshold; then all three items are grouped together.# Add the grouped strings:companies['deduplicated_name']=group_similar_strings(companies['Company Name'],ignore_index=True)# Show items with most duplicates:companies.groupby('deduplicated_name')['Line Number'].count().sort_values(ascending=False).head(10)deduplicated_name
ADVISORS DISCIPLINED TRUST                                      1824
AGL LIFE ASSURANCE CO SEPARATE ACCOUNT                           183
ANGELLIST-ART-FUND, A SERIES OF ANGELLIST-FG-FUNDS, LLC          116
AMERICREDIT AUTOMOBILE RECEIVABLES TRUST 2001-1                   87
ACE SECURITIES CORP. HOME EQUITY LOAN TRUST, SERIES 2006-HE2      57
ASSET-BACKED PASS-THROUGH CERTIFICATES SERIES 2004-W1             40
ALLSTATE LIFE GLOBAL FUNDING TRUST 2005-3                         39
ALLY AUTO RECEIVABLES TRUST 2014-1                                33
ANDERSON ROBERT E /                                               28
ADVENT INTERNATIONAL GPE VIII LIMITED PARTNERSHIP                 28
Name: Line Number, dtype: int64Thegroup_similar_stringsfunction also works with IDs: imagine aDataFrame(customers_df) with the following content:# Create a small set of artificial customer names:customers_df=pd.DataFrame([('BB016741P','Mega Enterprises Corporation'),('CC082744L','Hyper Startup Incorporated'),('AA098762D','Hyper Startup Inc.'),('BB099931J','Hyper-Startup Inc.'),('HH072982K','Hyper Hyper Inc.')],columns=('Customer ID','Customer Name')).set_index('Customer ID')# Display the data:customers_dfCustomer NameCustomer IDBB016741PMega Enterprises CorporationCC082744LHyper Startup IncorporatedAA098762DHyper Startup Inc.BB099931JHyper-Startup Inc.HH072982KHyper Hyper Inc.The output ofgroup_similar_stringscan be directly used as a mapping table:# Group customers with similar names:customers_df[["group-id","name_deduped"]]=\group_similar_strings(customers_df["Customer Name"])# Display the mapping table:customers_dfCustomer Namegroup-idname_dedupedCustomer IDBB016741PMega Enterprises CorporationBB016741PMega Enterprises CorporationCC082744LHyper Startup IncorporatedCC082744LHyper Startup IncorporatedAA098762DHyper Startup Inc.AA098762DHyper Startup Inc.BB099931JHyper-Startup Inc.AA098762DHyper Startup Inc.HH072982KHyper Hyper Inc.HH072982KHyper Hyper Inc.Note that herecustomers_dfinitially had only one column "Customer Name" (before thegroup_similar_stringsfunction call); and it acquired two more columns "group-id" (the index-column) and "name_deduped" after the call through a "setting with enlargement" (apandasfeature).Simply compute the cosine similarities of pairs of stringsSometimes we have pairs of strings that have already been matched but whose similarity scores need to be computed.  For this purpose we provide the functioncompute_pairwise_similarities:# Create a small DataFrame of pairs of strings:pair_s=pd.DataFrame([('Mega Enterprises Corporation','Mega Enterprises Corporation'),('Hyper Startup Inc.','Hyper Startup Incorporated'),('Hyper Startup Inc.','Hyper Startup Inc.'),('Hyper Startup Inc.','Hyper-Startup Inc.'),('Hyper Hyper Inc.','Hyper Hyper Inc.'),('Mega Enterprises Corporation','Mega Enterprises Corp.')],columns=('left','right'))# Display the data:pair_sleftright0Mega Enterprises CorporationMega Enterprises Corporation1Hyper Startup Inc.Hyper Startup Incorporated2Hyper Startup Inc.Hyper Startup Inc.3Hyper Startup Inc.Hyper-Startup Inc.4Hyper Hyper Inc.Hyper Hyper Inc.5Mega Enterprises CorporationMega Enterprises Corp.# Compute their cosine similarities and display them:pair_s['similarity']=compute_pairwise_similarities(pair_s['left'],pair_s['right'])pair_sleftrightsimilarity0Mega Enterprises CorporationMega Enterprises Corporation1.0000001Hyper Startup Inc.Hyper Startup Incorporated0.6336202Hyper Startup Inc.Hyper Startup Inc.1.0000003Hyper Startup Inc.Hyper-Startup Inc.1.0000004Hyper Hyper Inc.Hyper Hyper Inc.1.0000005Mega Enterprises CorporationMega Enterprises Corp.0.826463The StringGrouper classThe four functions mentioned above all create aStringGrouperobject behind the scenes and call different functions on it. TheStringGrouperclass keeps track of all tuples of similar strings and creates the groups out of these. Since matches are often not perfect, a common workflow is to:Create matchesManually inspect the resultsAdd and remove matches where necessaryCreate groups of similar stringsTheStringGrouperclass allows for this without having to re-calculate the cosine similarity matrix. See below for an example.company_names='/media/chris/data/dev/name_matching/data/sec_edgar_company_info.csv'companies=pd.read_csv(company_names)Create matches# Create a new StringGrouperstring_grouper=StringGrouper(companies['Company Name'],ignore_index=True)# Check if the ngram function does what we expect:string_grouper.n_grams('McDonalds')['McD', 'cDo', 'Don', 'ona', 'nal', 'ald', 'lds']# Now fit the StringGrouper - this will take a while since we are calculating cosine similarities on 600k stringsstring_grouper=string_grouper.fit()# Add the grouped stringscompanies['deduplicated_name']=string_grouper.get_groups()Suppose we know that PWC HOLDING CORP and PRICEWATERHOUSECOOPERS LLP are the same company. StringGrouper will not match these since they are not similar enough.companies[companies.deduplicated_name.str.contains('PRICEWATERHOUSECOOPERS LLP')]Line NumberCompany NameCompany CIK Keydeduplicated_name478441478442PRICEWATERHOUSECOOPERS LLP                              /TA1064284PRICEWATERHOUSECOOPERS LLP                              /TA478442478443PRICEWATERHOUSECOOPERS LLP1186612PRICEWATERHOUSECOOPERS LLP                              /TA478443478444PRICEWATERHOUSECOOPERS SECURITIES LLC1018444PRICEWATERHOUSECOOPERS LLP                              /TAcompanies[companies.deduplicated_name.str.contains('PWC')]Line NumberCompany NameCompany CIK Keydeduplicated_name485535485536PWC CAPITAL INC.1690640PWC CAPITAL INC.485536485537PWC HOLDING CORP1456450PWC HOLDING CORP485537485538PWC INVESTORS, LLC1480311PWC INVESTORS, LLC485538485539PWC REAL ESTATE VALUE FUND I LLC1668928PWC REAL ESTATE VALUE FUND I LLC485539485540PWC SECURITIES CORP                                     /BD1023989PWC SECURITIES CORP                                     /BD485540485541PWC SECURITIES CORPORATION1023989PWC SECURITIES CORPORATION485541485542PWCC LTD1172241PWCC LTD485542485543PWCG BROKERAGE, INC.67301PWCG BROKERAGE, INC.We can add these with the add function:string_grouper=string_grouper.add_match('PRICEWATERHOUSECOOPERS LLP','PWC HOLDING CORP')companies['deduplicated_name']=string_grouper.get_groups()# Now lets check again:companies[companies.deduplicated_name.str.contains('PRICEWATERHOUSECOOPERS LLP')]Line NumberCompany NameCompany CIK Keydeduplicated_name478441478442PRICEWATERHOUSECOOPERS LLP                              /TA1064284PRICEWATERHOUSECOOPERS LLP                              /TA478442478443PRICEWATERHOUSECOOPERS LLP1186612PRICEWATERHOUSECOOPERS LLP                              /TA478443478444PRICEWATERHOUSECOOPERS SECURITIES LLC1018444PRICEWATERHOUSECOOPERS LLP                              /TA485536485537PWC HOLDING CORP1456450PRICEWATERHOUSECOOPERS LLP                              /TAThis can also be used to merge two groups:string_grouper=string_grouper.add_match('PRICEWATERHOUSECOOPERS LLP','ZUCKER MICHAEL')companies['deduplicated_name']=string_grouper.get_groups()# Now lets check again:companies[companies.deduplicated_name.str.contains('PRICEWATERHOUSECOOPERS LLP')]Line NumberCompany NameCompany CIK Keydeduplicated_name478441478442PRICEWATERHOUSECOOPERS LLP                              /TA1064284PRICEWATERHOUSECOOPERS LLP                              /TA478442478443PRICEWATERHOUSECOOPERS LLP1186612PRICEWATERHOUSECOOPERS LLP                              /TA478443478444PRICEWATERHOUSECOOPERS SECURITIES LLC1018444PRICEWATERHOUSECOOPERS LLP                              /TA485536485537PWC HOLDING CORP1456450PRICEWATERHOUSECOOPERS LLP                              /TA662585662586ZUCKER MICHAEL1629018PRICEWATERHOUSECOOPERS LLP                              /TA662604662605ZUCKERMAN MICHAEL1303321PRICEWATERHOUSECOOPERS LLP                              /TA662605662606ZUCKERMAN MICHAEL1496366PRICEWATERHOUSECOOPERS LLP                              /TAWe can remove strings from groups in the same way:string_grouper=string_grouper.remove_match('PRICEWATERHOUSECOOPERS LLP','ZUCKER MICHAEL')companies['deduplicated_name']=string_grouper.get_groups()# Now lets check again:companies[companies.deduplicated_name.str.contains('PRICEWATERHOUSECOOPERS LLP')]Line NumberCompany NameCompany CIK Keydeduplicated_name478441478442PRICEWATERHOUSECOOPERS LLP                              /TA1064284PRICEWATERHOUSECOOPERS LLP                              /TA478442478443PRICEWATERHOUSECOOPERS LLP1186612PRICEWATERHOUSECOOPERS LLP                              /TA478443478444PRICEWATERHOUSECOOPERS SECURITIES LLC1018444PRICEWATERHOUSECOOPERS LLP                              /TA485536485537PWC HOLDING CORP1456450PRICEWATERHOUSECOOPERS LLP                              /TAPerformanceSemilogx plots of run-times ofmatch_strings()vs the number of blocks (n_blocks[1]) into which the right matrix-operand of the dataset (663 000 strings from sec__edgar_company_info.csv) was split before performing the string comparison.  As shown in the legend, each plot corresponds to the numbern_blocks[0]of blocks into which the left matrix-operand was split.String comparison, as implemented bystring_grouper, is essentially matrix
multiplication.  A pandas Series of strings is converted (tokenized) into a
matrix.  Then that matrix is multiplied by itself (or another) transposed.Here is an illustration of multiplication of two matricesDandMT:It turns out that when the matrix (or Series) is very large, the computer
proceeds quite slowly with the multiplication (apparently due to the RAM being
too full).  Some computers give up with anOverflowError.To circumvent this issue,string_groupernow allows the division of the Series
into smaller chunks (or blocks) and multiplies the chunks one pair at a time
instead to get the same result:But surprise ... the run-time of the process is sometimes drastically reduced
as a result.  For example, the speed-up of the following call is about 500%
(here, the Series is divided into 200 blocks on the right operand, that is,
1 block on the left × 200 on the right) compared to the same call with no
splitting [n_blocks=(1, 1), the default, which is what previous versions
(0.5.0 and earlier) ofstring_grouperdid]:# A DataFrame of 668 000 records:companies=pd.read_csv('data/sec__edgar_company_info.csv')# The following call is more than 6 times faster than earlier versions of# match_strings() (that is, when n_blocks=(1, 1))!match_strings(companies['Company Name')],n_blocks=(1,200))Further exploration of the block number space (see plot above) has revealed that for any fixed
number of right blocks, the run-time gets longer the larger the number of left
blocks specified.  For this reason, it is recommendednotto split the left matrix.In general,total runtime=n_blocks[0]×n_blocks[1]×mean runtime per block-pair=Left Operand Size×Right Operand Size×mean runtime per block-pair/ (Left Block Size×Right Block Size)So for given left and right operands, minimizing thetotal runtimeis the same as minimizing theruntime per string-pair comparison≝mean runtime per block-pair/ (Left Block Size×Right Block Size)Below is a log-log-log contour plotof theruntime per string-pair comparisonscaled by its value
atLeft Block Size=Right Block Size= 5000.  Here,Block Sizeis the number of strings in that block, andmean runtime per block-pairis the time taken for the following call to run:# note the parameter order!match_strings(right_Series,left_Series,n_blocks=(1,1))whereleft_Seriesandright_Series, corresponding toLeft BlockandRight Blockrespectively, are random subsets of the Seriescompanies['Company Name')]from thesec__edgar_company_info.csvsample data file.It can be seen that whenright_Seriesis roughly the size of 80 000 (denoted by the
white dashed line in the contour plot above), the runtime per string-pair comparison is at
its lowest for any fixedleft_Seriessize.  AboveRight Block Size= 80 000, the
matrix-multiplication routine begins to feel the limits of the computer's
available memory space and thus its performance deteriorates, as evidenced by the increase
in runtime per string-pair comparison there (above the white dashed line).  This knowledge
could serve as a guide for estimating the optimum block numbers —
namely those that divide the Series into blocks of size roughly equal to
80 000 for the right operand (orright_Series).So what are the optimum block number values foranygiven Series? That is
anyone's guess, and may likely depend on the data itself.  Furthermore, as hinted above,
the answer may vary from computer to computer.We however encourage the user to make judicious use of then_blocksparameter to boost performance ofstring_grouperwhenever possible.