Package: textpack
What is this?TextPack efficiently groups similar values in large (or small) datasets. Under the hood, it builds a document term matrix of n-grams assigned a TF-IDF score. It then uses matrix multiplication to quickly calculate the cosine similarity between these values. For a technical explination,I wrote a blog post.Why do I care?If you're a analyst, journalist, data scientist or similar and ever had a spreadsheet, SQL table or JSON string filled with inconsistent inputs like this:rowfullname1John F. Doe2Esquivel, Mara3Doe, John F4Whyte, Luke5Doe, John FrancisAnd you've wanted to perform some kind of analysis – perhaps in a Pivot Table or a Group By statement – but are hindered by the deviations in spelling and formatting, you can use TextPack to comb hundreds to thousands of cells in seconds and create a third column like this:rowfullnamename_groups1John F. DoeDoe John F2Esquivel, MaraEsquivel Mara3Doe, John FDoe John F4Whyte, LukeWhyte Luke5Doe, John FrancisDoe John FWe can then group byname_groupsand perform our analysis.You can also group across multiple columns. For instance, given the following:rowmakemodel1ToyotaCamry2toytacamry DXV3FordF-1504ToyotaTundra5HondaAccordYou can group acrossmakeandmodelto create:rowmakemodelcar_groups1ToyotaCamrytoyotacamry2toytacamry DXVtoyotacamry3FordF-150fordf1504ToyotaTundratoyotatundra5HondaAccordhondaaccordBoom.How do I use it?Installationpip install textpackImport modulefrom textpack import tpInstantiate TextPacktp.Textpack(df, columns_to_group, match_threshold=0.75, ngram_remove=r'[,-./]', ngram_length=3)Class parameters:df(required): A Pandas' DataFrame containing the dataset to groupcolumns_to_group(required): A list or string matching the column headers you'd like to parse and groupmatch_threshold(optional): This is a floating point number between 0 and 1 that represents the cosine similarity threshold we'll use to determine if two strings should be grouped. The closer the threshold to 1, the higher the similarity will need to be.ngram_remove(optional): A regular expression you can use to filter characters out of your strings when we build our n-gramsngram_length(optional): The length of our n-grams. This can be used in tandem withmatch_thresholdto find the sweet spot for grouping your dataset. If TextPack is running slow, it's usually a sign to consider raising the n-gram length.TextPack can also be instantiated using the following helpers, each of which is just a wrapper that converts a data format to a Pandas DataFrame and then passes it to TextPack. Thus, they all require a file path,columns_to_groupand take the same three optional parameters as callinTextPackdirectly.tp.read_csv(csv_path, columns_to_group, match_threshold=0.75, ngram_remove=r'[,-./]', ngram_length=3)tp.read_excel(excel_path, columns_to_group, sheet_name=None, match_threshold=0.75, ngram_remove=r'[,-./]', ngram_length=3)tp.read_json(json_path, columns_to_group, match_threshold=0.75, ngram_remove=r'[,-./]', ngram_length=3)Run Textpack and group valuesTextPack objects have the following public properties:df: The dataframe used internally by TextPack – manipulate as you see fitgroup_lookup: A Python dictionary built bybuild_group_lookupand then used byadd_grouped_column_to_datato lookup each value that has a group. It looks like this:{ 
    'John F. Doe': 'Doe John F',
    'Doe, John F': 'Doe John F',
    'Doe, John Francis': 'Doe John F'
}Textpack objects also have the following public methods:build_group_lookup(): Runs the cosine similarity analysis and buildsgroup_lookup.add_grouped_column_to_data(column_name='Group'): Uses vectorization to map values to groups viagroup_lookupand add the newGroupcolumn to the DataFrame. The column header can be set viacolumn_name.set_match_threshold(match_threshold): Modify the match threshold internally.set_ngram_remove(ngram_remove): Modify the n-gram regex filter internally.set_ngram_length(ngram_length): Modify the n-gram length internally.run(column_name='Group'): A helper function that callsbuild_group_lookupfollowed byadd_grouped_column_to_data.Export our grouped datasetexport_json(export_path)export_csv(export_path)A simple examplefrom textpack import tp

cars = tp.read_csv('./cars.csv', ['make', 'model'], match_threshold=0.8, ngram_length=5)

cars.run()

cars.export_csv('./cars-grouped.csv')How does it work?As mentioned above, under the hood, we're building a document term matrix of n-grams assigned a TF-IDF score. We're then using matrix multipcation to quickly calculate the cosine similarity between these values.I wrotethis detailed blog postto explian how TextPack works behind the scene and why it's fast. Check it out!