Package: bln-tools
Big Local News Tools for JournalistsHarmonizer: attempts to standardize data.Labeler: machine learning assisted data labeling / categorization.PowerBI: tools for scraping PowerBI dashboards.HarmonizerThe harmonizer attempts to standardize data. For instance, a column of data
like this:Apple Inc.APPLE Inc.APPLE INCAPPLEWould be standardized to "Apple Inc.", so all four entries would have the same
value. The methodology, usage, and examples are below. Understanding the
methodology will help to understand how it is used.Methodology:Harmonizing a column consists of two phases:OPTIONAL: Identify "stop words." Stop words are commonly occurring words
that carry little semantic value; in normal language, these are words
like "a", "the", "of", etc., but in the context of something like
corporate names, they may be words like "LLC", "CO", "INTERNATIONAL",
"GROUP", "DBA", etc. Identifying and removing these reduces the similarity
between unrelated companies, i.e. "ACME INTERNATIONAL" and "APPLE
INTERNATIONAL" might be ~50% similar, but once you strip "INTERNATIONAL"
from the names, they are 0% similar, which is most often what is desired.Standardize the names; this consists of several steps:clean the target column:uppercase all tokens (words)remove punctuationremove stop words (loaded fromstop_words.csvgenerated in optional
step 1; if this file doesn't exist, it doesn't remove any stop words)sort the target column (allows this algorithm to run in O(1) time)compare the current value to the previous value and calculate their
similarity (this program uses the harmonic mean of the partial ratio and
the sorted token ratio, see the python libraryfuzzywuzzyfor the meaning of
these)if the similarity is above the given threshold, it assigns the sameharmonizer_idto the value, otherwise it creates a new IDidentify the longest cleaned name for each group by ID and assign the
original name to that groupUse:Create astop_words.csvfile:harmonizer stop_words <csv_name> <csv_column>Harmonize the desired field:harmonizer harmonize <csv_name> <csv_column> -t 0.85the-t 0.85is optional and specifies a threshold between 0 and 1, with
values closer to 1 requiring a stricter match in order to assign the same
IDHelp:General help:harmonizer -hStop words help:harmonizer stop_words -hHarmonize help:harmonizer harmonize -hExamples:cd <package-dir>/harmonizer/examples # change into examples directoryH1B data:harmonizer stop_words h1b_datahubexport-2019.csv Employeroutputs:stop_words.csvharmonizer harmonize h1b_datahubexport-2019.csv Employertime: requires about ~18s on a normal laptopuses:stop_words.csvoutputs:h1b_datahubexport-2019_harmonized.csvWARN data:harmonizer stop_words Alaska_warn_raw.csv 'Company Name'outputs:stop_words.csvharmonizer harmonize Alaska_warn_raw.csv 'Company Name'time: requires about ~1s on a normal laptopuses:stop_words.csvoutputs:Alaska_warn_raw_harmonized.csvTuning:This script outputs the original file with the following columns added:<column>_harmonizer_cleaned: contains the cleaned version of the
target column<column>_harmonizer_score: contains the similarity score that
compares the current row to the previous row<column>_harmonizer_id: contains the assigned harmonizer ID<column>_harmonizer_standardized: contains the standardized valueLook at the<column>_harmonizer_score, which represents the similarity
between the current and previous rows' values; you can raise or lower the
threshold with the-t <value>argument, i.e. raise it if you think two
things shouldn't be a match and lower it if you think two things should be a
matchCaveats:This measure is not perfect; for instance, these companies probably will not
be identified as the same (although this doesn't appear to happen often in
H1B data):ACME GROUP / SPECIAL DIVISION XACME GROUP / REAL ESTATEACME GROUP / AGRICULTUREAlternatives:Attempted to use thisapproach,
but found that using a similarity matrix and affinity propagation doesn't
work, except for very small datasets (i.e. H1B data with ~51,000 rows crashes
a pretty decent computer); their algorithm runs in space and time of O(N^2),
while the one implemented here runs in O(N)LabelerThis tool helps label free form text. It takes a csv with raw text and a list
of labels and uses user input to train a model periodically, which then
predicts labels for unlabeled texts.Helplabeler -hUselabeler start <csv> <text-column-name> <label-1> <label-2>...[OPTIONS]labeler resume <checkpoint-pkl-path>Examplescd <package-dir>/labeler/examples # change into examples directorylabeler start examples/contraband.csv contraband alcohol drugs other weaponsruns labeler oncontraband.csvwith labelsalcohol,drugs,other, andweaponslabeler start examples/contraband.csv contraband alcohol drugs other weapons -xorruns labeler oncontraband.csvwith labelsalcohol,drugs,other, andweapons; however, this time, each record can only have 1 label, i.e. the
labels are mutually exclusivefollow the menu to label text or use the model to automatically label the
remaining texts, and then to save the labeled texts as a csvthis program also permits saving checkpoints, so labeling can be resumed at a
later point; this option can be accessed from the main menu