Repository: ipums/hlink (language: python)

Issue:
Add a function to show the model parameters and labels after training
We'd like to add this function for debugging and getting more information about the results of training. This Stack Overflow post may be helpful: https://stackoverflow.com/questions/42935914/how-to-map-features-from-the-output-of-a-vectorassembler-back-to-the-column-name.
Add testing
This should lock in what currently is getting returned, with a little room for variance. Asserts everything exists and is reasonable. Can certainly be edited, this isn't exactly exhaustive testing.


Discussion / hints:

Fix patch:
diff --git a/docs/_sources/config.md.txt b/docs/_sources/config.md.txt
index bf41d6b..8747c52 100644
--- a/docs/_sources/config.md.txt
+++ b/docs/_sources/config.md.txt
@@ -688,7 +688,9 @@ splits = [-1,0,6,11,9999]
   * `use_training_data_features` -- Type: `boolean`. Optional. If the identifiers in the training data set are not present in your raw input data, you will need to set this to `true`, or training features will not be able to be generated, giving null column errors.  For example, if the training data set you are using has individuals from 1900 and 1910, but you are about to train a model to score the 1930-1940 potential matches, you need this to be set to `true` or it will fail, since the individual IDs are not present in the 1930 and 1940 raw input data.  If you were about to train a model to score the 1900-1910 potential matches with this same training set, it would be best to set this to `false`, so you can be sure the training features are created from scratch to match your exact current configuration settings, although if you know the features haven't changed, you could set it to `true` to save a small amount of processing time.
   * `output_suspicious_TD` -- Type: `boolean`.  Optional.  Used in the `model_exploration` link task.  Outputs tables of potential matches that the model repeatedly scores differently than the match value given by the training data.  Helps to identify false positives/false negatives in the training data, as well as areas that need additional training feature coverage in the model, or need increased representation in the training data set.
   * `split_by_id_a` -- Type: `boolean`.  Optional.  Used in the `model_exploration` link task.  When set to true, ensures that all potential matches for a given individual with ID_a are grouped together in the same train-test-split group. For example, if individual histid_a "A304BT" has three potential matches in the training data, one each to histid_b "B200", "C201", and "D425", all of those potential matches would either end up in the "train" split or the "test" split when evaluating the model performance.
-  * `feature_importances` -- Type: `boolean`. Optional, and currently not functional.  Whether to record feature importances for the training features when training or evaluating an ML model.
+  * `feature_importances` -- Type: `boolean`. Optional.  Whether to record
+    feature importances or coefficients for the training features when training
+    the ML model. Set this to true to enable training step 3.
 
 
 ```
diff --git a/docs/_sources/link_tasks.md.txt b/docs/_sources/link_tasks.md.txt
index f31a9e4..34e27ba 100644
--- a/docs/_sources/link_tasks.md.txt
+++ b/docs/_sources/link_tasks.md.txt
@@ -30,15 +30,21 @@ as they are read in.
 Train a machine learning model to use for classification of potential links. This
 requires training data, which is read in in the first step. Comparison features
 are generated for the training data, and then the model is trained on the data
-and saved for use in the Matching task.
+and saved for use in the Matching task. The last step optionally saves some metadata
+like feature importances or coefficients for the model to help with introspection.
 
 ### Task steps
 
-The steps in each of these tasks are the same:
+The first three steps in each of these tasks are the same:
 * Step 0: Ingest the training data from a CSV file.
 * Step 1: Create comparison features.
 * Step 2: Train and save the model.
 
+The last step is available only for Training, not for Household Training.
+* Step 3: Save the coefficients or feature importances of the model for inspection.
+  This step is skipped by default. To enable it, set the `training.feature_importances`
+  config attribute to true in your config file.
+
 ### Related Configuration Sections
 
 * The [`training`](config.html#training-and-models) section is the most important
diff --git a/docs/config.html b/docs/config.html
index 11c14b8..9d31aa7 100644
--- a/docs/config.html
+++ b/docs/config.html
@@ -760,7 +760,9 @@ <h2>Training and <a class="reference internal" href="models.html"><span class="d
 <li><p><code class="docutils literal notranslate"><span class="pre">use_training_data_features</span></code> – Type: <code class="docutils literal notranslate"><span class="pre">boolean</span></code>. Optional. If the identifiers in the training data set are not present in your raw input data, you will need to set this to <code class="docutils literal notranslate"><span class="pre">true</span></code>, or training features will not be able to be generated, giving null column errors.  For example, if the training data set you are using has individuals from 1900 and 1910, but you are about to train a model to score the 1930-1940 potential matches, you need this to be set to <code class="docutils literal notranslate"><span class="pre">true</span></code> or it will fail, since the individual IDs are not present in the 1930 and 1940 raw input data.  If you were about to train a model to score the 1900-1910 potential matches with this same training set, it would be best to set this to <code class="docutils literal notranslate"><span class="pre">false</span></code>, so you can be sure the training features are created from scratch to match your exact current configuration settings, although if you know the features haven’t changed, you could set it to <code class="docutils literal notranslate"><span class="pre">true</span></code> to save a small amount of processing time.</p></li>
 <li><p><code class="docutils literal notranslate"><span class="pre">output_suspicious_TD</span></code> – Type: <code class="docutils literal notranslate"><span class="pre">boolean</span></code>.  Optional.  Used in the <code class="docutils literal notranslate"><span class="pre">model_exploration</span></code> link task.  Outputs tables of potential matches that the model repeatedly scores differently than the match value given by the training data.  Helps to identify false positives/false negatives in the training data, as well as areas that need additional training feature coverage in the model, or need increased representation in the training data set.</p></li>
 <li><p><code class="docutils literal notranslate"><span class="pre">split_by_id_a</span></code> – Type: <code class="docutils literal notranslate"><span class="pre">boolean</span></code>.  Optional.  Used in the <code class="docutils literal notranslate"><span class="pre">model_exploration</span></code> link task.  When set to true, ensures that all potential matches for a given individual with ID_a are grouped together in the same train-test-split group. For example, if individual histid_a “A304BT” has three potential matches in the training data, one each to histid_b “B200”, “C201”, and “D425”, all of those potential matches would either end up in the “train” split or the “test” split when evaluating the model performance.</p></li>
-<li><p><code class="docutils literal notranslate"><span class="pre">feature_importances</span></code> – Type: <code class="docutils literal notranslate"><span class="pre">boolean</span></code>. Optional, and currently not functional.  Whether to record feature importances for the training features when training or evaluating an ML model.</p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">feature_importances</span></code> – Type: <code class="docutils literal notranslate"><span class="pre">boolean</span></code>. Optional.  Whether to record
+feature importances or coefficients for the training features when training
+the ML model. Set this to true to enable training step 3.</p></li>
 </ul>
 </li>
 </ul>
diff --git a/docs/link_tasks.html b/docs/link_tasks.html
index be03e42..6765262 100644
--- a/docs/link_tasks.html
+++ b/docs/link_tasks.html
@@ -71,16 +71,23 @@ <h3>Overview<a class="headerlink" href="#id1" title="Permalink to this heading">
 <p>Train a machine learning model to use for classification of potential links. This
 requires training data, which is read in in the first step. Comparison features
 are generated for the training data, and then the model is trained on the data
-and saved for use in the Matching task.</p>
+and saved for use in the Matching task. The last step optionally saves some metadata
+like feature importances or coefficients for the model to help with introspection.</p>
 </section>
 <section id="id2">
 <h3>Task steps<a class="headerlink" href="#id2" title="Permalink to this heading">¶</a></h3>
-<p>The steps in each of these tasks are the same:</p>
+<p>The first three steps in each of these tasks are the same:</p>
 <ul class="simple">
 <li><p>Step 0: Ingest the training data from a CSV file.</p></li>
 <li><p>Step 1: Create comparison features.</p></li>
 <li><p>Step 2: Train and save the model.</p></li>
 </ul>
+<p>The last step is available only for Training, not for Household Training.</p>
+<ul class="simple">
+<li><p>Step 3: Save the coefficients or feature importances of the model for inspection.
+This step is skipped by default. To enable it, set the <code class="docutils literal notranslate"><span class="pre">training.feature_importances</span></code>
+config attribute to true in your config file.</p></li>
+</ul>
 </section>
 <section id="id3">
 <h3>Related Configuration Sections<a class="headerlink" href="#id3" title="Permalink to this heading">¶</a></h3>
diff --git a/docs/searchindex.js b/docs/searchindex.js
index 9ee66a1..8ccae2c 100644
--- a/docs/searchindex.js
+++ b/docs/searchindex.js
@@ -1,1 +1,1 @@
-Search.setIndex({"docnames": ["column_mapping_transforms", "comparison_types", "config", "feature_selection_transforms", "index", "installation", "introduction", "link_tasks", "models", "pipeline_features", "running_the_program", "substitutions", "use_examples"], "filenames": ["column_mapping_transforms.md", "comparison_types.md", "config.md", "feature_selection_transforms.md", "index.rst", "installation.md", "introduction.md", "link_tasks.md", "models.md", "pipeline_features.md", "running_the_program.md", "substitutions.md", "use_examples.md"], "titles": ["Column mapping transforms", "Comparison types, transform add-ons, aggregate features, and household aggregate features", "Configuration", "Feature Selection transforms", "Welcome to hlink\u2019s documentation!", "Installation", "Introduction", "Link Tasks", "Models", "Pipeline generated features", "Running hlink", "Substitutions", "Advanced Workflow Examples"], "terms": {"each": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10], "header": [0, 1, 2, 3, 9, 11], "below": [0, 1, 2, 3, 8, 9, 10], "repres": [0, 1, 2, 3, 9, 10], "type": [0, 2, 3, 4, 7, 8, 10, 11, 12], "ar": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 12], "us": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 12], "context": [0, 1, 3, 9], "column_map": [0, 2, 7], "list": [0, 1, 2, 3, 4, 8, 10, 11], "oper": [0, 2], "singl": [0, 2, 10, 12], "input": [0, 1, 2, 3, 6, 7, 10, 11], "output": [0, 1, 2, 3, 6, 7, 10, 12], "some": [0, 1, 3, 6, 7, 10], "suffix": [0, 4], "b": [0, 1, 2, 4, 10], "These": [0, 1, 2, 3, 6, 7, 8, 9], "mean": [0, 2], "appli": [0, 2, 3, 7, 12], "from": [0, 1, 2, 4, 6, 7, 8, 9, 10, 12], "onli": [0, 1, 2, 7, 12], "one": [0, 1, 2, 4, 7], "two": [0, 1, 2, 3, 6, 7, 9, 10, 12], "dataset": [0, 1, 2, 6, 7, 10, 12], "link": [0, 1, 2, 4, 6, 8], "A": [0, 1, 2, 9, 10], "most": [0, 1, 7, 10], "both": [0, 1, 2, 7, 12], "independ": [0, 2], "more": [0, 1, 2, 9, 10, 12], "than": [0, 1, 2, 8], "can": [0, 1, 2, 5, 6, 7, 8, 10, 12], "order": [0, 2, 7], "thei": [0, 1, 2, 7, 10], "re": 0, "configur": [0, 1, 6, 10, 12], "file": [0, 1, 4, 6, 7, 10, 11, 12], "so": [0, 1, 2, 5, 12], "mai": [0, 2, 6, 7, 10], "anoth": [0, 1, 3, 7], "format": 0, "The": [0, 1, 2, 3, 5, 7, 8, 9, 10, 12], "letter": [0, 4], "t": [0, 1, 2, 3, 12], "u": 0, "arbitrari": 0, "specifi": [0, 1, 2, 6, 7, 9, 10, 11], "column_nam": [0, 1, 2, 11], "attribut": [0, 1, 2, 3, 7, 8, 9, 10, 11], "under": [0, 1, 2], "its": [0, 1, 6, 10], "section": [0, 1, 2, 12], "name": [0, 1, 2, 4, 10, 11], "i": [0, 1, 2, 3, 5, 6, 7, 8, 10, 11, 12], "alia": [0, 2, 7], "along": [0, 1], "which": [0, 1, 2, 3, 6, 7, 9, 10, 12], "must": [0, 1, 2, 8, 9, 11], "addit": [0, 1, 2, 3, 5, 6, 10], "vari": [0, 2], "inform": [0, 1, 2, 10], "given": [0, 1, 2, 3, 8, 12], "often": [0, 10], "just": [0, 1, 2, 10, 12], "valu": [0, 1, 2, 4, 8, 9, 10, 11], "expect": [0, 1], "an": [0, 1, 2, 3, 6, 8, 10], "exampl": [0, 1, 2], "namefrst_split": [0, 2], "namefrst_clean": [0, 2], "add": [0, 4], "_": [0, 1, 2, 3, 4, 8, 9, 10], "numer": [0, 1], "11": [0, 2, 5, 9], "concat": [0, 4], "concaten": [0, 1], "string": [0, 1, 2, 3, 7, 8, 10, 11], "end": [0, 1, 2, 3, 11], "lowercas": [0, 4], "strip": [0, 4, 7], "clean": [0, 6], "convert": [0, 1, 2], "alphabet": 0, "charact": 0, "lower": [0, 1], "case": [0, 1, 2, 3, 6], "white": 0, "space": [0, 2, 3, 11], "start": [0, 11], "ration": [0, 4], "word": [0, 4], "replac": [0, 1, 4], "sinc": [0, 2], "peopl": [0, 1, 6, 10], "": [0, 1, 2, 6, 7, 10, 11], "raw": [0, 2, 7, 10], "censu": [0, 7, 12], "data": [0, 1, 4, 6, 7, 10], "contain": [0, 1, 11], "lead": 0, "better": [0, 6], "match": [0, 1, 4, 6, 10, 11, 12], "remov": [0, 4], "qmark": [0, 4], "hyphen": [0, 4], "punctuat": [0, 4], "thi": [0, 1, 2, 5, 6, 7, 9, 10, 12], "apostroph": [0, 4], "altern": [0, 2, 4], "If": [0, 1, 2, 3, 7, 8, 10, 11], "surround": 0, "all": [0, 1, 2, 3, 7, 8, 9, 10], "follow": [0, 1, 6, 10, 11, 12], "them": [0, 1, 2, 7], "jr": [0, 2], "sr": [0, 2], "ii": [0, 2], "iii": [0, 2], "stop": [0, 4], "last": [0, 1, 9], "street": [0, 1], "avenu": [0, 11], "blvd": 0, "circl": 0, "court": 0, "road": 0, "prefix": [0, 4], "like": [0, 2, 10], "m": [0, 1], "mr": 0, "In": [0, 1, 6, 10, 12], "ah": 0, "chines": 0, "condens": [0, 4], "whitespac": [0, 4, 7], "take": [0, 1, 2, 3, 7, 10], "non": 0, "leav": 0, "behind": 0, "arrai": [0, 2, 4, 8, 9], "index": [0, 4, 5], "select": [0, 1, 4, 6, 10, 12], "element": 0, "posit": [0, 1, 2, 4, 6], "first": [0, 1, 2, 5, 7, 10, 11], "second": [0, 1, 2, 11], "1": [0, 1, 2, 4, 7, 8, 9, 10, 12], "item": 0, "namefrst": [0, 1, 2], "set": [0, 1, 2, 3, 6, 7, 10, 12], "Then": [0, 5], "0": [0, 1, 2, 7, 8, 9, 10, 12], "give": [0, 2], "initi": [0, 1, 10], "person": [0, 1, 6], "probabl": [0, 2, 8], "middl": [0, 1], "namefrst_mid_init": [0, 1], "multipl": [0, 1, 2, 10], "otherwis": [0, 1, 9, 12], "known": 0, "recod": 0, "birthyr": [0, 2], "clean_birthyr": [0, 2, 3], "9999": [0, 2, 9], "1999": [0, 2], "9998": 0, "divid": [0, 4], "int": [0, 1, 2, 3, 4, 8], "integ": [0, 1, 2, 9], "It": [0, 1, 2, 6, 10, 12], "result": [0, 1, 6, 9, 10, 12], "For": [0, 1, 2, 7, 10, 12], "instanc": [0, 8], "birthplac": 0, "variabl": [0, 1, 2, 12], "detail": [0, 2, 10], "version": [0, 5, 12], "gener": [0, 1, 4, 6, 7, 10], "least": [0, 1], "signific": 0, "digit": 0, "make": [0, 1, 2, 5, 12], "we": [0, 1, 10, 12], "simpli": [0, 2], "drop": [0, 2, 10], "100": [0, 2, 12], "round": [0, 2], "lowest": 0, "whole": [0, 6], "number": [0, 1, 2, 7, 8, 10], "floor": [0, 4], "function": [0, 1, 2, 6, 10], "bpl": [0, 1, 2], "bpl_root": 0, "when": [0, 1, 2, 3, 4, 7, 12], "condit": [0, 1, 2, 3, 4, 7], "logic": 0, "work": [0, 1, 2, 5, 7, 10, 12], "sql": [0, 1, 2, 3, 4, 7, 10], "express": [0, 1, 2], "claus": [0, 1], "if_valu": 0, "else_valu": 0, "race": [0, 1, 2, 9, 12], "ipum": [0, 6], "code": [0, 1, 2, 5], "categori": [0, 8], "get": [0, 1, 4, 10], "down": [0, 6, 12], "nearest": 0, "produc": [0, 10], "relat": [0, 1, 2], "between": [0, 1, 2, 6, 7, 10, 12], "hundr": 0, "300": 0, "child": [0, 8], "household": [0, 4, 6, 8, 10, 12], "head": 0, "301": 0, "302": 0, "adopt": 0, "303": 0, "step": [0, 1, 2, 6], "usual": [0, 7, 12], "need": [0, 1, 2, 7, 10, 12], "2": [0, 1, 2, 3, 7, 8, 11, 12], "spous": 0, "3": [0, 1, 2, 5, 8, 9, 12], "4": [0, 1, 8], "law": 0, "5": [0, 1, 2, 8, 9, 10, 12], "parent": [0, 1, 11], "6": [0, 2, 8, 9, 12], "7": [0, 1, 2, 8, 12], "sibl": 0, "12": 0, "relate_div_100": [0, 1, 2], "page": [1, 10], "ha": [1, 2, 6, 10, 12], "differ": [1, 2, 4, 6, 7], "avail": [1, 2, 3, 5, 7, 8, 9, 12], "comparison_featur": [1, 2, 7], "relatematch": [1, 2], "comparison_typ": [1, 2], "categor": [1, 2, 8, 9], "true": [1, 2, 3, 9, 11, 12], "maximum": [1, 8], "jaro": [1, 9], "winkler": [1, 9], "find": [1, 7, 12], "greatest": 1, "among": 1, "cartesian": 1, "product": [1, 6, 12], "column": [1, 3, 4, 7, 9, 10, 11, 12], "namelast": [1, 2], "would": [1, 2, 12], "return": [1, 3, 8, 10], "four": 1, "namefrst_a": 1, "namefrst_b": 1, "namelast_b": 1, "namelast_a": 1, "requir": [1, 2, 3, 4, 7, 9, 10, 11], "maximum_jw": 1, "score": [1, 2, 7, 9], "compar": [1, 2, 6, 7], "namefrst_jw": [1, 2, 12], "geograph": 1, "filter": [1, 4, 7, 11], "major": [1, 10], "locat": [1, 2, 10], "chang": [1, 2, 5, 10, 12], "befor": [1, 2, 3, 5, 7], "boundari": 1, "zero": 1, "jw_street": 1, "enum_dist": 1, "max": [1, 8, 10], "member": [1, 7], "neighborhood": 1, "surnam": 1, "related_individual_max_jw": 1, "namefrst_rel": 1, "assert": [1, 10], "same": [1, 2, 6, 7, 10], "NOT": 1, "distinct": 1, "f1": 1, "evalu": [1, 2, 6, 7, 8], "either": [1, 2, 6, 11], "ani": [1, 2, 5, 8], "potenti": [1, 4, 7], "mismatch": 1, "queri": [1, 2], "fi": 1, "OR": 1, "mi0": 1, "mi1": 1, "THEN": 1, "els": [1, 2, 3], "first_init_col": 1, "namefrst_init": 1, "mid_init_col": 1, "namefrst_mid_init_2": 1, "f2": 1, "empti": 1, "null": [1, 2, 3], "AND": [1, 2], "individu": [1, 2, 7, 12], "mainli": 1, "caution": [1, 9], "flag": [1, 9, 10, 12], "f": [1, 10], "sp": 1, "m_caution": [1, 2, 9, 12], "mbpl": 1, "mother_birthyr": 1, "stepmom": 1, "momloc": 1, "comp_a": [1, 2], "comp_b": [1, 2], "comp_c": 1, "parent_step_chang": 1, "comp_d": 1, "check": [1, 10], "sign": 1, "boolean": [1, 2, 3, 11, 12], "form": [1, 7, 11], "cast": 1, "col": 1, "namelast_equal_as_int": 1, "namelast_clean": [1, 2, 3], "whether": [1, 2, 11], "join": [1, 11], "across": 1, "record": [1, 2, 6, 7], "being": [1, 7], "exact": [1, 2], "namefrst_unstd": [1, 2], "present": [1, 2, 9], "nonzero": 1, "primarili": [1, 7], "indic": [1, 12], "kind": 1, "incompar": 1, "akin": 1, "miss": [1, 10], "see": [1, 2, 5, 10, 12], "also": [1, 2, 5, 6, 7, 9, 10, 12], "univers": [1, 4, 7], "similar": 1, "fbpl_nomatch": 1, "fbpl": 1, "allow": [1, 2, 7, 12], "up": [1, 2, 10, 11], "sub": 1, "object": [1, 2, 6, 10], "document": [1, 8, 10, 12], "option": [1, 2, 3, 6, 8, 10, 12], "sp_caution": [1, 2, 12], "spouse_bpl": 1, "spouse_birthyr": 1, "durmarr": [1, 2], "new_marr": [1, 2], "appear": 1, "street_jw": [1, 2, 12], "counti": 1, "statefip": [1, 2], "9": 1, "multipli": 1, "togeth": [1, 2], "after": [1, 2, 4, 8, 10], "float": [1, 2, 8], "have": [1, 2, 5, 6, 7, 8, 10, 12], "comp": 1, "c": 1, "sploc": 1, "012": 1, "fals": [1, 2, 3, 4, 6, 10], "d": 1, "specif": [1, 2, 10], "circumst": 1, "should": [1, 2, 8, 9, 10], "mid_init_match": 1, "either_1": 1, "nativ": 1, "either_0": 1, "gen": 1, "imm": [1, 2, 12], "immigr": 1, "look": [1, 10, 11], "foreign": 1, "born": 1, "sgen": [1, 2, 12], "rel": [1, 2, 12], "scala": 1, "determin": [1, 7], "greater": [1, 5], "jw_threshold": 1, "ag": [1, 2, 3], "less": [1, 2], "age_threshold": 1, "sex": [1, 2, 11], "sampl": 1, "related_individual_row": 1, "unrel": 1, "depend": [1, 2, 5, 12], "name_col": 1, "birthyr_col": 1, "namefrst_related_row": 1, "replaced_birthyr": [1, 2, 3], "extra": 1, "children": 1, "who": 1, "base": [1, 2, 7], "count": [1, 10, 12], "suspect": [1, 6], "relate_col": 1, "histid_col": 1, "id": [1, 2], "birth": 1, "year": [1, 2, 3, 4], "year_b": 1, "wa": [1, 12], "taken": 1, "minimum": [1, 8], "accept": [1, 2, 12], "consid": [1, 8], "histid": [1, 2, 12], "1910": [1, 2, 12], "8": [1, 2, 5, 10], "rate": 1, "calcul": [1, 12], "percentag": 1, "seen": 1, "neighbor": 1, "meet": 1, "95": 1, "nbor": [1, 2, 12], "namelast_neighbor": 1, "05": [1, 2], "ad": [1, 2], "namelast_popularity_sum": 1, "namelast_popular": 1, "length": [1, 2, 9], "size": 1, "ab": 1, "diff": 1, "absolut": 1, "you": [1, 2, 5, 10, 11, 12], "invalid": [1, 8], "instead": [1, 2, 5, 7], "marriag": 1, "durat": 1, "99": [1, 2], "placehold": 1, "unknown": 1, "exclud": 1, "those": [1, 2], "consider": 1, "byrdiff": [1, 2, 12], "mardurmatch": [1, 2], "14": 1, "minu": [1, 2], "subtract": 1, "geo": 1, "distanc": [1, 8], "lookup": 1, "tabl": [1, 2, 4, 7, 10, 12], "hlink": [1, 2, 5, 6, 7, 12], "core": [1, 7, 10, 12], "dist_tabl": 1, "py": [1, 2], "There": [1, 3, 7], "sever": [1, 6], "wai": [1, 5, 10], "kei": [1, 7, 10], "key_count": 1, "secondari": 1, "serv": 1, "back": 1, "primari": [1, 6], "doe": [1, 7, 12], "particularli": 1, "state": [1, 6], "much": [1, 7], "fewer": [1, 8], "combin": [1, 2, 3, 7], "thu": 1, "risk": 1, "fill": 1, "aren": 1, "ex": 1, "even": 1, "though": 1, "distances_fil": 1, "path": [1, 2, 10, 11, 12], "table_nam": 1, "what": [1, 2, 10, 12], "want": [1, 2, 10, 12], "do": [1, 3, 10, 12], "read": [1, 2, 7, 10], "onc": [1, 10], "loc_a": 1, "where": [1, 7, 10, 12], "come": 1, "loc_b": 1, "distance_col": 1, "source_column_a": 1, "sourc": [1, 4, 7, 10, 12], "source_column_b": 1, "loc_a_0": 1, "loc_a_1": 1, "loc_b_0": 1, "loc_b_1": 1, "secondary_key_count": 1, "backup": 1, "secondary_table_nam": 1, "secondary_distances_fil": 1, "secondary_source_column": 1, "secondary_loc_a": 1, "secondary_loc_b": 1, "secondary_distance_col": 1, "state_dist": 1, "state_distance_lookup": 1, "county_state_dist": 1, "csv": [1, 2, 7, 10, 11, 12], "statecode1": 1, "statecode2": 1, "dist": 1, "county_dist": [1, 2, 12], "county_distance_lookup": 1, "county_1900_1910_distances_km": 1, "from_icpsrctyi": 1, "to_icpsrctyi": 1, "from_statefip": 1, "to_statefip": 1, "distance_km": 1, "state_1900_1910_distances_km": 1, "fetch": 1, "neither": 1, "nor": 1, "mpre": 1, "m_namefrst": 1, "accord": 1, "niu": 1, "other": [1, 2, 12], "mfbplmatch": 1, "multi": 1, "search": 1, "special": 1, "simplifi": 1, "particular": [1, 2], "constraint": 1, "num_col": 1, "whose": 1, "templat": 1, "n": [1, 8, 9], "per": [1, 2, 8, 9, 10], "current": [1, 2, 10], "respect": [1, 7], "jw_col_templ": 1, "jw": 1, "pair": [1, 12], "equal_and_not_null_templ": 1, "final": [1, 2, 12], "comput": [1, 3, 7], "_namefrst": 1, "_bpl": 1, "_sex": 1, "25": 1, "nvl": 1, "sm_namefrst": 1, "sn_namefrst": 1, "sm_bpl": 1, "sn_bpl": 1, "sm_sex": 1, "sn_sex": 1, "pass": [1, 2, 7, 8], "default": [1, 2, 8, 10], "flexibl": 1, "user": [1, 10], "write": [1, 10, 12], "own": [1, 2], "favor": 1, "reason": 1, "good": 1, "fallback": 1, "defin": [1, 7, 8, 9, 10], "spark": [1, 2, 5, 8, 9, 10, 12], "builtin": 1, "argument": [1, 10, 12], "namelast_jw_max": 1, "namelast1": 1, "namelast2": 1, "namelast3": 1, "abov": [1, 5], "extend": 1, "beyond": 1, "standard": [1, 11], "top": [1, 4], "level": [1, 4, 10], "everi": 1, "jw_f": [1, 2, 12], "father_namefrst": 1, "rais": [1, 3], "exponenti": 1, "squar": 1, "county_distance_squar": [1, 2, 12], "county_a": 1, "county_b": 1, "upper": 1, "gt": 1, "btwn": 1, "addl": 1, "var": [1, 2], "program": [1, 2, 7, 12], "report": [1, 4, 6, 10], "addl_var": 1, "check_val_expr": 1, "else_v": 1, "volumn": 1, "datasourc": [1, 2, 10], "yrimmig": 1, "immyear_diff": [1, 2, 9, 12], "To": [1, 5, 10], "includ": [1, 2, 7, 9, 10], "train": [1, 4, 6, 8, 10], "independent_var": [1, 2, 12], "config": [1, 4, 7, 10, 12], "id_column": [1, 2], "_a": 1, "mult": 1, "exist": [1, 2, 10], "within": [1, 2, 6, 10, 11], "hh_train": [1, 2, 7, 10, 12], "hh": 1, "highest": [1, 2], "against": [1, 11], "ten": [1, 2], "tell": 2, "how": [2, 7], "descript": [2, 8, 10], "refer": 2, "here": [2, 7, 10, 12], "tutori": [2, 10], "script": [2, 6, 10], "discuss": 2, "readm": 2, "note": 2, "written": [2, 6], "toml": [2, 6, 10], "abl": 2, "json": [2, 10], "datasource_a": [2, 7], "datasource_b": [2, 7], "transform": [2, 4, 6, 7], "lowercase_strip": 2, "add_to_a": 2, "10": [2, 5, 12], "age_2": 2, "derived_from": 2, "expand_length": 2, "explod": [2, 7], "jaro_winkl": 2, "namelast_jw": [2, 12], "threshold": [2, 8, 12], "feature_nam": 2, "79": 2, "84": 2, "complex": 2, "machin": [2, 6, 7, 10, 12], "learn": [2, 6, 7, 10, 12], "probabilist": [2, 6], "drop_data_from_scored_match": 2, "us1900": 2, "us1900m_usa": 2, "p": 2, "parquet": [2, 7], "us1910": 2, "us1910m_usa": 2, "training_data_subset": 2, "serialp": 2, "rationalize_name_word": 2, "remove_qmark_hyphen": 2, "replace_apostroph": 2, "remove_suffix": 2, "remove_alternate_nam": 2, "condense_strip_whitespac": 2, "split": [2, 3, 4, 7, 8, 9, 12], "namefrst_std": [2, 11], "array_index": 2, "bpl_orig": 2, "divide_by_int": 2, "get_floor": 2, "statefip_h": 2, "output_typ": 2, "substitution_column": [2, 7, 11], "join_column": [2, 11], "join_valu": [2, 11], "substitution_fil": [2, 11], "name_std": [2, 11], "male": [2, 11], "femal": [2, 11], "feature_select": [2, 3, 7], "input_column": [2, 3, 9], "output_column": [2, 3, 9], "sql_condit": 2, "namelast_bigram": 2, "bigram": [2, 4], "bpl_clean": 2, "bpl_str": 2, "washington": 2, "bpl2_str": 2, "53": 2, "region": [2, 12], "attach_vari": 2, "region_dict": 2, "col_to_join_on": 2, "col_to_add": 2, "null_fil": 2, "col_typ": 2, "potential_matches_univers": [2, 7], "birthyr_3": 2, "namefrst_std_jw": [2, 12], "75": [2, 8, 12], "comparis": 2, "post": [2, 7], "hh_comparison": [2, 7], "threshold_expr": 2, "fetch_a": 2, "sex_equ": 2, "equal": [2, 11], "relate_a": [2, 9], "pipeline_featur": [2, 7, 9], "sex_region_interact": 2, "transformer_typ": [2, 9], "interact": [2, 4, 7, 12], "relatetyp": [2, 9], "bucket": [2, 7], "hit": [2, 10, 12], "scale_data": [2, 12], "training_data": [2, 10], "dependent_var": [2, 12], "score_with_model": [2, 12], "use_training_data_featur": [2, 7, 12], "split_by_id_a": [2, 12], "decis": [2, 4, 8, 12], "drop_duplicate_with_threshold_ratio": [2, 12], "n_training_iter": [2, 7, 12], "output_suspicious_td": [2, 12], "param_grid": [2, 12], "model_paramet": [2, 7, 8, 12], "random_forest": [2, 12], "maxdepth": [2, 8, 12], "numtre": [2, 8, 12], "005": 2, "threshold_ratio": [2, 8, 12], "logistic_regress": [2, 12], "50": [2, 12], "65": 2, "80": 2, "chosen_model": [2, 8, 12], "prediction_col": 2, "predict": [2, 12], "hh_col": 2, "hh_training_data_1900_1910": 2, "probit": [2, 4], "go": [2, 10], "your": [2, 5, 10, 12], "uniqu": 2, "identifi": [2, 6, 12], "full": [2, 7, 12], "short": 2, "alphanumer": 2, "convert_ints_to_long": 2, "automat": [2, 5, 7], "long": [2, 11], "especi": 2, "assum": 2, "schema": 2, "sometim": 2, "term": 2, "bigint": 2, "thing": 2, "my_fil": 2, "subset": [2, 11], "limit": 2, "extract": 2, "new": [2, 12], "iv": 2, "v": 2, "vi": 2, "vii": 2, "viii": 2, "namelast_clean_bigram": [2, 3], "creat": [2, 6, 7, 9, 10, 11, 12], "fed": [2, 7], "prep": 2, "df": [2, 10], "men": 2, "describ": [2, 10], "newli": 2, "attempt": 2, "duplic": [2, 8], "row": 2, "conjuct": 2, "Will": 2, "conjunct": 2, "rang": [2, 9], "original_valu": 2, "plu": 2, "1870": 2, "expand": 2, "1867": 2, "1868": 2, "1869": 2, "1871": 2, "1872": 2, "1873": 2, "kept": 2, "keep": 2, "appropri": 2, "treat": [2, 9], "import": [2, 7, 10, 12], "dure": [2, 7], "hot": 2, "encod": [2, 3], "vector": [2, 9], "stage": 2, "well": 2, "upper_threshold": 2, "cannot": 2, "robust": 2, "ml": [2, 4, 8, 9], "typic": [2, 7], "leverag": 2, "api": [2, 6, 9], "piplin": 2, "regionf": 2, "sex_regionf_interact": 2, "immyear_caut": [2, 9], "myriad": 2, "explor": [2, 4, 6, 10], "part": [2, 7], "task": [2, 4, 6, 8, 12], "drop_duplicate_a": 2, "time": [2, 7, 10], "out": [2, 7, 12], "best": [2, 7], "smallest": 2, "possibl": 2, "ratio": [2, 8], "beta": [2, 8], "test": [2, 7, 12], "model_explor": [2, 10, 12], "hyper": [2, 6, 12], "paramet": [2, 6, 7, 8, 10, 12], "eval": 2, "skip": [2, 7], "apply_model": 2, "run_all_step": [2, 10, 12], "command": [2, 6, 10, 12], "try": 2, "creation": 2, "iter": 2, "scale": 2, "error": [2, 9], "1900": [2, 12], "about": [2, 10, 12], "1930": [2, 12], "1940": [2, 12], "fail": 2, "were": 2, "sure": [2, 5, 10], "scratch": 2, "although": 2, "know": 2, "haven": 2, "could": 2, "save": [2, 7, 12], "small": 2, "amount": 2, "process": [2, 6, 10], "repeatedli": 2, "help": [2, 10], "neg": [2, 4, 6], "area": 2, "coverag": 2, "increas": [2, 9], "represent": [2, 7], "ensur": 2, "group": [2, 7], "a304bt": 2, "three": 2, "b200": 2, "c201": 2, "d425": 2, "perform": [2, 6, 7, 11], "feature_import": [2, 12], "srace": [2, 9, 12], "race_interacted_srac": [2, 9, 12], "hits2": [2, 12], "exact_mult": [2, 12], "ncount": [2, 3, 12], "ncount2": [2, 3, 12], "f_interacted_jw_f": [2, 12], "f_caution": [2, 12], "f_pre": [2, 12], "fbplmatch": [2, 12], "m_interacted_jw_m": [2, 9, 12], "jw_m": [2, 9, 12], "m_pre": [2, 9, 12], "mbplmatch": [2, 12], "sp_interacted_jw_sp": [2, 12], "jw_sp": [2, 12], "sp_pre": [2, 12], "mi": [2, 12], "fsoundex": [2, 12], "lsoundex": [2, 12], "oth": [2, 12], "imm_interacted_immyear_caut": [2, 12], "1900_1910_training_data_20191023": 2, "jw_max_a": 2, "jw_max_b": 2, "f1_match": 2, "f2_match": 2, "byrdifcat": 2, "racematch": 2, "bplmatch": 2, "imm_interacted_bplmatch": 2, "sexmatch": 2, "relatetype_interacted_relatematch": 2, "checkpoint": 3, "override_column_a": 3, "override_column_b": 3, "set_value_column_a": 3, "set_value_column_b": 3, "no_first_pad": 3, "don": 3, "prepend": 3, "provid": [3, 6, 7, 9, 10], "namefrst_unstd_bigram": 3, "namelast_frst_bigram": 3, "namelast_clean_soundex": 3, "input_col": 3, "output_col": 3, "expon": 3, "introduct": 4, "overview": 4, "instal": 4, "pypi": 4, "preprocess": [4, 6, 10, 12], "model": [4, 6, 10], "run": [4, 5, 6, 7, 12], "librari": [4, 6], "mode": [4, 5, 12], "advanc": 4, "workflow": 4, "export": [4, 7, 10], "featur": [4, 6, 7, 8, 10], "reus": 4, "basic": 4, "map": [4, 7, 9], "substitut": [4, 7], "block": [4, 7], "comparison": [4, 7], "pipelin": 4, "substr": 4, "ons": 4, "aggreg": 4, "union": 4, "soundex": 4, "power": 4, "regex": 4, "random": [4, 8], "forest": [4, 8], "logist": [4, 8], "regress": [4, 8], "tree": [4, 8], "gradient": [4, 8], "boost": [4, 8], "system": 5, "python": [5, 6, 10], "java": 5, "integr": 5, "apach": 5, "via": [5, 6], "pyspark": [5, 8, 9, 10], "packag": 5, "org": 5, "latest": 5, "pip": 5, "easiest": [5, 10], "through": [5, 7, 9, 10], "instruct": [5, 10], "But": 5, "clone": 5, "github": 5, "repositori": 5, "root": 5, "project": 5, "directori": [5, 10, 12], "develop": [5, 6], "e": 5, "dev": 5, "edit": 5, "made": 5, "built": 5, "tool": [5, 6], "line": [6, 10], "share": 6, "characterist": [6, 7], "correspond": [6, 7], "real": 6, "world": 6, "determinist": [6, 7], "rule": [6, 7], "algorithm": [6, 7], "At": [6, 7], "been": 6, "unit": 6, "census": 6, "hierarch": [6, 10], "structur": 6, "nest": 6, "howev": [6, 12], "tailor": 6, "ignor": 6, "common": [6, 7, 12], "highli": [6, 7], "languag": 6, "further": [6, 12], "broken": 6, "smaller": 6, "sequenc": 6, "linkrun": [6, 10], "prepar": [6, 7, 10], "research": 6, "experi": 6, "understand": 6, "tune": [6, 12], "relationship": 6, "varieti": 7, "normal": 7, "abbrevi": [7, 11], "regist": [7, 10], "datafram": [7, 10, 12], "request": 7, "classif": [7, 8], "ingest": 7, "mani": [7, 10], "aspect": [7, 10], "extens": 7, "longest": 7, "definit": 7, "reduc": 7, "drastic": 7, "improv": 7, "runtim": 7, "separ": 7, "total": 7, "potential_match": [7, 10], "satisfi": 7, "elig": 7, "support": [7, 8], "reshap": 7, "thought": 7, "ahead": 7, "chosen": 7, "experiment": [7, 10], "focus": 7, "demograph": 7, "moment": 7, "veri": [7, 12], "anyon": 7, "percent": 7, "remain": 7, "popul": 7, "pull": 7, "fix": 7, "width": 7, "crosswalk": 7, "construct": 7, "alpha": 8, "hyperparamet": [8, 12], "de": 8, "param": [8, 12], "label": 8, "doc": [8, 9], "commonli": 8, "explan": 8, "randomforestclassifi": 8, "depth": 8, "20": 8, "featuresubsetstrategi": 8, "node": 8, "auto": 8, "onethird": 8, "sqrt": 8, "log2": 8, "15": 8, "generalizedlinearregress": 8, "famili": 8, "binomi": 8, "85": [8, 10], "logisticregress": 8, "decisiontreeclassifi": 8, "mininstancespernod": 8, "caus": 8, "left": 8, "right": [8, 10], "discard": 8, "maxbin": 8, "bin": 8, "discret": 8, "continu": [8, 9, 12], "gbtclassifi": 8, "mother": 9, "point": [9, 12], "With": 9, "x": [9, 10], "y": 9, "hold": 9, "except": 9, "strictli": 9, "inf": 9, "explicitli": 9, "cover": 9, "doubl": 9, "outsid": 9, "job": 10, "high": 10, "class": 10, "handl": 10, "main": 10, "complet": 10, "access": [10, 12], "link_run": 10, "factori": 10, "sparkfactori": 10, "load_config": 10, "load_conf_fil": 10, "sparksess": 10, "now": 10, "let": 10, "load": 10, "our": 10, "my_conf": 10, "lr": 10, "prep_step": 10, "get_step": 10, "enumer": 10, "print": 10, "input_table_nam": 10, "output_table_nam": 10, "run_step": 10, "get_tabl": 10, "matches_df": 10, "hh_model_explor": 10, "method": [10, 12], "interfac": 10, "easili": 10, "conveni": 10, "adjust": 10, "set_loc": 10, "set_num_cor": 10, "set_executor_memori": 10, "5g": 10, "ll": 10, "dictionari": 10, "modul": 10, "pleas": 10, "enabl": 10, "reproduc": 10, "consol": 10, "cpu": 10, "usag": 10, "h": 10, "executor_memori": [10, 12], "execute_task": 10, "execute_command": 10, "conf": [10, 12], "show": 10, "messag": 10, "exit": 10, "memori": 10, "executor": 10, "begin": 10, "execut": 10, "seri": 10, "excute_command": 10, "filepath": 10, "sai": 10, "fullcount_1870_1880": 10, "pattern": 10, "full_count_1870_1880": 10, "prompt": 10, "enter": 10, "text": 10, "unstabl": 10, "topic": 10, "analyz": [10, 12], "set_preexisting_t": 10, "x_persist": 10, "borrow_t": 10, "get_task": 10, "set_print_sql": 10, "x_sql": 10, "x_sqlf": 10, "ipython": 10, "showf": 10, "x_summari": 10, "desc": 10, "x_crosswalk": 10, "x_tab": 10, "q": [10, 12], "x_hh_tfam": 10, "x_tfam": 10, "drop_al": 10, "reload": 10, "x_hh_tfam_2a": 10, "x_tfam_raw": 10, "drop_all_prc": 10, "x_hh_tfam_2b": 10, "x_union": 10, "drop_all_temp": 10, "x_load": 10, "get_set": 10, "set_link_task": 10, "x_parquet_from_csv": 10, "organ": 10, "hierarchi": 10, "five": 10, "hh_match": 10, "someth": 10, "choic": 10, "preexist": 10, "prepped_df_a": 10, "prepped_df_b": 10, "raw_df_b": 10, "raw_df_a": 10, "training_featur": [10, 12], "scored_potential_match": 10, "potential_matches_prep": 10, "exploded_df_b": 10, "exploded_df_a": 10, "predicted_match": 10, "hh_training_featur": [10, 12], "hh_training_data": 10, "hh_predicted_match": 10, "hh_scored_potential_match": 10, "hh_potential_match": 10, "hh_blocked_match": 10, "hh_potential_matchs_prep": 10, "model_eval_training_vector": 10, "model_eval_training_data": 10, "model_eval_repeat_fp": 10, "model_eval_training_featur": 10, "model_eval_training_result": 10, "model_eval_repeat_fn": 10, "hh_model_eval_training_vector": 10, "hh_model_eval_repeat_fp": 10, "hh_model_eval_repeat_fn": 10, "hh_model_eval_training_result": 10, "hh_model_eval_training_featur": 10, "hh_model_eval_training_data": 10, "persist": 10, "hidden": 10, "intermedi": 10, "yet": 10, "databas": 10, "tablenam": 10, "istemporari": 10, "task_nam": 10, "num": 10, "finish": 10, "By": 10, "put": [10, 12], "launch": [10, 12], "my": [10, 12], "subhead": 11, "suppli": 11, "regex_word_replac": 11, "variant": 11, "av": 11, "7th": 11, "swap": 11, "still": 11, "anywher": 11, "proceed": 11, "street_unstd": 11, "dir": 11, "substitutions_street_abbrev": 11, "span": 12, "1920": 12, "deriv": 12, "necessari": 12, "scenario": 12, "copi": 12, "use_potential_matches_featur": 12, "full_count_1900_1910": 12, "50g": 12, "ask": 12, "arg": 12, "partit": 12, "training_data_1900_1910_hlink_featur": 12, "might": 12, "shut": 12, "framework": 12, "etc": 12, "relev": 12, "matrix": 12, "implement": 12, "regular": 12, "training_data_1900_1910": 12, "weren": 12, "ident": 12, "manual": 12, "updat": 12, "isn": 12, "analysi": 12, "training_result": 12, "hh_training_result": 12, "1900_1910_training_result": 12, "repeat_fp": 12, "repeat_fn": 12, "hh_repeat_fp": 12, "hh_repeat_fn": 12, "1900_1910_potential_fp": 12, "1900_1910_potential_fn": 12, "prefer": 12, "ve": 12}, "objects": {}, "objtypes": {}, "objnames": {}, "titleterms": {"column": [0, 2], "map": [0, 2], "transform": [0, 1, 3, 9], "add_to_a": 0, "concat_to_a": 0, "concat_to_b": 0, "lowercase_strip": 0, "rationalize_name_word": 0, "remove_qmark_hyphen": 0, "remove_punctu": 0, "replace_apostroph": 0, "remove_alternate_nam": 0, "remove_suffix": 0, "remove_stop_word": 0, "remove_prefix": 0, "condense_strip_whitespac": 0, "remove_one_letter_nam": 0, "split": 0, "array_index": 0, "substr": 0, "divide_by_int": 0, "when_valu": 0, "get_floor": 0, "comparison": [1, 2], "type": [1, 9], "add": 1, "ons": 1, "aggreg": 1, "featur": [1, 2, 3, 9, 12], "household": [1, 2, 7], "maximum_jaro_winkl": 1, "jaro_winkl": 1, "jaro_winkler_street": 1, "max_jaro_winkl": 1, "equal": 1, "f1_match": 1, "f2_match": 1, "not_equ": 1, "equals_as_int": 1, "all_equ": 1, "not_zero_and_not_equ": 1, "time": 1, "caution_comp_3": 1, "caution_comp_3_012": 1, "caution_comp_4": 1, "caution_comp_4_012": 1, "any_equ": 1, "either_are_1": 1, "either_are_0": 1, "second_gen_imm": 1, "rel_jaro_winkl": 1, "extra_children": 1, "jaro_winkler_r": 1, "sum": 1, "length_b": 1, "abs_diff": 1, "b_minus_a": 1, "geo_dist": 1, "fetch_a": 1, "fetch_b": 1, "present_both_year": 1, "neither_are_nul": 1, "present_and_matching_categor": 1, "present_and_not_equ": 1, "present_and_equal_categorical_in_univers": 1, "multi_jaro_winkler_search": 1, "sql_condit": [1, 3], "alia": 1, "power": [1, 3], "threshold": 1, "lower_threshold": 1, "upper_threshold": 1, "gt_threshold": 1, "btwn_threshold": 1, "look_at_addl_var": 1, "hit": 1, "hits2": 1, "exact_mult": 1, "jw_max_a": 1, "jw_max_b": 1, "configur": [2, 4, 7], "basic": 2, "config": 2, "file": 2, "advanc": [2, 12], "top": 2, "level": 2, "data": [2, 11, 12], "sourc": [2, 5], "filter": 2, "substitut": [2, 11], "select": [2, 3], "potenti": [2, 12], "match": [2, 7], "univers": 2, "block": 2, "pipelin": [2, 9], "gener": [2, 9, 12], "train": [2, 7, 12], "model": [2, 7, 8, 12], "bigram": 3, "arrai": 3, "union": 3, "soundex": 3, "welcom": 4, "hlink": [4, 10], "": 4, "document": 4, "api": 4, "instal": 5, "requir": 5, "from": 5, "pypi": 5, "introduct": 6, "overview": [6, 7], "link": [7, 10, 12], "task": [7, 10], "preprocess": 7, "step": [7, 10], "relat": 7, "section": 7, "explor": [7, 12], "report": 7, "random_forest": 8, "probit": 8, "logistic_regress": 8, "decision_tre": 8, "gradient_boosted_tre": 8, "interact": [9, 10], "bucket": 9, "run": 10, "us": 10, "librari": 10, "mode": 10, "start": 10, "program": 10, "exampl": [10, 12], "workflow": [10, 12], "1": 11, "tabl": 11, "regex": 11, "word": 11, "replac": 11, "export": 12, "after": 12, "reus": 12, "differ": 12, "year": 12, "ml": 12, "list": 12, "fals": 12, "posit": 12, "neg": 12, "fp": 12, "fn": 12}, "envversion": {"sphinx.domains.c": 2, "sphinx.domains.changeset": 1, "sphinx.domains.citation": 1, "sphinx.domains.cpp": 6, "sphinx.domains.index": 1, "sphinx.domains.javascript": 2, "sphinx.domains.math": 2, "sphinx.domains.python": 3, "sphinx.domains.rst": 2, "sphinx.domains.std": 2, "sphinx": 56}})
\ No newline at end of file
+Search.setIndex({"docnames": ["column_mapping_transforms", "comparison_types", "config", "feature_selection_transforms", "index", "installation", "introduction", "link_tasks", "models", "pipeline_features", "running_the_program", "substitutions", "use_examples"], "filenames": ["column_mapping_transforms.md", "comparison_types.md", "config.md", "feature_selection_transforms.md", "index.rst", "installation.md", "introduction.md", "link_tasks.md", "models.md", "pipeline_features.md", "running_the_program.md", "substitutions.md", "use_examples.md"], "titles": ["Column mapping transforms", "Comparison types, transform add-ons, aggregate features, and household aggregate features", "Configuration", "Feature Selection transforms", "Welcome to hlink\u2019s documentation!", "Installation", "Introduction", "Link Tasks", "Models", "Pipeline generated features", "Running hlink", "Substitutions", "Advanced Workflow Examples"], "terms": {"each": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10], "header": [0, 1, 2, 3, 9, 11], "below": [0, 1, 2, 3, 8, 9, 10], "repres": [0, 1, 2, 3, 9, 10], "type": [0, 2, 3, 4, 7, 8, 10, 11, 12], "ar": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 12], "us": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 12], "context": [0, 1, 3, 9], "column_map": [0, 2, 7], "list": [0, 1, 2, 3, 4, 8, 10, 11], "oper": [0, 2], "singl": [0, 2, 10, 12], "input": [0, 1, 2, 3, 6, 7, 10, 11], "output": [0, 1, 2, 3, 6, 7, 10, 12], "some": [0, 1, 3, 6, 7, 10], "suffix": [0, 4], "b": [0, 1, 2, 4, 10], "These": [0, 1, 2, 3, 6, 7, 8, 9], "mean": [0, 2], "appli": [0, 2, 3, 7, 12], "from": [0, 1, 2, 4, 6, 7, 8, 9, 10, 12], "onli": [0, 1, 2, 7, 12], "one": [0, 1, 2, 4, 7], "two": [0, 1, 2, 3, 6, 7, 9, 10, 12], "dataset": [0, 1, 2, 6, 7, 10, 12], "link": [0, 1, 2, 4, 6, 8], "A": [0, 1, 2, 9, 10], "most": [0, 1, 7, 10], "both": [0, 1, 2, 7, 12], "independ": [0, 2], "more": [0, 1, 2, 9, 10, 12], "than": [0, 1, 2, 8], "can": [0, 1, 2, 5, 6, 7, 8, 10, 12], "order": [0, 2, 7], "thei": [0, 1, 2, 7, 10], "re": 0, "configur": [0, 1, 6, 10, 12], "file": [0, 1, 4, 6, 7, 10, 11, 12], "so": [0, 1, 2, 5, 12], "mai": [0, 2, 6, 7, 10], "anoth": [0, 1, 3, 7], "format": 0, "The": [0, 1, 2, 3, 5, 7, 8, 9, 10, 12], "letter": [0, 4], "t": [0, 1, 2, 3, 12], "u": 0, "arbitrari": 0, "specifi": [0, 1, 2, 6, 7, 9, 10, 11], "column_nam": [0, 1, 2, 11], "attribut": [0, 1, 2, 3, 7, 8, 9, 10, 11], "under": [0, 1, 2], "its": [0, 1, 6, 10], "section": [0, 1, 2, 12], "name": [0, 1, 2, 4, 10, 11], "i": [0, 1, 2, 3, 5, 6, 7, 8, 10, 11, 12], "alia": [0, 2, 7], "along": [0, 1], "which": [0, 1, 2, 3, 6, 7, 9, 10, 12], "must": [0, 1, 2, 8, 9, 11], "addit": [0, 1, 2, 3, 5, 6, 10], "vari": [0, 2], "inform": [0, 1, 2, 10], "given": [0, 1, 2, 3, 8, 12], "often": [0, 10], "just": [0, 1, 2, 10, 12], "valu": [0, 1, 2, 4, 8, 9, 10, 11], "expect": [0, 1], "an": [0, 1, 2, 3, 6, 8, 10], "exampl": [0, 1, 2], "namefrst_split": [0, 2], "namefrst_clean": [0, 2], "add": [0, 4], "_": [0, 1, 2, 3, 4, 8, 9, 10], "numer": [0, 1], "11": [0, 2, 5, 9], "concat": [0, 4], "concaten": [0, 1], "string": [0, 1, 2, 3, 7, 8, 10, 11], "end": [0, 1, 2, 3, 11], "lowercas": [0, 4], "strip": [0, 4, 7], "clean": [0, 6], "convert": [0, 1, 2], "alphabet": 0, "charact": 0, "lower": [0, 1], "case": [0, 1, 2, 3, 6], "white": 0, "space": [0, 2, 3, 11], "start": [0, 11], "ration": [0, 4], "word": [0, 4], "replac": [0, 1, 4], "sinc": [0, 2], "peopl": [0, 1, 6, 10], "": [0, 1, 2, 6, 7, 10, 11], "raw": [0, 2, 7, 10], "censu": [0, 7, 12], "data": [0, 1, 4, 6, 7, 10], "contain": [0, 1, 11], "lead": 0, "better": [0, 6], "match": [0, 1, 4, 6, 10, 11, 12], "remov": [0, 4], "qmark": [0, 4], "hyphen": [0, 4], "punctuat": [0, 4], "thi": [0, 1, 2, 5, 6, 7, 9, 10, 12], "apostroph": [0, 4], "altern": [0, 2, 4], "If": [0, 1, 2, 3, 7, 8, 10, 11], "surround": 0, "all": [0, 1, 2, 3, 7, 8, 9, 10], "follow": [0, 1, 6, 10, 11, 12], "them": [0, 1, 2, 7], "jr": [0, 2], "sr": [0, 2], "ii": [0, 2], "iii": [0, 2], "stop": [0, 4], "last": [0, 1, 7, 9], "street": [0, 1], "avenu": [0, 11], "blvd": 0, "circl": 0, "court": 0, "road": 0, "prefix": [0, 4], "like": [0, 2, 7, 10], "m": [0, 1], "mr": 0, "In": [0, 1, 6, 10, 12], "ah": 0, "chines": 0, "condens": [0, 4], "whitespac": [0, 4, 7], "take": [0, 1, 2, 3, 7, 10], "non": 0, "leav": 0, "behind": 0, "arrai": [0, 2, 4, 8, 9], "index": [0, 4, 5], "select": [0, 1, 4, 6, 10, 12], "element": 0, "posit": [0, 1, 2, 4, 6], "first": [0, 1, 2, 5, 7, 10, 11], "second": [0, 1, 2, 11], "1": [0, 1, 2, 4, 7, 8, 9, 10, 12], "item": 0, "namefrst": [0, 1, 2], "set": [0, 1, 2, 3, 6, 7, 10, 12], "Then": [0, 5], "0": [0, 1, 2, 7, 8, 9, 10, 12], "give": [0, 2], "initi": [0, 1, 10], "person": [0, 1, 6], "probabl": [0, 2, 8], "middl": [0, 1], "namefrst_mid_init": [0, 1], "multipl": [0, 1, 2, 10], "otherwis": [0, 1, 9, 12], "known": 0, "recod": 0, "birthyr": [0, 2], "clean_birthyr": [0, 2, 3], "9999": [0, 2, 9], "1999": [0, 2], "9998": 0, "divid": [0, 4], "int": [0, 1, 2, 3, 4, 8], "integ": [0, 1, 2, 9], "It": [0, 1, 2, 6, 10, 12], "result": [0, 1, 6, 9, 10, 12], "For": [0, 1, 2, 7, 10, 12], "instanc": [0, 8], "birthplac": 0, "variabl": [0, 1, 2, 12], "detail": [0, 2, 10], "version": [0, 5, 12], "gener": [0, 1, 4, 6, 7, 10], "least": [0, 1], "signific": 0, "digit": 0, "make": [0, 1, 2, 5, 12], "we": [0, 1, 10, 12], "simpli": [0, 2], "drop": [0, 2, 10], "100": [0, 2, 12], "round": [0, 2], "lowest": 0, "whole": [0, 6], "number": [0, 1, 2, 7, 8, 10], "floor": [0, 4], "function": [0, 1, 2, 6, 10], "bpl": [0, 1, 2], "bpl_root": 0, "when": [0, 1, 2, 3, 4, 7, 12], "condit": [0, 1, 2, 3, 4, 7], "logic": 0, "work": [0, 1, 2, 5, 7, 10, 12], "sql": [0, 1, 2, 3, 4, 7, 10], "express": [0, 1, 2], "claus": [0, 1], "if_valu": 0, "else_valu": 0, "race": [0, 1, 2, 9, 12], "ipum": [0, 6], "code": [0, 1, 2, 5], "categori": [0, 8], "get": [0, 1, 4, 10], "down": [0, 6, 12], "nearest": 0, "produc": [0, 10], "relat": [0, 1, 2], "between": [0, 1, 2, 6, 7, 10, 12], "hundr": 0, "300": 0, "child": [0, 8], "household": [0, 4, 6, 8, 10, 12], "head": 0, "301": 0, "302": 0, "adopt": 0, "303": 0, "step": [0, 1, 2, 6], "usual": [0, 7, 12], "need": [0, 1, 2, 7, 10, 12], "2": [0, 1, 2, 3, 7, 8, 11, 12], "spous": 0, "3": [0, 1, 2, 5, 7, 8, 9, 12], "4": [0, 1, 8], "law": 0, "5": [0, 1, 2, 8, 9, 10, 12], "parent": [0, 1, 11], "6": [0, 2, 8, 9, 12], "7": [0, 1, 2, 8, 12], "sibl": 0, "12": 0, "relate_div_100": [0, 1, 2], "page": [1, 10], "ha": [1, 2, 6, 10, 12], "differ": [1, 2, 4, 6, 7], "avail": [1, 2, 3, 5, 7, 8, 9, 12], "comparison_featur": [1, 2, 7], "relatematch": [1, 2], "comparison_typ": [1, 2], "categor": [1, 2, 8, 9], "true": [1, 2, 3, 7, 9, 11, 12], "maximum": [1, 8], "jaro": [1, 9], "winkler": [1, 9], "find": [1, 7, 12], "greatest": 1, "among": 1, "cartesian": 1, "product": [1, 6, 12], "column": [1, 3, 4, 7, 9, 10, 11, 12], "namelast": [1, 2], "would": [1, 2, 12], "return": [1, 3, 8, 10], "four": 1, "namefrst_a": 1, "namefrst_b": 1, "namelast_b": 1, "namelast_a": 1, "requir": [1, 2, 3, 4, 7, 9, 10, 11], "maximum_jw": 1, "score": [1, 2, 7, 9], "compar": [1, 2, 6, 7], "namefrst_jw": [1, 2, 12], "geograph": 1, "filter": [1, 4, 7, 11], "major": [1, 10], "locat": [1, 2, 10], "chang": [1, 2, 5, 10, 12], "befor": [1, 2, 3, 5, 7], "boundari": 1, "zero": 1, "jw_street": 1, "enum_dist": 1, "max": [1, 8, 10], "member": [1, 7], "neighborhood": 1, "surnam": 1, "related_individual_max_jw": 1, "namefrst_rel": 1, "assert": [1, 10], "same": [1, 2, 6, 7, 10], "NOT": 1, "distinct": 1, "f1": 1, "evalu": [1, 2, 6, 7, 8], "either": [1, 2, 6, 11], "ani": [1, 2, 5, 8], "potenti": [1, 4, 7], "mismatch": 1, "queri": [1, 2], "fi": 1, "OR": 1, "mi0": 1, "mi1": 1, "THEN": 1, "els": [1, 2, 3], "first_init_col": 1, "namefrst_init": 1, "mid_init_col": 1, "namefrst_mid_init_2": 1, "f2": 1, "empti": 1, "null": [1, 2, 3], "AND": [1, 2], "individu": [1, 2, 7, 12], "mainli": 1, "caution": [1, 9], "flag": [1, 9, 10, 12], "f": [1, 10], "sp": 1, "m_caution": [1, 2, 9, 12], "mbpl": 1, "mother_birthyr": 1, "stepmom": 1, "momloc": 1, "comp_a": [1, 2], "comp_b": [1, 2], "comp_c": 1, "parent_step_chang": 1, "comp_d": 1, "check": [1, 10], "sign": 1, "boolean": [1, 2, 3, 11, 12], "form": [1, 7, 11], "cast": 1, "col": 1, "namelast_equal_as_int": 1, "namelast_clean": [1, 2, 3], "whether": [1, 2, 11], "join": [1, 11], "across": 1, "record": [1, 2, 6, 7], "being": [1, 7], "exact": [1, 2], "namefrst_unstd": [1, 2], "present": [1, 2, 9], "nonzero": 1, "primarili": [1, 7], "indic": [1, 12], "kind": 1, "incompar": 1, "akin": 1, "miss": [1, 10], "see": [1, 2, 5, 10, 12], "also": [1, 2, 5, 6, 7, 9, 10, 12], "univers": [1, 4, 7], "similar": 1, "fbpl_nomatch": 1, "fbpl": 1, "allow": [1, 2, 7, 12], "up": [1, 2, 10, 11], "sub": 1, "object": [1, 2, 6, 10], "document": [1, 8, 10, 12], "option": [1, 2, 3, 6, 7, 8, 10, 12], "sp_caution": [1, 2, 12], "spouse_bpl": 1, "spouse_birthyr": 1, "durmarr": [1, 2], "new_marr": [1, 2], "appear": 1, "street_jw": [1, 2, 12], "counti": 1, "statefip": [1, 2], "9": 1, "multipli": 1, "togeth": [1, 2], "after": [1, 2, 4, 8, 10], "float": [1, 2, 8], "have": [1, 2, 5, 6, 7, 8, 10, 12], "comp": 1, "c": 1, "sploc": 1, "012": 1, "fals": [1, 2, 3, 4, 6, 10], "d": 1, "specif": [1, 2, 10], "circumst": 1, "should": [1, 2, 8, 9, 10], "mid_init_match": 1, "either_1": 1, "nativ": 1, "either_0": 1, "gen": 1, "imm": [1, 2, 12], "immigr": 1, "look": [1, 10, 11], "foreign": 1, "born": 1, "sgen": [1, 2, 12], "rel": [1, 2, 12], "scala": 1, "determin": [1, 7], "greater": [1, 5], "jw_threshold": 1, "ag": [1, 2, 3], "less": [1, 2], "age_threshold": 1, "sex": [1, 2, 11], "sampl": 1, "related_individual_row": 1, "unrel": 1, "depend": [1, 2, 5, 12], "name_col": 1, "birthyr_col": 1, "namefrst_related_row": 1, "replaced_birthyr": [1, 2, 3], "extra": 1, "children": 1, "who": 1, "base": [1, 2, 7], "count": [1, 10, 12], "suspect": [1, 6], "relate_col": 1, "histid_col": 1, "id": [1, 2], "birth": 1, "year": [1, 2, 3, 4], "year_b": 1, "wa": [1, 12], "taken": 1, "minimum": [1, 8], "accept": [1, 2, 12], "consid": [1, 8], "histid": [1, 2, 12], "1910": [1, 2, 12], "8": [1, 2, 5, 10], "rate": 1, "calcul": [1, 12], "percentag": 1, "seen": 1, "neighbor": 1, "meet": 1, "95": 1, "nbor": [1, 2, 12], "namelast_neighbor": 1, "05": [1, 2], "ad": [1, 2], "namelast_popularity_sum": 1, "namelast_popular": 1, "length": [1, 2, 9], "size": 1, "ab": 1, "diff": 1, "absolut": 1, "you": [1, 2, 5, 10, 11, 12], "invalid": [1, 8], "instead": [1, 2, 5, 7], "marriag": 1, "durat": 1, "99": [1, 2], "placehold": 1, "unknown": 1, "exclud": 1, "those": [1, 2], "consider": 1, "byrdiff": [1, 2, 12], "mardurmatch": [1, 2], "14": 1, "minu": [1, 2], "subtract": 1, "geo": 1, "distanc": [1, 8], "lookup": 1, "tabl": [1, 2, 4, 7, 10, 12], "hlink": [1, 2, 5, 6, 7, 12], "core": [1, 7, 10, 12], "dist_tabl": 1, "py": [1, 2], "There": [1, 3, 7], "sever": [1, 6], "wai": [1, 5, 10], "kei": [1, 7, 10], "key_count": 1, "secondari": 1, "serv": 1, "back": 1, "primari": [1, 6], "doe": [1, 7, 12], "particularli": 1, "state": [1, 6], "much": [1, 7], "fewer": [1, 8], "combin": [1, 2, 3, 7], "thu": 1, "risk": 1, "fill": 1, "aren": 1, "ex": 1, "even": 1, "though": 1, "distances_fil": 1, "path": [1, 2, 10, 11, 12], "table_nam": 1, "what": [1, 2, 10, 12], "want": [1, 2, 10, 12], "do": [1, 3, 10, 12], "read": [1, 2, 7, 10], "onc": [1, 10], "loc_a": 1, "where": [1, 7, 10, 12], "come": 1, "loc_b": 1, "distance_col": 1, "source_column_a": 1, "sourc": [1, 4, 7, 10, 12], "source_column_b": 1, "loc_a_0": 1, "loc_a_1": 1, "loc_b_0": 1, "loc_b_1": 1, "secondary_key_count": 1, "backup": 1, "secondary_table_nam": 1, "secondary_distances_fil": 1, "secondary_source_column": 1, "secondary_loc_a": 1, "secondary_loc_b": 1, "secondary_distance_col": 1, "state_dist": 1, "state_distance_lookup": 1, "county_state_dist": 1, "csv": [1, 2, 7, 10, 11, 12], "statecode1": 1, "statecode2": 1, "dist": 1, "county_dist": [1, 2, 12], "county_distance_lookup": 1, "county_1900_1910_distances_km": 1, "from_icpsrctyi": 1, "to_icpsrctyi": 1, "from_statefip": 1, "to_statefip": 1, "distance_km": 1, "state_1900_1910_distances_km": 1, "fetch": 1, "neither": 1, "nor": 1, "mpre": 1, "m_namefrst": 1, "accord": 1, "niu": 1, "other": [1, 2, 12], "mfbplmatch": 1, "multi": 1, "search": 1, "special": 1, "simplifi": 1, "particular": [1, 2], "constraint": 1, "num_col": 1, "whose": 1, "templat": 1, "n": [1, 8, 9], "per": [1, 2, 8, 9, 10], "current": [1, 2, 10], "respect": [1, 7], "jw_col_templ": 1, "jw": 1, "pair": [1, 12], "equal_and_not_null_templ": 1, "final": [1, 2, 12], "comput": [1, 3, 7], "_namefrst": 1, "_bpl": 1, "_sex": 1, "25": 1, "nvl": 1, "sm_namefrst": 1, "sn_namefrst": 1, "sm_bpl": 1, "sn_bpl": 1, "sm_sex": 1, "sn_sex": 1, "pass": [1, 2, 7, 8], "default": [1, 2, 7, 8, 10], "flexibl": 1, "user": [1, 10], "write": [1, 10, 12], "own": [1, 2], "favor": 1, "reason": 1, "good": 1, "fallback": 1, "defin": [1, 7, 8, 9, 10], "spark": [1, 2, 5, 8, 9, 10, 12], "builtin": 1, "argument": [1, 10, 12], "namelast_jw_max": 1, "namelast1": 1, "namelast2": 1, "namelast3": 1, "abov": [1, 5], "extend": 1, "beyond": 1, "standard": [1, 11], "top": [1, 4], "level": [1, 4, 10], "everi": 1, "jw_f": [1, 2, 12], "father_namefrst": 1, "rais": [1, 3], "exponenti": 1, "squar": 1, "county_distance_squar": [1, 2, 12], "county_a": 1, "county_b": 1, "upper": 1, "gt": 1, "btwn": 1, "addl": 1, "var": [1, 2], "program": [1, 2, 7, 12], "report": [1, 4, 6, 10], "addl_var": 1, "check_val_expr": 1, "else_v": 1, "volumn": 1, "datasourc": [1, 2, 10], "yrimmig": 1, "immyear_diff": [1, 2, 9, 12], "To": [1, 5, 7, 10], "includ": [1, 2, 7, 9, 10], "train": [1, 4, 6, 8, 10], "independent_var": [1, 2, 12], "config": [1, 4, 7, 10, 12], "id_column": [1, 2], "_a": 1, "mult": 1, "exist": [1, 2, 10], "within": [1, 2, 6, 10, 11], "hh_train": [1, 2, 7, 10, 12], "hh": 1, "highest": [1, 2], "against": [1, 11], "ten": [1, 2], "tell": 2, "how": [2, 7], "descript": [2, 8, 10], "refer": 2, "here": [2, 7, 10, 12], "tutori": [2, 10], "script": [2, 6, 10], "discuss": 2, "readm": 2, "note": 2, "written": [2, 6], "toml": [2, 6, 10], "abl": 2, "json": [2, 10], "datasource_a": [2, 7], "datasource_b": [2, 7], "transform": [2, 4, 6, 7], "lowercase_strip": 2, "add_to_a": 2, "10": [2, 5, 12], "age_2": 2, "derived_from": 2, "expand_length": 2, "explod": [2, 7], "jaro_winkl": 2, "namelast_jw": [2, 12], "threshold": [2, 8, 12], "feature_nam": 2, "79": 2, "84": 2, "complex": 2, "machin": [2, 6, 7, 10, 12], "learn": [2, 6, 7, 10, 12], "probabilist": [2, 6], "drop_data_from_scored_match": 2, "us1900": 2, "us1900m_usa": 2, "p": 2, "parquet": [2, 7], "us1910": 2, "us1910m_usa": 2, "training_data_subset": 2, "serialp": 2, "rationalize_name_word": 2, "remove_qmark_hyphen": 2, "replace_apostroph": 2, "remove_suffix": 2, "remove_alternate_nam": 2, "condense_strip_whitespac": 2, "split": [2, 3, 4, 7, 8, 9, 12], "namefrst_std": [2, 11], "array_index": 2, "bpl_orig": 2, "divide_by_int": 2, "get_floor": 2, "statefip_h": 2, "output_typ": 2, "substitution_column": [2, 7, 11], "join_column": [2, 11], "join_valu": [2, 11], "substitution_fil": [2, 11], "name_std": [2, 11], "male": [2, 11], "femal": [2, 11], "feature_select": [2, 3, 7], "input_column": [2, 3, 9], "output_column": [2, 3, 9], "sql_condit": 2, "namelast_bigram": 2, "bigram": [2, 4], "bpl_clean": 2, "bpl_str": 2, "washington": 2, "bpl2_str": 2, "53": 2, "region": [2, 12], "attach_vari": 2, "region_dict": 2, "col_to_join_on": 2, "col_to_add": 2, "null_fil": 2, "col_typ": 2, "potential_matches_univers": [2, 7], "birthyr_3": 2, "namefrst_std_jw": [2, 12], "75": [2, 8, 12], "comparis": 2, "post": [2, 7], "hh_comparison": [2, 7], "threshold_expr": 2, "fetch_a": 2, "sex_equ": 2, "equal": [2, 11], "relate_a": [2, 9], "pipeline_featur": [2, 7, 9], "sex_region_interact": 2, "transformer_typ": [2, 9], "interact": [2, 4, 7, 12], "relatetyp": [2, 9], "bucket": [2, 7], "hit": [2, 10, 12], "scale_data": [2, 12], "training_data": [2, 10], "dependent_var": [2, 12], "score_with_model": [2, 12], "use_training_data_featur": [2, 7, 12], "split_by_id_a": [2, 12], "decis": [2, 4, 8, 12], "drop_duplicate_with_threshold_ratio": [2, 12], "n_training_iter": [2, 7, 12], "output_suspicious_td": [2, 12], "param_grid": [2, 12], "model_paramet": [2, 7, 8, 12], "random_forest": [2, 12], "maxdepth": [2, 8, 12], "numtre": [2, 8, 12], "005": 2, "threshold_ratio": [2, 8, 12], "logistic_regress": [2, 12], "50": [2, 12], "65": 2, "80": 2, "chosen_model": [2, 8, 12], "prediction_col": 2, "predict": [2, 12], "hh_col": 2, "hh_training_data_1900_1910": 2, "probit": [2, 4], "go": [2, 10], "your": [2, 5, 7, 10, 12], "uniqu": 2, "identifi": [2, 6, 12], "full": [2, 7, 12], "short": 2, "alphanumer": 2, "convert_ints_to_long": 2, "automat": [2, 5, 7], "long": [2, 11], "especi": 2, "assum": 2, "schema": 2, "sometim": 2, "term": 2, "bigint": 2, "thing": 2, "my_fil": 2, "subset": [2, 11], "limit": 2, "extract": 2, "new": [2, 12], "iv": 2, "v": 2, "vi": 2, "vii": 2, "viii": 2, "namelast_clean_bigram": [2, 3], "creat": [2, 6, 7, 9, 10, 11, 12], "fed": [2, 7], "prep": 2, "df": [2, 10], "men": 2, "describ": [2, 10], "newli": 2, "attempt": 2, "duplic": [2, 8], "row": 2, "conjuct": 2, "Will": 2, "conjunct": 2, "rang": [2, 9], "original_valu": 2, "plu": 2, "1870": 2, "expand": 2, "1867": 2, "1868": 2, "1869": 2, "1871": 2, "1872": 2, "1873": 2, "kept": 2, "keep": 2, "appropri": 2, "treat": [2, 9], "import": [2, 7, 10, 12], "dure": [2, 7], "hot": 2, "encod": [2, 3], "vector": [2, 9], "stage": 2, "well": 2, "upper_threshold": 2, "cannot": 2, "robust": 2, "ml": [2, 4, 8, 9], "typic": [2, 7], "leverag": 2, "api": [2, 6, 9], "piplin": 2, "regionf": 2, "sex_regionf_interact": 2, "immyear_caut": [2, 9], "myriad": 2, "explor": [2, 4, 6, 10], "part": [2, 7], "task": [2, 4, 6, 8, 12], "drop_duplicate_a": 2, "time": [2, 7, 10], "out": [2, 7, 12], "best": [2, 7], "smallest": 2, "possibl": 2, "ratio": [2, 8], "beta": [2, 8], "test": [2, 7, 12], "model_explor": [2, 10, 12], "hyper": [2, 6, 12], "paramet": [2, 6, 7, 8, 10, 12], "eval": 2, "skip": [2, 7], "apply_model": 2, "run_all_step": [2, 10, 12], "command": [2, 6, 10, 12], "try": 2, "creation": 2, "iter": 2, "scale": 2, "error": [2, 9], "1900": [2, 12], "about": [2, 10, 12], "1930": [2, 12], "1940": [2, 12], "fail": 2, "were": 2, "sure": [2, 5, 10], "scratch": 2, "although": 2, "know": 2, "haven": 2, "could": 2, "save": [2, 7, 12], "small": 2, "amount": 2, "process": [2, 6, 10], "repeatedli": 2, "help": [2, 7, 10], "neg": [2, 4, 6], "area": 2, "coverag": 2, "increas": [2, 9], "represent": [2, 7], "ensur": 2, "group": [2, 7], "a304bt": 2, "three": [2, 7], "b200": 2, "c201": 2, "d425": 2, "perform": [2, 6, 7, 11], "feature_import": [2, 7, 12], "srace": [2, 9, 12], "race_interacted_srac": [2, 9, 12], "hits2": [2, 12], "exact_mult": [2, 12], "ncount": [2, 3, 12], "ncount2": [2, 3, 12], "f_interacted_jw_f": [2, 12], "f_caution": [2, 12], "f_pre": [2, 12], "fbplmatch": [2, 12], "m_interacted_jw_m": [2, 9, 12], "jw_m": [2, 9, 12], "m_pre": [2, 9, 12], "mbplmatch": [2, 12], "sp_interacted_jw_sp": [2, 12], "jw_sp": [2, 12], "sp_pre": [2, 12], "mi": [2, 12], "fsoundex": [2, 12], "lsoundex": [2, 12], "oth": [2, 12], "imm_interacted_immyear_caut": [2, 12], "1900_1910_training_data_20191023": 2, "jw_max_a": 2, "jw_max_b": 2, "f1_match": 2, "f2_match": 2, "byrdifcat": 2, "racematch": 2, "bplmatch": 2, "imm_interacted_bplmatch": 2, "sexmatch": 2, "relatetype_interacted_relatematch": 2, "checkpoint": 3, "override_column_a": 3, "override_column_b": 3, "set_value_column_a": 3, "set_value_column_b": 3, "no_first_pad": 3, "don": 3, "prepend": 3, "provid": [3, 6, 7, 9, 10], "namefrst_unstd_bigram": 3, "namelast_frst_bigram": 3, "namelast_clean_soundex": 3, "input_col": 3, "output_col": 3, "expon": 3, "introduct": 4, "overview": 4, "instal": 4, "pypi": 4, "preprocess": [4, 6, 10, 12], "model": [4, 6, 10], "run": [4, 5, 6, 7, 12], "librari": [4, 6], "mode": [4, 5, 12], "advanc": 4, "workflow": 4, "export": [4, 7, 10], "featur": [4, 6, 7, 8, 10], "reus": 4, "basic": 4, "map": [4, 7, 9], "substitut": [4, 7], "block": [4, 7], "comparison": [4, 7], "pipelin": 4, "substr": 4, "ons": 4, "aggreg": 4, "union": 4, "soundex": 4, "power": 4, "regex": 4, "random": [4, 8], "forest": [4, 8], "logist": [4, 8], "regress": [4, 8], "tree": [4, 8], "gradient": [4, 8], "boost": [4, 8], "system": 5, "python": [5, 6, 10], "java": 5, "integr": 5, "apach": 5, "via": [5, 6], "pyspark": [5, 8, 9, 10], "packag": 5, "org": 5, "latest": 5, "pip": 5, "easiest": [5, 10], "through": [5, 7, 9, 10], "instruct": [5, 10], "But": 5, "clone": 5, "github": 5, "repositori": 5, "root": 5, "project": 5, "directori": [5, 10, 12], "develop": [5, 6], "e": 5, "dev": 5, "edit": 5, "made": 5, "built": 5, "tool": [5, 6], "line": [6, 10], "share": 6, "characterist": [6, 7], "correspond": [6, 7], "real": 6, "world": 6, "determinist": [6, 7], "rule": [6, 7], "algorithm": [6, 7], "At": [6, 7], "been": 6, "unit": 6, "census": 6, "hierarch": [6, 10], "structur": 6, "nest": 6, "howev": [6, 12], "tailor": 6, "ignor": 6, "common": [6, 7, 12], "highli": [6, 7], "languag": 6, "further": [6, 12], "broken": 6, "smaller": 6, "sequenc": 6, "linkrun": [6, 10], "prepar": [6, 7, 10], "research": 6, "experi": 6, "understand": 6, "tune": [6, 12], "relationship": 6, "varieti": 7, "normal": 7, "abbrevi": [7, 11], "regist": [7, 10], "datafram": [7, 10, 12], "request": 7, "classif": [7, 8], "ingest": 7, "mani": [7, 10], "aspect": [7, 10], "extens": 7, "longest": 7, "definit": 7, "reduc": 7, "drastic": 7, "improv": 7, "runtim": 7, "separ": 7, "total": 7, "potential_match": [7, 10], "satisfi": 7, "elig": 7, "support": [7, 8], "reshap": 7, "thought": 7, "ahead": 7, "chosen": 7, "experiment": [7, 10], "focus": 7, "demograph": 7, "moment": 7, "veri": [7, 12], "anyon": 7, "percent": 7, "remain": 7, "popul": 7, "pull": 7, "fix": 7, "width": 7, "crosswalk": 7, "construct": 7, "alpha": 8, "hyperparamet": [8, 12], "de": 8, "param": [8, 12], "label": 8, "doc": [8, 9], "commonli": 8, "explan": 8, "randomforestclassifi": 8, "depth": 8, "20": 8, "featuresubsetstrategi": 8, "node": 8, "auto": 8, "onethird": 8, "sqrt": 8, "log2": 8, "15": 8, "generalizedlinearregress": 8, "famili": 8, "binomi": 8, "85": [8, 10], "logisticregress": 8, "decisiontreeclassifi": 8, "mininstancespernod": 8, "caus": 8, "left": 8, "right": [8, 10], "discard": 8, "maxbin": 8, "bin": 8, "discret": 8, "continu": [8, 9, 12], "gbtclassifi": 8, "mother": 9, "point": [9, 12], "With": 9, "x": [9, 10], "y": 9, "hold": 9, "except": 9, "strictli": 9, "inf": 9, "explicitli": 9, "cover": 9, "doubl": 9, "outsid": 9, "job": 10, "high": 10, "class": 10, "handl": 10, "main": 10, "complet": 10, "access": [10, 12], "link_run": 10, "factori": 10, "sparkfactori": 10, "load_config": 10, "load_conf_fil": 10, "sparksess": 10, "now": 10, "let": 10, "load": 10, "our": 10, "my_conf": 10, "lr": 10, "prep_step": 10, "get_step": 10, "enumer": 10, "print": 10, "input_table_nam": 10, "output_table_nam": 10, "run_step": 10, "get_tabl": 10, "matches_df": 10, "hh_model_explor": 10, "method": [10, 12], "interfac": 10, "easili": 10, "conveni": 10, "adjust": 10, "set_loc": 10, "set_num_cor": 10, "set_executor_memori": 10, "5g": 10, "ll": 10, "dictionari": 10, "modul": 10, "pleas": 10, "enabl": [2, 7, 10], "reproduc": 10, "consol": 10, "cpu": 10, "usag": 10, "h": 10, "executor_memori": [10, 12], "execute_task": 10, "execute_command": 10, "conf": [10, 12], "show": 10, "messag": 10, "exit": 10, "memori": 10, "executor": 10, "begin": 10, "execut": 10, "seri": 10, "excute_command": 10, "filepath": 10, "sai": 10, "fullcount_1870_1880": 10, "pattern": 10, "full_count_1870_1880": 10, "prompt": 10, "enter": 10, "text": 10, "unstabl": 10, "topic": 10, "analyz": [10, 12], "set_preexisting_t": 10, "x_persist": 10, "borrow_t": 10, "get_task": 10, "set_print_sql": 10, "x_sql": 10, "x_sqlf": 10, "ipython": 10, "showf": 10, "x_summari": 10, "desc": 10, "x_crosswalk": 10, "x_tab": 10, "q": [10, 12], "x_hh_tfam": 10, "x_tfam": 10, "drop_al": 10, "reload": 10, "x_hh_tfam_2a": 10, "x_tfam_raw": 10, "drop_all_prc": 10, "x_hh_tfam_2b": 10, "x_union": 10, "drop_all_temp": 10, "x_load": 10, "get_set": 10, "set_link_task": 10, "x_parquet_from_csv": 10, "organ": 10, "hierarchi": 10, "five": 10, "hh_match": 10, "someth": 10, "choic": 10, "preexist": 10, "prepped_df_a": 10, "prepped_df_b": 10, "raw_df_b": 10, "raw_df_a": 10, "training_featur": [10, 12], "scored_potential_match": 10, "potential_matches_prep": 10, "exploded_df_b": 10, "exploded_df_a": 10, "predicted_match": 10, "hh_training_featur": [10, 12], "hh_training_data": 10, "hh_predicted_match": 10, "hh_scored_potential_match": 10, "hh_potential_match": 10, "hh_blocked_match": 10, "hh_potential_matchs_prep": 10, "model_eval_training_vector": 10, "model_eval_training_data": 10, "model_eval_repeat_fp": 10, "model_eval_training_featur": 10, "model_eval_training_result": 10, "model_eval_repeat_fn": 10, "hh_model_eval_training_vector": 10, "hh_model_eval_repeat_fp": 10, "hh_model_eval_repeat_fn": 10, "hh_model_eval_training_result": 10, "hh_model_eval_training_featur": 10, "hh_model_eval_training_data": 10, "persist": 10, "hidden": 10, "intermedi": 10, "yet": 10, "databas": 10, "tablenam": 10, "istemporari": 10, "task_nam": 10, "num": 10, "finish": 10, "By": 10, "put": [10, 12], "launch": [10, 12], "my": [10, 12], "subhead": 11, "suppli": 11, "regex_word_replac": 11, "variant": 11, "av": 11, "7th": 11, "swap": 11, "still": 11, "anywher": 11, "proceed": 11, "street_unstd": 11, "dir": 11, "substitutions_street_abbrev": 11, "span": 12, "1920": 12, "deriv": 12, "necessari": 12, "scenario": 12, "copi": 12, "use_potential_matches_featur": 12, "full_count_1900_1910": 12, "50g": 12, "ask": 12, "arg": 12, "partit": 12, "training_data_1900_1910_hlink_featur": 12, "might": 12, "shut": 12, "framework": 12, "etc": 12, "relev": 12, "matrix": 12, "implement": 12, "regular": 12, "training_data_1900_1910": 12, "weren": 12, "ident": 12, "manual": 12, "updat": 12, "isn": 12, "analysi": 12, "training_result": 12, "hh_training_result": 12, "1900_1910_training_result": 12, "repeat_fp": 12, "repeat_fn": 12, "hh_repeat_fp": 12, "hh_repeat_fn": 12, "1900_1910_potential_fp": 12, "1900_1910_potential_fn": 12, "prefer": 12, "ve": 12, "metadata": 7, "coeffici": [2, 7], "introspect": 7, "inspect": 7}, "objects": {}, "objtypes": {}, "objnames": {}, "titleterms": {"column": [0, 2], "map": [0, 2], "transform": [0, 1, 3, 9], "add_to_a": 0, "concat_to_a": 0, "concat_to_b": 0, "lowercase_strip": 0, "rationalize_name_word": 0, "remove_qmark_hyphen": 0, "remove_punctu": 0, "replace_apostroph": 0, "remove_alternate_nam": 0, "remove_suffix": 0, "remove_stop_word": 0, "remove_prefix": 0, "condense_strip_whitespac": 0, "remove_one_letter_nam": 0, "split": 0, "array_index": 0, "substr": 0, "divide_by_int": 0, "when_valu": 0, "get_floor": 0, "comparison": [1, 2], "type": [1, 9], "add": 1, "ons": 1, "aggreg": 1, "featur": [1, 2, 3, 9, 12], "household": [1, 2, 7], "maximum_jaro_winkl": 1, "jaro_winkl": 1, "jaro_winkler_street": 1, "max_jaro_winkl": 1, "equal": 1, "f1_match": 1, "f2_match": 1, "not_equ": 1, "equals_as_int": 1, "all_equ": 1, "not_zero_and_not_equ": 1, "time": 1, "caution_comp_3": 1, "caution_comp_3_012": 1, "caution_comp_4": 1, "caution_comp_4_012": 1, "any_equ": 1, "either_are_1": 1, "either_are_0": 1, "second_gen_imm": 1, "rel_jaro_winkl": 1, "extra_children": 1, "jaro_winkler_r": 1, "sum": 1, "length_b": 1, "abs_diff": 1, "b_minus_a": 1, "geo_dist": 1, "fetch_a": 1, "fetch_b": 1, "present_both_year": 1, "neither_are_nul": 1, "present_and_matching_categor": 1, "present_and_not_equ": 1, "present_and_equal_categorical_in_univers": 1, "multi_jaro_winkler_search": 1, "sql_condit": [1, 3], "alia": 1, "power": [1, 3], "threshold": 1, "lower_threshold": 1, "upper_threshold": 1, "gt_threshold": 1, "btwn_threshold": 1, "look_at_addl_var": 1, "hit": 1, "hits2": 1, "exact_mult": 1, "jw_max_a": 1, "jw_max_b": 1, "configur": [2, 4, 7], "basic": 2, "config": 2, "file": 2, "advanc": [2, 12], "top": 2, "level": 2, "data": [2, 11, 12], "sourc": [2, 5], "filter": 2, "substitut": [2, 11], "select": [2, 3], "potenti": [2, 12], "match": [2, 7], "univers": 2, "block": 2, "pipelin": [2, 9], "gener": [2, 9, 12], "train": [2, 7, 12], "model": [2, 7, 8, 12], "bigram": 3, "arrai": 3, "union": 3, "soundex": 3, "welcom": 4, "hlink": [4, 10], "": 4, "document": 4, "api": 4, "instal": 5, "requir": 5, "from": 5, "pypi": 5, "introduct": 6, "overview": [6, 7], "link": [7, 10, 12], "task": [7, 10], "preprocess": 7, "step": [7, 10], "relat": 7, "section": 7, "explor": [7, 12], "report": 7, "random_forest": 8, "probit": 8, "logistic_regress": 8, "decision_tre": 8, "gradient_boosted_tre": 8, "interact": [9, 10], "bucket": 9, "run": 10, "us": 10, "librari": 10, "mode": 10, "start": 10, "program": 10, "exampl": [10, 12], "workflow": [10, 12], "1": 11, "tabl": 11, "regex": 11, "word": 11, "replac": 11, "export": 12, "after": 12, "reus": 12, "differ": 12, "year": 12, "ml": 12, "list": 12, "fals": 12, "posit": 12, "neg": 12, "fp": 12, "fn": 12}, "envversion": {"sphinx.domains.c": 2, "sphinx.domains.changeset": 1, "sphinx.domains.citation": 1, "sphinx.domains.cpp": 6, "sphinx.domains.index": 1, "sphinx.domains.javascript": 2, "sphinx.domains.math": 2, "sphinx.domains.python": 3, "sphinx.domains.rst": 2, "sphinx.domains.std": 2, "sphinx": 56}})
\ No newline at end of file
diff --git a/hlink/linking/matching/link_step_score.py b/hlink/linking/matching/link_step_score.py
index 2c1ee0e..250c00d 100644
--- a/hlink/linking/matching/link_step_score.py
+++ b/hlink/linking/matching/link_step_score.py
@@ -4,7 +4,7 @@
 #   https://github.com/ipums/hlink
 
 import logging
-from pyspark.sql import Row, Window
+from pyspark.sql import Window
 from pyspark.sql import functions as f
 
 import hlink.linking.core.comparison_feature as comparison_feature_core
@@ -89,8 +89,6 @@ def _run(self):
             f"{table_prefix}potential_matches_pipeline"
         ).select(*pp_required_cols)
         score_tmp = plm.transform(pre_pipeline)
-        # TODO: Move save_feature_importances to training or model evaluation step
-        # _save_feature_importances(self.spark, score_tmp)
 
         alpha_threshold = chosen_model_params.get("threshold", 0.5)
         threshold_ratio = threshold_core.get_threshold_ratio(
@@ -109,22 +107,6 @@ def _run(self):
         self._save_predicted_matches(config, id_a, id_b)
         self.task.spark.sql("set spark.sql.shuffle.partitions=200")
 
-    def _save_feature_importances(self, spark, score_tmp):
-        config = self.task.link_run.config
-        if not config[f"{self.task.training_conf}"].get("feature_importances", False):
-            return
-        cols = (
-            score_tmp.select("*").schema["features_vector"].metadata["ml_attr"]["attrs"]
-        )
-        list_extract = []
-        for i in cols:
-            list_extract += cols[i]
-
-        varlist = spark.createDataFrame(Row(**x) for x in list_extract)
-        varlist.write.mode("overwrite").saveAsTable(
-            f"{self.task.table_prefix}features_list"
-        )
-
     def _save_table_with_requested_columns(self, pm, pmp, predictions, id_a, id_b):
         # merge back in original data for feature verification
         table_prefix = self.task.table_prefix
diff --git a/hlink/linking/model_exploration/link_step_get_feature_importances.py b/hlink/linking/model_exploration/link_step_get_feature_importances.py
deleted file mode 100644
index e859f55..0000000
--- a/hlink/linking/model_exploration/link_step_get_feature_importances.py
+++ /dev/null
@@ -1,85 +0,0 @@
-# This file is part of the ISRDI's hlink.
-# For copyright and licensing information, see the NOTICE and LICENSE files
-# in this project's top-level directory, and also on-line at:
-#   https://github.com/ipums/hlink
-
-from pyspark.ml import PipelineModel
-from hlink.linking.link_step import LinkStep
-from pathlib import Path
-
-
-class LinkStepGetFeatureImportances(LinkStep):
-    def __init__(self, task):
-        super().__init__(
-            task,
-            "get feature importances",
-            input_table_names=[
-                f"{task.table_prefix}training_features",
-                f"{task.table_prefix}training_vectorized",
-            ],
-            output_table_names=[f"{task.table_prefix}training_results"],
-        )
-
-    def _run(self):
-        training_conf = str(self.task.training_conf)
-        table_prefix = self.task.table_prefix
-        config = self.task.link_run.config
-
-        self.task.spark.sql("set spark.sql.shuffle.partitions=1")
-
-        if "feature_importances" in config[training_conf]:
-            if config[training_conf]["feature_importances"]:
-
-                # retrieve the saved chosen model
-                print("Loading chosen ML model...")
-                model_path = Path(config["spark_tmp_dir"]) / "chosen_model"
-                try:
-                    plm = PipelineModel.load(str(model_path))
-                except:
-                    print(
-                        "Model not found!  You might need to run step_2 to generate and train the chosen model if you haven't already done so."
-                    )
-
-                # make look at the features and their importances
-                print("Retrieving model feature importances or coefficients...")
-                try:
-                    feature_imp = plm.stages[-2].coefficients
-                except:
-                    try:
-                        feature_imp = plm.stages[-2].featureImportances
-                    except:
-                        print(
-                            "This model doesn't contain a coefficient or feature importances parameter -- check chosen model type."
-                        )
-                    else:
-                        label = "Feature importances"
-                else:
-                    feature_imp = feature_imp.round(4)
-                    label = "Coefficients"
-
-                varlist = self.task.spark.table(
-                    f"{table_prefix}features_list"
-                ).toPandas()
-                for i in varlist["idx"]:
-                    varlist.at[i, "score"] = feature_imp[i]
-                varlist.sort_values("score", ascending=False, inplace=True)
-                vl = self.spark.createDataFrame(varlist)
-                vl.write.mode("overwrite").saveAsTable(
-                    f"{table_prefix}feature_importances"
-                )
-
-                print(
-                    f"{label} have been saved to the Spark table '{table_prefix}feature_importances'."
-                )
-                print(varlist)
-
-            else:
-                print(
-                    f"'feature_importances' not set == true in '{training_conf}' section of config! Calculation of feature importances or coefficients not completed!"
-                )
-        else:
-            print(
-                f"'feature_importances' not included and set == true in '{training_conf}' section of config! Calculation of feature importances or coefficients not completed!"
-            )
-
-        self.task.spark.sql("set spark.sql.shuffle.partitions=200")
diff --git a/hlink/linking/model_exploration/model_exploration.py b/hlink/linking/model_exploration/model_exploration.py
index bd5df6a..489740c 100644
--- a/hlink/linking/model_exploration/model_exploration.py
+++ b/hlink/linking/model_exploration/model_exploration.py
@@ -8,7 +8,6 @@
 from .link_step_ingest_file import LinkStepIngestFile
 from .link_step_create_features import LinkStepCreateFeatures
 from .link_step_train_test_models import LinkStepTrainTestModels
-from .link_step_get_feature_importances import LinkStepGetFeatureImportances
 
 
 class ModelExploration(LinkTask):
@@ -22,5 +21,4 @@ def get_steps(self):
             LinkStepIngestFile(self),
             LinkStepCreateFeatures(self),
             LinkStepTrainTestModels(self),
-            LinkStepGetFeatureImportances(self),
         ]
diff --git a/hlink/linking/table_definitions.csv b/hlink/linking/table_definitions.csv
index 686cc9e..a2fbf77 100644
--- a/hlink/linking/table_definitions.csv
+++ b/hlink/linking/table_definitions.csv
@@ -18,6 +18,7 @@ training_data,Training: Raw training data as read from specified file,1
 hh_training_data,Household Training: Raw HH training data as read from specified file,1
 training_features,Training: Transformed data with features,1
 hh_training_features,Household Training: Transformed HH training data with features,1
+training_feature_importances,Training: Feature importances for the trained model,0
 model_eval_training_data,Model Exploration: Raw training data as read from specified file,1
 model_eval_training_vectorized,Model Exploration: Training data after applying comparison feature and pipeline transformations,1
 model_eval_training_results,Model Exploration: Results of ML model exploration train/test splits,0
diff --git a/hlink/linking/training/link_step_save_model_metadata.py b/hlink/linking/training/link_step_save_model_metadata.py
new file mode 100644
index 0000000..4c1189d
--- /dev/null
+++ b/hlink/linking/training/link_step_save_model_metadata.py
@@ -0,0 +1,98 @@
+# This file is part of the ISRDI's hlink.
+# For copyright and licensing information, see the NOTICE and LICENSE files
+# in this project's top-level directory, and also on-line at:
+#   https://github.com/ipums/hlink
+
+from hlink.linking.link_step import LinkStep
+
+
+class LinkStepSaveModelMetadata(LinkStep):
+    """Save metadata about the trained machine learning model.
+
+    By default this step is skipped. The training.feature_importances config
+    attribute enables it. The step saves either coefficients or feature
+    importances for the trained model, depending on the type of model. It saves
+    these data to the training_feature_importances Spark table.
+
+    This step is primarily helpful for debugging issues with machine learning
+    features and understanding how those features affect scoring of potential
+    matches.
+    """
+
+    def __init__(self, task):
+        super().__init__(
+            task,
+            "save metadata about the model",
+            output_table_names=[f"{task.table_prefix}training_feature_importances"],
+            input_model_names=[f"{task.table_prefix}trained_model"],
+        )
+
+    def _run(self):
+        training_conf = str(self.task.training_conf)
+        table_prefix = self.task.table_prefix
+        config = self.task.link_run.config
+
+        do_get_feature_importances = config[training_conf].get("feature_importances")
+
+        if do_get_feature_importances is None or not do_get_feature_importances:
+            print(
+                "Skipping the save model metadata training step. "
+                "To run this step and save model metadata like feature importances, "
+                "set feature_importances = true in the [training] section of your "
+                "config file."
+            )
+            return
+
+        print("Loading chosen ML model...")
+        try:
+            pipeline_model = self.task.link_run.trained_models[
+                f"{table_prefix}trained_model"
+            ]
+        except KeyError as e:
+            new_error = RuntimeError(
+                "Model not found!  Please run training step 2 to generate and "
+                "train the chosen model. The model does not persist between runs "
+                "of hlink."
+            )
+
+            raise new_error from e
+
+        # The pipeline model has three stages: vector assembler, classifier, post
+        # transformer.
+        vector_assembler = pipeline_model.stages[0]
+        classifier = pipeline_model.stages[1]
+
+        print("Retrieving model feature importances or coefficients...")
+        try:
+            feature_imp = classifier.coefficients
+        except:
+            try:
+                feature_imp = classifier.featureImportances
+            except:
+                print(
+                    "This model doesn't contain a coefficient or feature importances parameter -- check chosen model type."
+                )
+                return
+            else:
+                label = "Feature importances"
+        else:
+            label = "Coefficients"
+
+        column_names = vector_assembler.getInputCols()
+        # We need to convert from numpy float64s to Python floats to avoid type
+        # issues when creating the DataFrame below.
+        feature_importances = [
+            float(importance) for importance in feature_imp.toArray()
+        ]
+
+        features_df = self.task.spark.createDataFrame(
+            zip(column_names, feature_importances),
+            "feature_name: string, coefficient_or_importance: double",
+        ).sort("coefficient_or_importance", ascending=False)
+
+        feature_importances_table = (
+            f"{self.task.table_prefix}training_feature_importances"
+        )
+        features_df.write.mode("overwrite").saveAsTable(feature_importances_table)
+
+        print(f"{label} have been saved to the {feature_importances_table} table")
diff --git a/hlink/linking/training/training.py b/hlink/linking/training/training.py
index a17cf95..ec2eb95 100644
--- a/hlink/linking/training/training.py
+++ b/hlink/linking/training/training.py
@@ -8,6 +8,7 @@
 from .link_step_ingest_file import LinkStepIngestFile
 from .link_step_create_comparison_features import LinkStepCreateComparisonFeatures
 from .link_step_train_and_save_model import LinkStepTrainAndSaveModel
+from .link_step_save_model_metadata import LinkStepSaveModelMetadata
 
 
 class Training(LinkTask):
@@ -21,4 +22,5 @@ def get_steps(self):
             LinkStepIngestFile(self),
             LinkStepCreateComparisonFeatures(self),
             LinkStepTrainAndSaveModel(self),
+            LinkStepSaveModelMetadata(self),
         ]
diff --git a/sphinx-docs/config.md b/sphinx-docs/config.md
index bf41d6b..8747c52 100644
--- a/sphinx-docs/config.md
+++ b/sphinx-docs/config.md
@@ -688,7 +688,9 @@ splits = [-1,0,6,11,9999]
   * `use_training_data_features` -- Type: `boolean`. Optional. If the identifiers in the training data set are not present in your raw input data, you will need to set this to `true`, or training features will not be able to be generated, giving null column errors.  For example, if the training data set you are using has individuals from 1900 and 1910, but you are about to train a model to score the 1930-1940 potential matches, you need this to be set to `true` or it will fail, since the individual IDs are not present in the 1930 and 1940 raw input data.  If you were about to train a model to score the 1900-1910 potential matches with this same training set, it would be best to set this to `false`, so you can be sure the training features are created from scratch to match your exact current configuration settings, although if you know the features haven't changed, you could set it to `true` to save a small amount of processing time.
   * `output_suspicious_TD` -- Type: `boolean`.  Optional.  Used in the `model_exploration` link task.  Outputs tables of potential matches that the model repeatedly scores differently than the match value given by the training data.  Helps to identify false positives/false negatives in the training data, as well as areas that need additional training feature coverage in the model, or need increased representation in the training data set.
   * `split_by_id_a` -- Type: `boolean`.  Optional.  Used in the `model_exploration` link task.  When set to true, ensures that all potential matches for a given individual with ID_a are grouped together in the same train-test-split group. For example, if individual histid_a "A304BT" has three potential matches in the training data, one each to histid_b "B200", "C201", and "D425", all of those potential matches would either end up in the "train" split or the "test" split when evaluating the model performance.
-  * `feature_importances` -- Type: `boolean`. Optional, and currently not functional.  Whether to record feature importances for the training features when training or evaluating an ML model.
+  * `feature_importances` -- Type: `boolean`. Optional.  Whether to record
+    feature importances or coefficients for the training features when training
+    the ML model. Set this to true to enable training step 3.
 
 
 ```
diff --git a/sphinx-docs/link_tasks.md b/sphinx-docs/link_tasks.md
index f31a9e4..34e27ba 100644
--- a/sphinx-docs/link_tasks.md
+++ b/sphinx-docs/link_tasks.md
@@ -30,15 +30,21 @@ as they are read in.
 Train a machine learning model to use for classification of potential links. This
 requires training data, which is read in in the first step. Comparison features
 are generated for the training data, and then the model is trained on the data
-and saved for use in the Matching task.
+and saved for use in the Matching task. The last step optionally saves some metadata
+like feature importances or coefficients for the model to help with introspection.
 
 ### Task steps
 
-The steps in each of these tasks are the same:
+The first three steps in each of these tasks are the same:
 * Step 0: Ingest the training data from a CSV file.
 * Step 1: Create comparison features.
 * Step 2: Train and save the model.
 
+The last step is available only for Training, not for Household Training.
+* Step 3: Save the coefficients or feature importances of the model for inspection.
+  This step is skipped by default. To enable it, set the `training.feature_importances`
+  config attribute to true in your config file.
+
 ### Related Configuration Sections
 
 * The [`training`](config.html#training-and-models) section is the most important


Test patch:
diff --git a/hlink/tests/training_test.py b/hlink/tests/training_test.py
index 43668b2..6d9b400 100644
--- a/hlink/tests/training_test.py
+++ b/hlink/tests/training_test.py
@@ -85,6 +85,7 @@ def test_all_steps(
 
     # training_conf["training"]["use_potential_matches_features"] = True
     training_conf["training"]["score_with_model"] = True
+    training_conf["training"]["feature_importances"] = True
     training_conf["spark_tmp_dir"] = spark_test_tmp_dir_path
 
     training.link_run.trained_models["trained_model"] = None
@@ -110,6 +111,20 @@ def test_all_steps(
     assert row.prediction == 0
     assert row.state_distance_imp.round(0) == 1909
 
+    training.run_step(3)
+    tf = spark.table("training_feature_importances").toPandas()
+    for var in training_conf["training"]["independent_vars"]:
+        assert not tf.loc[tf["feature_name"].str.startswith(f"{var}", na=False)].empty
+    assert all(
+        [col in ["feature_name", "coefficient_or_importance"] for col in tf.columns]
+    )
+    assert (tf["coefficient_or_importance"] >= 0).all() and (
+        tf["coefficient_or_importance"] <= 1
+    ).all()
+    assert 0.4 <= tf.loc[0, "coefficient_or_importance"] <= 0.5
+    assert 0.2 <= tf.loc[1, "coefficient_or_importance"] <= 0.3
+    assert (tf.iloc[2:, 1] <= 0.1).all()
+
 
 def test_step_2_bucketizer(spark, main, conf):
     """Test a bucketized feature using spark pipeline function"""
@@ -236,3 +251,36 @@ def test_step_2_interaction(spark, main, conf):
     assert prepped_data.query("var1 == 3")["interacted_vars012"].iloc[0][0] == 18
 
     main.do_drop_all("")
+
+
+def test_step_3_skipped_on_no_feature_importances(training_conf, training, capsys):
+    """Step 3 is skipped when there is no training.feature_importances attribute
+    in the config."""
+    assert "feature_importances" not in training_conf["training"]
+
+    training.run_step(3)
+
+    output = capsys.readouterr().out
+    assert "Skipping the save model metadata training step" in output
+
+
+def test_step_3_skipped_on_false_feature_importances(training_conf, training, capsys):
+    """Step 3 is skipped when training.feature_importances is set to false in
+    the config."""
+    training_conf["training"]["feature_importances"] = False
+
+    training.run_step(3)
+
+    output = capsys.readouterr().out
+    assert "Skipping the save model metadata training step" in output
+
+
+def test_step_3_model_not_found(training_conf, training):
+    """Step 3 raises an exception when the trained model is not available."""
+
+    training_conf["training"]["feature_importances"] = True
+    with pytest.raises(
+        RuntimeError,
+        match="Model not found!  Please run training step 2 to generate and train the chosen model",
+    ):
+        training.run_step(3)