String Theory
By: Nicholas Teague
Tags: Data Science, Machine Learning

5. Family Tree Aggregations

An example composition of string parsing transformation aggregations, including generations and branches of derivations by way of entries to the family tree primitives, are now demonstrated for the root transformation category ‘or19’ [Fig. 4], which is available for assignment to source columns in context of an automunge(.) call. This transformation set is intended to automatically extract grammatical context from tabular data categoric features with a bounded range of unique entries.

Figure 4: Example family tree aggregations for bounded categoric sets

The sequence of four character keys represent transformation functions based on the transformation categories applied to a column. Note that each key represents a set of functions which may include one for application to train/test set(s) in an automunge(.) call for initial fitting of transformations to properties of a train set or a corresponding function for processing of a comparably formatted test set in postmunge(.) on the basis of properties from the train set. The steps of transformations for each returned column are logged by way of transformation function specific suffix appenders to the original column headers. Note also that some of the intermediate steps of transformations may not be retained in the returned set based on the configuration of family tree primitive entries.

The original application of an ‘UPCS’ transform serves the purpose of converting categoric entry strings to all uppercase characters, thus consolidating entries with different case configurations, e.g. a received set with unique entry strings {‘usa’, ‘Usa’, ‘USA’} would all be considered an equivalent entry (there may be some domains where this convention is not preferred in which case a user may deactivate a parameter to exclude this step). Adjacent to the ‘UPCS’ transform is a ‘NArw’ which returns boolean activations for those rows corresponding to infill based on the root category “processdict” defined type of source column values that will be target for infill (processdict is a data structure discussed further below, whose entries may either be a default or user specified). The included ‘1010’ transform for binary encoding distinguishes all distinct entries prior to any string parsing aggregations to ensure full information retention after the ‘UPCS’ transform both for purposes of ML training and also to support any later inversion operation to recover the original format of data pre-transforms as is supported in the postmunge(.) function. An alternate configuration could replace ‘1010’ with another categoric transform such as one-hot or ordinal encodings. The ‘nmc7’ transformation function is similar to ‘nmcm’ discussed earlier, but makes use of the added assumption that the set of unique entries in the test set will be the same or a subset of those values from the train set which allows for a more efficient application in the postmunge(.) function. These numeric extractions are followed by a ‘nmbr’ function for z-score normalization. Note that in some cases a numerical extract, such as those derived here from zip codes of passed addresses, may in fact be more suitable to a categoric encoding instead of numeric normalization. Such alternate configurations may easily be defined with the family tree primitives.

The remaining branches downstream of the ‘UPCS’ start with a ‘spl9’ function performing string parsing replacement operations comparable to the ‘spl2’ demonstrated above, but with the assumption that the set of unique entries in the test data will be the same or a subset of the train set for efficiency considerations. The ‘spl9’ parses through unique entry strings to identify character subset overlaps and replaces entries with the longest identified overlap, which results in returned columns with a fewer number of unique entries by aggregating entries with shared overlaps into a common representation, which may then be numerically encoded such as shown here with an ‘ord3’ transform. The ‘or19’ family tree also incorporates a second tier of string parsing with overlap replacement by use of the ‘sp10’ transform (comparable to ‘spl5’ with similar added assumptions for efficiency considerations as ‘spl9’). ‘sp10’ differs from ‘spl9’ in that unique entries without overlap are replaced in aggregate with an infill plug value to avoid unnecessary redundancy between encodings, again resulting in a reduced number of unique entries which may then be numerically encoded such as with ‘ord3’. Note that Fig. 4 also demonstrates an alternate root category configuration as ‘or20’ in which an additional tier of ‘spl9’ is incorporated prior to the ‘sp10’.

Figure 5: Demonstration of ‘or19’ returned data

Fig. 5 demonstrates the numerical encodings as would be returned from the application of the ‘or19’ root category to a small example feature set of categoric strings. It might be worth restating that due to the complexity scaling of the string parsing operation this type of operation is intended preferably for categoric sets with a bounded range of unique entries in the train set. The composition of returned sets are derived based on properties of the source column received in a designated train set, and these same bases are applied to consistently prepare data for test sets, such as sets that may be intended for an inference operation. In other words, when preparing corresponding test data, the same type and order of columns are returned, with equivalent encodings for corresponding entries and equivalent activations for specific string subset overlap partitions that were found in the train set.