Getting Caught Up
By: Nicholas Teague
Tags: Data Science, Machine Learning

3. A few new methods for preparing categoric data:

So we’ve been rolling out some really cool methods lately for extracting structure from categorical string sets by way of evaluation for overlaps between string compositions — using our ‘splt’ family of transformations. (As a quick asterisk these methods aren’t really intended for long form text, as they basically sequentially parse within embedded loops to identify overlaps in character compositions. However if you’ve got a bounded range of unique values in a categoric set, this can prove useful for identifying internal grammatical structures.) A few deviations include those that identify overlaps by way of an activation column for each train set identified overlap (such as splt/spl8), those that create a new column in which values with overlaps present are replaced with the (shorter length) overlap (such as spl2/spl9), or another deviation comparable to last but those values that are not replaced with overlaps are reset to zero (such as spl5/sp10) — with a key distinction being that in the cases of splt/spl2/spl5 the test set values are consistently parsed to the train set values to identify overlaps, and in spl8/spl9/sp10 the test set processing makes an assumption for efficiency that the set of unique values in the test set is the same or a subset of those values which were found in the train set. And using these functions we’ve assembled a few defined sets intended as an improvement for raw categoric embedding of bounded string sets of unknown structure by way of extracting grammatical patterns and encoding them for inference by the predictive algorithms, such that embeddings are based on properties of data found in the train set and consistently encoded for subsequent test set data.

An example of transformation trees for categoric sets, for more variations please see our READ ME. The orange boxes represent the internal derivations and the blue boxes the returned outputs. I probably overuse this color scheme, you know Florida Gators it’s kind of a thing.

I’ll demonstrate here what we’ve come up with in the family trees defined in our library for root categories or19 and or20. So basically if you want to apply these transforms, you can assign a column to the root category in the “assigncat” dictionary passed to an automunge(.) call. And the result will be a series of returned sets originating form that source column, with transforms based on property of data found in the train set, consistently applied for subsequent data in designated ‘test’ sets passed to automunge(.) or postmunge(.). As shown in the image above, the transform starts by converting all strings to uppercase characters via the UPCS function, which is based on the assumption that differences in string cases can be ignored (note there are also alternates of these methods in the library that omit the UPCS precursor). The set of distinct uppercase values are binary encoded via the ‘1010’ transform into a set of 1/0 designated columns. The ‘nmc8’ transform is another kind of string parsing in which unique values are parsed to extract any numeric character entries such as to return a numeric set, which in this case is followed by a z-score normalization via the ‘nmbr’ category. And then the string parsing for overlap methods are shown here making use of ‘spl9’ and ‘sp10’, wherein or19 has two tiers of overlap parsing, and or20 has three tiers of overlap parsing, in each case the parsing followed by an ordinal (integer) encoding with the ‘ord3’ transform which sorts entries by frequency of occurrence. The output would be distinct columns for each of the ord3 and nmbr encodings, and then the number of columns returned for the 1010 encoding would be a function of the number of distinct values found in the original set (after conversion to UPPERCASE).