Machine Learning for Statistical Astrology 1 of 3 - An XOR Strategy for Filtering Too Much Data
Channel: AK1
Date: 2022-01-15

**NOTE: The actual machine learning model I build is in video 2, not this one.** This video outlines a filtering strategy which you may find more useful than building bad models on too much data to begin with.

This is the first of two videos (plus a postscript) where I build a machine learning model in Knime to study astrological data. Whether or not you are interested in astrology, this video will show you some neat tricks for working with the Knime Analytics platform. Also, if you are interested in basic machine learning you can do yourself (without any code), this video may end up being more useful to you sooner or later.

To solve a problem that had been blocking my work for a couple of years, I devised an XORing strategy to quickly filter data which is too big to easily process. The explanation of what it is and how to implement it is between 29:07 - 59:40.

Introduction to the data and its difficulties 
5:30 Column Filter - narrowing the investigation space
7:20 Column Splitter, Column Expressions, Column Spliter, Table Row to Variable - capturing topic with a generic column name, then saving the name as a variable
Getting data into 1s and 0s for machine learning 
13:28 Rule Engine - recoding category to 0 and 1
14:33 (A Strategy for avoiding wide data: turning data sideways and processing in (tall data) pieces)
17:00 Column List Loop, One-to-Many,  Column Rename (Regex) , String Manipulation (Variable) - one hot encoding categorical columns in pieces, capturing each column name and adding it to the table
22:47 Transpose, Loop End - rotate table to stack so many columns (memory hog) as rows instead, then use the regular Loop End to pile on *rows (even though we began with columns). Doing takes 90 seconds. If we had used the one-hot node on the whole table to begin with, it would have taken 10x longer (~17 minutes on my computer  instead of 1.5 minutes)
26:25 A new problem: how do you know which of so many factors to look at
29:07 **XOR 1 of 3*** What is the XOR strategy? (at 34:05 I should have said "closer to 3" not 6)
35:54 **XOR 2 of 3** Column Aggregator, Cross Joiner, Create Bit Vector, Transpose, RowID - XORing lots of data in Knime then restoring meaningful labels
44:41 **XOR 3 of 3** Normalization, Row Filter - determine what constitutes normal, and eliminate these--keeping only extremes
52:04 Sorter, RowID, String Replacer, GroupBy, Row Filter - cutting out misleading (slow moving) asteroids by determining who appears too many times (out of 12 here) on the extremes. (If the body in a particular sign is actually special then it should appear only once out of the 12)
59:40 Reference Row Filter - keep only succesfully XORed, non-misleading features from the original massive number of features available
1:04:00 Concatenate, Transpose - put the label back onto the features to prepare data for ML...