What I’ve Learned: Microsoft Malware Prediction Competition on Kaggle
By: Tim Yee
Tags: Machine Learning, Data Science, Kaggle

Since many of the integers/floats could be converted further into unsigned integers (uint), the memory footprint could be reduced even further. (unsigned integers can hold 2x the positive value than that of signed integers, i.e. -128 to 127 versus 0 to 255 for int8, and uint8 respectively.) However, some consideration was taken for feature engineering and several numerical features retained floating point 32(float32) dtype because most hardware and software do not natively support below float32. In almost all scenarios float64/int64 was avoided since the features are not inherently large and scientific computation was unnecessary. All binary features (0,1) were converted either to int8 or uint8. To put these numbers into perspective, if a single cell of data has a dtype int8, then it takes 8bits, or 1Byte of storage capacity. If we load a training dataset with 9,000,000 rows and and assume that the entire database is stored using a int8 dtype, then it would take 9 MB (9,000,000 * 1Byte) per feature, or 729 MB memory in the RAM. However, that calculation is the absolute minimum.

The next challenge was exporting the optimized Pandas dataframe object for fast retrieval. Pickle was out of the option since it cannot handle large file sizes. So I did some research and I found Feather:

Feather is a fast, lightweight, and easy-to-use binary file format for storing data frames. It has a few specific design goals: Lightweight, minimal API: make pushing data frames in and out of memory as simple as possible Language agnostic: Feather files are the same whether written by Python or R code. Other languages can read and write Feather files, too. High read and write performance. When possible, Feather operations should be bound by local disk performance.

How fast is Feather compared to using pandas to read the dataframe? Using a Solid State Drive (SSD), I reduced loading time from about 1 minute per train/test dataset to under 2–3 seconds per train/test dataset on average. I can now load large datasets into working memory and execute code with minimal downtime. That is the difference between reading a .csv file using Pandas and reading a .ft file using feather. Don’t believe me? Try it out yourself.

# Loading the dataframe object using feather

train = feather.read_dataframe('train.ft') # Saving the dataframe object using feather

train.to_feather('train.ft')

Missing Values

Often times datasets contain missing values due to a variety of reasons. An example would be — a company was not actively collecting information until a certain business unit pointed out that it may be a good piece of information to collect, study and perhaps use it to improve or build a ML model. Or it could be — information just does not exist for that particular data point. An example may be not filling in optional information in a fill-able form.

First how do we figure out how to determine which rows are missing? A simple way to summarize all missing values is using .isnull() within the Pandas library.

# a simple way to summarize all missing values along the column

train[train.isnull().any(axis=1)] # summarize all missing values along the row/index

train[train.isnull().any(axis=0)]

So now that we know which values are missing, we have a couple of options to in our toolkit to handle missing values.

Drop missing values — depending on the percentage of data points that are missing in a feature (column), we may or may not want to drop that entire column. Impute missing values — this is when you replace the missing value with a systematic method. Options include: mean, median, mode, or model based.

Mean Imputation — replace the missing values with the mean of the non-missing values in the feature if it is a numerical feature.

Median Imputation — replace the missing values with the median of the non-missing values in that feature if it is a numerical feature.

Mode Imputation — replace the missing value with the mode of the non-missing values for both numerical and categorical features.

Model Based Imputation — build a model to predict (classification/regression) the missing values.

I should mention that some common ways to represent a missing value outside of the fact that Pandas internally handles missing values by representing them as np.nan, is that you may have a feature that internally identifies missing values. They can be “unknown” or “none” for a categorical, or -1 for a numerical. Each dataset is different and understanding the different ways a missing value can be represented can be helpful. In the real world, you may be working with a database engineer that may know the inner workings of how the data was stored and understand the intricacies. For this dataset, I had to discover everything myself.

Of course anytime you impute a missing value, it may be a good idea to also indicate whether you imputed that value through an additional binary feature[0,1]. Sometimes it is beneficial and can help improve a machine learning model a tiny bit.

Categorical Features

The next major hurdle was to plug and chug the dataset into any machine learning model. However, categorical features represented a major problem to overcome given that I had not yet worked on categorical features in my ML course. My initial response was just to one-hot-encode all the categorical features. I quickly realized that this would present more problems than it solves, namely I was dealing with categorical features with high cardinality. High cardinality is when a categorical feature contains highly unique values. For example ‘AvSigVersion’ is a feature representing anti-virus definition versions, which may be updated daily. Such a categorical feature contains tens of thousands of unique features. All One-Hot-Encoding will do is create 10k or more dummy features encoded in binary format — for that feature alone. This is no small issue. Imagine 9 million rows multiplied by 10k features times 1Byte. That is performing one-hot-encoding on this one feature alone will turn 9MB of data into 90GB. There is, of course, a way around using dense representation of a sparsely encoded feature, namely using sparse matrices. A sparse matrix stores only the non-zero values into three arrays. One stores the non-zero data value, another stores the row index and another stores column index. So, while we can one-hot-encode the entire train and test into sparse int8 dtypes, not all ML models will perform well. Moreover, time complexity may very well be unmanageable, despite solving space complexity with the sparse format. Really, the only options were mean target encoding, ordinal encoding, or frequency encoding.

The pros and cons are as follows:

Frequency Encoding : replaces the unique category with the frequency at which that unique value occurs throughout training data.

: replaces the unique category with the frequency at which that unique value occurs throughout training data. Ordinal Encoding : replaces the unique category with integer encoding, i.e. 1, 2, 3, … A downside would be, there is no ordinal relationship between categories, for example — SSD has no ordinal relationship with HDD and thus we are telling the machine learning algorithm that there is a natural relationship between the two categories.

: replaces the unique category with integer encoding, i.e. 1, 2, 3, … A downside would be, there is no ordinal relationship between categories, for example — SSD has no ordinal relationship with HDD and thus we are telling the machine learning algorithm that there is a natural relationship between the two categories. Mean Target Encoding: replaces the unique category with the mean response of that particular category. The mean response of a particular anti-virus signature version could be 0.50 with 0 being minimum and 1 being maximum. In this scenario, if there are an equal number of Detections (1) as there are no Detections (0), say 500 and 500 respectively, the mean response would come out to 500/1000 or 0.5. This is the mean response or target. The major drawback with using such an encoding method is that, while it can work well, it can over-fit the model to training data and perform worse in out of sample data. Precautions may be taken to prevent over-fitting to training data by calculating mean-response of a category using out-of-sample data or a cross-validation k-fold like approach. In other words, we only use a portion of the training data to calculate a different portion of training data, in the case the mean response/target.

# source: https://www.kaggle.com/artgor/is-this-malware-eda-fe-and-lgb-updated def frequency_encoding(variable):

t = pd.concat([train[variable], test[variable]]).value_counts().reset_index()

t = t.reset_index()

t.loc[t[variable] == 1, 'level_0'] = np.nan

t.set_index('index', inplace=True)

max_label = t['level_0'].max() + 1

t.fillna(max_label, inplace=True)

return t.to_dict()['level_0'] for col in tqdm(categorical):

freq_enc_dict = frequency_encoding(col)

train[col] = train[col].map(lambda x: freq_enc_dict.get(x, np.nan)).astype('float32')

test[col] = test[col].map(lambda x: freq_enc_dict.get(x, np.nan)).astype('float32')

Important Note: the code I have shared above combines train and test to utilize data leakage for competition purposes only. In the real world, such an approach cannot and should not be used. Data leakage is a real concern for every machine learning engineer and data scientist. It is when information from a testing set that has yet to be seen become available at training time. With the foresight of information that training data should not be aware of at the time of building the model, it can lead to bias, or worse, leading a practitioner to believe that their machine learning model generalization is better than it actually is. Such a practice can be dangerous and should be avoided in real world applications.

Thankfully, LightGBM handles categorical features internally by integer (ordinal) encoding¹, which is not a major issue with tree-based models. CatBoost on the other hand, also handles categorical features internally, except using a variation of the Mean Target Encoding method described above, with a smoothing constant.²

In the Jupyter notebook, it is shown that Frequency Encoding was performed, followed by sorting the frequencies of each category in descending order. On top of sorted frequencies, label encoding was performed. First of all, what benefit does frequency encoding offer? Nothing, or perhaps negative performance if used blindly/incorrectly. If used correctly, sorted frequency encoding may offer the benefit of improving model performance.

How can sorted frequency encoding offers the benefit of improving model performance? If sorted frequency correlates with the target which we are trying to predict (Malware detection rate), then sorted frequency encoding will be a smarter form of encoding.

Note: based on the problems of One-Hot-Encoding I just mentioned, during the competition, a competitor, Vladislav Bogorod published a Kaggle kernel that showed how to perform LightGBM using One-Hot Encoding. His elegant solution runs entirely within the Kaggle kernel using only 17GB of RAM. Please follow the link below if you would like to see his brilliant solution. https://www.kaggle.com/bogorodvo/lightgbm-baseline-model-using-sparse-matrix#L209

Feature Engineering

We can theoretically plug and chug the entire training data into LightGBM once we account for missing values (NaN). However, a primary factor in improving model performance is handcrafting new features that a machine learning model can extract meaning from. To give you an idea of newly crafted features, I took numerical features such as CPU core count and divided by the RAM capacity.

# create a new feature that combines the interaction between RAM and CPU

train['cpu_ram_ratio'] = train['Census_TotalPhysicalRAM'] / train['Census_ProcessorCoreCount']

Another numerical feature displays aspect ratio.

# Horizontal Resolution/ Vertical Resolution

train['aspect_ratio'] = train['Census_InternalPrimaryDisplayResolutionHorizontal'] / train['Census_InternalPrimaryDisplayResolutionVertical']

Finally, an example of feature engineering for the categorical feature is splitting a string containing a version: AvSigVersion — ‘1.273.1144.0’ into four separate features. Since a decimal separates the numbers, we can use the .split() method.

# this extracts the numerical value before the first decimal place train['AvSigVersion1'] = train['AvSigVersion].apply(lambda x: x.split('.')[0]



Feature engineering is a difficult topic to discuss in length since it is part art-form and part science. In order to fully appreciate feature engineering and how to best perform it, it would require detailed analysis of specific features including charts breaking down feature-feature interactions, and trial and error measurements on whether the engineered feature provided additional benefit to a model. I have opted to skip the discussion of this step since I am by no means an expert in this area nor does my experience allow much room for detailed discussions.

Because I was still learning and did not understand how to properly create feature-feature interactions during the competition, I will link to a top competitor discussing their solution post-competition. Because they are best able to explain the intricate details that I cannot. full write-up and code block

Non-Stationary (Unstable) Features

Perhaps one of the most important considerations in this competition is that the dataset contained at minimum 5 features that have non-stationary or unstable values across time. AvSigVersion (Anti-Virus Signature Version), EngineVersion, AppVersion, Census_OSVersion (Operating System Version), and Census_OSBuildRevision contained values in a test that was unseen in training data. So much so, that a simple Adversarial Validation could find a strong separation between train and test data using those 5 features alone. Adversarial Validation is a technique that combines training and testing datasets into a single dataset and applies a pseudo label of 0 to train and 1 to test. If a simple classifier can achieve high AUC for example, then it can be said that train and test datasets are not from the same distribution. In order to achieve reliable cross-validation (CV) that can generalize performance well on unseen (test) data, the training and testing datasets should not be separable by a simple machine learning classifier.

# Create Pseudo-Label

train['label'] = 1

test['label'] = 0

train['label'] = train['label'].astype('uint8')

test['label'] = test['label'].astype('uint8') # combine train and test

total = pd.concat([train, test], axis=0)

psuedo_label = total['label']

total.drop(['label'], inplace=True, axis=1)

total.reset_index(drop=True, inplace=True)

By using LightGBM on all 17 million data points between train and test, LightGBM was able to achieve well over 0.9 AUC, which indicates that some or all features are contributing to the differences between train and test data. In order to investigate which features contributed to the high degree of train/test separability as indicated by the high AUC metric, I performed forward feature selection by dropping the top feature by importance and re-ran LightGBM using default parameters.

Based on my Adversarial Validation, combined with forwarding feature selection, and further EDA on the matter. I decided that there were five unstable features. In theory, there should be no way a model can take advantage of a feature containing unseen values.

Real life problems are rarely without these non-stationary problems. Learning to detect unstable features by adding Adversarial Validation to my ML arsenal has given me confidence that I will know how to tackle future problems with non-stationarity.

Adversarial Validation alone, however, is not enough to uncover the nature of differences between training and testing (out of sample) data. To illustrate, we will look to AvSigVersion dates scraped from Microsoft’s website.