Deep Learning for EEGs and BCI : (How I Managed to get 94% score model.)
By: Peijin Chen
Tags: Machine Learning, Deep Learning, Eeg, Time Series Analysis, Bci

Severe Class Imbalance is Hell

Part of what made me spend a lot of time on treating this as a multi-class classification problem is that I thought I could fix this by either under or over-sampling. This introduced me to two Python libraries, imblearn and smote-variants. Let’s say that you have 1.5 million data points and about 90% of them are the ‘0’ category, which is [0,0,0,0,0] in original vector form. Well if you undersample it to about 50,000, then oversample the others, you could get something even. There are a few problems with this:

In theory, if the data set is balanced, the learner could learn each one almost equally, which would mean it would do well on any test data. However, in reality, there are certain categories that happen more than others. If you took value counts and just learned the top 3 well, you’d probably do decently in the test data (assuming test data isn’t wacky). How much? Category 0 has a1 million data points, and the smallest category might only have 2000 points. So you have to oversample using the SMOTE or other such methods. Now imblearn has classic, borderline and KNN type SMOTEs, and smote-variants has more SMOTE than you can shake a stick at. I thought I’d try one or two that I’d read about in papers, but the thing is, it’s a losing battle. Computational cost — only random undersampling is fast. Other methods took too long for my taste. Random undersampling can discard useful data. You have no idea if it’s outlier junk or decision boundary defining data. I tried using outlier detection to trim off the outliers before doing some undersampling…but again, there’s too much flying in the dark for me.

Nevertheless, I persisted

I wanted to use Keras. Now, using imblearn means we have this thing called BalancedBatchGenerator — just for keras — which is a data generator like the kind you find in keras for time series or images, most notably. Rather than augmenting data, it mostly just creates balanced training mini-batches. Now if you’ve already done some a priori balancing, like I did before, you might not think there is a need for this — but that depends on how balanced you want things. If you decided YES, I will use BalancedBatchGenerator (hereafter BBG), you will have to decide how to achieve balance in the mini-batch — you can either use the over or under-sampling (or combo) methods in imblearn — but remember each one of these comes with a cost. If you do CondensedNearestNeighbour under sampler in auto-mode, it will have to re-sample EVERY class in the mini-batch until you get your perfect balance. tl;dr: NOT WORTH IT, takes forever. Too much computational cost just to get one friggin training mini-batch. I thought: this way I don’t have to undersample ahead of time — locking in my data loss, a priori. If you randomly undersample you will probably, on average, get mini-batches that have ‘good’ data that will help define a clear decision function. Still, in the end, I never got good results when I submitted my predictions to the Kaggle site. I never got about 65% on the AUC. The high score was 98%.

Here’s what worked, and fairly easily too:

Considering this a time series classification problem. First, preprocess using MNE library to bandpass from .1 to 40Hz. I thought using the mu bands, eg 8–12Hz and 16–30 Hz, might be enough. You could concatenate them or even just combine using 8–30Hz bandpass. But I never got great results. I also thought that just using C3, C4, Cz electrodes would be enough, since they have a lot to do with left/right hand motor functions. Still, not great results. So I stuck with 0.1–40Hz, then normalized. Here’s some code:

X_gen = mne.filter.filter_data(data = Xt, sfreq=500, l_freq=0.1, h_freq=40, picks = None)

somescaler = StandardScaler()

X = somescaler.fit_transform(X)

Then: segment the signal into ‘epochs’ (in the signal processing sense), which means short segments, maybe 0.5 seconds, maybe 3 seconds. I used keras’ TimeSeriesGenerator — which is awesome — to sample the last 1000 points, but at a rate of 10. This means I am using 100 data points to cover the last two seconds of signal, or put another way, I’m working at 50hz. I figured that two seconds before or after a signal is enough time to see your brain waves doing the ‘desynchronization’ thing or at least just ‘gearing up’ to do whatever motion you’re told to do.

Furthermore, I used a stride of 125 seconds or so — which means I made overlapping 2 seconds epochs every 0.25 seconds. Here’s the code:

trainX, valX, trainY, valY = train_test_split(X, y, test_size=0.30, random_state=3249, shuffle = False)

print(trainX.shape, trainY.shape, valX.shape, valY.shape) train_gen = TimeseriesGenerator(trainX, trainY, length = 1000,

stride = 50,

sampling_rate = 10,

shuffle = True,

batch_size = 1000)

val_gen = TimeseriesGenerator(valX, valY, length = 1000,

stride = 50,

sampling_rate = 10,

shuffle = True,

batch_size = 1000)

The Shootout: LSTMs vs Temporal Convolutional Networks vs EEGNet.

This is the LSTM model:

i = Input(shape=(100,32))

out = Bidirectional(LSTM(8,return_sequences = False, dropout = 0.2))(i)

#out = GlobalAveragePooling1D()(out)

out = Dense(6, kernel_initializer = 'he_uniform')(out)

model = Model(inputs=[i],outputs=[out])

This is my generic temporal conv1D type architecture:

from keras.layers import AveragePooling1D

model_m = Sequential()

model_m.add(Dropout(0.50, input_shape=(100,32)))

model_m.add(Conv1D(128, 4, kernel_constraint = max_norm(1.),padding = 'same'))

model_m.add(Conv1D(128, 4, kernel_constraint = max_norm(1.),padding = 'same',)) model_m.add(BatchNormalization(axis = -1))

model_m.add(Activation("relu"))

model_m.add(MaxPooling1D(2))

model_m.add(Dropout(0.50)) model_m.add(Conv1D(128, 10, kernel_constraint = max_norm(1.),padding = 'same'))

model_m.add(Conv1D(128, 10, kernel_constraint = max_norm(1.),padding = 'same'))

model_m.add(BatchNormalization(axis=-1))

model_m.add(Activation("relu"))

model_m.add(MaxPooling1D(10))

model_m.add(Flatten())

model_m.add(Dropout(0.50))

model_m.add(Dense(64, kernel_regularizer = l2(1e-5), activation = 'relu', kernel_constraint = max_norm(1.)))

model_m.add(Dense(6, kernel_regularizer = l2(1e-5),activation = 'sigmoid',kernel_constraint = max_norm(1.)))

model_m.summary()

model_m.compile(loss = "binary_crossentropy", optimizer = adam(lr = 0.001), metrics = ["acc"])

Here is the temporal convolutional network from the awesome implementation of Phillipe Remy:

i = Input((100,32))

o = TCN(nb_filters = 64, kernel_size = 8, nb_stacks = 1, dropout_rate = 0.4, dilations = [2,4,6,8])(i)

#o = Dense(64, activation = 'elu', kernel_initializer = 'he_normal')(o)

o = Dense(6, activation = 'sigmoid', kernel_initializer = 'he_normal')(o)

model1 = Model(inputs = [i], outputs = [o])

And here is my implementation of EEGNet. This is based on the 2018 paper by Lawhern et. al. Now, EEGNet is touted as a general EEG/BCI type classification architecture. It’s meant to be fairly simple and shallow, but achieve good results. Which it does. The original implementation in keras, by the authors of the paper, uses channels_first, but I could never quite get used to it, so I changed it to channels_last. Some other details, like dropout rate and pooling size, might also be different from their ‘standard’ model:

F1 = 16

from keras.layers import SpatialDropout2D

D = 4 ; F2 = D * F1

nb_classes = 6; norm_rate = 1.

kernLength = 10

Chans = 32

Samples = 100 input1 = Input((100,32))

block1 = Lambda(lambda x: K.expand_dims(x), output_shape = (100,32,1))(input1)

#block1 = ExpandDimension(axis = -1)(input1)

block1 = Conv2D(F1, (1, kernLength), padding = 'same',

input_shape = (1, Chans, Samples),

use_bias = False)(block1)

block1 = BatchNormalization(axis = -1)(block1)

block1 = DepthwiseConv2D((1,Chans), use_bias = False,

depth_multiplier = 4,

depthwise_constraint = max_norm(1.))(block1) block1 = BatchNormalization(axis = -1)(block1)

block1 = Activation('elu')(block1)

block1 = AveragePooling2D((4,1))(block1)

block1 = SpatialDropout2D(0.50)(block1) block2 = SeparableConv2D(F2, (1, 16), use_bias = False, padding = 'same')(block1)

block2 = BatchNormalization(axis = -1)(block2)

block2 = Activation('elu')(block2)

block2 = AveragePooling2D((5,1))(block2)

blok2 = SpatialDropout2D(0.50)(block2) block2 = Flatten()(block2)

out = Dense(6, activation = 'sigmoid')(block2) ##################################################################

model2 = Model(inputs = [input1], outputs = [out])

model2.summary()

Caveats before the final conclusion

The training was per-subject — I never got around to training an overall model. Also, I didn’t train them equal number of epochs — sometimes 40, sometimes 60. I used cosine-annealing sometimes, didn’t other times. Mostly it was Adam at .001 learning rate, with gradient clipping set at .10.

Some of my final scores.

Conclusion

Overall, the best score was the conv_bn one — which corresponds to my generic TCN model with a few layers of convolutions and batch normalization. EEGnet and the longer temporal convolutional network, with various dilations built in, did similarly well. The LSTM based architecture can do over 90%, which isn’t shabby — however, it was a shallow LSTM. I didn’t have the patience to train some deep stacked LSTM type network. The second place winner of the actual competition used recurrent convolutional combinations, the exact form of which I am unsure, so there is something to explore there.

This whole thing took me damn well near 4 months and I wish it hadn’t, but I did learn some good things from all the pitfalls. I’m especially psyched by learning about the SAX_VSM, WEASEL, and other such methods of time series classification, all of which involve discretization of time series data into ‘words’ which are then classified using tf-idf, bag of words type methods. Another thing worth looking into is Learning Shapelets method of time series classification. Stay tuned!