Feature Importance with Time Series and Recurrent Neural Network | by Marco Cerliani | Towards Data Science

Neural networks are often considered black-box algorithms. This is true but with few tricks and some useful external inference techniques we can extract lots of information from them. Amazing examples come from the application of Deep Learning with images. The extraction of internal features from the network is useful to understand how the network works: famous are heatmap representations which point in the image the areas of interest for the model to make a decision.
The concepts behind this process of knowledge extraction can be generalized for every domain of application of deep learning, like NLP and Time Series forecasting. Practically speaking, we can query our trained neural network with few lines of code with the scope of check the importance of output filters towards the final prediction.
In this post, I investigate the decision taken by a neural network trained to forecast the future. I used a recurrent structure to automatically learn information also from the time dimension. With some simple steps, we extract all we needed to understand the output of our model. We do this without the usage of any additional external library but simply squeezing what the model has learned.
The data for our experiment is collected from the UCI repository. The Beijing PM2.5 Dataset stores, as suggested by the name, the pm2.5 hourly observations (from 2010 to 2015) together with other weather regressors like temperature, pressure, wind and rain levels. Our scope is to forecast the exact value of pm2.5 one step ahead (the concentration of the next hour).
We use the first three years at our disposal as trainset and the latter two respectively for validation and test.
To train our sequential neural network we rearrange the data properly as 3D sequences of dimension: sample x time dimension x features. Below an example of a generated sequence of length 30 hours, where the values of each series are standardized with standard scaling.
This is an example of input samples for our model. We also standardize the target with mean and standard deviation.
The neural network structure we use for time series forecasting is as follow:
def get_model(params):        inp = Input(shape=(sequence_length, len(columns)))    x = GRU(params['units_gru'], return_sequences=True)(inp)    x = AveragePooling1D(2)(x)    x = Conv1D(params['units_cnn'], 3,                activation='relu', padding='same',                name='extractor')(x)    x = Flatten()(x)    out = Dense(1)(x)        model = Model(inp, out)    model.compile(optimizer=Adam(lr=params['lr']), loss='mse')        return model
A recurrent layer processes the initial sequences returning the full output in form of sequences. An average pooling compresses the output which is then treated by a one-dimensional convolutional layer. In the end, all is flattened to turn in a 2D dimension before generating the final predictions. Below a comparison between predictions and real values in a subsample of the test set (this is a demo, we are not interested in performances).
With our neural network trained we are ready to inspect the predictions. I provide two interpretations of the model outputs which can be easily retrieved for every test sequence.
Our idea is to check the contribution of each single input feature on the final prediction output. The contribution in our case is given by the value of the gradients obtained from the differentiation operation of the input sequences on the forecasts.
With Tensorflow, the implementation of this method is only 3 steps:
use the GradientTape object to capture the gradients on the input;
get the gradients with tape.gradient: this operation produces gradients of the same shape of the single input sequence (time dimension x features);
obtain the impact of each sequence feature as average over the time dimension.
def gradient_importance(seq, model):    seq = tf.Variable(seq[np.newaxis,:,:], dtype=tf.float32)    with tf.GradientTape() as tape:        predictions = model(seq)    grads = tape.gradient(predictions, seq)    grads = tf.reduce_mean(grads, axis=1).numpy()[0]        return grads
Some graphical examples of results are shown below:
The higher the average gradients are, the bigger is the impact of the feature on the final prediction. So for each of the cases shown above, we can identify which variable, in the input sequence, weighs more for the prediction of the next hour value of pm2.5 concentration.
Visualizing which input feature influences the most a prediction can help detect weird behaviors. However, it gives fewer insights into why a neural network makes a decision. This method tends to underline what specific part of the input is influencing the output value.
The process of reaching these conclusions is based on the same assumption made before. In other words, we need to operate gradient importance. The difference now it’s that this operation is computed with some internal filters instead of with the raw input sequence. Given our internal convolutional/recurrent filters, we operate the gradients towards the outputs. To establish the importance of each filter in the decision, we take the average of its weights (gradients importance) and multiply each map by its corresponding weights. If an activation map has been lightened up during forward pass and if its gradients are large, it means the region which is activated has a large impact on the decision.
Speaking in code language, little changes have to be made; particularly in the end where we operate normalization and we need to adjust the dimensions, to match the input sequences.
def activation_grad(seq, model):        seq = seq[np.newaxis,:,:]    grad_model = Model([model.inputs],                        [model.get_layer('extractor').output,                         model.output])    # Obtain the predicted value and the intermediate filters    with tf.GradientTape() as tape:        seq_outputs, predictions = grad_model(seq)    # Extract filters and gradients    output = seq_outputs[0]    grads = tape.gradient(predictions, seq_outputs)[0]    # Average gradients spatially    weights = tf.reduce_mean(grads, axis=0)    # Get a ponderated map of filters according to grad importance    cam = np.ones(output.shape[0], dtype=np.float32)    for index, w in enumerate(weights):        cam += w * output[:, index]    time = int(seq.shape[1]/output.shape[0])    cam = zoom(cam.numpy(), time, order=1)    heatmap = (cam - cam.min())/(cam.max() - cam.min())        return heatmap
In this post, I’ve built a sequential neural network for time series forecasting. Our purpose was to extract useful information from it. To do this, we’ve provided 2 different methods, based on gradients computation to operate inference after training. I’ve shown that also for sequential neural networks (recurrent/convolutional), in a time series application, it’s possible to investigate the behavior of deep models with simple tricks and few lines of code.
CHECK MY GITHUB REPO
Keep in touch: Linkedin