Are Deeper Networks Better? —A Case Study | by Timothy Tan | Towards Data Science

Have you ever wondered about the effects of building deeper networks and what the changes entail?
I know I have.
I’m talking about the added complexity in terms of additional layers to your model and scrutinizing what downstream changes occur.
Ceteris paribus:
What effects (if any) does an additional layers have on your training/validation loss curve?To what extent does an additional layer affect your downstream prediction outcomes?What is the relationship between the number of input features versus the complexity of a model?Are there any trade-offs involved here?
What effects (if any) does an additional layers have on your training/validation loss curve?
To what extent does an additional layer affect your downstream prediction outcomes?
What is the relationship between the number of input features versus the complexity of a model?
Are there any trade-offs involved here?
In this post, I will be sharing a series of experiments and their respective outcomes to answer these questions.
Let’s get down to it!
Before we begin, I’ll need to explain the experimental setup involved.
The setup comprises of 3 parts:
The training data set usedThe model architecture built for the experimentThe experiment methodology
The training data set used
The model architecture built for the experiment
The experiment methodology
The data set used for the experiments comes from a Kaggle competition; tweet sentiment extraction.
The goal?
To extract out phrases that supports the given sentiment.
Here’s a glimpse of the training data:
I will not be going in-depth into the data preparation methodology as I will be writing about the Kaggle experience in another post.
In essence, we cleaned the text up following the Tweeter GloVe methodology and created our own labels for the downstream Named Entity Recognition (NER) task.
Once we had a modelling ready data set, I started to build a baseline seq2seq model.
The model architecture consists of 3 main segments:
Pre-attention Bi-LSTMMulti-head Self-attention Encoder Stack(s)Post-attention LSTM
Pre-attention Bi-LSTM
Multi-head Self-attention Encoder Stack(s)
Post-attention LSTM
The model starts with a pre-attention Bi-LSTM whose outputs are fed into encoder stack(s).
The outputs from these encoder stacks are then fed into a post-attention LSTM to generate sequential predictions for the NER task at hand.
Why encoder stacks?
The inspiration of using encoder stacks came from the the paper “Attention Is All You Need” by Vaswani et. al.
The paper details how a transformer model looks like.
In short, the transformer model consists of 6 encoder stacks and 6 decoder stacks. I won’t go into the details of the paper but if you want to know the architecture that underlies the famed Bidirectional Encoder Representations from Transformers (BERT). I recommend the read.
Anyway, here’s how an encoder stack looks like:
As you can see, each encoder stack consists of two sub-layers.
The first being a multi-head self-attention mechanism and the second, a fully connected feed-forward network.
Around each sub-layer, a residual connection followed by a layer normalization was employed. This makes sense because the residual connection sums up the pre-attention Bi-LSTM outputs with the context learned by the multi-head self-attention layer.
So how does our resulting model architecture look like?
If I were to draw it out, it will look like this:
As a baseline, the architecture starts with only a single encoder stack.
Here were the layer parameters used across experiments which were kept constant:
Pre-attention Bi-LSTM:preatn_forward={type="recurrent", n=64, act='tanh', init='xavier', rnnType='LSTM',outputType='samelength', reversed=False, dropout=0.3}preatn_backward={type="recurrent", n=64, act='tanh', init='xavier', rnnType='LSTM',outputType='samelength', reversed=True, dropout=0.3}concatenation_layer={type="concat"}
Encoder Stack:attention_layer={type="mhattention", n=128, act='gelu', init='xavier', nAttnHeads=16}residual_layer1={type="residual"}norm_layer1=layer={type="layernorm"}fully_connected_layer={type="fc", n=128, act='relu', init='xavier', dropout=0.3}residual_layer2={type="residual"}norm_layer2=layer={type="layernorm"}
Post-attention LSTM:postatn_forward={type="recurrent", n=128, act='tanh', init='xavier', rnnType='LSTM',reversed=False, dropout=0.3};
The methodology used were as follows. Across experiment cases, I needed a way to:
Simulate the model complexity of deep networksSimulate different input featuresUse a standard evaluation metric to measure prediction outcomes
Simulate the model complexity of deep networks
Simulate different input features
Use a standard evaluation metric to measure prediction outcomes
Simulate Model Complexity
To systematically add model complexity, I added an encoder stack to the model architecture while keeping all other hyper-parameters constant for every experiment.
For example, a single experiment set for tweeter glove 25d would have 4 distinct architectures.
Notice how the number of encoder stacks in each experiment increases systematically.
The hyper-parameters used were also kept constant throughout:
Optimizer hyper-parameters:miniBatchSize=32stagnation = 10algorithm = {method = ‘adam’, gamma = 0.9, beta1 = 0.9, beta2 = 0.999, learningRate = 0.001, clipGradMax = 100, clipGradMin = -100, lrPolicy = ‘step’, stepSize=10}regL2 = 0.0007maxEpochs = 10dropout=0.3
Simulate Input Features
To test if feature-rich input data has any effect on outcomes, I ran the model through tweeter glove 25d, 50d, 100d and 200d.
This would mean for each glove dimension, I would have ran 4 sets of experiments and logged their results respectively.
Baseline Model
Experiment 1: Baseline + 1 additional Encoder Stack
Experiment 2: Baseline + 2 additional Encoder Stacks
Experiment 3: Baseline + 3 additional Encoder Stacks
Therefore, there were 16 (4 architectures * 4 different glove dimension sets) different model configurations in total that were ran.
Standardized Evaluation Metric
Lastly, I decided to use F1-Score as the standardized metric of comparison as it is the harmonic mean between Precision and Recall.
If you need a refresher on Precision, Recall & F1-Score. I found a good video on it.
For this section, I will first share the experiment results within each tweeter glove dimensions. Then, I will share the results of each experiment across tweeter glove dimensions.
From here on out, I will display a series of result outputs as images and will add comments as I deem fit.
For every result output, take note of the Precision, Recall, F1 scores and how the loss curves look like.
Remember, the only change across experiments is an additional encoder stack.
Do take some time to interpret the results yourself as you go along. :)
For this result set, it is a little difficult to see the effects of a deeper network across the experiments.
I do have an idea and sensing on what is occurring but will hold off my comments till the pattern is clearer.
Note: It gets clearer in the next result set :)
Having said that, there is an interesting pattern in the F1-Scores of the positive tag (P).
As the model gets more complex, the F1-Scores for the Positive Tag (P) seem to be slowly increasing at every experiment.
I suspect it has to do with the deeper network learning more complex relationships.
For example, co-references, long-term dependencies and the like.
Now let’s increase the number of features from 25D to 50D to get a better picture on what is happening as the model gets deeper.
This is definitely a better result set to interpret compared to the previous one.
First off, notice how the validation loss starts to diverge from the training loss curve as the model gets deeper?
Why do you think this is the case?
Similar to the previous result set (albeit very subtle), this has something to do with the bias-variance trade off in machine learning.
At gist, as model complexity increases, the model starts to learn better on the training data causing bias to decrease.
This is evident from the results by the training loss (blue line) decreasing ever so slightly across experiments.
Note: Training bigger networks is a standard mitigation effort to lower bias. This results clearly shows how bigger networks reduces bias.
However...
As the model begins to overfit the training data, it becomes less generalizable causing the variances to increase.
Note the behavior of the validation loss curves diverging across experiments.
To get a more concrete example on the trade off, just take a look at Experiments 2 and 3.
Note how the training loss falls from 0.1618 to 0.1595 while the validation error starts to spike. i.e. bias decreasing, variance increasing
Now on to the next finding!
The next finding stems from the general F1-Scores across the experiments.
Compared to the previous result set, all the F1-Scores are better.
F1-Scores for Positive Tag (P) averages about ~32% while Negative Tag (N) averages ~27%.
Contrasting this to the previous results, the maximum F1-Scores for Positive Tag (P) was ~28% and Negative Tag (N) was only ~18%.
Remember that the only change here was the a richer input set was used. i.e. Tweeter Glove 25D swapped to 50D.
Everything else remained constant.
This is telling of the relationship between the number of input features and prediction outcomes.
In this case, using a richer set of features improved the model’s overall performance.
So what would we see by increasing features from 50D to 100D?
This result set actually validates our previous point on the relationship between input features and prediction outcomes.
But.. not in the way you’d expect.
In this case, the model is sightly better at predicting the Positive Tag (P); F1-Scores across experiments are higher than previous results.
However, it looks like the model is not doing so well at predicting the Negative Tag (N); F1-Scores across experiments dropped compared to previous results.
It is possible that the added features, 50D to 100D, might be allowing the model to learn Positive Tags (P) better but have added noise when learning Negative Tags (N) instead.
This would be one of the reasons why I think we see a fall in F1-Scores across experiments compared to the previous result.
The other reason (which might be more plausible) is that the F1-Scores are fluctuating due to the model losing it’s generalizability.
Notice how erratic the validation loss curves are.
To get a better result, we would probably need to tune other hyper-parameters before looking at the F1-Scores again.
However, the purpose of this experiment was to only change the model architecture by adding more layers and note the downstream changes thus I will not be doing any hyper-parameter tuning here.
The next finding is pretty obvious.
Hint: It’s got to do with the validation loss curves.
Notice how the validation loss is typically below the training loss across experiments?
This is an uncommon pattern but it does happen.
On this point, the next logical question is...
Why?
Why is the validation loss below training loss?
Couple of reasons:
It could have something to do with how validation loss is calculated during the training process. For each epoch, validation loss is calculated after training loss. i.e. After the model has updated the weights, the validation loss is calculated.I used dropout layers in the training set. Dropout layers have a regularization effect. This means that it penalizes the weights so as to allow predictions to be more generalizable. During the validation calculation, dropout is disabled.
It could have something to do with how validation loss is calculated during the training process. For each epoch, validation loss is calculated after training loss. i.e. After the model has updated the weights, the validation loss is calculated.
I used dropout layers in the training set. Dropout layers have a regularization effect. This means that it penalizes the weights so as to allow predictions to be more generalizable. During the validation calculation, dropout is disabled.
Now that we are done with 100D, let’s see what happens with 200D!
Food for thought.
Can having too many features be a problem?
Let’s find out shall we?
Apparently it can!
Looks like having 200D caused the models to overfit tremendously across the experiments.
Let’s take a step back and think for a moment.
What happened here?
The only change was that a more feature-rich input set was used. i.e. 100D to 200D features.
Shouldn’t having more features give us better results?
Not exactly.
Having an extended set of features might have inevitably introduced more noise to the model.
This is evident by the over-fitting we are seeing.
This is what I think is happening.
Every input feature can be mapped to other concepts like “man”, “woman”, “crown”, “royal” etc.
The learned embeddings for each feature (those numbers) reflects whether the input term meets the criteria of those features or not. This is usually in a range of [-1,1].
For example, for the input term “King”.
Is a king a man?
Yes. So the learned embeddings has a score of 0.98 for the feature “man”.
Is a King a woman?
No. So the learned embeddings has a score of 0.01 for the feature “woman”.
So on and so forth.
I think at 200D, there were too many features causing too much noise.
For example, features like “red” or “green” might be just noise to the model.
So what can we take away from this?
Adding more input features does not guarantee better models or better results. Sometimes, having more features mean your model starts to learn the noise instead.
I’ve also noticed that across all 16 experiments, the model always does poorly on the Negative Tags (N).
Take a look at the F1-Scores across 25D, 50D, 100D and 200D.
The F1-Scores for the Negative Tags (N) is always below that of the Positive Tags (P).
Why would that be so?
I reckon the Positive Tag (P) training examples might be easier to learn compared to the Negative Tag (N) training examples.
Perhaps learning negation or certain negative phrases is proving hard for the model. This would explain why (N) does not seem to be getting much performance boost.
Alright! We have come to the end of the results within each input dimension.
For the final segment of the post, I want to give you a glimpse of how the results look across experiments.
This will give you a different view of the results.
Just take note on how the curves changes as input features increases.
Remember that everything else is constant. The only change here are the input features.
I will let you view the results on your own without any commentary.
Before I conclude this post, I thought I’d summarize some takeaways from the experiment.
The bias-variance trade off very much still exists in deep learning. It can be mitigated if you actively reduce the bias and variance by applying mitigation efforts as mentioned in the linked post.Deeper networks reduce bias in general but if everything else is held constant, variances increases. It would be best to iterate through and apply regularization/add more training data the deeper you build your network.Sometimes the validation loss can appear below training loss. Don’t be taken aback by that result.Adding more input features does not guarantee a better model or better results. 200D introduced more noise into the training set resulting in a worse off outcome compared to 100D results.
The bias-variance trade off very much still exists in deep learning. It can be mitigated if you actively reduce the bias and variance by applying mitigation efforts as mentioned in the linked post.
Deeper networks reduce bias in general but if everything else is held constant, variances increases. It would be best to iterate through and apply regularization/add more training data the deeper you build your network.
Sometimes the validation loss can appear below training loss. Don’t be taken aback by that result.
Adding more input features does not guarantee a better model or better results. 200D introduced more noise into the training set resulting in a worse off outcome compared to 100D results.
So after reviewing the results of this experiment...
Are Deeper Networks Better?
It depends.
There seems to be an optimal depth which ties back to the amount of training data you have got to begin with.
In the case of this tweeter dataset with 27,481 rows, it looks like the model architecture for Experiment 1 with 100D input features is the optimum.
For easy reference, Experiment 1’s architecture looks like this with 2 encoder stacks.
Well, that’s it then!
It definitely took a while to run all the experiments to write this post.
Hope you found this post insightful!
If it was, let me know so I’d know what types of posts to continue writing. :)
Till the next post...
Sayonara!
LinkedIn Profile: Timothy Tan