NVIDIA Model Overfitting to training data

bibekx: <p>I want a CNN designed by NVIDIA to work on a regression task of “Driving image to steering angle prediction”. Although the model (image 1) was designed for a slightly different task (with driving images and other signals), I was planning to try this first and modify along for my purposes.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/c/5c6bad40226d1d8bb3310c134d85c23f07535b5f.png" data-download-href="https://discuss.pytorch.org/uploads/default/5c6bad40226d1d8bb3310c134d85c23f07535b5f" title="Screen Shot 2021-08-04 at 8.09.43 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/c/5c6bad40226d1d8bb3310c134d85c23f07535b5f_2_354x500.png" alt="Screen Shot 2021-08-04 at 8.09.43 PM" data-base62-sha1="dbACFpiHKCdPsuVFnIlBp6TWr9B" width="354" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/c/5c6bad40226d1d8bb3310c134d85c23f07535b5f_2_354x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/c/5c6bad40226d1d8bb3310c134d85c23f07535b5f_2_531x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/c/5c6bad40226d1d8bb3310c134d85c23f07535b5f_2_708x1000.png 2x" data-dominant-color="D5D5D5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2021-08-04 at 8.09.43 PM</span><span class="informations">870×1228 61.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Hyperparamters I have been using are:</p>
<ol>
<li>Learning rate: 0.0001</li>
<li>Train and Val batch sizes: 64, 64</li>
<li>Optimizer: Adam</li>
<li>Activations: ELU</li>
</ol>
<p>I have studied code from various people who implemented this model to try a bunch of different strategies:</p>
<ol>
<li>Normalize pixels in range (0,1), in the range (-1,1) and z-score normalization (separately)</li>
<li>One Dropout 0.2 between CONV 2 and 3 layers (also tried 0.4)</li>
<li>One Dropout of 0.2 between CONV and Linear layers (also tried 0.4, 0.5)</li>
<li>Batch normalization after each convolutional layer</li>
<li>Reduction of model complexity (removal of CONV and Linear layers)</li>
<li>Different values of LR, Batch</li>
<li>Different variations of filter sizes, stride and linear layers</li>
</ol>
<p>However, despite trying about many many variations of the model for the entire day today, I am still not able to get validation loss to decrease (see figure 2)<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/3/13ec906164c84d3855ed0afb3208ec47f5be13c0.png" data-download-href="https://discuss.pytorch.org/uploads/default/13ec906164c84d3855ed0afb3208ec47f5be13c0" title="qi3DCDZP"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/3/13ec906164c84d3855ed0afb3208ec47f5be13c0_2_690x215.png" alt="qi3DCDZP" data-base62-sha1="2QfUi7PgnrBloc8guUe4sZgxSY8" width="690" height="215" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/3/13ec906164c84d3855ed0afb3208ec47f5be13c0_2_690x215.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/3/13ec906164c84d3855ed0afb3208ec47f5be13c0_2_1035x322.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/1/3/13ec906164c84d3855ed0afb3208ec47f5be13c0_2_1380x430.png 2x" data-dominant-color="FDFCFC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">qi3DCDZP</span><span class="informations">1600×500 34 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Some more strategies that I have in mind but am not sure about their effectiveness are:</p>
<ol>
<li>Implementing a decaying learning rate scheme</li>
</ol>
<p>Any help would be greatly appreciated.<br>
Thanks</p>

eqy: <p>Although the model architecture looks relatively nonstandard (pure convolution, extensive use of 5x5 convolutions, no residual layers), the overfitting might be attributable to the dataset. What is the extent of the training and validation set here, and is the data distribution comparable? For example, are the road conditions similar, orientation of camera comparable, etc.?</p>
<p>If collecting more data is not an option, there are various data augmentations that might be applicable in this setting:<br>
<a href="https://pytorch.org/vision/stable/transforms.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/vision/stable/transforms.html</a></p>

bibekx: <p>Thank you for your reply. Sorry, I forgot to mention about the dataset.</p>
<p>Its from Honda and Waymo (on real road images not in simulation), each image has size (220, 66, 3)<br>
There are about 2,50,000 images in training set and about 43,000 in validation.</p>
<p>And yes the data distribution is comparable. Similar angles and road conditions. Although inherently driving data is a little bit non identically distributed.</p>
<p>Yes, Data Augmentation with:</p>
<ol>
<li>Random selection of various lighting, brightness, contrast, saturation and hue conditions</li>
<li>Spatial shift, rotation, flip and perspective</li>
</ol>
<p>are in my considerations for the future. However, I was under the impression that I should be able to get at least some generalization with the current version of the model and pipeline as well. And that Augmentation would help improve that performance.</p>
<p>Please feel free to add your thoughts and comments on this.<br>
I appreciate you taking the time to answer.</p>

eqy: <p>To check if it is indeed a distribution or issue of problem difficulty, I would try to first debug the system by replacing the validation set with the training set and seeing if the results match the training set.<br>
If this works I would try splitting the training set in half and seeing if training on the first half of the training set generalizes well to the second.</p>

bibekx: <p>Thank you for your insight,<br>
The result of both Train and Val sets= Training data<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/a/ba35cafa88bd643953caf7512980de896f40db5e.png" data-download-href="https://discuss.pytorch.org/uploads/default/ba35cafa88bd643953caf7512980de896f40db5e" title="Dc6Ybo1w"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/a/ba35cafa88bd643953caf7512980de896f40db5e_2_517x161.png" alt="Dc6Ybo1w" data-base62-sha1="qzi3JwdJif2O2W582rnFmrZjmG2" width="517" height="161" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/a/ba35cafa88bd643953caf7512980de896f40db5e_2_517x161.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/a/ba35cafa88bd643953caf7512980de896f40db5e_2_775x241.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/a/ba35cafa88bd643953caf7512980de896f40db5e_2_1034x322.png 2x" data-dominant-color="FDFCFC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Dc6Ybo1w</span><span class="informations">1600×500 33.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Next, the result of Train = first half of Training data, Val = second half of Training data<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/1/c15b1962ddf11d367634bde5e30a667d05c2de06.png" data-download-href="https://discuss.pytorch.org/uploads/default/c15b1962ddf11d367634bde5e30a667d05c2de06" title="rEnlSFDB"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/1/c15b1962ddf11d367634bde5e30a667d05c2de06_2_517x161.png" alt="rEnlSFDB" data-base62-sha1="rAvkvOwS9MjCyBLgyBcEYZOjGHc" width="517" height="161" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/1/c15b1962ddf11d367634bde5e30a667d05c2de06_2_517x161.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/1/c15b1962ddf11d367634bde5e30a667d05c2de06_2_775x241.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/c/1/c15b1962ddf11d367634bde5e30a667d05c2de06_2_1034x322.png 2x" data-dominant-color="FDFCFC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">rEnlSFDB</span><span class="informations">1600×500 32.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Note the validation error is very high.<br>
Now would it be correct for me to say, “The problem we face is not due to distribution shift between Training and Validation sets but rather the problem difficulty of learning with generalization”. Because, even when the data was from the same distribution it did not generalize.</p>
<p>Next I am going to try data augmentation (which I believe has been proven to improve generalization as well as to better performance in training). Are there any specific strategies you know that I should follow?</p>

eqy: <p>I don’t think there is any silver bullet for what types of augmentation will work best e.g., <a href="https://arxiv.org/abs/1909.13719v2" rel="noopener nofollow ugc">[1909.13719v2] RandAugment: Practical automated data augmentation with a reduced search space (arxiv.org)</a></p>
<p>However, there may still be parts of the model architecture that make it more prone to overfitting. For example, a flatten operation rather than average pooling (or some other kind of layer that removes spatial dependencies) before the fully connected layers may cause the model to be more dependent on non-generalizable features. You might want to see if the results improve with a modified ResNet or other popular vision architecture.</p>

bibekx: <p>Thank you for the pointers to the paper.</p>
<p>I did a data augmentation where for every image there is a transformed image with one of the 13 transforms from this list: [Random Color Jitter (Hue, Saturation, Contrast, Brightness), Horizontal Flip, Vertical Flip, Histogram Equalization, Auto Contrast, Adjust Shaprness, Solarize, Posterize , Invert, Affine Transformation, Random rotation , Random Perspective shift, Gaussian Blur]</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/c/7ce3a7ef58a660040937ff18579dbad99abdaa82.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/7ce3a7ef58a660040937ff18579dbad99abdaa82" title="v-5UN0Xf"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/c/7ce3a7ef58a660040937ff18579dbad99abdaa82_2_517x294.jpeg" alt="v-5UN0Xf" data-base62-sha1="hOOXkVzsEtS6zkkhvJxlIIoxYNY" width="517" height="294" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/c/7ce3a7ef58a660040937ff18579dbad99abdaa82_2_517x294.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/7/c/7ce3a7ef58a660040937ff18579dbad99abdaa82_2_775x441.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/7/c/7ce3a7ef58a660040937ff18579dbad99abdaa82_2_1034x588.jpeg 2x" data-dominant-color="9FA1A5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">v-5UN0Xf</span><span class="informations">1589×907 324 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>However, the training still does not improve.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/4/7496e5b56ae480a3ff944d3bac736640dcf131f6.png" data-download-href="https://discuss.pytorch.org/uploads/default/7496e5b56ae480a3ff944d3bac736640dcf131f6" title="wJHqH-jD"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/4/7496e5b56ae480a3ff944d3bac736640dcf131f6_2_517x161.png" alt="wJHqH-jD" data-base62-sha1="gDoGjHCERkoctwYe8EgPpBNRZD8" width="517" height="161" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/4/7496e5b56ae480a3ff944d3bac736640dcf131f6_2_517x161.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/4/7496e5b56ae480a3ff944d3bac736640dcf131f6_2_775x241.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/7/4/7496e5b56ae480a3ff944d3bac736640dcf131f6_2_1034x322.png 2x" data-dominant-color="FCFCFC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">wJHqH-jD</span><span class="informations">1600×500 37.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Next, I coded up the ResNet 50 architecture but seems like it will take a while to train (23 million parameters and train data size = 4,30,000). Will update after overnight of running on ResNet.</p>