Please help me understand not improving *training* (not test) with more layers

rob-miller: <p>Please pardon my pre-deep-learning graduate training.  I am trying to understand whether I have a bug, or what the theoretical reason(s) is(are) for the behavior I am seeing (and suggestions for improvements).</p>
<p>I am exploring multiple models and expect to see overfitting as I add more layers, but instead the deep nets just seem to hit a wall:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/d/3dccec5cf4e07dd4cc7d56f3159f5ed29daba4fa.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/3dccec5cf4e07dd4cc7d56f3159f5ed29daba4fa" title="Screenshot 2022-10-17 at 08.44.06"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/d/3dccec5cf4e07dd4cc7d56f3159f5ed29daba4fa_2_690x447.jpeg" alt="Screenshot 2022-10-17 at 08.44.06" data-base62-sha1="8OIaEvzTGd4Fnvhl8I66ZodXDh8" width="690" height="447" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/d/3dccec5cf4e07dd4cc7d56f3159f5ed29daba4fa_2_690x447.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/3/d/3dccec5cf4e07dd4cc7d56f3159f5ed29daba4fa_2_1035x670.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/d/3dccec5cf4e07dd4cc7d56f3159f5ed29daba4fa_2_1380x894.jpeg 2x" data-dominant-color="F3F3F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2022-10-17 at 08.44.06</span><span class="informations">1920×1245 134 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The black trace is the simplest network, defned as:</p>
<pre><code class="lang-auto">        self.linear_relu_stack = nn.Sequential(
            nn.Linear(config["input_dim"], config["input_dim"]),
            nn.ReLU(),
            nn.Linear(config["input_dim"], config["input_dim"]),
            nn.ReLU(),
            nn.Linear(config["input_dim"], config["output_dim"]),
        )

</code></pre>
<p>Some specifics are:</p>
<pre><code class="lang-auto">1259837 training samples
139982 test samples
input dimension 9078
output dimension 76
</code></pre>
<p>Loss function is MSE, optimizer is SGD.<br>
Outputs are real values, not classes.</p>
<p>The model for the pink trace has one more linear + ReLU pair, while the blue and yellow traces have 11 linear+ReLU layers with learning rates of 0.001 and 0.0001, and trying different physical GPUs.</p>
<p>I’m not understanding why (1) the MSE looks higher overall apparently as a function of the number of layers, and (2) why the deeper nets hit a wall even on the training data.</p>
<p>Why can’t I overfit?  Is this to be expected?</p>
<p>Thank you.</p>

ptrblck: <p>The <a href="https://arxiv.org/abs/1512.03385">ResNet paper</a> discusses the training issues with deeper models and also shows the same effect in Figure 1 directly which shows the deeper model to have a higher training error while the opposite might have been the expectation due to the increase in model capacity.</p>