Unconverging loss for a simple regression task

username: Saleh_qaryan
cooked: <p>I am working on a project to predict soccer player values from a set of inputs. The data consists of about 19,000 rows and 8 columns (7 columns for input and 1 column for the target) all of numerical values. Here is a glimpse:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/8/b8f3ec834147afd9f11c5ab5a9f26be11d996583.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/b8f3ec834147afd9f11c5ab5a9f26be11d996583" title="head"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/8/b8f3ec834147afd9f11c5ab5a9f26be11d996583_2_690x180.jpeg" alt="head" data-base62-sha1="qoassCZdQM6egbCKOhuScdbl00r" width="690" height="180" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/8/b8f3ec834147afd9f11c5ab5a9f26be11d996583_2_690x180.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/b/8/b8f3ec834147afd9f11c5ab5a9f26be11d996583_2_1035x270.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/8/b8f3ec834147afd9f11c5ab5a9f26be11d996583.jpeg 2x" data-dominant-color="F6F5F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">head</span><span class="informations">1056×276 30 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I am using a fully connected Neural Network for the prediction but the problem is the loss is not decreasing as it should. This is the result I get:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/c/3c6daabdfaa25e149c7f4053c2f06d62c6ed4f37.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/3c6daabdfaa25e149c7f4053c2f06d62c6ed4f37" title="result"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/c/3c6daabdfaa25e149c7f4053c2f06d62c6ed4f37_2_257x500.jpeg" alt="result" data-base62-sha1="8CzBJukrStEJ2fH8GXpo6WwoI07" width="257" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/c/3c6daabdfaa25e149c7f4053c2f06d62c6ed4f37_2_257x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/3/c/3c6daabdfaa25e149c7f4053c2f06d62c6ed4f37_2_385x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/3/c/3c6daabdfaa25e149c7f4053c2f06d62c6ed4f37.jpeg 2x" data-dominant-color="E9EEF1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">result</span><span class="informations">428×831 62.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>This is how I am creating the dataset and the dataloaders:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/d/3d137db7676fd9fec7b56b800a2e46dc797d3ede.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/3d137db7676fd9fec7b56b800a2e46dc797d3ede" title="dataset"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/d/3d137db7676fd9fec7b56b800a2e46dc797d3ede_2_306x500.jpeg" alt="dataset" data-base62-sha1="8IiSSupyd558i0fla5v5jrnHivs" width="306" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/d/3d137db7676fd9fec7b56b800a2e46dc797d3ede_2_306x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/3/d/3d137db7676fd9fec7b56b800a2e46dc797d3ede_2_459x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/3/d/3d137db7676fd9fec7b56b800a2e46dc797d3ede.jpeg 2x" data-dominant-color="F2F2F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">dataset</span><span class="informations">566×924 57.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>This is the code I am using to run the model:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/8/a8211d289528e2732c89c40aa6e4db933d6a051f.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/a8211d289528e2732c89c40aa6e4db933d6a051f" title="gradient"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/8/a8211d289528e2732c89c40aa6e4db933d6a051f_2_690x386.jpeg" alt="gradient" data-base62-sha1="nZl9ZO152m8NV5XC0NV6qTZdnEb" width="690" height="386" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/8/a8211d289528e2732c89c40aa6e4db933d6a051f_2_690x386.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/a/8/a8211d289528e2732c89c40aa6e4db933d6a051f.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/8/a8211d289528e2732c89c40aa6e4db933d6a051f.jpeg 2x" data-dominant-color="EEEFEE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">gradient</span><span class="informations">964×540 74.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/b/0bebd622a005a3d6f04b6074b536d9bd150b2642.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/0bebd622a005a3d6f04b6074b536d9bd150b2642" title="model"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/b/0bebd622a005a3d6f04b6074b536d9bd150b2642_2_689x500.jpeg" alt="model" data-base62-sha1="1HswPiaeUs5y3EWmnOx4DEk8dX4" width="689" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/b/0bebd622a005a3d6f04b6074b536d9bd150b2642_2_689x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/0/b/0bebd622a005a3d6f04b6074b536d9bd150b2642_2_1033x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/b/0bebd622a005a3d6f04b6074b536d9bd150b2642.jpeg 2x" data-dominant-color="F2F3F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">model</span><span class="informations">1115×809 89.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I tried changing the learning rate but it is still bad.</p>
<p>What could be the reason behind this? Thank you!</p>

username: user_123454321
cooked: <p>Is <code>ValueEUR</code> the target variable? It seems to have values in <code>10e7</code> . Thus, the loss can be pretty big if it is used directly (without scaling).</p>

username: eqy
cooked: <p>The design of the network seems a bit odd, as usually in fully connected models there is a projection to a higher dimensional space (but here the maximum dimension is 7 throughout the model). You might want to try a simple sanity check which is to just use a very small subset of the dataset (e.g., two data points) and use it as both the train and test set to verify that you can easily overfit a small amount of data which would rule out trivial bugs in the training loop. After that you could try increasing the size of the hidden layers of the model or other common techniques like normalizing the dataset (though in theory linear layers with ReLU activations should be able to “learn around” unnormalized data).</p>