Poor model performance

pothak: <p>Hello all,</p>
<p>I have been developing a model and test dataset to segment and classify trees from drone imagery. The performance (OA, Recall, etc.) has been exceptionally poor for the new model I have been attempting to use. I have been troubleshooting various things that I believe may be wrong however I am scratching my head as to how my OA is so poor at 4-5%, the previous model (similar architecture in Tensorflow) seemed to perform better with some classes reaching OA of 80-90%. While the dataset may not be one hundred percent accurate or voluminous I imagine the cause for such poor performance comes from some steps not performing correctly.</p>
<p>Some background:</p>
<ul>
<li>The dataset it high-resolution RGB drone imagery of a forest, with two additional layers of a texture index and canopy height model (So five layers in total).</li>
<li>There are nine labeled classes being [Elm, Oak, Willow, Locust, Ash, Hackberry, Hickory, Elevation under 10m, and Canopy gaps] where the elements start at 1 (Elm) in the imagery labels</li>
<li>I am using a U-Net architecture in PyTorch and performing this in a multi-class manner for all nine classes</li>
<li>I have not included code snippets within this post as the scripts are fairly long, but have added them via a <a href="https://drive.google.com/drive/folders/1_HqSGXFrcfzWJnHEMaoNrgZe_w1EYecF?usp=sharing" rel="noopener nofollow ugc">Google Drive</a> link with some examples of both imagery and labels.</li>
</ul>
<p>When training my validation loss and training loss follow each other for only a few epochs (&lt; 10) before they start diverging. The training loss plateaus very quickly (&lt; 20 epochs) and even the accuracy on the training class per epoch never seems to increase. While I don’t get any outright error  the below are some things I have been looking at:</p>
<ul>
<li>The indexing or the image and label get mixed up. This seemed to be the case with one of the ways I created the dataset however I have reverted to one I know is correct (made manually) and checked the outputs of the train and val data-loader and they appear to be coming into the network correctly. I have tried several different ways of sorting lists before the data-loader yet don’t see any difference.</li>
<li>Introduced image distortion through some processing. Again, I have looked at the data throughout different points of the model (correcting issues when found) yet it seems to be fine and I am able to visually display both data and labels correctly with no distortion</li>
<li>I have gone through and checked data types and dimensionality throughout multiple parts of the model and it does not seem to get distorted.</li>
<li>I have tried different learning rates, loss functions, and schedules yet they don’t seem to make an impressionable difference in the outcome</li>
<li>The dataset is inherently too variable to train on, this could potentially be true however I have had limited success in the past with this same dataset and imagine classification accuracies should be at least better than randomly assigning a pixel a class…</li>
</ul>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/b/1b58e725bcfd5471d9403f4e8e750db3cf1e1f64.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/1b58e725bcfd5471d9403f4e8e750db3cf1e1f64" title="sdfasdfasd.PNG"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/b/1b58e725bcfd5471d9403f4e8e750db3cf1e1f64_2_587x500.jpeg" alt="sdfasdfasd.PNG" data-base62-sha1="3TVmEk6rhWwwv2jiAKQt0rQarsw" width="587" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/b/1b58e725bcfd5471d9403f4e8e750db3cf1e1f64_2_587x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/1/b/1b58e725bcfd5471d9403f4e8e750db3cf1e1f64_2_880x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/1/b/1b58e725bcfd5471d9403f4e8e750db3cf1e1f64.jpeg 2x" data-dominant-color="4B522C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">sdfasdfasd.PNG</span><span class="informations">901×767 219 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>This is currently where I am at, in the past, I have had problems with overfitting my train data yet I don’t seem to be able to “learn” at all. I am currently training via a binary approach for each tree genus to see if that is any better and will update. Some things regarding the scripts. One only needs to run the model_load.py script to initiate the training. I am admittedly quite novice at ML and programming and don’t have any people I can talk with to help me work through these issues so I appreciate the help!</p>