Expand model training

username: paraujo
cooked: <p>Hi everyone, I am a beginner in DL and PyTorch, and I would like to ask for help in a specific situation. I built a model and trained it for 200 epochs using a specific dataset. Now, I have a different dataset that supplements the previous one. Should I load the pretrained weights and keep training in the new dataset? Should I retrain the model merging both datasets? Should I train two different models and ensemble them? What is the easiest strategy? Thank you!</p>

username: KFrank
cooked: <p>Hi Paulo!</p>
<aside class="quote no-group" data-username="paraujo" data-post="1" data-topic="118847" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/paraujo/48/37348_2.png" class="avatar"> paraujo:</div>
<blockquote>
<p>Now, I have a different dataset that supplements the previous one. Should I load the pretrained weights and keep training in the new dataset?</p>
</blockquote>
</aside>
<p>If your new data is basically the same as your original dataset – if the<br>
data is basically just more of the same but you didn’t have it when you<br>
first started training – you should keep the pre-trained weights and then<br>
continue training with the combined dataset.</p>
<p>If your new dataset if for a similar, but distinct problem, and you want<br>
to build a model for the new problem, keep the pre-trained weights to<br>
get a head start, but only train with the new dataset.</p>
<p>(I can’t think of a non-contrived version of this use case where I would<br>
train two separate models and combine them.)</p>
<p>Best.</p>
<p>K. Frank</p>

username: paraujo
cooked: <p>Hi <a class="mention" href="/u/kfrank">@KFrank</a>,</p>
<p>My second dataset was developed to a similar problem. However, its normalization parameters are a bit different due to the quality of the cameras and the weather. My expectation is to enhance the performance of the model in the first dataset, but supplementing it with the second dataset. I am not sure if this is possible or right. I did a few tests keeping the weights as you suggested, and the performance with the first dataset got worse. So, I thought of merging both datasets to train all together.</p>

username: KFrank
cooked: <p>Hi Paulo!</p>
<aside class="quote no-group quote-modified" data-username="paraujo" data-post="3" data-topic="118847" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/paraujo/48/37348_2.png" class="avatar"> paraujo:</div>
<blockquote>
<p>Hi <a class="mention" href="/u/kfrank">@KFrank</a>,<br>
its normalization parameters are a bit different due to the quality of the cameras and the weather.</p>
</blockquote>
</aside>
<p>If the difference is primarily one of normalization, you might be able<br>
to normalize one or the other of the two datasets to make them match<br>
better.  But see below.</p>
<aside class="quote no-group">
<blockquote>
<p>My expectation is to enhance the performance of the model in the first dataset, but supplementing it with the second dataset.</p>
</blockquote>
</aside>
<p>The important question is what does the real-world data to which you’ll<br>
be applying your model for inference look like?</p>
<p>You have two (potentially competing) goals:  You want to have the data<br>
you train on be as representative as possible of the data on which you<br>
will be running inference.  But you also want to have enough training<br>
data to be able to do a good job of training your model.</p>
<p>If you have more than enough training data, stick to the data that most<br>
closely matches your inference data.  But if you’re short on training data<br>
it can be helpful to add more data, even if it’s not a perfect match for<br>
your real-world data.</p>
<p>Which dataset best matches the normalization of your real-world data?<br>
If both datasets otherwise match reasonably well, I would adjust the<br>
normalization of the dataset whose normalization matches less well.</p>
<p>As a final comment, let’s say that your first dataset is really rather<br>
different than your real-world data (and your second dataset matches<br>
the real-world data reasonably well), but you spent a lot of time training<br>
with it.  Even if it would not make sense to use a model trained on the<br>
first dataset on your real-world data, you can still use it as a pre-trained<br>
model that you “fine tune” by further training it on the second dataset,<br>
potentially saving significant time in comparison to training from scratch<br>
on the second dataset.</p>
<aside class="quote no-group">
<blockquote>
<p>I did a few tests keeping the weights as you suggested, and the performance with the first dataset got worse.</p>
</blockquote>
</aside>
<p>Remember, your goal is not narrowly to have the model perform<br>
well on either the first or second dataset, but to have it perform well<br>
on your real-world data.  So you want it to perform well on whatever<br>
test / training data best represents your real-world data.</p>
<p>Best.</p>
<p>K. Frank</p>

username: paraujo
cooked: <p>Hi <a class="mention" href="/u/kfrank">@KFrank</a> , thanks for the explanation. If I correctly understand, if I train a model using dataset 1 and test it in dataset 2, which is a similar dataset, the model should generalize to a new dataset, correct?</p>
<p>When I test the model in the second dataset, the results are quite wrong. Is it an indication that the dataset is wrong or the model lacks training?</p>
<p>Another question I have, I am using <code>transforms.Normalize</code> to normalize the first dataset during training. Do I need to calculate the mean and std of the second dataset to retrain the model?</p>

username: cpeters
cooked: <p>If I understand it correctly, you have two datasets taken in different conditions, and you want your model to apply well to both of them?</p>
<p>If you believe that the model should apply well to both (the subjects are similar, for instance) but the conditions they were taken in are interfering, I can see two main options:</p>
<ol>
<li>If you can know in advance if any unseen data will come from a good source or bad source (picture metadata etc., image statistics), then you can preprocess it to resemble Dataset 1.</li>
<li>If you can’t predict this or run such preprocessing, then you can combine the two datasets and normalize them, then train on the combined dataset. This will, however, probably have worse accuracy than a model trained <em>only</em> on Dataset 1 or <em>only</em> on Dataset 2 when tested on those.</li>
</ol>
<aside class="quote no-group" data-username="paraujo" data-post="5" data-topic="118847">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/paraujo/48/37348_2.png" class="avatar"> paraujo:</div>
<blockquote>
<p>Hi <a class="mention" href="/u/kfrank">@KFrank</a> , thanks for the explanation. If I correctly understand, if I train a model using dataset 1 and test it in dataset 2, which is a similar dataset, the model should generalize to a new dataset, correct?</p>
<p>When I test the model in the second dataset, the results are quite wrong. Is it an indication that the dataset is wrong or the model lacks training?</p>
</blockquote>
</aside>
<p>If the second dataset is sufficiently similar to the first, then the hope is that a model trained on the first dataset will generalise. If the results are wrong, then it might mean that the model has <em>too much</em> training and has overfit your first dataset (it doesn’t generalise), it might mean the second dataset isn’t similar enough to the first (it would generalise to unseen data, but only if it’s actually similar), or it could mean that the second dataset hasn’t been correctly processed (the model would generalise and the dataset is similar, but the format is off).</p>
<p>You <em>could</em> combine the two datasets into one and train a single model on them, but at that point your test set has become part of your training and validation set and you need new test data–but if it represents a large category you want to generalise to with or without additional image preprocessing, it might be worth it rather than having a narrowly specialised model.</p>

username: paraujo
cooked: <p>HI <a class="mention" href="/u/cpeters">@cpeters</a> , interesting points! My datasets vary in image quality (brightness, colors, etc) and in scene elements (cars, people, etc). I am working on a model to estimate a robot’s pose. In the first dataset, it is known that it lacks image sequences when the robot is at high speeds. Thus, I am trying to create another dataset to supplement the first one. However, in a different scenario and weather conditions (image quality). So, I am not sure if this strategy works. Maybe, I can merge both datasets and test the trained model in a separated dataset, just as you suggested. Do you think it is a reasonable alternative?</p>

username: cpeters
cooked: <p>It certainly seems worth a try; the obvious problem being that you might have it learning the noise as a differentiating measure, so if you got poor-quality low-speed poses or high-quality fast poses it might have trouble learning them.</p>
<p>It sounds like you might want to try processing the images in both datasets first to equalise features like brightness etc.?</p>

username: paraujo
cooked: <blockquote>
<p>It sounds like you might want to try processing the images in both datasets first to equalize features like brightness etc.?</p>
</blockquote>
<p>I think it will take some time, and I am unsure if it will enhance the final results. Probably I will merge them and calculate the mean and std to use the <code>transforms.Normalize</code>.</p>