Is torch.save() limited to overwriting?

Ammar_ul_hassan: <p>In my particular scenario, I want torch.save() to append the model state rather than overwrite it whenever it is called.<br>
Assume I have a pre-trained cGAN model on 0 to 5 mnist dataset classes with classes as conditions (saved using torch.save() with the filename “trained.pt”).<br>
Now I want to fine-tune this pre-trained model so that it can learn the remaining 6 to 9 classes. I saved the model in the same “trained.pt” file but noticed that it was overwriting the model state, i.e. it can generate images with 6-9 digits but ignores the 0-5 classes which were learned by the pre-trained model.<br>
I tried to search but couldn’t come up with a solution so far. Is there a way so that I can append my finetuned model in the pre-trained model.</p>

ptrblck: <p>I don’t completely understand what</p>
<aside class="quote no-group" data-username="Ammar_ul_hassan" data-post="1" data-topic="133675">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ammar_ul_hassan/48/42816_2.png" class="avatar"> Ammar_ul_hassan:</div>
<blockquote>
<p>append my finetuned model in the pre-trained model</p>
</blockquote>
</aside>
<p>means.<br>
Assuming you are storing the <code>state_dict</code> of the model, it would contain all trained parameters and buffers on the current use case. If you then fine-tune the model on another use case (e.g. numbers between 6 to 9) your model might “forget” the previously learned use case and the parameters might be updated to learn the currently trained use case. You wouldn’t be able to “append” this new model to the old one as it’s basically still the same model with another parameter set, so could you explain your use case a bit more, please?</p>

Ammar_ul_hassan: <p>Thanks for responding.<br>
So basically my use case is to generate 0-9 mnist numbers using cGAN. However, I want to learn these 0-9 numbers (classes and corresponding images) in two steps.<br>
Step1: The model learns 0 to 5 classes first.<br>
Step2: The model learns the remaining classes i.e. 6 to 9 however WITHOUT forgetting step 1 learned classes i.e. after step 2 my model can generate images of all classes 0-9.</p>
<p>I have recently moved from tensorflow to pytorch. In TF I can easily do this without any extra code.<br>
From the below thread I can see that in Pytorch I cannot do this because torch.save() overwrites and doesn’t provide any append functionality.</p>
<aside class="onebox githubissue" data-onebox-src="https://github.com/pytorch/pytorch/issues/16737">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/issues/16737" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/pytorch</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewbox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/16737" target="_blank" rel="noopener nofollow ugc">torch.save overwrite</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2019-02-04" data-time="23:14:36" data-timezone="UTC">11:14PM - 04 Feb 19 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/shayneobrien" target="_blank" rel="noopener nofollow ugc">
          <img alt="shayneobrien" src="https://avatars.githubusercontent.com/u/19914514?v=4" class="onebox-avatar-inline" width="20" height="20">
          shayneobrien
        </a>
      </div>
    </div>

    <div class="labels">
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          high priority
        </span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          triaged
        </span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          enhancement
        </span>
    </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">## 🚀 Feature

Add an overwrite feature to torch.save (default: True) or throw <span class="show-more-container"><a href="" rel="noopener" class="show-more">…</a></span><span class="excerpt hidden">a warning that a file will not overwrite an already existing file.

## Motivation
The current [workaround](https://discuss.pytorch.org/t/how-to-save-a-model-from-a-previous-epoch/20252/4) is to save a new model at each epoch. This is helpful in many applications, but saving a single model file when you know overfitting will not be an issue (e.g. when reimplementing a paper) would save a lot of memory when training large models as well as remove the need for a workaround using os / shutil functions. The default in all other applications I have seen is to overwrite the file.

cc @ezyang @gchanan @zou3519</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


ptrblck: <p>Could you describe what TensorFlow is “appending”?<br>
Storing two sets of parameters wouldn’t make sense unless you want to use two different models, so I’m unsure how this can work.</p>

Ammar_ul_hassan: <p>TF doesn’t append specifically I mean its the default behavior of TF. I mean in TF I can do these two steps and after step 2 my model remembers 0-9 classes however in Pytorch it forgets the Step1 classes.</p>

ptrblck: <p>That would depend on the training and is should be unrelated to <code>torch.save</code>.<br>
I would recommend to check the training itself and see why your model forgets the previous cases.<br>
Generally I would expect a pre-trained model (e.g. a ResNet trained on ImageNet) to forget the 1000 classes after I sufficiently trained it on my new use case (e.g. a 5 class guitar classifier).</p>
<p>Based on your use case I assume your model has 10 output neurons, while you are ignoring a subset in both steps?</p>

Ammar_ul_hassan: <p>Exactly, my model has 10 output neurons, and I am ignoring half of them in these two steps. A little explanation is below.</p>
<p>Assume that prior to training the model, there will be a total of ten classes (the size of the target one-hot vector is 10). However, in step 1, I show only the first five classes (the one-hot vector’s last 5 values remain 0, and the first five change depending on the input class) and train and save the model.</p>
<p>In step 2, I reload the model (which has already learned the first five classes) and learn the remaining classes without learning the previously learned classes (the one-hot vector first 5 values always remain 0, and the last five change depending on the input class), retraining and saving it.</p>
<p>Then, I expect the model to know all ten classes at this point, but in my experiment, it forgets the step 1 learned classes.</p>

ptrblck: <p>I think the expectation wouldn’t hold, since the model could learn to extract features only for the current task.<br>
While the weights for the 5 previously classes in the classification layer wouldn’t be updated, the feature extractor would.<br>
I would assume the model to forget the previously learned classes especially if the features between the tasks differ a lot.<br>
E.g. an artificial examples would be to try to learn to classify 5 bird classes first and 5 cancer classes based on CT scans later. The model could try to extract a completely different feature set for these images and I would expect it to decrease its performance on the bird dataset.<br>
Of course, your use case is different, i.e. your samples come from the same dataset, but I would still think the model could focus on different feature sets.<br>
Do you have any resources which have evaluated the ability to “remember a task” of DL models?</p>

Bridget_Murphy: <p>Thank you for sharing good luck on your project.</p>