How to do transfer learning using CPU with a model trained using DataParallel?

username: yx0123
cooked: <p>I found a pretrained model online that was trained using DataParallel and I want to retrain the final layer using my own dataset. I load the model and modify the last layer as follows:</p>
<pre><code class="lang-auto">model_ft = torch.load(base_model, map_location={'cuda:0': 'cpu'})
for param in model_ft.module.parameters():
    param.requires_grad = False
num_ftrs = model_ft.module.classifier.in_channels
features = list(model_ft.module.classifier.children())[:-1] # remove last later
features.extend([nn.Conv2d(num_ftrs, n_classes*n_atom_outputs, kernel_size=1, bias=True)])
model_ft.module.classifier = nn.Sequential(*features)

</code></pre>
<p>When I run the code, I get the following error during forward propagation: Traceback (most recent call last):<br>
File “code/train.py”, line 157, in train_model<br>
outputs = model(inputs)<br>
File “/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py”, line 325, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
…<br>
…<br>
…<br>
File “/usr/local/lib/python3.5/dist-packages/torch/nn/parallel/_functions.py”, line 94, in _get_stream<br>
if _streams[device] is None: IndexError: list index out of range</p>
<p>I believe this is because the model has been wrapped with DataParallel but I do not have a GPU. May I know how can I remove DataParallel from a pretrained model? Or is there a better way to resolve the error? Thank you!</p>

username: user_123454321
cooked: <p>Can you try this…</p>
<pre><code class="lang-auto">model_dict = torch.load(model_path, map_location='cpu')
for (key, val) in list(model_dict.items()):
  value = val.clone()
  del model_dict[key]
  model_dict[key.replace('module.', '')] = value
model.load_state_dict(model_dict)
</code></pre>

username: yx0123
cooked: <p>I tried and got the following error: File “/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py”, line 366, in <strong>getattr</strong><br>
type(self).<strong>name</strong>, name))<br>
AttributeError: ‘DataParallel’ object has no attribute ‘items’</p>
<p>I also tried using</p>
<pre><code class="lang-auto">for (key, val) in list(model_dict.module.items()):
</code></pre>
<p>and got this error: File “/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py”, line 366, in <strong>getattr</strong><br>
type(self).<strong>name</strong>, name))<br>
AttributeError: ‘DPN’ object has no attribute ‘items’</p>

username: user_123454321
cooked: <p>Ah…so the model is not saved as dictionary, but the object itself is saved ? Try this…</p>
<pre><code class="lang-auto">model = torch.load(model_path, map_location='cpu')
model = model.module
</code></pre>

username: yx0123
cooked: <p>I have already manually replaced model_ft with model_ft.module in my code. I am able to access the model attributes but the IndexError is actually during the forward propagation step.</p>

username: user_123454321
cooked: <p>Oh ok…<br>
It works for a small module I tried…<br>
Can you send me the model file ?<br>
If not, you can try opening in google colab using GPU and then save it without the data parallel…I know, not very elegant…</p>

username: yx0123
cooked: <p>Hi, you can download the model file by going to this link <a href="https://s3.amazonaws.com/hello-tc/spacenet3-code/trained_models/model01.pth" rel="nofollow noopener">https://s3.amazonaws.com/hello-tc/spacenet3-code/trained_models/model01.pth</a></p>
<p>Could you send me your code if it works with this model?  Really appreciate your help! <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>
<p>This is the source code that was used to train the original models <a href="https://github.com/SpaceNetChallenge/RoadDetector/blob/master/pfr-solution/code/train.py" rel="nofollow noopener">https://github.com/SpaceNetChallenge/RoadDetector/blob/master/pfr-solution/code/train.py</a><br>
I’m using the same code but edited it to load the pretrained model instead of the DPN model, and froze the first few layers of the pretrained model.</p>

username: yx0123
cooked: <p>I tried this and it works! I guess when I replaced model with model.module manually I missed out some of them, thanks!</p>