Imagenet example with inception v3

Nadav_Bhonker: <p>When I run the pytorch_examples/imagenet example with inception_v3 model I get:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/home/nadavb/pytorch_examples/examples/imagenet/main.py", line 287, in &lt;module&gt;
    main()
  File "/home/nadavb/pytorch_examples/examples/imagenet/main.py", line 130, in main
    train(train_loader, model, criterion, optimizer, epoch)
  File "/home/nadavb/pytorch_examples/examples/imagenet/main.py", line 166, in train
    output = model(input_var)
  File "/home/nadavb/anaconda2/lib/python2.7/site-packages/torch/nn/modules/module.py", line 206, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/nadavb/anaconda2/lib/python2.7/site-packages/torch/nn/parallel/data_parallel.py", line 61, in forward
    outputs = self.parallel_apply(replicas, inputs, kwargs)
  File "/home/nadavb/anaconda2/lib/python2.7/site-packages/torch/nn/parallel/data_parallel.py", line 71, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs)
  File "/home/nadavb/anaconda2/lib/python2.7/site-packages/torch/nn/parallel/parallel_apply.py", line 45, in parallel_apply
    raise output
RuntimeError: CHECK_ARG(input-&gt;nDimension == output-&gt;nDimension) failed at torch/csrc/cudnn/Conv.cpp:275
</code></pre>
<p>It looks like I’m missing something obvious which is related to the input size. I’m not sure what exactly…<br>
Is there a different pre-processing done for inception models?</p>

fmassa: <p>Inception requires the <a href="https://github.com/pytorch/vision/blob/master/torchvision/models/inception.py#L76">input size to be <code>299x299</code></a>, while all other networks requires it to be of size <code>224x224</code>. Also, if you are using the standard preprocessing of torchvision (mean / std), then you should look into passing the <a href="https://github.com/pytorch/vision/pull/101"><code>transform_input</code> argument</a></p>

achaiah: <p><a class="mention" href="/u/fmassa">@fmassa</a>  There seems to be another problem with customizing Inception.  Even if the image size is adjusted looks like the Inception output is a tuple so the code to compute loss fails.</p>
<p>Code for loss computation:</p>
<pre><code class="lang-auto">output = model(input_var)
loss = criterion(output, target_var)
</code></pre>
<p><code>RuntimeError: expected a Variable argument, but got tuple</code></p>
<p>Looking at the inception implementation it seems like the following lines are being executed:</p>
<pre><code class="lang-auto">if self.training and self.aux_logits:
            return x, aux
</code></pre>
<p>Do you know what the <code>aux_logits</code> parameter is refering to?  Should it simply be ignored for loss computation?</p>
<p>thanks</p>

smth: <p>you need to compute loss on both x and aux (separately). aux is the auxillary classifier referred to in the inception paper.</p>

achaiah: <p>I see.  Do you have an idea of what specifically needs to be adapted to make it work?  I’m not familiar with the exact computation of loss in the inception network so I’m not sure how to change the pytorch code.</p>
<p>Thanks.</p>

wangg12: <p><a class="mention" href="/u/achaiah">@achaiah</a> Have you figured out how to use the inception-v3 model? How can I compute the aux loss and do backpropagation?</p>

achaiah: <p>Not completely.  Adjusting the image size is straightforward (just change the transform to reflect the new size) but waiting to see if <a class="mention" href="/u/smth">@smth</a> has any pointers on how to compute (and propagate) the loss.</p>
<p>size adjustment:</p>
<pre><code class="lang-auto">transforms.Compose([
                transforms.Scale(299),
                transforms.ToTensor(),
                transforms.Normalize(mean, std)
            ])
</code></pre>

wangg12: <p>OK, thanks anyway. I’ve already figured out how to use inception v3.</p>

achaiah: <p><a class="mention" href="/u/wangg12">@wangg12</a> would you mind sharing your findings?</p>

wangg12: <p>I just add the loss and aux loss (say total_loss), in the train phase, backprop the total_loss.</p>

achaiah: <p>Ok, so you just do loss1 + loss2 like here <a href="https://discuss.pytorch.org/t/how-to-extract-features-of-an-image-from-a-trained-model/119/55">How to extract features of an image from a trained model</a> ?  Then backprop them?</p>

achaiah: <p>If so, do you use the same criterion with same targets for both losses?</p>

wangg12: <p>Yes. I use the same criterion with same targets for both losses. And it seems to work well for my task.</p>

achaiah: <p>Thanks, I appreciate the feedback.  I was just about to try the same thing.</p>

ahkarami: <p>According to the useful guidelines of <a class="mention" href="/u/achaiah">@achaiah</a> &amp; <a class="mention" href="/u/wangg12">@wangg12</a>, I can fine tune the <em>inception v3</em> model. However, I can’t save this model correctly and then reuse it again. Would you please help me?<br>
I have tested both of the methods described at <a href="http://pytorch.org/docs/notes/serialization.html#recommended-approach-for-saving-a-model" rel="noopener nofollow ugc">Recommended approach for saving a model</a>, but they don’t work correctly for <em>inception v3</em> model.</p>
<p>In fact, when I used the</p>
<blockquote>
<p>torch.save(myInceptionModel, ‘./model.pth’)</p>
</blockquote>
<p>approach, when I loaded the model and re use it (i.e., test some new images via the model and forward them to the loaded model), the following error has occurred:</p>
<blockquote>
<p>result = i.index(self.index)<br>
IndexError: index 1 is out of range for dimension 0 (of size 1)</p>
</blockquote>
<p>And when I used the following approach for saving the model</p>
<blockquote>
<p>torch.save(myInceptionModel.state_dict(), ‘./model.pth’)</p>
</blockquote>
<p>the loaded model give me wrong answers (i.e., can’t predict the label of the images correctly in most of the samples).</p>

achaiah: <p>There are two issues here to look out for.</p>
<ol>
<li>Make sure you replace the FC and Bias layers just like you did when fine-tuning the model, since the original model still has the default number of classes.</li>
<li>Make sure you somehow preserve and restore the class_to_idx mapping. Remember that the model doesn’t know anything about names of actual classes so you have store it yourself. A simple mistake (that I’ve made) is to have fewer classes in the test set than in the train set. You’d wouldn’t catch the problem at first but if you’re using an ImageFolder then it will create a different class_to_idx mapping from your test set than from your train set if they have a different number of classes.</li>
</ol>

Dondon_Jie: <p>I face  the same issue…</p>
<pre><code>---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
&lt;ipython-input-14-3fe681399c1a&gt; in &lt;module&gt;()
  1 t1 = time.time()
  2 print(datetime.now().strftime("%Y-%m-%d %H:%M:%S"))
----&gt; 3 model_conv = train_model(model_conv, criterion, optimizer_conv,exp_lr_scheduler, num_epochs=10)
  4 t2 = time.time()
  5 print(datetime.now().strftime("%Y-%m-%d %H:%M:%S"))

&lt;ipython-input-8-c5340d5d5434&gt; in train_model(model, criterion, optimizer, lr_scheduler, num_epochs)
 49                 loss = None
 50                 # for nets that have multiple outputs such as inception
---&gt; 51                 outputs = model(inputs)
 52                 if isinstance(outputs, tuple):
 53                     loss = sum((criterion(o,labels) for o in outputs))

/home/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
204 
205     def __call__(self, *input, **kwargs):
--&gt; 206         result = self.forward(*input, **kwargs)
207         for hook in self._forward_hooks.values():
208             hook_result = hook(self, input, result)

/home/anaconda3/lib/python3.6/site-packages/torchvision/models/inception.py in forward(self, x)
 72             x = x.clone()
 73             x[0] = x[0] * (0.229 / 0.5) + (0.485 - 0.5) / 0.5
---&gt; 74             x[1] = x[1] * (0.224 / 0.5) + (0.456 - 0.5) / 0.5
 75             x[2] = x[2] * (0.225 / 0.5) + (0.406 - 0.5) / 0.5
 76         # 299 x 299 x 3

/home/anaconda3/lib/python3.6/site-packages/torch/autograd/variable.py in __getitem__(self, key)
 67                 type(key.data).__name__ == 'ByteTensor'):
 68             return MaskedSelect()(self, key)
---&gt; 69         return Index(key)(self)
 70 
 71     def __setitem__(self, key, value):

/home/anaconda3/lib/python3.6/site-packages/torch/autograd/_functions/tensor.py in forward(self, i)
 14     def forward(self, i):
 15         self.input_size = i.size()
---&gt; 16         result = i.index(self.index)
 17         self.mark_shared_storage((i, result))
 18         return result

IndexError: index 1 is out of range for dimension 0 (of size 1)
</code></pre>
<p>What can I do?</p>

godisboy: <p>I face the same question while fine-tune the InceptionV3.<br>
And I transform the image to 299*299.<br>
Pytorch version=0.2.0+de24bb4</p>
<p>Traceback (most recent call last):<br>
File “<a href="http://inceptionv3.py" rel="nofollow noopener">inceptionv3.py</a>”, line 263, in <br>
train(train_loader, model, criterion, optimizer, epoch)<br>
File “<a href="http://inceptionv3.py" rel="nofollow noopener">inceptionv3.py</a>”, line 126, in train<br>
outputs = model(input_var)<br>
File “/usr/local/lib/python2.7/dist-packages/torch/nn/modules/module.py”, line 252, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/usr/local/lib/python2.7/dist-packages/torch/nn/parallel/data_parallel.py”, line 60, in forward<br>
outputs = self.parallel_apply(replicas, inputs, kwargs)<br>
File “/usr/local/lib/python2.7/dist-packages/torch/nn/parallel/data_parallel.py”, line 70, in parallel_apply<br>
return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])<br>
File “/usr/local/lib/python2.7/dist-packages/torch/nn/parallel/parallel_apply.py”, line 67, in parallel_apply<br>
raise output<br>
RuntimeError: size mismatch at /home/scw4750/AIwalker/pytorch/source/pytorch/torch/lib/THC/generic/THCTensorMathBlas.cu:243</p>

lolongcovas: <p>Yes,<br>
If I am not wrong, I think the inception v3 model was trained with different image preprocessing. After unroll the equation:</p>
<pre><code class="lang-auto">new_x = (original_x*0.299) / 0.5 + (0.485-0.5)/0.5  --&gt;
new_x = (0.299*original_x + 0.485) / 0.5 - 1
</code></pre>
<p>the <code>original_x</code> is:</p>
<pre><code class="lang-auto">original_x = ((new_x + 1) * 0.5 - 0.485) / 0.299
</code></pre>
<p>which is mean subtraction and one unit deviation normalization by ( 0.485 and 0.299 resp.) this means that new_x was ranged between -1 and 1.</p>

micklexqg: <p>inception v3 has two outputs. so how to get the final predict or accuracy with the two outputs?</p>