Input format for pretrained torchvision models

username: M_S
cooked: <p>Hi,<br>
Many torchvision models are also available in a fully trained mode.<br>
As it is not obvious, I’d like to know what’s the input format that the models are trained on.<br>
I’m assuming it’s color images so 3 channel tensors, but is it RGB or BGR (like in OpenCV \ cv2)?<br>
Also what’s the range values? Is it uint8 ([0:255]) or a float type?<br>
If it’s float, is it normalized to [0:1] or to [-1:1]?<br>
Please specify these details so it would be easy to reproduce the results.</p>
<p>Thanks</p>

username: ptrblck
cooked: <p>From the <a href="https://pytorch.org/docs/stable/torchvision/models.html">docs</a>:</p>
<blockquote>
<p>All pre-trained models expect input images normalized in the same way, i.e. mini-batches of 3-channel RGB images of shape (3 x H x W), where H and W are expected to be at least 224. The images have to be loaded in to a range of [0, 1] and then normalized using  <code>mean = [0.485, 0.456,0.406]</code>  and  <code>std = [0.229, 0.224, 0.225]</code> .</p>
</blockquote>

username: M_S
cooked: <p>Thank you for the detailed answer!</p>

username: M_S
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, I was wondering, isn’t it preferred to normalize the data to the [-1:1] range instead of the [0:1] range?</p>

username: ptrblck
cooked: <p>Not sure, if <code>[-1:1]</code> is preferred over <code>[0:1]</code>, but the implemented normalization standardizes the data, such that it has a zero mean and a unit variance (sometimes known as z-score).</p>

username: tazz
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> What does one do if you have to export the model to onnx ? Can one just assume the batch size for a model? and specify the batch size later? or is it possible to not mention the batch size while transferring it to ONNX?</p>

username: ptrblck
cooked: <p>You could use the <code>dynamic_axes</code> argument to specify that the batch size is not fixes as used in <a href="https://pytorch.org/tutorials/advanced/super_resolution_with_onnxruntime.html">this tutorial</a>.</p>

username: tazz
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> thanks a lot. That helped!</p>

username: M_S
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>,</p>
<p>In the docs, mean and STD values are specified for classification, segmentation, and video classification.<br>
They are not specified for object detection models: <a href="https://pytorch.org/docs/stable/torchvision/models.html#object-detection-instance-segmentation-and-person-keypoint-detection" rel="nofollow noopener">https://pytorch.org/docs/stable/torchvision/models.html#object-detection-instance-segmentation-and-person-keypoint-detection</a></p>
<p>Can we get their values as well?</p>

username: ptrblck
cooked: <p>You could manually pass the <code>image_mean</code> and <code>image_std</code> to the model creation based, if you’ve calculated them manually on your datase.<br>
If you don’t pass them, <a href="https://github.com/pytorch/vision/blob/75f5b57e680549d012b3fc01b356b2fb92658ea7/torchvision/models/detection/faster_rcnn.py#L227-L230">these values</a> will be used, which apparently are the <code>ImageNet</code> stats.</p>
<p>CC <a class="mention" href="/u/pmeier">@pmeier</a> for more information.</p>

username: pmeier
cooked: <p><a class="mention" href="/u/m_s">@M_S</a></p>
<p>For detection models we do <a href="https://github.com/pytorch/vision/blob/75f5b57e680549d012b3fc01b356b2fb92658ea7/references/detection/train.py#L51-L56" rel="nofollow noopener"><strong>not</strong> use any normalization</a>. I don’t know why not and can’t comment if this would be beneficial or not. You’ll have to ask <a class="mention" href="/u/fmassa">@fmassa</a> about that <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: M_S
cooked: <p>Thank you. Indeed I tried inference on a pretrained model with no normalization and it worked. Some notes though:</p>
<ol>
<li>Consider explaining why the pretrained detection models are the only ones that don’t require image normalization (I understand that the training set was not normalized. But again, why?)</li>
<li>Worth mentioning that no normalization is needed. The classification, segmentation and detection pretrained models are trained on ImageNet, so one may think all of them require ImageNet normalization, when in fact only the classification and segmentation models require normalization.<br>
Perhaps it’s best to put this info in a table, since the pretrained video models also have a normalization, but different.</li>
</ol>

username: pmeier
cooked: <p>Those are valid points. Do you want to open an <a href="https://github.com/pytorch/vision/issues/new?assignees=&amp;labels=&amp;template=---documentation.md&amp;title=" rel="nofollow noopener">issue</a> for that or should I do it in your name?</p>

username: M_S
cooked: <p>I would appreciate if you could do it. I’m mattans on GitHub. Thanks!</p>

username: pmeier
cooked: <p>For reference: <a href="https://github.com/pytorch/vision/issues/2397" rel="nofollow noopener">https://github.com/pytorch/vision/issues/2397</a></p>

username: ptrblck
cooked: <p>The the sake of completeness:<br>
Based on <a href="https://github.com/pytorch/vision/issues/2397#issuecomment-654286088">Francisco’s answer</a>, the normalization is used internally in <a href="https://github.com/pytorch/vision/blob/e212cc86b80baf1a46681442db1312ebce5a21bb/torchvision/models/detection/transform.py#L104-L105">these lines of code</a>, which is using the <code>GeneralizedRCNNTransform</code> from my previous link.</p>