Tensor dtype for input batches to pytorch networks

papoo13: <p>When adding torchvision.transform.ToTensor() to our dataloaders transformation, ToTensor() sets the datatype of input images X to torch.float32 and labels y to torch.int64. Is there a specific reason for this?</p>

ptrblck: <p><code>float32</code> is the default <code>dtype</code> in PyTorch and floating point types are expected in training a neural network, which is why the integer image types are transformed to <code>float32</code>.</p>
<aside class="quote no-group" data-username="papoo13" data-post="1" data-topic="155097">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/f19dbf/48.png" class="avatar"> papoo13:</div>
<blockquote>
<p>and labels y to torch.int64</p>
</blockquote>
</aside>
<p>I don’t think that’s the case, as <code>ToTensor()</code> would expect an image-line input as seen <a href="https://github.com/pytorch/vision/blob/fb7f9a16628cb0813ac958da4525247e325cc3d2/torchvision/transforms/functional.py#L137-L141">here</a>.</p>

papoo13: <p>thank you for your answer.<br>
So in this piece of code, after applying the transformations on the data set, we get the aforementioned types for x and y:</p>
<blockquote>
<p>training_data = torchvision.datasets.CIFAR10(<br>
root=“data”,<br>
train=True,<br>
download=True,<br>
transform=ToTensor()<br>
)</p>
<p>test_data = torchvision.datasets.CIFAR10(<br>
root=“data”,<br>
train=False,<br>
download=True,<br>
transform=ToTensor()<br>
)</p>
</blockquote>
<blockquote>
<p>batch_size = 64</p>
</blockquote>
<blockquote>
<p>train_dataloader = DataLoader(training_data, batch_size=batch_size)<br>
test_dataloader = DataLoader(test_data, batch_size=batch_size)</p>
<p>for X, y in test_dataloader:</p>
<pre><code>print(f"Shape of X [N, C, H, W]: {X.shape} {X.dtype}")
print(f"Shape of y: {y.shape} {y.dtype}")
break
</code></pre>
</blockquote>
<p>output is:</p>
<blockquote>
<p>Shape of X [N, C, H, W]: torch.Size([64, 3, 32, 32]) torch.float32<br>
Shape of y: torch.Size([64]) torch.int64</p>
</blockquote>
<p>so I concluded that theses stypes are set after applying ToTensor() to the dataset. is torch.int64 the default data type for labels? what happens if we change these types to torch.float64 and torch.int32?</p>

ptrblck: <p>The <code>CIFAR10</code> dataset applies the <code>transform</code> only to the data samples as seen <a href="https://github.com/pytorch/vision/blob/fb7f9a16628cb0813ac958da4525247e325cc3d2/torchvision/datasets/cifar.py#L117-L118">here</a> and uses <code>target_transform</code> for the targets:</p>
<pre data-code-wrap="python"><code class="lang-python">        if self.transform is not None:
            img = self.transform(img)

        if self.target_transform is not None:
            target = self.target_transform(target)

        return img, target
</code></pre>
<p>The targets are stored as integers in a <code>list</code> and the <code>DataLoader</code>’s <code>collate_fn</code> would stack them to tensors, which would then default to <code>int64</code>:</p>
<pre data-code-wrap="python"><code class="lang-python">torch.tensor(1).dtype
# torch.int64
</code></pre>
<p><code>nn.CrossEntropyLoss</code> as well as <code>nn.NLLLoss</code> expect <code>int64</code> tensors as the targets which is why it’s the default type (unless <code>CrossEntropyLoss</code> is used with probabilities in newer PyTorch releases).</p>
<aside class="quote no-group" data-username="papoo13" data-post="3" data-topic="155097">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/f19dbf/48.png" class="avatar"> papoo13:</div>
<blockquote>
<p>what happens if we change these types to torch.float64 and torch.int32?</p>
</blockquote>
</aside>
<p>While the data loading would work, your loss calculation might fail depending which loss you are using.</p>

papoo13: <p>Thank you very much for the through answer.</p>