Using your own dataset on PyTorch

Ismail_Elezi: <p>I have a dataset in numpy format (actually, I just modified the CIFAR-10/MNIST datasets given on PyTorch). The dimensions of it consist with the dimensions a normal CNN expects. For example, if I write:</p>
<p><code>print(our_dataset.shape, our_labels.shape)</code></p>
<p>I get:</p>
<p><code>(10000, 3, 32, 32) (10000,)</code></p>
<p>which is fine. Now I cast the data info torch format using:</p>
<pre><code>train_data = torch.from_numpy(our_dataset)
our_labels = torch.from_numpy(our_labels)
</code></pre>
<p>encapsulate it into a TensorDataset:</p>
<p><code>train = torch.utils.data.TensorDataset(train_data, our_labels)</code></p>
<p>and finally into a DataLoader:</p>
<p><code>trainloader = torch.utils.data.DataLoader(train, batch_size=128, shuffle=True)</code></p>
<p>All fine here. Now I build the neural network, and then when I do the training, I get the error:</p>
<p><code>TypeError: DoubleSpatialConvolutionMM_updateOutput received an invalid combination of arguments - got (int, torch.DoubleTensor, torch.DoubleTensor, torch.FloatTensor, torch.FloatTensor, torch.DoubleTensor, torch.DoubleTensor, long, long, int, int, int, int), but expected (int state, torch.DoubleTensor input, torch.DoubleTensor output, torch.DoubleTensor weight, [torch.DoubleTensor bias or None], torch.DoubleTensor finput, torch.DoubleTensor fgradInput, int kW, int kH, int dW, int dH, int padW, int padH)</code></p>
<p>If I am working on cuda, then the error changes to:</p>
<p><code>_cudnn_convolution_full_forward received an invalid combination of arguments - got (torch.cuda.DoubleTensor, torch.cuda.FloatTensor, torch.cuda.FloatTensor, torch.cuda.DoubleTensor, tuple, tuple, int, bool), but expected (torch.cuda.RealTensor input, torch.cuda.RealTensor weight, torch.cuda.RealTensor bias, torch.cuda.RealTensor output, std::vector&lt;int&gt; pad, std::vector&lt;int&gt; stride, int groups, bool benchmark)</code></p>
<p>Anyone has seen these errors before? In addition, is the right way of using a new dataset by first casting it to torch format, then building a TensorDataset and finally a DataLoader?</p>

sarthak1996: <p>Can you paste the output for <code>our_dataset.dtype</code> and <code>our_labels.dtype</code><br>
And also did you send the data and labels to the GPU while training?<br>
For example:<br>
<code>inputs, labels = data</code><br>
<code>inputs, labels = Variable(inputs).cuda(), Variable(labels).cuda()</code></p>
<p><code>our_dataset.dtype</code> should be of type <code>(np.float32)</code><br>
<code>our_labels.dtype</code> should be of type <code>('int64')</code></p>

edgarriba: <p>you need to cast your data to double in that case</p>

Ismail_Elezi: <p>If I type:</p>
<p><code>print(our_dataset.dtype, our_labels.dtype)</code></p>
<p>I get:</p>
<p><code>float32 int64</code></p>
<p>I sent the data to cuda (and gave the Error in that case). However, if I don’t send the net (and data) to cuda, I get the other error that I posted in my original post.</p>
<p><a class="mention" href="/u/edgarriba">@edgarriba</a></p>
<p>Well, the data is already in float.</p>

edgarriba: <p>right, but the error says that expect a DoubleTensor</p>

sarthak1996: <p>Try <code>np.double</code> for <code>our_dataset</code></p>

Ismail_Elezi: <p>If I case the our_dataset to float32 and labels to int64, then the error changes to:</p>
<p><code>multi-target not supported at /data/users/soumith/miniconda2/conda-bld/pytorch-0.1.7_1485444530918/work/torch/lib/THNN/generic/ClassNLLCriterion.c:20</code></p>
<p>but the shape of the labels is (10000,).</p>
<p>Hmm, this is becoming weird.</p>

sarthak1996: <p>In the training function:<br>
where you have written:<br>
<code>inputs, labels = data</code><br>
<code>inputs, labels = Variable(inputs), Variable(labels)</code><br>
add this:<br>
<code>labels=labels[:,0]</code><br>
The error is because the labels are of size <code>batch_sizex1</code> you need to convert it to <code>'batch_size'</code></p>

Ismail_Elezi: <aside class="quote no-group" data-username="sarthak1996" data-post="8" data-topic="536">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/43a26b/48.png" class="avatar"> sarthak1996:</div>
<blockquote>
<p>labels=labels[:,0]</p>
</blockquote>
</aside>
<p>It seems that this works.</p>

Shine_George: <p>the version of 0.3.0 can well solve the problem of the problem of ‘received an invalid combination of arguments’</p>