pin_memory=True but data unavailable on GPU

username: cpeters
cooked: <p>I have a code section which creates a data loader as follows:</p>
<pre><code class="lang-auto">    val_loader = torch.utils.data.DataLoader(
        datasets.ImageFolder(valdir, transforms.Compose([
            transforms.RandomResizedCrop(224),
            transforms.ToTensor(),
            normalize,
        ])),
        batch_size=args.batch_size, shuffle=False,
        num_workers=args.workers, pin_memory=True)
</code></pre>
<p>This is essentially the same loader that I’ve been using in other files without problem. Similarly, the code within the validate method in question is:</p>
<pre><code class="lang-auto">for i, (input, target) in enumerate(val_loader):
    target = target.cuda(device=None, non_blocking=False)
    with torch.no_grad():
        input_var = torch.autograd.Variable(input)
        target_var = torch.autograd.Variable(target)

        # compute output
        with torch.cuda.amp.autocast():
            output = model(input_var)
            loss = criterion(output, target_var)
</code></pre>
<p>Yet, despite the pinned memory, it throws the following error stack when I attempt to run it:</p>
<blockquote>
<p>RuntimeError                              Traceback (most recent call last)<br>
 in <br>
238<br>
239 if <strong>name</strong> == ‘<strong>main</strong>’:<br>
 → 240     main()</p>
<p> in main()<br>
126     for run in range(1):<br>
127     # evaluate on validation set<br>
 → 128         acc = validate(val_loader, model, criterion)<br>
129         prec1 = prec1 + acc[0]<br>
130         prec5 = prec5 + acc[1]</p>
<p> in validate(val_loader, model, criterion)<br>
159<br>
160             with torch.cuda.amp.autocast():<br>
 → 161                 output = model(input_var)<br>
162                 loss = criterion(output, target_var)<br>
163 #             scaler.unscale(loss)</p>
<p>~.conda\envs\pytorch\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)<br>
725             result = self._slow_forward(*input, **kwargs)<br>
726         else:<br>
 → 727             result = self.forward(*input, **kwargs)<br>
728         for hook in itertools.chain(<br>
729                 _global_forward_hooks.values(),</p>
<p>~\Documents\Pytorch\COCO Autoencoder\places365\caffemodel2pytorch.py in forward(self, data, **variables)<br>
119                                 assert name in variables, ‘Variable [{}] does not exist. Pass it as a keyword argument or provide a layer which produces it.’.format(name)<br>
120                         inputs = [variables[name] if propagate_down else variables[name].detach() for name, propagate_down in zip(module.caffe_input_variable_names, module.caffe_propagate_down)]<br>
 → 121                         outputs = module(*inputs)<br>
122                         if not isinstance(outputs, tuple):<br>
123                                 outputs = (outputs, )</p>
<p>~.conda\envs\pytorch\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)<br>
725             result = self._slow_forward(*input, **kwargs)<br>
726         else:<br>
 → 727             result = self.forward(*input, **kwargs)<br>
728         for hook in itertools.chain(<br>
729                 _global_forward_hooks.values(),</p>
<p>~\Documents\Pytorch\COCO Autoencoder\places365\caffemodel2pytorch.py in forward(self, x)<br>
330                         convert_to_gpu_if_enabled(self)<br>
331                         init_weight_bias(self, requires_grad = requires_grad)<br>
 → 332                 return super(Convolution, self).forward(x)<br>
333<br>
334         def set_parameters(self, weight = None, bias = None):</p>
<p>~.conda\envs\pytorch\lib\site-packages\torch\nn\modules\conv.py in forward(self, input)<br>
421<br>
422     def forward(self, input: Tensor) → Tensor:<br>
 → 423         return self._conv_forward(input, self.weight)<br>
424<br>
425 class Conv3d(_ConvNd):</p>
<p>~.conda\envs\pytorch\lib\site-packages\torch\nn\modules\conv.py in _conv_forward(self, input, weight)<br>
417                             weight, self.bias, self.stride,<br>
418                             _pair(0), self.dilation, self.groups)<br>
 → 419         return F.conv2d(input, weight, self.bias, self.stride,<br>
420                         self.padding, self.dilation, self.groups)<br>
421</p>
<p>RuntimeError: Input type (torch.FloatTensor) and weight type (torch.cuda.HalfTensor) should be the same</p>
</blockquote>
<p>This is despite having identical code for loading data and feeding it to the model that works perfectly fine, with the sole difference being in the <em>model</em> loading–but the model is correctly on the GPU, rather than the data?</p>

username: cpeters
cooked: <p>The problem appears to be because the new model was moved onto the GPU using <code>model.cuda()</code> rather than <code>model = torch.nn.DataParallel(model).cuda()</code>, though I don’t know why this is the case (or what is correct practice now, as this is based on quite old code).</p>
<p><strong>Edit:</strong> Rather, what leaves me more confused now is how the latter causes the inputs to be copied to the GPU automatically, whereas pinning memory just reserves space for fast transfers in main memory…</p>