akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
CUDA memory out

username: oasjd7
cooked: <p>When I use one GPU,<code> 'CUDA memory out'</code> occurs.<br>
so I add one more GPU with <code>nn.DataParallel(model)</code> but still memory out occurs.<br>
I searched google about this, then found output_device settings like<br>
<code>model = nn.DataParallel(model, output_device=1)</code>.<br>
<code>model2 = nn.DataParallel(model2, output_device=1)</code>.</p>
<p>but I got a runtime error.<br>
How can I solve it except decreasing batch size ?</p>
<blockquote>
<p>RuntimeError: Assertion `THCTensor_(checkGPU)(state, 4, input, target, output, total_weight)’ failed. Some of weight/gradient/input tensors are located on different GPUs. Please move them to a single one. at /pytorch/aten/src/THCUNN/generic/ClassNLLCriterion.cu:28</p>
</blockquote>

username: albanD
cooked: <p>This error happens because you Give to your <code>NLLLoss</code> inputs and targets that are on different devices. They all need to be on the same GPU.</p>
Tensor not being moved to GPU.. What am I doing wrong?

username: Umair_Javaid
cooked: <pre><code class="lang-auto">data_preprocess = transforms.Compose([
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ])

def preprocess_input(img):
  print(img.shape)
  x = data_preprocess(img)
  x.unsqueeze_(0)
  x.to(device)
  print("x.size(): ",x.size(),x.type())
  return x
</code></pre>
<p><strong>output:</strong></p>
<pre><code class="lang-auto">(375, 500, 3)
x.size():  torch.Size([1, 3, 375, 500]) torch.FloatTensor
</code></pre>
<p>Hence getting the <strong>Error:</strong><br>
<code>RuntimeError: Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same</code></p>
user_suspended: yes

username: Umair_Javaid
cooked: <p>fixed it by <strong>returning x.to(device)</strong>:</p>
<pre><code class="lang-auto">def preprocess_input(img):
  print(img.shape)
  x = data_preprocess(img)
  x.unsqueeze_(0)
  #x.to(device)
  print("x.size(): ",x.size(),x.type())
  return x.to(device)
</code></pre>
user_suspended: yes

username: albanD
cooked: <p>Yes, the <code>.to()</code> operation is not inplace. So you need to use the returned value!</p>
A question about achieving complex weight decay in PyTorch

username: alan_ayu
cooked: <p>Hello everyone,</p>
<p>I am reading the paper “Bag of tricks for image classification with convolutional neural networks”. And I want to test the impact of “No bias decay”, which means only applying regularization to weights of convolution, and other parameters, including the biases and the parameters in BN layers are left unregularized.</p>
<p>I checked the code of convolution in PyTorch, but I found may there is no way to apply different regularization to conv’s weights and bias, respectively.</p>
<p>The code I found about the definition of Convolution,<br>
class _ConvNd(Module):</p>
<pre><code>__constants__ = ['stride', 'padding', 'dilation', 'groups', 'bias',
                 'padding_mode', 'output_padding', 'in_channels',
                 'out_channels', 'kernel_size']

def __init__(self, in_channels, out_channels, kernel_size, stride,
             padding, dilation, transposed, output_padding,
             groups, bias, padding_mode):
    super(_ConvNd, self).__init__()
    if in_channels % groups != 0:
        raise ValueError('in_channels must be divisible by groups')
    if out_channels % groups != 0:
        raise ValueError('out_channels must be divisible by groups')
    self.in_channels = in_channels
    self.out_channels = out_channels
    self.kernel_size = kernel_size
    self.stride = stride
    self.padding = padding
    self.dilation = dilation
    self.transposed = transposed
    self.output_padding = output_padding
    self.groups = groups
    self.padding_mode = padding_mode
    if transposed:
        self.weight = Parameter(torch.Tensor(
            in_channels, out_channels // groups, *kernel_size))
    else:
        self.weight = Parameter(torch.Tensor(
            out_channels, in_channels // groups, *kernel_size))
    if bias:
        self.bias = Parameter(torch.Tensor(out_channels))
    else:
        self.register_parameter('bias', None)
    self.reset_parameters()
</code></pre>
<p>If you have any idea about that, please shear something in this post. Thank you !</p>

username: Eta_C
cooked: <p>In my understanding, this code would work. I have not test it, but it is something worth trying.</p>
<pre><code class="lang-python">optim.SGD([
        {'params': other_parameters}
        {'params': model.conv_layer.weight, 'lr': 1e-3, 'weight_decay': 0.9},
        {'params': model.conv_layer.bias, 'lr': 1e-3, 'weight_decay': 0.0}
    ], lr=1e-2, momentum=0.9, weight_decay=0.98)
</code></pre>
How to convert F.interpolate to onnx?

username: dalalaa
cooked: <p>I’ve converted an east model into ONNX，but when checking model by <code>onnx.checker.check_model()</code>, I got the following error:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "east_onnx_demo.py", line 223, in &lt;module&gt;
    onnx.checker.check_model(net)
  File "/home/dai/py36env/lib/python3.6/site-packages/onnx/checker.py", line 86, in check_model
    C.check_model(model.SerializeToString())
onnx.onnx_cpp2py_export.checker.ValidationError: Node () has input size 4 not in range [min=2, max=2].

==&gt; Context: Bad node spec: input: "901" input: "924" input: "924" input: "923" output: "925" op_type: "Resize" attribute { name: "coordinate_transformation_mode" s: "pytorch_half_pixel" type: STRING } attribute { name: "cubic_coeff_a" f: -0.75 type: FLOAT } attribute { name: "mode" s: "linear" type: STRING } attribute { name: "nearest_mode" s: "floor" type: STRING }
</code></pre>
<p>The problem could come from the following code:</p>
<pre><code class="lang-auto">def forward(self, x1,x2,x3,x4):
        y = F.interpolate(
            x4, scale_factor=2.0, mode="bilinear", align_corners=False
        )
        y = torch.cat((y, x3), 1)
        y = self.relu1(self.bn1(self.conv1(y)))
        y = self.relu2(self.bn2(self.conv2(y)))

        y = F.interpolate(
            y, scale_factor=2.0, mode="bilinear", align_corners=False
        )
        y = torch.cat((y, x2), 1)
        y = self.relu3(self.bn3(self.conv3(y)))
        y = self.relu4(self.bn4(self.conv4(y)))

        y = F.interpolate(
            y, scale_factor=2.0, mode="bilinear", align_corners=False
        )
        y = torch.cat((y, x1), 1)
        y = self.relu5(self.bn5(self.conv5(y)))
        y = self.relu6(self.bn6(self.conv6(y)))

        y = self.relu7(self.bn7(self.conv7(y)))
        return y
</code></pre>
<p>Environment: Pytorch1.3.1, ONNX1.5.0</p>
<p>How can I solve this problem?</p>