akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Cast int64 Tensor to float32 in pre-trained model?

Aeroxander: <p><img src="https://www.redditstatic.com/desktop2x/img/renderTimingPixel.png" alt width="1" height="1"></p>
<p>So I’m trying to load a certain ONNX model in OpenCV (started in PyTorch), but it crashes with this error:</p>
<pre><code class="lang-auto">OpenCV(4.1.0) Error: Assertion failed (inputs.size()) in cv::dnn::dnn4_v20190122::Layer::getMemoryShapes, file c:\build\master_winpack-build-win64-vc15\opencv\modules\dnn\src\dnn.cpp, line 3616
</code></pre>
<p>It gives me this error when I try to convert that ONNX model to Tensorflow:</p>
<pre><code class="lang-auto">ValueError: Tensor conversion requested dtype float32 for Tensor with dtype int64: 'Tensor("Mul_3:0", shape=(), dtype=int64)'
</code></pre>
<p>I suppose this is why it isn’t working in OpenCV, is there any way to cast the int64 Tensor to a float32 one? I can try to modify the float32 &lt;-&gt; float16 conversion function so that it can also convert int64 to float16, but think this could be quite complex or impossible (I would have to move the weight up by 19 digits?).</p>
<p><strong>So I’m basically asking how I could cast at int64 to a float32 in an existing pre-trained Pytorch model, while getting the same result in the neural network (don’t want to retrain it)</strong></p>

ecdrid: <p>By forcibly type casting the weights and resaving the model?</p>

Aeroxander: <p>Is there any tutorial on how to cast it and re-save the model? I don’t need to train the model again for this right?</p>
<p>In OpenCV I get this error now:</p>
<p>OpenCV(4.1.0) Error: Assertion failed (inputs.size()) in cv::dnn::dnn4_v20190122::Layer::getMemoryShapes, file c:\build\master_winpack-build-win64-vc15\opencv\modules\dnn\src\dnn.cpp, line 3616</p>
<p>I see that someone converting the model to Tensorflow somehow solved this issue, but I can’t even manage to convert it to Tensorflow… Although casting and resaving the model should fix it!</p>
ATen .options() and .type()

dashesy: <p>What is the difference between <code>.options()</code> and <code>.type()</code>?</p>
<pre><code class="lang-python">auto w  = at::empty({batches, channels, height, width}, xy.type());
</code></pre>
<p>vs</p>
<pre><code class="lang-python">auto w  = at::empty({batches, channels, height, width}, xy.options());
</code></pre>
<p>Would they both inherit the same device from <code>xy</code>?</p>
<p>I asked a <a href="https://discuss.pytorch.org/t/aten-create-tensor-of-the-same-type-options/30817">similar question</a> earlier that I realized I can use <code>.options</code>.</p>

LeviViana: <p>According to the <a href="https://pytorch.org/cppdocs/notes/tensor_creation.html#configuring-properties-of-the-tensor" rel="nofollow noopener">documentation</a>, the <code>options</code> carry 4 construction axes: <code>type</code>, <code>layout</code>, <code>device</code> and <code>requires_grad</code>.</p>

gchanan: <p>don’t use type(), it’s deprecated.</p>

dashesy: <p>But does that mean <code>type</code> itself does not have <code>device</code> or <code>layout</code>? please note that <code>type</code> is different from <code>dtype</code>. I guess <code>.type()</code> is what now <code>.options()</code> provides with better details.</p>

dashesy: <p>I have also lots of</p>
<pre><code class="lang-auto">AT_DISPATCH_FLOATING_TYPES(xy.type() ...)
</code></pre>
<p>should I update them as well?</p>

LeviViana: <p>Nope. Type is related to the numeric encoding of the tensor (i.e. int32, int64, float32, etc). Layout relates to the memory layout of your Tensor storage, it can be strided or sparse.</p>

LeviViana: <p>You should use <code>AT_DISPATCH_FLOATING_TYPES(xy.scalar_type() ...)</code> now.</p>
Change the latent space of autoencoder while training

shivangi: <p>I am training one autoencoder for two classes : real and fake. I have a latent space of 256 dimensions. I just wish to activate first 128 dimensions for the real class and last 128 for fake ones.<br>
My forward function looks something like below. How can this be done.</p>
<pre><code class="lang-auto">    def forward(self, x):
        """
        pass input through the encoder and reconstruct with decoder
        :param x: original input
        :return: x_recon : reconstructed input, z : latent representation
        """
        unpool_info = []

        for m in self.encoder:
            if isinstance(m, nn.MaxPool2d):
                output_size = x.size()
                x, pool_idx = m(x)
                unpool_info.append({'output_size': output_size,
                                    'indices': pool_idx})
            else:
                x = m(x)
        z = x
        for m in self.decoder:
            if isinstance(m, nn.MaxUnpool2d):
                x = m(x, **unpool_info.pop())
            else:
                x = m(x)
        x_recon = x
        return z, x_recon

</code></pre>

shivangi: <p>The solution</p>
<pre><code class="lang-auto">def forward(self, x, y):
        """
        pass input through the encoder and reconstruct with decoder
        :param x: original input
        :param x: original target
        :return: x_recon : reconstructed input, z : latent representation
        """
        unpool_info = []
        real = torch.zeros(256).cuda()
        real[:128] = 0
        fake = torch.zeros(256).cuda()
        fake[128:] = 0
        mult = [real, fake]

        for m in self.encoder:
            if isinstance(m, nn.MaxPool2d):
                output_size = x.size()
                x, pool_idx = m(x)
                unpool_info.append({'output_size': output_size,
                                    'indices': pool_idx})
            else:
                x = m(x)
        z = x
        for i, y_i in enumerate(y):
            z[i] = z[i] * mult[y_i]

        for m in self.decoder:
            if isinstance(m, nn.MaxUnpool2d):
                x = m(x, **unpool_info.pop())
            else:
                x = m(x)
        x_recon = x
        return z, x_recon

</code></pre>
The code is the same, the input is the same, but why the computed loss is different?

yunpei: <p>When I use the alexnet to train the Cifar10, I use the following code to set the rand  seed:</p>
<p><em>random.seed(666)</em><br>
<em>np.random.seed(666)</em><br>
<em>torch.manual_seed(666)</em><br>
<em>torch.cuda.manual_seed_all(666)</em><br>
<em>torch.cuda.manual_seed(666)</em><br>
<em>torch.backends.cudnn.benchmark = True</em><br>
<em>torch.backends.cudnn.deterministic = True</em></p>
<p>And I train it twice, and print the input and loss every iter. But I find the loss of the two different running is different, however, the input is the same. Just as the content in red box.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/f/f5b618e7e09ed64cd6e5248c2ccef1e3124f75d9.png" data-download-href="https://discuss.pytorch.org/uploads/default/f5b618e7e09ed64cd6e5248c2ccef1e3124f75d9" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/f/f5b618e7e09ed64cd6e5248c2ccef1e3124f75d9_2_690x305.png" alt="image" data-base62-sha1="z3F7yAkiT80G9liLsXrcJNLf20V" width="690" height="305" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/f/f5b618e7e09ed64cd6e5248c2ccef1e3124f75d9_2_690x305.png, https://discuss.pytorch.org/uploads/default/optimized/2X/f/f5b618e7e09ed64cd6e5248c2ccef1e3124f75d9_2_1035x457.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/f/f5b618e7e09ed64cd6e5248c2ccef1e3124f75d9_2_1380x610.png 2x" data-dominant-color="262121"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1440×637 661 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><em><strong>The code is the same, the input is the same, but why the computed loss is different?</strong></em></p>
<p>Do someone know how it comes? Thank you very much!!</p>

ptrblck: <p><code>torch.backends.cudnn.benchmark</code> should be set to <code>False</code>.<br>
If that doesn’t help, you might see some floating precision issues to to atomic aperations (<a href="https://pytorch.org/docs/stable/notes/randomness.html" rel="nofollow noopener">docs</a>).</p>
Updating torchvision after upgrading to PyTorch 1.0.1

Deeply: <p>I noticed that some <code>transforms</code> were missing from <code>torchvision.transforms</code>; for example, for example,  <code>transforms.RandomPerspective</code>.</p>
<pre><code class="lang-auto">torch.__version__
Out[17]: '1.0.1'

torchvision.__version__
Out[18]: '0.2.1'
</code></pre>
<p>I need to ask before trying anything, would updating <code>torchvision</code> via<br>
<code>conda update pytorch torchvision</code><br>
solve the problem?</p>

ptrblck: <p><code>RandomPerspective</code> is quite new (<a href="https://github.com/pytorch/vision/pull/781" rel="nofollow noopener">PR</a>), so you should update <code>torchvision</code> to get this functionality.<br>
If you are concerned about the <code>conda update</code> command and any side effects, I would recommend to create a new virtual environment and try to install the latest lib versions there.</p>
Implementing DeepLab V3+ in PYTORCH

ganLover: <p>I am trying to implement DeepLab V3+ in PYTORCH, but I am confused in some parts of the network.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/5/56ab90914d256b1e3c8b1dd467f88357513fda1e.png" alt="image" data-base62-sha1="cmIDz7JzZPBfmjPULe1BxqazKdU" width="666" height="391"></p>
<ol>
<li>
<p>Is “1*1 conv” -. Global Average Pooling as mentioned in DeepLab V3</p>
</li>
<li>
<p>What exactly is “Image Pooling” operation?</p>
</li>
<li>
<p>As Dilated convolutions of different Rates are applied on the same feature map, the resulting feature map will have different dimensions. Is padding applied during these convolutions, so that the final results can be concatenated?</p>
</li>
</ol>
<p>PLS help…</p>

brookisme: <p>Hi - I’m doing the same right now.</p>
<p>Here’s my understanding:</p>
<ul>
<li>The 1x1 conv is just that, a 1x1 conv.</li>
<li>The “Image Pooling” is the Global Average Pooling.</li>
<li>the dilations do <em>not</em> result in different dimensions but you need to have different padding for  each.</li>
</ul>
<p>Let me go into a little more detail for the padding issue.  For a normal NxN-Conv (dilation rate=1) choosing a padding of <code>(N-1)//2</code> will give you the same size output as input. Now consider the a 3x3 Conv with dilation rate=2.  Draw a little picture and you can convince yourself that the field of view is the same as the field of view for a 5x5 Conv and therefore, the padding required to have the same size output as input is not <code>(3-1)//2=1</code>, as with the <code>rate=1</code> case, but rather <code>(5-1)//2=2</code></p>
<p>It turns out that the generalized formula for “same” padding for atrous convs is</p>
<pre><code class="lang-python">effective_kernel_size=kernel_size+((kernel_size-1)*(rate-1))
padding=int((effective_kernel_size-1)//2)
</code></pre>
<p>Which for <code>kernel_size=3</code>, <code>rate=2</code> simplifies to <code>(3+2*1-1)/2=(5-1)/2=2</code> as before.</p>
Scatter_sample like operation?

dragen: <p>i have a tensor <code>x = [2, 3, 2, 1, 2, 3, 2]</code><br>
I want to randomly sample a number from all same number, e.g. randomly sample the  <code>2</code> from all 2’s index, and get index may be <code>0</code>, namely sampled the first 2; randomly sample the <code>3</code>, and get index may be <code>5</code>. and randomly sample the <code>1</code>, get index <code>3</code> since only single <code>1</code> exists. so the return index may like <code>[0, 5, 3]</code> or mask: <code>[1, 0, 0, 1, 0, 1, 0]</code>.<br>
I dnt know how to implement this logic by pytorch?  please kindly help me.</p>
<p>Thank you so much.</p>

ptrblck: <p>You could get all occurrences of the current number you would like to draw and randomly permute these indices:</p>
<pre><code class="lang-python">x = torch.tensor([2, 3, 2, 1, 2, 3, 2])
to_sample = torch.as_tensor(2)
idx_all = (x == to_sample).nonzero()
idx = idx_all[torch.randperm(len(idx))[0]]
</code></pre>

dragen: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thanks.<br>
However, if i want to iterate over all unique number, i have to use a python <code>for</code> loop:</p>
<pre><code class="lang-auto">x = torch.tensor([2, 3, 2, 1, 2, 3, 2])
unique = torch.unique(x)
for u in unique: #!!! time consuming!
    to_sample = u
    idx_all = (x == to_sample).nonzero()
    idx = idx_all[torch.randperm(len(idx))[0]]
</code></pre>
<p>any concise way to finish it without python <code>for</code> loop?<br>
Thank you.</p>

ecdrid: <p>using numpy might be faster; why iterating over your list?<br>
Something like this,</p>
<pre><code class="lang-auto">import numpy as np
values = np.array([1,2,3,1,2,4,5,6,3,2,1])
searchval = 3
ii = np.where(values == searchval)[0]
</code></pre>

ptrblck: <p>In that case, I would just sample the values using <code>torch.multinomial</code>:</p>
<pre><code class="lang-python">x = torch.tensor([2, 3, 2, 1, 2, 3, 2])
unique = torch.unique(x)
weight = x.unsqueeze(0) == unique.unsqueeze(1)
torch.multinomial(weight.float(), num_samples=1)
</code></pre>
Dense layer with different inputs for each neuron

erap129: <p>Hello, I am trying to create this dense layer:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/0/063d990affbf0cf9cc4e605a1b40fe92552b316f.jpeg" alt="image" data-base62-sha1="TcPVy7NxO1CZU5ujnxZbHdH4cv" width="326" height="320"><br>
where each neuron receives as input only a portion of the previous layers (my goal is to create a learned weighted average of the previous layers).<br>
Shown below is the custom layer I created for this purpose but the network, using this layer, doesn’t seem to be learning.<br>
Can anyone point out what I got wrong here and if another solution exists for my problem? Thank you.</p>
<pre><code class="lang-auto">class LinearWeightedAvg(torch.nn.Module):
    def __init__(self, n_neurons):
        super(LinearWeightedAvg, self).__init__()
        self.neurons = []
        for neur_idx in range(n_neurons):
            self.neurons.append(torch.nn.Linear(2, 1))

    def forward(self, *input):
        outputs = []
        for neur_idx, neuron in enumerate(self.neurons):
            concat_inp = [[] for i in range(len(input[0]))]
            for inp in input:
                for example_idx, example in enumerate(inp):
                    concat_inp[example_idx].append(example[neur_idx])
            outputs.append(neuron.forward(torch.tensor(concat_inp)))
        return torch.stack(outputs).view(-1, len(self.neurons))
</code></pre>

albanD: <p>Wouldn’t a simpler (and faster) implementation be:</p>
<pre><code class="lang-auto">class LinearWeightedAvg(torch.nn.Module):
  def __init__(self, n_neurons):
    super(LinearWeightedAvg, self).__init__()
    self.weight_inp1 = torch.nn.Parameter(torch.randn(1, n_neurons))
    self.weight_inp2 = torch.nn.Parameter(torch.randn(1, n_neurons))

  def forward(self, input1, input2):
    return input1 * self.weight_inp2 + input2 * self.weight_inp2
</code></pre>
<p>Also I would be careful with the weight initialization. Here I use randn for simplicity but something else should be used for sure.<br>
Also the weight initialization of a Linear Layer depends on the sizes so might behave unexpectedly for very small numbers like you do.</p>

erap129: <aside class="quote no-group quote-modified" data-username="albanD" data-post="2" data-topic="47348">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p>class LinearWeightedAvg(torch.nn.Module): def <strong>init</strong>(self, n_neurons): super(LinearWeightedAvg, self).<strong>init</strong>() self.weight_inp1 = torch.nn.Parameter(torch.randn(1, n_neurons)) self.weight_inp2 = torch.nn.Parameter(torch.randn(1, n_neurons)) def forward(self, input1, input2): return input1 * self.weight_inp2 + input2 * self.weight_inp2</p>
</blockquote>
</aside>
<p>awesome! wasn’t sure about the usage of torch.nn.Parameter but this made it clear. Used xavier for initialization and seems to be working.</p>
Weight doesn't update even though weight.grad is not none

JS_Lee: <p>Hi I made a simple code to track nn.Linear’s parameter’s update.</p>
<pre><code class="lang-auto">import torch


optimizer = torch.optim.Adam(qwe.parameters(), lr=0.001)
optimizer.zero_grad()

qwe = nn.Linear(1,1)

xx = torch.tensor(torch.ones(1,1), requires_grad = True)
yy =qwe(xx)
zz = yy**2

zz.backward()

print('xx requires grad: ',xx.requires_grad)
print(xx.grad) 
print('yy requires grad : ',yy.requires_grad)
print(yy.grad) 
print("qwe's weight:",qwe.weight.clone())
print("qwe's grad",qwe.weight.grad)
print('zz requires grad :',zz.requires_grad)
print(zz.grad) 
optimizer.step()
print("qwe's weight:",qwe.weight.clone())
</code></pre>
<p>The result is</p>
<pre><code class="lang-auto">xx requires grad:  True
tensor([[0.5291]])
yy requires grad :  True
None
qwe's weight: tensor([[0.3074]], grad_fn=&lt;CloneBackward&gt;)
qwe's grad tensor([[1.7211]])
zz requires grad : True
None
qwe's weight: tensor([[0.3074]], grad_fn=&lt;CloneBackward&gt;)
</code></pre>
<p>You can see qwe’s grad is not none but there was no update in qwe’s weight.<br>
How can I see weight’s update ?<br>
Is it possible to catch weight update?</p>

albanD: <p>Hi,</p>
<p>I think you should give <code>qwe.parameters()</code> to the optimizer, not <code>net.parameters()</code>.</p>

JS_Lee: <p>you are right…<br>
and I should have put</p>
<pre><code class="lang-auto">qwe = nn.Linear(1,1)
</code></pre>
<p>before optimizer clause.</p>
<p>Thanks a lot for picking up my mistake</p>

weiwei: <p>hi, albanD, i have a question about share memory, can we disable share memory usage in pytorch? thank you very much.</p>

albanD: <p>Could you please open a new topic for unrelated questions please.</p>

weiwei: <p>hi, i am quite new to pytorch forum, so how can  i open a new topic for my question , i have start a question two day ago, but found no reply ? so , does my problem can not seen by you ? i will start a new topic :<br>
the topic link is show here:</p><aside class="quote" data-post="1" data-topic="47342">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/w/f9ae1b/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/can-we-disable-share-memory-usage-in-pytorch/47342">Can we disable share memory usage in pytorch?</a> 
  </div>
  <blockquote>
    since i am not able to adjust the share memory usage in the remote server, can we disable share memory usage in pytorch. the same experiment run with tensorflow without shm size problem, so i just want to find a solution for this problem.
  </blockquote>
</aside>


weiwei: <p>i have open a new topic for the problem , thanks very much !</p>
ValueError: Expected target size (4, 170), got torch.Size([4])

chrome: <p>I am running into an issue of dimension mismatch after F.log_softmax layer when calculating the error. I am using a batch size of 4.</p>
<pre><code class="lang-auto"># inside forward()
fc_classify = fc_classify.view(self.batch_size, -1, fc_classify.shape[1])  # torch.Size([1108, 170]). This is the shape of fc_classify which is a fully connected layer
# 170 is the number of labels, self.batch_size=4

scores = F.log_softmax(fc_classify, dim=1)  #torch.Size([4, 277, 170]) of scores
# 277 is the max length of each sentence

loss_classification = nn.NLLLoss()

loss = loss_classification(label_pred, target_label) # where label_pred is the output of the log_softmax function and its shape is torch.Size([4, 277, 170]) whereas the target_label shape is torch.Size([4])

</code></pre>
<p>So the issue is with the additional dimension of 277. I don’t know how to solve this issue. This was working when I didn’t use any batching.</p>
What is the difference between net.cuda and net.to('cuda')?

yunpei: <p>What is the difference between net.cuda and net.to(‘cuda’)?</p>
<p>Do someone know? Thank you</p>

albanD: <p>Hi,</p>
<p>The first one is the old notation, the second one is the new unified notation. They do the same thing.</p>

yunpei: <p>Thank you very much.</p>
Where is implementation for common autograd operator

yukino: <p>I want to know how pytorch implement autograd to ALLl operator, such as add, multiple, power and so on.</p>

albanD: <p>Hi,</p>
<p>For python functions, users define their own <code>autograd.Function</code>. I don’t think there are much of these left though in the core except for things like <a href="https://github.com/pytorch/pytorch/blob/ffdce79078438c8722eccc87d9d68396559b97e4/torch/utils/checkpoint.py#L55" rel="nofollow noopener">Checkpointing</a>.<br>
For cpp implementations in the core, the aten library contains all the cpp functions and <a href="https://github.com/pytorch/pytorch/blob/12528990f8c56deb7ce1c699e6da63d82c115968/tools/autograd/derivatives.yaml" rel="nofollow noopener">this file</a> defines the mapping between the forward and backward functions.</p>

yukino: <p>Thanks for your replies.<br>
And I would like to know where is overriding apply method for these operator</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/e87f77def637edf85aa5191f8422892415e4f5ae/torch/csrc/autograd/function.h#L39-L46" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/e87f77def637edf85aa5191f8422892415e4f5ae/torch/csrc/autograd/function.h#L39-L46" target="_blank" rel="nofollow noopener">pytorch/pytorch/blob/e87f77def637edf85aa5191f8422892415e4f5ae/torch/csrc/autograd/function.h#L39-L46</a></h4>
<pre class="onebox"><code class="lang-h"><ol class="start lines" start="39" style="counter-reset: li-counter 38 ;">
<li>///~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~</li>
<li>///                               Function</li>
<li>///~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~</li>
<li>/// A `Function` is an abstract class that represents an operation taking zero</li>
<li>/// or more input `Variable`s and producing zero or more output `Variable`s. All</li>
<li>/// functions in PyTorch's autograd machinery derive from this class and</li>
<li>/// override its `apply` method. Instances of such subclasses will then be</li>
<li>/// invokeable via the call operator.</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


albanD: <p>This is some automatically generated code based on the yaml file linked above.<br>
If you have a local install, you can find these in <code>torch/csrc/autograd/generated/Functions.cpp</code>.</p>
Tensor split accross multiple dimensions

jfm: <p>I have a 4-d (batch, channel, W, H) tensor that I’d like to split into equal sized tensor while preserving the batch and channel dimensioinality. I was wondering if there’s a better way of doing this instead of nesting two <code>torch.split</code> calls. My ultimate goal is to apply the same type of transformation to each of these chunks (this transformation is not a convolution). Maybe there’s a way to avoid the “explicit” tensor splitting and apply the transformation directly in a chunk-based fashion? This is what I’ve done:</p>
<pre><code class="lang-auto">a = torch.randn(64, 3, 16, 16)

# I'd like to split it into [64,3,4,4] chunks

chunk_dim = 4
a_x_split = torch.chunk(a, chunk_dim, dim=2)

chunks = []
for cnk in a_x_split:
    cnks = torch.chunk(cnk, chunk_dim, dim=3)
    for c_ in cnks:
        chunks.append(c_)

print(len(chunks)) # prints '16'
print(chunks[0].shape) # prints 'torch.Size([64, 3, 4, 4])'

# then apply same transform to each chunk individually
...
</code></pre>

ptrblck: <p><code>tensor.unfold</code> should yield the same result:</p>
<pre><code class="lang-python">a = torch.randn(64, 3, 16, 16)
kernel_size = 4
kernel_stride = 4
a = a.unfold(2, kernel_size, kernel_stride).unfold(3, kernel_size, kernel_stride)
a = a.contiguous().view(a.size(0), a.size(1), -1, a.size(4), a.size(5))
print(a.shape)
&gt; torch.Size([64, 3, 16, 4, 4])
</code></pre>
<p>Not that the patches are now in dim2. You can of course permute the tensor as you wish.<br>
<code>chunks[0]</code> now corresponds to <code>a[:, :, 0]</code>.</p>

jfm: <p>Thank you for the above. What would be the best way to revert this operation while perserving the internal orderingn of the data? sounds silly but I’m struggling with this o.O)</p>

windson: <aside class="quote no-group" data-username="jfm" data-post="3" data-topic="45406">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/b2d939/48.png" class="avatar"> jfm:</div>
<blockquote>
<p>orderingn of the data</p>
</blockquote>
</aside>
<p>Hi, I think in that way the internal order is preserved, no?<br>
In my case, imagine its a high-dimension image batch [8,64,64,64], I wanna crop this “image” tensor into 4 pieces, just like cropping images into 4 parts. So each partial “images” tensor will have [8,64,32,32]. I think torch.unfold is doing right, but am also not 100% sure. <a class="mention" href="/u/ptrblck">@ptrblck</a> could you please clarify that? Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>

ptrblck: <p>Have a look at <a href="https://discuss.pytorch.org/t/patch-making-does-pytorch-have-anything-to-offer/33850/11">this post</a> to see how to recreate the original input.<br>
As <a class="mention" href="/u/windson">@windson</a> said, the internal order should be preserved.</p>
Why is sourceTensor.clone() not a leaf variable?

StefanKMdA: <p>I noticed recently that sourceTensor.clone() is not a leaf variable and is part of the original computational graph and I wondered why this could be useful?</p>

sbelharbi: <p>To allow in-place operations for example.</p>
<p><a href="https://discuss.pytorch.org/t/how-to-perform-indexing-with-autograd/47225/2">This</a> for instance. Instead of adding <code>0.</code>, one can use <code>*.clone()</code> as in <a href="https://discuss.pytorch.org/t/leaf-variable-was-used-in-an-inplace-operation/308/2">here</a>.</p>

StefanKMdA: <p>Okay, thank you!<br>
Best regards,<br>
Stefan</p>
Lstm many-to-one

slavavs: <p>How do I make many-to-one so that the linear layer is connected to the last lstm block?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/3/398159349cb4c4e65d0a155d388cbf01440d6bb0.png" data-download-href="https://discuss.pytorch.org/uploads/default/398159349cb4c4e65d0a155d388cbf01440d6bb0" title="rnn7"><img src="https://discuss.pytorch.org/uploads/default/original/2X/3/398159349cb4c4e65d0a155d388cbf01440d6bb0.png" alt="rnn7" data-base62-sha1="8cIliQ7FyTiUCWUOqDFV9JUUsqk" width="690" height="299" data-dominant-color="F3F3F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">rnn7</span><span class="informations">1244×540 6.74 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Should I manually average gradients when training with multiprocessing?

bluehood: <p>Hi,<br>
I really don’t know where else to ask but here, and could not find an answer already on the forum.</p>
<p>Do optimizers work transparently in multiprocess runs or do I need to average the gradients of each process manually?</p>
<p>The <a href="https://github.com/pytorch/examples/blob/master/imagenet/main.py" rel="nofollow noopener">imagenet example</a> in the pytorch/examples repo does not do explicit gradient averaging between processes, but the <a href="https://pytorch.org/tutorials/intermediate/dist_tuto.html#distributed-training" rel="nofollow noopener">example on distributed training</a> in pytorch’s tutorials does.</p>
<p>Thanks in advance for any help!<br>
Enrico</p>
Different results of nearest interpolation

WangQi: <p>I am doing image segmentation using pytorch. I found that, for nearest interpolation, the result of nn.functional.interpolate is different from scipy.ndimage.zoom.</p>
<p>My prediction is p. And the ground truth is gt. Then p is interpolated to the same size as gt. I use nn.functional.interpolate and scipy.ndimage.zoom, respectively. Then I calculate dice or IoU. The results are different. And zoom gave me a better performance. But they are both nearest interpolation. Why?</p>
I am trying to calculate the auc value but getting error.How to resolve it?

SHASHANK_KUMAR_MISHR: <p>My code is below:-<br>
def test(dataloader,model):<br>
y_score=[]<br>
y=[]<br>
for data in dataloader:<br>
inputs, labels = data<br>
outputs = model(inputs)<br>
_, predicted = torch.max(outputs.data, 1)<br>
c = (predicted == labels).squeeze().cpu().numpy()<br>
y_score.append(outputs.cpu())<br>
y.append(labels.cpu().numpy())<br>
return y_score,y</p>
<p>y_score,y=test(test_loader,vgg)</p>
<p>b = y_score[0].detach().numpy()</p>
<p>c=np.array(y)</p>
<p>from sklearn import metrics<br>
fpr, tpr, thresholds = metrics.roc_curve(c,b)<br>
roc_auc = auc(fpr, tpr)</p>
<p>Error which I got is below:-<br>
ValueError: unknown format is not supported</p>
<p>In this b is 2class classification and c is 5class classification.How to make c of 2 class from 5 class classification</p>

ecdrid: <p>please reformat your code using the back ticks</p>

SHASHANK_KUMAR_MISHR: <p><a class="mention" href="/u/ecdrid">@ecdrid</a> Aditya Can u please let me know what do u mean by reformatting the code?</p>

ecdrid: <p>like wrap your code with the backticks ``` …<br>
Plus roc-auc isn’t defined like that for multi-class/multi-label classification…<br>
Check this out <a href="https://stats.stackexchange.com/questions/2151/how-to-plot-roc-curves-in-multiclass-classification" rel="nofollow noopener">link</a>…</p>

SHASHANK_KUMAR_MISHR: <p>Sure I will do that Thanks for ur response</p>
I got a size mismatch problem

Pattarapol_Terng: <p>RuntimeError: size mismatch, m1: [64 x 1024], m2: [2048 x 1024]<br>
this is my code.<br>
class nnModel(nn.Module):</p>
<pre><code>def __init__(self, num_classes=32):
    super(nnModel, self).__init__()
    self.para1 = resnet18()
    self.para2 = resnet18()
    self.para3 = nn.Sequential(
        nn.Conv2d(1, 8, 5),
        nn.BatchNorm2d(8),
        nn.LeakyReLU(),
        nn.MaxPool2d(3, stride=2)
    )
    self.para3_fc = nn.Sequential(
        nn.Linear(95048, 64),
        nn.BatchNorm1d(64),
        nn.LeakyReLU()
    )
    self.NN1 = nn.Sequential(
        nn.Linear(2048, 1024),
        nn.BatchNorm1d(1024),
        nn.LeakyReLU()
    )
    self.fc1 = nn.Sequential(
        nn.Linear(1024 + 64 + 5, 1024),
        nn.BatchNorm1d(1024),
        nn.LeakyReLU()
    )
    self.fc2 = nn.Sequential(
        nn.Linear(1024, 512),
        nn.BatchNorm1d(512),
        nn.LeakyReLU()
    )
    self.fc3 = nn.Sequential(
        nn.Linear(1024 + 512, 248),
        nn.BatchNorm1d(248),
        nn.LeakyReLU(),
        nn.Linear(248, num_classes)
    )

def forward(self, faces, par1, par2, orientation, height, width, cam_x, cam_y):
    x_l = self.para1(par1)
    x_r = self.para2(par2)
    x_e = torch.cat((x_l, x_r), 1)
    x_e = self.NN1(x_e)
    x_f = self.para3(faces)
    x_f = x_f.reshape(x_f.size(0), -1)
    x_f = self.para3_fc(x_f)
    x = torch.cat((x_e, x_f, orientation.unsqueeze(1), height.unsqueeze(1), width.unsqueeze(1), cam_x.unsqueeze(1),
                   cam_y.unsqueeze(1)), 1)
    x_fc1 = self.fc1(x)
    x_fc2 = self.fc2(x_fc1)
    x = self.fc3(torch.cat((x_e, x_fc2), 1))
    return x</code></pre>

ecdrid: <p>The error is very very clear; just change your Linear layers dimensions properly; plus why hard coding the dims like that?</p>
<p>(the error probably lies in this line <code>x_e = self.NN1(x_e)</code>)</p>
<p>(Check the attribute <code>nn.linear.out_features</code>) (valid for other layers as well(at least the commonly used ones))</p>
<blockquote>
<p>m1: [64 x 1024], m2: [2048 x 1024]</p>
</blockquote>
<p>this says that you are having some error in your dimension calculation and matrix multiplication can’t happen; the tip is to use the traceback and fix the dims (and it should look like <code>[64*1024], [1024*1024] or [64*2048], [2048*1024]</code>)</p>
IMDB & WIKI gender dataset not converging

Diaa_elsayed_ziada: <p>I’m working on gender detection with 3 datasets IMBD<a href="https://data.vision.ee.ethz.ch/cvl/rrothe/imdb-wiki/" rel="nofollow noopener">IMDB &amp; WIKI</a> consist of more than 500K image and <a href="http://cvit.iiit.ac.in/projects/IMFDB/" rel="nofollow noopener">IMFDB</a> about 3000 images, when using all datasets together model not converging, in case of using imfdb dataset only model reach 90% accuracy, in case of  wiki or imbd dataset model not converging, in case of using subset of wiki dataset to be same as imfdb (about 3000 from wiki) model still not converging, what is the reason for this<br>
<a href="https://colab.research.google.com/drive/1X99zL0iUQyBaNyiUvZAb8Fc_v8D5bWxK" rel="nofollow noopener">colab notebook</a></p>
Multiplying every row of a 2D tensor with every matrix in a 3D tensor

yottabytt: <pre><code class="lang-auto">'''
Let's say we have a 2D tensor A and a 3D tensor B. 

Let A.size() equals (2, 3) and B.size()  equals (2, 3, 3)

I expect the result C to be a 2D tensor with C.size() equals (2, 3). 
'''

C[0] = torch.matmul(B[0], A[0])
C[1] = torch.matmul(B[1], A[1])
</code></pre>
<p>Can I vectorize the above operation instead of looping over 0, 1, …?</p>

yottabytt: <pre><code class="lang-auto">'''
Seems I can leverage torch.bmm to get it done !! 

torch.bmm can take two tensors of (b, n, m) and (b, m, p) 
to give a tensor of (b, n, p)

So, let me put A in the proper shape to leverage torch.bmm
'''

A = torch.reshape(2, 3, 1)
C = torch.bmm(B, A).reshape(2, 3)
</code></pre>
How to fine tune a model from an MTCNN?

RickBakker: <p>I’m making a system that can detect faces from artistic paintings. I use for this the face.evoLVe repo from<br>
<a href="https://github.com/ZhaoJ9014/face.evoLVe.PyTorch" rel="nofollow noopener">https://github.com/ZhaoJ9014/face.evoLVe.PyTorch</a></p>
<p>face.evoLVe uses an MTCNN for the facial detection and works brialliantly on real world images. However, I want to fine tune it using transfer learning to work on artistic painting such as the Mona Lisa.</p>
<p>I tried the tutorial on the PyTorch website, <a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html</a></p>
<p>However, I’m not very sure on where to include the model from face.evoLVe. Since it is an MTCNN, I assume I only have to fine tune the pnet.npy.</p>
<p>For the dataset on what it should fine tune I was thinking about having a dataset with two categories: faces and not-faces.</p>
DataLoader num_workers vs torch.set_num_threads

mshuaibi: <p>Is there a difference between the parallelization that takes place between these two options? I’m assuming num_workers is solely concerned with the parallelizing the data loading. But is setting torch.set_num_threads for training in general? Trying to understand the difference between these options. Thanks!</p>

albanD: <p>Hi,</p>
<p>Your assumption is correct:</p>
<ul>
<li>num_workers will set the number of processes (EDITED thanks <a class="mention" href="/u/simonw">@SimonW</a>)   used to load and preprocess data in the dataloader</li>
<li>set_num_threads sets the number of threads that can be used to perform cpu operations like conv or mm (usually used by OpenMP or MKL).</li>
</ul>

SimonW: <p>Also, data loader’s <code>num_workers</code> specifies the number of multiprocessing workers, and has nothing to do with threads.</p>
Higher order derivatives

smu226: <p>Hello! I have the following code to take higher order derivatives (simplified example):</p>
<pre><code class="lang-auto">import torch
from torch.autograd import grad

x = torch.tensor([3.], requires_grad=True)
y = x**4
for i in range(5):   
    print(i,y)
    grads = grad(y, x, create_graph=True)[0]
    y = grads.sum()
</code></pre>
<p>This is the output:</p>
<pre><code class="lang-auto">0 tensor([81.], grad_fn=&lt;PowBackward0&gt;)
1 tensor(108., grad_fn=&lt;SumBackward0&gt;)
2 tensor(108., grad_fn=&lt;SumBackward0&gt;)
3 tensor(72., grad_fn=&lt;SumBackward0&gt;)
4 tensor(24., grad_fn=&lt;SumBackward0&gt;)
</code></pre>
<p>It does what I want, except for the fact that it prints 108 twice. Why does it do that?</p>

colesbury: <p>Because:</p>
<p>0: f(x) = (x ** 4) = 81<br>
1: f’(x) = 4 * (x ** 3) = 108<br>
2: f’’(x) = 12 * (x ** 2) = 108<br>
3: f’’’(x) = 24 * x = 72<br>
4: f’’’’(x) = 24</p>
How to put a module in different GPUs

Shengyu_Feng: <p>I’m currently training a word2vec model with a large vocabulary (50 million). When I use a embedding layer with 64 dimension, the volume of the embedding weight (64x50milliton) will exceed the memory of a single GPU, so I wanna put the weight on different GPUs.</p>
<p>I’ve already tried DataParallel but it still doesn’t work.</p>
<pre><code class="lang-auto">        self.in_embed = nn.Embedding(self.num_classes, self.embed_size, sparse=True)
        self.in_embed.weight = Parameter(t.FloatTensor(self.num_classes, self.embed_size).uniform_(-1, 1))
        self.in_embed = DataParallel(self.in_embed,device_ids=[0,1,2])
        self.in_embed.cuda()
</code></pre>

ptrblck: <p><code>DataParallel</code> won’t work in this case, as this will copy the model to each GPU and split the data in dim0 sending each chunk to the corresponding GPU.<br>
I haven’t tested this approach, but it might be possible to split the weigth matrix and push the splits onto different GPUs (similar to model sharding, but with a single layer).<br>
Here is a dummy code (currently CPU only) to test for equal results:</p>
<pre><code class="lang-python"># Reference embedding, which is too large on single GPU
emb = nn.Embedding(10, 300)
emb1 = nn.Embedding(5, 300)
emb2 = nn.Embedding(5, 300)

# Copy weigths so that we can compare both approaches
with torch.no_grad():
    emb1.weight.copy_(emb.weight[:5])
    emb2.weight.copy_(emb.weight[5:])

# Initialize some input randomly
x = torch.randint(0, 10, (3, 7))

# Split input and call corresponding embedding layer (on different GPUs)
emb1_idx = (x &lt; 5).nonzero()
emb2_idx = (x &gt;= 5).nonzero()

out1 = emb1(x[emb1_idx.split(1, 1)])
out2 = emb2(x[emb2_idx.split(1, 1)] - 5)

# Concatenate output to check for equal results
out_cat = torch.zeros(x.size(0), x.size(1), 300)
out_cat[emb1_idx.split(1, 1)] = out1
out_cat[emb2_idx.split(1, 1)] = out2

# Get reference output
out = emb(x)

# Compare outputs
print((out == out_cat).all())

# Call backward and compare gradients
out_cat.mean().backward()
out.mean().backward()

print((torch.cat((emb1.weight.grad, emb2.weight.grad), 0) == emb.weight.grad).all())
</code></pre>
<p>In your code, you would have to push the embedding “sublayers” as well as the corresponding input chunks to different devices.</p>
<p>Let me know, if I’m missing something obvious.</p>
How to reduce the overhead of conv layers?

cwan: <p>I’m trying to optimize the inference latency, but found that there’s a strange overhead happens in one single conv layer. Even though it still fits <code>y=kx+b</code> relationship, but the <code>b</code> is unacceptable large. How should I reduce such <code>b</code>? Thanks in advance!</p>
<p>The profiling result goes as follows.</p>
<div class="md-table">
<table>
<thead>
<tr>
<th>[filter, filter, input_channels, output_channels]</th>
<th>Latency</th>
</tr>
</thead>
<tbody>
<tr>
<td>[3, 3, 16, 8]</td>
<td>0.024658</td>
</tr>
<tr>
<td>[3, 3, 16, 16]</td>
<td>0.032011</td>
</tr>
<tr>
<td>[3, 3, 16, 32]</td>
<td>0.031948</td>
</tr>
<tr>
<td>[3, 3, 16, 64]</td>
<td>0.037025</td>
</tr>
<tr>
<td>[3, 3, 16, 128]</td>
<td>0.049538</td>
</tr>
<tr>
<td>[3, 3, 16, 256]</td>
<td>0.062251</td>
</tr>
<tr>
<td>[3, 3, 16, 512]</td>
<td>0.105888</td>
</tr>
</tbody>
</table>
</div><p>The code for profiling goes as follows:</p>
<pre><code class="lang-python">for i in range(7):
  shape = [3,3,16,2**(i+3)]
  kernel_value = np.random.rand(shape[0],shape[1],shape[2],shape[3]).astype(np.float32)
  kernel = torch.as_tensor(np.transpose(kernel_value, (3,2,0,1)))
  input_value = np.random.rand(1,16,32,32).astype(np.float32)
  x = torch.as_tensor(input_value)

  before = datetime.datetime.now()
  for j in range(100):
    if j==50:
      before = datetime.datetime.now()
    tmp = F.conv2d(x, weight=kernel,bias=None,stride=1,padding=(3-1)//2)
  after = datetime.datetime.now()
  interval = after-before
  print(str(shape)+"\t"+str(get_seconds(interval)))
</code></pre>

cwan: <p>I just tried it on some other machines (CPU only). The problem appears on them to.</p>
How to creat a classification model with multiple input and 2 output?

HamedH25: <p>Hi,<br>
I want to create a model with 2*7 input and it has 2 outputs which have been classified from 4096 classes. How can I do it? It is my model. Thank you in advance.</p>
<pre><code>class FeedForwardNNModel(nn.Module): 
        def __init__(self, input_size, hidden_size, num_classes): 
              super(FeedForwardNNModel, self).__init__() 
    
    
              # Linear Function
              self.fc1 = nn.Linear(2 * 7, hidden_dim)
    
             # Non-Linearity
             self.sigmoid = nn.Sigmoid()
    
             # Linear function (readout)
             self.fc2 = nn.Linear(hidden_dim, 4096)

       def forward(self, x): 
    
             # Linear function  
             out = self.fc1(x)
    
             # Non-linearity  
             out = self.sigmoid(out)
    
             # Linear function (readout)  
             out = self.fc2(out)
    
             return out</code></pre>