akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Torch equivalent of np.where

username: Karan_Chhabra
cooked: <p>Hi,</p>
<p>I am working with 4-dimensional tensors, and I am using for loop to check the conditional flow, I wanted to know how can I implement the below code efficiently</p>
<pre><code class="lang-auto">## object_temp Shape -&gt; (24, 7, 7, 30)
## Object Shape -&gt; (24, 7, 7)
## target Shape -&gt; ( 24, 7, 7, 30)

        for i in range(object.shape[0]):
          for j in range(object.shape[1]):
            for k in range(object.shape[2]):
              if ((target[i,j,k,4] &gt;0) | (target[i,j,k,9] &gt;0)):
                object[i,j,k] = 1
                object_temp[i,j,k,:] = 1
</code></pre>

username: JuanFMontesinos
cooked: <aside class="onebox allowlistedgeneric">
  <header class="source">
      <a href="https://pytorch.org/docs/stable/generated/torch.where.html?highlight=where#torch.where" target="_blank" rel="noopener nofollow ugc">pytorch.org</a>
  </header>
  <article class="onebox-body">
    <img src="" class="thumbnail" width="" height="">

<h3><a href="https://pytorch.org/docs/stable/generated/torch.where.html?highlight=where#torch.where" target="_blank" rel="noopener nofollow ugc">torch.where — PyTorch 1.6.0 documentation</a></h3>



  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


username: Karan_Chhabra
cooked: <p>It does not take in input of 1 and 0,</p>
<p>I want to create a tensor of the same dimensions as the original tensor <strong>target</strong>.</p>
<p>I can only create one less dimension tensor using torch.where. How can I also get the 4 dimension?</p>
<pre><code class="lang-auto"># Wanted object_temp Shape -&gt; ( 24, 7, 7, 30)
#target Shape -&gt; ( 24, 7, 7, 30)
</code></pre>

username: JuanFMontesinos
cooked: <p>I don’t understand at all why you need to do a nested loop. You are not permuting indices and you are always taking i,j,k in an ordered way. You can just do something like</p>
<pre><code class="lang-auto">mask = target[...,4]&gt;0 |  target[...,9]
object[mask]=1
object[mask.expand_as(object_temp)]=1
</code></pre>
Gathering batches efficiently

username: iiSeymour
cooked: <p>I have a fairly standard workflow with sequence data <em>(size (N, C, T) representing a single example)</em> where N can vary in size. I have written a keyed batching pipeline that will gather up examples so every forwards on the model will run with a fixed batch size <em>(B, C, T)</em>. Many examples might fit into a single batch or one example might span multiple batches.</p>
<p>The problem I’m having is my <code>unbatchify</code> step adds a lot of latency into the pipeline so there is significant deadtime on the gpu before the next batch is run.</p>
<p>Any recommendations to gather batches and not block?</p>
<pre><code class="lang-auto">import torch
from random import randint
from itertools import groupby
from operator import itemgetter

BATCH, CHANNELS, TIME = 64, 512, 1000

def generate_data(examples=8):
    """ create example data of various batch sizes """
    for idx in range(1, examples + 1):
        yield ('key-%s' % idx, torch.rand((randint(2, 48), CHANNELS, TIME)))

def batchify(items, batchsize, dim=0):
    """ Batch up multiple examples up to `batch_size`. """
    stack, pos = [], 0
    for k, v in items:
        breaks = range(batchsize - pos, v.shape[dim], batchsize)
        for start, end in zip([0, *breaks], [*breaks, v.shape[dim]]):
            sub_batch = v[start:end]
            stack.append(((k, (pos, pos + end - start)), sub_batch))
            if pos + end - start == batchsize:
                ks, vs = zip(*stack)
                yield ks, torch.cat(vs)
                stack, pos = [], 0
            else:
                pos += end - start
    if len(stack):
        ks, vs = zip(*stack)
        yield ks, torch.cat(vs, dim)

def unbatchify(batches, dim=0):
    """ reconstruct batches to original examples """
    batches = (
        (k, v[start:end])
        for sub_batches, v in batches
        for k, (start, end) in sub_batches
    )
    return (
        (k, torch.cat([v for (k, v) in group], dim))
        for k, group in groupby(batches, itemgetter(0))
    )

def model(data):
    """ dummy model """
    return data + 1

batches = batchify(generate_data(), batchsize=64)
results = ((key, model(data)) for key, data in batches)

for key, res in unbatchify(results):
    print(key, res.shape)
</code></pre>
How to build libtorch with cmake

username: mehditlili
cooked: <p>I am trying to build libtorch in order to get what comes in this zip <a href="https://download.pytorch.org/libtorch/nightly/cpu/libtorch-shared-with-deps-latest.zip" rel="nofollow noopener">https://download.pytorch.org/libtorch/nightly/cpu/libtorch-shared-with-deps-latest.zip</a> , mainly libs, headers and a FindCMake in order to be able to link to the libraries when building my c++ inference code.<br>
Until now I couldn’t find any tutorial or example on how to do that, I am not interested in running python scripts that build everything, I just want to create a build directory and run cmake with some flags, then make. I’ve seen it done in some other questons but details were lacking.</p>

username: jackyfaster
cooked: <p>Hi,<br>
I am interested in it, but i am confused, and no idea how to do it. Have you get advanced about it ?</p>
Ensemble loss not going down proportionally

username: annafel
cooked: <p>Hi,</p>
<p>I have implemented an ensemble consisting of 3-layer MLPs with the following architecture:</p>
<pre><code class="lang-auto">super(MLP, self).__init__()
        self.linear1 = torch.nn.Linear(D_in, H)
        self.relu1 = torch.nn.ReLU()
        self.batch1 = torch.nn.BatchNorm1d(H)
        self.hidden1 = torch.nn.Linear(H, H)
        self.relu2 = torch.nn.ReLU()
        self.batch2 = torch.nn.BatchNorm1d(H)
        self.hidden2 = torch.nn.Linear(H, H)
        self.hidden3 = torch.nn.Linear(H, D_out)
        self.logSoftMax = torch.nn.LogSoftmax(dim=1)
        self.SoftMax = torch.nn.Softmax(dim=1)
</code></pre>
<p>When testing the loss for the ensemble I don’t see the loss decreasing when the number of models is increased. For a single model, it starts out with a reasonable value and then as the ensemble increases it goes up and then down. I think it might be something wrong with how we add the predictions. This is how we are doing it:</p>
<pre><code class="lang-auto">def avg_evaluate(models):
    loss_fn = torch.nn.NLLLoss()
    loss = 0
    total = 0
    for batch_idx, (data, target) in enumerate(test_loader):
        y_preds = []
        for idx, model in enumerate(models):
            model = model.eval()
            data = data.view(data.shape[0], -1)

            y_pred, _ = model(data)
            y_preds.append(y_pred)

        loss = loss + loss_fn(torch.div(torch.stack(y_preds, dim=0).sum(dim=0), len(models)), target).item()

    print("Final loss")
    print(loss / len(test_loader))
    loss = loss / len(test_loader)
</code></pre>
<p>Does anyone have any idea of what could be wrong? Would appreciate any help! Thanks.</p>
PyTorch solution for TimeDistributedDense

username: jerin
cooked: <p>How do I realize something similar to keras TimeDistributedDense (<a href="https://github.com/fchollet/keras/issues/1029" rel="nofollow noopener">https://github.com/fchollet/keras/issues/1029</a>) in pytorch?</p>

username: chsasank
cooked: <p>Because of pytorch’s dynamic graph, you don’t need <code>TimeDistributedDense</code> like in Keras. You can just use <code>Linear</code>. LSTM networks become very straight forward. See the tutorial for pos tagging with LSTM: <a href="http://pytorch.org/tutorials/beginner/nlp/sequence_models_tutorial.html#example-an-lstm-for-part-of-speech-tagging" rel="nofollow noopener">http://pytorch.org/tutorials/beginner/nlp/sequence_models_tutorial.html#example-an-lstm-for-part-of-speech-tagging</a></p>

username: sid-ls
cooked: <pre><code class="lang-auto"># 24 fc timedistributed
num = 24
fc = nn.ModuleList([nn.Linear(8, 1) for i in range(num)])

# forward pass 
x = np.zeros(64, 24, 8)
outs=[]
for i in range(x.shape[1]):
 outs.append(fc[i](x[:, i, :].unsqueeze(1)))
outs=torch.cat(outs, axis=1)
</code></pre>

username: yongen9696
cooked: <p>Hi, could you further explain more about it?</p>
How to perform dot product along a dimension of the same batch?

username: paula_gomez_duran
cooked: <p>I have a batch = 256 of pair of vectors of dimension 32. The shape is so [256, 2, 32].</p>
<p>What I would want would be to perform the dot product between each of the pair of vectors of the batch, so that I end up with a tensor of shape [256, 32].</p>
<p>How should I do it ?</p>
<p>Thank you in advance!</p>

username: alexgo
cooked: <p>I don’t understand how you wish to end up with such dimensions. If one vector is X[:,0,:] and another is X[:,1,:], and you want to dot product them, the result should be a either a scalar, or a vector of length 256 or a vector of length 32 (if you want to perform dot product in one dimension).</p>
<p>Maybe you want elementwise product?<br>
In that case it can be achieved by<br>
<code>torch.prod(X, dim=1)</code></p>

username: KFrank
cooked: <p>Hi Paula (and Alex)!</p>
<aside class="quote no-group" data-username="paula_gomez_duran" data-post="1" data-topic="99990" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/paula_gomez_duran/48/30050_2.png" class="avatar"> paula_gomez_duran:</div>
<blockquote>
<p>What I would want would be to perform the dot product between each of the pair of vectors of the batch, so that I end up with a tensor of shape [256, 32].</p>
</blockquote>
</aside>
<p>Building on Alex’s suggestion, I believe that</p>
<pre data-code-wrap="python"><code class="lang-python">X.prod (dim = 1).sum (dim = 1)
</code></pre>
<p>should do what you want.</p>
<p>Best.</p>
<p>K. Frank</p>

username: paula_gomez_duran
cooked: <p>Yes, thank you both!  You solved my doubts.</p>
<p>Best,</p>
<p>Paula</p>
How to process two images with the same augmentation?

username: mathfinder
cooked: <p>We have a requirement that processes two images with the SAME augmentation. How to use torchvision.transforms to implement in an elegant way?</p>

username: InnovArul
cooked: <p>I am not sure about <code>torchvision.transforms</code>.<br>
<code>imgaug</code> python library may be of help to you.<br>
</p><aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="16" height="16">
      <a href="https://github.com/aleju/imgaug" target="_blank" rel="noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars2.githubusercontent.com/u/11698516?s=400&amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/aleju/imgaug" target="_blank" rel="noopener">aleju/imgaug</a></h3>

<p>Image augmentation for machine learning experiments. - aleju/imgaug</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


username: ptrblck
cooked: <p>You could use the functional API via <code>torchvision.transforms.functional</code> as seen in <a href="https://discuss.pytorch.org/t/torchvision-transfors-how-to-perform-identical-transform-on-both-image-and-target/10606/7">this example</a>.</p>

username: mathfinder
cooked: <p>Hi, thx. But I can’t clip into the link. Could you paste the URL?</p>

username: ptrblck
cooked: <p>Sorry, I had a typo in the link. Should work now.</p>
Memory gpu fill to fast

username: MehdiZouitine
cooked: <p>I have an problem : I need to store many outputs of a network in a buffer (for reinforcement learning)</p>
<pre><code class="lang-auto">for i in range 1000:
    buffer.append(network[x])
compute mean loss of the elements in the buffer
</code></pre>
<p>x = [B,C,H,W] = [64,1,30,30]<br>
network[x] is shaped like [64,1]</p>
<p>So the buffer contains tensor of very small dimension.</p>
<p>However the memory of the gpu fills up extremely fast.</p>
<p>I suspect that by adding the output to the buffer I am also adding all the computational graph.</p>
<p>So I add the graph hundreds of times and it makes my memory explode.</p>
<p>Do you have any solutions?</p>
<p>Thanks you</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="MehdiZouitine" data-post="1" data-topic="100593">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mehdizouitine/48/15365_2.png" class="avatar"> MehdiZouitine:</div>
<blockquote>
<p>I suspect that by adding the output to the buffer I am also adding all the computational graph.</p>
</blockquote>
</aside>
<p>Yes, you are storing the complete computation graphs in the <code>list</code>, if you are not detaching the tensors.</p>
<aside class="quote no-group" data-username="MehdiZouitine" data-post="1" data-topic="100593">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mehdizouitine/48/15365_2.png" class="avatar"> MehdiZouitine:</div>
<blockquote>
<p>Do you have any solutions?</p>
</blockquote>
</aside>
<p>It depends on your use case. If you need to store the computation graphs to call <code>backward</code> later, then you could reduce the number of iterations. Alternatively, if you don’t need to compute the gradients, you could store the tensors after calling <code>detach()</code> on them.</p>

username: MehdiZouitine
cooked: <p>Ok thanks,</p>
<p>I need to call backward later so I cant detach.</p>
<p>I have another question : if I have a batch of 20 then the memory used by the graph will be 2 time greater than a batch of 10 ?</p>
<p>Thank you</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="MehdiZouitine" data-post="3" data-topic="100593">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mehdizouitine/48/15365_2.png" class="avatar"> MehdiZouitine:</div>
<blockquote>
<p>if I have a batch of 20 then the memory used by the graph will be 2 time greater than a batch of 10 ?</p>
</blockquote>
</aside>
<p>More of less. You would have to take e.g. memory fragmentation into consideration. Also, if you are using cudnn with <code>benchmark=True</code>, different algorithms might be picked for different batch sizes depending on their speed, so you might end up with a different memory footprint.</p>
Using only subset of class labels for training

username: eyeris
cooked: <p>Hi all, I have an issue related to following error. The cause for the error is known but I cannot find a workaround as I need to run the algorithm that way.</p>
<pre><code class="lang-auto">RuntimeError: cuda runtime error (59) : device-side assert triggered at /tmp/pip-req-build-4baxydiv/aten/src/THCUNN/generic/ClassNLLCriterion.cu:110
</code></pre>
<p>I have a large set of classes in a dataset and I want to train the model with only a subset at a time. Lets say I have 12 classes [0-11] and classifier’s last layer output shape is 5. With data subset with data labels [0,1,2,3,4], model works perfectly. However, it throws the above error for following class label combinations.</p>
<p>[1,2,3,4,5]<br>
[2,5,6,8,11]</p>
<p>I am using <code>cross_entropy</code> criterion and last layer is a linera layer. I am fairly new to Pytorch and also searched some solutions in the forum so cound’t find anything related to my requirement. Appreciate any help that can be provided.</p>

username: ptrblck
cooked: <p>The error is expected in this case, since <code>nn.CrossEntropyLoss</code> expects targets in the range <code>[0, nb_classes-1]</code> in order to use it as indices to calculate the loss.</p>
<p>If you want to train the model with different subsets, your model should most likely still output the logits for all 12 classes, shouldn’t it?<br>
I guess your current approach wants to map the new labels to the 5 outputs of your model, which would mix different targets for the same class output of the model so I’m unsure if that’s really what you want.</p>
Cuda Out of Memory with Esrgan

username: Niels_Gielingh
cooked: <p>Hey guys!<br>
I’m quite new with Python and Pytorch but installed it all through tutorials.<br>
I use Esrgan in combination with image ImageEnhancingUtility.<br>
Because I want to upscale my old cartoons!<br>
Every worked fine but since a few days ago I got the Runtime Error</p>
<p>I’m really a noob in Python that’s why I ask you guys to help!</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/c/ccec140ae3b05b0fefe228b46bc060b73f6c499b.png" data-download-href="https://discuss.pytorch.org/uploads/default/ccec140ae3b05b0fefe228b46bc060b73f6c499b" title="Schermopname (70)"><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/c/ccec140ae3b05b0fefe228b46bc060b73f6c499b.png" alt="Schermopname (70)" data-base62-sha1="tePcmpi3VZwCtneOm8hTkBxQHUf" width="690" height="361" data-dominant-color="0D0D0D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Schermopname (70)</span><span class="informations">977×512 33.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: Niels_Gielingh
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/f/cfbfa7bc87001dbda394c57086f312ed2597fc11.png" data-download-href="https://discuss.pytorch.org/uploads/default/cfbfa7bc87001dbda394c57086f312ed2597fc11" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/f/cfbfa7bc87001dbda394c57086f312ed2597fc11.png" alt="image" data-base62-sha1="tDPsg5FjolMsyLw9LaYkCLLipb3" width="690" height="360" data-dominant-color="0A0A0A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">979×512 16.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>Your current code is trying to allocate too much memory and is thus raising this error.<br>
Did you change anything in the model, such as the spatial size of the input or the batch size?</p>
<p>A workaround is to reduce the memory usage so that the workload can fit in your 6GB GPU e.g. by reducing the batch size or by using <code>torch.utils.checkpoint</code> to trade compute for memory.</p>
Training with data parallel , validation on 1 gpu

username: Btorch
cooked: <p>Hi, I’m using hooks for validation, and with data parallel only the packet of gpu=0 returns, I saw that there is not advance with that issue.<br>
So I thought on a hack, I will train with data-parallel but when doing validation I’ll remove the data parallel, and send the model to 1 gpu only.<br>
No matter what I tried to do it did not succeed, “to(‘cuda:0’)”, sending it to cpu, no matter what it returns half packet.<br>
So now to my question, how can I take a model with data-parallel wrapper, and remove the data-parallel wrapper of him?!!?</p>

username: ptrblck
cooked: <p>After you’ve wrapped the model into <code>nn.DataParallel</code> you can get the non-parallel model back via:</p>
<pre><code class="lang-python">model = model.module
</code></pre>
<p>Let me know, if you are still seeing any issues.</p>
How to build a convolutional recurrent model in pytorch?

username: zeyilain
cooked: <p>Hi,<br>
I have no idea how to build a recurrent layer which looks like this and the autograd can run smoothly.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/d/2dea838847760a1fe746939a871cf1894235bed6.png" data-download-href="https://discuss.pytorch.org/uploads/default/2dea838847760a1fe746939a871cf1894235bed6" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/d/2dea838847760a1fe746939a871cf1894235bed6.png" alt="image" data-base62-sha1="6ybVDDlw1Rzzg38sZo2WtLMR7ZY" width="690" height="244" data-dominant-color="F2F2F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1409×499 13.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><strong>x</strong> is a 1-d tensor create by previous layer, in each time step <strong>t</strong>, the kernel will stride one step and do convolution with <strong>x</strong> until <code>time_step = len(x)</code>, the result of the convolution becomes one value and add with <strong>xt</strong>, <strong>xt</strong> should be update by each time step, then becomes the input of next time step.<br>
After the convolutional recurrent layer, for the autograd part, generate data x and real data only use L1loss (which means <code>real data - x</code>).</p>
<p>Thanks in advance,<br>
Zeyilain</p>
How to extract weight/bias tensor from torch._C.Graph?

username: icyhearts
cooked: <p>I want to load a model saved by torch.jit.save and then convert the model to other deep learning framework.<br>
It is necessary to extract graph and weight/bias and other tensors from graph.</p>
<p>I only know that the weith/bias tensor is stored in <code>state_dict</code>.<br>
However, in pytorch 1.5.0, the order of <code>torch._C.Value</code> in graph.inputs() and the order of <code>state_dict</code> keys are not same.<br>
For example, for mobilenetv2 model in torchvision, the first few lines of its graph generated by torch.jit is:</p>
<pre><code class="lang-auto">graph(%input.10 : Float(10, 3, 224, 224),
      %1 : Float(1280, 320, 1, 1),
      %2 : Float(1280),
      %3 : Float(1280),
      %4 : Float(1280),
      %5 : Float(1280),
      %6 : Float(160, 960, 1, 1),
      %7 : Float(160),
      %8 : Float(160),
      %9 : Float(160)
</code></pre>
<p>but the first few keys of state_dict is</p>
<p><code>['features.0.0.weight', 'features.0.1.weight', 'features.0.1.bias', 'features.0.1.running_mean', 'features.0.1.running_var'] </code></p>
<p>and I have</p>
<pre><code class="lang-auto">model.state_dict()[list(model.state_dict().keys())[0]].shape

torch.Size([32, 3, 3, 3])

model.state_dict()[list(model.state_dict().keys())[1]].shape

torch.Size([32])

model.state_dict()[list(model.state_dict().keys())[2]].shape

torch.Size([32])
</code></pre>
<p>It is obvious that the order in graph.inputs() and order in state_dict are not the same.<br>
So, is there any method to correctly extract the Tenso/weight/bias values from <code>torch._C.Graph</code>?</p>
Covert Conv-TasNet from torch to onnx successfully, but not work

username: oneway234
cooked: <p>I coverted Conv-TasNet from torch to onnx. Follow steps from website (Pytoch onnx). Then used $“testing.assert_almost_equal” to verify output. The orignal TasNet pre-trained model worked, however the result of onnx are zeros. Its weights seems to be all zero, and the output dimension also different.  I cann’t figure which step was wrong.</p>
<p>Here is my <a href="https://github.com/oneway234/torch_to_onnx/blob/main/README.md" rel="noopener nofollow ugc">project</a>.<br>
Run <a href="https://github.com/oneway234/torch_to_onnx/blob/main/demo_tasnet.py" rel="noopener nofollow ugc">demo_tasnet.py</a> onnx will be save in checkpoints.<br>
Also write a test.py to campare result.</p>

username: oneway234
cooked: <p>Hi,<br>
I tried to figure out the problem. So I checked the code again.</p>
<blockquote>
<p>torch.onnx.export(model, input,<br>
“.path/to/conv_tasnet.onnx”,<br>
export_params=True<br>
)</p>
</blockquote>
<p>I set up export_params=True, and test .onnx with onnxruntime the ouput are still all zeros. It seems that the weights are not be saved.</p>
Extracting tensor from tuple

username: catlady
cooked: <p>I am trying to extract tensor data from the MNIST data set. Essentially I want to remove the data labels from the tuple. The standard ConcatDataset is set up using the following:</p>
<p>def <strong>init</strong>(self, *datasets):<br>
self.datasets = datasets</p>
<p>def <strong>getitem</strong>(self, i):<br>
return tuple(d[i] for d in self.datasets)</p>
<p>def <strong>len</strong>(self):<br>
return min(len(d) for d in self.datasets)</p>
<p>The issue is I’m not sure how to use indexing to extract the tensors from d[i] to remove the labels. d[i] has structure of tuple((tensor(), label),(tensor(), label)), so I should be able to extract the tensors and append to a new tuple to get rid of the labels, but this has tripped me up a bit. Any help would be appreciated.</p>
How is a Conv1d with groups=1 different from a Linear layer?

username: shamoons
cooked: <p>If I have:</p>
<pre><code class="lang-auto">self.layer1 = torch.nn.Conv1d(in_channels=512, out_channels=512, kernel_size=1)
</code></pre>
<p>isn’t that equivalent to</p>
<pre><code class="lang-auto">self.layer1 = torch.nn.Linear(512, 512)
</code></pre>
<p>?</p>

username: ptrblck
cooked: <p>Yes, should be the case:</p>
<pre><code class="lang-python"># Setup
conv = torch.nn.Conv1d(in_channels=512, out_channels=512, kernel_size=1).double()
lin = torch.nn.Linear(512, 512).double()

# use same param values
with torch.no_grad():
    lin.weight = nn.Parameter(conv.weight.squeeze(2))
    lin.bias = nn.Parameter(conv.bias)

# forward
x = torch.randn(2, 512, 20).double()
out_conv = conv(x)

# permute for linear
x_lin = x.permute(0, 2, 1)
out_lin = lin(x_lin)

# check forward output
print(torch.allclose(out_lin.permute(0, 2, 1), out_conv))
&gt; True

print((out_lin.permute(0, 2, 1) - out_conv).abs().max())
&gt; tensor(1.2212e-15, dtype=torch.float64, grad_fn=&lt;MaxBackward1&gt;)

# check backward
out_conv.mean().backward()
out_lin.mean().backward()
print(torch.allclose(conv.weight.grad.squeeze(2), lin.weight.grad))
&gt; True
print(torch.allclose(conv.bias.grad, lin.bias.grad))
&gt; True
</code></pre>

username: shamoons
cooked: <p>Thanks so much. So there’s literally no difference, not even in terms of computation?</p>

username: ptrblck
cooked: <p>There is most likely a difference in computation in particular if you are using CUDA operations. E.g. convolutions would be dispatched to cudnn, if you are using an NVIDIA GPU, which could internally call into cublas (same as in the linear layer), but isn’t guaranteed.<br>
I don’t know, which methods are exactly called on the CPU.</p>
<p>For my code snippet the convolution would use <code>cudnn::cnn::implicit_convolve_dgemm</code>, while the linear layer would call into <code>volta_dgemm_128x64_tn</code>.</p>
Using libtorch on an embedded environment

username: AustinBH
cooked: <p>I need to export a an MLP network that I wrote in python and get it working on an msp432. I have setup the include paths and the libraries but to no avail. I keep getting the error that namespace “std” has no member “mutex” and other similar errors. Has anyone gotten libtorch working in code composer studio by TI? I am use version 10.0 and c++ 14.</p>
How DDP allocate CPUs

username: zzzf
cooked: <p>Suppose I have a machine with 8 GPUs and 64 CPUs.</p>
<p>Using DistributedDataParallel, it will run 8 processes and each process uses a single GPU. I’m wondering how about CPUs? Are they evenly distributed across the 8 processes? Can we specify which process use how many CPUs?</p>

username: mrshenli
cooked: <p>Hey <a class="mention" href="/u/zzzf">@zzzf</a>, DDP does not do any specific thing to allocate CPUs.</p>
<p>cc <a class="mention" href="/u/ptrblck">@ptrblck</a> is it possible to pin CPU affinity for a PyTorch process</p>

username: ptrblck
cooked: <p>I’ve seen approaches to set the CPU affinity for a GPU device using <code>nvml</code> as described <a href="https://discuss.pytorch.org/t/why-does-pytorch-only-find-one-physical-cpu/92977/5">here</a>.<br>
However, I don’t know if and how this approach would work for a general PyTorch process and if you would benefit from it.</p>

username: mrshenli
cooked: <p><a class="mention" href="/u/pritamdamania87">@pritamdamania87</a> also suggested <code>os.sched_setaffinity </code>.</p>
<p><a class="mention" href="/u/zzzf">@zzzf</a> please let us know if these solutions would help. Thx!</p>

username: zzzf
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
<a class="mention" href="/u/mrshenli">@mrshenli</a></p>
<p>Thanks for your reply. These are very helpful!</p>

username: zzzf
cooked: <p>Hi Shen, could you please also take a look at my latest post which is relevant to your tutorial: <a href="https://discuss.pytorch.org/t/distributeddataparallel-model-weights-and-grads-not-synchronized-with-multiple-forward-backward-pass/100441" class="inline-onebox">DistributedDataParallel: model weights and grads not synchronized with multiple forward backward pass</a>? Thanks!</p>

username: mrshenli
cooked: <p>yep, commented there. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>DDP was supposed to be used with alternating fw and bw passes. I am a little surprised that it didn’t throw any error. Please let us know the version of PyTorch you are using, we might have recently accidentally disabled the check for some code paths.</p>
Pytorch Indexing is very slow

username: Dzhange
cooked: <p>Hi,<br>
Recently in my project, I needed to lift a point cloud of shape  <code>(n, 3)</code> from an Image of shape <code>(3, H, W)</code>, whose RGB color represents the spatial coordinates. I also have a mask with shape <code>(240, 320)</code> to indicate which points I need. I used an indexing operation to lift it:</p>
<pre><code class="lang-auto">masked = mask &gt; threshold
point_cloud = img[:, masked].transpose(0, 1)
</code></pre>
<p>However, this operation is very slow, took nearly 0.1 seconds. Is there any way to improve the performance here?</p>
<p>Thanks!</p>

username: ChrisLiu2
cooked: <p>I don’t know why exactly it’s slow, but did you try indexing on different devices? For many of my purposes indexing on GPU is faster than on CPU</p>

username: Dzhange
cooked: <p>I’m working on a tesla V100, so I think may it’s not the problem with the device? and the <code>masked</code> would contain like 10k index.</p>
About the error "auto" not allowed in function prototype in libtorch

username: Chen0729
cooked: <p>Hi, All</p>
<p>I have met an issue in libtorch when I use auto as the type for unlabel dataset as the input argument of the function.<br>
The error is " “auto” not allowed in function prototype"</p>
<p>The context is:</p>
<p>function1(auto &amp;unlabel)<br>
{<br>
unlabel=Unlabelset();<br>
}<br>
torch::Tensor Unlabelset(){<br>
auto raw_dataset= torch::data::datasets::MNIST("…/data", torch::data::datasets::MNIST::Mode::kTrain).map(torch::data::transforms::Normalize&lt;&gt;(0.13707,0.3081)).map(torch::data::transforms::Stack&lt;&gt;());<br>
}</p>
<p>Another issue is that it seems that the function Unlabelset() is not recognized even if I define it explicitly. Any comments or suggestions ?</p>
<p>Many thanks.</p>
Weight features during data prep

username: THEblob
cooked: <p>Hello there,</p>
<p>I was wondering if a network is able to recognize small difference within a feature.<br>
Let’s assume this feature is the only feature for a binary problem, it’s the numeric representation of occurrence of this feature in the original data.<br>
To be labeled as A it needs to be 0 everything else would be B.<br>
But also B just ranges from 1 to 5 for example. So the difference between A and B<br>
could just be 1. Is a differentiation between these two still possible or would it be better<br>
to put a weight to every occurrence of this feature so that there is a more distinct difference between the two ?</p>
How do I install Captum on Google Colab from a custom branch in a fork?

username: ProGamerGov
cooked: <p>I’ve tried cloning and running this along with all the other tricks I could find online with no luck:</p>
<pre><code class="lang-auto">#git clone https://github.com/pytorch/captum.git
#cd captum
#pip install -e .

git clone https://github.com/UserName/captum
cd captum
git checkout custom-branch
pip install -e .
</code></pre>
<p>If I use the above code, I can import captum but can’t import any submodules.</p>
<p>Any suggestions?</p>

username: ptrblck
cooked: <p>Are you able to build captum from master and import submodules?<br>
If so, I guess your custom branch might be broken.</p>

username: ProGamerGov
cooked: <p>I just tested installing the latest release of Captum and it seems to have worked:</p>
<pre><code class="lang-auto">!git clone https://github.com/pytorch/captum 

%cd captum

!git checkout "v0.2.0"

!pip3 install -e .

import sys
sys.path.append('/content/captum')

%cd ..
</code></pre>
<p>Testing the imports:</p>
<pre><code class="lang-auto">import captum

from captum.attr import (
    GradientShap,
    DeepLift,
    DeepLiftShap,
    IntegratedGradients,
    LayerConductance,
    NeuronConductance,
    NoiseTunnel,
)

from captum import custom_module # This fails
</code></pre>
<p>The master branch seems to install correctly as well, but my custom branch still does not. I haven’t changed anything on my custom branch relating to the submodule organization or setup.py, so I don’t understand why I can’t import the one submodule?</p>
<p><strong>Edit:</strong> The import in the custom module’s <code>__init__.py</code> were changed from relative to absolute, and that looks like it might be the cause.</p>
How to do manual update on parameters after optimizer?

username: Jacfger
cooked: <p>I am new to PyTorch and had little experience on this library.</p>
<p>So I was trying to replicate the result of Deep dream and I found a implementation in GitHub. That works beautifully. However, I read some sources saying I should try to use gaussian filter on training as well (as a “regularization” method) and things started to fall apart after that.</p>
<pre><code class="lang-auto">def getFilter():
    ### some code here to generate a filter content
    gaussian_filter = nn.Conv2d(in_channels=channels, out_channels=channels, kernel_size=kernel_size, groups=channels, bias=False, padding=int(kernel_size / 2))

    gaussian_filter.weight.data = gaussian_kernel
    gaussian_filter.weight.requires_grad = False
    return gaussian_filter

processed_image = getImage(input_path)
optimizer = SGD([processed_image], lr=12, weight_decay=1e-4)
filter_ = getFilter().cuda()
for i in range(1, steps + 1):
    optimizer.zero_grad()
    out = model(processed_image)
    loss = # skipped the calculation
    loss.backward()
    optimizer.step()
    processed_image = filter_(processed_image)
    print(loss)
</code></pre>
<p>The <code>print(loss)</code> in the function is quite important here because this program will run this line exactly twice. Then, the program will report “RuntimeError: Trying to backward through the graph a second time, but the saved intermediate results have already been freed. Specify<br>
retain_graph=True when calling backward the first time.” in the loss.backward().</p>
<p>The program would run perfectly fine without <code>processed_image = filter_(processed_image)</code>.</p>
<p>Does anyone know how to avoid the error?</p>
Custom CPU binaries for Intel Knights Landing for Pytorch

username: Brando_Miranda
cooked: <p>Are there optimized pytorch binaries for hardware using Intel’s Knights Landing CPUs?</p>
<p>related:</p>
<ul>
<li><a href="https://github.com/pytorch/pytorch/issues/32909" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/issues/32909</a></li>
<li><a href="https://www.quora.com/unanswered/How-does-one-install-custom-binaries-for-the-Knights-Landing-CPU-architecture-for-PyTorch" rel="noopener nofollow ugc">https://www.quora.com/unanswered/How-does-one-install-custom-binaries-for-the-Knights-Landing-CPU-architecture-for-PyTorch</a></li>
<li><a href="https://www.reddit.com/r/pytorch/comments/jhsy7w/how_does_one_install_custom_binaries_for_the/" rel="noopener nofollow ugc">https://www.reddit.com/r/pytorch/comments/jhsy7w/how_does_one_install_custom_binaries_for_the/</a></li>
</ul>
What architecture would I use to go from n 512-dimensional vectors to a n*400 vector?

username: shamoons
cooked: <p>I am trying to invert <code>wav2vec 2.0</code> and it looks like it takes 400 samples and converts to a 512-dimensional vector. I’m having a hard time figuring out how to invert it. I tried doing a straight mapping from 512 =&gt; 400, but it doesn’t give great results, even when overfitting to a handful of samples. I think it’s because I need to include more temporal information.</p>
<p>So if 400 samples (25ms) converts to a single 512-dim vector, then 1200 samples (75ms) will give me 3 512-dim vectors.</p>
<p>How can I take those 3 vectors and convert back to 1200 samples? What architecture would you recommend?</p>