akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
TensorFlow tf.losses.softmax_cross_entropy() equivalence in PyTorch

username: dllacer
cooked: <p>Hi everyone!</p>
<p>I’m trying to reproduce a model originally programmed in TensorFlow, but I’m using PyTorch.</p>
<p>My doubt is: Which is the equivalent TF softmax_cross_entropy() function in PyTorch?</p>
<p>Thanks in advance!</p>

username: KFrank
cooked: <p>Hi Deep!</p>
<aside class="quote no-group" data-username="dllacer" data-post="1" data-topic="91206" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/50afbb/48.png" class="avatar"> dllacer:</div>
<blockquote>
<p>Which is the equivalent TF softmax_cross_entropy() function in PyTorch?</p>
</blockquote>
</aside>
<p>In short, there is no <em>direct</em> equivalent, so you have to write your own.</p>
<p>A quick glance at the tensorflow documentation suggests that<br>
<code>tf.losses.softmax_cross_entropy()</code> has been deprecated<br>
in favor of <code>tf.nn.softmax_cross_entropy_with_logits()</code>,<br>
but that both of these take <code>labels</code> of shape <code>[nBatch, nClass]</code><br>
that are probabilities (sometimes called “soft labels”).</p>
<p>In contrast, pytorch’s <code>torch.nn.CrossEntropyLoss</code> (or its function<br>
version <code>torch.nn.functional.cross_entropy()</code>) takes integer<br>
class labels of shape <code>[nBatch]</code>.</p>
<p>(Both the tensorflow and pytorch versions take <em>logits</em>, rather than<br>
probabilities, for the predictions you pass is, so that part’s the same.)</p>
<p>If your problem uses, in fact, integer class labels, just use pytorch’s<br>
<code>CrossEntropyLoss</code>.  But if your labels are probabilities, then you<br>
can write your own “soft-label” version, as described here:</p>
<aside class="quote quote-modified" data-post="2" data-topic="69501">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/soft-cross-entropy-loss-tf-has-it-does-pytorch-have-it/69501/2">Soft Cross Entropy Loss (TF has it does Pytorch have it)</a> 
  </div>
  <blockquote>
    Hello Raaj! 

I do not believe that pytorch has a “soft” cross-entropy function built in. 
But you can implement it using pytorch tensor operations, so you should 
get the full benefit of autograd and gpu acceleration. 
See this (pytorch version 0.3.0) script: 
import torch
torch.__version__

# define "soft" cross-entropy with pytorch tensor operations
def softXEnt (input, target):
    logprobs = torch.nn.functional.log_softmax (input, dim = 1)
    return  -(target * logprobs).sum() / input.shap…
  </blockquote>
</aside>

<p>Good luck.</p>
<p>K. Frank</p>
Trainig a conditional GAN - UNet

username: Alex_Dima
cooked: <p>Hello everyone! Continuing my tests with UNet,  I am trying to create a GAN similar to pix2pix. So, I am using a UNet as generator and a discriminator with the structure usual to conditional GANs. While training, I noticed that the losses remain around the same numbers. When I had a small dataset (20 samples) the generator loss would go down to about 0.7 and the results were surpsisingly not too bad. But now that I added more samples, it’s not even doing that.<img src="https://discuss.pytorch.org/uploads/default/original/3X/a/8/a85f5cd7a6f11a6875dc54f9c02be54f6b063346.png" alt="ganA_25_500" data-base62-sha1="o1uwLXb1Q7jWo9jU1cOxfC24cYe" width="321" height="252">  (this is with 20 samples and 500 epochs, the right being the predicted image).<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/e/a/ea2868c76138f7ad3d770d8dc3d2d7b04a274c04.png" alt="40 samples_400 epoch" data-base62-sha1="xpsiSRp8NSUlaqjfwOETW0ZUsqU" width="372" height="264">  (these are the losses with the new dataset)<br>
I also tried to use a scheduler, in case it is a learning rate problem, but I ’ m not quite sure how to control it. I know my question is quite broad, but if you have any suggestions I would be happy to try them. Also this is a link to my notebook and samples, if you want to have a look.<br>
<a href="https://drive.google.com/drive/folders/1tACWs1eUfdggATAkeMfKr6M-OQsYiSpW?usp=sharing" class="onebox" target="_blank" rel="noopener nofollow ugc">https://drive.google.com/drive/folders/1tACWs1eUfdggATAkeMfKr6M-OQsYiSpW?usp=sharing</a></p>
Pytorch equivalent for np.multiply.at

username: zzhu
cooked: <p>Hi,</p>
<p>I’m looking for pytorch equivalent for np.multiply.at but couldn’t find any option. I found the “index_add_” similar as np.add.at but nothing for multiply. Wondering if there is a efficient way to do select an index tensor (may have same index) from a base tensor and then multiply with another tensor.</p>
<p>In Numpy, this is how I achieve it:</p>
<pre><code class="lang-auto">base = torch.ones((2, 3, 5, 5))
to_multiply = torch.arange(120).view(2, 3, 4, 5)
index = torch.tensor([[0, 2, 4, 2], [0, 3, 3, 2]])
np.multiply.at(base1, (np.arange(2)[:,None,None],np.arange(3)[:,None],index[:,None,:]), to_multiply)
</code></pre>
<p>Thanks!</p>

username: aluo-x
cooked: <p>Would <a href="https://github.com/rusty1s/pytorch_sparse#sparse-dense-matrix-multiplication" rel="nofollow noopener">this library</a> work? I think spmm supports gradients, but <a href="https://github.com/rusty1s/pytorch_sparse/issues/45" rel="nofollow noopener">spspmm does not</a>. Might want to double check.</p>

username: zzhu
cooked: <p><a class="mention" href="/u/aluo-x">@aluo-x</a> thanks! though it doesn’t seem to fit my needs, but when searching for similar packages I found torch_scatter.scatter_mul and it seems to work: <a href="https://pytorch-scatter.readthedocs.io/en/1.4.0/functions/mul.html" rel="nofollow noopener">https://pytorch-scatter.readthedocs.io/en/1.4.0/functions/mul.html</a></p>
"AttributeError: 'ReLU' object has no attribute 'threshold' "

username: bemorept
cooked: <p>Hi, All.</p>
<p>I have met this error when run pytorch inference in Jetson nano.</p>
<p>"AttributeError: ‘ReLU’ object has no attribute ‘threshold’ ".</p>
<p>What wrong to me?</p>
<p>Jetson nano, Ubuntu 18.04LTS aarch64, Pytorch 1.1.0, Torchvision 0.3.0</p>
<p>Thanks.<br>
<span class="mention">@bemoregt</span>.</p>

username: ptrblck
cooked: <p>Could you post the code snippet which raises this error?</p>

username: lianzhang132
cooked: <p>Have you solved it?I also ran into it when I called the model</p>

username: arnabsinha
cooked: <p>I ran into the issue when I am running the following code using the export.pkl model I had saved in my directory:</p>
<pre><code class="lang-auto">import fastai
from fastai.vision import *

def main():

    classes = ['Potato___Late_blight',
               'Cherry_(including_sour)___healthy',
               'Apple___Black_rot',
               'Apple___Apple_scab',
               'Cherry_(including_sour)___Powdery_mildew',
               'Potato___healthy',
               'Apple___Cedar_apple_rust',
               'Potato___Early_blight',
               'Apple___healthy']

    #load predictor
    predictor = load_learner('.')
    img = open_image('apple_black_rot1.jpg')
    pred,pred_idx,probs = predictor.predict(img)

    print(classes[pred_idx])

if __name__=="__main__":
    main()
</code></pre>
<p>The same error pops up along with various warnings. Is there any fix to this yet?</p>
Data Augmentation for Audiofiles

username: Kla
cooked: <p>Hello everyone,</p>
<p>I’m trying to make a data augmentation for my audio DNN but unfortunately I’m not finding the right solution. I have found only torch.nn.Transformer but it seems to be only for photos. Can someone please help me?</p>

username: smth
cooked: <p>have a look at TorchAudio, which has a lot of audio preprocessing facilities:</p>
<ul>
<li><a href="https://github.com/pytorch/audio">https://github.com/pytorch/audio</a></li>
<li><a href="https://pytorch.org/audio/">https://pytorch.org/audio/</a></li>
</ul>

username: Kla
cooked: <p>Thank you <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Explain the concept behind this line. I guess it's adding a 3D tensor!

username: Blizzard_boi2020
cooked: <pre><code>x = x.float()
X= torch.sum(((x[:, 0, :, :-1] - x[:, 0, :, 1:])**2 + (x[:, 1, :, :-1] - x[:, 1, :, 1:])**2)**0.5)</code></pre>

username: ElToto
cooked: <p>We need more code and context for a better explanation but what is happening in this line:</p>
<p>x is a tensor with dimesnion = 4 (if it’s an image we have batchsize, channels, widht,height)</p>
<pre><code>(x[:, 0, :, :-1] - x[:, 0, :, 1:])**2
</code></pre>
<p>this subtracts x[all, 0, all, from 0 to the second last index) with x[all, 0, all from 1 to the last index] and then uses the power of two.</p>
<p>The same is happening here</p>
<pre><code class="lang-auto">x[:, 1, :, :-1] - x[:, 1, :, 1:])**2
</code></pre>
<p>But now we use in the second dimension the second index</p>
<p>All of that is now used with the power of 0.5 (this is the square root) and then all calculated values are summed.</p>
<p>In the end you have only one value</p>

username: Blizzard_boi2020
cooked: <p>Also can you please explain:-</p>
<p>What’s the meaning? (x[:,:,0,:,:].shape[-2])</p>

username: ElToto
cooked: <p>Sure.</p>
<p>x is a tensor with 5 dimensions.<br>
We take the zero index in third dimension so x is now a tensor with 4 dimensions.</p>
<p>tensor.shape returns the size of each 4 dimensions as a list --&gt; [dim1 size, dim2 size, dim3 size, dim4 size] if we use shape[-2] we take the list but only until the second last index so we get [dim1 size, dim2 size]</p>

username: Hmrishav_Bandyopadhy
cooked: <p>Hi,</p>
<p>You can look <a href="https://numpy.org/doc/stable/reference/arrays.indexing.html" rel="nofollow noopener">this</a> up for a better explanation on slicing and indexing. Although the implementation is in numpy, it’s pretty much the same for PyTorch.</p>
Unable to do next(model.parameters()) with Pyro models

username: Hyunjin_Cho
cooked: <p>Hello,</p>
<p>I am trying to convert <code>myPyTorchModel</code> into a Bayesian <code>Pyro</code> model (see code below).<br>
However, when I execute the line <code>next(myPyTorchModel.parameters()</code>, <code>StopIteration</code> error pops up. I am wondering if there is any way to prevent <code>Pyro</code> models from generating the <code>StopIteration</code> error.</p>
<pre><code class="lang-auto">for m in myPyTorchModel.modules():

    for name, value in list(m.named_parameters(recurse=False)):

        options =  dict(dtype=torch.double, device="cpu")
        prior_loc = torch.zeros(1, 1, **options)
        prior_scale = torch.ones(1, 1, **options)
        zs = module.PyroSample(dist.Normal(prior_loc, prior_scale).to_event(1))
        zs.double()
        setattr(m, name, zs)
        
# generates an error
next(myPyTorchModel.parameters()).dtype

OUT:
Traceback (most recent call last):

  File "&lt;ipython-input-37-f9ae77a21ef4&gt;", line 1, in &lt;module&gt;
    next(myPyTorchModel.parameters()).dtype

StopIteration
</code></pre>
<p>How can I fix this error?</p>
<p>Thanks,</p>

username: tom
cooked: <p>Apparently your myPyTorchModel doesn’t have parameters.<br>
Also, if you make things double, you expect to have double dtype, no?</p>

username: Hyunjin_Cho
cooked: <p>Hello,</p>
<p>Thank you for your reply. If <code>myPyTorchModel.parameters()</code> happens to be an empty generator, would it be okay for me to add a fake parameter onto the generator  <code>myPyTorchModel.parameters()</code> just so that I wouldn’t get the <code>StopIteration</code> error?</p>
<p>If yes, how can I add a fake parameter onto the generator  <code>myPyTorchModel.parameters()</code>?<br>
:s thank you,</p>

username: tom
cooked: <p><code>myPyTorchModel = torch.nn.Parameter(torch.zeros(1))</code> or so</p>

username: Hyunjin_Cho
cooked: <p>Hello,</p>
<p>Thank you again for your reply.</p>
<p>So <code>myPyTorchModel</code> currently has no parameters.</p>
<p>When I execute the code below, the code generates an error:</p>
<pre><code class="lang-auto"># add a fake parameter to prevent the StopIteration error        
myPyTorchModel = torch.nn.Parameter(torch.tensor([0.], requires_grad=True,
                               dtype=torch.float64))

loss = myPyTorchModel(input_ids = input, labels = labels)[0]
OUT:
Traceback (most recent call last):
File "&lt;ipython-input-20-52d7dadcb74d&gt;", line 1, in &lt;module&gt;
    mc_loss = model(input_ids = input, labels = labels)[0]

TypeError: 'Parameter' object is not callable
</code></pre>
<p>The error above is generated because <code>myPyTorchModel = torch.nn.Parameter(torch.tensor([0.], requires_grad=True, dtype=torch.float64))</code> does not add a callable (fake) parameter to <code>myPyTorchModel</code>.</p>
<p>Is there any way that I can add a fake “callable” parameter to <code>myPyTorchModel</code>? If yes, how can I do it?</p>
<p>Thank you very much again for your help.</p>

username: tom
cooked: <p>PyTorch parameters aren’t callable, so I’ll probably have to pass and think you need a Pyro expert.</p>
Unable to use multiple GPU's with DataParallel

username: maan198
cooked: <p>I have wrapped my model with DataParallel to use both of my RTX2080Ti’s but I can only see 1 firing up and out of memory as I start the training.</p>
<p>I’m using num_workers=0 (due to an error LMDB is giving me when I have num_workers&gt;0)</p>
<p>Would this be the issue or is there anything else I’m missing?</p>
<pre><code class="lang-auto">def train(hierarchy, epochs, lr, batch_size):
    
    if hierarchy=='top':
        model=PixelSnail(
            [32,32],512,512,5,4,5,512,dropout=0.1,n_out_res_stack=5,
        )
        if torch.cuda.device_count()&gt;1:
            print('using:',torch.cuda.device_count(),'gpus')
            model=nn.DataParallel(model)
            model.to(device)
</code></pre>
<p>(The print there is giving me 2 gpus.)</p>

username: ptrblck
cooked: <p><code>nn.DataParallel</code> might create an imbalanced memory usage as described <a href="https://medium.com/huggingface/training-larger-batches-practical-tips-on-1-gpu-multi-gpu-distributed-setups-ec88c3e51255">here</a>.<br>
This could yield an out of memory issue on one device, which would stop the script execution.<br>
You could lower the batch size (if it’s not already 1 per device) and rerun the script to check the device usage.</p>
Saving and loading a model split on different gpus?

username: sicroci
cooked: <p>I have a model split on different gpus and I would like to save it on disk and load it. How could I do that?</p>

username: ptrblck
cooked: <p>I would recommend to transfer the models first to the CPU and store their <code>state_dict</code> as described <a href="https://pytorch.org/docs/stable/notes/serialization.html">here</a>.<br>
This would make sure you can recreate the models and load their <code>state_dicts</code> on different systems even if they don’t have a GPU.</p>
Depthwise Convolution in Pytorch

username: adi8862
cooked: <p>Hi,</p>
<p>I was using nvprof to profile the calls for various layers of MobilenetV2 in the forward and backward pass. For the Depthwise convolution the profiling says it uses the “spatialDepthwisConvolutionUpdateOutput” function, and from various other topics I have seen that PyTorch uses its own kernel for Depthwise convs instead of a cudnn kernel.</p>
<p>I was wondering which algorithm (GEMM, FFT, or WINOGRAD) is used to perform the convolution in Pytorch’s version?</p>

username: ptrblck
cooked: <p>It seems to use the im2col and matmul approach as seen <a href="https://github.com/pytorch/pytorch/blob/985fd970aad85799a568b76c2a1b75a1c422fe53/aten/src/THCUNN/SpatialDepthwiseConvolution.cu#L28-L103">here</a>.</p>
How to temporarily detach parameters when using DataParallel?

username: LuChengTHU
cooked: <p>I need to do something like this:</p>
<pre><code class="lang-auto">class MyOp(torch.autograd.Function):
        @staticmethod
        def forward(ctx, net1, net2, x):
            ctx.net1 = net1
            ctx.net2 = net2
            ctx.save_for_backward(x)
            return net1(x)

        @staticmethod
        def backward(ctx, grad):
            net1 = ctx.net1
            net2 = ctx.net2
            x = ctx.saved_tensors
            # disable backward for parameters in net2, because I only need the gradient for x by net2.
            for params in net2.parameters():
                params.requires_grad_(False)
            with torch.enable_grad():
                y = net2(x)
             y.backward(torch.ones_like(x).to(x))
             gradx = x.grad.clone().detach()
             # enable backward for net2, because it needs to be used in other computations.
             for params in net2.parameters():
                 params.requires_grad_(True)
             return (None, None, gradx)
</code></pre>
<p>This code works well for single-GPU. However, when I use DataParallel with Multi-GPUs, the gradient is wrong.</p>
<p>I guess maybe it is because there is no lock for multi-processes and there are some gradients backwarded to parameters in net2. How can I correct my code for DataPrallel models?</p>

username: ruotianluo
cooked: <p>My guess the reason why it doesn’t work, is you can no longer get parameters on DataParallel replica.<br>
One workaround, (my guess), is to use torch.autograd.grad instead of backward.<br>
you can do:</p>
<pre><code class="lang-auto">gradx = torch.autograd.grad(y, x, torch.ones_like(x).to(x))[0]
</code></pre>

username: pritamdamania87
cooked: <p>Can you set grads to True for net2’s parameters before you start the forward and then set them to false after you are done with the forward? This way the grads should be False for the entire backward pass despite concurrent execution of the backward pass across multiple GPUs.</p>

username: LuChengTHU
cooked: <p>Thanks! I’ve tried your suggestion and it works!</p>
Padding : nn.ZeroPad2d makes Error

username: moonchildz
cooked: <p>Hello. I am now making GC-net-like own network,</p>
<p>where GC-net is : <a href="https://arxiv.org/pdf/1703.04309.pdf" rel="nofollow noopener">https://arxiv.org/pdf/1703.04309.pdf</a></p>
<p>I’m now having difficulty at 'cost-volume’making phase.</p>
<p>the code of the phase is :</p>
<pre><code>def cost_volume(self, imgl, imgr):
    xx_list = []
    pad_opr1 = nn.ZeroPad2d((0, self.max_d, 0, 0))
    xleft = pad_opr1(imgl)
    for d in range(self.max_d):  # maxdisp+1 ?
        pad_opr2 = nn.ZeroPad2d((d, self.max_d - d, 0, 0))
        xright = pad_opr2(imgr)
        xx_temp = torch.cat((xleft, xright), 1)
        xx_list.append(xx_temp)
    xx = torch.cat(xx_list, 1)
    xx = xx.view(1, self.max_d, 64, int(self.height / 2), int(self.width / 2) + self.max_d)
    xx0 = xx.permute(0, 2, 1, 3, 4)
    xx0 = xx0[:, :, :, :, :int(self.width / 2)]
    return xx0
</code></pre>
<p>Here,  ‘xleft = pad_opr1(imgl)’ code line makes error of following :</p>
<p>xleft = pad_opr1(imgl)<br>
File “C:\Users\Administrator\PycharmProjects\eps\venv\lib\site-packages\torch\nn\modules\module.py”, line 550, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “C:\Users\Administrator\PycharmProjects\eps\venv\lib\site-packages\torch\nn\modules\padding.py”, line 17, in forward<br>
return F.pad(input, self.padding, ‘constant’, self.value)<br>
File “C:\Users\Administrator\PycharmProjects\eps\venv\lib\site-packages\torch\nn\functional.py”, line 3389, in _pad<br>
assert len(pad) // 2 &lt;= input.dim(), ‘Padding length too large’<br>
AttributeError: ‘tuple’ object has no attribute ‘dim’</p>
<p>I cannot understand the error and guess how to deal with it.</p>
<p>Please anybody give me hints and breakthroughs and thank you very much.</p>

username: ptrblck
cooked: <p>Based on the error message it seems <code>imgr</code> is a <code>tuple</code> instead of a tensor.<br>
Could you check the type and make sure to pass a tensor to the padding module?</p>
LSTM implementation

username: amin
cooked: <p>I am new with deep learning especially LSTM, I have time-series data for 62 days of 63 station and each station has divided into 8 parts in each day. it means I have 62 * 63 * 8 columns and the period of data is weekly for example every Sunday the station number 4 in part 1 of the day has a similar act. my question is how can set the LSTM parameters like input_size, seq_length, batch_size, n_layers,…<br>
I should note that I don’t have all stations in all days for example in some days I have data from 58 stations and I fill my table with zero for the rest of the stations<br>
thanks for your quick response!</p>
How to generate variable-length mask?

username: KelleyYin
cooked: <p>Supposing a is a=tensor([2, 1, 2]), I want to generate a mask based on tensor a .<br>
The mask is like this:</p>
<pre><code class="lang-auto">mask = tensor([[ 0,  0,  0,  1,  1],
               [ 0,  0,  0,  0,  1],
               [ 0,  0,  0,  1,  1]])
</code></pre>
<p>Is there any simple and convenient  method ?</p>

username: KelleyYin
cooked: <p>The question have solved.<br>
Interesting !</p>
<pre><code class="lang-auto">a = torch.tensor([2, 3, 1])
b = torch.arange(10, 0, step=-1).long().expand(3, 4)
a = a.unsqueeze(1).expand_as(c)
mask = b&lt;=a
</code></pre>

username: Yongfei_Yan
cooked: <p>I found this seems to work fine. <a href="https://stackoverflow.com/questions/53403306/how-to-batch-convert-sentence-lengths-to-masks-in-pytorch" rel="nofollow noopener">https://stackoverflow.com/questions/53403306/how-to-batch-convert-sentence-lengths-to-masks-in-pytorch</a></p>
<pre><code class="lang-auto">def length_to_mask(length, max_len=None, dtype=None):
    """length: B.
    return B x max_len.
    If max_len is None, then max of length will be used.
    """
    assert len(length.shape) == 1, 'Length shape should be 1 dimensional.'
    max_len = max_len or length.max().item()
    mask = torch.arange(max_len, device=length.device,
                        dtype=length.dtype).expand(len(length), max_len) &lt; length.unsqueeze(1)
    if dtype is not None:
        mask = torch.as_tensor(mask, dtype=dtype, device=length.device)
    return mask

</code></pre>

username: Yuheng_Zhi
cooked: <p>Great link! Prefer the other answer though, personally.</p>
<blockquote>
<p><code>torch.arange(max_len)[None, :] &lt; lens[:, None]</code></p>
</blockquote>
Intermediate Layer

username: truestory1226
cooked: <p>Hi,</p>
<p>I am using a resnet101 model from</p>
<pre><code class="lang-auto">import torchvision.models as models
models.resnet101
</code></pre>
<p>I want to make certain modifications to the model in such a way that I remove certain layers from the model towards the end and replace them by some other layers(I have successfully done this).</p>
<p>Next, I wanted to take output of one of the intermediate layer and perform some activation on it.</p>
<p>The final output of the model would be the output from activation(intermediate layer) and the one obtained at the end of the network. However, I am not aware of any suitable way of doing this operation where I could take the output of intermediate layer and then perform some operation on it and then return it back.</p>
<p>I have come across some methods on forum where we can register hook and obtain the outcome, but I want to perform further operations after obtaining the output.</p>
<p>Thanks in advance.</p>

username: ptrblck
cooked: <p>I think for many modifications a custom ResNet implementation would be more suitable than using hooks and other modifications.<br>
You can copy the code from <a href="https://github.com/pytorch/vision/blob/master/torchvision/models/resnet.py">here</a> and change the model as you wish.</p>

username: truestory1226
cooked: <p>Hi ptrblck,</p>
<p>Thank you for your answer. I will try doing that.</p>
Calculating accuracy gives wonky result

username: Radytin
cooked: <p>Hi,</p>
<p>I have a neural network that uses PyTorch. When I run it the accuracy seems a bit wonky (constant 1.00).</p>
<p>epoch=20, loss=0.00002, val_loss=33.33334, acc=1.00000, val_acc=0.66667<br>
.<br>
.<br>
epoch=70, loss=0.00000, val_loss=2.69771, acc=1.00000, val_acc=0.66667</p>
<p>Also val_acc doesn’t seem to change. Here’s my code I use to calculate accuracy:</p>
<pre><code>for epoch in range(epochs):
    loss = n_correct = 0
    model3.train()
    for batch, target in train_loader:
        batch,target = batch.view(-1, n_metabolites).to(device),target.to(device)
        batch = batch.view(-1, n_metabolites).to(device)
        optimizer3.zero_grad()
        outputs = model3(batch)
        train_loss = objective3(outputs, target)
        loss += train_loss.item()
        n_correct += (target == (outputs.reshape(-1) &gt; 0.5).float()).sum()
        train_loss.backward()
        optimizer3.step()
    
    loss = loss / len(train_loader)   
    acc = (n_correct.float() / len(train)).cpu().numpy()
    epoch += 1
        
    model3.eval();
    val_loss = val_n_correct = 0
    with torch.no_grad():
        for batch, target in val_loader:
            batch,target = batch.view(-1, n_metabolites).to(device),target.to(device)
            batch = batch.view(-1, n_metabolites).to(device)
            outputs = model3(batch)
            val_loss += objective3(outputs, target)
            val_n_correct += (target == (outputs.reshape(-1) &gt; 0.5).float()).sum()
        val_loss = (val_loss / len(val_loader)).cpu().numpy()
        val_acc = (val_n_correct.float() / len(val)).cpu().numpy()

    
    if (epoch % print_stats_interval) == 0 or epoch == epochs:
        print(f'epoch={epoch:.0f}, loss={loss:.5f}, val_loss={val_loss:.5f}, acc={acc:.5f}, val_acc={val_acc:.5f}')
    log3.append((epoch, loss, val_loss, acc, val_acc))
log3 = pd.DataFrame(log3, columns=['epoch', 'loss', 'val_loss', 'acc', 'val_acc'])
</code></pre>
<p>I’ve used similar accuracy calculation before, but now it seems something’s going wrong. I’d be happy if someone can give suggestions!</p>

username: tom
cooked: <p>100% accuracy may just mean your network has learnt perfectly on (or maybe memorized / overfit?) the training training set. I’d be more concerned about the apparent fixed accuracy on the validation set, probably…</p>

username: Radytin
cooked: <p>Thank you for the reply! Yes, the validation accuracy also concerns me.  It’s calculated the same way as the accuracy, I’m planning on running this with larger dataset to see if that would fix the issue (in case it’s due to overfitting), I’ve trained the network with smaller dataset to for shorter run time for testing etc. If you have any suggestions I would be happy to hear them!</p>
Representing Point Clouds to work with 3d Convolutions

username: Stanley_C
cooked: <p>I’m currently trying to work on a project which I’m working with point clouds that were created with a LiDAR scanner. I know that occupancy grids seems to be popular and straightforwards to work with, but they require an extremely large input, since every possible point has to be either marked as filled or empty. However, I could also use depth map, but since it is a flat image, where the grayscale color represents, but this isn’t true 3d data, more like 2.5d.  First, would it be useful to convert a 2d input layer that takes a depth map, to a 3d convolution, and would this help solve my issue of not having true 3d data? Second, how would this be implemented, and could someone please link an example of how to upscale a 2d convolution to a 3d convolution? Third, is there any other efficient method that can be used to convert a 3d point cloud into a true 3d format, that can be fed into a 3d convolutional neural network?</p>

username: aluo-x
cooked: <p>I have also been working on (synthetic) point clouds recently.</p>
<p>There is <a href="https://github.com/NVIDIA/MinkowskiEngine" rel="nofollow noopener">this library</a> that works with Pytorch.</p>
<p>There is also <a href="https://github.com/rusty1s/pytorch_geometric" rel="nofollow noopener">Pytorch Geometric</a> which can work with point clouds in both <a href="https://pytorch-geometric.readthedocs.io/en/latest/modules/data.html#torch_geometric.data.Batch.from_data_list" rel="nofollow noopener">data loading</a> using their custom <a href="https://pytorch-geometric.readthedocs.io/en/latest/modules/data.html#torch_geometric.data.DataLoader" rel="nofollow noopener">dataloader</a> &amp; <a href="https://pytorch-geometric.readthedocs.io/en/latest/modules/nn.html#torch_geometric.nn.conv.XConv" rel="nofollow noopener">convolutions</a> which uses their own <a href="https://pytorch-geometric.readthedocs.io/en/latest/modules/nn.html#torch_geometric.nn.data_parallel.DataParallel" rel="nofollow noopener">DataParallel</a>.</p>

username: Stanley_C
cooked: <p>Thank you for the reply. So are you saying the repo that you linked to make working with depth map and occupancy grids easier, or are you recommending that I use a feature, like graph CNNs for my project?</p>

username: aluo-x
cooked: <p>I’m not sure about the specifics of your project. But from what you described, I feel like what you want is to un-project the depth map (using whatever camera parameters) to a 3D point cloud. If that is the case, I think you would want graph CNNs / pointnet / pointCNN.</p>

username: Stanley_C
cooked: <p>So basically, I’m trying to upscale point clouds. So could I use a graph CNNs to convert a point cloud list  to a 3d cnn layer, and use a standard image upscaling network, but instead, use 3d cnn instead of 2d cnn. Also, I know that graph CNNs are good for non euclidean data, like social networks, but why would graph CNNs be useful to covert a point cloud list to a 3d CNN, given that point clouds are euclidean data?</p>
Feature Fusion with code

username: 111186
cooked: <p>I want to use Feature Fusion to improve the VGG19’s performance in classification. my plan is to extract the Conv layers 8，12, 16 and use the torch.cat() to imply the feature fusion. but i don’t konw how to change my code to finish it. this is my code, it will be very grateful to tell me how to do it.</p>
<pre><code class="lang-auto">class VGG(nn.Module):
    def __init__(self, vgg_name):
        super(VGG, self).__init__()
        self.features = self._make_layers(cfg[vgg_name])
        self.classifier = nn.Linear(512, 7)

    def forward(self, x):
        out = self.features(x)
        out = out.view(out.size(0), -1)
        out = F.dropout(out, p=0.5, training=self.training)
        out = self.classifier(out)
        return out

    def _make_layers(self, cfg):
        layers = []
        in_channels = 3
        for x in cfg:
            if x == 'M':
                layers += [nn.MaxPool2d(kernel_size=2, stride=2)]
            else:
                layers += [nn.Conv2d(in_channels, x, kernel_size=3, padding=1),
                           nn.BatchNorm2d(x),
                           nn.ReLU(inplace=True)]
                in_channels = x
            # print(layers)
        layers += [nn.AvgPool2d(kernel_size=1, stride=1)]
        return nn.Sequential(*layers)
</code></pre>

username: ptrblck
cooked: <p>Could you explain what “feature fusion” means?<br>
E.g. how would these features be fused, if they have different shapes or are you making sure the mentioned layers return activations in the same shape?</p>

username: 111186
cooked: <p>thank you so much,  I have solve this problem. by use the torch.cat().</p>
Canonical way to use jit.script w/ DataParallel

username: aluo-x
cooked: <p>Have been looking at using jit.script to potentially speed up some operations during training time. But the documentation is a bit sparse.</p>
<p>I have the following questions:</p>
<ol>
<li>Does it matter if we call<pre><code class="lang-auto">model = model.cuda()
model = torch.jit.script(model)
</code></pre>
versus<pre><code class="lang-auto">model = torch.jit.script(model)
model = model.cuda()
</code></pre>
Fundamentally, I guess the question is does the jit process depend on the device of the module?</li>
<li>Calling DataParallel on the jitted module. It seems like you cannot jit a DataParallel object. But are there any gotchas using DataParallel on a jitted module correct?</li>
<li>Saving state dicts<br>
On a normal model located on the GPU (with DataParallel), I can call<pre><code class="lang-auto">torch.save(net.module.cpu().state_dict(), STATE_PATH)
net.cuda()
</code></pre>
Is this correct for jitted networks?</li>
<li>Loading state dicts<br>
On a normal model located on the GPU (with DataParallel), I can call<pre><code class="lang-auto">net.module.load_state_dict(torch.load(STATE_PATH, map_location=str(net.device)))
</code></pre>
Is this still correct? Does the jit process introduce any subtleties that require caution?</li>
</ol>
Help with OSError caffe2_detectron_ops_gpu

username: cramberiecookies
cooked: <p>Hello. I keep getting the error</p>
<blockquote>
<p>OSError: [WinError 126] The specified module could not be found. Error loading “C:\Users\Username\AppData\Local\Programs\Python\Python38\lib\site-packages\torch\lib\caffe2_detectron_ops_gpu.dll” or one of its dependencies.</p>
</blockquote>
<p>I tried updating form cuda 10.1 to 10.2 but it didnt fix the issue. i also have cudnn installed . I uninstalled and reistalled torch multiple times with the commands found on the getting started page. Can someone help me resolve this i’ve being stuck for 2 days now</p>

username: cramberiecookies
cooked: <p>Problem solved cudnn64_7.dll was missing from system32 folder</p>
Update parameters of a network with a loss that is not directly computed by that network

username: sdota16
cooked: <p>Hi,</p>
<p>I have a network -&gt; net1 whose output is used to compute a mask, M.</p>
<p>The input (In2) to another network -&gt; net2 is multiplied by this mask and then passed into the net2.</p>
<p>In2 = In2 * M</p>
<p>How do I train net1 using the output of net2? I cannot train them sequentially beacause there is a function between net1 and net2 that uses the output of net1 to compute a mask.</p>
<p>net1 -&gt; mask_compute -&gt; In2*M -&gt; net2</p>

username: ptrblck
cooked: <p>If <code>mask_compute</code> is not using PyTorch methods, which would thus detach the computation graph and Autograd won’t be able to create the backward pass automatically, you would have to implement a custom <code>torch.autograd.Function</code> for it and provide the <code>forward</code> and <code>backward</code> for these operations.<br>
<a href="https://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_custom_function.html">Here</a> is an example.</p>

username: sdota16
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>, Thankyou for your response.</p>
<p>So the mask_compute function takes the output of net1 (size = Nx1) and creates a mask of size (Nx1) where the elements of the mask are 1 if the value of the corresponding element of the output of net1 is above a threshold and 0 if it’s below a threshold.</p>
<p>I am not sure how the forward and backward pass can be implemented for this?</p>
Speeding up tensor concatenation

username: bloodymeli
cooked: <p>Hi,</p>
<p>I need to concatenate a long list of small tensors. Each small tensor is a slice of a given (quite simple) constant matrix. Here is the code:</p>
<pre><code class="lang-auto">max_node, counter = 0, 0
batch_size, n_days = (1000, 10)
n_interactions_in = torch.randint(low=100,high=200,size=(batch_size,n_days), dtype=torch.long)
max_interactions = n_interactions_in.max()
delay_table = torch.arange(n_days, device=device, dtype=torch.float).expand([max_interactions, n_days]).t().contiguous()
delay_table = n_days - delay_table - 1
edge_delay_buf = []
for b in range(batch_size):
     delay_vec = [delay_table[d, :n_interactions_in[b, d]] for d in range(n_days)]
     edge_delay_buf.append(torch.cat(delay_vec))
res = torch.cat(edge_delay_buf)
</code></pre>
<p>This takes a lot of time. Is there a way to effeciently parrallize the creation of each element in the edge_delay_buf?<br>
I have tried multiple variants, such as replacing the for loop with a list concatenation, where the result is a list of lists, then flattening the list and applying torch.cat on the flattened list. However, it didn’t improve by much. For some reason the slicing operation takes too long.</p>
<p>Is there a way to make the slicing faster? Is there a way to make the loop more efficient / parallel?</p>
Not able to run quantized model on android

username: dklvch
cooked: <p>I have a quantized model which works on intel cpu and can be traced but fails to be run on android. Float32 model works fine on mobile though. Unfortunately, I cannot share the model. I get the following error:</p>
<p><code>java.lang.IllegalArgumentException: at::Tensor scalar type is not supported on java side</code></p>
<h2>Environment</h2>
<p>PyTorch version: 1.7.0.dev20200727<br>
Is debug build: No<br>
CUDA used to build PyTorch: 10.2</p>
<p>OS: Ubuntu 20.04 LTS<br>
GCC version: (Ubuntu 8.4.0-3ubuntu2) 8.4.0<br>
CMake version: version 3.16.3</p>
<p>Python version: 3.7<br>
Is CUDA available: Yes<br>
CUDA runtime version: 10.2.89<br>
GPU models and configuration: GPU 0: GeForce GTX 1070<br>
Nvidia driver version: 440.100<br>
cuDNN version: Could not collect</p>
<p>Versions of relevant libraries:<br>
[pip3] numpy==1.19.0<br>
[pip3] torch==1.7.0.dev20200727<br>
[pip3] torchvision==0.8.0.dev20200727<br>
[conda] Could not collect</p>
<p>cc <a class="mention" href="/u/jerryzh168">@jerryzh168</a> <span class="mention">@jianyuh</span> <a class="mention" href="/u/dzhulgakov">@dzhulgakov</a> <a class="mention" href="/u/raghuramank100">@raghuramank100</a> <span class="mention">@jamesr66a</span> <span class="mention">@vkuzo</span></p>

username: dzhulgakov
cooked: <p>Most likely you’re trying to return a quantized tensor (we should improve the error message <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> ). We don’t have Java binding for quantized tensor yet. You can try to dequantize the tensor within your model (something like <code>result.dequantize()</code>) or return individual components of the tensor (<code>result.int_repr(), result.q_scale(), result.q_zero_point()</code>)</p>
Best way to cache large pre-processed dataset in disk

username: ucals
cooked: <p>After a long pre-processing procedure (~1h), I generate ~3.5 million samples to train a network in an image segmentation task. This dataset is too large to fit into memory, so I’d like to cache the pre-processed samples in disk. Saving one file per sample is very inefficient, taking over 50 GB in disk.</p>
<p><strong>What is the recommended/best way to solve this problem? Does PyTorch have something like TFRecord? Is there a ready-to-use library to cache large batches of samples?</strong></p>
<p>All the solution I thought basically involving coding a solution from scratch.</p>
<p>Thanks!</p>

username: JuanFMontesinos
cooked: <p>Hi,<br>
I’m not aware at all about how TF records work. Have you think of using numpy mmap? if the images are same size you can create a big numpy array and read it on-demand and even make your own cache in ram.</p>
<p>Another option to optimize the process could be using nvidia dali <a href="https://developer.nvidia.com/DALI" rel="nofollow noopener">https://developer.nvidia.com/DALI</a></p>