akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
The function to get size of the tensor?

kaiseryet: <p>So I need to get the size of a list of tensors… that’s right, <code>[tensor_1, tensor_2, tensor_3]</code>; apparently, <code>.size()</code> will not work for this, so I need to use <code>map</code> function eventually I guess. But what’s the function to get the size of the tensor?</p>

albanD: <p>I guess you can use a lambda to do this: <code>lambda t: t.size()</code> ?</p>

kaiseryet: <p>call up lambda each time will be slow tho</p>

albanD: <p>Not sure it will be slower than any other python invocation.<br>
Why would it be so slow?</p>

kaiseryet: <p>Basically everytime (in a loop) you use lambda, you are basically defining a function again. So predefine a function and use it later will be much more efficient than lambda.</p>

albanD: <p>Fun fact, the lambda is actually significantly faster than creating a function and reusing it <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<pre><code class="lang-python">In [1]: import torch

In [2]: def gs(t):
   ...:     return t.size()
   ...:

In [3]: t = torch.rand(10, 3, 2)

In [4]: %timeit gs(t)
692 ns ± 12.6 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

In [5]: %timeit gs(t)
740 ns ± 5.03 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

In [6]: %timeit lambda x: x.size()
228 ns ± 3.08 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

In [7]: %timeit lambda x: x.size()
235 ns ± 3.41 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
</code></pre>
How are torchvision pretrained models converted from version to version?

Naman-ntc: <p>If I try to load pytorch 0.3 or 0.2 version in pytorch 0.4, batchnorm gives the track running status error.</p>
<p>Are torchvision pretrained models trained everytime when new release occurs or do the transfer weights ‘in some way’ across versions?</p>

tom: <p>This is relatively rare and most likely considered a bug. In your case <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/batchnorm.py#L72" rel="nofollow noopener">PyTorch master has a bugfix to help BatchNorm load older snapshots</a>, 
don’t tell anyone, but you might try just copypasting from that
.</p>
<p>Best regards</p>
<p>Thomas</p>

remissio: <p>I’m also struggling with models trained on older vesions (0.4). Wish there was a solution for that.</p>
Get filename when using ImageFolder

cmplx96: <p>Hi,</p>
<p>I am using ImageFolder to load a test dataset to submit to Kaggle.<br>
I need the filename because it contains the id.<br>
Is there any way I can get this with ImageFolder?</p>
<p>Thanks!</p>

rwightman: <p>Since there shouldn’t be any shuffling when you run through your dataset you can just match up the filenames in the <code>imgs</code> member of the dataset after you’ve run inference. It’s less messy than trying to return the filename as an extra element in the getitem tuple.</p>
<p>I usually add a member like the one below to my dataset class, you could also implement as a free function and direcly access the <code>imgs</code> member from outside the dataset class.</p>
<pre><code class="lang-auto">    def filenames(self, indices=[], basename=False):
        if indices: 
            # grab specific indices
            if basename:
                return [os.path.basename(self.imgs[i][0]) for i in indices]
            else:
                return [self.imgs[i][0] for i in indices]
        else:
            if basename:
                return [os.path.basename(x[0]) for x in self.imgs]
            else:
                return [x[0] for x in self.imgs]
</code></pre>
<p>A very minimal example that zips the filenames with the labels from an inference loop here: <a href="https://github.com/rwightman/pytorch-image-models/blob/master/inference.py" rel="nofollow noopener">https://github.com/rwightman/pytorch-image-models/blob/master/inference.py</a></p>

nofreewill: <blockquote>
<p>dataset = ImageFolder(…)<br>
dataset.imgs</p>
</blockquote>
<p><strong>dataset.imgs</strong> contains all the filenames (dataset[0] → dataset.imgs[0], …)</p>
Batching training Denoising Autoencoder

Dylan_Yung: <p>I have this code for training a denoising autoencoder that uses an LSTM for encoder and decoder and operates on names</p>
<pre><code class="lang-auto">def denoise_train(x: DataLoader):
    loss = 0.
    noisy_x = list(map(lambda s: noise_name(s), x))

    rnn_x = to_rnn_tensor(x, DECODER_COUNT)
    rnn_noisy_x = to_rnn_tensor(noisy_x, ENCODER_COUNT)

    encoder_hidden = encoder.init_hidden(batch_size=BATCH_SZ)

    for i in range(rnn_noisy_x.shape[0]):
        _, encoder_hidden = encoder(rnn_noisy_x[i].unsqueeze(0), encoder_hidden)

    decoder_input = strings_to_tensor([SOS] * BATCH_SZ)

    decoder_hidden = encoder_hidden

    name = ''

    for i in range(rnn_x.shape[0]):

        decoder_probs, decoder_hidden = decoder(decoder_input, decoder_hidden)

        _, nonzero_indexes = rnn_x[i].topk(1)

        # TODO!!! Need to fix rest of code for batch

        best_index = torch.argmax(decoder_probs, dim=2).item()

        loss += criterion(decoder_probs[0], nonzero_indexes[0])

        name += ALL_CHARS[best_index]

        decoder_input = torch.zeros(1, 1, LETTERS_COUNT)

        decoder_input[0, 0, best_index] = 1.

    loss.backward()
    return name, noisy_x, loss.item()
</code></pre>
<p>The x that gets passed in is the next iteration of an iter(DataLoader). The main thing I’m trying to do is get the argmax for all the decoder_probs, which is of size name length x batch size x output length. So I need best_index to be the argmax for all entries in the batch and decoder_input should be a 1xbatch size x output where all the best chars = 1.</p>
Log_softmax always returns 0 for index 0(background)

lzy_pub: <p>Hi I am currently using albunet from <a href="https://github.com/ternaus/robot-surgery-segmentation" class="inline-onebox" rel="noopener nofollow ugc">GitHub - ternaus/robot-surgery-segmentation: Wining solution and its improvement for MICCAI 2017 Robotic Instrument Segmentation Sub-Challenge</a> with a pretrained resnet as the encoder to train on a custom dataset. Regardless how many epochs(I tried 1-4), the predicted log_softmax value of class 0(background) is always 1 which shouldn’t be the case since that would mean it’s probability is 1. I have 4 classes including the background. The training code uses NLLLoss with log softmax.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/c/a/ca01eee28ffa2dbbe46218c0419fcd047e1fe505.png" alt="image" data-base62-sha1="sP2ACVWOWZSQ3YMQABWDwKfVrrD" width="571" height="464"></p>

ptrblck: <p>Could you check the outputs at the beginning of the training?<br>
Are you seeing random log probabilities, which then converge towards class0?<br>
If so, are you dealing with an imbalanced dataset, which could explain the overfitting to the majority class.</p>

lzy_pub: <p>Oh it seems you are right since it converges quickly to 0 and hovers around there.</p>
<p>However since I am doing semantic segmentation, I am likely to have much more background than objects as most of my objects are small-medium sized, how would I be able to prevent it converge towards class 0 then?</p>
<p>Also another thing, since I can reobtain the probabilities by just doing torch.exp(log probs) wouldn’t that mean that the probability for class 0 be 1 and hence the other should be 0 or -inf in log probs? Or is it because it is rounded up that’s why the other can be large negative values?</p>
<p>Btw, appreciate your help for here and around the forums, it has helped a lot of the problems I have faced when learning pytorch.</p>

ptrblck: <p>For a segmentation use case <a href="https://discuss.pytorch.org/t/how-to-implement-focal-loss-in-pytorch/6469">focal loss</a> might help or a pixel-wise weighting of the minority classes.</p>
<p>The large negative values are approx. zero as probabilities and I would assume an Inf somewhere in the computation might have some bad side effects. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
<p>Thanks, I’m glad my posts are helpful. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

lzy_pub: <p>Oh okay I’ll take a look, thanks!</p>
<p>Yeah that’s true,t hat makes sense haha.</p>