akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Equivalence between CTC loss gradients

username: Kiriakos_Shiarli
cooked: <p>I am working  on a variant of the CTC loss. It is my understanding that one can compute the forward variables (in log space) and simply perform a logsumexp operation in the last two alpha variables of the sequence at the last timestep. This gives us the log-likelihood of the sequence under the model.</p>
<p>My current implementation is in pytorch and takes this as the loss and simply calls backward to get the gradients. I.e I don’t have a specific implementation for the backward pass.</p>
<p>Looking at the native cuda implementation here: <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/cuda/LossCTC.cu#L9" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/cuda/LossCTC.cu#L9</a>, the full forward-backward recursion used in the paper is performed.</p>
<p>The question is: Will the gradients in the above implementation be equivalent to performing the forward pass and then using autograd only (much) slower? Or is there something in the backward recursion that will produce different gradients. If so, what would that be?</p>
<p>Already talked with <a class="mention" href="/u/tom">@tom</a> about this but it wasn’t clear to me whether or not a pure pytorch implementation of the forward pass and then loss.backward() would suffice.</p>

username: tom
cooked: <p>My impression was that the implied CTC backward has numerical stability problems, but in the end I didn’t dig into it too much.<br>
You can just implement the CTC function, call it with double and call <code>gradcheck</code> if you’re interested in exploring this.</p>
<p>Best regards</p>
<p>Thomas</p>

username: Kiriakos_Shiarli
cooked: <p>Ah gradcheck sounds like a good idea! thanks!</p>

username: vadimkantorov
cooked: <p><a class="mention" href="/u/kiriakos_shiarli">@Kiriakos_Shiarli</a> I implemented a CTC that computes gradients via pure autograd (it’s quite slow):<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicon.ico" class="site-icon" width="32" height="32">
      <a href="https://github.com/vadimkantorov/ctc" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars2.githubusercontent.com/u/1041752?s=400&amp;v=4" class="thumbnail onebox-avatar" width="400" height="400">

<h3><a href="https://github.com/vadimkantorov/ctc" target="_blank" rel="nofollow noopener">vadimkantorov/ctc</a></h3>

<p>Primer on CTC implementation in pure Python PyTorch code - vadimkantorov/ctc</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>If you replace logadd by PyTorch logsumexp and use 1e-16 instead of float(’-inf’), double backward may just happen to work</p>
How to determine gradient clip value

username: Dean_Sumner
cooked: <p>Is there a way to assess the appropriate value to clip the gradients to, e.g can I return the gradient values from my models and compute the nth percentile and set this to the clip value?</p>
<blockquote>
<p>for epoch in enumerate(n_epochs):</p>
<p>encoder = …<br>
decoder = …<br>
&lt;&lt;&lt; SOME TRAINING CODE … &gt;&gt;&gt;</p>
<p><span class="hashtag-raw">#update</span> new loss<br>
loss += new_loss</p>
<p><span class="hashtag-raw">#BackProp</span><br>
loss.backward()</p>
<p><span class="hashtag-raw">#Clip</span> gradients: gradients are modified in place<br>
clip = <strong>some_value based on nth percentile of all gradients</strong><br>
_ = nn.utils.clip_grad_norm_(encoder.parameters(), clip)<br>
_ = nn.utils.clip_grad_norm_(decoder.parameters(), clip)</p>
<pre><code># Adjust model weights
encoder_optimizer.step()
decoder_optimizer.step()    

# Increment the schedulers
scheduler_encoder.step()
scheduler_decoder.step()

current_learning_rate = []
for param_group in encoder_optimizer.param_groups:
    current_learning_rate.append(param_group['lr'])

return loss,  current_learning_rate
</code></pre>
</blockquote>

username: Dean_Sumner
cooked: <p>I’m not sure if percentiles are appropriate to use here. Open to other suggestions.</p>
Last layer of the Discriminator

username: oasjd7
cooked: <p>Hi, I have two questions about the discriminator.</p>
<ol>
<li>Which size is right for the last layer of discriminator?</li>
</ol>
<pre><code class="lang-auto">last_fc = nn.Linear(100,2)
or
last_fc = nn.Linear(100,1)
</code></pre>
<ol start="2">
<li>After last_fc, both sigmoid and softmax can be used with no differences?</li>
</ol>

username: tom
cooked: <p>So the two things are mathematically equivalent, but you have to choose a consistent setup. For the equivalence, look here:</p>
<pre><code class="lang-python">weight_softmax = torch.randn(2, 100)
bias_softmax = torch.randn(2)
inp = torch.randn(1, 100) / 50

probs_real_1 = torch.softmax(torch.nn.functional.linear(inp, weight_softmax, bias_softmax), dim=1)[:, 0]
probs_real_2 = torch.sigmoid(torch.nn.functional.linear(inp, weight_softmax[:1] - weight_softmax[1:], bias_softmax[:1] - bias_softmax[1:]))
probs_real_1, probs_real_2
</code></pre>
<p>Of course, you can also do the computation with variables, pencil, and paper to have a proof. The softmax gradient typically has mean zero, so they’re equivalent in computation (and by corollary by gradient).</p>
<p>Given that the relation is linear, one would expect the equivalence to broadly hold. However, things taking norms like spectral normalization or LAMB-style optimizers will work in a (slightly) different way.</p>
<p>Best regards</p>
<p>Thomas</p>
Defining an object inside autograd.Function

username: jinujanu
cooked: <p>Hi, I’m trying to define an object inside the autograd.Function, but when i’m calling the object inside the backward, it’s saying the object is not defined…</p>
<p>e.g.</p>
<pre><code class="lang-auto">class PermutationLayer(autograd.Function):
    @staticmethod
    def forward(ctx, x, x1, x2):
        ctx.save_for_backward(x, x1, x2)
    
    @staticmethod
    def backward(ctx, grad_output, ...):
        x, x1, x2 = ctx.saved_tensors

        grad_x1 = want_to_add(x1)

        return None, grad_x, grad_x1, grad_x2

    def want_to_add(ctx, x_in):
        return...
</code></pre>
<p>Could someone tell me the reason and a way to implement an intended def object to be used inside the autograd.Function?</p>
<p>Thank you for reading!</p>

username: albanD
cooked: <p>Hi,</p>
<p>You want to double check how python classes work.<br>
But because we use static method, we don’t create instances of this class.</p>
<p>To use this function, you can either define it outside of the class. Or as another static method and call it with <code>PermutationLayer. want_to_add()</code>.</p>
PyTorch - Tensorboard - keyword attribute 'name' has the wrong type

username: precipice
cooked: <p>Am working on getting Tensorboard and Pytorch working.  Have been following the tutorial at <a href="https://pytorch.org/tutorials/intermediate/tensorboard_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/intermediate/tensorboard_tutorial.html</a>.  However, I cannot get past the error <strong>RuntimeError: required keyword attribute ‘name’ has the wrong type</strong>. (See full dump below). Any suggestions.  Thanks</p>
<p>I am on Anaconda with Python 3.7.5/ Torch 1.3.1 and the following Tensorboard<br>
tensorboard                             2.0.2<br>
tensorflow                              2.0.0<br>
tensorflow-estimator                    2.0.1<br>
termcolor                               1.1.0<br>
testpath                                0.4.4<br>
torch                                   1.3.1<br>
torchvision                             0.4.2</p>
<p>Conda Info<br>
conda version : 4.8.0<br>
conda-build version : 3.18.11<br>
python version : 3.7.4.final.0<br>
virtual packages : __osx=10.15.2<br>
base environment : /usr/local/anaconda3  (writable)<br>
package cache : /usr/local/anaconda3/pkgs<br>
/Users/jph/.conda/pkgs<br>
envs directories : /usr/local/anaconda3/envs<br>
/Users/jph/.conda/envs<br>
platform : osx-64<br>
user-agent : conda/4.8.0 requests/2.22.0 CPython/3.7.4 Darwin/19.2.0 OSX/10.15.2<br>
UID:GID : 501:20<br>
netrc file : None<br>
offline mode : False</p>
<p>Error Message<br>
File “/Users/jph/Documents/tmp/torch/cnn/CNNNet.py”, line 174, in <br>
writer.add_graph(net, images)</p>
<p>File “/usr/local/anaconda3/envs/pyt/lib/python3.7/site-packages/torch/utils/tensorboard/writer.py”, line 682, in add_graph<br>
self._get_file_writer().add_graph(graph(model, input_to_model, verbose))</p>
<p>File “/usr/local/anaconda3/envs/pyt/lib/python3.7/site-packages/torch/utils/tensorboard/_pytorch_graph.py”, line 243, in graph<br>
list_of_nodes = parse(graph, args)</p>
<p>File “/usr/local/anaconda3/envs/pyt/lib/python3.7/site-packages/torch/utils/tensorboard/_pytorch_graph.py”, line 212, in parse<br>
nodes_py.append(NodePyOP(node))</p>
<p>File “/usr/local/anaconda3/envs/pyt/lib/python3.7/site-packages/torch/utils/tensorboard/_pytorch_graph.py”, line 90, in <strong>init</strong><br>
self.attributes = str({k: node_cpp[k] for k in node_cpp.attributeNames()}).replace("’", ’ ')</p>
<p>File “/usr/local/anaconda3/envs/pyt/lib/python3.7/site-packages/torch/utils/tensorboard/_pytorch_graph.py”, line 90, in <br>
self.attributes = str({k: node_cpp[k] for k in node_cpp.attributeNames()}).replace("’", ’ ')</p>
<p>File “/usr/local/anaconda3/envs/pyt/lib/python3.7/site-packages/torch/onnx/utils.py”, line 731, in _node_getitem<br>
return getattr(self, sel)(k)</p>
<p><strong>RuntimeError: required keyword attribute ‘name’ has the wrong type</strong></p>

username: ucals
cooked: <p>Same issue here… can anyone help how to fix this issue?</p>
'NoneType' object has no attribute 'vgg'

username: 111215
cooked: <p>I now want to use “refinedet” for target detection, loading my own dataset, but the following problem occurred when using vgg16, the backbone network.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/6/161a3a3f831b1ecd1340e2ba6eea250da0fd5012.png" data-download-href="https://discuss.pytorch.org/uploads/default/161a3a3f831b1ecd1340e2ba6eea250da0fd5012" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/6/161a3a3f831b1ecd1340e2ba6eea250da0fd5012_2_690x137.png" alt="image" data-base62-sha1="39wHbDQZ6rnZAVjquw5IoRUJrtE" width="690" height="137" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/6/161a3a3f831b1ecd1340e2ba6eea250da0fd5012_2_690x137.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/6/161a3a3f831b1ecd1340e2ba6eea250da0fd5012_2_1035x205.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/1/6/161a3a3f831b1ecd1340e2ba6eea250da0fd5012.png 2x" data-dominant-color="EFEAF3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1269×253 38.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: albanD
cooked: <p>This happens because <code>refinedet_net</code> is <code>None</code> is your code? You want to check why this happens? Did you forgot a <code>return</code> statement maybe?</p>

username: 111215
cooked: <p>Thanks for your warning. I’ve found the problem and I’m trying to solve it.</p>
How to optimize a slow LSTM

username: maria
cooked: <p>I’m training a language model in colab on GPU but it’s really slow, How can I optimize my code?</p>
<p>This is my code:</p>
<ol>
<li>The training:</li>
</ol>
<pre><code class="lang-auto">   for epoch in range(epochs):
        states = (torch.FloatTensor(2, bs, 200).\
                uniform_(-0.1, 0.1).to(device),\
            torch.FloatTensor(2, bs, 200).\
                uniform_(-0.1, 0.1).to(device))
        states = detach(states)
        for i, batch in enumerate(train_iter):
            model.train()     
            inputs, labels = batch.text, batch.target  
            inputs, labels = inputs.to(device), labels.to(device)     
            outputs, states = model(inputs, states)
            loss = loss_fn(outputs, labels.view(-1))
            model.zero_grad()
            loss.backward(retain_graph=True)
            clip_grad_norm_(model.parameters(), 0.5)
            optimizer.step()
            if not  i%100:
                    print('i {}, Epoch [{}/{}], Loss: {:.4f}, Perplexity: {:5.2f}'
                    .format(i, epoch+1, epochs, loss.item(), np.exp(loss.item())))
</code></pre>
<ol start="2">
<li>The model:</li>
</ol>
<pre><code class="lang-auto">class RNNModel(nn.Module):
    def __init__(self,
                 rnn_type,
                 vocab_size=10001,
                 n_layers=2,
                 n_hidden=200,
                 emb_size=300,
                 p_dropout=0.,
                 batch_size=bs):
        super(RNNModel, self).__init__()
        self.embedding = nn.Embedding.from_pretrained(TEXT.vocab.vectors, freeze=True)#(vocab_size, emb_size)
        #self.embedding.weight.data.copy_(TEXT.vocab.vectors)
        self.rnn = getattr(nn, rnn_type)(input_size=emb_size,
                                        num_layers=n_layers,
                                        hidden_size=n_hidden,
                                        dropout=p_dropout,
                                        batch_first=True)
        self.linear = nn.Linear(n_hidden,
                                vocab_size)
    
    def forward(self, x, h):
        x = self.embedding(x)
        output, (h, c) = self.rnn(x, h)
        output = output.reshape(output.size(0)*output.size(1), 
                                output.size(2))
        output = self.linear(output)
        return output, (h, c)
</code></pre>

username: G.M
cooked: <p>Python loops are  very  slow, you  should try to use something to replace that. You can also use Touchscript to optimize it. Take a look <a href="https://github.com/pytorch/pytorch/blob/master/benchmarks/fastrnns/custom_lstms.py" rel="nofollow noopener">here</a>.</p>
How to parallel multiple models

username: LonglongVIP
cooked: <p>I have a model framework. Models have different inputs, but their corresponding tags are the same. The framework is shown in the figure. How should I design models and connect their fc layers?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/4/8474252c04de08a6c681d12f659f849ee705a4da.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/8474252c04de08a6c681d12f659f849ee705a4da" title="model%20(2)"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/4/8474252c04de08a6c681d12f659f849ee705a4da_2_666x500.jpeg" alt="model%20(2)" data-base62-sha1="iTJRH5HEUhOWzem4KDm3zyJnxKq" width="666" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/4/8474252c04de08a6c681d12f659f849ee705a4da_2_666x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/8/4/8474252c04de08a6c681d12f659f849ee705a4da_2_999x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/4/8474252c04de08a6c681d12f659f849ee705a4da_2_1332x1000.jpeg 2x" data-dominant-color="969693"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">model%20(2)</span><span class="informations">4160×3119 2.23 MB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>Your architecture looks a bit like a model ensemble, so <a href="https://discuss.pytorch.org/t/custom-ensemble-approach/52024/4">this post</a> might be a good starter for you.</p>
<p>Let me know, if this code helps or if you need more input. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: LonglongVIP
cooked: <p>I watched it, I can understand it, I will try it and  thanks for your reply! Maybe I could ask other question s later</p>
Can nn.Upsample do differentiation?

username: feiyuhuahuo
cooked: <p>When I was debugging, I noticed the grad_fn for a tensor is ‘UpsampleBilinear2DBackward’. I was confused that can nn.Upsample do differentiation? I know that nn.Conv2d and nn.ConvTranspose2d() can do differentiation and get the gradients because they have parameters.<br>
Besides, how can I take a look that how ops computes gradients in my IDE (PyCharm) or at somewhere else?</p>
Strange loss function (VGG)

username: kubslo
cooked: <p>Hi,<br>
During training neural network, a loss function looks strange. I used VGG architecture from torchvision, but I modified it to work with BCEloss and CEL ( CrossEntropyLoss) at the same time.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/0/8/0847655a8da8e7d79905a7218de40a21a502a513.png" alt="image" data-base62-sha1="1beMEMdx9MH628KpmjxmFgESbjZ" width="565" height="372"></p>
<p>The loss function is now seen as : L = (1-k) BCEloss + k CEL.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/e/2eedc96379c56bddb4b5bd834fe24121c9205bda.png" alt="image" data-base62-sha1="6H9pZyITMt7KW1QOw6dBDWBUOeu" width="474" height="287"></p>
<p>Is that possible that most neurons are dead at the beginning of training?<br>
I think that is not related to data or dataloader because I load the same data to ResNet and the loss function looks normal there.</p>
<p>Thanks in advance for suggestions.</p>
Adam Optimizer and learning rate

username: farhad-bat
cooked: <p>Hello,</p>
<p>I use Adam Optimizer for training my network but when I print learning rate I realized that learning rate is constant and doesn’t change. Adam optimizer is adaptive learning rate algorithm but I don’t see such behavior in results.</p>
<pre><code class="lang-auto">    if optimizer_name == 'SGD':
        # optimizer
        optimizer = torch.optim.SGD(
            params=params,
            lr=optimizer_arguments['lr'],
            momentum=optimizer_arguments['momentum'],
            weight_decay=optimizer_arguments['weight_decay'])
    elif optimizer_name == 'Adam':
        # Adam optimizer
        optimizer = torch.optim.Adam(
            params=params,
            lr=optimizer_arguments['lr'],
            weight_decay=optimizer_arguments['weight_decay'])
    else:
        raise ValueError('Optimizer name is incorrect!')
</code></pre>

username: albanD
cooked: <p>Hi,</p>
<p>It is adaptive for each weight. The global <code>lr</code> does not change but the statistics for each weight does. You might want to double check the paper.</p>
Unable to install latest torchaudio using pip

username: torslim
cooked: <p>I am trying to install the latest torchaudio version from <a href="https://download.pytorch.org/whl/nightly/torch_nightly.html" rel="nofollow noopener">this link</a> using the following pip command:</p>
<pre><code class="lang-auto">pip install torchaudio_nightly -f https://download.pytorch.org/whl/nightly/torch_nightly.html
</code></pre>
<p>This downloads and installs torchaudio version <code>0.4.0.dev20190801</code>. However, on the given link there are multiple wheels available for a torchaudio 0.4.0 dev version with a later date (all the way up to 20200101). How can I install one of the later versions (preferably the latest one) using pip?</p>

username: albanD
cooked: <p>Do you have some dependency that fix your torchaudio version to this one? like torch_nightly?</p>

username: torslim
cooked: <p>No, when using a completely empty environment and installing torchaudio_nightly using the command above it the latest version pip downloads is 0.4.0-dev20190801, see <a href="https://colab.research.google.com/drive/1esAfqlbSoSqY0fG8OZgPvX9IAsi21BC4" rel="nofollow noopener">this Colab notebook</a>. There clearly are torchaudio nightly versions available from a newer data as can be seen on the html nightly link. Conda has a newer version available from <a href="https://anaconda.org/pytorch-nightly/torchaudio" rel="nofollow noopener">this link</a> (v0.4.0.dev20200102).</p>

username: albanD
cooked: <p>cc <a class="mention" href="/u/vincentqb">@vincentqb</a> who is working on audio</p>
How to visualize the feature map of each layer of the convolutional network

username: Foolish
cooked: <p>How do I replace the following code with my own network model?<br>
########################################################<br>
self.pretrained_model = models.vgg16(pretrained=True).features</p>

username: Eta_C
cooked: <p>1.Use <code>register_forward_hook</code> to get feature map</p><aside class="quote" data-post="2" data-topic="64639">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/spanev/48/14892_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-to-register-hook-on-conv2d/64639/2">How to register_hook on Conv2d?</a> 
  </div>
  <blockquote>
    Hi <a class="mention" href="/u/umair_javaid">@Umair_Javaid</a>, 
Simply create a hook function and register it to the by calling register_forward_hook.
  </blockquote>
</aside>
<p>
2. Use <a href="https://captum.ai/" rel="noopener nofollow ugc">Captum</a> or <a href="https://github.com/utkuozbulak/pytorch-cnn-visualizations" rel="noopener nofollow ugc">pytorch-cnn-visualizations</a> to visualize the feature map</p><aside class="quote" data-post="2" data-topic="65617">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-to-use-heatmaps-in-predicting-where-the-classifier-is-focusing-on-i-need-some-help-here-is-the-code-how-can-i-go-about-it/65617/2">How to use heatmaps in predicting where the classifier is focusing on i need some help ,here is the code.how can i go about it?</a> 
  </div>
  <blockquote>
    <a href="https://captum.ai/" rel="noopener nofollow ugc">Captum</a> might give you a good toolset for model interpretability.
  </blockquote>
</aside>
<aside class="quote" data-post="3" data-topic="65617">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/eta_c/48/17667_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-to-use-heatmaps-in-predicting-where-the-classifier-is-focusing-on-i-need-some-help-here-is-the-code-how-can-i-go-about-it/65617/3">How to use heatmaps in predicting where the classifier is focusing on i need some help ,here is the code.how can i go about it?</a> 
  </div>
  <blockquote>
    Before Captum, I use <a href="https://github.com/utkuozbulak/pytorch-cnn-visualizations" rel="noopener nofollow ugc">pytorch-cnn-visualizations</a>.<img width="20" height="20" src="https://discuss.pytorch.org/images/emoji/apple/laughing.png?v=12" title="laughing" alt="laughing" class="emoji">
  </blockquote>
</aside>
How to fix GAN class errors

username: cruu
cooked: <p>Hello,<br>
i have a few errors in my GAN class, which i did not have in earlier versions of pytorch.<br>
Can someone show / explain me how to fix them?<br>
Code: <a href="https://hastebin.com/ninibexava.py" rel="nofollow noopener">Link</a><br>
Errors: <a href="https://hastebin.com/izezirokaz.sql" rel="nofollow noopener">Link</a><br>
Running on GTX 1060 6GB<br>
Windows 10 - Pycharm<br>
Should be latest version of Pytorch</p>
<p>Dear</p>

username: ptrblck
cooked: <p>Could you check the length of your <code>Dataset</code> and make sure it’s larger than the batch size of 64?<br>
Since you are using this values for the batch size and <code>drop_last=True</code>, this line of code:</p>
<pre><code class="lang-python">x = next(iter(data))
</code></pre>
<p>will raise the exception for datasets with fewer samples.</p>
Normalization for images and other input data

username: Hong_Cheng
cooked: <p>Here is my custom DATASET</p>
<p>In each sample, I have the image data and other data (float number)<br>
I hope I can do the normalization for <code>image</code>, <code>tr_angles</code> and <code>pls</code>.</p>
<p>Thanks a lot.</p>
<pre><code class="lang-auto">
class PLDataset(Dataset):
    """Face Landmarks dataset."""

    def __init__(self, csv_file, root_dir, transform=None):
        """
        Args:
            csv_file (string): Path to the csv file with annotations.
            root_dir (string): Directory with all the images.
            transform (callable, optional): Optional transform to be applied
                on a sample.
        """
        self.landmarks_frame = pd.read_excel(csv_file)
        self.root_dir = root_dir
        self.transform = transform

    def __len__(self):
        return len(self.landmarks_frame)

    def __getitem__(self, idx):
        if torch.is_tensor(idx):
            idx = idx.tolist()

        img_names = os.path.join(self.root_dir,
                                 self.landmarks_frame.iloc[idx, 0])
        image = io.imread(img_names)

        pls = self.landmarks_frame.iloc[idx, 5:9]  # not [:, 5:9]
        tr_angles = self.landmarks_frame.iloc[idx, 9:14]
        positions = self.landmarks_frame.iloc[idx, 3:5]

        image = image.astype('float')  # .values
        pls = pls.values.astype('float')
        tr_angles = tr_angles.values.astype('float')
        positions = positions.values.astype('float')

        sample = {'image': image,
                  'pls': pls,
                  'tr_angles': tr_angles,
                  'positions': positions}

        if self.transform:
            sample = self.transform(sample)

        return sample

</code></pre>
<p>Now I just have my own <code>ToTensor()</code> function,</p>
<pre><code class="lang-auto">class ToTensor(object):
    """Convert ndarrays in sample to Tensors."""

    def __call__(self, sample):
        image, pls, tr_angles, positions = \
            sample['image'], sample['pls'], sample['tr_angles'], sample['positions']
        image = image[np.newaxis, :, :]
        # swap color axis because
        # numpy image: H x W x C
        # torch image: C X H X W
        # image = image.transpose((2, 0, 1)) # single channel
        return {'image': torch.from_numpy(image),
                'pls': torch.from_numpy(pls),
                'tr_angles': torch.from_numpy(tr_angles),
                'positions': torch.from_numpy(positions)
                }


</code></pre>
Feature extraction using alexnet

username: Nikhil_Kaishal
cooked: <p>I am right now trying to implement a project called face recognition on google collab where I want to do feature extraction using AlexNet model and save the feature extraction vectors in a csv file. Then apply SVM for classificiation. I am struck at downloading and applying AlexNet model in google collab.Could anyone please provide step to step guidelines for implementing AlexNet for feature extraction using google collab</p>

username: Eta_C
cooked: <p>Maybe…you can just train AlexNet as a classifier…and then remove the classifier layers…I think there must have some tutorials…</p>
How to get center of features?

username: 1414b35e42c77e0a57dd
cooked: <p>In supervised manner, How can I get center of the features on each classes?<br>
Do I need to use clustering algorithms like k-means?<br>
or Can I get center feature using only labeled features?</p>

username: Eta_C
cooked: <p>Why use k-means in supervised manner?</p>

username: 1414b35e42c77e0a57dd
cooked: <aside class="quote no-group" data-username="1414b35e42c77e0a57dd" data-post="1" data-topic="65630">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/1414b35e42c77e0a57dd/48/7689_2.png" class="avatar"> 1414b35e42c77e0a57dd:</div>
<blockquote>
<p>features</p>
</blockquote>
</aside>
<p>How can I get the center of the features per classes?<br>
in mini-batch,<br>
outputs = model(inputs)<br>
(sum of all outputs which have same class) / mini_batch_size is center?</p>

username: Eta_C
cooked: <p>Sorry, I don’t know.<br>
I am currently studying face recognition. In face recognition, such as ArcFace, <strong>center of features of each class</strong> is a learnable matrix, which shape is (embedding_len, num_classes).</p>
Applying ToTensor() rotates the image CW 90degrees

username: n0obcoder
cooked: <pre><code class="lang-auto">import torch
from PIL import Image
from torchvision import transforms

img = Image.open(filename)
print(img.size)
content_transform = transforms.Compose([
        transforms.ToTensor(),
        transforms.Lambda(lambda x: x.mul(255))
    ])

content_image = content_transform(content_image)
print(content_image.shape)
</code></pre>
<p>When i load an image using PIL and appply ToTensor() transform on it, the tensor rotates 90degrees Clockwise</p>
<pre><code class="lang-auto">img.size: (2592,1944)
content_image.shape: torch.Size([3,1994,2592])
</code></pre>
<p>Any idea why is this happening or how can i fix this?<br>
Thanks in advance <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: ptrblck
cooked: <p><code>PIL</code> returns the size as <code>[width, height]</code> without the number of channels.<br>
<code>ToTensor()</code> permutes the image array to <code>[channels, height, width]</code>, which is the expected input shape for image tensors for 2-dimensional modules, so your shape looks alright. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
How does copy.deepcopy initilize modules?

username: penelope24
cooked: <p>From my understanding, deepcopy copies modules with the same initial params.<br>
So is it unsafe to use copy.deep copy if I want modules with same architecture but are initialized independently?</p>

username: ptrblck
cooked: <p><code>copy.deepcopy</code> should copy the model directly, i.e. with exactly the same values in all parameters. The copied module will therefore not be reinitialized.<br>
Let me know, if I misunderstood your question.</p>
Overwhelmed by PyTorch and NLP

username: Raphy-N
cooked: <p>I’m not sure if this is a suitable forum for seeking general help and advice. I’m new to the field of Applied DL in NLP. I’ve studied basics of DL and implemented CNNs in Keras. I’ve started a project which involves trying out various NN architectures (LSTMs, Bi-LSTM-CNN-CRFs, Transformers-BERT etc.) on news articles data.</p>
<p>However, I’m being overwhelmed by the many architectures to understand and especially the lengthy PyTorch codes. There aren’t many online courses that teach how to implement advanced models in PyTorch and I just can’t make sense of how to use GitHub repos provided in research papers like “Attention is All you Need” or that of BERT (Devlin et al.)</p>
<p>Any help related to this would be highly appreciated. Thanks</p>

username: vdw
cooked: <p>Since I’m kind of in the same boat, here is just my personal opinion: At the end of the day, you simple have to bite the bullet. Deep learning and Natural Language Processing are both complex topics on their own and even more so when put together. Here just some pointers based on my personal experience:</p>
<ul>
<li>
<p><strong>Understand the basics:</strong> Many modern architectures simple say that “…we use attention…” like it is a well-established concept. And maybe it is. Still, as a newcomer it’s important to go back to see where attention is coming from. It’s often too easy to simply copy/paste a tutorial or other existing code that does something without really knowing what. I’m certainly guilty of that.</p>
</li>
<li>
<p><strong>Be critical/skeptical of available code:</strong> With AI/DL/ML such hot topics, everyone seems to get on the bandwagon. The “problem” is that it’s actually very easy to get started. Install PyTorch, download a dataset, copy some tutorial code, and you quickly get first results. However, I’ve seen a lot of code on Github that was not correct. As long as the dimensions of your tensors you pass between layers, no errors are thrown. However, that doesn’t mean that it’s correct. For example, take a multilayer LSTM. A subsequent linear layer doesn’t care if you give it the hidden state of the first (wrong) or the last (correct) layer. Another common culprit is a “blind” usage of, e.g., <code>view()</code> just to get the dimensions right, no matter how.</p>
</li>
<li>
<p><strong>Even peer-reviewed work is not perfect:</strong> The “publish or perish” culture in academia is alive and well, and again, AI/DL/ML is the topic of current times. I’ve read papers that are simply not doing what they promise – that’s not just armchair reasoning; I re-implemented and evaluated them. It’s also too easy to make established network models “just a bit more complex” and publish the results again.</p>
</li>
</ul>
<p>In short, feeling overwhelmed is natural. New architectures – or at least sold as such – are presented almost weekly.</p>

username: zhangguanheng66
cooked: <p>If you are new to PyTorch, there are a lot of tutorials to understand the basics and give some models a try. Some tutorials may be useful <a href="https://pytorch.org/tutorials/" rel="nofollow noopener">here</a>.</p>
Transfer Learning: Fine tuning the convnet vs ConvNet as feature extractor

username: ucals
cooked: <p>In the <a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html#convnet-as-fixed-feature-extractor" rel="nofollow noopener">Transfer Learning for Computer Vision Tutorial: ConvNet as fixed feature extractor</a>, do we need the <code>.fc</code> in the code below?</p>
<pre><code class="lang-auto"># Observe that only parameters of final layer are being optimized as
# opposed to before.
optimizer_conv = optim.SGD(model_conv.fc.parameters(), lr=0.001, momentum=0.9)
</code></pre>
<p>Or it can be re-written using <code>model_conv.parameters()</code>?</p>
<p>I’m asking because I tested with and without and results look like the same. And most important, as the tutorial had previously frozen all parameters from all layers except the ones in the <strong>fc</strong>, <code>model_conv.fc.parameters()</code> should be equivalent to <code>model_conv.parameters()</code>.</p>
<p>Can someone confirm please? Thanks!</p>

username: ptrblck
cooked: <p>The behavior will be the same in this use case. However, have a look at <a href="https://discuss.pytorch.org/t/passing-params-to-an-optimizer/63191/2">this post</a> for a bit more detail on edge cases.</p>
Can I work with torchaudio in windows

username: Vijay
cooked: <p>I can’t import torchaudio module in python 3 base conda does it support windows or it works then how do I use I am new in pytorch</p>

username: vincentqb
cooked: <p>Thanks for asking! We don’t officially offer binaries yet for torchaudio in windows, see <a href="https://github.com/pytorch/audio/issues/50" rel="nofollow noopener">open issue</a>. You can try compiling from source using the instructions <a href="https://github.com/pytorch/audio/blob/master/README.md" rel="nofollow noopener">here</a>. Please do post about issues you may run into while doing so <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Probability regression models

username: Bhavya_Bhatt
cooked: <p>I have 2-d data points and have probabilities for those data points (which are of the order of 10e-8). Now I have some functional form of the probability distribution (which has a complicated parameterisation of mean and covariance matrix - total number of parameters are 14). Now I want to find optimal parameters of the model which fits these observed probabilities. I am using PyTorch for doing this.</p>
<p>total number of data points: 6000<br>
for 2-D data points I have<br>
range of column 1: 0 to 6<br>
range of column 2: 0 to 5</p>
<p>I am currently using mean squared error (which is probably wrong) and the loss at initialisation is very large and with epochs loss is decreasing and is converging to some value which is still very large<br>
for eg. from 10e12 to 10e8.<br>
I tried for 1000 epochs but nothing much improved. I even tried KL divergence loss but in that the loss instead of converging showed fluctuations. I am using Adam optimiser.</p>
<p>I have couple of questions:</p>
<ul>
<li>How to initialise my parameters ?</li>
<li>Appropriate loss function for such probability regression problems.</li>
<li>Will SGD be a good optimisation algorithm for this task.</li>
<li>And what other optimisation algorithms are good for such tasks.</li>
<li>Standard models (and literature pointer) for such probability regression problems.</li>
</ul>
<p>I am new to this problem and searched a lot of resourced but did not got any satisfactory answer and have been stuck for many days now.</p>
<p>Thanks in advance.</p>
Efficient way to obtain values from specified indexes

username: G.M
cooked: <p>I have a tensor A of shape(L, W). I also have another tensor B of shape(L, 1). I want to use B as indexes to select from A, which would result in a tensor C of shape(L). For example, say A is:<code>[ [0, 1], [2, 3], [5, 6] ]</code>, and B is<code>[ [1], [1], [0] ]</code>, I want to make C contains: <code>[1, 3, 5]</code>. I didn’t find any function that suits my needs. Currently, I’m achieving it by doing this:</p>
<pre><code class="lang-auto">        mask: torch.Tensor = torch.zeros_like(A, dtype=torch.uint8)
        mask.scatter_(1, B, 1)
        C = A.masked_select(mask)
</code></pre>
<p>I feel that this method isn’t very efficient as it first needs to convert B into one-hot and then selecting. So is there a better way?</p>

username: Eta_C
cooked: <p>Try this</p>
<pre><code class="lang-python">import torch
A = torch.tensor([[0, 1], [2, 3], [5, 6]]).float()
B = torch.tensor([[1], [1], [0]]).long()
print(torch.gather(A, 1, B))
</code></pre>

username: G.M
cooked: <aside class="quote no-group" data-username="Eta_C" data-post="2" data-topic="65593">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/eta_c/48/17667_2.png" class="avatar"> Eta_C:</div>
<blockquote>
<p>B = torch.tensor([[1], [1], [0]]).long()</p>
</blockquote>
</aside>
<p>It works, thx. I was long troubled by what does torch.gather do <img src="https://discuss.pytorch.org/images/emoji/apple/rofl.png?v=12" title=":rofl:" class="emoji" alt=":rofl:" loading="lazy" width="20" height="20"></p>
LSTM Cell weights from LSTM

username: Binbose
cooked: <p>How can I load the weights of an LSTM module to an LSTMCell?</p>
<p>I tried :<br>
<code>self.lstm_cell.load_state_dict(self.lstm.state_dict)</code></p>
<p>but I get the following error:</p>
<pre><code class="lang-auto">self.lstm_cell.load_state_dict(self.lstm.state_dict)
  File "/home/luca/anaconda3/envs/rdkit_env/lib/python3.7/site-packages/torch/nn/modules/module.py", line 812, in load_state_dict
    state_dict = state_dict.copy()
AttributeError: 'function' object has no attribute 'copy'
</code></pre>

username: G.M
cooked: <p>You should probably take another look at the Doc. I remembered state_dict is not a attribute nor a property: it is a function, as described in the traceback. So, you should use <code>state_dict = lstm.state_dict()</code></p>

username: Binbose
cooked: <p>Hi, true, that was stupid. However, now I get the error:</p>
<pre><code class="lang-auto">RuntimeError: Error(s) in loading state_dict for LSTMCell:
	Missing key(s) in state_dict: "weight_ih", "weight_hh", "bias_ih", "bias_hh". 
	Unexpected key(s) in state_dict: "weight_ih_l0", "weight_hh_l0", "bias_ih_l0", "bias_hh_l0".
</code></pre>
<p>I guess this is, because a LSTM has several layer, and the l0 indicates, that it is the first one?</p>

username: G.M
cooked: <p>Yes.</p>
<p>Also, say u have 2 lstm: A, B. These two  have different structures. I guess what u r doing is using the weights from A to initialize B, which of course won’t work.</p>
Debugging `RuntimeError: dependency not found for torch::autograd::AccumulateGrad`

username: Xuechen_Li
cooked: <p>I’m getting this error when running <code>torch.autograd.grad</code> in the static <code>backward</code> function of <code>torch.autograd.Function</code>.</p>
<p>Since my codebase is too large, I won’t be posting it here. But as clearly stated, this runtime error should be coming from the C end, and I would like to get some advice on debugging it. Thanks!</p>

username: Xuechen_Li
cooked: <p>I could be wrong, but this seems like a synchronization bug. I have searched everywhere on the web, and haven’t found any related issues.</p>

username: Xuechen_Li
cooked: <p>Moreover, the crash type seems to be non-deterministic. I’m sometimes getting this runtime error, but at other times getting segfault.</p>

username: albanD
cooked: <p>Can you give more details on what you are doing? Do you have a code sample?</p>
Debugging mutex lock error

username: Xuechen_Li
cooked: <p>Hi, when using <code>torch.autograd.Function</code>, I am getting the following error:</p>
<pre><code class="lang-auto">python3: ../nptl/pthread_mutex_lock.c:117: __pthread_mutex_lock: Assertion `mutex-&gt;__data.__owner == 0' failed.
</code></pre>
<p>Are there any suggested ways to debug this?</p>

username: albanD
cooked: <p>Hi,</p>
<p>Do you have a code sample that reproduces this please?</p>
Reading image from url is taking too long

username: Upgrade_Yourself
cooked: <p>Here’s the code I used:</p>
<pre><code class="lang-auto">import requests
from PIL import Image
import io
import cv2
response = requests.get(df1.URL[0]).content
#print(response)

im = Image.open(io.BytesIO(response))

</code></pre>
<p>The image is very large. Is there a way to fasten things?</p>

username: ptrblck
cooked: <p>I’m not sure, if it’s possible to speedup <code>request</code> calls, and I think you might get a better answer on e.g. stackoverflow, as this is not PyTorch-specific. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: Upgrade_Yourself
cooked: <aside class="quote no-group" data-username="Upgrade_Yourself" data-post="1" data-topic="65577" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/upgrade_yourself/48/17961_2.png" class="avatar"> Upgrade_Yourself:</div>
<blockquote>
<p>Here’s the code I used:</p>
<pre><code class="lang-auto">import requests
from PIL import Image
import io
import cv2
response = requests.get(df1.URL[0]).content
#print(response)

im = Image.open(io.BytesIO(response))

</code></pre>
<p>The image is very large. Is there a way to fasten things?</p>
</blockquote>
</aside>
<p>Thanks I’ll do that.</p>
LSTM input and output dimentions

username: VikasRajashekar
cooked: <p>I am a bit confused about LSTM input and output dimensions</p>
<pre><code class="lang-auto">Here is my network:
Intent_LSTM(
  (embedding): Embedding(41438, 400)
  (lstm): LSTM(400, 512, num_layers=2, batch_first=True, dropout=0.5)
  (dropout): Dropout(p=0.5, inplace=False)
  (fc): Linear(in_features=512, out_features=3, bias=True)
).
</code></pre>
<p>Also this is my forward pass:</p>
<pre><code class="lang-auto">def forward(self, x):
        """
        Perform a forward pass

        """
        batch_size = x.size(0)

        x = x.long()
        embeds = self.embedding(x)

        lstm_out, hidden = self.lstm(embeds)

    
        # stack up lstm outputs
        lstm_out = lstm_out.contiguous().view(-1, self.hidden_dim)


        out = self.dropout(lstm_out)
        out = self.fc(out)


        
        # reshape to be batch_size first
        out = out.view(batch_size, -1,3)
        #print("sig_out",sig_out.shape)
        out = out[:, -1,:] # get last batch of labels
        
        # return last sigmoid output and hidden state
        return out
</code></pre>
<p>Here the shape of my embdeddings is : [50, 150, 400] 50 being batch size, 150 seq lenth of my input. 400 being my embedded dimensions. I am feeding this into my LSTM. But when I was going through pytorch documentation. It states that input has to be in the form :</p>
<p><a href="https://pytorch.org/docs/stable/nn.html" rel="nofollow noopener"><code>**input**  of shape (seq_len, batch, input_size)</code></a></p>
<p>So should the input be converted to this format. ([150,50,400]) ?</p>
<p>If yes how do I do that?</p>

username: vdw
cooked: <p>You define your LSTM with <code>batch_first=True</code>. This changes the expected dimensions if the input to <code>(batch, seq_len, input_size)</code>. Try setting <code>batch_first=False</code>, and you will see that an error is thrown due to incorrect dimensions.</p>
Multi-label text classification

username: Er_Hall
cooked: <p>Hi all,</p>
<p>Can someone explain me what are the various strategies for solving text multilabel classification problems with Deep Learning models?</p>
<p>Is it right to “convert” the problem to multiclass classification problem? What I mean?</p>
<p>If for example I have 3 labels and an instance can belong to one, two or even three labels or a combination of these 3 labels I can convert the problem as a multiclass classification problem of 7 classes:<br>
(A), (B), ©, (AB), (AC), (BC), (ABC).</p>
<p>Is it right? If not how can I handle it?</p>

username: vdw
cooked: <p>The problem I see with this approach is that the 7 classes are not independent. Admittedly, I cannot give a solid, “scientific-y” explanation why this causes issues.</p>
<p>Intuitively, I would go with a “one-vs-rest” approach – have 3 binary classifiers, one for each label A, B, and C:</p>
<ol>
<li>Should the text be labelled with A? Yes/No</li>
<li>Should the text be labelled with B? Yes/No</li>
<li>Should the text be labelled with C? Yes/No</li>
</ol>
<p>You only need to adjust your training data for each of the 3 classifiers accordingly.</p>
Pytorch multiprocessing CUDA Initialization error

username: Peter_Ham
cooked: <p>I’m trying to run multiple threads in pytorch with GPU enabled. In each thread, I am trying to create a CUDA tensor from numpy array using the following code:</p>
<p>tensor = torch.from_numpy(array).cuda().float()</p>
<p>this triggers the following error report:</p>
<p>RuntimeError: CUDA error: initialization error</p>
<p>Any help would be greatly appreciated!</p>

username: Peter_Ham
cooked: <p>sorry I just solved this problem by using</p>
<pre><code class="lang-auto">    mp.set_start_method('spawn')
</code></pre>
About the behavior of dataloader

username: phantom90
cooked: <p>Hi there,</p>
<p>I am current using the dataloader (torch.utils.data.DataLoader) to get batches for model training.</p>
<p>By default, is each sample selected exactly once in one epoch? A epoch is defined as follows:</p>
<pre><code class="lang-auto">for epoch in range(10):
    for j, (inputs, labels, _) in enumerate(dataloader):
        something
</code></pre>
<p>Thanks!</p>

username: ptrblck
cooked: <p>By default the <code>DataLoader</code> will use a <code>SequentialSampler</code>, if <code>shuffle=False</code>, otherwise <code>RandomSampler</code> with the default argument <code>replacement=False</code>, which would yield each sample only once in each epoch.</p>

username: phantom90
cooked: <p>Just one more question:</p>
<p>In the follow combinations, each sample is selected <strong>exactly once</strong> in one epoch, is that correct?</p>
<ol>
<li>Default DataLoader  with  SequentialSampler, shuffle=False</li>
<li>Dataloader with RandomSampler, shuffle=True, replacement=False.</li>
</ol>
<p>Thanks again.</p>

username: ptrblck
cooked: <p>Yes, that’s correct.</p>