akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How can I download torchaudio?

username: Victor_Dobra
cooked: <p>I am currently trying to download torchaudio for Conda to train an RNN on audio, but I can’t download it. I used the command <code>conda install -c pytorch torchaudio </code>, and also downloaded all of the required libraries, but when I try to download it, it says <code>PackagesNotFoundError: The following packages are not available from current channels: torchaudio</code>. Why does it happen, and how can I download torchaudio succesfully?</p>

username: albanD
cooked: <p>Hi,</p>
<p>Which python version are you using?<br>
Also have you tried using pip or installing the nightly builds?</p>

username: Victor_Dobra
cooked: <p>I am using Python 3.7, and yes, I have tried to use pip and install the nightly builds and it didn’t work.</p>

username: albanD
cooked: <p>This is surprising indeed, have you tried in an empty conda environment? You might have conflicting packages?</p>
<p><a class="mention" href="/u/vincentqb">@vincentqb</a> is there something special about the packages?</p>

username: Victor_Dobra
cooked: <p>Yes, I have tried and it doesn’t work.</p>

username: mpariente
cooked: <p><code>pip install torchaudio</code> works at this moment, do you have an error with it?</p>

username: Victor_Dobra
cooked: <p>I have found out that torchaudio is not compatible with Windows 10, which I am using. Thank you for the help, anyway.</p>

username: vincentqb
cooked: <p>What platform are you using? torchaudio is currently available for linux and macos. Windows support is in progress, see <a href="https://github.com/pytorch/audio/issues/425" rel="nofollow noopener">this</a>.</p>

username: peterjc123
cooked: <p>It is usable but without mp3 support.</p>
Backward very slow

username: netaglazer
cooked: <p>hi, i built a loss function, and the backward is very slow:</p>
<pre><code class="lang-auto"> def loss_function(p, y, b):
     eps = 0.0000000001
     losses = 0
     k = len(y[0])
     ones = torch.ones(k).cuda()
     for i in range(b):
         loss1 = -((ones-y[i])@(((ones-p[i])+(eps)).log()))
         prod = (ones-y[i])*k - y[i]*((p[i]+ eps).log())
         loss2 = torch.min(prod)
         losses = losses + (loss2 + loss1)
     return (losses/b)
</code></pre>
<p>and also, what should be the ratio between the size of the data, and the size of the batch?<br>
i have ~300,000 rows in my dataset, maybe a better choice of batch size would help here?</p>
<p>my model is a CNN, with 3 conv layers and 2 linear layers</p>

username: albanD
cooked: <p>Hi,</p>
<p>The most likely reason is that the for loop that you use in your loss function is slow and creates many small operations in the backward as well.<br>
Maybe you want to change it to work with the whole batch at once?</p>

username: netaglazer
cooked: <p>how?<br>
this is a multi-label problem, so i calculate the distance between each pair of prediction+label in the loop</p>

username: albanD
cooked: <p>Assuming that your inputs are 2D, the code below shows how to do it</p>
<pre><code class="lang-python"> import torch

def loss_function(p, y, b):
    eps = 0.0000000001
    losses = 0
    k = len(y[0])
    ones = torch.ones(k)
    for i in range(b):
        loss1 = -((ones-y[i])@(((ones-p[i])+(eps)).log()))
        prod = (ones-y[i])*k - y[i]*((p[i]+ eps).log())
        loss2 = torch.min(prod)
        losses = losses + (loss2 + loss1)
    return (losses/b)

def new_loss_function(p, y, b):
    eps = 0.0000000001
    losses = 0
    k = len(y[0])
    ones = torch.ones(1, 1).expand(b, k)
    loss1 = -((ones-y)*(((ones-p)+(eps)).log())).sum(dim=1)
    prod = (ones-y)*k - y*((p+ eps).log())
    loss2 = torch.min(prod, dim=1)[0]
    losses = (loss1 + loss2).sum()
    return losses / b

b = 10
k = 5

# Are there the right input size?
p = torch.rand(b, k)
y = torch.rand(b, k)

old_res = loss_function(p, y, b)
new_res = new_loss_function(p, y, b)

print((old_res - new_res).abs().max())


</code></pre>

username: netaglazer
cooked: <p>thank you so much!</p>
<p>just a little question:<br>
why torch.ones(1,1).expand(x,y) and not torch.ones(x,y)?<br>
does it matter?</p>

username: albanD
cooked: <p>This is just a small trick to reduce memory use. The <code>expand</code> function does not actually allocate new memory but use tricks on stride to get the final Tensor.<br>
You can use both and should not see any difference in the result. The one with expand will just use a bit less memory.</p>

username: netaglazer
cooked: <p>hi, i have one more question about my loss, maybe you could help me</p>
<p>do you now how can i add L2 regularization to thin loss?<br>
tnx!</p>

username: albanD
cooked: <p>The simplest way is just to use the <code>weight_decay</code> parameter of the optimizer if you use one from torch.optim.</p>
<p>Otherwise, just compute the l2 by hand and add it to the loss before running the backward.</p>
Why does CrossEntropyLoss consume so many memory in Transformer

username: Walter1
cooked: <p>I’m trying to train a Transformer here with limited resource.</p>
<p>In general way, the loss function is:</p>
<pre><code class="lang-auto">lang_prediction_scores = self.cls(feats)

masked_lm_loss = CrossEntropyLoss(ignore_index=-1)(
                lang_prediction_scores.view(-1, self.config.vocab_size),
                masked_lm_labels.view(-1)
            )
</code></pre>
<p>Where ignore_index ignores all label with -1.</p>
<p>But I find this cost many GPU memory, whitch is unacceptable given my limited GPU memory. For example, when the length of tokens to be computed in loss function is 36, the training is fine. But when up to 72, It will consume 6GB more GPU meomory more or less. The number of input tokens is always 72, it’s just the tokens in loss fuction changed here.</p>
<p>I find a solution for this. Since transformer doesn’t really care for order, I move all tokens to be predicted in together. Then I just need predict much less tokens. Like this:</p>
<pre><code class="lang-auto">lang_prediction_scores = self.cls(feats[:, TOKENS_TO_BE_PREIDCTED:])

masked_lm_loss = CrossEntropyLoss(ignore_index=-1)(
                lang_prediction_scores.view(-1, self.config.vocab_size),
                masked_lm_labels.view(-1)
            )
</code></pre>
<p>Now this reduce the memory cost in a considerate way, allow me continue to train the model.</p>
<p>Now my question is:</p>
<ol>
<li>Why CrossEntropyLoss costs so many memory? It’s just a loss function, and ignore all index with -1 anyway. In my opinion, loss will be summed and then averaged. Should it not consume more memory in the backward process?</li>
<li>Am I doing this correct? Does this have a difference in computing gradient than just use ignore_index?</li>
</ol>
<p>Really need help here. <img src="https://discuss.pytorch.org/images/emoji/apple/neutral_face.png?v=9" title=":neutral_face:" class="emoji" alt=":neutral_face:"></p>
<p>To clear something up, in the example above, the input token is 72 always. I just need to compute CrossEntropyLoss on all tokens now. So this memory consumption is not caused by the MaxLength. I’m confident the number of tokens in loss function is primary cause here.</p>

username: zhangguanheng66
cooked: <p>One of the reasons, I think about, is that the attention weights and so on within transformer model are doubled when the input length is doubled. Depending on the task, it may increase it by a factor of four.</p>

username: Walter1
cooked: <p>As I mentioned above, the input length doesn’t change. The only thing changes is the number of tokens computed in CrossEntropyLoss.<br>
That’s wierd.</p>
Building Libtorch from source

username: copythatpasta
cooked: <p>Hello all,<br>
I am attempting to rebuild pytorch from source to get the standalone libtorch libraries(c++)<br>
I have checked out v1.3.0 and compiled with gcc 4.8. How do i link the libs/ and includes/ for a makefile without using cmake? Also which version of Pytorch would have a better libtorch build for gcc 4.8.5?<br>
This is my requirement for compiler.</p>
<p>Also, have noticed that when building libtorch using build_libtorch script on the newest version, the TorchVersion file indicates it builds Torch 1.1. However if i build pytorch using the setup.py build command, the libtorch version will match the checked out version. Whats going on here?</p>
Parallelization of multiple cost functions

username: daner
cooked: <p>Hi</p>
<p>I have a RNN and a set of time series data organized in n sequences as (y_1, u_1), (y_2, u_2), …, (y_n, u_n), where y_i is vector of data outputs and u_i is matrix where each column is a signal and each row a time sample of all signals.</p>
<p>The cost function to be minimized is in the form of a sum</p>
<p>min f_1(y_1, u_1) +  f_2(y_2, u_2) + f_3(y_3, u_3) + … + f_n(y_n, u_n)</p>
<p>where each cost function f_i is different for each sequence i.</p>
<p>I was wondering if someone has any experience, or can help me find where to start look how to make each term f_i(y_i, u_i) in the cost-function to be evaluated in parallel using multiple cpus/gpus?</p>

username: mrshenli
cooked: <p>Hey <a class="mention" href="/u/daner">@daner</a> if you are using a single machine with multiple GPUs, you can try scatter + parallel_apply + gather. The implementation of <code>DataParallel</code> can serve as an example. [<a href="https://github.com/pytorch/pytorch/blob/df8d6eeb19423848b20cd727bc4a728337b73829/torch/nn/parallel/data_parallel.py#L151-L156" rel="nofollow noopener">link</a>]</p>
Compressing Emebedding Size

username: mohitsharma29
cooked: <p>I have a question for all of you. In NLP, say you use some embedding method. and say for a paragraph you get say 768*n matrix, where n is dependent on, let’s say the number of words or sentences or context window of words<br>
And you want to work in lower dimensions. What are some ways to reduce this dimensionality, one way is to take mean across each axis to reduce it to a 768 dimension vector?</p>
Does fast.ai teach me how to use Pytorch because it seems fast.ai teaches their library?

username: JonathanSum
cooked: <p>I am currently taking lesson 2 on Fast.AI, and I see they teach GAN later too. However, it seems to be what they are using is not Pytorch, but their library or a library on top of Pytorch. But I just want to learn Pytorch that I can do like GAN, Time series prediction, NPL and more. Is taking their course to allow me to learn Pytorch more deeply in these fields?</p>

username: snip3r77
cooked: <p>I’m kinda half way through fast.ai but I think I want to learn the proper steps as fast.ai I feel is the ‘wham bam’ type. What is good I think it’s the one-cyce-policy that is implemented but I did do a search and there are 3rd party codes for this in pytorch, my 2 cents</p>
How to decode labels generated by pytorchvision

username: SamyTahar
cooked: <p>I am new on Pytorch and I was wondering how to <strong>decode</strong> the <strong>labels</strong> when using <strong>pytorchvision</strong> and <strong>datasets.ImageFolder</strong>.</p>
<p>As far I understood that <strong>labels</strong> are <strong>created</strong> with the image <strong>folder</strong> <strong>name</strong></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/1/f1c011c5ae9dc7f28019c8c0d0ee6e648105f2a9.png" data-download-href="https://discuss.pytorch.org/uploads/default/f1c011c5ae9dc7f28019c8c0d0ee6e648105f2a9" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/1/f1c011c5ae9dc7f28019c8c0d0ee6e648105f2a9_2_613x500.png" alt="image" data-base62-sha1="yuCzMPiGOHQAtww9Irwn0xzNoy5" width="613" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/1/f1c011c5ae9dc7f28019c8c0d0ee6e648105f2a9_2_613x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/1/f1c011c5ae9dc7f28019c8c0d0ee6e648105f2a9_2_919x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/f/1/f1c011c5ae9dc7f28019c8c0d0ee6e648105f2a9.png 2x" data-dominant-color="E9F0F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1102×898 80.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I have <strong>43 folders</strong> so <strong>43 classes</strong> that are loaded the folder <strong>structure</strong> is the following <strong>data_img/train/Classes/image_from_classes</strong>.</p>
<p><strong>Here is the code line</strong> that i have used to load the images:<br>
<code>train_data = datasets.ImageFolder('data_img/train/', transform=train_transforms)</code></p>
<p>When the <strong>model</strong> is <strong>calculated</strong> and that I try to <strong>pass</strong> an <strong>image</strong> and <strong>predict</strong> the result I was <strong>expecting</strong> the <strong>label</strong> <strong>name</strong> however I have a <strong>tensor</strong> as a <strong>return</strong> “Image predicted as tensor([15])”.</p>
<p>I was <strong>wondering</strong> how to <strong>decode</strong> this number to <strong>get</strong> the <strong>class</strong> <strong>name</strong>?</p>
<p>Thanks for your help,<br>
Samy</p>

username: yash1994
cooked: <p>Hi,</p>
<p>You can use <code>class_to_idx</code> attribute of <code>ImageFolder</code>. So <code>train_data.class_to_idx</code> will give you a dictionary with key as class name and id as its value. You can find class names by checking ids within this dictionary.</p>

username: SamyTahar
cooked: <p>Thanks I will try that right away.</p>
<p>Samy</p>
Combined CNN and LSTM

username: GerryNoble69
cooked: <p>New to Pytorch and CNN</p>
<p>What is the best way to combine a CNN with a LSTM model?</p>
<p>Should I define the CNN class<br>
Define the LSTM class<br>
Then define a model that combines both</p>
<p>OR</p>
<p>Should I wrap them all in one class?</p>
<p>Thanks in advance for response.</p>
<p>Gerry</p>

username: ptrblck
cooked: <p>I would argue the implementation depends on your coding style and how you would like to use the model.<br>
E.g. if you want to call the submodules separately in the future or would like to easily inspect the output of the first module, you could use the split approach.<br>
However, usually you would implement the model as once class.</p>

username: GerryNoble69
cooked: <p>Many thanks for replying  Greatly appreciated.</p>
<p>Do you have any suggestions about where I could find good examples of combined Models…or a good site/book to read on the subject matter f combination models.</p>
<p>So far I’m using pytorch site, deeplizard, towards datascience  and blog posts.</p>
<p>Many thanks</p>
<p>Gerry</p>

username: ptrblck
cooked: <p>We have a couple of forum posts about <a href="https://discuss.pytorch.org/search?q=ConvLSTM">ConvLSTM</a>, so these posts might have a good starter code.</p>

username: GerryNoble69
cooked: <p>Thanks again…it’s kind of you to help others starting out.  Greatly appreciated.</p>
Best way to index along a dimension

username: dhecloud
cooked: <p>Hi, apologies if this has been asked before.</p>
<p>I have a tensor <code>x</code> of shape <code>(batch_size, seq_len, vocab_size)</code>, say shape <code>(1, 3, 2)</code>. I have a tensor <code>i</code> which is <code>(batch_size, vocab_indices)</code> , say shape <code>(1,3)</code> which are the indices of the vocab size dimension in <code>x</code>.  I want to get values from the last dimension (vocab_size).</p>
<p>For example,</p>
<pre><code class="lang-auto">x = torch.tensor( [[ [0.1, 0.2], [0.3, 0.4], [0.5, 0.6] ]]) # shape = (1,3,2)
i = torch.tensor([[1,0,1]]) # shape = (1,3)
</code></pre>
<p>I want to get</p>
<pre><code class="lang-auto">y =  torch.tensor([[0.2,0.3,0.6]]) #shape = (1,3)
</code></pre>
<p>Is there a clean way to do this other than using a for loop? Thank you very much!</p>

username: ptrblck
cooked: <p>This should work:</p>
<pre><code class="lang-python">x = torch.tensor( [[ [0.1, 0.2], [0.3, 0.4], [0.5, 0.6] ]]) 
i = torch.tensor([[1,0,1]])

x[:, torch.arange(x.size(1)), i]
</code></pre>

username: dhecloud
cooked: <p>Hi, thanks for your reply!</p>
<p>I tried your solution, but for more than one batches, ie like</p>
<pre><code class="lang-auto">x = torch.tensor( [[ [0.1, 0.2], [0.3, 0.4], [0.5, 0.6]], [ [0.7, 0.8], [0.9, 1.0], [1.1, 1.2]]]) # shape = (2,3,2)
i = torch.tensor([[1,0,1],[1,1,1]]) # shape = (2,3)
</code></pre>
<p><code>x[:, torch.arange(x.size(1)), i]</code> broadcasts the last dim and gives</p>
<pre><code class="lang-auto">tensor([[[0.2000, 0.3000, 0.6000],
         [0.2000, 0.4000, 0.6000]],

        [[0.8000, 0.9000, 1.2000],
         [0.8000, 1.0000, 1.2000]]])
# shape (2,2,3)
</code></pre>
<p>where it should be</p>
<pre><code class="lang-auto">tensor([[0.2000, 0.3000, 0.6000],
         [0.8000, 1.0000, 1.2000]])
#shape (2,3)
</code></pre>
<p>was trying to index it again but i guess i still need a loop?</p>

username: ptrblck
cooked: <p>No, you wouldn’t need a for loop and for batched input this should work:</p>
<pre><code class="lang-python">x[torch.arange(x.size(0)).unsqueeze(1), torch.arange(x.size(1)), i]
</code></pre>

username: dhecloud
cooked: <p>thanks so much, this worked like a charm!</p>
How to calculate accuracy

username: netaglazer
cooked: <p>hi, i have a semi - multi label problem</p>
<p>my specific problem is a bit different from a classic multi-label problem<br>
i  <strong>want</strong>  to minimize my loss when the prediction is correct in only one class (or more)<br>
i  have a costume loss for my problem:</p>
<pre><code class="lang-auto">def new_loss_function(p, y, b):
    eps = 0.000000000000000000000000001
    losses = 0
    k = len(y[0])
    ones = torch.ones(1, 1).expand(b, k).cuda()
    loss1 = -((ones-y)*(((ones-p)+(eps)).log())).sum(dim=1)
    prod = (ones-y)*k - y*((p+ eps).log())
    loss2 = torch.min(prod, dim=1)[0]
    losses = (loss1 + loss2).sum()
    return losses / b
</code></pre>
<p>it means, if the model was right in one class:<br>
label = [1,1,0,0,1]<br>
predication = [1,0,0,0,0]<br>
<strong>in my case</strong> this is a success</p>
<p>im not sure how to calculate the accuracy of the model in that case</p>

username: ptrblck
cooked: <p>Based on your description you could probably use:</p>
<pre><code class="lang-python">if (prediction == label).any():
    nb_correct += 1
</code></pre>
<p>to calculate the number of correct samples and the accuracy by dividing it by the number of samples.</p>

username: netaglazer
cooked: <p>without threashold?<br>
the predictions are between 0 and 1</p>

username: ptrblck
cooked: <p>In your first post you’ve posted the predictions as zeros and ones, so I assumed you’ve already applied a threshold. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
<p>If that’s not the case, you should use a threshold to get the predictions.<br>
The simplest case would be <code>0.</code> for logits and <code>0.5</code> for probabilities (after <code>sigmoid</code>).<br>
The threshold can then be tuned using the ROC etc.</p>

username: netaglazer
cooked: <p>thank you for your answer!<br>
i just wonder why after sigmoid?</p>

username: ptrblck
cooked: <p>If you are using a sigmoid activation for your model output, you could use the default threshold of 0.5.<br>
On the other hand, if you are returning the raw logits, you could use 0.0.</p>
__getitem__ indexes following a prepared file/list

username: barakb
cooked: <p>Hi, I want that my dataloader will follow and sample indexes by using a prepared order.<br>
Means, I don’t want my dataloader to sample indexes randomly and not by sequential order, but by using a file or list that I prepared from advance.<br>
Any idea of how to implement such a thing?<br>
Thanks.</p>

username: simaiden
cooked: <p>You could prepare a dictionary that map the index in <strong>getitem</strong> to your own order</p>

username: barakb
cooked: <p>It was ok if in each epoch we had the same mapping, but I’m planning to give different order in each epoch, just like shuffling the data.</p>
Calculating mean and std for custom dataset with `RandomWeightedSampler`

username: SurajSubramanian
cooked: <p>I am using the following code to calculate the mean and std for my custom dataset. <a href="https://discuss.pytorch.org/t/about-normalization-using-pre-trained-vgg16-networks/23560/6" class="inline-onebox">About Normalization using pre-trained vgg16 networks</a> (thanks to ptrblck)<br>
However my dataset is unbalanced and hence I have used <code>RandomWeightedSampler</code> to create a dataloader.<br>
Should I calcuate mean and std using this dataloader or calculate it using a dataloader that does not use any sampler. (Calculating mean and std using the sampler dataloader gives varying results each time)<br>
Thanks in advance !</p>
Question about data range when calculating loss function

username: moravel
cooked: <p>Hi! My network use image data scaled from (0, 255) to (0, 1) as input and the output data is in (0, 1) too. <code>torch.optim.Adam</code> is used as optimizer and I calculate loss using output and ground truth in (0, 1).</p>
<p>Should I scale the data back to (0, 255) when calculating loss function? Is it useful in training process?</p>
Accessing the LSTM properties

username: peps0791
cooked: <p>Hi,</p>
<p>I have a seq2seq encoder model (Custom class), of which I want to access the RNN properties such as:</p>
<ol>
<li>hidden state size</li>
<li>number of layers</li>
<li>number of directions.</li>
</ol>
<p>I am able to get the LSTM object (‘torch.nn.modules.rnn.LSTM’) by invoking the <strong>._modules</strong> property on the encoder model. On printing the LSTM object, I see this output:</p>
<pre><code>LSTM(500, 250, num_layers=2, dropout=0.3, bidirectional=True)
</code></pre>
<p>How do I access these items from the LSTM object?</p>

username: ptrblck
cooked: <p>You can directly access these properties and print them:</p>
<pre><code class="lang-python">model = nn.LSTM(10, 10, 2, 0.3, bidirectional=True)

print(model.hidden_size)
print(model.num_layers)
print(model.bidirectional)
</code></pre>
Parallelising models with variable size inputs across multiple GPUs

username: CompRhys
cooked: <p>I have written a module for small graphs that has a custom collate_batch function so that I have batches of 128 graphs at a time however the graphs are different sizes and so we have a variable number of nodes. I am using torch_scatter for some operations.</p>
<p>My aim is to parallelise my code over multiple GPUs to reduce the real time for training. Naively using <code>nn.DataParallel(model)</code> gives me an error as I believe it requires the sub-tensors it splits up the tensor into to all be the same size which is not the case here as while each GPU gets 64 graphs the number of nodes varies. Is there a way to do this currently across multiple GPUs or do I need to just take the hit and train on one GPU?</p>

username: mrshenli
cooked: <aside class="quote no-group" data-username="CompRhys" data-post="1" data-topic="73977">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/comprhys/48/9863_2.png" class="avatar"> CompRhys:</div>
<blockquote>
<p>Naively using <code>nn.DataParallel(model)</code> gives me an error as I believe it requires the sub-tensors it splits up the tensor into to all be the same size which is not the case here as while each GPU gets 64 graphs the number of nodes varies.</p>
</blockquote>
</aside>
<p>I am inclined to think this shouldn’t cause a problem for <code>DataParallel</code>. Because <code>DataParallel</code> would replicate your model and scatter your input on the first (batch) dimension. Even though different input element (graph) can contain different node, as long as your model forward function can handle it properly, it should not hit error.</p>
<p>Could you please share a min repro for the error you saw?</p>
How the handles are passed around when autograd

username: Arrogan
cooked: <p>This is a beginner question. I come from C++ and a python beginner. I’m confused by how the handles are passed around.<br>
Here is the sample code<br>
import torch<br>
import torch.nn.functional as F</p>
<p>x_data = torch.Tensor(torch.Tensor([[1.0], [2.0], [3.0], [4.0]]))<br>
y_data = torch.Tensor(torch.Tensor([[0.], [0.], [1.], [1.0]]))</p>
<p>class Model(torch.nn.Module):<br>
def <strong>init</strong>(self):<br>
super(Model, self).<strong>init</strong>()<br>
self.linear = torch.nn.Linear(1, 1)</p>
<pre><code>def forward(self, x):
    y_pred = F.sigmoid(self.linear(x))
    return y_pred
</code></pre>
<p>model = Model()</p>
<p>criterion = torch.nn.BCELoss(reduction=‘mean’)<br>
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)</p>
<p>for epoch in range(1000):<br>
y_pred = model(x_data)<br>
loss = criterion(y_pred, y_data)<br>
print(epoch, loss.data, type(loss.data))</p>
<pre><code>optimizer.zero_grad()
loss.backward()
optimizer.step()
</code></pre>
<p>hour_var = torch.Tensor([[1.0]])<br>
print(hour_var)</p>
<p>My question is that, when loss.backward() is called, how does python know it needs to look into model? criterion(y_pred, y_data) receives two tensors and call BCELoss.<strong>forward</strong>(). But how loss knows the calculation inside Model (e.g., Model.forward())? Is this related to the computational graph? I learned the basics, but I don’t know how the computational graph is implemented in this sample code. For example, does the tensor y_pred preserve the information about how y_pred is calculated?</p>
<p>Thanks in advance!</p>

username: ptrblck
cooked: <p>The computation graph is created during the forward pass dynamically and you could traverse it back using the <code>.grad_fn</code> of the output.<br>
Based on this graph, the <code>backward()</code> call can backpropagate through all operations and calculate the gradients for all parameters.</p>
How to apply gradient to tuneable gamma parameter in loss function

username: geschaftsreise
cooked: <p>Take the following pseudocode, attempting to define a global tunable parameter ‘gamma’ which is incorporated in the loss function. Rough solution is edited below.</p>
<pre><code class="lang-auto">class someModel(nn.Module):
    def __init__(self, process):
        super(someModel, self).__init__()
        self.process = SomeNnFunction()
        self.gamma = nn.Parameter(torch.ones(1), requires_grad=True)

    def forward(self, x):
        xhat = self.process(x)
        gamma = self.gamma
    
        return xhat, gamma

set up 'loader'
for i, (x, _) in enumerate(loader):
        model.zero_grad()
        x = x.to(device)
        xhat, gamma = model(x)
        gamma=gamma.mean()
        loss = loss_function(xhat, x, gamma)
        loss.backward()
        optimizer.step()
</code></pre>

username: Tony-Y
cooked: <aside class="quote quote-modified" data-post="12" data-topic="39681">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tony-y/48/8373_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-to-learn-the-weights-between-two-losses/39681/12">How to learn the weights between two losses?</a> 
  </div>
  <blockquote>
    In order to avoid numerical instability, we should use a variable change: 
eta = log(sigma)

The new variable eta can be defined within (-oo, +oo). 
 <a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/9/9445b48d1c15d05bae1b1da799f5b5c14ddda9b9.png" data-download-href="https://discuss.pytorch.org/uploads/default/9445b48d1c15d05bae1b1da799f5b5c14ddda9b9" title="multitask_loss.png">[multitask_loss]</a> 
Sample code: 
import torch
import torch.nn as nn
import torch.optim as optim

class MultiTaskLoss(nn.Module):
    def __init__(self, model, loss_fn, eta):
        super(MultiTaskLoss, self).__init__()
        self.model = model
        self.loss_fn = loss_fn
        self.eta = nn.Parameter(torch.Tensor(eta))

    def forward(self,…
  </blockquote>
</aside>

<p>You can find an PyTorch implementation for such a loss in this thread.</p>

username: geschaftsreise
cooked: <p>Nevermind, I solved this. Added the rough solution to the question for posterity.</p>
What's the behavior of BCEWithLogitsLoss for multi-label data

username: Wenxuan_Guo
cooked: <p>Hi,</p>
<p>I have a noob question about the behavior of BCEWithLogitsLoss. I tested its loss with some data <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/e/5e808ca96673ac1ea3f89db9b3d55f4cb1cb0627.png" data-download-href="https://discuss.pytorch.org/uploads/default/5e808ca96673ac1ea3f89db9b3d55f4cb1cb0627" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/e/5e808ca96673ac1ea3f89db9b3d55f4cb1cb0627_2_605x500.png" alt="image" data-base62-sha1="du0iuabVQcMW6vTmwdxJOWlHeSz" width="605" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/e/5e808ca96673ac1ea3f89db9b3d55f4cb1cb0627_2_605x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/e/5e808ca96673ac1ea3f89db9b3d55f4cb1cb0627_2_907x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/e/5e808ca96673ac1ea3f89db9b3d55f4cb1cb0627_2_1210x1000.png 2x" data-dominant-color="A7A7A8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1696×1400 253 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div>.</p>
<p>My understanding is that the loss between 0.999(x) and 1(y) should be 0.3135 and loss between 0.999(x) and 0(y) 1.3125. Thats the case when target and output are of size(1, <span class="hashtag-raw">#sample</span>), which is binary classification. However, when I reshape the data to size(<span class="hashtag-raw">#size</span>, <span class="hashtag-raw">#class</span>) to simulate multi-label, the loss is different (0.3143, 0.6931).</p>
<p>So how will BCEWithLogitsLoss calculate loss for multilabel. My understanding for the doc is they should be the same. Thank you!</p>

username: ptrblck
cooked: <p>You are passing the <code>output</code> and <code>target</code> in the wrong order in your second example.<br>
Pass the <code>output</code> first, then the <code>target</code>, and you’ll get the same result. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: Wenxuan_Guo
cooked: <p>Thank you! I feel bad for myself lol</p>
Loss.backward() in pytorch hooks

username: pytorchnewbie
cooked: <p>How come Kaggle says here (in the image attached below) that “loss.backward(retain_graph=True)  # doesn’t work with backward hooks”? I didn’t quite understand their explanation, especially since lower down they do call loss.backward() (not pictured but is in the article) and it seems to work. I got this from the kaggle article : <a href="https://www.kaggle.com/sironghuang/understanding-pytorch-hooks/notebook" class="inline-onebox" rel="noopener nofollow ugc">Understanding Pytorch hooks | Kaggle</a></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/a/2a891d9617cbf64905c24326266c97464c277c2c.png" data-download-href="https://discuss.pytorch.org/uploads/default/2a891d9617cbf64905c24326266c97464c277c2c" title="Screen Shot 2020-03-22 at 8.53.31 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/a/2a891d9617cbf64905c24326266c97464c277c2c_2_556x500.png" alt="Screen Shot 2020-03-22 at 8.53.31 PM" data-base62-sha1="64hOZKzEtth16Dbd9TyQqwhrPQg" width="556" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/a/2a891d9617cbf64905c24326266c97464c277c2c_2_556x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/a/2a891d9617cbf64905c24326266c97464c277c2c_2_834x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/2/a/2a891d9617cbf64905c24326266c97464c277c2c_2_1112x1000.png 2x" data-dominant-color="DCDCDC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2020-03-22 at 8.53.31 PM</span><span class="informations">1644×1478 288 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Thanks so much!</p>
How does loss over sigmoid works in skip gram model?

username: shashankMadan
cooked: <p>Here is the code I have been referring to<br>
<a href="https://github.com/fanglanting/skip-gram-pytorch/blob/master/model.py" rel="nofollow noopener">skip gram source code</a></p>
<pre><code class="lang-auto">class skipgram(nn.Module):
  def __init__(self, vocab_size, embedding_dim):
    super(skipgram, self).__init__()
    self.u_embeddings = nn.Embedding(vocab_size, embedding_dim, sparse=True)   
    self.v_embeddings = nn.Embedding(vocab_size, embedding_dim, sparse=True) 
    self.embedding_dim = embedding_dim
    self.init_emb()
  def init_emb(self):
    initrange = 0.5 / self.embedding_dim
    self.u_embeddings.weight.data.uniform_(-initrange, initrange)
    self.v_embeddings.weight.data.uniform_(-0, 0)
  def forward(self, u_pos, v_pos, v_neg, batch_size):

    embed_u = self.u_embeddings(u_pos)
    embed_v = self.v_embeddings(v_pos)

    score  = torch.mul(embed_u, embed_v)
    score = torch.sum(score, dim=1)
    log_target = F.logsigmoid(score).squeeze()
    
    neg_embed_v = self.v_embeddings(v_neg)
    
    neg_score = torch.bmm(neg_embed_v, embed_u.unsqueeze(2)).squeeze()
    neg_score = torch.sum(neg_score, dim=1)
    sum_log_sampled = F.logsigmoid(-1*neg_score).squeeze()

    loss = log_target + sum_log_sampled

    return -1*loss.sum()/batch_size
</code></pre>
<p>As we can see the author of the code has directly taken the sigmoid of the scores along with negative sigmoid adds them takes the mean and sends if off…<br>
I could have understood if we would have a cross-entropy or NLLloss over here but there is nothing as such…</p>
<p>does anyone have an intuitive explanation on why this works (because it does)<br>
thank you</p>
Hessian computation in GANs training

username: vittorio
cooked: <p>Hi guys i hope someone can help me on this one,<br>
So i am trying to implement an algorithm for GANs training, i have defined my generator and discriminator lets say:<br>
G = mygenerator()<br>
D = mydiscriminator()<br>
loss = myloss()<br>
Now going through one step of what my training routine should be, once i have computed the loss (which is function of both G and D) i need to compute the gradient of this w.r.t the parameters of both networks. i will refer with X to the parameters of G and with Y to the ones of D. I do this as:</p>
<pre><code class="lang-auto">grad_x = autograd.grad(loss, G.parameters(), create_graph=True, retain_graph=True, allow_unused= True)
grad_x_vec = torch.cat([g.contiguous().view(-1) for g in grad_x])
</code></pre>
<p>Similar code for grad_y, now in order to update the weights(i’ll refer only to the ones of X here) i need to compute the product Dxy * grad_y, where with Dxy i mean the derivative of the loss first w.r.t X and then w.r.t Y.<br>
I think it’s possible to compute this product in one step by the correct use of the grad_outputs parameter in autograd.grad.<br>
Here’s what is confusing me:<br>
if i call:</p>
<pre><code class="lang-auto">autograd.grad(grad_y_vec, G.parameters(), grad_outputs = grad_y_vec)
</code></pre>
<p>In my mind i would obtain Dyx * grad_y_vec, but by simulating this same problem in a small dimensional environment where i can actually see whats inside the tensors i’ve realized that the output of the code above is Dxy*grad_y (which is what i actually need), but doesn’t make much sense to me since i feel like i am differentiating first w.r.t Y and then wrt X, does someone understands what i am getting wrong?</p>
<p>Thanks for your time</p>
Siamese Net questions

username: joekid
cooked: <p>Excuse me, maybe an stupid question. I am new in DL, because I have a few training data for a classification project I proposed to use Siamese network (It allows few-shot learning). After watching deeplearning.ai videos I got confused:</p>
<p>Are these correct?</p>
<p>Siamese is trained in two steps:</p>
<ol>
<li>First, training step, using the triplet loss to learn the model parameters (<a href="https://www.youtube.com/watch?v=g3UGGAEbw_Q" rel="nofollow noopener">https://www.youtube.com/watch?v=g3UGGAEbw_Q</a>)</li>
<li>Next, recognition step, add a Sigmoid activation to get a binary classifier (<a href="https://www.youtube.com/watch?v=0NSLgoEtdnw" rel="nofollow noopener">https://www.youtube.com/watch?v=0NSLgoEtdnw</a>). In this case, I must  retrain the net, in this case using another loss, ex. Hinge Loss?</li>
</ol>
Grad_func implementations of primitive tensor operations

username: nirandaperera
cooked: <p>Hi,</p>
<p>I was wondering if I could see the grad_func implementations of primitive tensor operations in the PT codebase?<br>
Ex: for x.addmm(y), the grad_func is AddmmBackward but I wasn’t able to find this anywhere in the codebase (in both python/cpp)?</p>
<p>Best<br>
Niranda</p>
Truncated BPTT for Language Modelling

username: marco_zaror
cooked: <p>Hi,<br>
I’ve working with N-gram Language Modelling, taking n words and predicting the word n+1. My code is working but I’ve got a doubt in particular. Specifically, for each word I’m obtaining the output and the hidden state, and then passing those again for n steps. Finally, I compute the loss and update the gradients. My question is: Because I’m renaming the hidden state with every word. I’m doing truncated bptt? (Because I’m only gonna have the last hidden vector available when do the backprop?)</p>
<pre><code class="lang-auto">def train(text_x_tensor1,label1):#, text_x_tensor2, label2):
    text_x_tensor1, label1 = text_x_tensor1.to(device), label1.to(device)
    rnn1.train()
    hidden_1 = rnn1.initHidden()
    hidden_1 = hidden_1.to(device)
    text_x_tensor1 = text_x_tensor1.permute(1,0,2)
    for i in range(len(text_x_tensor1)): #For each word
        output_1, hidden_1 = rnn1(text_x_tensor1[i], hidden_1) 
    loss1 = criterion(output_1,label1)
    optimizer1.zero_grad()
    loss1.backward()#retain_graph=True) #Keep the intermediate buffers
    torch.nn.utils.clip_grad_norm_(rnn1.parameters(),1)
    optimizer1.step()
    return output_1, loss1,hidden_1
</code></pre>
<p>Thanks in advance!</p>
Interpreting Loss value with my sensitivty and specificty

username: ykukkim
cooked: <p>Hi all,</p>
<p>I am currently developing a LSTM model for binary classification, and I am at a point where I am not too sure what to expect from my loss value.</p>
<p>Here is my <code>loss</code> from training and validation: I run 1 epoch of validation before training loop to see how the model performs and as wanted validation has better performance as I have <code>drop out = 0.1 </code></p>
<pre><code class="lang-auto">Train Epoch: 47	Loss: 0.418231	TPR: 90.32258064516128	TNR: 93.67773677736777	Time: 0.015681
Val_loss: 0.402811	TPR: 90.625	TNR: 94.61122047244095	Time: 0.433588

Val_loss: 0.319000	TPR: 93.54838709677419	TNR: 94.24354243542436	Time: 0.443668

Train Epoch: 48	Loss: 0.444865	TPR: 90.625	TNR: 92.96259842519686	Time: 0.015649
Val_loss: 0.326329	TPR: 93.54838709677419	TNR: 94.31734317343174	Time: 0.443036

Val_loss: 0.331961	TPR: 90.32258064516128	TNR: 94.36654366543665	Time: 0.439770

Train Epoch: 49	Loss: 0.401345	TPR: 90.32258064516128	TNR: 93.82533825338253	Time: 0.016455
Val_loss: 0.369906	TPR: 90.625	TNR: 94.04527559055119	Time: 0.418198
</code></pre>
<p>The thin I am not too sure about is the loss value, I do not know whether this value is large or small but I can see that my TPR and TNR is high which is a good thing.</p>
<pre><code class="lang-auto">                pred = predictions.detach()
                pred[pred &gt;0.5] = 1
                pred[pred&lt;=0.5] = 0

                correct_indx_positive = pred[target == 1]                       # Should have batch_size * 1's
                correct_indx_negative = pred[target == 0]                       # Should have (batch_size*seq_length-batch_size) * 0's
  
                TPR = len(correct_indx_positive[correct_indx_positive == 1])/len(correct_indx_positive) 
                TNR = len(correct_indx_negative[correct_indx_negative == 0])/len(correct_indx_negative)
</code></pre>
<p>What I want to know is, what I would have to do in order to decrease loss value, is it just matter of trial and error with hyperparameters?</p>
<p>Any comments will be really helpfeul!</p>
Learning rate decay during training

username: Imran_Rashid
cooked: <p>I am trying to implement a particular learning rate decay on the Adam optimizer with each training step (<code>global step</code>) according to the function below:</p>
<pre><code class="lang-auto">from math import pow


init_learning_rate = 1e-3
min_learning_rate = 1e-5
decay_rate = 0.9999

lr = ((init_learning_rate - min_learning_rate) *
      pow(decay_rate, global_step) +
      min_learning_rate)

optimizer = torch.optim.Adam(model.parameters(), lr=lr)
</code></pre>
<p>Would I have to redefine the optimizer at every time step?</p>

username: Tony-Y
cooked: <aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://github.com/cmpark0126/pytorch-polynomial-lr-decay" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars3.githubusercontent.com/u/17241893?s=400&amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/cmpark0126/pytorch-polynomial-lr-decay" target="_blank" rel="nofollow noopener">cmpark0126/pytorch-polynomial-lr-decay</a></h3>

<p>Polynomial Learning Rate Decay Scheduler for PyTorch - cmpark0126/pytorch-polynomial-lr-decay</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


username: Tony-Y
cooked: <pre><code class="lang-auto">import torch
import torchvision
from math import pow

model = torchvision.models.AlexNet()

def lr_decay(global_step,
    init_learning_rate = 1e-3,
    min_learning_rate = 1e-5,
    decay_rate = 0.9999):
    lr = ((init_learning_rate - min_learning_rate) *
          pow(decay_rate, global_step) +
          min_learning_rate)
    return lr

lr0 = lr_decay(0)
optimizer = torch.optim.Adam(model.parameters(), lr=lr0)
scheduler = torch.optim.lr_scheduler.LambdaLR(optimizer,
    lr_lambda=lambda step: lr_decay(step)/lr0)
for step in range(10):
    print(step, optimizer.param_groups[0]['lr'], lr_decay(step))
    assert optimizer.param_groups[0]['lr'] == lr_decay(step)
    # loss.backward()
    optimizer.step()
    scheduler.step()
</code></pre>
Spconv with torchscript

username: Jakov_Gaspar
cooked: <p>Hi,</p>
<p>I’m having trouble using torchscript with my model which uses spconv. I need spconv, since I’m working with large sparse matrices. However,</p>
<pre><code class="lang-auto">traced_script_module= torch.jit.trace(model, example)
</code></pre>
<p>throws the following error:</p>
<pre><code class="lang-auto">RuntimeError: Found an unsupported argument type c10::List&lt;at::Tensor&gt; in the JIT tracer.
</code></pre>
<p>I’m also getting a lot of transformation errors like:</p>
<pre><code class="lang-auto">TracerWarning: Converting a tensor to a NumPy array might cause the trace to be incorrect.
</code></pre>
<p>Any Idea what could cause this?</p>
<p>Thanks.</p>
Getting NaN when using either BatchNorm or Dropout or both

username: divyesh_rajpura
cooked: <p>I am trying to implement TDNN based speaker verification system (<a href="https://www.danielpovey.com/files/2018_icassp_xvectors.pdf" rel="nofollow noopener">X-VECTORS: ROBUST DNN EMBEDDINGS FOR SPEAKER RECOGNITION</a>)</p>
<p>I have found some implementation of TDNN from github <a href="https://github.com/jonasvdd/TDNN" rel="nofollow noopener">jonasvdd</a>, <a href="https://github.com/cvqluu/TDNN" rel="nofollow noopener">cvqluu</a>, <a href="https://github.com/SiddGururani/Pytorch-TDNN" rel="nofollow noopener">SiddGururani</a>. I am using implementation given by <a href="https://github.com/jonasvdd/TDNN" rel="nofollow noopener">jonasvdd</a>. I verified that this implementation is perfectly matching the purpose of TDNN. The TDNN layer contains series of layers, namely, Conv1D, ReLU, Dropout, and BatchNorm.<br>
<strong>TDNN ( Conv1D --&gt; ReLU --&gt; Dropout --&gt; BatchNorm )</strong></p>
<p>Complete model is as follows<br>
<strong>Model ( TDNN1 --&gt; TDNN2 --&gt; TDNN3 --&gt; TDNN4 --&gt; TDNN5 --&gt; StatsPool --&gt; Linear --&gt; ReLU --&gt; Linear )</strong></p>
<p>Here StatsPool layers calculates the mean and standard deviation of output of TDNN5 layer.</p>
<p>My input to the network is (64, 480, 60), where 64 is batch size, 480 is sequence length, and 60 is feature size. As this is multi-class classification task, I am using CrossEntropy Loss.</p>
<p>The problem is when I use either BatchNorm or Dropout or both in TDNN, it gives me NaN after some iteration (after 8 to 10 batches only). But, when I remove both the layers from the network it works perfectly fine.</p>
<p>As suggested in other queries on this forums, I checked that my input does not contain NaN. I also tried by applying gradient clipping, that also doesn’t work.</p>
<p>I have seen some queries regarding effect of changing sequence of layers. Is there any improper sequence of layers in my model?<br>
Please suggest solution.</p>
<p>Thank You.</p>

username: divyesh_rajpura
cooked: <p>Can someone please help me with this??</p>

username: divyesh_rajpura
cooked: <p>I think applying Dropout before BatchNorm in TDNN was the problem. I interchange sequence of that two layers and now model is working perfectly. Can anyone please state the for that ?</p>
Permuted MNIST class. AttributeError: can't set attribute

username: Haneensu
cooked: <p>Running this code for getting permuted MNIST raises an error on the bolded if statement:<br>
AttributeError: can’t set attribute<br>
Can anyone help me with understanding the reason behind error and how to fix it?</p>
<pre><code class="lang-auto">import random
import torch
from torchvision import datasets


class PermutedMNIST(datasets.MNIST):

    def __init__(self, root="~/.torch/data/mnist", train=True, permute_idx=None):
        super(PermutedMNIST, self).__init__(root, train, download=True)
        assert len(permute_idx) == 28 * 28
        **if self.train:**
**            self.train_data = torch.stack([img.float().view(-1)[permute_idx] / 255 for img in self.train_data()])**
        else:
            self.test_data = torch.stack([img.float().view(-1)[permute_idx] / 255 for img in self.test_data()])

    def __getitem__(self, index):

        if self.train:
            img, target = self.train_data[index], self.train_labels[index]
        else:
            img, target = self.test_data[index], self.test_labels[index]

        return img, target

    def get_sample(self, sample_size):
        sample_idx = random.sample(range(len(self)), sample_size)
        return [img for img in self.train_data[sample_idx]]
</code></pre>

username: Tony-Y
cooked: <pre><code class="lang-auto">    @property
    def train_data(self):
        warnings.warn("train_data has been renamed data")
        return self.data
</code></pre>
<p>self.train_data is a read-only property. Try another name such as <code>self.my_data</code>.</p>