akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Let custom autograd Function know if create_graph is enabled

username: vmoens
cooked: <p>Hi<br>
Assume you have a custom autograd.Function class that during backward calls autograd.grad.<br>
How can this call to backward know if in the ‘main code’ the option create_graph was set on?<br>
One option could be to always set on create_graph in the backward function but that has obviously unwanted consequences, especially in term of memory usage.<br>
Here is the solution i came up with, not super elegant i must admit.</p>
<pre><code class="lang-auto">import torch
x = torch.tensor([1,2.0,3.0], requires_grad=True)

CREATE_GRAPH=False
class set_create_graph:
    def __init__(self, value):
        self.value = value
    def __enter__(self, *args):
        global CREATE_GRAPH
        self.prev = CREATE_GRAPH
        CREATE_GRAPH=self.value
    def __exit__(self, *args):
        global CREATE_GRAPH
        CREATE_GRAPH=self.prev
        
class f(torch.autograd.Function):
    @staticmethod
    def forward(ctx, x):
        with torch.set_grad_enabled(True):
            y = x.tanh()
            ctx.save_for_backward(x,y)
        return y.detach().requires_grad_(y.requires_grad)
    
    @staticmethod
    def backward(ctx, gd):
        with torch.set_grad_enabled(True):
            x, y = ctx.saved_tensors
            return torch.autograd.grad(y, x, gd, True, CREATE_GRAPH)
s = f.apply(x)
with set_create_graph(True):
    g = torch.autograd.grad(s, x, torch.ones_like(s, requires_grad=True), True, True)
torch.autograd.grad(g, x, torch.ones(3), True)

</code></pre>
<p>Would be nice to have a way to replace this global variable CREATE_GRAPH by something that would not require me to put things in a specific context.<br>
Thanks for the help!</p>

username: albanD
cooked: <p>Hi,</p>
<p>You can check if <code>torch.is_grad_enabled()</code> in the backward and if <code>gd.requires_grad</code>.<br>
That will tell if you something wants the gradients to be computed for your function. Namely if grad mode is enabled and the input requires_grad then you should create the graph. Otherwise, it is not needed.</p>
Set_index() for tensor.device()

username: Stonepia
cooked: <p>I want to write a code to explicitly delegate index to all tensors and the result is that every tensor should have a default device().index() 0.</p>
<p>So I wrote the code as follows:</p>
<pre><code class="lang-auto">  if (! rawTensor.device().has_index()){
    rawTensor.device().set_index(0);
  }
</code></pre>
<p>However, this seems not work. Does this mean that I must explicitly call tensor.to( ‘cpu:0’) ? If so, what is the point of offering a set_index() function?</p>

username: albanD
cooked: <blockquote>
<p>If so, what is the point of offering a set_index() function?</p>
</blockquote>
<p>I am not a specialist of that part of the code but I would expect this function to simply update the device object that was returned (so that you can pass it to a further constructor or .to function). I don’t expect it to modify the Tensor itself.<br>
Also note that all these functions that move Tensors across devices are applied out of place so you need to get the new object they return to get the new Tensor.</p>
Pollen prediction with Pytorch

username: d10genes
cooked: <p>Greetings! I just started using pytorch and so far I’ve been really impressed with the software and grateful for how over-the-top helpful the community has been.</p>
<p>I wrote up a pollen forecaster with RNNs using pytorch <a href="http://d10genes.github.io/blog/2017/04/05/pollen/" rel="nofollow noopener">here</a>. There’s not too much that’s novel given all the code examples and documentation, but I thought perhaps someone could find it useful (assuming this is the correct forum for such things)</p>
<p>Thanks!</p>

username: Li_Ziming
cooked: <p>The link URL had been 404?</p>
RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 0. pls help

username: pparitoshh
cooked: <p>Hi All,<br>
I am getting an error -<br>
File “c:\Users\Paritosh\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py”, line 232, in <br>
return [default_collate(samples) for samples in transposed]<br>
File “c:\Users\Paritosh\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py”, line 218, in default_collate<br>
return torch.stack([torch.from_numpy(b) for b in batch], 0)<br>
RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 0. Got 5 and 8 in dimension 1 at c:\a\w\1\s\tmp_conda_3.6_122405\conda\conda-bld\pytorch_1549283197917\work\aten\src\th\generic/THTensorMoreMath.cpp:1307</p>
<p>While I am using batchsize=1 then I am not getting any error.</p>
<p>Could anybody help me with the issue?</p>

username: chenyuntc
cooked: <p>You should check the size of all data returned in your <code>Dataset</code>, and you will find the one with different shapes (5 or 8).</p>

username: pparitoshh
cooked: <p>It was tensor shape issues. Thanks Yun Chen:grinning:</p>

username: wangyilan
cooked: <p>Could I ask the stupid question, what’s the meaning of “different shapes (5 or 8).” ? For e.g., My input shape is (3,512,512)(chw) and the outpust shape is (80, 128,128), if some data get shape 5 or 8, which dimension of the data is 5 or 8 <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>

username: codearts
cooked: <p>I am having the same error which also doesn’t occur when I use a batch size of 1.</p>
<p>Looped through the dataset and can confirm all images are 3x224x224 and all annotations are 1x224x224.</p>
<p><code>RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 0. Got 224 and 244 in dimension 2 at /tmp/pip-req-build-9oilk29k/aten/src/TH/generic/THTensor.cpp:689</code></p>
<p>How is that happening? It even says 224 and 224 in dimension 2 which doesn’t make sense. Shouldn’t this exception occur when they aren’t matching?</p>

username: ptrblck
cooked: <p>The error states that the shapes are <code>224</code> vs. <code>244</code> (note the changed 2 and 4), so I guess you might have a similar typo somewhere in your code? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: codearts
cooked: <p>Thanks a lot! I already solved this.</p>
<p>I asked this as a separate question here and on SO. Then I noticed that it is not 224x224 but 224x244 and got mad. I am so blind. Anyways changing that one image manually in my case (or using a resize transformation) did the job.</p>
<p>PS: I have noticed how much you help on this forum even before signing up. You are the real MVP.</p>
IndexError: index_select(): Index is supposed to be a vector

username: raoashish10
cooked: <pre><code class="lang-auto">for batch_id, (data, target) in enumerate(tqdm(train_loader)):
        print(target)
        print('Entered for loop')
        
        target = torch.sparse.torch.eye(10).index_select(dim=0, index=target)
        data, target = Variable(data), Variable(target)
</code></pre>
<p>The line which contains the index_select function gives this error and I am not able to find a solution to it anywhere. The target variable on printing looks like this:</p>
<p>tensor([[4],<br>
[1],<br>
[8],<br>
[5],<br>
[9],<br>
[5],<br>
[5],<br>
[8],<br>
[4],<br>
[6]])</p>
<p>How do I convert the target variable into a vector? Isn’t it already a vector?</p>

username: Satish_Ad47
cooked: <p><a class="mention" href="/u/raoashish10">@raoashish10</a>   u  got  any solution?</p>

username: raoashish10
cooked: <p>Yes I did! I think the index attribute expects a tensor of all the classes for example if I have 10 classes, then it expects a tensor([0,1,2,3,4,5,6,7,8,9]). So I declared a tensor in this manner and it worked for me. Let me know if it works for you too.</p>

username: Satish_Ad47
cooked: <p>Thank you very much.</p>

username: raoashish10
cooked: <p>I want to update my answer because the previous solution was a little misleading. You can use the flatten() function. In my case, I used it as <code>  target = torch.sparse.torch.eye(10).index_select(dim=0, index=target.long().flatten())</code> The problem with the previous solution is that it erases the error but will decrease your accuracy considerably.</p>
Understanding transfer learning implementation

username: offset-null1
cooked: <p>Hey, I encountered <a href="https://github.com/pytorch/examples/blob/master/cpp/transfer-learning/main.cpp" rel="noopener nofollow ugc">this</a>. I want to implement transfer learning however, I’m not able to understand what the following code does.</p>
<pre><code class="lang-auto">Acc += acc.template item&lt;float&gt;();
mse += loss.template item&lt;float&gt;();
</code></pre>
<p>Can anyone please help me understand what’s <code>.template item&lt;float&gt;()</code>  doing here?</p>

username: ptrblck
cooked: <p>The <code>template</code> usage is explained in <a href="https://discuss.pytorch.org/t/understanding-target-eq-pred-sum-template-item-int64-t/83479/2">this post</a>. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
Number of threads when using DataParallel

username: 111413
cooked: <p>Hi. When I use nn.DataParallel for training my model, I have a problem with the number of threads.</p>
<p>Even though I set then number of threads for the process (by using torch.set_num_threads), it takes a lot of threads.<br>
I checked that the torch.set_num_threads works well for using single gpu.</p>
<p>Is there any setting that I missed? How can I deal with the problem?</p>
<p>Thank you</p>

username: ptrblck
cooked: <p>Are you sure these threads are used by <code>nn.DataParallel</code> or could e.g. the <code>DataLoader</code> spawn new workers in your setup?</p>
Unpickling torch tensor from Database

username: raceee
cooked: <p>I have used pickle to store my tensors in a database. I am trying to get them out and un-serialize them using pickle again.</p>
<pre><code class="lang-auto">pickle.loads(val[3])
</code></pre>
<p>Where <code>val</code> represents the pytorch tensor in serialized string form. I get this error:<br>
<code>TypeError: a bytes-like object is required, not 'str'</code><br>
Are any of you storing your tensors in a database? How do you  do this?</p>

username: ptrblck
cooked: <p>You could try to <code>decode</code> the string first via:</p>
<pre><code class="lang-python">unpickled = pickle.loads(codecs.decode(pickled, "base64"))
</code></pre>
<p>using the right encoding.</p>
MLPerf multi thread: Does pytorch support MLPerf multi threads in multi core cpu

username: yuanzhangzhu
cooked: <p>I recently  test MLPerf , and use pytorch as runtime.  I found there’s only 1 core could be used the mutlthread scenario.<br>
It’s said that py::gil_scoped_release release is needed to enable python multithread.<br>
Does anybody know how to enable multi thread in pytorch.</p>

username: ptrblck
cooked: <p>Could you explain what “MLPerf multi threads” are and where you are using multi-threading in your code?</p>
Transformer for audio classification unpack error

username: 111412
cooked: <p>Hi. I’m using transformer for audio classification. but there is problem with multi_head_attention_forward function.</p>
<p>/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in multi_head_attention_forward(query, key, value, embed_dim_to_check, num_heads, in_proj_weight, in_proj_bias, bias_k, bias_v, add_zero_attn, dropout_p, out_proj_weight, out_proj_bias, training, key_padding_mask, need_weights, attn_mask, use_separate_proj_weight, q_proj_weight, k_proj_weight, v_proj_weight, static_k, static_v)<br>
4132                 q_proj_weight=q_proj_weight, k_proj_weight=k_proj_weight,<br>
4133                 v_proj_weight=v_proj_weight, static_k=static_k, static_v=static_v)<br>
-&gt; 4134     tgt_len, bsz, embed_dim = query.size()<br>
4135     assert embed_dim == embed_dim_to_check<br>
4136     # allow MHA to have different sizes for the feature dimension</p>
<p>ValueError: too many values to unpack (expected 3)</p>
<p>please help.</p>

username: ptrblck
cooked: <p>The <code>nn.MultiHeadAttention</code> module returns <code>attn_outpu</code> and <code>attn_output_weights</code> as described in the <a href="https://pytorch.org/docs/stable/generated/torch.nn.MultiheadAttention.html?highlight=multiheadattention#torch.nn.MultiheadAttention">docs</a>. Based on the error message it seems you might be trying to unpack a wrong number of return values.</p>
Distributed Data Parallel

username: RundErr24
cooked: <p>Hello there, I am student and we could say beginner to the topic of machine learning.<br>
I am currently looking into problematic of parallel training on multiple GPUs. I understand DataParallel, but cant make Distributed Data Parallel works.<br>
The part I dont understand is communication through backend and connecting two nodes, for example do they need to be on same cluster? Or is static IP enough for master node. Can I somehow use public IP, and how?</p>
<p>My question really is, if you could provide me some good sources of knowledge, tutorials, etc.</p>
<p>I would be really grateful.<br>
Lukas</p>

username: rvarm1
cooked: <p>Hi,</p>
<p>To concretely answer your question around communication, basically nodes need some way to discover each other, whether that is through a shared-filesystem approach, or through a main IP address that every node can talk to.</p>
<p>Here are some helpful tutorials around PyTorch distributed and DDP:<br>
PyTorch DDP tutorial: <a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html" rel="noopener nofollow ugc">https://pytorch.org/tutorials/intermediate/ddp_tutorial.html</a><br>
PyTorch distributed overview: <a href="https://pytorch.org/tutorials/beginner/dist_overview.html" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/dist_overview.html</a></p>
The problem of DataParallel outputs

username: zyh3826
cooked: <p>When I use DataParalle, I find the first dim of outputs is batch_size * gpu_nums, and when I calculate the loss, the error coming</p>
<blockquote>
<p>ValueError:  Expected input batch_size (32) to match target batch_size (16).</p>
</blockquote>
<p>my code is:</p>
<blockquote>
<p>model = DataParallel(model, device_ids=gpus, output_device=gpus[0])<br>
model.to(config.cuda_id)<br>
outputs = model(input_ids, token_type_ids, attention_mask)<br>
loss = loss_fct(outputs, labels.cuda(config.cuda_id))</p>
</blockquote>
<p>I think the first dim of outputs should be batch_size<br>
How to fix this, anybody can help me? thx</p>

username: rvarm1
cooked: <p>Thanks! Checking some other sources: <a href="https://discuss.pytorch.org/t/valueerror-expected-input-batch-size-1-to-match-target-batch-size-64/43071/2" class="inline-onebox">ValueError: Expected input batch_size (1) to match target batch_size (64)</a> and <a href="https://stackoverflow.com/questions/56719867/pytorch-expected-input-batch-size-12-to-match-target-batch-size-64" rel="noopener nofollow ugc">https://stackoverflow.com/questions/56719867/pytorch-expected-input-batch-size-12-to-match-target-batch-size-64</a>, and <a href="https://discuss.pytorch.org/t/valueerror-expected-input-batch-size-324-to-match-target-batch-size-4/24498/3" class="inline-onebox">ValueError: Expected input batch_size (324) to match target batch_size (4)</a>, there is likely a bug in how you’ve defined the shapes in the implementation of your forward pass.</p>
<p>If your model works without DataParallel but breaks with it, it’s likely due to your model implicitly hardcoding a specific batch size it expects, likely in the beginning of the forward pass (maybe somwhere in <code>self.bert()</code>.</p>
Training additional layer without hurting original model performance

username: lazybut
cooked: <p>Hi, I trying to train auxiliary task using original model’s mid-block features without influence to original model’s performance. I use detach(), however original model’s performance is decreased.</p>
<p>there are two model</p>
<ol>
<li>the main model</li>
<li>the auxiliary model that takes main model’s mid block</li>
</ol>
<p>The main network have  Block 1 and fully-connected layer like below,<br>
input -&gt; B1 -&gt; FC1 -&gt; score<br>
then, auxiliary model have only fully-connected layer<br>
input(it will be B1) -&gt; FC2</p>
<p>I want to train moth model simultaneously, auxiliary model doesn’t affect to main model.</p>
<p>here’s my code</p>
<h1>two model</h1>
<p>MainModel / AuxModel</p>
<h1>two optimizer</h1>
<p>opM = optimizer(MainModel)<br>
opA = optimizer(AuxModel)</p>
<h1>training main model</h1>
<p>MainModel .zero_grad()<br>
<span class="hashtag">#when</span> I forward to main_model, I got score and midBlock<br>
mainLoss, B1 = MainModel(input)<br>
mainLoss.backward()<br>
opM.step()</p>
<h1>training aux model</h1>
<p>AuxModel.zero_grad()<br>
B1 = B1.detach()<br>
prediction= AuxModel(B1)<br>
auxLoss = criterion(prediction)<br>
auxLoss.backward()<br>
opA.step()</p>
<p>I try many things include each model’s param.requires_grad = False, however it still affect to original model’s performance.<br>
could you help me with any advices?</p>

username: simaiden
cooked: <p>If you are using MainModel only for get the mid block it’s not necessary to train it (if i understand the question), so you can do:</p>
<pre><code class="lang-auto">opA = optimizer(AuxModel)

with torch.no_grad():
   _, B1 = MainModel(input)
   
opA.zero_grad()
B1 = B1.detach()
prediction= AuxModel(B1)
auxLoss = criterion(prediction)
auxLoss.backward()
opA.step()
</code></pre>

username: lazybut
cooked: <p>Thank you for the reply!<br>
I need to train main model from scratch.<br>
my question is the different performance between</p>
<ol>
<li>training only main_model</li>
<li>training main_model and aux model at same time</li>
</ol>

username: stroncea
cooked: <p>You could initialize the second optimizer with only the FC2 parameters, i.e excluding main model parameters. This should mean that the 2nd optimizer won’t update the weights of B1.</p>
<p>You would still need to make sure that during training of auxiliary model, gradients are not stored in B1 during the backward pass, and I think if you turn off the gradients in one model you will turn them off in the second model as well if they share underlying memory.</p>
<p>Off the top of my head, a possible solution would be implementing some locking mechanism between the 2 models s.t when you train 1st model, requires_grad is True, and False for 2nd model. There might be easier solutions though.</p>
How to make a loss function for nn.Transformer

username: Amy_Lapi
cooked: <p>I’m working on building a transformer that translates polynomials into specific sequences of numbers. However, I’m running into a lot of trouble with the loss function.</p>
<p>I understand that when you use nnTransformer, the source input needs to be size (S,N,E) (where S is the length of the input) and the corresponding target needs to be (T,N,E) (where T is the target length).</p>
<p>The transformer returns an output of size (T,N,E).</p>
<p>I tried to implement nn.CrossEntropyLoss  to calculate the loss, but when I do I get this error:</p>
<pre><code class="lang-auto">IndexError: Target -1 is out of bounds.
</code></pre>
<p>I don’t understand what is causing this error. Could anyone help me understand what it means so I can fix the problem? The tensors that I am sending to nn.CrossEntropyLoss are sizes [25, 32, 512] and [25, 512].</p>
<p>What could be causing this error? Any help is appreciated! Thanks!</p>

username: Abhilash_Srivastava
cooked: <ol>
<li>Make sure that the target vocab is created correctly. The target labels for a batch are as expected. Also, if you’ve created your custom loss function that might have an issue.</li>
<li>Is possible share your code.</li>
</ol>
Making pynvml match torch device ids (CUDA_VISIBLE_DEVICES)

username: stas
cooked: <p>Yesterday, I discovered <code>pynvml</code> doesn’t respect <code>CUDA_VISIBLE_DEVICES</code>, so the ids of torch.cuda and nvml don’t match if you need to change which gpus are used via <code>CUDA_VISIBLE_DEVICES</code>, which breaks the software that uses <code>pynvml</code>.</p>
<p>I wrote a re-mapper to solve the problem:</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/stas00/ipyexperiments/blob/a60a23141e2ecb11c0554f8be535fe6ccb831604/ipyexperiments/utils/mem.py#L33-L44" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/stas00/ipyexperiments/blob/a60a23141e2ecb11c0554f8be535fe6ccb831604/ipyexperiments/utils/mem.py#L33-L44" target="_blank" rel="noopener nofollow ugc">stas00/ipyexperiments/blob/a60a23141e2ecb11c0554f8be535fe6ccb831604/ipyexperiments/utils/mem.py#L33-L44</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="33" style="counter-reset: li-counter 32 ;">
<li>def get_nvml_gpu_id(torch_gpu_id):</li>
<li>    """</li>
<li>    Remap torch device id to nvml device id, respecting CUDA_VISIBLE_DEVICES. </li>
<li>
</li><li>    If the latter isn't set return the same id</li>
<li>    """</li>
<li>    # if CUDA_VISIBLE_DEVICES is used automagically remap the id since pynvml ignores this env var</li>
<li>    if "CUDA_VISIBLE_DEVICES" in os.environ:</li>
<li>        ids = list(map(int, os.environ.get("CUDA_VISIBLE_DEVICES", "").split(",")))</li>
<li>        return ids[torch_gpu_id] # remap</li>
<li>    else:</li>
<li>        return torch_gpu_id</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>may be it might be of use to others.</p>
<p>Usage:</p>
<pre><code class="lang-auto">    [...]
    torch_gpu_id = torch.cuda.current_device()
    nvml_gpu_id = get_nvml_gpu_id(torch_gpu_id)
    handle = pynvml.nvmlDeviceGetHandleByIndex(nvml_gpu_id)
    [....]
</code></pre>
<p>I also suggested for pynvml to integrate it directly <a href="https://github.com/gpuopenanalytics/pynvml/issues/28" rel="noopener nofollow ugc">https://github.com/gpuopenanalytics/pynvml/issues/28</a></p>
A loss function to estimate the difference between two distributions?

username: be7f984b2f2e66ba7969
cooked: <p>Hi,</p>
<p>I know there is KL divergence loss in pytorch.</p>
<p>but the limitation with KL divergence is that sum of the distribution should be 1.</p>
<p>however, my target distribution is like 1d-Gaussian distributions where the sum of the distribution is not equal to1.</p>
<p>for example,</p>
<pre><code class="lang-auto">{0,0,0,0.01,0.02,...0.8,0.9,1.0,0.9,0.8........0.02,0.01,0,0,0,0}
</code></pre>
<p>I tried to model this as a multi-label sigmoid cross entropy. for example,</p>
<pre><code class="lang-auto">target * (pred) + 1 - target(1-pred)
</code></pre>
<p>however, I found out this is not a good loss to model this problem.</p>
<p>for example, say that one of the target probability is 0.6.  when pred is approaching to 0.6, the loss is not approaching to 0.</p>
<pre><code class="lang-auto">-(0.6  * (torch.log(0.6)) + 0.4 * (torch.log(0.4)))= 0.9163
</code></pre>
<p>I tried to model this problem as L1 loss, but the result is pretty bad.</p>
<p>any good advice for modeling this problem?</p>

username: Preetham_R_Patlolla
cooked: <p>May be you could use KL Div loss on the normalized data (instead of applying it on the original data). So that the sum is always 1.</p>
Pause and Resume .backward() pass

username: Kabir_Nagrecha
cooked: <p>How would we run a “partial” backwards on one subgraph, pause, then continue upward on a previous subgraph in the <em>same overall</em> backward pass?</p>
<p>My current idea, which works to some extent, is to freeze all but one subgraph, call backwards, then refreeze it, unfreeze the next section, and recall backwards. So essentially repeated backwards calls to compose a part-by-part overall backwards call.</p>
<p>However, this necessitates the use of retain_graph. Due to memory constraints, I’d prefer not to do this - instead I’d like to figure out a way for all of these to be part of the same backward pass. Is this possible? I thought perhaps it might require rewriting the execution engine, but is it possible with the higher-level APIs?</p>
Difference between functional.mse_loss and nn.MSELoss (stateless loss functions)

username: NightRain
cooked: <p>Is there any difference between calling</p>
<blockquote>
<p>functional.mse_loss(input, target)</p>
</blockquote>
<p>and</p>
<blockquote>
<p>nn.MSELoss(input, target)?</p>
</blockquote>
<p>Same question applies for l1_loss and any other <em>stateless</em> loss function.</p>

username: KFrank
cooked: <p>Hi Night!</p>
<aside class="quote no-group quote-modified" data-username="NightRain" data-post="1" data-topic="102865" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/n/df705f/48.png" class="avatar"> NightRain:</div>
<blockquote>
<p>Is there any difference …</p>
</blockquote>
</aside>
<p>Roughly speaking, there is no difference.</p>
<aside class="quote no-group">
<blockquote>
<blockquote>
<p>nn.MSELoss(input, target)?</p>
</blockquote>
</blockquote>
</aside>
<p>Note, as written, this won’t work.  This calls the <code>MSELoss</code> constructor<br>
with invalid arguments.  You need:</p>
<pre data-code-wrap="python"><code class="lang-python">loss_value = torch.nn.MSELoss() (input, target)
# or
loss_function = torch.nn.MSELoss()
loss_value = loss_function (input, target)
</code></pre>
<p>That is, you have to construct an <code>MSELoss</code> object first, and then call<br>
(apply) it.</p>
<aside class="quote no-group">
<blockquote>
<p>Same question applies for l1_loss and any other <em>stateless</em> loss function.</p>
</blockquote>
</aside>
<p><code>MSELoss</code> objects (and similar loss-function objects) are “stateless” in<br>
the sense that they don’t remember anything from one application<br>
(<code>loss_function (input, target)</code>) to the next.  But they do have<br>
state, in that they are constructed with certain “control” parameters<br>
set (for example, the <code>reduction</code> property of <code>MSELoss</code> and other<br>
loss-function objects) that are stored internally, and used repeatedly<br>
from one application to the next.  (These properties can be changed<br>
after construction, but I don’t think that that’s common practice.)</p>
<p>Best.</p>
<p>K. Frank</p>

username: NightRain
cooked: <p>Thanks a lot for your answer!</p>
Error using instance normalization

username: edshkim98
cooked: <p>Hello everyone, I am currently doing a project using instance normalization as my batch size is 1.<br>
However, whenever I start training, I get the following error below:</p>
<p>ValueError: InstanceNorm1d returns 0-filled tensor to 2D tensor.This is because InstanceNorm1d reshapes inputs to(1, N * C, …) from (N, C,…) and this makesvariances 0.</p>
<p>I know that this error occurs as my input shape after max pooling is (bs * C) while instancenorm1d only allows input shape (bs * C * N).<br>
Does anyone have any idea how to solve this issue?<br>
Or is there any way to train my model in batch size 1 with batch norm instead?<br>
Thank you.</p>
After 2nd iteration: always same result when training in a loop #8569

username: PhilipMay
cooked: <p>I train a BERT model on a binary classification task. I do the training 4 times in a row. With same train and validation data and with the exact same hyperparameters. I use no seeds.</p>
<p>The results of the 2nd, 3rd and 4th iteration are 100% the same. The result of the 1st run is unique. This behavior is 100% reproducible.</p>
<p>Due to the random initialization I would expect all runs to have different results. As with the 1st vs other runs. The resons why the 2nd and all following runs are equal must be a bug.</p>
<p>Did anybody alse see a problem like this? Am I doing anything wrong or is there in deed a major bug going on?</p>
<p>Full bug description here:<br>
</p><aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/huggingface/transformers/issues/8569" target="_blank" rel="noopener nofollow ugc">github.com/huggingface/transformers</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/huggingface/transformers/issues/8569" target="_blank" rel="noopener nofollow ugc">After 2nd iteration: always same result when training in a loop</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2020-11-16" data-time="15:51:11" data-timezone="UTC">03:51PM - 16 Nov 20 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/PhilipMay" target="_blank" rel="noopener nofollow ugc">
          <img alt="PhilipMay" src="https://avatars3.githubusercontent.com/u/229382?v=4" class="onebox-avatar-inline" width="20" height="20">
          PhilipMay
        </a>
      </div>
    </div>
  </div>
</div>

<div class="github-row">
  <p class="github-content">I train a BERT model on a binary classification task. I do the training 4 times in a row. With same...</p>
</div>

<div class="labels">
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
Linear Regression Model--Problems with Loss

username: rgodbey
cooked: <p>I am working on a linear model to make predictions. The problem I am having is that my loss is higher than my actual values to be predicted. To see if it is a problem with the data I have printed at several spots throughout trying to find if there are any disparities in the data but I find none. I have even tried leaking the expected values, making the inputs and labels the same. Loss is still extremely high. Hoping for some guidance. Below is the code for my model.</p>
<pre><code class="lang-auto">train_target = th.tensor(df["Label"].values.astype(np.float32))
train = th.tensor(df.drop("Label",axis=1).values.astype(np.float32))
train_data = TensorDataset(train, train_target)
train_dl = DataLoader(train_data, batch_size = batch_size, shuffle=True)

print(train_target.shape)
class LinReg(nn.Module):
    # Init layers
    def __init__(
        self,
        in_dim: int = 13,
        out_dim: int = 1,
        latent_base: int = 128,
        dropout: float = 0.5):

        super(LinReg, self).__init__()
        self.net = nn.Sequential(
            nn.Dropout(dropout),
            nn.Linear(in_dim, latent_base),
            nn.PReLU(),
            nn.Dropout(dropout),
            nn.Linear(latent_base, 1)
        )
    
    # Forward
    def forward(self, x: th.Tensor) -&gt; th.Tensor:
        return self.net(x)


model = LinReg()
model.cuda()

optimizer = th.optim.RMSprop(model.parameters(), lr = 1e-3)
criterion = nn.L1Loss()

for i in range(epochs):
    for inputs, labels in train_dl:
        y_pred = model(inputs.to(device))
        loss = criterion(y_pred, labels.to(device))
        loss.backward()
        optimizer.zero_grad()
        optimizer.step()
        if i == (epochs-1):
            pred =  y_pred
            y = labels
    if i%1==0:
        print('Epoch {}, Loss: {}'.format(i, loss.item()))
</code></pre>

username: googlebot
cooked: <p>I’m not sure that such use of dropout is correct. And use of L1Loss instead of MSELoss may aggravate the issue.</p>

username: pchandrasekaran
cooked: <pre><code class="lang-auto">optimizer.zero_grad()
loss.backward()
optimizer.step()
</code></pre>
<p>I believe this may fix your issue. You need to clear all the gradients before you perform a backward pass again. Also as Alex mentioned, using L1Loss() may exacerbate the issue in some cases.</p>

username: Preetham_R_Patlolla
cooked: <p>Try as suggested by <a class="mention" href="/u/pchandrasekaran">@pchandrasekaran</a></p>
<p>Also, how’s your model behaving when you try to run it on CPU? In my personal experience running some models on GPU resulted in exploding loss sometimes (the reason for which is still unclear to me). Try running it on CPU once.</p>
Torch::cuda::is_available return false in libtorch 1.5

username: Si_laopi
cooked: <p>After I update libtorch 1.5(C++, cuda10.1), <code>torch::cuda::is_available()</code> keeps returning false, and I couldn’t find the reason. Before that, I was using libtorch 1.3(C++, cuda10.1), and everything is OK.</p>

username: peterjc123
cooked: <p>Which OS are you using?</p>

username: Si_laopi
cooked: <p>Windows 10 1903, build 18362.778<br>
Nvidia GTX GeForce 1070 GPU</p>

username: peterjc123
cooked: <p>You’ll need to pass an additional argument to the linker.<br>
<code>-INCLUDE:?warp_size@cuda@at@@YAHXZ</code></p>

username: Si_laopi
cooked: <p>It worked. Thank you very much.</p>

username: powderluv
cooked: <p>I see the same issue with Linux. Is there an equivalent linker magic flag to pass ?</p>

username: OronG13
cooked: <p>Hello,<br>
I have the sample problem with Linux on my NVIDIA Jetson AGX Xavier, JetPack 4.4 and liborch 1.6.<br>
Did you solve this issue on your Linux?<br>
Did you find the “magic flag”?</p>
<p>Thanks</p>
How to make n random len masks on time axis?

username: hadaev8
cooked: <p>For example i have batch of [batch_size, lens, features] and [batch_size,] for x and x_lens.<br>
So masking should be like this:</p>
<pre><code>x = torch.rand(32, 1000, 512)
lens = (torch.rand(32,)*1000).long()
power=3
masks=10
x = x.clone()
batch_size, length, features = x.size()
T = lens // power // masks
mean = (x.detach().sum(1) / lens.unsqueeze(-1)).unsqueeze(1).expand(-1, length, -1)
mask_start = (torch.rand((batch_size, masks), device=lens.device)*(lens-T).unsqueeze(1)).long()
mask_end = mask_start + (torch.rand((batch_size, masks), device=lens.device)*T.unsqueeze(1)).long()
mask = torch.arange(0, length, device=lens.device).unsqueeze(0).expand(batch_size, -1)
# here im stuck
mask = (mask &gt;= mask_start) &amp; (mask &lt; mask_end)
x[mask] = mean[mask]
</code></pre>
<p>Im getting<br>
RuntimeError: The size of tensor a (1000) must match the size of tensor b (10) at non-singleton dimension 1</p>
<p>I cant figure out how to apply all these masks.</p>
Detach certain columns of a tensor

username: _Thorsten
cooked: <p>I’m trying to detach certain columns in a 2d tensor.</p>
<pre><code class="lang-auto">masked_encoding_prob = torch.randn(6272, 48, requires_grad=True)

num_group = 4
num_classes_per_group = 12

startcolumn = num_group * num_classes_per_group 
endcolumn = startcolumn + num_classes_per_group 

keep_grads = masked_encoding_prob[:, startcolumn:endcolumn]

masked_encoding_prob = masked_encoding_prob.detach()

print(masked_encoding_prob.requires_grad)  #False

masked_encoding_prob[:, startcolumn:endcolumn] = keep_grads

print(masked_encoding_prob.requires_grad)  #True
</code></pre>
<p>Is there a way to see which variables require gradients in a tensor individually? Or how would I go about verifying if this works?</p>

username: albanD
cooked: <p>Hi,</p>
<p>I am afraid Tensors are seen as “elementary” objects from the point of view of the autograd and you can only control if the whole Tensor requires grad or not.</p>
<p>Note though that if some values are “constant” in there, they will get a gradient of 0 though so that should still give you the result you’re looking for no?</p>
HELP! When using DDP，some troubles in params

username: saijunhu
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/f/6f0eb0c573ed89be37807d20a5cd0174f1b6bcce.png" data-download-href="https://discuss.pytorch.org/uploads/default/6f0eb0c573ed89be37807d20a5cd0174f1b6bcce" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/f/6f0eb0c573ed89be37807d20a5cd0174f1b6bcce_2_690x104.png" alt="image" data-base62-sha1="fQstm7XLDl0XsmDk6DXFiijvMNU" width="690" height="104" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/f/6f0eb0c573ed89be37807d20a5cd0174f1b6bcce_2_690x104.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/f/6f0eb0c573ed89be37807d20a5cd0174f1b6bcce_2_1035x156.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/6/f/6f0eb0c573ed89be37807d20a5cd0174f1b6bcce_2_1380x208.png 2x" data-dominant-color="383537"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1413×215 36.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>here is my mode, my problem is when i only load the pretrained params in rank=0 process, the other params such as ‘acc_max’ in checkpoint can’t be sysc cross the process,it seems only sysc the model weights cross ranks, so what should i do to make sure the all kinds of params saved in checkpoint will be loaded on all process.<br>
(ps: if i remove the ‘args.local_rank==0’ judge, i watch the gpustats and see two process running on the gpu0. That case is not my desired. )<br>
thanks bro!</p>

username: rvarm1
cooked: <p>Hi,</p>
<p>As I understand it you’d like to broadcast the value of <code>acc_max</code> to all ranks from rank 0?<br>
In that case, you can simply convert it to a tensor and call <code>dist.broadcast</code>:</p>
<pre><code class="lang-auto">acc_max = checkpoint['acc_max']
acc_tensor = torch.zeros(1) if args.local_rank != 0 else torch.tensor([acc_max])
torch.distributed.broadcast(acc_tensor)
acc_max_from_rank_0 = acc_tensor.item()
</code></pre>
<aside class="onebox allowlistedgeneric">
  <header class="source">
      <a href="https://pytorch.org/docs/master/distributed.html#torch.distributed.broadcast" target="_blank" rel="noopener nofollow ugc">pytorch.org</a>
  </header>
  <article class="onebox-body">
    <img src="" class="thumbnail" width="" height="">

<h3><a href="https://pytorch.org/docs/master/distributed.html#torch.distributed.broadcast" target="_blank" rel="noopener nofollow ugc">Distributed communication package - torch.distributed — PyTorch 1.7.0...</a></h3>



  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
AMP working with custom op

username: as754770178
cooked: <p>I want use amp in <code>https://github.com/open-mmlab/mmcv/blob/master/mmcv/ops/roi_align.py</code>.</p>
<p>I refer to <code>https://pytorch.org/docs/stable/notes/amp_examples.html#functions-with-multiple-inputs-or-autocastable-ops</code>, and Apply custom_fwd and custom_bwd (with no arguments) to forward and backward respectively. I modify code :</p>
<pre><code class="lang-auto">class RoIAlignFunction(Function):
    ... ...
    @staticmethod
    @custom_fwd
    def forward(ctx,
                input,
                rois,
                output_size,
                spatial_scale=1.0,
                sampling_ratio=0,
                pool_mode='avg',
                aligned=True):
        ctx.output_size = _pair(output_size)
        ctx.spatial_scale = spatial_scale
        ... ...
 
    @staticmethod
    @once_differentiable
    @custom_bwd
    def backward(ctx, grad_output):
        rois, argmax_y, argmax_x = ctx.saved_tensors
        ... ...
</code></pre>
<p>The code still report error:</p>
<pre><code class="lang-auto">  File "/usr/local/python16/lib/python3.7/site-packages/mmcv/ops/roi_align.py", line 71, in forward
    aligned=ctx.aligned)
RuntimeError: expected scalar type Half but found Float
</code></pre>
<p>what did <code>custom_fwd</code> and <code>custom_bwd </code> do specifically?</p>
<p>This refer to issue <code>https://github.com/pytorch/pytorch/issues/47906</code></p>
Separate Torchscript model into two parts

username: Jungmo_Ahn
cooked: <p>Hello, all.</p>
<p>Please imagine that we want to separate the process of the Deep learning model into the feature extraction process and the classification process.<br>
For example, the feature extraction process of AlexNet is from the first convolution layer and the last pooling layer which is located in front of the first fully connected layer. The remaining part from the first fully connected layer to the end of the Deep learning model is the classification process.</p>
<p>I checked that this partitioning can be achieved for the pre-trained AlexNet model in the Torchvision package. The feature extraction process is implemented in <em><strong>features</strong></em> which is a nn.Sequential instance and the classification process is implemented in <em><strong>classifier</strong></em> which is also a nn.Sequential instance.<br>
Thus, model.features.forward(input data) gives me the result of the feature extraction and model.classifier.forward(feature data) gives me the result of the classification.</p>
<p>My question is whether this is also possible for the Torchscript model.<br>
According to the tutorial of Torchscript, we use torch.jit.save(PyTorch model, filename) to convert a PyTorch model to a Torchscript model.<br>
Then, when I load the saved Torchscript model, it is impossible to access <em><strong>features</strong></em> directly like the case of PyTorch model.</p>
<p>The reason what I thought is that the PyTorch model is eager mode and the Torchscript model is graph mode. Thus, the graph mode model might not able to be separated.</p>
<ul>
<li>If my idea is correct, is there no way to separate graph mode model into two parts without conducting torch.jit.save for <em><strong>features</strong></em> and <em><strong>classifier</strong></em>?</li>
<li>If my idea is not correct, please share your idea.</li>
</ul>
<p>Many thanks !</p>

username: vferrer
cooked: <p>Hi</p>
<p>You can export custom methods with TorchScript. So, I would make your model to have a “extract_features” method that extracts them y mark it as <code>@jit.export</code> . So, you can call <code>model.extract_features((x)</code>.</p>
Anomaly detection with synthetic data

username: mr_joe
cooked: <p>Hello everyone,</p>
<p>I’m working on a project in which I need to detect anomalies in a particular scene (two background scenes). The anomaly could be anything (bolts, pliers, glasses, etc.). However, I have generated synthetic data training with unity because I have very few realistic images and here comes the problem. I was looking throughout different techniques like Domain Adaptation since I need to find a solution that implies training on synthetic and doing inference on real ones, but each of them seems to be focused on the class of the image and thus could fail in finding particular anomaly. I’m not an expert in this field and I’d like to hear an expert’s opinion  since I am a little bit stuck <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"> . thank you very much for you answers!</p>

username: Preetham_R_Patlolla
cooked: <p>Hey <a class="mention" href="/u/mr_joe">@mr_joe</a>,</p>
<p>One suggestion from myside is to use the Transfer Learning paradigm. You can use the pretrained models for better generalization. You have mentioned that you are generating synthetic data to train your model. In that case try using the techniques like Domain Randomization for generating the dataset that is more diverse and robust. Please do refer to the paper below.</p>
<aside class="onebox pdf">
  <header class="source">
      <a href="https://openaccess.thecvf.com/content_cvpr_2018_workshops/papers/w14/Tremblay_Training_Deep_Networks_CVPR_2018_paper.pdf" target="_blank" rel="noopener nofollow ugc">openaccess.thecvf.com</a>
  </header>
  <article class="onebox-body">
    <a href="https://openaccess.thecvf.com/content_cvpr_2018_workshops/papers/w14/Tremblay_Training_Deep_Networks_CVPR_2018_paper.pdf" target="_blank" rel="noopener nofollow ugc"><span class="pdf-onebox-logo"></span></a>
<h3><a href="https://openaccess.thecvf.com/content_cvpr_2018_workshops/papers/w14/Tremblay_Training_Deep_Networks_CVPR_2018_paper.pdf" target="_blank" rel="noopener nofollow ugc">Tremblay_Training_Deep_Networks_CVPR_2018_paper.pdf</a></h3>

<p class="filesize">1557.36 KB</p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


username: mr_joe
cooked: <p>Hello <a class="mention" href="/u/preetham_r_patlolla">@Preetham_R_Patlolla</a>,</p>
<p>Thank you for your answer! The idea is good but I think it won’t be suitable for my task because the goal is to detect anomalies (even unknown ones) while the paper is about detecting training-time known objects. Moreover, I didn’t mention that the synthetic data have been provided to me by someone else but there is plenty of it (more than 15k against 350 real ones). I was thinking about anomaly detector with GANs. My intuition is to try to translate synthetic images in realistic ones (maybe using StyleGAN) and then train an anomaly detector on synthetic realistic images without any anomalies in order to detect them at inference time.</p>
Masked_select vs indexing with boolean in aten

username: dashesy
cooked: <p>I want to use a boolean as index with aten, there is a <code>masked_select</code> that does the job but flattens the output (I can use view to correct that).<br>
How is indexing with a boolean array implemented in aten (does it call masked_select and then change the view)?</p>
<p>IOW, <code>masked_select</code> returns a different shape than using <code>[]</code>, what is the equivalent of <code>[]</code> in aten/C++?</p>
<pre><code class="lang-python">    x = torch.randn(3, 4)
    mask = torch.tensor([[1], [0], [1]], dtype=torch.uint8)
    print(x.masked_select(mask).shape)
    print(x[mask.view(-1)].shape)
</code></pre>

username: oli4100
cooked: <p>Hi,</p>
<p>Did you find out how to do this? I am having the same issue and I am pondering what the best method is (e.g. using masked_select and then view, or converting the boolean mask to int and doing index_select)</p>

username: oli4100
cooked: <p>For anyone stumbling upon this topic, the solution is to use .index, e.g. :</p>
<pre><code class="lang-auto">auto boolean_selector = (A == B);
auto C = D.index({boolean_selector});
</code></pre>
<p>and for a two-dimensional array where one wants to select along the second dimension of D:</p>
<pre><code class="lang-auto">auto boolean_selector = (A == B).squeeze();
auto C = D.index({Slice(), boolean_selector});
</code></pre>
<p>with D of size [m x n] and A, B of size [n x 1].</p>