akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to initialize the hidden state of a LSTM?

username: Peter_Ham
cooked: <p>in order to use LSTM, you need a hidden state and a cell state, which is not provided in the first place. My question is how to you initialize the hidden state and the cell state for the first input? If it is randomly initialized then if I feed into the second input, the same initialization should also work to predict the next output. But it does not make sense to me that inputting different hidden/cell state would come up with the same output.</p>

username: tom
cooked: <p>Even though that is in the context of BPTT, I think similar considerations apply as discussed the first item in:</p>
<aside class="quote quote-modified" data-post="2" data-topic="58384">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tom/48/3162_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/initialization-of-first-hidden-state-in-lstm-and-truncated-bptt/58384/2">Initialization of first hidden state in LSTM and truncated BPTT</a> 
  </div>
  <blockquote>
    Yes, zero initial hiddenstate is standard so much so that it is the default in nn.LSTM if you don’t pass in a hidden state (rather than, e.g. throwing an error).  Random initialization could also be used if zeros don’t work. Two basic ideas here: 

If your hidden state evolution is “ergodic”, the state will move closer to some “steady distribution” anyways, so it doesn’t matter as much.
You want the initial hidden state handling to be somewhat consistent between training and inference.
The fan…
  </blockquote>
</aside>

<p>Best regards</p>
<p>Thomas</p>
Error when exporting to onnx: "Auto nesting doesn't know how to process an input object of type maskrcnn_benchmark.structures.image_list.ImageList. Accepted types: Tensors, or lists/tuples of them"

username: kasen_zhou
cooked: <p>pytorch:1.0.0<br>
model: faster-RCNN(<a href="https://github.com/facebookresearch/maskrcnn-benchmark" rel="nofollow noopener">https://github.com/facebookresearch/maskrcnn-benchmark</a>)<br>
when export to onnx, get the error:</p>
<pre><code class="lang-auto">……
File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/maskrcnn_benchmark/modeling/detector/generalized_rcnn.py", line 50, in forward
    proposals, proposal_losses = self.rpn(images, features, targets)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 487, in __call__
    result = self._slow_forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 464, in _slow_forward
    input_vars = tuple(torch.autograd.function._iter_tensors(input))
  File "/opt/conda/lib/python3.6/site-packages/torch/autograd/function.py", line 284, in _iter
    for var in _iter(o):
  File "/opt/conda/lib/python3.6/site-packages/torch/autograd/function.py", line 293, in _iter
    if condition_msg else ""))
ValueError: Auto nesting doesn't know how to process an input object of type maskrcnn_benchmark.structures.image_list.ImageList. Accepted types: Tensors, or lists/tuples of them
</code></pre>
nn.TransformerDecoder and nn.TransformerEncoder inference doesn't work

username: xdwang0726
cooked: <p>I am using customized Transformer with nn.TransformerEncoder and nn.TransformerDecoder layer . It seem like nn.TransformerDecoder layer doesn’t support inference process(generation/testing), like sending token id one by one with fixed memory generated from nn.TransformerEncoder layer. I am wondering is there a tutorial that I can refer to as I didn’t find a tutorial in the official documents. Thank you in advance for your help!<br>
Encoder is Bert plus nn.TransformerEncoder and Decoder is nn.TransformerDecoder<br>
Here’s my codes:</p>
<pre><code class="lang-auto">    def greedy_inference(self,
                          src_input_ids,
                          max_tgt_seq_len,
                          src_token_type_ids,
                          src_attention_mask,
                          tgt_input_ids=None,
                          inference=True):
         with torch.no_grad():
             batch_size = 1
    
             if tgt_input_ids is not None:
                 tgt_seq_len = tgt_input_ids.shape[1]
             else:
                 assert max_tgt_seq_len is not None, 'Target sequence length do not defined'
                 tgt_seq_len = max_tgt_seq_len
    
             # encoder
             encoder_vec = self.bert_encoder(src_input_ids, src_token_type_ids, src_attention_mask)
             # create initial tokens: shape tensor([[101]])
             generated_seq = torch.full((batch_size, 1), Constants.BOS, dtype=torch.long, device=self.device)
    
             for i in range(1, tgt_seq_len):
                 decoder_outputs = self.bert_decoder(encoder_vec, generated_seq, inference)
                 _, lsm_score = self.generator(decoder_outputs)
                 # Take token with largest probability and use it for the next words
                 generated_token_index = lsm_score[-1, :, :]  # lsm_score shape [num_tokens, bs, vocab_size]
                 _, generated_token_index = torch.topk(generated_token_index, 1, dim=-1)
                 # Concatenate generated token with sequence
                 generated_seq = torch.cat((generated_seq, generated_token_index), dim=-1)
    
         return generated_seq
</code></pre>
Need Help/Contributions to PyTorch C++ Tutorial for Deep Learning

username: prabhuomkar
cooked: <p>Hi everyone!</p>
<p>I am working on C++ Implementation of PyTorch Tutorial for Deep Learning Researchers. I aim to help C++ programmers with easy to follow tutorials which cover all basics, machine learning examples and wide range of Deep Learning examples as well.</p>
<p>I am following <a href="https://github.com/yunjey/pytorch-tutorial" rel="nofollow noopener">https://github.com/yunjey/pytorch-tutorial</a> as the base for my C++ tutorials. Programmers who would like to contribute can pick up topics from Table of Contents and can work on it. Also, feel free to add more topics <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>GitHub: <a href="https://github.com/prabhuomkar/pytorch-cpp" rel="nofollow noopener">https://github.com/prabhuomkar/pytorch-cpp</a></p>

username: ptrblck
cooked: <p>Hi Omkar!</p>
<p>Just wanted to ping <a class="mention" href="/u/krshrimali">@krshrimali</a>, as he started writing a blog series on the <a href="https://krshrimali.github.io/Announcing-PyTorch-CPP-Series/" rel="nofollow noopener">PyTorch C++ API</a>. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: prabhuomkar
cooked: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> will connect <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
Relation between Tensor and FloatTensor

username: ralphmao
cooked: <p>The doc says " <a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor" rel="noopener nofollow ugc"> <code>torch.Tensor</code> </a> is an alias for the default tensor type ( <code>torch.FloatTensor</code> )"<br>
However I found it is not that trivial. When I run the following code:</p>
<blockquote>
<p>a = torch.zeros(3,4)<br>
isinstance(a, torch.Tensor) # Return True<br>
isinstance(a, torch.FloatTensor) # Return True</p>
</blockquote>
<p>Things look good so far. However, if I check the mro of both classes:</p>
<blockquote>
<p>torch.Tensor.__ mro__ # Return (torch.Tensor, torch.<em>C.<em>TensorBase, object)<br>
torch.FloatTensor.</em></em>__ mro__ # Return (torch.FloatTensor, object)</p>
</blockquote>
<p>There seems to be no inheritence between FloatTensor and Tensor. Then how could isinstance return True for both two classes?</p>
<p>Could someone point out these two classes’ relationship, or anything wrong with my understanding of isinstance() and <strong>mro</strong>?</p>

username: albanD
cooked: <p>Hi,</p>
<p>It’s non trivial because it also changes the cpp behavior. And so this link is rooted in cpp code.<br>
I am not sure how this works exactly, but I can dig into if you really want to <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: ralphmao
cooked: <p>Thank you for your reply! I just want to know what magic makes isinstance() return True for both classes even if they do not inherit each other. It would be very helpful if you could answer it!</p>

username: albanD
cooked: <p>Well in cpp, you have a single <a href="https://github.com/pytorch/pytorch/blob/aa49aa856c559e7aef87d9935e968a6c4850e2ce/torch/csrc/tensor/python_tensor.cpp#L140" rel="nofollow noopener">metaclass</a> and then <a href="https://github.com/pytorch/pytorch/blob/aa49aa856c559e7aef87d9935e968a6c4850e2ce/torch/csrc/tensor/python_tensor.cpp#L308-L310" rel="nofollow noopener">for each tensor type</a>, <a href="https://github.com/pytorch/pytorch/blob/aa49aa856c559e7aef87d9935e968a6c4850e2ce/torch/csrc/tensor/python_tensor.cpp#L162-L163" rel="nofollow noopener">set the parent type to be the metaclass</a>.<br>
This metaclass is here to make this <code>isinstance</code> magic work using <a href="https://github.com/pytorch/pytorch/blob/aa49aa856c559e7aef87d9935e968a6c4850e2ce/torch/csrc/tensor/python_tensor.cpp#L75-L99" rel="nofollow noopener">this function</a> that is called when you call isinstance because of <a href="https://github.com/pytorch/pytorch/blob/aa49aa856c559e7aef87d9935e968a6c4850e2ce/torch/csrc/tensor/python_tensor.cpp#L126" rel="nofollow noopener">this</a>.</p>
<p>Hope this helps <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Loss increases, converges particular result

username: srslysirius
cooked: <p>I’m trying to figure out how to run a tensor through a function, do some math inside that function, and compare the output to a predetermined “fitness goal” to be used in the loss function. I have started with a very simple example, but already run into issues.</p>
<p>This example is a bread shopping simulation, where every 3rd day bread goes on sale for 90% off. The input and output values just determine what % of your budget to spend on bread each day. Ideally it should end up [0, 0, 1, 0, 0, 1, 0, 0, 1] and so forth</p>
<pre><code>def buybread(args):
    
    budget = 1000
    cost = 10
    totalbread = 0
    days = range((len(args[0])))
    
    for i in days:
        if (i+1) % 3 == 0:
            cost = 1
        else:
            cost = 10
    
        spend = budget
        bought = int((args[0][i] * spend)/cost)
        totalbread += bought
        budget -= (bought * cost)
        
    argsum = torch.sum(args)
    fitness = torch.add(argsum, totalbread)
    return(fitness)

class TwoLayerNet(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(21,100)
        self.fc2 = nn.Linear(100,21)
    def forward(self, x):
        x = F.sigmoid(self.fc1(x))
        x = F.sigmoid(self.fc2(x))
        return x

m = TwoLayerNet()
loss_fn = nn.MSELoss()
optimizer = optim.Adam(m.parameters(), lr=0.01)

training_epochs = 30
fitnessgoal = torch.tensor(1000)
blanktensor = torch.zeros([1, 21])

   
for i in range(training_epochs):

    fpass = m(blanktensor)
    fitness = buybread(fpass)
    loss = loss_fn(fitness, fitnessgoal.float())
    print(i, fitness, loss)
    optimizer.zero_grad()
    
    loss.backward()
    
    optimizer.step()
</code></pre>
<p>Ideally it should learn to buy bread only on the discount days, which is every 3 days when a loaf costs 10x less. If played perfectly, the optimal result is buying 1,000 loaves for $1 each.</p>
<p>Instead, it converges on 129 loaves, no matter what I set the fitness goal to. Oftentimes it will buy 260 or more loaves on the first epoch, just by virtue of randomness, but usually settles on 129 within ten epochs.</p>
<p>I assume that what’s actually happening is that the variable totalbread is being disregarded entirely, and pytorch is optimizing for the highest value for argsum. When I add print(fpass) to the loop, I generally get something like this for the last epoch:</p>
<pre><code>tensor([[0.9997, 0.9997, 0.9996, 0.9996, 0.9996, 0.9997, 0.9997, 0.9997, 0.9997,
     0.9997, 0.9997, 0.9996, 0.9996, 0.9996, 0.9996, 0.9996, 0.9996, 0.9996,
     0.9996, 0.9996, 0.9996]], grad_fn=&lt;SigmoidBackward&gt;)
</code></pre>
<p>So it’s trying to spend almost the maximum allowed budget every day, including day 0 where most of it gets used up.</p>
<p>I had previously tried simply setting fitness = totalbread, but that ran into all sorts of problems with being the wrong type of variable, and eventually the loss not changing at all.</p>
How to get input&output tensor of a module in backward process

username: pinaggle
cooked: <p>Hi everyone,</p>
<p>I’m working on a project which requires me to get input and output tensors of intermediate layers for further analysis. However, I want to get them in backpropagation process for the convenience of analysis, although they are calculated in forward process.<br>
The <em>register_backward_hook</em> function might be useful, but it only return <em>grad_input</em> and <em>grad_output</em> tensors, which are not the tensors I want. Can anyone give some advice to me? Any information will be appreciated.</p>

username: mailcorahul
cooked: <p>May I know why you need it in the backpropagation process?<br>
You can simply save the forward call results for intermediate layers using <code>register_forward_hook</code> and then use it in <code>register_backward_hook</code>.</p>

username: pinaggle
cooked: <p>Hi mailcorahul,</p>
<p>Thanks for your advice!<br>
I want to implement an algorithm which calculates effective path of neural network, which is proposed by a recent paper. In fact I don’t really need to do it in backpropagation process, but as the algorithm processes the network from output layer to input layer, I think implementing it in backpropagation process might be natural.<br>
However, I still have some questions on that solution. First, where should I put the intermediate layers’ results, since the hook doesn’t need ctx as a parameter? Besides, sometimes a module will be used several times in a model, will that cause some problems?</p>

username: mailcorahul
cooked: <p>You can store the forward results in an attribute. Simply saving resnet18’s output after first 7x7 conv…</p>
<pre><code class="lang-auto">class Net(nn.module):
    def __init__(self):
        self.net = models.resnet18(pretrained=True)
        self.remove_handle = self.net._modules['relu'].register_forward_hook(self.forward_hook)
        self.forward_output = None
    def forward_hook(self, module, input, output):
        # save any modules output in an attribute and later use it in backprop.
        self.forward_output = output
</code></pre>
<p>Afraid I won’t be able to answer your question on 'will multiple calls to a module cause problems" since I don’t have much idea about the paper(on effective path of neural network) you’re talking about. But regarding a module being called several times, the forward hook will be invoked on every call.</p>

username: pinaggle
cooked: <p>Thanks very much for your help! I’ll have a try.</p>

username: sebamenabar
cooked: <p>Hi, I made a library to get the intermediate results, minimal modifications to your model (none for simple use cases) and if a module is called more than once all the results are saved. Check it out. <a href="https://pypi.org/project/torch-intermediate-layer-getter/" rel="nofollow noopener">Pytorch-Intermediate-Layer-Getter</a></p>
How to get intermediate features from pre-trained model

username: 1414b35e42c77e0a57dd
cooked: <pre><code class="lang-auto">trained_model have
    self.conv1 = ~~~
    self.conv2 = ~~~
    self.conv3 = ~~~
    self.conv4 = ~~~
    ...

then,

trained_model = torch.load('trained_net')
half_model = nn.Sequential(trained_model.conv1, trained_model.conv2)
</code></pre>
<p>Hi all,<br>
I’ve tried to get intermediate features from pre-trained model.<br>
Is it correct way to get intermediate features ?</p>

username: blackberry
cooked: <p>If you need conv2 output, yes it would work.</p>
<p>Another  way is to override the <code>forward</code> function of the model class in a subclass and run the input through the layers (conv1, conv2) only.</p>

username: mailcorahul
cooked: <p>You can use <code>register_forward_hook</code> for whichever intermediate layer you want. Check out my previous post below on extracting features.</p><aside class="quote quote-modified" data-post="5" data-topic="57302">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mailcorahul/48/15919_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-to-get-input-output-tensor-of-a-module-in-backward-process/57302/5">How to get input&amp;output tensor of a module in backward process</a> <a class="badge-category__wrapper " href="/c/autograd/7"><span data-category-id="7" data-drop-close="true" class="badge-category " title="A category of posts relating to the autograd engine itself."><span class="badge-category__name">autograd</span></span></a>
  </div>
  <blockquote>
    You can store the forward results in an attribute. Simply saving resnet18’s output after first 7x7 conv… 
class Net(nn.module):
    def __init__(self):
        self.net = models.resnet18(pretrained=True)
        self.remove_handle = self.net._modules['relu'].register_forward_hook(self.forward_hook)
        self.forward_output = None
    def forward_hook(self, module, input, output):
        # save any modules output in an attribute and later use it in backprop.
        self.forward_output = outp…
  </blockquote>
</aside>


username: sebamenabar
cooked: <p>Hi! I made a library for this, it’s called <a href="https://pypi.org/project/torch-intermediate-layer-getter/" rel="nofollow noopener">Pytorch-Intermediate-Layer-Getter</a> (I know, cool name), check it out.</p>
[JIT] NamedTuple's _fields attribute

username: zetyquickly
cooked: <p>Don’t know is it a bug or just partial coverage but the _fields attribute of NamedTuple unavailable while scripting.</p>
<pre><code class="lang-auto">class _Contents(NamedTuple):
    a : Optional[torch.Tensor]
    b : Optional[torch.Tensor]
    c : Optional[torch.Tensor]
    d : Optional[torch.Tensor]

@torch.jit.script
class Container:
   def __init__(self, fields):
      # type: (Tuple[int, int], _Contents)
      self.fields = fields
  def has(self, name: str) -&gt; bool:
        """
        Returns:
            bool: whether the field called `name` exists.
        """
        return name in self.fields._fields
</code></pre>
<pre><code class="lang-auto">RuntimeError: 
Unknown attribute to named tuple:
at container.py:91:23
    def has(self, name: str) -&gt; bool:
        """
        Returns:
            bool: whether the field called `name` exists.
        """
        return name in self.fields._fields
                       ~~~~~~~~~~~~~~~~~~~~ &lt;--- HERE
</code></pre>

username: Michael_Suo
cooked: <p>We have not bound <code>_fields</code> into NamedTuples. If you file a feature request on GitHub we can take a look at implementing it <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
DataParallel changes parameter names, issue with load_state_dict()

username: xvdp
cooked: <p>Im working on multigpu, single gpu and cpu on the same model.<br>
I noticed that dataparallel changes the parameter names appending the names to a “module.” prefix.<br>
eg. from “conv1.weight” to “module.conv1.weight”<br>
This is fine for training, until one needs to load the parameters in a model that is not running DataParallel.</p>
<p>Short of rebuilding the OrderedDict splitting “module.” out of the names; or building a generalized trained model loader that matches partial names and tensor shapes, Is there a generalized way of handling this?<br>
its no biggie, but i rather use a generalized handler if one is available. Im testing inference times on cpu only devices but my training setup is multigpu.</p>
<p>thanks</p>

username: JuanFMontesinos
cooked: <p>You can just modify dic names by hand and manually removing module.</p>
<pre><code class="lang-auto">            state_dict = torch.load(directory, map_location=lambda storage, loc: storage)
        new_state_dict = OrderedDict()

        for k, v in state_dict.items():
            if k[:7] == 'module.':
                name = k[7:]  # remove `module.`
            else:
                name = k
            new_state_dict[name] = v
</code></pre>

username: xvdp
cooked: <p>right, thanks,  i was asking if there was a generalized loader that didnt require that. I was hacking it like this,</p>
<pre><code class="lang-python">from collections import OrderedDict as OD
#...
          if device == "cuda":
                self.model = torch.nn.DataParallel(self.model)
                self.model.load_state_dict(state_dict)
            else:
                state = OD([(key.split("module.")[-1], state_dict[key]) for key in state_dict])

                self.model.load_state_dict(state)
#...
</code></pre>
Understanding autograd calculation of backprop

username: JakeStevens
cooked: <p>I am attempting to re-implement backpropagation on my own for didactic purposes, but am running into some issues. I am trying to work backwards from a simple network, starting with LogSoftmax + NLLLoss, but I am unable to match the calculated gradient of the input to the LogSoftmax layer as calculated by autograd.</p>
<pre><code class="lang-auto">import torch
import torch.nn

new_relu_feats = torch.Tensor([[1,0,3]])
new_relu_feats.requires_grad = True
logits = torch.nn.LogSoftmax(dim=1)(new_relu_feats)
logits.retain_grad()
label = torch.LongTensor([1])
loss = torch.nn.NLLLoss()(logits, label)
loss.backward()

sm = torch.nn.Softmax(dim=1)(new_relu_feats)
dloss_dlogits = torch.Tensor([[0,-1,0]])
logits.grad # This matches above
dlogits_dnew_relu_feat = torch.Tensor([[[1-sm[0][0], -sm[0][0], -sm[0][0]], [-sm[0][1], 1-sm[0][1], -sm[0][1]], [-sm[0][2], -sm[0][2], 1-sm[0][2]]]])

dloss_dlogits * dlogits_dnew_relu_feat
# This does not entirely match above, but the middle column does (corresponding to the correct class)
new_relu_feats.grad
</code></pre>
<p>Is this correct, but the matrix is eventually reshaped for efficiency by simply selecting the portions that are non-zero (aka corresponding to the correct class)?</p>

username: albanD
cooked: <p>Hi,</p>
<p>You can use triple backticks ``` before and after your code to have nicer formatting.</p>
<p>The combination of gradients if not an element wise product but a matrix matrix multiplications.<br>
You can do <code>torch.bmm(dlogits_dnew_relu_feat, dloss_dlogits.unsqueeze(-1)).squeeze(-1)</code> to get what you want.<br>
The unsqueeze/squeeze in the last dimension is just to have a dummy dimension of size 1 to make <code>bmm</code> happy <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: JakeStevens
cooked: <p>Thank you so much! That did the trick and I can match these layers, along with a different example that includes a Relu before the softmax. I am still a little unclear why it is dlogits_dnew_relu_feat @ dloss_dlogits rather than the other way round (dloss_dlogits @ dlogits_dnew_relu_feat) from a conceptual point of view.</p>

username: JakeStevens
cooked: <p>Ah, I see now. The derivative for LogSoftMax was transposed from how I was thinking about it conceptually:</p>
<pre><code class="lang-auto">dlogits_dnew_relu_feat = torch.Tensor([[[1-sm[0][0], -sm[0][0], -sm[0][0]], [-sm[0][1], 1-sm[0][1], -sm[0][1]], [-sm[0][2], -sm[0][2], 1-sm[0][2]]]])
</code></pre>
<p>should  be</p>
<pre><code class="lang-auto">dlogits_dnew_relu_feat = torch.Tensor([[[1-sm[0][0], -sm[0][1], -sm[0][2]], [-sm[0][0], 1-sm[0][1], -sm[0][2]], [-sm[0][0], -sm[0][1], 1-sm[0][2]]]])
</code></pre>
<p>and now you can do dloss_dlogits @ dlogits_dnew_relu_feat with no need for squeeze/unsqueeze</p>
U-Net training with mixed precision fails with cublasGemmEx RuntimeError

username: tastyminerals
cooked: <p>We are trying to train a U-Net based network for box classification using cross entropy loss. We are using  new NVIDIA APEX mixed precision library and having the following issue during training. After couple of epochs (depending on the APEX apm optimization level) the training stops due to the following error message:</p>
<pre><code class="lang-auto">...
  File "/home/pavel/.local/lib/python3.7/site-packages/torch/tensor.py", line 150, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/pavel/.local/lib/python3.7/site-packages/torch/autograd/__init__.py", line 99, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling `cublasGemmEx( handle, opa, opb, m, n, k, &amp;falpha, a, CUDA_R_16F, lda, b, CUDA_R_16F, ldb, &amp;fbeta, c, CUDA_R_16F, ldc, CUDA_R_32F, CUBLAS_GEMM_DFALT_TENSOR_OP)`
</code></pre>
<p>Could there be a bug in <code>backward</code> which is triggered by this particular network because <strong>resnet18</strong> trains fine with apex using any optimization level.<br>
Does anyone have a clue of what could be the reason?</p>
<p>Here is where the error is getting thrown:</p>
<pre><code class="lang-python">        for epoch in range(1, self.num_epochs + 1):
            logger.info(f"running epoch {epoch}")
            avg_train_loss = 0

            self.model.train()
            for step, sample_batch in enumerate(self.train_data, start=1):
                sample_batch = self._sample_to_device(sample_batch)
                self.optimizer.zero_grad()

                doc_id_batch = sample_batch[DOC_ID]

                logits_dict = self.model(sample_batch)
                loss = self.criterion(logits_dict, sample_batch)
                with amp.scale_loss(loss, self.optimizer) as scaled_loss:
                    scaled_loss.backward()  # &lt;&lt;&lt; exception is thrown

                self.optimizer.step()

                avg_train_loss += loss.item()

            epoch_end_time = timeit.default_timer()
            epoch_time = epoch_end_time - epoch_start_time
</code></pre>
<p>The original issue discussion and logs are here: <a href="https://github.com/NVIDIA/apex/issues/580#issuecomment-549487887" rel="nofollow noopener">https://github.com/NVIDIA/apex/issues/580#issuecomment-549487887</a></p>

username: ptrblck
cooked: <p>I’ve commented in the GitHub issue.<br>
Your error might be related to <a href="https://github.com/pytorch/pytorch/issues/24018" rel="nofollow noopener">this issue</a>, if your input size is too large.</p>
Reusing Tensor and Optimizer in nested loops

username: atouchet
cooked: <p>My question pertains to the implementation of adversarial attacks that used Adam or SGD optimizers on the perturbation tensor. In particular, attacks such as Carlini-Wagner (<a href="https://github.com/BorealisAI/advertorch/blob/73610dce2d4c65e3fa0e5bef92b68caea2d8f95e/advertorch/attacks/carlini_wagner.py#L218" rel="nofollow noopener">advertorch implementation</a>). Attacks that require two levels of optimization have the general structure:</p>
<pre><code class="lang-auto">for outer_step in range(outer_max):
    delta = nn.Parameter(torch.zeros_like(x))
    optimizer = optim.SGD([delta], lr=0.01)
    for inner_step in range(inner_max):
        # Some stuff
        loss.backward()
        optimizer.step()
</code></pre>
<p>I would like to do the following instead:</p>
<pre><code class="lang-auto"># Initialize a tensor and its optimizer outside the loop
delta = torch.zeros_like(x,requires_grad=True)
optimizer = ch.optim.SGD([delta],lr=.001)
for outer_step in range(outer_max):
    # Zero the tensor instead of creating new one
    with ch.no_grad():
        delta.data.sub_(delta)
    for inner_step in range(inner_max):
        # Some stuff
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
</code></pre>
<p>I’ve done basic testing successfully, but to be honest I’m not sure under which considers this might break.</p>
<p>Does anyone have any suggestions on if this is safe to do, and if not is there any alternative that still uses optimizers?</p>

username: albanD
cooked: <p>You should not (never) use <code>.data</code>. The <code>with torch.no_grad()</code> is good here.</p>
<p>This looks good to me. Keep in mind though that if you use optimizers that have states (sgd with momentum, adam, etc) these states will persist from one inner loop to the other.</p>

username: atouchet
cooked: <p>I originally had resorted to using <code>.data</code> because of the error <code>a leaf Variable that requires grad has been used in an in-place operation</code>. But I see that <code>with no_grad()</code> takes of this problem!</p>
<p>As for the states of the optimizer, that is another challenge I’m working on by building optimizer specific to attacks. Which is only tangentially related to this topic.</p>
<p>Thanks!</p>
TypeError: fileCopyLocalRepository() takes 1 positional argument but 2 were given

username: ashokkumarsoliyappan
cooked: <p>Here is my code</p>
<p>class FileUpload():</p>
<pre><code>def fileCopyLocalRepository(requiredImgFiles):
	userLocalRepo = os.mkdir(os.environ['USERPROFILE'] + "\Desktop","dummy")
	print("success")
	exit()
	for filePath in requiredImgFiles :
		print(filePath)
		destinationLocation = userLocalRepo
		try:
			shutil.copyfile(filePath, destinationLocation)
			print("File copied successfully.")
		except shutil.SameFileError: 
			print("Source and destination represents the same file.")
		except IsADirectoryError: 
			print("Destination is a directory.")
		except PermissionError: 
			print("Permission denied.")
		except: 
			print("Error occurred while copying file.")

def processStart(self):
	requiredImgFiles = []
	definedFileFormat = "**/*.txt"
	currentSystemUser = getpass.getuser()
	userSourcePath = os.getcwd() #to find the user python directory
	desiredPath = userSourcePath + "\AutomationProject"
	requiredImgFiles = [f for f in glob.glob(desiredPath + definedFileFormat, recursive=False)] #to find partiular extension file format
	self.fileCopyLocalRepository(requiredImgFiles)

def gitUserLogin(self):
	print(config.gitUserCred['userName'])
	exit()	
</code></pre>
<p>obj = FileUpload()<br>
data = obj.processStart()</p>

username: ptrblck
cooked: <p>The error doesn’t seem to be related to PyTorch, does it?<br>
Anyway, note that you are calling <code>self.fileCopyLocalRepository(requiredImgFiles)</code> in <code>processStart</code>, while the posted <code>fileCopyLocalRepository</code> method isn’t defines as a class method.<br>
Maybe you are calling the class method with the same name?</p>
Does calling backward on two different variables affect how gradients are calculated?

username: jwillette
cooked: <p>I saw someone’s code that went like this while operating on the same model…</p>
<pre><code class="lang-auto">loss_one = criterion(...)
loss_one.backward()
loss_two = criterion(...)
loss_two.backward()
optimizer.step()
</code></pre>
<p>My feeling would be to code it like this…</p>
<pre><code class="lang-auto">loss = criterion(...) + criterion(...)
loss.backward()
optimizer.step()
</code></pre>
<p>I cannot explain to myself if there is any difference in these two flows…what is the truth?</p>

username: albanD
cooked: <p>Hi,</p>
<p>There is no difference between the two in the gradients that are computed.</p>
<p>Because of some optimization we do (to reduce memory usage), you might have to pass <code>retain_graph=True</code> to the first backward to avoir errors.</p>
How to use nn.init.orthogonal_ with more than 2 dimensions

username: gaoliming
cooked: <p>I want to use nn.init.orthogonal_ get orthogonal matrix. For 2 dimensions, such as this code:</p>
<pre><code class="lang-auto">torch.nn.init.orthogonal_(torch.empty(2, 2))
tensor([[ 0.8164,  0.5775],
        [ 0.5775, -0.8164]])
</code></pre>
<p>But for more than 2 dimensions, such as this:</p>
<pre><code class="lang-auto">torch.nn.init.orthogonal_(torch.empty(3, 2, 2))
tensor([[[ 0.1439,  0.1052],
         [ 0.1259, -0.9759]],

        [[-0.4479, -0.0665],
         [ 0.8906,  0.0417]],

        [[ 0.1099, -0.9895],
         [-0.0143, -0.0923]]])
</code></pre>
<p>obviously, for every 2 * 2 matrix not orthogonal matrix, but I want get three 2 * 2 orthogonal  matrix in the above code. I read the document <a href="https://pytorch.org/docs/stable/nn.init.html?highlight=orthogonal#torch.nn.init.orthogonal_" rel="nofollow noopener">torch.nn.init.orthogonal_</a> , but I can not understand the mean of flattened. Anyone can help me?</p>

username: albanD
cooked: <p>You can check the source <a href="https://pytorch.org/docs/stable/_modules/torch/nn/init.html#orthogonal_" rel="nofollow noopener">here</a> but it basically does <code>inp = inp.view(inp.size(0), -1)</code> And then optionally transpose to make sure to be able to find an orthogonal matrix.<br>
If you want many 2x2 orthogonal matrices, you might have to call <code>orthogonal_</code> multiple times.</p>
Visualize learnt pattern

username: vainaijr
cooked: <p>at which layer of a neural network, do I know the pattern that it has learnt, for example if I pass an image of cat to a neural network, and it gives it label of cat, by giving high probability at the final layer, which layer would show the pattern of cat it has learnt?<br>
Similarly if I pass image of dog to the neural network, and it correctly labels it as dog, which layer would show the pattern it has learnt?<br>
same thing for capsule networks also, how do I see what pattern has a capsule learnt?</p>

username: ptrblck
cooked: <p>The kernels and activations do not directly show a cat or dog feature.<br>
However, you could e.g. try to visualize the gradients using e.g. <a href="https://github.com/MisaOgura/flashtorch" rel="nofollow noopener">flashtorch</a>.</p>
Save elapsed time per epoch to txt/csv file?

username: astri
cooked: <p>Hello all<br>
I am a beginner in deep learning and pytorch and train my model like code below. I want to ask is it possible to save my elapsed time per epoch in a file . I want to do it because i want to see the max,min,and average elapsed time.</p>
<pre><code class="lang-auto">epochs = 5
train_losses=[]

for epochs in range(epochs):
    t0 = time.time()
    model.train()
    for batch_idx, (x,y) in enumerate(loader):
        x,y = x.to(device), y.to(device)
        optimizer.zero_grad()
        outputs = model(x)
        loss = loss_fn(outputs,y)
        loss.backward()
        optimizer.step()
        StopatLossValue()
        
        # program timing
        torch.cuda.synchronize() 
        t1 = time.time()
        eta = t1-t0

        if batch_idx % 100 == 0:
            print('Train Epoch: {} \t Loss: {:.6f} \t Elapsed time :{:.2f} ms'.format(epochs, loss.item(), ((eta)*1000)))
</code></pre>

username: Nikronic
cooked: <p>Hi,</p>
<p>Your question is not directly affecting PyTorch, so you can save the <code>eta</code> variable in a list with same length of number of <code>epochs</code> and at the end of the loops, write it into a file using python <code>open(file_path, 'w+')</code> method.<br>
There are other ways to do it though.</p>
<p>But something you should consider is that writing to a file is a slow process in comparison to tensor operations, so it is not good idea to write into files within your training or testing workflow, so just save the value in a array and when the loop finished, write to disk.</p>
<p>Bests<br>
Nik</p>

username: astri
cooked: <p>thank you for your reply. I have managed to save it</p>
EOFError in multiprocessing/connection

username: keldLundgaard
cooked: <p>Dear forum,</p>
<p>I’m getting the attached error message, which I have a hard time to decrypt. Pytorch will throw this error, but continue most of the time. Should I suppress it or does it tell me that I should construct my data loader in some other way?</p>
<p>Running Pytorch 0.4 on p2.xlarge instance.</p>
<p>Let me know if more information about my code is needed to help.</p>
<pre><code class="lang-auto">Exception ignored in: &lt;bound method _DataLoaderIter.__del__ of &lt;torch.utils.data.dataloader._DataLoaderIter object at 0x7fcc2fed5f60&gt;&gt;
Traceback (most recent call last):
  File "/data/miniconda3/envs/vae_cf/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 349, in __del__
    self._shutdown_workers()
  File "/data/miniconda3/envs/vae_cf/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 328, in _shutdown_workers
    self.worker_result_queue.get()
  File "/data/miniconda3/envs/vae_cf/lib/python3.6/multiprocessing/queues.py", line 337, in get
    return _ForkingPickler.loads(res)
  File "/data/miniconda3/envs/vae_cf/lib/python3.6/site-packages/torch/multiprocessing/reductions.py", line 70, in rebuild_storage_fd
    fd = df.detach()
  File "/data/miniconda3/envs/vae_cf/lib/python3.6/multiprocessing/resource_sharer.py", line 57, in detach
    with _resource_sharer.get_connection(self._id) as conn:
  File "/data/miniconda3/envs/vae_cf/lib/python3.6/multiprocessing/resource_sharer.py", line 87, in get_connection
    c = Client(address, authkey=process.current_process().authkey)
  File "/data/miniconda3/envs/vae_cf/lib/python3.6/multiprocessing/connection.py", line 493, in Client
    answer_challenge(c, authkey)
  File "/data/miniconda3/envs/vae_cf/lib/python3.6/multiprocessing/connection.py", line 732, in answer_challenge
    message = connection.recv_bytes(256)         # reject large message
  File "/data/miniconda3/envs/vae_cf/lib/python3.6/multiprocessing/connection.py", line 216, in recv_bytes
    buf = self._recv_bytes(maxlength)
  File "/data/miniconda3/envs/vae_cf/lib/python3.6/multiprocessing/connection.py", line 407, in _recv_bytes
    buf = self._recv(4)
  File "/data/miniconda3/envs/vae_cf/lib/python3.6/multiprocessing/connection.py", line 383, in _recv
    raise EOFError
EOFError:
</code></pre>

username: Michael_Gussert
cooked: <p>I see this as well, no idea why.  It appears to happen intermittently and I can’t seem to reproduce it consistently</p>

username: ttrun
cooked: <p>It may be a pytorch version issue；There is this problem in version 1.3.0，but I change the pytorch version to 1.0.1，it‘s ok。</p>

username: ptrblck
cooked: <p>Do you have a reproducible code snippet raising this error so that we could debug it?</p>

username: ttrun
cooked: <p>I ran the  NMT code from FaceBook in <a href="https://github.com/facebookresearch/UnsupervisedMT" rel="nofollow noopener">https://github.com/facebookresearch/UnsupervisedMT</a>.<br>
It had the EOFError in pytorch version 1.3.0, but it’s ok in version 1.0.1.</p>
Is it okay to create a Loss function within the forward method?

username: coyote
cooked: <p>Hi everyone, I have a model that uses HuggingFace’s Bert implementation as a submodule. (I share the relevant parts of the model below.) My question is about the proper place for loss function initialization. I’ll use the <code>nn.CrossEntropyLoss()</code> as a loss function of the model. Is it okay if I use it as below? Or should I initialize it as a field of the model within the <code>__init__</code> function (like self.loss_function = nn.CrossEntropyLoss()) and then use it inside the forward function? I asked this because when I initialize it within the forward function, I use a new loss function at every forward call and I don’t know if this breaks something (maybe loss function needs to remember something ? )</p>
<pre><code class="lang-auto">class MyModel(torch.nn.Module):

    def __init__(self,bert_dim,n_labels,pretrained_weights,dp=0.5):
        super(MyModel,self).__init__()
        self.num_labels = n_labels
        self.bert1 = BertModel.from_pretrained(pretrained_weights)
        self.bert2 = BertModel.from_pretrained(pretrained_weights)
        self.dropout = nn.Dropout(dp)
        self.classifier = nn.Linear(2*bert_dim, n_labels)
        
        
    def forward(self, input_ids=None, attention_mask=None, token_type_ids=None,
               labels=None,c_input_ids=None,c_attention_mask=None,c_token_type_ids=None):

        output1 = self.bert1(input_ids,
                            attention_mask=attention_mask,
                            token_type_ids=token_type_ids,

        output2 = self.bert2(c_input_ids,
                            attention_mask=c_attention_mask,
                            token_type_ids=c_token_type_ids,

         logits = # get logits by using output1 and output2 here for sake of simplicity. 
         loss_fct = nn.CrossEntropyLoss()
         loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1))
        return (logits),loss
</code></pre>

username: Nikronic
cooked: <p>Hi,</p>
<p>I could not find any information about buffering data from <code>Loss</code> (maybe necessary in particular loss). But the ideal form is to use initializations within <code>__init__</code> method and usage in forward, because initializing a <code>object</code> has its own overhead problems when for every single run in train or test loop, the object needs to be initialized (may cause memory issues too, because previously initialized objects are no longer useful and garbage collector should handle it I think).</p>
<p>So I think as you used this convention for <code>nn.Dropout</code> or the other parts, I think you should do it for <code>Loss</code> too.</p>
<p>Bests<br>
Nik</p>
Saving Spliting Dataset

username: Gedamu_Alemu
cooked: <p>test_to_train, test_to_test = indices_test[split_test:], indices_test[:split_test]<br>
test_to_train = SubsetRandomSampler(test_to_train)<br>
test_to_test = SubsetRandomSampler(test_to_test)</p>
<p>How to save these two split Datasets and is it possible to save the split datasets to load them later?</p>
How can I get the gradients as an integration with DataParallel (Multiple GPUs) according to input sequence?

username: AndrewSoul
cooked: <p>I used DataParallel with 2 GPUs.  I register_hook in model, like this:</p>
<blockquote>
<p>def forward(self, x):</p>
<pre><code>    x = self.features(x)
    x.register_hook(self.save_gradient)
    inter = x
  
    x = x.view(x.size(0), -1)
    x = self.classifier(x)
</code></pre>
</blockquote>
<p>When I do the backward, the gradients will be split into two parts.<br>
I tried to collect the grad in a list, but how can I make them an integration according to the input sequence?   Does the DataParallel compute the gradient according to gpus_id order squentially?<br>
Thanks!</p>

username: fanl
cooked: <p>I got the same problem, do you have a solution? Thanks.</p>
Index a tensor using two tensors

username: kcang2
cooked: <p>Hi, I have an N x IC x IH x IW input, and I want to index the IH and IW dimensions using two N x M tensors to get a N x IC x M output. How to do it?</p>

username: tom
cooked: <p><a href="https://docs.scipy.org/doc/numpy/reference/arrays.indexing.html#advanced-indexing" rel="nofollow noopener">Advanced indexing</a> (links to the NumPy reference, but PyTorch wants to be compatible here) should do the trick:</p>
<pre><code class="lang-python">values = torch.arange(2 * 3 * 5 * 5).view(2, 3, 5, 5)
idx_x = torch.randint(0, 5, (4,))
idx_y = torch.randint(0, 5, (4,))
values[:, :, idx_x, idx_y]
</code></pre>
<p>Best regards</p>
<p>Thomas</p>

username: kcang2
cooked: <p>Hi, thanks for replying, but in my case <code>idx_x</code> and <code>idx_y</code> have to be <code>2x4</code> instead of just <code>4</code>, meaning each example in the batch has its own indices. Is it still possible with advanced indexing? Or do we have to use a for-loop to loop through the examples in a batch?</p>
Is there a way to parallelize meta-update for MAML?

username: hzw
cooked: <p>Hi, I’m trying to make my MAML code more efficient by parallelizing the meta-update. Suppose I have a meta batch size of b, then I have to use a for-loop to iterate through all b train-test pairs for one meta-update. However, in Tensorflow, you could use tf.map_fn to parallelize the for loop and save a lot of time. I’m wondering how I could do this in PyTorch. Thanks!</p>
About Cuda data allocate problems

username: Steve_Hu
cooked: <ol>
<li>how to check the model on which specific cuda device, not just is_cuda, is there a solution?</li>
<li>if the input on cuda, but the model hasn’t been explicitly declared on cuda, in this time, i execute mode(input),what happens? if model and input are not on the same device ,it’ll be wrong?</li>
<li>about the details of DataParallel, what is the difference between the host cuda device data and other devices data, the model and input allocate and sth , what is the inner mechanics of DataParallel, it seems host device uses more memory.</li>
</ol>
<p>totally,i feel a little confused on the details of data allocate…someone can help me ? thx!!</p>

username: cdancette
cooked: <p>Hi,</p>
<ol>
<li>
<p>You can use tensor.device. It returns you the current device of the tensor, and the index of the GPU if the device is cuda.</p>
</li>
<li>
<p>If model and input are not in the same device, it will crash. You cannot do operations between tensors in cpu and tensors in cuda.</p>
</li>
</ol>
<p>Exemple:</p>
<pre><code class="lang-auto">In [12]: torch.rand(10, 10).cuda() + torch.rand(10, 10)                                                                                                                                                            
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-12-4280bdc898ed&gt; in &lt;module&gt;
----&gt; 1 torch.rand(10, 10).cuda() + torch.rand(10, 10)
RuntimeError: expected device cuda:0 and dtype Float but got device cpu and dtype Float
</code></pre>
<ol start="3">
<li>In DataParallel, the batch will be split among the gpus. At the end, the gradients will be sent back to the host and summed, and the step happens here.<br>
So every device is hosting the whole model, plus a portion of the batch.</li>
</ol>

username: Steve_Hu
cooked: <p>thanks , i understand it now,<img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=9" title=":smile:" class="emoji" alt=":smile:"></p>
How does broadcasting 3-D tensors work?

username: Preyas
cooked: <p>Let’s begin with a simple 2-D tensor:<br>
t1=torch.tensor([1,2,3,5],dtype=torch.int16).<br>
t1=t1.reshape((2,2))</p>
<p>Next I create two 3-D tensors by re-shaping t1, and then try to perform an element-wise operation on them.<br>
t11=t1.unsqueeze(1) # (shape: (2,1,2)<br>
t12=t1.unsqueeze(2) # (shape: (2,2,1)</p>
<p>final= t12 + t11</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/b/6b1468d76e85be675cbc736701f63967b322a21c.png" alt="Screenshot%20from%202019-11-06%2021-04-58" data-base62-sha1="fhgOCTt48Hw2pMguKmjRnR0MKCg" width="386" height="95"></p>
<p>I expected the final output to be of shape (2,2,2), but I don’t understand how did pytorch broadcast t12 and t11 to perform the element-wise addition operation. Could someone please explain how does broadcasting happen with 3D tensors, and how did it result in the final tensor shown above?</p>
<p>P.S.: I am aware how broadcasting works with 2-D tensors, I just cant seem to visualise it happening with 3-D tensors.</p>
<p>Thanks!</p>

username: albanD
cooked: <p>What happens when you have the same number of dimension and in one dimension, the size don’t match but one is 1, you expand to the new size.<br>
In you case, you have <code>final = t12.expand(2,2,2) + t11.expand(2,2,2)</code>.<br>
This basically repeats the existing Tensor along the dimension that was of size 1.</p>