akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Error in Logistic regression

Data_Research: <p>I am trying to implement basic logistic regression using pytorch. For loss function I am using nn.BCEWithLogitLoss().<br>
I am getting an error of  “result type Double can’t be cast to the desired output type Long” I tried many ways, but I am unable to overcome this.</p>
<pre><code>class LogisticRegression(nn.Module):
  def __init__(self,input_size,num_classes):
    super(LogisticRegression, self).__init__()
    self.linear=nn.Linear(input_size,num_classes)

  def forward(self, x):
    out=self.linear(x)
    return out
model=LogisticRegression(input_size, num_classes)

criterion =nn.BCEWithLogitsLoss() 
optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)  

# Training the Model
for epoch in range(num_epochs):
    for i, (dataX, labels) in enumerate(trainloader1):
        dataX = Variable(dataX.view(-1, 34))
        labels = Variable(labels)

        dataX=torch.tensor(dataX, dtype=torch.float64)
        labels=torch.tensor(labels, dtype=torch.long)

        print(labels.squeeze(1).size())

    
        # Forward + Backward + Optimize
        optimizer.zero_grad()
        outputs = model(dataX)
        #loss = criterion(outputs, labels.squeeze(1))
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()
        
        if (i+1) % 100 == 0:
           print ('Epoch: [%d/%d], Step: [%d/%d], Loss: %.4f' 
                   % (epoch+1, num_epochs, i+1, len(X_1)//batch_size, loss.data.item())) # a//b 
means integer(a/b)
        

# Test the Model
correct = 0
total = 0
for dataX2, labels in validloader1:
    dataX2 = Variable(dataX2.view(-1, 34))
    outputs = model(dataX2)
    _, predicted = torch.max(outputs.data, 1)
    total += labels.size(0)
    correct += (predicted == labels).sum()
    
print('Accuracy of the model on the 10000 test images: %d %%' % (100 * correct / total))
</code></pre>
<p>Any Help is highly appreciated</p>

Eta_C: <p>Note the example in <a href="https://pytorch.org/docs/stable/nn.html?highlight=bcewithlogitsloss#torch.nn.BCEWithLogitsLoss" rel="noopener nofollow ugc">documentation</a>.</p>
<blockquote>
<p>Examples:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; target = torch.ones([10, 64], dtype=torch.float32)  # 64 classes, batch size = 10
&gt;&gt;&gt; output = torch.full([10, 64], 0.999)  # A prediction (logit)
&gt;&gt;&gt; pos_weight = torch.ones([64])  # All weights are equal to 1
&gt;&gt;&gt; criterion = torch.nn.BCEWithLogitsLoss(pos_weight=pos_weight)
&gt;&gt;&gt; criterion(output, target)  # -log(sigmoid(0.999))
tensor(0.3135)
</code></pre>
</blockquote>
<p>Maybe <code>loss = criterion(outputs, labels.float())</code> or <code>loss = criterion(outputs, labels.double())</code>can solve your problem.</p>
When increase batchsize,RuntimeError: cuda runtime error (59) : device-side assert triggered

chunchun: <p>There will be an error when I increase the batchsize.But there is nothing without changing.<br>
Why?</p>

InnovArul: <p>Can you give some more info on this?</p>
<p>Like batch sizes, the GPU memory that you are using, a sample code to reproduce the issue if possible, etc.,</p>

chunchun: <p>I have solved this problem.<br>
I found when memory wasn’t enough,it would occur this problem</p>
'train_one_epoch gives error' while using COCO annotations

madiltalay: <p>I am using the COCO dataset for training with annotations available at the COCO website.<br>
While using the code from: <a href="https://github.com/pytorch/vision/issues/url" rel="noopener nofollow ugc">https://github.com/pytorch/vision/blob/master/references/detection/engine.py</a>, I get an error:</p>
<blockquote>
<p>AttributeError: ‘list’ object has no attribute ‘items’</p>
</blockquote>
<p>for the code snippet:<br>
<code>targets = [{k: v.to(device) for k, v in t.items()} for t in targets]</code></p>
<p>Further digging into the issue, I find that the ‘targets’ I receive from the ‘for loop’:<br>
<code>for images, targets in metric_logger.log_every(data_loader, print_freq, header):</code></p>
<p>are in tuple format, with length equal to the batch_size.<br>
Moreover, each item in this tuple is a list, and each list consists of seven dictionaries containing the annotation information.<br>
When I apply this code to an individual object, it works fine:</p>
<pre><code class="lang-auto">target = targets[0]
obj_1 = target[0]
dict_1 = [{k: v for k, v in obj_1.items()}]
</code></pre>
<p>So I suppose the code might be written as follows:<br>
<code>targets = [[{k: v for k, v in obj.items()} for obj in target] for target in targets]</code></p>
<p>Can anyone please confirm this and provide help in this regard?</p>
How to do a complex index selection operation in a simple way

yajha: <p>Here I have a N*4 matrix  F with data from 0 to 1,I want to get a (N,) index tensor which requires<br>
for every row in F , return the first value’s index which value &gt;=R<br>
if there is no such value in this row , return the max value’s index<br>
Here’s an example(R=0.7)<br>
[<br>
[0.5 0.8 0.9 0.3],<br>
[0.5 0.8 0.7 0.3],<br>
[0.6 0.4 0.3 0.4],<br>
]<br>
will get<br>
[1,1,0]</p>
<p>Thank you very much!</p>

LeviViana: <p>The following snippet solves your problem:</p>
<pre><code class="lang-python">import torch

a = torch.tensor(
[
    [0.5, 0.8, 0.9, 0.3],
    [0.5, 0.8, 0.7, 0.3],
    [0.6, 0.4, 0.3, 0.4],
])
r = 0.7

torch.min(a.max(1)[1], ((a &gt; r).cumsum(1) &lt; 1).sum(1))
</code></pre>
<p>Note that (at least for torch 1.1) <code>torch.zeros(n).argmax() == n - 1</code> and <code>torch.zeros(n).cuda().argmax() == 0</code>… which are logically both valid, but aren’t consistent. If your operations only use cpu tensors, you can do it a little simpler with the following:</p>
<pre><code class="lang-python">torch.min(a.max(1)[1], (a &gt; r).cumsum(1).argmin(1) + 1)
</code></pre>

yajha: <p>Thanks for your time.</p>
<p>My pytorch is 1.2.0 and I find argmax always returns the last index in the above case(in CPU), so I didnot try to think about the consistency. I will follow your answer.</p>
<p>Again, thanks for your reply.</p>
Batch Scalar multiplication

Bhavya_Bhatt: <p>I have a matrix A with shape (N, 1) and a matrix B with shape (2, 2). I want that each entry in the A matrix (column vector) is multiplied with the B matrix (each component will be a value so scalar multiplication of that value with the B matrix) to get a matrix with shape (N, 2, 2) where each matrix along the first dimension will be the resultant scalar multiplied matrix.</p>
<p>I am really confused how should the broadcasting should work to get the desired result.</p>
<p>Thanks in advance !!</p>

LeviViana: <p>This way:</p>
<pre><code class="lang-python">import torch

n = 3
a = torch.arange(n).unsqueeze(1) # [n, 1]
b = torch.ones(4).view(2, 2) # [2, 2]

a[:, :, None] * b # [n, 2, 2]
</code></pre>

Bhavya_Bhatt: <p>Thanks a lot <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"> it worked like a charm !!</p>
How to know a pretrained model's input_shape

Lychee: <p>hi,dear,<br>
I’m newer to the Pytorch,<br>
I do not know how to get the input_shape of a pretrained model,<br>
Could you help me ?<br>
any advice or suggestion will be appreciated.</p>
<p>thx</p>

ptrblck: <p>I assume you are using some pretrained models from <code>torchvision</code>?<br>
If so, the usual input shape is <code>[batch_size, 3, 224, 224]</code> with the exception of Inception, which expects <code>[batch_size, 3, 299, 299]</code>.<br>
Since the models use an adaptive pooling layer before flattening the output of the last conv or pooling layer, the spatial size of your input images is more flexible.<br>
However, if you are using a pretrained model, note that the kernels were trained using the aforementioned shapes, so using completely different spatial sizes might result in bad performance.</p>
<p>Let us know, if you have any more questions. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Should backward() function be in the loop of epoch or batch?

whj123: <p>When training nn models using Pytorch, is there a difference regarding where we place the backward method? For example, which one of below is correct?</p>
<p>Calculate gradient across the batch:</p>
<pre><code class="lang-auto">for e in range(epochs):
    for i in batches_list:
        out = nn_model(i)
        loss = loss_function(out, actual)
        loss_sum += loss.item()
        lstm.zero_grad()
        loss.backward()
        optimizer.step()
loss_list.append(loss_sum / num_train_obs)
</code></pre>
<p>Calculate gradient across the epoch:</p>
<pre><code class="lang-auto">for e in range(epochs):
    for i in batches_list:
        out = nn_model(i)
        loss = loss_function(out, actual)
        loss_sum += loss.item()
    lstm.zero_grad()
    loss_sum.backward()
    optimizer.step()     
loss_list.append(loss_sum / num_train_obs)
</code></pre>

crowsonkb: <p>The former is correct. The latter will compute the gradient for the last minibatch only.</p>

whj123: <p>Thanks <a class="mention" href="/u/crowsonkb">@crowsonkb</a> . I think I had a typo in the second code. The backward() should be based on loss_sum, not loss. In this case, would your opinion still hold?</p>

crowsonkb: <p>To accumulate gradients across an entire epoch, try something along the lines of the following code:</p>
<pre><code class="lang-python">for e in range(epochs):
    optimizer.zero_grad()
    for i in batches_list:
        out = nn_model(i)
        loss = loss_function(out, actual)
        loss.backward()
        loss_sum += loss.item()
    optimizer.step()     
    loss_list.append(loss_sum / num_train_obs)
</code></pre>

whj123: <p>I guess your pseudocode and my second code are both correct in terms of accumulating gradients across the entire epoch? <a class="mention" href="/u/crowsonkb">@crowsonkb</a></p>

crowsonkb: <p>I think your second code will need <code>loss_sum += loss</code> instead of <code>loss.item()</code> to work at all? I think mine may be more efficient, allowing the intermediate computational graph for each batch to be freed at the end of each batch, by the <code>loss.backward()</code> call. Though someone with more experience with autograd might want to chime in here.</p>

ptrblck: <p>Your explanation is correct and <a class="mention" href="/u/alband">@albanD</a> posted some examples in <a href="https://discuss.pytorch.org/t/why-do-we-need-to-set-the-gradients-manually-to-zero-in-pytorch/4903/20">this post</a> a while ago.</p>
Zero weight in cross entropy is same as ignore index

hadaev8: <p>I need alluxary classification in very imbalanced data.<br>
And dont need major class.<br>
Before i had linear layer with 9 neurons and ignore index - 1.<br>
But now i think it is better to have 10 classes (this values still used in another part of model).<br>
Is it same to set weight zero for major class as i used to ignore it with ignore index?</p>

ptrblck: <p><code>ignore_index</code> will ignore valid as well as invalid target indices.<br>
E.g. if you are dealing with 10 classes, your model output would have a dimension of <code>[batch_size, 10]</code> in the vanilla multi-class classification setup, while the target will be a <code>LongTensor</code> of shape <code>[batch_size]</code> containing class indices in the range <code>[0, 9]</code>.<br>
If your target contains any other values outside this range, you’ll get a <code>RuntimeError</code>:</p>
<pre><code class="lang-python">RuntimeError: Expected cur_target &gt;= 0 &amp;&amp; cur_target &lt; n_classes to be true, but got false.  (Could this error message be improved?  If so, please report an enhancement request to PyTorch.)
</code></pre>
<p>Now, if you specify a special class index in <code>ignore_index</code>, your target might also contain this index without throwing an error.</p>
<p>On the other hand, setting a weight of zero to a specific valid class, will create zero gradients for this class logit.<br>
Setting <code>ignore_index</code> to the same valid class will yield the same behavior.</p>
<p>While the use cases might be different, if you use a valid class index for both approaches, you should see the same result.</p>
NaN values after transformer layer

isaacmg: <p>Hi this is a follow-up to my other question I now have the architecture but I am getting NaN values after the first gradient update and after the transformer layer.</p>
<pre><code class="lang-auto">class SimpleTransformer(torch.nn.Module):
    def __init__(self, n_time_series, d_model=128):
        super().__init__()
        self.dense_shape = torch.nn.Linear(n_time_series, d_model)
        self.pe = SimplePositionalEncoding(d_model)
        self.transformer = Transformer(d_model, nhead=8)
        self.final_layer = torch.nn.Linear(d_model, 1)
    def forward(self, x, t, tgt_mask):
        x = self.dense_shape(x)
        x = self.pe(x)
        t = self.dense_shape(t)
        t = self.pe(t)
        x = x.permute(1,0,2)
        t = t.permute(1,0,2)
        x = self.transformer(x, t, src_mask=tgt_mask, tgt_mask=tgt_mask)
        print(torch.isnan(x)) # This returns true after the first gradient update
        x = self.final_layer(x)
        return x
        
class SimplePositionalEncoding(torch.nn.Module):
    def __init__(self, d_model, dropout=0.1, max_len=5000):
        super(SimplePositionalEncoding, self).__init__()
        self.dropout = torch.nn.Dropout(p=dropout)

        pe = torch.zeros(max_len, d_model)
        position = torch.arange(0, max_len, dtype=torch.float).unsqueeze(1)
        div_term = torch.exp(torch.arange(0, d_model, 2).float() * (-math.log(10000.0) / d_model))
        pe[:, 0::2] = torch.sin(position * div_term)
        pe[:, 1::2] = torch.cos(position * div_term)
        pe = pe.unsqueeze(0).transpose(0, 1)
        self.register_buffer('pe', pe)

    def forward(self, x):
        x = x + self.pe[:x.size(0), :]
        return self.dropout(x)
</code></pre>
<pre><code class="lang-auto">def generate_square_subsequent_mask(sz):
        r"""Generate a square mask for the sequence. The masked positions are filled with float('-inf').
            Unmasked positions are filled with float(0.0).
        """
        mask = (torch.triu(torch.ones(sz, sz)) == 1).transpose(0, 1)
        mask = mask.float().masked_fill(mask == 0, float('-inf')).masked_fill(mask == 1, float(0.0))
        return mask
</code></pre>
<p>Training loop</p>
<pre><code class="lang-auto">for epoch in range(max_epochs):
    running_loss = 0.0
    for src, trg in data_loader:
        mask = generate_square_subsequent_mask(10)
        optimizer.zero_grad()
        output = a(src.float(), mask)
        #output = s(src.float(), trg.float(), mask)
        labels = trg[:, :, 0]
        loss = criterion(output.view(-1, 10), labels.float())
        loss.backward()
        #torch.nn.utils.clip_grad_norm_(s.parameters(), 0.5)
        optimizer.step()
        running_loss += loss.item()
        i+=1
</code></pre>
<p>Can anyone help?<br>
Thanks</p>

akashs: <p>I am also facing a similar issue, did you able to solve the problem?</p>

isaacmg: <p>Yeah I was able to solve it by applying StandardScaler on my the data and filling all NaN values. Large values even more so than with traditional RNNs seem to cause loss to explode.</p>
DDP for model parallelism

uertenli: <p>Hi,</p>
<p>I am trying to implement the main idea in <a href="https://arxiv.org/abs/1806.03863" rel="noopener nofollow ugc">Massively Parallel Video Networks</a>. In the paper, the authors implement model parallelism for training video networks by giving the outputs of each layer to the next layer as usual but to the next time step. This way, they can process layers independently on different GPUs. The following figure shows the most basic case:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/b/0b827525be2bf106b4aee341c9c8681dcfbfd1a3.png" alt="Screenshot%20from%202019-12-10%2020-45-48" data-base62-sha1="1DOKTfFaMJu2vyFHqtBDzhhQbsv" width="142" height="203"></p>
<p>This photo shows the network unrolled over time. At every time instant, for this case, we have 4 different layers and each of these 4 layers can be processed independently on separate GPUs.</p>
<p>To do this I want to use hopefully, something simple. However, I am having trouble understanding if I can use DDP for this. Essentially, I want to divide my model into independent blocks and pass the gradient in the direction of the arrows. Can I use DDP with model parallel as given <a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html#combine-ddp-with-model-parallelism" rel="noopener nofollow ugc">here</a>. If so, how to do this for a big model where I can choose how I define my sub-blocks.</p>
<p>As a further question: In the paper, they also implement the same idea on a CPU. Is there some function in PyTorch which can achieve this task.</p>
<p>Note: We cannot use <code>nn.DataParallel()</code> because the training setting is an online training and we want to process frames one by one as they come.</p>
<p>Thanks in advance.</p>

pritamdamania87: <p>IIUC, what we’re looking for here is pipeline parallelism. PyTorch currently doesn’t have native support for pipeline parallelism. There are a few projects that have built something similar on top of PyTorch: <a href="https://github.com/kakaobrain/torchgpipe" rel="nofollow noopener">https://github.com/kakaobrain/torchgpipe</a> and <a href="https://github.com/msr-fiddle/pipedream" rel="nofollow noopener">https://github.com/msr-fiddle/pipedream</a>. You could also use the <a href="https://pytorch.org/docs/master/rpc.html" rel="nofollow noopener">Distributed RPC Framework</a> to build something like this.</p>
How to set the glove path in an object torch.load ()?

Antoine_Compagnie: <p>I tried to load an object from a file with a torch to create an embedded word:</p>
<pre><code>infersent = torch.load('InferSent/encoder/infersent1.pkl', map_location=lambda storage, loc: storage)
</code></pre>
<p>And then I tried to establish the Glove path as it does [Alvira Swalin in her article] <a href="https://towardsdatascience.com/building-a-question-answering-system-part-1-9388aadff507" rel="nofollow noopener">1</a>.</p>
<pre><code>infersent.set_glove_path("InferSent/dataset/GloVe/glove.840B.300d.txt")
</code></pre>
<p>But it gives me:</p>
<pre><code>---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
&lt;ipython-input-14-1a6368709557&gt; in &lt;module&gt;()
      1 infersent = torch.load('InferSent/encoder/infersent1.pkl', map_location=lambda storage, loc: storage)
----&gt; 2 infersent.set_glove_path("InferSent/dataset/GloVe/glove.840B.300d.txt")
      3 

AttributeError: 'collections.OrderedDict' object has no attribute 'set_glove_path'
</code></pre>

albanD: <p>Hi,</p>
<p><code>torch.load()</code> just returns the object that was saved. I am not sure in this case what was the original object that was saved. But the file you read contains an <code>OrderedDict</code> from the <code>collections</code> package. This object does not have a method <code>set_glove_path</code>. I think you are not loading the proper <code>.pkl</code> file.</p>

Antoine_Compagnie: <p><a class="mention" href="/u/alband">@albanD</a> Thank for your answer ! Yes it was <code>infersent.allnli.pickle</code> before, as you can see on <a href="https://towardsdatascience.com/building-a-question-answering-system-part-1-9388aadff507" rel="nofollow noopener">Alvira Swalin’s tutorial</a>, but it is no more a pkl filed provided by Infersent, the sentence embeddings method that provides semantic representations for English sentences. They changed it for infersent1.pkl and infersent2.pkl.<br>
Therefore … I don’t know what to do, I just wanted to do and update Alvira Swalin’s tutorial.</p>

albanD: <p>I am not familiar with this repo so I don’t know how to help. Sorry.</p>

Ajinkya_Aj: <p>did you find solution? Im having the same issue</p>

Faeze_Zps: <p>Hi.<br>
I had this problem too. I tried to read the README.md from <a href="https://github.com/facebookresearch/InferSent" rel="nofollow noopener">https://github.com/facebookresearch/InferSent</a> carefully.<br>
I found out that some changes have been made in the previous file. Therefore, you have to download all the files. Then, you should rewrite your code with respect to the explanation on the GitHub page.<br>
As they have explained there, this the right way that you should set the path:<br>
W2V_PATH = ‘fastText/crawl-300d-2M.vec’<br>
infersent.set_w2v_path(W2V_PATH)</p>

amir_mohammadi: <p>Tnx Faeze; it was useful!</p>
How does DistributedDataParallel handle ignore classes when averaging gradients?

marcman411: <p>How does gradient averaging work in DistributedDataParallel training? I am particularly interested in what happens when the batches have masked or ignored data, e.g. with semantic segmentation.</p>
<p>For example: let’s say I have 4 GPUs and I am training a semantic segmentation network with a dataset with an ignore class. As I understand it, in the DataParallel setting, the outputs are aggregated on GPU0, the loss computed, and then the gradient is backpropagated back through each GPU’s model. In the DistributedDataParallel case,  L0, L1, L2, L3 are each computed for each GPU’s share of the batch, the losses are backpropagated back through their respective GPU’s model, and the gradients along the way are averaged.</p>
<p>Using DataParallel, the presence of an ignore class makes no difference. Even if one GPU’s mini-batch has a lopsided amount of ignore pixels, the loss is computed as the weighted average. However, what happens when you have a lopsided distribution of ignore pixels on one GPU using DistributedDataParallel? There does not seem to be any mechanism for weighting the average of the gradients. Yet in this case, L0, L1, L2, and L3 ought to have their contributions weighted by the ratio of valid pixels when averaging gradients during backpropagation.</p>
<p>Is there some way to handle this ignore class imbalance during distributed training?</p>

mrshenli: <blockquote>
<p>How does gradient averaging work in DistributedDataParallel training?</p>
</blockquote>
<ol start="0">
<li>Every DDP instance will have its own copy of the local model, and DDP will setup post autograd hooks on every parameter (i.e., DDP hooks).</li>
<li>In every forward pass, DDP feeds the input data to its own local model, and returns the local output to the application.</li>
<li>The application uses the local output to compute the local loss, and calls backward on the local loss, which kicks off the local autograd engine to compute gradient for all parameters. When one local gradient becomes ready, that will trigger the corresponding DDP hook. The DDP hook will run allreduce on the given gradients, and write the averaged grads back to the parameter.grad field.</li>
<li>When backward is done, parameter.grad should all be globally averaged gradients. Optimizer can then consume that grad to update parameters.</li>
</ol>
<blockquote>
<p>Is there some way to handle this ignore class imbalance during distributed training?</p>
</blockquote>
<p>DDP simply averages (sum and then divide by the number of DDP world size) all local gradients. So, it should work as long as the ignored data do not contribute to the local gradients.</p>
Ignite TypeError: 'Events' object is not callable

Upgrade_Yourself: <p>I get this error when I try to execute the example <a href="https://github.com/pytorch/ignite/blob/master/examples/mnist/mnist_with_tensorboardx.py" rel="nofollow noopener">https://github.com/pytorch/ignite/blob/master/examples/mnist/mnist_with_tensorboardx.py</a></p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/content/mnist_with_tensorboardx.py", line 130, in &lt;module&gt;
    args.log_interval, args.log_dir)
  File "/content/mnist_with_tensorboardx.py", line 76, in run
    @trainer.on(Events.ITERATION_COMPLETED(every=log_interval))
TypeError: 'Events' object is not callable
</code></pre>
<p>I’m working on google colab so I had to install some packages and I also used  TensorBoard notebook extension:</p>
<pre><code class="lang-auto">!pip install --pre pytorch-ignite
!pip install 'grpcio==1.24.3'
!pip install -q tf-nightly-2.0-preview
%load_ext tensorboard 
!python /content/mnist_with_tensorboardx.py --log_dir=/tmp/tensorboard_logs
%tensorboard --logdir {/tmp/tensorboard_logs}
</code></pre>
<p>According to this issue <a href="https://github.com/pytorch/ignite/issues/687" rel="nofollow noopener">https://github.com/pytorch/ignite/issues/687</a>, the problem is solved when the latest nightly version is used however for me it’s not the case.</p>

vfdev-5: <p><a class="mention" href="/u/upgrade_yourself">@Upgrade_Yourself</a> I replied on this issue here too : <a href="https://github.com/pytorch/ignite/issues/687" rel="nofollow noopener">https://github.com/pytorch/ignite/issues/687</a><br>
Please, tell me if it works now for you.</p>

Upgrade_Yourself: <p>I haven’t yet looked at the code so I don’t know what you changed exactly but yes it works. That was a quick answer <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

vfdev-5: <p>Actually, there is nothing changed in the code, except a typo introduced while previously refactored some stuff… You can see the change here too : <a href="https://github.com/pytorch/ignite/commit/8f9ca9059e3e5341527f4f6d05923508b20812f0" rel="nofollow noopener">https://github.com/pytorch/ignite/commit/8f9ca9059e3e5341527f4f6d05923508b20812f0</a></p>
<p>Glad, that it works for you now.</p>
1D Convolution using 2D Convolution

antimora: <p>Here is a problem I am currently facing. I have a model that uses 1D convolution that I would like to export to ONNX and use pytorch/glow or tvm to compile. But both projects currently do not support 1D convolution (see <a href="https://github.com/pytorch/glow/issues/3246" rel="nofollow noopener">pytorch/glow/issues/3246</a> and <a href="https://github.com/apache/incubator-tvm/issues/4587" rel="nofollow noopener">tvm/issues/4587</a>. A workaround that has been suggested (<a href="https://github.com/apache/incubator-tvm/commit/aa4241390e870880392bf388883377e3367f03a8#diff-fe67a468a757f6154fb40a510fa3e521R67-R83" rel="nofollow noopener">see an example</a>) is to use 2D Convolution with height 1 for 1D convolution</p>
<p>Has anyone done this before using PyTorch? The <a href="https://pytorch.org/docs/stable/nn.html#conv2d" rel="nofollow noopener">PyTorch documentation for Conv2d</a> suggests it would be possible to have “non-square kernels and unequal stride and with padding and dilation”. It seems I can have a wrapper class for Conv1d that uses Conv2d underneath. Is there any downside to it? Does math checks out? Also it looks like I have to do the same for BatchNorm1d  and pooling.</p>

JuanFMontesinos: <p>Yes ofc. Think of a convolution as an sliding window. You can slide the kernel through a 1d signal right? You can think of a 2d convolution as an spatial kernel right? If one of the dimensions is 1 then it’s the same than a 1d convolution.<br>
In fact you can do a sanity check which is applying a 1d convolution and use its weights to emulate the same result with a 2d convolution.</p>
Apply class weights to a custom loss function

geekyrakshit: <p>I am trying to train Enet(<a href="https://arxiv.org/abs/1606.02147" rel="nofollow noopener">https://arxiv.org/abs/1606.02147</a>) on the CamVid dataset using the <a href="https://github.com/bermanmaxim/LovaszSoftmax/blob/master/pytorch/lovasz_losses.py#L153" rel="nofollow noopener">Lovasz Softmax Loss</a>. Originally Enet uses the following class weight strategy:</p>
<pre><code class="lang-auto">def get_class_weights(loader, num_classes, c=1.02):
    _, y= next(iter(loader))
    y_flat = y.flatten()
    each_class = np.bincount(y_flat, minlength=num_classes)
    p_class = each_class / len(y_flat)
    return 1 / (np.log(c + p_class))
</code></pre>
<p>I am applying this class strategy using:</p>
<pre><code class="lang-auto">class_weights = get_class_weights(train_loader, 12)
criterion = CrossEntropyLoss(
    weight=torch.FloatTensor(class_weights).to(device)
)
</code></pre>
<p>How do I apply class weights to a custom loss function such as Lovasz Softmax which doesn’t accept the parameter <code>weight</code>?</p>
TypeError: Cannot handle this data type when changing the Dataset's data

Sangwon_Jake: <p>trainloader = utilsxai.load_data_cifar10(batch_size=1,test=False)<br>
this is my dataset from torchvision</p>
<p><code>trainloader.dataset.data.shape = (50000, 32, 32, 3)</code></p>
<p>I want to replace this with</p>
<p><code>sal_maps_hf.shape = (50000, 3, 224, 224)</code></p>
<p>So I try to replace it like this</p>
<pre><code>trainloader.dataset.data = sal_maps_hf
</code></pre>
<p>but got error message following</p>
<pre><code>for idx,(img,target) in enumerate(trainloader):
   print(img)
</code></pre>
<blockquote>
<p>----&gt; 1 for idx,(img,target) in enumerate(trainloader):</p>
<p>TypeError: Cannot handle this data type</p>
<p>TypeError                                 Traceback (most recent call<br>
last)  in <br>
----&gt; 1 for idx,(img,target) in enumerate(trainloader):<br>
2     img = img.squeeze()<br>
3     <span class="hashtag-raw">#pritn</span>(img)<br>
4     img = torch.tensor(img)<br>
5     print(img.shape)</p>
<p>~/venv/lib/python3.7/site-packages/torch/utils/data/dataloader.py in<br>
<strong>next</strong>(self)<br>
344     def <strong>next</strong>(self):<br>
345         index = self._next_index()  # may raise StopIteration<br>
 → 346         data = self._dataset_fetcher.fetch(index)  # may raise StopIteration<br>
347         if self._pin_memory:<br>
348             data = _utils.pin_memory.pin_memory(data)</p>
<p>~/venv/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py in<br>
fetch(self, possibly_batched_index)<br>
42     def fetch(self, possibly_batched_index):<br>
43         if self.auto_collation:<br>
—&gt; 44             data = [self.dataset[idx] for idx in possibly_batched_index]<br>
45         else:<br>
46             data = self.dataset[possibly_batched_index]</p>
<p>~/venv/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py in<br>
(.0)<br>
42     def fetch(self, possibly_batched_index):<br>
43         if self.auto_collation:<br>
—&gt; 44             data = [self.dataset[idx] for idx in possibly_batched_index]<br>
45         else:<br>
46             data = self.dataset[possibly_batched_index]</p>
<p>~/venv/lib/python3.7/site-packages/torchvision/datasets/cifar.py in<br>
<strong>getitem</strong>(self, index)<br>
120         # doing this so that it is consistent with all other datasets<br>
121         # to return a PIL Image<br>
 → 122         img = Image.fromarray(img)<br>
123<br>
124         if self.transform is not None:</p>
<p>~/venv/lib/python3.7/site-packages/PIL/Image.py in fromarray(obj,<br>
mode)    2645             mode, rawmode = _fromarray_typemap[typekey]<br>
2646         except KeyError:<br>
 → 2647             raise TypeError(“Cannot handle this data type”)    2648     else:    2649         rawmode = mode</p>
<p>TypeError: Cannot handle this data type</p>
</blockquote>
Pytorch > ONNX > TensorRT

siva_raju: <p>I am able to convert pre-trained models(pfe.onnx and rpn.onnx) into tensorrt. But I am not able to convert our models into tensorrt.</p>
<h2>ONNX IR version:  0.0.4<br>
Opset version:    9<br>
Producer name:    pytorch<br>
Producer version: 1.1<br>
Domain:<br>
Model version:    0<br>
Doc string:</h2>
<p>While parsing node number 16 [Squeeze -&gt; “175”]:<br>
ERROR: /home/erisuser/p4sw/sw/gpgpu/MachineLearning/DIT/release/5.0/parsers/onnxOpenSource/builtin_op_importers.cpp:1570 In function importSqueeze:<br>
[8] Assertion failed: axis != BATCH_DIM<br>
ERROR: failed to parse onnx file</p>
<p>Please address this issue. Thanks in advance.</p>

YCAyca: <p>Hi, I see your problem is on another level but I wanted to ask you how you done the onnx -&gt; tensorrt conversion. Because I could convert my sample neural network code written in pytorch to onnx but I can’t go through with the next step. Thanks!</p>

jinfagang: <p>this website have a lot of examples showing on tensorrt examples.</p>
<p>all include training codes of pytorch:</p>
<p><a href="http://manaai.cn" class="onebox" target="_blank" rel="nofollow noopener">http://manaai.cn</a></p>

Joe_Smith: <p>Hi <a class="mention" href="/u/jinfagang">@Jinfagang</a>, i am new with <a href="http://manaai.com" rel="noopener nofollow ugc">manaai.com</a>, and i want to join, would you verify me via wechat security barcode.</p>
<p>Thanks!<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/9/093757b7ccd47b9143ed86b7d5eff22155aa9e6a.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/093757b7ccd47b9143ed86b7d5eff22155aa9e6a" title="Screenshot_20191223-161252_WeChat"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/9/093757b7ccd47b9143ed86b7d5eff22155aa9e6a_2_243x500.jpeg" alt="Screenshot_20191223-161252_WeChat" data-base62-sha1="1jwRPMSirSzVbSKDkYglg60BnqG" width="243" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/9/093757b7ccd47b9143ed86b7d5eff22155aa9e6a_2_243x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/0/9/093757b7ccd47b9143ed86b7d5eff22155aa9e6a_2_364x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/0/9/093757b7ccd47b9143ed86b7d5eff22155aa9e6a_2_486x1000.jpeg 2x" data-dominant-color="ECECEC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot_20191223-161252_WeChat</span><span class="informations">1440×2960 530 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

jinfagang: <p><a class="mention" href="/u/joe_smith">@Joe_Smith</a> Pls add me wechat via: jintianiloveu</p>
<p>(If you can not loggin wechat, pls contact via this forum: <a href="http://t.manaai.cn" rel="nofollow noopener">t.manaai.cn</a></p>
Strange GPU memory consumption in case of multiple models on one GPU

Chen_Li: <p>I’m trying to deploy multiple models on one GPU in a single service. When I add one model on a GPU, the GPU memory consumption is 819M; when I add another model on the same GPU,  the GPU memory consumption increases just 84M, to 903M. Is it a normal phenomenon? Why?</p>
<p>Below is the demo code, I use Pytorch1.0, Ubuntu 18.04.</p>
<pre><code class="lang-auto">import PIL
import time
import torch
import torchvision.models as models
import torchvision.transforms as transforms


class Classifier(object):
    def __init__(self, gpuid):
        self.device = torch.device("cuda:{}".format(gpuid))
        self.model = models.densenet201(pretrained=True)
        self.model.to(self.device)
        self.model.eval()

        ### forward a fake sample to allocate GPU memory
        fake = torch.zeros((1, 3, 224, 224), dtype=torch.float32)
        self.model(fake.to(self.device)).cpu()


c1 = Classifier(0)  # add one classifier, GPU memory: 819M
c2 = Classifier(0)  # add another classifier, GPU memory: 903M

time.sleep(100)
</code></pre>

ptrblck: <p>Most likely the main part of the memory allocation after initializing the first model will be grabbed by the CUDA context for this device.<br>
The second model will thus just use the necessary space for all parameters.</p>
A question on one of the usages of detach()

oran.sh: <p>Hey,</p>
<p>In the <a href="https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html" rel="nofollow noopener">dcgan tutorial</a> from the pytorch website, in this piece of code that basically generates some evaluation every 500 steps:</p>
<pre><code class="lang-auto">        # Check how the generator is doing by saving G's output on fixed_noise
        if (iters % 500 == 0) or ((epoch == num_epochs-1) and (i == len(dataloader)-1)):
            with torch.no_grad():
                fake = netG(fixed_noise).detach().cpu()
            img_list.append(vutils.make_grid(fake, padding=2, normalize=True))

        iters += 1
</code></pre>
<p><strong>Question is: why is the detach on the netG output needed?</strong><br>
From what I understand fake will be ran-over in the next iteration anyway, so the fake created in this piece of code will not be inside the computational graph.</p>
<p>Thanks!<br>
Oran</p>

ptrblck: <p>It’s not really necessary, since the forward pass is also wrapped in a <code>torch.no_grad()</code> block.<br>
Thus the model output shouldn’t even have a <code>grad_fn</code> you could detach.<br>
However, it should be a no-op in this case anyway. At least I cannot think of a use case where this could be necessary.</p>

oran.sh: <p>thanks so much!</p>
<p>oran</p>
Filling torch tensor with zeros after certain index

antimora: <p>I am looking efficient and PyTorch idiomatic approach to zeroing out values beyond certain indices.</p>
<p>I have found <a href="https://stackoverflow.com/questions/57548180/filling-torch-tensor-with-zeros-after-certain-index" rel="nofollow noopener">one solution</a> but it seems more complicated then needs to be.</p>
<p>Here is an example:</p>
<p>Given</p>
<pre><code class="lang-auto">x: tensor([[[0.7418, 0.3182, 0.4222, 0.0584, 0.0477]],

        [[0.4293, 0.3079, 0.2928, 0.8873, 0.9470]],

        [[0.6137, 0.3592, 0.2576, 0.8944, 0.4743]],

        [[0.6279, 0.2723, 0.2599, 0.6904, 0.9212]],

        [[0.2126, 0.2689, 0.2441, 0.8755, 0.4720]]])

x.shape: torch.Size([5, 1, 5])
</code></pre>
<p>and</p>
<pre><code class="lang-auto">z: tensor([3, 3, 1, 4, 1])

z.shape: torch.Size([5])
</code></pre>
<p>Modify x such that values are zeroes beyond indices specified in z</p>
<pre><code class="lang-auto">x^: tensor([[[0.7418, 0.3182, 0.4222, 0.0000, 0.0000]],

        [[0.4293, 0.3079, 0.2928, 0.0000, 0.0000]],

        [[0.6137, 0.0000, 0.0000, 0.0000, 0.0000]],

        [[0.6279, 0.2723, 0.2599, 0.6904, 0.0000]],

        [[0.2126, 0.0000, 0.0000, 0.0000, 0.0000]]])

x^: torch.Size([5, 1, 5])

</code></pre>

LeviViana: <p>The following is the same idea, but with less code:</p>
<pre><code class="lang-python">import torch
x = torch.tensor([[[0.7418, 0.3182, 0.4222, 0.0584, 0.0477]],
        [[0.4293, 0.3079, 0.2928, 0.8873, 0.9470]],
        [[0.6137, 0.3592, 0.2576, 0.8944, 0.4743]],
        [[0.6279, 0.2723, 0.2599, 0.6904, 0.9212]],
        [[0.2126, 0.2689, 0.2441, 0.8755, 0.4720]]])

z = torch.tensor([3, 3, 1, 4, 1])

# one line trick
(torch.arange(x.size(2)) &lt; z[..., None]).unsqueeze(1) * x
</code></pre>
<p>This gives the desired result.</p>

antimora: <p>Thank you, Levi. This checks out and works.</p>
<p>Here is a slight modification to specify device and improve consistency:</p>
<pre><code class="lang-auto">(torch.arange(x.size(2), device=z.device)&lt;z.unsqueeze(1)).unsqueeze(1)*x
</code></pre>

antimora: <p>The solution works beautifully but if you know a way to achieve this without creating a new tensor (arange), it would be even better.</p>
<p>The reason I needed it is because I am trying to use pytorch/glow project but the do not support dynamically creating new tensors (see <a href="https://github.com/pytorch/glow/issues/3932" rel="nofollow noopener">glow/issues/3932</a>)</p>
Bug of register_backward_hook

Zichao_Li: <p>Hi, during training I use Dataparallel to trian a model and I’d like to get the gradient of the input of the first layer of the model. I use register_backward_hook. But when I modified some unrelated codes, sometimes the gradient can be gotten, sometimes not. I am extremely confused.</p>

mailcorahul: <p>Can you post the code where <code>register_backward_hook</code> is used?</p>

MrRobot: <p>Could you provide a minimal working example to demonstrate your issues? It is really difficult to help without enough context.</p>

Zichao_Li: <pre><code class="lang-auto">    def forward_hook(module, input, output):
        layer_inputs.append(input[0])
        layer_outputs.append(output)

    def backward_hook(module, grad_input, grad_output):
        grad_inputs.append(grad_input[0].detach().clone())
        grad_outputs.append(grad_output[0].detach().clone())

    # register_index = 0
    for p in model.modules():
        if isinstance(p, nn.Conv2d):
            p.register_forward_hook(forward_hook)
            p.register_backward_hook(backward_hook)
</code></pre>
<p>I just want to get the grad data of every convolution layer in ResNet101<br>
There should be 208 items in the grad list but there are sometimes 207 or 206 or 205.</p>
Proper way to deal with loss of batch

two_Two: <p>Hi,I would like to know the proper way to write the custom loss function and return the correct value for backward pass.<br>
Here’s simplified code based on this repo:<br>
<a href="https://github.com/kuangliu/pytorch-retinanet" rel="nofollow noopener">pytorch-retinanet</a></p>
<p>custom loss function:</p>
<pre><code class="lang-auto">class Focal_loss(nn.Module):
    def __init__(self,num_classes):
        super().__init__()
        self.num_classes = num_classes
    def binary_focal_loss(self,x,y,stabilization ="None"):
        gamma = 2 
        alpha = 0.25 
        y_true = one_hot_embedding(y.data.cpu(),self.num_classes+1)
        y_true = y_true[:,1:]
        y_true = Variable(y_true).cuda()
        if stabilization=="sigmoid":
            x = x.sigmoid()
        x = torch.clamp(x,1e-4, 1.0 - 1e-4)
        alpha_weight = alpha*y_true+(1.-x)*(1.-alpha)
        focal_weight = y_true*(1.-x)+(1.-y_true)*x
        focal_weight = focal_weight*alpha_weight

        bce_loss = -y_true*torch.log(x)+(1-y_true)*torch.log(1-y_true)
        bce_loss = bce_loss*focal_weight
        return bce_loss.sum()
        
    def forward(self,cls_preds,cls_targets):
        cls_preds: (tensor) predicted class confidences, sized [batch_size, #anchors, #classes].
        cls_targets: (tensor) encoded target labels, sized [batch_size, #anchors].'''
        pos = cls_targets&gt;0
        #for normalize 
        num_pos = pos.data.long().sum()
        
        pos_neg = cls_targets&gt;-1
        mask_cls_preds = cls_preds[pos_neg]
        cls_loss = self.binary_focal_loss(mask_cls_preds,cls_targets[pos_neg])

        total_loss = cls_loss/num_pos
        return total_loss
</code></pre>
<p>Notice that the both <code>cls_preds</code> and <code>cls_targets</code> shape start with <code>[batch_size,....]</code><br>
So,First question:<br>
1.Does this mean that pytorch would automatically pass the batch into network?Should I divide the number of batch after calculate loss?</p>
<p>In this repo,I found that</p>
<pre><code class="lang-auto">trainloader = torch.utils.data.DataLoader(trainset, batch_size=8, shuffle=True, num_workers=8, collate_fn=trainset.collate_fn)
criterion = FocalLoss()
for batch_idx, (inputs, loc_targets, cls_targets) in enumerate(trainloader):
    loss = criterion(cls_preds, cls_targets)
</code></pre>
<p>The second question:<br>
The last line of code <code>loss = criterion(cls_preds, cls_targets)</code>,does this return the loss of single image or the batch of images ?</p>
<p>Thanks in advance!</p>

mailcorahul: <p>The loss returned by <code>criterion</code> call will usually be average over all images in the batch. <code>reduction</code> parameter to the <code>criterion</code> decides whether to average or sum them.</p>

two_Two: <p>Yeah,the tensors passed into loss function would contains batch size and this explains all.for how to deal with it simply depends on what you want!</p>
ANN with arbitrary number of layers and outputs

empty: <p>Hi, I am trying to build a multitask learning neural network similar to the following image.</p>
<p><img src="https://miro.medium.com/max/565/0*ZwCyanzdxYyWXLKK.png" alt width="565" height="424"></p>
<p>For a small number of tasks, I can easily manually code each layer and task. However, for a large number of tasks, e.g., the number of tasks &gt; 100, I don’t know how to scale my code. Is there any guideline on how to properly build a network with an arbitrary number of layers and outputs?</p>
<p>Thanks</p>
What's dim=-1 means

tsungruihon: <p>Notice that in PyTorch version, a tensor and index operation could be written like below</p>
<pre><code class="lang-auto">tensor_test = torch.randn(1, 27)
tensor_test[:, 5:]
</code></pre>
<p>In C++ libtorch version, i noticed that i have to set the <code>dim=-1</code> in slice function so that i could get the right answer. But i don’t understand what <code>dim=-1</code> means.</p>
<pre><code class="lang-auto">tensor_test = torch::randn({1, 27})
tensor_test.slice(-1, 5, tensor_test.sizes()[1])
</code></pre>

ptrblck: <p>I assume it has the same meaning as in the Python code, i.e. “negative indexing”.<br>
<code>-1</code> would thus map to the last dimension, <code>-2</code> to the preceding one, etc.</p>
Fix some parameters in a layer

Mason-Qin: <p>I’m trying to train a network with some strategies and needs to fix some params in one layer during training. For example, a conv layer that is 6 channel 3x3, and only params in channel 1, or even the top 2x2 in channel 1 should be updated. Wandering how to achieve this.</p>

ptrblck: <p>I think the easiest way would be to zero out the gradients of all frozen parameters before calling <code>optrimizer.step()</code>.</p>

numbpy: <p>Is there any way to automate this process. I saw people mentioning backward_hooks but it seems to have a lot of issues. Any way to get it work or a simple workaround?</p>

ptrblck: <p>You could register a hook directly on the parameter, e.g. as:</p>
<pre><code class="lang-python">model = nn.Conv2d(3, 6, 3, 1, 1)
mask = torch.randint(0, 2, (6, 3, 3, 3)).float()
model.weight.register_hook(lambda x: x * mask)

model(torch.randn(1, 3, 4, 4)).mean().backward()
print(model.weight.grad)
</code></pre>
Question about Dataloader and GPU memory usage

Jayson_Ng: <p>for batch_i, (_, imgs, targets) in enumerate(dataloader):</p>
<pre><code>        batches_done = len(dataloader) * epoch + batch_i

        imgs = Variable(imgs.to(device))
        targets = Variable(targets.to(device), requires_grad=False)
</code></pre>
<p>Question 1:<br>
If using num working = 4, batch_size=8 for the dataloader, does it mean that at the same time I will have 32 data assigned to the GPU memory?</p>
<p>Question 2:<br>
What is pin_memory in dataloader?</p>

InnovArul: <ol>
<li>
<p>Batch size indicates the size of data that comes out of dataloader at a time. i.e., in your case, you will have count=8 data assigned to the GPU. num_workers defines the number of threads spawned to read the data.</p>
</li>
<li>
<p>Take a look at this: <a href="https://discuss.pytorch.org/t/when-to-set-pin-memory-to-true/19723/8" class="inline-onebox">When to set pin_memory to true?</a></p>
</li>
</ol>
About Multi-gpu

oasjd7: <p>Hi,all<br>
I want to speed up for training time.<br>
Now I’m using only one GPU.<br>
Here is my nvidia-smi info.</p>
<pre><code class="lang-auto">Memory-Usage: 1143MiB / 12189MiB 
Volatile GPU-Util: 45~60%
</code></pre>
<p>In this case, using multi-gpu can speed up my training time?</p>

InnovArul: <p>Training speed depends on many factors including the input loading time (due to number of threads, RAM, CPU speed etc), Batch size used etc.</p>
<p>With the nvidia-smi info,I could see that the GPU is under-utilized. Atleast memory-wise.<br>
First step would be to increase the batch size and see if the speed is sufficient enough.</p>