akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to avoid confusion between two classes

username: sparshgarg23
cooked: <p>I am using a pretrained HRNet model  from timm for multi -class classification problem.My current dataset’s distribution is as follows</p>
<pre><code class="lang-auto">{'C1': 480, 'C2: 444, 'C3: 348, 'C4: 479, 'C5': 456, 'C6: 372, 'C7: 468, 'C8 396
And I am using weighted sample in the dataloader with weights as follows
[0.0021, 0.0023, 0.0029, 0.0021, 0.0022, 0.0027, 0.0021, 0.0025
</code></pre>
<p>My model classifier is as follows</p>
<pre><code class="lang-auto">(classifier): Sequential(
    (0): Linear(in_features=2048, out_features=625, bias=True)
    (1): ReLU()
    (2): Dropout(p=0.3, inplace=False)
    (3): Linear(in_features=625, out_features=256, bias=True)
    (4): ReLU()
    (5): Linear(in_features=256, out_features=128, bias=True)
    (6): ReLU()
    (7): Linear(in_features=128, out_features=8, bias=True)
</code></pre>
<p>I have noticed that when it comes to class2,class 4, class 6 and 8,the model ends up getting confused between these classes.<br>
Images for class C2 and C4</p>
<pre><code class="lang-auto">C2
![model_44_3|500x500](upload://gHbVdyPDZFgpogwNQbjw40sQTLu.jpeg)

C4
![model_20_18|500x500](upload://uhS4xVOYPSXSmu9UMyR3ruv08l2.jpeg)


</code></pre>
<p>For example for the two images shown below</p>
<pre><code class="lang-auto">C6
![model_24_7|500x500](upload://15qkWoRfFDYsLkVCOlI0E62UJqO.jpeg)

C8
![model_14_8|500x500](upload://uDXEu7Q9FTZJAck2rBGJkJopa4n.jpeg)


When I use the model in inference stage the prediction for class 6 come out to be as follows
C6 score .33 C8 score .67
Prediction result for C8
C8 score .67 C6 score .33
</code></pre>
<p>I have trained the model using cross entropy loss,and I have used weighted sampling to handle class imbalance.</p>
<p>Any ideas on what I can do to ensure that the model is able to discrimante between C6 and C8 sufficently. Note this happens only for these four classes for the rest of the classes the confusion isn’t that significant</p>
How to convert PyTorch tensor to C++ torch::Tensor vice versa?

username: xiaoguoer
cooked: <p>Hi, I am working on a project which need transfer the Pytorch tensor to libtorch and vice versa.<br>
I already knew that it may need following functions :</p>
<pre><code class="lang-auto">PyObject * THPVariable_Wrap(at::Tensor t);
at::Tensor&amp; THPVariable_Unpack(PyObject* obj);
</code></pre>
<p>However, I tried use them like these:</p>
<pre><code class="lang-auto">void receive_tensor_from_py(PyObject* obj){
  at::Tensor &amp;t = THPVariable_Unpack(obj);
  std::cout &lt;&lt; &amp;t &lt;&lt; std::endl;
}
</code></pre>
<p>But it doesn’t work, can anyone gives a small sample code?</p>
Creating Dataloader using custom Dataset

username: thaophan1
cooked: <p>Currently, I have a Dataloader initialized with a dataset of an array of tensors that does not inherit from Dataset. So far my model has not been learning at all. I was wondering if it is necessary to pass in a dataset class that inherits from Dataset for Dataloader to work or not.</p>

username: eqy
cooked: <p>I don’t think it should really matter unless your dataloader is missing some functions (which should throw an error). If the data and label tensors returned by your dataset are reasonable it is likely the problem is somewhere else.</p>

username: jackyjoy123
cooked: <p>thanks for the awesome information.</p>
Allow in-place operations

username: JorJorWell
cooked: <p>Hi,</p>
<p>I was trying to run my code and ran into an error about an in-place operation. I’ve simplified it to the bare minimum so it’s easier to understand. If I’m not wrong, autograd doesn’t like that I modify the tensor y in-place, even if I don’t modify the same elements. Is there a way to allow it? Or is there another way to do what I’m trying to do (use previous slice and parameter u to compute next slice in a differentiable way)?</p>
<p>I hope it’s somewhat clear (it’s my first post).</p>
<p>Thanks</p>
<pre><code class="lang-python">import torch

def dydt(u, y):

    dy = u * y

    return dy

u = torch.ones(1).requires_grad_()

y = torch.zeros(3, 1)

a = dydt(u, y[0])

y[1] = a + y[0]

b = dydt(u, y[1])

y[2] = b + y[1]

y[2].backward()

print(u.grad)
</code></pre>

username: tom
cooked: <p>So to compute the gradient of <code>u</code>, PyTorch needs the intermediate version of y. As the versions (available through the not-official-api-use-at-your-own-risk <code>y._version</code>) are only kept per-tensor and not per-entry, you are seeing that the backward of the product <code>u * y</code> complains about the version of <code>y</code> being different from what it had in the forward pass.</p>
<p>I’d probably work with a list and <code>torch.stack</code> the result at the end if you need the <code>y</code>-tensor. You could also clone y every here or there so that the inplace modifications don’t hit the <code>y</code> tensor used at critical places in the forward.</p>
<p>It is quite operation-specific which inputs and outputs are saved for the backward, but here it will be the multiplication (<code>*</code>), and indeed if you follow the debugging hint to enable anomaly detection, this is what you are pointed at.</p>
<p>Best regards</p>
<p>Thomas</p>

username: JorJorWell
cooked: <p>Thank you for this answer! I’ll go with a list and then stack the elements at the end.</p>
<p>Best,</p>
<p>JorJorWell</p>
How to Integrate Bi-LSTM with 2D-CONV and 2d-Max Pooling

username: Bahaa_Kattan
cooked: <p>I want to apply this paper : <a href="https://arxiv.org/abs/1611.06639v1" rel="noopener nofollow ugc">Text Classification Improved by Integrating Bidirectional LSTM with Two-dimensional Max Pooling</a>;  for multi-class text classification problem (news classification). But <strong>I don’t know how to create the architecture</strong>.</p>
<p>please if you know how to build it I will be so pleasure to see how.<br>
Any help or advice will appreciated.</p>
<p>Here is my code if some one want a good starter:</p>
<pre><code class="lang-auto">from torch import nn
from torch.nn import functional as F
import torch


class Model(nn.Module):
    def __init__(self, vocab_size, embedding_dim, hidden_size, n_filters, n_layers, output_size, p):
        super().__init__()
        self.Embedding = nn.Embedding(vocab_size, embedding_dim)
        self.LSTM = nn.LSTM(embedding_dim, hidden_size, num_layers=n_layers, bidirectional=True, batch_first=True, dropout=p)
        self.Conv2d = nn.Conv2d(hidden_size, n_filters, kernel_size=2)
        self.MaxPool2d = nn.MaxPool2d(2)

    def forward(self, x, features=None):
        Embedding = self.Embedding(x)
        output, (hidden, cell) = self.LSTM(Embedding)
        print(self.Conv2d(hidden))

vocab_size = 5
embedding_dim = 55
hidden_size = 300
n_filters = 25
n_layers = 2
output_size = 2
p = 0.5
clf = Model(vocab_size, embedding_dim, hidden_size, n_filters, n_layers, output_size, p)

a = torch.ones((1, 5), dtype=torch.long)

print('a: ', a)
print('clf: ', clf(a))
</code></pre>

username: Bahaa_Kattan
cooked: <p>I figured it out finally.</p>

username: Vishhvak_Srinivasan
cooked: <p>How did you do it? Could use some help.</p>
Using .pth file to predict

username: narendrago
cooked: <p>Hey, I am new to image processing using PyTorch and have been trying to run a model to deblur images. However, I am not able to test with my own image.</p>
<p><strong>Error:</strong></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/d/cd29222fdcedc90e628c9e55a1af6112bb5fb8a5.png" data-download-href="https://discuss.pytorch.org/uploads/default/cd29222fdcedc90e628c9e55a1af6112bb5fb8a5" title="Screenshot 2022-03-20 at 6.44.15 AM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/d/cd29222fdcedc90e628c9e55a1af6112bb5fb8a5_2_689x500.png" alt="Screenshot 2022-03-20 at 6.44.15 AM" data-base62-sha1="tgW0B68krly2iBUDhvsRmeVtodT" width="689" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/d/cd29222fdcedc90e628c9e55a1af6112bb5fb8a5_2_689x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/d/cd29222fdcedc90e628c9e55a1af6112bb5fb8a5_2_1033x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/c/d/cd29222fdcedc90e628c9e55a1af6112bb5fb8a5_2_1378x1000.png 2x" data-dominant-color="F1F4F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2022-03-20 at 6.44.15 AM</span><span class="informations">1564×1134 245 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><strong>Relevant Code:</strong></p>
<pre><code class="lang-auto">class DeblurCNN(nn.Module):
    def __init__(self):
        super(DeblurCNN, self).__init__()
        self.conv1 = nn.Conv2d(3, 64, kernel_size=9, padding=2)
        self.conv2 = nn.Conv2d(64, 32, kernel_size=1, padding=2)
        self.conv3 = nn.Conv2d(32, 3, kernel_size=5, padding=2)
    def forward(self, x):
        x = F.relu(self.conv1(x))
        x = F.relu(self.conv2(x))
        x = self.conv3(x)
        return x
</code></pre>
<pre><code class="lang-auto">model=DeblurCNN()
model.load_state_dict(torch.load(model_path))
predicted=model.forward(image_path)
</code></pre>
<p>NOTE: I have given the correct model and image paths.</p>

username: ptrblck
cooked: <p>The model expects an input tensor while you are passing an image path as a <code>str</code> to it.<br>
Load the image first, process it, and transform to a tensor before feeding it to the model.<br>
Also, don’t use <code>model.forward</code> explicitly, but call the model directly via <code>output = model(input_tensor)</code>.</p>
How to get random subset of original dataset for each epoch?

username: Samue1
cooked: <p>I would like to know if there is an option to train a model on a different random subset of a larger training set each epoch?</p>
<p>Currently I use <code>Subset()</code> to obtain a smaller version of the original data set.</p>
<pre><code class="lang-auto">    trainset = torchvision.datasets.CIFAR10()
    subset_length = 1000
    trainset = Subset(trainset, range(subset_length))
    trainloader = DataLoader(trainset)
</code></pre>
<p>But this subset is fix for all epochs (the same 1000 images). I would like to have a more dynamic subset, that allows me to train on a fraction of the original data set using a random subset for each epoch.</p>
<p>Do I have to modify the <code>DataLoader()</code>'s <code>__iter__()</code> method to achieve this? Or is there a simpler way?</p>

username: ptrblck
cooked: <p>The simple approach would be to create a new <code>sampler</code> and re-create a new <code>DataLoader</code> with it for each new epoch. Assuming you are lazily loading the data in the <code>Dataset</code> the overhead of re-creating the <code>DataLoader</code> should be small.</p>
How many lines of code does PyTorch consist of?

username: K_D
cooked: <p>How many lines of code has PyTorch in opensource? Approximately. How many manpower (measured in man hours) were spent to make PyTorch? Answer to 2nd question optional. Thanx</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="K_D" data-post="1" data-topic="146937">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/k_d/48/45844_2.png" class="avatar"> K_D:</div>
<blockquote>
<p>How many lines of code has PyTorch in opensource?</p>
</blockquote>
</aside>
<p>I think <code>git ls-files | xargs wc -l</code> should work, which shows <code>220718</code> lines in total.<br>
You could adapt this command to filter specific file types etc. if needed.</p>
CNN + RNN + Fully Connected

username: shwe87
cooked: <p>Hi all,</p>
<p>I have a doubt about hidden dimensions. If I create a neural network like:</p>
<p>Layer 1 --&gt; Convolutional Network<br>
Layer 2 --&gt; RNN (GRU or LSTM)<br>
Layer 3 --&gt; Fully connected linear</p>
<p>How do I handle the hidden outputs used by the RNN because CNN won’t need them…</p>
<p>Thanks.</p>

username: harsha_g
cooked: <p>Sorry, I am not sure if I follow the question clearly. If the order of layers in 1 --&gt; 2 --&gt; 3 which means RNN comes after CNN and then of course CNN won’t need anything from RNN. Instead, if the order is 3 --&gt; 2 --&gt; 1 then you can just choose not to use the hidden outputs from RNN when going into CNN.</p>

username: shwe87
cooked: <p>for example:</p>
<p>output = conv2d<br>
output, h = rnn(x, h)</p>
<p>this h should be proceeded as usual in the training layer and the forward function?</p>

username: harsha_g
cooked: <p>If the <code>h</code> has no role to play in the fully connected linear layer <code>output, _ = rnn(x)</code> should suffice unless you want to initialize <code>h</code> yourself at which point <code>output, _ = rnn(x, h)</code> will do.</p>

username: shwe87
cooked: <p>But then we aren’t taken maximum advantage of the hidden layers of the RNN, that’s why I am asking…</p>

username: harsha_g
cooked: <p>Taking advantage of the hidden outputs is driven by the choice of the architecture.</p>

username: shwe87
cooked: <p>Thanks, then my question was wrong formulated.</p>
<p>If I want to take advantage of the hidden outputs because I am working on sequential data. Is this correct?</p>
<pre><code class="lang-auto">ModelClass.....
.......


   def forward(self, x,  hidden):
         x = Conv2d....
         x, h = rnn(x,h)
         ......
        return x,h
</code></pre>

username: harsha_g
cooked: <p>If you want to use the hidden outputs outside your forward function (for example, if you want to do seq2seq where the hidden outputs from the encoder are used in a decoder) then yes your formulation is fine (I am assuming the <code>hidden</code> passed into the <code>forward</code> function is passed on to the <code>rnn</code> as <code>h</code>).</p>

username: Balazs_Gonczy
cooked: <p>Hi, I am having the same issue. I would like to do a CNN-GRU speaker identification task on preprocessed spectograms. How could you connect the two different NN-s?</p>
Torch.norm with dim=(1,2) gives nan grads

username: kuzand
cooked: <p>I am getting some weird behavior when using <code>torch.norm</code> with <code>dim=(1,2)</code> in my loss computation:</p>
<pre><code class="lang-auto">  m = nn.Linear(3, 9)
  nn.init.constant_(m.weight, 0)
  nn.init.eye_(m.bias.view(3, 3))

  x = torch.rand((2, 3))
  out = m(x).view((2, 3, 3))

  I = torch.eye(3).unsqueeze(0)
  diff = out - I
  loss = torch.mean(torch.norm(diff, dim=(1, 2)))
  loss.backward()

  print(m.bias.grad)
</code></pre>
<p>gives <code>tensor([nan, nan, nan, nan, nan, nan, nan, nan, nan])</code>.</p>
<p>However if instead I use the equivalent<br>
<code>loss = torch.mean(torch.norm(torch.norm(diff, dim=1), dim=1))</code><br>
it gives the correct result:<br>
<code>tensor([0., 0., 0., 0., 0., 0., 0., 0., 0.])</code>.</p>
<p>What could be the reason for this?</p>

username: eqy
cooked: <p>I think this could be because the default norm (Frobenius) has a sqrt that causes issues when the input is near zero e.g., consider the following snippets:</p>
<pre><code class="lang-auto">$ cat example.py
import torch

b = torch.zeros(2,2,2, requires_grad=True)
out = torch.mean(torch.sqrt(b**2))
print(out)
out.backward()
print(b.grad)
</code></pre>
<pre><code class="lang-auto">$ python3 example.py
tensor(0., grad_fn=&lt;MeanBackward0&gt;)
tensor([[[nan, nan],
         [nan, nan]],

        [[nan, nan],
         [nan, nan]]])
</code></pre>
<pre><code class="lang-auto">$ cat example2.py
import torch

b = torch.zeros(2,2,2, requires_grad=True)
out = torch.mean(torch.sqrt(b**2 + 1e-12))
print(out)
out.backward()
print(b.grad)
</code></pre>
<pre><code class="lang-auto">$ python3 example2.py
tensor(1.0000e-06, grad_fn=&lt;MeanBackward0&gt;)
tensor([[[0., 0.],
         [0., 0.]],

        [[0., 0.],
         [0., 0.]]])
</code></pre>
<pre><code class="lang-auto">$ cat example0.py
import torch
from torch import nn
m = nn.Linear(3, 9)
nn.init.constant_(m.weight, 0)
nn.init.eye_(m.bias.view(3, 3))

x = torch.rand((2, 3))
out = m(x).view((2, 3, 3))

I = torch.eye(3).unsqueeze(0)
diff = out - I
loss = torch.mean(torch.norm(diff + 1e-12, dim=(1, 2)))
loss.backward()

print(m.bias.grad)
</code></pre>
<pre><code class="lang-auto">$ python3 example0.py
tensor([0.3333, 0.3333, 0.3333, 0.3333, 0.3333, 0.3333, 0.3333, 0.3333, 0.3333])
</code></pre>

username: kuzand
cooked: <p>Indeed it seems that sqrt is the reason. However it’s strange that using <code>torch.mean(torch.norm(torch.norm(diff, dim=1), dim=1))</code> works fine.</p>
Training with images too big for memory

username: Paul_Chee
cooked: <p>Hi,</p>
<p>I am currently working in a project which requires me to train image classification models on medical images. However, these are hi-res images that are too big to fit into memory.</p>
<p>We could chop up the images but that would create more problems and seem very inefficient.</p>
<p>I would like to ask if there exists a way in pytorch to allow partial loading of images but still essentially has the algorithm run over the entire picture?</p>
<p>Any input is appreciated</p>
<p>Thanks<br>
Paul</p>
This is a question about classification labels.

username: shenao1994
cooked: <p>I am dividing a set of image data into five categories. I want to set the label like this, because these data are related. Can you tell me how to modify the related code?My original label is [0, 1, 2, 3, 4]<br>
<strong>1 1 0 0 0 0<br>
2 1 1 0 0 0<br>
3 1 1 1 0 0<br>
4 1 1 1 1 0<br>
5 1 1 1 1 1</strong></p>
<p>Thanks！</p>

username: Hmrishav_Bandyopadhy
cooked: <p>Hi,<br>
This might be a bit inefficient and might take some time depending on the size of the dataset, but it will get the work done:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; previous_label=torch.tensor([0,1,2,3,4])
&gt;&gt;&gt; x=previous_label.size()[0]
&gt;&gt;&gt; new_label=torch.zeros([x,x])
&gt;&gt;&gt; for i,val in enumerate(previous_label):
...     new_label[i,:val+1]=1
... 
&gt;&gt;&gt; new_label
tensor([[1., 0., 0., 0., 0.],
        [1., 1., 0., 0., 0.],
        [1., 1., 1., 0., 0.],
        [1., 1., 1., 1., 0.],
        [1., 1., 1., 1., 1.]])
</code></pre>
<p>Let me know if this is what you were looking for <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: shenao1994
cooked: <p>Thank you very much!</p>
Allreduce takes 100% gpu utilization if it is getting

username: amsword
cooked: <p>As GPU utilization can also go to 100% if it is stuck at allreduce, how to figure out how much computational resource is taken? If it is close to 100%, it may also be waiting for allreduce or for networking communication.</p>

username: ptrblck
cooked: <p>You could profile your code with the PyTorch profiler or e.g. Nsight Systems to see when NCCL is waiting for a communication and if other computations are executed.</p>
Why the gradient is so strange after weight_norm?

username: Andrew_Chen
cooked: <p>I’m studying how to use weight_norm in pytorch. But, I found a weird problem.</p>
<p>this is my code:</p>
<pre><code class="lang-auto">W = weight_norm(nn.Linear(5,3, bias = False))
x  = torch.ones(5)
d = torch.sum(W(x))
d.backward()
W.weight_v.grad
</code></pre>
<p>After doing that, W.weight_v.grad should all be one, but, the gradient is so strange:</p>
<pre><code class="lang-auto">W.weight_v.grad
tensor([[0.2237, 1.4274, 0.7918, 0.8525, 0.1773],
        [0.4810, 1.4245, 0.7913, 0.4601, 1.0554],
        [0.9531, 0.8751, 0.9691, 1.0501, 1.1175]])
</code></pre>
<p>And another weird thing happened:</p>
<pre><code class="lang-auto">W.weight_g.grad
tensor([[-1.2358],
        [-0.8875],
        [-0.1872]])
</code></pre>
<p>I set bias = False, why W.weight_g have gradient??</p>

username: Andrew_Chen
cooked: <p>I’m so sorry… I should read the document carefully before asking question…</p>
<blockquote>
<p>Weight normalization is a reparameterization that decouples the magnitude of a weight tensor from its direction. This replaces the parameter specified by <code>name</code> (e.g. <code>'weight'</code> ) with two parameters: one specifying the magnitude (e.g. <code>'weight_g'</code> ) and one specifying the direction (e.g. <code>'weight_v'</code> ).</p>
</blockquote>
<p>weight_g is not bias, weight_v is not weight matrix!!!</p>
How to fix the Dropout mask for different batch

username: fixedrl
cooked: <p>Suppose we have 2 minibatch (each with 10 data point). When turning on the dropout for forward pass of first minibatch, the dropout mask with dimension 10 is generated. What if we want to use the same mask for the second batch of data ?</p>

username: smth
cooked: <p>you can do this dropout operation yourself, instead of using <code>nn.Dropout</code>.</p>
<p>You can generate a bernoulli mask of numbers using <code>torch.bernoulli</code> and then multiply your both mini-batches with the same mask.</p>
<p>For example:</p>
<pre><code class="lang-python"># generate a mask of same shape as input1
mask = Variable(torch.bernoulli(input1.data.new(input1.data.size()).fill_(0.5)))

output1 = input1 * mask
output2 = input2 * mask
</code></pre>

username: fixedrl
cooked: <p>Is it correct to rescale the mask to output the same magnitude in following way ?</p>
<pre><code class="lang-auto">mask = Variable(torch.bernoulli(input1.data.new(input1.data.size()).fill_(0.4)))/0.6
</code></pre>

username: tom
cooked: <p>Looks good to me…</p>
<p>Best regards</p>
<p>Thomas</p>

username: rdroste
cooked: <p>For future readers I would like to mention that the rescaling is not correct.<br>
Please note that the Bernoulli distribution samples 0 with the probability (1-p), contrary to dropout implementations, which sample 0 with probability p.</p>
<p>Therefore, if you want dropout with p=0.4, the mask has to be<br>
<code>mask = Bernoulli(torch.full_like(input1, 0.6)).sample()/0.6</code></p>
<p>For dropout with p=0.6 the mask is<br>
<code>mask = Bernoulli(torch.full_like(input1, 0.4)).sample()/0.4</code></p>

username: seyeeet
cooked: <p>what is .sample() ? in your code?</p>

username: Torcione
cooked: <p>Hi,<br>
if I understand correctly, this way I generate a different mask (i.e. dropout) for each element in the batch (since <code>input1.data.size()</code> also contains the size relative to the number of images in the batch); shouldn’t I rather apply the same mask to each element in the batch?</p>
Unet multiclass segmentation, inputs look not linear correlation to lables

username: Bruce_Kieu
cooked: <p>Hi Folks,<br>
I read a paper that they use <em>Unet CNN</em> to build a model using inputs of 870 images size(200x600x1) &amp; 870 label images which is 6-12 layers/classes with the same shape as inputs.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/5/05f1b226c348307da940c2cbc4a9be04678c1f20.png" data-download-href="https://discuss.pytorch.org/uploads/default/05f1b226c348307da940c2cbc4a9be04678c1f20" title="pls see the image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/5/05f1b226c348307da940c2cbc4a9be04678c1f20_2_690x285.png" alt="pls see the image" data-base62-sha1="QAdxFXGaISQQw4v73HBRAZEcBG" width="690" height="285" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/5/05f1b226c348307da940c2cbc4a9be04678c1f20_2_690x285.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/5/05f1b226c348307da940c2cbc4a9be04678c1f20_2_1035x427.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/5/05f1b226c348307da940c2cbc4a9be04678c1f20.png 2x" data-dominant-color="DEDEDE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">pls see the image</span><span class="informations">1187×492 61.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div>.<br>
I have several concerns as below:<br>
Is it a multi-class classification (6-12 classes) process <em>(crossentropyloss function)</em> or regression <em>(mean_square_error)?</em><br>
I just wonder that do they need to build a mask dataset? that is a bridge connect input’s pixels with labels’ pixels. It because I see there is no linear correlation between input &amp; label in term of pixels/color values so far. I still believe that CNN Unet will generate latent features from down-sampling &amp; up-sampling process though kernel &amp; activation functions to come up with the best weight set to predict output, then no need the mask dataset as other semantic/instance segmentation matter?!<br>
I’m new to this field so please correct me if I’m wrong &amp; kindly share your thought to the above question. Thank you very much!</p>
Extract key, value pairs from different documents

username: ngap_wei_Tham
cooked: <p>Assume I have an ocr engine which could detect the bounding boxes of each sentences, characters and recognize them, but the ocr engine do not know how to associate the key value pairs, how could I train a model to associate the key value pairs? Which direction I should search for? What are the recommend solutions today?</p>
<p>Azure has a form recognizer, the ads mentioned you can train your custom model to associate key value pairs, could it deal with forms with different layout? Sometimes keys on top, values on bottom, sometimes keys on left, values on rights, those keys and values could be everywhere. If it could, what are the solutions Azure possible using? I only need to deal with two languages at once, Japanese, English and numbers(0-9).</p>
<p>Thanks</p>
Nvidia A100 slow training speed

username: Rajat_Sharma1
cooked: <p>I moved my model from Quadro RTX 6000 to A100 GPU for the training. But I couldn’t see any speedup in the training process. I am training the model in mixed precision on both machines. I can say training on RTX is marginally faster than A100. What could be the reason as A100 are claimed to be the have lesser training time. What could be the reason for this?</p>

username: eqy
cooked: <p>It likely depends on if your model has enough computation to fully utilize the GPU or if there is a bottleneck somewhere else in the system. If for example the model or the batch size is too small to fully utilize the GPU, then you can see cases where performance really depends on factors like the peak clock rate attained by each setup. What is the utilization reported by <code>nvidia-smi</code>?</p>

username: Rajat_Sharma1
cooked: <p>I am using a batch size of 1 as my input to the model is 128x128x128. The gpu is consuming around 7.5GB out of 40GB.</p>

username: eqy
cooked: <p>Could you check the utilization reported by <code>nvidia-smi</code>? You might want to increase the batch size until you are close to using all available memory or the throughput in examples/sec is no longer improving.</p>

username: tensar
cooked: <p>Hey,</p>
<p>did you get the A100 to speed up?</p>
After using dynamic quantization and static quantization, the accuracy of the model decreases a lot

username: Ritian-Li
cooked: <p>Hi all, I just started learning model quantification.<br>
The model I use is divided into front-end(based on CNN) and back-end(based on GRU), so I used static quantization and dynamic quantization. The ACC will decrease a little if using dynamic quantization which seems like a correct result. But coupled with static quantization, the accuracy of the model will decline significantly. I don’t know whether there is a problem with my usage.</p>
<p>Here is the way I used:</p>
<pre><code class="lang-auto">model = CNN_GRU()
model.eval()
list_mix = [['...', '...']]
# dynamic
model = torch.quantization.quantize_dynamic(
    model, {nn.GRU, nn.Linear}, dtype=torch.qint8)

# static
model.qconfig = torch.quantization.get_default_qconfig('fbgemm')
torch.backends.quantized.engine = 'fbgemm'
model_fp32_fused = torch.quantization.fuse_modules(model, list_mix)
model_fp32_prepared = torch.quantization.prepare(model_fp32_fused)

for _, input in enumerate(loader):
     video = input.get('video')
     border = input.get('duration').float()
     model_fp32_prepared(video, border)
model_int8 = torch.quantization.convert(model_fp32_prepared)

# inference
pred = model_int8 (video, border)
</code></pre>
<p>Besides, for some ops doesn’t support yet like maxpool3d, i used dequant() in the forward().</p>
<pre><code class="lang-auto">class Model():
    def __init__(self):
         ...
         self.pool = nn.MaxPool3d()
         ...
         self.bn = nn.BatchNorm1d()
         ...
        self.quant = QuantStub()
        self.dequant = DeQuantStub()
     def forward(self, x):
         x = self.quant(x)
         ...
         x = self.dequant(x)
         x = self.pool(x)
         x = self.quant(x)
         ...
         x = self.dequant(x)
         x = self.bn(x)
         x = self.quant(x)
         ...
         x = self.dequant(x)
         return x

</code></pre>
<p>After doing this, the code could run, and the size of the model decreased by about 3/4, but the ACC is nearly decreased to 2%.</p>
Ignore zero in weights of model for inference

username: ori
cooked: <p>I have pruned a model and after pruning the model has lot of zeros in layers. After doing inference there is no improvement in speed, So is there any way to ignore zeros preset in layers and achieve after inference time.<br>
Just to give more ideas here are some layers info</p>
<pre><code class="lang-auto">electra.embeddings.word_embeddings.weight | nonzeros = 19355528 / 23887104 ( 81.03%) | total_pruned = 4531576 | shape = (31103, 768)
electra.embeddings.position_embeddings.weight | nonzeros =  177670 /  393216 ( 45.18%) | total_pruned =  215546 | shape = (512, 768)
electra.embeddings.token_type_embeddings.weight | nonzeros =     132 /    1536 (  8.59%) | total_pruned =    1404 | shape = (2, 768)
electra.embeddings.LayerNorm.weight | nonzeros =     768 /     768 (100.00%) | total_pruned =       0 | shape = (768,)

</code></pre>
<p>I’d greatly appreciate any kind of feedback on this.<br>
Thank you all in advance</p>
Sum of same values return different result with PyTorch but not Numpy

username: Flavioh
cooked: <p>Hello!</p>
<p>I really cannot explain myself what is going on here: I have two Tensors of size (1000, 2, 2) and (1000, 2 , 5) which share the same values for the first two slices of the third dimension, i.e. [:,:,:2].</p>
<pre><code class="lang-auto">z0 = torch.rand(1000, 2, 2)
z1 = torch.cat([z0, torch.rand(1000, 2, 3)], dim=2)

s0 = z0[:, :, :2].sum()
s1 = z1[:, :, :2].sum()
</code></pre>
<p>But when I sum over the same slice I do not get the same result, I have a feeling there’s some floating point precision issue but I don’t see how it can apply in this case since the elements of the tensor are exactly equal. Numpy, on the other hand, does not show this behaviour.</p>
<pre><code class="lang-auto">&gt;&gt;&gt; torch.all(z0[:, :, :2] == z1[:, :, :2])
True
&gt;&gt;&gt; s0 == s1
False
&gt;&gt;&gt; (s0.sum() - s1.sum()).abs().sum().numpy()
0.017578125
&gt;&gt;&gt; z0.numpy()[:, :, :2].sum() == z1.numpy()[:, :, :2].sum()
True
</code></pre>
<p>Any clue on why is this happening?</p>
<p>EDIT: if I switch the dimensions such that now I am summing over the first, i.e. z1.shape = [5,2,1000] and s1=z1[:2,:,:].sum() (same applies for z0 and s1) I don’t have the numerical issue. Why?</p>

username: ptrblck
cooked: <p>My guess would be that different algorithms could be picked depending on the memory layout of the data and its alignment.<br>
<code>(s0.sum() - s1.sum()).abs().max().numpy()</code> returns <code>~array(1.6212463e-05, dtype=float32)</code> which fits the range of the expected <code>float32</code> floating point precision.</p>
Taking gradient w.r.t multi loss function for a single variable

username: kbtorcher
cooked: <p>My loss function is defined as follows: $Total loss = MSE(w, \phi) + L2norm(F) + MSE(L2norm(\phi), 1). <strong>w</strong> is a d-dimensional vector which I am learning. I want to optimize <strong>w</strong> by taking the gradient of total loss w.r.t <strong>w</strong>. Is there a way to do so without creating a single layer neural network?</p>

username: tom
cooked: <p>You can just define <code>w</code> to be a tensor that requires gradient.<br>
You can instantiate the optimizer with <code>[w]</code> instead of <code>model.parameters()</code> or you could do the optimization yourself in a <code>torch.no_grad</code> block.</p>
<p>Best regards</p>
<p>Thomas</p>
Pytorch on GeForce RTX 3090 GPUs

username: SHRAMAN_PRAMANICK
cooked: <p>Hi, I am working with GeForce RTX 3090 GPUs. Recently, I am facing an error that I have never seen before. When creating a new virtual env, upgrading pip, and trying to install PyTorch by</p>
<pre><code class="lang-auto">pip3 install torch==1.11.0+cu113 torchvision==0.12.0+cu113 torchaudio==0.11.0+cu113 -f https://download.pytorch.org/whl/cu113/torch_stable.html
</code></pre>
<p>I am getting this error:</p>
<pre><code class="lang-auto">ERROR: Could not find a version that satisfies the requirement torch==1.11.0+cu113 (from versions: 1.0.0, 1.0.1, 1.0.1.post2, 1.1.0, 1.2.0, 1.3.0, 1.3.1, 1.4.0, 1.5.0, 1.5.1, 1.6.0, 1.7.0, 1.7.1, 1.8.0, 1.8.1, 1.9.0, 1.9.1, 1.10.0, 1.10.0+cu113, 1.10.1, 1.10.1+cu113, 1.10.2, 1.10.2+cu113)
ERROR: No matching distribution found for torch==1.11.0+cu113
</code></pre>
<p>I can install nightly by <code>pip3 install --pre torch torchvision torchaudio -f https://download.pytorch.org/whl/nightly/cu113/torch_nightly.html</code>, but then I face this error:</p>
<pre><code class="lang-auto">/cis/home/shraman/multimodal_self_supervision/experiment_codes/ssl_env/lib/python3.6/site-packages/torch/cuda/__init__.py:143: UserWarning:
GeForce RTX 3090 with CUDA capability sm_86 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_70.
If you want to use the GeForce RTX 3090 GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/
</code></pre>
<p>and the code stops<br>
<code>RuntimeError: CUDA error: no kernel image is available for execution on the device</code></p>
<p>I have used the same GPUs many many times with the different virtual environments a month ago. What is the problem now?</p>

username: tom
cooked: <aside class="quote no-group" data-username="SHRAMAN_PRAMANICK" data-post="1" data-topic="146848">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shraman_pramanick/48/27149_2.png" class="avatar"> SHRAMAN_PRAMANICK:</div>
<blockquote>
<p><code>python3.6</code></p>
</blockquote>
</aside>
<p>I think PyTorch 1.11 is 3.7+ only, as Python 3.6 is not supported by <a href="http://Python.org" rel="noopener nofollow ugc">Python.org</a> anymore.</p>
<p>Best regards</p>
<p>Thomas</p>