akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Getting the Data Points Classified Correctly

username: Maria_Fernanda_De_La
cooked: <p>If I wanted to observe which data-points are being classified correctly at every epoch, is there a simple way to do this? I’m thinking I would get torch.max(net(inputs,1) but how would I obtain the indices of these data-points. Given that in:</p>
<p><code>max_vals, max_indices = torch.max(net(inputs,1)</code></p>
<p>max_indices actually returns the index of the correctly classified label and not the the index of the datapoint, is there a better way to do this?</p>

username: albanD
cooked: <p>Hi,</p>
<p>What do you mean by the “index of the datapoint” ? The index in your Dataset?</p>

username: Maria_Fernanda_De_La
cooked: <p>Hello, thank you for your response. Yes exactly! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: albanD
cooked: <p>One approach will be for your dataset to return the index in the <code>__getitem__</code> function and then you can have access to it.</p>
<pre><code class="lang-auto"># In dataset definition
  def __getitem__(self, idx):
    return foo, bar, idx # To replace just foo, bar

# In train loop
  for foos, bars, idxs in dataloader:
    # Now you have access to the indices in the dataset of each sample.
</code></pre>
<p>Another approach is to get a dataloader that does not shuffle the samples, and then you know that the indices are given in increasing order from 0 to dataset_size by groups of batch_size.</p>
Simple networks does not reach zero loss

username: lfolle
cooked: <p>Why is this simple network not converging despite same inputs and labels each epoch?</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.optim as optim
import torch.nn.functional as F
import numpy as np
import matplotlib.pyplot as plt


class SimpleModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.lin1 = nn.Linear(30, 20)
        self.lrelu1 = nn.LeakyReLU()
        self.lin2 = nn.Linear(20, 9)
        self.lrelu2 = nn.LeakyReLU()
        self.act = nn.Softmax(dim=1)

    def forward(self, x):
        x = self.lin1(x)
        x = self.lrelu1(x)
        x = self.lin2(x)
        x = self.lrelu2(x)
        x = x.reshape(3, 3)
        return self.act(x)
            

model = SimpleModel()            
x = torch.randn(1, 30)
y = torch.tensor([[1., 0., 0.],
                  [0., 1., 0.],
                  [0., 0., 1.]])

criterion = nn.BCEWithLogitsLoss()
optimizer = optim.SGD(model.parameters(), lr=1e-3)
epochs = 40000

loss_history = []
for epoch in range(epochs):
    optimizer.zero_grad()
    output = model(x)
    loss = criterion(output, y)
    loss_history.append(loss)
    loss.backward()
    optimizer.step()
    print('Loss: {:.3f}'.format(loss.item()))
    
print("target: {}".format(y))
print("output: {}".format(output))

plt.plot(np.linspace(1, epochs, epochs), loss_history)
plt.xlabel("epoch")
plt.ylabel("loss")
plt.show()
</code></pre>

username: SimonW
cooked: <p>Because BCEWithLogitsLoss expects logits, not probabilities. Remove your softmax.</p>

username: lfolle
cooked: <p>Thank you.<br>
This did the trick for me:</p>
<pre><code class="lang-auto">criterion = nn.BCELoss()
</code></pre>
How to obtain the worst possible test accuracy?

username: TinfoilHat0
cooked: <p>I’m training a simple model on MNIST data where my goal is to obtain the worst possible test accuracy.</p>
<p>Intuitively, given my model reaches to %99 accuracy when it’s trained properly, the worst possible accuracy can be at most %1.  I can simply obtain this accuracy by picking some random value that is different than my model’s prediction.</p>
<p>However, if I want to train the model to do this instead of messing with its inference phase, how can I do it? I naively attempted to negate the output of my loss function (negative log-likelihood) with a goal of maximizing the training loss. However, I ended up with a model that reaches %10 accuracy which always predicts the same digit.</p>
<p>So long story short, what’s the proper way to to train a model where the goal is to obtain the worst possible test accuracy? Should I pick a loss function that is bounded (both from above and below) and try to negate that instead? If yes, what’s a suitable candidate?</p>

username: Anuj_Chopra
cooked: <p>Turn off all the regularization effects. The model will overfit then. Romeve dropout, use batch-size = 1, Learning rate should be small (you can get there in steps). Further try using wide and shallow network.</p>

username: JuanFMontesinos
cooked: <p>It’s (I would say) not possible to get an accuracy less than by chance unless you train the network to properly classify items and later on switch classes.</p>
<p>This means, train the network to semantically learn that 1 is 2 and so on and later on just use the proper labels.</p>
<p>Think that if the network learn to missclasify it is implicitly learning to classify.</p>

username: TinfoilHat0
cooked: <p>Thanks Juan, that’s what I had in my mind too as an alternative to maximizing the loss. With that method, my model reaches to about %0.2 accuracy which is low enough.</p>
Help with initial setup

username: Hamhub7
cooked: <p>Hey guys, I have been interested in NN in AI for a while now and stumbled across PyTorch C++ in a Discord Server. I am familiar with C++, and am following the C++ tutorials in the docs (<a href="https://pytorch.org/cppdocs/installing.html" rel="nofollow noopener">https://pytorch.org/cppdocs/installing.html</a>) but for some reason when I attempt to build my sample program it fails. It Coulld NOT find torch (missing: TORCH_LIBRARY). I have my logs attached as a github gist, so maybe you guys can find something there? my cmake command is pointing to the libtorch folder that I exctracted, as a side note. Thanks!<br>
PS: I am running Windows 10 Pro</p>
<p>Logs: <a href="https://gist.github.com/hamhub7/2665464f67ab6be6acf34388050a3e6e" rel="nofollow noopener">https://gist.github.com/hamhub7/2665464f67ab6be6acf34388050a3e6e</a></p>
<p>Edit: If I could get libtorch installed in Visual Studio 2019, that would be much better personally</p>
Plotting random image in validation set for GANs

username: tomt
cooked: <p>Hi,</p>
<p>I’m training a GAN to upscale low resolution image to high resolution image. For my validation set, I want to randomly plot images, but I always get the same one. My dataloader for my validation set has batch size one.</p>
<pre><code class="lang-auto">i_val = 0
for _,imgs in enumerate(val_loader):
            i_val +=1
            imgs_lr = Variable(imgs["lr"].type(Tensor))
            imgs_hr = Variable(imgs["hr"].type(Tensor))
            # VISUALIZING when  for loop is done
              if (i_val % len(val_loader) == 0):
                        with torch.no_grad():
                              index = torch.tensor([np.random.randint(imgs_lr.shape[0], size=1)[0]]).to(cfg.device)
                              lr_i = torch.index_select(imgs_lr, 0, index, out=None)
                            


</code></pre>

username: ptrblck
cooked: <p>Since your batch size is 1, I assume <code>imgs_lr.shape[0]</code> is also 1.<br>
If that’s the case, <code>np.random.randint(imgs_lr.shape[0], size=1)[0]</code> will always return a 0, which would explain the same plotted image.</p>

username: tomt
cooked: <p>Yeah that makes sense. It looks like my if statement will only lead me to plot out the last image. Is there any workaround where I can plot a random image instead?</p>

username: ptrblck
cooked: <p>You could sample some variable from <code>[0, len(val_loader)-1]</code> and compare it directly to <code>i_val</code> inside the data loader loop.<br>
Then just use the same variable to get the corresponding image.</p>
<p>Alternatively, you could also remove the plotting completely from the data loader loop and just sample from the <code>Dataset</code> directly using a random number.</p>

username: tomt
cooked: <p>What a splendid idea! Continue being awesome <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Mask input / skip row in conv layer

username: jbo
cooked: <p>Hi,</p>
<p>is there a way to skip/mask a particular row/column or even pixel of the input in a Conv2D layer?<br>
Further, is it possible to skip/mask only the backward pass?</p>
<p>An example:<br>
Given an input with (h,w)=(10,10)  e.g. the 5th row should be ignored, so that the output size after appropriately padded convolution is (h,w) = (9,10).</p>

username: albanD
cooked: <p>Hi,</p>
<p>You can manually remove the row with (for a 1D Tensor) <code>new = torch.cat([old[:idx], old[idx+1:]], 0)</code>.</p>

username: jbo
cooked: <p>Thanks, but the row/column/pixel is needed for the convolution of neighboring rows/columns/pixels. It just should not produce an output where it’s at the center of the convolution kernel (especially not on the backward pass)</p>

username: albanD
cooked: <p>You can delete that line after the convolution computation then?</p>

username: jbo
cooked: <p>For the forward pass yes, but what happens to the backward pass then?</p>

username: albanD
cooked: <p>Since you deleted this output’s row, it’s gradient will be 0. And so it won’t have any impact on the input.</p>

username: jbo
cooked: <p>Perfect, many thanks!</p>
<p>What is with individual pixels, if I want to ignore them for the backward pass?</p>

username: albanD
cooked: <p>What do you mean by ignore them for the backward pass exactly? Do you want to force the gradients for these pixels to be 0?</p>

username: jbo
cooked: <p>Yes kind of.<br>
I need those pixels in the forward pass, but when the kernel is centered on those pixels it (the entire kernel at that location) should not contribute to the gradient.</p>

username: albanD
cooked: <p>The masking will do what you want. As it will fill the gradient for these pixels with <code>0</code>s and the backward of the conv will propagate <code>0 * w = 0</code> for these entries.</p>

username: jbo
cooked: <p>sorry, which masking do you mean?</p>

username: albanD
cooked: <p>Any of them <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> But in particular the one discussed above using <code>.cat</code> will work.</p>
PyTorch and Visual Studio 2019. Exception in Dimname.h

username: JaMM
cooked: <p>When I start my project everything is fine but when I start debug mode I get the following message:</p>
<p>“An unhandled exception occurred at address 0x000007FEFD7FB3DD in NerualNetwork.exe: Microsoft C ++ exception: std :: length_error at memory address 0x000000000014F420.”</p>
<p>This message is issued about the line code “static Symbol kWildcard = Symbol::dimname(”*");" in the file “Dimname.h”</p>
<p>How can I fix it?</p>

username: JaMM
cooked: <p>The question is closed. The error was that I used the release version in the debug project.</p>
Help me,solve this problem

username: Loring
cooked: <p>I want to try the distribution of pytorch, I used two nodes, but reported an error during the startup of the second node:<br>
Traceback (most recent call last):<br>
File “test.py”, line 205, in <br>
main()<br>
File “test.py”, line 198, in main<br>
rank=args.rank,<br>
File “/home/loring/anaconda3/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py”, line 410, in init_process_group<br>
timeout=timeout)<br>
File “/home/loring/anaconda3/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py”, line 478, in _new_process_group_helper<br>
timeout=timeout)<br>
RuntimeError: [/opt/conda/conda-bld/pytorch_1570710822989/work/third_party/gloo/gloo/transport/tcp/pair.cc:761] connect [::1]:14728: Connection refused</p>
<p>who can help me?thanks</p>

username: albanD
cooked: <p>Hi,</p>
<p>It seems like the there is connection issues between your nodes. Do you have the necessary permissions to connect from one computer to the other at the port you specified?</p>
Dataloader stuck when adding shuffle or with bigger dataset

username: Tomer
cooked: <p>Hello,<br>
My program got stuck when i increased my dataset size, i notice that me dev_loader work fine.<br>
the only different is that he use (shuffle=False), when i removed the shuffle from the train_loader it works fine.<br>
small datset with/without shuffle work.<br>
big datset without shuffle work.<br>
big datset with shuffle <strong>don’t</strong> work.<br>
Anyone khow i can solve it?</p>
<pre><code class="lang-python">class SliceData(Dataset):
    def __init__(self, root, transform, sample_rate=1):
        self.transform = transform

        self.examples = []
        files = list(pathlib.Path(root).iterdir())
        if sample_rate &lt; 1:
            random.shuffle(files)
            num_files = round(len(files) * sample_rate)
            files = files[:num_files]
        for fname in sorted(files):
            img = h5py.File(fname, 'r')['data']
            num_slices = img.shape[2]
            self.examples += [(fname, slice) for slice in range(num_slices)]

    def __len__(self):
        return len(self.examples)

    def __getitem__(self, i):
        fname, slice = self.examples[i]
        with h5py.File(fname, 'r') as data:
            img = data['data'][:, : , slice, :]
            bvecs = data['bvecs'][:]
            return self.transform(img, bvecs, fname.name, slice)
</code></pre>

username: JuanFMontesinos
cooked: <p>Shuffle does nothing but shuffling the indices which are gonna be called. Seems that you have troubles reading non-sequential data. Why don’t you try to change your data format?<br>
Besides, I don’t usually use h5py but it seems you are reading  the whole file in getitem, if it’s very big that may be problematic.</p>
Plotting multiple tensors

username: Lame
cooked: <p>Hello Pytorch community <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>I have a question about drawing/plotting a group of images tensors.</p>
<p>So if I had a combined tensor that has 5 different tensors (each time I add one by one to it using <code>torch.cat((combined, Y))</code>)</p>
<p>And I want to plot it like one beside of each other (like one row of 5 columns)</p>
<p>What I tried is first to plot just one image in order just to see it but I had an error of <code>TypeError: Invalid shape (8, 1, 28, 28) for image data</code> where <code>[8, 1, 28, 28]</code> is the shape of one tensor image</p>
<p>And about plotting them all together, I thought of maybe using:</p>
<pre><code class="lang-auto">poly.subplots(something here)
make_grid(combined tensor or other things)
</code></pre>
<p>But still, I’m not fully convinced about how I can do it.</p>
<p>Any help please with that?</p>

username: tom
cooked: <aside class="quote no-group" data-username="Lame" data-post="1" data-topic="60167">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/l/c57346/48.png" class="avatar"> Lame:</div>
<blockquote>
<p>each time I add one by one to it using <code>torch.cat((combined, Y))</code></p>
</blockquote>
</aside>
<p>That isn’t really a good idea to do at scale. It’s better to keep a list and cat at the end.</p>
<p>Another thing: Isn’t that a batch of 8 images rather than one?</p>
<p>I use <code>pyplot.imshow</code> a lot, it wants HWC format (and not batch), so you would need to do something like <code>combined = combined.permute(0, 2, 3, 1).reshape(-1, combined.size(3), combined.size(1))</code> (or <code>permute(2, 0, 3, 1).reshape(combined.size(2, -1, combined.size(1)</code>.</p>
<p>torchvision has <a href="https://pytorch.org/docs/master/torchvision/utils.html#torchvision.utils.make_grid" rel="noopener nofollow ugc"><code>make_grid</code></a> that can do the layout for you more easily than the above.</p>
<p>Best regards</p>
<p>Thomas</p>
Cublas runtime error : library not initialized at work/aten/src/THC/THCGeneral.cpp:250

username: sryim
cooked: <p>Hello. I’m having some trouble with running pytorch with GPU.</p>
<p>The code is as below:<br>
import torch<br>
device = ‘cuda’<br>
a = torch.Tensor([[1, 2]]).to(device)<br>
fc1 = torch.nn.Linear(2, 4).to(device)<br>
print(fc1(a))</p>
<p>It is a very simple code, however, it came up with this error message:<br>
Traceback (most recent call last):<br>
File “test.py”, line 6, in <br>
print(fc1(a))<br>
File “lib/python2.7/site-packages/torch/nn/modules/module.py”, line 489, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “lib/python2.7/site-packages/torch/nn/modules/linear.py”, line 67, in forward<br>
return F.linear(input, self.weight, self.bias)<br>
File “lib/python2.7/site-packages/torch/nn/functional.py”, line 1352, in linear<br>
ret = torch.addmm(torch.jit._unwrap_optional(bias), input, weight.t())<br>
RuntimeError: cublas runtime error : library not initialized at …conda/conda-bld/pytorch_1549614443593/work/aten/src/THC/THCGeneral.cpp:250</p>
<p>The code runs fine with CPU.<br>
Also, simple operations such as bias+bias work normally.<br>
The error message seems strange since the file conda/condabld/pytorch_1549614443593/work/aten/src/THC/THCGeneral.cpp:250 even doesn’t exist.</p>
<p>I found some solutions saying that removing ~/.nv or monitoring GPU usage with nvidia-smi, but they did not work for me.</p>
<p>Any help would be appreciated.<br>
Thank you!</p>

username: albanD
cooked: <p>Hi,</p>
<p>I would make sure that you only have one cuda install on your system. And that you have a version of pytorch that is built for that same version of cuda.</p>

username: sryim
cooked: <p>Thank you for your reply.<br>
My home directory was on NFS and there was an issue about NFS file locking, which prevented ~/.nv to be written.<br>
I got a hint from the link:<br>
<a href="https://devtalk.nvidia.com/default/topic/1003878/problem-with-cuda-8-with-381-09-drivers-on-ubuntu-16-04-gtx-1080ti/" class="onebox" target="_blank" rel="nofollow noopener">https://devtalk.nvidia.com/default/topic/1003878/problem-with-cuda-8-with-381-09-drivers-on-ubuntu-16-04-gtx-1080ti/</a></p>
Loading the mode and training is same as keeping it training?

username: Yupjun
cooked: <p>Let’s say I have one autoencoder model that should be trained on 20 epochs. (any model can be done)<br>
Does training 20 epochs straight is the same as saving model at 10 epochs and re-run it on for 10 epochs?</p>
<p>I had set the seed part like this.</p>
<pre><code class="lang-auto">torch.manual_seed(0)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
np.random.seed(0)
</code></pre>
<p>I am using the “load_state_dict” method for loading the model. What I’m concerning  is that loading the model could affect the model performance</p>

username: JuanFMontesinos
cooked: <p>It doesn’t affect the performance if you properly do it. Models are supposed to be robust to randomness.<br>
The fact is that you also have to save optimizer state_dict as some of them has running parameters which affects loss, eg, Adam.</p>
<p>If you don’t do so, you will observe a peak in the loss (which should be recovered).</p>
About torch.no_grad()

username: WormWakedEarlyEatenB
cooked: <p>I tried to use Resnet50 to extract images’ feature and classify it by our own CNNNet.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/1/2/1278983c672e2c886cfc848dfa5f55c7c642acf9.png" alt="no_grad()" data-base62-sha1="2DoXYx6v4zRzNRb99yeGEN9U3y1" width="488" height="383"><br>
I want to know whether <strong>totch.no_grad()</strong> makes <strong>optimizer.step()</strong> invalid during updating parameters of Resnet50.<br>
If yes, how could I change it, because pycharm warns out of memory without <strong>torch.no_grad</strong> .</p>

username: JuanFMontesinos
cooked: <p>Indeed if you compute backpropagation you needs more memory. There is no issue about that. You may need to use a lighter network</p>

username: WormWakedEarlyEatenB
cooked: <p>so Resnet50 can’t be updated with torch.no_grad(), right?</p>

username: JuanFMontesinos
cooked: <p>no_grad disable autograd machinery. Nothing can be updated with torch no grad.<br>
It is used to speed up evaluation and optimize memory usage in inference. For regular training you cannot use that and it won’t work as you have seen.</p>

username: WormWakedEarlyEatenB
cooked: <p>ty, it seems that I need to deal with memory</p>
Why does the normal distribution have `loc` and `scale` as attributes?

username: jwillette
cooked: <p>The “normal” thing to call them IMO would be <code>mu</code> and <code>sigma</code> or <code>mean</code> and <code>std</code>, why was <code>loc</code> and <code>scale</code> chosen? Is there some background to this?</p>
<p>I was very confused by the names the first time I had to use the distributions…</p>

username: tom
cooked: <p>Superficially, like many things in PyTorch, this is in alignment with NumPy, or here <a href="https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.norm.html#scipy.stats.norm" rel="nofollow noopener"><code>scipy.stats</code></a>.</p>
<p>Going more in depth, this is to generalize shifting (loc) and coordinate-scaling (scale) distribution from a “standard” location and scale. For example, the lognormal distribution’s scale parameter is doing scaling, too, and not just changing σ.<br>
Wikipedia has an entry on <a href="https://en.wikipedia.org/wiki/Scale_parameter" rel="nofollow noopener">scale parameter</a> elaborating on the concept.</p>
<p>Best regards</p>
<p>Thomas</p>
Generator output

username: Frida
cooked: <p>Hi,<br>
What is in general the last layer of a generator?</p>

username: ptrblck
cooked: <p>If might depend on your use case, but e.g. DCGAN uses a tanh as the generator output.</p>

username: Frida
cooked: <p>I have an imbalanced dataset and I want to generate more images from one type and train on it as well. However, it seems like that the different statistics of the generated images differ from the original dataset. Any idea how can I align them?</p>
Output shape doesn't match to broadcast shape

username: Ahmad_Nur_Riza
cooked: <p>Hello everyone, I have a problem when I want to test my training file in real-time. When I grab the picture from the camera, there is error like this<br>
Traceback (most recent call last):<br>
File “/home/riza/Documents/P R O G R A M /NEW_GAN/Test.py”, line 145, in <br>
real_img = transform(real_img).cuda()<br>
File “/usr/local/lib/python3.5/dist-packages/torchvision/transforms/transforms.py”, line 60, in <strong>call</strong><br>
img = t(img)<br>
File “/usr/local/lib/python3.5/dist-packages/torchvision/transforms/transforms.py”, line 163, in <strong>call</strong><br>
return F.normalize(tensor, self.mean, self.std, self.inplace)<br>
File “/usr/local/lib/python3.5/dist-packages/torchvision/transforms/functional.py”, line 208, in normalize<br>
tensor.sub_(mean[:, None, None]).div_(std[:, None, None])<br>
RuntimeError: output with shape [1, 512, 512] doesn’t match the broadcast shape [3, 512, 512]<br>
Please your help. Thank you</p>

username: ptrblck
cooked: <p>Are you dealing with some gray images (single channel), while others are RGB  (3 channels)?<br>
Based on the error message, it seems <code>mean</code> and <code>std</code> contain three values (for each channel), while the <code>tensor</code> seems to be a single channel tensor.</p>
<p>If that’s the case, you could load the images as color images (<code>PIL</code> will repeat the single channel) using <code>img = img.convert("RGB")</code>.</p>

username: Ahmad_Riza
cooked: <p>The image is in RGB, because I take the picture directly from a camera, and I have set mean and std contain three values<br>
This is my list code of transform image<br>
transform = tv.transforms.Compose([   tv.transforms.Resize(512),<br>
tv.transforms.ToTensor(),<br>
tv.transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))<br>
])</p>

username: ptrblck
cooked: <p>It still seems <em>some</em> images might be loaded as grayscale images.<br>
Here is a small code snippet to reproduce your error message:</p>
<pre><code class="lang-python">mean = [0.5, 0.5, 0.5]
std = [0.5, 0.5, 0.5]

trans = transforms.Normalize(mean, std)

x = torch.randn(3, 224, 224) # works
out = trans(x)

y = torch.randn(1, 224, 224) # throws your error
out = trans(y)
&gt; RuntimeError: output with shape [1, 224, 224] doesn't match the broadcast shape [3, 224, 224]
</code></pre>

username: Ahmad_Nur_Riza
cooked: <p>sorry, what are x and y?</p>

username: ptrblck
cooked: <p><code>x</code> and <code>y</code> just represent some dummy image tensors to reproduce the error.</p>

username: Ahmad_Nur_Riza
cooked: <p>Oke, I see, Thank you for your help. I have find the problem. Because I train with the different network so I can’t load the training file.</p>

username: Ahmad_Nur_Riza
cooked: <p>I am so sorry about it</p>
Shorthand way to convert an ndarray of 2D torch tensors to completely torch tensor

username: saruftw
cooked: <p>I have some data of the following form,</p>
<p>shape = (batch_size X max_seq_len X embedding_dim)</p>
<pre><code class="lang-auto">np.ndarray([torch.tensor([torch.tensor(), torch.tensor(), ....]), ...])
</code></pre>
<p>Is there a convenient way to convert it to,</p>
<pre><code class="lang-auto">torch.tensor([torch.tensor([torch.tensor(), torch.tensor(), ....]), ...])
</code></pre>
<p>Right now, I do</p>
<pre><code class="lang-auto">torch.from_numpy(devX)
</code></pre>
<p>and get the error</p>
<pre><code class="lang-auto">TypeError: can't convert np.ndarray of type numpy.object_. The only supported types are: float64, float32, float16, int64, int32, int16, int8, uint8, and bool.
</code></pre>

username: tom
cooked: <p>Tensors fundamentally cannot hold arbitrary objects, but assuming you have a 1d array of 2d tensors, you can use <code>stack</code> to get a 3d Tensor.</p>
<p>Best regards</p>
<p>Thomas</p>

username: saruftw
cooked: <p>Thanks for the reply! Could you elaborate a little more? I’m sort of new to the framework. Thanks.</p>

username: tom
cooked: <p>Well, so ideally you would have shown more about how your data looks like (I must admit I don’t understand that exactly), but say it is similar to</p>
<pre><code class="lang-python">a,b,c = torch.randn(3, 2, 10) # three 2x10 tensors
arr = numpy.array([a,b,c], dtype=object) # array of size 3 of 2x10 tensors
</code></pre>
<p>then you can do</p>
<pre><code class="lang-python">t = torch.stack(list(arr), dim=0) # 3x2x10 tensor
</code></pre>
<p>Best regards</p>
<p>Thomas</p>
Why is Training error not changing? Loss is decreasing and weights are changing

username: Maria_Fernanda_De_La
cooked: <pre><code class="lang-auto">import torch
import torch.nn as nn
import numpy as np
from sklearn.datasets.samples_generator import make_blobs
from sklearn.model_selection import train_test_split

x, y = make_blobs(n_samples=1000, centers=2,random_state=0)
x, xtest, y, ytest = train_test_split(x, y, test_size=0.15, random_state=42)  
        
x_data = torch.Tensor(x)
y_data = torch.Tensor(y)
x_data.requires_grad_(True)
y_data.requires_grad_(True)

x_data_test = torch.Tensor(xtest)
y_data_test = torch.Tensor(ytest)
x_data_test.requires_grad_(True)
y_data_test.requires_grad_(True)


class Net(nn.Module):
  def __init__(self):
      super(Net, self).__init__()
      self.layer1 = torch.nn.Linear(2, 32)
      self.layer2 = torch.nn.Linear(32, 16)
      self.layer3 = torch.nn.Linear(16, 1)

  def forward(self,x):
      x = self.layer1(x)
      x = self.layer2(x)
      x = self.layer3(x)
      return x


net = Net()
optimizer = torch.optim.SGD(net.parameters(), lr=0.01)
loss_fn = torch.nn.BCEWithLogitsLoss()


def error_criterion(outputs,labels):
    max_vals, max_indices = torch.max(outputs,1)
    error = (max_indices != labels).float().sum()/max_indices.size()[0]
    return error


net.train()
y_data = y_data.unsqueeze(1)
for t in range(900):
    outputs = net(x_data)
    
    loss = loss_fn(outputs, y_data)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()


    if t % 100 == 0:
        print("Loss:", loss.item())
        print("Error:", error_criterion(outputs, y_data.squeeze(1)).item())
      

</code></pre>

username: ptrblck
cooked: <p>Since you are using a single output neuron for your binary classification task, <code>torch.max(outputs, 1)</code> will always yield zeros.</p>
<p>Instead you could use a threshold (e.g. 0, as you are dealing with logits) to calculate the error:</p>
<pre><code class="lang-python">def error_criterion(outputs,labels):
    preds = outputs &gt; 0.
    error = (preds.float() != labels).float().mean()
    return error

for t in range(900):
    outputs = net(x_data)
    
    loss = loss_fn(outputs, y_data)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()


    if t % 100 == 0:
        print("Loss:", loss.item())
        print("Error:", error_criterion(outputs, y_data).item()) # I removed the squeeze() here
</code></pre>
<p>Also, note that you are basically using a single linear layer, since you don’t apply any activation function after <code>layer1</code> and <code>layer2</code>.</p>
a error about the model of pytorch transform to libtorch

username: chen1234520
cooked: <pre><code class="lang-auto">    # load net
    num_classes = len(labelmap) + 1                      # +1 for background
    net = build_refinedet('test', int(args.input_size), num_classes)            # initialize SSD
    net.load_state_dict(torch.load(args.trained_model,map_location='cuda:0'))    # model = torch.load(model_path, map_location='cuda:0')
    net.eval()
    print('Finished loading model!')

    # 向模型中输入数据以得到模型参数
    example = torch.rand(1,3,320,320).cuda()
    traced_script_module = torch.jit.trace(net,example)

    # 保存模型
    traced_script_module.save("torch_script_eval.pt")
</code></pre>
<p>i want to make the refinedet model transform to libtorch,but it have a error about “RuntimeError: Attempted to trace Detect_RefineDet, but tracing of legacy functions is not supported”</p>
<p>Thank you for helping me！</p>

username: tom
cooked: <p>As the error message says, the model is using PyTorch 0.1.2-style Functions. (Newer autograd.Functions cannot be traced either, but that is another story…)<br>
You probably want to convert the model to use the nms functions provided by TorchVision, those have tracing support.</p>
<p>Best regards</p>
<p>Thomas</p>
Get parameters of modules in a dictionary

username: Yunxiang_LI
cooked: <p>If a module contains a dictionary which has other two modules as following, can I get parameters of model_dict[‘model1’] and model_dict[‘model2’] with outer_network.state_dict()? Or is there any solutions to get parameters of modules from a dictionary?</p>
<pre><code class="lang-auto">class inner_network_1(nn.Module):
    def __init__(self):
        super(inner_network_1, self).__init__()
        self.inner_fc = nn.Linear(20, 30)

class inner_network_2(nn.Module):
    def __init__(self):
        super(inner_network_2, self).__init__()
        self.inner_fc = nn.Linear(20, 30)

class outer_network(nn.Module):
    def __init__(self):
        super(outer_network, self).__init__()
        self.fc = nn.Linear(10, 20)
        model_dict = {}
        model_dict['model1'] = inner_network_1()
        model_dict['model2'] = inner_network_2()
</code></pre>

username: vainaijr
cooked: <p>do you mean something like this,</p>
<pre><code class="lang-auto">class outer_network(nn.Module):
  def __init__(self):
    super().__init__()
    self.fc = nn.Linear(10, 20)
    self.inner_network_1 = nn.ModuleDict([['inner_fc', nn.Linear(20, 30)]])
    self.inner_network_2 = nn.ModuleDict([['inner_fc', nn.Linear(20, 30)]])

model = outer_network()
model.state_dict()
</code></pre>
<p>or something like this,</p>
<pre><code class="lang-auto">class inner_network_1(nn.Module):
    def __init__(self):
        super(inner_network_1, self).__init__()
        self.inner_fc = nn.Linear(20, 30)

class inner_network_2(nn.Module):
    def __init__(self):
        super(inner_network_2, self).__init__()
        self.inner_fc = nn.Linear(20, 30)

class outer_network(nn.ModuleDict):
    def __init__(self):
        super(outer_network, self).__init__()
        self.fc = nn.Linear(10, 20)
        self['model1'] = inner_network_1()
        self['model2'] = inner_network_2()

model = outer_network()
model.state_dict()
</code></pre>

username: Yunxiang_LI
cooked: <p>Thank you for these two examples! I didn’t know ModuleDict, which is exactly what I need!</p>
What is THCudaTensor_data corresponding function in Aten

username: Bohang_Li
cooked: <p>I’m converting a Pytorch 0.4 version model to Pytorch 1.3. I failed to build some self defined .c files, because Pytorch deprecated THC, and use ATen as CUDA lib. Can anyone tell me what is the corresponding THCudaTensor_data function in ATen? Or how should I modify the C files?</p>

username: SimonW
cooked: <p>probably <code>tensor.data_ptr()</code></p>

username: Bohang_Li
cooked: <p>Hi <a class="mention" href="/u/simonw">@SimonW</a>, thanks for your reply. It seems right, both functions return the address of given tensor. However, I want to use it in c++, should I use something like at::tensor.data_ptr()?</p>

username: SimonW
cooked: <p>yeah, I actually meant the cpp code. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: Bohang_Li
cooked: <p>Thank you! will give a try.</p>
This implementation of a kind of Dense block is right or wrong?

username: SugarMasuo
cooked: <p>I recently referred to a kind of dense block implementation (not in a classic way), but what puzzles me is his use of nn.ModuleList. His implementation is shown below.</p>
<p>I think, just the first nn.Sequential in the layers (nn.ModuleList) is used, is right？Look forward replying to help me understand it. Thanks!</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/5/d5d183413a392a58ee89c07a328cec06faccd14e.png" data-download-href="https://discuss.pytorch.org/uploads/default/d5d183413a392a58ee89c07a328cec06faccd14e" title="%E5%BE%AE%E4%BF%A1%E6%88%AA%E5%9B%BE_20191105142734"><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/5/d5d183413a392a58ee89c07a328cec06faccd14e.png" alt="%E5%BE%AE%E4%BF%A1%E6%88%AA%E5%9B%BE_20191105142734" data-base62-sha1="uvwzY05Zg4yNnWnssgHuH3yE1t4" width="643" height="500" data-dominant-color="FAF9F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">%E5%BE%AE%E4%BF%A1%E6%88%AA%E5%9B%BE_20191105142734</span><span class="informations">672×522 14.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>link is here: <a href="https://github.com/zhixinshu/DeformingAutoencoders-pytorch/blob/master/DAENet.py" class="inline-onebox" rel="noopener nofollow ugc">DeformingAutoencoders-pytorch/DAENet.py at master · zhixinshu/DeformingAutoencoders-pytorch · GitHub</a></p>

username: ptrblck
cooked: <p>I’m not familiar with the code base and am not sure, how the model is supposed to work, but you could use forward hooks to check, which layer is being called:</p>
<pre><code class="lang-auto">model = DenseBlockEncoder(1, 3)

def print_layer(name):
    def hook(model, input, output):
        print("{} called".format(name))
    return hook    
    
for idx in range(len(model.layers)):
    model.layers[idx].register_forward_hook(print_layer(str(idx)))

x = torch.randn(1, 1, 4, 4)
model(x)
&gt; 0 called
</code></pre>
<p>In this example, only the first sequential block is being called.</p>

username: SugarMasuo
cooked: <p>Thank you so much! That means what I thought is right.</p>
Training loss and validation loss does not change during training

username: muhibkhan
cooked: <p>I’m running an embedding model. I have a embedding model that I am trying to train where the training loss and validation loss does not go down but remain the same during the whole training of 1000 epoch. while i’m also using: lr = 0.001,  optimizer=SGD.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/a/ea74539de49393bda91dddd7dcfc6969e9c3ced6.png" data-download-href="https://discuss.pytorch.org/uploads/default/ea74539de49393bda91dddd7dcfc6969e9c3ced6" title="2019-09-20_0935"><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/a/ea74539de49393bda91dddd7dcfc6969e9c3ced6.png" alt="2019-09-20_0935" data-base62-sha1="xs4XjH0LMMjZQmeFZoHC3QTnzdI" width="275" height="500" data-dominant-color="0E0E0E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">2019-09-20_0935</span><span class="informations">353×640 8.93 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: mailcorahul
cooked: <p>Can you post the code here?</p>

username: muhibkhan
cooked: <p>Brother How I upload it? Its huge and multiple team. I didn’t have access some of the modules.</p>

username: Prashant_Kalikotay
cooked: <p>I too faced the same problem, the way I went debugging  it was:<br>
You can check your codes output after each iteration,<br>
if the output is same then there is no learning happening.<br>
maybe some of the parameters of your model which were not supposed to be detached might have got detached.</p>

username: mailcorahul
cooked: <p>Check the code where you pass model parameters to the optimizer and the training loop where ‘optimizer.step()’ happens. Also see if the parameters are changing after every step.</p>

username: 111179
cooked: <p>hi，are you solve the prollem? I have met the same problem with you!</p>

username: muhibkhan
cooked: <p><a class="mention" href="/u/111179">@111179</a> Yeah I was detaching the tensors from gpu to cpu before the model starts learning. So in that case the optimizer and the learning rate does affect anything. What particularly your model is doing?</p>

username: 111179
cooked: <p>The training loss and validation loss doesn’t change, I just want to class the car evaluation</p>

username: muhibkhan
cooked: <p>use dropout between layers. After passing the model parameters use optimizer.step() to evaluate it in each iteration (the parameters should changing after each iteration)</p>

username: 111179
cooked: <p>yep,I have already use optimizer.step(), can you see my code? Thank you.</p>
Norm of a Tensor

username: Audrey
cooked: <p>For a 2-dimensional tensor, I want to normalize each row vector of it. In the formula, it’s like:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/2/5244fd4a734747019fa276b015adbefa6eac7181.jpeg" alt="%E6%89%B9%E6%B3%A8%202019-11-05%20205242" data-base62-sha1="bJMXCRc3UGJg2WYd9cL0jtb1A4N" width="363" height="32"></p>
<p>I know such code below can solve easily:</p>
<pre><code class="lang-auto">embedding_norm = Torch.norm(embedding.weight.data, dim=1, keepdim=True)
</code></pre>
<p>But I don’t understand why the value of parameter <em>dim</em> = <em>1</em>. I’ve read doc. where it says the value of <em>dim</em> must be an int to calculate vector norm, but why the number is <em>1</em> here?</p>

username: JuanFMontesinos
cooked: <p>dim refers to which dimension normalize with respect<br>
your two dimensional tensor has 2 dims, (x,y) which corresponds to dims (0,1)<br>
if you set 0 it’s row-wise normalization.<br>
If you set 1 it’s colum.wise normalization<br>
extrapolable for N-dim tensors</p>

username: JuliousHurtado
cooked: <p>I don’t know the complete example, but most likely the dimension 0 in your example is the batch size, so you want to get the norm of each vector from the batch.</p>

username: Audrey
cooked: <p>Thank you for your help!</p>

username: Audrey
cooked: <p>Yes, it seems like <em>dim</em> should be 0 here. The example code is implemented by others and they may made a mistake. Thanks!</p>
Is there any way to compute second order grad for DataParallel models?

username: FlyingRoastDuck
cooked: <p>Hi, I’m currently working on a project related to paralleled MAML training. I need to compute second-order grad for DataParallel models. However, the only way I find to solve this problem is<br>
<code>torch.autograd.grad(loss, model.module.params())</code><br>
(params is a generator function to output all parameters, which is implemented by myself.), which may cause imbalanced GPU utilization. Is there any way to solve this problem?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/d/ad8673a155538b8d69a8a4c1f225bf1a33eaecb2.png" data-download-href="https://discuss.pytorch.org/uploads/default/ad8673a155538b8d69a8a4c1f225bf1a33eaecb2" title="%E6%8D%95%E8%8E%B7"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/d/ad8673a155538b8d69a8a4c1f225bf1a33eaecb2_2_690x198.png" alt="%E6%8D%95%E8%8E%B7" data-base62-sha1="oL4F9AsMWekWJhnt7wxL9pSjAfo" width="690" height="198" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/d/ad8673a155538b8d69a8a4c1f225bf1a33eaecb2_2_690x198.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/d/ad8673a155538b8d69a8a4c1f225bf1a33eaecb2_2_1035x297.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/d/ad8673a155538b8d69a8a4c1f225bf1a33eaecb2.png 2x" data-dominant-color="120808"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">%E6%8D%95%E8%8E%B7</span><span class="informations">1242×357 31 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: albanD
cooked: <p>Hi,</p>
<p>I would say this is expected as the gradients are all gathered back to the main GPU after being computed (so that you can do the weight update).</p>

username: FlyingRoastDuck
cooked: <p>HI, thanks for your reply. In my project, the imbalanced GPU utilization is quite serious. I want to optimize my model on 4 GPUs at the same time. Is there any way to solve this problem? Thank you.</p>
The model only improves Precision/Recall AUC

username: amirhf
cooked: <p>I have a CNN model for an imbalanced image classification problem. I’m experimenting with a theory that is supposed to improve the accuracy of the model. Since I’m dealing with imbalanced data, I’m reporting f1, prec/rec auc and the average of recalls for each class (mean of the confusion matrix diagonal values). It seems like the new modification only improves PRAUC but f1 and average recall still stays comparable to the baseline. How can I interpret this?</p>
Re-assign part of the tensor object

username: Sara_Wasl
cooked: <p>Hi all,</p>
<p>Is there a way to re-assign the batch_size of a tensor?</p>
<p>For example: if I have a tensor <code>x</code> of shape <code>[1,200,1,1]</code>, is there a way like <code>x.shape[0] = 20</code></p>
<p>So it becomes of shape <code>[20,200,1,1]</code></p>
<p>Any help/suggestion with that, please?</p>

username: albanD
cooked: <p>Hi,</p>
<p>What do you expect this new Tensor to contain?</p>

username: Sara_Wasl
cooked: <p><a class="mention" href="/u/alband">@albanD</a> it contains a randoms numbers</p>

username: albanD
cooked: <p>when you say random you mean truly random or undefined values?</p>

username: Sara_Wasl
cooked: <p><a class="mention" href="/u/alband">@albanD</a> truly random as been generated previously (before many steps with <code>torch.randn</code>)</p>

username: albanD
cooked: <p>The simplest would be to just re-generate it with <code>x = torch.randn(new_size)</code>.</p>

username: Sara_Wasl
cooked: <p><a class="mention" href="/u/alband">@albanD</a> I understand you, but no</p>
<p>this random tensor are not directly generated with <code>torch.randn</code> (it come after a number of calculations of another tensor been generated with <code>torch.randn</code></p>

username: albanD
cooked: <p>But if you want 20 times more samples (as in the example you gave initially), you need to call your code that generates these numbers with the bigger input to get more random numbers.<br>
Or do you want 20 times the same samples?</p>

username: Sara_Wasl
cooked: <p><a class="mention" href="/u/alband">@albanD</a> Yes exactly, I want 20 times the same samples</p>

username: albanD
cooked: <p>Hoooo</p>
<p><code>x.expand([20,200,1,1])</code> then <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: Sara_Wasl
cooked: <p><a class="mention" href="/u/alband">@albanD</a> Oh it works! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>Many thanks!</p>
Feeding single channel images to three channel pre-trained model

username: MrRobot
cooked: <h2><a name="problem-1" class="anchor" href="#problem-1"></a>Problem</h2>
<p>I am trying to use ResNet pretrained model provided in <code>torchvision.models</code>, which is pre-trained on ImageNet where images are RGB. However, my dataset consists of CT scan images, which are grayscale. I am wondering how to feed my data into this model to fine-tune the parameters.</p>
<p>Three are several thoughts</p>
<ul>
<li>Fill the other two channels with all 0’s or 1’s.</li>
<li>Copy the single channel images three times.</li>
</ul>
<p>I tried to do those with MNIST dataset and got the following, where the first one corresponds to copying three times while the other two correspond to padding with all 1 and padding with all 0, respectively.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/b/6bff65c2fdd91d3c842e590a01c55b35825aa64d.png" data-download-href="https://discuss.pytorch.org/uploads/default/6bff65c2fdd91d3c842e590a01c55b35825aa64d" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/b/6bff65c2fdd91d3c842e590a01c55b35825aa64d_2_690x239.png" alt="image" data-base62-sha1="fpoh6PVAuUG2QaNuuXkfr68T3B3" width="690" height="239" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/b/6bff65c2fdd91d3c842e590a01c55b35825aa64d_2_690x239.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/b/6bff65c2fdd91d3c842e590a01c55b35825aa64d_2_1035x358.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/b/6bff65c2fdd91d3c842e590a01c55b35825aa64d.png 2x" data-dominant-color="73733E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1156×401 28.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>From this, it seems that copying the single channel three times make more sense since they look essentially the same as original image. But I am not sure someone encountered this problem and found a empirically reasonable solution.</p>
<p>Thank you in advance!</p>

username: JuanFMontesinos
cooked: <p>It’s difficult as you are applying a human semantic perspective assuming that cloning channels “looks the same” It’s better than setting random numbers but I don’t think it will provide good results. You should finetune the network</p>

username: MrRobot
cooked: <p>I tried two methods for MNIST images generated by copying them 3 times.</p>
<ul>
<li>Just change the output layers but freeze all previous layers</li>
<li>Change the output layer and fine-tune all other layers</li>
</ul>
<p>Additional fine-tuning gives an about 4% accuracy boost, from around 95% to 99% for individual category.</p>