akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Random_seed caution

username: Juna
cooked: <p>Hi<br>
I get this caution when I use random seed.</p>
<p>UserWarning: You have chosen to seed training. This will turn on the CUDNN deterministic setting, which can slow down your training considerably! You may see unexpected behavior when restarting from checkpoints.</p>
<p>What I want to do is to get some multiple outputs with the model I get after training with a random seed.<br>
Plus I want to save the model first before I get multiple outputs.</p>
<ol>
<li>Should I use the same random seed when I get the outputs?</li>
<li>If I should, then is this the only thing I should do?</li>
<li>With the answers to the above questions, can I get the same outputs whenever I use the same input? (e.g. the outputs I get with the same input and the same model but multiple times)</li>
</ol>

username: Juna
cooked: <p>By the way, I used the imagenet classifier example in pytorch github tutorial.</p>

username: ptrblck
cooked: <ol>
<li>If all runs should return the same output, you would have to use the same seed. A seed is usually used to get reproducible results.</li>
<li>Using the script, it seems other deterministic options will also be enabled. Have a look at the <a href="https://pytorch.org/docs/stable/notes/randomness.html">Reproducibility docs</a> for more information</li>
<li>If you don’t use the mentioned non-deterministic ops (from the repro docs), you should get the same answers.</li>
</ol>

username: Juna
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="3" data-topic="61623">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>mentioned non-deterministic ops</p>
</blockquote>
</aside>
<p>It seems to work properly. Thanks a lot!!</p>
CNN With 3 Conv Layer

username: Davood_Kalami
cooked: <p>Hi<br>
I want to make a CNN with 3 convolution layers and 3 pooling layers, the net is like this :<br>
convolution-pooling-convolution-pooling-convolution-pooling-fc1-fc2<br>
the pooling is 2x2 and its stride is 1 and the convolutions are 5x5<br>
the output sizes are respectively : 32,64,128,512,64<br>
my network code is this :</p>
<p>class Net(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(Net, self).<strong>init</strong>()<br>
self.conv1 = nn.Conv2d(3, 32, 5)<br>
self.pool = nn.AvgPool2d(2, 2, 1)<br>
self.conv2 = nn.Conv2d(32, 64, 5)<br>
self.conv3 = nn.Conv2d(64, 128, 5)<br>
self.fc1 = nn.Linear(128 * 5 * 5, 512)<br>
self.fc2 = nn.Linear(512, 64)</p>
<pre><code>def forward(self, x):
    x = self.pool(F.relu(self.conv1(x)))
    x = self.pool(F.relu(self.conv2(x)))
    x = self.pool(F.relu(self.conv3(x)))
    x = x.view(-1, 128 * 5 * 5)
    x = F.relu(self.fc1(x))
    x = F.relu(self.fc2(x))
    return x
</code></pre>
<p>I got this error :   x = x.view(-1, 128 * 5 * 5)<br>
RuntimeError: invalid argument 2: size ‘[-1 x 3200]’ is invalid for input with 512 elements at /opt/conda/conda-bld/pytorch_1524577177097/work/aten/src/TH/THStorage.c:37</p>

username: ptrblck
cooked: <p>What is you input size? Note that you’ll lose 2 pixels at each conv layer, as your <code>kernel_size=5</code> and you don’t use any <code>padding</code>.<br>
Also the arguments for <code>nn.AvgPool2d</code> are <code>kernel_size</code>, <code>stride</code>, <code>padding</code>. So currently you use <code>stride=2</code> and add <code>padding=1</code>.</p>
<p>To easily debug your shape, just add <code>print(x.shape)</code> before the <code>.view</code> op to see the current output shape.</p>

username: Davood_Kalami
cooked: <p>My input size is 32x32<br>
What should be the numbers in x.view?</p>

username: zeakey
cooked: <p>Just as <a class="mention" href="/u/ptrblck">@ptrblck</a> pointed out, <code>print(x.shape)</code> before <code>.view()</code> to check the numbers.</p>

username: Davood_Kalami
cooked: <p>Thanks , it got solved</p>

username: Rengoku
cooked: <p>How did you resolve it?</p>
Embedding in the NLP tutorial

username: SamRoss
cooked: <p>I am trying to follow the third NLP tutorial <a href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html" rel="nofollow noopener">here</a>. I’m having trouble understanding the relationship between the input and hidden feature sizes in the encoder module.</p>
<p>The relevant code is:</p>
<pre><code class="lang-auto">
class EncoderRNN(nn.Module):
    def __init__(self, input_size, hidden_size):
        super(EncoderRNN, self).__init__()
        self.hidden_size = hidden_size

        self.embedding = nn.Embedding(input_size, hidden_size)
        self.gru = nn.GRU(hidden_size, hidden_size)

    def forward(self, input, hidden):
        embedded = self.embedding(input).view(1, 1, -1)
        output = embedded
        output, hidden = self.gru(output, hidden)
        return output, hidden

    def initHidden(self):
        return torch.zeros(1, 1, self.hidden_size, device=device)
</code></pre>
<p>Won’t the shape of <code>embedded</code> be <code>(1, 1, input_size * hidden_size)</code>? If so, how can this be fed into a <code>GRU</code> expecting and input of shape <code>(1, 1, hidden_size)</code>?</p>
<p>I’m trying something like:</p>
<pre><code class="lang-auto">
model = EncoderRNN(input_size=5, hidden_size=2)
model(input=torch.tensor([[[1,2,3,4,0]]]), 
      hidden=torch.tensor([[[0,0,0,0,0,0]]]))
</code></pre>
<p>but I get <code>RuntimeError: input.size(-1) must be equal to input_size. Expected 2, got 10</code>, as expected.</p>
<p>So I am probably not calling <code>model</code> correctly. What am I missing?</p>
Why does CenterCrop allows me to save my images?

username: alx
cooked: <p>Hey guys, I’m trying to resize all my images to 256. I modified the code I was using to train my model as follow:</p>
<pre><code class="lang-auto">from torchvision import transforms

trans = transforms.Compose([
    transforms.Resize(256),
    transforms.ToTensor(),
])
</code></pre>
<p>And saving by using</p>
<pre><code class="lang-auto"> for img_batch, label_batch, path_batch in dataloader:
    for img, path in zip(img_batch, path_batch):
      save_image(img, filename)
</code></pre>
<p>But that doesn’t work, it says there’s an issue with the dimensions. If I add a CenterCrop to my transformations it works.</p>
<pre><code class="lang-auto">    transforms.Resize(256),
    transforms.CenterCrop(256),
    transforms.ToTensor(),
</code></pre>
<p>Can anyone explain this behavior?</p>
<p>Thanks</p>

username: ptrblck
cooked: <p><a href="https://pytorch.org/docs/stable/torchvision/transforms.html#torchvision.transforms.Resize">transforms.Resize</a> uses a <code>sequence</code> of sizes or an <code>int</code>:</p>
<blockquote>
<p>Desired output size. If size is a sequence like (h, w), output size will be matched to this. If size is an int, smaller edge of the image will be matched to this number. i.e, if height &gt; width, then image will be rescaled to (size * height / width, size)</p>
</blockquote>
<p>Since you are providing an <code>int</code> I would assume the smaller edge of the image will be matched. Could this be the issue?</p>

username: alx
cooked: <p>Maybe, so how can I resize and save without using crop?</p>

username: ptrblck
cooked: <p>Providing both numbers (<code>Resize((256, 256))</code>) will return this exact spatial size.</p>
Why isn't torch::jit::script::Module::forward const?

username: kb1ooo
cooked: <p>Wondering why the forward method isn’t const?  And since it’s not, what’s the proper way to get back to the loaded state of the module without having to actually reload?</p>
<p>Thanks,<br>
Marc</p>

username: yf225
cooked: <p><a class="mention" href="/u/kb1ooo">@kb1ooo</a> Would you like to post this question in <a href="https://discuss.pytorch.org/c/jit">https://discuss.pytorch.org/c/jit</a>? The JIT team monitors that channel more closely and we would be able to have a response there.</p>

username: kb1ooo
cooked: <p><a class="mention" href="/u/yf225">@yf225</a> ok, thanks.  Is there a way to just move it there or do I repost?</p>

username: yf225
cooked: <p><a class="mention" href="/u/kb1ooo">@kb1ooo</a> Reposting might be the easiest, thanks and sorry for the bad experience <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"></p>

username: kb1ooo
cooked: <p><a class="mention" href="/u/yf225">@yf225</a> oops, ok I saw that I could change the category so that’s what I did.  Maybe not as effective as a repost.</p>

username: driazati
cooked: <p><code>Module</code>s can have stateful data attached to themselves as parameters or attributes. These can be mutated during <code>forward</code> (or any other method that is run on the <code>Module</code>), so it doesn’t make sense to have it be <code>const</code>.</p>
<p>We don’t have any way of backing out changes to <code>Module</code> state, so re-loading is the only way to get back a known state. We may in the future have a user-accessible way of stating that some method is pure and has no mutable operations, but that’s still something we’re discussing.</p>

username: kb1ooo
cooked: <p>Ok, understood, thank you.</p>
Transforms in c++

username: varghese_alex
cooked: <p>Hi,</p>
<p>I trained a model using transforms 1) Resize (128,64), 2) To Tensor 3) Normalizing using ImageNet stats.</p>
<p>I would like to now port the model to C++. Hence I need to use transforms in C++. Could someone please help me with this?</p>

username: ptrblck
cooked: <p><a class="mention" href="/u/krshrimali">@krshrimali</a> released some blog posts about the PyTorch C++ API and <a href="https://krshrimali.github.io/Blog-Dogs-VS-Cats/">this one</a> could be a good starter.</p>
Weird behaviour of Training loss

username: mojzaar
cooked: <p>I have modeled a NN which contains conv3D, LSTM, and conv2D. I’m using MSELoss to map a tensor data with 3 dimension as data dimension to a 2d dimension data where each pixel has a positive value at target. Therefore, the output at each pixel should have the same value. The training loss plot for a single data is something like following picture.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/3/9388c71194afec0b21cc4566b4d578dd393848d0.png" alt="index" data-base62-sha1="l39er1S2WcHFSr6mTmRZWHIZhp6" width="376" height="273"></p>
<p>I normalized the input data. have used xavier uniform to initialize the weights. Moreover, I decrease the learning rate with scheduler and clip the gradients with</p>
<pre><code class="lang-auto">torch.nn.utils.clip_grad_norm_(model.parameters(), 1000)
</code></pre>
<p>Any suggestion?</p>
Error : one of the variables needed for gradient computation has been modified by an inplace operation

username: YeongHwa_Jin
cooked: <p>I implemented the code like below:</p>
<pre><code class="lang-auto">for layer in self.model.features:
    if isinstance(layer, torch.nn.Conv2d):
        kernel_vec = layer.weight.reshape(layer.out_channels, -1)
        kernel_norm = torch.norm(kernel_vec, p=1, dim=1)
        for i in range(kernel_norm.shape[0]):
            kernel_vec[i] = kernel_vec[i] / kernel_norm[i]
        inner_prod = torch.mm(kernel_vec, kernel_vec.t())
        inner_prod = inner_prod - torch.diag(inner_prod)
        inner_loss = torch.sum(torch.abs(inner_prod))
        loss += inner_loss

optimizer.zero_grad()
loss.backward()
optimizer.step()
</code></pre>
<p>But it doesn’t work. I have the error code like this:</p>
<blockquote>
<p>one of the variables needed for gradient computation has been modified by an inplace operation</p>
</blockquote>
<p>I found this error occurred by the in-place operation. But I don’t know how fix the code…<br>
Could you help me?</p>

username: albanD
cooked: <p>Hi,</p>
<p>Which version are you using? This error in the latest version contains much more informations than that <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
Also you can enable the anomaly detection to know which forward methods had a Tensor it needs to use that was modified inplace.</p>

username: YeongHwa_Jin
cooked: <p>Thank you for reply.<br>
I use the pytorch 1.2<br>
Full error code is this:</p>
<blockquote>
<p>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [4608]], which is output 0 of SelectBackward, is at version 514; expected version 513 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).</p>
</blockquote>
<p>and if you don’t mind, could you let me know how use anomaly detection?</p>

username: albanD
cooked: <p>You might want to upgrade your pytorch version to get a more useful error message.</p>
<p>The doc is <a href="https://pytorch.org/docs/stable/autograd.html#torch.autograd.detect_anomaly">here</a>, but you can simply add at the beginning of your file (before the forward pass) <code>torch.autograd.set_detect_anomaly(True)</code>.</p>

username: YeongHwa_Jin
cooked: <p>Thank you for your advice!</p>
Why the use of nn.Sequential(nn.Sequential())

username: Upgrade_Yourself
cooked: <p>Hi, I need some help understanding this code:</p>
<blockquote>
<p>model = nn.Sequential(<br>
nn.Sequential(full.layer0, full.layer1, full.layer2, <strong>full.layer3[:3])</strong>,<br>
nn.Sequential(<strong>full.layer3[3:]</strong>, full.layer4),<br>
get_head(2048, n_classes))</p>
</blockquote>
<p>Why not do something like:</p>
<blockquote>
<p>model = nn.Sequential(<br>
full.layer0, full.layer1, full.layer2, <strong>full.layer3,</strong><br>
get_head(2048, n_classes))</p>
</blockquote>
<p>Thank you for the help</p>

username: albanD
cooked: <p>Hi,</p>
<p>The only possible reason I can see is if you want to only train the last layers and you want to pass these parameters to the optimizer with <code>model[1].parameters()</code>.<br>
Otherwise. I don’t really see a good reason to do this.</p>
How to load a dataset from google drive to google colab

username: hunar
cooked: <p>FileNotFoundError: [Errno 2] No such file or directory: ‘/content/gdrive/My Drive/train’</p>
<p>i’ve tried to load a dataset from google drive to google colab , but it provide the error above ?! folder name in my google drive is( train )</p>
<p>train_dataset = datasets.ImageFolder(root="/content/gdrive/My Drive/train" , transform=train_transfor)</p>
<p>thanks</p>
Understanding need for reinitializing weights in code

username: Upgrade_Yourself
cooked: <p>I’ve been reading a piece of code found on github. I specifically don’t understand why we keep on verifying if the model’s layers have got weights and bias (<strong>is not None</strong>) when we’ve just created it. And why would we do (nn.init.uniform_(module.weight)) if there are weights. What’s the logic here? Thanks</p>
<blockquote>
<p>def get_head(nf: int, n_classes):</p>
</blockquote>
<pre><code>    model = nn.Sequential(
    nn.ReLU(),
    nn.AdaptiveAvgPool2d(1),
    Flatten(),
    nn.BatchNorm1d(nf),
    nn.Dropout(p=0.25),
    nn.Linear(nf, n_classes)
    )
    for i, module in enumerate(model):
        if isinstance(module, (nn.BatchNorm1d, nn.BatchNorm2d)):
             if module.weight is not None:
                  nn.init.uniform_(module.weight)
             if module.bias is not None:
                  nn.init.constant_(module.bias, 0)
        if isinstance(module, nn.Linear):
             if getattr(module, "weight_v", None) is not None:
                  print("Initing linear with weight normalization")
                  assert model[i].weight_g is not None
             else:
                 nn.init.kaiming_normal_(module.weight)
                 print("Initing linear")
             if module.bias is not None:
                 nn.init.constant_(module.bias, 0)
</code></pre>
<blockquote>
<p>return model</p>
</blockquote>

username: albanD
cooked: <p>Hi,</p>
<p>This is because the batchnorm can be created without the affine transformation (see the code <a href="https://github.com/pytorch/pytorch/blob/f7b12a985812f278b225fd58549ac95c7864a22c/torch/nn/modules/batchnorm.py#L26-L31">here</a>). In that case, the weight and bias will be None.<br>
For the linear layer, the <code>.bias</code> can be None for the same reason.<br>
For the <code>weight_v</code> attribute. I guess this is something that the user added himself as it does not exist in the original Linear layer.</p>

username: albanD
cooked: <p><code>torch.Tensor(num_features)</code> returns uninitialized memory. So you definitely want to put some sensible values in there before using it <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Trouble with simple RNN

username: SamRoss
cooked: <p>I’m trying to create my own toy example. I want a single RNN unit that takes a 1d input and has a 1d hidden state which gets updated.</p>
<p>I set the rnn block to be</p>
<p><code>rnn_b = nn.RNN(input_size=1, hidden_size=1, num_layers=1, bias=False)</code></p>
<p>Then my net is as follows:</p>
<pre><code class="lang-auto">class MyRNN(nn.Module):
    def __init__(self, input_size, hidden_size):
        super(MyRNN, self).__init__()
        self.input_size = input_size
        self.hidden_size = hidden_size

    def forward(self, input, hidden):
        out, hidden = rnn_b(input, hidden)  # update the hidden state
        return hidden

</code></pre>
<p>Next, I do</p>
<p><code>rnn = MyRNN(1, 1)</code></p>
<p>Now when I try to train, I do</p>
<pre><code class="lang-auto">
criterion = nn.MSELoss()
learning_rate = 0.0005
hidden = torch.tensor([[[1.0]]])
def train(input, target):
    rnn.zero_grad()
    loss = 0
    global hidden
    hidden = rnn(input, hidden)
    l = criterion(hidden, target)
    loss += l
    loss.backward()
    for p in rnn.parameters():
        p.data.add_(-learning_rate, p.grad.data)
    return hidden, loss
</code></pre>
<p>I would like to train it to learn some sequence of numbers, say</p>
<pre><code class="lang-auto">inputs = [1, 2, 3, 4, ..., 1000]
targets = [2, 3, ..., 1000]
for i in range(0, 999):
    hidden, loss = train(torch.tensor([[[inputs[i]]]]), torch.tensor([[[targets[i+1]]]]))
</code></pre>
<p>This gives me the error: "Trying to backward through the graph a second time, but the buffers have already been freed. " I have seen many threads about this error, but I don’t quite understand it. Besides that, I get the feeling I am not doing things correctly here, so please give me some advice here <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: richard
cooked: <p>After the first call to <code>train</code>, <code>hidden</code> becomes a tensor that has <code>requires_grad=True</code> and is a part of the computation graph.</p>
<p>During the second call to <code>train</code>, the <code>backward</code> step tries to propagate gradients through <code>hidden</code>. However, since we already did a backward(), the computation graph of <code>hidden</code> doesn’t exist anymore.</p>
<p>Perhaps you did not mean to propagate gradients through <code>hidden</code> during the second call to <code>train</code>? Something like the following:</p>
<pre><code class="lang-auto">hidden, loss = train(torch.tensor([[[inputs[i]]]]), torch.tensor([[[targets[i+1]]]]))
hidden = hidden.detach_()  # detach it from the computation graph for the next iteration of training
</code></pre>
PyTorch weight initialization

username: Er_Hall
cooked: <p>Hi all,</p>
<p>Can I ask you something about weight initialization in PyTorch because I have been confused?</p>
<p>First of all, is it mandatory to initialize weights in PyTorch or can the framework initialize the layers for us?</p>
<p>Also,  because many posts are outdated, in default initialization what kind of initialization the framework use? Xavier?</p>
<p>Moreover, all layers provide default initialization (LSTMS, GRUs, CNNs layers etc.)</p>
<p>Finally, if I use init function to ‘manually’ initialize the layers. Where I must put this function? inside the class where I define the layers?</p>

username: albanD
cooked: <p>Hi,</p>
<p>All of our layers use a default initialization that is the most commonly used in the literature. You can see in the <a href="https://pytorch.org/docs/stable/nn.html#linear">doc</a> what is the default initialization.<br>
If you want to change the default initialization, you can use the <a href="https://pytorch.org/docs/stable/nn.init.html">nn.init</a> module’s functions to change the weights of your modules after creating them. There is no hard rule how to do this. You can simply update the .weight after you create it. Or do whatever feels best for your use case.</p>
Optimize only for slice of tensor (or single entries)

username: Nio
cooked: <p>I’m trying to use Autograd in order to optimize some parameters stored inside a tensor.<br>
Let’s say I have a tensor of size 10 and I would like to optimize for only the first parameter of that tensor.<br>
So I call:<br>
<code>torch.optim.Adam([variables[0]], lr=self.lr)</code></p>
<p>The problem is that when selecting the entry I performed an operation that is tracked by the autograd graph and I don’t have a leaf node anymore.<br>
What can be an easy way to solve this?<br>
I need this kind of setup because my variables contain lots of information that I need to visualize and compute the loss, but I don’t want to optimize for all the entries in the tensor.</p>
<p>Help is appreciated.</p>

username: JuliousHurtado
cooked: <p>Hello Nio,</p>
<p>I am not sure if I understand the problem, but you can try to create two tensors, one with the variables that you need to optimize and the other with the other set of variables, and when using the vector concatenate those two tensors.</p>
<p>I don’t know if that applicable to your problem, but maybe it could help.</p>
RuntimeError: Given groups=1, weight of size 16 16 3 3, expected input[16, 64, 222, 222] to have 16 channels, but got 64 channels instead

username: Ravikant_Gautam
cooked: <p>I try to run the following program for image classification problems in Pytorch. I am new to PyTorch and I am not getting what’s wrong with the code. I tried reshaping the images but no help. I am running this code with Cuda. I have around 750 classes and 10-20 images in one class. My dataset is a benchmark dataset and every image has a size of 60*160.</p>
<p>Getting this error and I don’t know where to make changes.  <strong>Given groups=1, the weight of size 16 16 3 3, expected input[16, 64, 222, 222] to have 16 channels but got 64 channels instead.</strong></p>

username: albanD
cooked: <p>The input you give is of size [16, 64, 222, 222]. Where by convention, the dimensions are <code>batch, channels, heigh, width</code>.<br>
As you can see, your image has 64 channels.<br>
But the convolution you give it to has been created to expect 16 channels for the input. Hence the error.</p>
<p>You need to make sure that the number of channels expected by the convolution and in the image you give are the same.</p>
What is statistical power of study

username: PANKAJ_JADHAV
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/7/071165351af26d23a46fd4044a2f75ba48556be2.png" data-download-href="https://discuss.pytorch.org/uploads/default/071165351af26d23a46fd4044a2f75ba48556be2" title="19%20PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/7/071165351af26d23a46fd4044a2f75ba48556be2_2_690x404.png" alt="19%20PM" data-base62-sha1="10wBV94FWe2LG1SVdYhWSPkK5N0" width="690" height="404" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/7/071165351af26d23a46fd4044a2f75ba48556be2_2_690x404.png, https://discuss.pytorch.org/uploads/default/original/3X/0/7/071165351af26d23a46fd4044a2f75ba48556be2.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/7/071165351af26d23a46fd4044a2f75ba48556be2.png 2x" data-dominant-color="E8EEF3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">19%20PM</span><span class="informations">1031×605 121 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div> I am classifying sleep stages into five classes wake,N1, N2, N3 and REM. The confusion matrices obtained using six different classifier is attached. What is statistical power of study? How to compute it from confusion matrices.</p>
Three dimensional input for DataLoader

username: Raphikowski
cooked: <p>Hi,</p>
<p>I have the following set-up:<br>
I have a 3D data tensor with the axes (different_samples, timeseries_length, feature_dimensionality), i.e.,<br>
the first dimension is the number of different timeseries samples I have, the second dimension is the length of the timeseries’ and the third dimension is the dimensionality of each timestep.<br>
I want to plug this 3D tensor into the Dataloader. I have written my own BatchSampler, however, only for a 2D tensor containing a single timeseries.</p>
<p>What I want to do now, however, is that my custom BatchSampler first selects an index to choose one of the timeseries in the 3D tensor (i.e., an index for the first dimension) and then selects a starting index for the batch (i.e., an index for the second dimension) within the timeseries that was selected at first, which means that the BatchSampler has to return a 2D object.</p>
<p>My BatchSampler (which is still only able to handel 2D data input) currently returns a list of indices (I think the Sampler classes always have to return a list). And as far as I understand, this list of indices is then passed to the <strong>getitem</strong> method of the dataset object. So for my new set-up with a 3D data tensor, I would now require my BatchSampler to return something like a 2D object: The first dimension providing the index that selects the specific timeseries, and the second index dimension providing the indices that select the batch from that chosen timeseries. However, there are no 2D lists…<br>
So how would I do that?<br>
Or is there a completely different way to do so?<br>
Help is very much appreciated!</p>
Self defined loss print bracket

username: coincheung
cooked: <p>HI,</p>
<p>I am defining an customer operator, with backward:</p>
<pre><code class="lang-python">class LabelSmoothSoftmaxCEFunction(torch.autograd.Function):

    @staticmethod
    def forward(ctx, logits, label, lb_smooth):
        num_classes = logits.size(1)
        scores = torch.softmax(logits, dim=1)
        logs = torch.log(scores)
        lb_one_hot = torch.zeros_like(logits).scatter_(1, label.unsqueeze(1), 1)
        label = (1. - lb_smooth) * lb_one_hot + lb_smooth / num_classes

        ctx.lb_smooth = lb_smooth
        ctx.num_classes = num_classes
        ctx.scores = scores
        ctx.lb_one_hot = lb_one_hot

        loss = -torch.sum(logs * label, dim=1)
        return loss.mean()

    @staticmethod
    def backward(ctx, grad_output):
        print(ctx.saved_variables)
        lb_smooth = ctx.lb_smooth
        num_classes = ctx.num_classes
        scores = ctx.scores
        lb_one_hot = ctx.lb_one_hot
        loss = scores - lb_smooth / num_classes
        #  loss[lb_one_hot == 1] = -loss[lb_one_hot]
        loss = torch.where(lb_one_hot == 1, -loss, loss)
        return loss, None, None

class LabelSmoothSoftmaxCEV2(nn.Module):

    def __init__(self, lb_smooth):
        super(LabelSmoothSoftmaxCEV2, self).__init__()
        self.lb_smooth = lb_smooth

    def forward(self, logits, label):
        return LabelSmoothSoftmaxCEFunction.apply(logits, label, self.lb_smooth)
</code></pre>
<p>When I tried to print the loss, I got:</p>
<pre><code class="lang-python">loss = criteria(logits, label)
print(loss.item())
</code></pre>
<p>()<br>
8.125082015991211<br>
()<br>
8.200040817260742</p>
<p>Why is there a <code>()</code> when I print the value, how could I make it work as other loss?</p>

username: coincheung
cooked: <p>Sorry, It is my fault, I have solved this:)</p>
LibTorch with cuda raising an exception on windows

username: erict
cooked: <p>Hi,</p>
<p>i am on win 10 vs 2017 LibTorch 1.3 with debug cuda 10.1. I want to train a model with pytorch c++. With cpu is ok but when i’am trying to use Gpu i got</p>
<pre><code class="lang-auto">Unhandled exception at 0x00007FFD2A759129 in train.exe: Microsoft C++ exception: c10::Error at memory location 0x000000085539C5E0
</code></pre>
<p>cuda is availabe because i check it with if (torch::cuda::is_available()).</p>
<p>Code to reproduce the error</p>
<pre><code class="lang-auto">
struct TestImpl : nn::Module {


	TestImpl() : conv1(register_module("conv1", nn::Conv2d(nn::Conv2dOptions(1, 32, 4).stride(2).padding(1)))){

		register_parameter("prelu1", prelu1.fill_(0.25));

	
	
	}

	torch::Tensor forward(torch::Tensor x) {
	
		x = torch::prelu(x, prelu1);

		return x;
	
	}


	nn::Conv2d conv1;

	torch::Tensor prelu1 = torch::ones({ 1 });



};
TORCH_MODULE(Test);


int main{
Test testModel;
testModel-&gt;to(torch::DeviceType::CUDA); //Exception Here
}

</code></pre>
<p>Someone have an idea how to solve?</p>
<p>Thanks</p>
Storing a torch tensor without pickle

username: Raphikowski
cooked: <p>Hi all,</p>
<p>I want to store a tensor containing my results to disc. However, when using torch.save and torch.load it seems like I have to store the tensor in the same directory as the python file from which I want to load the saved file, otherwise it wont load the file. This seems to come from the fact, that pytorch internally uses pickle.<br>
<a href="https://pytorch.org/tutorials/beginner/saving_loading_models.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/saving_loading_models.html</a></p>
<p>I therefore want to store my torch tensor in a different way (I dont need to take care of grad), so which would be the recommended way (simply storing the values from the tensor and the loading it again into a torch tensor)?</p>

username: dejanbatanjac
cooked: <blockquote>
<p>However, when using torch.save and torch.load it seems like I have to store the tensor in the same directory as the python file from which I want to load the saved file,</p>
</blockquote>
<p>Have you tried to set the path already to something different ?</p>
<pre><code class="lang-auto">torch.save(the_model, PATH)
the_model = torch.load(PATH)
</code></pre>

username: Raphikowski
cooked: <p>So the problem that pickle and torch save and load has is that the directory structure that was present at the time when the tensor was stored, has to be maintained otherwise it cannot be loaded as I understood it. So I really just want to know a simple way to store the data contained in a tensor and how I can make a tensor out of it again when loading it</p>

username: dejanbatanjac
cooked: <p>Yes, when you <code>import torch</code> it will <code>import pickle</code> for you and you don’t need to call <code>pickle.dump()</code> and <code>pickle.load()</code> since <code>torch.save()</code> and <code>torch.load()</code> will wrap <code>pickle.dump()</code> and <code>pickle.load()</code> for you.</p>
<p>So strange your question is.</p>

username: Raphikowski
cooked: <p>But isnt it possible to just store it as a CSV or JSON or sth alike? And if so, which is probably the most usefull way?</p>

username: dejanbatanjac
cooked: <p>Maybe <a href="https://docs.python-guide.org/scenarios/serialization/" rel="nofollow noopener">this</a> is what You may try.</p>
ONXX model and wrong ir_version

username: Mackou
cooked: <p>Hello everyone.</p>
<p>I have converted a tensorflow model to ONXX but when I try to use it I get the following error : <a href="https://pastebin.com/VNMc2EH5" rel="nofollow noopener">https://pastebin.com/VNMc2EH5</a></p>
<p>It seems that the ir_version of my model is 0.0.4 and the software I need needs 0.0.3. How do I do to convert my model to a right ir_version ?</p>
<p>Thanks a lot !</p>
Different filters for 2dConv

username: razvanc92
cooked: <p>I’m having an input of shape (B(atch), F(features), N(odes), T(timestamps)). Right now if I apply a 2d convolution with a kernel of shape (1,2) I will have a total of (F_out, F_in, 1,2) weights to learn which is alright. I want to extend this so that for each Node in the input I have it’s own 1,2 filter. Does any of you have any idea where should I start from?</p>

username: ptrblck
cooked: <p>If seems you are using the <code>Nodes</code> and <code>Timestamps</code> dimensions as the spatial dimension at the moment.<br>
If you just need a <code>1x2</code> filter, you could <code>permute</code> and <code>reshape</code> the input, so that <code>Nodes</code> is accumulated in dim0 together with the <code>Batch</code> and apply a <code>nn.Conv1d</code> layer.</p>

username: razvanc92
cooked: <p>I don’t think that I quite understand your solution. let’s take a simple example, 2 nodes with 2 features each, 2 time stamps and a kernel of size 1x2 filled with ones. As you can see from the image the two timestamps are convolved into one, but there is no information going from node 1 to node 2 which is intended. Now I would like for each node to have it’s own filter, so features 1,2 are going to be multiplied with a filter and 2,4 with another, and same thing for the other channel.</p>
<p>Thank in advance for the response, and I really appreciate your help.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/8/f89554f50dd66794bc6018403386d5be1fbb21aa.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/f89554f50dd66794bc6018403386d5be1fbb21aa" title="DSC_0695"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/8/f89554f50dd66794bc6018403386d5be1fbb21aa_2_345x194.jpeg" alt="DSC_0695" data-base62-sha1="zt4lZwuNpEVo0x5KUdGZqRkipWG" width="345" height="194" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/8/f89554f50dd66794bc6018403386d5be1fbb21aa_2_345x194.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/f/8/f89554f50dd66794bc6018403386d5be1fbb21aa_2_517x291.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/8/f89554f50dd66794bc6018403386d5be1fbb21aa_2_690x388.jpeg 2x" data-dominant-color="837E7A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">DSC_0695</span><span class="informations">3840×2160 2.09 MB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
General unfold function with dilation

username: deJQK
cooked: <p>PyTorch provide the powerful function unfold, through both <a href="https://pytorch.org/docs/stable/nn.html#id23" rel="nofollow noopener">torch.nn.functional.unfold</a> and the builtin function for tensor <a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor.unfold" rel="nofollow noopener">torch.Tensor.unfold</a>. It seems the latter is easier to use, and it is more general as it is not restricted to 4D tensor. However, the former implementation support dilation, while the latter does not.</p>
<p>Does anyone has method to augment the builtin function implementation to make it support dilation?</p>

username: SimonW
cooked: <p>They are really doing different things. nn.Unfold gets the sliding window from image tensors. Tensor.unfold just unfolds along a single dimension. (You can think of the former as doing the latter along dim 2 and dim 3 at the same time). For Tensor.unfold, step arg is probably what you want.</p>

username: deJQK
cooked: <p>Thanks. step arg is stride, not dilation. Also, doing nn.unfold twice does not give the same result as tensor.unfold.</p>

username: ToughMind
cooked: <p>I am looking for a <code>unfold</code> function with dilation too. Hope some one can giva more imformation!</p>

username: tom
cooked: <p>What does nn.Unfold not do what you need when you give it a dilation parameter?</p>

username: ToughMind
cooked: <p>After some research and trail，nn.Unfold with stride parameter seems to do what I want. I am still trying…</p>
What can i do to make train loss, accuracy and validation loss, accuracy normal?

username: yuaner
cooked: <p>hello，it is my first time to ask a question in pytorch forum!<br>
when i train a 3dunet in cremi dataset to segment synapses, lower accuracy is better (update now, best score is about 50). However, my results are very bad just like the  picture below.<br>
i want know what reason will result in so bad result ?<br>
and how can i solve ?<br>
thank you !<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/f/8f4c94685e82af4763c4ad740ce45ec0c39461d6.png" data-download-href="https://discuss.pytorch.org/uploads/default/8f4c94685e82af4763c4ad740ce45ec0c39461d6" title="%E6%88%AA%E5%9B%BE"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/f/8f4c94685e82af4763c4ad740ce45ec0c39461d6_2_690x312.png" alt="%E6%88%AA%E5%9B%BE" data-base62-sha1="krGlFIv2WH7F46Q3k4L7rLDZ3Rc" width="690" height="312" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/f/8f4c94685e82af4763c4ad740ce45ec0c39461d6_2_690x312.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/f/8f4c94685e82af4763c4ad740ce45ec0c39461d6_2_1035x468.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/f/8f4c94685e82af4763c4ad740ce45ec0c39461d6.png 2x" data-dominant-color="FBF9F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">%E6%88%AA%E5%9B%BE</span><span class="informations">1112×503 105 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
How to add Scalar input to AlexNet

username: B4D_0M3N
cooked: <p>Hello,<br>
I am trying to retrain AlexNet, however, in order to check if I could improve accuracy, I would like to feed it a scalar information, namely before the last fully connected layer. Nonetheless, I cannot find any way to flag “input layers” using pyTorch, so I have no idea how to achieve this. Does anybody tried anything similar?</p>

username: ptrblck
cooked: <p>You can simply modify the <code>forward</code> method of your model and pass an arbitrary number of parameters.<br>
PyTorch does not use specific “input” tensors etc.</p>
CNN never converges (implementation issue suspected)

username: jb4earth
cooked: <p>Hi all, I am having trouble getting this network to work as desired.  I have tried so many iterations of this model and yet cannot get a reasonable error (it never fits, can’t even get it to overfit).</p>
<p>Where have I gone wrong?  Any help would be greatly appreciated</p>
<p>For reference, there are 12 input ‘images’ (they’re actually surface elevation) of shape 49,9 and 12 labels of shape 1,9.</p>
<p>In case you’d like to see the data or all of my failed attempts: <a href="https://gitlab.com/jb4earth/effonn/" rel="nofollow noopener">https://gitlab.com/jb4earth/effonn/</a></p>
<pre><code class="lang-auto">net = []
class Net(torch.nn.Module):
    def __init__(self, kernel_size):
        super(Net, self).__init__()
        mid_size = (49*49*9)
        self.predict = torch.nn.Sequential(
            nn.Conv2d(
                        in_channels=1,
                        out_channels=mid_size,
                        kernel_size=kernel_size,
                        stride=1,
                        padding=(0, 0)
                    ),
            nn.ReLU(),
            nn.MaxPool2d(1),
            nn.ReLU(),
            nn.Conv2d(
                        in_channels=mid_size,
                        out_channels=1,
                        kernel_size=kernel_size,
                        stride=1,
                        padding=(0, 0)
                    ),
            nn.ReLU()
        )
        

    def forward(self, x):
        x = self.predict(x)
        return x

def train_network(x,y,optimizer,loss_func):
    prediction = net(x)    
    loss = loss_func(prediction, y.squeeze())     
    optimizer.zero_grad()  
    loss.backward()     
    optimizer.step()    
    return prediction, loss


net = Net((1,1))
optimizer = torch.optim.Adam(net.parameters(), lr=0.01)
loss_func = torch.nn.MSELoss()
cnt = 0
t = True
while t == True:
    # get_xy in place of DataLoader
    (x,y) = get_xy(input_data,output_data,cnt)
    # x.shape is 1,1,49,9
    # y.shape is 1,1,1,9
    
    # train and predict
    (prediction,loss) = train_network(x,y,optimizer,loss_func)
    
    # prediction shape different than desired so averaging all results
    prediction_ = torch.mean(prediction)

    # only 12 IO's so loop through 
    cnt += 1
    if cnt &gt; 11:
        cnt = 0
</code></pre>

username: ptrblck
cooked: <p>I’m not familiar with your use case, but I assume you are working on some regression problem using images.</p>
<p>Based on your model definition, if seems you are increasing the number of channels quite aggressively (going from 1 input channel to 21609 output channels), which is quite uncommon.</p>
<p>Also, <code>nn.MaxPool2d(1)</code> won’t have any effect, as the kernel size is set to 1, which will basically just return the same values. The second <code>ReLU</code> activation also won’t change anything.</p>
<p>I would recommend to increase the number of channels a bit smoother and also to use other base models such as <code>VGG16</code> as a starter and adapt them to your use case.</p>
Inceptionv3 out of GPU memory error, even with memory management

username: whydoineedausername
cooked: <pre><code class="lang-auto">inceptionv3.to(device) # 'cuda'
inceptionv3.eval()

inceptionv3.requires_grad_ = False

output = []

for batch in loader:
    torch.cuda.empty_cache()
    out = inceptionv3(batch[0].to(device))
    output.append(out.detach().cpu().numpy())
    del out
</code></pre>
<p>Somehow, this still winds up with a memory error. Adjusting batch size makes no difference.</p>
<p>Let me add that I’m using the facenet modification of inception v3, and there might be some error internal to the way the model was modified for the facenet data.</p>

username: ptrblck
cooked: <p><code>inceptionv3</code> itsemf doesn’t have the <code>requires_grad</code> attribute, which is an attribute for <code>tensors</code>.<br>
Setting it to <code>False</code> will just create a new attribute without any effect.</p>
<p>If you want to run inference only, you should wrap the code in a <code>with torch.no_grad()</code> block:</p>
<pre><code class="lang-python">with torch.no_grad():
    out = inceptionv3(batch)
</code></pre>
<p>This will save some memory by avoiding to store the intermediate activations, which would be needed for the backward pass.</p>
<p>Calling <code>torch.cuda.empty_cache</code> shouldn’t give you more memory, as it will only clear the cache, which could be reused otherwise.</p>

username: whydoineedausername
cooked: <p>Thank you. Is there a tutorial where these sorts of memory management details are made available that you know of? I haven’t seen anything too advanced or useful with pytorch tutorials for this.</p>

username: ptrblck
cooked: <p>I’m not sure, if we have a tutorial specifically targeting “memory management”.<br>
What would you like to see in such a tutorial from a user’s perspective?</p>

username: whydoineedausername
cooked: <p>Just common pitfalls, explanations about what parts of a model and tensors and parameters and scheduling and so on get loaded to the GPU and can thus accumulate, how to avoid accumulation, etc.</p>
What is the difference between computing on tensor and tensor.data

username: coincheung
cooked: <p>Hi,</p>
<p>I noticed in some code that some people like to compute on the tensor.data, what is its difference compared to compute directly on tensor?</p>
<pre><code class="lang-auto">backup = {}
for name, param in model.named_parameters():
    backup[name] = param.data.clone() # other people's code
    backup[name] = param.clone().detach()  # my implementation if I am on this task
</code></pre>

username: ptrblck
cooked: <p>The usage of <code>.data</code> might yield silent errors and is thus not recommended.<br>
<code>.clone().detach()</code> should work fine, so please stick to your approach. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
One output for binary classification and the loss function in this case

username: Mainul
cooked: <p>Regarding binary Classification, I have two question,</p>
<ol>
<li>
<p>Lets say, I only care about one class, If my model gives one output (using sigmoid) rather than two (using softmax), will it be a better idea. If so, could you please explain the reason?</p>
</li>
<li>
<p>For one output, what loss function should I use? It seems that the PyTorch implementation of cross-entropy is not suitable for this. Because, according to the official document, as <code>target</code>  I have to provide the <code>indices</code> of the correct classes and for one output that is not possible. What should I do in this case? Is there any option I am missing?</p>
</li>
</ol>

username: Mainul
cooked: <p>Answer to the question 2: use <a href="https://pytorch.org/docs/stable/nn.html#bceloss" rel="nofollow noopener">BCELoss</a></p>

username: KFrank
cooked: <p>Hi Mainul!</p>
<aside class="quote no-group" data-username="Mainul" data-post="1" data-topic="61608" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mainul/48/30342_2.png" class="avatar"> Mainul:</div>
<blockquote>
<ol>
<li>Lets say, I only care about one class, If my model gives one output (using sigmoid) rather than two (using softmax), will it be a better idea. If so, could you please explain the reason?</li>
</ol>
</blockquote>
</aside>
<p>It will be better to use a single output for a binary-classification<br>
problem, rather than two (which you could logically do).  By<br>
doing so you eliminate one redundancy, and the parameters<br>
that go along with it, from your network.  This will provide at<br>
least a modest advantage.</p>
<aside class="quote no-group">
<blockquote>
<ol start="2">
<li>For one output, what loss function should I use? It seems that the PyTorch implementation of cross-entropy is not suitable for this.</li>
</ol>
</blockquote>
</aside>
<p>The natural loss function to use would be <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.BCEWithLogitsLoss" rel="noopener nofollow ugc">BCEWithLogitsLoss</a><br>
without passing your output through a <code>sigmoid()</code>.  If you insist,<br>
for whatever reason, on using a <code>sigmoid()</code>, you would use<br>
<a href="https://pytorch.org/docs/stable/nn.html#torch.nn.BCELoss" rel="noopener nofollow ugc">BCELoss</a>.</p>
<p>Good luck.</p>
<p>K. Frank</p>