akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Does tensors on gpu perform non-arithmetic operations faster than tensors on cpu?

darknight: <p>As we all know, gpu can accelerate arithmetic operations of tensors, such as <code>+ - * /</code>.</p>
<p>In that way, do tensors on gpu perform non-arithmetic operations faster than tensors on cpu?</p>
<p>For example, I want to change all elements of the tensor which are smaller than 10000 to 123:</p>
<pre><code class="lang-auto">a = torch.tensor([i for i in range(100000)])
a[(a&lt;10000).nonzero(as_tuple=True)] = 123 # operation 1
</code></pre>
<pre><code class="lang-auto">a = torch.tensor([i for i in range(100000)]).cuda()
a[(a&lt;10000).nonzero(as_tuple=True)] = 123 # operation 2
</code></pre>
<p>Is operation 2 necessarily faster than operation 1?</p>

darknight: <p>Would anyone like to share your opinion?</p>
TypeError only integer tensors of a single element can be converted to an index

RicardLee9808: <p>Dear pytorch users<br>
As a freshman in pytorch.I am now design my first UNet to help me understand pytorch, I copy an UNet code from YOUTUBE and now i am going to print  torchsize of different layer,But  something goes wrong.<br>
Would you mind give me some su</p>

CedricLy: <p>print(my_unetmodel) give you already some information.</p>
Bitwise-XOR of all elements in a multi-dimensional Tensor

Manvi_Gupta: <p>Hi everyone!<br>
Does PyTorch provide a way to do the bitwise-xor of all elements in a tensor and return the output value?</p>
<p>I need to do this for a tensor of tensors, hence need a way to parallelize this computation, instead of using loops. Any kind of help would be great.</p>
<p>Thanks in advance.</p>

Eta_C: <p>Do you mean</p>
<pre><code class="lang-python">import torch

x = torch.rand(2, 3, 4, 5) &gt; 0.5
y = torch.rand(3, 4, 5) &gt; 0.5
print(x)
print(y)
z = x | y
print(z)
</code></pre>

Manvi_Gupta: <p>Hi <a class="mention" href="/u/eta_c">@Eta_C</a><br>
Thanks for replying.<br>
No, actually I have to do something like this:</p>
<pre><code class="lang-python">import torch

# Sample tensor
X = torch.tensor([[1, 2, 3, 4, 5], [6, 7, 8, 9, 10]])
lst = []
for i in X:
    XorI = 0    
    for elem in i:
        XorI = elem ^ XorI    
    lst.append(XorI)
</code></pre>
<p>But without loops.</p>
<p>P.S. It would be better if there’s way for more than 2 dimensions as well, where I can reduce in 1 specified direction.</p>

Eta_C: <p>Sorry, since pytorch only have <code>logical_xor</code> and <code>bitwise_xor</code>, I have to find some “math expression” for this operator. See <a href="https://math.stackexchange.com/questions/15141/expressing-bitwise-operations-in-terms-of-other-functions/613514#613514" rel="noopener nofollow ugc">here</a>, it is too complicated!</p>
<p>I think the best way is to make a <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html" rel="noopener nofollow ugc">pytorch extension</a>.</p>

Manvi_Gupta: <blockquote>
<p>See <a href="https://math.stackexchange.com/questions/15141/expressing-bitwise-operations-in-terms-of-other-functions/613514#613514" rel="noopener nofollow ugc">here </a>, it is too complicated!</p>
</blockquote>
<p>Yeah, this one seems a bit too complicated. <img src="https://discuss.pytorch.org/images/emoji/apple/worried.png?v=12" title=":worried:" class="emoji" alt=":worried:" loading="lazy" width="20" height="20"></p>
<blockquote>
<p>I think the best way is to make a <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html" rel="noopener nofollow ugc">pytorch extension </a>.</p>
</blockquote>
<p>I never knew about this feature of PyTorch. Can give it a try!</p>
<p>Thanks, <a class="mention" href="/u/eta_c">@Eta_C</a> <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=12" title=":smile:" class="emoji" alt=":smile:" loading="lazy" width="20" height="20"> ; though let’s keep the discussion open for any other possible solutions.</p>
Perform operations on a 4D tensor

Hdk: <p>I have a 4D tensor of shape say (100, 10, 5, 5).</p>
<p>I want to perform operations in last to dimensions.<br>
e.g. on each (5, 5) subtensor.</p>
<p>Let’s say…<br>
Mean of all 5 rows, mean of all 5 colums =&gt; 10 values<br>
Max. of each row and column = &gt; 10 values<br>
Min. of each row and column = &gt; 10 values<br>
So, I have now 10+10+10= 30 values, which can be viewed as (5,6) subtensor.</p>
<p>So, I want to replace original (5, 5) subtensors by these (5, 6) subtensors such that the shape of that 4D tensor now becomes (100, 10, 5, 6) or maybe (100, 10, 30).</p>
<p>I want to avoid for loops because backward method is very slow in that case.</p>

Eta_C: <p>Could you show how to implement it using for loops? I couldn’t understand</p>
<blockquote>
<p>10+10+10= 30 values, which can be viewed as (15,15) subtensor.</p>
</blockquote>
<p>30 values could be viewed as (5, 6).</p>

Hdk: <p>Hi <a class="mention" href="/u/eta_c">@Eta_C</a> ,</p>
<p>I was thinking to view it as a 3D tensor maybe and iterate through each (5, 5) sub tensors with for loop.<br>
But that way would be extremely slow as I mentioned. <img src="https://discuss.pytorch.org/images/emoji/apple/unamused.png?v=12" title=":unamused:" class="emoji" alt=":unamused:" loading="lazy" width="20" height="20"></p>
<aside class="quote no-group" data-username="Eta_C" data-post="2" data-topic="109736">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/eta_c/48/17667_2.png" class="avatar"> Eta_C:</div>
<blockquote>
<p>30 values could be viewed as (5, 6</p>
</blockquote>
</aside>
<p>I corrected the mistake <img src="https://discuss.pytorch.org/images/emoji/apple/grin.png?v=12" title=":grin:" class="emoji" alt=":grin:" loading="lazy" width="20" height="20"><br>
Thanks.</p>

nibor: <p>Hi <a class="mention" href="/u/hdk">@Hdk</a></p>
<p>I has a similar use case where a tensor lets say A, A.shape = (100,10,5,5)<br>
I wanted to add the last two dimensions such that new Ashape = (100,10,5) and following did the trick</p>
<blockquote>
<p>A.sum(axis=2)</p>
</blockquote>
<p>Hopefully it give you more ideas.</p>
<p>regards,</p>

Eta_C: <p>What about</p>
<pre><code class="lang-python">result = torch.cat(
    [x.mean(-1), x.mean(-2),
     x.min(-1)[0], x.min(-2)[0], 
     x.max(-1)[0], x.max(-2)[0]], 
    dim=-1)
</code></pre>
How to use root='./data'?

1315933876: <p>E:\Python35&gt;python work\train.py<br>
After executing the above code, there is the following bug.<br>
My system is windows, and My cifar data is in the file folder, E:\Python35\work\data<br>
The error is as follows: Thank you in advance.<br>
Traceback (most recent call last):<br>
File “work\train.py”, line 39, in <br>
trainset = torchvision.datasets.CIFAR10(root=‘E:\Python35\work\data’, train=True, download=False, transform=transform_train) <span class="hashtag">#训练数据集</span><br>
File “E:\Python35\lib\site-packages\torchvision\datasets\cifar.py”, line 62, in <strong>init</strong><br>
’ You can use download=True to download it’)<br>
RuntimeError: Dataset not found or corrupted. You can use download=True to download it<br>
Exception ignored in: &lt;function WeakValueDictionary.<strong>init</strong>..remove at 0x0000027BF28DAF28&gt;<br>
Traceback (most recent call last):<br>
File “E:\Python35\lib\weakref.py”, line 117, in remove<br>
TypeError: ‘NoneType’ object is not callable</p>

ptrblck: <p>You might have to use <code>\\</code> or <code>/</code> in the path, since <code>\</code> might be understood as an escape sequence.<br>
If the error is still raised, you could try to download it again via <code>download=True</code>.</p>
Multiple GPUs to run the same instance of a model with different inputs

Sherine_Brahma: <p>Hi,</p>
<p>First, I want to thank you for your effort in maintaining this forum which made learning PyTorch much easier for me. I had a question regarding using two GPUs to run the same instance of a model with different inputs (think of it as different samples of a distribution). I am using two GPUs because I have a computationally heavy framework and running my network once fills up the capacity of one GPU.</p>
<p>My question is whether it possible to update the same network, gen_net, based on different inputs? I had this question because if I used different GPUs, while backpropagating, I thought Pytorch will update separate instances of my network stored in different GPUs. What I want is to update the same network but using different GPUs.</p>
<h1><a name="code-snippet-1" class="anchor" href="#code-snippet-1"></a>Code snippet</h1>
<blockquote>
<p>gen_net = gen_net.to(torch.device(0))<br>
by_gen_1 = gen_net(bx) # bx is my input sample</p>
<p>gen_net = gen_net.to(torch.device(1))<br>
bx = bx.to(torch.device(1))<br>
by_gen_2 = gen_net(bx) # This becomes a different input because we add noise along with bx inside gen_net</p>
<p>loss = loss_func(by_gen_1, by_gen_2)<br>
loss.backward()</p>
</blockquote>

ptrblck: <p>If I understand your use case correctly, you are trying to clone the same model onto GPU0 and GPU1, use different inputs, and calculate the loss based on both outputs.<br>
Once this loss is calculated, you would like to calculate the gradients for both models.<br>
How should these gradients be handled? Since the inputs were different, the model clone would get different gradients.</p>
<p>If you would like to reduce the gradients to a single model, I think you should be able to use a distributed approach e.g. via DDP.</p>

Sherine_Brahma: <p>Thank you for the reply. I would want the gradients to accumulate to the same model but I do not know how will that happen if I have two models in different GPUs.</p>
Does Pytorch 1.7 support CUDA Toolkit 11.0 or CUDA Toolkit 11.0 Update1?

1315933876: <p>In this web page, there are two versions,  CUDA Toolkit 11.0 and CUDA Toolkit 11.0 Update1 . <a href="https://developer.nvidia.com/cuda-toolkit-archive" class="inline-onebox" rel="noopener nofollow ugc">CUDA Toolkit Archive | NVIDIA Developer</a><br>
I am confused. If I want to install Pytorch version with CUDA, Which CUDA should I choose?</p>

ptrblck: <p>Double post from <a href="https://discuss.pytorch.org/t/v1-1-0-of-pytorch-with-cuda-or-the-version-v1-6-0-pytorch-no-cuda/109063/4">here</a> with a potential answer.<br>
Cross-posting:</p>
<ul>
<li>you don’t need a local CUDA toolkit installation if you don’t plan on building PyTorch from source or any CUDA extension, just the NVIDIA driver.</li>
<li>the binaries use CUDA11.0.3, so 11.0 Update 1. If you plan on building custom CUDA extensions, you should install the same local toolkit</li>
</ul>
Minimising a simple function

Omroth: <p>Hi again.</p>
<p>I’m attempting to minimise x*x using autograd in c++:</p>
<pre><code class="lang-auto">torch::Tensor dynamic_parameters = torch::tensor(/*value=*/{17.2}, torch::dtype(torch::kFloat64).requires_grad(true));

torch::Tensor target = torch::tensor(/*value=*/{0}, torch::dtype(torch::kFloat64).requires_grad(false));

torch::optim::SGD optimizer({dynamic_parameters}, /*lr=*/0.001);

pybind11::gil_scoped_release no_gil;

for (int s = 0; s &lt; 20; s ++)
{
    torch::Tensor prediction = dynamic_parameters * dynamic_parameters;

    torch::Tensor loss = torch::nn::functional::mse_loss(prediction, target);

    std::cout &lt;&lt; "guess:  " &lt;&lt; dynamic_parameters &lt;&lt; std::endl;
    std::cout &lt;&lt; "loss:  " &lt;&lt; loss &lt;&lt; std::endl;

    optimizer.zero_grad();

    loss.backward();

    optimizer.step();
}
</code></pre>
<p>Steps 10-20 output:</p>
<pre><code class="lang-auto">guess:  -2.5730
[ CPUDoubleType{1} ]
loss:  43.827
[ CPUDoubleType{} ]
guess:  -2.5048
[ CPUDoubleType{1} ]
loss:  39.3659
[ CPUDoubleType{} ]
guess:  -2.4420
[ CPUDoubleType{1} ]
loss:  35.5603
[ CPUDoubleType{} ]
guess:  -2.3837
[ CPUDoubleType{1} ]
loss:  32.2869
[ CPUDoubleType{} ]
guess:  -2.3295
[ CPUDoubleType{1} ]
loss:  29.4501
[ CPUDoubleType{} ]
guess:  -2.2790
[ CPUDoubleType{1} ]
loss:  26.9751
[ CPUDoubleType{} ]
guess:  -2.2316
[ CPUDoubleType{1} ]
loss:  24.8023
[ CPUDoubleType{} ]
guess:  -2.1872
[ CPUDoubleType{1} ]
loss:  22.8843
[ CPUDoubleType{} ]
guess:  -2.1453
[ CPUDoubleType{1} ]
loss:  21.1824
[ CPUDoubleType{} ]
guess:  -2.1058
[ CPUDoubleType{1} ]
loss:  19.6651
[ CPUDoubleType{} ]
guess:  -2.0685
[ CPUDoubleType{1} ]
loss:  18.3065
[ CPUDoubleType{} ]
guess:  -2.0331
[ CPUDoubleType{1} ]
loss:  17.0851
[ CPUDoubleType{} ]
guess:  -1.9995
[ CPUDoubleType{1} ]
loss:  15.9829
[ CPUDoubleType{} ]
</code></pre>
<ol>
<li>
<p>It explodes with a learning rate of 0.005, which I guess surprises me a little.</p>
</li>
<li>
<p>The convergence seems quite slow - do we just expect to be running an optimiser step hundreds of times even for a simple function?</p>
</li>
<li>
<p>I note the loss from MSE is not… the prediction minus the target squared.  Purely out of interest, what does that loss represent?</p>
</li>
</ol>
<p>Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Best architecture/optimizer for 3-class prediction?

machine_learning: <p>Hello!</p>
<p>I am trying to get good scores on a 3-class pneumonia(x-ray images) classification problem which is:<br>
class0 : normal<br>
class1 : bacterial pneumonia<br>
class3 : viral pneumonia</p>
<p>I have tried multiple models(resnet, vgg, etc), pretrained and not, but I cannot surpass 81% testing accuracy.<br>
Is there any model that can perform better on such problem? Or a better optimizer/transformation?</p>
<p>Thank you in advace!</p>
Loss definition

Javier_Rodriguez_Pui: <p>Hi I am using such a loss:</p>
<pre><code class="lang-auto">            nll = torch.pow(y_hat - y, 2) * 0.5
            e_log_sigma_2 = torch.exp(-log_sigma_2)
            u_a = nll * (e_log_sigma_2) + log_sigma_2 * 0.5
            loss = u_a.mean()
</code></pre>
<p>After training a while. I get some nan and I cannot understand why. Maybe Somebody have face this problem already.</p>
<p>Any suggestion is welcome =)</p>
ValueError: Expected target size (5, 50, 76, 80), got torch.Size([5, 2])

jdlin: <p>I am using a transfer learning from a pretrained model. When I tried to do this:</p>
<pre><code class="lang-auto">loss_f = nn.CrossEntropyLoss(weight=torch.tensor([1,1,1]))
output = model(volumes)
     
labels = torch.FloatTensor(labels).long()   #deleted .cuda()
labels = labels.cuda()

loss = loss_f(output, labels)
</code></pre>
<p>The error came up with:</p>
<pre><code class="lang-auto">ValueError: Expected target size (5, 50, 76, 80), got torch.Size([5, 2])
</code></pre>
<p>I printed the output and labels shape as follows:</p>
<pre><code class="lang-auto">output_shape = torch.Size([5, 2, 50 ,76, 80)]
labels_shape = torch.Size([5, 2])
</code></pre>
<p>It seems that with the default documentation from CrossEntropyLoss is (N, C, D1, D2, …, Dk), which k is the dimensions of the image. I am trying to do binary classification, so the classes are 2. I am facing the problem with the shape differences between the output and the labels, could anyone comment on how to solve the problem.</p>
<p>Thanks!!</p>

ptrblck: <p>The output shape indicates a multi-classification use case for volumetric data as it would correspond to logits in <code>[batch_size, nb_classes, depth, height, width]</code>.<br>
If you are working an a binary classification use case the output is expected to have the shape <code>[batch_size, nb_classes2]</code> while the target should have the shape <code>[batch_size]</code> and contain values in the range <code>[0, nb_classes=2]</code>.<br>
Currently your <code>label</code> tensor also seems to be one-hot encoded so use <code>labels = torch.argmax(labels,. dim=1)</code> if that’s the case.</p>
How does Conv2d sees the input data?

Nelyo: <pre><code class="lang-auto">  (conv): Sequential(
    (0): Conv2d(1, 64, kernel_size=(4, 4), stride=(1, 1))
    (1): ReLU()
    (2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
</code></pre>
<p>Imagine the input shape looks like 1x7x7<br>
How does the input data gets transformed/scaled into the In/Out channels of Conv2d?</p>
<p>I’m sure I sound silly but all explanations came with the perspective of images where the input was larger than the channels. My case is Connect4 (with 6x7 + 1x7 action space). I am trying to get a pattern recognition.</p>

ptrblck: <p>Are you interested in the general method how conv layers are working?<br>
If so, I think <a href="https://cs231n.github.io/convolutional-networks/#conv">CS231n - Conv</a> is providing a good explanation how each filter is used in the layer.</p>

mobassir94: <p>after checking the resource that <a class="mention" href="/u/ptrblck">@ptrblck</a> provided,you can check this as well : <a href="https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/160147" class="inline-onebox" rel="noopener nofollow ugc">SIIM-ISIC Melanoma Classification | Kaggle</a></p>

Nelyo: <p>Thanks guys, I dive further</p>
Torchscript w/ custom CUDA operator

Duncan_Freeman: <p>Are there instructions for properly including CUDA kernels in custom TorchScript ops built using CMake? I haven’t been able to find any, and I’ve hit an odd <code>nvcc</code>-related compilation error when I attempt to do so (See <a href="https://pastebin.com/TpejJwfr" rel="noopener nofollow ugc">the log file</a> and the <a href="https://github.com/Masterchef365/torchscript_custom_op_rust_example/tree/cuda" rel="noopener nofollow ugc">repository</a> which I am using to test these concepts)</p>
<p>For my use-case, I need to serialize a model into TorchScript which references custom CUDA ops. Said model must also be executable from C++/Rust. The instructions for compiling a C++ operator for TorchScript <a href="https://pytorch.org/tutorials/advanced/torch_script_custom_ops.html" rel="noopener nofollow ugc">here</a> work well for pure C++ operators, and the op is both traceable in python and useable from C++ and Rust. However, I must also be able to include C++ operators that execute CUDA kernels (in fact, I’m actually porting a neural net that used the JIT method for compiling these - but in C++/Rust I need to link against a DLL or shared object so I cannot use this afaik).</p>
<p>Thanks!</p>
V1.1.0 of Pytorch with CUDA or the version v1.6.0 Pytorch No CUDA?

1315933876: <p>Thank you very much.<br>
I would like to install Pytorch on windows.<br>
Which Pytorch version is suitable for me?<br>
CUDA version is 9.0 on my laptop computer.<br>
The latest version of Pytorch is v.1.6.0, which only supports CUDA versions 9.2 or newer versions.<br>
The version of Pytorch v.1.1.0 can support CUDA version 9.0.<br>
So I am not sure whether I should  install the  version v1.1.0 of Pytorch with CUDA or the version v1.6.0 Pytorch No CUDA?<br>
Which version is suitable for me?<br>
If I install Pytorch v.1.1.0, then if the codes are based Pytorch v.1.6.0 in one article, will I meet troubles when I try to copy the codes and do numerical experiments?</p>

ptrblck: <p>The latest PyTorch release is currently <code>1.7.1</code> and I would recommend to stick to it instead of falling back to old versions.<br>
Your local CUDA installation will not be used to run PyTorch, as the conda binaries and pip wheels ship with their own CUDA runtime. You would have to make sure to use the appropriate NVIDIA driver on your system to use the CUDA runtime.<br>
Your local CUDA toolkit would be used, in case you want to build PyTorch from source or any custom CUDA extensions.</p>

1315933876: <p>Thank you very much.</p>

1315933876: <p>In this web page, there are two versions, CUDA Toolkit 11.0 and CUDA Toolkit 11.0 Update1 . <a href="https://developer.nvidia.com/cuda-toolkit-archive" rel="noopener nofollow ugc">CUDA Toolkit Archive | NVIDIA Developer</a><br>
I am confused. If I want to install Pytorch version 1.7.1 with CUDA, Which CUDA should I choose?</p>

ptrblck: <p>If you want to build custom CUDA extensions, I would recommend to stick to the matching one which ships with the binaries, so 11.0.</p>
Optimizing parameters of function generating convolution kernel instead of raw weights

G.S_Luimstra: <p>I am currently working on my thesis and have a situation where I have a function that generates a convolution kernel. For now, let us assume that this function takes the location in the kernel and multiplies it by a constant. As an example, consider f(x, y) = c(x + y). Rather than optimize and alter the raw weights of the kernel, I would instead like to make autograd update c during the backprop.</p>
<p>I have tried creating a custom ‘DumbKernel’ as follows:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
from torch import Tensor
import numpy as np

class DumbKernel(nn.Conv2d):

    def __init__(self, *args, **kwargs):

        super(DumbKernel, self).__init__(*args, **kwargs)

        # Initialize parameter 'c' for each kernel
        self.parameters = []
        for out_channel in range(self.out_channels):
            sub_list = []
            for in_channel in range(self.in_channels):
                sub_list.append([
                    nn.Parameter(torch.tensor([1.0], requires_grad=True)),
                ])
                self.register_parameter(name='{}{}c'.format(out_channel, in_channel), param=sub_list[-1][0])
            self.parameters.append(sub_list)
        self.parameters = np.array(self.parameters, copy=False)

        # Initialize the filters based on parameters 'c'
        self.initialize_filters()

    def initialize_filters(self):
        for out_channel in range(self.out_channels):
            for in_channel in range(self.in_channels):
                self.weight[out_channel, in_channel] = self.generate_filter(*self.parameters[out_channel, in_channel])

    def generate_filter(self, c):

        #Obtain the right kernel size
        kernel_size = self.kernel_size[0]

        # Calculate the boundaries to add to the center element
        x_max = kernel_size // 2
        y_max = kernel_size // 2
        x_min = -x_max
        y_min = -y_max

        kernel = torch.ones((kernel_size, kernel_size))

        for y in range(y_min, y_max + 1):
            for x in range(x_min, x_max + 1):
                kernel[y_max - y][x_max - y] = c * (y + x)
        
        return kernel

    def forward(self, input: Tensor) -&gt; Tensor:
        return super()._conv_forward(input, self.weight)
        

</code></pre>
<p>I hoped that autograd would pick up on the fact that the weights are in fact generated by a parameter ‘c’, through the computation graph, but this is not true. When I used the above code in a training loop, I run into the following error: <code>ValueError: can't optimize a non-leaf Tensor</code>. If I add <code>self.weight = nn.Parameter(self.weight)</code> at the end of the constructor, it does work but it then changes the raw kernel weights instead of ‘c’. Is there any way to fix this? In theory it should be possible using the chain rule!</p>
<p>Kind regards,<br>
Gerrit</p>

googlebot: <p>You should probably use nn.functional.conv2d, with it you can use any <em>tensor</em> as kernel .</p>

G.S_Luimstra: <p>Thank you, this was it!</p>
Issue with pretrained resnet (FIXED)

gurhar1133: <p>For some reason I am getting this message:</p>
<pre><code class="lang-auto">self = reduction.pickle.load(from_parent)
AttributeError: Can't get attribute 'CassDataset' on &lt;module '__main__' (built-in)&gt;

</code></pre>
<p>in my terminal but not in my jupyter notebook. I am not sure what it even telling me.</p>
<p>It is referencing my dataset class CassDataset. The code for that class is:</p>
<pre><code class="lang-auto">class CassDataset(Dataset):
    def __init__(self, Dir, FNames, Labels, Transform):
        self.dir = Dir
        self.fnames = FNames
        self.transform = Transform
        self.lbs = Labels
        
    def __len__(self):
        return len(self.fnames)

    def __getitem__(self, index):
        x = Image.open(os.path.join(self.dir, self.fnames[index]))
        if "train" in self.dir: 
            res  = self.transform(image=asarray(x))
            image = res['image'].astype(np.float32)
            return image, self.lbs[index]            
        elif "test" in self.dir:            
            return self.transform(image=asarray(x)), self.fnames[index]

</code></pre>
<p>And here are some of my model parameters:</p>
<pre><code class="lang-auto">model_ft = models.resnet18(pretrained=True)
num_ftrs = model_ft.fc.in_features
# Here the size of each output sample is set to 2.
# Alternatively, it can be generalized to nn.Linear(num_ftrs, len(class_names)).
model_ft.fc = nn.Linear(num_ftrs, 5)

#model_ft = model_ft.to(device)

criterion = nn.CrossEntropyLoss()

# Observe that all parameters are being optimized
optimizer_ft = optim.SGD(model_ft.parameters(), lr=1e-2, momentum=0.9)

</code></pre>

gurhar1133: <p>I think that might have helped! Im not getting the error but also not sure if the model is doing anything</p>

1chimaruGin: <p>can I see dataloader code snippet?</p>

gurhar1133: <p>I fixed by putting the dataset class in a separate text file and importing the class in my main training program</p>
How do I concatenate 1.2 million 2 dimensional tensors into a single tensor?

ProGamerGov: <p>So, I have a list of 1.2 million tensors that I’d like to concatenate into a single tensor. When I use <code>torch.cat</code> on the list of tensors, I get an out of memory error. Each tensor has a shape of [1, n_channels]. Ex: <code>torch.Size([832, 1])</code></p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "cat_samples.py", line 10, in &lt;module&gt;
    torch.save(torch.cat(samples, 1), "1m_samples.pt")
RuntimeError: CUDA out of memory. Tried to allocate 3.97 GiB (GPU 0; 8.00 GiB total capacity; 4.28 GiB already allocated; 2.44 GiB free; 4.28 GiB reserved in total by PyTorch)
</code></pre>
<p>How can I concatenate my list of 1.2 million 2 dimensional tensors into a single tensor?</p>

ProGamerGov: <p>So, it looks like I can avoid an out of memory error by moving the tensors to the CPU before using <code>torch.cat</code>.</p>
MKLDNN support for bidirectional LSTM

nibor: <p>I have PyTorch 1.7.1 and looking at this: <a href="https://github.com/pytorch/pytorch/blob/master/torch/utils/mkldnn.py" class="inline-onebox" rel="noopener nofollow ugc">pytorch/mkldnn.py at master · pytorch/pytorch · GitHub</a> I don’t see support for MKLDNN for bidirectional LSTMs on CPUs. Also, when I run my code in verbose, I find calls only for MKL not for MKLDNN.<br>
Am I missing something or there is no support of MKLDNN for bidirectional LSTMs on CPUs in Pytorch?</p>
<p>I am doing a comparison of LSTMs on Tensorflow/Keras and PyTorch so just want to be sure that I am not missing anything.</p>
BCELoss and BCEWithLogits stop decreasing

Adri: <p>I have multi label classification task, i have 90 labels, and every data have 1.2 label on average. I try to use BCELoss and BCEWithLogits and it’s stuck on early epoch and the model will give score 0.5 on every single label prediction.</p>
<p>So if i round the score, it means every single data doesnt have label cause the prediction is 0 (zero). It means my micro-f1 score will be 0 too.</p>
<p>Do tou have any suggestion to this problem?</p>

Adri: <p>But on several data point, the data has multiple label. I want my model can predict multiple label</p>

neuralpat: <p>I’m sorry, I was wrong about BCELoss, I’ll withdraw my post.</p>
<p>Would you be able to show us your code, so we might be able to spot any issues ?</p>
How to save/load Torch models?

Russel_Russel: <p><code>torch.save(object, f)</code>. I don’t understand the documentation explanation of <code>"f". What is "f"?</code> Can you give an example?  Are there better ways to save torch models other than the save function?  Thanks a lot</p>

apaszke: <p><code>f</code> should be a file-like object (e.g. obtained from a call to <code>open</code>), or a path to the file where the model will be saved. <code>torch.save</code> is exactly what you should use, but we recommend serializing only the <code>model.state_dict()</code>. You can later load it using <code>load_state_dict</code>.</p>

kindlychung: <p>Could you provide an example for serializing and loading <code>state_dict</code>?</p>

apaszke: <p><a href="https://github.com/pytorch/examples/blob/0984955bb8525452d1c0e4d14499756eae76755b/imagenet/main.py#L139-L145">ImageNet example</a>.</p>

YunTang: <p>Is there any reason to save model using state_dict instead of the model itself?<br>
One obvious advantage to save the model directly is the model can be loaded without creating a prototype model, while the state_dict needs to create a prototype model with same configure (same <span class="hashtag">#layers</span>, <span class="hashtag">#neuron</span> size etc)<br>
One potential advantage I can see to use state_dict to save/load model is that you can use a updated scripts to load the old model if the parameters are the same.</p>
<p>Thanks</p>
<p>Yun</p>

smth: <p>state_dict is easy to manipulate and load into other models. it’s also a simpler structure. Your model class definition might also not be picklable (possible in some complicated models)</p>
<p>Other than this, it’s a matter of taste.</p>

Nick_Brandaleone: <p>Is there any work to export Pytorch models to new Apple format (CoreML), or convert to Keras (so it can be used by Apple devices)?</p>
<p>Nick</p>

Brando_Miranda: <p>did you find out what is <code>f</code>? Is it a string or an actual file pointer?</p>

allenye0119: <p><code>f</code> can be either a string or a file pointer.<br>
<a href="http://pytorch.org/docs/master/torch.html#torch.load" class="onebox" target="_blank" rel="nofollow noopener">http://pytorch.org/docs/master/torch.html#torch.load</a></p>

rajarsheem: <p>Is there any way to load a trained model without declaring the class definition before ?<br>
I want the model architecture as well as parameters to be loaded.</p>

smth: <p><a class="mention" href="/u/rajarsheem">@rajarsheem</a> no, you have to load the class definition before, this is a python pickling limitation.</p>

kirill-pinigin: <p>Please tell me. Does Pytorch save model in h5df format file or another format?  What I must do to save dict model in h5df format?</p>

singleroc: <p>It seems the default format is the plk. However, hdf5 is also possible,refer: <a href="https://github.com/deepmind/torch-hdf5" rel="nofollow noopener">https://github.com/deepmind/torch-hdf5</a></p>

dhanyashri_b_v: <p>When you save the pytorch model in .pth format. The entire model structure will get saved in it right. Again why do we have to keep the .py script along with .pth file?</p>
<p>Thank you,</p>
Gradient accumulation

NadiaMe: <p>Let’s say I want to experiment with large batch size that does not fit in memory.<br>
Would batch accumulation be equivalent to running model with a large batch size?</p>
<p>For example, say batch is of size 10 and I have the following pseudo code</p>
<pre><code class="lang-auto">for i, batch in enumerate(train_loader):
      loss = compute_loss(batch)
      loss.backward()

      if i%9 == 0:
            optimizer.step()
            scheduler.step()
</code></pre>
<p>Would this be equivalent to running just</p>
<pre><code class="lang-auto">for i, batch in enumerate(train_loader):
      loss = compute_loss(batch)
      loss.backward()
      optimizer.step()
      scheduler.step()
</code></pre>
<p>if we had the size of batch 100?</p>

ptrblck: <p>It would be equivalent, if you are not using batch size - dependent layers such as batchnorm layers.<br>
These layers update their internal running stats using the stats of the current input batch and the <code>momentum</code> term, which would be different for smaller and larger batch sizes.<br>
Besides that, random operations, such as dropout layers could also change the final results, since the outputs would be randomly dropped in both approaches.</p>

mobassir94: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> you are right,i was using gradient accumulation with filter response normalization and with batch size = 4 and it didn’t perform well,then used same model with batch size = 8 and without gradient accumulation and it worked fine there</p>
How to slice Kinetics400 training dataset?

111429: <p>Hi,<br>
I am trying to run the official <a href="https://github.com/pytorch/vision/blob/master/references/video_classification/train.py" rel="noopener nofollow ugc">script</a> for video classification.<br>
I want to tweak some functions and running through all examples would cost me too much time.<br>
I wonder how can I slice the training kinetics dataset based on that script.<br>
This is the code I added before<br>
<code>train_sampler = RandomClipSampler(dataset.video_clips, args.clips_per_video)</code><br>
in the script: (let’s say I just want to run 100 examples.)</p>
<pre><code class="lang-auto">tr_split_len = 100
dataset = torch.utils.data.random_split(dataset, [tr_split_len, len(dataset)-tr_split_len])[0]
</code></pre>
<p>Then when hitting <code>train_sampler = RandomClipSampler(dataset.video_clips, args.clips_per_video)</code><br>
, it pops out the error:</p>
<pre><code class="lang-auto">AttributeError: 'Subset' object has no attribute 'video_clips'
</code></pre>
<p>Yeah, so the type of <code>dataset</code> converts from <code>torchvision.datasets.kinetics.Kinetics400</code> to <code>torch.utils.data.dataset.Subset</code>.<br>
I understand. So how can I do it? (hopefully not the way using <code>break</code> in the dataloader loop).<br>
Thanks.</p>

ptrblck: <p>Based on your code snippet it seems <code>dataset</code> is supposed to contain an attribute called <code>video_clips</code>.<br>
Since you are passing this <code>dataset</code> to <code>random_split</code>, I assume each clip would be returned in the <code>__getitem__</code> or would it return a sample for one clip?<br>
In the latter case, I assume the usage of <code>random_split</code> is wrong, as it would remove clips, which the <code>RandomClipSampler</code> is for.</p>
Matrix multiplication in c++

Omroth: <p>I have</p>
<pre><code class="lang-auto">torch::Tensor c = torch::mm(a, b);
</code></pre>
<p>Where a is of shape [30,1000] and b is of shape [1000]</p>
<p>On running I get:</p>
<pre><code class="lang-auto">RuntimeError: mat2 must be a matrix
</code></pre>
<p>But matrix multiplication of [30,1000] and [1000] should be valid and result in a tensor of shape [30], if I’m not missing something…</p>

Omroth: <p>I should be using torch::matmul, not mm</p>
Unable to `import torch`

0xdd: <p>PyTorch version on my system is <code>1.7.1</code> installed via conda. The CUDA version on my system is <code>11.2</code> and driver version is <code>460.32.03</code>. But on doing <code>import torch</code>, I get an <code>OSError</code> as attached below</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/c/8cff6f40b420128b023a4a6fdf46c794ddff91ef.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/8cff6f40b420128b023a4a6fdf46c794ddff91ef" title="Screenshot from 2021-01-22 12-02-13"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/c/8cff6f40b420128b023a4a6fdf46c794ddff91ef_2_690x388.jpeg" alt="Screenshot from 2021-01-22 12-02-13" data-base62-sha1="k7k6WfLyWkGygRCLGpIP6IsEmaH" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/c/8cff6f40b420128b023a4a6fdf46c794ddff91ef_2_690x388.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/8/c/8cff6f40b420128b023a4a6fdf46c794ddff91ef_2_1035x582.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/c/8cff6f40b420128b023a4a6fdf46c794ddff91ef_2_1380x776.jpeg 2x" data-dominant-color="050709"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2021-01-22 12-02-13</span><span class="informations">1920×1080 273 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>It used to work perfectly but then CUDA Toolkit got updated. Does anybody know that is it due to the version mismatch or some other error I am unaware of? I am currently running Fedora 33 with NVIDIA GeForce GTX 1050 GPU. Any help will be appreciated. Thanks!</p>

ptrblck: <p>Could you create a new virtual environment and reinstall the PyTorch binaries there, please?</p>

0xdd: <p>Thanks for the response. I get the same error again.</p>
Do tensors containing NaNs affect the loss.backward()?

manurare: <p>Hi,</p>
<p>I have a loss function that looks like this:</p>
<pre><code class="lang-auto"> disp_diff = (torch.max(eval_target[:, -1:, ...], eval_source[:, -1:, ...]) /
 torch.min(eval_target[:, -1:, ...], eval_source[:, -1:, ...])) - 1

disp_diff[disp_diff != disp_diff] = 0

loss = torch.sum(disp_diff, dim=[2, 3])
</code></pre>
<p>The max/min operation can be (0/0) = NaN. That’s why I set those position to 0 afterwards. The problem arises that when I try to do the loss.backward() in the next iteration the network only predicts NaNs. Is this because eval_target and eval_source have requires_grad=True and their NaNs can be propagated during the backward pass?</p>

ptrblck: <p>I think you would have to avoid running into these invalid values, as manipulating the output afterwards would still run into the invalid operation in the backward pass, if I’m not mistaken.<br>
Here is a small code example:</p>
<pre><code class="lang-python">x = torch.randn(10, requires_grad=True)
d = torch.randn(10)
d[0] = 0.

out = x / d
print(out)
&gt; tensor([   -inf, -0.9076, -0.6809, -0.0224, -0.4553, -1.4387, -1.9699,  0.2929,
         0.5049, -0.8641], grad_fn=&lt;DivBackward0&gt;)

out[~torch.isfinite(out)] = 0
print(out)
&gt; tensor([ 0.0000, -0.9076, -0.6809, -0.0224, -0.4553, -1.4387, -1.9699,  0.2929,
         0.5049, -0.8641], grad_fn=&lt;IndexPutBackward&gt;)

out.mean().backward()
print(x.grad)
&gt; tensor([    nan, -0.1204,  0.0864, -0.0858, -0.1498,  0.1549,  0.1525,  0.2894,
         0.0665, -1.0095])
</code></pre>

manurare: <p>Didnt know this. Thanks!</p>
Numerical stability of BCEWithLogitsLoss

Sourabh_Daptardar: <p>What are differences with respect to numerical stability, for the implementation of binary cross entropy with logits loss</p>
<p>in PyTorch as defined <a href="https://github.com/pytorch/pytorch/blob/d5a7e304fa858136eeffa1ebe35e22ed886bf256/torch/nn/functional.py#L859" rel="nofollow noopener">here</a></p>
<pre><code>max_val = (-input).clamp(min=0)
loss = input - input * target + max_val + ((-max_val).exp() + (-input - max_val).exp()).log()
</code></pre>
<p>versus formulation used in tensorflow, defined <a href="https://www.tensorflow.org/api_docs/python/tf/nn/sigmoid_cross_entropy_with_logits" rel="nofollow noopener">here</a></p>
<pre><code>max(x, 0) - x * z + log(1 + exp(-abs(x)))
</code></pre>
<p>Are there any pros and cons of choosing one formulation over the other ?</p>

smth: <p>the first formulation is a two-pass formulation that is generally known to be more stable.<br>
This page might help understand it better <a href="https://en.wikipedia.org/wiki/LogSumExp">https://en.wikipedia.org/wiki/LogSumExp</a></p>
3D matrix data plot reconstruction

sikhar: <p>I have a 3D matrix when i plot it show some peaks as shown in the screenshot. <img src="https://discuss.pytorch.org/uploads/default/original/3X/3/4/34424a1ce0f9363724a8aeceee87433aa4c579bc.png" alt="Screenshot_312" data-base62-sha1="7siQTcbYqMUf0nKI78eBrtoxJgg" width="429" height="300"> .<br>
What i am doing i am adding some noise(0dB to -20dB) in the matrix and trying to reconstruct their peaks. probably i am getting the good reconstruction results as well. but i stuck on one thing that i wish to know the shifting the heights of the peaks in comparison to Original matrix and reconstructed matrix. anyone please help me about this.</p>
No lr_scheduler in libtorch?

botelho: <p>Isn’t there a facility for learning-rate scheduling in C++ libtorch, similar to <code>torch.optim.lr_scheduler</code> in Python? I can’t seem to find anything in the documentation or in the code. If not, any idea if this is in the works for an upcoming release? Thanks.</p>

ptrblck: <p>It’s not implemented (yet) and you can track the progress <a href="https://github.com/pytorch/pytorch/issues/50577">here</a>.</p>