akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to convert model's parameters to tensor and keep grad?

Tessa_van_der_Heiden: <p>Hi all,</p>
<p>I’d like to compute the second derivative of the loss w.r.t. a model (nn.module), but I cannot do this:</p>
<pre><code class="lang-auto">params = torch.cat([p.flatten() for p in policy.parameters()], dim=0)

for i in range(200):
  y = get_expected_return(policy)
  grad = torch.autograd.grad(y, params, create_graph=True)[0]

  with torch.no_grad():
    params += grad
</code></pre>
<p>So, I need to do this:</p>
<pre><code class="lang-auto">for i in range(200):
  y = get_expected_return(policy)
  grad = torch.autograd.grad(y, policy.parameters(), create_graph=True)

  with torch.no_grad():
    for p, g in zip(policy.parameters(), grad):
      p += g
</code></pre>
<p>But, the first solution is required for the second derivative. So, anybody knows how to make the first code work?</p>

albanD: <p>Hi,</p>
<p>Why would the first one be required?<br>
The second one looks fine to me.</p>

Tessa_van_der_Heiden: <p>Hi albanD,</p>
<p>Thanks for you answer, sorry for my late response (did not get a notification).</p>
<p>The second solution works, but now how do I compute the second derivative?</p>
<pre><code class="lang-auto">
for i in range(200):

  y = get_expected_return(policy)
  grad = torch.autograd.grad(y, policy.parameters(), create_graph=True)

  for gvec, (name,theta) in zip(grad, policy.named_parameters()):
    gvec=gvec.flatten()
    
    H = [torch.autograd.grad(g, theta, create_graph=True)[0].unsqueeze(0) for g in gvec]
</code></pre>
<p>Is this correct?</p>
<p>Consider two parameter sets, x1 and x2. The H should be computed by:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/f/2/f2f3d9cde9f7d0ba1a4f26fe4320afbdd3b151f8.png" alt="image" data-base62-sha1="yFfZJKVhRoxADYZ7swU37zJWUGc" width="401" height="212"></p>
<p>In my code, where am I computing right upper and left lower elements?</p>

albanD: <blockquote>
<p>Is this correct?</p>
</blockquote>
<p>Yes that looks correct.</p>
<blockquote>
<p>In my code, where am I computing right upper and left lower elements?</p>
</blockquote>
<p>You are computing the whole thing.<br>
You have two loops over the params creating a full size n x n matrix.</p>

Tessa_van_der_Heiden: <p>Hi Alban,</p>
<p>Yes, and but if there are two weight matrices Wx, Wr, with sizes R<em>X, R</em>R (RNN).</p>
<p>Therefore, n=(R<em>X+R</em>R)**2. So, each row is of size r=R<em>X+R</em>R</p>
<p>How can this now be used to update Wx and Wr? Or can you slice?</p>
<p>E.g. H[:, :R<em>X] for Wx and H[:, :R</em>X] for Wr?</p>
RuntimeError: Found dtype Long but expected Float

akib62: <p>You can see the <code>dtype</code> of the <code>b_y</code> is <code>Int64</code> and I am getting an error</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/Opps_0/Desktop/MNIST/src/train.py", line 69, in &lt;module&gt;
    train(NB_EPOCS, model, loaders)
  File "/home/Opps_0/Desktop/MNIST/src/train.py", line 60, in train
    loss.backward()    
  File "/home/Opps_0/.local/lib/python3.8/site-packages/torch/tensor.py", line 245, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "/home/Opps_0/.local/lib/python3.8/site-packages/torch/autograd/__init__.py", line 145, in backward
    Variable._execution_engine.run_backward(
RuntimeError: Found dtype Long but expected Float
</code></pre>
<p>Code</p>
<pre><code class="lang-auto">def train(NB_EPOCS, model, loaders):
    model.train()
    # Train the model
    total_step = len(loaders['train'])
    for epoch in range(NB_EPOCS):
        for i, (images, labels) in enumerate(loaders['train']):
            # gives batch data, normalize x when iterate train_loader
            b_x = Variable(images)   # batch x
            b_y = Variable(labels)   # batch y
            print(np.shape(b_x), b_x.dtype, np.shape(b_y), b_y.dtype)
            ohe_target = torch.nn.functional.one_hot(b_y, num_classes=10)
            output = model(b_x)[0]   
            loss = criterion(output, ohe_target)
</code></pre>
<pre><code class="lang-auto">torch.Size([100, 1, 28, 28]) torch.float32 torch.Size([100]) torch.int64
</code></pre>
<p>If I change the dtype of the b_y to <code>Float</code> by the following code</p>
<pre><code class="lang-auto">b_y=b_y.float()
</code></pre>
<p>Then I am getting another error,</p>
<pre><code class="lang-auto">RuntimeError: one_hot is only applicable to index tensor.
</code></pre>
<p>What I have to do to solve the errors?</p>

ptrblck: <p>Double post from <a href="https://discuss.pytorch.org/t/how-to-get-continuous-value-from-output-layer/131977/8">here</a> with solution.</p>
How to get number of images per class from ConcatDataset

talhaanwarch: <p>I have a ConcatDataset and i want to know how many number of images per class so that i can apply WeightedRandomSampler</p>

tom: <p>I don’t think there is a better <em>generic</em> solution than iterating over the dataset and recording the images per class. That said, you’ll be decoding all images once just for nothing, to a solution tailored to your dataset will be much more efficient.</p>
<p>Best regards</p>
<p>Thomas</p>

talhaanwarch: <p>i tried this way</p>
<pre><code class="lang-auto">counts=[]
for d in data_list:
     print(len(np.unique(d.targets, return_counts=True)[1]))
     counts.append(np.unique(d.targets, return_counts=True)[1])
</code></pre>
Making libtorch

Joe_Harrison: <p>I’m following the steps here <a href="https://pytorch.org/cppdocs/installing.html" rel="noopener nofollow ugc">https://pytorch.org/cppdocs/installing.html</a></p>
<p>My folder structure is as follows:</p>
<ul>
<li>build/</li>
<li>libtorch/</li>
<li>CmakeLists.txt</li>
<li>example-app.cpp</li>
</ul>
<p>The latter files are as described in the link. build is empty and for the libtorch folder I downloaded the gpu version from the main download page.</p>
<p>When I run the following command inside the build folder:</p>
<pre><code class="lang-auto">cmake -DCMAKE_PREFIX_PATH=../libtorch ..
</code></pre>
<p>I get the following message:</p>
<pre><code class="lang-auto">-- Caffe2: CUDA detected: 10.1
-- Caffe2: CUDA nvcc is: /usr/bin/nvcc
-- Caffe2: CUDA toolkit directory: /usr
-- Caffe2: Header version is: 10.1
-- Found cuDNN: v8.0.4  (include: /usr/local/cuda/include, library: /usr/local/cuda/lib64/libcudnn.so)
-- Autodetected CUDA architecture(s):  6.1
-- Added CUDA NVCC flags for: -gencode;arch=compute_61,code=sm_61
-- Configuring done
CMake Warning at CMakeLists.txt:7 (add_executable):
  Cannot generate a safe runtime search path for target example-app because
  files in some directories may conflict with libraries in implicit
  directories:

    runtime library [libnvToolsExt.so.1] in /usr/lib/x86_64-linux-gnu may be hidden by files in:
      /usr/local/cuda/lib64
    runtime library [libcufft.so.10] in /usr/lib/x86_64-linux-gnu may be hidden by files in:
      /usr/local/cuda/lib64
    runtime library [libcurand.so.10] in /usr/lib/x86_64-linux-gnu may be hidden by files in:
      /usr/local/cuda/lib64

  Some of these libraries may not be found correctly.


-- Generating done
-- Build files have been written to: /home/joe/Documents/torchscript/build
</code></pre>
<p>I am a complete noob with Linux and c++/cmake. I have looked far and wide but cannot find out how to solve this. I don’t understand enough what information I should provide for help but here’s what I know.</p>
<p>Nvidia-smi and nvcc --version tell me I have CUDA 11.1 and driver version 455.23.05. My LD_LIBRARY_PATH is:</p>
<pre><code class="lang-auto">/usr/local/cuda-11.1/lib64. 
</code></pre>
<p>My PATH is:</p>
<pre><code class="lang-auto">usr/local/cuda-11.1/bin:/home/joe/anaconda3/bin:/home/joe/bin:/home/joe/.local/bin:/home/joe/anaconda3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
</code></pre>
<p>I installed Pytorch 1.6 with Anaconda if that matters.</p>
<p>I am out of ideas on how to fix this. Any help is welcome!</p>

ptrblck: <p>Based on the output it seems the build files have been written successfully.<br>
Do you get an error in any step or just this warning?</p>

Joe_Harrison: <p>You are right, it does work! I just get these warnings</p>
Tensorboard problem1

lxm-001: <p>when i import torch.utils.tensorboard.SummaryWriter ,it tell me “cannot import name ‘SummaryWriter’ from partially initialized module ‘torch.utils.tensorboard’”. But i don’t know how to deal this problem.  if you know how to deal with it ,please tell me .thankyou.</p>

ptrblck: <p>Could you try to use:</p>
<pre><code class="lang-python">from torch.utils.tensorboard import SummaryWriter
</code></pre>
<p>instead?</p>

lxm-001: <p>i have found the solution of my question.It is because the file name has the same name as the library name.And thank you for answer my question <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=10" title=":smiley:" class="emoji" alt=":smiley:"></p>
How to use CrossEntropyLoss()

InfT: <p>Good morning.<br>
I’m using a Variational Autoencoder with this loss function:</p>
<pre><code class="lang-auto">def loss_function(self, x_hat, x, mu, logvar, β=1):
	# Reconstruction + β * KL divergence losses summed over all elements and batch
	loss = nn.CrossEntropyLoss()
	CE = loss(x_hat, x)
	KLD = 0.5 * torch.sum(logvar.exp() - logvar - 1 + mu.pow(2))
	return CE + β * KLD
</code></pre>
<p><code>x</code> is my input image of [batch_size, 1, 16, 16] size. When I start the training phase, it returns this error:</p>
<p><code>RuntimeError: Expected object of scalar type Long but got scalar type Float for argument #2 'target' in call to _thnn_nll_loss2d_forward</code></p>
<p>So I edit in this way: <code>CE = loss(x_hat, x.long())</code>, but returns this error:</p>
<p><code>RuntimeError: 1only batches of spatial targets supported (3D tensors) but got targets of size: : [256, 1, 16, 16]</code></p>
<p>How can I use cross entropy correctly?</p>

huahuanZ: <p>what is <code>x_hat</code>? The second argument of CE loss should be the labels.</p>

gphilip: <p>CrossEntropyLoss expects arguments of certain shapes and value ranges. Search for “Shape:” in <a href="https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html#torch.nn.CrossEntropyLoss" rel="noopener nofollow ugc">the documentation for this class</a>, for a precise description of what this criterion expects.</p>
How can I detect parking lots on an aerial picture?

Heg0Damask: <p>Hello,</p>
<p>I would like to know how to detect this type of object (car park see pictures below)<br>
) on arial pictures ? How can I create my train dataset and which model I have to use (maskRCNN, Unet other…) ?</p>
<p><a href="https://raw.githubusercontent.com/Heg0Damask/simpletest/master/picture1.png" rel="noopener nofollow ugc">Picture 1</a><br>
<a href="https://raw.githubusercontent.com/Heg0Damask/simpletest/master/picture2.png" rel="noopener nofollow ugc">Picture 2</a></p>
<p>Thank you for your help</p>
Efficient way to multiply two tensor with different lengths

huahuanZ: <p>Here is a mini example</p>
<pre><code class="lang-python"># A in normal representation: [[1., 2., 3.], [1., 2.], [1., 2., 3., 4.]]
# A in my situation
A = tensor[(1., 2., 3., 1., 2., 1., 2., 3., 4.])
# len_A indicates the real length of each element in A
len_A = [3, 2, 4]
weight_A = tensor([0.2, 0.3, 0.5])
# how can I efficiently make the calculation without a loop
# 3, 5, 9 are computed with len_A
A[:3] *= weight_A[0]
A[3:5] *= weight_A[1]
A[5:9] *= weight_A[2]
</code></pre>

tom: <p>Quite likely, one could use some index trickery, but what is a good way to achieve things really depends on what realistic shapes are.<br>
So A is a list of … tensors of shape … to … or so, each multiplied with a separate scalar.</p>
Setting a random seed or passing a generator to Modules using sampling

khatrishubham: <p>I would like to obtain deterministic/reproducible weight initialization when working with say Con1d, while limiting the deterministic state to the weight [default] initialization part only and retaining the random state for the rest of the torch modules. Following, non-functional, code expresses the idea I am trying to achieve</p>
<pre><code class="lang-auto">gen1 = torch.Generator()
gen1.manual_seed(123)
conv = nn.Conv1d(16, 33, 3, stride=2)
nn.init.kaiming_uniform_(conv.weight, a=0, mode='fan_in', nonlinearity='leaky_relu', generator=gen1)


gen2 = torch.Generator()
gen2.manual_seed(123)
conv2 = nn.Conv1d(16, 33, 3, stride=2)
nn.init.kaiming_uniform_(conv2.weight, a=0, mode='fan_in', nonlinearity='leaky_relu', generator=gen2)

assert conv2.weight == conv1.weight
</code></pre>
<p>Expected: True</p>
<p>Unfortunately <code>kaiming_uniform_</code> does not accept a generator. Is there a way to this?</p>
<p>This would make sure that the sampling from the distribution is same.</p>

ptrblck: <p>This sound like a valid feature request and I think a similar one was already created. In any case I would recommend to create this feature request also on GitHub so that it can be discussed with the code owners as well.</p>
<p>For now you could use the same approach of passing a <code>generator</code> to directly initialize tensors as a workaround e.g. via:</p>
<pre><code class="lang-python">x = torch.empty(3, 3)
gen = torch.Generator()

gen.manual_seed(2809)
x.normal_(mean=1., std=2., generator=gen)

x.normal_(mean=1., std=2.)

gen.manual_seed(2809)
x.normal_(mean=1., std=2., generator=gen)
</code></pre>

khatrishubham: <p>Thanks! I will set this up. I can also contribute directly there, although I am not familiar with torch development guidelines <img src="https://discuss.pytorch.org/images/emoji/apple/stuck_out_tongue.png?v=10" title=":stuck_out_tongue:" class="emoji" alt=":stuck_out_tongue:"></p>
<p>Any references?</p>

ptrblck: <p>Sounds great! After discussing the feature request in the GitHub issue, explain that you would be interested in working on this and check out the <a href="https://pytorch.org/docs/stable/community/contribution_guide.html">Contribution guide</a>, <a href="https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md">this doc</a>, and in case you are interested in a tutorial how to write your first PR, check out <a class="mention" href="/u/tom">@tom</a>’s great video <a href="https://lernapparat.de/fixing-your-first-pytorch-bug/">here</a>.</p>

khatrishubham: <p>Thanks! I will check these out <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Why is there a GPU with a utilization rate of 0 when running pytorch's DDP task?

siaimes: <p>I used four RTX3090 graphics cards to train the resnet18 model, but I don’t know why the utilization of one of the graphics cards is always 0.</p>
<p>Following this doc <a href="https://github.com/pytorch/examples/tree/master/imagenet" rel="noopener nofollow ugc">https://github.com/pytorch/examples/tree/master/imagenet</a> to run the following command:</p>
<pre><code class="lang-auto">python main.py --arch resnet18 --multiprocessing-distributed --world-size 1 --rank 0 --batch-size 1024 --epochs 90
</code></pre>
<p>The GPU utilization results recorded by Grafana are as follows:</p>
<p><img src="https://user-images.githubusercontent.com/34199488/133264499-827f286b-b57c-421e-a8c1-d66430de3fa5.png" alt="image" width="690" height="322"></p>

gphilip: <aside class="quote no-group" data-username="siaimes" data-post="1" data-topic="132001">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/siaimes/48/42198_2.png" class="avatar"> siaimes:</div>
<blockquote>
<p>why the utilization of one of the graphics cards is always 0</p>
</blockquote>
</aside>
<p>I am not familiar with the application that you used to make these graphs, but to me the gray box in the image seems to be reporting the usage at a particular instant (21:06:04), not for all the time. Is this not so?</p>

siaimes: <p>My statement may not be appropriate, but you can view the green curve, which is almost always at the bottom.</p>
Computational graph's working during updating parameters

Dmitriy1: <p>Hello everyone!<br>
Work with part of the code below:</p>
<pre><code class="lang-auto">#data initialization
x = torch.randn(10, 10)
y = torch.randn(10, 1)

# param initialization
w = torch.randn(10, 1, requires_grad=True)

print('W before updating is leaf == ', w.is_leaf)


n_epoch = 1
for epoch in range(n_epoch):
  loss = ((x @ w) ** 2).sum()
  loss.backward()
  w = w - w.grad * 0.01
  print('W after updating is leaf == ', w.is_leaf)
</code></pre>
<p>Let’s, that I wanna do updating parameters without context manager torch.no_grad().</p>
<pre><code class="lang-auto">w = w - w.grad * 0.01
</code></pre>
<p>Does it cause to building new computational graph? Because this operation is using requires_grad=True variable?<br>
Moreover, why <strong>w</strong> after updating become non-leaf variable?</p>
<p>I apologize in advance for the stupid question! Thank you!</p>
Weird behaviors when setting pre-trained weights on CNN networks

sjyk112: <p>Hi.</p>
<p>I’m training Mini-ImageNet datasets using VGG16 with pre-trained ImageNet parameters.<br>
First, load_state_dict() method was used to set pre-trained parameters in network.<br>
At this time, accuracy scored 55% in 1st epoch and it trained continuously.</p>
<p>Secondly, i set pre-trained parameters in network by assign weight tensor in module’s weight and bias attribute directly, like the code below. The ‘tensor’ in code block is just example.</p>
<pre><code class="lang-auto">for m in model.features.modules():

        if isinstance(m, nn.Conv2d) :
            with torch.no_grad():
                m.weight = nn.Parameter(tensor)
                m.bias = nn.Parameter(tensor)
        
        elif isinstance(m, nn.BatchNorm2d) :
            with torch.no_grad():
                m.weight = nn.Parameter(tensor)
                m.bias = nn.Parameter(tensor)
                m.running_mean = tensor
                m.running_var = tensor

</code></pre>
<p>At this time, 85% was scored in 1st epoch.</p>
<p>I checked all the assigned weights an biases are same each other between two method, after setting pre-trained parameters.</p>
<p>I don’t understand this behavior, and I wonder if there is any difference between the two methods.</p>
<p>Thanks for your attention.</p>
Finetunning model to downstream task. Attribute does not exist!

neuralNoob: <p>Hello, I have a pretrained model that I am trying to fine tune to a downstream task. However, when I create this new class and pass my model through the constructor. See below:</p>
<pre><code class="lang-auto">class MyModel(nn.Module):

 def __init__(self, pretrained_model, model_settings):

        super(MyModel, self).__init__()
   
        self.orig_d_a, self.orig_d_v = model_settings.orig_d_a, model_settings.orig_d_v
        self.d_a, self.d_v = 130, 130
        self.output_dim = model_settings.output_dim
        self.pre_trained = pretrained_model
        self.proj_a = nn.Conv1d(self.orig_d_a, self.d_a, kernel_size=1, padding=0, bias=False)
        self.proj_v = nn.Conv1d(self.orig_d_v, self.d_v, kernel_size=1, padding=0, bias=False)

    def forward(self, a, v):

        x_a = a.transpose(1, 2)
        x_v = v.transpose(1, 2)
        proj_x_a = x_a if self.orig_d_a == self.d_a else self.proj_a(x_a)
        proj_x_v = x_v if self.orig_d_v == self.d_v else self.proj_v(x_v)
        output = self.pre_trained(proj_x_a, proj_x_v)
        
        return output
</code></pre>
<p>I keep getting the following error:</p>
<pre><code class="lang-auto">  File "/localdisk//finetunning/src/models.py", line 31, in forward
    output = self.pre_trained(proj_x_a, proj_x_v)
  File "/localdisk/conda/miniconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in __getattr__

    raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'MyModel' object has no attribute 'pre_trained'
</code></pre>
<p>Any ideas why this is happening? thanks</p>

ptrblck: <p>I cannot reproduce the issue using your code snippet and <code>models.resnet18()</code> as the pretrained model after removing undefined code parts. Could you update your code and make it executable to that we could reproduce the error?</p>
Run-time increase in linux cluster

hanabi: <p>Hi,<br>
I am running my program for CPU-only inference in the linux cluster (LSF).<br>
The problem is that the more jobs are distributed, the more time is taken for forward() function.<br>
When the only job is running it takes only 0.01s for forward(), but if I submit several jobs, the time increases proportionally.<br>
I set torch::set_num_threads() but it does not work.<br>
If all jobs are distributed in the different machines it seems the time does not increase. But the nodes are limited, I cannot avoid time increment.<br>
Is there anyone who knows how to prevent time increasing in this case or suffering from the same problem?</p>

hanabi: <p>Let me answer myself <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<pre><code class="lang-auto">at::set_num_threads(int num)
</code></pre>
<p>solves the problem.</p>
Converting a tensor into a list of tensors

saran: <p>I have one tensors which have many tensors in it onto  which i want to iterate  through loop but i have to convert it to a “list of tensors” before iterating ,so how i can convert a tensors into a list of tensors for example P1 is a tensor  with 60 values in it and i want  a list of tensors with 60 tensors in it.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/f/b/fbfa8edcd1f078a64f4ab9c8df8cd46bc586c232.png" alt="2" data-base62-sha1="zX6F7vBlbLIrE1hlB0Cegfw6jwS" width="591" height="462"></p>

gphilip: <p>Does the following help?</p>
<pre><code class="lang-auto">tensors = torch.randn(10,10)
print(tensors.shape)
for tensor in tensors:
    print(tensor.shape)
</code></pre>

saran: <aside class="quote no-group" data-username="gphilip" data-post="2" data-topic="131998">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/g/858c86/48.png" class="avatar"> gphilip:</div>
<blockquote>
<pre><code class="lang-auto">tensors = torch.randn(10,10)
print(tensors.shape)
for tensor in tensors:
    print(tensor.shape)
</code></pre>
</blockquote>
</aside>
<p>thank you for  response actually i need a list of tensors a list with 10 tensors in it (this example) unfortunately i cant append the tensors</p>

gphilip: <p>Does this work:</p>
<pre><code class="lang-auto">tensors = torch.randn(10,10)
tensorList = [None] * 10
print(tensors.shape)
for idx, tensor in enumerate(tensors):
    tensorList[idx] = tensor

for tensor in tensorList:
    print(tensor.shape)
</code></pre>

ryli: <pre><code class="lang-auto">p2 = []

torch.manual_seed(0)
p1 = torch.rand(60)

num = p1.shape[0]
for i in range(num):
   p2.append(p1[i])
</code></pre>
<p>This may be crass, but it’ll populate p2 with tensors of every element in p1</p>
<p>Would something like this be useful to iterate through p1?</p>
<pre><code class="lang-auto">for i in torch.arange(p1.shape[0]):
     print(p1[i])
</code></pre>

saran: <p>thank you yes its actually worked but now i realized that i should actually get a tensors of tensors</p>

ryli: <p>I’m not entirely sure what you’re wanting the output to look like, but you could simply try reshaping p1 with <a href="https://pytorch.org/docs/stable/generated/torch.reshape.html" rel="noopener nofollow ugc">torch.reshape</a> or <a href="https://pytorch.org/docs/stable/generated/torch.Tensor.view.html" rel="noopener nofollow ugc">torch.view</a></p>
[AV Discussion] How can Lidar segmentation work for vegetation? Why bother with Lidar Segmentation for AV?

tlim: <p>It seems that most Autonomous Vehicles benchmarks for Lidar Segmentation have included the vegetation class. Example:</p>
<ol>
<li>Semantic Kitti: <a href="http://www.semantic-kitti.org/dataset.html#classes" class="inline-onebox" rel="noopener nofollow ugc">SemanticKITTI - A Dataset for LiDAR-based Semantic Scene Understanding</a>
</li>
<li>NuScenes: <a href="https://www.nuscenes.org/nuscenes?externalData=all&amp;mapData=all&amp;modalities=Any" rel="noopener nofollow ugc">https://www.nuscenes.org/nuscenes?externalData=all&amp;mapData=all&amp;modalities=Any</a>
</li>
</ol>
<p>How is it possible that point cloud data can discriminate for vegetation? It seems to me that this is task is only viable for camera RGB segmentation.</p>
<p>For example, if the vegetation is overgrown onto a part of the road, how can the Lidar tell the differences between debris and vegetation? I am also curious about how can Lidar differentiate between terrain and sidewalk (+ is this important?).</p>
<p>What is the core purpose of Lidar Segmentation for AV? Besides drivable space,  all other detections (ex: cars &amp; humans) that it makes seem like it requires to be post-processed to 3D boxes to be of any use downstream (might else well do 3DOD). <strong>Are there any developers in this space (AV) deploying Lidar Segmentation model for any particular use cases?</strong></p>
Cannot import name 'TranslationDataset' from 'torchtext.datasets'

albertjohnson: <p>When I import TranslationDataset form torchtext.data, I get the following error:<br>
ImportError: cannot import name ‘TranslationDataset’ from ‘torchtext.datasets’<br>
I have installed pytorch correctly and can import torchtext.data and  torchtext.datasets, but why can’t I import TranslationDataset.<br>
My pytorch version is 1.9.0+cu11</p>

ptrblck: <p><code>TranslationDataset</code> was moved in <a href="https://github.com/pytorch/text/pull/1172">this PR</a> to the <code>legacy</code> namespace as seen <a href="https://github.com/pytorch/text/blob/e3799a6eecef451f6e66c9c20b6432c5f078697f/torchtext/legacy/datasets/translation.py#L13">here</a> and which is also why you cannot find it in the current docs anymore, so you might be able to import it via <code>torchtext.legacy.datasets</code>.</p>
Loading in a large tensor just to take a small crop takes too long

josh1: <p>I’m trying to train a GAN and loading each batch takes about 3-4x as long as the actual training for each batch. Is this normal? It doesn’t seem to be the result of the transformations I apply. The data is stored as PyTorch tensors in .pt files.</p>
<p>Each tensor is of shape (3601, 3601), which is massive, but I only load each one in order to take a single random crop of size (256, 256), which then gets put into the batch. I strongly suspect the issue is that I am using torch.load() to load in these massive images even though I end up needing only like 0.5% of their data.</p>
<p>Is there any easy way to do this crop without loading in the entire image? Even if it requires me changing the way I have my data stored. If there is no easy way, would it be doable to write my own loading function that builds the cropped image directly from the.pt file containing the larger image?</p>
<p>Thanks in advance for any advice.</p>

ptrblck: <p>You could check, if <a href="https://numpy.org/doc/stable/reference/generated/numpy.memmap.html"><code>np.memmap</code></a> could work to load only parts of a larger file into memory, which might speed up your data loading.</p>
Iteration over a 0-d tensor

Amin_Asghari: <p>Hi, I use pytorch 1.4.0 but I have a problem:</p>
<pre><code> raise TypeError('iteration over a 0-d tensor')
 TypeError: iteration over a 0-d tensor
</code></pre>
<p>How can I solve this?</p>

ptrblck: <p>You cannot iterate a 0-dim tensor, which is a single scalar.<br>
Either unsqueeze the tensor to create a single dimension or use the value directly.</p>

Amin_Asghari: <p>Thank you for your reply but I did it on colab and it’s error is:<br>
PyTorch 0.4.1 is not supported (needs to be 0.3.1)</p>

ptrblck: <p>I’m a bit confused, as your first post mentioned you are using <code>1.4.0</code>, while now you are downgrading PyTorch to <code>0.4.1</code> (or <code>0.3.1</code>)?</p>
<p>Try to stick with <code>1.4.0</code> and either unsqueeze the tensor or don’t try to iterate a scalar value.</p>

Zubair: <p>In case of loss function, if you try to return more values than the loss function is returning then this error erupts.</p>
Efficiency of predictions using GPU vs CPU

andreys42: <p>I have Inference routine consist of prediction using GPU:</p>
<pre><code class="lang-auto">for i in data: 
  pred_subcube=  model(data[i].cuda())
</code></pre>
<p>Where data[i] is item (Tensor of shape  5x1x128x128x128, where 5 is batchsize, 1 channels, and 128x128x128 is shape of my 3D image) of DataLoader class, model is my U-Net like 3D CNN model.<br>
And then I would like to set ‘pred’ as part of numpy array :</p>
<p><code>resulting_cube[i] = pred_subcube.detach().cpu().numpy()</code></p>
<p>Is it efficient way in my case to predict using GPU and then sent it to CPU to fill my “resulting_cube” array? Isn’t it more efficient to predict using CPU to not waste time to detach.</p>

gphilip: <p>I am not an expert on this so take it for what it is worth, but my hunch is that if the model is bigger than some small limit then the time for detaching the results will be negligible compared to the time for computing them on the CPU.</p>

andreys42: <p>that’s exactly the idea of my question) how deep have to be my model to use it with GPU and then detach</p>

gphilip: <p>The one sure way to figure this out is to run some timed tests in either mode and compare. I am sorry, but I don’t know enough to make a good guess.</p>

andreys42: <p>yeah, thats nice suggestion, but I thought there are some thoughts from experts who already tested thouse routines</p>
How to avoid -inf while implementing binary cross entropy loss?

zihaog: <p>Hi,<br>
I found sometimes the bce loss hit -inf with my bce loss. My implementation is shown below:</p>
<pre><code class="lang-auto"># imagine the model ouputs a tensor with shape (N, 2)
# GT is one-hot encoding of shape (N, 2)
prob = torch.softmax(output, dim=1)
loss = torch.sum(-torch.log(prob) * gt, dim=1)
loss = torch.mean(loss)
</code></pre>
<p>In some cases, the model is overconfident and outputs two values with a very large difference, e.g. tensor([[-100, 100]]). The -100 will cause the softmax function give 0 and then torch.log() gives -inf.</p>
<p>Is there any way to deal with it ?  Thanks a lot.</p>
<p>Best,</p>

ptrblck: <p>You could use the built-in loss functions, such as <code>nn.BCEWithLogitsLoss</code> or <code>nn.CrossEntropyLoss</code>, which increase the numerical stability e.g. via <code>logsumexp</code> or by clipping.</p>

zihaog: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>,<br>
I‘ve tried the built-in functions and they are indeed numerical stable.<br>
But I’m curious that why is my implementation not numerical stable. How to make it more stable?</p>
<p>Best,</p>

ptrblck: <p><a href="https://timvieira.github.io/blog/post/2014/02/11/exp-normalize-trick/">This blog post</a> and <a href="https://en.wikipedia.org/wiki/LogSumExp">this Wikipedia article</a> explain the subtraction of the <code>max</code> value for numerical stability.</p>
Python Order of execution of instructions

saran: <p>How python can print instructions sequentially fallowing the order  ?<br>
I am running a lengthy code  of python in which there are many loops and return functions and have a lot of computations  but when I  am printing some of the values to have a look of the output in pycharm IDE (run  console)   the order of execution is not sequential (maybe due to the computational time of different functions).Due to this the printed data is mixed and and some values are lost how I cant print all the values step by step (according to the written sequence of program)</p>

googlebot: <p>that’s crazy, you’ll only get mixed output if you’re running python functions concurrently, in which case they are unordered by definition. you can only reconcile order of finished jobs, but this implies that you’re explicitly launching jobs and then collecting results in an ordered fashion (what you want to print should be a part of these results, not printed directly from ‘worker’ threads/processes)</p>

saran: <p>yes you are right I  was running functions concurrently I will study about reconcile order of jobs thank you very much</p>
Loading weights from a pretrained model to a new model with added layer

avalon1511: <p>Hi, I have model A which is pretrained and Model B which is new. Model B has all of the layers of model A  + an extra layer.</p>
<p>I want to load the weights from Model A → B for the layers they have in common.</p>
<p>I implemented the following:</p>
<pre><code class="lang-auto">pretrained_path = torch.load(path to pretrained model)


new_model_dict = model.state_dict()

pretrained_weights = { k:v for k , v in pretrained_path.items() if k in new_model_dict}

new_model_dict.update(pretrained_weights)

model.load_state_dict(pretrained_weights, strict = False)
</code></pre>
<p>I had to use <code>strict = False</code> because the extra layer in model B was giving me an error. But this has made my results pretty bad.</p>
<p>How do I fix this? I want to avoid <code>strict = False</code></p>

ptrblck: <p>In case you are using the same layer names, you could iterate the <code>named_modules</code> in both models and load the <code>state_dict</code> per layer. If that’s not the case, you could create a mapping between the names of the pretrained <code>modelA</code> layers and the <code>modelB</code> layers and do the same. This would avoid the <code>strict=False</code> usage.</p>

avalon1511: <p>Thanks, this approach worked!</p>
Plotting a tensor

Nil_MSh: <p>Hello! I have a tensor <code>avg_act['a']</code> of size (64,2000) and I try</p>
<pre><code class="lang-auto">plt.imshow(avg_act['a'][idx1])
</code></pre>
<p>and get this error: <code> Invalid shape (128,) for image data</code><br>
How can I fix this?</p>

Unity05: <p>Hi,<br>
your tensor needs to have at least 2 dimension for imshow. Check <a href="https://matplotlib.org/stable/api/_as_gen/matplotlib.pyplot.imshow.html" rel="noopener nofollow ugc">the docs</a> for reference.</p>

Nil_MSh: <p>now that I have one dimension what should I use? Is there anything?</p>

Unity05: <p>You could just unsqueeze the data you pass to imshow.</p>

Nil_MSh: <p>I don’t know how. Could you please explain?</p>

Unity05: <p>By using <a href="https://pytorch.org/docs/stable/generated/torch.unsqueeze.html" rel="noopener nofollow ugc"><code>example_tensor.unsqueeze(-1)</code></a> for instance.</p>

Nil_MSh: <p>thank you! I did that and now the code works without errors but it doesn’t show any plots. what is wrong? I’m using python 3.7 on CPU.</p>

Unity05: <p>Have you forgotten to use <code>plt.show()</code>?</p>

Nil_MSh: <p>Yes! Thank you it works now</p>
Getting error when loading adam optimizer state dict

jkking: <p>Hi,</p>
<p>I’m working on training a light GAN on Google Colab and thus need to resume training over a couple of sessions. I can save and load the generator and discriminator state dicts A-OK, but get the following error when trying to load the adam optimizer state dicts:</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
&lt;ipython-input-27-cbfbe7dc0130&gt; in &lt;module&gt;()
      4 G.cuda()
      5 D.cuda()
----&gt; 6 g_optimizer.load_state_dict(torch.load('/content/drive/MyDrive/GAN_Temp/g_optim.pth'))
      7 d_optimizer.load_state_dict(torch.load('/content/drive/MyDrive/GAN_Temp/d_optim.pth'))

/usr/local/lib/python3.7/dist-packages/torch/optim/optimizer.py in load_state_dict(self, state_dict)
    135         # Validate the state_dict
    136         groups = self.param_groups
--&gt; 137         saved_groups = state_dict['param_groups']
    138 
    139         if len(groups) != len(saved_groups):

TypeError: 'method' object is not subscriptable
</code></pre>
<p>I would save the the state dicts like so:</p>
<pre><code class="lang-auto">torch.save(G.state_dict(), '/content/drive/MyDrive/GAN_Temp/G.pth')
torch.save(D.state_dict(), '/content/drive/MyDrive/GAN_Temp/D.pth')
torch.save(g_optimizer.state_dict, '/content/drive/MyDrive/GAN_Temp/g_optim.pth')
torch.save(d_optimizer.state_dict, '/content/drive/MyDrive/GAN_Temp/d_optim.pth')
</code></pre>
<p>Originally I was trying to save them at the end of every so-and-so batch iterations in my training loop, but I’ve also tried to let a full few epochs complete and run the save steps above, then reload the state dict before a new running the training loop again – same results.</p>
<p>I feel like I’m probably just doing something silly, but any ideas?</p>

ptrblck: <p>You are storing the <code>state_dict</code> methods in:</p>
<pre><code class="lang-python">torch.save(g_optimizer.state_dict, '/content/drive/MyDrive/GAN_Temp/g_optim.pth')
torch.save(d_optimizer.state_dict, '/content/drive/MyDrive/GAN_Temp/d_optim.pth')
</code></pre>
<p>and would need to call this method to store the actual <code>state_dict</code>:</p>
<pre><code class="lang-python">torch.save(g_optimizer.state_dict(), '/content/drive/MyDrive/GAN_Temp/g_optim.pth')
torch.save(d_optimizer.state_dict(), '/content/drive/MyDrive/GAN_Temp/d_optim.pth')
</code></pre>

jkking: <p>I am a fool! Not sure why I was thinking those were attributes, but I’ll leave my ignorance here for others. Thanks!</p>
DistributedDataParallel hurts the model performance

xdwang0726: <p>I am wondering is it reasonable that using multi-gpus parallel hurts the model’s performance? (I am using DistributedDataParallel, btw)</p>
<p>For single gpu, I use the batch size 32, and to test on 2 gpus, I set batch size equals to 16 for each gpu, so the overall batch size will be 32 which is same to the single gpu. (other hyper parameters stay the same)  The performance of the model drops a lot when using 2 gpus. I am wondering is there any possible reason why it happens?</p>
<p>Thanks!</p>

ptrblck: <p>Your model’s performance might depend on the batch size e.g. if it’s using batchnorm layers.<br>
Since you are using a smaller batch size on each rank now, the performance could change, and you could check <code>SyncBatchNorm</code> if that’s indeed the root cause.</p>
Let's discuss **project architecture**

alexandre-blanc: <p>I spend a lot of time building projects in PyTorch and have not yet found a nice structure that encapsulates everything I want to do in a clean, decoupled way.</p>
<p>Mostly I want to decouple :</p>
<ul>
<li>model definition</li>
<li>data management (preprocessing, dataloaders, I/O)</li>
<li>training</li>
<li>evaluation</li>
<li>logging</li>
</ul>
<p>My current approach would be to use a class handling all data management issues, and a class containing the network definition.</p>
<p>Training and evaluation would be performed by different classes, let’s call them “engines”. Those engines would have the Data and Model class as attributes, and a logging as well.</p>
<p>How do you structure your own project ? Are there certain caveats to having an object as an attribute of multiple different objects ?</p>
<p>Let’s start a discussion !</p>

ptrblck: <p>Your approach sounds a bit like a higher-level API, such as <code>Catalyst</code>, <code>Ignite</code>, or <code>Lightning</code>, which uses layers of abstractions for these training, logging, checkponting etc. routines so you might want to take a look at them.</p>
Resources for PyTorch

Mohamed_Taha: <p>I’m new here, I used to learn AI from scratch, and then I used the Tensorflow framework, I found that TensorFlow is more API like blocks build together to make the entire model, you can’t learn strange things, you can’t build the imagination model you would like, you can’t even make Reinforcement learning algorithms and can’t make complex functions and learning algorithms, so I decided to learn PyTorch.<br>
I didn’t find free sources to learn PyTorch<br>
I would like to get free sources starting from tensors and ends with more difficult algorithms in PyTorch</p>

gphilip: <p>The excellent (and free) <a href="http://d2l.ai/" rel="noopener nofollow ugc">Dive into Deep Learning book</a> has a Pytorch track.</p>

Mohamed_Taha: <p>Special thanks to you, and if there is videos or lessons please</p>
.to('cuda') taking super long time

thavens: <p>device: gtx 1070<br>
tensor info:<br>
variable shape of approx: [150, 6]<br>
dtype: torch.int16<br>
I’ve also tested float32 but got the same results</p>
<pre><code class="lang-auto">tq = tqdm(trainset)
for i in tq:
    optimizer.zero_grad()
    tic = time.time()
    pred = model(i[0].to(device))
    toc = time.time()
    #This is what is taking so long
    i[1] = i[1].cuda()#.to(device)
    tooc = time.time()
    loss = test(pred, i[1])
    
    loss[0].backward()
    optimizer.step()
    times = loss[1]
    times['inference'] = int((toc-tic) * 1000)
    times['to cuda!?!?'] = int((tooc-toc) * 1000)
    tq.set_postfix(times)
</code></pre>
<p>output:<br>
build anchors=40, big loop=114, inference=90, <strong>to cuda!?!?=573</strong><br>
units: ms.</p>

gphilip: <p>See if</p>
<pre><code class="lang-auto">i[1] = i[1].cuda(non_blocking=True)
</code></pre>
<p>makes a difference.</p>
<p>I <em>think</em> that for this to make a difference you have to specify <code>pin_memory=True</code> as an argument while constructing the <code>DataLoader</code> object, but I am not entirely sure. If merely setting <code>non_blocking=True</code> does not give any improvement, and if you do use a <code>DataLoader</code> object to construct <code>trainset</code>, then try passing <code>pin_memory=True</code> to its constructor, as well.</p>

ptrblck: <p>CUDA operations are executed asynchronously, so you would need to synchronize the code before starting and stopping each timer via <code>torch.cuda.synchronize()</code>. Otherwise the times will be accumulated in blocking operations and are thus wrong.</p>
Torchscript Dict problems

DSL: <p>Hello guys,</p>
<p>I have the following problem. I want to convert my model with torchscript, but it throw a convert error.</p>
<pre><code class="lang-auto">This attribute exists on the Python module, but we failed to convert Python type: 'dict' to a TorchScript type. Cannot infer concrete type of torch.nn.Module. Its type was inferred; try adding a type annotation for the attribute.):
</code></pre>
<p>I have a dictonary e2c = {} and this dictonary will be used like this for example:</p>
<pre><code class="lang-auto">e2c["5"] = Cube2Equirec(512, 1024, 2)
</code></pre>
<p>later it should be called like this in the forward method:</p>
<pre><code class="lang-auto">equi = c2e["5"](cube)
</code></pre>
<p>How i program it right that it can be converted by torchscript?</p>
spaCy dependency break in its release version 2.2.4

Dennis_LD: <p>In this tutorial <a href="https://pytorch.org/tutorials/beginner/translation_transformer.html" rel="noopener nofollow ugc">Machine translation with transformer</a>, the colab cell break because spaCy dependency break, that leads to can’t load tokenizer error.</p>
<p>The original installation is this</p>
<pre><code class="lang-python"># pip install -U spacy
# python -m spacy download en_core_web_sm
# python -m spacy download de_core_news_sm
</code></pre>
<p>The fix is this (validated by me)</p>
<pre><code class="lang-python">pip install spacy==2.2.1
python -m spacy download en
python -m spacy download de
</code></pre>
Grad is none outside torch.no_grad

Mox_P: <p>Hi, I’m facing an issue about the usage of torch.no_grad. I’m using something like this :</p>
<pre><code class="lang-auto"> with torch.no_grad():
                    logits ,_,_ = self.model(input_images)
                    logits = logits.detach()


_, loss_1, _ = self.model(images_second)

....

_, loss_2, _ = self.model(input_images, gt=logits)

total_loss = loss_1 + loss_2
total_loss.backward()
</code></pre>
<p>When I’m printing the gradient of my network, I get something like :</p>
<pre><code class="lang-auto">Grad is None :
...
params_X.weight
params_Y.weight
...


Grad is not None :
...
params_X.norm.weight
params_Y.norm.bias
....
</code></pre>
<p>I don’t understand why the grad of the parmaters are None, when I remove the torch.no_grad(), all params have a grad. Could anyone helps me on this ?</p>
<p>Thanks</p>

gphilip: <p>If you could provide a small executable example (with a toy network and random inputs), then I can try figuring it out.</p>
How to get computing resource freely

sunshine1: <p>The GPU is so expensive so that it is unavailable for me. I have to wait very long time for a training. How to solve this problem?I also want to  learn deep learning</p>

gphilip: <p>You could try the free GPU time offered by <a href="https://colab.research.google.com/" rel="noopener nofollow ugc">Google Colab</a> or <a href="https://www.kaggle.com" rel="noopener nofollow ugc">Kaggle</a>.</p>
Convert conv1d keras to pytorch

Oui_Zal: <p>I want to convert this model from keras to pythorch, but I have problem that I don’t know the output of each layer, please can someone convert it to me or tell me how can I calculate the output of each layer. Thank you</p>
<pre><code class="lang-auto">nclass = 5
    inp = Input(shape=(187, 1))
    #img_1= Sequential()(inp)
    img_1 = Convolution1D(64, kernel_size=2, activation=activations.relu, padding="valid")(inp)
    img_1 = Convolution1D(64, kernel_size=2, activation=activations.relu, padding="valid")(img_1)
    img_1= BatchNormalization()(img_1)
    img_1 = MaxPool1D(pool_size=2)(img_1)
    img_1 = Dropout(rate=0.1)(img_1)
    img_1 = Convolution1D(128, kernel_size=3, activation=activations.relu, padding="valid")(img_1)
    img_1 = Convolution1D(128, kernel_size=3, activation=activations.relu, padding="valid")(img_1)
    img_1= BatchNormalization()(img_1)
    img_1 = MaxPool1D(pool_size=2)(img_1)
    img_1 = Dropout(rate=0.1)(img_1)
    img_1 = Convolution1D(128, kernel_size=3, activation=activations.relu, padding="valid")(img_1)
    img_1 = Convolution1D(128, kernel_size=3, activation=activations.relu, padding="valid")(img_1)
    img_1= BatchNormalization()(img_1)
    img_1 = MaxPool1D(pool_size=2)(img_1)
    img_1 = Dropout(rate=0.1)(img_1)
    img_1 = Convolution1D(256, kernel_size=3, activation=activations.relu, padding="valid")(img_1)
    img_1 = Convolution1D(256, kernel_size=3, activation=activations.relu, padding="valid")(img_1)
    img_1= BatchNormalization()(img_1)
    img_1 = GlobalMaxPool1D()(img_1)
    img_1 = Dropout(rate=0.1)(img_1)
    


    dense_1 = Dense(1000, activation=activations.relu, name="dense_1")(img_1)
    dense_1 = Dense(1000, activation=activations.relu, name="dense_2")(dense_1)
    dense_1 = Dense(nclass, activation=activations.softmax, name="dense_3_mitbih")(dense_1)

    model = models.Model(inputs=inp, outputs=dense_1)
    opt = tf.optimizers.Adam(0.001)

    model.compile(optimizer=opt, loss=losses.sparse_categorical_crossentropy, metrics=['acc'])
    model.summary()
    return model
</code></pre>

gphilip: <p>You could print out the shape of the output computed by each layer. In PyTorch this would look like <code>print(img_1.shape)</code>. You could use the equivalent keras code.</p>