akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to make channels?

Sarath_Raveendran: <p>Hi,<br>
I am new to PyTorch. I have a doubt about converting a data frame with 6 columns and 50,000 rows. I want to form 6 channels with these 6 columns and 100 rows such that it can be given as input to a CNN that takes input with 6 channels.  Could any of you please help me to sort this out?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/0/10fc9dd2ab6ef2b52a7bb78736e09766b63cb27b.png" data-download-href="https://discuss.pytorch.org/uploads/default/10fc9dd2ab6ef2b52a7bb78736e09766b63cb27b" title="Screenshot from 2020-10-11 13-31-17"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/0/10fc9dd2ab6ef2b52a7bb78736e09766b63cb27b_2_690x182.png" alt="Screenshot from 2020-10-11 13-31-17" data-base62-sha1="2qgRO6wbYFvB5ie9W3fEm9hNKwr" width="690" height="182" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/0/10fc9dd2ab6ef2b52a7bb78736e09766b63cb27b_2_690x182.png, https://discuss.pytorch.org/uploads/default/original/3X/1/0/10fc9dd2ab6ef2b52a7bb78736e09766b63cb27b.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/1/0/10fc9dd2ab6ef2b52a7bb78736e09766b63cb27b.png 2x" data-dominant-color="F0F1F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2020-10-11 13-31-17</span><span class="informations">959×253 23.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

a_d: <p>I assume you would want your data sample to be of the shape - <code>[batch, 6, 100]</code><br>
So Initialize a NumPy array of that shape<br>
<code>data = numpy.zeros((1, 6, 100)) #batch size equal to 1</code><br>
and then populate this NumPy array the way you want.</p>
<p>Then convert this Numpy array to a torch tensor using -<br>
<code>torch_tensor = torch.from_numpy(data)</code></p>
<p>if you want to batch it, you could stack these tensors on top of each other at the <code>0th</code> dimension making the input of shape <code>batch_size, 6, 100</code></p>
Can loss vary when overfitting a single batch?

spadel: <p>I am trying to overfit a single batch in order to test, whether my network is working as intended. I would have expected, that the loss should keep decrease as long as the learning rate isn’t too high. What I observe, however, is that the loss in fact decreases over time, but it fluctuates strongly. Is that a sign, that I have a flaw in my architecture?</p>

Kushaj: <p>Use small batch size (like 2). Also, this test only tells if the model has enough capacity to learn the data, so if you are able to reach a loss of 0, then it means that you passed the test.</p>

spadel: <p>I am trying that. But I can’t reach zero. My question is exactly the following: Should the loss <strong>strictly</strong> decrease provided a sufficiently small learning rate while overfitting or can it vary?</p>

Kushaj: <p>It should gradually decrease. (some fluctuation is ok, but not strong fluctuation as in your case). If the model is correct the fluctuation might be due to bad hyperparameters (probably lr or momentum). Also, do not use any data augmentation, weight decay, dropout or any other fancy regularization trick.</p>

spadel: <p>I can’t believe I oversaw that - I had a random augmentation activated within the Dataloder! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> Thanks for pointing that out!</p>
Why do we have ~torch?

JonathanSum: <p>I see people do it in     mask = ~torch.eye(n_samples, device=sim.device).bool()<br>
What does the ~ for?<br>
Thank you.</p>

ptrblck: <p>The tilde operation is a bitwise negation and should yield the same output as <code>torch.bitwise_not</code>.</p>
Is there a way to use pretrained networks on CIFAR10 without altering the Image size from 32 x 32 to 224 x 244?

Kalyan: <p>I would like to exploit the pre-trained networks on CIFAR10 but don’t want to alter the image size from 32 x 32 to 224 x 244.</p>

ptrblck: <p>For some models the input might be too small since some intermediate activations might be smaller than a specific conv kernel.<br>
If you encounter these issues, you would have to increase the input up to a specific size until at least a single pixel is produced by the feature extractor.<br>
Depending on the model architecture the minimal size might be smaller than <code>224x224</code>.</p>
Embedding Layer with 4D input

shashank_gupta1: <p>I have a input with dimension (Batch Size * MAX_NUMBER_SENTENCES * MAX_SENT_LENGTH). Basically a list-of-list-of token indices.</p>
<p>I have initialized an embedding layer as</p>
<p>self.query_embedding = nn.Embedding(len(word_lookup_table), inp_dim)<br>
I have initialized it with pretrained word embeddings. Now when I pass an input of the above dim, I get as output:</p>
<p>(BatchSize * Max_Num_Sents * Max_Sent_Len * Embed_Dim).</p>
<p>Is this the correct output?</p>

ptrblck: <p>Yes, that’s the expected output as shown in the <a href="https://pytorch.org/docs/stable/generated/torch.nn.Embedding.html#torch.nn.Embedding">docs</a>.<br>
<code>nn.Embedding</code> layers are adding the <code>embedding_dim</code> dimension as the last additional dimension to the output tensor.</p>

shashank_gupta1: <p>Thank you for your response.</p>
Python block/halt on importing torch

yude: <p>Hi everyone, I have developed a deep-learning object-detect program based on <code>pytorch</code> and it works very well. Today I deploy this program on a PC, everything goes well, but the program cannot be launched. Debug and find out that, the program blocks, or halts, on importing <code>pytorch</code>.</p>
<p>Simply start a python prompt, type <code>import torch</code> and the prompt blocks. <code>top</code> command shows that CPU/memory usage is very low. Press <code>ctrl-c</code> cannot stop the prompt. While other library importing is fine. I have tried <code>pycrypto</code> and the one I wrote myself, all work but <code>pytorch</code> cannot.</p>
<p>I have deployed more than 100 times, but never meet this situation. I also tried to reinstall pytorch, from 1.6 to 1.4, torchvision from 0.7 to 0.5, still not work. No error printed, no complain shown.</p>
<p>Environment:</p>
<ul>
<li>OS: centos 7.4</li>
<li>CUDA: 10.0</li>
<li>NVIDIA driver: 440.82</li>
<li>GPU: GTX 1660</li>
<li>python: 3.6</li>
</ul>
<p><code>pytorch</code> version: 1.6 and 1.4.</p>
<p>Any information are welcome, thanks in advance.</p>

ptrblck: <p>Did you deploy on this setup and system before?<br>
If so, what could have changed between the successful deployment and the issue now, i.e. did you change something in the OS/system?<br>
Also, could you check if you are able to run a simple PyTorch script in a docker container?</p>
Best practices multiple net output

donJuan: <p>So I have a task where the net outputs different variables. Some outputs are bounded like [0,1], [0, +inf] and [-inf, +inf]. I want the network to handle these restrictions. My thought was to do</p>
<pre><code class="lang-auto">def forward(self, x):
	x = self.net(x) 
	#print(x.shape), (bs, 5)

	# say idx 0 is the variable with [0,1] bound so
	x[:, 0] = nn.Sigmoid(x[:, 0])

	# idx 1-3 is [0, +inf] bound so
	x[:, 1:4] = nn.ReLU(x[:, 1:4])

	# idx 5 is [-inf, +inf] so just left as is.

	return x

</code></pre>
<p>Is this a valid approach? The model will be saved with jit.trace and served using the c++ api.</p>
<p>Offtopic. What you guys think of training with only MSE loss? Sure you can combine cross entrpoy loss for the 0-1 vars and MSE for the others but I wonder if it will have a major effect?</p>

ptrblck: <p>Using different activation functions sounds reasonable. However, I would recommend to check if your code would raise errors for invalid inplace operations before porting it to <code>libtorch</code>.<br>
If so, you might assign the results to temporal tensors and concatenate them afterwards.</p>
torchvision.transforms.Compose

jy_mo: <p>The function in the corresponding libtorch torchvision.transforms.Compose</p>

ptrblck: <p>I think <code>transforms</code> are still not completely ported and while some transformations already work on tensors instead of <code>PIL.Image</code>s, I don’t think <code>Compose</code> was ported yet.<br>
<a href="https://github.com/pytorch/vision/issues/885">This PR</a> seems related and I’m sure contributions are welcome in case you would be interested to port some functions. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
<p>CC <a class="mention" href="/u/fmassa">@fmassa</a> to correct me in case I’m missing something.</p>
Differential privacy library opacus in Pytorch error

Michelle_Barbosa: <p>Hello, I have been trying to implement Opacus to my models optimiser. I have run it on colab. At the time of training I run into this error.<br>
RuntimeError: CUDA error: CUBLAS_STATUS_ALLOC_FAILED when calling <code>cublasCreate(handle)</code><br>
Could someone please guide me on this.<br>
Thanks!</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/5/85cd90a3486a7765a0952b53cb495342c378442c.png" data-download-href="https://discuss.pytorch.org/uploads/default/85cd90a3486a7765a0952b53cb495342c378442c" title="Screen Shot 2020-10-09 at 11.03.33 AM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/5/85cd90a3486a7765a0952b53cb495342c378442c_2_690x488.png" alt="Screen Shot 2020-10-09 at 11.03.33 AM" data-base62-sha1="j5FVjU3sg587dsIJa32i2r3kipC" width="690" height="488" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/5/85cd90a3486a7765a0952b53cb495342c378442c_2_690x488.png, https://discuss.pytorch.org/uploads/default/original/3X/8/5/85cd90a3486a7765a0952b53cb495342c378442c.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/5/85cd90a3486a7765a0952b53cb495342c378442c.png 2x" data-dominant-color="F0F0F0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2020-10-09 at 11.03.33 AM</span><span class="informations">859×608 64.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>This error might be thrown if you are running out of memory and cublas isn’t able to create its handle.<br>
Could you check the memory usage via <code>nvidia-smi</code> and check if you are close to the device limit?<br>
If that’s the case, try to lower the batch size and rerun the code.</p>
PYTORCH1.8 build hava error

liu: <p>WINDOWS10 VS2017 build error Q:\pytorch1\pytorch\aten\src\ATen/core/boxing/impl/boxing.h(150): error C2210: “_Ty”: 包扩展不能被用作别名模板中非打包参数的自变量 (编译源文件 Q:<br>
pytorch1\pytorch\torch\csrc\jit\frontend\schema_type_parser.cpp) [Q:\pytorch1\pytorch\build\caffe2\torch_cpu.vcxproj]</p>

ptrblck: <p>Please use an online translation service, so that more users can help you. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>From Google translate:</p>
<pre><code class="lang-auto">WINDOWS10 VS2017 build error Q:\pytorch1\pytorch\aten\src\ATen/core/boxing/impl/boxing.h(150): error C2210: “_Ty”: Package extension cannot be used as non-packaged parameter in alias template Argument (compiled source file Q:
pytorch1\pytorch\torch\csrc\jit\frontend\schema_type_parser.cpp) [Q:\pytorch1\pytorch\build\caffe2\torch_cpu.vcxproj]
</code></pre>
Faster-RCNN input data in training mode

Sergey_Gryaznov: <p><strong>Hello there</strong>. Can anyone help me to understand docs <a href="https://github.com/pytorch/vision/blob/master/torchvision/models/detection/faster_rcnn.py" rel="noopener nofollow ugc">link</a>.<br>
documentation says:</p>
<blockquote>
<p>The input to the model is expected to be a list of tensors, each of shape [C, H, W], one for each<br>
image, and should be in 0-1 range. Different images can have different sizes.<br>
The behavior of the model changes depending if it is in training or evaluation mode.<br>
During training, the model expects both the input tensors, as well as a targets (list of dictionary),<br>
containing:<br>
- boxes (FloatTensor[N, 4]): the ground-truth boxes in [x1, y1, x2, y2] format, with values<br>
between 0 and H and 0 and W<br>
- labels (Int64Tensor[N]): the class label for each ground-truth box</p>
</blockquote>
<p>Unfortunately, I still don’t understand the input format <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=12" title=":smile:" class="emoji" alt=":smile:" loading="lazy" width="20" height="20">  Based on documentation, class expect:</p>
<pre><code class="lang-auto"># list of tensor 
images = [tensor([C,H,W]), tensor([C,H,W]), tensor([C,H,W])]

# list of dictionary
targets = [{'boxes': tensor([1,4]), 'labels': tensor([1])}, 
 'boxes': tensor([1,4]), 'labels': tensor([1])},
 'boxes': tensor([1,4]), 'labels': tensor([1])}]
</code></pre>
<p>And if we have more bounding boxes for each images then boxes will be <code>tensor([N_boxes, 4])</code> and label <code>tensor([N])</code>, I’m right?</p>
<p>And one more question about <code>__getitem__</code> . I don’t know how to return a list of dictionary in this format…<br>
I do</p>
<pre><code class="lang-auto">targets = [{'boxes': bbox, 'labels': label}]
return img, targets
</code></pre>
<p>and this is wrong, I suppose…<br>
i.e. I have only one bounding box and label for each image and I get shapes (where 8 is batch size):</p>
<pre><code class="lang-auto"># shape of images
torch.Size([8, 3, 224, 224])
# shape of targets
[{'boxes': torch.Size([8, 1, 4]),
 'labels': torch.Size([8])}]
</code></pre>
<p>Thanks for your attention</p>

Sergey_Gryaznov: <p>I solved problems</p>
<ol>
<li>Yes, class expect a list of dictionary</li>
<li>
<code>__getitem__</code> is correct, the problem was in DataLoader and batches which he returned</li>
</ol>
<p>solution</p>
<pre><code class="lang-auto">def collate(batch):
    return tuple(zip(*batch))

dataloader = DataLoader(dataset, batch_size=5, collate_fn=collate)
</code></pre>
<p>And I get list of dictionary</p>

xtian_carlos: <p>Thank you so much!. I had stuck in this part.</p>
How remove filter from layers and initialize custom filter weights to layers

Adeel: <p>I have to remove filter from layers.<br>
VGG16 model, here are two layer<br>
model = models.vgg16(pretrained=True)<br>
print(model)<br>
print(model.features[2].weight.shape)           #   features[2].weight has 64 filters<br>
torch.Size([64, 64, 3, 3])<br>
print(model.features[5].weight.shape)           #   features[5].weight has 128 filters<br>
torch.Size([128, 64, 3, 3])</p>
<p>Is there any way (function or method) to remove filter#53 from model.features[2].weight and filter#109 from features[5].weight.</p>
<p><strong>Also:</strong><br>
How to initialize model.features[5].weight of VGG16 with custom filter weights of same shape.</p>
<p>model = models.vgg16(pretrained=True)<br>
print(model)<br>
print(model.features[5].weight.shape)<br>
torch.Size([128, 64, 3, 3])</p>
<p>Thanks</p>

Kushaj: <p>You can remove features by doing <code>torch.cat([a[:53], a[54:]])</code>.</p>
<p>For  initialization you can directly operate on <code>model.features[5].weight</code> tensor or pass the tensor to some method of <code>torch.nn.init</code> to initialize it.</p>

Adeel: <p>Thank you very much Kushajveer Singh</p>
Very huge memory of batch multiplication and addition

wxystudio: <p>I want to do torch.bmm for tensor[12000, 1200, 1] and tensor[12000, 1, 1920], so I get tensor[12000, 1200, 1920], then I want to do torch.sum( , dim = 0) to it, finally I get tensor[1200, 1920].<br>
But the process is very memory-consuming, How can I do it <strong>without for loop?</strong></p>

googlebot: <p>that’s equivalent to a single matrix multiplication: 1200x12000 @ 12000x1920 = 1200,1920</p>
Not all model parameters are transfered to GPU

mortonjt: <p>I’m noticing that if I run <code>model.to(device)</code>, the tensors used in the <code>forward</code> are not necessarily transferred over the GPU. Below is a minimal example of this</p>
<pre><code class="lang-python">import torch
from torch.distributions import MultivariateNormal

class SimpleModel(torch.nn.Module):
    def __init__(self, q):
        super(SimpleModel, self).__init__()
        self._zero_mean = torch.zeros(2 * q)
        self._eye_covar = torch.eye(2 * q)
        self.mvn = MultivariateNormal(self._zero_mean, self._eye_covar)
    def reparameterize(self, mu, logv):
        eps = torch.randn_like(mu)  
        z = mu + eps * torch.exp(logv)  
        logp = self.mvn.log_prob(eps) 
        return z, logp        
    def forward(self, x):
        return self.reparameterize(x, x)

model = SimpleModel(10).to('cuda')
x = torch.ones(20).to('cuda')
model(x)
</code></pre>
<p>This gives the following error</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-15-bd0200007a4a&gt; in &lt;module&gt;
----&gt; 1 model(x)

~/miniconda3/envs/mavi/lib/python3.8/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    548             result = self._slow_forward(*input, **kwargs)
    549         else:
--&gt; 550             result = self.forward(*input, **kwargs)
    551         for hook in self._forward_hooks.values():
    552             hook_result = hook(self, input, result)

&lt;ipython-input-10-7cdded9571ea&gt; in forward(self, x)
     14         return z, logp
     15     def forward(self, x):
---&gt; 16         return self.reparameterize(x, x)

&lt;ipython-input-10-7cdded9571ea&gt; in reparameterize(self, mu, logv)
     11         eps = torch.randn_like(mu)
     12         z = mu + eps * torch.exp(logv)
---&gt; 13         logp = self.mvn.log_prob(eps)
     14         return z, logp
     15     def forward(self, x):

~/miniconda3/envs/mavi/lib/python3.8/site-packages/torch/distributions/multivariate_normal.py in log_prob(self, value)
    205         if self._validate_args:
    206             self._validate_sample(value)
--&gt; 207         diff = value - self.loc
    208         M = _batch_mahalanobis(self._unbroadcasted_scale_tril, diff)
    209         half_log_det = self._unbroadcasted_scale_tril.diagonal(dim1=-2, dim2=-1).log().sum(-1)

RuntimeError: expected device cuda:0 but got device cpu
</code></pre>
<p>I know that I can manually override the <code>to()</code> and <code>cuda()</code> methods, so there is a solution. But I’m curious why the <code>to()</code> and <code>cuda()</code> can’t already handle these sorts of edge cases (or if there is something that I am overlooking).</p>

albanD: <p>Hi</p>
<p>Yes I don’t think the distribution objects are moved around by the nn.Modules.<br>
But If you want <code>_zero_mean</code> and <code>_eye_covar</code> to be moved around, you need to register them as buffers in the nn.Module.</p>
C++17 compatibility

oni: <p>Currently, the C++ API forces C++14 flag during compilation. (-std=c++14)</p>
<p>Are there any plans to support C++17 in the near future?<br>
Cuda 11.1 is already C++17 compatible.</p>
<p>Thx</p>

oni: <p>Hi All,</p>
<p>after changing “c++14” to “c++17” in all occurrences within /usr/lib/cmake/Caffe2/*.cmake, i can compile and run code successfully in C++17 mode.</p>
<p>It seems that the libtorch code base is already prepared for C++17, its just the compatibility w.r.t the CUDA env. which has to be maintained. In my case, i’m using CUDA 11.1, which is C++17 ready.</p>
<p>I guess it should not be too hard to enable C++17 when the installed CUDA version supports it… (e.g., by checking in cmake scripts…)</p>
<p>Cheers,<br>
Oni</p>
My pre-trained model is giving negative and floating point value instead of positive (MNIST digit reognizer)

akib62: <p>I have a <code>test_dataloader</code> and it contains <code>14000 tensors</code>. My test images are 28000 and I am taking <code>batch = 2</code>. I have a <code>pre-trained model</code> and now I am trying to test my existing model.</p>
<p>However, my <code>testing.py</code> output is showing <code>negative floating point number</code> but it must be <code>positive integer number</code>. Full code <a href="https://github.com/mostafiz67/MNIST-PyTorch" rel="noopener nofollow ugc">Github Link</a></p>
<p>Could tell me what I have to change?</p>
<pre><code class="lang-auto">final_output = []
for i, data in enumerate(test_data):
    data = data.unsqueeze(1)
    output = model(data).cpu().detach().numpy()
    data = None

    final_output.append(output)
result = np.concatenate(final_output)

print((result))
</code></pre>
<p>The output is like this:</p>
<pre><code class="lang-auto">[[-4.397916    0.7076113   2.1967683  ...  0.06060949 -2.8013513
 -8.800405  ]
[-3.533296   -3.1798646  -5.6163416  ... -4.7265635  -1.8589627
  0.5682605 ]
[-1.8575612   3.9310014  -4.122321   ... -1.2687542  -3.5150855
 -5.7542324 ]
...
[-8.762509   -8.240637   -2.7152536  ... -1.5188062  -5.932935
  2.6340218 ]
[-1.9312052  -2.675097   -2.2223709  ... -1.4572031  -8.078956
 -4.047556  ]
[-1.9931098  -2.840486   -3.620531   ... -2.5536153  -1.735633
 -2.317892  ]]
</code></pre>
<p>Any kind of suggestion is appreciable.</p>

Kushaj: <p>I think you meant to do <code>data = data.unsqueeze(0)</code>. Also, it might be softmax is not included in the pretrained model (as softmax is usually combined with the loss function, so we have to manually add softmax at test time).</p>

akib62: <p><a class="mention" href="/u/kushaj">@Kushaj</a> after giving <code>0</code> getting the error</p>
<pre><code class="lang-auto">RuntimeError: Given groups=1, weight of size [32, 1, 3, 3], expected input[1, 2, 28, 28] to have 1 channels, but got 2 channels instead

</code></pre>

akib62: <p>Soved the problem using <code>LongTensor</code></p>
Export computation graph

amiyamandal-dev: <p>is there any way to export the actual computation graph of a model that is trained or not? Since ‘model.state_dict()’ give the weights in an ordered dict. By some manual coding, It can be made as JSON. Theoretically, weights can be sent as JSON over the net. If it possible to generate sent computation graph as follows</p>
<pre><code class="lang-auto">class linearRegression(nn.Module):
    def __init__(self):
        super(linearRegression, self).__init__()
        self.linear = nn.Linear(1, 1)

    def forward(self, x):
        out = self.linear(x)
        return out

model = linearRegression()

model.state_dict()

#OrderedDict([('linear.weight', tensor([[0.6289]])),
#            ('linear.bias', tensor([0.7113]))])
</code></pre>
<p>but when I have to export to libtorch(C++), I have to make an exact same copy of the class to use the model. I guess there might be an easy way which I don’t know that to an export graph from different language</p>

ptrblck: <p>You can use the JIT and script the model as described <a href="https://pytorch.org/tutorials/advanced/cpp_export.html">here</a>.</p>
I got an error after training lenet5 model

chamanthi_aki: <p>hey guys , i got error during the training of model. can you help me!<br>
<a href="https://jovian.ml/chamschamanthi36/lenet5-m1" rel="noopener nofollow ugc">here is the notebook</a></p>

ptrblck: <p>Your current notebook shows a <code>KeyboardInterrupt</code>, which seems to be created by the user.<br>
Could you post the error you are seeing here?</p>
Multiple input in VGG16

anotherone_one: <p>Hello everyone!<br>
I wanted to ask how could I add a first layer to the pre-trained vgg16 in a way that my input to the network would be 3 RGB images, so my input shape would be 500x500x3x3 ! I only was able to see how to change this first layer to change the dimension of the image, for example from 500x500x3 to 500x500x1, but I don’t know how to add that 4th dimension.</p>
<p>Thank you in advance</p>

ptrblck: <p>How should this layer work for 3 input images?<br>
I.e. do you want to apply a <code>nn.Conv3d</code> layer and treat the multiple images as the depth dimension or apply an <code>nn.Conv2d</code> layer for each image?<br>
In the latter case, how would the output be defined?</p>
Convolution with different kernel for each pixel

DenisDiachkov: <p>Hello, I want to implement Conv2d with different kernels for each pixel. Is there any way to make it in pytorch and keep it as efficient as regular convolution?</p>

ptrblck: <aside class="quote no-group" data-username="DenisDiachkov" data-post="1" data-topic="98669">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/denisdiachkov/48/29566_2.png" class="avatar"> DenisDiachkov:</div>
<blockquote>
<p>Is there any way to make it in pytorch</p>
</blockquote>
</aside>
<p>Yes, you could start by implementing the desired work flow using e.g. nested for loops and once it’s working use <code>unfold</code> to apply a matmul to speed it up, if that fits your use case.</p>
<aside class="quote no-group" data-username="DenisDiachkov" data-post="1" data-topic="98669">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/denisdiachkov/48/29566_2.png" class="avatar"> DenisDiachkov:</div>
<blockquote>
<p>and keep it as efficient as regular convolution?</p>
</blockquote>
</aside>
<p>That won’t be straightforward, but you could e.g. write a custom CUDA extension for your convolution to get another speedup.</p>
Load list of directory to Dataloader

Zila_Tajudin: <p>Hello,</p>
<p>Is there any way to pass a list of directories contain in a text file to the data loader?<br>
Because I only want a certain files from the folder and not all of it.<br>
If I use root it will load all the files in the folder.</p>
<p>Thanks</p>

ptrblck: <p>You can create a custom <code>Dataset</code> as described on <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html">this tutorial</a>, where you could filter out or specify your files directly.</p>
WHY batch_size=1 and num_workers>0 still facilitate loading speed

t.ouyang: <p>Recently in my project, I need to go deeper on how <code>Dataloader</code> works.<br>
I know that when setting <code>num_workers&gt;0</code> can make loading data into RAM faster,<br>
BUT when using <code>batch_size=1</code> why the loading speed still follows <code>num_workers=4</code>&gt;<code>num_workers=2</code>&gt;<code>num_workers=0</code>?</p>
<p>The dataset defines commonly as:</p>
<pre><code class="lang-auto">def __getitem__(self, index):
    path=get_datafile(index)
    data=pil_loader(path)
    return data, get_target(index)
</code></pre>
<p>Are workers caching data into RAM when <code>num_workers&gt;0</code> and how do they know which <code>index</code> to cache?<br>
Or one sample (<code>batch_size</code>=1) is loaded by several workers?</p>

ptrblck: <p>Each worker loads a complete batch, i.e. <code>batch_size</code> samples.<br>
The used indices are specified by the sampler or if no sampler is defined by the <code>shuffle</code> option in the <code>DataLoader</code>, which will internally create a <code>SequentialSampler</code> or a <code>RandomSampler</code>.</p>
<p>How did you profile the code and what does</p>
<aside class="quote no-group" data-username="t.ouyang" data-post="1" data-topic="98649">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/t.ouyang/48/14275_2.png" class="avatar"> t.ouyang:</div>
<blockquote>
<p>BUT when using <code>batch_size=1</code> why the loading speed still follows <code>num_workers=4</code> &gt; <code>num_workers=2</code> &gt; <code>num_workers=0</code> ?</p>
</blockquote>
</aside>
<p>mean?</p>
Parallelizing a training code using a couple of P100/V100 GPUs

Mona_Jalal: <p>Is there a way to parallelize the code in this repo which is written in PyTorch? The training approximately takes 1 month on a 1080Ti GPU, so I am wondering if I can run it much faster say on 8 P100 or V100 GPUs?<br>
I am not familiar which part of code I should change to do so?<br>
</p><aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="16" height="16">
      <a href="https://github.com/furkanbiten/GoodNews" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars3.githubusercontent.com/u/22663927?s=400&amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/furkanbiten/GoodNews" target="_blank" rel="noopener nofollow ugc">furkanbiten/GoodNews</a></h3>

<p>Good News Everyone! - CVPR 2019 . Contribute to furkanbiten/GoodNews development by creating an account on GitHub.</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>I am running it with this command:<br>
$ python train.py --cnn_weight [YOUR HOME DIRECTORY]/.torch/resnet152-b121ed2d.pth</p>
<p>I am using PyTorch 1.6.0</p>
<p>Their train.py code is here:<br>
</p><aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/furkanbiten/GoodNews/blob/master/train.py" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/furkanbiten/GoodNews/blob/master/train.py" target="_blank" rel="noopener nofollow ugc">furkanbiten/GoodNews/blob/master/train.py</a></h4>
<pre><code class="lang-py"># Use tensorboard

from __future__ import absolute_import
from __future__ import division
from __future__ import print_function

import torch
import torch.nn as nn
from torch.autograd import Variable
import torch.optim as optim
import time
from six.moves import cPickle
import opts
import models
from dataloader import *
import eval_utils
import misc.utils as utils
try:
    import tensorflow as tf

</code></pre>

  This file has been truncated. <a href="https://github.com/furkanbiten/GoodNews/blob/master/train.py" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


ptrblck: <p>You could use a data parallel approach to use multiple GPUs for the training.<br>
<a href="https://pytorch.org/tutorials/beginner/dist_overview.html#data-parallel-training">This tutorial</a> gives you an overview of different approaches.</p>
ResNet32 Architecture

Angelina_Robert: <p>Hi, I tried to replace ResNet50 architecture with ResNet32 in one model to find the accuracy of the model. But it generated the following error. Any help in this regard would be appreciated.</p>
<p>size mismatch, m1: [256 x 10], m2: [512 x 512] at /opt/conda/conda-bld/pytorch_1565272271120/work/aten/src/THC/generic/THCTensorMathBlas.cu:273</p>

Kushaj: <p>What is your input size and other sizes? An easy way to debug your problem is to add <code>print(tensor.shape)</code> statements in the model.</p>
Average of vector words

slavavs: <p>My transformer network gives me a vector of size 100x128x250 (100 is the number of words, 128 is the batch, 250 is feature_size). I want to convert forward (). 100x128x250 -&gt; 128x250 (128 average). Then 128x250 -&gt; 128x3 (here I know it as Linear (250, 3). How to do this? 100x128x250 -&gt; 128x250</p>

Kushaj: <p><code>torch.mean(x, axis=1)</code></p>
How to take weighted combination of models and calculate the gradient wrt combination weight?

vkverma: <p>Suppose that I have two model M1 and M2. I want to make a joint model M=a1<em>M1+a2</em>M2, later I want to calculate the gradient of M wrt a1 and a2 (i.e. wrt combination weight).</p>
<p>Could anyone suggest how to do that?</p>
<p>Thanks</p>

ptrblck: <p>If you define <code>a1</code> and <code>a2</code> as tensors with <code>requires_grad=True</code> or as <code>nn.Parameters</code>, Autograd will compute their gradients in the backward pass.<br>
I’m not sure how the "gradient of <code>M</code> wrt <code>aX</code>" would be calculated in your example, but if you compute a loss using <code>M</code>, you would be able to compute the gradient of this loss function with respect to the parameters.</p>
<p>I’m not familiar with your use case, but note that optimizing these parameters (<code>aX</code>) could just push them to negative values, which would then “minimize” the loss.</p>

vkverma: <p>Thanks, <a class="mention" href="/u/ptrblck">@ptrblck</a> for the reply. My main concern is how to make joint model M so that model knows about all the M1, M2etc.<br>
I tried M=a1<em>M1.parameters()+a2</em>M2.parameters()<br>
But now model M does not know about M1 and M2.<br>
Here my objective is using a convex combination of a1 and a2 to learn a joint model that minimize the loss.</p>
<p>Thanks</p>

googlebot: <aside class="quote no-group" data-username="vkverma" data-post="3" data-topic="98566">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/v/8797f3/48.png" class="avatar"> vkverma:</div>
<blockquote>
<p>M=a1 <em>M1.parameters()+a2</em> M2.parameters()</p>
</blockquote>
</aside>
<p>This line suggests that you want to do parameter averaging with replicated networks. That won’t work as network re-runs converge to distinct local minima.</p>
<p>OTOH, network output averaging (ensembles) can work, but you normally need a staged training for that. I.e. combination weights should be trained with independently pre-trained contributing models.</p>

vkverma: <p>Thanks, <a class="mention" href="/u/googlebot">@googlebot</a> for your reply.</p>
<p>Actually, my idea is something different. Suppose that I initialise the a1=a2=0.5, then make the gradient of M1 and M2 false and pass to the data to M and find out y=M(x). Now calculate the entropy (E) of y take the gradient of E wrt a1 and a2. Then based on the gradient of a1 and a2 decide which model will minimise the entropy. In this way, I can select the model M1 or M2 just by a single pass, instead of passing data to all model.</p>
<p>For the two models, it looks costly but if we have many models then it can be much efficient. Instead of linear computation growth wrt the number of model, it will be much efficient. The idea is something like <a href="https://arxiv.org/abs/2006.14769" rel="noopener nofollow ugc">https://arxiv.org/abs/2006.14769</a> (NeurIPS, 20)</p>
<p>Thanks</p>

googlebot: <p>That paper links to a pytorch implementation. Looks pretty intrusive, but perhaps that’s necessary.</p>
Optical flow Forward Warping

SamTX: <p>Hi,</p>
<p>From papers related to optical flow &amp; respective code, it’s clear how backward warping works e.g. say you have 2 images I1 and I2, using the forward flow map F12, backward warping BW(I2, F12) results in I1. I want to know how one could use such flow maps to get reproduce I2? so this I believe is forward warping.</p>
<p>I tried to follow the following thread: <a href="https://discuss.pytorch.org/t/solved-how-to-do-the-interpolating-of-optical-flow/5019" class="inline-onebox">[Solved]How to do the interpolating of optical flow?</a><br>
but no solution was posted from what I saw. Any help would be highly appreciated</p>
Save and load model pytoch

adam_kossemtini: <p>hello guys,<br>
i had been save a train model ResNet in collab but if i load this model with spyder (ubuntu) i have this error (RuntimeError: [enforce fail at inline_container.cc:144] . PytorchStreamReader failed reading zip archive: failed finding central directory)<br>
this is the save and the load !<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/6/3/63372cd2708ba9ca2dd19c673ef74a1bcde1bb45.png" alt="image" data-base62-sha1="e9HtRsX3Bfo97uKEqi6lTK3fnUx" width="386" height="50"></p>

Keyv_Krmn: <p>Are you using torchscript model? if so, you should use torch.jit.save for saving a torchscript model.</p>

ptrblck: <p>Besides what <a class="mention" href="/u/keyv_krmn">@Keyv_Krmn</a> mentioned, check if you are using the same PyTorch versions, since loading a <code>state_dict</code> should be backwards compatible but not necessarily forwards compatible.<br>
I.e. storing a <code>state_dict</code> in an older PyTorch version and loading it with a newer one should work, while the opposite direction could fail.</p>
Pseudo Labelling for self-training

jwngo: <p>Hi,</p>
<p>I am trying to implement pseudo labelling for self-training.</p>
<p>I have a trained model which I use to run evaluations on unlabeled data. After getting the logits, i softmax them which gives me a tensor of [7x368x640].</p>
<p>However, when i do torch.save() to save these predictions for future training with a model with both labeled and unlabeled dataset, I realise that these files are 76M for just 1 image.</p>
<p>This is too large for a dataset &gt;100k images.</p>
<p>Is there another way I should implement pseudolabelling?</p>

ptrblck: <p>Each of these tensors should take approx. 6.3MB as seen here:</p>
<pre><code class="lang-python">x = torch.randn(7, 368, 640)
print('expected size ', x.nelement() * 4 / 1024**2, 'MB')
&gt; expected size  6.2890625 MB
torch.save(x, 'tmp.pt')

print('file size ', os.stat('tmp.pt').st_size / 1024**2, 'MB')
&gt; file size  6.289786338806152 MB
</code></pre>
<p>Even if you could reduce the size by zipping the data, you would still need to process it later and thus unzip it.<br>
Based on the shape it seems you won’t be able to compress the outputs using an image format.</p>
Data loader struct pack issue(overflow)

rui_zhang_331: <p>I have created a issue in pytorch repo also: <a href="https://github.com/pytorch/pytorch/issues/43467" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/issues/43467</a></p>
<p>I am using pytorch data loader and runs into some strange error as below, we generate training data in each hour. and I run this experiment to just read training data csv file and parse line by line, no real training stuff is going. we define the dataloader  <code>__get_item_ </code> to get a single file out of bunch of files(&gt;5000 files). and each worker dataloader is suppose to read that file and parse csv.</p>
<p>I checked it should not be  <code>__get_item</code>  index out of boundary, is it because some training file we loaded is too large? it is not always repeatable error, so I am not sure it may be related to some training data? is it possible one training file was too large and basically the multi worker data loader overflow?</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/home/miniconda/lib/python3.6/multiprocessing/queues.py", line 240, in _feed
    send_bytes(obj)
  File "/home/miniconda/lib/python3.6/multiprocessing/connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "/home/miniconda/lib/python3.6/multiprocessing/connection.py", line 393, in _send_bytes
    header = struct.pack("!i", n)
struct.error: 'i' format requires -2147483648 &lt;= number &lt;= 2147483647
</code></pre>
<p>looks like multi processing can not handle more than 2G data, anyway we can enlarge multi processing data handle size?</p>
I don't know how to use GRU Moves but does not learn

skerlet_flandorle: <p>I don’t know how to use GRU<br>
It moves but doesn’t learn<br>
This is the first time I’ve used an RNN layer in pytorch, so I think this is the cause.<br>
Please tell me if there is not enough information I will add it</p>
<p>code<br>
<span class="hashtag">#Definition</span><br>
self.rnn=nn.GRU(self.hidden_size1,gru_hidden)<br>
self.norm=nn.LayerNorm(gru_hidden)<br>
self.Relu=swish(0.7)</p>
<p><span class="hashtag">#runtime</span><br>
x=self.Linear(x)<br>
x,hidden=self.rnn(x,hidden)<br>
onry_x=self.Relu(self.norm(x))</p>
<p>Causes i can think of<br>
1<br>
Insufficient understanding of gru and proper initialization (for example, he、Set bias to 0)<br>
I don’t even know how to access weight bias</p>
<p>2<br>
There is a tanh in the GRU (in the official description)<br>
3<br>
Something is missing</p>
<p>​</p>

shartzog: <p>I’m having trouble understanding your issue.  Could you post some executable code?</p>

skerlet_flandorle: <p>If you put all of them, it will be long, so put only a part</p>
<pre><code>loss=torch.sum(output.permute(1,2,0),dim=2)-torch.sum(targets.detach().permute(1,2,0),dim=2)
    
optimizer.zero_grad()
loss=loss__(loss)#(loss**2)/2
loss.backward()
torch.nn.utils.clip_grad_norm_(mainQN.parameters(),1,2)
optimizer.step()
scheduler.step()
self.tortal_losses+=loss.clone().detach().to('cpu')</code></pre>

skerlet_flandorle: <p>target.size=time,batch,output<br>
output.size=time,batch,output</p>
<p>input.size=time,batch,20</p>
<h1></h1>
<p>model</p>
<pre><code>  Definitionself.rnn = nn.GRU（self.hidden_​​size1、gru_hidden）
  self.norm = nn.LayerNorm（gru_hidden）
  self.Relu = swish（0.7）
  
  #runtime
  x = self.Linear（x）
  x、hidden = self.rnn（x、hidden）
  onry_x = self.Relu（self.norm（x）
</code></pre>
<p>I’m doing something a little different after GRU. Is this a problem?<br>
There should be no problem in itself as it is from the code before the introduction of GRU</p>
<pre><code>   L = self.L(x).view(bacth_size*timee, self.num_outputs,self.num_outputs)
    tril_mask = torch.tril(torch.ones(self.num_outputs,self.num_outputs), diagonal=-1).unsqueeze(0).to("cuda:0")
    diag_mask = torch.diag(torch.diag(torch.ones(self.num_outputs,self.num_outputs))).unsqueeze(0).to("cuda:0")
    L = L * tril_mask.expand_as(L) + torch.exp(L) * diag_mask.expand_as(L)
    P = torch.bmm(L, L.transpose(2, 1))</code></pre>

skerlet_flandorle: <p>In the explanation of pytorch, it looks like this</p>
<pre><code>   \begin{array}{ll}
    r = \sigma(W_{ir} x + b_{ir} + W_{hr} h + b_{hr}) \\
    z = \sigma(W_{iz} x + b_{iz} + W_{hz} h + b_{hz}) \\
    n = \tanh(W_{in} x + b_{in} + r * (W_{hn} h + b_{hn})) \\
    h' = (1 - z) * n + z * h
    \end{array}
</code></pre>
<p>I don’t want to use tanh<br>
Is there a way to change to Relu or swish?</p>
Feature maps in CNN and discriminative loss

Hdk: <p>Suppose we have a CNN with five convolutional layers<br>
defined as (in_features, out_features, kernel, stride,padding) and no linear layers. Consider a dataset say Cifar10.</p>
<p>Conv1: (3, 32, 5, 1, 0)<br>
Conv2: (32, 64, 5, 1, 0)<br>
Conv3: (64, 128, 5, 1, 0)<br>
Conv4: (128, 256, 5, 1, 0)<br>
And output layer as convolutional layer itself.<br>
Conv5: (256, 10, *, 1, 0)</p>
<ol>
<li>
<p>Is it possible to get feature maps of one convolutional layer to be far different then its following convolutional layer? If so, how can we do it?</p>
</li>
<li>
<p>I want to calculate the loss of feature maps with respect to original image for each convolutional layer. And add it up to as usual cross entropy loss. i.e.</p>
</li>
</ol>
<p>Loss1 = my_Loss_func (feature maps, images)<br>
Loss2 = torch.nn.CrossEntropyLoss(outputs,labels)<br>
loss = Loss1 + Loss2<br>
.<br>
loss.backward()<br>
…</p>
<p>But I think there’ll be error of size mismatch in Loss1 because now for 1 image we have feature maps= out_features(say 32 if it’s conv1).</p>
<ol start="3">
<li>Does, the way I am defining Loss1, make it a discriminative loss? To be honest, I am not really sure about it. But I have to use discriminative loss, and I didn’t find much resources on how it can be implemented in context of feature maps.</li>
</ol>
Complex<double> access violation, cannot understand this one!

Anders_G: <p>I am calling some torchlib functions from c# and want to send some data from c# to libtorch. For this Im using torch::from_blob function with some pinned memory in c#. And it works really well, except for one case. If the data is ComplexDouble I get an access violation. But accessing the same memory with any other datatype is no problem at all.</p>
<p>I’ve traced the issue to assigning a complex to the memory buffer that comes from c#. But the really strange thing is that assigning anything else to this buffer is no problem at all. Only if it is a complex(double). In the code below the <em>data</em> is a memory corresponding to 2 complex doubles (i.e. 32 bytes).</p>
<pre><code class="lang-auto">        double* ddata = (double*)data;
        ddata[0] = 2.0;
        ddata[1] = 3.0;

        complex&lt;float&gt;* cfdata = (complex&lt;float&gt;*) data;
        cfdata[0] = complex&lt;float&gt;(2, 3);
        cfdata[1] = complex&lt;float&gt;(2, 3);

        complex&lt;double&gt;*cdata = (complex&lt;double&gt;*) data;
        cdata[0] = complex&lt;double&gt;(2, 3);
</code></pre>
<p>Running this code I get “Exception thrown at 0x00007FFFD839DF02 (cstorchlib.dll) in Examples.exe: 0xC0000005: Access violation reading location 0xFFFFFFFFFFFFFFFF.”. I get this error on the last of the lines above. And no matter what I do I cannot get an exception if I assign with some other datatype. I even tried creating my own complex double data structure and it was no problem to use.</p>
<p>So does anyone have any idea what is going on? Im quite lost and also surprised by the fact that I get a reading violation rather than writing.</p>

Anders_G: <p>I found the issue. c# has 8 byte aligned data (x64 mode) in arrays while complex&lt;double&gt; requires 16 bytes alignment. Unfortunately there is no way to make c# align to 16 bytes …</p>
Initialize hidden state with identity matrix

smu226: <p>Hello! I have an LSTM which looks like this:</p>
<pre><code class="lang-auto">class BD_LSTM(nn.Module):
    def __init__(self, nl):
        super().__init__()
        self.nl = nl
        self.rnn = nn.LSTM(1, n_hidden, nl, bidirectional=True) 
        self.l_out = nn.Linear(n_hidden*2, n_classes)
        self.init_hidden(bs)
        
    def forward(self, input):
        outp,h = self.rnn(input.view(len(input), bs, -1), self.h)
        return F.log_softmax(self.l_out(outp),dim=2)
    
    def init_hidden(self, bs):
        self.h = (V(torch.zeros(self.nl*2, bs, n_hidden)),
                  V(torch.zeros(self.nl*2, bs, n_hidden)))

model = TESS_LSTM(2).cuda()
</code></pre>
<p>I want to initialize the hidden layer of the LSTM to the identity matrix (I read this is better for convergence purposes). When I run this:</p>
<p><code>?model.rnn</code></p>
<p>one of the attributes (which should be the one I need) is this: weight_hh_l. This is the full output relevant for my question:</p>
<pre><code class="lang-auto">Attributes:
    weight_ih_l[k] : the learnable input-hidden weights of the :math:`\text{k}^{th}` layer
        `(W_ii|W_if|W_ig|W_io)`, of shape `(4*hidden_size x input_size)`
    weight_hh_l[k] : the learnable hidden-hidden weights of the :math:`\text{k}^{th}` layer
        `(W_hi|W_hf|W_hg|W_ho)`, of shape `(4*hidden_size x hidden_size)`
    bias_ih_l[k] : the learnable input-hidden bias of the :math:`\text{k}^{th}` layer
        `(b_ii|b_if|b_ig|b_io)`, of shape `(4*hidden_size)`
    bias_hh_l[k] : the learnable hidden-hidden bias of the :math:`\text{k}^{th}` layer
        `(b_hi|b_hf|b_hg|b_ho)`, of shape `(4*hidden_size)`
</code></pre>
<p>However, when I do this:</p>
<p><code>model.rnn.weight_hh_l[0].data.copy_(torch.eye(n_hidden))</code></p>
<p>I am getting this error:</p>
<p><code>AttributeError: 'LSTM' object has no attribute 'weight_hh_l'</code></p>
<p>What am I doing wrong? Thank you!</p>

Sanquen: <p>you don’t need to include the brackets, just suffix with the layer index directly<br>
for layer 0:<br>
<code> rnn.weight_hh_l0</code><br>
for layer 1<br>
<code>rnn.weight_hh_l1</code><br>
etc</p>