akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Padding 'same' exists for Conv1d but not for ConvTranspose1d

username: jpainam
cooked: <p>Hi,<br>
In <code>Conv1d</code>, the <code>padding</code> parameter can take the value of <code>same</code>. This doesn’t work for <code>convTranpose1d</code>. What is the best way to achieve this:</p>
<pre><code class="lang-python">conv1 = .nn.Conv1d(3, 16, kernel_size=25, padding='same')
x = torch.randn(4, 3, 15)
out = conv1(x)
&gt;&gt;&gt; out.shape
torch.Size([4, 16, 15])
conv1dtranspose = nn.ConvTranspose1d(16, 3, kernel_size=25)
v = conv1dtranspose(out)
&gt;&gt;&gt; v.shape
torch.Size([4, 3, 39])
</code></pre>
<p>What’s the best way to get <code>torch.Size([4, 3, 15])</code> from <code>conv1dtranspose</code>?<br>
Thank you</p>
Training retinanet_resnet50_fpn model

username: user30
cooked: <p>Hi everyone!<br>
Im trying to train this model: <a href="https://pytorch.org/vision/stable/_modules/torchvision/models/detection/retinanet.html" class="inline-onebox" rel="noopener nofollow ugc">torchvision.models.detection.retinanet — Torchvision 0.11.0 documentation</a>.</p>
<p>During inference (model.eval()) the model returns some “scores”.</p>
<ol>
<li>How can i obtain the loss values from the model in evaluation mode? Can calculate the loss = (1 - score)?</li>
<li>Do this scores relate to classification or for bbox_regression?</li>
</ol>
<p>Thank you!</p>
Training stops after first epoch

username: Eshta
cooked: <p>I am working on compute canada platform. Initialiy had pytorch 1.10 Cuda 11.4 installed in the torch environment. I was successfully able to run the RetinaNet model for 200 epoch on a dataset of 465 training images. however when I ran the model for next 50 epochs I got CUDA OOM on the same cloud cluster. The tech team asked me to switch to torch1.9 and cUDA 10.1. The CUDA OOM error was gone but the training doesn’t go beyond first epoch and no error message displayed. The job shows timeout because it doesnt move beyond first epoch. I am using the resources as under<br>
no. of gpus -2<br>
no. of workers -6<br>
memory 64G<br>
no. of nodes -1<br>
total images -465 , reduced the batch size to 4</p>
<p>I have been stuck here for long and not found any solution . Ay help will be highly appreciated.</p>
<p>Thanks in advance.</p>

username: pbelevich
cooked: <p><a class="mention" href="/u/eshta">@Eshta</a> could you please share the code and environment(pytorch version/libraries)? Also, why are there 6 workers on 2 gpus?</p>
Retrieve the PyTorch model from a PyTorch lightning model

username: emiled16
cooked: <p>I have trained a PyTorch lightning model that looks like this:</p>
<pre><code>In [16]: MLP
Out[16]:
DecoderMLP(
  (loss): RMSE()
  (logging_metrics): ModuleList(
    (0): SMAPE()
    (1): MAE()
    (2): RMSE()
    (3): MAPE()
    (4): MASE()
  )
  (input_embeddings): MultiEmbedding(
    (embeddings): ModuleDict(
      (LCLid): Embedding(5, 4)
      (sun): Embedding(5, 4)
      (day_of_week): Embedding(7, 5)
      (month): Embedding(12, 6)
      (year): Embedding(3, 3)
      (holidays): Embedding(2, 1)
      (BusinessDay): Embedding(2, 1)
      (day): Embedding(31, 11)
      (hour): Embedding(24, 9)
    )
  )
  (mlp): FullyConnectedModule(
    (sequential): Sequential(
      (0): Linear(in_features=60, out_features=435, bias=True)
      (1): ReLU()
      (2): Dropout(p=0.13371112461182535, inplace=False)
      (3): LayerNorm((435,), eps=1e-05, elementwise_affine=True)
      (4): Linear(in_features=435, out_features=435, bias=True)
      (5): ReLU()
      (6): Dropout(p=0.13371112461182535, inplace=False)
      (7): LayerNorm((435,), eps=1e-05, elementwise_affine=True)
      (8): Linear(in_features=435, out_features=435, bias=True)
      (9): ReLU()
      (10): Dropout(p=0.13371112461182535, inplace=False)
      (11): LayerNorm((435,), eps=1e-05, elementwise_affine=True)
      (12): Linear(in_features=435, out_features=435, bias=True)
      (13): ReLU()
      (14): Dropout(p=0.13371112461182535, inplace=False)
      (15): LayerNorm((435,), eps=1e-05, elementwise_affine=True)
      (16): Linear(in_features=435, out_features=435, bias=True)
      (17): ReLU()
      (18): Dropout(p=0.13371112461182535, inplace=False)
      (19): LayerNorm((435,), eps=1e-05, elementwise_affine=True)
      (20): Linear(in_features=435, out_features=435, bias=True)
      (21): ReLU()
      (22): Dropout(p=0.13371112461182535, inplace=False)
      (23): LayerNorm((435,), eps=1e-05, elementwise_affine=True)
      (24): Linear(in_features=435, out_features=435, bias=True)
      (25): ReLU()
      (26): Dropout(p=0.13371112461182535, inplace=False)
      (27): LayerNorm((435,), eps=1e-05, elementwise_affine=True)
      (28): Linear(in_features=435, out_features=435, bias=True)
      (29): ReLU()
      (30): Dropout(p=0.13371112461182535, inplace=False)
      (31): LayerNorm((435,), eps=1e-05, elementwise_affine=True)
      (32): Linear(in_features=435, out_features=435, bias=True)
      (33): ReLU()
      (34): Dropout(p=0.13371112461182535, inplace=False)
      (35): LayerNorm((435,), eps=1e-05, elementwise_affine=True)
      (36): Linear(in_features=435, out_features=1, bias=True)
    )
  )
)
</code></pre>
<p>I need the corresponding PyTorch model to use in one of my other applications.</p>
<p>Is there a simple way to do that?</p>
<p>I thought of saving the checkpoint but then I don’t know how to do it.</p>
<p>Can you please help?<br>
Thanks</p>
Graph Classification via Random Forest

username: Gianmarco_Luchetti_S
cooked: <p>Hello everybody!<br>
I’m a medicinal chemistry undergraduate student who is preparing his dissertation. My idea would be to create a classifier that can distinguish anticancer drugs as active or inactive and distinguish those active in three classes, describing the molecules as a graph. My supervisor suggested me to use the <a href="https://en.wikipedia.org/wiki/Random_forest" rel="noopener nofollow ugc">random forest</a> classifier, and to do this I need to convert my graph into a vector trying to keep as many characteristics as possible.</p>
<p>I start from a molecular graph dataset like this:</p>
<pre><code class="lang-auto">Data(x=[9, 9], edge_index=[2, 18], edge_attr=[18, 2], y=[0], smiles='COC(=O)C=CN1CC1')
</code></pre>
<p>where <code>x</code>, <code>edge_index</code> and <code>edge_attr</code> are a <a href="https://pytorch.org/docs/stable/tensors.html" rel="noopener nofollow ugc">torch tensor</a>, and <code>y</code> is the label (<code>0</code> is inactive, <code>1</code> is activity of class one, <code>2</code> is activity of class two, …). To run a random forest classifier I think I must convert them into a vector like a <code>np.array</code>, to do that I think I must to compute a kernel for my graphs, but I have no idea how to do it . Has anyone had experience on this task?</p>

username: anantguptadbl
cooked: <p><a class="mention" href="/u/gianmarco_luchetti_s">@Gianmarco_Luchetti_S</a> You should refer to this paper for your project</p><aside class="onebox pdf" data-onebox-src="https://arxiv.org/pdf/1806.06988.pdf">
  <header class="source">

      <a href="https://arxiv.org/pdf/1806.06988.pdf" target="_blank" rel="noopener nofollow ugc">arxiv.org</a>
  </header>

  <article class="onebox-body">
    <a href="https://arxiv.org/pdf/1806.06988.pdf" target="_blank" rel="noopener nofollow ugc"><span class="pdf-onebox-logo"></span></a>

<h3><a href="https://arxiv.org/pdf/1806.06988.pdf" target="_blank" rel="noopener nofollow ugc">1806.06988.pdf</a></h3>

  <p class="filesize">630.24 KB</p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>I have tried it and this will give you ideas on how to approach your problem</p>
Regularization in test phase?

username: Aymen_Sadraoui
cooked: <p>Hello, I want to add L1 regularisation to my model but I am confused should I calculate loss function in test phase with adding the weights penalty?</p>
<p>in train:<br>
loss=criterion(output,y)<br>
parameters = [parameter.view(-1) for parameter in model.parameters()]<br>
l1 = l1_lambda<em>model.compute_l1_loss(torch.cat(parameters))<br>
loss += l1 # Add L1 loss component<br>
in test:<br>
should I calculate the loss like this?:<br>
loss=criterion(output,y)<br>
parameters = [parameter.view(-1) for parameter in model.parameters()]<br>
l1 = l1_lambda</em>model.compute_l1_loss(torch.cat(parameters))<br>
loss += l1 # Add L1 loss component<br>
or like this?<br>
loss=criterion(output,y)</p>
<p><span class="hashtag">#Regularization</span> <span class="hashtag">#L1_regularization</span> <span class="hashtag">#test_phase</span></p>

username: pascal_notsawo
cooked: <p>You don’t need to do it when testing your model. It is done during training for the update of the parameters, which is not done during the test of the model.</p>

username: mMagmer
cooked: <p>I think the second one is the way to go. But it only matters if you’re going to use it as an early stop metric. In classification applications, loss has direct relation to error rate (it also considers by how much margin the classifier predict correct output), so the model with lower loss is better estimator for class probabilities. Regularization technique i.e. L1, max entropy regularizer (label smoothing) are not related to error rate in any way.<br>
This is my take.</p>

username: Aymen_Sadraoui
cooked: <p>but when i calculate the loss in train and test I find a lot of differences like:<br>
loss_train: 658.215<br>
test_loss: 0.6542</p>

username: mMagmer
cooked: <p>How do you compute and add the regularization term.<br>
It should be something like this.</p>
<pre><code class="lang-auto">loss = criterion(output,y) + l1_lambda*L1(parameters)
</code></pre>
<p>Where l1_lambda is  a small number, i.e. <code>1e-5</code>.<br>
Also check it without any regularization to see if some other thing may cause the problem (like different batch size in train and test)</p>

username: Aymen_Sadraoui
cooked: <p>yes that’s is, i compute the sum of the absolute value of the weights and I add it to the calculated loss.</p>
similar behaviour as output_attentions for a custom finetuned model

username: asadisaghar
cooked: <p>I am fine tuning a BERT classifier with my own data and save the state_dict (following <a href="https://curiousily.com/posts/sentiment-analysis-with-bert-and-hugging-face-using-pytorch-and-python/" rel="noopener nofollow ugc">this tutotrial</a>). Now for evaluating the model, I would like to use <a href="https://github.com/jessevig/bertviz" rel="noopener nofollow ugc">bertViz</a>, and so one thing I need is for an option like huggingface’s “output_attentions=True” for a custom model class but I’m not sure how to do that.</p>
Using torch.load() isn't loading my pre-trained model on a Docker Image, but loads the model on my local machine in PyCharm

username: kylemabry
cooked: <p>Hello,</p>
<p>I’m seeing the following error when trying to load a pre-trained model on a Docker image using torch.load():</p>
<p>“ModuleNotFoundError: No module named ‘evaluate’”</p>
<p>It looks like this happens during the unpickler.load() step within torch.serialization.py<br>
(see image below for a more detailed description).</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/8/b8bdb590a46422c0b86a25875d8762dacdef57d3.png" data-download-href="https://discuss.pytorch.org/uploads/default/b8bdb590a46422c0b86a25875d8762dacdef57d3" title="No Module named 'evaluate'"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/8/b8bdb590a46422c0b86a25875d8762dacdef57d3_2_690x241.png" alt="No Module named 'evaluate'" data-base62-sha1="qmiiTnYo5taHSmBDhDNO9NnLXs7" width="690" height="241" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/8/b8bdb590a46422c0b86a25875d8762dacdef57d3_2_690x241.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/8/b8bdb590a46422c0b86a25875d8762dacdef57d3_2_1035x361.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/8/b8bdb590a46422c0b86a25875d8762dacdef57d3_2_1380x482.png 2x" data-dominant-color="EFEFEF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">No Module named 'evaluate'</span><span class="informations">2062×722 98.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>My code looks like this:</p>
<p>model.load_state_dict(torch.load(“path/to/model.pth”, map_location=torch.device(‘cpu’)), strict=False)</p>

username: ptrblck
cooked: <p>Did you store the model object in the <code>model.pth</code> directly or anything else with might depend on the actual file structure? Based on the error it seems as if the <code>torch.load</code> is directly failing due to a change in the source files, which could happen e.g. if the model is directly stored (instead of the <code>state_dict</code>).</p>

username: kylemabry
cooked: <p>Hi,</p>
<p>No the state_dict is what was used to save the model during training. I should mention that everything works fine when I’m running my application on my local machine.</p>

username: ptrblck
cooked: <p>Since it’s working on your local machine it would indicate that some source files might still be needed. You are Indexing the loaded object (probably a <code>dict</code>) via <code>['model']</code>. What are the other objects stored in the <code>dict</code>? Could you save the <code>state_dict</code> alone and check if loading it would work?</p>
Quantizing an RNN with Glow tools

username: babbage
cooked: <p>How would I go about doing profile guided quantization of an RNN with Glow? All of the user guides I have seen make reference to an image-classifier tool: <a href="https://github.com/pytorch/glow/blob/master/docs/Quantization.md" rel="noopener nofollow ugc">https://github.com/pytorch/glow/blob/master/docs/Quantization.md</a></p>
<p>My input is not images but looks something like: <code>[x, h_1, h_2, ..., h_n]</code><br>
And the output <code>[y, h_1, h_2, ..., h_n]</code><br>
Where the <code>h_1, h_2,...</code> correspond to the hidden states of the network</p>
KeyError: 'Missing utterance

username: Kuldeep_Rana
cooked: <p>When I am training the model then this error is coming. It says missing utterance. How can I track missing utterances? Thank you so much</p>
<p>Traceback (most recent call last):</p>
<p>File “main/nnet/trainnew.py”, line 93, in </p>
<p>run(args)</p>
<p>File “main/nnet/trainnew.py”, line 55, in run</p>
<p>trainer.run(train_loader, dev_loader, num_epochs=args.epochs)</p>
<p>File “/workdir/denoiser/main/nnet/libs/trainer.py”, line 236, in run</p>
<p>tr = self.train(train_loader)</p>
<p>File “/workdir/denoiser/main/nnet/libs/trainer.py”, line 188, in train</p>
<p>for egs in data_loader:</p>
<p>File “/workdir/denoiser/main/nnet/libs/dataset.py”, line 143, in <strong>iter</strong></p>
<p>for chunks in self.eg_loader:</p>
<p>File “/opt/conda/lib/python3.6/site-packages/torch/utils/data/dataloader.py”, line 388, in <strong>next</strong></p>
<p>data = self._next_data()</p>
<p>File “/opt/conda/lib/python3.6/site-packages/torch/utils/data/dataloader.py”, line 428, in _next_data</p>
<p>data = self._dataset_fetcher.fetch(index) # may raise StopIteration</p>
<p>File “/opt/conda/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py”, line 44, in fetch</p>
<p>data = [self.dataset[idx] for idx in possibly_batched_index]</p>
<p>File “/opt/conda/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py”, line 44, in </p>
<p>data = [self.dataset[idx] for idx in possibly_batched_index]</p>
<p>File “/workdir/denoiser/main/nnet/libs/dataset.py”, line 42, in <strong>getitem</strong></p>
<p>ref = [reader[key] for reader in self.ref]</p>
<p>File “/workdir/denoiser/main/nnet/libs/dataset.py”, line 42, in </p>
<p>ref = [reader[key] for reader in self.ref]</p>
<p>File “/workdir/denoiser/main/nnet/libs/audio.py”, line 120, in <strong>getitem</strong></p>
<p>raise KeyError(“Missing utterance {}!”.format(index))</p>
<p>KeyError: ‘Missing utterance GX_20180328_032_BGD_150204_040_BUS.CH1_snr10_fileid_11233!’</p>
<p>training finished</p>

username: anantguptadbl
cooked: <p><a class="mention" href="/u/kuldeep_rana">@Kuldeep_Rana</a> looks like a missing file in the training folder/data</p>
Dependence of the model output on the batch size

username: ushakovegor
cooked: <p>Hello,<br>
On the inference I noticed that output heatmap of my model depence of the batch size of the dataloader.<br>
Here it is images:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/5/a/5a430b4af688685960e573ed1dd26a325428eaa1.png" alt="images" data-base62-sha1="cSuC31jyKLhdCgEknk8zsnHutAl" width="600" height="158"><br>
The left image with batch_size=1, the right: batch_size=5(it is batch_size I used for training)</p>
<p>When I use the native batch size(5), the quality of the heatmap is better.<br>
Here is my model: <a href="https://www.dropbox.com/s/ms3mvh030t7n74a/latest3.pth" rel="noopener nofollow ugc">https://www.dropbox.com/s/ms3mvh030t7n74a/latest3.pth</a></p>
<p>I believe, the model should not depend of the batch size, but it does. Can you give me any suggestions about this? It is normal behavior of the model?<br>
If you need, I can attach the code of my dataset and code of the inference my model.</p>

username: anantguptadbl
cooked: <p><a class="mention" href="/u/ushakovegor">@ushakovegor</a> you are right when you say that the model should not depend on the batch size, but when you are updating gradients, anything can happen</p>
<ul>
<li>
<p>If your dataset is skewed then model will oscillate during each batch execution</p>
</li>
<li>
<p>Batch size of 1 does not make sense as there will be very little room for the model to play around</p>
</li>
<li>
<p>If you use very large batch sizes then you will overfit</p>
</li>
<li>
<p>Most use a low batch_size of 16 and a high batch size of 64</p>
</li>
</ul>

username: ushakovegor
cooked: <p>Hello,<br>
Thanks for your response.The problems with this model behavior was related to BatchNorm. I forgot to switch model to eval mode. This switching helps me to resolve this problem.<br>
This problem was discussed here too:<br>
<a href="https://discuss.pytorch.org/t/output-varies-when-changing-batch-size-during-test/10439">https://discuss.pytorch.org/t/output-varies-when-changing-batch-size-during-test/10439</a></p>
Bottleneck on data loading

username: marioo
cooked: <p>Hi,</p>
<p>I have a bottleneck on the dataloading during training. I run cProfiler and these are the results:</p>
<pre><code class="lang-auto">        1    0.012    0.012 1820.534 1820.534 models.py:15(fit)
       56    0.001    0.000 1808.163   32.289 dataloader.py:775(__next__)
       52    0.001    0.000 1807.264   34.755 dataloader.py:742(_get_data)
      392    0.016    0.000 1807.263    4.610 dataloader.py:711(_try_get_data)
      392    0.006    0.000 1807.178    4.610 queues.py:91(get)
      392    0.002    0.000 1806.842    4.609 connection.py:253(poll)
      392    0.002    0.000 1806.840    4.609 connection.py:413(_poll)
      392    0.009    0.000 1806.837    4.609 connection.py:906(wait)
      392    0.004    0.000 1806.810    4.609 selectors.py:402(select)
      392 1806.805    4.609 1806.805    4.609 {method 'poll' of 'select.poll' objects}
        4    0.000    0.000    6.452    1.613 dataloader.py:274(__iter__)
        4    0.016    0.004    6.452    1.613 dataloader.py:635(__init__)
      128    0.007    0.000    5.553    0.043 process.py:101(start)
      128    0.001    0.000    5.531    0.043 context.py:221(_Popen)
      128    0.003    0.000    5.530    0.043 context.py:274(_Popen)
</code></pre>
<p>I am using 32 workers (I have 40 cpus available).<br>
Do you what is causing the dataloading to be slow? Do you know what are the files queues.py and connetctions.py? Functions there seem to be taking great part of the time.</p>
<p>Cheers <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: ptrblck
cooked: <p>Have a look at <a href="https://discuss.pytorch.org/t/how-to-prefetch-data-when-processing-with-gpu/548/19">this post</a>, which explains some potential bottlenecks and workarounds. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: Ara_Jafarzadeh
cooked: <aside class="quote no-group" data-username="marioo" data-post="1" data-topic="90445">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/marioo/48/13447_2.png" class="avatar"> marioo:</div>
<blockquote>
<p>Do you know what are the files queues.py and connetctions.py</p>
</blockquote>
</aside>
<p>Both <a href="https://github.com/python/cpython/blob/main/Lib/multiprocessing/connection.py" rel="noopener nofollow ugc">connections.py</a> and <a href="https://github.com/python/cpython/blob/main/Lib/multiprocessing/queues.py" rel="noopener nofollow ugc">queues.py</a> seem to be from the multiprocessing library of Python.</p>
Regarding "forward" for custom models

username: tanabe.d.zc32s
cooked: <p>Hellow there.</p>
<p>I’m currently trying to build a custom object detection model based on Faster R-CNN.<br>
The structure of the custom CNN model is such that the final output is the class of the object, the boundary box, and the attributes of the object .<br>
So, I wrote the following code.</p>
<pre><code class="lang-auto">class CustomPredictor(nn.Module):
    def __init__(self,in_channels,num_classes):
        super(CustomPredictor,self).__init__()
          
        self.cls_score = nn.Linear(in_channels, num_classes)
        self.bbox_pred = nn.Linear(in_channels, num_classes * 4)
        self.additional_layer = nn.Linear(in_channels,1) #this is the additional layer
        
        
        
    def forward(self,x):
        if x.dim() == 4:
            assert list(x.shape[2:]) == [1, 1]
        x = x.flatten(start_dim=1)
        
        
        pred_leaf_age = self.additional_layer(x)
        scores = self.cls_score(x)
        bbox_deltas = self.bbox_pred(x)
        return scores, bbox_deltas, pred_leaf_age

    
    
class CustomModel(nn.Module):
    def __init__(self):
        
        super(CustomModel, self).__init__()
        self.model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)
        self.in_features = self.model.roi_heads.box_head.fc7.out_features
        self.model.roi_heads.box_predictor = CustomPredictor(self.in_features,6)
        
        
    def forward(self, x):

</code></pre>
<p>The problem here is how to define the foward of the CustomModel class.<br>
Could you give me any advice?</p>
<p>Best regards.</p>

username: tanabe.d.zc32s
cooked: <p>I may have resolved this probrem in myself .<br>
Thank you for watching.</p>
SVCCA on LSTM output

username: andcarnivorous
cooked: <p>Hi everyone,</p>
<p>I am kind of a newbie to PyTorch and ML. I have been trying to check for nodes correlation using the svcca module from google <a href="https://github.com/google/svcca" rel="nofollow noopener">here</a>.</p>
<p>The examples work well and if I try to use random tensors converted into np arrays I’m able to get the svcca values.</p>
<p>I then tried to get the activations from a layer of an LSTM net by adding a <code>self.activations</code> variable to the model, which saves the activations of the LSTM layer output.</p>
<pre><code class="lang-auto">    def forward(self, text):

        embedded = self.embedding(text)
        output, hidden = self.rnn(embedded)
        
        self.activations = output
        
        assert torch.equal(output[-1,:,:], hidden[0].squeeze(0))
        out = self.linear(hidden[0].squeeze(0))
        return out
</code></pre>
<p>Then during evaluation I save the activations to a global variable:</p>
<pre><code class="lang-auto">    model.eval()
    
    with torch.no_grad():
    
        for batch in iterator:

            predictions = model(batch.text).squeeze(1)
            activations.append(model.activations)

</code></pre>
<p>Then, just to test it, I take one of the activations and transpose it as required by the svcca module, and check the cca similarity with itself which should give a coefficient of 1</p>
<pre><code class="lang-auto">a = activations[1].cpu().detach().numpy()
a = a[0,:,:]
a = np.transpose(a, (1,0))

results = cca_core.get_cca_similarity(a, a, verbose=True)
</code></pre>
<p>What I get is this error:</p>
<pre><code class="lang-auto">  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "/home/main/Projects/svcca/cca_core.py", line 295, in get_cca_similarity
    verbose=verbose)
  File "/home/main/Projects/svcca/cca_core.py", line 162, in compute_ccas
    u, s, v = np.linalg.svd(arr)
  File "&lt;__array_function__ internals&gt;", line 6, in svd
  File "/home/main/.local/lib/python3.6/site-packages/numpy/linalg/linalg.py", line 1636, in svd
    u, s, vh = gufunc(a, signature=signature, extobj=extobj)
ValueError: On entry to DLASCL parameter number 4 had an illegal value
</code></pre>
<p>I checked for NaNs and Infs and there seem to be none.</p>
<p>But if I make random matrices with the same shape (65, 128), I can get the cca similarity just fine. Anyone has any idea where I might be getting stuff wrong?  Is there something wrong in how I get the activations from torch tensors to np arrays?</p>
<p>Thanks a lot.</p>
What loss function for binary unet?

username: meshghi
cooked: <p>Hi, I’m trying to do a semantic segmentation for a binary case (<strong>i.e. background=0, and class-of-interest=1</strong>) using unet. my last layer of deconvolution looks as follow:</p>
<p><code>self.final2 = nn.Conv2d(num_filters, num_classes=1, kernel_size=1)</code></p>
<p>My output from the model has one channel (i.e. <code>output(N,1,256,256)</code>), in which higher probability means more like class 1, and background otherwise. I was wondering:</p>
<p>1- what loss function should I use?<br>
2- Is my way of calculating the confusion matrix below is correct?</p>
<pre><code class="lang-auto">probability_class = torch.exp(output)
prediction = torch.argmax(probability_class, 1)
tn, fp, fn, tp = confusion_matrix(gt.view(-1), prediction.view(-1)).ravel()
</code></pre>

username: Eta_C
cooked: <p>While your output has only one channel, use <code>probability_class = torch.sigmoid(output)</code> to norm them between [0, 1] and then <code>prediction = probability &gt; threshold</code> may be a better choice. About the Loss function, <code>Sigmoid</code> + <code>MSELoss</code> is OK.</p>
<p>Note that <code>output</code> has one channel, so <code>probability_class</code> will also has only one channel, that means your code <code>torch.argmax(probability_class, 1)</code> will always return zero.</p>
<p>Or your can use <code>self.final2 = nn.Conv2d(num_filters, num_classes=2, kernel_size=1)</code> and <code>nn.CrossEntropyLoss</code>. In this case, direct <code>argmax</code> (eval mode) is OK.</p>

username: meshghi
cooked: <p>Thanks <a class="mention" href="/u/eta_c">@Eta_C</a>. If I choose <code>num_classes=2</code> and use <code>nn.CrossEntropyLoss</code>, does <code>argmax</code> at eval mode give me indices 0 and 1? I am asking because my groundtruth data is 0 and 1 and I want my prediction output to match.</p>

username: Eta_C
cooked: <p>Expected YES.</p>
<p>Also, as a semantic segmentation task, there are some other loss can be tried. For example, Focal Loss, DICE Loss, IoU Loss, Tversky Loss, Lovasz Loss, and so on.</p>
Do I need to apply a softmax when using nn.CrossEntropy()

username: OBouldjedri
cooked: <p>I am working on multiclass classification problem so the natural choice in pythorch is nn.CrossEntropy, but the docs indicates that the loss function include a softmax function I got confused,   I was wondering if applying the softmax  at the end of my neural network model will make it used twice instead of only 1 use ?</p>

username: Eta_C
cooked: <p>No.</p>
<blockquote>
<p>Note that this case is equivalent to the combination of LogSoftmax and NLLLoss.</p>
</blockquote>
Unable to enter debugging with hooks

username: Valiox
cooked: <p>I’m unable to enter breakpoints (using PyCharm) inside my hook functions, e.g.:</p>
<pre><code class="lang-auto">class MyModule(nn.Module):
    def __init__(self, train: bool):
        super().__init__()
        
        def hook(module, grad_input, grad_output):
            print('hello there')
        self.register_backward_hook(hook)
</code></pre>
<p>I cannot break on the print but it will still execute. Does anyone know why that is and if there is a possible way to circumvent this?</p>

username: smth
cooked: <p>The hooks are called as PyFunction calls from the C++ side. It’s possible that PyCharm cannot handle this yet.</p>

username: JeongHyunho
cooked: <p>Turning off cuda is a possible workaround. (cpu only)<br>
For that, you could set CUDA_VISIBLE_DEVICES empty.</p>
Use CLIP as a pretrained module -- Should/Can I use `requires_grad=False`?

username: nullgeppetto
cooked: <p>I’m interested in using OpenAI’s CLIP as a pre-trained module, part of a network with other trainable module. Input will be first passed through CLIP (e.g., image encoder) for feature extraction, which will subsequently be used as input to a trainable module. The network needs to be trained using back-propagation under some criterion and so on.</p>
<p>My question is, whether it suffices to use CLIP in eval mode (<code>.eval()</code>) or whether I need to set its parameters <code>requires_grad</code> to false as well. More importantly, do I necessarily NEED to have CLIP’s parameters with <code>requires_grad=True</code> in order to back-propagate and update the trainable network’s weights? I’m asking because if I don’t set <code>requires_grad=False</code> I get error during <code>loss.backward()</code> call, where it suggests to use <code>retain_graph=True</code> etc. It feels that I don’t need to use <code>retain_graph=True</code>  in the backwards call, but I might be wrong obviously.</p>
<p>Thank you!</p>
Initialization for AlexNet

username: just_started_coding
cooked: <p>I tried to initialize the weights and bias of AlexNet exactly as described in the paper. Is this the correct way to do it?</p>
<h1>
<a name="initialize-weight-and-bias-1" class="anchor" href="#initialize-weight-and-bias-1"></a>Initialize weight and bias</h1>
<p>def initialize_weights(m):<br>
for layer_count,layers in enumerate(m.children()):<br>
if isinstance(layers, nn.Conv2d):<br>
nn.init.normal_(layers.weight, mean=0, std=0.01)<br>
nn.init.constant_(layers.bias, 0)<br>
if layer_count in [1,3,4,5,6]:<br>
nn.init.constant_(layers.bias, 1)</p>
Smoothed Update of Network Parameters

username: ssha
cooked: <p>Hello,</p>
<p>I have two networks, target_value and value_state dict.</p>
<p>I am trying to update parameters of the target_value network, I want the update to be a smoothed version of target_value and value_state. I have the following code, which works well if my networks do not contain a batch normalisation layer.</p>
<pre><code class="lang-auto">    def update_network_parameters(self, tau=None):
        if tau is None:
            tau = self.tau

        target_value_params = self.target_value.named_parameters()
        value_params = self.value.named_parameters()

        target_value_state_dict = dict(target_value_params)
        value_state_dict = dict(value_params)

        for name in value_state_dict:
            value_state_dict[name] = tau*value_state_dict[name].clone() + \
                                     (1-tau)*target_value_state_dict[name].clone()

        self.target_value.load_state_dict(value_state_dict)
</code></pre>
<p>However, once I include the batch normalisation layers, I am returned with the following error:</p>
<pre><code class="lang-auto">RuntimeError: Error(s) in loading state_dict for ValueNetwork:
	Missing key(s) in state_dict: "feature_extractor.extractors.price_features.co1.bn1.running_mean", "feature_extractor.extractors.price_features.co1.bn1.running_var", "feature_extractor.extractors.price_features.co2.bn1.running_mean", "feature_extractor.extractors.price_features.co2.bn1.running_var"
</code></pre>
<p>I understand that the error is present since calling <code>.named_parameters()</code> will not load the batch norm state dict.</p>
<p>My question’s are:</p>
<ol>
<li>After performing the smooth update of parameters, what is the correct way to update the batch norm statistics i.e. running mean and running_var</li>
<li>Are you able to able provide some code snippet so I can understand how to do the above</li>
</ol>
<p>Thank you!</p>
<p>p.s. <a class="mention" href="/u/ptrblck">@ptrblck</a> I saw a similar topic you replied to in the past, however, I can no longer find the thread.</p>

username: googlebot
cooked: <blockquote>
<p>skip rewriting these with strict=False argument</p>
</blockquote>
<p>disregard that, your problem is that you’re not including buffers in state_dict (try Module.buffers()). I’m not sure about fixing the stats, I think correct values won’t be available without re-evaluation, but maybe error won’t be significant.</p>

username: mMagmer
cooked: <p>Hi <a class="mention" href="/u/ssha">@ssha</a>,<br>
what you want to do is really close to <a href="https://pytorch.org/blog/pytorch-1.6-now-includes-stochastic-weight-averaging" rel="noopener nofollow ugc">stochastic-weight-averaging</a>.</p>
<pre><code class="lang-auto">torch.optim.swa_utils
</code></pre>
<p>You can check the source to see how they implementing it.<br>
About how to update running means, you can see function <code>update_bn</code> in blog post.</p>
Multiple sequences with a single label

username: mscorp
cooked: <p>Is it possible to train a pytorch LSTM with a sequence containing several features, but a single label?</p>
<p>If so, how does one format the data, and what size is the input for the LSTM?</p>
<p>time1 feature1, feature2, feature3, feature4<br>
time2 feature1, feature2, feature3, feature4<br>
time3 feature1, feature2, feature3, feature4, label</p>
<p>Assume the data is in a csv, each row is a time step and contains a single sequence of 4 features, and every 3 rows has a single label.</p>
<p>I’ve tried asking this question several different ways on this forum and never get a response. So either i’m asking the question poorly, or no one understands what i’m asking. I could really use some help.    For reference, i’ve read (i’m fairly sure) all of the pytorch tutorials on the lstm topic.  None of them seem to address what i’m trying to do, which is use several sequences with a single label, instead of labeling every sequence.   That’s because in my case, we have a cause, and later, an effect, and they are displaced in time (a time series).  The classification isn’t known until the end of the sequence, thus, several sequences per label.  Importantly, every sequence has several features.<br>
Is this possible? What’s the proper name for this kind of approach?</p>
<p>Any help would be  very much appreciated. Thank you.</p>
How to reshape a tensor back to it previous shape after using view()

username: rubijade
cooked: <p>Hello everyone. Hope you are doing well.<br>
I have reshaped my tensor from (b, c, H, W)  to (b, c, H*W) before sending it to a conv1D module.<br>
So I want to reshape it back to its previous shape (b, c, H, W) in order to send it to a conv2D.<br>
I was trying reshape() and view(), even squeeze().<br>
Any idea? Thank you in advance.</p>

username: rubijade
cooked: <p>so the tensor is on the shape (80, 16, 112, 112) and I reshape to (80, 16, 12542) before sending to the conv1D.<br>
To reshape it back to (80, 16, 112, 112), first, i use unsqueeze(0) and get (1,80,16,12542) and then view as follow</p>
<pre><code class="lang-auto">x = x.unsqueeze(0)
d, b, c, hw = x.shape
x = x.view(b, c, round(math.sqrt(hw)), round(math.sqrt(hw)))
</code></pre>
<p>But i got this error</p>
<pre><code class="lang-auto">Exception has occurred: RuntimeError
shape '[80, 16, 112, 112]' is invalid for input of size 16053760
</code></pre>

username: my3bikaht
cooked: <aside class="quote no-group" data-username="rubijade" data-post="2" data-topic="143468">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/ec9cab/48.png" class="avatar"> rubijade:</div>
<blockquote>
<p><code>16053760</code></p>
</blockquote>
</aside>
<p>112*122 = 12544<br>
16053760 / 80 / 16 = 12542</p>
<p>You’re probably lost 1 padding in conv layer before.</p>

username: rubijade
cooked: <p>that is right. Thank you<br>
Adding a padding =1 to the conv1d solve it.</p>
Tensor elements won't update in a loop

username: mv000
cooked: <p>Hi experts. I cannot update tensors in loops over zipped and flattened pairs, and I am confused by this.</p>
<p>If I run</p>
<pre><code class="lang-auto">for x, y in zip(xx.flatten(), yy.flatten()):
    t = torch.tensor([0, 0, 0, 0])
    print(x,y)
    t[1]=torch.tensor(x)
    t[2]=torch.tensor(y)
    print(t)   
</code></pre>
<p>The output is:</p>
<pre><code class="lang-auto">-0.05 -0.05
tensor([0, 0, 0, 0])
-0.03888888888888889 -0.05
tensor([0, 0, 0, 0])
-0.02777777777777778 -0.05
tensor([0, 0, 0, 0])
-0.01666666666666667 -0.05
tensor([0, 0, 0, 0])
...
...
</code></pre>
<p>But if I access the elements individually, the tensor is changed. Or if I loop over</p>
<pre><code class="lang-auto">for x, y in zip( range(10), range(10)):
    t = torch.tensor([0, 0, 0, 0])
    print(x,y)
    t[1]=torch.tensor(x)
    t[2]=torch.tensor(y)
    print(t) 
</code></pre>
<p>The output is:</p>
<pre><code class="lang-auto">0 0
tensor([0, 0, 0, 0])
1 1
tensor([0, 1, 1, 0])
2 2
tensor([0, 2, 2, 0])
3 3
tensor([0, 3, 3, 0])
4 4
tensor([0, 4, 4, 0])
...
...
</code></pre>
<p>Is there something about flattening that would cause tensors to fail to update?</p>
<p>The xx, and yy are formed by mesh grid in case that helps:</p>
<pre><code class="lang-auto"> xx, yy = np.meshgrid(np.linspace(-0.05, 1.05, 100), np.linspace(-0.05, 1.05, 100))
</code></pre>

username: pascal_notsawo
cooked: <p>In the second case, it works because the elements you give to <code>t[1]</code> and <code>t[2]</code> are integers, and <code>t</code> is of type <code>torch.int64</code>.<br>
In the first case, as <code>t</code> is of type <code>torch.int64</code>, only the integer parts of <code>x</code> and <code>y</code> are assigned to <code>t[1]</code> and <code>t[2]</code> respectively.</p>
<pre><code class="lang-auto">xx, yy = np.meshgrid(np.linspace(-0.05, 1.05, 2), np.linspace(-0.05, 1.05, 2))

for x, y in zip(xx.flatten(), yy.flatten()):
    t = torch.tensor([0, 0, 0, 0])
    print(x,y)
    t[1]=torch.tensor(x)
    t[2]=torch.tensor(y)
    print(t)   

"""
-0.05 -0.05
tensor([0, 0, 0, 0])
1.05 -0.05
tensor([0, 1, 0, 0])
-0.05 1.05
tensor([0, 0, 1, 0])
1.05 1.05
tensor([0, 1, 1, 0])
"""
</code></pre>
<p>But :</p>
<pre><code class="lang-auto">xx, yy = np.meshgrid(np.linspace(-0.05, 1.05, 2), np.linspace(-0.05, 1.05, 2))

for x, y in zip(xx.flatten(), yy.flatten()):
    t = torch.tensor([0, 0, 0, 0]).float()
    #or t = torch.FloatTensor([0, 0, 0, 0])
    print(x,y)
    t[1]=torch.tensor(x)
    t[2]=torch.tensor(y)
    print(t)   

"""
-0.05 -0.05
tensor([ 0.0000, -0.0500, -0.0500,  0.0000])
1.05 -0.05
tensor([ 0.0000,  1.0500, -0.0500,  0.0000])
-0.05 1.05
tensor([ 0.0000, -0.0500,  1.0500,  0.0000])
1.05 1.05
tensor([0.0000, 1.0500, 1.0500, 0.0000])
"""
</code></pre>

username: mv000
cooked: <p>That’s got it! Thank you!</p>
ONNX Export Difference

username: Orcun_Deniz
cooked: <p>I’ve exported my model to onnx and verifying the exported model by <code>torch.linalg.norm( torch_out - onnx_out) </code>. These outputs are inferred from torch.ones tensors. I get difference norms below,</p>
<pre><code>center diff: 0.0001215097145177424, tensor shape: torch.Size([1, 2, 468, 468])
center_z diff: 0.000555071048438549, tensor shape: torch.Size([1, 1, 468, 468])
dim diff: 0.0008262245682999492, tensor shape: torch.Size([1, 3, 468, 468])
rot diff: 0.0005295916926115751, tensor shape: torch.Size([1, 2, 468, 468])
hm diff: 0.0025102065410465, tensor shape: torch.Size([1, 3, 468, 468])
VFE diff: 0.12264608591794968, tensor shape: torch.Size([10000, 64])
</code></pre>
<p>Torch export and inference were performed while model was in <code>eval()</code> and <code>with torch.no_grad()</code></p>
<p>Are these differences expected and reasonable or might something be wrong?</p>
<p>Thanks!</p>

username: anantguptadbl
cooked: <p><a class="mention" href="/u/orcun_deniz">@Orcun_Deniz</a> can you provide the model source code. This will help us understand where the mismatch might be getting triggered</p>

username: Orcun_Deniz
cooked: <p>Here, I uploaded the project to git; <a href="https://github.com/OrcunCanDeniz/OpenPCDet_Exports/blob/main/centerpoint_exporter.py" rel="noopener nofollow ugc">https://github.com/OrcunCanDeniz/OpenPCDet_Exports/blob/main/centerpoint_exporter.py</a> does the export. You can comment this <a href="https://github.com/OrcunCanDeniz/OpenPCDet_Exports/blob/3ed8d843fb261bce1ef5c34d6fe81dea07b7f7f3/centerpoint_exporter.py#L110" rel="noopener nofollow ugc">line</a> to avoid loading weights, this does not change the behaviour.<br>
I break down the model into 2 parts and took 2 exports. One of those seem to be work ok. But the one on this <a href="https://github.com/OrcunCanDeniz/OpenPCDet_Exports/blob/3ed8d843fb261bce1ef5c34d6fe81dea07b7f7f3/centerpoint_exporter.py#L155" rel="noopener nofollow ugc">block</a> is the problematic one. And <a href="https://github.com/OrcunCanDeniz/OpenPCDet_Exports/blob/main/pcdet/models/backbones_3d/vfe/pillar_vfe.py" rel="noopener nofollow ugc">this</a> is the file is where source of that module. Finally that’s my <a href="https://drive.google.com/file/d/1Z_6y-faPZxkdgtJtsxwXcbvwL0gGE_gu/view?usp=sharing" rel="noopener nofollow ugc">onnx model</a>.</p>
<p>Note: I am also exporting a TorchScript of the model and do not face any issue.</p>
How to convert .ckpt to .mlmodel

username: arnab_siam
cooked: <p>I am trying to use <a href="https://github.com/saic-mdal/lama" rel="noopener nofollow ugc">https://github.com/saic-mdal/lama</a> in my ios project. They have given a .ckpt file. I need to convert this .ckpt file to .mlmodel. But I don’t know how can I achieve that.</p>
How to instantiate axis-aligned multivariate Gaussian?

username: Dane_Mitrev
cooked: <p>It is straightforward to do the following and wrap a Gaussian into an <code>Independent</code>. So if we are given a</p>
<pre><code class="lang-auto">loc = [batch_size, event_shape]
scale = [batch_size]


loc = torch.zeros(5, 3, 2)
scale = torch.ones(2)
normal = Normal(loc, scale)
normal.batch_shape, normal.event_shape
(torch.Size([5, 3, 2]), torch.Size([]))

ind = Independent(normal, 1)
ind.batch_shape, ind.event_shape
(torch.Size([5, 3]), torch.Size([2]))
</code></pre>
<p>This <code>Independent</code> distribution is identical to a <code>MultivariateNormal</code> defined as:<br>
<code>mvn = MultivariateNormal(mu, torch.diag(scale))</code></p>
<p>But, what if I have the following:</p>
<pre><code class="lang-auto">
mu = torch.zeros(5, 2)
log_sigma = torch.ones(5, 2)
Independent(Normal(loc=mu, scale=torch.exp(log_sigma)), 1)
</code></pre>
<p>This works fine, however how to use MultivariateNormal is not clear:</p>
<p><code>mvn = MultivariateNormal(mu, ???)</code></p>
<p>Is it possible to use <code>MultivariateNormal</code> and get the same distribution? How do I provide the <code>covariance_matrix</code> argument given <code>log_sigma</code> of shape <strong>[batch_size, event_shape]</strong></p>
<hr>
<p><strong>EDIT:</strong></p>
<pre><code class="lang-auto">mu = torch.zeros(5, 2)
log_sigma = torch.ones(5, 2)
cov = torch.stack([torch.diag(sigma) for sigma in torch.exp(log_sigma)])
</code></pre>
<p><code>mvn = MultivariateNormal(mu, cov)</code></p>
<p><strong>=&gt; would this result in an equivalent to</strong></p>
<p><code>Independent(Normal(loc=mu, scale=torch.exp(log_sigma)), 1)</code>  ??</p>

username: christian-cahig
cooked: <p>I think <a href="https://pytorch.org/docs/stable/generated/torch.diag_embed.html#torch-diag-embed" rel="noopener nofollow ugc"><code>torch.diag_embed</code></a> is useful in constructing the tensor of diagonal covariance matrices. Hence,</p>
<pre><code class="lang-python">import torch
from torch.distributions import MultivariateNormal

mu = torch.zeros(5, 2)
log_sigma = torch.ones(5, 2)
cov = torch.diag_embed(log_sigma)
mvn = MultivariateNormal(mu, covariance_matrix=cov)
</code></pre>
Gather 4d tensor

username: murloren
cooked: <p>Hi! I want to know if there exits a better way with torch.gather or something of doing the following:</p>
<p>I have x,y tensors with the indices of the correspondent samples of the batch, which represent the x-y coordinates of the respective samples i. So, if the batchsize = 8, they are a tensor like of 8 components respectively, whose values are between W and H respectively<br>
x = voxel_indices [:, 0, u, v].long(), y = voxel_coordinates [:, 1, u, v].long()</p>
<p>Then, I want to use this indices in order to index a feature map (Bs, Ch, W, H) like this<br>
features_map[:, :, x, y]</p>
<p>However, this gives me a tensor of [8, 256, 8], while I only want a tensor of [8, 256], with one feature map per each sample of the batch.</p>
Using StaticRuntime with libtorch

username: victor-shepardson
cooked: <p>How can I use StaticRuntime from C++ using libtorch?</p>
<p>following e.g. <a href="https://github.com/pytorch/pytorch/blob/4d08db0cb272093af6dbdd7038f92407f8741ddf/torch/csrc/jit/runtime/static/impl.h#L176" class="inline-onebox" rel="noopener nofollow ugc">pytorch/impl.h at 4d08db0cb272093af6dbdd7038f92407f8741ddf · pytorch/pytorch · GitHub</a> I just get a compile error: <code>use of undeclared identifier 'StaticModule'</code></p>
<p>the nightly libtorch has no <code>include/torch/csrc/jit/runtime/static</code> directory, nor does searching the whole tree for “staticmodule” turn up any results. it doesn’t seem to be there at all. Do I need to build pytorch myself? With what options?</p>

username: victor-shepardson
cooked: <p>I didn’t find exactly why the static runtime headers are left out of the libtorch build, but I did find that I could copy them into the nightly release at <code>include/torch/csrc/jit/runtime/static</code> from the corresponding commit to <code>master</code>.</p>