akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
DataLoader returning training set in size of validation set

username: ptinn
cooked: <p>Hi,</p>
<p>I am encountering a strange issue where my DataLoader is returning the training set in the same size as my validation set.  With the below implementation on my data, I get a 5911-1478 split.  But coming out from the DataLoader with batch=16, I get the lengths of both in 93, which is based on 1478/16.   What might have happened to the training set with 5911 samples?</p>
<pre><code class="lang-auto">def split_data(data):
  train = int(len(data)* .8)
  val = len(data) - train
  train_set, val_set = random_split(data, [train, val])
  
  return train_set, val_set

train_loader = DataLoader(train_set, batch_size=16, shuffle=True)
val_loader = DataLoader(val_set, batch_size=16, shuffle=True)

</code></pre>

username: ptrblck
cooked: <p>Your code looks correct and also works for me:</p>
<pre><code class="lang-python">def split_data(data):
  train = int(len(data)* .8)
  val = len(data) - train
  train_set, val_set = torch.utils.data.random_split(data, [train, val])
  
  return train_set, val_set


dataset = TensorDataset(torch.randn(5911+1478))
train_set, val_set = split_data(dataset)

print(len(train_set))
# 5911
print(len(val_set))
# 1478

train_loader = DataLoader(train_set, batch_size=16, shuffle=True)
val_loader = DataLoader(val_set, batch_size=16, shuffle=True)
print(len(train_loader))
# 370 = 369 full batches and 1 batch with 7 samples
print(len(val_loader))
# 93 = 92 full batches and 1 batch with 6 samples
</code></pre>

username: ptinn
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="152940">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p><code>orch.utils.data.ra</code></p>
</blockquote>
</aside>
<p>Thank you for verifying <a class="mention" href="/u/ptrblck">@ptrblck</a> . It was an obvious mistake on my end for returning one set as both.  My bad <img src="https://discuss.pytorch.org/images/emoji/apple/see_no_evil.png?v=12" title=":see_no_evil:" class="emoji" alt=":see_no_evil:" loading="lazy" width="20" height="20"></p>
Multi-Class cross entropy loss batch size error

username: Shashank.Mishra
cooked: <p>Hi,<br>
I am working on nuscenes dataset and for one of the output head using cross entropy loss.<br>
input size torch.Size([8, 3, 10, 159, 159])<br>
target size torch.Size([8, 10, 159, 159])<br>
8 - batch size<br>
3 - classes (specific to head)<br>
10 - d1 ( these are overall classes; for each class, we can have 3 values specifically as mentioned above)<br>
159 - d2 (height)<br>
159 -d3 (width)<br>
We have output heads in our model and the model predicts all the attributes for the main 10 different classes.</p>
<p>If I am using a batch size of 1 then everything is working fine.<br>
input [1, 3, 10 159, 159]<br>
target [1, 10, 159, 159]</p>
<p>But when I increase the batch size cross-entropy loss results in the error <strong>"RuntimeError: The size of tensor a (8) must match the size of tensor b (10) at non-singleton dimension 1"</strong></p>
<p>I am using torch 1.7.1.</p>
<p>Am I missing something from the definition of cross-entropy loss?<br>
Any suggestion on how I can correct it?</p>

username: ptrblck
cooked: <p>Both approaches should work as seen here:</p>
<pre><code class="lang-python">criterion = nn.CrossEntropyLoss()

input = torch.randn(1, 3, 10, 159, 159, requires_grad=True)
target = torch.randint(0, 3, (1, 10, 159, 159))
loss = criterion(input, target)
print(loss)
# tensor(1.3898, grad_fn=&lt;NllLoss2DBackward0&gt;)

input = torch.randn(8, 3, 10, 159, 159, requires_grad=True)
target = torch.randint(0, 3, (8, 10, 159, 159))
loss = criterion(input, target)
print(loss)
# tensor(1.3896, grad_fn=&lt;NllLoss2DBackward0&gt;)
</code></pre>
<p>Are you changing the shape of these tensors internally somehow?</p>
Tensor format to utilize tensor cores?

username: jayz
cooked: <p>Hi,</p>
<p>in the performance guide (<a href="https://pytorch.org/tutorials/recipes/recipes/tuning_guide.html#use-mixed-precision-and-amp" class="inline-onebox" rel="noopener nofollow ugc">Performance Tuning Guide — PyTorch Tutorials 2.1.1+cu121 documentation</a>), it says:</p>
<blockquote>
<p>To use Tensor Cores: set sizes to multiples of 8 (to map onto dimensions of Tensor Cores)</p>
</blockquote>
<p>Does this mean when I have a tensor BCHW with (32,15,10,256), operations on this tensor within the autocast() context manager will not be mapped at all to tensor cores, because the C and H dimensions are not multiples of 8? In my application the dimension H=10 is a fixed parameter that must not be changed. Does that mean, I won’t be able to utilize tensor cores, at all?</p>
<p>Furthermore, I also saw in the performance guide that the channels_last memory format may be preferable to work with tensor cores. So, following the guide (<a href="https://pytorch.org/tutorials/recipes/recipes/tuning_guide.html#enable-channels-last-memory-format-for-computer-vision-models" class="inline-onebox" rel="noopener nofollow ugc">Performance Tuning Guide — PyTorch Tutorials 2.1.1+cu121 documentation</a>), I convert my existing model:</p>
<pre data-code-wrap="python"><code class="lang-python">model = model.to(device='cuda', memory_format=torch.channels_last)
</code></pre>
<p>and in the training loop:</p>
<pre data-code-wrap="python"><code class="lang-python">for i,(X,y) in enumerate(dataloader): 
     X = X.to(device='cuda', memory_format=torch.channels_last, non_blocking=True)
     with autocast(): 
            out = model(X)
     # ...
</code></pre>
<p>Is that about right to make the model tensor core compatible?</p>
<p>Thanks!</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="jayz" data-post="1" data-topic="152916">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jayz/48/46742_2.png" class="avatar"> jayz:</div>
<blockquote>
<p>Does this mean when I have a tensor BCHW with (32,15,10,256), operations on this tensor within the autocast() context manager will not be mapped at all to tensor cores, because the C and H dimensions are not multiples of 8?</p>
</blockquote>
</aside>
<p>Not necessarily as e.g. cuDNN could pad your inputs and use TensorCores internally.</p>
<aside class="quote no-group" data-username="jayz" data-post="1" data-topic="152916">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jayz/48/46742_2.png" class="avatar"> jayz:</div>
<blockquote>
<p>Is that about right to make the model tensor core compatible?</p>
</blockquote>
</aside>
<p>Yes, the channels-last usage looks correct.</p>
Dataloader batching index

username: WenXiaowei
cooked: <p>Hi there,<br>
I am trying to do this:<br>
Given a sequence of images, let’s say: pic1, pic2… pic 32, with batch size of 4, I will get batches like:<br>
[pic1, pic2, pic3, pic4]. [pic5, pic6, pic7, pic8)] etc.<br>
But what I want is that the next batch does not start at batchsize+1 element, but having some element in common with the previous batch. e.g.<br>
[pic1, pic2, pic3, pic4]. [pic4, pic5, pic6, pic7].</p>
<p>Or just start batches at random position of the sequence of the image.</p>
<p>Cheers.<br>
Xw</p>

username: saeed_arabzadeh
cooked: <p>Hi WenXiaowei,<br>
You wanted to select some baches of your sequence at a random position, as you mentioned above. You can shuffle them first; after that, select them. For the problem, I think the torch.unfold will be helpful with a little trick to use it. It would help if you put all your picture in a tensor and for the first tdimension of it, use the torch.unfold to fold the first dimension to your batch with 25 percent overlap.</p>

username: WenXiaowei
cooked: <p>Hi, thank you for reply.</p>
<p>I cannot shuffle them because the order it important.<br>
I’ll try the second option <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"></p>
<p>Ty</p>

username: WenXiaowei
cooked: <p>Solve, by using a custom batch_sampler while creating dataloader.  instead of a simple sampler.</p>
<pre><code class="lang-auto">loader = DataLoader(
        dataset,
        batch_sampler=SequenceSampler(len(dataset), batch_size),
        num_workers=num_workers,
        pin_memory=pin_memory,
        collate_fn=collate_fn
    )
</code></pre>
nn.quantized.Conv1d; RuntimeError: [QConv2D] each dimension of output tensor should be greater than 0

username: 111357
cooked: <p>torch version: 1.9.1+cpu</p>
<p>When I run the demo in <a href="https://pytorch.org/docs/stable/generated/torch.nn.quantized.Conv1d.html#torch.nn.quantized.Conv1d" rel="noopener nofollow ugc"><code>torch.nn.quantized.Conv1d</code></a></p>
<pre><code class="lang-auto">m = nn.quantized.Conv1d(16, 33, 3, stride=2)
input = torch.randn(20, 16, 100)
# quantize input to quint8
q_input = torch.quantize_per_tensor(input, scale=1.0, zero_point=0,dtype=torch.quint8)
output = m(q_input)
</code></pre>
<p>I got the following error, do you have any suggestions?</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-36-b9b5fcef4003&gt; in &lt;module&gt;
      8 # quantize input to quint8
      9 q_input = torch.quantize_per_tensor(input, scale=1.0, zero_point=0,dtype=torch.quint8)
---&gt; 10 output = m(q_input)

~/.local/lib/python3.6/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1049         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1050                 or _global_forward_hooks or _global_forward_pre_hooks):
-&gt; 1051             return forward_call(*input, **kwargs)
   1052         # Do not call functions when jit is used
   1053         full_backward_hooks, non_full_backward_hooks = [], []

~/.local/lib/python3.6/site-packages/torch/nn/quantized/modules/conv.py in forward(self, input)
    328             input = F.pad(input, _reversed_padding_repeated_twice,
    329                           mode=self.padding_mode)
--&gt; 330         return ops.quantized.conv1d(input, self._packed_params, self.scale, self.zero_point)
    331 
    332     @classmethod

RuntimeError: [QConv2D] each dimension of output tensor should be greater than 0

</code></pre>

username: ptrblck
cooked: <p>Could you update to the latest stable or nightly release, please, as the code works fine in <code>1.12.0.dev20220425</code>?</p>
Maxout layer in pytorch

username: cham-3
cooked: <p>I  want to us maxout activation in pytorch, and I use torch.max() function to implement.</p>
<pre><code>class Maxout(nn.Module):
    def __init__(self):
        super(Maxout, self).__init__()
    def forward(self, x, y):
        return x.max(y)
</code></pre>
<p>Is it right?</p>

username: Quang_Tran_Ngoc
cooked: <p>I also have the same question. In my case, I use the code below. Can anyone help?</p>
<pre><code class="lang-auto">class Maxout(nn.Module):
         def __init__(self):
              super(Maxout, self).__init__()
        def forward(self, x, y):
              return torch.max(x,y)```</code></pre>

username: JuanFMontesinos
cooked: <p>You can define a layer like that but it’s not necesary, you can call torch max in forward and it will do the same.</p>

username: Abhisek_Panigrahi
cooked: <p>Check this: <a href="https://github.com/paniabhisek/maxout/blob/master/maxout.py" rel="nofollow noopener">maxout-layer</a></p>

username: yairschiff
cooked: <p>You can also check out <a href="https://github.com/yair-schiff/stochastic-orders-ICMN/blob/a5435840899b8683a913cb0156ca941d7da85b25/src/models.py#L10-L31" rel="noopener nofollow ugc">this MaxOut2D implementation</a> that performs maxout on the channels for an input that is C x H x W</p>
GradCam visualisation on modified EfficientNet_b0

username: kspyrikos
cooked: <p>Hello, I am currently working on my thesis and I am working with medical images. I want to add some GradCam visualisation on the outcome of my model. I am using a pretrained EfficientNet_b0 with ‘features_only=True’ (timm library):</p>
<pre><code class="lang-auto">class EfficientNet(torch.nn.Module):
        def __init__(self):
            super().__init__()
            # base model
            self.feature_extractor = timm.create_model('efficientnet_b0', pretrained=True, features_only=True)
            # Get the number input features that the classifier receive. 
            # define the classifier. Note that the feature extractor keep 
            self.classification_label = nn.Sequential(
                nn.Linear(self.feature_extractor.feature_info.channels()[-1], 1280),
                torch.nn.ReLU(True),
                torch.nn.Dropout(),
                torch.nn.Linear(1280, 1280),
                torch.nn.ReLU(True),
                torch.nn.Dropout(),
                torch.nn.Linear(1280, 4) # the labels are 4 
                )
            
            self.classification_reason = nn.Sequential(
                nn.Linear(self.feature_extractor.feature_info.channels()[-1], 1280),
                torch.nn.ReLU(True),
                torch.nn.Dropout(),
                torch.nn.Linear(1280, 1280),
                torch.nn.ReLU(True),
                torch.nn.Dropout(),
                torch.nn.Linear(1280, 3) # the reasons are 3
                )
          
            
            self.flat_gap = torch.nn.Sequential(
                torch.nn.AdaptiveAvgPool2d(1),
                torch.nn.Flatten()
            )
        def forward(self, x):
            features = self.flat_gap(self.feature_extractor(x)[-1])
            label = self.classification_label(features)
            reason = self.classification_reason(features)
            
            return label, reason
</code></pre>
<p>My model gets the features of the last convolution layer and then forward passes them to two classifiers. One classifier for the diagnoses (labels) of the images and one classifier for the reasons for bad quality (bad_light, blurry, low_resolution) of the images.</p>
<p>After training my model, I want to load the saved weights and present a GradCam visualisation of the second classifier (for bad quality reasons). I want to get a heatmap representation for each reason. Can someone help me with the implementation?</p>
<p>Any help will be appreciated. Thank you in advance <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
Detectron2 inference on CPU in C++

username: Sylvain_Ard
cooked: <p>Hi all,<br>
I would to do inference with a detectron2 mask_rcnn_X_101_32x8d_FPN_3x.yaml model on CPU  in C++ with Visual Studio 2019 please ?<br>
Thank you<br>
Best regards</p>

username: Sylvain_Ard
cooked: <p>I saw solutions for C++ detectron2 inference with Torchscript but all are with GPU, is it possible to use it with CPU ?</p>

username: ptrblck
cooked: <p>I assume in your code you are explicitly moving the data and model to the GPU via <code>.to(torch::kCUDA)</code> so remove this op and the training should run on the CPU by default.</p>

username: Sylvain_Ard
cooked: <p>And for take the model I cited please ?</p>
How to replace customized module in FX?

username: Liangzhi_Tang
cooked: <pre><code class="lang-auto">MyOldMod(torch.nn.Module):
    def __init__(self):
        super().__init__()

    def forward(self, x):
        print("IN MyOldMod")
        y = x * 1
        return y


class MyNewMod(torch.nn.Module):
    def __init__(self):
        super().__init__()

    def forward(self, x):
        print("IN MyNewMod")
        y = x + 2
        return y

class MyNet(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1a = MyOldMod()
        self.bn1 = torch.nn.BatchNorm2d(1)

    def forward(self, x):
        print("In MyNet!")
        return self.bn1(self.conv1a(x))
</code></pre>
<p>This is just a trivial example! My goal is to replace MyOldMod to MyNewMod, while torch.fx seems can’t detect customized module, i.e. MyOldMod here, what should I do?</p>

username: Liangzhi_Tang
cooked: <p>Anyone can help to solve this problem?</p>
[Differential Privacy] Does Autograd need the input tensor when performing the backward pass?

username: elliotto
cooked: <p>Hi.</p>
<p>We are undertaking research into the applications of Differential Privacy (DP) to neural networks. Building on some existing work [1] we are attempting to understand the internals of Pytorch’s Autograd, to see where it may be appropriate to inject DP to different parts of a neural net.</p>
<p>One of the core principles of DP is that once a DP mechanism has been applied to an input, any postprocessing applied to its output will remain DP, provided that the postprocessing does not involve information derived from the original dataset. That is to say that once we have privatized some data, we can’t ‘look’ at the original dataset again, or else we have broken privacy.</p>
<p>One problem we are stuck on, is whether the backward() function requires multiple ‘looks’ at the original Tensor. From watching lectures and reading Autograd’s documentation, it appears that the backward pass is calculating a graph with each node representing a gradient calculation, and that we only require the original Tensor to be used a single time time when traversing this graph.</p>
<p>However, it has been difficult to find direct code sources or references to validate this. [2] appears to specify that the input to the backward function only needs to be the Tensors that the forward function returns. However, this is quite abstract, and it has been difficult to find specific implementations within the codebase.</p>
<p>I was hoping there would be someone who would know generally whether we require using the input Tensor multiple times when calculating a gradient, and if possible could provide a reference to a part of the codebase that would show this clearly?</p>
<p>Thank you <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>[1]: Martin Abadi, Andy Chu, Ian Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and<br>
Li Zhang. 2016. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security (CCS ’16). Association for Computing Machinery, Vienna, Austria, 308–318. DOI: 10.1145/2976749.2978318.<br>
[2]: <a href="https://github.com/pytorch/pytorch/blob/089203f8bc71c64688def521284e32aee3fb5989/torch/autograd/function.py#L344" class="inline-onebox" rel="noopener nofollow ugc">pytorch/function.py at 089203f8bc71c64688def521284e32aee3fb5989 · pytorch/pytorch · GitHub</a></p>
What is an efficient way to replace specific columns in tensor

username: mhadnanali
cooked: <p>I need to apply a mask on specific columns of a tensor. I could not find any helping function for that, so i am creating a subset and then applying mask on the subset. Now i need to merge the subset into oringal tensor but i can not find some efficient way.</p>
<pre><code class="lang-auto">import torch
X=torch.rand(10,9) 
tensorsize = X.size()
indices = torch.tensor([0,3,7,5,4])  #sorting them will make the process faster?
candidateCF=torch.index_select(X,1,indices)
mask=torch.FloatTensor(candidateCF.size()).uniform_() &gt;= 0.3
output=candidateCF.mul(mask)
############
X[indices]= output    #is there any way to replace output subset into X on specified indices,  
</code></pre>
<p>For loop is too time-consuming. currently my code takes 12 seconds for one epoch… <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"></p>

username: Imbesat_Rizvi
cooked: <p>If column-wise or row-wise replacements are required you can do:</p>
<pre><code class="lang-auto">X[:, indices] = output # for column replacements
X[indices, :] = output # for row replacements
</code></pre>
<p>Does this serves your purpose?</p>

username: mhadnanali
cooked: <p>That’s what I was looking for. Thank you.</p>
RuntimeError: Event device type CUDA does not match blocking stream's device type CPU

username: pan24n
cooked: <p>when I do the unittest of test_autograd.py on pytorch1.11.0, it throw an RuntimeError: Event device type CUDA does not match blocking stream’s device type CPU. I want to know what causes this mistake.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/4/d4e5a0eda40ed307c0f1332d2a5193a38bd61dcb.png" data-download-href="https://discuss.pytorch.org/uploads/default/d4e5a0eda40ed307c0f1332d2a5193a38bd61dcb" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/4/d4e5a0eda40ed307c0f1332d2a5193a38bd61dcb_2_690x148.png" alt="image" data-base62-sha1="unncrRHPBTAHQsClUcK1fi7tsfN" width="690" height="148" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/4/d4e5a0eda40ed307c0f1332d2a5193a38bd61dcb_2_690x148.png, https://discuss.pytorch.org/uploads/default/original/3X/d/4/d4e5a0eda40ed307c0f1332d2a5193a38bd61dcb.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/d/4/d4e5a0eda40ed307c0f1332d2a5193a38bd61dcb.png 2x" data-dominant-color="303030"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">860×185 74.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>I cannot reproduce the issue in <code>v1.11.0</code> and a current source build from master and get:</p>
<pre><code class="lang-python">root@dd61f7adc653:/opt/pytorch/pytorch/test# python test_autograd.py -v -k test_type_conversions
test_type_conversions (__main__.TestAutograd) ... ok

----------------------------------------------------------------------
Ran 1 test in 3.296s

OK
</code></pre>

username: pan24n
cooked: <p>Is this test still passable on ROCM?</p>

username: ptrblck
cooked: <p>Oh, I don’t know as I’m not familiar with ROCm.<br>
If this test is failing on your system using ROCm, please feel free to open an <a href="https://github.com/pytorch/pytorch/issues">issue of GitHub</a> so that the code owners can take a look at it.</p>
Cannot compile TorchLib on Windows 10 with GNU compiler

username: Valentin
cooked: <p>Hello. Trying to compile test  torch.cpp file that includes TorchLib on Windows 10 , and it fails.</p>
<p>compile command:<br>
<code>g++ torch.cpp -I include -L lib</code></p>
<p>file contents:</p>
<pre><code class="lang-auto">#include &lt;torch/script.h&gt; // One-stop header.

#include &lt;iostream&gt;
#include &lt;memory&gt;

int main(int argc, const char* argv[]) {
  if (argc != 2) {
    std::cerr &lt;&lt; "usage: example-app &lt;path-to-exported-script-module&gt;\n";
    return -1;
  }


  torch::jit::script::Module module;
  try {
    // Deserialize the ScriptModule from a file using torch::jit::load().
    module = torch::jit::load(argv[1]);
  }
  catch (const c10::Error&amp; e) {
    std::cerr &lt;&lt; "error loading the model\n";
    return -1;
  }

  std::cout &lt;&lt; "ok\n";
}
</code></pre>
<p>Errors:<br>
a lot of<br>
include/c10/macros/Macros.h:252:5: error: ‘__assert_fail’ was not declared in this scope<br>
and warnings -<br>
sorry can’t post them here</p>
<p>Thanks for any help.</p>
New attributes in torch.Parameter are thrown away when dumping with pickle

username: Michael-H777
cooked: <p>sorry, can’t find the correct topic for issue described.</p>
<p>please see attached image<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/9/0956b8e55495714e6e7a030c8786c304c9e76a96.png" data-download-href="https://discuss.pytorch.org/uploads/default/0956b8e55495714e6e7a030c8786c304c9e76a96" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/9/0956b8e55495714e6e7a030c8786c304c9e76a96.png" alt="image" data-base62-sha1="1kC67LUR3xNCyV7atv72ijeMdVk" width="690" height="369" data-dominant-color="382624"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">840×450 30.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>what would be the proper way to store new attributes with torch.parameter? Let me be specific, i need to create new attribute to pytorch parameters, and store into pickle.</p>
<p>EDIT:</p>
<pre><code>import torch 
import pickle 
test_param = torch.nn.Parameter(torch.rand(2, 2))
test_param._new_param = 10 
print(f'{test_param._new_param = }') # this prints 'test_param.new_param = 10'
pickle.dump(test_param, open('test.pickle', 'wb')) 
new_param = pickle.load(open('test.pickle', 'rb'))
print(f'{hasattr(new_param, "_new_param") = }') # this prints 'hasattr(new_param, "new_param") = False'
print(f'{new_param._new_param = }') # this bugs with attribute error
# what would be the correct way to store new attributes into torch.Parameter 
# and be able to access it when loading from pickle file?
</code></pre>

username: iordanis
cooked: <p>You should paste a fully working code example.</p>

username: Michael-H777
cooked: <p>please elaborate, the code above is able to reproduce the issue. what would qualify as ‘working code’? this bug would break the code, therefore there is no ‘working code’.</p>

username: KFrank
cooked: <p>Hi Michael!</p>
<aside class="quote no-group" data-username="Michael-H777" data-post="3" data-topic="152868" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/michael-h777/48/36056_2.png" class="avatar"> Michael-H777:</div>
<blockquote>
<p>please elaborate</p>
</blockquote>
</aside>
<p>At the risk of putting words into <a class="mention" href="/u/iordanis">@iordanis</a>’s mouth, perhaps he means<br>
that you shouldn’t post images of textual information, as doing so breaks<br>
copy-paste (as well as accessibility and searchability).</p>
<p>If I were to try running your code (working or not), I would soon discover<br>
that pasting an image into a python interpreter does me no good.</p>
<p>Best.</p>
<p>K. Frank</p>

username: Michael-H777
cooked: <p>Thank you, I have put the code required to reproduce the bug in text.</p>
Torchvision questions

username: Sylvain_Ard
cooked: <p>Hello,<br>
I saw recently torchvision. What is it exactly, how recent is it and have you books about it ?<br>
Thank you<br>
Best regards</p>

username: AlphaBetaGamma96
cooked: <p>The documentation is always a good start before getting books: <a href="https://pytorch.org/vision/stable/" class="inline-onebox" rel="noopener nofollow ugc">torchvision — Torchvision 0.12 documentation</a></p>
BCELoss error with mps

username: sejung
cooked: <p>I’m using BCELoss. But when I use mps, it gives me the following error. It runs well without mps.</p>
<p>loc(“reductionMeanTensor”("(mpsFileLoc): /AppleInternal/Library/BuildRoots/560148d7-a559-11ec-8c96-4add460b61a6/Library/Caches/com.apple.xbs/Sources/MetalPerformanceShadersGraph/mpsgraph/MetalPerformanceShadersGraph/Core/Files/MPSGraphUtilities.mm":281:0)): error: invalid axes: 2<br>
LLVM ERROR: Failed to infer result type(s).</p>
Deploy PyTorch models with flutter (android & ios)

username: Emad_Elazhary
cooked: <p>Flutter has a very big promise of writing one code base and delopying it to Android, ios and web, of course Google (the owner of flutter) has added the capability to include a tensorflow model to your flutter projects.</p>
<p>When can we see a flutter package that would allow us to import a pytorch model and do on device inference in Android or ios?</p>

username: RicCu
cooked: <p>There’s a <a href="https://pub.dev/packages/torch_mobile" rel="nofollow noopener">package on pub.dev</a> for pytorch mobile, however it only works for android and it hasn’t been updated since its release in October.<br>
I’ve been meaning to work on a plugin as well, but I’ve been swamped with other stuff <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:">  If I ever find the time to work on it I’ll be sure to update this thread.<br>
And note that there’s no PyTorch runtime for the browser, so Flutter web is out of reach (until someone comes through with a version of torch.js).</p>

username: MohamedAliRashad
cooked: <p><a class="mention" href="/u/riccu">@RicCu</a> Is there any updates on this ?</p>
<p>PyTorch has released PyTorch Live, it may be useful to you</p>
Classification with unbalanced classes and big images

username: Sylvain_Ard
cooked: <p>Hello,<br>
I am looking a code for a classification program with very unbalanced classes and big images (2500x1500) where details are very importants so with no resizing before enter in the neural network.<br>
Thank you<br>
Best regards</p>

username: Sylvain_Ard
cooked: <p>nobody has an answer ?</p>

username: KFrank
cooked: <p>Hi Sylvain!</p>
<aside class="quote no-group" data-username="Sylvain_Ard" data-post="1" data-topic="152519" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sylvain_ard/48/30714_2.png" class="avatar"> Sylvain_Ard:</div>
<blockquote>
<p>I am looking a code for a classification program with very unbalanced classes</p>
</blockquote>
</aside>
<p>You say “classification” so I assume that you are dealing with (potentially<br>
multi-class) classification, but not semantic segmentation.</p>
<p>Unless you have a very large number of classes, you would typically<br>
use <code>CrossEntropyLoss</code> as your loss criterion.  If you have enough<br>
samples of your under-represented classes, I would suggest sampling<br>
those classes more heavily, e.g., with <code>WeightedRandomSampler</code>.  If you<br>
don’t have enough samples to use <code>WeightedRandomSampler</code> without<br>
duplicating samples within a batch, then user the <code>weight</code> constructor<br>
argument of <code>CrossEntropyLoss</code> to reweight the underrepresented<br>
classes in the loss.</p>
<aside class="quote no-group">
<blockquote>
<p>and big images (2500x1500) where details are very importants so with no resizing</p>
</blockquote>
</aside>
<p>2500x1500 is not excessively big.  Have you tried any of the common<br>
image classification networks on images of this size?</p>
<p>I was able to forward and backward pass a batch of two (three-channel)<br>
2500x1500 images through <code>torchvision.models.resnet18</code> on a gpu<br>
4 GB of memory.</p>
<p>Note that torchvision’s resnet networks have are agnostic about the size<br>
of the input image (as they have an <code>AdaptiveAvgPool2d</code> layer just<br>
before their final fully-connected layer).</p>
<p>Best.</p>
<p>K. Frank</p>

username: Sylvain_Ard
cooked: <p>Ok thank you I have two trainings to do, one with 600 classes and the other with 10 classes. 600 classes is what you call very lot classes ?<br>
Thank you very much</p>

username: Sylvain_Ard
cooked: <p>Are all images must have same size before enter into resnet ?</p>
<p>Thank you</p>
How to multiply each channel by the sum of its pixel values?

username: Andrew_Legend
cooked: <p>I have the standard NxCxWxH feature map. And I want to multiply each channel by the sum of all pixel values of each channel. How can I write a function that can do such a thing?</p>

username: Matias_Vasquez
cooked: <p>If I understood correctly, you want to add every value per each channel. Meaning, that for an RGB image you would add all of the R values together, all of G is another sum and B would be a third sum. Each of these results is then multiplied to the original values, such that if the sum of R=50, then you multiply every value in the R channel by 50.</p>
<p>If this is correct, then the following code should do what you want.</p>
<p>If that is not what you want, please let me know.</p>
<pre><code class="lang-python">import torch

img = torch.rand(1, 3, 10, 10)

def mult_by_channel_sum(img):
    sum_per_channel = img.sum(dim=3).sum(dim=2).unsqueeze(2).unsqueeze(3)
    return img * sum_per_channel

print(mult_by_channel_sum(img).shape)
</code></pre>
<pre><code class="lang-python"># Output:
torch.Size([1, 3, 10, 10])
</code></pre>

username: Andrew_Legend
cooked: <p>It seems what I want but will such a thing work on different batch sizes other than 1?</p>

username: Matias_Vasquez
cooked: <p>Yes, this should work for any batch size.</p>
<p>This should also work for any number of channels, any H and W.</p>
<p>Also if you have more dimensions, and you only care that it is the last two dimensions, then you could change it to this.</p>
<pre><code class="lang-python">def mult_by_channel_sum(img):
    sum_per_channel = img.sum(dim=-1).sum(dim=-1).unsqueeze(-1).unsqueeze(-1)
    return img * sum_per_channel
</code></pre>
I want to create yolo V5 with pytorch, and create a function that recognizes a face with my own data and returns the recognized size

username: YuhskeHujisaki
cooked: <p>I want to create yolo V5 with pytorch, and create a function that recognizes a face with my own data and returns the recognized size.<br>
So I have three questions.</p>
<ol>
<li>How can I make yolo v5 by myself?</li>
<li>How can I return the recognized size?</li>
</ol>
Type Error Expected Scalar Type Long but found float INT

username: ANIL_PUDASAINI
cooked: <p>def train(net, data, model_name, batch_size=10, seq_length=50, lr=0.001, clip=5,<br>
print_every_n_step=50, save_every_n_step=5000):<br>
net.train()<br>
opt = torch.optim.Adam(net.parameters(), lr=lr)<br>
criterion = nn.CrossEntropyLoss()</p>
<pre><code>if train_on_gpu:
    net.cuda()

n_chars = len(net.chars)
counter = epoch = 0   
loss_history = []
start = time.time()    
while True:
    epoch += 1
    h = net.init_hidden(batch_size)
    
    for x, y in get_batches(data, batch_size, seq_length):
        counter += 1
        x = one_hot_encode(x, n_chars)
        inputs, targets = torch.from_numpy(x), torch.from_numpy(y)
        if train_on_gpu:
            inputs, targets = inputs.cuda(), targets.cuda()

        h = tuple([each.data for each in h])

        net.zero_grad()
        
        output, h = net(inputs, h)
        loss = criterion(output,targets.view(batch_size*seq_length) )
        loss.backward()
        nn.utils.clip_grad_norm_(net.parameters(), clip)
        opt.step()
        
        # loss stats
        if counter % print_every_n_step == 0:
            print(f"Epoch: {epoch:5} | Step: {counter:6} | Loss: {loss.item():.4f} | Elapsed Time: {time_since(start)}")

        if counter % save_every_n_step == 0:
            print(f"Epoch: {epoch:5} | Step: {counter:6} | Loss: {loss.item():.4f} | Elapsed Time: {time_since(start)}")
            print(" --- Save checkpoint ---")
            checkpoint = {
                'n_hidden': net.n_hidden,
                'n_layers': net.n_layers,
                'state_dict': net.state_dict(),
                'tokens': net.chars,
                'loss_history': loss_history
            }
            torch.save(checkpoint, open(f"{model_name}/epoch_{epoch}.pth", 'wb'))                
    loss_history.append(loss.item())</code></pre>
Torch.no_grad and register_hook inside forward

username: Gaoyuan_Yang
cooked: <p>Hi, I want to change part of intermediate layer’s activation to zero during forward pass,<br>
suppose my forward function is like this:</p>
<pre><code class="lang-python">def forward(self, x):
    x = self.l1(x)
    x = F.relu(x)
    x[:10,:] = 0

    def x_hook(grad):
        grad_clone = grad.clone()
        grad_clone[:10,:]= 0
        return grad_clone
    x.register_hook(x_hook)
    
    output = F.log_softmax(x,dim=1)
    return output
</code></pre>
<p>this runs well inside train function, however inside my test function</p>
<pre><code class="lang-python">def test(model, device, test_loader):
    model.eval()
    ...
    with torch.no_grad():
        for data, target in test_loader:
            ...
            output = model(data)
            ...
</code></pre>
<p>However, while running the code inside test model, it gives out an error<br>
<strong>cannot register a hook on a tensor that doesn’t require gradient</strong><br>
My question is：<br>
(1)Can we put register_hook of torch.tensor inside forward function?<br>
(2)Why register_hook function called inside torch.no_grad() still calculate gradients here?</p>
<p>Any suggestion would be appreciated!</p>

username: albanD
cooked: <p>Hi,</p>
<p>To fix your original question,<br>
You want to add the hook only if <code>x.requires_grad == True</code>.<br>
There are no gradient computed and so to avoid the user registering a hook and having it not being called, we raise an error.</p>
<p>Note that since you override the values of x in a differentiable manner, the gradients for these entries will be zeroed out by the backward of <code>x[:10,:] = 0</code>. So you don’t actually need the hook here!</p>

username: hsong
cooked: <p>also solved my problem. Thank you!</p>
Gradients in loss function and efficient calculation

username: jloverich
cooked: <p>I’m using the l2 norm of a pde as the loss function, the input are variables (x,y) and the output are some quantities (r, u, v), and I want to calculate the jacobian dr/dx, dr/dy, du/dx … and I’d like to do it in batch - (I believe it’s very slow otherwise, though cant be sure since I haven’t been able to try!).  Currently I use the jacobian function and loop over each sample in the batch, as shown below. I’d like to do this without the for loop (or much more efficiently) can this be done as a simple batch operation?  If so how?</p>
<pre><code>    x, y = batch
    x.requires_grad_(True)
    y_hat = self(x)

    jacobian_list = []
    
   for inp in x:
        inp = inp.unsqueeze(dim=0)
        jacobian = torch.autograd.functional.jacobian(
            self, inp, create_graph=True, strict=False, vectorize=False
        )
        jacobian_list.append(jacobian)

    gradients = torch.reshape(torch.stack(jacobian_list), (-1, 3, 2))
</code></pre>
<p>Thanks!</p>

username: jloverich
cooked: <p>Looks like this has also been discussed here <a href="https://discuss.pytorch.org/t/computing-batch-jacobian-efficiently/80771/4" class="inline-onebox">Computing batch Jacobian efficiently - #4 by albanD</a>, not sure if there are any updates.</p>

username: AlphaBetaGamma96
cooked: <p>If you want batched Jacobian (or higher order) gradient functions you can check out functorch, their github page is <a href="https://github.com/pytorch/functorch" rel="noopener nofollow ugc">here</a>.</p>
Suggested loss function for One-Hot Encoding Target

username: Jacob_Dallas
cooked: <p>I realize this has been discussed at nauseam, but I just wanted to verify that if the output of my network is a vector [0.1, 0.5, 0.3, 0.1] and my target is a one-hot encoded vector [0, 0, 1, 0] I would use the nn.NLLLoss() function correct? Oh and for clarification, the last layer of my network is a LogSoftMax layer.</p>

username: eqy
cooked: <p>Yes, although you may want to consider returning the raw logits and combining the two with <code>CrossEntropyLoss</code>: <a href="https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html" class="inline-onebox" rel="noopener nofollow ugc">CrossEntropyLoss — PyTorch 1.11.0 documentation</a></p>

username: Jacob_Dallas
cooked: <p>Thanks! I was avoiding that so that I could see the output of the softmax (petty I know haha)</p>
Regression with semi-supervised VAE

username: Tavados
cooked: <p>I’m currently working on a VAE that does regression with semi-supervised learning. So there is labeled and unlabeled data, and the goal is to achieve acceptable labels for the unlabeled data.</p>
<p>The data (images) is being normalized to the [0,1] range and Adam is used as optimizer.<br>
The architecture is:</p>
<ol>
<li>Encoder with convolutions and ReLUs</li>
<li>Three fully connected layers with the bottleneck in the middle<br>
3a) Decoder with ConvTranspose, BatchNorms, ReLUs and Sigmoid<br>
3b) Regressor with one linear layer, dropout and Softplus</li>
</ol>
<p>My questions are:</p>
<ol>
<li>Does it make sense to have 3b separated from the decoder and to feed the last fully connected layer (i.e. the one behind the bottleneck, not the bottleneck itself) to it? I also tried to put the result in an additional image layer of the decoder, taking the average of that layer as the prediction result.</li>
<li>Does it make sense to just add the MSE loss of the regressor to the BCE+KLD loss of the VAE and scale it by some amount, or should in this case MSE also be used for the decoder loss?</li>
<li>Are there any suggestions or tips to this architecture? At the moment it looks strange to me with the MSE loss becoming relatively small compared to the BCE loss, which is not really going down, since the data isn’t binary, but spread over the whole greyscale.</li>
</ol>
<p>Thanks in advance. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: Idan_Cohen
cooked: <p><a class="mention" href="/u/tavados">@Tavados</a> Have you had any sucess with that my friend? I am currently working on a same project and wondered about your second question as well.</p>

username: Ziqi.Yang394
cooked: <p>Hi, anybody solved the problems? I am focusing on regression problem with discrete data with VAE now, and I tried to use MSE loss instead of BCE as well. The loss value is really low, but output results looked not good. I wonder the reason of this problem.</p>
How to create banded matrices from multiple bands

username: LeanderK
cooked: <p>so I have multiple tensors containing bands (each different width). I now want to construct the full matrix from those bands. I have the “offset” of each band but I am unsure how to do it. I know <a href="https://pytorch.org/docs/stable/generated/torch.diag.html" class="inline-onebox" rel="noopener nofollow ugc">torch.diag — PyTorch 1.11.0 documentation</a>, but it assumes width = 1. Am I overlooking something?</p>
Use visdom in the background?

username: fermat97
cooked: <p>I use visdom on a remote server and I connect via ssh. As far as I know we have to start the visdom by the command <code>$ visdom</code>.<br>
My problem is my remote server kicked me out if I am not active on the terminal. How I can run <code>visdom</code> to avoid such problem, since it takes a bit to every time call (after each epoch) the plotting procedure by <code>visdom</code>.</p>

username: ptrblck
cooked: <p>I’m not sure, if this would avoid your disconnections, but I often use <a href="https://help.ubuntu.com/community/Screen" rel="nofollow noopener">screen</a> to start something on a remote server and just detach from the current session.</p>

username: mohit_kushwah
cooked: <p>Thank you so much. This worked for me</p>