akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
StopIteration is not working in Custom Iterable Dataloader

username: Abdul
cooked: <p>I have written a custom Iterable Dataloader for my project. While testing it for a single epoch, it goes through the whole data, then resets the counter to zero and starts iterating again bypassing the “raise StopIteration” command. I am a bit confused why it is working this way. Any help would be highly appreciated.</p>
<p>The <strong>next</strong>() function is given below</p>
<pre><code class="lang-auto">def __next__(self):
        if(self._counter &lt; len(self.index)):
            self.sample = self._data[self.index[self._counter] : self.index[self._counter] + self._chunk_size, :]
            self.label = self._labels[self.index[self._counter]]
            
            if self._mean_std_normalize:
                self.sample = (self.sample - 17041.3242) / (131.5367)
            
            print(self._counter)
            self._counter += 1
            return self.sample, self.label
        else:
             self._counter = 0
             raise StopIteration
</code></pre>
<p>While testing it using a DataLoader, it works completely fine with a <strong>batch_size = 5</strong>, and runs in infinite loop for other batch_sizes. The testing script is</p>
<pre><code class="lang-auto">train_data = IterDataset(root_path = 'D:/idrad', folder = 'train',
                          chunk_size = 30, mean_std_normalize = True, shuffle = True)
train_loader = DataLoader(dataset = train_data, batch_size = 10)

epochs = 1
for i in range(epochs):
    for idx, (md, target) in enumerate(train_loader):
        pass

</code></pre>

username: Abdul
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>  kindly help me to sort out this issue.</p>
<p>Other members are also encouraged to help me.</p>
<p>Thanking in advance</p>

username: Abdul
cooked: <p>This solved my problem.</p>
<pre><code class="lang-auto">def __next__(self):
        if(self._counter &lt; len(self.index)):
            self.sample = self._data[self.index[self._counter] : self.index[self._counter] + self._chunk_size, :]
            self.label = self._labels[self.index[self._counter]]
            
            if self._mean_std_normalize:
                self.sample = (self.sample - 17041.3242) / (131.5367)
            
            print(self._counter)
            self._counter += 1
            return self.sample, self.label
        else:
             raise StopIteration
</code></pre>
<p>and moving <code>self._counter = 0</code> to <code>__iter__</code> method. Now, my <code>__iter__</code> method is</p>
<pre><code class="lang-auto">def __iter__(self):
    self._counter = 0
    return self
</code></pre>
Documentation for CUDA backend

username: kevs
cooked: <p>I am interested in reading about how the CUDA backend for pytorch works but I cannot orient myself in the source code. Specifically, i am interested in how the reduction kernels work for operations like sum with axis parameters. Can someone point me towards do the documentation for this or at least where to look in the source code?</p>
<p>I also wonder if the kernels are JIT compiled? But I could probably answer this question myself if someone has a reference.</p>
Deepcopy of pre-trained model failing with Pickle Error

username: Shashank_Holla
cooked: <p>Hi, for a neural style problem, Im trying to deepcopy a pre-trained model. The pre-trained model is a jit.traced model.<br>
While performing the deepcopy, its failing with Pickle error. Request you help in debugging this error.</p>
<p>Stack trace-</p>
<pre><code class="lang-auto">4     # Deep copy vgg model
----&gt; 5     cnn = copy.deepcopy(cnn)
      6     normalization = Normalization(normalization_mean, normalization_std).to(device)
      7     # Collect metrics

/usr/lib/python3.6/copy.py in deepcopy(x, memo, _nil)
    167                     reductor = getattr(x, "__reduce_ex__", None)
    168                     if reductor:
--&gt; 169                         rv = reductor(4)
    170                     else:
    171                         reductor = getattr(x, "__reduce__", None)

/usr/local/lib/python3.6/dist-packages/torch/jit/__init__.py in __getstate__(self)
   1746                 "ScriptModules cannot be deepcopied using copy.deepcopy or saved using torch.save. " +
   1747                 "Mixed serialization of script and non-script modules is not supported. " +
-&gt; 1748                 "For purely script modules use my_script_module.save(&lt;filename&gt;) instead.")
   1749 
   1750         # Python magic methods do method lookups on an object's class type, instead of looking up

PickleError: ScriptModules cannot be deepcopied using copy.deepcopy or saved using torch.save. Mixed serialization of script and non-script modules is not supported. For purely script modules use my_script_module.save(&lt;filename&gt;) instead.
</code></pre>

username: ptrblck
cooked: <p>Based on the error message, it seems like a known limitation, which apparently needs you to save and load the model via <code>my_script_module.save(&lt;filename&gt;)</code>:</p>
<blockquote>
<p>PickleError: ScriptModules cannot be deepcopied using copy.deepcopy or saved using torch.save. Mixed serialization of script and non-script modules is not supported. For purely script modules use my_script_module.save() instead.</p>
</blockquote>

username: Geoffrey_Payne
cooked: <p>I found I got this error by trying to save a model without putting it into a dictionary object first.</p>
Convolution Backward with unfold and fold

username: kraken-d
cooked: <p>There is an implementation of conv2d forward pass in the unfold documentation page and it works very wellhelp. Could the same be done for a conv2d backward pass using fold and unfold. If so please help me to implement it. I found out the gradients of weight and bias and it matches the pytorch ones but the gradient of the input seems to be not the same. Please help.</p>

username: evinpinar
cooked: <p>Any solutions for that?</p>
Sample tensor at specific coordinates using Spatial Transformer Networks

username: Linus_Tomasulo
cooked: <p>Hello,</p>
<p>I have two tensors:</p>
<ul>
<li>features - with shape [batch_size, channels, height, width]</li>
<li>coordinates - with shape [batch_size, num_coordinates, 2]</li>
</ul>
<p>The coordinates tensor has values in range [-1,1] and every row indicates a location in the features tensor (height and width are also considered to be normalized between -1 and 1]).</p>
<p>I would like to sample the features tensor at the coordinates described by the coordinates tensor using STNs. The result must have the shape: [batch_size, channels, num_coordinates].</p>
<p>Does anyone has a simple solution code?</p>
<p>Thank you!</p>

username: Linus_Tomasulo
cooked: <p>Solved with:</p>
<pre><code class="lang-auto">coordinates = coordinates.unsqueeze(1)
features = torch.nn.functional.grid_sample(features, coordinates, align_corners=True)
features = features.squeeze(2)
</code></pre>
Horizontal compression and vertical elongation as Data Augmentation

username: elibon
cooked: <p>Reading Yann LeCun’s report “Gradient-based learning applied to document recogniition” i saw he used a couple of data augmentation techniques on the MNIST dataset, including <strong>squeezing</strong> (“<em>simultaneous horizontal compression and vertical elongation, or the reverse</em>”). I would like to implement <strong>squeezing</strong> as data augmentation in my own PyTorch project, however I can’t seem to find a proper way to do it. I have been searching around but haven’t found anything useful.</p>
<p>Anyone who knows how to implement <strong>squeezing</strong> in pytorch, either with the pre-built torchvision transformations or as a custom transformation?</p>

username: KFrank
cooked: <p>Hi Elias!</p>
<aside class="quote no-group" data-username="elibon" data-post="1" data-topic="117709" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/elibon/48/30358_2.png" class="avatar"> elibon:</div>
<blockquote>
<p>Anyone who knows how to implement <strong>squeezing</strong> in pytorch, either with the pre-built torchvision transformations or as a custom transformation?</p>
</blockquote>
</aside>
<p>I think that <a href="https://pytorch.org/vision/0.9/transforms.html#torchvision.transforms.Resize" rel="noopener nofollow ugc">torchvision.transforms.Resize</a> will do what you want.  If you<br>
want the shape of your squeezed image to be the same as the shape<br>
of the untransformed image, you will have to pad and crop the result<br>
of <code>Resize</code>.</p>
<p>Best.</p>
<p>K. Frank</p>

username: elibon
cooked: <p>Tanks a lot KFrank!</p>
<p>I will definitely try that out and let everybody know if it works.</p>
<p>Kind Regards,<br>
Elias</p>
Write custom non maximum suppression operator for TorchScript

username: teotfw
cooked: <p>Hi there,</p>
<p>I’m trying to write a custom nms operator for TorchScript. The original code is here: <a href="https://github.com/lucastabelini/LaneATT/tree/main/lib/nms" rel="noopener nofollow ugc">https://github.com/lucastabelini/LaneATT/tree/main/lib/nms</a>.</p>
<p>My code for building nms operator is here: <a href="https://github.com/hthoai/LaneATT/tree/torchscript/lib/nms_torchscript" rel="noopener nofollow ugc">https://github.com/hthoai/LaneATT/tree/torchscript/lib/nms_torchscript</a></p>
<p>I caught this error:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/8/e83376c084d60be8b1b2deb1f4553f44d16c235b.png" data-download-href="https://discuss.pytorch.org/uploads/default/e83376c084d60be8b1b2deb1f4553f44d16c235b" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/8/e83376c084d60be8b1b2deb1f4553f44d16c235b_2_690x183.png" alt="image" data-base62-sha1="x8926EjdQZAFm4CMRnEplMvYugP" width="690" height="183" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/8/e83376c084d60be8b1b2deb1f4553f44d16c235b_2_690x183.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/8/e83376c084d60be8b1b2deb1f4553f44d16c235b_2_1035x274.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/8/e83376c084d60be8b1b2deb1f4553f44d16c235b_2_1380x366.png 2x" data-dominant-color="31353D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1393×371 45.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Is there anyone know how to fix it?!</p>

username: ptrblck
cooked: <p>It seems that <code>Python.h</code> is missing on your system, so you might want to install e.g. <code>libpython-dev</code>.</p>
Affine Transform bug in torch.distributions

username: nmourdou
cooked: <p>Hi guys,</p>
<p>I have the distribution of a <strong>MxN</strong> random matrix, say <strong>A</strong>, and I want to get the distribution of  <strong>Av</strong>, where <strong>v</strong> is a fixed <strong>N</strong>-dimensional vector. Obviously, the output of this should be an <strong>M</strong> dimensional vector.</p>
<p>But, when I use the command,</p>
<pre><code class="lang-auto">TransformedDistribution( A,  AffineTransform(0, v))
</code></pre>
<p>I get again a distribution over <strong>MxN</strong> matrices. It is as if the <strong>TransformedDistribution</strong> does not work.</p>
<p>Any ideas on why is this happening?</p>
<p>Thank you</p>
Simulate nn.BatchNorm2d of pytorch with python code(batch normalization in pytorch)

username: mahnazbanoo
cooked: <p>i want to simulate BatchNorm2d and write this code and compare the result with result of nn.BatchNorm2d function of pytorch when i print the output of my code to the output of nn.BatchNorm2d,each channels of the output tensor denoted by hk1,hk2 is a coefficient of the output channels of the nn.BatchNorm2d function.i want to understand what are these coefficients?</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

m1 = nn.BatchNorm2d(2)
inp1 = torch.tensor([[[[1.,0.,3.,2.]],[[0.,2.,0.,4.]]],[[[0.,3.,0.,2.]],[[0.,1.,0.,1.]]],[[[0.,5.,0.,2.]],[[0.,1.,3.,1.]]]])

ot0 = torch.mean(inp1,(0,2,3))
jazr0 = torch.var(inp1,(0,2,3))

gt1 = inp1[:,0,:,:]
rd1=gt1-ot0[0]
hk1 = rd1/jazr0[0]

gt2 = inp1[:,1,:,:]
rd2=gt2-ot0[1]
hk2 = rd2/jazr0[1]

print('****batchnorm2d****:\n',m1(inp1))

btch0 = m1(inp1)[:,0,:,:]
btch1 = m1(inp1)[:,1,:,:]
print(hk1/btch0)
print(hk2/btch1)
</code></pre>

username: mahnazbanoo
cooked: <p>the output of this code is:</p>
<pre><code class="lang-auto">****batchnorm2d****:
 tensor([[[[-0.3216, -0.9649,  0.9649,  0.3216]],

         [[-0.8628,  0.7301, -0.8628,  2.3230]]],


        [[[-0.9649,  0.9649, -0.9649,  0.3216]],

         [[-0.8628, -0.0664, -0.8628, -0.0664]]],


        [[[-0.9649,  2.2514, -0.9649,  0.3216]],

         [[-0.8628, -0.0664,  1.5266, -0.0664]]]],
       grad_fn=&lt;NativeBatchNormBackward&gt;)
tensor([[[0.5897, 0.5897, 0.5897, 0.5897]],

        [[0.5897, 0.5897, 0.5897, 0.5897]],

        [[0.5897, 0.5897, 0.5897, 0.5897]]], grad_fn=&lt;DivBackward0&gt;)
tensor([[[0.7301, 0.7301, 0.7301, 0.7301]],

        [[0.7301, 0.7301, 0.7301, 0.7301]],

        [[0.7301, 0.7301, 0.7301, 0.7301]]], grad_fn=&lt;DivBackward0&gt;)

Process finished with exit code 0

</code></pre>

username: ptrblck
cooked: <p>I’m not sure I understand your code correctly, but if you would like to manually implement a batchnorm layer, you could use <a href="https://github.com/ptrblck/pytorch_misc/blob/master/batch_norm_manual.py">this code</a> I’ve written some time ago.</p>
What's the exactly meaning of 18(ie, 2*3*3) channels of the offset in a deformable convolution?

username: Naruto-Sasuke
cooked: <p>I want to visualize the offset of the deformable convolution with kernel size 3*3.<br>
So It’s essential for me to know what’s the exact meaning of these channels.</p>
<p>I write down something possible here:</p>
<pre><code class="lang-python">upper-left: ul
upper-right: ur
bottom-left: bl
bottom-right: br
up: u
bottom: b
right: r
left: l
center: c


offset: 
delta_ul_x, delta_ul_y, delta_u_x, delta_u_y, delta_ur_x, delta_ur_y;
delta_l_x, delta_l_y, delta_c_x, delta_c_y, delta_r_x, delta_r_y;
delta_bl_x, delta_bl_y, delta_b_x, delta_b_y, delta_br_x, delta_br_y;

</code></pre>

username: ptrblck
cooked: <p>Could you add a bit more background to the question, please?<br>
Is “deformable convolution” a custom modules, which uses these parameters?</p>

username: Naruto-Sasuke
cooked: <p>Hi, ptrblck<br>
It is a newly added module in torchvision, however, the document about the offset is not clear.</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/vision/blob/master/torchvision/ops/deform_conv.py" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/vision/blob/master/torchvision/ops/deform_conv.py" target="_blank" rel="noopener nofollow ugc">pytorch/vision/blob/master/torchvision/ops/deform_conv.py</a></h4>
<pre><code class="lang-py">import math

import torch
from torch import nn, Tensor
from torch.nn import init
from torch.nn.parameter import Parameter
from torch.nn.modules.utils import _pair
from typing import Optional, Tuple
from torchvision.extension import _assert_has_ops


def deform_conv2d(
    input: Tensor,
    offset: Tensor,
    weight: Tensor,
    bias: Optional[Tensor] = None,
    stride: Tuple[int, int] = (1, 1),
    padding: Tuple[int, int] = (0, 0),
    dilation: Tuple[int, int] = (1, 1),
    mask: Optional[Tensor] = None,
</code></pre>

  This file has been truncated. <a href="https://github.com/pytorch/vision/blob/master/torchvision/ops/deform_conv.py" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


username: ptrblck
cooked: <p>Thanks for the reference! Could you point me to the <code>2*3*3</code> usage from your original post?</p>
How to select feature maps of resnet50FPN corresponding to region proposals in a faster-rcnn

username: jaeger94
cooked: <p>Hi all,</p>
<p>I have trained a FRCNN on my custom dataset and I just want the output of RPN and ROI-pool.<br>
I have used resnet50 FPN as the backbone.</p>
<p>For every image, I just want the region proposals from the network and their corresponding feature maps so that I can run my custom classifier that I have developed separately on this features that am getting.</p>
<p>The issue is -<br>
initially I used a simple RESNET - non-FPN architecture and I was able to do what I wanted since the output of the backbone was a single set of feature maps, however when using FPN am getting multiple feature maps the name and shape of which am showing below :<br>
‘’’<br>
0 torch.Size([1, 256, 200, 200])<br>
1 torch.Size([1, 256, 100, 100])<br>
2 torch.Size([1, 256, 50, 50])<br>
3 torch.Size([1, 256, 25, 25])<br>
pool torch.Size([1, 256, 13, 13]) ‘’’</p>
<p>Now I don’t understand which of these feature maps to use and I saw a formula k=k0+log2(sqrt(wh)/224) but am not sure how and where to use this.</p>
<p>Also when I trained using simple Non-FPN architecture i had used the same backbone for my FRCNN as well as the custom classifier so now if I were to re-train my classifier using ResnetFPN that yields multiple feature maps how should I do that?</p>
<p>Thank You</p>
<p>ps : the custom classifier is an OCC-one class classifier</p>
Loss is None during training

username: igoneal
cooked: <p>class SyNet(sy.Module):<br>
def <strong>init</strong>(self, torch_ref):<br>
super(SyNet, self).<strong>init</strong>(torch_ref=torch_ref)<br>
self.layer1 = self.torch_ref.nn.Linear(in_dim, 20)<br>
self.layer2 = self.torch_ref.nn.Linear(20, 30)<br>
self.out = self.torch_ref.nn.Linear(30, out_dim)</p>
<pre><code>def forward(self, X):
    X = self.torch_ref.nn.functional.relu(self.layer1(X))
    X = self.torch_ref.nn.functional.relu(self.layer2(X))
    output = self.torch_ref.nn.functional.relu(self.out(X), dim=1)
    return output
</code></pre>
<p>local_model = SyNet(torch)<br>
remote_model = local_model.send(duet)<br>
remote_torch =duet.torch<br>
params = remote_model.parameters()<br>
optim = remote_torch.optim.Adam(params=params, lr=0.00000001)<br>
#%%pixie_debugger<br>
start_time = time.time()<br>
<span class="hashtag">#import</span> pdb; pdb.set_trace()<br>
<span class="hashtag">#from</span> IPython.core.debugger import Tracer; Tracer()()<br>
def train(iterations, model, torch_ref, optim, data_ptr, target_ptr):</p>
<pre><code>losses = []

for i in range(iterations):

    optim.zero_grad()

    output = model(data_ptr)

    loss = torch_ref.nn.functional.mse_loss(output, target_ptr.long())

    loss_item = loss.item()

    loss_value = loss_item.get(
        reason="To evaluate training progress", request_block=True, timeout_secs=5
    )

    if i % 10 == 0:
        print("Epoch", i, "loss", loss_value)

    losses.append(loss_value)

    loss.backward()

    optim.step()

return losses
</code></pre>
<p>print(“Computation Time:- %s seconds -” % (time.time() - start_time))<br>
iteration = 100<br>
losses = train(iteration, remote_model, remote_torch, optim, data_ptr, target_ptr)<br>
I am getting this from the training<br>
poch 0 loss None<br>
Epoch 10 loss None<br>
Epoch 20 loss None<br>
Epoch 30 loss None<br>
Epoch 40 loss None<br>
Epoch 50 loss None<br>
Epoch 60 loss None<br>
Epoch 70 loss None<br>
Epoch 80 loss None<br>
Epoch 90 loss None</p>
Convert eras to pytorch

username: Pramod_Singh
cooked: <p>I am trying to implement machine translation using pytorch but without torch text i.e. without bucketiterator etc.<br>
from keras.layers import Input, LSTM, Embedding, Dense<br>
from keras.models import Model</p>
<p>encoder_inputs = Input(shape=(None,))<br>
enc_emb =  Embedding(num_encoder_tokens, latent_dim, mask_zero = True)(encoder_inputs)<br>
encoder_lstm = LSTM(latent_dim, return_state=True)<br>
encoder_outputs, state_h, state_c = encoder_lstm(enc_emb)</p>
<h1>
<a name="we-discard-encoder_outputs-and-only-keep-the-states" class="anchor" href="#we-discard-encoder_outputs-and-only-keep-the-states"></a>We discard <code>encoder_outputs</code> and only keep the states.</h1>
<p>encoder_states = [state_h, state_c]</p>
<p>I am trying to code in pytorch<br>
encoder_input = torch.randn(1,None,)<br>
enc_emb = nn.Embedding(num_encoder_tokens, latent_dim)<br>
but I am not able to convert it<br>
keras code is from <a href="https://www.kaggle.com/aiswaryaramachandran/english-to-hindi-neural-machine-translation" class="inline-onebox" rel="noopener nofollow ugc">English to Hindi Neural Machine Translation | Kaggle</a><br>
Any one help me how to implement it in pytorch</p>
How to store nn.Parameter as class attribute?

username: Sam_Lerman
cooked: <p>Normally, <code>setattr</code> allows you to set a class  attribute. However, when I’m in a class that extends <code>Module</code> and run:</p>
<p><code>setattr(self, "attribute_name", Parameter(torch.empty([5])))</code></p>
<p>the class attribute <code>attribute_name</code> does not get created. I can print the class attributes with <code>print("attribute_name" in self.__dict__)</code> but it’s not there.</p>
<p>Incidentally, <code>setattr(self, "attribute_name", torch.empty([5]))</code> does work.</p>

username: crcrpar
cooked: <p>Hi,</p>
<p>Did you try <a href="https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.register_parameter" rel="noopener nofollow ugc">register_parameter method — PyTorch 1.8.1 documentation</a>?</p>

username: Sam_Lerman
cooked: <p>Thank you, I’m interested in trying that, but I’m not sure I understand the purpose of the <code>register_parameter</code> method.</p>
Using a pre-trained network for input with more than 3 channels

username: Anupama_Rajkumar
cooked: <p>Hi,</p>
<p>I have a satellite dataset with images containing &gt; 3 channels. Since the data is limited, I am using pre-trained models. Also, I am not training the pre-trained model from scratch due to small dataset and freezing the gradient for initial layers. In such a case, is there a way to use/fuse the remaining channels of my image and use it with the pre-trained model?<br>
Any inputs would be appreciated.</p>
<p>Regards,<br>
Anupama</p>

username: ptrblck
cooked: <p>It would depend on the use case and what kind of information is stored in each channel.<br>
You could certainly try to reduce the channels to three, but would need to check, if you are losing valid information. Another approach would be to either replace the first layer with a new conv accepting more input channels, manipulating the kernels in the first layer and add new randomly initialized channels, or to add a complete new “0th” layer to transform the images to 3 channels (let’s call this a “trainable reduction”) using e.g. a <code>1x1</code> kernel.</p>

username: Anupama_Rajkumar
cooked: <p>Thanks for the input. I can try adding a new 0th layer or manipulating the first layer as you suggested. I would need the information from the other channels as they contain information from infra red band and help in understanding the landscape information.<br>
Since I am a PyTorch novice, could you point me to an example where this is done? It would help me a great deal!</p>
<p>Thanks in advance.</p>
<p>Regards</p>

username: ptrblck
cooked: <p>Something like this should work:</p>
<pre><code class="lang-python">model = models.resnet18()
conv0 = nn.Conv2d(4, 3, 1)

x = torch.randn(1, 4, 224, 224)
out = conv0(x)
out = model(out)
</code></pre>
<p>You could of course wrap it in a new custom module, which would be a bit cleaner.</p>
Torch::save throws std::bad_alloc error. Advice?

username: Ipsum
cooked: <p>I am a longtime python user who is now trying to save a tensor in c++. I am simply running the following code:</p>
<pre><code class="lang-auto">torch::Tensor a = torch::zeros({ 2,3 });
torch::save(a, "my_tensor.pt");
</code></pre>
<p>I am thrown the following error: “std::bad_alloc error”, followed by a hexadecimal memory address. This is the exact error message:</p>
<pre><code class="lang-auto">Unhandled exception at 0x00007FF8AC46D759 in CStack.exe: Microsoft C++ exception: std::bad_alloc at memory location 0x00000053318FF290.

</code></pre>
<p>I believe that I have configured my CMake appropriately because I can create tensors without error (as well as perform other functions on the tensors).</p>
<p>I am coding in visual studio, and I would appreciate any advice that you have to offer. Thank you!</p>

username: Ipsum
cooked: <p>Figured it out – Had to switch the compiler from debug mode to release mode within visual studio. I hope that this post helps anyone else experiencing the same problem!</p>
Does TorchScript / JIT support __iter__? Odd NotImplementedError

username: ProGamerGov
cooked: <p>So, I ran into the following error while using a model that have been converted to JIT via <code>torch.jit.script</code>:</p>
<pre><code class="lang-auto">     15     def __init__(self, module_list: Iterable[nn.Module]) -&gt; None:
---&gt; 16         self.module_dict = dict.fromkeys(module_list, None)

/usr/local/lib/python3.7/dist-packages/torch/jit/_script.py in __iter__(self)
    616 
    617         def __iter__(self):
--&gt; 618             return self.forward_magic_method("__iter__")
    619 
    620         def __getitem__(self, idx):

/usr/local/lib/python3.7/dist-packages/torch/jit/_script.py in forward_magic_method(self, method_name, *args, **kwargs)
    612                 RecursiveScriptModule, method_name
    613             ):
--&gt; 614                 raise NotImplementedError()
    615             return self_method(*args, **kwargs)
    616
</code></pre>
<p>I assume that the cause of this issue might be <code>__iter__</code>, but I’m not entirely.</p>
<p>It says on the PyTorch Github repo here that <code>__iter__</code> is not supported yet: <a href="https://github.com/pytorch/pytorch/blob/master/docs/source/jit_python_reference.rst" class="inline-onebox" rel="noopener nofollow ugc">pytorch/jit_python_reference.rst at master · pytorch/pytorch · GitHub</a>, but this file hasn’t been updated in over a year.</p>

username: ProGamerGov
cooked: <p>I figured out the issue! It seems that any module run through <code>torch.jit.script</code> gains the <code>__iter__</code> attribute that one would expect in lists, tuples, tensors, etc…</p>
<pre><code class="lang-auto">print(hasattr(model.layer, "__iter__")) # False

jit_model = torch.jit.script(model)

print(hasattr(jit_model.layer, "__iter__")) # True
</code></pre>

username: ProGamerGov
cooked: <p>Though, how do I check if I have a list of JIT model layers or a single layer?</p>
Broadcasting issue in Custom DoubleBackward

username: AlphaBetaGamma96
cooked: <p>Hi All,</p>
<p>I’m having a weird broadcasting issue when calculating my gradients with a Custom Autograd Function. I’ve written a custom version of the <code>torch.slogdet</code> with both a Custom Backward and Custom DoubleBackward extension. However, I’m having an issue with actually using the backward due to this error,</p>
<pre><code class="lang-auto">RuntimeError: The size of tensor a (4) must match the size of tensor b (6) at non-singleton dimension 3
</code></pre>
<p>I can see where it is, but I’m not sure how to resolve it. The code of the custom <code>torch.slogdet</code> is here,</p>
<pre><code class="lang-auto">class CustomSLogDeterminant(Function):
  
  @staticmethod
  def forward(ctx, A):
    ctx.save_for_backward(A)
    return torch.slogdet(A)    
    
  @staticmethod
  def backward(ctx, sgnbar, logabsdetbar):
    A, = ctx.saved_tensors
    return CustomSLogDeterminantBackward.apply(A, sgnbar, logabsdetbar) 
    
class CustomSLogDeterminantBackward(Function):

  @staticmethod
  def forward(ctx, A, sgnbar, logabsdetbar):
    #A shape [4096, 4, 6, 6]
    #sgnbar Shape [4096, 4]
    #logabsdetbar Shape [4096, 4]
    ctx.save_for_backward(A, sgnbar, logabsdetbar) 
    return logabsdetbar * torch.transpose(torch.linalg.inv(A), -2, -1)
    
  @staticmethod
  def backward(ctx, InverseTransposeAbar):
    A, sgnbar, logabsdetbar = ctx.saved_tensors
    
    InverseAbar = torch.transpose(InverseTransposeAbar, -2, -1)
    TransposeInverseA = torch.transpose(torch.linalg.inv(A), -2, -1)
        
    second_derivative = -(TransposeInverseA @ InverseAbar @ TransposeInverseA)
    return (logabsdetbar * second_derivative), None, InverseAbar @ TransposeInverseA
</code></pre>
<p>Now the error above clearly comes from <code>return logabsdetbar * torch.transpose(torch.linalg.inv(A), -2, -1)</code> as <code>logabsdetbar</code> has shape <code>[4096, 4]</code> and the other term has <code>[4096,4,6,6]</code>. Now I would need to broadcast this to each element within <code>dim</code> 2 and 3 for the second <code>Tensor</code> term. The 2nd <code>Tensor</code> of size <code>[4096,4,6,6]</code> represents a batch of <code>4096</code> inputs of <code>4</code> matrices which are of size <code>[6, 6]</code> each. I did modifying the line to <code>unsqueeze dim</code> 2 and 3 although that just pushed the error further along. It was modified as follows,</p>
<pre><code class="lang-auto">    return logabsdetbar.unsqueeze(2).unsqueeze(3) * torch.transpose(torch.linalg.inv(A), -2, -1)
</code></pre>
<p>But then that just returns a simply error of,</p>
<pre><code class="lang-auto">RuntimeError: Function CustomSLogDeterminantBackwardBackward returned an invalid gradient at index 2 - got [4096, 4, 6, 6] but expected shape compatible with [4096, 4]
</code></pre>
<p>How exactly can I solve this?</p>
<p>Thank you in advance!</p>
Batch normalisatio

username: Sethu
cooked: <p>I am a beginner to pytorch. I wanted to use BatchNorm1d in my code. wondering what would be the batch size by default as I could not find any user specified input for batch size.</p>

username: ptrblck
cooked: <p>PyTorch modules do not depend on the batch size, so you could pick this value.<br>
However, as is shown in a lot of posts in this forum, a too small batch size might yield skewed running stats and your model might perform poorly on the validation and test set. The <a href="https://arxiv.org/abs/1803.08494">GroupNorm paper</a> explains this in more detail and proposes the <code>nn.GroupNorm</code> layer.</p>

username: Sethu
cooked: <p>Thanks for your reply. But I am still not able to figure out where to specify batch size.<br>
In this ,         torch.nn.BatchNorm1d (num_features,eps=1e-05, momentum=0.1,affine=True,track_running_stats=True)     where to specify batch size as there is no provision for specifying that</p>

username: patrickwilliams3
cooked: <p>Batch Normalization does not need batch size. The learnable of batch normalization are of shape 2xC where C is the channel dimension of a 3D input (NxCxL) and the 2 is for the mean and variance at each channel. If the input is 2D then the learnable parameters of batch norm will be of shape (2x1).</p>
<p>Batch Norm will compute the mean and variance across N where N is the batch size so it is an unimportant parameter for batch normalization.</p>
Determining if a dataset is iterable-style or map-style

username: trooper
cooked: <p>Hey there<br>
1.I want to know ,is there any way to understand that our dataset is iterable-style or map-style?</p>
<p>2.Secondly, when we used "torch.utils.data.DataLoader " for a “train_set” is there a way to show us the contents of "torch.utils.data.DataLoader " is iterable or map styled?</p>
<p>3.I think that if we give the ARG ‘sampler’ in  "torch.utils.data.DataLoader " ,’=Sampler’ (i.e. sampler=‘Sampler’), we define that our dataset is map-style, and if we set it as ‘Iterable’, we define that our dataset is iterable, AM I RIGHT?</p>
<p>4.I did define a map-style dataset by setting   sampler=‘Sampler’ arg in  "t=torch.utils.data.DataLoader " . but when I want to call ‘t’ by indexs(i.e. t[i] (like t[0])) it raise the error “‘DataLoader’ object is not subscriptable”. I also used ‘t=torch.utils.data.Sampler’, but it also raised that error too. PLS help me.</p>
<p>I’m so confused by them. I even can’t see the size of my dataloader properly, BCS I don’t like ‘len’ func as it just show the biggest length not the matrix exact size.</p>
<p>PLEASE HELP ME. THANKS A LOT</p>
Center crop in PyTorch?

username: Zhengtian
cooked: <p>Hi, all</p>
<p>I want to crop the image after using <code>nn.ConvTranspose2d</code>, but I can not find some relative function…</p>

username: chenyuntc
cooked: <p>seems that there aren’t such methods, try something like this:</p>
<pre><code class="lang-auto">def crop(variable,tw,th):
        w, h = variable.size
        x1 = int(round((w - tw) / 2.))
        y1 = int(round((h - th) / 2.))
        return variable[x1:x1+tw,y1:y1+th]
</code></pre>

username: smth
cooked: <p>you could also use F.pad with negative number for cropping, but you will have to figure out the amount of padding manually: <a href="http://pytorch.org/docs/nn.html#torch.nn.functional.pad">http://pytorch.org/docs/nn.html#torch.nn.functional.pad</a></p>

username: pkmandke
cooked: <p>Hi,</p>
<p>Has center cropping (and/or negative padding) been added to torch.nn API in the recent versions?</p>
<p>I am having a similar use case as the OP and would prefer to use the nn API rather F.pad.</p>
<p>I found some old threads (<a href="https://github.com/pytorch/pytorch/issues/4939" rel="nofollow noopener">here</a> and <a href="https://github.com/pytorch/pytorch/issues/1331" rel="nofollow noopener">here</a>) about adding these features but couldn’t find anything readily in the docs. Any pointers would be helpful.</p>
<p>Thank you!</p>

username: pkmandke
cooked: <aside class="quote no-group" data-username="pkmandke" data-post="4" data-topic="3214">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pkmandke/48/16436_2.png" class="avatar"> pkmandke:</div>
<blockquote>
<p>Hi,</p>
<p>Has center cropping (and/or negative padding) been added to torch.nn API in the recent versions?</p>
<p>I am having a similar use case as the OP and would prefer to use the nn API rather F.pad.</p>
<p>I found some old threads (<a href="https://github.com/pytorch/pytorch/issues/4939" rel="noopener nofollow ugc">here</a> and <a href="https://github.com/pytorch/pytorch/issues/1331" rel="noopener nofollow ugc">here</a>) about ad</p>
</blockquote>
</aside>
<p><a class="mention" href="/u/smth">@smth</a> Any heads up on this?<br>
TIA.</p>

username: faizan_shaikh
cooked: <p>Maybe (<a href="https://pytorch.org/docs/stable/_modules/torchvision/transforms/transforms.html#CenterCrop" rel="nofollow noopener">transforms.CenterCrop</a>) func should suffice for your usecase.</p>

username: jayce
cooked: <p><a href="https://pytorch.org/vision/stable/transforms.html#torchvision.transforms.functional.center_crop" rel="noopener nofollow ugc">center_crop</a><br>
this method is helpful</p>
Problem importing Torch

username: Haider_Ali_Shuvo
cooked: <p>While executing import torch statement i get the following error</p>
<p>OSError                                   Traceback (most recent call last)<br>
 in <br>
----&gt; 1 import torch<br>
2 <span class="hashtag">#assert</span> ‘.’.join(torch.<strong>version</strong>.split(’.’)[:2]) == ‘1.4’<br>
3 print(torch.<strong>version</strong>)<br>
4 import torch.nn as nn<br>
5 import torch.optim as optim</p>
<p>~.conda\envs\DeepLearning\lib\site-packages\torch_<em>init</em>_.py in <br>
79     dlls = glob.glob(os.path.join(th_dll_path, ‘*.dll’))<br>
80     for dll in dlls:<br>
—&gt; 81         ctypes.CDLL(dll)<br>
82<br>
83</p>
<p>~.conda\envs\DeepLearning\lib\ctypes_<em>init</em>_.py in <strong>init</strong>(self, name, mode, handle, use_errno, use_last_error)<br>
362<br>
363         if handle is None:<br>
–&gt; 364             self._handle = _dlopen(self._name, mode)<br>
365         else:<br>
366             self._handle = handle</p>
<p>OSError: [WinError 127] The specified procedure could not be found</p>
<p>Is there any solution to this ?</p>
Mini Batches respecting class proportion/frequency

username: Fra
cooked: <p>Hi All,</p>
<p>I would like to know if there is an efficient way to create mini-batches which respect class proportion ( that is, if my training is formed by 95% samples in class 1 and 5% class2, I would like to have mini-batches respecting these frequencies).</p>
<p>Thank you!</p>

username: Fra
cooked: <p>I have currently found the answer here:</p>
<aside class="quote" data-post="1" data-topic="911">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/gaurav_pandey/48/6139_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-to-enable-the-dataloader-to-sample-from-each-class-with-equal-probability/911">How to enable the dataloader to sample from each class with equal probability</a> 
  </div>
  <blockquote>
    The dataloader utility in torch (courtesy of Soumith Chintala) allowed one to sample from each class with equal probability. I was wondering, if there is a straightforward approach to enable the same in pytorch dataloaders.
  </blockquote>
</aside>