akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Tensor shifts in torch.roll

meta: <p>Hi all!</p>
<p>I have multiple images (or feature maps). I want roll the images using torch.roll.</p>
<pre><code class="lang-auto"> torch.roll(images, shifts, dims=(2,3))
</code></pre>
<p>shifts must be int or tuple of ints.<br>
But I have shifts tensor with different amount for each image.<br>
Is there a way to do roll each image with different indices in vectorized manner?</p>
<p>Thank you!</p>

KFrank: <p>Hi Meta!</p>
<aside class="quote no-group" data-username="meta" data-post="1" data-topic="170655" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/f17d59/48.png" class="avatar"> meta:</div>
<blockquote>
<p>Is there a way to do roll each image with different indices in vectorized manner?</p>
</blockquote>
</aside>
<p>Yes, you can create “identity” indices for each of the four dimensions of<br>
your <code>images</code> tensor.  Then “roll” the identity indices for your height and<br>
width dimensions (your <code>dims = (2, 3)</code>) on a batch basis and create your<br>
rolled <code>images</code> by using pytorch tensor indexing to index into <code>images</code> with<br>
your “rolled” indices.</p>
<p>Here is an illustration:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch

nBatch = 5
nChannels = 3
h = 24
w = 32

images = torch.randn (nBatch, nChannels, h, w)
shifts = torch.stack ((torch.randint (h, (nBatch,)), torch.randint (w, (nBatch,))), -1)

ind0 = torch.arange (nBatch)[:, None, None, None].expand (nBatch, nChannels, h, w)
ind1 = torch.arange (nChannels)[None, :, None, None].expand (nBatch, nChannels, h, w)
ind2 = torch.arange (h)[None, None, :, None].expand (nBatch, nChannels, h, w)
ind3 = torch.arange (w)[None, None, None, :].expand (nBatch, nChannels, h, w)

rolled_images = images[ind0, ind1, (ind2 + shifts[:, 0, None, None, None]) % h, (ind3 + shifts[:, 1, None, None, None]) % w]
</code></pre>
<p>Best.</p>
<p>K. Frank</p>
Efficiently Get Gradient Of All Neurons wrt Input

Max_Torop: <p>I’m looking to get the gradient of every single neuron in a network f wrt the input x. I was wondering if there is an efficient way to do this? One naive approach would be to set x.requires_grad = True and then separately backprop the output of each neuron in a for loop, recording the values of x.grad. However, with such an approach a lot of computation would be repeated. I feel like there should be a much more efficient way to do this.</p>
<p>Any help would be greatly appreciated !</p>
GPU memory usage increases even when overwriting variables

Mole_m7b5: <p>I’m currently training a ResUnet with 3 encoding blocks, 1 bottleneck, 4 decoding blocks, and an output layer and I’m using an RTX 3090. I seem to be having trouble fitting the model, training, and validation within the available 24GB memory.</p>
<p>The model size itself isn’t that large, and I have a batch_input.shape() of (4,3,604,513) and a batch_target.shape() which is also (4,3,604,513). These also don’t take up that much space in memory.</p>
<p>Within my forward pass it seems the memory usage for the GPU seems to continue to increase even though I’m overwriting the x variable used for input and output of each layer - shown below</p>
<pre><code class="lang-auto">    def forward(self, x):
        skip_connections = []
        s = x.clone()
        """ initial conv block"""
        x = self.conv1(x) 
        x = self.bn1(x)  
        x = self.conv2(x) 
        s = self.conv_skip(s)  
        x = x + s
        skip_connections.append(x)
        """ Encoding blocks"""
        for encode in self.encodes:
            x = encode(x)
            skip_connections.append(x)
        """Bottle neck"""
        x = self.bottleneck(x)  
        """Decoder"""
        skip_connections = skip_connections[::-1]  
        for idx, decode in enumerate(self.decodes):
            x = decode(x, skip_connections[idx])
        """output"""
        x = self.output(x)
        return x
</code></pre>
<p>I understand that certain “versions” of <code>x</code> will be larger than others, but then when I return to my original output size the memory usage remains large. Is there something to do with how PyTorch cache’s memory on the GPU that I’m missing?</p>

ptrblck: <p>During training Autograd will store intermediate forward activations since these are needed for the gradient computation. If you don’t want to train the model or compute gradients you could disable this behavior by using <code>with torch.no_grad()</code> which will delete the intermediates.</p>
Torchvision transforms for videos

braindotai: <p>How can I apply transformations like, Resize, CenterCrop, RandomCrop, RandomHorizontalFlip etc… to a read video, of type torch.tensor with four dimensions -&gt; (channels, frames, height, width). (Its okay if I’d have to reshape this…)</p>

ptrblck: <p>You could create custom transformations, which would apply the <code>torchvision.transforms</code> in a loop on each sample (or rewrite the transformations so that they would work on batched inputs).<br>
<code>torchvision</code> has some internal <a href="https://github.com/ehnryx/vision/blob/be6f398c0612c245b0019a286a99f80aca81de7d/torchvision/transforms/_transforms_video.py">video transforms</a>. Since the API isn’t finalized, this code might break and shouldn’t be used, if you rely on backwards compatibility.</p>

lab176344: <p>Don’t know if you are still looking for a solution, might be this will help <a href="https://github.com/facebookresearch/pytorchvideo/" class="inline-onebox" rel="noopener nofollow ugc">GitHub - facebookresearch/pytorchvideo: A deep learning library for video understanding research.</a></p>

braindotai: <p>Yup I’ve also recently discovered this… and its awesome <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
Thanks for your time.</p>

Sara176: <p>hi<br>
i am a beginner , my inputs are videos processed and stored in tensors of shape<br>
[batch size, number of channels , number of frames , height , width]<br>
i need to apply transformations to use pretrained resnet , is there anything else i can try ?</p>

ptrblck: <p>What did you already try and what kind of issues were you seeing?</p>
Meaning of multiprocessing_context in dataloader.py

Vinayaka_Hegde: <p>Hi, what is the meaning of `multiprocessing_context’ here  ? What other values can it take?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/d/1d11398c4d0c5a9bedb3a84571b21dc041c033aa.png" data-download-href="https://discuss.pytorch.org/uploads/default/1d11398c4d0c5a9bedb3a84571b21dc041c033aa" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/d/1d11398c4d0c5a9bedb3a84571b21dc041c033aa_2_690x106.png" alt="image" data-base62-sha1="498Kxj7Xparr9Ac4bwBYePo2TpM" width="690" height="106" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/d/1d11398c4d0c5a9bedb3a84571b21dc041c033aa_2_690x106.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/d/1d11398c4d0c5a9bedb3a84571b21dc041c033aa_2_1035x159.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/1/d/1d11398c4d0c5a9bedb3a84571b21dc041c033aa.png 2x" data-dominant-color="F7F7F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1356×209 20.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
1DCNN variable length batch input

Nick95: <p>Hi,<br>
I have a dataset composed of sequences of different length.<br>
For training an LSTM network I have configured the dataloader to reshape each batch with shape: [batch, sequence, channels], where number of channels is 1, so [batch, sequence, 1].<br>
Then I have used the pad_sequence and pad_packed_sequence to prepare the batch input to the network. Until here everything it is fine.<br>
Now I have tried to fed the same dataset to a 1DCNN with 1 channel as input. I have changed the batch shape as [batch, 1, sequence] to fit the requirements of the 1DCNN as input, however when I try to pack the sequence it gives me this error:<br>
“RuntimeError: The size of tensor a (86) must match the size of tensor b (95) at non-singleton dimension 1”<br>
How can I solve?<br>
Thanks</p>

J_Johnson: <p>It’s tough to determine without seeing your code. Can you make a short reproducible code sample?</p>

Nick95: <p>Thanks for the reply, I managed to found a viable solution. I post it here, so that maybe can be helpful for someone:</p>
<pre><code class="lang-auto">final_tensor = torch.zeros(self.output_size)
final_tensor[0, :len(original_tensor[0])] = original_tensor
</code></pre>
How to get the file name in dataloader

cat-loves-donuts: <p>Well, I create d a test data set which contains 13 different objects. After the training I want to use those 13 objects to test my model. I printed confusion matrix for each test data, so I need to get the name of each test data. However I used shuffle in dataloader, which called data_loader_test, when I read test data set. I used <strong>data_loader_test.dataset.training_files</strong> inside epoch loop to get the file name in each epoch. But it looks like the file name is the same order which all those test data sorted in the file. How can I get the file name order after shuffle?</p>

ptrblck: <p>You could create a custom <code>Dataset</code> and return the image name with the data and target samples.<br>
This would make sure that, even after shuffling, you would still get the corresponding file names for the current data and target batch.</p>
<p><a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html">This tutorial</a> gives you a good overview on writing a custom <code>Dataset</code>. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

cat-loves-donuts: <p>Oh, thank you so much. I will try this one. <img src="https://discuss.pytorch.org/images/emoji/apple/rofl.png?v=9" title=":rofl:" class="emoji" alt=":rofl:"></p>

zhangyu-python: <p>hellow, did you succeed? I also meet the problem now.</p>

cat-loves-donuts: <p>Yeah, it works, when I pre-process my .npz data, I save those data into a tuple and add the file name into it too. So when I pass them into the data loader, the file name is also in it.</p>

zhangyu-python: <p>Thank you so much for your reply，and it works in my dataset ,too</p>

aemonge: <p>For anyone who’s interested in the code directly, it would be in the <strong>CustomDataset</strong></p>
<pre><code class="lang-auto">    return image, [class_tensor, img_name] 
</code></pre>
<p>And <strong>loading</strong> it as follows:</p>
<pre><code class="lang-auto">for (images, labels) in iter(loaders)
    labels[0] # is the class_tensor
    labels[1] # is the img_name
</code></pre>