akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Exception when adding tensors

username: Isabela_Marinho
cooked: <p>I’m converting a python code to C++ and I have to add 2 tensors, one of them (z_vals) shaped [64] and another one (rand_mult) shaped [100, 100, 64]. In python the broadcasting is made automatically, but in C++ I did it manually, converting the first tensor to [100, 100, 64]:</p>
<p><code>torch::Tensor z_vals_broad = torch::broadcast_to(z_vals, {100, 100, 64});</code></p>
<p>This way, I should be able to add them up, but every time I try, an exception is called, without much information about what the problem is. It only says <strong>Error at memory location</strong>.</p>
<p>I’ve tried:</p>
<p><code>z_vals = z_vals_broad.add(rand_mult);</code></p>
<p>and</p>
<p><code>torch::Tensor z_vals_b = torch::add(z_vals_broad, rand_mult);</code></p>
<p>but none of it works. Am I missing something in the broadcasting part? Or is the problem elsewhere?</p>

username: ptrblck
cooked: <p>I cannot reproduce the issue and am able to directly add these tensors:</p>
<pre><code class="lang-auto">#include &lt;torch/script.h&gt; // One-stop header.

#include &lt;iostream&gt;
#include &lt;memory&gt;

int main(int argc, const char* argv[]) {
  auto a = torch::randn({64});
  auto b = torch::randn({100, 100, 64});

  auto c = a + b;
  std::cout &lt;&lt; c.sizes() &lt;&lt; std::endl;

  return 0;
}
</code></pre>
<p>Output:</p>
<pre><code class="lang-auto">./example-app 
[100, 100, 64]
</code></pre>

username: Isabela_Marinho
cooked: <p>I managed to solve it here, thanks! The problem was one of the tensors was allocated at CUDA.</p>
How my did my training loss go from .19 to .03 just by changing my number of works from zero to 4?

username: StrafeNDestroy
cooked: <p>Hello I was training my fasterrcnn model a couldn’t get past a training loss of .19. At this time my number of works for my data loader was zero. I decided to try 4 for my number of works and suddenly my training loss got down to .03! How did this happen? I thought a number of workers was only for contributing to the speed of the model.</p>

username: ptrblck
cooked: <p>Yes, your assumption is correct that the number of workers should not change any hyperparameters etc. However, it would be interesting to check if you are accidentally reusing the same random transformations e.g. by not seeding the code properly.<br>
See <a href="https://pytorch.org/docs/stable/data.html#randomness-in-multi-process-data-loading">this section</a> of the docs.</p>

username: StrafeNDestroy
cooked: <p>For transformations are you talking about image augmentation? I don’t have anything for transforms other than resizing the image from 416 to 1024. Are there any other links about this seeding and why it is important?</p>

username: ptrblck
cooked: <p>If no random transformations are used, the seeds shouldn’t matter as the samples should be transformed to the same outputs.<br>
Since no randomness should be introduced, you could compare both <code>DataLoader</code>s and see where the samples would differ.<br>
Iterate both loaders, store the samples in corresponding <code>list</code>s, and compare the samples for differences afterwards. You might need to disable shuffling (or store the index and sort it afterwards) and depending on the size of the dataset you might want to check subset(s).</p>
Pytorch Tensorboard logging requires TensorBoard version 1.15 Import Error

username: justkittenaround
cooked: <p>So I have a similar problem to other people on this logging version error, except the previously posted solutions are not working for me. I just fresh installed pytorch using official site instructions then <code>pip3 install tensorboard==1.15.0</code>. I do not have tensorflow or tf-gpu installed.</p>
<pre><code class="lang-auto">python            3.6
pytorch           1.60+cu101
torchvision       0.7.0+cu101
tensorboard       1.15.0
cuda              10.1
nvidia driver     450.51.06
ubuntu            18.04

</code></pre>
<p>Having both tensorboard and pytorch throws the following error:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; from torch.utils import tensorboard
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "/home/usrname/.local/lib/python3.6/site-packages/torch/utils/tensorboard/__init__.py", line 4, in &lt;module&gt;
    raise ImportError('TensorBoard logging requires TensorBoard version 1.15 or above')
ImportError: TensorBoard logging requires TensorBoard version 1.15 or above
&gt;&gt;&gt; 
</code></pre>
<p>If I uninstall tensorboard with<code> python3 -m pip uninstall tensorboard</code> and pytorch with <code>python3 -m pip uninstall torch</code> (x2) and then reinstall pytorch, I get the same error.</p>

username: knslee07
cooked: <p>Hi, I am having the same issue. Have you found a way to solve it by any chance?</p>

username: ptrblck
cooked: <p>Double post from <a href="https://discuss.pytorch.org/t/importerror-tensorboard-logging-requires-tensorboard-version-1-15-or-above/156743">here</a> which cannot be reproduced in the latest stable release.</p>
Pip installer doesn't seem to work for A6000 GPU on Linux

username: Sam_Lerman
cooked: <p>I get this error and then it bugs:</p>
<pre><code class="lang-auto">`UserWarning: `
`NVIDIA RTX A6000 with CUDA capability sm_86 is not compatible with the current PyTorch installation.`
`The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_70.`
</code></pre>

username: ptrblck
cooked: <p>You are most likely installing the PyTorch binaries with the CUDA 10.2 runtime, while your Ampere GPU needs CUDA 11. Select the CUDA 11.3 or 11.6 runtime in the install box, use the created install command, and it should work.</p>
Terminate called after throwing an instance of 'c10::Error' what(): poolINTERNAL ASSERT FAILED

username: MINGGUI
cooked: <p>this case happened by accident when I run model train with Pycharm IDE</p>
<p>but it must appear when I run on Linux terminal with python xxx.py</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/f/7ffa7f9da9310f2acfc7b9548a679a6e7b15f92f.png" data-download-href="https://discuss.pytorch.org/uploads/default/7ffa7f9da9310f2acfc7b9548a679a6e7b15f92f" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/f/7ffa7f9da9310f2acfc7b9548a679a6e7b15f92f.png" alt="image" data-base62-sha1="ig9kpR41rQvUC5DjQen6YUj0CTl" width="690" height="108" data-dominant-color="302F2F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1587×249 22.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>is it a version inconsistency or something else?</p>

username: ptrblck
cooked: <p>It seems <code>OpenMP</code> fails with this error. If you are seeing the same issue in the latest PyTorch release, could you create a GitHub issue so that we can track and debug it, please?</p>

username: MINGGUI
cooked: <p>hi, ptrblck!<br>
my torch version is ‘1.12.0’<br>
Additionally,  I adopt python’s Multiprocess class to implement the parallel model training over different gpu device.<br>
I don’t know whether there exist conflict between torch’s OpenMP and the Multiprocess</p>
Make custom datasets with some arbitrary indexes

username: maryma
cooked: <p>Hi. I wanted to train model with different datasets based on a whole dataset, it is like k fold. My data is made based on custom datasets. Every time I have a list that contains the indexes that I want from my custom datasets. for example mylist is [0,1,2,3] that I need these indexes from whole dataset. My whole dataset has 57 images but I just need 0th,1th,2th and 3th image. This is my custom dataset. I can not understand how can I make a connection between index and mylist. If someone had the same issue please help.</p>
<pre><code class="lang-auto">class DatasetBatch(Dataset):  
    def __init__(self, image_dataset, time, event, mylist):
        self.image_dataset = image_dataset
        self.time, self.event = tt.tuplefy(time, event).to_tensor()
        
        
    def __len__(self):
        return len(self.mylist)

    def __getitem__(self, index):
        if not hasattr(index, '__iter__'):
            index = mylist
   
        img = [self.image_dataset[i]['image']  for i in mylist]
        img = torch.stack(img)
        
        return tt.tuplefy(img, (self.time[mylist], self.event[mylist]))
</code></pre>
<p>mylist = [0,1,2,3]<br>
dataset1 = DatasetBatch (our_ds, *our_target, mylist)</p>

username: ptrblck
cooked: <p>I’m not sure I understand the use case correctly, but it sounds as if you are looking for the <code>torch.utils.data.Subset</code> class, which would allow you to pass indices to it and only return samples from these indices.</p>
Different speed of models when loading from snapshot and when randomly initializing weights

username: nasretdinovr
cooked: <p>I have 2 models with the same architecture. The first model is with trained weights and the second is with all weights (including biases) from normal distribution. And the second one is 50% faster than the first. I measured time on CPU.<br>
I even used torch.profiler and realized that individual operations like conv2d are faster in the 2nd case. Is it normal behavior for a model?</p>

username: ptrblck
cooked: <p>Could you check if changing <code>torch.set_flush_denormal</code> would speed up the slower model?</p>

username: nasretdinovr
cooked: <p>I checked, it haven’t changed anything.</p>

username: ptrblck
cooked: <p>Unfortunately, I wouldn’t know what else might cause a performance difference on the CPU for exactly the same models using different parameter values.</p>
Distributed Data Parallel Hangs

username: Hussein
cooked: <p>Hello<br>
I hope you all are doing well.<br>
Could you please help me to understand the following strange behavior of Pytorch DDP.</p>
<p>I have two models that I train using one optimizer. When I run the training on one machine<br>
that has two Gpus there are no problems what so ever. It runs using both nccl and gloo.</p>
<p>On the other hand, It hangs when I use one machine with four Gpus using nccl. The training hangs when I warp each of the models with DDP. as the following code.</p>
<pre><code class="lang-auto">model = nn.parallel.DistributedDataParallel(model,
                                                device_ids=[gpu],broadcast_buffers=False, find_unused_parameters=True)
</code></pre>
<p>When I switch to gloo, it runs but its extremely slow. The Gpus configurations as follows.<br>
I tried to use sub set of two Gpus out of the four but it hangs as well at different line of the code.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/2/224abc2faf1872cc81af8600dde67ad280fc72ce.png" data-download-href="https://discuss.pytorch.org/uploads/default/224abc2faf1872cc81af8600dde67ad280fc72ce" title="Screen Shot 2022-07-15 at 9.42.34 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/2/224abc2faf1872cc81af8600dde67ad280fc72ce_2_690x217.png" alt="Screen Shot 2022-07-15 at 9.42.34 PM" data-base62-sha1="4TmlTKOB6uWmOF3jIJNHQi9v5sq" width="690" height="217" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/2/224abc2faf1872cc81af8600dde67ad280fc72ce_2_690x217.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/2/224abc2faf1872cc81af8600dde67ad280fc72ce_2_1035x325.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/2/2/224abc2faf1872cc81af8600dde67ad280fc72ce_2_1380x434.png 2x" data-dominant-color="F1F1F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2022-07-15 at 9.42.34 PM</span><span class="informations">1532×482 74.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Thank you very much</p>
Multiple images for one label

username: samuel_fipps
cooked: <p>Hello I have a dataset that is binary, but each example within the dataset has multiple images. Another way of explaining it is, 1 image but sliced up into multiple images(to get rid of white space). How do I go about putting this into one tensor? Would i just combined them all using numpy? Or are you supposed to use a different dimension, where all 60 images combined would equal 1 label.</p>

username: samuel_fipps
cooked: <p>To explain this better you would have image 1 that is label 0. This image is split into 60 slices to get rid of white space. How do I make all 60 slices equal the same label, because all 60 of the slices might not contain what i’m looking for.</p>

username: ptrblck
cooked: <p>I don’t completely understand the use case and what “white spaces” refers to in your example.<br>
How did you split the image into 60 slices and what does each slice represent?</p>

username: samuel_fipps
cooked: <p>This is the dataset and this is how they are sliced. <a href="https://www.kaggle.com/code/iuryck/quick-blood-clot-slicing" class="inline-onebox" rel="noopener nofollow ugc">Quick Blood Clot Slicing | Kaggle</a></p>
Can we wrap existing Tensor Functions by a Module Class

username: viash
cooked: <p>Hi,</p>
<p>Currently when we define a model we have the flexibility of using a torch.flatten or torch.nn.Flatten.</p>
<p>A user may choose to use either of them. I agree that torch.nn.Flatten is a module class and uses torch.flatten inside the forward call.</p>
<p>The advantage torch.nn.Flatten offers is that it has capability of forward hooks.</p>
<p>What I would like to do is either register a forward hook to the Tensor or wrap the tensor in nn.Module Class so I can register a hook to it.</p>
<p>tensor.flatten() is just an example. Ideally I would want to wrap all the <strong>torch_function</strong> to the nn.Module .</p>
<p>This should not change the way user defines the model. Any suggestions would be helpful.</p>

username: jerryzh168
cooked: <p>does feel like this is related to quantization, can you add a different tag?</p>

username: viash
cooked: <p>I might have to repost this as it is not letting me edit the post. The only logical label seems to be uncategorized.</p>
Mask R-CNN: bbox and segmentation good, but classification very bad

username: Joysn
cooked: <p>I trained a model from scratch, no pretrained has been used. The dataset contains 2 classes, each of them has 10k grayscale images, but very small ones. 24x40. I embedded those images into bigger images, which have been randomly created, as well as embedded them into some other bigger grayscale images (512x640) (plus augmentation with affine transformations).<br>
When I test the model with those images, too embedded randomly into other images, the bbox and segmentation is very good. But classification is around 0.5 for both classes, with a tendency towards the correct class.<br>
How can i improve the classification accuracy?</p>
<pre><code class="lang-auto">optimizer = torch.optim.SGD(params, lr=0.005, momentum=0.9, weight_decay=0.0005)
lr_scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=3, gamma=0.1)
</code></pre>
<p>Or is the object size too small and the model architecture needs to be adapted?</p>
<p>At the moment I only adapted the anchor size to</p>
<pre><code class="lang-auto">anchor_sizes = ((8,), (16,), (32,), (64,), (128,), )
</code></pre>
<p>Even when I run this 10 epochs up to 10 times, effectifely running 100 epochs, the situation does not change.</p>
Extract features before classification

username: mathwseg
cooked: <p>I need to extract the features before the last layer or before classifying it from this model</p>
<p>can I do this?</p>
<pre><code class="lang-auto">PATH = 'focal-base-is224-ws7.pth'
model = FocalTransformer(pretrained=True)
model.load_state_dict(torch.load(PATH),strict=False)
model.head = nn.Identity()
</code></pre>

username: mxahan
cooked: <p>It should work. Did you run into any errors afterward?</p>

username: ParGG
cooked: <p>As <a class="mention" href="/u/mxahan">@mxahan</a> said, if <code>model.head</code> is your last layer (the one before which you want to get the get the activations), it should work.</p>
<p>An alternative solution is to register a forward hook to your layer of interest without modifying the rest of the network: <a href="https://pytorch.org/docs/stable/generated/torch.nn.modules.module.register_module_forward_hook.html" class="inline-onebox" rel="noopener nofollow ugc">torch.nn.modules.module.register_module_forward_hook — PyTorch 1.12 documentation</a>.</p>

username: mathwseg
cooked: <p>i edit the post with the architecture of the model. excuse me if I need the features before the classification layer, should I remove the head only and make <code>avgpool layer</code> be the last layer</p>

username: ParGG
cooked: <aside class="quote no-group" data-username="mathwseg" data-post="4" data-topic="156693">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mathwseg/48/39221_2.png" class="avatar"> mathwseg:</div>
<blockquote>
<p>excuse me if I need the features before the classification layer, should I remove the head only and make <code>avgpool layer</code> be the last layer</p>
</blockquote>
</aside>
<p>Yes, either that or register a forward hook in the avgpool layer.</p>
Checkpoint in DDP

username: shrutishrestha
cooked: <p>Do we have to save checkpoint after every epoch in DDP(DistributedDataParallel), or we can just store best checkpoint in a variable and save it after all the iterations are over?</p>

username: ptrblck
cooked: <p>I don’t think you <em>have</em> to store checkpoints during the training, but it might be a good way to store some intermediate checkpoints in case your training experiences any issues and crashes.</p>

username: shrutishrestha
cooked: <p>Thank you, will do that.</p>
Missing doc pages

username: Ashvant_Ram_Selvam
cooked: <p>There used to be a page under this link: <a href="https://pytorch.org/cppdocs/api/file_build_aten_src_ATen_Functions.h.html#file-build-aten-src-aten-functions-h" rel="noopener nofollow ugc">https://pytorch.org/cppdocs/api/file_build_aten_src_ATen_Functions.h.html#file-build-aten-src-aten-functions-h</a></p>
<p>It listed all the aten operators with named parameters. This seems to be unavailable now. Is there an alternative to this? Will this be available?</p>
Increase amount of anchors in FasterRCNN

username: vahvero
cooked: <p>One can alter the sizes of anchors in FasterRCNN by</p>
<pre><code class="lang-auto">anchor_sizes = (
      (
          (32,),
          (64,),
          (128,),
          (512,),
          (728,),
      ), # resnet fpn only accepts 5 inputs
)
aspect_ratios = ((0.5, 0.75, 1.0, 1.33, 2.0),) * len(anchor_sizes)
anchor_generator = AnchorGenerator(anchor_sizes, aspect_ratios)

backbone = resnet_fpn_backbone(
    backbone,
    pretrained=True,
)

model = FasterRCNN(
        backbone,
        num_classes=num_classes,
        rpn_anchor_generator=anchor_generator,
)
</code></pre>
<p>In my data with large fluctations of target sizes, this however still leads to inaccurate bounding boxes in large observations.</p>
<p>With mobilenet I can alter the amount of boxes trivially</p>
<pre><code class="lang-auto">anchor_sizes = (
    (
        32,
        64,
        128,
        512,
        728,
        882, # 6 box sizes not 5
    ),
)
aspect_ratios = ((0.5, 0.75, 1.0, 1.33, 2.0),) * len(anchor_sizes)
anchor_generator = AnchorGenerator(anchor_sizes, aspect_ratios)

backbone = mobilenet_backbone(
    "mobilenet_v3_large",
    pretrained=True,
    fpn=False,
)
model = FasterRCNN(
    backbone,
    num_classes=num_classes,
    rpn_anchor_generator=anchor_generator,
)
</code></pre>
<p>However, is it possible to use alter ResNet backbone to allow arbitary number of boxes?</p>
<p>For example, this does not work, but can backbone be altered to accept the anchor generator?</p>
<pre><code class="lang-auto">anchor_sizes = (
      (
          (32,),
          (64,),
          (128,),
          (512,),
          (728,),
          (882,),
          (1028,),
      ), # here we have seven inputs -&gt; model does not work
)
aspect_ratios = ((0.5, 0.75, 1.0, 1.33, 2.0),) * len(anchor_sizes)
anchor_generator = AnchorGenerator(anchor_sizes, aspect_ratios)

backbone = resnet_fpn_backbone(
    backbone,
    pretrained=True,
)

model = FasterRCNN(
        backbone,
        num_classes=num_classes,
        rpn_anchor_generator=anchor_generator,
)
</code></pre>
AttributeError: 'str' object has no attribute 'shape'

username: NagaYu
cooked: <pre><code class="lang-auto">model.eval()
with torch.no_grad():
    step = 10
    for i in range(0, len(test), step):
        test_data = test.tolist()[i: i+step]
        
        # 推論
        outputs = model(test_data)
        # ソフトマックス関数を使って確率を出力
        preds = F.softmax(outputs, dim=1)[:, 1].tolist()
        
        image_paths.append(img)
        predictions.append(preds[0])
    
res = pd.DataFrame({
    'ImageID': test_data,
    'predictions': predictions})

res.sort_values(by='test_data', inplace=True)
res.reset_index(drop=True, inplace=True)

# CSV出力
res.to_csv('test_data/sample_submission.csv', index=False)
</code></pre>
<p>AttributeError                            Traceback (most recent call last)<br>
Input In [15], in &lt;cell line: 2&gt;()<br>
5 test_data = test.tolist()[i: i+step]<br>
7 # 推論<br>
----&gt; 8 outputs = model(test_data)<br>
9 # ソフトマックス関数を使って確率を出力<br>
10 preds = F.softmax(outputs, dim=1)[:, 1].tolist()</p>
<p>File ~/opt/anaconda3/lib/python3.9/site-packages/torch/nn/modules/module.py:1110, in Module._call_impl(self, *input, **kwargs)<br>
1106 # If we don’t have any hooks, we want to skip the rest of the logic in<br>
1107 # this function, and just call forward.<br>
1108 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks<br>
1109         or _global_forward_hooks or _global_forward_pre_hooks):<br>
 → 1110     return forward_call(*input, **kwargs)<br>
1111 # Do not call functions when jit is used<br>
1112 full_backward_hooks, non_full_backward_hooks = [], []</p>
<p>File ~/opt/anaconda3/lib/python3.9/site-packages/torchvision/models/detection/ssd.py:323, in SSD.forward(self, images, targets)<br>
321 original_image_sizes: List[Tuple[int, int]] = []<br>
322 for img in images:<br>
 → 323     val = img.shape[-2:]<br>
324     assert len(val) == 2<br>
325     original_image_sizes.append((val[0], val[1]))</p>
<p>AttributeError: ‘str’ object has no attribute ‘shape’</p>
TypeError: expected Tensor as element 1 in argument 0, but got int

username: areeb_adnan
cooked: <p>I am facing an error on concatenating two different datasets my one dataset is from Fall_detection dataset and the other from the UP_FALL dataset. Now I looked on stackoverflow and got something like different datasets having different shapes but I don’t see any problem in combing both of them since both of the  datasets have their own CSV files with image names and Activity. Is there any way that this problem can be resolved or we can’t just combine them in any way</p>
<p>Link for Google Colab: <a href="https://colab.research.google.com/drive/12vCXui3Ulkk9olACYZiixwZfIW2UGWBl?usp=sharing" class="inline-onebox" rel="noopener nofollow ugc">Google Colab</a></p>
GRU for Multi-dimensional Input

username: se7en.dreams
cooked: <p>Hello, I have an input of shape <code>(14, 10, 30, 300)</code>,<br>
where, <code>14</code> is <code>batch_size</code>, <code>10</code> is the <code>seq_len</code>, <code>30</code> is the <code>num_tokens</code> in each element in the sequence, and <code>300</code> is the <code>embedding_dim</code> for each token. (I know, it’s complicated!)</p>
<p>I want to process each of the <code>10</code> elements in the sequence through a <code>GRU</code> and convert it to a <code>512</code> dimensional output. i.e., I wanna take each <code>30 x 300</code> in the sequence and learn to map it to a <code>512</code> dimensional vector.</p>
<p>Here’s what I have. I’m using the output of gru’s last hidden layer. That’s because my input represents <em>textual features</em> and I’ll be combining it with <em>visual features</em> later. Would love some feedback on whether this is the correct approach. Thank you.</p>
<pre><code class="lang-python">gru = nn.GRU(
    input_size=300,
    hidden_size=256,
    batch_first=True,
    bidirectional=True
)

# Reshape input from (14, 10, 30, 300) -&gt; (14*10, 30, 300)
inputs = inputs.reshape(14*10, 30, 300)
_, final_hidden_layer = gru(inputs)
outputs = final_hidden_layer.permute(1, 0, 2).contiguous().flatten(start_dim=1) # (140, 512)
outputs = outputs.reshape(14, 10, 512)
</code></pre>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a> <a class="mention" href="/u/pargg">@ParGG</a></p>

username: ParGG
cooked: <p>The GRU, with <code>bath_first=True</code> takes a sequence with dimensions (BatchSize, SequenceLengh, InputFeatures): <a href="https://pytorch.org/docs/stable/generated/torch.nn.GRU.html" class="inline-onebox" rel="noopener nofollow ugc">GRU — PyTorch 2.1 documentation</a>. If you want to map your <code>30 x 300</code> to <code>512</code>. Then you need to reshape your input as <code>inputs = inputs.reshape(14, 10, 30 * 300)</code>.</p>
<aside class="quote no-group quote-modified" data-username="se7en.dreams" data-post="1" data-topic="156682">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/a4c791/48.png" class="avatar"> se7en.dreams:</div>
<blockquote>
<p>I’m using the output of gru’s last hidden layer. That’s because my input represents <em>textual features</em>.</p>
</blockquote>
</aside>
<p>I would suggest you to check the documentation about what the two output of rat GRU represent. Once you will have multiple layers, it might not match your expectations.</p>
Average Pool to get rid of max_len dim

username: Pritish_Mishra
cooked: <p>I have a tensor with shape <code>(batch_size, max_len, latent_dim)</code> I want to make this tensor of shape <code>(batch_size, latent_dim)</code>. I want to get rid of max_len dim.</p>
<p>Why I want to do this?</p>
<p>I am doing sentiment analysis so initially, I have tensor of shape <code>(batch_size, max_len)</code> → then I gave this to embedding so it became <code>(batch_size, max_len, emb_dim)</code> → then I gave it to LSTM so it became <code>(batch_size, max_len, latent_dim)</code> → then I gave it to the final dense layer with 2 neurons for ‘positive’ and ‘negative’ sentiment so I can have two classes but I am getting shape <code>(batch_size, max_len, 2)</code> but I want <code>(batch_size, 2)</code> so then I can come to a conclusion what class my neural network has predicted.</p>
<p>In the <a href="https://www.tensorflow.org/tutorials/keras/text_classification#create_the_model" rel="noopener nofollow ugc">tensorflow tutorial here</a> you can see they have done some average pooling so <code>(batch_size, max_len, latent_dim)</code> becomes <code>(batch_size, latent_dim)</code>. But I want to use Pytorch, how can I achieve this?</p>
<p>Thanks!</p>
Chainer.grad() how to implement this function in Pytorch

username: shyZam
cooked: <p>I am writing to migrate code from chainer framework to pytorch. I came across the below code:</p>
<p>chainer.grad([loss_func(F.clipped_relu(X2,z=1.0),Yp)], [X2], set_grad=True, retain_grad=True)<br>
I tired looking at the pytorch documentation but was not able to find an equivalent function to implement the above code.</p>
<p><a href="https://docs.chainer.org/en/stable/reference/generated/chainer.grad.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://docs.chainer.org/en/stable/reference/generated/chainer.grad.html</a><br>
The above link explains what this function does in chainer framework. I would appreciate if someone could guide me on how to go about migrating the above chainer code to pytorch.</p>

username: ptrblck
cooked: <p>I think <a href="https://pytorch.org/docs/stable/generated/torch.autograd.grad.html#torch.autograd.grad"><code>torch.autograd.grad</code></a> would be the corresponding function.</p>

username: shyZam
cooked: <p>Hello, thank you for this answer.</p>
<p>I am able to calculate the gradient using torch.autograd.grad() function but this does not populate .grad field.</p>
<p>Could you tell me how I can populate this feild using the gradient calculate from torch.autograd.grad()?</p>
<p>Any help will be highly appreciated</p>

username: ptrblck
cooked: <p>If you want to populate the <code>.grad</code> attribute of specific parameters, you could use the <code>inputs</code> argument in the <code>backward</code> call as e.g. given <a href="https://discuss.pytorch.org/t/backpropagate-the-loss-of-a-certain-layer/156495/5">here</a>.</p>

username: shyZam
cooked: <p>Thank you for this. So this will update the .grad feild of those parameters using the gradient that was calculated using autograd,grad() function?</p>
<p>Thank you once again</p>

username: ptrblck
cooked: <p>Yes, that should be the case.</p>
How to degenerate SGD to primitive GD

username: robinho
cooked: <p>how to degenerate SGD to primitive GD? Or actually SGD(momentum=0, dampening=0, weight_decay=0, nesterov=False, maximize=False, foreach=None)=GD, where there’s no stochasticity?</p>

username: ptrblck
cooked: <p>The “stochastic” part refers to the mini-batch usage, so you could pass the entire dataset to the model and should get GD.</p>
Toggle between multiple dataloaders during the training

username: nini
cooked: <p>Hi,<br>
I have multiple dataloaders and I want to randomly select a batch from a dataloader randomly during the training.</p>
<p>For example, the first batch is from dataset 1 and the second batch is from dataset 2.<br>
The batch size and the input image size is the same across all the dataloaders.</p>
<p>Please let me know if there is any possible example or method.</p>

username: ptrblck
cooked: <p>You could create <code>iterators</code> for each <code>DataLoader</code> and get the next batches via <code>next(data_loader_iter)</code> from the corresponding <code>DataLoader</code>. Using this approach you would need to make sure there are enough batches available or catch the <code>StopIteration</code>.</p>
torchScript - Need help running a function a large number of times on gpu in parallel

username: FA321
cooked: <p>I’ve been trying to learn torchScript and have been running into some issues trying to apply it to a notebook I’m working on in Google Colab. Currently, I have a gaussian transformation function shown below:</p>
<pre><code class="lang-auto">def gaussian(x,mu,sigma):
    return 1-(sigma/(((x-mu)**2) + sigma**2) / np.pi)
</code></pre>
<p>Ideally, I want to run this function on gpu in parallel a large number of times (Ex. 100-1000 times). Here is my current torchScript code, mostly from <a href="https://pytorch.org/tutorials/advanced/torch-script-parallelism.html:" rel="noopener nofollow ugc">https://pytorch.org/tutorials/advanced/torch-script-parallelism.html:</a></p>
<pre><code class="lang-auto">import numpy as np
import torch
from torch import nn
from torchvision.transforms import ToTensor
from timeit import default_timer as timer

@torch.jit.script
def example(x):
    start = timer()
    futures: List[torch.jit.Future[torch.Tensor]] = []
    for _ in range(1000):
        futures.append(torch.jit.fork(gaussian, x, 0, 1))

    results = []
    for future in futures:
        results.append(torch.jit.wait(future))

    end = timer()
    print(f'elapsed time: {end - start}')
    return torch.sum(torch.cat(results))

x_ax = np.linspace(-100, 100, 101)
cen = 2
sigma = 10

y_ax = gaussian(x_ax,cen,sigma)
</code></pre>
<p>However, when I try to run this, I get the following error:</p>
<pre><code class="lang-auto">RuntimeError: 
Python builtin &lt;built-in function perf_counter&gt; is currently not supported in Torchscript:
  File "&lt;ipython-input-5-280f74a36289&gt;", line 13
@torch.jit.script
def example(x):
    start = timer()
            ~~~~~ &lt;--- HERE
    futures: List[torch.jit.Future[torch.Tensor]] = []
    for _ in range(1000):
</code></pre>
<p>Is there a way to fix this? Is torchScript even compatible with this function definition at all (If not, are there any alternatives)? Any help would appreciated.</p>

username: ptrblck
cooked: <p>I don’t think TorchScript is able to understand the <code>timer()</code> function so you might need to remove it.</p>
Conv1d - Expected 3-dimensional input for 3-dimensional weight [9, 16, 2], but got 2-dimensional input of size [32, 160] instead

username: Kezzy
cooked: <p>Hi Everyone,</p>
<p>I am facing a problem probably in data loading or maybe making some error in Convd layer.</p>
<p>Problem Definition - I have some understanding of AE and now trying to move towards Convolution AE. I have (32000,160) data in a single file and I want to use my ConvDLayer (subclass of Encoder class) and I get the error mentioned in the headline.</p>
<p>originally, I have (32000,16) data but using somelength of signal at a time, I am making 160 feature (for example, response length = 10 , number of sensor = 16, so taking every 10 steps for 16 sensor and making first row, second row and so on - 1st sensor - some values only till 10 response length/sensor values , 2nd sensor some values till 10 response length, 3rd sensor - some values  and so on till 16th sensor data)</p>
<pre><code class="lang-auto">class ConvDLayer(nn.Module):
    def __init__(self, input_channels, output_channels, kernel_size, dilation, l_in):
        super(ConvDLayer, self).__init__()
        stride = 1
        # dilation = 1
        padding = int((l_in * (stride - 1) - stride + dilation * (kernel_size - 1) + 1) / 2)
    
        self.conv = nn.Conv1d(input_channels, output_channels, kernel_size, padding=(padding,), dilation=dilation)
        self.batchnorm = nn.BatchNorm1d(output_channels)
        self.activation = nn.ReLU()
        self.dropout = nn.Dropout(p=0.2)

    def forward(self, x):
        x = self.conv(x)
        x = self.activation(x)
        return x
 

class EncoderAEC(nn.Module):
    def __init__(self, num_sensors, channels, kernel_sizes, dilation, response_length):
        super(EncoderAEC, self).__init__()
        self.layers = nn.ModuleList()
        in_channel = num_sensors
        i = 0
        for channel in channels:
            self.layers.append(ConvDLayer(in_channel, channel, kernel_sizes[i], dilation[i], response_length))
            in_channel = channel
            i += 1

    def forward(self, x):
        for layer in self.layers:
            x = layer(x)
        return x
</code></pre>
<p>Kernel_sizes_encoder = (2,)<br>
Channels_encoder= [9]<br>
in_channel = 16</p>
<p>Can anyone please help me here ?</p>

username: ptrblck
cooked: <p>I’m not sure I completely understand the data format but it seems you are passing it as <code>[batch_size=32, features*seq_len=160]</code>? If so, you might want to split <code>dim1</code> into a temporal and feature dimension and pass the input as <code>[batch_size, channels, seq_len]</code>.</p>
Lambda inline function

username: robinho
cooked: <p>There’s an inline function f = lambda x: 1+n(x), where n is a neural net defined by n=nn.Sequential(nn.Linear(1,10), nn.Sigmoid()).<br>
n needs to be trained and during training this f function is called all the time. Is this f function evolving with the n being trained, or is f just endowed with a frozen n without training?</p>

username: ptrblck
cooked: <p>Yes, it seems to work as I’m seeing weight updates:</p>
<pre><code class="lang-python">n = nn.Sequential(nn.Linear(1,10), nn.Sigmoid())
optimizer = torch.optim.SGD(n.parameters(), lr=1.)

f = lambda x: 1+n(x)

for _ in range(3):
    print(n[0].weight.abs().sum())
    optimizer.zero_grad()
    
    x = torch.randn(1, 1)
    out = f(x)
    out.mean().backward()
    
    print(n[0].weight.grad.abs().sum())
    optimizer.step()
</code></pre>

username: robinho
cooked: <p>thank you, really nice proving that a lambda function is dynamic</p>
RuntimeError: CUDA out of memory + gpu memory management best practices? + colab

username: Prachi
cooked: <p>Hello all, can anyone explain to me what is happening here, I am a novice in this topic of memory management + colab:<br>
I am training a pre-trained inception V3 model for cifar10 data for <strong>epochs=10</strong> on a colab.<br>
Following is higher level script code:</p>
<p>for each epoch:<br>
model.train()<br>
…<br>
model.eval()<br>
with torch.no_grad():<br>
…<br>
…</p>
<p>After the <strong>first epoch</strong> of training colab throws error related to <em>RuntimeError: CUDA out of memory</em> for batch_size:[8,16,32,64], just before running <strong>evaluation on test</strong>.<br>
How should I fix it if I want to run in one script? What are the best practices for this, memory mgmt? Answers would be appraciated.</p>
<p>Thanks,<br>
Prachi</p>

username: thecho7
cooked: <p>YOLOv5 supports what you are looking for (<a href="https://github.com/ultralytics/yolov5/blob/master/utils/autobatch.py" class="inline-onebox" rel="noopener nofollow ugc">yolov5/autobatch.py at master · ultralytics/yolov5 · GitHub</a>)</p>

username: Prachi
cooked: <p>Hi Suho, thanks for your prompt reply.<br>
The code provides estimating apt batch size to use <code>fraction</code> of available CUDA memory, probably to avoid running OOM. is it right? It is helpful in a way. However, do you know if in a <strong>script</strong> I can run training+evaluation subsequently for each epoch?<br>
Prachi</p>

username: ptrblck
cooked: <p>If the validation loop raises the out of memory error, you are either using too much memory in the validation loop directly (e.g. the validation batch size might be too large) or you are holding references to the previously executed training run.<br>
Python frees variables once they leave a function scope so you could either wrap the training and validation on own functions or <code>del</code> unneeded tensors manually if you are using a global scope.</p>

username: Prachi
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, big fan of your answers on PyTorch.</p>
<p>Yes, you are right, I overlooked the validation batch size, which is 1000 in my case. For training, I am using 64 (the current GPU can handle), so big difference. I tried running with test_batch_size = 250 and it worked fine.<br>
What suggestions would you recommend other than making train and eval functions separately?</p>
<p>Thanks,<br>
Prachi</p>

username: ptrblck
cooked: <p>Besides making sure unneeded references are freed, I would check if the validation loop is already wrapped into a <code>with torch.no_grad()</code> guard to avoid storing intermediates.</p>

username: Prachi
cooked: <p>Yes, that helps. Thanks!!</p>
Cannot install the gpu version of torch and torchvision in poetry due to a dependency problem

username: Hika-Kondo
cooked: <p>I am trying to create a virtual environment for machine learning using poetry. So, I am using pytorch as a framework for deep learning. I will extract the relevant part of my pyproject.toml.</p>
<pre><code class="lang-auto">[tool.poetry.dependencies].
python = "^3.8"
torch = { url = "https://download.pytorch.org/whl/cu111/torch-1.8.0%2Bcu111-cp38-cp38-linux_x86_64.whl"}
torchvision = { url = "https://download.pytorch.org/whl/cu111/torchvision-0.9.0%2Bcu111-cp38-cp38-linux_x86_64.whl" }
</code></pre>
<p>Since pytroch uses the GPU, you need to install it by specifying the whl file. If you install it this way, the version of <code>pytroch</code> will be <code>1.8.0+cu111</code>. <code>torchvision</code> corresponding to <code>1.8.0</code> of pytorch is <code>0.9.0</code>. The version of pytroch that this <code>torchvision</code> depends on is <code>1.8.0</code> (without <code>cu111</code>). Therefore, I cannot create a virtual environment using poetry with the following error.</p>
<pre><code class="lang-auto">
  SolverProblemError

  Because torchvision (0.8.0) depends on torch (1.7.0)
   and mdwithpriorenergy depends on torch (1.8.0+cu111), torchvision is forbidden.
  So, because mdwithpriorenergy depends on torchvision (0.8.0), version solving failed.

So, because [env name] depends on torchvision (0.8.0), version solving failed. 

</code></pre>
<p>I also made the following changes to torchvision in pyproject.toml above, but they did not work.</p>
<pre><code class="lang-auto">[tool.poetry.dependencies].
python = "^3.8"
torch = { url = "https://download.pytorch.org/whl/cu111/torch-1.8.0%2Bcu111-cp38-cp38-linux_x86_64.whl"}
- torchvision = { url = "https://download.pytorch.org/whl/cu111/torchvision-0.9.0%2Bcu111-cp38-cp38-linux_x86_64.whl"}
+ torchvision = "*"
</code></pre>
<p>In this case, I received the following error</p>
<pre><code class="lang-auto">  AttributeError

  'EmptyConstraint' object has no attribute 'allows'.

</code></pre>
<p>Please tell me how to solve this error.</p>

username: sarat
cooked: <p>Are you able to solve this problem? I am facing the same issue.</p>

username: phithangcung
cooked: <p>Have you solve this problem ? I met the same problem now.</p>
How to Do Semi-Asynchronous or Asynchronous Training with Pytorch

username: lsy643
cooked: <p>When PyTorch is used for distributed training, DDP is normally good enough for most situations. However, when if performance of different nodes differs, the performance of the whole training will be decided by the worst node. E.g. worker 0 needs 1 second for a forward and backward pass while worker 1 needs 2 seconds, the time for one step will be 2 seconds.</p>
<p>So I am wondering if there is way to do semi-asynchronous training with Pytorch?</p>
<p>There is a similar library called hivemind, but it is designed for Internet while we prefer to run the training job in our cluster.</p>

username: aazzolini
cooked: <p>Hi Siyuan , thanks for your question.</p>
<p>PyTorch distributed package offers a Remote Procedure Call (RPC) API that you could use to implement your asynchronous training algorithm.</p>
<p>Here’s an example of how to implement a parameter server using RPC: <a href="https://pytorch.org/tutorials/intermediate/rpc_param_server_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Implementing a Parameter Server Using Distributed RPC Framework — PyTorch Tutorials 1.12.0+cu102 documentation</a></p>
<p>Let us know if that makes sense for you.</p>
Unable to load torch.nn.Module on sagemaker

username: Ben_Nicholl
cooked: <p>I created a model using torch.nn.Module inheritance below</p>
<pre><code class="lang-auto">class Model(torch.nn.Module):
    def __init__(self, model, logit_layer):
        super(Model, self).__init__()
        
        self.model = model
        
        self.logit_layer = logit_layer

    def forward(self, input_ids, mask, tokens):
        pre_logits_mask = torch.reshape(mask, (mask.shape[0], mask.shape[1], 1) )
        
        outputs = self.model(input_ids, attention_mask=mask, token_type_ids=tokens)
        last_hidden_state = outputs.last_hidden_state
        last_hidden_state_zero_layer = torch.mul(last_hidden_state, pre_logits_mask)
        summed_final_hidden_state = torch.sum(last_hidden_state_zero_layer, 1)
        
        logits = self.logit_layer(summed_final_hidden_state)
        probs = torch.sigmoid(logits)
        
        return probs
</code></pre>
<p>I works fine when saving and loading locally, but sagemaker is unable to load this object when I use sagemakers PyTorch wrapper function. When I checked the type of my above model, it read</p>
<pre><code class="lang-auto">type(entire_model)
Out[116]: __main__.Model
</code></pre>
<p>So it doesn’t look like the above model I created is technically a PyTorch object, which may be the cause of the sagemaker model not loading error. Is there a workaround for this?</p>
Timm models default_cfg

username: pradnesh
cooked: <p>Hi there. I am currently using the beit model from timm as backbone. Here in the <strong>default_cfg</strong> it shows that the size should be 224x224x3, interpolation should be BICUBIC, mean and std should be 0.5. I am attaching a figure as well as shown below.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/a/8af9f93ccbc733b7e3f524ab83364feaaa0d2f91.png" data-download-href="https://discuss.pytorch.org/uploads/default/8af9f93ccbc733b7e3f524ab83364feaaa0d2f91" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/a/8af9f93ccbc733b7e3f524ab83364feaaa0d2f91_2_690x177.png" alt="image" data-base62-sha1="jPrsk9sGWWMgw7IWsfSt3jAQUpz" width="690" height="177" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/a/8af9f93ccbc733b7e3f524ab83364feaaa0d2f91_2_690x177.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/a/8af9f93ccbc733b7e3f524ab83364feaaa0d2f91_2_1035x265.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/a/8af9f93ccbc733b7e3f524ab83364feaaa0d2f91_2_1380x354.png 2x" data-dominant-color="F8F8F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2422×622 38 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>So, my doubt is → should we write transforms like this (shown in the code below) to handle the default_config as shown in the image above or will the transforms like BICUBIC and normalize be handled automatically by the loaded timm model?</p>
<pre><code class="lang-auto">scale_transform = transforms.Resize(size=(224, 224), interpolation=transforms.InterpolationMode.BICUBIC)
channel_transform = transforms.ToTensor()
normalize = transforms.Normalize(mean=[0.5, 0.5, 0.5],
                                     std=[0.5, 0.5, 0.5])
composed_transform = transforms.Compose([scale_transform, channel_transform, normalize])
</code></pre>
<p>I am not able to find relevant results on the web. Please help.</p>
<p>Thanks,<br>
Pradnesh</p>
Pytorch autograd second derivative

username: tingw
cooked: <p>Hi. When I was testing the following toy example</p>
<p>q = torch.rand(10, 2, requires_grad=True)<br>
p = torch.rand(10, 1, requires_grad=True)<br>
input = torch.cat((q, p), dim=1)<br>
u = torch.sum(input, dim=1)<br>
u_q = torch.autograd.grad(u, q, grad_outputs=torch.ones_like(u), create_graph=True, retain_graph=True)[0]<br>
u_p = torch.autograd.grad(u, p, grad_outputs=torch.ones_like(u), create_graph=True, retain_graph=True)[0]<br>
u_qq = torch.autograd.grad(u_q, q, grad_outputs=torch.ones_like(u_q), create_graph=True, retain_graph=True)[0]</p>
<p>I got the error message:</p>
<blockquote>
<p>element 0 of tensors does not require grad and does not have a grad_fn</p>
</blockquote>
<p>However, if I change u = torch.sum(input, dim=1) to u = torch.sum(input**1, dim=1), the error disappears. Can anyone explain why?</p>

username: soulitzer
cooked: <p>Probably because u is linear wrt its inputs, so when you take the first derivative that produces a constant output, which won’t have a grad_fn. When you use input**1 that will probably just get you a gradient that is zero anyway.</p>

username: tingw
cooked: <p>I agree. This seems to be what is going on here. But it is strange that one needs to add **1 in order for autograd to work properly for such a simple computation.</p>

username: soulitzer
cooked: <p>I’d actually say that this is expected behavior since you’d rather explicitly know that your graph will always produce zero gradients. I agree that it can be confusing, however.</p>