akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Constraints on Parameters Update

username: sfr000
cooked: <p>I would like to make sure the error is decreasing at each iteration of a module I invented.<br>
Let’s say it is made of iterative processes which are called for example for 10 times, so my architecture is made of the unfolded version of these iterative processes.</p>
<p>To learn the parameters of this unfolded architecture, I am using the back propagation (through time) of the error computed at the end of this architecture.<br>
I would like to make sure that at each parameter update, the error is decreasing through this unfolded architecture.</p>
<p>Is there a way to be sure the error is decreasing at each iteration step, though we are doing a backpropagation of the error from the end of the network?</p>
<p>I know it is a tricky question, however it could be the case that somebody has faced this problem,<br>
thanks.<br>
S</p>
Train vs eval mode

username: twkim0812
cooked: <p>Hi!<br>
I have a basic question.</p>
<p>I’m trying to learn visuomotor policy for robot application, where the robot takes a series of images as inputs and the policy network outputs motor velocities to perform a target task.</p>
<p>I imported the pretrained ResNet18 model from torchvision for feature extraction,<br>
and the features are fed into the following fully connected layers.</p>
<p>In training phase, the imported ResNet18 runs by .train() mode, whereas .eval() mode in test phase.</p>
<p>What I want to know is that if I use the ResNet18 for feature extraction only(not fine-tune), should I set it to .eval() mode in training phase? or my current approach is correct?<br>
(I know the difference between the modes is about the batch norm and dropout)</p>
<p>Or, .train() mode with freezing all the layers is enough?</p>
<p>Though there are a lot of topics about train vs eval mode, but little bit confusing in details…</p>
Dataloader pickle error because *must* use multiprocessing_context = 'spawn' instead of 'fork'

username: harpone
cooked: <p>I’m working with a library that does some heavy multithreading and it hangs if I use the ‘fork’ multiprocessing context, so I need to use ‘spawn’ (<strong>not</strong> using windows jfc…).</p>
<p>Then getting the classic “AttributeError: Can’t pickle local object ‘main..collate_fn’” when defining the collate function something like this inside the <code>main()</code>:</p>
<pre><code class="lang-auto">def collate_fn(x):
    return functions.collate(collate_params, x)
</code></pre>
<p>What can I do?</p>
<p>Someone somewhere mentioned using dill instead of pickle (it can serialize anything I guess), but this is a problem with pytorch using pickle, so can’t just use dill I think…</p>
<p>EDIT: OK the form of the collate function is not really important, but that it’s defined inside the main() and then right after I define the <code>Dataloader(collate_fn=collate_fn, ...)</code></p>

username: nivek
cooked: <p>Can you elaborate on what is making the function unpicklable?</p>
<p>One potentially solution is to create a callable object with custom <code>__getstate__</code> and <code>__setstate__</code> methods that will make that the object pickable (or use dill within). Then use that callable object as <code>collate_fn</code>.</p>

username: harpone
cooked: <p>Ah OK that sounds cool, need to try that out!</p>
<p>I don’t know why it’s unpicklable… it’s not a lambda expression after all.</p>
<p>Anyway, I actually managed to use ‘fork’ by going around the multiprocessing issue so AOK (will stick with ‘fork’ if I only can).</p>
<p>Thanks for the interesting tip though <a class="mention" href="/u/nivek">@nivek</a>!</p>
Gradient accumulation convergence slowing

username: junqiangchen
cooked: <p>Hello,</p>
<p>I use 3D Unet that including Conv3d，GroupNorm,Dropout3d layers for medical image segmentation,input size is 256x240x256x1</p>
<p>in order to train the model，i use Mixed Precision Training and Gradient accumulation Training from here <a href="https://pytorch.org/docs/stable/notes/amp_examples.html" class="inline-onebox" rel="noopener nofollow ugc">CUDA Automatic Mixed Precision examples — PyTorch 1.13 documentation</a></p>
<pre><code class="lang-auto">scaler = GradScaler()

for epoch in epochs:
    for i, (input, target) in enumerate(data):
        with autocast(device_type='cuda', dtype=torch.float16):
            output = model(input)
            loss = loss_fn(output, target)
            loss = loss / iters_to_accumulate

        # Accumulates scaled gradients.
        scaler.scale(loss).backward()

        if (i + 1) % iters_to_accumulate == 0:
            # may unscale_ here if desired (e.g., to allow clipping unscaled gradients)

            scaler.step(optimizer)
            scaler.update()
            optimizer.zero_grad()
</code></pre>
<p>i find Gradient accumulation Training method lead to slow convergence.<br>
Has anyone encountered this situation?</p>
Different networks with same optimizer

username: qiminchen
cooked: <p>Suppose I have 4 <strong>independent</strong> networks and only 1 optimizer</p>
<pre><code class="lang-auto">network_1 = network()
network_2 = network()
network_3 = network()
network_4 = network()

optimizer = torch.optim.Adam(list(network_1.parameters()) + 
                             list(network_2.parameters()) +
                             list(network_3.parameters()) +
                             list(network_4.parameters()), lr=0.0001)

input = torch.rand((2, 2, 2))
loss_1 = gt_1 - network_1(input)
loss_2 = gt_2 - network_2(input)
loss_3 = gt_3 - network_3(input)
loss_4 = gt_4 - network_4(input)
loss_1.backward()
loss_2.backward()
loss_3.backward()
loss_4.backward()
optimizer.step()
</code></pre>
<p>Loss of each network is computed independently, when doing the backpropagation, the gradient of each loss should only affect its corresponding network even if I use the same optimizer, right? i.e. <code>network_1</code> parameters will be updated only based on the gradient computed from <code>loss_1</code> and will NOT be affect by other losses.</p>

username: ptrblck
cooked: <p>Yes, independent computation graphs won’t have any effect on each other.</p>
Detectron2 in C++

username: Sylvain_Ard
cooked: <p>I am trying to convert detectron 2 inference in C++, CPU and VS 2019 with this code : <a href="https://github.com/facebookresearch/detectron2/tree/main/tools/deploy" class="inline-onebox" rel="noopener nofollow ugc">detectron2/tools/deploy at main · facebookresearch/detectron2 · GitHub</a>. Cmake successfully find OpenCV 4 and libtorch but I haven’t found on Internet the libtorchvision, where it is please ?<br>
thank you<br>
best regards<br>
PS : I found Libtorch here : <a href="https://pytorch.org/get-started/locally/" class="inline-onebox" rel="noopener nofollow ugc">Start Locally | PyTorch</a></p>

username: solarflarefx
cooked: <p><a class="mention" href="/u/sylvain_ard">@Sylvain_Ard</a> Were you able to figure this out?</p>

username: KRG
cooked: <p>You will have to build the touch vision c++. Please refer - <a href="https://discuss.pytorch.org/t/building-torchvision-c-api-from-source/68281" class="inline-onebox">Building Torchvision C++ API from source</a></p>
<p>Below is a sample of commands I used in Windows using anaconda.</p>
<ul>
<li>Clone pybind11 and install to C:\pybind11</li>
</ul>
<pre><code class="lang-auto">    git clone https://github.com/pybind/pybind11.git
    cd pybind11
    mkdir build
    cd build
    cmake -DPYBIND11_TEST=OFF -DCMAKE_INSTALL_PREFIX="C:\Users\pybind11" ..
</code></pre>
<ul>
<li>Clone torchvision</li>
</ul>
<pre><code class="lang-auto">   git clone https://github.com/pytorch/vision.git
</code></pre>
<ul>
<li>Cmake the project.</li>
</ul>
<pre><code class="lang-auto">    cd vision
    mkdir build
    cd build    
    ## Enable CUDA. 
    cmake -DWITH_CUDA=on -DUSE_PYTHON=on -DCMAKE_PREFIX_PATH="C:\libtorch-win-shared-with-deps-1.10.2+cu113\libtorch;C:\Users\pybind11" -DCMAKE_BUILD_TYPE=Release ..
</code></pre>
<ul>
<li>
<p>Create torchvison dll file : cd C:\Users\user\vision\build and open torchvision.sln using Visual Studio. Build the ‘tourchvision’ project to create DLL file. DLL file ‘torchvision.dll’ is created in '.\Release' folder.</p>
</li>
<li>
<p>Install C++ files : cd C:\Users\user\vision\build and install torchvision c++ using ‘cmake -P cmake_install.cmake’. C++ files will be created in C:\Program Files (x86)\torchvision.</p>
</li>
</ul>
How to view the commands that were used to build a Pytorch wheel?

username: Vedant_Roy
cooked: <p>I’m trying to figure out what settings were used to build a particular Pytorch wheel. Is there any way to tell?</p>
<p>The reason is, I’d like to be able to build Pytorch from source, while closely following the official build configuration.</p>

username: ptrblck
cooked: <p>The <a href="https://github.com/pytorch/builder/tree/main/manywheel">pytorch/builder</a> repository contains the code used to build the binaries so you might want to take a look at it.</p>
How to make targets go through the same augmentations as the images?

username: Rafa
cooked: <p>Hi all,</p>
<p>I’ve been trying the <a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html" rel="noopener nofollow ugc">instance segmentation tutorial</a> and I have a question about augmentations. Basically, I want to perform augmentations to the images, but the targets (boxes, masks) also need to suffer the same augments. If I do them individually everything comes out wrong, because of the randomness of these operations.</p>
<p>The tutorial has some helper functions that, from what I understand, are not part of the main torchvision module. They do what I want, but only some very specific augmentations are provided. Horizontal flip is provided but not vertical, for example.</p>
<p>Is there any way or any library that does this? I’ve looked at albumentations but it doesn’t support masks for instance segmentation (it expects a single segmentation mask).</p>
<p>I’d want something like in the tutorial, that does everything in a single line, like:</p>
<pre><code class="lang-auto">        target = {}
        target["boxes"] = boxes
        target["labels"] = labels
        target["masks"] = masks
        target["image_id"] = image_id
        target["area"] = area
        target["iscrowd"] = iscrowd

        if self.transforms is not None:
            img, target = self.transforms(img, target)
</code></pre>

username: ptrblck
cooked: <p>One possible approach would be to use the functional transforms API and sample the “random” parameters once and apply them to the input image as well as all targets.<br>
<a href="https://discuss.pytorch.org/t/torchvision-transfors-how-to-perform-identical-transform-on-both-image-and-target/10606/7">This post</a> gives you an examples and you could also use the reference implementation from <a href="https://github.com/pytorch/vision/blob/ad128b753c7e8cc0c600dfddac22ff48fc73c9d9/references/detection/transforms.py#L30-L45">here</a> to add more transformations.</p>
PyTorch DDP but with only one process to load data

username: amirhf
cooked: <p>Hello,</p>
<p>I understand that with PyTorch DDP, each process loads its own instance of data from the disk. However, my dataset is very large (a very large parquet file that loads into a dataframe) and I can’t have each process load it into memory with limited RAM capacity. Is there a shared memory implementation so that one process loads the data into RAM and then each process uses the same loaded data from the first process?</p>
<p>I also thought of splitting but I can’t split (.iloc) data until after all the data is loaded.</p>

username: wanchaol
cooked: <p>Does your use case need the entire dataset to be available in RAM to start off training? For DDP and general PyTorch training, usually it’s doing batched gradient descent and it only needs a batch of data available in the memory at a time, so the memory requirement on each worker should be fairly small when using PyTorch’s <a href="https://pytorch.org/docs/stable/data.html" rel="noopener nofollow ugc">dataloader</a>, could you try streaming the parquet file and use PyTorch’s dataloader to load the data?</p>

username: amirhf
cooked: <p>Yes, the data needs to be in RAM. That’s what makes my application complicated.<br>
How do I stream a parquet file?</p>

username: f10w
cooked: <p>I am also looking for a solution to this problem. If anybody knows about this, please help!</p>

username: fduwjj
cooked: <p>Hi, I guess there are two things you can try:</p>
<ol>
<li>You can try the distributed sampler.</li>
<li>If it does not work for you, you probably need to have your customized logic for the data loading. cc: <a class="mention" href="/u/ejguan">@ejguan</a> <a class="mention" href="/u/vitalyfedyunin">@VitalyFedyunin</a>
</li>
</ol>

username: ejguan
cooked: <p>I think <code>torchdata</code> does provide a <code>DataPipe</code> to stream parquet file. <a href="https://github.com/pytorch/data/blob/2dfdcfb861971f08d5a0673ce17d6a979bc5de8d/torchdata/datapipes/iter/util/dataframemaker.py#LL95C5-L95C5" class="inline-onebox" rel="noopener nofollow ugc">data/dataframemaker.py at 2dfdcfb861971f08d5a0673ce17d6a979bc5de8d · pytorch/data · GitHub</a></p>
<p>But, it requires some implementation changes on your data pipeline. We proposed to make data pipeline using composable <code>DataPipes</code>. You can find more detailed doc in <a href="https://pytorch.org/data/beta/tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Tutorial — TorchData 0.5.0 (beta) documentation</a></p>
<p>cc: <a class="mention" href="/u/nivek">@nivek</a></p>
Incorrect Conv2D Output

username: Sairam954
cooked: <p>Hi,</p>
<p>I was trying to understand and mimic conv2D. However, I am not able to comprehend the output value of the conv2d.<br>
I am defining my input and also assigning the custom weights to the conv2d.<br>
Below is the code:</p>
<pre><code class="lang-auto">import torch


m = torch.nn.Conv2d(1, 1, 3, stride= 1)
input = torch.randn(1, 1, 3, 3)

print(input.dtype)
input = torch.randint(0,2,(1,1,3,3), dtype= torch.float32)
weight = torch.randint(0,2,(1,1,3,3), dtype= torch.float32)
print("Default Conv Weight ", m.weight.data)
m.weight.data= weight
true_output = m(input)
print("Assigned Conv Weight ", m.weight)
print("Conv Weight ", weight)
print("Conv input ", input)
print('True Output',true_output)
print('True Output Shape', true_output.shape)
</code></pre>
<p>Output Image:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/2/220f11777661df9f08c7509c6faee902a21998f8.png" alt="image" data-base62-sha1="4Riw5nedO4yzvzlv5kF787uLkzC" width="641" height="333"></p>
<p>From the output image, I was expecting the output to be 3, but it shows 3.16. I am not sure what is happening under the module.<br>
<a class="mention" href="/u/hdcharles">@HDCharles</a> <a class="mention" href="/u/jcaip">@jcaip</a>  <a class="mention" href="/u/jerryzh168">@jerryzh168</a></p>
<p>Thanks</p>

username: ptrblck
cooked: <p>You are not considering the <code>bias</code> which will be added to your output activation.</p>

username: Sairam954
cooked: <p>Yes, I completely missed that. Thanks</p>
Training Hogwild with SGD parallel process stalls in CPU

username: jlopez
cooked: <p>I am training the MNIST dataset with HOGWILD Async SGD as per your example. I am using a CPU with 64 cores and a GPU A100.</p>
<p>When running the program for CPU, the more processors I use the more time takes to run. I have tried with 3,4,5,6,…and more and the process stalls. As per the figure you can appreciate that when running the program with 6 processes the processing time becomes “infinite” (represented here by 999).I can attest that the number of processes translates to the number of cores used to run the program, because when I monitor the cpu usage I can see that the cores that are busy equals the number of processes specified in the run. I thought that the more processors used the faster the training would be, but is not the case, at first I thought that the communication increased and could be the culprit, but I am not sure, do you know why? Is this a bug, or I am missing something?</p>
<p>Also, for the case of running the program with --cuda, what is the significance of number of processes?, if there is only one GPU available? What I observed is that even though the process does not stall, the processing time grows with the increased number of processors as well.</p>
<p>Thanks for your help.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/c/2cc76ee515f99e2a7c501667890f83169cc4f4e7.png" data-download-href="https://discuss.pytorch.org/uploads/default/2cc76ee515f99e2a7c501667890f83169cc4f4e7" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/c/2cc76ee515f99e2a7c501667890f83169cc4f4e7_2_690x384.png" alt="image" data-base62-sha1="6o8i7GXShlbYGLJHgfGk15fnBXN" width="690" height="384" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/c/2cc76ee515f99e2a7c501667890f83169cc4f4e7_2_690x384.png, https://discuss.pytorch.org/uploads/default/original/3X/2/c/2cc76ee515f99e2a7c501667890f83169cc4f4e7.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/2/c/2cc76ee515f99e2a7c501667890f83169cc4f4e7.png 2x" data-dominant-color="F7E6E5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">884×493 22.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: jlopez
cooked: <p>Can someone from Pytorch reply to this post please?</p>
Does the model.train() change the sub-modules?

username: changanjie1
cooked: <p>Hello, I have this model class with two submodules, self.extractor and self.mlp.</p>
<pre><code class="lang-auto">class Model(nn.Module):
     def __init__(self, ):
        super(Model, self).__init__()
        self.extractor = ....
        self.extractor.eval()
        self.mlp = .....
</code></pre>
<p>The extractor is used to extract feature vector from the image and the self.mlp is used to perform the classification. In my case, I’d like the extractor to be in eval() mode all the time, but only train the self.mlp. However, in my training loop, I set up the Model into train():</p>
<pre><code class="lang-auto">model = Model()
def train_one_batch():
    model.train()
    ......
</code></pre>
<p>Now, I am confused if this line ‘model.train()’ will set the self.extractor to the train() mode and fails my goal.</p>

username: ConvolutionalAtom
cooked: <p>Hi,<br>
Yes, it also sets the submodules’ mode to training mode.</p>
<p>For the relevant code see here (search for train method in the page),<br>
<a href="https://pytorch.org/docs/stable/_modules/torch/nn/modules/module.html#Module" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/_modules/torch/nn/modules/module.html#Module</a><br>
where it goes through its children module</p>
<p>Edit: suggestion: for your problem, maybe you could still use train method and in the next line set extractor to eval mode so every submodule is to training mode except for extractor.</p>

username: changanjie1
cooked: <p>Thank you for the response, this really helps a lot!</p>
TypeError: relu(): argument 'input' (position 1) must be Tensor, not tuple. I believe it's because I have an LSTM layer. How can I modify the code so it will work fine? Any help?

username: sam5
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/d/7da565c3dda548de9495a3aed9cdaeb70d073279.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/7da565c3dda548de9495a3aed9cdaeb70d073279" title="pytorch model"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/d/7da565c3dda548de9495a3aed9cdaeb70d073279_2_690x217.jpeg" alt="pytorch model" data-base62-sha1="hVw2SctlDNEVe0sZmhProXXYxsZ" width="690" height="217" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/d/7da565c3dda548de9495a3aed9cdaeb70d073279_2_690x217.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/7/d/7da565c3dda548de9495a3aed9cdaeb70d073279_2_1035x325.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/7/d/7da565c3dda548de9495a3aed9cdaeb70d073279_2_1380x434.jpeg 2x" data-dominant-color="F3F4F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">pytorch model</span><span class="informations">1920×604 118 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: srishti-git1110
cooked: <p>Please ensure you are putting a PyTorch tensor inside relu and not a python tuple or any other data structure.</p>

username: sam5
cooked: <p>Hi I’m not sure how can I do that? When I remove LSTM layer and add a linear layer, it works fine. But I need to keep LSTM layer. So how can I correct it?</p>

username: srishti-git1110
cooked: <p>Can you please post a minimum executable snippet?</p>

username: AlphaBetaGamma96
cooked: <p>Hi <a class="mention" href="/u/sam5">@sam5</a>,</p>
<p>What’s the shape of <code>x_training</code>? Could that be the Tuple you’re looking for?</p>
<p>Also, can you share the code within three backticks (```)? Posting screenshots isn’t legible.</p>

username: sam5
cooked: <p>It’s solved. Thank you.</p>
CNN gives negative output

username: Ioannis_D
cooked: <p>Hello,</p>
<p>I have trained my model using CrossEntropyLoss thus I havent used a softmax at the end or a sigmoid (it is binary classification but I have used softmax).<br>
However, when I test the model, it gives negative output (probability) and I assume this is wrong.<br>
Am I supposed to have a softmax at the end of the model during testing? If yes how can I tune it then?</p>
<p>Thank you</p>

username: ptrblck
cooked: <p>Since you are using <code>nn.CrossEntropyLoss</code> the model is supposed to return logits, which contains arbitrary values. The negative values are thus not wrong but expected.<br>
To get the predicted class you could directly use <code>preds = torch.argmax(output, dim=1)</code> on the raw logits. To see the probability, you could apply <code>softmax</code> on the output, but do not use these probabilities in the loss function.<br>
Also, note that <code>torch.argmax</code> returns the same predicted classes for logits or probabilities, since the max. value index won’t change.</p>
RuntimeError: NCCL communicator was aborted on rank 0

username: Debasish_Behera
cooked: <p>I was doing distributed training of my HRNet model using my 2 GPU. so after certain iteration of batches, it showed this issue. In the meanwhile, when i checked using nvidia-smi on cmd, it showed me this error:- Unable to determine the device handle for GPU 0000:67:00.0: Unknown Error…</p>
<p>I have been saving my checkpoint every 2500 iteration, and my model is of size 1.67GB.</p>
<p>RuntimeError: NCCL communicator was aborted on rank 0.<br>
[E ProcessGroupNCCL.cpp:325] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data. To avoid this inconsistency, we are taking the entire process down.<br>
terminate called after throwing an instance of ‘std::runtime_error’<br>
what():  [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(OpType=ALLGATHER, Timeout(ms)=4800000) ran for 4806903 milliseconds before timing out.</p>
<p>Please i needed help on this</p>

username: fduwjj
cooked: <aside class="quote no-group" data-username="Debasish_Behera" data-post="1" data-topic="166009">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/debasish_behera/48/46065_2.png" class="avatar"> Debasish_Behera:</div>
<blockquote>
<p>nicator was aborted on rank 0.<br>
[E ProcessGroupNCCL.cpp:325] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data. To avoid this inconsistency, we are taking the entire process down.<br>
terminate called after throwing an instance of ‘std::runtime_error’<br>
what(): [Rank 0] Watch</p>
</blockquote>
</aside>
<p>Can you share your code so we can better help you? What env and setup is your training job? Thanks!</p>
WeightedRandomSampler with weights per class

username: hond86
cooked: <p>Hi all,<br>
I’m currently using the WeightedRandomSampler by passing a weights tensor with the same size of the number of samples, so one weight per sample and it’s working fine. I was wondering if there’s a way to specify weights per classes, since I’m currently working with millions of samples and the weights tensor is therefore of the same size.</p>
<p>Thanks.</p>

username: ptrblck
cooked: <p>No, I don’t think that’s currently possible. Note however, that 1 million <code>float32</code> values take approx. ~4MB of memory, so unsure how much memory you would save in the end.</p>

username: hond86
cooked: <p>Thanks for the prompt reply. Indeed, memory is not an issue, I have ~150m samples, so that is ~570MB, still feasible, the time-consuming part is to scan all the samples at the beginning of the training to get the class and create the weight tensor, but I guess I would need to find a smarter way to do that.</p>
<p>Thanks again.</p>

username: hond86
cooked: <p>Side note: the WeightedRandomSampler is based on torch.multinomial and I just realized that torch.multinomial doesn’t support arrays larger than 2^24 (&lt;17M)  samples.</p>
Why there are two LayerNorm consecutively in TransformerEncoder?

username: Mahdi_Amrollahi
cooked: <p>Why there are two LayerNorm consecutively in TransformerEncoder? What does the second one do? Also there are two dropout layer in the end which I do not understand the reason.</p>
<pre><code class="lang-auto">TransformerEncoder(
  (layers): ModuleList(
    (0): TransformerEncoderLayer(
      (self_attn): MultiheadAttention(
        (out_proj): NonDynamicallyQuantizableLinear(in_features=512, out_features=512, bias=True)
      )
      (linear1): Linear(in_features=512, out_features=2048, bias=True)
      (dropout): Dropout(p=0.1, inplace=False)
      (linear2): Linear(in_features=2048, out_features=512, bias=True)
      (norm1): LayerNorm((512,), eps=1e-05, elementwise_affine=True)
      (norm2): LayerNorm((512,), eps=1e-05, elementwise_affine=True)
      (dropout1): Dropout(p=0.1, inplace=False)
      (dropout2): Dropout(p=0.1, inplace=False)
    )
  )
)
</code></pre>
XLA-TPU - Extremely slow

username: pere
cooked: <p>I am trying XLA on the TPUs for the first time. Have been running all my other training on Flax. I need to move a Whisper finetuning script from GPU to TPU (and finally TPU Pods). I am using a simple TPU VM v4 .</p>
<p>I am able to set up the TPU. The sample scripts run at correct speed. I am also able to start my script without any errors. For me it seems that the xla-hook-script is initiated, and I set the variable “xla”=True in the Trainer script. No error messages. But everything is extremely slow. On a standard A6000 I am able to run around 16s/it. On the TPU I am running 800s/it! It is as if I am training on CPU.</p>
<p>Could anyone give me some basic tips in how to debug this? If you are interested, the script is available here: <a href="https://huggingface.co/NbAiLab/whisper" class="inline-onebox" rel="noopener nofollow ugc">NbAiLab/whisper · Hugging Face</a></p>

username: Liyang_Lu
cooked: <p>Hi pere,</p>
<p>Such a slow down with PyTorch/XLA usually indicates there are excessive recompilations or CPU fallbacks.</p>
<p>Could you trying out the steps in the <a href="https://github.com/pytorch/xla/blob/master/TROUBLESHOOTING.md" rel="noopener nofollow ugc">pytorch/xla troubleshooting guide</a>, especially the “Perform A Auto-Metrics Analysis” and the “Get A Metrics Report” parts. It’s usually a good start point to show if excessive recompilations or CPU fallbacks are happening.</p>
<p>Also, would you mind opening a github issue under <a href="https://github.com/pytorch/xla" class="inline-onebox" rel="noopener nofollow ugc">GitHub - pytorch/xla: Enabling PyTorch on Google TPU</a> for better tracking purpose?</p>
<p>Thanks,<br>
Liyang</p>

username: pere
cooked: <p>Thanks you. I have opened an issue here: <a href="https://github.com/pytorch/xla/issues/4203" class="inline-onebox" rel="noopener nofollow ugc">XLA-TPU - Extremely slow · Issue #4203 · pytorch/xla · GitHub</a></p>
Distribution Error: The value argument must be within the support

username: shivammehta007
cooked: <p>So, I found this cool Normalizing flow tutorial in PyTorch and I was trying the first tut itself link <a href="https://github.com/acids-ircam/pytorch_flows/blob/master/flows_01.ipynb" rel="noopener nofollow ugc">here</a></p>
<pre><code class="lang-auto">import torch.distributions as distrib
import torch.distributions.transforms as transforms


x = np.linspace(-4, 4, 1000)
z = np.array(np.meshgrid(x, x)).transpose(1, 2, 0)
z = np.reshape(z, [z.shape[0] * z.shape[1], -1])
</code></pre>
<p>And the transformations like</p>
<pre><code class="lang-auto"># Initial distribution
q0 = distrib.MultivariateNormal(torch.zeros(2), torch.eye(2))
# Defining Affine Transformation
f1 = transforms.ExpTransform()
# Transforming
q1 = distrib.TransformedDistribution(q0, f1)

---&gt; q1.log_prob(torch.Tensor(z))
</code></pre>
<p>The last line raises <code>ValueError: The value argument must be within the support</code> which is not the case in the notebook did something change? Because I tried running the whole notebook on colab and it broke there too and on my system too.<br>
Any help what might be wrong?</p>

username: Muhammad_Maaz
cooked: <p>Any workaround for this?</p>

username: M3114K
cooked: <p>Hello,</p>
<p>It is late for your work, but I hope it will be helpful for others.<br>
You can check this link : <a href="https://github.com/pytorch/pytorch/issues/59228" class="inline-onebox" rel="noopener nofollow ugc">ValueError: The value argument must be within the support · Issue #59228 · pytorch/pytorch · GitHub</a></p>
<p>you should just add (validate_args=False) :<br>
q0 = distrib.MultivariateNormal(torch.zeros(2), torch.eye(2),validate_args=False)</p>
<h1>
<a name="define-an-affine-transform-1" class="anchor" href="#define-an-affine-transform-1"></a>Define an affine transform</h1>
<p>f1 = transform.ExpTransform()<br>
q1 = distrib.TransformedDistribution(q0, f1, validate_args=False)</p>
Performance degradation with PyTroch compared to TF

username: Omega_Ma
cooked: <p>I have tried PyTorch (PT) and Tensorflow (TF) to implement the same idea of an unsupervised learning task. The network structures are the same. The training data keep the same distribution. And I use the same loss function and other hyper-parameters. The strange thing is that under the same number of training batches,  the testing performance obtained by TF is much better than that of the PT.  The testing parameters also keep the same.  I have tried training more times in PT but still got poor performance.  Have anyone observed such a phenomenon? I have checked the codes in PT and TF again and again, and there is nothing wrong with the codes. I really can’t understand why this happened.</p>

username: khushi-411
cooked: <p>Hey,<br>
Could you please share the minimal executable code?</p>
<p>Also, could you try profiling the code and share the outputs here? You can refer to <a href="https://pytorch.org/tutorials/recipes/recipes/profiler_recipe.html" rel="noopener nofollow ugc">this link</a> for PyTorch. It’s a great tutorial to profile models. I’m not sure about TensorFlow, please check the similar approach in their documentation.</p>
<p>Thanks!</p>
What's the difference between ToTensor and PILToTensor transforms?

username: f3ba
cooked: <p>What are the differences between the transforms:</p>
<pre><code class="lang-auto">torchvision.transforms.ToTensor
</code></pre>
<p>and</p>
<pre><code class="lang-auto">torchvision.transforms.PILToTensor
</code></pre>
<p>?</p>

username: srishti-git1110
cooked: <p><code>PILToTensor</code> transforms a PIL Image to a PyTorch tensor and the resulting tensor is of the same dtype, whereas <code>ToTensor</code> returns a FloatTensor after normalizing in the range [0, 1].</p>
<p>Please also refer to the official docs for better understanding.</p>
Pandas Dataframe for duplicate values

username: usama_meraj1
cooked: <p>Hello,</p>
<p>i have a very short query regarding pandas dataframe<br>
my dataframe looks like</p>
<hr>
<p>True Class, Predicted Class, Confi<br>
800 750 0.67<br>
800 800 0.42<br>
800 750 0.32<br>
800 750 0.98<br>
800 900 0.34<br>
800 900 0.45<br>
. . .<br>
. . .<br>
. . .<br>
800 750 0.77</p>
<hr>
<p>i would like to add the percentages of repeated prediction class values and plot predicted class vs (added percentage per prediction class) so 900 = (0.34+0.45) and then 800 = 0.42 etc etc if you know what i mean.</p>
<p>im struggling to group by this data could you please help me with this creating these two arrays by any chance.</p>
A way to detect whether code is executed within "with torch.no_grad():"

username: hubsiiiii
cooked: <p>Hello,</p>
<p>I wanted to ask whether there is a way to check if code is executed within a <code>with torch.no_grad():</code> block?<br>
I have the following problem: I am evaluating my model within <code>with torch.no_grad():</code>. However, my model executes <code>loss.backward()</code> in its forward pass since I want to adapt my deployed model on the unlabelled test data. After adapting my model, I want to evaluate it on the same data using my test function but I am getting the following error: “<em>RuntimError: element 0 of tensors does not require grad and does not have a grad_fn</em>”.</p>
<p>I assume this error is thrown because my model is calling <code>loss.backward()</code> within <code>with torch.no_grad():</code>.</p>
<p>Therefore, I need something like this</p>
<pre><code class="lang-auto">#Basic structure of my test function
def test(model, dataloader):
    with torch.no_grad():
         for data, label in dataloader:
             out = model(data) # this is calling my forward function of my model
             ...

#Custom Forward function of my model
def forward_mymodel(self, input):
     out = model(x)
     if not TODO:InTorchNoGrad:
        loss = softmax_entropy(out).mean(0)
        loss.backward()
        optimizer.step()
        optimizer.zero_grad()      
</code></pre>
<p>I guess, I could just check one of my input tensors and see if it requires grad but I was wondering if there is an elegant solution for this?</p>
<p>I would appreciate if someone could help me.</p>
<p>Thank you! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

username: ptrblck
cooked: <p><code>torch.is_grad_enabled()</code> should work:</p>
<pre><code class="lang-python">print(torch.is_grad_enabled())
# True

with torch.no_grad():
    print(torch.is_grad_enabled())
    # False
    
with torch.inference_mode():
    print(torch.is_grad_enabled())
    # False
</code></pre>

username: hubsiiiii
cooked: <p>Thank you so much for your fast response! <img src="https://discuss.pytorch.org/images/emoji/apple/partying_face.png?v=12" title=":partying_face:" class="emoji" alt=":partying_face:" loading="lazy" width="20" height="20"></p>
<p>It works! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
How do I convert a tensorflow model to pytorch?

username: YuhskeHujisaki
cooked: <p>How do I convert a tensorflow model to pytorch?</p><aside class="onebox githubblob" data-onebox-src="https://github.com/IoBT-VISTEC/MIN2Net/blob/main/min2net/model/MIN2Net.py">
  <header class="source">

      <a href="https://github.com/IoBT-VISTEC/MIN2Net/blob/main/min2net/model/MIN2Net.py" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/IoBT-VISTEC/MIN2Net/blob/main/min2net/model/MIN2Net.py" target="_blank" rel="noopener nofollow ugc">IoBT-VISTEC/MIN2Net/blob/main/min2net/model/MIN2Net.py</a></h4>


      <pre><code class="lang-py">from tensorflow.keras.layers import Concatenate, AveragePooling2D, BatchNormalization, Conv2D, Conv2DTranspose, Dense, Input, Reshape, Flatten
from tensorflow.keras.constraints import max_norm
from tensorflow.keras.models import Model
from tensorflow.keras.callbacks import CSVLogger, ModelCheckpoint, ReduceLROnPlateau
from tensorflow.keras.callbacks import EarlyStopping
from tensorflow.keras.optimizers import Adam
import tensorflow as tf
from tensorflow.keras import backend as K

import numpy as np
import os
import time
from sklearn.metrics import classification_report, f1_score
from min2net.loss import mean_squared_error, triplet_loss, SparseCategoricalCrossentropy
from min2net.utils import TimeHistory, compute_class_weight

class MIN2Net:
    def __init__(self,
                input_shape=(1,400,20), 
                num_class=2, 
</code></pre>



  This file has been truncated. <a href="https://github.com/IoBT-VISTEC/MIN2Net/blob/main/min2net/model/MIN2Net.py" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
I want to write MIN2Net with pytorch, how should I convert it?</p>

username: khushi-411
cooked: <p>Hi, You can check out <a href="https://pytorch.org/tutorials/beginner/blitz/neural_networks_tutorial.html" rel="noopener nofollow ugc">this tutorial</a> for an example!</p>
<p>Similar question (posting for reference that will help you): <a href="https://discuss.pytorch.org/t/how-to-convert-this-code-to-pytorch-format/165829" class="inline-onebox">How to convert this code to pytorch format</a>.</p>
<p>Feel free to ping me if you need further help or are stuck in error. Thanks!</p>
What's the best practice to make a tensor A that A[i]=B[C[i]]

username: SJoJoK
cooked: <p>I got tensor C (of size N ) and tensor B (of size M), and I want to make a tensor A which<br>
A[i]=B[C[i]],<br>
Actually I can use A = [B[x] for x in C], which create a list(of size N) of tensor(scalar) but not a tensor(of size N), and for-loop seems slow without vectorization.<br>
I think there must have a better way to do it, but i’m new to torch, so i’m looking for help.<br>
Thanks in advance.</p>

username: ptrblck
cooked: <p>Direct indexing should work:</p>
<pre><code class="lang-python">N, M = 4, 5

C = torch.randint(0, M, (N,))
B = torch.randn(M)

A_slow = torch.stack([B[x] for x in C])

A = B[C[torch.arange(C.size(0))]]

print((A_slow == A).all())
# tensor(True)
</code></pre>

username: SJoJoK
cooked: <p>That works, Thanks a lot!</p>
Convert Kitti labels to Yolo

username: Tornike
cooked: <p>Hi, Trying to convert Kitti label format to Yolo. But after converting the bbox is misplaced. this is kitti bounding box</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/b/bb4bdeda46afa7196bd0715a23b77262afae1e41.png" alt="download (1)" data-base62-sha1="qITQ0nLifwkxolnTpoaBmwXatPj" width="204" height="252"></p>
<p>this is conversion code:</p>
<pre><code class="lang-auto">def convertToYoloBBox(bbox, size):
# Yolo uses bounding bbox coordinates and size relative to the image size.
# This is taken from https://pjreddie.com/media/files/voc_label.py .
dw = 1. / size[0]
dh = 1. / size[1]
x = (bbox[0] + bbox[1]) / 2.0
y = (bbox[2] + bbox[3]) / 2.0
w = bbox[1] - bbox[0]
h = bbox[3] - bbox[2]
x = x * dw
w = w * dw
y = y * dh
h = h * dh
return (x, y, w, h)


convert =convertToYoloBBox([kitti_bbox[0],kitti_bbox[1],kitti_bbox[2],kitti_bbox[3]],image.shape[:2])
</code></pre>
<p>The function does some normalization which is essential for yolo and outputs following:</p>
<blockquote>
<p>(0.14763590391908976, 0.3397063758389261, 0.20452591656131477, 0.01810402684563757)</p>
</blockquote>
<p>but when i try to check if the normalization is being done correctly with this code:</p>
<pre><code class="lang-auto">x = int(convert[0] * image.shape[0])
y = int(convert[1] * image.shape[1])
width = x+int(convert[2] * image.shape[0]) 
height = y+ int(convert[3] * image.shape[1])

cv.rectangle(image, (int(x), int(y)), (int(width), int(height)), (255,0,0), 2 )
</code></pre>
<p>Bounding box is misplaced:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/9/c/9ca9a7bdac971b5a03baacdcc11c38c90ecd7ddd.png" alt="download (2)" data-base62-sha1="mlTY8UH1Krkx6JBXzrEBbGLVtSJ" width="204" height="252"></p>
<p>Any suggestions ? Is conversion fucntion correct? or the problem is in the checking code ?</p>

username: Albert1404
cooked: <p>I reckon the problem is located when doing</p>
<pre><code class="lang-python">x = int(convert[0] * image.shape[0])
y = int(convert[1] * image.shape[1])
</code></pre>
<p>As <code>convert[0]</code> contains the x position but <strong>centered</strong>. Same happens with <code>convert[1]</code>. This center position introduces the offset in the bbox you drawed.</p>

username: Albert1404
cooked: <p><strong>UPDATE</strong><br>
Based on your code, replace as follows</p>
<pre><code class="lang-python">x = int((convert[0] - convert[2]/2))*im.size[0]
y = int((convert[1] - convert[3]/2))*im.size[1]
width = int(convert[2]*im.size[0])
height = int(convert[3]*im.size[1])

cv.rectangle(image, (int(x), int(y)), (int(width), int(height)), (255,0,0), 2 )```</code></pre>
Libtorch equivalent of torch.cuda.memory_reserved()

username: harshitAgr
cooked: <p>Hi, I tried to find the equivalent of torch.cuda.memory_reserved() for libtorch but could not find it. I tried torch::cuda::memory_reserved() but it gives error: ‘memory_allocated’ is not a member of ‘torch::cuda’</p>

username: ptrblck
cooked: <p>You could try to use:</p>
<pre><code class="lang-python">const DeviceStats stats =
      c10::cuda::CUDACachingAllocator::getDeviceStats(device);
</code></pre>
<p>and then access its attributes:</p>
<pre><code class="lang-python">stats.reserved_bytes;
stats.allocated_bytes;
...
</code></pre>

username: harshitAgr
cooked: <p>Thanks for fast reply.<br>
One can follow this as well:  <a href="https://github.com/pytorch/pytorch/issues/85436" class="inline-onebox" rel="noopener nofollow ugc">C++ Memory Management API · Issue #85436 · pytorch/pytorch · GitHub</a></p>
<p>Best,</p>
How `torch.jit` optimizes the element-wise operation

username: fangwei123456
cooked: <p>I find that even if I write a CUDA kernel manually, the kernel is still slower than <code>torch.jit</code>. Thus, I am interested in how <code>torch.jit</code> optimizes the element-wise operation. Can the developers introduce some reasons about that?</p>

username: ptrblck
cooked: <p>In the current release scripted models use nvFuser to code-gen elementwise operations for CUDA kernel.<br>
<a href="https://pytorch.org/blog/introducing-nvfuser-a-deep-learning-compiler-for-pytorch/">This blog post</a>, <a href="https://pytorch.org/tutorials/intermediate/nvfuser_intro_tutorial.html">this tutorial</a>, and <a href="https://www.nvidia.com/en-us/on-demand/session/gtcspring21-s31952/">this GTC talk</a> give you more information about its technology.</p>
RuntimeError Inplace Operation in Recursive Structure

username: malioboro
cooked: <p>I create blocks of layers and combine them in a tree structure. The blocks are stored in a list:</p>
<pre><code class="lang-auto">blocks = [block0, block1, block2]
</code></pre>
<p>And then, the forward function will hierarchically use these blocks:</p>
<pre><code class="lang-auto">        -&gt; block1 -&gt; output1
block0-|
        -&gt; block2 -&gt; output2
</code></pre>
<p>When I implement this without recursion, it works:</p>
<pre><code class="lang-auto">    def forward(self, x):
        x = self.blocks[0](x)
        y1 = self.blocks[1](x)
        y2 = self.blocks[2](x)
        self.outputs = [y1, y2]
        return self.outputs
</code></pre>
<p>But, when I implement it in recursion, it raises a RuntimeError:</p>
<pre><code class="lang-auto">    def forward(self, x):
        x = self.blocks[0](x)
        self.traverse(0, x)
        return self.outputs

    def traverse(self, now, x):
        leaves = True
        if now*2+1 &lt; self.nodes:   # self.nodes for this case is 3
            y = self.blocks[now*2+1](x)
            self.traverse(now*2+1, y)
            leaves = False
        if now*2+2 &lt; self.nodes:   # self.nodes for this case is 3
            y = self.blocks[now*2+2](x)
            self.traverse(now*2+2, y)
            leaves = False
        if leaves:
            self.outputs.append(x)
</code></pre>
<p>It raises this error:</p>
<pre><code class="lang-auto">RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [100, 10]], which is output 0 of AsStridedBackward0, is at version 2; expected version 1 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!
</code></pre>
<p>My training part:</p>
<pre><code class="lang-auto">model.train()
with torch.autograd.detect_anomaly():
    for i in range(epoch):
        for image, label in tqdm(trainloaders):
            image = image.to(device)
            label = label.to(device)
            
            out = model(image)
            out = torch.mean(torch.stack(out), dim=0)
            loss = criterion(out, label)
    
            optimizer.zero_grad()
            loss.backward(retain_graph=True)
            optimizer.step()
</code></pre>
<p>Can someone help me understand why this error happens, please?</p>

username: malioboro
cooked: <p>I found the problem that I forgot to re-initialize the <code>self.outputs</code> in the recursion, it works now</p>