akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
ShufflerIterDataPipe but different random seed per distributed rank

AlbertZeyer: <p>As I understand the (somewhat obfuscated logic) of <code>DataLoader</code> iterators and specifically what this means to <code>ShufflerIterDataPipe</code> if I have that in my data pipeline:</p>
<p>Creating a new iterator on <code>DataLoader</code> will get a new random number from the current default Torch RNG, and use this as seed for all data pipes. This seed is broadcasted from rank 0 to all other workers, i.e. all ranks use the same seed. It then calls <code>torch.utils.data.graph_settings.apply_random_seed</code>, which will call <code>set_seed</code> on all random data pipes (i.e. also on my <code>ShufflerIterDataPipe</code>).</p>
<p>Calling <code>__iter__</code> on any of the data pipes will then call <code>reset()</code> (via a very obfuscated way…) and there actually use this seed and (re)initialize any own internal RNGs. This is also what <code>ShufflerIterDataPipe</code> does.</p>
<p>However, what I want is to have actually a different random seed per worker, or in general more control over the random seed, e.g. making it also dependent on the current epoch, or so. How would I do that?</p>
<p>I also understand that these data pipes are deprecated (probably for good reason, the code is quite difficult to read). So what would be the alternative? I need sth like <code>ShufflerIterDataPipe</code>.</p>
Performance regarding `group` argument in p2p comm

Fan_Hong: <p>In P2P communication APIs like <code>torch.distributed.send(tensor, dst, group=None, tag=0)</code>, no matter what <code>group</code> is, ranks have to be the ones in the global process group.</p>
<p>Suppose there are two ranks in a smaller subgroup than the global one. Are there any differences regarding performance if using the global group in the APIs compared to using the smaller subgroup?</p>
How to add value to tensor multiple times by index

dronnet1: <p>Hi all.</p>
<p>Example:</p>
<pre><code class="lang-auto">x = torch.zeros(10)
x
tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])

indices = torch.tensor([1,1,3,3,3])
</code></pre>
<p>and I want to add value by index</p>
<pre><code class="lang-auto">x[indices] += 1
</code></pre>
<p>I get this</p>
<pre><code class="lang-auto">x
tensor([0., 1., 0., 1., 0., 0., 0., 0., 0., 0.])
</code></pre>
<p>But I want to get</p>
<pre><code class="lang-auto">x
tensor([0., 2., 0., 3., 0., 0., 0., 0., 0., 0.])
</code></pre>
<p>May be this impossible ?</p>

Eduardo_Lawson: <p>Try doing like this:</p>
<pre><code class="lang-auto">x = torch.zeros(10)
indices = torch.tensor([1,1,3,3,3])
x.scatter_add_(0, indices, torch.ones_like(indices, dtype=torch.float32))
</code></pre>

dronnet1: <p>Thank you. Its really works.<br>
But how make it on 2d tensor ?<br>
Now I do it with double cycle ‘for’. Very slow.<br>
Is it possible to add values ​​to a tensor using broadcast?</p>
<pre><code class="lang-auto">x = torch.zeros(5, 5)
x
tensor([[0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0.]])
</code></pre>
<pre><code class="lang-auto">indices = torch.tensor([[2, 1],
                        [2, 1],
                        [2, 1],
                        [1, 1],
                        [1, 1]
                       ], 
                       dtype=torch.int)
</code></pre>
<p>And need to broadcast add value by each index</p>
<pre><code class="lang-auto">x[indices] += 1
</code></pre>
<p>it should turn out like this</p>
<pre><code class="lang-auto">x
tensor([[0., 0., 0., 0., 0.],
        [0., 1., 0., 0., 0.],
        [0., 3., 0., 0., 0.],
        [0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0.]])

</code></pre>
<p>Maybe you know how to do this without cycles?</p>
Persistent error in loss.backward() during joint-training of two models using two different but dependent loss functions

Goutam: <p>I am trying to train a joint model for a crowd counting network, I ma computing two losses, one is loss_predict (depends on counter output and refiner output), one is loss_refine(depends on refiner output and ground truth). I want to train the model in a way that : Counter is updated using loss_predict only, but Refiner is updated using a weighted sum of loss_predict &amp; loss_refine. So i am trying to train them one by one, first the counter then refiner. But getting the error message as :<br>
" Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed). Saved intermediate values of the graph are freed when you call .backward() or autograd.grad(). Specify retain_graph=True if you need to backward through the graph a second time or if you need to access saved tensors after calling backward." I also tried using retain_graph = True. Still the error says, some component needed to compute gradient is updated, so error.<br>
Any suggestion how to resolve this?? Any help will be appreciated.</p>
<p>Training code :</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/d/3db4085e6be45f34ad9d0a6ff92d9f7a375fa71d.png" data-download-href="https://discuss.pytorch.org/uploads/default/3db4085e6be45f34ad9d0a6ff92d9f7a375fa71d" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/d/3db4085e6be45f34ad9d0a6ff92d9f7a375fa71d_2_690x447.png" alt="image" data-base62-sha1="8NQQkaVgAkEn8Vd46YwAlKczreB" width="690" height="447" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/d/3db4085e6be45f34ad9d0a6ff92d9f7a375fa71d_2_690x447.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/d/3db4085e6be45f34ad9d0a6ff92d9f7a375fa71d_2_1035x670.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/d/3db4085e6be45f34ad9d0a6ff92d9f7a375fa71d_2_1380x894.png 2x" data-dominant-color="E8EBED"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1679×1090 149 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Layer Normalization gradients

Nectralyx: <p>I am attempting to create my own custom Layer Normalization layer, and I intend on my implementation working identically to PyTorch’s nn.layerNorm. However, when I try to recreate the layer, I always get slightly different gradients for the input. This is the script I am using:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

class CustomLayerNorm(nn.Module):
    def __init__(self, normalized_shape, eps=1e-5):
        super(CustomLayerNorm, self).__init__()
        # Define learnable parameters gamma (scale) and beta (shift)
        self.gamma = nn.Parameter(torch.ones(normalized_shape))
        self.beta = nn.Parameter(torch.zeros(normalized_shape))
        self.eps = eps

    def forward(self, x):
        # Calculate mean and variance along dimension 2
        mean = x.mean(dim=2, keepdim=True)
        variance = x.var(dim=2, unbiased=False, keepdim=True)
        
        # Normalize the input
        x_normalized = (x - mean) / torch.sqrt(variance + self.eps)
        
        # Scale and shift
        out = self.gamma * x_normalized + self.beta
        return out

# Initialize custom layer norm and PyTorch's layer norm
custom_layer_norm = CustomLayerNorm(normalized_shape=(5,))
pytorch_layer_norm = nn.LayerNorm(normalized_shape=(5,))

# Set PyTorch's gamma and beta to match our custom layer norm

# Define the input tensor
x = torch.tensor(
    [[[76.1738, 77.1738, 76.1738, 77.1738, 76.1738],
      [77.0152, 76.7141, 76.1989, 77.1735, 76.1744],
      [77.0831, 75.7576, 76.2240, 77.1725, 76.1750],
      [76.3149, 75.1838, 76.2491, 77.1709, 76.1757],
      [75.4170, 75.5201, 76.2741, 77.1687, 76.1763]]], requires_grad=True
)

# Forward pass for both layers
custom_output = custom_layer_norm(x)
pytorch_output = pytorch_layer_norm(x)

# Compare the output of the custom layer to the PyTorch layer
print("Custom LayerNorm Output:\n", custom_output)
print("PyTorch LayerNorm Output:\n", pytorch_output)

# Backward pass
custom_output.sum().backward()
print("\nInput Gradients - Custom LayerNorm:\n", x.grad)

pytorch_output.sum().backward()

# Compare gradients

print("Input Gradients - PyTorch LayerNorm:\n", x.grad)
</code></pre>
<p>Am I doing this wrong? Why don’t the input gradients match?</p>
Kernel for Pytorch Linear Regression appears to have died

nagarajvsn: <p>I am using the following simple code to start with</p>
<p>import numpy as np<br>
import matplotlib.pyplot as plt<br>
import torch<br>
w=torch.tensor(-10.0,requires_grad=True)<br>
x=torch.arange(-3,3,0.1).view(-1,1)<br>
f=-3*x</p>
<p>plt.plot(x.numpy(),f.numpy())<br>
plt.show()</p>
<p>when I try to plot, I get an error message "Kernel for Pytorch apperas to have dies. It will restyart automatically</p>
<p>I tried uninstalling torchvision and reinstalling the same, but error persists</p>

ptrblck: <p>Try to run the code in a terminal to get a proper error message.</p>
Initializating parameter with registered parametrization

octave: <p>Hello, I’m having trouble initializing a module that contains parameters with registered parametrization.</p>
<p><em>My understanding:</em><br>
The initialization functions in <code>torch.nn.init</code> modify the tensors in place. When using parametrization the attribute becomes a property and the tensor returned by the getter method is the output of the parametrization’s <code>forward</code> call: modifying this tensor in place doesn’t modify the original tensor values and doesn’t trigger the call to <code>right_inverse</code>.</p>
<p>A trivial examples with a ReLU parametrization applied to the weight of a Linear layer:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.nn.utils.parametrize as parametrize

class ReLUParametrization(torch.nn.Module):
    def forward(self, x: torch.Tensor):
        return torch.nn.functional.relu(x)
    
    def right_inverse(self, x: torch.Tensor):
        return torch.clamp(x, 0)

class MyModule(torch.nn.Linear):
    def __init__(self, in_features: int, out_features: int) -&gt; None:
        super().__init__(in_features, out_features, bias=False, device=None, dtype=None)
        torch.nn.utils.parametrize.register_parametrization(self, "weight", ReLUParametrization())

def initialize(module: torch.nn.Module):
    if isinstance(module, MyModule):
        torch.nn.init.uniform_(module.weight, 0, 1)
</code></pre>
<pre><code class="lang-auto">if __name__ == "__main__":
    my_module = MyModule(3, 1)
    print(my_module.weight)
    my_module.apply(initialize)
    print(my_module.weight)

# tensor([[0.2571, 0.0000, 0.3429]], grad_fn=&lt;ReluBackward0&gt;)
# tensor([[0.2571, 0.0000, 0.3429]], grad_fn=&lt;ReluBackward0&gt;)
</code></pre>
<p>Could you please explain the recommended way to initialize parameters with registered parametrization ?</p>

ptrblck: <p>I’m unsure if I understand your use case correctly, but you could try to initialize the original parameter via <code>module.parametrizations.weight.original</code>.</p>

octave: <p>Thank you for the help, I’ll look into that!</p>
How to parallelise this multidimensional indexing

Vassil17: <p>I have tensor pos which is (N,M,L,3), a second tensor env_scans (which is a grid) of size (N,K,3), and a third tensor indices (N,M,L) which contains indices (from 0 to K-1). I want to do the following:</p>
<pre><code class="lang-auto">  for n in range(N):
      for m in range(M):
          for l in range(L):
              pos[n,m,l, 2] = env_scans[n, indices[n, m, l], 2]
</code></pre>
<p>Is there a way to vectorise this to avoid the for loops? So far I managed to vectorise across the batch dimension (N):</p>
<pre><code class="lang-auto">for m in range(M):
        for l in range(L):
                pos[:,m, l, 2] = env_scans[torch.arange(N),indices[:,m, l], 2]
</code></pre>
<p>and this works, but I’m not sure how to expand it further for the remaining dimensions as well. I’d appreciate any clues/help!</p>
Nvidia-drivers, cuda version and pytorch version compatibility

zkining: <p>I have 4 A100 graphics cards in the lab</p>
<ul>
<li>GPU driver is 470.256.02</li>
<li>cuda version is 11.4</li>
<li>pytorch version is 1.13.<br>
But now I want to use functions such as torch.compile() which need pytorch verision &gt;2.0.0. Since the GPU driver in the lab cannot be updated, the GPU driver is still 470.256.02.</li>
</ul>
<ol>
<li>One way is to install cuda 11.8. Although the nvidia official website states that GPU drivers &gt;450 are compatible with 11.x versions of cuda, some functions are lost. I am not sure whether there will be compatibility issues if I use GPU driver 470.256.02, cuda version 11.8 and pytorch version 2.1.</li>
<li>The second method is to keep cuda version 11.4 and directly install pytorch 2.1 that supports cuda 11.8.<br>
I would like to ask if there is a better way to use the latest version of pytorch without updating the graphics card driver.</li>
</ol>

ptrblck: <aside class="quote no-group" data-username="zkining" data-post="1" data-topic="212567">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/z/2bfe46/48.png" class="avatar"> zkining:</div>
<blockquote>
<p>I would like to ask if there is a better way to use the latest version of pytorch without updating the graphics card driver.</p>
</blockquote>
</aside>
<p>Your second approach should work, but I would highly recommend updating the drivers to any one supporting CUDA 12.x applications as the drivers were released at the beginning of 2023 already.</p>
How to use Mosaic streaming with TimeseriesDataset and PyTorch Forecasting?

KornkamonGib: <p>Hi, I’m working on forecasting daily product demand for a retailer using the Temporal Fusion Transformer (TFT) model from the PyTorch Forecasting library. My dataset is huge, with around 10 billion rows, and is stored in a Spark DataFrame on Databricks.</p>
<p>I’m encountering an issue when I try to convert the Spark DataFrame to Pandas, resulting in a “driver out of memory” error. To tackle this, I attempted to use Mosaic streaming for efficient data loading, but the challenge is that the TimeseriesDataset in PyTorch Forecasting requires a Pandas DataFrame.</p>
<p>Could anyone guide me on how to use Mosaic streaming with the TimeseriesDataset and PyTorch Forecasting? If possible, I’d greatly appreciate an example to understand how to apply it.</p>
<p>Thank you in advance!</p>

akoetsier: <p><a class="mention" href="/u/kornkamongib">@KornkamonGib</a> have you been able to solve this?</p>
Issue With Forward Hooks in Deterministic Multi-GPU training

Tanay_Arora: <p>Hello. I have been trying to debug this issue for a while but have not gotten very far. I have achieved multi-gpu training (with DDP) through the following launch file:</p>
<pre><code class="lang-auto">import torch
import torch.multiprocessing as mp
import torch._dynamo as dynamo
import torch.distributed as dist

import os
import sys

import random

def setup_distribute(rank, world_size): 
    os.environ['MASTER_ADDR'] = 'localhost'
    os.environ['MASTER_PORT'] = '12355'
    dist.init_process_group("nccl", rank = rank, world_size = world_size)

def cleanup_distribute():
    dist.destroy_process_group()

def dist_wrapper(rank, world_size, func, name: str, SEED: int):
    setup_distribute(rank, world_size)
    torch.cuda.set_device(rank)
    set_dynamo_cfg()
    reset_deterministic(SEED)
    try:
        func(rank, world_size, name)
    finally:
        cleanup_distribute()

def set_dynamo_cfg():
    dynamo.config.capture_scalar_outputs = True
    dynamo.config.cache_size_limit = 256
    dynamo.config.guard_nn_modules = True
    dynamo.config.suppress_errors = True

def set_non_deterministic():
    torch.backends.cudnn.benchmark = True

def reset_deterministic(SEED):
    import numpy as np
    random.seed(SEED)
    np.random.seed(SEED)
    torch.manual_seed(SEED)
    torch.use_deterministic_algorithms(True)
    torch.backends.cudnn.benchmark = False
    torch.backends.cudnn.deterministic = True

def main(func, name: str, SEED: int):
    world_size = torch.cuda.device_count()
    mp.spawn(dist_wrapper, args=(world_size, func, name, SEED), nprocs=world_size, join=True)
</code></pre>
<p>This all works fine (note that DistributedSampler is also seeded properly). However, I start to run into issues with forward hooks. Specifically, when i attach the following hooks:</p>
<pre><code class="lang-auto">    def _activation_hook(self, module, input, output: torch.Tensor) -&gt; None:
        self.act_w.append(output.clone().detach().cpu().cuda().to(torch.float64).mean(dim = 0).view(-1))
        return
    
    def _fake_activation_hook(self, module, input, output: torch.Tensor) -&gt; None:
        self.act_w.append(F.relu(output.clone().detach().cpu().cuda()).to(torch.float64).mean(dim = 0).view(-1))
</code></pre>
<p>Through testing, I have noticed that the gradients are slightly different across instances with the hooks and without them.</p>
<p>Note that the device transfers part is not working either. It was inspired by noticing that if I just transfered everything to cpu, it would remain deterministic. However, without any device transfers or with the one to cpu and back to cuda, it is still not deterministic. The rest of the training (like initialization and data) is still remaining deterministic.</p>
<p>Furthermore, if I run it with and without the hook, calling a simple torch.randint(0, 100000, (1,)) (on either device), this still remains the same. Can anyone help me out?</p>
Asynchronous execution multiple GPUs

Manvel_Petrosyan: <p>Dear friends, I am using pytorch for linear algebra task to accelerate some calculations with GPUs. I have some function which do some calculations with given two tensors for example A and B. I have created two instances of this function with two pairs of tensors allocated on two different GPUs</p>
<p>some_fun(Tensor_A1_GPU0,Tensor_B1_GPU0,GPU_0) # First instance<br>
some_fun(Tensor_A2_GPU1,Tensor_B2_GPU1,GPU_1) # Second instance</p>
<p>As I understand pytorch by default executes commands asynchronous, but for my case for some reason it waits first instance to be done only than it runs second one, even though they are completely independent. What could be the cause for this behavior?</p>

ptrblck: <p><code>some_fun</code> might be synchronizing the CPU and is thus blocking it from advancing and scheduling the second <code>some_fun</code> call.<br>
You could use <a href="https://pytorch.org/docs/stable/generated/torch.cuda.set_sync_debug_mode.html"><code>torch.cuda.set_sync_debug_mode</code></a> to debug if any synchronizing calls are used in <code>some_fun</code>.</p>

Manvel_Petrosyan: <p>Thank You, I will try.</p>
Is it possible to see the total amounts of memory reads and writes

yabber303: <p>I am using the Pytorch memory profiling functions to analyse my model. I can see all memory allocations using the export_chrome_trace function of torch.profiler. I was wondering whether this contains all memory transactions or does Pytorch use the same allocation more than once?</p>

ptrblck: <p>PyTorch uses a caching allocator and is thus reusing the already allocated memory if possible. If you are interested in the actual memory reads and writes on a specific CUDA kernel you could profile it as described e.g. <a href="https://docs.nvidia.com/cuda/profiler-users-guide/#memory-statistics">here</a>.</p>

yabber303: <p>Thank you for your response. Is there also such a tool for actual memory reads and writes when using only an intel CPU?</p>
Autograd on a specific layer's parameters

Klae_zhou: <p>Hi, I am trying to get the Jacobian matrix of a specific layer’s parameters. The below is my network model and i apply functional_call on it.</p>
<pre><code class="lang-auto">def fm(params,input):
                    return functional_call(self.model,params,input.unsqueeze(0)).squeeze(0)
                
def floss(func_params,input):
                    fx = fm(func_params, input)
                    return fx
</code></pre>
<p>and I used to compute the Jacobian of all the parameters by this way</p>
<pre><code class="lang-auto">func_params=dict(self.model.named_parameters())
per_sample_grads =vmap(jacrev(floss,0), (None, 0))(func_params, input)
</code></pre>
<p>Right now, I  need just get the gradient on a specific layer’s parameters, here is my approach.</p>
<pre><code class="lang-auto">def grad(f,param):
            return torch.autograd.grad(f,param)    
out = vmap(floss,(None,0))(func_params,input)
gradf=vmap(grad,(0,None))(out,func_params['model.0.weight'])
</code></pre>
<p>However, the error saying “element 0 of tensors does not require grad and does not have a grad_fn”<br>
Because, i have tried</p>
<pre><code class="lang-auto">grad=self.grad(out[0],func_params['model.0.weight'])
</code></pre>
<p>and it works. I don’t really know how to fix this problem.</p>
Pytorch Weighted Regression Loss Functions

Ajaikrish: <p>Is there an built-in weighted loss function for regression tasks? If there are third party weighted loss functions, please let me know.</p>
<p>I am trying to use Weighted Mean Absolute Error for Unet  for semantic segmentation. I need to assign weights so that my model ignores background and focuses on poorly predicted regions.</p>

ptrblck: <p>You can use <code>nn.L1Loss</code> with <code>reduction='none'</code> and add your weights directly to the computed loss before reducing it.<br>
Adding an internal weight for floating point inputs does not make sense as this loss function does not have any class knowledge (e.g. what “background” means for your use case).</p>

Ajaikrish: <p>Thank you for your response.</p>
<p>May be, I can explain it better. I would like to have weighted loss such that if the target has a value of range 1 to 0.  I would like to have lower weights for targets with value 1.0 (background) and increase the weights as the target value reduces from 1.</p>
<p>I wrote a custom loss function for this. Please let me know if there are any other suggestions.</p>
<p>class WeightedMAELoss (nn.Module):</p>
<pre><code>def __init__(self, beta=1.0, reduction='mean'):
    super(WeightedMAELoss, self).__init__()
    self.beta = beta
    self.reduction = reduction

def forward(self, inputs, targets):
    diff = torch.abs(inputs-targets)

    with torch.no_grad():
        weights = torch.exp(-self.beta * targets)
    weighted_diff = weights * diff

    if self.reduction == 'mean':
        return weighted_diff.mean()
    elif self.reduction == 'sum':
        return weighted_diff.sum()
    else:
        return weighted_diff
</code></pre>

ptrblck: <p>You might want to normalize the loss e.g. by dividing by the <code>weights.sum()</code> as otherwise the target distribution (the frequency of 0s and 1s) would influence the loss scale.</p>
DDP barrier has no effect when more than one is used

seer_mer: <p>When using barrier in multi-gpu DDP (I am using <code>torchrun --standalone --nnodes=1 --nproc_per_node=gpu example.py</code> on 6 gpus machine), I expect that whenever I use barrier, all ranks should be blocked until all ranks arrive this barrier. However, only the first barrier successfully blocked all processes, the second barrier just falls through as if it does not exist. See example:</p>
<pre><code class="lang-auto"># example.py
import time
from typing import Tuple

from torch import distributed as dist


def ddp_setup() -&gt; Tuple[int, int]:
    dist.init_process_group("nccl")
    world_size = dist.get_world_size()
    rank = dist.get_rank()
    return rank, world_size


def ddp_cleanup() -&gt; None:
    dist.destroy_process_group()


def main():
    rank, world_size = ddp_setup()

    if rank == 0:
        time.sleep(1)
        for i in range(3):
            time.sleep(1)
            print(f"[RANK {rank}] 1-{i}", flush=True)
        time.sleep(1)

    print(f"[RANK {rank}] before barrier1", flush=True)
    dist.barrier()
    print(f"[RANK {rank}] after barrier1", flush=True)

    time.sleep(3)

    if rank == 0:
        time.sleep(1)
        for i in range(3):
            time.sleep(1)
            print(f"[RANK {rank}] 2-{i}", flush=True)
        time.sleep(1)

    print(f"[RANK {rank}] before barrier2", flush=True)
    dist.barrier()
    print(f"[RANK {rank}] after barrier2", flush=True)

    ddp_cleanup()


if __name__ == "__main__":
    main()
</code></pre>
<p>output:</p>
<blockquote>
<p>[RANK 5] before barrier1<br>
[RANK 4] before barrier1<br>
[RANK 1] before barrier1<br>
[RANK 3] before barrier1<br>
[RANK 2] before barrier1<br>
[RANK 0] 1-0<br>
[RANK 0] 1-1<br>
[RANK 0] 1-2<br>
[RANK 0] before barrier1<br>
[RANK 4] after barrier1<br>
[RANK 2] after barrier1<br>
[RANK 5] after barrier1<br>
[RANK 1] after barrier1<br>
[RANK 0] after barrier1<br>
[RANK 3] after barrier1<br>
[RANK 4] before barrier2<br>
[RANK 4] after barrier2<br>
[RANK 2] before barrier2<br>
[RANK 2] after barrier2<br>
[RANK 5] before barrier2<br>
[RANK 5] after barrier2<br>
[RANK 1] before barrier2<br>
[RANK 1] after barrier2<br>
[RANK 3] before barrier2<br>
[RANK 3] after barrier2<br>
[RANK 0] 2-0<br>
[RANK 0] 2-1<br>
[RANK 0] 2-2<br>
[RANK 0] before barrier2<br>
[RANK 0] after barrier2</p>
</blockquote>
<p>Notice that during barrier1, all ranks are waiting for rank0 to complete its task, and “after barrier1” is displayed after “[RANK 0] before barrier1”, which is <strong>as expected</strong>. However, for barrier2, which does the exact same thing, “after barrier2” for ranks 1-5 are displayed <strong>before</strong> rank 0 does its task. (note: adding even more barrier also has no effect, only first barrier works)</p>
<p>Why is this happening? How can I use barriers multiple times?</p>
<p>Python 3.11.10<br>
torch                   2.4.1<br>
torchaudio              2.4.1<br>
torchvision             0.19.1</p>
<p>OS: Ubuntu 18.04.1</p>
Fasterrcnn_mobilenet_v3_large_fpn for binary class classification

shanalikhan: <p>Hi,<br>
Please suggest how to change the last layers to tweak the model <code>fasterrcnn_mobilenet_v3_large_fpn</code>  for binary class only - e.g 0 for background and 1 for defect.</p>
<p>Here is the sample code.</p>
<pre><code class="lang-auto">import numpy as np
from torchmetrics.detection import IntersectionOverUnion
from torchmetrics.detection import MeanAveragePrecision
import math

class CocoDNN(L.LightningModule):
    def __init__(self):
        super().__init__()
        self.model = models.detection.fasterrcnn_mobilenet_v3_large_fpn(weights="DEFAULT")

        self.metric = MeanAveragePrecision(iou_type="bbox",average="macro",class_metrics = True, iou_thresholds=[0.5, 0.75],extended_summary=True, backend="faster_coco_eval")  


    def forward(self, images, targets=None):
        return self.model(images, targets)

    def training_step(self, batch, batch_idx):
        imgs, annot = batch
        batch_losses = []
        for img_b, annot_b in zip(imgs, annot):
            #print(len(img_b), len(annot_b))
            if len(img_b) == 0:
                continue
            loss_dict = self.model(img_b, annot_b)
            losses = sum(loss for loss in loss_dict.values())
            #print(losses)
            batch_losses.append(losses)
        batch_mean  = torch.mean(torch.stack(batch_losses))
        self.log('train_loss', batch_mean, on_step=True, on_epoch=True, prog_bar=True, logger=True)
        return batch_mean
    
    def validation_step(self, batch, batch_idx):
        imgs, annot = batch
        targets ,preds = [], []
        for img_b, annot_b in zip(imgs, annot):
            if len(img_b) == 0:
                continue
            if len(annot_b)&gt; 1:
                targets.extend(annot_b)
            else:
                targets.append(annot_b[0])

            #print(f"Annotated : {len(annot_b)} - {annot_b}")
            #print("")
            loss_dict = self.model(img_b, annot_b)
        
            #print(f"Predicted : {len(loss_dict)} -  {loss_dict}")
            if len(loss_dict)&gt; 1:
                preds.extend(loss_dict)
            else:
                preds.append(loss_dict[0])
            #preds.append(loss_dict)

        self.metric.update(preds, targets)
        map_results = self.metric.compute()
        #self.log_dict('logs',map_results)
        
        print(map_results)
        #print(map_results['map_50'].float().item())
        self.log('recall', map_results['recall'].mean().float().item(),on_step=True, on_epoch=True, prog_bar=True, logger=True)
        self.log('map_50', map_results['map_50'].float().item(),on_step=True, on_epoch=True, prog_bar=True, logger=True)
        self.log('map_75', map_results['map_75'].float().item(),on_step=True, on_epoch=True, prog_bar=True, logger=True)
        return map_results['map_75']

    def configure_optimizers(self):
        return optim.SGD(self.parameters(), lr=0.001, momentum=0.9, weight_decay=0.0005)
</code></pre>
Torch.bfloat16 < how does it work? in bf 16 model

minjun_jo: <p>I want to experiment on fp32 model, int8 model, and bf16 model, how can I get bf16??</p>
<p>One of the ways I found it is to make the fp32 model bf16 with torch.bfloat16.</p>
<p>I’m wondering if it’s expressed as bf16 using the fp32 value value as it is, and what specific rule does it count as. Also, I don’t know if I can specify this as bf16 in paper (without fine tuning)</p>
<pre><code class="lang-auto">model_squeezenet = models.squeezenet1_1(pretrained=True)
model_squeezenet.eval()

# Convert to BF16 and move to GPU if available
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model_squeezenet = model_squeezenet.to(device).to(torch.bfloat16)```</code></pre>

Vasiliy_Kuznetsov: <blockquote>
<p>I’m wondering if it’s expressed as bf16 using the fp32 value value as it is, and what specific rule does it count as.</p>
</blockquote>
<p>When you convert a model from float32 to bfloat16 with <code>model.to(torch.bfloat16)</code>, every parameter and buffer will be cast to bfloat16, with rounding to nearest even.  If you are interested in the details, the code is here: <a href="https://github.com/pytorch/pytorch/blob/d3fc13a9dd186ceb8d1b56b0968a41686ea645cd/c10/util/BFloat16-inl.h#L35" class="inline-onebox" rel="noopener nofollow ugc">pytorch/c10/util/BFloat16-inl.h at d3fc13a9dd186ceb8d1b56b0968a41686ea645cd · pytorch/pytorch · GitHub</a> .</p>