akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Processing Sequences with Linear layers Behaviour

nicaushaz: <p>Hi there.</p>
<p>I have implemented a sequential neural network that has an MLP feature extractor prior to the recurrent blocks. Intuitively I found out that the torch.nn.Linear can handle data of shapes (L, B, N)<br>
where L is the time-series length, B is the batch size and N is the feature shape.<br>
For instance the following block works:</p>
<pre><code class="lang-auto">rand_tens = torch.randn(12, 4, 16)
linear_layer = torch.nn.Linear(16, 8)
output = linear_layer(rand_tens)
</code></pre>
<p>and its output is torch.Size([12, 4, 8]).</p>
<p>However, normally we would require flattening of the temporal dimension and the batch dimension, allowing for the Linear layer to operate at a shape (L*B, N). My questions are the following:</p>
<ol>
<li>Is this flattening (and unflattening) what torch.nn.Linear does under the hood?</li>
<li>Is my thought process for processing  (L, B, N) shaped data at the feature dimension with linear layers correct?</li>
</ol>
<p>If you could point me a reference for that or the part of the documentation that describes it I would be greatful.</p>
<p>Thank you!</p>

ptrblck: <aside class="quote no-group" data-username="nicaushaz" data-post="1" data-topic="206515">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/n/49beb7/48.png" class="avatar"> nicaushaz:</div>
<blockquote>
<p>Is this flattening (and unflattening) what torch.nn.Linear does under the hood?</p>
</blockquote>
</aside>
<p>Yes, this should be the case as seen <a href="https://github.com/pytorch/pytorch/blob/6d9f74f0af54751311f0dd71f7e5c01a93260ab3/aten/src/ATen/native/Linear.cpp#L102C14-L108">here</a> and <a href="https://github.com/pytorch/pytorch/blob/6d9f74f0af54751311f0dd71f7e5c01a93260ab3/aten/src/ATen/native/Linear.cpp#L55-L70">here</a>.</p>
I am observing different results when using `nn.Linear` on different GPU configurations with the same input data

wanxin: <h3><a name="environment-1" class="anchor" href="#environment-1"></a>Environment</h3>
<ul>
<li><strong>PyTorch Version:</strong> 1.8.1+cu111</li>
<li><strong>GPU:</strong> A30 mig12g and mig6g</li>
<li><strong>Python Version:</strong> 3.8.8</li>
</ul>
<h3><a name="steps-to-reproduce-2" class="anchor" href="#steps-to-reproduce-2"></a>Steps to Reproduce</h3>
<ol>
<li>Initialize a model with <code>nn.Linear</code> layer.</li>
<li>Perform a forward pass with the same input data on different GPU configurations.</li>
<li>Observe different output features.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/8/b/8b50a368c80c81daf1ae4ecda0dfdcbd0e7b2fae.png" alt="image" data-base62-sha1="jSr8nFOvi1gC9zmjDRwiNUBeb1A" width="490" height="426"></li>
</ol>
<p>what is the reason?</p>

KFrank: <p>Hi Wanxin!</p>
<aside class="quote no-group" data-username="wanxin" data-post="1" data-topic="206442" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/w/c6cbf5/48.png" class="avatar"> wanxin:</div>
<blockquote>
<ol start="3">
<li>Observe different output features.</li>
</ol>
</blockquote>
</aside>
<p>You can’t expect exact equality across different architectures (or versions,<br>
gpu vs. cpu, batch sizes, etc.)  Your results differ by amounts consistent<br>
with floating-point round-off error, as is to be expected.</p>
<p>Best.</p>
<p>K. Frank</p>

wanxin: <p>Thank you for your response.<br>
You are correct, but I still have a question. When I only use convolutional layers, this situation does not occur. Is it because the underlying implementation mechanisms of linear layers and convolutional layers are different?</p>

KFrank: <p>Hi Wanxin!</p>
<aside class="quote no-group" data-username="wanxin" data-post="3" data-topic="206442" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/w/c6cbf5/48.png" class="avatar"> wanxin:</div>
<blockquote>
<p>When I only use convolutional layers, this situation does not occur. Is it because the underlying implementation mechanisms of linear layers and convolutional layers are different?</p>
</blockquote>
</aside>
<p>Well, yes, strictly speaking, linear and convolutional layers are different.<br>
But I don’t think that’s really the point.</p>
<p>I fully believe that on some sets of architectures, you will get round-off-error<br>
discrepancies with convolutional layers, just as you saw with linear layers.<br>
I would say that it <em>happened to be the case</em> that for your particular<br>
convolutional layers and your particular gpu configurations, pytorch chose<br>
to use the same orderings of floating-point operations and therefore no<br>
discrepancy arose.</p>
<p>That is, what you saw was happenstance, rather than convolutional layers<br>
having some special behavior.</p>
<p>Best.</p>
<p>K. Frank</p>
Compilation error for Pytorch 2.5 with Numpy 1.24

mahmoodn: <p>Hi,<br>
I am trying to install Pytorch 2.5 (by cloning the repo) with numpy 1.24.6, Python 3.12, GCC-12.3 and CUDA 12.2 on a Conda environment.<br>
In the latest steps, it terminates with the following error:</p>
<pre><code class="lang-auto">[ 96%] Building CXX object test_jit/CMakeFiles/test_jit.dir/test_upgrader_utils.cpp.o
[ 96%] Building CXX object test_lazy/CMakeFiles/test_lazy.dir/test_ir_util.cpp.o
In file included from /usr/include/c++/12/bits/stl_uninitialized.h:63,
                 from /usr/include/c++/12/memory:65,
                 from /home/mnaderan/pytorch/third_party/googletest/googletest/include/gtest/gtest.h:57,
                 from /home/mnaderan/pytorch/test/cpp/api/dataloader.cpp:1:
In static member function ‘static _Tp* std::__copy_move&lt;_IsMove, true, std::random_access_iterator_tag&gt;::__copy_m(const _Tp*, const _Tp*, _Tp*) [with _Tp = long unsigned int; bool _IsMove = false]’,
    inlined from ‘_OI std::__copy_move_a2(_II, _II, _OI) [with bool _IsMove = false; _II = const long unsigned int*; _OI = long unsigned int*]’ at /usr/include/c++/12/bits/stl_algobase.h:495:30,
    inlined from ‘_OI std::__copy_move_a1(_II, _II, _OI) [with bool _IsMove = false; _II = const long unsigned int*; _OI = long unsigned int*]’ at /usr/include/c++/12/bits/stl_algobase.h:522:42,
    inlined from ‘_OI std::__copy_move_a(_II, _II, _OI) [with bool _IsMove = false; _II = __gnu_cxx::__normal_iterator&lt;const long unsigned int*, vector&lt;long unsigned int&gt; &gt;; _OI = __gnu_cxx::__normal_iterator&lt;long unsigned int*, vector&lt;long unsigned int&gt; &gt;]’ at /usr/include/c++/12/bits/stl_algobase.h:529:31,
    inlined from ‘_OI std::copy(_II, _II, _OI) [with _II = __gnu_cxx::__normal_iterator&lt;const long unsigned int*, vector&lt;long unsigned int&gt; &gt;; _OI = __gnu_cxx::__normal_iterator&lt;long unsigned int*, vector&lt;long unsigned int&gt; &gt;]’ at /usr/include/c++/12/bits/stl_algobase.h:620:7,
    inlined from ‘std::vector&lt;_Tp, _Alloc&gt;&amp; std::vector&lt;_Tp, _Alloc&gt;::operator=(const std::vector&lt;_Tp, _Alloc&gt;&amp;) [with _Tp = long unsigned int; _Alloc = std::allocator&lt;long unsigned int&gt;]’ at /usr/include/c++/12/bits/vector.tcc:244:21:
/usr/include/c++/12/bits/stl_algobase.h:431:30: error: argument 1 null where non-null expected [-Werror=nonnull]
  431 |             __builtin_memmove(__result, __first, sizeof(_Tp) * _Num);
      |             ~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/include/c++/12/bits/stl_algobase.h:431:30: note: in a call to built-in function ‘void* __builtin_memmove(void*, const void*, long unsigned int)’

</code></pre>
<p>Any idea on what to do?</p>
Why torch.__version__ returns `2.3.1.post300`

Rhett-Ying: <p>Hi, I am wondering in what kind of scenarios that we may install a pytorch version that <code>torch.__version__</code> returns <code>2.3.1.post300</code> instead of <code>2.3.1</code>. Is it a post-release version with additional minor bug fixes? Is it a stable version?</p>
What is the formula for binary cross entropy loss with label smoothing?

Ngo_Huy: <p>hello, can anyone help me to implement binary cross entropy loss with label smoothing?</p>

Brock_Brown: <p>If you’re okay with <code>CrossEntropyLoss</code> instead of <code>BCELoss</code>, <code>CrossEntropyLoss</code> <a href="https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html" rel="noopener nofollow ugc">comes with an optional <code>label_smoothing</code> parameter</a>. Otherwise, you can try using this:</p>
<pre data-code-wrap="python"><code class="lang-python">eps = 0.1
y_true = y_true * (1 - eps) + (eps / 2)
</code></pre>
<p><sub>Binary cross entropy label smoothing formula taken from <a href="https://stackoverflow.com/questions/73157016/label-smoothing-in-pytorch-using-bce-loss-doing-it-with-the-data-itself" rel="noopener nofollow ugc">here</a>.</sub></p>

Ngo_Huy: <p>I found in YOLOv9 uses BCE label smoothing but I dont know how to implement it to my code :v can you help me to provide a sample</p>
Optimization help needed!

gradientlord: <p>These are the variables we have,</p>
<pre><code class="lang-auto">M = 5   # this is input from user.
data = torch.rand(batch_size, 1024)
multipliers = [torch.rand(1024, 512) for _ in range(M)]

multipliers_to_multiply_each_row_of_data = [
	[1,2],
	[2,4],
	…
	…
	…
	[1,4],
	[3,2],
]
</code></pre>
<p>We want to do a selective tensor contraction of <code>data</code> and <code>Multipliers</code> as given by <code>multipliers_to_multiply_each_row_of_data</code>. So the last row <code>[3,2]</code> means, we’ll need to do a tensor contraction of last row of <code>data</code> with the 3rd and 2nd matrix given in <code>multipliers</code>. In this case, we have two multipliers per data row, but it’s essentially a hyperparameter, let’s say, <code>K</code>.</p>
<p>The best I could come up was this. Here I tried to execute a <code>multiplier</code> just a single time.:</p>
<pre><code class="lang-auto">one_hot_multiplier_mask = F.one_hot(multipliers, num_classes=self.num_multipliers)  # (T, K, M)
one_hot_multiplier_mask = one_hot_multiplier_mask.permute(2, 1, 0)  # (M, K, T)

where_multiplier, where_T, where_k = torch.where(one_hot_multiplier_mask)

for midx in range(self.num_multipliers):
    selection_mask1 = where_multiplier == midx
    selection_mask2 = where_k[selection_mask1]

    if selection_mask2.nelement() &gt; 0:
        A = data[selection_mask2]  # this is the bottleneck!!
        current_data = A @ multipliers[multiplier_idx]

        result[selection_mask2, midx * self.multiplier_size:(midx+1) * self.multiplier_size] = current_data

</code></pre>
<p>Pytorch profiler indicated that the bottleneck is <code>A = data[selection_mask2]</code>.<br>
Looking at the pytorch docs, I found that indexing through a tensor actually copies the result as opposed to indexing through an integer or a slice which returns a view.</p>
<p>Can this be done any faster?</p>
How to avoid repeating a dimention before some customized multiplication?

YagaoDirac: <p>a.shape is [2,5,3], b.shape is [2,5]. I repeated b into [2,5,3] only to do element wise mul with a. Any more effecient way to do this? Thanks.<br>
Real code:</p>
<pre><code class="lang-auto">g_in_reshaped = g_in.unsqueeze(dim=-1).repeat(1, 1, in_features)
the_heaviest_tensor = the_heaviest_tensor.mul(g_in_reshaped)
</code></pre>

gradientlord: <p>Here are some benchmarks:</p>
<p>Using Expand:</p>
<pre><code class="lang-auto">C = B.unsqueeze(dim=-1).expand(-1, -1, 3)
%timeit A * C
</code></pre>
<p>2.85 µs ± 16.7 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)</p>
<p>Adding a dummy dimension:</p>
<pre><code class="lang-auto">C = B[:, :, None]
%timeit A * C
</code></pre>
<p>2.95 µs ± 52.4 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)</p>
<p>Using Repeat:</p>
<pre><code class="lang-auto">C = B.unsqueeze(dim=-1).repeat(1, 1, 3)
%timeit A * C
</code></pre>
<p>2.67 µs ± 31.1 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)</p>
<p>All look pretty much same to me. I’d avoid <a href="https://pytorch.org/docs/stable/generated/torch.Tensor.repeat.html" rel="noopener nofollow ugc"><code>repeat</code></a> since it creates a copy of the tensor while <a href="https://pytorch.org/docs/stable/generated/torch.Tensor.expand.html" rel="noopener nofollow ugc"><code>expand</code></a> returns a <code>view</code>.</p>

YagaoDirac: <p>Thank you. <img src="https://discuss.pytorch.org/images/emoji/apple/+1.png?v=12" title=":+1:" class="emoji" alt=":+1:" loading="lazy" width="20" height="20"> <img src="https://discuss.pytorch.org/images/emoji/apple/+1.png?v=12" title=":+1:" class="emoji" alt=":+1:" loading="lazy" width="20" height="20"> <img src="https://discuss.pytorch.org/images/emoji/apple/+1.png?v=12" title=":+1:" class="emoji" alt=":+1:" loading="lazy" width="20" height="20"></p>
How to get class imbalanced ImageNet

Baraa: <p>I am trying to build a class that would produce a class imbalanced variant of ImageNet. It is not working. Here is my code inspired by: <a href="https://github.com/kaidic/LDAM-DRW/blob/master/imbalance_cifar.py" rel="noopener nofollow ugc">https://github.com/kaidic/LDAM-DRW/blob/master/imbalance_cifar.py</a></p>
<pre><code class="lang-auto">import torch
import torchvision
import torchvision.transforms as transforms
import numpy as np

class IMBALANCEIMAGENET(torchvision.datasets.ImageFolder):
    cls_num = 1000
    def __init__(self, root, transform=None):
        super().__init__(root)
        self.imb_type='exp'
        self.imb_factor=0.01
        self.rand_number=0
        self.train=True
        self.transform=None
        self.target_transform=None
        self.download=False
        np.random.seed(self.rand_number)
        img_num_list = self.get_img_num_per_cls(self.cls_num, self.imb_type, self.imb_factor)
        self.gen_imbalanced_data(img_num_list)

    def get_img_num_per_cls(self, cls_num, imb_type, imb_factor):
        img_max = len(self.imgs) / cls_num
        img_num_per_cls = []
        if imb_type == 'exp':
            for cls_idx in range(cls_num):
                num = img_max * (imb_factor**(cls_idx / (cls_num - 1.0)))
                img_num_per_cls.append(int(num))
        elif imb_type == 'step':
            for cls_idx in range(2):   #Nummber of majority classes. cls_num//2
                img_num_per_cls.append(int(img_max))
            for cls_idx in range(8):   #Nummber of minority classes.
                img_num_per_cls.append(int(img_max * imb_factor))
        else:
            img_num_per_cls.extend([int(img_max)] * cls_num)
        print('img_num_per_cls:')
        print(img_num_per_cls)
        return img_num_per_cls
    def gen_imbalanced_data(self, img_num_per_cls):
        new_data = [] # List[Tuple[str, int]]: List[Tuple[str, int]]: samples of a form (path_to_sample, class)
        new_targets = []
        targets_np = np.array(self.targets, dtype=np.int64)
        classes = np.unique(targets_np)
        # np.random.shuffle(classes)
        self.num_per_cls_dict = dict()
        for the_class, the_img_num in zip(classes, img_num_per_cls):
            self.num_per_cls_dict[the_class] = the_img_num
            idx = np.where(targets_np == the_class)[0] 
            print('idx:')
            print(idx)
            # np.random.shuffle(idx)
            selec_idx = idx[:the_img_num]
            print('selec_idx:')
            print(selec_idx)
            new_data.append(self.imgs[selec_idx, ...])
            new_targets.extend([the_class, ] * the_img_num)
        new_data = np.vstack(new_data)
        self.imgs = new_data
        self.targets = new_targets
        print('imgs shape: ',self.imgs.shape)
        print('self.target length: ',len(self.data))
</code></pre>
<p>The error message is:<br>
new_data.append(self.imgs[selec_idx, …])<br>
TypeError: list indices must be integers or slices, not tuple.<br>
Any help is really appreciated.</p>
Decreasing FPS of PyTorch Model Prediction in Inference Time

abdulkadirkaraca: <p>Hi. I am getting an output by using anomaly detection model which is unsupervised learning model. When I control with time.time(), model inference is looking fast, but when I would like to get image score from model output by using torch.max() or something, fps is decreasing suddenly. What is cause of that. I think this state relevant to autograd (or grad_mode.py call) by pytorch or like this but I cannot ensure. I do not have knowledge enough. Help me please.</p>

ptrblck: <aside class="quote no-group" data-username="abdulkadirkaraca" data-post="1" data-topic="206388">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/abdulkadirkaraca/48/57460_2.png" class="avatar"> abdulkadirkaraca:</div>
<blockquote>
<p>When I control with time.time(), model inference is looking fast, but when I would like to get image score from model output by using torch.max() or something, fps is decreasing suddenly.</p>
</blockquote>
</aside>
<p>Assuming your are using your GPU for the model execution, using host timers requires a device synchronization since CUDA operations are executed asynchronously.</p>

abdulkadirkaraca: <p>Thank you for your reply.</p>
Seg fault with torch::jit::load on Windows

benbellick: <p>Has anyone had success loading a model in lib torch using <code>torch::jit::load</code> on Windows?</p>
<p>I tried doing so following along the instructions given <a href="https://pytorch.org/tutorials/advanced/cpp_export.html" rel="noopener nofollow ugc">here</a> and would either get a segmentation fault, or a file not found error with a file name mysteriously given as the empty string.</p>
<p>You can see more details about the issue <a href="https://github.com/pytorch/pytorch/issues/130874" rel="noopener nofollow ugc">here</a>.</p>

BiggestZocker: <p>I think your argument might me the problem. It looks like you start your .exe from a differnt folder with a relative path and give the file location with a similar relative path. But your program evalutes the path to the model from the location of the .exe file, not from the folder from where you are calling the .exe. Try using a global path or a local path from the .exe file. Hope that helps.</p>

benbellick: <p>Yeah so I suspected that as well, and using an absolute path instead also yielding the same error.</p>
<p>I actually just opened a new issue which is slightly different but I believe more fundamental. Which is that you cannot even construct a Module in C++ <img src="https://discuss.pytorch.org/images/emoji/apple/upside_down_face.png?v=12" title=":upside_down_face:" class="emoji" alt=":upside_down_face:" loading="lazy" width="20" height="20"></p>
<p><a href="https://github.com/pytorch/pytorch/issues/130951" rel="noopener nofollow ugc">issue</a></p>
Can an ensemble of NNs with the same architecture be boiled down to one?

pumplerod: <p>I feel like I’m probably missing something very obvious, but if I have an ensemble of say 100 Neural Networks, all with the same architecture, the only difference being the weights, should there not be a way to combine all the weights resulting in a single neural net producing the same results as the ensemble.  Maybe there’s an order of operations involved between layers which might make this impossible.  Apologies for not be able to process the high school algebra involved, I’m sure greater minds than mine have dealt with this and I’m wondering if there’s a solution or approximation that might suffice.</p>
No module named 'torchvision.transforms.functional_tensor'

Sevimli: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/9/49590cbb714098f4484032b00bba73d6eea78bcb.png" data-download-href="https://discuss.pytorch.org/uploads/default/49590cbb714098f4484032b00bba73d6eea78bcb" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/9/49590cbb714098f4484032b00bba73d6eea78bcb_2_563x500.png" alt="image" data-base62-sha1="asRDTGVXO9GjtT9Dufb8ipxEB4f" width="563" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/9/49590cbb714098f4484032b00bba73d6eea78bcb_2_563x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/9/49590cbb714098f4484032b00bba73d6eea78bcb_2_844x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/9/49590cbb714098f4484032b00bba73d6eea78bcb_2_1126x1000.png 2x" data-dominant-color="303030"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1606×1424 269 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Could you help me for it please?</p>

Eduardo_Lawson: <p>I searched in Pytorch docs and only find this function <a href="https://pytorch.org/vision/0.9/transforms.html#torchvision.transforms.functional.to_tensor" rel="noopener nofollow ugc">torchvision.transforms.fucntional.to_tensor</a>.</p>
<p>I didn´t find any function with that name, so maybe you are trying to import this one…<br>
Here is how you should do it:</p>
<pre><code class="lang-auto">import torchvision.transforms.functional.to_tensor as F_t
</code></pre>

ptrblck: <p><code>torchvision.transforms.functional_tensor</code> was deprecated and is private now after <a href="https://github.com/pytorch/vision/pull/7269">this PR</a>.</p>
Custom Loss when using DDP

Bhavana: <p>Hi,</p>
<p>I am trying to implement custom loss using nn.Module. I have to train the model using multiple GPUs. My forward method of the custom loss takes the (labels, logits) as input. My question is, do I need to gather and concatenate the inputs i.e labels and as well as logits inside the forward method before actually calculating the loss?</p>
<p>Below is the basic skeleton of my custom loss.</p>
<p>def multigpu_gather(input, rank, world_size):<br>
“”"<br>
Gather all the tensors from all the processes and concatenate.<br>
“”"<br>
if world_size == 1:<br>
return input<br>
gather_list = [torch.empty_like(input) for _ in range(world_size)]<br>
dist.all_gather(gather_list, input)<br>
gather_list[rank] = input<br>
output = torch.cat(gather_list, dim=0).contiguous()<br>
return output</p>
<p>class CustomLoss(nn.Module):<br>
def <strong>init</strong>(self, ignore_index, *args, **kwargs):<br>
super().<strong>init</strong>(*args, **kwargs)<br>
self.ignore_index = ignore_index</p>
<pre><code>    # Custom Losses
    self.loss1 = Loss1()
self.loss2 = Loss2()
	
    self.world_size = dist.get_world_size()

def forward(self, logits, labels, dynamic_w2):
    # Question: DO I need to gather in multi GPU setting?
rank = dist.get_rank()
    labels = multigpu_gather(labels, rank, self.world_size)
    logits = multigpu_gather(logits, rank, self.world_size)

    # Loss1
    loss = self.loss1(logits, labels)   # another nn.Module class

    # Loss 2
    loss2 = self.loss2(logits, labels)  # another nn.Module class
	
    final_loss = loss1 + dynamic_w2 * loss2

    return final_loss.mean()
</code></pre>
<p>And, in the training loop, I am calling loss.backward().</p>
<p>Could you please kindly help me understand this? Thanks in advance.</p>
<p>Best,<br>
Bhavana</p>

gradientlord: <p>If you’re using DDP, it takes care of everything.<br>
The following code should be scalable to any number of GPUs.</p>
<pre><code class="lang-auto">def train_one_step():
    opt.zero_grad()
    
    x, y = get_next_batch()
    y1 = model(x)
    
    loss = TheLossFunction(y1, y)
    loss.backward()

    opt.step()

</code></pre>
<p>You do not need to gather/concat labels. The loss is calculated on every GPU and gradients are gathered, averaged and then backpropagated on all GPUs by DDP.</p>
<p>If you want to log/print loss/metrics, you may need to bring them to the master GPU/node, and that’s when you need to gather them from all GPUs.</p>
<pre><code class="lang-auto">if ddp:
    dist.all_reduce(loss, op=dist.ReduceOp.AVG)
</code></pre>
<p>I find using frameworks like Lightling helpful.</p>

Bhavana: <p>Thanks for the clarification.</p>
Iam trying to change this code from tensorflow to pytorch but i cant

Ziad_Sharawy: <p>if someone can help me it would be fantastic, I can send you the code</p>

ptrblck: <p>Could you post the code you have trouble with here and explain what you’ve tried so far and where you are stuck?</p>

Ziad_Sharawy: <p>def inference_batch(self, images=None, target_ys=None, scale=None, par_inp=None):</p>
<pre><code>    """Report loss and label probabilities, and patched images for a batch.

	Args:
	  images: A batch of images to train on, it loads if not present.
	  target_ys: The target_ys for loss calculation, TARGET_ONEHOT if not present."""
    if images is None:
        images = image_loader.get_images()

    if target_ys is None:
        target_ys = TARGET_ONEHOT
    if par_inp is None:
        par_inp = [1., 0.]

    feed_dict = {self._image_input: images, self._target_ys: target_ys, self._par_inp: par_inp, }
    loss_per_example, ps, ims, output_layer = self._run([self._loss_per_example,
                                                         self._probabilities, self._inp,
                                                         # self._patched_input,
                                                         self.ouput_layer],
                                                        feed_dict, scale=scale)

    return loss_per_example, ps, ims, output_layer
</code></pre>
<p>I have tried to change this function from tensor flow  to pytorch  but with no good results</p>

ptrblck: <p>Your code snippet only shows a TF session being executed. PyTorch does not use sessions or other objects to “run” the model, but can execute code directly, similar to e.g. <code>numpy</code>. You can just run the forward pass of a model by passing an input to it:</p>
<pre data-code-wrap="python"><code class="lang-python"># create model instance
model = models.resnet18()

# create random inputs
x = torch.randn(1, 3, 224, 224)

# execute forward pass
output = model(x)

# compute gradients in backward pass
output.mean().backward()
</code></pre>
<p><a href="https://pytorch.org/tutorials/beginner/basics/buildmodel_tutorial.html">This tutoral</a> might be helpful to get started.</p>
Get import error when compile a third op by using CppExtension

Shan19900305: <p>Get import error when compile a third class op by using CppExtension.</p>
<p>A op class is inherited from CustomClassHolder, and using macro TORCH_LIBRARY to bind to pytorch. But after compiled by CppExtension, this module can’t import to python side.</p>
<p>Anyway to fix this error?</p>
<p>Error message:</p>
<pre><code class="lang-auto">python -c "import myCopyOp"
Traceback (most recent call last):
  File "&lt;string&gt;", line 1, in &lt;module&gt;
ImportError: dynamic module does not define module export function (PyInit_myCopyOp)
</code></pre>
<p>setup.py</p>
<pre><code class="lang-auto">from setuptools import setup, find_packages
from torch.utils import cpp_extension

setup(name="myCopyOp",
      packages=find_packages(),
      ext_modules=[cpp_extension.CppExtension('myCopyOp',
                                              sources=['self_ops.cpp'],
                                              )],
      cmdclass={'build_ext': cpp_extension.BuildExtension})
</code></pre>
<p>C++ code:</p>
<pre><code class="lang-auto">#include "torch/torch.h"
#include "torch/custom_class.h"

class MyCopyFunction : public torch::CustomClassHolder {
  public:
    MyCopyFunction(int64_t size): size_(size) {
      weight_ = torch::randn({size}, at::kCPU);
    }

    at::Tensor forward(const at::Tensor&amp; self) {
      at::Tensor dst = torch::mul(self, weight_);
      return dst;
    }

  private:
    at::Tensor weight_;
    const int64_t size_;
};


//  PYBIND11_MODULE(TORCH_EXTENSION_NAME, m) {
//    m.def("forward", &amp;MyCopyFunction::forward, "LLTM forward");
//  }

TORCH_LIBRARY(my_copy_function, m) {
  m.class_&lt;MyCopyFunction&gt;("MyCopyFunction")
   .def(torch::init&lt;int64_t&gt;())
   .def("forward", &amp;MyCopyFunction::forward)
  ;
}
</code></pre>

fxmarty: <p><a class="mention" href="/u/shan19900305">@Shan19900305</a> did you manage to solve this?</p>
<p>On my end adding</p>
<pre data-code-wrap="python"><code class="lang-python">PYBIND11_MODULE(TORCH_EXTENSION_NAME, m) {
}
</code></pre>
<p>solves the issue, but then on python side one still needs <code>import myCopyOp</code> so as to use <code>torch.ops.my_copy_function.MyCopyFunction</code>.</p>
Is it normal so see many [pt_main_thread] <defunct>?

oarend: <p>I’m currently training (or trying to) a small transformer model, and when looking at the processes running, I don’t just see the name of my python process, but also several zombie (<code>&lt;defunct&gt;</code>) processes just called <code>[pt_main_thread]</code>. Their number correlates with the number of dataloader workers, and they keep respawning and disappearing, since their PIDs change over time. There are no error messages in the output. I am training the model on a single GPU and have tried varying numbers of dataloaders.</p>
<p>Maybe this is to be expected, but I wanted to confirm before looking for bugs in my code or reporting any to Pytorch.</p>

hassonofer: <p>While I can’t provide a definitive answer, I’ve observed a similar phenomenon when training a large vision model on a WebDataset using <code>torchrun</code>.</p>
<p>During the first epoch, typically midway through, I notice a temporary slowdown in training. Concurrently, a few <code>[pt_main_thread] &lt;defunct&gt;</code> processes appear. After several seconds, training resumes at approximately 90% of the original pace. Full performance is usually restored within a few minutes, and the remainder of the training proceeds without issues.</p>
<p>This behavior might be related to your observation, though I can’t confirm if it’s the same underlying cause. It could be worth investigating if this is a known issue or expected behavior in PyTorch.</p>
Extending torchscript a custom class with member variables in torch c++

xiaooo-jian: <p>How can I create a custom class with member variables in c++ and use the variables in python?</p>
<p>Now i construct this cpp code.</p>
<pre><code class="lang-auto">class Loader : public torch::CustomClassHolder{
 public:
  Loader(const std::string&amp; path);
  std::vector&lt;int64_t&gt; get_config(const std::string&amp; filename);
  std::vector&lt;int64_t&gt; meta_json;
};

# ------------------

TORCH_LIBRARY(name, ops) {
     ops.class_&lt;Loader&gt;("Loader")
        .def(torch::init&lt;const std::string&gt;())
        .def("get_config", &amp;Loader::get_config);
}
</code></pre>
<p>but i can’t use “Loader().meta_json” in python</p>
<pre><code class="lang-auto">model = torch.classes.name.Loader("xxx.path")
print(model.meta_json) # error with "meta_json" not found.
</code></pre>
<p>So, how can i register the class member in python?</p>
<p>Thanks for your help~</p>
Omp_set_num_threads issue

Olek: <p>Hi, have an issue after ugprading libTorch from 2.2.1 to latest 2.3.1 where running forward call on my model causes several popups to appear that say " Omp_set_num_threads should only be called in serial regions." Forward call is done on a tuple.<br>
Any help is appreciated, thank you.</p>
Combining `pytorch` native layers with custom non-`pytorch` ones wrapped by a `torch.nn.Module`

RupertSciamenna: <p>Hello,</p>
<p>as the title says I am trying to build a <code>torch.nn.Sequential</code> model that stacks native <code>pytorch</code> layers, e.g. <code>torch.nn.Linear</code>, with a custom layer that is not compatible with <code>torch.Tensor</code>, which runs for example on a device that does not support tensors.</p>
<p>Say for example, that I have wrapped this custom module:</p>
<pre data-code-wrap="python"><code class="lang-python">class CustomModule(torch.nn.Module):

    def forward(self, x: torch.Tensor) -&gt; torch.Tensor:
        # convert the inputs, for instance to numpy  
        x = x.detach().numpy()
        # here I would actually like to save the gradients before 
        # detaching and in some sense reattach them to the ouptuts of the
        # module

        # perform some operations
        x = do_some_non_pytorch_things(x)

        # convert back to pytorch
        return torch.as_tensor(x)

    def backward(self, out_grad: torch.Tensor) -&gt; torch.Tensor:
         # here I would externally calculate the gradients of the 
         # non-`pytorch` operations and combine them with the gradients 
         # saved above
         gradients = custom_gradients(out_grad)
         return gradients
</code></pre>
<p>I would like to be able to run it inside of a model like the following:</p>
<pre data-code-wrap="python"><code class="lang-python">model = torch.nn.Sequential(
    torch.nn.Linear(n, k),
    torch.nn.ReLU(),
    CustomModule(),
    torch.nn.Linear(i, j),
)
</code></pre>
<p>while preserving the flow of the gradients and making the <code>CustomModule</code> to completely mimic a standard <code>pytorch</code> layer. Is this possible by any means? What would be the best way to achieve this?</p>
<p>This might be related in some part to <a href="https://discuss.pytorch.org/t/detach-and-re-attach-variable/41364">this other discussion</a>.</p>
<p>Many thanks to anyone who will take time to look into this!</p>

AlphaBetaGamma96: <p>Hi <a class="mention" href="/u/rupertsciamenna">@RupertSciamenna</a>,</p>
<p>You need to define your custom function as a <code>torch.autograd.Function</code> object (docs <a href="https://pytorch.org/docs/stable/autograd.html#torch.autograd.Function" rel="noopener nofollow ugc">here</a>) and then you can just wrap that custom function object within a <code>nn.Module</code> object (like you have above, but without the <code>backward</code> method as that’ll be define in your <code>torch.autograd.Function</code> object.</p>
<p>With the ‘non-pytorch’ operations, you’ll need to manually derive the backward formula and define it in the <code>torch.autograd.Function</code> too.</p>

RupertSciamenna: <p>Thanks, thus even detaching the input tensor in the <code>forward</code> will not break the flow and the layers before that are still going to be trained correctly, right?</p>

AlphaBetaGamma96: <p>Assuming you’ve derived the correct derivative, it should be fine.</p>
Keras training speed issue (PyTorch is a lot slower than TensorFlow)

aabyssx: <p>This might not be a PyTorch issue because I am using Keras but I have no idea what could be the problem and maybe someone can steer me towards the right direction.</p>
<p>I am on native Windows and I used old Keras with TensorFlow 2.10 (GPU accelerated) before. I wanted to try Keras 3 with PyTorch backend. Can someone please help me why this model trains 10x slower with Keras 3.4.1 and PyTorch 2.3.1 backend? With my GPU a single epoch takes a little more than 2 minutes with TF, and over 20 minutes with PyTorch.</p>
<pre><code class="lang-auto">import os
os.environ["KERAS_BACKEND"] = "torch"
import torch
torch.cuda.is_available() # &lt;-- returns True

import keras
from keras.models import Sequential
from keras.layers import Dense
from keras.layers import Dropout
from keras.layers import LSTM
from keras import optimizers
from keras.regularizers import l2

x_train, y_train = np.float32(x_train), np.float32(y_train)
x_val, y_val = np.float32(x_val), np.float32(y_val)

model=Sequential()
reg=0.00001
model.add(LSTM( 80, return_sequences=True , dropout=0.0, kernel_regularizer=l2(reg), recurrent_regularizer=l2(reg), input_shape=(x_train.shape[1], x_train.shape[2]) ))
model.add(LSTM( 80, return_sequences=False, dropout=0.0, kernel_regularizer=l2(reg), recurrent_regularizer=l2(reg) ))
model.add(Dense(40))
model.add(Dense(40))
model.add(Dense(1))
opt = optimizers.Adam(learning_rate=lrate)
model.compile(optimizer=opt, loss='mean_squared_error')

from keras.callbacks import ModelCheckpoint
from keras.callbacks import BackupAndRestore
savecallback = ModelCheckpoint(basefolder+"/"+modelfile, save_best_only=False, monitor='val_loss', mode='min', verbose=1)
backupcallback = BackupAndRestore(basefolder+"/tmp/backup_"+modelfile)

hist=model.fit(x_train, y_train, validation_data=(x_val, y_val), batch_size=batchsize, epochs=20, callbacks=[savecallback, backupcallback])
</code></pre>
<p>If someone would like to test it, you can generate a small sample using this:</p>
<pre><code class="lang-auto">x_train=np.random.uniform(-1, 1, size=(1976, 400, 14))
y_train=np.random.uniform(-1, 1, size=(1976, 1, 1))
x_val=np.random.uniform(-1, 1, size=(1137, 400, 14))
y_val=np.random.uniform(-1, 1, size=(1137, 1, 1))
</code></pre>
<p>With this small sample one epoch takes ~30 seconds with PyTorch and 2-3 seconds with TF.</p>

aabyssx: <p>I was experimenting with jit_compile=True to force torch.compile to use the “inductor” backend, but I was unable to get it working on Windows.</p>
PyTorch on device learning

Markus_1992: <p>I was looking in to on device learning and I think it might be possible in TensorFlow-Lite however I love PyTorch and all models I am currently using are in PyTorch. Is there a way of retraining the models on device like an Android Phone for example using either PyTorch mobile or ExecuTorch?</p>
<p>For some reason, I can’t find any information about this.</p>
Wrong cantor pairing value when using torch.tensor

Joeyoung: <p>Hi,<br>
This is a simple but weird problem I encountered.</p>
<p>def cantor( a,b):<br>
return 0.5*(a+b)*(a+b+1)+b<br>
print( cantor(5804,5) )   <span class="hashtag-raw">#--</span> (1)<br>
print( cantor( torch.tensor(5804), torch.tensor(5) ) )  #–(2)</p>
<p>At (1), it gives the correct answer, 16875150.0.<br>
At (2), it gives the wrong answer,  tensor( 16875148. )</p>
<p>Hmm… confusing me…<br>
Can someone enlighten me?</p>
<p>Regards,<br>
Joe</p>

Joeyoung: <p>Never mind. I have found the reason.</p>

ptrblck: <p>For anyone else wondering: <code>float32</code> rounds to multiples of 2 for <code>values &gt; 2**24</code> which is most likely what is happening here:</p>
<pre data-code-wrap="python"><code class="lang-python">0.5*(a+b)*(a+b+1) &gt; 2**24
# tensor(True)
</code></pre>
Do I need to consider the batch_size when defining own module

king_wang: <p>From the tutorial, I got that <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/1/1d2712da0016ce57bf913c7643afef7595f6a430.png" data-download-href="https://discuss.pytorch.org/uploads/default/1d2712da0016ce57bf913c7643afef7595f6a430" title="Snipaste_2018-04-12_09-38-49"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/1/1d2712da0016ce57bf913c7643afef7595f6a430_2_690x388.png" alt="Snipaste_2018-04-12_09-38-49" data-base62-sha1="49TyOvVaoNJzxdjNsql7Eh0PJvO" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/1/1d2712da0016ce57bf913c7643afef7595f6a430_2_690x388.png, https://discuss.pytorch.org/uploads/default/optimized/2X/1/1d2712da0016ce57bf913c7643afef7595f6a430_2_1035x582.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/1/1d2712da0016ce57bf913c7643afef7595f6a430_2_1380x776.png 2x" data-dominant-color="DDDBDB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Snipaste_2018-04-12_09-38-49</span><span class="informations">1435×809 173 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>we need to consider the batch_size when using the Linear layer. Then if I define own module ,do i need to consider the batch_size, e.g if the original input is a  input_feature  Tensor, if we consider the batch_size, then the input will be batch_size*input_feature Tensor</p>

ptrblck: <p>You don’t need to consider the batch size when initializing the <code>Modules</code>. The <code>Linear</code> layer for example takes <code>in_features</code> as an argument, which would be dimension 1 for <code>x = torch.randn(10, 20)</code>.</p>
<p>However, when you need another <code>view</code> on the <code>Tensor</code>, e.g when you need to flatten the <code>Tensor</code> coming from a <code>Conv2d</code>, you most likely want to keep the batch size and flat all remaining dimensions.<br>
You would do it in the <code>forward</code> method:</p>
<pre><code class="lang-auto">x = self.conv(x)
x = x.view(x.size(0), -1) #keep batch size
x = self.fc(x)
</code></pre>
<p>Does this explanation make it clearer?</p>

Dung_Hu_nh_Anh: <p>How about a model from any paper? I currently working on replicate a model from a paper. And do i need to consider the batch size?</p>

ptrblck: <p>The implementation of models “from any paper” depends on the definition of the authors. While built-in layers (and other standard layers) do not hard-code the batch size, it’s impossible to speak for all authors.</p>
Can someone help me understand how to use a .pt file?

anonymous_struggler: <p>I am really new to this field and have a few questions. I am trying to quantize a model using MIT Han lab AWQ quantization method. I have successfully pulled the model from huggingface and applied quantization inside a docker container. I now have a file titled as “Llama-2-7b-hf-w4-g128-awq-v2.pt.” I need desperate help on how to use this model from here. I need to benchmark this models performance. Any help would be greatly appreciateed. I do not know where to start so if anyone can help guide me(provide links or resources) I would be eternally grateful.</p>
<p>Thank you in advance!!</p>

ptrblck: <p>I assume your script has created this file? If so, check which object was passed to the <code>torch.save</code> method. I would guess it’s either the <code>model.state_dict()</code>, which will contain all trained parameters and buffers, or a custom <code>dict</code> containing the <code>state_dict</code> as well as other objects and data, e.g. the <code>optimizer.state_dict()</code> etc.<br>
The file itself is just an archive and can be loaded via <code>torch.load</code>. Load it in a new script and check its content.</p>

AlphaBetaGamma96: <p>Just to add to <a class="mention" href="/u/ptrblck">@ptrblck</a>’s comment, the <code>.pt</code> file doesn’t save the model itself (just the weights, as <a class="mention" href="/u/ptrblck">@ptrblck</a> said). So, if you don’t have the original source code for the model you’ll need to re-code the model and then load the weights.</p>
Error running Libtorch C++ CUDA code (MKL dlls are missing)

DavidHr: <p>fter Installing MKL I can’t find those dll files. And the error continues.</p>
<p>Intel MKL FATAL ERROR: cannot load mkl_vml_avx2.1.dll or mkl_vml_def.1.dll, at runtime.</p>
<p>Libtorch Distrib I’m using is: CUDA 12, C++.</p>
<p>Any help would be appreciated, as I’m not able to run the code using the NVIDIA CUDA.</p>
<p>I have no errors running the same code for CPU.</p>
Mse loss averaging over every pixel instead of batch

HaziqRazali: <p>I am using the mse loss described here.</p>
<p><a href="https://pytorch.org/docs/stable/nn.html?highlight=mse#torch.nn.functional.mse_loss" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/nn.html?highlight=mse#torch.nn.functional.mse_loss</a></p>
<blockquote>
<p><code>torch.nn.functional.</code> <code>mse_loss</code> ( <em>input</em> ,  <em>target</em> ,  <em>size_average=None</em> ,  <em>reduce=None</em> ,  <em>reduction=‘mean’</em> ) → Tensor</p>
</blockquote>
<p>My input and target are of size [16, 2, 48, 120] i.e. a batch size of 16 where each item is a tensor of size [2, 48, 120].</p>
<p>Supplying the argument <code>reduction="sum"</code> returns <code>tensor(3304.8472, grad_fn=&lt;MseLossBackward&gt;)</code><br>
whereas the argument <code>reduction="mean"</code> returns <code>tensor(0.0747, grad_fn=&lt;MseLossBackward&gt;)</code></p>
<p>This probably means that the method is averaging over the total number of pixel instead of averaging only by the batch size. What can I do to sum over every pixel then divide by the batch size ?</p>

ptrblck: <p>You could divide it manually by the batch size:</p>
<pre><code class="lang-python">loss = F.mse_loss(output, target, reduction='sum') / output.size(0)
</code></pre>

HaziqRazali: <p>I thought of that but I wondered if it will have any effect on the gradients due to the division operation. I guess not, thanks !</p>
Cross Attention for images

NolimitDougie: <pre><code class="lang-auto">class CrossAttentionLayer(nn.Module):
    def __init__(self, embed_dim, num_heads, ff_dim, dropout=0.1):
        super(CrossAttentionLayer, self).__init__()
        self.multihead_attn = nn.MultiheadAttention(embed_dim, num_heads, dropout=dropout, batch_first=True)
        self.norm1 = nn.LayerNorm(embed_dim)
        self.ffn = nn.Sequential(
            nn.Linear(embed_dim, ff_dim),
            nn.ReLU(),
            nn.Dropout(dropout),
            nn.Linear(ff_dim, embed_dim),
            nn.Dropout(dropout)
        )
        self.norm2 = nn.LayerNorm(embed_dim)

    def forward(self, query, key, value):
        # Multi-head cross-attention
        attn_output, _ = self.multihead_attn(query, key, value)
        attn_output = self.norm1(attn_output + query)

        # Feed-forward network
        ffn_output = self.ffn(attn_output)
        output = self.norm2(ffn_output + attn_output)

        return output
</code></pre>
<p>Hello, I have one source image and 5 similar images I get back from my retrieval model doing contrasitive learning  I’m trying to set up cross attention to have my source image attend to the other images to get a better representation of my source image. Please if you have any suggestions please feel free to share. This enhance representation will make up a hidden state for my diffusion model. Should I connate the similiar images across the seq length then do the attn or do the attn one by one so src image attn to img1 and so on</p>

NolimitDougie: <p>Also if I’m doing feature aggregation should I concatenate across the sequence length or embed dimension</p>
Lifetime of CPU tensors which use pinned memory

embg: <p>Does this code have a use-after-free bug?</p>
<pre><code class="lang-auto">def foo():
    A = torch.rand(128, device="cpu", pin_memory=True)
    B = A.cuda(non_blocking=True)
    return B

C = foo()
# do stuff with C
</code></pre>
<p>My concern is that when <code>foo()</code> returns, the refcount of CPU tensor <code>A</code> will hit zero and its pinned memory will be returned to PyTorch’s allocator.</p>
<p>Does PyTorch make sure that <code>B</code> keeps a reference to <code>A</code> (until the next stream sync) to prevent this issue? Or as the user, do I need to manually hold a reference to <code>A</code> until I know it’s safe to free the pinned memory?</p>

embg: <p>Turns out this is not a use-after-free. PyTorch keeps the pinned memory alive until the memcpy is complete. Reference: <a href="https://github.com/pytorch/pytorch/issues/130785#issuecomment-2229635072" class="inline-onebox" rel="noopener nofollow ugc">[CUDA] Remove footgun related to non-blocking copies · Issue #130785 · pytorch/pytorch · GitHub</a></p>
Omp_set_num_threads should only be called in serial regions

jfulem: <p>Please, when I call a torch model forward() method I get several pop-up messages with this statement: <strong>omp_set_num_threads should only be called in serial regions</strong>. The input to that forward method is a 3D tensor, where the first dimension is a batch size. Please, does anybody know what the issue is? Thank you.</p>

Olek: <p>Hi, did you ever resolve this issue?<br>
Thanks</p>