akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
libTorch cannot pass a const module into function

CuriousCat-7: <pre><code class="lang-cpp">   22 void get_optical_flow(                                                                                                                                                                               
    23     const cv::Mat &amp; image1,                                                                                                                                                                          
    24     const cv::Mat &amp; image2,                                                                                                                                                                          
    25     cv::Mat &amp; flow, //output                                                                                                                                                                         
 &gt;&gt; 26     const torch::jit::script::Module &amp; module,                                                                                                                                                       
    27     const int module_input_h,                                                                                                                                                                        
    28     const int module_input_w){
</code></pre>
<p>get error</p>
<pre><code class="lang-bash">error: passing ‘const Module {aka const torch::jit::Module}’ as ‘this’ argument discards qualifiers [-fpermissive]
   at::Tensor output = module.forward(inputs).toTensor();
</code></pre>
<p>looks like the <code>forward</code> will change the module</p>

weitaoliu: <p>Hey, I found the same issue. If I provide the module with const, the forward function throws the same error as you got. I am unsure if it is a good idea that const cannot be applied to the module.</p>
An AVX512 capable cpu mandatory for pytorch 2/pytorch-lightning?

dip4fish: <p>Hi,<br>
My computer has a non AVX CPU (xeon x5570). Can a pytorch 2.x based code run on my computer?<br>
Thanks</p>

smth: <p>AVX512 is not mandatory. The minimal assumption is AVX1 I believe.</p>
<p>The default PyTorch binaries are built with AVX1, AVX2 and AVX512 optimizations but AVX512 is gated behind runtime dispatch.</p>
<p>If for some reason you are seeing <code>illegal instruction</code> errors, you can build PyTorch from source.</p>

dip4fish: <p>ho, so AVX is mandatory for pytorch 2. My machine have a non-AVX xeon x5570 CPU.</p>

smth: <p>yea if your processor doesn’t support even AVX1, its too old for PyTorch.</p>
Ampere sgemm 128x32 sliced1x4 nn

hst123: <p>When pytorch executes resnet50 on the GPU, it executes this operator ampere_sgemm_128x32_sliced1x4_nn. The cuda source code of this operator is in the location of pytorch.</p>
<p>I can see this kernel by nvidia-nsight</p>
<div class="md-table">
<table>
<thead>
<tr>
<th>5.40%</th>
<th>110.754 μs</th>
<th>7</th>
<th>15.822 μs</th>
<th>14.688 μs</th>
<th>14.561 μs</th>
<th>22.561 μs</th>
<th>2.972 μs</th>
<th>7   32    1</th>
<th>128    1    1</th>
<th>ampere_sgemm_32x32_sliced1x4_nn</th>
</tr>
</thead>
<tbody>
<tr>
<td>4.80%</td>
<td>98.178 μs</td>
<td>42</td>
<td>2.337 μs</td>
<td>2.336 μs</td>
<td>2.240 μs</td>
<td>2.720 μs</td>
<td>70 ns</td>
<td>1    1    1</td>
<td>128    1    1</td>
<td>void at::native::vectorized_elementwise_kernel&lt;(int)4, at::native::::batch_norm_calc_invstd(const at::Tensor &amp;, const at::Tensor &amp;, double)::[lambda() (instance 1)]::operator ()() const::[lambda() (instance 4)]::operator ()() const::[lambda(float) (instance 1)], at::detail::Array&lt;char *, (int)2&gt;&gt;(int, T2, T3)</td>
</tr>
<tr>
<td>4.40%</td>
<td>90.498 μs</td>
<td>3</td>
<td>30.166 μs</td>
<td>29.889 μs</td>
<td>29.888 μs</td>
<td>30.721 μs</td>
<td>480 ns</td>
<td>1   16   10</td>
<td>256    1    1</td>
<td>ampere_sgemm_64x32_sliced1x4_nn</td>
</tr>
</tbody>
</table>
</div>

ptrblck: <p>Could you explain what the question is?</p>
What is the equivalent to conv1D with kernel size 1x1 in pytorch geometric convolutions?

Mary_H: <p>I would like to know how to apply equivalent operation as conv1D with kerenl size 1x1 in graph data by using e.g., graphSAGE?<br>
Or can I simply just use nn.conv1d(in,out,kernel=1) to the graph nodes?</p>
Torch.Compile different fx graph for inductor and custom backend

Tiru_B: <p>Hi</p>
<p>I have registered custom backend in pytorch 2.0 and running GPT-J model for  both  inductor and custom backed named ABCD.</p>
<pre><code class="lang-auto">tokenizer = AutoTokenizer.from_pretrained("hf-internal-testing/tiny-random-gptj")
model = GPTJForCausalLM.from_pretrained("hf-internal-testing/tiny-random-gptj")
inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
outputs = model(**inputs, labels=inputs["input_ids"])
input_ids = inputs["input_ids"]
start_time = time.time()
print("cpu.................")
fn_cpu = torch.compile(model.generate,backend="inductor")
with torch.no_grad():
    output_cpu = fn_cpu(input_ids, do_sample=True, temperature=0.9, max_length=200)
print(output_cpu)
print("--- %s pytorch with compile CPU seconds ---" % (time.time() - start_time))
torch._dynamo.reset()
</code></pre>
<p><strong>Above same code i ran it for backend ABC</strong></p>
<p>now I am capturing fx grahp for both the backends</p>
<p>For Inductor you can see below code</p>
<pre><code class="lang-auto"> def __call__(self, model_, inputs_):
        from torch._inductor.compile_fx import compile_fx
        print("CPU Graphs")
          model_ir = model_.print_readable(print_output=False)
          with open(f"pt_graph_fwd_cpu.ir", 'w') as file:
              file.write(model_ir)
          from torch.fx.passes import graph_drawer
          gd = graph_drawer.FxGraphDrawer(model_, 'f')
          pydot_graph = gd.get_dot_graph()
          pydot_graph.write_png(f"pt_graph_fwd_cpu.png")

        return compile_fx(model_, inputs_, config_patches=self.config) ```

for custom backed end 

```@register_backend
def ABC_backend(model:GraphModule, inputs:List[FakeTensor]):
    compiled_graph = None
    def fwd(*args):
        nonlocal model
        nonlocal compiled_graph
        if compiled_graph is None:
        model_ir = model_.print_readable(print_output=False)
        with open(f"pt_graph_fwd_cpu.ir", 'w') as file:
            file.write(model_ir)
        from torch.fx.passes import graph_drawer
        gd = graph_drawer.FxGraphDrawer(model_, 'f')
        pydot_graph = gd.get_dot_graph()
        pydot_graph.write_png(f"pt_graph_fwd_cpu.png")
            compiled_graph = ABCDBACKENDCLASS(model,inputs,args)
            del model
        return compiled_graph(*args)

    return fwd```

Now this issues is , I am seeing different fx graphs for both flows, and what i have observed is in Inductor flow compile_fx is being called multiple times so all the sub grahps are are merged where as for ABC backend i can see only once its being called.</code></pre>
How to get relationship between api and operator？

hny.wineus: <p>I am trying to reduce the test work in  the task which called "every version of pytorch or cuda published, run the all testcase to make sure it does not make our model bad " . It’s too much and too expensive. So I am going to find the relationship. While any code changed in one of those relationship, all I need to do is  run the testcase those include the api which involed. If the testcase ok , our Model OK.</p>
<p>So can I get the relationship from anywhere, like a doc or use python/Java/C++ .</p>
<p>Thanks anyone who see or give the advice!</p>
Building geodataset of Sentinel-2 timeseries data

Reza_Maleki: <p>Hello! I wanted to build a geodataset from Sentinel-2 timeseries, when I wanted to intersect them with &amp;, I got this error:</p>
<h1><a name="runtimeerror-datasets-have-no-spatiotemporal-intersection-1" class="anchor" href="#runtimeerror-datasets-have-no-spatiotemporal-intersection-1"></a>RuntimeError: Datasets have no spatiotemporal intersection</h1>
<p>I know this error comes from “mint” “maxt” mismatching between two sentinel2 data but how could I build timeseries dataset from Sentinel-2 data?</p>
<p>I used “|” (union dataset) but datasampler couldn’t consider timeseries.</p>
<p>Name                    Version                   Build  Channel<br>
torchgeo                  0.5.0                    pypi_0    pypi</p>
Problems compiling with Libtorch v2.0.1 and Visual Studio 2015 (platform toolset v140)

rak: <p>Hello!</p>
<p>Background: I am working on a big-legacy code that runs on the VS 2015 (toolset v140), but not above. I want to merge this legacy code with the libtorch v2.0.1</p>
<p>Problem: I had no problem of compiling libtorch v2.0.1 on VS 2022 version. However, when I try to compile libtorch v2.0.1 on VS 2015, errors regarding “constexpr” show up.</p>
<p>I found a previous topic discussion related to this issue, but the discussion ended without any helpful replies.<br>
<a href="https://discuss.pytorch.org/t/problems-compiling-with-libtorch-v1-5-0-and-vs-toolset-v140/82351">https://discuss.pytorch.org/t/problems-compiling-with-libtorch-v1-5-0-and-vs-toolset-v140/82351</a></p>
<p>Following the previous post, I tried the minimal code below:</p>
<blockquote>
<p>example-app.cpp</p>
</blockquote>
<pre><code class="lang-auto">#include &lt;torch/torch.h&gt;
#include &lt;iostream&gt;

int main() {
  torch::Tensor tensor = torch::rand({2, 3});
  std::cout &lt;&lt; tensor &lt;&lt; std::endl;
}
</code></pre>
<blockquote>
<p>Warnings: (only the first one)</p>
</blockquote>
<pre><code class="lang-auto">Severity	Code	Description	Project	File	Line	Suppression State
Warning	C4814	'c10::reverse_iterator&lt;_Iterator&gt;::operator =': in C++14 'constexpr' will not imply 'const'; consider explicitly specifying 'const'	minimal_2015ver_libtorch	c:\users\user\downloads\common\libtorch-win-shared-with-deps-debug-2.0.1+cu118\libtorch\include\c10\util\reverse_iterator.h	88	
</code></pre>
<blockquote>
<p>Errors: (selecting first few errors)</p>
</blockquote>
<pre><code class="lang-auto">Severity	Code	Description	Project	File	Line	Suppression State
Error	C3249	illegal statement or sub-expression for 'constexpr' function	minimal_2015ver_libtorch	c:\users\user\downloads\common\libtorch-win-shared-with-deps-debug-2.0.1+cu118\libtorch\include\c10\core\dispatchkey.h	533	
Error	C3249	illegal statement or sub-expression for 'constexpr' function	minimal_2015ver_libtorch	c:\users\user\downloads\common\libtorch-win-shared-with-deps-debug-2.0.1+cu118\libtorch\include\c10\core\dispatchkey.h	532	
Error	C3250	'count': declaration is not allowed in 'constexpr' function body	minimal_2015ver_libtorch	c:\users\user\downloads\common\libtorch-win-shared-with-deps-debug-2.0.1+cu118\libtorch\include\c10\core\dispatchkey.h	560
Error	C3249	illegal statement or sub-expression for 'constexpr' function	minimal_2015ver_libtorch	c:\users\user\downloads\common\libtorch-win-shared-with-deps-debug-2.0.1+cu118\libtorch\include\c10\core\dispatchkey.h	560	
</code></pre>
<p>My environment:<br>
Windows 11 Home<br>
Compiler: VS 2022 version and VS 2015</p>
<p>I will appreciate any help, please.</p>
<p>Thanks,<br>
Rak</p>

rak: <p>I am writing this in case anyone faces the same problem as I.</p>
<p>I was able to use other c-binding deep learning library with the Visual Studio 2015 (platform toolset v140). The library is tensorflow. It works, although it has its own problems.</p>
Data format for Input to Opacus

Ashutosh: <p>I am trying to use Opacus to train distilgpt2 on my data with DP-SGD. I have my data loaded and it has the below format.<br>
datasets.Dataset({<br>
features: [‘input_ids’, ‘attention_mask’, ‘labels’],<br>
num_rows: 139<br>
})</p>
<p>I am struggling to convert it into the required format for this code to work.<br>
data_loader = DataLoader(data, batch_size=2)<br>
privacy_engine = PrivacyEngine()<br>
model, optimizer, train_data = privacy_engine.make_private(<br>
module=model,<br>
optimizer=optimizer,<br>
data_loader=data_loader,<br>
noise_multiplier=1.0,<br>
max_grad_norm=1.0<br>
)</p>
<p>I am able to train distilgpt2 without Opacus. I am new to this and any help would be appreciated.</p>

Ashutosh: <p>I figured out the training part using a custom training loop instead of using Trainer. I am having issues with generation as I get the below error.<br>
AttributeError: ‘GradSampleModule’ object has no attribute ‘generate’<br>
So, do I having to implement my own generate method?</p>
Maintenance of pytorch/cpuinfo

Prashanth_Swaminatha: <p>Hi folks - it was recommended I reach out here from <a href="https://github.com/pytorch/cpuinfo/pull/210" class="inline-onebox" rel="noopener nofollow ugc">Fix chipset enum name to include 'vendor_' by prashanthswami · Pull Request #210 · pytorch/cpuinfo · GitHub</a> - recently, the primary maintainer of pytorch/cpuinfo has transitioned off the project, so there isn’t an official maintainer anymore that has bandwidth to review patches. malfet@ on the above PR has been kind to  review a number of RISC-V patches myself and others have been sending over, but it’s consuming a lot of bandwidth and a proper maintainer needs to be assigned to balance the load.</p>
<p>While there are a few folks contributing patches, my stake here is Android RISC-V and by extension, libraries like XNNPACK that are used on Android that would need RISC-V support. I’m flagging this purely as a “there’s an active need from the community to make changes in service of enabling RISC-V”. There are already gaps appearing that would break future rolls of pytorch for RISC-V: <a href="https://github.com/pytorch/cpuinfo/pull/212" class="inline-onebox" rel="noopener nofollow ugc">Fix RISC-V Linux build by markdryan · Pull Request #212 · pytorch/cpuinfo · GitHub</a></p>
<p>Of course, I don’t want to just drop a resource request out of the blue! I’m here to ask what the process is and what sort of help can be provided to find a maintainer.</p>
<ol>
<li>Is there some committee that identifies / assigns new maintainers?</li>
<li>Can I follow along somewhere to be informed if a new maintainer is selected?</li>
<li>If someone needs to be found, and I find someone who is interested, can they offer to help maintain this library / who would I point them to that could evaluate / approve them?</li>
</ol>

smth: <p>this is a solid topic for the developer mailing list <a href="https://dev-discuss.pytorch.org/">https://dev-discuss.pytorch.org/</a><br>
I suspect, one of the existing CPU maintainers (mostly folks from Intel) might step up.</p>
<p>I suggest you re-post the topic there, and we can continue the discussion.</p>

Prashanth_Swaminatha: <p>Thank you for the redirect! I’ve re-posted this topic here: <a href="https://dev-discuss.pytorch.org/t/maintenance-of-pytorch-cpuinfo/1767" class="inline-onebox" rel="noopener nofollow ugc">Maintenance of pytorch/cpuinfo - PyTorch Dev Discussions</a></p>
Similarity or distance?

papoo13: <p>I am trying to understand <a href="https://arxiv.org/abs/1806.10574" rel="noopener nofollow ugc">this paper</a> through <a href="https://github.com/cfchen-duke/ProtoPNet" rel="noopener nofollow ugc">the oficial code implementation</a>. The learn prototypes and compute the distance input image feature patches and those prototypes. In the code, however, this distance is implemented as the following (refer to model.py, lines 154 to 177):</p>
<pre><code class="lang-auto">    def _l2_convolution(self, x):

        '''
        apply self.prototype_vectors as l2-convolution filters on input x
        '''
        # x.shape: torch.Size([80, 128, 7, 7])
        pdb.set_trace() 
        x2 = x ** 2 # why do they do this? : x2.shape: torch.Size([80, 128, 7, 7])
        x2_patch_sum = F.conv2d(input=x2, weight=self.ones) # torch.Size([80, 2000, 7, 7])
        
        # self.prototype_vectors shape: torch.Size([2000, 128, 1, 1]) - random values
        p2 = self.prototype_vectors ** 2
        # compute the squared L2 norm of the prototype_vectors
        p2 = torch.sum(p2, dim=(1, 2, 3)) #torch.Size([2000]) 
        # p2 is a vector of shape (num_prototypes,)
        # then we reshape it to (num_prototypes, 1, 1)
        p2_reshape = p2.view(-1, 1, 1) #torch.Size([2000, 1, 1])

        xp = F.conv2d(input=x, weight=self.prototype_vectors) #torch.Size([80, 2000, 7, 7])
        intermediate_result = - 2 * xp + p2_reshape  # use broadcast
        # x2_patch_sum and intermediate_result are of the same shape
        distances = F.relu(x2_patch_sum + intermediate_result)

        return distances
</code></pre>
<p>I am not sure how this is computing the distance! To me this line <code>xp = F.conv2d(input=x, weight=self.prototype_vectors)</code> computes the similarity between patches of the features and the initialized prototypes (in the very first step). I want to get a second opinion on this. I think I’m missing something. Thank you in advance <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
Torch.jit.script getting stuck in torchvision.transforms

Vidit_Agarwal: <p>Here the <code>t</code> isn’t a tensor rather a function. I also tried to resolve it by passing example_inputs, but still it’s giving same error.</p>
<pre data-code-wrap="sh"><code class="lang-sh">RuntimeError: 
cannot call a value of type 'Tensor':
  File "/opt/homebrew/Caskroom/mambaforge/base/envs/mlx/lib/python3.11/site-packages/torchvision/transforms/transforms.py", line 95
    def __call__(self, img):
        for t in self.transforms:
            img = t(img)
                  ~ &lt;--- HERE
        return img
</code></pre>
I got different results from different GPUs

Betsi: <p>I built a deep-learning model using Pytorch for a specific task and got a good result(68.7%) on the RTX 3090 GPU. Similarly, I built a model on GeForce RTX 2080 without changing the source code and dataset, but I got a bad result(5.4%). I’m new to GPU. I want to know why the result differs and How can I solve it?<br>
thanks<br>
** NVIDIA GeForce RTX 3090:**<br>
Driver Version: 470.86 and CUDA Version: 11.4<br>
nvcc --version result on GeForce RTX 3090 :<br>
Cuda compilation tools, release 11.2, V11.2.67<br>
Build cuda_11.2.r11.2/compiler.29373293_0<br>
torch version:1.7.1</p>
<p><strong>and  NVIDIA GeForce RTX 2080:</strong><br>
Driver Version: 515.57  and CUDA Version: 11.7<br>
nvcc --version result on GeForce RTX 2080 :<br>
Cuda compilation tools, release 11.6, V11.6.124<br>
Build cuda_11.6.r11.6/compiler.31057947_0<br>
torch version:1.7.1+cu110</p>

ptrblck: <p>Check your input data, the model, and everything else in your script as we’ve seen similar questions in the past, which were caused e.g. by a different folder structure of the dataset and never by different GPUs.</p>

Betsi: <p>The problem is solved. Some dataset folders were empty in the GeForce RTX 2080-based remote server. that was the problem. thank you for your useful suggestion and response.</p>
How to load using torch.load without source class (using which model was created)?

braindotai: <p>Hi there, in first file I’m defining the model class as “Classifier” and training the model and then saving it using <code>torch.save(model, 'model.pt', dill)</code>. Code in first script looks like-</p>
<pre><code class="lang-python">class Classifier(nn.Module):
    def __init__(self):
        super(Classifier, self).__init__()
        # a toy example model
        self.model = nn.Sequential(
            nn.Linear(10, 5),
            nn.LeakyReLU(0.2),
            nn.Linear(5, 10))
    
    def forward(self, x):
        return self.model(x)

    def a_use_full_function(self):
        """
        A function I'm using while training the model,
        so I wanna save and load this as well so that I can use it
        for continuing training my model in the second script
        """

model = Classifier()
model.cuda()
# training the model....
# evaluating and testing it....
# and saving as-

import dill
torch.save(model, 'model.pt', dill)
</code></pre>
<p>In the second script I’m trying to loading the model without any <code>Classifier</code> definition, so code in second script looks exactly like-</p>
<pre><code class="lang-python">from torch import nn
import dill
model = torch.load('model.pt', 'cuda', dill)
</code></pre>
<p>Running it is showing error as “<code>AttributeError: Can't get attribute 'Classifier' on </code>”.<br>
I understood this error so when I copy and paste the <code>Classifier</code> class in this second script, then it works, but of course, this is not loading model without model definition.</p>
<p>So my question is how can I save and load my model so that I don’t have to deal with the original source code of the model definition.</p>
<p><strong>One note</strong>- when I tried saving and loading the model build just using <code>torch.nn.Sequential</code> then it got loaded in the second scripts without any error.</p>

ptrblck: <p>You could save the jitted model, which can then be loaded in other applications.<br>
As far as I know, there is no way of storing the model directly without recreating the file structure.<br>
Generally this workflow is also not recommended and you should store and load the <code>state_dict</code> instead.</p>
Help! Has anyone ever gotten AVSR (Audio-Visual Speech Recognition) to function with the PytorchAudio AVSR Examples?

blevlabs: <p>Referencing this: <a href="https://github.com/pytorch/audio/tree/main/examples/avsr" rel="noopener nofollow ugc">https://github.com/pytorch/audio/tree/main/examples/avsr</a><br>
Has anyone ever gotten AVSR to function with the PytorchAudio AVSR Examples?</p>
<p>I have been trying for a few days to get an Av_Hubert AVSR model functioning for inference like the ones shown here: <a href="https://facebookresearch.github.io/av_hubert/" rel="noopener nofollow ugc">https://facebookresearch.github.io/av_hubert/</a></p>
<p>for audio-visual speech recognition with a video and audio stream, however, the system provided on Pytorch does not appear to have any support for running inference on these models.</p>
<p>Any guidance to get an AVSR pre-trained model functioning would be great, as there does not appear to be a lot of support for such a system.</p>
Bug in backward hook: grad_input[1] (dW) is none

Eden_Tan: <p>I have this simple class but dW is None, even when the parameter is still being updated. After calling x.require_grad_(True), the hook is printing dW. This behavior is confusing</p>
<pre><code class="lang-auto">import torch.nn as nn
import torch.nn.functional as F
import torch

def hook_fn(module, grad_input, grad_output):
    print(f"{module} has dW {grad_input[1]} and scaler value {module.scaler}")
    
    
def factor_balance(mid_blksz, out_blksz):
    total = mid_blksz * out_blksz

class Scaler(nn.Module):
    def __init__(self, out_features):
        super().__init__()
        self.scaler = nn.Parameter(torch.zeros(1))
        
    def forward(self, x):
        # x.requires_grad_(True)
        x = self.scaler * x
        # x = F.layer_norm(x, x.shape[1:])
        # layernorm to avoid vanishing gradient
        return x

x = torch.ones(100, 100, dtype=torch.float32)
y = torch.full((100, 100), 2, dtype=torch.float32)
model = Scaler(100)
model.register_backward_hook(hook_fn)
optimizer = torch.optim.SGD(model.parameters(), lr=0.1)
for i in range(100):
    loss = F.mse_loss(model(x), y)
    loss.backward()
    optimizer.step()
  
</code></pre>
<p>Thanks in advance!</p>

soulitzer: <p>In your case need to look at grad_input[0], grad_input[1] corresponds to the grad wrt x, which does not require grad, and hence receives None for gradients.</p>
<p>Also, note that Module.register_backward_hook is deprecated, you should be using Module.register_full_backward_hooks instead.</p>

Eden_Tan: <p>I thinks it’s a bug because it returns both dw and dx for some modules. grad_input[0] has the same shape as x. Using register_full_backward_hooks makes no difference.<br>
<a class="mention" href="/u/ptrblck">@ptrblck</a> could you please take a look?</p>

soulitzer: <p>It returns both dw and dx because your module’s last operation is a mul between x and w. This is expected behavior.</p>
<p>Sorry full backward hook does something different (this is also expected behavior). It only returns the gradients wrt the inputs to your module, it will not include gradients wrt params</p>
<p>If you want the gradient wrt to a weight, you could register a hook directly to the parameter.</p>
<p>If you would like the hook to be fired after gradients wrt all parameters to be fired, you can register a multi-hook.</p>
Links to internal FB pages in blogpost

azik1725: <p>In blogpost <a href="https://pytorch.org/blog/empowering-models-performance/" rel="noopener nofollow ugc">empowering-models-performance</a> I found that some links lead to internal Facebook pages, for example:</p>
<ul>
<li><a href="https://fburl.com/dp0c31pl" class="inline-onebox" rel="noopener nofollow ugc">Internal Login</a></li>
</ul>
<p>Could you fix the links?</p>

azik1725: <p>And also on the topic of this blogpost, it seems that the COMPILE-TIME GRAPH SEARCH paragraph ends too abruptly</p>
<blockquote>
<p>With our approach, the search time is roughly 60 seconds for one of our largest internal models. It is manageable for on-the-fly tasks and</p>
</blockquote>

ptrblck: <p>Thanks for letting us know. CC <a class="mention" href="/u/malfet">@malfet</a></p>

malfet: <p>Thank you for reporting. This is just an URL shortener for <a href="https://pytorch.org/blog/optimizing-production-pytorch-performance-with-graph-transformations/" class="inline-onebox" rel="noopener nofollow ugc">Optimizing Production PyTorch Models’ Performance with Graph Transformations | PyTorch</a></p>

azik1725: <p>Thank you very much for the correct link. There is another link in this post that leads to internal resources - <a href="https://fburl.com/wiki/2kypn658" class="inline-onebox" rel="noopener nofollow ugc">Internal Login</a></p>
<p>And can you tell me when you (or maybe authors of blogpost?) can correct the part of the article about COMPILE-TIME GRAPH SEARCH, because this part seems to me to be unfinished as i said earlier.</p>
Network return torch.int64 and target variable is float.64

shakeel608: <pre><code class="lang-auto">RuntimeError: "log_softmax_lastdim_kernel_impl" not implemented for 'Long'

output = model(batch)
_, pred = torch.max(output, 1)   
loss(pred, targets)
print(f"pred output = {pred.dtype}, target={targets.dtype}")
</code></pre>
<p>I am getting the following error <em>RuntimeError: “log_softmax_lastdim_kernel_impl” not implemented for ‘Long’</em></p>
<p>When I checked the data type of “pred”  and “targets” and it gives me torch.int64 and torch.float64.</p>
<p>I tried to use target.long() but it is saying  <code>RuntimeError: Expected floating point type for target with class probabilities, got Long</code></p>
<p>When I use pred = pred.to(torch.float32), it works but then I am getting error in loss.backward()</p>
<pre><code class="lang-auto">RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

</code></pre>
<p>in the loss function .</p>
<p>How to fix this ?</p>

ptrblck: <p><code>pred</code> represents the indices corresponding to the max. values in <code>output</code>. This <code>argmax</code> operation is not differentiable so pass the logits directly to your loss function if it’s e.g. <code>nn.CrossEntropyLoss</code>.</p>
<p>The targets should then contain either class indices as <code>long</code> or in newer PyTorch versions could alternatively contain probabilities as <code>float</code> to represent soft targets.</p>

shakeel608: <p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a> . It solved my issue</p>
Why the default negative_slope for kaiming_uniform initialization of Convolution and Linear layers is √5?

lihx: <p>I noticed that the default initialization method for Conv and Linear layers in Pytorch is Kaimiing_uniform.</p>
<p>I just don’t understand why the default value of negative_slope(the default act is leaky_relu) is √5.</p>
<p>Is it written just for simplicity or for some specific reason?</p>
<pre><code>def reset_parameters(self):
    init.kaiming_uniform_(self.weight, a=math.sqrt(5))
    if self.bias is not None:
        fan_in, _ = init._calculate_fan_in_and_fan_out(self.weight)
        bound = 1 / math.sqrt(fan_in)
        init.uniform_(self.bias, -bound, bound)</code></pre>

tom: <p>My understanding (<a href="https://github.com/pytorch/pytorch/pull/9038" rel="nofollow noopener">the last update is here</a>) is that this is mainly from casting the <em>init as it has always been done</em> to the <code>kaiming_uniform_</code> initialization scheme. I would <em>not</em> read it as a recommendation for that particular activation function.</p>
<p>As such, for me there are two takeways:</p>
<ul>
<li>This is a scheme that has been used for quite some time as the default,</li>
<li>if you believe that (Kaiming) He’s initialization scheme is a good thing, including the parametrization, and your activation doesn’t happen to be leaky_relu with slope √5, you might consider overriding the initialization with a <code>kaiming_init_</code> aligned with your activation function.</li>
</ul>
<p>Best regards</p>
<p>Thomas</p>
<p>Acknowledgement: Thanks to Vishwak and <a class="mention" href="/u/fmassa">@fmassa</a> who were involved in the effort of documenting and streamlining how the initializations are coded, for their response when I asked about this, any errors and bad ideas above are my own.</p>

lihx: <p>Thanks for you reply.</p>
<blockquote>
<p>this is mainly from casting the  <em>init as it has always been done</em>  to the  <code>kaiming_uniform_</code>  initialization scheme.</p>
</blockquote>
<p>This actually solves my confusion.</p>

liangbright: <p>The default values of the implementation are confusing</p>
<p><a href="https://pytorch.org/docs/stable/" rel="nofollow noopener">https://pytorch.org/docs/stable/</a><em>modules/torch/nn/init.html#kaiming_uniform</em><br>
def kaiming_uniform_(tensor, a=0, mode=‘fan_in’, nonlinearity=‘leaky_relu’)<br>
Args:<br>
tensor: an n-dimensional <code>torch.Tensor</code><br>
a: the negative slope of the rectifier used after this layer (0 for ReLU<br>
by default)</p>
<p><a href="https://pytorch.org/docs/stable/nn.html#leakyrelu" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/nn.html#leakyrelu</a><br>
LeakyReLU( negative_slope=0.01 , inplace=False* )</p>
<p>a=0 for Relu, but  nonlinearity=‘leaky_relu’<br>
the default value of negative_slope is 0.01</p>

ado_sar: <p>Yep, I have found it confusing also. If gain is what is only affected, then only parameter <code>a</code> should be there.</p>
AttributeError: module 'torch.nn.utils.parametrizations' has no attribute 'weight_norm'

ninodv: <p>Hello,<br>
Recently tried to install tortoise fast tts, and installed all the corresponding packages with it, includig PyTorch. While trying to launch it, I got the following error: "AttributeError: module ‘torch.nn.utils.parametrizations’ has no attribute ‘weight_norm’ ".<br>
My question is, should I search for help here or at the tortoise side? And if you have an idea, how can I fix this?<br>
I have python 3.9.13,  torch 1.13.1 (Which was needed for tortoise). I installed torch with pip.<br>
I am on a windows pc, win32? and my license is not activated, if that information would help.<br>
Thx for the help<br>
Greetings, ninodv</p>
I get a shape error

pak_key: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/1/1152a7c7b9a81849487b0b4fb3a778f5c2f6f16c.png" data-download-href="https://discuss.pytorch.org/uploads/default/1152a7c7b9a81849487b0b4fb3a778f5c2f6f16c" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/1/1152a7c7b9a81849487b0b4fb3a778f5c2f6f16c_2_690x410.png" alt="image" data-base62-sha1="2tfcJoh82f1b77Nt9B5UyebQwgY" width="690" height="410" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/1/1152a7c7b9a81849487b0b4fb3a778f5c2f6f16c_2_690x410.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/1/1152a7c7b9a81849487b0b4fb3a778f5c2f6f16c_2_1035x615.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/1/1/1152a7c7b9a81849487b0b4fb3a778f5c2f6f16c_2_1380x820.png 2x" data-dominant-color="262727"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1419×845 55.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
How to achieve Back Propagation a complex-value model?

109390177: <p>I try to use a simple mo  del to rebuild a complex-value signal, I split the real and imag to two real numbers, and the label also has two parts, the real part is labelr, and the imag part is labeli, here’s the model, based on <a href="https://github.com/wavefrontshaping/complexPyTorch/blob/master/complexPyTorch/complexLayers.py" rel="noopener nofollow ugc">https://github.com/wavefrontshaping/complexPyTorch/blob/master/complexPyTorch/complexLayers.py</a></p>
<blockquote>
<p>class model(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(model,self).<strong>init</strong><br>
self.conv1=ComplexConv2d(1,64,3,1,1);<br>
self.conv2=ComplexConv2d(64,1,3,1,1);<br>
def forward(self,xr,xi):<br>
xr,xi=self.conv1(xr,xi);<br>
xr,xi=self.conv2(xr,xi);<br>
return xr,xi</p>
</blockquote>
<p>And I use mse loss:</p>
<blockquote>
<p>loss1=loss_fn(yr,labelr);<br>
loss2=loss_fn(yi,labeli);<br>
loss=loss1+loss2<br>
loss.backward()</p>
</blockquote>
<p>I don’t know if it’s the correct way to work, or I need to modify the loss function or backward function to make it fit complex-values? I don’t know where to change the code.</p>

J_Johnson: <p>When it comes to loss for complex numbers, it’s a field being studied. There are several ways to approach it.</p>
<p>If you’re attempting to predict a complex value, for signal processing, a Cartesian approach would be to calculate the distance between two points. This would be done via:</p>
<p>d = ((x2 - x1)^2 + (y2 - y1)^2)^0.5</p>
<p>Or in code:</p>
<pre><code class="lang-auto">def cart_loss(yr, yi, labelr, labeli):
    return ((yr - labelr)**2 + (yi - labeli)**2)**0.5
</code></pre>
<p>Another approach that is more natural for complex values is to find the polar loss. That is the difference between the radii and the angles. This recognizes that complex values are not simply x, y coordinates. We can obtain this via:</p>
<pre><code class="lang-auto">def polar_loss(yr, yi, labelr, labeli):
    y = torch.complex(yr,yi)
    label = torch.complex(labelr, labeli)
    rad_loss = torch.abs(y.abs() - label.abs())
    angle_loss = torch.abs(torch.angle(label/y))
    return rad_loss + angle_loss
</code></pre>
<p>You may find it beneficial to scale the radial loss within the same range as the angle loss. This can be done with a modified version of the above:</p>
<pre><code class="lang-auto">def polar_loss(yr, yi, labelr, labeli, eps = 1e-8):
    y = torch.complex(yr,yi)
    label = torch.complex(labelr, labeli)
    rad_loss = torch.abs(y.abs() - label.abs())/((torch.max(y.abs(), label.abs()) + eps) # normalizes the value in a range of 0 to 1
    angle_loss = torch.abs(torch.angle(label/y))/math.pi # normalizes the value to a range of 0 to 1
    return (rad_loss + angle_loss)/2
</code></pre>
What is k-dimensional crossentropyloss?

Goldname: <p>In the docs <a href="https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html#torch.nn.CrossEntropyLoss" class="inline-onebox" rel="noopener nofollow ugc">CrossEntropyLoss — PyTorch 2.1 documentation</a> they discuss k-dimensional loss.</p>
<p>Can someone explain to me what k-dimensional loss is? Is this related to contrastive loss or symmetric loss or something like that?</p>

J_Johnson: <p>If you have a problem where you have multiple nodes needing classification, then you may need k-dimensional CE loss.</p>
<p>A good example of this is object detection in a picture. Every pixel can be treated separately for classification. One pixel may be a “nothing” class, while another is “cat” and another “dog”, and so on.</p>

vdw: <p>Another example would be sequence labeling of text data, for example:</p>
<ul>
<li>
<p>Part-of-Speech Tagging <em>or</em></p>
</li>
<li>
<p>Named Entity Recognition</p>
</li>
</ul>
<p>where in both cases you want to predict a class label for each word in your sentence.</p>

Goldname: <p>Got it thanks. Do you know of any source I can read more about the math behind this? It’s still not clear to me how it works and whether or not it’s the loss functionality i’m looking for</p>

J_Johnson: <p>As far as loss is concerned, the number of k-dims is irrelevant as those will just get flattened, anyway.</p>
<p>It doesn’t differ from Cross Entropy Loss where we have only one classification made.</p>
<p>Suppose we have some photo for segmentation, that is evaluating each pixel separately for what type of object it contains. Each pixel has a corresponding index as a target, the correct class. And each pixel’s loss is evaluated independently.</p>
<p>Whether we keep the image 2d or flatten it doesn’t change anything since each pixel is evaluated individually.</p>
<p>And then you use the <code>reduction</code> argument to convey how to deal with those individual losses. It defaults to mean.</p>
RuntimeError: Shared memory manager connection has timed out

HELLORPG: <p>During my training process, I received the following error (probably appeared at the junction of two epochs):</p>
<pre data-code-wrap="shell"><code class="lang-shell">Traceback (most recent call last):
  File "anaconda3/envs/SORTIP/lib/python3.11/multiprocessing/queues.py", line 244, in _feed
    obj = _ForkingPickler.dumps(obj)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "anaconda3/envs/SORTIP/lib/python3.11/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
  File "anaconda3/envs/SORTIP/lib/python3.11/site-packages/torch/multiprocessing/reductions.py", line 417, in reduce_storage
    metadata = storage._share_filename_cpu_()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "anaconda3/envs/SORTIP/lib/python3.11/site-packages/torch/storage.py", line 297, in wrapper
    return fn(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "anaconda3/envs/SORTIP/lib/python3.11/site-packages/torch/storage.py", line 334, in _share_filename_cpu_
    return super()._share_filename_cpu_(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Shared memory manager connection has timed out
</code></pre>
<p>This error occurs randomly, so I can not locate it in my code or even reproduce it stably.</p>
<p>Does anyone have any idea about this? Thanks a lot~</p>

HELLORPG: <p>I still do not know the reason for this error. But I guess it may be caused by the dataloader multiprocessing between adjacent epochs (maybe the multiprocessing startup or shutdown). I re-read the <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader" rel="noopener nofollow ugc">document</a> and find that <code>persistent_workers=True</code> can persistent the worker processes between epochs. If my assumption is correct, this option would bypass the error I met.</p>
<p>Now, I use <code>persistent_workers=True</code> in my dataloader, and I haven’t met this problem again so far. <strong>But I still don’t know what caused this, and I’m not sure it won’t happen again.</strong> If anyone can shed some light on this problem, it would be greatly appreciated.</p>
Get RuntimeError: One of the differentiated Tensors does not require grad in pytorch lightning

Yuanzhi: <p>This is the key part in configure_optimizers() function in pytorch lightning:</p>
<pre><code class="lang-python">params = []
train_names = []
print("Training only unet attention layers")
for name, module in self.model.diffusion_model.named_modules():
    if isinstance(module, CrossAttention) and name.endswith('attn2'):
        train_names.append(name)
        params.extend(module.parameters())
    else:
        # Set requires_grad=False for all other parameters
        for param in module.parameters():
            param.requires_grad = False
opt = torch.optim.AdamW(params, lr=lr)
</code></pre>

marksaroufim: <p>Guessing without a repro but can you try explicitly setting <code>param.requires_grad = True</code> in your if condition</p>

Yuanzhi: <p>Hi Marksaroufim,</p>
<p>Thank you for replying.</p>
<p>What I want to do here is only setting attention parameters requires_grad = True and passing them to the optimizer.<br>
It did work if I set all the parameters with requires_grad = True.</p>
<p>I just found out that this error seems to be caused by the custom-defined class CheckpointFunction(torch.autograd.Function)<br>
and I just made the code work by simply avoiding using this class</p>

Jing_Yang: <p>Hi, Yuanzhi, I met the same problem when I set the ‘attention layer’ trainable only. Could you clarify how to change the CheckpointFunction(torch.autograd.Function)?</p>

dorian_Zhang: <p>I also encountered the same situation when I debugged the code. Did you solve this problem? I look forward to receiving your reply</p>

200tortoise: <p>I’m facing the same issue here. The reason CheckpointFunction do not work is that it will try to get the grads of parameters that requires_grad=False.<br>
For those who want to use CheckpointFunction in this case, you can modify the backward function like this:</p>
<pre><code class="lang-auto">class CheckpointFunction(torch.autograd.Function):
     .......
    @staticmethod
    def backward(ctx, *output_grads):
        ......

        grads = torch.autograd.grad(
            output_tensors,
            ctx.input_tensors + [x for x in ctx.input_params if x.requires_grad],
            output_grads,
            allow_unused=True,
        )

        grads = list(grads)

        # Assign gradients to the correct positions, matching None for those that do not require gradients
        input_grads = []
        for tensor in ctx.input_tensors + ctx.input_params:
            if tensor.requires_grad:
                input_grads.append(grads.pop(0))  # Get the next computed gradient
            else:
                input_grads.append(None)  # No gradient required for this tensor

        del ctx.input_tensors
        del ctx.input_params
        del output_tensors
        return (None, None) + tuple(input_grads)
</code></pre>
BBOX get error after v2.transform

hysskm: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/b/db55be8979266a7f69a94e2d68908b7bbe29be4f.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/db55be8979266a7f69a94e2d68908b7bbe29be4f" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/b/db55be8979266a7f69a94e2d68908b7bbe29be4f_2_690x337.jpeg" alt="image" data-base62-sha1="vikgSLASGTRQIlJ0s0rq0uYFRbp" width="690" height="337" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/b/db55be8979266a7f69a94e2d68908b7bbe29be4f_2_690x337.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/d/b/db55be8979266a7f69a94e2d68908b7bbe29be4f.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/d/b/db55be8979266a7f69a94e2d68908b7bbe29be4f.jpeg 2x" data-dominant-color="382827"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">950×464 102 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
How to weight option for binary_cross_entropy

sroyyors: <p>Hi,<br>
I was wondering how in C++ I can specify the weight parameter in the binary_cross_entropy function.</p>
<p>I am using something like</p>
<pre><code class="lang-auto">auto loss_classification = torch::nn::functional::binary_cross_entropy(output_class, target_class);
</code></pre>
<p>I tried using</p>
<pre><code class="lang-auto">auto loss_classification = torch::nn::functional::binary_cross_entropy(output_class, target_class, torch::nn::BCELossOptions().weight("0.7"));
</code></pre>
<p>as a third argument but it did not work.</p>

sroyyors: <p>OK I was able to get this work using</p>
<pre><code class="lang-auto">torch::Tensor weight=torch::zeros(1);
weight.index_put_({0},0.7);
auto loss_classification = torch::nn::functional::binary_cross_entropy(output_class, target_class,torch::nn::functional::BinaryCrossEntropyFuncOptions().weight(weight));
</code></pre>
<p>felt like an overkill to make a tensor of 1 dimension, but this compiled and executed.</p>
Do we need to bother with BPTTs for basic LSTM implementations anymore?

lIlBrother: <p>I used to implement and use BPTT myself back in the day.<br>
I’m now using torch 2.0 or later,<br>
do I still need to implement BPTT myself for basic LSTM usage?</p>
<p>Should I implement something like truncated bptt myself?</p>
<p>If these features were added, when were they added?</p>
Help！！How to install pytorch（C++） in the vscode and Apple M1 ？！

Eric0: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/5/f5e8391fa4e7562bdbf568f5de84f17263c6d7a0.png" data-download-href="https://discuss.pytorch.org/uploads/default/f5e8391fa4e7562bdbf568f5de84f17263c6d7a0" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/5/f5e8391fa4e7562bdbf568f5de84f17263c6d7a0.png" alt="image" data-base62-sha1="z5ovZF0cUbmcf4hS9lEGOgAtTVu" width="690" height="91" data-dominant-color="2E2F37"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1096×146 14.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
custom calculation

aemrhb: <p>hi everyone<br>
I have a question about custom calculation before passing it to the loss function. Considering we have a network that generates a feature map I want to do some logic calculations without having any learnable parameter and pass it to the loss function. Would this calculation cause a problem for the backward method?<br>
thank you so much and Best Regards.</p>

ptrblck: <p>You shouldn’t see any issues if you are using differentiable operations on the model output.</p>

aemrhb: <p>thank you for your response.<br>
even if they are not part of the feed-forward they should be differentiable? I include the pseudo code here :</p>
<pre><code>        output, _ = generator(input) # trianable model 
        custom_output = logic_cacualtion(output)
        custom_output_gt = logic_cacualtion(GT)
     
        loss =  torch.nn.functional.mse_loss(custom_output,custom_output_gt)
</code></pre>

ptrblck: <p>Yes, as generally PyTorch/Autograd does not care if an operation is inside a model’s <code>forward</code> function or in a global context. The computation graph will be created using the tracked operations so they should be differentiable if you want to backpropagate through them.</p>

aemrhb: <p>I really appreciate your response.<br>
I have yet two other questions.<br>
1-So if I want to use unfold and reshape is it impossible? if yes how backpropagation works for this function?<br>
2- can I ensure the gradient is calculated by  :</p>
<pre><code class="lang-auto"> for name, param in model.named_parameters():
                 if param.grad is not None:
                      print(model - Parameter: {name}, Gradient Norm: {param.grad.norm()}')
</code></pre>
<p>Best Regards.</p>

soulitzer: <ul>
<li>unfold and reshape are differentiable, so autograd should work</li>
<li>not sure what you mean by your second question, do you want to compute second order gradients?</li>
</ul>

aemrhb: <p>thank you for your response.<br>
actually, I want to be sure that the gradient for each parameter is calculated. so can I use that code to print the gradient of model parameters?</p>