akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Bifurcation of a pretrained model and error in blockwise forward pass

Prafful_Kumar: <p>Aim: To extract the intermediate features (say from block_num=4) from a pretrained resnet50. Then feed the same extracted feature to the next blocks (In between I have to do some manipulation, but I have not done it here). Even the shape of the block feature is correct that is required for the next layer. I am unable to solve this issue. Kindly help.</p>
<p>import torch<br>
import torch.nn as nn</p>
<p>from torchvision import models</p>
<p>model = models.resnet50(pretrained=True)</p>
<p>model.eval()</p>
<p>inputs = torch.randn(2, 3, 224, 224)</p>
<p>block_num = 4</p>
<p>feature_extractor = nn.Sequential(*list(model.children())[:block_num + 1])<br>
rest_of_network = nn.Sequential(*list(model.children())[block_num+1 :])<br>
block_features = feature_extractor(inputs)<br>
print(block_features.shape)</p>
<p>perturbed_block_features = rest_of_network(block_features) → ERROR</p>
<p>terminal output:</p>
<p>torch.Size([2, 256, 56, 56])<br>
Traceback (most recent call last):<br>
File “trial.py”, line 22, in <br>
perturbed_block_features = rest_of_network(block_features)<br>
File “/home/prafful/scratch/conda_envs/MRG/lib/python3.8/site-packages/torch/nn/modules/module.py”, line 1051, in _call_impl<br>
return forward_call(*input, **kwargs)<br>
File “/home/prafful/scratch/conda_envs/MRG/lib/python3.8/site-packages/torch/nn/modules/container.py”, line 139, in forward<br>
input = module(input)<br>
File “/home/prafful/scratch/conda_envs/MRG/lib/python3.8/site-packages/torch/nn/modules/module.py”, line 1051, in _call_impl<br>
return forward_call(*input, **kwargs)<br>
File “/home/prafful/scratch/conda_envs/MRG/lib/python3.8/site-packages/torch/nn/modules/linear.py”, line 96, in forward<br>
return F.linear(input, self.weight, self.bias)<br>
File “/home/prafful/scratch/conda_envs/MRG/lib/python3.8/site-packages/torch/nn/functional.py”, line 1847, in linear<br>
return torch._C._nn.linear(input, weight, bias)<br>
RuntimeError: mat1 and mat2 shapes cannot be multiplied (4096x1 and 2048x1000)</p>
<p>when run using cuda, I got this error → RuntimeError: CUDA error: CUBLAS_STATUS_INVALID_VALUE when calling<br>
<code>cublasSgemm( handle, opa, opb, m, n, k, &amp;alpha, a, lda, b, ldb, &amp;beta, c, ldc)</code></p>
<p>Please help me to solve this issue. I am not able to understand why this simple code is not working.</p>
<p>Best Regards,</p>
<p>Prafful</p>

ptrblck: <p>Wrapping submodules into an <code>nn/Sequential</code> container will often fail as you would be missing all functional API calls. For <code>resnet50</code> you would miss <a href="https://github.com/pytorch/vision/blob/0b41ff0b0a08229a10cfe1ca6987b4386d68bd9c/torchvision/models/resnet.py#L279">this <code>torch.flatten</code> operation</a> which is most likely causing the issue.<br>
Either add an <code>nn.Flatten</code> module manually to your <code>nn.Sequential</code> container or derive a custom model and override the <code>forward</code> method with your custom approach.</p>

Prafful_Kumar: <p>Thank you, it works fine.</p>
What am I missing in my understanding of the role of "persistent_id" in PyTorch

jinwoosorj: <p>I am digging into the torch.save and the data flow of the PyTorch just because I am self-interested how this is using python and c++ to store its data.</p>
<p>So, in my short studying (it’s been just a couple of weeks), I am kinda understood what Pickle does to serialize the data. And now I am stuck at understanding the persistent_id(obj) in torch.save.</p>
<p>By merely googling what is “persistent_id” in Pickle (which is not implemented at first in the Pickle.py), I understood that it makes not to be duplicated so that we don’t serialize the same objects.</p>
<p>However, when I look at the code inside of the “persistent_id(obj)” in serialization.py in PyTorch. It does something else. But, I am totally lost to fully understand what the reason is that it exists.</p>
<p>So, to just briefly explain how I understood the method, I will shortly go over the flow of the code.</p>
<ol>
<li>Assume that I call torch.save(model.state_dict(), PATH)</li>
<li>Assume that the object goes through _save(obj)</li>
<li>In _save(obj), it initializes Pickler itself and goes into Pickle.py</li>
<li>In Pickle.py, it uses persistent_id made by Serialization.py in PyTorch</li>
<li>Persistent_id(obj) firstly filters whether it is tensor or not by checking TypedStorage</li>
<li>if it is true, it makes object to untyped storage because tensors could be various types</li>
<li>Then, it is mapping the obj’s data pointer to store the address in memory so that it is not directly saving the tensor data to the memory</li>
<li>In the method, it stores the address and the index will be returned so that PyTorch can find the address later.</li>
<li>It returns storage type and index, then Pickle.py serialize the returned information.</li>
</ol>
<p>In my curiosity by looking at it, is it just made for organizing in the checkpoint file?<br>
This is because I guess persistent_id is not necessary for saving since when we can load the models by just using only Pickle (w/o persistent_id).<br>
<strong>What does actually "persistent_id(obj) do in PyTorch?</strong></p>
<p>I am sorry if I am questioning in wrong way in this community because this is my second question in this community and I have no where to ask my curiosity.</p>
Forward_pre Hook in Quantized Model

Ghadeer_Jaradat: <p>I am trying to upate the the input to the linear layer in BERT transformer using forward_pre_hooks,</p>
<blockquote>
<p>nn=model_dynamic_quantized.bert.encoder.layer[0].attention.self.key</p>
<p>def gethook(mask):<br>
def forward_pre_hook(model_dynamic_quantized, input):<br>
print(input)<br>
input=input*0<br>
return input</p>
<p>h=nn.register_forward_pre_hook(gethook(0))</p>
<p>classifier = pipeline(task=‘text-classification’, model=model_dynamic_quantized,<br>
tokenizer=tokenizer)</p>
<p>results = classifier(“A stirring, funny and finally transporting re-imagining of Beauty and the Beast and 1930s horror films”)<br>
print(results)</p>
<h1>detach the hooks</h1>
<p>h.remove()</p>
</blockquote>
<h2><a name="i-am-getting-this-error-1" class="anchor" href="#i-am-getting-this-error-1"></a>I am getting this error</h2>
<p><strong>TypeError: Kernel.raw_input() takes from 1 to 2 positional arguments but 3 were given</strong></p>
Fix DDP stuck issue by adding NCCL_P2P_LEVEL=NVL

lxysl: <p>I ran the DDP code and it got stuck. I opened this issue:<br>
<a href="https://github.com/rickyang1114/DDP-practice/issues/1" rel="noopener nofollow ugc">Solve stuck issue by setting NCCL_P2P_LEVEL=NVL · Issue #1 · rickyang1114/DDP-practice (github.com)</a><br>
I got the solution from this link:<br>
<a href="https://github.com/pytorch/pytorch/issues/73206" rel="noopener nofollow ugc">DDP gets stuck on A40 GPUs · Issue #73206 · pytorch/pytorch (github.com)</a><br>
My question is why setting NCCL_P2P_LEVEL=NVL is useful? Is that a bug to be fixed?</p>
Deploying models using torchserve workflow on aws

HemaDevaSagar: <p>Hello,</p>
<p>I have multiple pytorch models which form a DAG. Torchserve has workflows to deploy such DAG inference workflow on local, but I am not sure how to use it (torchserve workflows) to deploy them on aws.</p>
<p>Any thoughts on how this can be done is really appreciated</p>
<p>Thank you!</p>

marksaroufim: <p>We have 2 examples you can checkout here <a href="https://github.com/pytorch/serve/tree/master/examples/Workflows">https://github.com/pytorch/serve/tree/master/examples/Workflows</a></p>
<p>What specifically about deploying to AWS are you asking about?</p>

HemaDevaSagar: <p>I have already checked out the examples in the above mentioned repo. They seem to be serving the workflow on the local machine. I am wondering how to serve the workflow as an aws endpoint.</p>
<p>For example, I know how to deploy a single torch model using torchserve on aws as mentioned in this (<a href="https://aws.amazon.com/blogs/machine-learning/deploying-pytorch-models-for-inference-at-scale-using-torchserve/" rel="noopener nofollow ugc">Deploying PyTorch models for inference at scale using TorchServe | AWS Machine Learning Blog</a>). I am wondering if something like this can be done with torchserve workflow (when we have multi-models in a dag). I searched over internet to see if anyone have deployed uisng torch workflow on aws, but no luck. Thank you!</p>
(more) Dilation support for MaxPool / AvgPool?

zisisnotzis: <p>Currently, <code>AvgPool1d</code> doesn’t support dilation at all. While <code>MaxPool1d</code> do support dilation, it’s limited by:</p>
<pre><code class="lang-auto">In [6]: MaxPool1d(2,stride=1,padding=2,dilation=2)(torch.rand(2,4,8))
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Cell In[6], line 1
----&gt; 1 MaxPool1d(2,stride=1,padding=2,dilation=2)(torch.rand(2,4,8))

File ~/p10/lib/python3.10/site-packages/torch/nn/modules/module.py:1501, in Module._call_impl(self, *args, **kwargs)
   1496 # If we don't have any hooks, we want to skip the rest of the logic in
   1497 # this function, and just call forward.
   1498 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1499         or _global_backward_pre_hooks or _global_backward_hooks
   1500         or _global_forward_hooks or _global_forward_pre_hooks):
-&gt; 1501     return forward_call(*args, **kwargs)
   1502 # Do not call functions when jit is used
   1503 full_backward_hooks, non_full_backward_hooks = [], []

File ~/p10/lib/python3.10/site-packages/torch/nn/modules/pooling.py:92, in MaxPool1d.forward(self, input)
     91 def forward(self, input: Tensor):
---&gt; 92     return F.max_pool1d(input, self.kernel_size, self.stride,
     93                         self.padding, self.dilation, ceil_mode=self.ceil_mode,
     94                         return_indices=self.return_indices)

File ~/p10/lib/python3.10/site-packages/torch/_jit_internal.py:484, in boolean_dispatch.&lt;locals&gt;.fn(*args, **kwargs)
    482     return if_true(*args, **kwargs)
    483 else:
--&gt; 484     return if_false(*args, **kwargs)

File ~/p10/lib/python3.10/site-packages/torch/nn/functional.py:696, in _max_pool1d(input, kernel_size, stride, padding, dilation, ceil_mode, return_indices)
    694 if stride is None:
    695     stride = torch.jit.annotate(List[int], [])
--&gt; 696 return torch.max_pool1d(input, kernel_size, stride, padding, dilation, ceil_mode)

RuntimeError: max_pool1d() padding should be at most half of kernel size, but got padding=2 and kernel_size=2
</code></pre>
<p>Which I think do makes sense since adding dilation will “eat” more output, which should allow more padding as compensation.</p>
<p>BTW, it would even be better to allow one sided padding (instead of both sided).</p>
Different behavior of TorchScripted and python function

Amit_Gupta: <p>Hi, I have noticed that the following function yields different values for hessian/2nd derivative (<code>0.0</code> vs <code>nan</code>) depending on whether it was jitted or not. Is this intended behavior? I have replicated this behavior in torch 1.12, 1.13, and 2.1.0</p>
<pre data-code-wrap="In"><code class="lang-plaintext">
In [2]: torch.__version__
Out[2]: '2.1.0.dev20230309+cpu'

In [3]: from torch.autograd.functional  import hessian

In [4]: def ssp(x):
   ...:     return torch.nn.functional.softplus(x) -  0.69314
   ...: 

In [5]: x = torch.tensor(120.0)

In [6]: hessian(ssp, x)
Out[6]: tensor(0.)

In [7]: hessian(ssp, x)
Out[7]: tensor(0.)

In [8]: ssp = torch.jit.script(ssp)

In [9]: hessian(ssp, x)
Out[9]: tensor(0.)

In [10]: hessian(ssp, x)
Out[10]: tensor(nan)
</code></pre>
<p>Note: In some cases the second call to hessian yield <code>nan</code> but in most cases the jitted function always give <code>nan</code> on second derivative (specially it jitted using decorator <code>@torch.jit.script</code>)</p>
Need help in torch tensorrt

k191459: <p>I’m bit new in torch tensorrt, but need help with how I’m gonna run torch tensorrt on any models.<br>
Can someone provide me with an example? I’m trying to convert hf model into torch tensorrt but couldn’t convert it successfully, here is the code please convert this code into running torch tensorrt code:</p>
<p>from transformers import AutoModel AutoFeatureExtractor</p>
<p>checkpoint_name = “microsoft/resnet-50”<br>
feature_extractor = AutoFeatureExtractor.from_pretrained(checkpoint_name)<br>
torch_resnet = AutoModel.from_pretrained(checkpoint_name)<br>
x = torch.rand((1, 3, 224, 224))</p>
<h1><a name="from-here-to-implement-torhc-tensorrt-1" class="anchor" href="#from-here-to-implement-torhc-tensorrt-1"></a>FROM HERE TO IMPLEMENT TORHC TENSORRT</h1>

marksaroufim: <p>There’s an example you can follow form the official tensorRT docs here <a href="https://github.com/pytorch/TensorRT#python" class="inline-onebox">GitHub - pytorch/TensorRT: PyTorch/TorchScript/FX compiler for NVIDIA GPUs using TensorRT</a></p>
How to check which cuda version my pytorch is using

JaeJin_Cho: <p>Hello!</p>
<p>I have multiple CUDA versions installed on the server, e.g., /opt/NVIDIA/cuda-9.1 and /opt/NVIDIA/cuda-10, and /usr/local/cuda is linked to the latter one. I believe I installed my pytorch with cuda 10.2 based on what I get from running torch.version.cuda.</p>
<p>How can I check which version of CUDA that the installed pytorch actually uses in running? I set my CUDA_PATH=/opt/NVIDIA/cuda-9.1 but it still seems to run without any problem on a gpu.</p>
<p>Thanks,<br>
Jaejin Cho</p>

ptrblck: <p>You could check the linked CUDA version via <code>print(torch.version.cuda)</code>.</p>

JaeJin_Cho: <p>Thank you for the quick answer.</p>
<p>I actually used “torch.version.cuda” and checked it as in my original post (and it was 10.2). However, I was wondering why it still works with the setting “CUDA_PATH=/opt/NVIDIA/cuda-9.1”. In other words, how it finds the correct cuda version to use (or which environment variable it uses to do so) in run-time?</p>

ptrblck: <p><code>CUDA_PATH</code> might be the wrong env var. I’m using <code>CUDA_HOME</code>, which works fine.</p>

JaeJin_Cho: <p>Thank you for the answer!</p>
<p>Added this code in my script,</p>
<p>logging.warning(‘cuda version: {}’.format(torch.version.cuda))<br>
import os<br>
logging.warning(‘CUDA_PATH: {}’.format(os.environ[“CUDA_PATH”]))<br>
logging.warning(‘CUDA_HOME: {}’.format(os.environ[“CUDA_HOME”]))</p>
<p>Then, the outputs are like below:<br>
WARNING: cuda version: 10.2<br>
WARNING: CUDA_PATH: /opt/NVIDIA/cuda-9.1<br>
WARNING: CUDA_HOME: /opt/NVIDIA/cuda-9.1</p>
<p>It seems working fine but I am still curious what pytorch refers to for the correct CUDA version under the hood.</p>

ptrblck: <p>The <code>CUDA_HOME</code> env var is used during the build of PyTorch in order to compile the CUDA kernels, so you would need to rebuild it.</p>

doem97: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> I am a little confused by cuda version, torch version.</p>
<p>My CUDA_HOME currently points to <code>cuda v11.1</code></p>
<p>While my existing torch installation reads <code>torch.version.cuda=11.7</code></p>
<p>Which cuda executive lib it is calling? I am confused because this wrong-version torch seems running quite fine.</p>

ptrblck: <p>PyTorch ships with its own CUDA dependencies and your locally installed CUDA toolkit will be used if you build PyTorch from source or a custom CUDA extension.</p>
How to use if else condition in `torch.nn` module

palguna_gopireddy: <p>I am able to compare two tensors in <code>if-else</code> statement like:</p>
<pre><code class="lang-auto">if torch.gt(torch.tensor(3),torch.tensor(2)):
    print('Greater')

Greater
</code></pre>
<p>But I am not able to use the same in <code>nn.Module</code>. For example,</p>
<pre><code class="lang-auto">class user_def(nn.Module):
    def __init__(self):
        super(user_def, self).__init__()
  
    def forward(self, x):
        if torch.gt(x,torch.tensor(0)):
            return torch.tensor(2)
        else:
            return torch.tensor(0)
    
x = torch.linspace(-10, 10, 100)
k = user_def()
y = k(x)
  
# plot the function graph
plt.plot(x, y)
plt.grid(True)
plt.title('Function')
plt.xlabel('x')
plt.ylabel('y')
plt.show()
</code></pre>
<p>It is giving error:</p>
<pre><code class="lang-auto">Cell In[39], line 6, in user_def.forward(self, x)
      5 def forward(self, x):
----&gt; 6     if torch.gt(x,torch.tensor(0)):
      7         return torch.tensor(2)
      8     else:

RuntimeError: Boolean value of Tensor with more than one value is ambiguous
</code></pre>
<p>How to overcome this.</p>

ptrblck: <p><code>x</code> contains multiple values so the condition would return a <code>BoolTensor</code> containing <code>True/False</code> for each element, which Python cannot use in the <code>if</code> statement.<br>
Either reduce it e.g. via <code>.any()</code> or <code>.all()</code> or split the tensor manually based on the condition and concatenate it afterwards.</p>

palguna_gopireddy: <p>I do not know how to split the tensor manually.<br>
I used <code>torch.gt(x,torch.tensor(0)).all()</code> in the <code>if</code> condition. But when I plotted it using</p>
<pre><code class="lang-auto">x = torch.linspace(-10, 10, 100)
k = user_def()
y = k(x)
  
# plot the softplus function graph
plt.plot(x, y)
plt.grid(True)
plt.title(' Function')
plt.xlabel('x')
plt.ylabel('y')
plt.show()
</code></pre>
<p>It is giving error: <code>ValueError: x and y must have same first dimension, but have shapes torch.Size([100]) and (1,)</code></p>
<p>Basically I am trying to define an activation function, that results different values for input tensor x&gt;0, x&lt;0.<br>
Is there anyway to do it?</p>

ptrblck: <p>Yes, you can create a mask as described before:</p>
<pre><code class="lang-python">x = torch.randn(10, 10)
mask = x &gt; 0.
x_pos = x[mask]
x_neg = x[~mask]

# apply your different operations here
x_pos = x_pos + 100.
x_neg = x_neg - 100.

output = torch.empty_like(x)
output[mask] = x_pos
output[~mask] = x_neg

print(((output &gt; 0.) == (x &gt; 0.)).all())
# tensor(True)
</code></pre>
Loss in Physics Informed Neural Network goes to Nan

naz.ga: <p>I am solving the same differential equation using two different methods. One uses finite differences (<strong>code A</strong>) and the other uses a neural network in Nvidia Modulus, a deep learning framework built on PyTorch (<strong>code B</strong>). The problem is, that the differential equation in code A has a <strong>k</strong> term in <code>def rhs</code> , in the differential equation, and I do not know how to code for it in <strong>code B.</strong> Please help me in this area.</p>
<p>Below is my executable <strong>code A</strong> :</p>
<pre><code class="lang-auto">import numpy as np

nz = 50
dz = 1 / nz
nt = 50
t = np.linspace(0, 4, nt)
p = np.array([0.01, 0.5, 1.0])

#Initialize concentration and temperature arrays
C = np.zeros(nz)
T = np.zeros(nz)

#Set initial condition for concentration
C[0] = 1.0

#Initialize arrays to store results
c_results = np.zeros((nt, nz))
T_results = np.zeros((nt, nz))

#Define the Arrhenius rate constant function
def k(temperature):
arrh_expr = 4.68 * np.exp(-806 / ((600 - 273) * temperature + 273))
return arrh_expr

#Calculate time derivative at specific spatial points
def rhsC(c0, c1, c2, T1, dz, p):
return (p[0] * (c2 - 2 * c1 + c0) / dz**2 - p[1] * (c2 - c0) / (2 * dz) - p[2] * k(T1) * c1**2)

def rhsT(T0, T1, T2, c1, dz, p):
return (0.01 * (T2 - 2 * T1 + T0) / dz**2 - 0.5 * (T2 - T0) / (2 * dz) + 2.45 * k(T1) * c1**2)

#Time-stepping loop
for it in range(nt):
c_results[it, :] = C
T_results[it, :] = T

for iz in range(1, nz - 1):
# Calculate concentration and temperature derivatives
dc_dt = rhsC(C[iz - 1], C[iz], C[iz + 1], T[iz], dz, p)
dT_dt = rhsT(T[iz - 1], T[iz], T[iz + 1], C[iz], dz, p)

#Update concentration and temperature using forward Euler
C[iz] += dz * dc_dt

T[iz] += dz * dT_dt
</code></pre>
<p>Below is a snippet of my attempt in <strong>code B</strong> :</p>
<pre><code class="lang-auto">class PFR_equation2D(PDE): 
     name = "PFR_equation2D"
     
     def __init__(self):
         # z coordinate
         x = Symbol("x")
 
         #time
         t = Symbol("t")
 
         input_variables = {"x": x, "t": t}
         c = Function("c")(*input_variables)
         Temp = Function("Temp")(*input_variables) 
 
         def k(Temp):
             arrh_exp = 4.68 *exp(-806 / ((600 - 273) * Temp + 273))
             return arrh_exp
 
         self.equations = {}
         self.equations["PFR_equation_c"] =  c.diff(t,1) - 0.01*(c.diff(x,2)) + 0.5*(c.diff(x)) + 1.0*(c**2) 
         
         self.equations["PFR_equation_T"] =  Temp.diff(t,1) - 0.01*(Temp.diff(x,2)) + 0.5* (Temp.diff(x)) - 2.45*k(Temp)*(c**2)
</code></pre>
How do i print out the point in time for each tensor in forward and backward process?

Cru: <p>I wana to measure the point in time for each tensor in training. For exmaple, when i train a sample network in python api, i want to get the point in time for each tensor. I have inserted some “chrono” time api in “relu” func in Actiovation.cpp (I have not inserted the code into backward func part), but it is too intricate to insert some code in each implement file. Can it be reliazed by controled by a global (singleton pattern)  time counter in impl file or somewher? I am not realy sure about  this part code .</p>
How to convert Tensorflow Multi-head attention to Pytorch equivalent?

Yasaman_Salehi: <p>Both Tensorflow and Pytoch have functions for Transformer’s Multi-head attention and they gets arguments as follows:<br>
tensorflow.keras.layers.MultiHeadAttention(num_heads, key_dim)<br>
torch.nn.MultiheadAttention(embed_dim, num_heads)<br>
where num_heads stands for number of heads in multi-head attention module and key_dim and embed_dim are explained below:</p>
<p>Assume in a NLP example we have an input of size [1, 10, 512] which means we have a sentence with 10 words and each word is a [1, 512] vector after passing through the tokenizer. so the embed_dim is 512.<br>
now if we have a multi-head attention with num_heads = 8 , the input would be resized to [1, 10, 8, 64]. so, each of these heads would have dimension of [1, 10, 64]. if I understand it correctly, the key_dim in Tensorflow is this 64 in each head.</p>
<p>I have a Tensorflow transformer and I’m trying to convert it to Pytorch. The input data is a [1, 136, 4] matrix which is used for both query and key.<br>
In TF code, the multi-head attention gets num_head = 8 and key_dim = 4. it means the embed_dim in Pytorch must be key_dim * num_heads = 32 but my data is [1, 136, 4].<br>
Does in mean I have to repeat my input for 8 times and have the key and query of size [1, 136, 32]?<br>
how can I give a [1, 136, 4] data to a Transformer with 8 heads in Pytorch?<br>
I should mention the input is a sequence of 136 bounding box coordinates including x, y, w &amp; h.<br>
Thank you in advanced.</p>
Pytorch Bug Bounty

Ankush_Goel: <p>Hi,</p>
<p>I wish to inform you that I have identified a critical vulnerability within the PyTorch/Glow repository on GitHub. Could you please confirm if there is an established bug bounty program for this particular repository? Additionally, would it be appropriate to report this issue through the Meta/Facebook bug bounty program, or is there a preferred method or point of contact for such disclosures?</p>
<p>Awaiting your guidance.</p>

Mah_Neh: <p>Try <code>/security</code></p><aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/pytorch/pytorch/security">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/pytorch/pytorch/security" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <img width="500" height="500" src="https://github.githubassets.com/images/modules/open_graph/github-logo.png" class="thumbnail onebox-avatar">

<h3><a href="https://github.com/pytorch/pytorch/security" target="_blank" rel="noopener nofollow ugc">Build software better, together</a></h3>

  <p>GitHub is where people build software. More than 100 million people use GitHub to discover, fork, and contribute to over 330 million projects.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


Ankush_Goel: <p><a href="https://www.facebook.com/whitehat" rel="noopener nofollow ugc">https://www.facebook.com/whitehat</a> doesnt list pytorch as inscope. Is pytorch in scope or not? If so, is it eligible for rewards?</p>

Ankush_Goel: <p><a class="mention" href="/u/mah_neh">@Mah_Neh</a> <a href="https://www.facebook.com/whitehat" rel="noopener nofollow ugc">https://www.facebook.com/whitehat</a> doesnt list pytorch as inscope. Is pytorch in scope or not? If so, is it eligible for rewards?</p>
Determinism in pytorch across multiple files

Megh_Bhalerao: <p>Hi,</p>
<p>Suppose I have the following python files <code>main.py</code>, <code>utils.py</code>, <code>trainer.py</code>. My overall objective is to get deterministic behavior across all files.</p>
<p>My entry point to the code is <code>main.py</code>. <code>main.py</code> calls a function which is located in <code>utils.py</code> <code>set_random_seeds(seed)</code>. The function is as follows -</p>
<pre><code class="lang-auto">def set_random_seed(seed: int) -&gt; None:
    """
    Sets the seeds at a certain value.
    :param seed: the value to be set
    """
    print("Setting seeds ...... \n")
    torch.manual_seed(seed)
    torch.cuda.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)
    torch.backends.cudnn.benchmark = False
    torch.backends.cudnn.deterministic=  True
</code></pre>
<p>Now, <code>main.py</code> also calls a function <code>train_code()</code> from <code>train.py</code> after the <code>set_random_seed()</code> function.</p>
<p>I have a basic question - does setting the random seeds in one of the files guarantee that <code>torch</code>, <code>numpy</code>, <code>random</code> or any other package uses the same seed? If yes, how does python know that the same seed is to be used in all the different files? Is this the job of the linker or some related thing?</p>
<p>Thank you!</p>

ptrblck: <p>Your script will be executed in a main Python process, which would load the libraries, set the seeds etc. As long as this main process calls all methods, the seeds will be kept and the state of the script is consistent.<br>
If you use multiprocessing, you might need to re-seed the code as the behavior of forking or spawning new processes could be different between libraries. E.g. take a look at <a href="https://github.com/pytorch/pytorch/issues/5059">this issue</a> which describes numpy’s behavior.</p>
Kaiming He initialisation not preserving variance

niz: <p>Hello,</p>
<p>As I understand it He initialisation was developed to preserve the variance of layers - when used in conjunction with a ReLU activation function. I noticed that I was getting a vanishing gradient, with batch normalisation I got an exploding gradient. I did some analysis and found that the layers do not preserve the variance. The variance repeatedly shrank by a constant factor (<code>~0.8</code> for <code>a=0.2</code>) for each layer-activation pair.<br>
Am I missing something? Am I doing something wrong?</p>
<p>I was using a <code>a=0.2</code> in my activation function, however this problem persisted with <code>a=0</code>, and with <code>init.kaiming_unifrom_()</code>.<br>
Below is some code using a normal distribution as an input to showcase the situation (a uniformly distributed input had the same problem):</p>
<pre><code class="lang-auto">#A layer class with He initialisation
class LinearHe(nn.Linear):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        torch.nn.init.kaiming_normal_(self.weight, a=0.2)

#A Linear layer with 1024 input and output features
linear1 = LinearHe(1024, 1024) 
leaky_relu = nn.LeakyReLU(negative_slope=0.2)

#A normally distributed dataset of size 1024 with mean=0, std=1
a = torch.normal(torch.zeros(1, 1024), torch.ones(1, 1024))
a
&gt;&gt;&gt; tensor([[-1.8505,  0.7651,  1.8227,  ..., -0.3863,  0.6085,  0.6416]])
torch.var(a)
&gt;&gt;&gt; tensor(0.9165)

b = leaky_relu(linear1(a))
torch.var(b)
&gt;&gt;&gt; tensor(0.7133, grad_fn=&lt;VarBackward0&gt;)
#This is a repeating result, the variance is scaled by a factor of roughly 0.8
torch.mean(b)
&gt;&gt;&gt; tensor(0.4346, grad_fn=&lt;MeanBackward0&gt;)
</code></pre>
Predict a signal considering other related signal using RNN (LSTM)

Jalil: <p>Hi.<br>
Suppose we have a speed signal as a target in the regression problem.<br>
Having several related signals such as acceleration, etc., I want to train a recurrent neural network that predicts the speed signal by taking input of features (acceleration signals, etc.).</p>
<p>Now I have three questions:</p>
<ol>
<li>
<p>Is it reasonable to use RNNs like LSTM for such a problem? And in general, what networks do you recommend for such a task?</p>
</li>
<li>
<p>What is the best way to feed data to the model? Is it fine to give a Data-Frame where the features are in the columns and the rows represent the samples in time, as the input of the network?<br>
And how should hidden state and cell state be defined and update in the model so that the dependence of features and targets on the previous time step is considered.</p>
</li>
<li>
<p>In the sliding window used in RNN, does the model use previously predicted values ​​to predict new values ​​or does it use the actual value in the dataset?</p>
</li>
</ol>
<p>I will be happy to hear your comments.<br>
Thanks.</p>
Is it possible to limit driver_allocated_memory

Vincent-liuwingsang: <p>I’m using <a href="https://github.com/UKPLab/sentence-transformers" rel="noopener nofollow ugc">sentence-transformer</a> to turn strings into sentences with mps enabled. I noticed that driver_allocated_memory grows as I process more strings which I don’t quite understand. This behaviour doesn’t exist when mps is disabled</p>
<p>Is there a way to limit the driver_allocated_memory for a process?</p>
Isolate processess and instances on separate GPU's, same machine - How to?

lostalot: <p>I am running two separate pytyhon virtual environments in Linux, on a machine with three GPUs (GPU-0  is for the display).   nvidia-smi shows each of the non display GPUs shows its own 4 MB /usr/lib/xorg/Xorg process before I do anything.</p>
<p>When I launch a training run on one of the GPU-1, nvidia-smi shows  GPU-1 being loaded with 3680 MB of /bin/python3.11 program.  So far-so good.</p>
<p>When, from the second virtual python environment, I launch a second training program on GPU-2, nvidia-smi shows GPU-2 being loaded up with 3644 MB of /bin/python3.22 program.  HOWEVER, at the same time a new program appears on GPU-1 that nvidia-smi shows as being /bin/python3.11 for 306 MB.</p>
<p>Can someone explain the reason that launching a second training program, from a second virtual environment, on the second GPU on the same machine, causes a process to appear on the first GPU?</p>
<p>Is there something that can be done to completely isolate the processes?</p>
<blockquote>
<p>|    1   N/A  N/A      2154      G   /usr/lib/xorg/Xorg                            4MiB |<br>
|    1   N/A  N/A     62386      C   …/bin/python3.11                          3680MiB |<br>
|    1   N/A  N/A     62772      C   …/bin/python3.11                           306MiB |<br>
|    2   N/A  N/A      2154      G   /usr/lib/xorg/Xorg                            4MiB |<br>
|    2   N/A  N/A     62772      C   …/bin/python3.11                          3644MiB |</p>
</blockquote>

ptrblck: <aside class="quote no-group" data-username="lostalot" data-post="1" data-topic="187985">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/l/858c86/48.png" class="avatar"> lostalot:</div>
<blockquote>
<p>Can someone explain the reason that launching a second training program, from a second virtual environment, on the second GPU on the same machine, causes a process to appear on the first GPU?</p>
</blockquote>
</aside>
<p>You are most likely allowing the process to see all devices, might then select <code>cuda:1</code> for the main work but are also calling into methods, which would use the “default” device (<code>cuda:0</code>) in this case.<br>
As a quick workaround, use <code>CUDA_VISIBLE_DEVICES=1 python script.py</code> and use <code>cuda:0</code> internally.</p>

lostalot: <p>That works.  Thank you again <a class="mention" href="/u/ptrblck">@ptrblck</a>.</p>
Dataloader slowing down with each batch

zainsarwar865: <p>I have an image dataset of 30k images and I have loaded a previously trained Resnet50 model. My goal is to get gradients of each input sample individually for the last layer thus my batch size = 1.</p>
<p>Now, here’s my problem. Over the 30k batches, the dataloader slows down considerably. The first 10k batches are loaded and processed (forward pass + gradients for the last layer computed) in around 3 minutes but the next 10k take around 10 minutes and the remaining ten take about 20 minutes to load.</p>
<p>I am resizing and normalizing the image using PyTorch Transforms.</p>
<p>For reference, I am on a GPU cluster with Nvidia A100s. The machine has 8 such GPUs and none of them are being used. The dataset is being pulled in from an NFS server but the problem persists even when I moved the dataset to my local directory. I also resized my images before running my model so that I don’t have to do it on the go but the problem still persists.</p>
<p>Here’s another finding. If I break my dataloader at 10k iterations and then restart it again, the speed is again pretty good (3 minutes / 10k) so I suspect it has something to do with the dataloader somehow taking up memory.</p>
<p>I would really appreciate any help on this matter.</p>
<p>Thank you!</p>
Bug (?): aten ir nodes have wrong overload op

Nitzan_Shaked: <p>briefly: a fw_compiler gets a graph with aten-ir nodes, but some nodes with <code>node.op == "call_function"</code> have a <code>node.target</code> that points to the wrong <code>OpOverload</code>. Here’s the simplest example possible.</p>
<pre><code class="lang-auto">#!/usr/bin/env python3

from typing import Any, Callable

import torch
import torch.fx
from torch._functorch.aot_autograd import aot_module_simplified


def my_fw_compiler(
    gm: torch.fx.graph_module.GraphModule,
    sample_input: Any,
) -&gt; Callable[..., Any]:
    assert isinstance(gm, torch.fx.graph_module.GraphModule)
    for node in gm.graph.nodes:
        assert isinstance(node, torch.fx.node.Node)
        if node.op not in ("placeholder", "output"):
            print(node.op, node.target)
            print("    schema:  ", node.target._schema)
            print("    args:    ", "(", ", ".join(
                f"{type(a).__name__} {a}"
                for a in node.args
            ), ")")
    return gm.forward  # type: ignore


def my_dynamo_backend(
    gm: torch.fx.graph_module.GraphModule,
    sample_input: Any,
) -&gt; Callable[..., Any]:
    assert isinstance(gm, torch.fx.graph_module.GraphModule)
    return aot_module_simplified(
        gm,
        sample_input,
        fw_compiler=my_fw_compiler,
    )


class SimplestModule(torch.nn.Module):
    def forward(self, input: torch.Tensor) -&gt; torch.Tensor:
        return input + 5e-1


def main() -&gt; None:
    sample_input = torch.ones(4, dtype=torch.float32, device="cpu")

    regular_model = SimplestModule()
    regular_model.eval()
    regular_model_result = regular_model(sample_input)

    compiled_model = torch.compile(
        regular_model,
        backend=my_dynamo_backend,
        fullgraph=True,
    )
    assert isinstance(compiled_model, torch.nn.Module)
    compiled_model_result = compiled_model(sample_input)

    print()
    print(regular_model_result)
    print(compiled_model_result)


if __name__ == "__main__":
    main()
</code></pre>
<p>when running:</p>
<pre><code class="lang-auto">$ ./bug_repro.py
call_function aten.add.Tensor
    schema:   aten::add.Tensor(Tensor self, Tensor other, *, Scalar alpha=1) -&gt; Tensor
    args:     ( Node arg0_1, float 0.5 )

tensor([1.5000, 1.5000, 1.5000, 1.5000])
tensor([1.5000, 1.5000, 1.5000, 1.5000])
</code></pre>
<p>as can be seen, <code>node.target</code> is the “Tensor” overload, but the 2nd argument in <code>node.args</code> is clearly not a tensor…</p>
<p>Am I missing the obvious here? maybe such scary terms as boxing / wrappers / trace modes (came across those while digging through the code).</p>
<p>Context: I am toying with a “compiler” that generates some (non-Python) code from the graph along the way, and as part of that I implemented a “function call actual matches argument type” code, and to my surprise it failed.</p>
<p>Cheers!</p>
Explicit control orders of related static (un)scoped variables

jinglijing: <p>in torch：</p>
<ul>
<li>dispatcher(<strong>libtorch</strong>)</li>
<li>python_registrations(<strong>libtorch_python</strong>)</li>
</ul>
<p>manage registered kernel functions, like <code>PythonKernelHolder</code>, which contains <code>SafePyObject</code> that holds :</p>
<ul>
<li>self_interpreter(<strong>libtorch_python</strong>)</li>
<li>noop_vtable(<strong>libtorch</strong>)</li>
</ul>
<p>for python-aware lifetime management of PyObject. However, above 4 global/static scoped/unscoped variables will be destructed after the program exits with implicit orders, and:</p>
<ul>
<li><code>libtorch_python</code> links <code>libtorch</code>.</li>
<li><code>dispatcher</code> and <code>python_registrations</code> depend <code>self_interpreter</code>/<code>noop_vtable</code>.</li>
</ul>
<p>For now, implicit orders are normal after <a href="https://github.com/pytorch/pytorch/pull/95464" rel="noopener nofollow ugc">PR</a> in <code>DEBUG</code> mode. Should we be explicit about their dependeies? like:</p>
<ul>
<li>meyer’s singleton of get scoped static <code>noop_vtable</code> .</li>
<li>In <code>dispatcher</code>/<code>python_registrations</code>, explicit call get_noop_vtable ().</li>
</ul>

ptrblck: <p>I would recommend creating an issue on GitHub to discuss it with the code owners directly.</p>
How to Initialize network output

zshn25: <p>I have a regression problem where a neural network predicts the output which have a range of 0 to inf. I know that the output should be close to some heuristic and would like to initialize the output accordingly.</p>
<p>In what way can I do this?</p>

shivammehta007: <p>Change the weights of the last output layer to zero and bias to the initialisation value.</p>
<pre><code class="lang-python">last_layer = nn.Linear(10, 10)
last_layer.weight.data.zero_()
last_layer.bias.data.fill_(1.0) # replace 1.0 with the required value

x = torch.randn(10)
last_layer(x)
# tensor([1., 1., 1., 1., 1., 1., 1., 1., 1., 1.], grad_fn=&lt;AddBackward0&gt;)

</code></pre>
<p>Essentially, “biasing” the output of the network. Hope this helps <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
Can't install Pytorch on Ubuntu 23.04

smahi: <p>Hi,<br>
I am facing the following error when attempting to install <code>Pytorch</code> on <code>Ubuntu 23.04</code>, I am using <code>Python V 3.11</code> installed and managed via <a href="https://asdf-vm.com/" rel="noopener nofollow ugc">ASDF VERSION MANAGER</a></p>
<p><code>pip list</code> :</p>
<pre><code class="lang-auto">Package            Version
------------------ --------
...
setuptools         68.0.0
...
</code></pre>
<p>the command <code>pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118 </code> throw the following error</p>
<pre><code class="lang-auto">  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 1
  ╰─&gt; [3 lines of output]
      Looking in indexes: https://download.pytorch.org/whl/cu118
      ERROR: Could not find a version that satisfies the requirement setuptools&gt;=40.8.0 (from versions: none)
      ERROR: No matching distribution found for setuptools&gt;=40.8.0
      [end of output]

</code></pre>
<p>Please help</p>

smahi: <p>It’s seems to be an issue with <code>Python 3.11</code>, just changing the version to <code>3.10.12</code> make it working properly without issue.</p>
<p>If you are using <code>ASDF VERSION MANAGER</code> then do the following:</p>
<p><code>$ asdf install python 3.10.12</code></p>
<p>To to set a specific version of <code>Python</code> for your project, run the following from inside your project folder:</p>
<p><code>$ asdf local python 3.10.12</code></p>
<p>Then install <code>Pytorch</code>:</p>
<p><code>$ pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118</code></p>
<p>To verify the installation:</p>
<p><code>$ python</code></p>
<pre><code class="lang-auto">Python 3.10.12 (main, Jul 10 2023, 11:47:06) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
&gt;&gt;&gt; import torch
&gt;&gt;&gt; x = torch.rand(5, 3)
&gt;&gt;&gt; print(x)
tensor([[0.4309, 0.9209, 0.4374],
        [0.4121, 0.4171, 0.0754],
        [0.5129, 0.2636, 0.3915],
        [0.1323, 0.1035, 0.9279],
        [0.6893, 0.3870, 0.8031]])
&gt;&gt;&gt; 
</code></pre>

alierkan: <p>You can use Python 3.11.4 if you don’t update pip and setuptools. With pip==23.0.1 and setuptools==66.1.1, I installed pytorch 2.0.1(cuda 1.18) successfully.</p>
Question about masked MultiheadAttention

Behrad: <p>Hello,</p>
<p>I am training a ViT Encoder. <strong>For each input,</strong> I have two masks: First mask shows the padding area and the second mask is used to help the attention module attend more or less to some specific parts.</p>
<p>Input: (1, 706, 384) → 1 : batch size, 706: number of patch embeddings, 385: embedding dimensions<br>
padding_mask: (1, 706) : binary values to show with patches correspond the padding<br>
input_mask: (1, 706): float values [0-1] to show important patches</p>
<p>How can I define a self attention layer in pytorch?<br>
should I create one mask( including 0 for padding and a float value for importance) and give to <strong>key_padding_mask</strong>? Or do I need to use <strong>attn_mask</strong>?</p>
<p>Thank you in advance.</p>
Understand Memory Usage of Pytorch Tensors for Inference

Maxwellu: <p>I plan on translating large text corpora from various languages to english with Large Language Models. Therefore, I tried messing around a bit to see the computational limits of my machine.</p>
<p>Specifically, I try to translate a text corpus by using the No Language Left Behind translation model of Meta with the following code:</p>
<pre><code class="lang-auto">from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
device = "cpu"
tokenizer = AutoTokenizer.from_pretrained("facebook/nllb-200-distilled-600M")
model = AutoModelForSeq2SeqLM.from_pretrained("facebook/nllb-200-distilled-600M").to(device)

def translationPipeline(text):
    input_ids = tokenizer.encode(text, return_tensors="pt").to(device)
    outputs = model.generate(input_ids,forced_bos_token_id=tokenizer.lang_code_to_id["deu_Latn"])
    decoded = tokenizer.decode(outputs[0], skip_special_tokens=True)
    return decoded

# c = df_text.iloc[1,:].body.apply(translationPipeline)
with torch.no_grad():
    c = translationPipeline(df_text.iloc[0,:].body)
</code></pre>
<p>Here, <code> df_text.iloc[0,:].body</code> is one text corpus that equates to roughly 150,000 characters. The tokenizer transforms this to a Tensor of shape (1, 42809). If I execute the above code, I get a out-of-memory error on my 256 GB RAM machine.</p>
<p>I am aware that the token length of my text corpus exceeds the maximum length for this model. However, I am still wondering where this high RAM usage comes from. In its raw form, 150,000 characters should of course not consume 250 GB of memory.</p>
<p>Where does the additional RAM required come from? How much RAM would be needed to translate the whole text corpus at once?</p>
In reset method in c10/util/intrusive_ptr.h

jinglijing: <p>Refer to:</p><aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/6c7260407b35aadcaf7807cbc2ad320a16ced785/c10/util/intrusive_ptr.h#L272-L292">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/6c7260407b35aadcaf7807cbc2ad320a16ced785/c10/util/intrusive_ptr.h#L272-L292" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/6c7260407b35aadcaf7807cbc2ad320a16ced785/c10/util/intrusive_ptr.h#L272-L292" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/6c7260407b35aadcaf7807cbc2ad320a16ced785/c10/util/intrusive_ptr.h#L272-L292</a></h4>



    <pre class="onebox"><code class="lang-h">
      <ol class="start lines" start="272" style="counter-reset: li-counter 271 ;">
          <li>void reset_() noexcept {</li>
          <li>  if (target_ != NullType::singleton() &amp;&amp;</li>
          <li>      detail::atomic_refcount_decrement(target_-&gt;refcount_) == 0) {</li>
          <li>    // See comment above about weakcount. As long as refcount&gt;0,</li>
          <li>    // weakcount is one larger than the actual number of weak references.</li>
          <li>    // So we need to decrement it here.</li>
          <li>    bool should_delete =</li>
          <li>        target_-&gt;weakcount_.load(std::memory_order_acquire) == 1;</li>
          <li>    if (!should_delete) {</li>
          <li>      // justification for const_cast: release_resources is basically a</li>
          <li>      // destructor and a destructor always mutates the object, even for const</li>
          <li>      // objects. NOLINTNEXTLINE(cppcoreguidelines-pro-type-const-cast)</li>
          <li>      const_cast&lt;std::remove_const_t&lt;TTarget&gt;*&gt;(target_)-&gt;release_resources();</li>
          <li>      should_delete =</li>
          <li>          detail::atomic_weakcount_decrement(target_-&gt;weakcount_) == 0;</li>
          <li>    }</li>
          <li>    if (should_delete) {</li>
          <li>      delete target_;</li>
          <li>    }</li>
          <li>  }</li>
          <li>}</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
When decrement refcount_ from 1 to 0, why not <code>release_resources()</code> directly but additionally determine  <code>weakcount != 1</code>? Should weakcount_ controls the object itself, and refcount_controls the resources occupied by the object？</p>