akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to avoid re-decoding for multiple inputs that have shared prefixes

sim_inf: <p>My question is about Huggingface library, but it may have an answer at Pytorch level.<br>
I have a Huggingface pipeline for text generation (intentionally avoiding the details):</p>
<pre><code class="lang-auto">llm = pipeline('text-generation', model='gpt2-xl')
</code></pre>
<p>I would like to use the model above to do generation for a long list of input texts (for each entry individually). The data has a specific property, and that is all the entries of the list have an identical prefix string. So to be more specific, assume the simplified example below:</p>
<blockquote>
<p>INSERT A LONG PREFIX HERE first piece of text<br>
INSERT A LONG PREFIX HERE second piece of text<br>
INSERT A LONG PREFIX HERE third piece of text<br>
INSERT A LONG PREFIX HERE fourth piece of text<br>
.<br>
.<br>
.</p>
</blockquote>
<p>The naive way to do generation for each entry is to use a for loop and input each piece separately. But one can also think of decoding the prefix once, and then to re-use the cached state of the model to only decode the distinct part of each entry to get the output. I wonder if there is a straightforward way to do this?</p>
Not able to run pytorch parallel on Google colab

sunil_s: <p>Hi,</p>
<p>We are trying process video files to extract features from it, we are using keras3 with pytorch. We could run multiprocessing code on our local laptop but same code is not working on google colab, Please find the notebook with the parallel execution code –<a href="https://colab.research.google.com/drive/1e1CIo_GVqVy2XSdIBIW8DICMc-ACGOSB#scrollTo=7Rpcf2ExPpSs" class="inline-onebox" rel="noopener nofollow ugc">Google Colab</a></p>
<p>On our local laptop we can see sub processes start execution as we have print statements showing the progress of execution, but on colab it silently finishes processing. Can someone please help on the same</p>
Torch.eq VS == (equality operator)

stevethesteve: <p>If <code>a</code> is some tensor and <code>b</code> is a tensor or a number, is there a difference between <code>torch.eq(a, b)</code> and <code>a == b</code> ?<br>
If not, why does Pytorch API specify the <code>torch.eq</code> method?<br>
Should I avoid using the <code>==</code> operator and use <code>torch.eq</code> instead?</p>

ptrblck: <p>I think both approaches should yield the same result:</p>
<pre data-code-wrap="python"><code class="lang-python">a = torch.arange(10)
b = 2

print(a == b)
print(torch.eq(a, b))

b = torch.arange(10)
print(a == b)
print(torch.eq(a, b))
</code></pre>
<aside class="quote no-group" data-username="stevethesteve" data-post="1" data-topic="147847">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/stevethesteve/48/12812_2.png" class="avatar"> stevethesteve:</div>
<blockquote>
<p>If not, why does Pytorch API specify the <code>torch.eq</code> method?</p>
</blockquote>
</aside>
<p>I guess:</p>
<ul>
<li>convenience if you prefer to use the explicit <code>torch.*</code> methods, e.g. such as <code>torch.add</code> instead of <code>+</code></li>
<li>to allow users to pass the <code>out</code> tensor</li>
<li>I guess to map it easily to the internal <code>aten::eq</code> operator (but unsure if adding <code>torch.eq</code> actually makes it easier)</li>
</ul>

stevethesteve: <p>OK, I marked ptrblck’s answer as solution. Maybe adding a small hint about the equivalence of <code>==</code> and <code>torch.eq</code> would be something worth adding into the documentation?</p>

Stardance: <p>Actually, ==(equality operator) and torch.eq are <strong>different</strong>!<br>
You can try</p>
<pre><code class="lang-auto">print(b == a) 
print(torch.eq(b, a))
</code></pre>
<p>with</p>
<pre><code class="lang-auto">a = torch.arange(5)
b = 2
</code></pre>
<p>Then you will find*(in Python3.11 and PyTorch 2.2)* the equality operator can run while the torch.eq will give TypeError.</p>
<p>However, this is not the end of the story. as b is <code>int</code> in the above example, if we pass PyTorch tensor instead, both comparison will give the desired result.</p>
<pre><code class="lang-auto">a = torch.arange(5)
b = torch.arange(2,3)
print(b == a) 
# the output is tensor([False, False,  True, False, False])
print(torch.eq(b, a))
# the output is tensor([False, False,  True, False, False])
</code></pre>
<p>One final note, you can treat <code>a == b</code> as trying to run both <code>torch.eq(a, b)</code> and <code>torch.eq(b, a)</code>.</p>

Huawei_Zhu: <p>No difference in normal usage.</p>
<p>But there is a huge difference when you try to do model conversion using torchscript torch.jit.script() or onnx torch.onnx.export()</p>
<p>using signs like &gt;, &lt;, == will result in model conversion failture for torch.onnx.export(), and also somtetimes affect torch.jit.script()<br>
<a class="mention" href="/u/ptrblck">@ptrblck</a></p>
OOM memory due to computation graph when stacking embeddings

saxenarohit: <p>Hi,</p>
<p>I am trying out the following code which stacks embeddings for manipulation later.</p>
<pre><code class="lang-auto">from transformers import BartTokenizer, BartModel,BartForConditionalGeneration
import torch


def encode():

    print("Loading model")
    tokenizer = BartTokenizer.from_pretrained('facebook/bart-large-cnn')
    model = BartForConditionalGeneration.from_pretrained('facebook/bart-large-cnn')
    model.to("cuda")

    encoer = model.get_encoder() #returns object of BARTEncoder

    sentences_to_encode = [str(i)+" This is my paragraph."*50  for i in range(50)]
    tensorStack = []
    for idx, para in enumerate(sentences_to_encode):
        inputs = tokenizer.encode(para,return_tensors='pt',truncation=True,padding=True)
        tensorStack.append(encoer(inputs.to("cuda")).last_hidden_state)
        print("Encoded para "+str(idx),"Mem:", round(torch.cuda.memory_allocated()/1024**3,2),"GB")


if __name__ == '__main__':
    encode()

</code></pre>
<p>When stacking tensors that are the output of the Transformer encoder, I observe an increase in GPU memory after each iteration. I believe this is due to the storage of the computation graph.</p>
<p>Is there a way to prevent the creation of new intermediate tensors on GPU for each iteration and tie their weights or do checkpointing? I intend to perform backpropagation on these embeddings, so using <code>torch.no_grad()</code> is not a suitable solution for me.<br>
Thanks</p>

soulitzer: <p>You might be look for <a href="https://pytorch.org/docs/stable/checkpoint.html" class="inline-onebox" rel="noopener nofollow ugc">torch.utils.checkpoint — PyTorch 2.2 documentation</a></p>

saxenarohit: <p>Thanks! let me check if I can work this out.<br>
I was also thinking if I could distribute the encoder, but since it is the same reference of the encoder object going through paragraphs I am not entirely sure how to distribute computation to multiple gpus</p>
How do I pass a forward value to a backward function

Josh_Steve: <p>If there are two autograds function f and g and backward() function involving the composite (fog) is called, how can I make the backward(ctx, *grad_outputs) of f dependent on the forward value of g (not just its gradients)?</p>

soulitzer: <p>Hmm not sure what you mean by this. Could you talk a bit more about your example? Are you trying to produce higher-order gradients?</p>
C++ linking failure

caiomcg: <p>Dears, I am trying to link pytorch statically with a cmake project. For that I am using FetchContent to download Pytorch 2.2.2 and make it available to be compiled as such:</p>
<p>CMakeLists.txt:</p>
<pre><code class="lang-auto">...

include(FetchContent)

set(FETCHCONTENT_QUIET FALSE)

FetchContent_Declare(torch
    GIT_REPOSITORY https://github.com/pytorch/pytorch
    GIT_TAG v2.2.2)

set(BUILD_SHARED_LIBS OFF)
set(USE_KINETO OFF)

FetchContent_MakeAvailable(torch)

# Manually list all .h and .cpp files for the plugin
set(SourceFiles
    src/main.cpp
    )

add_executable("${PROJECT_NAME}" ${SourceFiles})

target_link_libraries("${PROJECT_NAME}"
    PRIVATE
    torch
    )

...
</code></pre>
<p>The issue is, once I invoke make, the pytorch library seems to be compiled however, I always see this linking error where ATen/Tensor.h is not found. Am I missing something on my cmake? Maybe include directories? Any help is appreciated.</p>
<p>Torch Build:</p>
<pre><code class="lang-auto">[ 35%] Linking CXX static library ../../../lib/libtorch_cpu.a
[ 35%] Built target torch_cpu
[ 35%] Building CXX object _deps/torch-build/caffe2/CMakeFiles/torch.dir/__/__/__/empty.cpp.o
[ 35%] Linking CXX static library ../../../lib/libtorch.a
warning: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: archive library: ../../../lib/libtorch.a the table of contents is empty (no object file members in the library define global symbols)
[ 35%] Built target torch
</code></pre>
<p>Linker error:</p>
<pre><code class="lang-auto">In file included from /Users/caiomcg/Workspace/project/src/main.cpp:2:
In file included from /Users/caiomcg/Workspace/project/build/_deps/torch-src/torch/csrc/api/include/torch/torch.h:3:
In file included from /Users/caiomcg/Workspace/project/build/_deps/torch-src/torch/csrc/api/include/torch/all.h:7:
In file included from /Users/caiomcg/Workspace/project/build/_deps/torch-src/torch/csrc/api/include/torch/autograd.h:3:
In file included from /Users/caiomcg/Workspace/project/build/_deps/torch-src/c10/../torch/csrc/autograd/autograd.h:3:
In file included from /Users/caiomcg/Workspace/project/build/_deps/torch-src/c10/../torch/csrc/autograd/variable.h:6:
In file included from /Users/caiomcg/Workspace/project/build/_deps/torch-src/c10/../torch/csrc/autograd/cpp_hook.h:2:

/Users/caiomcg/Workspace/project/build/_deps/torch-src/c10/../torch/csrc/autograd/function_hook.h:3:10: fatal error: 'ATen/Tensor.h' file not found
#include &lt;ATen/Tensor.h&gt;
</code></pre>
Unconsistent (Deep)Copying Behavior CPU/GPU?

hugosou: <p>Hello,<br>
This must be already known issue (but just in case it might be worth reporting it): Pytorch behavior is not consistent when working on GPU <em>Vs</em> CPU: parameters are deep copied only in the former case when initializing an <code>nn.Module</code></p>
<p>While I understand why this might be the case, it’s rather cumbersome. Below is a ‘minimal’ example where I build 2 modules with the same parameters on the CPU and on the GPU :</p>
<pre><code class="lang-auto">import torch.nn as nn

class dummy_class(nn.Module):
    def __init__(self, param):
        super().__init__()
        self.param = nn.Parameter(param)
    
    def print_param(self):
        return str(self.param[0].item())
        

# Initial Parameter
param = torch.tensor([0.5])

# Devices
cpu_device = torch.device("cpu")
gpu_device = torch.device("cuda")

# Build 2 dummy_class object on CPU and GPU
on_cpu = [dummy_class(param).to(cpu_device) for _ in range(2)]
on_gpu = [dummy_class(param).to(gpu_device) for _ in range(2)]
</code></pre>
<p>Now if, I modify only one object and print the result, the behavior differs:</p>
<pre><code class="lang-auto"># Print Param 
print_param = lambda x: [print('Object ' + str(ii) + '= ' + obji.print_param()) for ii, obji in enumerate(x)]

print('======')
print('On CPU')
print('======')
print('Initial Parameter')
print_param(on_cpu)
on_cpu[0].state_dict()['param'] += 1
print('After Modifying Object 0')
print_param(on_cpu)
print('')

print('======')
print('On GPU')
print('======')
print('Initial Parameter')
print_param(on_gpu)
on_gpu[0].state_dict()['param'] += 1
print('After Modifying Object 0')
print_param(on_gpu)
print('')

</code></pre>
<p>Outputs:</p>
<pre><code class="lang-auto">======
On CPU
======
Initial Parameter
Object 0= 0.5
Object 1= 0.5
After Modifying Object 0
Object 0= 1.5
Object 1= 1.5

======
On GPU
======
Initial Parameter
Object 0= 0.5
Object 1= 0.5
After Modifying Object 0
Object 0= 1.5
Object 1= 0.5


</code></pre>

ptrblck: <p>Yes, inplace operations do not work between different devices.</p>

hugosou: <p>Oh of course. The issue is that <code>param</code> was created on the CPU, not that the behavior differs. Thanks.</p>
What the meaning of `exitcode -6`

cokespace2: <p>I was training a Yolo model with DDP, and got DDP error with messages below:</p>
<pre><code class="lang-auto">[1]:
  time      : 2024-04-17_15:48:22
  host      : mccx6
  rank      : 4 (local_rank: 4)
  exitcode  : -6 (pid: 3882833)
  error_file: &lt;N/A&gt;
  traceback : Signal 6 (SIGABRT) received by PID 3882833
[2]:
  time      : 2024-04-17_15:48:22
  host      : mccx6
  rank      : 6 (local_rank: 6)
  exitcode  : -6 (pid: 3882835)
  error_file: &lt;N/A&gt;
  traceback : Signal 6 (SIGABRT) received by PID 3882835
--------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2024-04-17_15:48:22
  host      : mccx6
  rank      : 2 (local_rank: 2)
  exitcode  : -6 (pid: 3882831)
  error_file: &lt;N/A&gt;
  traceback : Signal 6 (SIGABRT) received by PID 3882831
=======================================================
</code></pre>
<p>It usally happened at the end of an epoch, and I don’t know what the exitcode -6 means.</p>
Implement C++ autograd function and use it in Python

G.M: <p>I’m new to the C++ version of Pytorch and I’m <strong>not very familiar</strong> with C++. I want to write a custom autograd function in C++ and use it in Python. My goal is similar to what this <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html#writing-a-c-extension" rel="nofollow noopener">tutorial</a> is doing, except that I don’t want to wrap the function in Python using autograd.Function, what I want is a custom C++ autograd Function that can be used directly.</p>
<p>Currently, I know how to use the autograd function in C++, according to this <a href="https://github.com/pytorch/pytorch/blob/master/torch/csrc/autograd/custom_function.h" rel="nofollow noopener">file</a>(My function is also implemented based on this file’s example). The problem with this is that I don’t know how to bind the <code>SOME_FUNC::apply</code> to python using pybind11.</p>
<pre><code class="lang-auto">namespace py = pybind11;
PYBIND11_MODULE(module, m) {
	py::class_&lt;SOME_FUNC&gt;(m, "PTanh")
		.def_static("forward", &amp;SOME_FUNC::forward)
                //.def_static("apply",&amp;SOME_FUNC::apply) //This line will produce an error.
		.def_static("backward", &amp;SOME_FUNC::backward);
}
</code></pre>
<p>Can someone tell me a way to bind the <code>apply</code> or some way to make the autograd function in C++ directly callable in Python. Thank u very much.</p>
<p>Note: I’m compiling using pybind11 with setuptools to compile the code, like how it is in the tutorial.</p>

G.M: <p>I managed to find a work around, which is to wrap the <code>SOME_FUNC::apply</code> in another C++ function and bind that function to Python. But I still hope to know a direct way.</p>

Abhijit_Balaji1: <p>Hey I am trying to do the same thing. Did you find a more direct way to do it?</p>
What would be a proper way of implementing this riemann gradient?

msqiqi10: <p>So here’s a toy example from this paper,<a href="https://arxiv.org/abs/2103.14974" rel="noopener nofollow ugc">Automatic differentiation for Riemannian optimization on low-rank matrix and tensor-train manifolds</a>:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

def f(X):
    return torch.sum(X**2)

def g(delta_U, delta_V, U, V, f):
    perturbed_matrix = U @ delta_V.t() + delta_U @ V.t()
    return f(perturbed_matrix)

def compute_riemannian_gradient(X):
    U, S, V = torch.svd(X)
    delta_U = U @ torch.diag(S)
    delta_V = torch.zeros_like(V)
    delta_U.requires_grad_(True)
    delta_V.requires_grad_(True)
    perturbed_value = g(delta_U, delta_V, U, V, f)
    perturbed_value.backward()

    return delta_U.grad, delta_V.grad

def apply_gauge_conditions(delta_U, delta_V, V):
    delta_V -= V @ (V.t() @ delta_V)
    return delta_U, delta_V
</code></pre>
<pre><code class="lang-auto">def riemannian_gradient(X):
    U, _, V = torch.svd(X)
    delta_U, delta_V = compute_riemannian_gradient(X)
    delta_U, delta_V = apply_gauge_conditions(delta_U, delta_V, V)
    return delta_U @ V.t() + U @ delta_V.t()

X = torch.randn(5, 3)

for i in range(10):
    rgrad = riemannian_gradient(X)
    X = X - 0.01*rgrad
    print(f(X))
</code></pre>
<p>So as you can see, in the training inference, we don’t need the gradient of X, or [U, S, V]. Instead, I need the gradient from delta_U and delta_V to update X. Therefore I’m not able to simply loop through the parameters registered in parameters if I want to integrate this piece of code into torch.optimizer module.</p>
<p>My question is what’s the proper way of implementing this optimizing algorithm in optim.step() function when the weight X is updated by gradients from other parameters?</p>
Compile and deepcopy in the context of early stopping

hectordomorozco: <p>Hey there! Sorry if this has been asked before (I did several searches and couldn’t find the answers I’m looking for). I have a training loop where I compile a model before training using the default parameters (this model has pre-trained weights already loaded):</p>
<pre data-code-wrap="python"><code class="lang-python">model = torch.compile(model)
</code></pre>
<p>In this training loop, I check for the training loss (training loss because this is a specific case where we do not have validation files) and do an early stopping based on the training loss (i.e., if the absolute change of the loss is below a certain threshold for a number of steps, then we finish training and output the model). Based on a flag, we return either the last trained model, or the model with the lowest training loss:</p>
<pre data-code-wrap="python"><code class="lang-python"># this is done at each epoch in the training loop
  if tr_loss_sum &lt; min_loss:
      min_loss = copy.copy(tr_loss_sum)
      best_model = copy.deepcopy(model)
      logger.debug(f"epoch: {epoch}, min_loss: {min_loss}")

  # early stopping
  if early_stopping and np.abs(tr_loss_sum - last_loss) &lt; tolerance:
      stopping_counter += 1
      if stopping_counter &gt;= patience:
          logger.debug(f"Performing early stopping at epoch {epoch}")
          torch.cuda.empty_cache()
          if return_best_model:
              return best_model
          else:
              return model
</code></pre>
<p>The question I have is as follows: I currently do a <code>deepcopy</code> of the model. I know this works when the model is not compiled, so we do output the best model in that situation. But, when the model is compiled, I understand that even <code>deepcopy</code> would fail because the model object itself points at the compiled code in a cache. I understand that compiled models can’t be saved as compiled models (<a href="https://discuss.pytorch.org/t/pt-2-0-are-compiled-models-savable/169871" class="inline-onebox">PT 2.0 - Are compiled models savable</a>), and I understand that, as of July 2023 compiled models couldn’t be cloned/copied (<a href="https://discuss.pytorch.org/t/clone-copy-compiled-model/183972" class="inline-onebox">Clone/Copy compiled model</a>). Is there a way to make sure I output the <strong>compiled model</strong> with the lowest training loss? Would this entail making copies of the cache? Thank you so much for your time; again, I apologize if this question has been answered anywhere else but I couldn’t find anything on the forum/documentation.</p>
What does ProbabilisticActor model output?

Vefery: <p>I followed <a href="https://pytorch.org/rl/tutorials/coding_ppo.html" rel="noopener nofollow ugc">this</a> guide and trained the model. Then I saved the full model and tried to use it for inference. But the loaded model outputs a tuple of 4 tensors (3 tensors of the length of actions number and one tensor with a single value). I can’t figure out what are those</p>
<p>I’m still a begginer, so I might not know some basic stuff. From the guide I understood that the model is trained to output mean and scale values to define a distribution (specifically normal destruction). Through experiments I figured that the first 2 tensors are mean and scale, I used them to make the distribution and it seemed to work. But what’s the purpose of the other 2 tensors?</p>
(Up to date) Transformer + Classifier example?

Matt-T: <p><img src="https://discuss.pytorch.org/images/emoji/apple/wave.png?v=12" title=":wave:" class="emoji" alt=":wave:" loading="lazy" width="20" height="20"> Hi there, I’m looking at adapting this tutorial here: <a href="https://pytorch.org/tutorials/beginner/transformer_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Language Modeling with nn.Transformer and torchtext — PyTorch Tutorials 2.2.0+cu121 documentation</a></p>
<p>Instead of outputting probabilities of the next word in a sequence, I would like outputs to be class / label indices instead. I have found a great, <a href="https://n8henrie.com/2021/08/writing-a-transformer-classifier-in-pytorch/" rel="noopener nofollow ugc">semi-dated example here</a>, but it relies on an old version of PyTorch, with some of the data imports no longer working and marked deprecated.</p>
<p>Would anyone know of a more up to date example that uses an nn.Transformer tailored towards classification, or be able to share generally how one might adapt the official torchtext example above to output classes instead?</p>
<p>Thank you <img src="https://discuss.pytorch.org/images/emoji/apple/pray.png?v=12" title=":pray:" class="emoji" alt=":pray:" loading="lazy" width="20" height="20"></p>

Matt-T: <p>By the way, this is an extension of someone’s suggestion here to try Transformers for a classification problem: <a href="https://discuss.pytorch.org/t/help-improving-sports-prediction-model/195878/5" class="inline-onebox">Help improving sports prediction model - #5 by J_Johnson</a></p>
Backward on scatter_reduce raising an error

loryruta: <p>Hi all,<br>
I’m trying to back-propagate the following code (snippet):</p>
<pre data-code-wrap="py"><code class="lang-py"># Dummy initialization for the sake of the post
batch_size = 200
batch_indices = torch.empty((5000,))
node_hiddens = torch.empty((5000, 100))

# Actual code
mol_repr = torch.zeros((batch_size, 100,))
mol_repr = mol_repr.scatter_reduce(
    dim=0,
    index=batch_indices.unsqueeze(1),  # (5000, 1,)
    src=node_hiddens,  # (5000, 100)
    reduce='mean'
)
return mol_repr
</code></pre>
<p>The idea is that I have  a bunch of <code>node_hiddens</code> that I’ve already computed earlier, and I have a <code>batch_indices</code> tensor telling where the i-th <code>node_hiddens</code> should go in the final <code>mol_repr</code>. <code>node_hiddens</code> with the same index should average out (i.e. <code>reduce='mean'</code>).</p>
<p>However, I get:</p>
<pre><code class="lang-auto">RuntimeError: Function ScatterReduceBackward0 returned an invalid gradient at index 1 - got [5000, 1] but expected shape compatible with [5000, 100]
</code></pre>
<p>as I have very little idea about the autograd inner workings, I can’t wrap my head around this error.<br>
Could anyone help me? Thank you,</p>
<hr>
<p>Separate question that regards the same code:<br>
the snippet above is from <code>def forward(...)</code> function. To my understanding, <code>mol_repr</code> doesn’t need <code>requires_grad=True</code> on initialization as I don’t want to accumulate gradient on it to perform gradient update (i.e. it’s not a parameter!). Is my guess correct?</p>

ptrblck: <p>You could work around this issue via:</p>
<pre data-code-wrap="python"><code class="lang-python">index=batch_indices.unsqueeze(1).repeat(1, 100),
</code></pre>
<p>as the unsqueezed index seems to cause the issue.</p>

loryruta: <p>Thank you,<br>
finally I went for using <a href="https://pytorch.org/docs/stable/generated/torch.Tensor.index_reduce_.html#torch.Tensor.index_reduce_" rel="noopener nofollow ugc">index_reduce</a>, which is backpropagating correctly (at least, without raising errors).<br>
I should have read more carefully <a href="https://pytorch.org/docs/stable/generated/torch.Tensor.scatter_reduce_.html" rel="noopener nofollow ugc">scatter_reduce_ documentation</a>:</p>
<pre><code class="lang-auto">The backward pass is implemented only for src.shape == index.shape.
</code></pre>
Cdp_simple_quicksort made the Cuda-context consumed 50MB more…why?and what’s the best way to sort in CUDA?

opengpu: <p>cdp_simple_quicksort function from here made the Cuda-context consumed 50MB more than not compiled cdp_simple_quicksort…WHY 50MB so much?</p><aside class="onebox githubblob" data-onebox-src="https://github.com/NVIDIA/cuda-samples/blob/master/Samples/3_CUDA_Features/cdpSimpleQuicksort/cdpSimpleQuicksort.cu">
  <header class="source">

      <a href="https://github.com/NVIDIA/cuda-samples/blob/master/Samples/3_CUDA_Features/cdpSimpleQuicksort/cdpSimpleQuicksort.cu" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/NVIDIA/cuda-samples/blob/master/Samples/3_CUDA_Features/cdpSimpleQuicksort/cdpSimpleQuicksort.cu" target="_blank" rel="noopener nofollow ugc">NVIDIA/cuda-samples/blob/master/Samples/3_CUDA_Features/cdpSimpleQuicksort/cdpSimpleQuicksort.cu</a></h4>


      <pre><code class="lang-cu">/* Copyright (c) 2022, NVIDIA CORPORATION. All rights reserved.
 *
 * Redistribution and use in source and binary forms, with or without
 * modification, are permitted provided that the following conditions
 * are met:
 *  * Redistributions of source code must retain the above copyright
 *    notice, this list of conditions and the following disclaimer.
 *  * Redistributions in binary form must reproduce the above copyright
 *    notice, this list of conditions and the following disclaimer in the
 *    documentation and/or other materials provided with the distribution.
 *  * Neither the name of NVIDIA CORPORATION nor the names of its
 *    contributors may be used to endorse or promote products derived
 *    from this software without specific prior written permission.
 *
 * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY
 * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
 * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
 * PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL THE COPYRIGHT OWNER OR
 * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
 * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
</code></pre>



  This file has been truncated. <a href="https://github.com/NVIDIA/cuda-samples/blob/master/Samples/3_CUDA_Features/cdpSimpleQuicksort/cdpSimpleQuicksort.cu" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>and what’s the best way to sort in CUDA?</p>
<p>Thanks!</p>

ptrblck: <p>The issue doesn’t seem to be related to PyTorch, so you might want to ask in the NVIDIA discussion board.</p>
What's the aten::index?

MartinZhang: <p>what’s the <code>aten::index</code> ?</p>
<p>Where can I find documentation to explain this?</p>
<p>thx.</p>

ptrblck: <p>You won’t be able to find a proper documentation as the <code>libtorch</code> docs are quite lacking. The function signature is defined <a href="https://pytorch.org/cppdocs/api/function_namespaceat_1af1152a41061673382ce204faa02f4755.html#_CPPv4N2at5indexERKN2at6TensorERKN3c104ListIN3c108optionalIN2at6TensorEEEEE">here</a>, which at least shows the expected input arguments.</p>

Samet_Kocbay: <p>“Page not found”. Do you have a new link maybe, sadly I can not find any documenation anywhere ?<br>
Thank you</p>

ptrblck: <p>The new link seems to be <a href="https://pytorch.org/cppdocs/api/function_namespaceat_1a38e2ed33d3b7c12966ef78f6c7c91bd1.html#_CPPv4N2at5indexERKN2at6TensorERKN3c104ListINSt8optionalIN2at6TensorEEEEE">this one</a>, but I’m unsure why the URL changed.<br>
If it breaks again, check the <a href="https://pytorch.org/cppdocs/notes/tensor_indexing.html"><code>tensor indexing</code> API</a> or try the <a href="https://pytorch.org/cppdocs/search.html?q=index&amp;check_keywords=yes&amp;area=default">search</a>, which is how I found the new link.</p>
Torch.numel behavior with torch.Size as input

wandering007: <p>The official doc on <a href="https://pytorch.org/docs/stable/torch.html?highlight=numel#torch.numel" rel="nofollow noopener"><code>torch.numel</code></a> notes that its input must be a <code>torch.Tensor</code>. I tried <code>torch.Size</code> as input and the result might be kind of <em>unexpected</em>.</p>
<p>The code snippet is shown as:</p>
<pre><code class="lang-auto">import torch
a = torch.Tensor(3, 4)
print(a.numel(), a.shape.numel())
</code></pre>
<p>The returned values of both <code>a.numel()</code> and <code>a.shape.numel()</code> are <code>12</code>.</p>
<p>But there is no documentation or explanation for the input type <code>torch.Size</code>. Whether it is a potential bug or feature?</p>

harsanyidani: <p>I also ran into the same issue.<br>
There is an open issue about this on github: <a href="https://github.com/pytorch/pytorch/issues/61231" class="inline-onebox" rel="noopener nofollow ugc">Document torch.Size and its methods (like torch.Size.numel) · Issue #61231 · pytorch/pytorch · GitHub</a></p>
For unbatched 2-D input, hx and cx should also be 2-D but got (3-D, 3-D) tensors

Ninlawat_Phuangchoke: <p>The scenario: I don’t want to use embedding layer in decoder part so that I did step like this</p>
<ol>
<li>In <code>def forward</code>, I change from <code>embedded = self.dropout(self.embedding(input))</code> to <code>embedded = self.dropout(input)</code> but I got an error <code>For unbatched 2-D input, hx and cx should also be 2-D but got (3-D, 3-D) tensors</code></li>
<li>Following that, I ask clude3 that say I should check if hidden and cell are 3D, if so, squeeze the first dimension with code</li>
</ol>
<pre><code class="lang-auto">    if hidden.ndim == 3:
        hidden = hidden.squeeze(0)  # hidden = [batch size, hidden dim]
    if cell.ndim == 3:
        cell = cell.squeeze(0)  # cell = [batch size, hidden dim]
</code></pre>
<p>Until now, it is still error that is the same as I got in step 1)<br>
Hence, Could you please help me to solve it and I’m not sure that it have other impacts on another part?</p>
<p>This is my last code:</p>
<pre><code class="lang-auto">class Decoder(nn.Module):
    def __init__(self, output_dim, embedding_dim, hidden_dim, n_layers, dropout):
        super().__init__()
        self.output_dim = output_dim
        self.hidden_dim = hidden_dim
        self.n_layers = n_layers
        self.rnn = nn.LSTM(embedding_dim, hidden_dim, n_layers, dropout=dropout)
        self.fc_out = nn.Linear(hidden_dim, output_dim)
        self.dropout = nn.Dropout(dropout)

    def forward(self, input, hidden, cell):
        # input = [batch size]
        # hidden = [n layers * n directions, batch size, hidden dim]
        # cell = [n layers * n directions, batch size, hidden dim]
        # n directions in the decoder will both always be 1, therefore:
        # hidden = [n layers, batch size, hidden dim]
        # context = [n layers, batch size, hidden dim]
        input = input.unsqueeze(0)
        # input = [1, batch size]
        embedded = self.dropout(input)
        # embedded = [1, batch size, embedding dim]
        output, (hidden, cell) = self.rnn(embedded, (hidden, cell))
        # output = [seq length, batch size, hidden dim * n directions]
        # hidden = [n layers * n directions, batch size, hidden dim]
        # cell = [n layers * n directions, batch size, hidden dim]
        # seq length and n directions will always be 1 in this decoder, therefore:
        # output = [1, batch size, hidden dim]
        # hidden = [n layers, batch size, hidden dim]
        # cell = [n layers, batch size, hidden dim]
        prediction = self.fc_out(output.squeeze(0))
        # prediction = [batch size, output dim]
        return prediction, hidden, cell
</code></pre>
Sparse matrix multiplication randomly causes NaNs

Viktor_Todosijevic: <p>Hi,</p>
<p>I’ve run into an issue where a sparse matrix multiplication causes NaN values for no good reason. I can’t share the source code and the data, plus it doesn’t always happen so it’s quite hard to reproduce.</p>
<p>I’m just wondering if anyone else has encountered anything similar? It’s so random and I’ve no idea why it sometime breaks and sometimes not.</p>
<p>Here’s an example of the code that breaks:</p>
<pre data-code-wrap="python"><code class="lang-python">    def my_foo(a: Tensor, b: Tensor, sparse_matrix: Tensor) -&gt; Tensor:

        c = a + 1j * b # torch.complex128
        assert not torch.isnan(c).any() # all good, no nans
        
        # originaly torch.int64 (a matrix of ones and zeros), sparse csr
        sparse_matrix = sparse_matrix.to(c.dtype)
        assert not torch.isnan(sparse_matrix .values()).any() # all good, no nans
        
        # the dimensions are c: (32700,), and sparse_matrix: (302200, 32700) so quite large
        result = sparse_matrix @ c

        assert not torch.isnan(result ).any() # this assertion fails, there are nans (some entries are nans, not all)
        return result
</code></pre>
<p>So neither <code>sparse_matrix</code> nor <code>c</code> contain NaNs, yet when multyplying them NaNs occur.</p>
Declaration error when compiling LibTorch with 3rd party

ssakaoglu: <p>I am developing a physics informed neural network code for an OpenFOAM application in my thesis studies. I am stuck when I include libtorch into the code, inside a specific class. The compiler complains about some declaration error, which is coming from the libtorch, however, if I include it into another class, it works well. Unfortunately, I need to include torch headers in that problematic class to get access to the objects I need for DNN inference.</p>
<pre><code class="lang-auto">/opt/openfoam7/src/OpenFOAM/lnInclude/className.H:40:34: error: two or more data types in declaration of ‘typeName_’
   40 |     static const char* typeName_() { return TypeNameString; }

/opt/openfoam7/src/OpenFOAM/lnInclude/typeInfo.H:72:19: error: ‘word’ does not name a type; did you mean ‘word_H’?
   72 |     virtual const word&amp; type() const { return typeName; }

/home/sergen/torch/files/libtorch/include/c10/util/typeid.h:437:38: error: expected unqualified-id before ‘noexcept’
  437 |   static c10::string_view TypeName() noexcept {

/home/sergen/torch/files/libtorch/include/c10/core/TensorImpl.h: In member function ‘T* c10::TensorImpl::data() const’:
/home/sergen/torch/files/libtorch/include/c10/core/TensorImpl.h:1184:27: error: ‘TypeName’ is not a member of ‘caffe2::TypeMeta’
 1184 |         caffe2::TypeMeta::TypeName&lt;T&gt;(),

/home/sergen/torch/files/libtorch/include/c10/core/TensorImpl.h:1184:37: error: expected primary-expression before ‘&gt;’ token
 1184 |         caffe2::TypeMeta::TypeName&lt;T&gt;(),

and so on
</code></pre>
<p>Any help is appreciated,</p>
<p>Kind regards</p>
Pytorch Addmv in cpu

Aruna: <p>Hi,<br>
when addmv_impl_cpu is being used in cpu and is it faster than addmm_impl_cpu interms of performance if the addmm shapes are small?<br>
The impl is available here (<a href="https://github.com/pytorch/pytorch/blob/03a05e791aabe4f2e3938626bff849641dc101ea/aten/src/ATen/native/Blas.cpp" class="inline-onebox" rel="noopener nofollow ugc">pytorch/aten/src/ATen/native/Blas.cpp at 03a05e791aabe4f2e3938626bff849641dc101ea · pytorch/pytorch · GitHub</a>)<br>
but I am not able to find its usage.<br>
Thanks</p>
Why bfloat16 matmul is significantly slower than float32?

nkjcqvcpi: <p>I am figuring out how should I use bfloat16 or float32 on CPU training and inference.</p>
<p>That’s the code I use to test.</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
import time

tensor_size = (1000, 1000)
num_iterations = 100


def perform_operations(data):
    start_time = time.time()
    squared = torch.square(data)
    squared_time = time.time() - start_time

    start_time = time.time()
    summed = torch.sum(data)
    sum_time = time.time() - start_time

    start_time = time.time()
    averaged = torch.mean(data)
    mean_time = time.time() - start_time

    start_time = time.time()
    transpose = data.t()
    transpose_time = time.time() - start_time

    start_time = time.time()
    matmul_result = torch.matmul(data, transpose)
    matmul_time = time.time() - start_time
    return squared_time, sum_time, mean_time, transpose_time, matmul_time


bfloat16_times = [0.0] * 5
float32_times = [0.0] * 5

for i in range(num_iterations):
    data = torch.randn(tensor_size, dtype=torch.float32)
    data_bfloat16 = data.to(torch.bfloat16)

    bfloat16_op_times = perform_operations(data_bfloat16)
    for j in range(5):
        bfloat16_times[j] += bfloat16_op_times[j]

    float32_op_times = perform_operations(data)
    for j in range(5):
        float32_times[j] += float32_op_times[j]

bfloat16_times = [time / num_iterations for time in bfloat16_times]
float32_times = [time / num_iterations for time in float32_times]

print("Average time for bfloat16:")
print("Squared: {:.6f} seconds".format(bfloat16_times[0]))
print("Sum: {:.6f} seconds".format(bfloat16_times[1]))
print("Mean: {:.6f} seconds".format(bfloat16_times[2]))
print("Transpose: {:.6f} seconds".format(bfloat16_times[3]))
print("Matmul: {:.6f} seconds".format(bfloat16_times[4]))
print()

print("Average time for float32:")
print("Squared: {:.6f} seconds".format(float32_times[0]))
print("Sum: {:.6f} seconds".format(float32_times[1]))
print("Mean: {:.6f} seconds".format(float32_times[2]))
print("Transpose: {:.6f} seconds".format(float32_times[3]))
print("Matmul: {:.6f} seconds".format(float32_times[4]))
</code></pre>
<p>And I got the result:</p>
<blockquote>
<p>Average time for bfloat16:<br>
Squared: 0.000355 seconds<br>
Sum: 0.000183 seconds<br>
Mean: 0.000237 seconds<br>
Transpose: 0.000027 seconds<br>
<strong>Matmul: 0.299079 seconds</strong></p>
</blockquote>
<blockquote>
<p>Average time for float32:<br>
Squared: 0.000585 seconds<br>
Sum: 0.000239 seconds<br>
Mean: 0.000123 seconds<br>
Transpose: 0.000010 seconds<br>
<strong>Matmul: 0.005496 seconds</strong></p>
</blockquote>
<p>I wonder why bfloat16 faster than float32 in all other operations except torch.matmul().<br>
The code runs on an intel 9th-gen core i7 cpu.</p>
How to finetune my quantized model

RonaldYuren: <p>Hi, I am <strong>trying to use quantized model to do finetuning in order to improve accuracy</strong>,<br>
but the original model is using floating point which means that there are problems with floating point and fixed point( from quantized model weights) , the error is showed below, please give me some suggestion on how to solve or let the finetuning is work:</p>
<h5><a name="below-are-error-message-using-pytorch-210-to-do-post-training-static-quantization-1" class="anchor" href="#below-are-error-message-using-pytorch-210-to-do-post-training-static-quantization-1"></a><strong>below are error message     Using pytorch 2.1.0 to do post-training static-quantization</strong>#####</h5>
<p>Traceback (most recent call last):<br>
File “Static_quantize5.py”, line 218, in <br>
train(hyp, opt = opt,device = torch.device(‘cpu’), tb_writer = tb_writer)<br>
File “/home/gene/YuRen/yolov7/train.py”, line 117, in train<br>
model.load_state_dict(quantized_weights, strict=False)<br>
File “/home/gene/miniconda3/envs/YuRen_copy1/lib/python3.8/site-packages/torch/nn/modules/module.py”, line 2152, in load_state_dict<br>
raise RuntimeError(‘Error(s) in loading state_dict for {}:\n\t{}’.format(<br>
RuntimeError: Error(s) in loading state_dict for Model:<br>
While copying the parameter named “model.0.conv.weight”, whose dimensions in the model are torch.Size([32, 3, 3, 3]) and whose dimensions in the checkpoint are torch.Size([32, 3, 3, 3]), an exception occurred : (‘Copying from quantized Tensor to non-quantized Tensor is not allowed, please use dequantize to get a float Tensor from a quantized Tensor’,).<br>
While copying the parameter named “model.1.conv.weight”, whose dimensions in the model are torch.Size([64, 32, 3, 3]) and whose dimensions in the checkpoint are torch.Size([64, 32, 3, 3]), an exception occurred : (‘Copying from quantized Tensor to non-quantized Tensor is not allowed, please use dequantize to get a float Tensor from a quantized Tensor’,).<br>
While copying the parameter named “model.2.conv.weight”, whose dimensions in the model are torch.Size([32, 64, 1, 1]) and whose dimensions in the checkpoint are torch.Size([32, 64, 1, 1]), an exception occurred : (‘Copying from quantized Tensor to non-quantized Tensor is not allowed, please use dequantize to get a float Tensor from a quantized Tensor’,).<br>
…there are many similar error about copying parameters<br>
########</p>
<p>Please Help Me !!   Thanks you very much!!!</p>
How does loss.backward() calculated in pytorch lightning

Xanthan: <p>I understand that when <code>training_step()</code> returns loss, the code for automatic optimization<br>
(<a href="https://lightning.ai/docs/pytorch/stable/common/optimization.html#automatic-optimization" rel="noopener nofollow ugc">link</a>) takes care of the loss.backward()</p>
<p>Can someone tell me what would be difference in the loss.backward() automatic optimization, for the following two scenarios for training_step():</p>
<p>Scenario 1:</p>
<pre><code class="lang-auto">def training_step(self, batch: list,epochidx):
     x,y = batch
     output = model(x)
     loss = self.loss_func(output,y)

     return loss

</code></pre>
<p>Scenario 2:</p>
<pre><code class="lang-auto">def training_step(self, batch: list,epochidx):
     x,y = batch
     output = model(x)
     loss = self.loss_func(output,y)
     metric = self.metric(output,y)

     train_log = {"loss":loss,"metric":metric}

     return train_log

</code></pre>
<p>What my worry is that loss.backward() in the 2nd scenario does backward for both loss and metric instead of just loss.</p>
<p>I opened the pytorch-lightning files in my conda environment to understand how the automatic optimization is happening if I send a dictionary instead of a Tensor but it didn’t lead to much.</p>
<p>Any help/hint is appreciated. Thanks!</p>

ptrblck: <p>Since this issue is PyTorch-Lightning-specific you might want to cross-post the question into their discussion board.</p>

Xanthan: <p>sure, I will put it their as well. thanks!</p>
<p>I would like to know in your experience, did a model training yield different results when tried with scenario 2 (compared with scenario 1)</p>

NilsB: <p>When you return a dict in the “training_step” method pytorch lightning will look specifically for the “loss” key in the dict for the automatic optimization and use only this for the backprop.<br>
It’s not that clearly stated, but you can also find it <a href="https://lightning.ai/docs/pytorch/stable/common/lightning_module.html#training-step" rel="noopener nofollow ugc">here</a> in the docs.<br>
So there wouldn’t be any differences in regard to optimization between your two scenarios.</p>

Xanthan: <p><a class="mention" href="/u/nilsb">@NilsB</a> , thanks for the amazing find. This is exactly what I wanted to know.</p>
<p>So, like you said and what’s mentioned in your link, there should ideally be <strong>no difference</strong> in training when encountered with scenario 1 and scenario 2.</p>
Tensor Tracking

sabiha_afroz: <p>I am using torch._dynamo and torch.compile to generate FX graph while training a resnet_50 model. I have attached a screenshot of the FX graph. This graph is showing torch operations and tensors used as parameters and output tensors for each torch operation. I have used red boxes to mark a few tensors. How do I track these tensors while I am training a model using pytorch?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/d/2dabaa91d04b5712d28b30c5278d279102a4d26b.png" data-download-href="https://discuss.pytorch.org/uploads/default/2dabaa91d04b5712d28b30c5278d279102a4d26b" title="fx_graph"><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/d/2dabaa91d04b5712d28b30c5278d279102a4d26b.png" alt="fx_graph" data-base62-sha1="6w1hknr6Pib3MlseLAY2vXN0D9N" width="690" height="366" data-dominant-color="2A2929"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">fx_graph</span><span class="informations">1436×763 87.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

vdw: <p>I think that maybe <a href="https://blog.paperspace.com/pytorch-hooks-gradient-clipping-debugging/" rel="noopener nofollow ugc">hooks</a> are what you’re looking for.</p>

sabiha_afroz: <p>When I will do the training, I do not know which tensors will get created and where in pytorch code. How do I attach hook for tensors? Can you guide me?</p>

vdw: <p>You can register hooks to any <code>nn.Module</code> to see which tensor goes in and what tensors goes out. <a href="https://discuss.pytorch.org/t/how-to-register-forward-hooks-for-each-module/43347">Here</a> is an example to register a hook to all modules.</p>
<pre><code class="lang-auto">self.hooks = {}
for name, module in model.named_modules():
    hooks[name] = module.register_forward_hook(self,hook_fn)
</code></pre>
<p>The <code>hook_fn</code> that gets passed can look something like this:</p>
<pre><code class="lang-auto">def hook_fn(module, input, output):
    print(input.shape)
    print(output.shape)
</code></pre>

GuWei007: <aside class="quote no-group" data-username="sabiha_afroz" data-post="1" data-topic="200812">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sabiha_afroz/48/64174_2.png" class="avatar"> sabiha_afroz:</div>
<blockquote>
<p>torch.compile</p>
</blockquote>
</aside>
<p>Has your problem been solved? I also want to know how to track tensor in torch.compile mode.</p>

sabiha_afroz: <p><a class="mention" href="/u/vdw">@vdw</a> Is it possible to identify tensor types such as whether a tensor is a weight tensor, or bias tensor, or input tensor or activation tensor?</p>

sabiha_afroz: <p>@ Gu Wei, we can use hook with the model. But I am not sure how we could track with torch.compile.</p>
Tried to do real simple nn.Linear model from scratch forward() is not giving correct answer

GGPYTORCH000: <p>i wrote small class Linear which is similar to nn.Linear interms of initialization and forwarding. It worked however when I switched my code from nn.Linear to my own Linear, forward compute is not giving correct result.<br>
I thought fwd of linear is just simple matmul between X and Weight + bias<br>
def forward (X:torch.tensor):<br>
out = torch.matmul(self.weight, X)<br>
<span class="hashtag-raw">#out</span> = torch.matmul(X, self.weight)<br>
out += self.bias<br>
return out</p>
<p>To give more intext when following X and W values are supplied to Linear (my own) class result is<br>
tensor([[0.3684, 1.2859]]), which I checkeked manually</p>
<blockquote>
<blockquote>
<blockquote>
<p>X=torch.tensor([[-0.9708,  0.9610]])<br>
W=torch.tensor([[ 0.6627, -0.4245],<br>
…         [ 0.5373,  0.2294]])<br>
b=torch.tensor([0.4954, 0.6533] )<br>
torch.matmul(X, W) + b<br>
tensor([[0.3684, 1.2859]])</p>
</blockquote>
</blockquote>
</blockquote>
<p>But with nn.linear class, it is returning<br>
tensor([[-0.3565, -0.2904]],</p>
<p>Now if the W is 0, then results of both are correct meaning, bias part is working (wondeful) it is just matmul part that is something wrong.</p>

ptrblck: <p>Your <code>matmul</code> is wrong as you need to pass the transposed <code>weight</code> to it:</p>
<pre data-code-wrap="python"><code class="lang-python">batch_size = 2
in_features = 3
out_features = 4

x = torch.randn(batch_size, in_features)
lin = nn.Linear(in_features, out_features)
print(lin.weight.shape)
# torch.Size([4, 3]) # [out_features, in_features]
print(lin.bias.shape)
# torch.Size([4]) # [out_features]

ref = lin(x)

w = lin.weight
b = lin.bias

out = torch.matmul(x, w.T) + b
print(out - ref)
# tensor([[0., 0., 0., 0.],
#         [0., 0., 0., 0.]], grad_fn=&lt;SubBackward0&gt;)
</code></pre>

GGPYTORCH000: <p>thx!! i am able to move fwd now. (salute)</p>