akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How do I get `torch.onnx.export` to work with `aot_module`?

Sam_Carroll: <p>I’m trying to use <code>aot_module</code> to compile forward and backward graph of a model.  I want to be able to export the graph to onnx but I’m having trouble figuring out how to do that.  Here is an example script</p>
<pre><code class="lang-auto">import torch
from functorch.compile import aot_module
from torch._subclasses.fake_tensor import FakeTensorMode
from torch.fx.experimental.symbolic_shapes import ShapeEnv
from functorch.compile import make_boxed_func
import torch._C._onnx as _C_onnx


class Linear(torch.nn.Module):
    def __init__(self):
        super(Linear, self).__init__()
        self.linear = torch.nn.Linear(128, 10)
        self.activation = torch.nn.ReLU()

    def forward(self, *inputs):
        input = self.linear(inputs[0])
        input = self.activation(input)
        return input


def compiler_fn(fx_module: torch.fx.GraphModule, example_inputs):
    print(fx_module)
    torch.onnx.export(
        fx_module,
        tuple(example_inputs),
        "model.onnx",
        training=_C_onnx.TrainingMode.PRESERVE,
        do_constant_folding=False,
        export_params=False,
    )
    return make_boxed_func(fx_module)


model = Linear()
model.train()
loss_fn = torch.nn.MSELoss()

with FakeTensorMode(allow_non_fake_inputs=True, shape_env=ShapeEnv()) as fake_mode:
    input = torch.randn((64, 128), requires_grad=True)
    labels = torch.randn((64, 10), requires_grad=True)

    c_model = aot_module(model, fw_compiler=compiler_fn, bw_compiler=compiler_fn)
    output = c_model(input)
    loss = loss_fn(output, labels)
    loss.backward()
</code></pre>
<p>I get</p>
<pre><code class="lang-auto">TypeError: f_(): incompatible function arguments. The following argument types are supported:
    1. (self: torch._C.Node, arg0: str, arg1: float) -&gt; torch._C.Node

Invoked with: %5 : Tensor = onnx::Gemm(%primals_3, %t, %primals_2), scope: torch.fx.graph_module.GraphModule.__new__.&lt;locals&gt;.GraphModuleImpl::
, 'alpha', i1 
(Occurred when translating addmm).
</code></pre>
<p>and have no idea what I’m doing wrong. Any guidance would be appreciated. Thanks!</p>

Sam_Carroll: <p>So, if i turn bias off in <code>LinearLayer</code> i get further.  This gets passed the first error on the forward graph.  Now I get something like</p>
<pre><code class="lang-auto">torch.onnx.errors.UnsupportedOperatorError: Exporting the operator 'aten::threshold_backward' to ONNX opset version 17 is not supported. Please feel free to request support or submit a pull request on PyTorch GitHub: https://github.com/pytorch/pytorch/issues.
</code></pre>
<p>So is the first error i showed in the original post a bug then?</p>
Differences between 'params.grad' and 'params.grad.data'

Karolina_Pondel-Sycz: <p>Hi everyone,<br>
I’m trying to understand the difference between ‘params.grad’ and ‘params.grad.data’. I asked ChatGPT and searched for info, unfortunately no answer was satisfactory. Could someone explain it? Thanks!</p>

KFrank: <p>Hi Karolina!</p>
<aside class="quote no-group" data-username="Karolina_Pondel-Sycz" data-post="1" data-topic="201234" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/karolina_pondel-sycz/48/65756_2.png" class="avatar"> Karolina_Pondel-Sycz:</div>
<blockquote>
<p>I’m trying to understand the difference between ‘params.grad’ and ‘params.grad.data’.</p>
</blockquote>
</aside>
<p>The short answer is that <code>.data</code> has been deprecated for public-facing use<br>
(so don’t use it).</p>
<p>Having said that – and not really being knowledgeable about what <code>.data</code><br>
is currently used for – <code>.data</code> appears to give you a tensor that shares its<br>
underlying data with the tensor it was called on.  If called on a <code>Parameter</code><br>
(a subclass of <code>Tensor</code>), it returns an ordinary <code>Tensor</code>.  It also gives you<br>
a tensor with <code>require_grad = False</code> whether or not <code>requires_grad</code><br>
was <code>True</code> for the original tensor.  I don’t think that any of this depends on<br>
whether the tensor <code>.data</code> is called on is the <code>.grad</code> of some other tensor.</p>
<p>Best.</p>
<p>K. Frank</p>
Gradients always 0 during training

ala_baccar: <p>Hi,<br>
I am training a classification neural network but the loss function is barely changing at each epoch, I checked the gradients of my network params and I found them to be 0 or e-05,e-10… I have tried normalization, decreasing hidden layers, changing activation functions but still no improvements, and I don’t think it is a problem of requires_grad=true, here is the code below</p>
<p>model:<br>
def <strong>init</strong>(self,input_dim,hidden_dim,output_dim=4):<br>
super(Sic_Detector,self,).<strong>init</strong>()<br>
self.input=nn.Linear(in_features=input_dim,out_features=hidden_dim)<br>
self.hidden1 = nn.Linear(in_features=hidden_dim, out_features=hidden_dim*3)<br>
self.output= nn.Linear(in_features=hidden_dim, out_features=output_dim)<br>
self.softmax=nn.Softmax(dim=0)</p>
<pre><code>def forward(self,x):
    x=self.input(x)
    #x=self.hidden1(relu(x))
    x=self.output(relu(x))
    return  self.softmax(x)
</code></pre>
<p>Training block:</p>
<p>criterion = nn.CrossEntropyLoss()<br>
optimizer = optim.Adam(model.parameters(),lr=0.005)</p>
<p>for epoch in range(10):<br>
running_loss = 0.0<br>
for i, data in enumerate(trainloader, 0):</p>
<pre><code>    inputs, labels = data
    inputs=inputs.requires_grad_(True)
    optimizer.zero_grad()
    outputs = model(inputs)
    loss = criterion(outputs, labels)

    loss.backward()

    running_loss += loss.item()
    optimizer.step()
</code></pre>

ala_baccar: <p>Gradient values:</p>
<p>Layer: input.weight | Size: torch.Size([5, 2]) | Gradients:<br>
tensor([[ 1.6471e-04,  6.5578e-05],<br>
[ 0.0000e+00,  0.0000e+00],<br>
[-8.7129e-05, -5.4855e-05],<br>
[-5.7131e-05, -4.4586e-06],<br>
[-2.4803e-06, -5.0209e-05]])<br>
Layer: input.bias | Size: torch.Size([5]) | Gradients:<br>
tensor([-1.5109e-04,  0.0000e+00,  5.9544e-06, -9.2206e-05,  3.3566e-05])<br>
Layer: output.weight | Size: torch.Size([4, 5]) | Gradients:<br>
tensor([[ 1.4060e-04,  0.0000e+00,  2.8602e-06, -1.1239e-04,  3.6894e-05],<br>
[-1.3274e-04,  0.0000e+00,  2.4924e-04, -5.3653e-05,  9.1451e-05],<br>
[-4.9264e-05,  0.0000e+00, -1.4066e-04,  9.7468e-05, -3.0539e-05],<br>
[ 6.1543e-05,  0.0000e+00, -7.1362e-05,  4.6664e-05, -9.2642e-05]])</p>

ptrblck: <p><code>nn.CrossEntropyLoss</code> expects raw logits so remove the <code>self.softmax</code> usage and pass the output of <code>self.output</code> to the criterion.</p>
Memory leak on training

THarIKH_M: <p>I was trying to use euler based backprop through time.</p>
<pre><code class="lang-auto">for i in tqdm(range(iterations)):
    x0 = sample(batch_size)
    loss = 0.0
    for t in np.arange(start_time,end_time+1e-5,step):
        u = policy(x0)
        loss= loss+ step*cost(x0,u)
        x0 = x0 + step * dynamics(x0, u)

    del x0
    del u

    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
</code></pre>
<p>Here sample function just samples batches from normal dist, policy is a small neural network, cost and dynamics are small functions (containing sin, cos and some small computations).</p>
<p>But its eating up my ram. Is there a way to clear ram after some iterations?<br>
Thanks in advance</p>

ptrblck: <p>Could you post a minimal and executable code snippet reproducing the issue?</p>

THarIKH_M: <p>I found out why it was eating up my ram, i had one statement which stored the loss value into a list</p>
<pre><code class="lang-auto">losses.append(loss) 
</code></pre>
<p>instead of loss.item() or detaching, i put loss directly into list which ate my ram</p>
Computer recognize torch but not torchvision

Pranjal.Pok7: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/3/53185d15fb53f5f015770fe56dfc63d8bde799ab.png" alt="Screenshot from 2024-04-19 17-57-23" data-base62-sha1="bR5Po7pu7OBhrB9anAccm3SqEm7" width="605" height="236"><br>
Installed pytorch from source. Now I am facing this problem. How do I solve this ?</p>

ptrblck: <p>Install <code>torchvision</code> too.</p>
How to use system cuda/cudnn

Li_Li: <p>I have a machine with cuda/cudnn compatible rocm device.</p>
<pre><code class="lang-auto">$ nvcc --version
HIPHSA: Author SUGON
HIP version: 5.4.23453
Cuda compilation tools, release 11.8, V11.8.89
clang version 15.0.0 (http://10.15.3.7/dcutoolkit/driverruntime/llvm-project.git 1be90618e508074abc746ab4963d7ad92710d6c5)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /public/software/compiler/dtk-23.10.1/llvm/bin
</code></pre>
<p>I then install pytorch 2.2 with cuda 11.8 by:</p>
<pre><code class="lang-auto">pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
</code></pre>
<p>But when I import torch, it can’t find cuda device:</p>
<pre><code class="lang-auto">$ python
Python 3.11.8 (main, Feb 26 2024, 21:39:34) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.cuda.device_count()
0
</code></pre>
<p>I think the problem is pytorch use the cuda/cudnn runtime lib of its own. But I want it to use system cuda.</p>
<p>I have set CUDA_HOME and LD_LIBRARY_PATH. But it seems not work.</p>

ptrblck: <p>Which device are you using?<br>
The binaries ship with their own CUDA runtime and your locally installed CUDA toolkit will be used if you build PyTorch from source.</p>
Torch + pytest leads to memory fragmentation: How to do proper integration testing of a lot of torch models?

orcunderscore: <p>Hi there,</p>
<p>we serve a bunch of torch models in production.<br>
To minimize issues, we have a whole bunch of integration tests. Basically, each model is loaded, some inference is done with it and the result is checked.<br>
Most of the tests live in different files.<br>
We start pytest by calling</p>
<pre><code class="lang-auto">pytest -s tests
</code></pre>
<p>where test is the name of the directory, where all the tests are located.</p>
<p>Recently, we ran into the following issue:</p>
<pre><code class="lang-auto">torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 26.00 MiB. GPU 0 has a total capacity of 14.58 GiB of which 10.96 GiB is free. Including non-PyTorch memory, this process has 0 bytes memory in use. Of the allocated memory 3.40 GiB is allocated by PyTorch, and 75.64 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
</code></pre>
<p>From my understanding, there is (at least according to this message) enough memory to allocate this.<br>
Also, all the tests run fine if I run the test-files one-by-one with pytest.</p>
<p>My best guess about what is happening:</p>
<ul>
<li>Pytest somehow keeps the torch-gpu-memory state</li>
<li>After some time torch suffers from memory fragmentation (I have read about this, but I don’t really know how to confirm if this is happening; it makes sense though, since we are loading many different models and data onto the GPU and then releasing them again, so lots of different things are happening within the pytest run).</li>
</ul>
<p>What I have so far tried to get  rid of this problem:</p>
<ul>
<li><code>torch.cuda.empty_cache()</code> → Helped briefly, but the issue came back when a test was added, which makes sense, since this only frees cache that torch should in principle free by itself if needed.</li>
<li>use numba to reset the gpu, by <code>from numba import cuda; cuda.get_current_device().reset(); cuda.close()</code> (done before torch is imported in each test file). This did not help.</li>
</ul>
<p>Our current solution is to split the test in multiple pytest calls, which doesn’t really solve the problem but works around it. I was wondering if there is a better way to do it. I saw that torch has internal test classes (e.g. <code>from torch.testing._internal.common_utils import TestCase</code>).</p>
<p>So how do the torch-devs solve this issue? I would imagine that this came up at some point.</p>
OH encoded features are lost when padding and packing

Danibahnani: <p>I’m trying to pass time series data to an LSTM network, the data uses both minmax scaling, and one hot encoding. I do this by splitting the data into sequences in the <code>Dataset</code> and then padding and packing those using <code>collate_fn</code> in the <code>DataLoader</code>. The network is then run through a training loop.</p>
<p>My issue is when I try to unpack the values from the lstm layer of the last batch and pass them into the network’s final linear layer in the forward function:</p>
<pre><code class="lang-auto">    def forward(self,x ):
        lstm = self.lstm
        batch_size = self.batch_size

        h0 = torch.zeros(self.num_layers,batch_size,self.hidden_size,)   
        c0 = torch.zeros(self.num_layers,batch_size,self.hidden_size,)
        print(f"Expected input size: {self.input_size}")
        print(f"Actual input size: {x.data.size()}")

        # ****** Fog feature contains only 0 for the last batch (first values, date[0]=-1) ******
        #Fog column is dropped during OH-encoding (or other preprocessing part) which causes input size inconsistency
        
        packed_lstm_out, (hn,cn) = lstm(x, (h0,c0))
        print(f"lstm_out size: {packed_lstm_out.data.size()}")
        unpacked_lstm_out, _ = pad_packed_sequence(sequence=packed_lstm_out,batch_first=True)
        print(f"Unpacked lengths: {[len(seq) for seq in unpacked_lstm_out]}")
        #unpacked_lstm_tensor = torch.stack(unpacked_lstm_out,dim=0).float().requires_grad_(True)

        print(f"Unpacked shape: {unpacked_lstm_out.shape}\n")

        output = self.fc1(unpacked_lstm_out[:,-1,:])

        return output
</code></pre>
<p>I am getting an error about the input size (which is supposed to be 9) being too small (8).<br>
When I check the data passed into the error-inducing batch I can see that one of the one-hot encoded features are missing.</p>
<p>How can I stop this column/feature from being dropped, so that the input size is consistent for all batches?</p>
Question about Sigmoid function

m7md_hka: <p>Hello, I want ask about sigmoid function and it’s general rule:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/4/34711ed15823dacb2e59634aa6f8037dd153a322.png" alt="image" data-base62-sha1="7tVbDLlrw9R4T1txgOk72OZC7rs" width="283" height="101"></p>
<p>is it in pytorch always centered on zero? and the <code>c1</code> is constant? or the parameters updated during training?!</p>

UMAR_MASUD: <p>Hi,<br>
As per the documentation - <a href="https://pytorch.org/docs/stable/generated/torch.nn.Sigmoid.html" class="inline-onebox" rel="noopener nofollow ugc">Sigmoid — PyTorch 2.2 documentation</a>, the above sigmoid function is always centred on zero. So basically c1=1 and c2=0 in your equation above. There are no learnable parameters in the sigmoid implementation, it is just a non-linear transformation applied to the input.</p>

Albertma: <p>Activations functions have not param to learn.</p>

m7md_hka: <p>No, you can set a parameter for certain  activation function like <code>LeakyReLU</code> using <code>nn.Parametem</code> and make the model to learn and select the best slope of the negative values.</p>

Albertma: <p>good point!<br>
Is there any paper that focus on learnable param of activation function?</p>
:An unexpected error occurred: module 'torch.cuda.amp' has no attribute 'supported'

Stefan_Ionica: <p>:An unexpected error occurred: module ‘torch.cuda.amp’ has no attribute ‘supported’<br>
torch version 2.2.2+cu118<br>
torch.cuda.amp is available.</p>
<p>Please Help</p>

ptrblck: <p><code>torch.cuda.amp.supported</code> is not a valid attribute and also not mentioned in the docs (at least I cannot find it). Could you add more information where this attribute is used?</p>

Stefan_Ionica: <p>Sorry, it all work after a restart , Thanks</p>
Transfer learning, requires_grad

Yusfal: <p>By setting “requires_grad == True” for a layer in a pretrained model. Does the layer become initilized randomly? Or does it get initilized with the pretrained parameters and then retrained during training?</p>

UMAR_MASUD: <p>Hi, I think initialisation is not dependent on what you set for <code>requires_grad==True</code>. If you don’t load the pretrained checkpoint or weights, the model will be initialised with random weights (Check this documentation - <a href="https://pytorch.org/vision/stable/models.html#initializing-pre-trained-models" rel="noopener nofollow ugc">Initializing pre-trained models</a>).<br>
Setting requires_grad as True will ensure the gradient calculation for whatever weight initialisation you do (random or loaded pretrained) for the future operation on them.</p>
Signed distance transform computation of predicted segmentation

tkepp: <p>Hey!<br>
I try to implement a new loss function, in which the calculation of the signed distance transform (<a href="https://en.wikipedia.org/wiki/Signed_distance_function" rel="nofollow noopener">https://en.wikipedia.org/wiki/Signed_distance_function</a>) of the predicted segmentation label is required. The SDT can be computed using the euclidean distance transform. A comparable function is available in the scipy package (<a href="https://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.ndimage.morphology.distance_transform_edt.html" rel="nofollow noopener">https://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.ndimage.morphology.distance_transform_edt.html</a>). Does anyone know how to compute the SDT of the predicted segmentation label to perform the backward operation afterwards?</p>

John1231983: <p><a class="mention" href="/u/tkepp">@tkepp</a>: Interesting! Have you find any solution. I used regression to do it and MSE loss but the performance is very poor. The output also has negative value. I did not find the way to force them to positive. Any idea</p>

tkepp: <p>Hey,<br>
I used the L1 loss for the regression of the SDTs. Unfortunately I haven’t found a better solution yet.</p>

John1231983: <p>So, SDTs used ndimage to compute. Am I right? I am looking at the idea <a href="https://arxiv.org/abs/1901.01238" rel="nofollow noopener">https://arxiv.org/abs/1901.01238</a> but it does not show any improvement in my case</p>

11185: <p>Have you addressed this problem? We also want to implement the distance transform with the pytorch. It would be great if you could share your implementation.</p>

tkepp: <p>A derivable SDT was recently presented: <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7972695/" class="inline-onebox" rel="noopener nofollow ugc">A Differentiable Convolutional Distance Transform Layer for Improved Image Segmentation</a></p>

Duane_Nielsen: <p>In case anyone comes here in the future.</p>
<p>This was implemented into the Kornia library.</p>
<p><a href="https://kornia.readthedocs.io/en/latest/contrib.html#kornia.contrib.DistanceTransform" class="onebox" target="_blank" rel="noopener nofollow ugc">https://kornia.readthedocs.io/en/latest/contrib.html#kornia.contrib.DistanceTransform</a></p>
Intuition Behind Nesterov Update Rule for SGD w/ Momentum

Andrew_Holmes: <p>In the docs for the <code>SGD</code> class that implements Stochastic Gradient Descent w/ Momentum, there’s a modification in the update rule and something I’m confused about when Nesterov Accelerated Gradient (NAG) is enabled. The docs mention:</p>
<p>PyTorch update:<br>
v_t+1=μ∗vt +gt+1<br>
​pt+1=pt−lr∗v t+1<br>
​<br>
​Typical momentum update:<br>
v_t+1=μ∗vt + lr*gt+1<br>
​pt+1=pt−vt+1</p>
<p>What’s the reason for this change/difference?</p>
<p>Also the Nesterov application is said to be from the paper with this question:<br>
vt+1 = µvt − ε∇f(θt + µvt) (3)<br>
θt+1 = θt + vt+1 (4)</p>
<p>From this formula, how do you arrive at gt = gt + μvt?</p>
<p><a href="https://pytorch.org/docs/stable/generated/torch.optim.SGD.html#:~:text=The%20implementation%20of,to%20all%20zeros." rel="noopener nofollow ugc">Document reference</a></p>
Shared parameters

gautamvarmadatla: <p>I would like to implement cross-layer parameter sharing similar to what’s done in ALBERT, between attention blocks. I basically want to share the same K,Q,V matrices… Can anyone please check this implement and confirm if it does the same?</p>
<p>import torch<br>
import torch.nn as nn<br>
import torch.nn.functional as F<br>
import math</p>
<p>class SharedAttention(nn.Module):<br>
def <strong>init</strong>(self, shared_Wq, shared_Wk, shared_Wv, input_dim, hidden_dim, num_heads):<br>
super(SharedAttention, self).<strong>init</strong>()<br>
self.input_dim = input_dim<br>
self.hidden_dim = hidden_dim<br>
self.num_heads = num_heads</p>
<pre><code>    self.shared_Wq = shared_Wq
    self.shared_Wk = shared_Wk
    self.shared_Wv = shared_Wv

def forward(self, input):
    # Linear transformation for Q, K, and V
    q = torch.matmul(input , self.shared_Wq)
    k = torch.matmul(input , self.shared_Wq)
    v = torch.matmul(input , self.shared_Wq)
    
    # Split into multiple heads
    q = q.view(input.size(0), -1, self.num_heads, self.hidden_dim // self.num_heads).transpose(1, 2)
    k = k.view(input.size(0), -1, self.num_heads, self.hidden_dim // self.num_heads).transpose(1, 2)
    v = v.view(input.size(0), -1, self.num_heads, self.hidden_dim // self.num_heads).transpose(1, 2)
    
    # Apply shared attention mechanism
    scores = torch.matmul(q , k.transpose(-2, -1)) / math.sqrt(self.hidden_dim)
    attention_weights = F.softmax(scores, dim=-1)
    output = torch.matmul(attention_weights, v)

    # Concatenate and reshape to original shape
    output = output.transpose(1, 2).contiguous().view(input.size(0), -1, self.hidden_dim)
    
    return output
</code></pre>
<p>class Model(nn.Module):<br>
def <strong>init</strong>(self, shared_Wq, shared_Wk, shared_Wv, input_dim, hidden_dim, output_dim, num_heads):<br>
super(Model, self).<strong>init</strong>()<br>
self.attention1 = SharedAttention(shared_Wq, shared_Wk, shared_Wv, input_dim, hidden_dim, num_heads)<br>
self.attention2 = SharedAttention(shared_Wq, shared_Wk, shared_Wv, input_dim, hidden_dim, num_heads)<br>
self.fc = nn.Linear(hidden_dim, output_dim)</p>
<pre><code>def forward(self, input):
    output1 = self.attention1(input)
    output2 = self.attention2(output1)
    output = self.fc(output2)
    return output
</code></pre>
<p>input_dim = 6<br>
hidden_dim = 12<br>
output_dim = 1<br>
num_heads = 2<br>
shared_Wq = nn.Parameter(torch.randn(input_dim, num_heads * hidden_dim))<br>
shared_Wk = nn.Parameter(torch.randn(input_dim, num_heads * hidden_dim))<br>
shared_Wv = nn.Parameter(torch.randn(input_dim, num_heads * hidden_dim))</p>
<h1><a name="instantiate-the-model-with-shared-weights-1" class="anchor" href="#instantiate-the-model-with-shared-weights-1"></a>Instantiate the model with shared weights</h1>
<p>model = Model(shared_Wq, shared_Wk, shared_Wv, input_dim=input_dim, hidden_dim=hidden_dim, output_dim=output_dim, num_heads=2)</p>
Effective multiplication over last dimension

mY_NAme: <p>I have two torch tensors - A of shape <code>(15, 100, 256)</code> and B of shape (120, 2010, 256). How can I make effective multiplication over last dimension and get tensor of shape <code>(15, 100, 120, 2010)</code>. I tried something like <code>torch.einsum('ijk, mnk -&gt; ijmn', A, B)</code>, but as I understand this method implicitly creates intermediate tensors and takes a lot of memory and time. I tried also <code>opt_einsum</code> library, but didn’t see big difference in time and memory usage comparing to <code>torch.einsum</code></p>
<p>Of course, I can just make it in a loop, but I wanna get solution that will be efficient either in time and in memory usage. Thank you in advance</p>

smth: <p>The problem is that because you’re broadcasting, and then it routes to a batch matmul, pytorch will materialize the fully broadcasted and expanded tensor into memory first and then run the batch matmul, which is not good for memory usage as you see.</p>
<p>The first thing I would try is to write it out without <code>torch.einsum</code>, i.e. just write it as <code>unsqueeze + matmul</code> and then wrap it with <code>torch.compile</code> and see if PyTorch is able to compile it.<br>
Something like this:</p>
<pre><code class="lang-auto">import torch

# Sample dimensions
i, j, k, m, n = 2, 3, 4, 5, 6
A = torch.randn(i, j, k)
B = torch.randn(m, n, k)

@torch.compile
def outermul(A, B):
    # Reshape and expand A and B for batched multiplication
    A_expanded = A.unsqueeze(2).unsqueeze(3)  # Shape: (i, j, 1, 1, k)
    B_expanded = B.unsqueeze(0).unsqueeze(0)  # Shape: (1, 1, m, n, k)
    B_expanded = B_expanded.transpose(3, 4)  # Now Shape: (1, 1, m, k, n)

    # Multiply using torch.matmul, automatically summing over the last dimension of A and the second-to-last of B
    result = torch.matmul(A_expanded, B_expanded).squeeze(3)  # Resulting shape: (i, j, m, n)
    return result

result = outermul(A, B)
result.shape
</code></pre>
<p>If that doesn’t quite work out to your liking, if you are inclined to try something a bit more advanced, you should write a Triton kernel for this, it should be efficient, while also being concise. Here’s a couple of links for you to get started with Triton:</p>
<ul>
<li><a href="https://triton-lang.org/main/getting-started/tutorials/01-vector-add.html" class="inline-onebox">Vector Addition — Triton documentation</a></li>
<li><a href="https://github.com/srush/Triton-Puzzles/blob/main/Triton-Puzzles.ipynb" class="inline-onebox">Triton-Puzzles/Triton-Puzzles.ipynb at main · srush/Triton-Puzzles · GitHub</a></li>
</ul>

mY_NAme: <p><a class="mention" href="/u/smth">@smth</a> The problem of matmul with unsqueeze is large memory usage. As I understand, with unsqueeze we add dimension and then matmul will create a copy of tensor under the hood and this tensor’s memory usage will be large.</p>

smth: <p>that’s why I suggested to <code>torch.compile</code> it, because the compiler might generate a memory-efficient fused kernel for it if we’re lucky.</p>
<p>Otherwise, writing a custom Triton kernel is the only way to go.</p>
Mask rcnn model returns undefined tensor

squareroot: <p>Why i got undefined mask tensor from forward ( CPUFloatType{0,1,640,480} ] )?</p>

ptrblck: <p>The shape indicates an empty tensor, not an undefined one.<br>
This could happen if the tensor is sliced out of its bounds:</p>
<pre data-code-wrap="python"><code class="lang-python">x = torch.randn(1, 1, 640, 480)
x[1:].shape
# torch.Size([0, 1, 640, 480])
</code></pre>
Torch.compile performance with recursive code

mathis_c99: <p>Hi all,</p>
<p>For the past two years I’ve been using JAX. I now see that Pytorch introduced torch.compile and torch.vmap and wanted to try this. In JAX, I could wrap the loops in <code>jax.lax.scan</code>, such that the JIT compiler knows to compile without unrolling the loop, therefore ensuring low compilation times. I tried to compile a function containing a for loop with torch.compile (no data-dependent control flow, just iterating through an input array and returning another array).</p>
<pre><code class="lang-auto">def test_func(x_seq):
    result = torch.empty_like(x_seq)
    for i, (x,y) in enumerate(zip(x_seq, torch.arange(0,1,x_seq.shape[0]))):
        result[i] = x*y
    return result
</code></pre>
<p>The compilation time increases with the length of x_seq, which is unusable for me because my sequences can be very long. Is there a torch primitive to avoid this ?</p>
Keras-like add_loss for torch.nn.Module

kmdalton: <p>I develop Bayesian models using Variational Inference and am coming from a TensorFlow / TensorFlow Probability background. I’ve found the <code>keras</code> loss aggregation pattern where layers call <code>self.add_loss(kl_div)</code> can be very useful for this application domain. It leads to more modular code where any model component can locally add a prior for an intermediate random variable. I’d like to implement a similar pattern for <code>torch.nn.Module</code>. Does anyone have any advice for how best to tackle this?</p>
<ul>
<li>Should I use <code>buffers</code>?</li>
<li>Should I just set some private attribute of the module and then aggregate it by traversing all the modules from the top-level module by calling <code>Module.modules()</code>?</li>
</ul>
Host system freezing while trying to install pytorch from souce

Pranjal.Pok7: <p>I am trying to install pytorch from source with cuda 12.4 and when i run the command<br>
“python setup.py develop” my system freezes after few minutes.<br>
How can i address this issue ?<br>
I have 16gb ram gtx 1660 ti</p>

ptrblck: <p>You might be running out of memory during the build and could reduce the number of jobs via <code>export MAX_JOBS=N</code> for e.g. <code>N=4</code>.</p>

Pranjal.Pok7: <p>Should I write it like <code>export MAX_JOBS=4 python setup.py develop</code> and then run the command in terminal?</p>
Partitioned matrix multiplication

mY_NAme: <p>I have two lists with tensors. Tensors in first list have shapes (3, 200), (5, 200), (11, 200). Tensors in second list have shapes (2, 200), (4, 200), (7, 200), (13, 200), (14, 200). I want to get a matrix of shape (3, 5) - score of each tensor in first list with each tensor in second list. I tried to concat tensors in first list, tensors in second list and get tensors of shapes (19, 200) and (40, 200), then multiply them, but there is no way to get tensor of shape (3, 5) at the end. How can I do that without padding</p>
drop_last=False and the last batch of data cannot be evenly distributed to each GPU

dkz: <p>In the multi-GPU dataloader, if I set drop_last=False and the last batch of data cannot be evenly distributed to each GPU, what will PyTorch do?</p>

ptrblck: <p>The <code>DistributedSampler</code> will drop the tail of the samples as seen <a href="https://github.com/pytorch/pytorch/blob/63c24f73ef53cc7994b7e25b4cdf6b761e8f04bf/torch/utils/data/distributed.py#L113-L114">here</a>.</p>

dkz: <p>I understand, thank you!</p>
Omit loss.backward for forward-only algos?

qqqllppp: <p>In modern deep learning, using an algo with backprop is the mainstream way of getting started. In pytorch, the update of params is done by calling <code>loss.backward()</code>. Currently, I am exploring “forward-only” algos, and as the name implies, forward-only algo does not use backprop to update the weights.</p>
<p>A few questions regarding this:</p>
<ol>
<li>
<p>In pytorch, is there an alternative way to update the params that does not use <code>loss.backward()</code>?</p>
</li>
<li>
<p>If I do not plan to update my weights via <code>loss.backward()</code>, is it safe to remove that line altogether?</p>
</li>
</ol>
<p>My concern with (2) is that even Hinton’s “forward-forward” pytorch <a href="https://github.com/loeweX/Forward-Forward/blob/da516f094c70a4a92042fbbc7ab33303aa7dbfd9/main.py#L26C1-L26C46" rel="noopener nofollow ugc">implementation</a> (one of many forward-only algos) uses <code>loss.backward()</code>, which confuses me a bit and makes me think the <code>backward</code> function does more than just updating params? (I understand the use of <a href="https://github.com/loeweX/Forward-Forward/issues/3#issuecomment-1620438278" rel="noopener nofollow ugc">detach</a>, but if we detach all params, then is it the same as not calling <code>loss.backward()</code> at all?)</p>

soulitzer: <p>loss.backward() is actually only responsible for populating the .grad fields of the parameters. The optimizer step is what is responsible for updating the parameters using the .grad field</p>
<p>So, re: (2), if you are not performing back-prop it is fine to not use .backward(), but if you still want to use existing optimizers, you’re now responsible for setting the .grad fields</p>

qqqllppp: <p>That makes sense! Thank you <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
Libtorch C++ build 'Could NOT find Torch (missing: TORCH_LIBRARY)'

neolei: <p>os: Windows 10<br>
ide: clion</p>
<pre><code class="lang-auto">====================[ Build | app | Debug ]=====================================
D:\software\cygwin64\bin\cmake.exe --build /cygdrive/d/cpp_extension/app/cmake-build-debug --target app -- -j 6
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - found
-- Found Threads: TRUE  
CMake Error at /usr/share/cmake-3.14.5/Modules/FindPackageHandleStandardArgs.cmake:137 (message):
  Could NOT find Torch (missing: TORCH_LIBRARY)
Call Stack (most recent call first):
  /usr/share/cmake-3.14.5/Modules/FindPackageHandleStandardArgs.cmake:378 (_FPHSA_FAILURE_MESSAGE)
  D:/cpp_extension/libtorch/share/cmake/Torch/TorchConfig.cmake:135 (find_package_handle_standard_args)
  CMakeLists.txt:5 (find_package)


-- Configuring incomplete, errors occurred!
See also "/cygdrive/d/cpp_extension/app/cmake-build-debug/CMakeFiles/CMakeOutput.log".
make: *** [Makefile:176: cmake_check_build_system] Error 1
</code></pre>
<p>cmakelist</p>
<pre><code class="lang-auto">cmake_minimum_required(VERSION 3.14)
project(app)

set(CMAKE_PREFIX_PATH "D:\\cpp_extension\\libtorch")
find_package(Torch REQUIRED)

add_executable(app main.cpp)
target_link_libraries(app "${TORCH_LIBRARIES}")
set_property(TARGET app PROPERTY CXX_STANDARD 14)
</code></pre>

neolei: <p><a class="mention" href="/u/alband">@albanD</a>  Is there any solution?</p>

albanD: <p>Hi,</p>
<p>I am afraid I don’t know, especially for Windows build system <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>

neolei: <p>thanks, libtorch maybe be buit by MSVC. so it did not support cygwin <img src="https://discuss.pytorch.org/images/emoji/apple/sob.png?v=9" title=":sob:" class="emoji" alt=":sob:"></p>

lavoos: <p>I had fixed such an error by modifying CMakeList.txt<br>
list(APPEND CMAKE_PREFIX_PATH “C:/path_to_the_pytorch_sources_root”)<br>
find_package(Torch REQUIRED)</p>
<p>On my system I’ve downloaded file libtorch-win-shared-with-deps-debug-2.2.2+cpu.zip<br>
and unarchived it to the folder c:/libs/libtorch-win-shared-with-deps-debug-2.2.2+cpu. The unarchived path has been used in CMakeList.txt</p>
Does not contain CUDA Unified Memory CPU page faults data

Andy_P: <p>Dear community,<br>
I replaced the CUDA memory allocation from <code>cudaMalloc</code> to <code>cudaMallocManaged</code> in c10/cuda/CUDACachingAllocator.cpp in the PyTorch open-source code, and successfully compiled it. It can be used as expected, and PyTorch version is v1.13.0 .</p>
<p>When training a GNN, I successfully oversubscribed the GPU memory.</p>
<p>When I use <code>nsys</code> to analyze a Python program,<br>
<code>nsys profile --stats=true --cuda-um-gpu-page-faults=true --cuda-um-cpu-page-faults=true --show-output=true  python reddit.py</code><br>
there are no occurrences of any page faults.</p>
<p>Analysis results are as follows:</p>
<pre><code class="lang-auto">...
[ 9/11] Executing 'um_sum' stats report
SKIPPED: /mypath/report90.sqlite does not contain CUDA Unified Memory CPU page faults data.
[10/11] Executing 'um_total_sum' stats report
SKIPPED: /mypath/report90.sqlite does not contain CUDA Unified Memory CPU page faults data.
[11/11] Executing 'um_cpu_page_faults_sum' stats report
SKIPPED: /mypath/report90.sqlite does not contain CUDA Unified Memory CPU page faults data.
</code></pre>
<p>Why is that?</p>
<p>Thanks.</p>

hawkheimmer: <aside class="quote no-group" data-username="Andy_P" data-post="1" data-topic="196224">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/andy_p/48/61281_2.png" class="avatar"> Andy_P:</div>
<blockquote>
<p>nsys</p>
</blockquote>
</aside>
<p>How did you edit the CUDACachingAllocator.cpp? Did you just replace <code>cudaMalloc()</code> with <code>cudaMallocManaged()</code> or did you replace <code>cudaMalloc_count{}</code>, <code>cudaMallocMaybeCapturing()</code>, <code>release_lock_on_cudamalloc()</code> and <code>cudaMallocAsync()</code> as well?</p>
__getitem__ returning random sample

whoab: <p>Is it good practice to return a random sample in getitem?</p>
<p>My use case is:<br>
I am doing training over videos, and I want to return random subsequences of videos within each minibatch.</p>
<p>I’m not sure the best way to do this.</p>

Albertma: <p>i have not trained a model on videos. But I trained models on images. I always set shuffle True and set random torch.manual_seed(10). So that the random sequence is the same for each time.</p>
High Latency Variance During Inference

SvenS: <p><strong>Problem</strong><br>
I have a  resnet that I want to apply in a loop for a real time application. During deployment I noticed that the time it takes for applying my model is very inconsistent. At first it only takes ~2ms but after some time it sometimes even spikes up to ~12ms.</p>
<p><strong>Example</strong><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/4/3/433a98eb10b390edc6a54cb11cd810474791ab31.png" alt="image" data-base62-sha1="9AJx9wmZlhJfvBWKsyTiSHK4KLT" width="612" height="471"></p>
<p><strong>To Reproduce</strong><br>
I already manged to narrow down the specific circumstances:</p>
<ul>
<li>This only happens if I simultaneously load data from my hard drive.</li>
<li>It happened on multiple windows systems but not on linux.</li>
<li>The latency comes from moving my tensor to(‘cuda’) and from cuda to(‘cpu’)</li>
</ul>
<p>I wrote a minimal example:</p>
<pre><code class="lang-auto">from time import perf_counter_ns

import torch
import cv2 as cv
import matplotlib.pyplot as plt
import numpy as np

model = torch.hub.load('pytorch/vision:v0.10.0', 'resnet18', pretrained=True)

model.eval()
model = model.to("cuda")
tensor = torch.Tensor(np.random.rand(1, 3, 224, 224).astype(np.float32))

torch.cuda.synchronize()

timestamps = []

for x in range(3000):
    start = perf_counter_ns()
    model(tensor.to("cuda")).to("cpu")
    d_t = perf_counter_ns() - start
    timestamps.append(d_t)
    cv.imread(r"path/to/some/image.png")

plt.plot(np.array(timestamps[1:]) * 1e-6)
plt.xlabel("Inference count")
plt.ylabel("Time [ms]")
plt.show()
</code></pre>
<p><strong>System Information</strong></p>
<ul>
<li>os: win11</li>
<li>gpu: rtx 4070 ti</li>
<li>python: 3.11.7</li>
<li>torch: 2.1.2+cu118</li>
<li>torchvison: 0.16.2+cu118</li>
</ul>
Possible Logical Reasons behind Wrapping Examples around Special Tokens (SOS | EOS) While Preparing the Training Dataset for LLM

Harsh_Mittal: <p>I’m in the process to train a small language model. I’m stuck in the data preparation step where I have seen how It’s a basic practice to wrap the example with special tokens (SOS, EOS).</p>
<p>I’d appreciate the logic behind this practice - also how it would affect the model performance If we don’t wrap the data with special tokens.</p>
<p>I’m attaching a sample data just to showcase my understanding for ref which I wrapped with the SOS token &amp; tried to generate X (input) &amp; Y (output) tensors.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/a/0aed90f4bc70ecc1011b39ee309cae3c37115651.png" data-download-href="https://discuss.pytorch.org/uploads/default/0aed90f4bc70ecc1011b39ee309cae3c37115651" title="Screenshot 2024-04-18 at 3.01.27 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/a/0aed90f4bc70ecc1011b39ee309cae3c37115651_2_690x341.png" alt="Screenshot 2024-04-18 at 3.01.27 PM" data-base62-sha1="1yFKZO1329IRDhackjO9znOAmiZ" width="690" height="341" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/a/0aed90f4bc70ecc1011b39ee309cae3c37115651_2_690x341.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/a/0aed90f4bc70ecc1011b39ee309cae3c37115651_2_1035x511.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/0/a/0aed90f4bc70ecc1011b39ee309cae3c37115651_2_1380x682.png 2x" data-dominant-color="F2F2F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2024-04-18 at 3.01.27 PM</span><span class="informations">1662×822 93.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Looking forward to hearing from the community <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"><br>
Cheers!</p>
How to solve this’OSError: [Errno 22] Invalid argument:’

Naveen_Hedalla_Arach: <p>I was trying to use a lpmodel using the below line of code,</p>
<p>model = lp.AutoLayoutModel(“lp://efficientdet/PubLayNet/tf_efficientdet_d0”)</p>
<p>But it returns the following error.</p>
<p>OSError: [Errno 22] Invalid argument: ‘C:\Users\navee/.torch/iopath_cache\s/ukbw5s673633hsw\publaynet-tf_efficientdet_d0.pth.tar?dl=1.lock’</p>
<p>How can I solve this?</p>
Is there a way of accessing grad from other parameters in torch.optim.Optimiter.step() function?

msqiqi10: <p>Say I have 2 parameters W1 and W2, for some reason when updating W1 using the TORCH.OPTIM.OPTIMIZER.STEP function, I have to use the grad from W2, when usually you can only loop through the parameters registered with optimizer.<br>
So I’m wondering is there a way doing this by creating a customized optimizer module myself?</p>

ptrblck: <p>Yes, you should be able to implement your own custom optimizer and could take a look at e.g. <a href="https://github.com/pytorch/pytorch/blob/c9ab9248ce0391c233e7189c0ee71f2aca6a786c/torch/optim/sgd.py#L301-L338"><code>sgd</code></a> as a template.</p>

msqiqi10: <p>Hi! Thanks for the reply.<br>
I’m wondering how would it be a good practice if I want to split all parameters into 3 sets, A, B and C.<br>
A’s normally updated and B’s gradients is grad_B + grad_C, C is not updated? How do you iterate them accordingly?</p>
Build lib upon major version of libtorch

Rhett-Ying: <p>We are building a library upon libtorch. I am wondering if it’s ok that we build the library upon torch 2.2.0 and run such library upon torch 2.2.x(such as 2.2.1, 2.2.2). Namely, we build upon the major version of torch(such as 2.0, 2.1, 2.2) and run upon any following minor versions(such as 2.0.x, 2.1.x, 2.2,x). Is it ok to assume no API/symbol change between minor versions?</p>

smth: <p>PyTorch does not follow semantic versioning. There could be API changes between minor versions as well (thought we try very hard not to introduce such changes)</p>
ANN Implementation for Nanofluid Thermal Conductivity Prediction

saleh1: <p>I am Saleh, a nanofluid researcher seeking help with ANN implementation for predicting the thermal conductivity of Al2O3/water-ethylene glycol hybrid nanofluid.</p>
<p>I have prepared a small dataset of 30 data points which I will divide into 15% for validation, 70% for training, and 15% for testing. I have selected the following specifications for the ANN based on other research trends :</p>
<ul>
<li>
<p>ANN morphology: Multilayer perceptron (MLP) network</p>
</li>
<li>
<p>Network type: Feed-forward Artificial Neural Network (ANN)</p>
</li>
<li>
<p>Training Method: Levenberg–Marquardt algorithm</p>
</li>
<li>
<p>Error Criteria: Mean Square Error (MSE)</p>
</li>
<li>
<p>Number of Hidden Layers: 1</p>
</li>
<li>
<p>Number of neurons in the hidden layer: Increase from 7 to 21 with a step size of 2</p>
</li>
<li>
<p>Hidden Layer function: Tangent-sigmoid function</p>
</li>
<li>
<p>Output Layer function: Linear (Purelin activation function)</p>
</li>
<li>
<p>Inputs: Volume fraction (φ) and temperature (T)</p>
</li>
<li>
<p>Outputs: Thermal conductivity of the nanofluid (knf)</p>
</li>
</ul>
<p>With my knowledge in nanofluids but limited coding skills, I kindly request assistance from the community in implementing the ANN for my project.</p>
<p>for anybody that is willing to help<br>
contact info below</p>
<p><a href="mailto:salehaltoaimi.s@gmail.com">salehaltoaimi.s@gmail.com</a></p>
Efficient shuffling buffer in pinned memory

tresan: <p>Hello everyone,</p>
<p>I have a question regarding the efficient implementation of a shuffling buffer during data loading. I am dealing with video data and I am using NVIDIA DALI for efficiently loading and transforming the data on the GPU. However, since my data is loaded sequentially, I must additionally implement some sort of shuffling buffer. In order to not fill GPU memory too much this buffer must reside on the CPU side.</p>
<p>The data flow therefore is:</p>
<ol>
<li>Process samples on GPU</li>
<li>Put samples into large buffer residing on CPU (device change <span class="hashtag-raw">#1</span>)</li>
<li>Randomly pick samples from buffer and transfer back to GPU (device change <span class="hashtag-raw">#2</span>)</li>
</ol>
<p>I would guess the most efficient way for such a structure would be to utilize pinned memory and non blocking device transfers as much as possible. Sample code:</p>
<pre><code class="lang-auto">...
data_pipe = make_data_pipe(...)
buffer = []
...
for sample in data_pipe:
    sample = sample.to('cpu', non_blocking=True)
    # needs synchronization ?
    # torch.cuda.current_stream().synchronize()
    buffer.append(sample)
    if len(buffer) &gt;= max_buffer_size:
        sample = randomly_pick_from_buffer(buffer)
        sample = sample.to('cuda', non_blocking=True)
        # or needs synchronization here ?
        # torch.cuda.current_stream().synchronize()
        yield sample
</code></pre>
<ol>
<li>Am I correct in the assumption that I need to synchronize the current cuda stream when transferring data from cuda to cpu (pinned) in a non blocking way?</li>
<li>If yes, is there a way to synchronize only the sample that is actually picked from the buffer? The buffer is pretty large, so there is no need to synchronize all the data transfer that is put in the buffer, as it might reside there for quite a while. So synchronizing every transfer put in the buffer at each iteration might be inefficient? Or is my general view about the cuda semantics entirely wrong?</li>
</ol>
<p>I would really appreciate some clarification and help!</p>
Ppo implementation

Krishna_Mohan: <aside class="onebox allowlistedgeneric" data-onebox-src="https://colab.research.google.com/drive/1jYhG8m8otFv1cHD0ydKVRJygGMT7D3LG">
  <header class="source">
      <img src="https://ssl.gstatic.com/colaboratory-static/common/ef9642ed288c98cc51e850391b18dd3f/img/favicon.ico" class="site-icon" width="128" height="128">

      <a href="https://colab.research.google.com/drive/1jYhG8m8otFv1cHD0ydKVRJygGMT7D3LG" target="_blank" rel="noopener nofollow ugc">colab.research.google.com</a>
  </header>

  <article class="onebox-body">
    <img width="260" height="260" src="https://colab.research.google.com/img/colab_favicon_256px.png" class="thumbnail onebox-avatar">

<h3><a href="https://colab.research.google.com/drive/1jYhG8m8otFv1cHD0ydKVRJygGMT7D3LG" target="_blank" rel="noopener nofollow ugc">Google Colaboratory</a></h3>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
This is the code I have implemented (ppo algo) but this code is not working, each time rewards are continuously decreasing<br>
It will be very helpful if someone can help me<br>
I am currently newbie to this<br>
Btw thanks</p>