akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Set nn.Parameter during training

Max_Unhold: <p>Hello,</p>
<p>I would like to set an <code>nn.Parameter</code> during training. What is the right way to do so, without<br>
breaking the connection to the optimizer (i.e. creating a copy of the tensor).</p>
<p>I tried to use <code>torch.where</code> but even if I use an <code>nn.Parameter(torch.tensor(1.))</code> as replacement, it throws an errror.</p>
<pre><code class="lang-auto">TypeError: cannot assign 'torch.FloatTensor' as parameter 'weight' (torch.nn.Parameter or None expected)
</code></pre>

ptrblck: <p>I’m unsure if I understand your use case correctly but in case you want to directly manipulate a trainable parameter (i.e. without calculating the gradients and using an optimizer), you could use a <code>no_grad</code> context as seen here:</p>
<pre><code class="lang-python">model = nn.Linear(10, 10)
optimizer = torch.optim.Adam(model.parameters(), lr=1.)

out = model(torch.randn(1, 10))
out.mean().backward()

print(model.weight.abs().sum())
# tensor(14.3759, grad_fn=&lt;SumBackward0&gt;)
optimizer.step()
print(model.weight.abs().sum())
# tensor(97.4504, grad_fn=&lt;SumBackward0&gt;)

model.zero_grad()
with torch.no_grad():
    model.weight.copy_(torch.ones_like(model.weight))
print(model.weight)

# make sure model is still updated
out = model(torch.randn(1, 10))
out.mean().backward()

print(model.weight.abs().sum())
# tensor(100., grad_fn=&lt;SumBackward0&gt;)
optimizer.step()
print(model.weight.abs().sum())
# tensor(100.3176, grad_fn=&lt;SumBackward0&gt;)
</code></pre>

Max_Unhold: <pre><code class="lang-python">class TinyModel(torch.nn.Module):

    def __init__(self):
        super(TinyModel, self).__init__()

        self.layer1 = torch.nn.Linear(1000, 100)
        self.relu = torch.nn.ReLU()
        self.layer2 = torch.nn.Linear(100, 10)

    def forward(self, x):
        with torch.no_grad():
            addition = torch.ones_like(model.weight)
            self.layer1.weight.copy_(self.layer1.weight + addition)

        x = self.layer1(x)
        x = self.relu(x)
        x = self.layer2(x)
        return x
</code></pre>
<p>So I could use the <code>copy</code> operation without running into troubles that the new weights are not registered by Adam under any circumstances (as long as I keep the shape of model.weights)?</p>

Aliadast_Lin: <p>Hi, I did a similar operation, but it seems that if the optimizer is not processed, the parameters will fall back to the previous values.</p>
Ignore a inner function with @torch.compile

marioo: <p>Hi all,</p>
<p>I need to torch.compile a function but it cannot be done because an inner function uses complex valued tensors, which are not supported by torch.compile. However, the outputs of this problematic function are floats, so the problem could be solved if I could do something like <span class="mention">@torch.compile.ignore</span> to this function.</p>
<p>Do you know any option for this? I have been google’ing and I have not find anything. It could be a nice feature.</p>

marksaroufim: <p>This tutorial might be helpful <a href="https://pytorch.org/docs/stable/torch.compiler_fine_grain_apis.html" class="inline-onebox">TorchDynamo APIs for fine-grained tracing — PyTorch 2.1 documentation</a></p>
<p>Depending on your use case you want to go with disable or disallow_in_graph</p>
UserWarning: ONNX export unsqueeze with negative axis -1 might cause the onnx model to be incorrect. Negative axis is not supported in ONNX.

boy_little: <p>UserWarning: ONNX export unsqueeze with negative axis -1 might cause the onnx model to be incorrect. Negative axis is not supported in ONNX. Axis is converted to 1 based on input shape at export time. Passing an tensor of different rank in execution will be incorrect.<br>
“Passing an tensor of different rank in execution will be incorrect.”)</p>
<p>I encountered this issue when exporting the pytorch model as an onnx model. It seems that the unsqueeze (-1) operation is not supported in onnx. After my tracking, it seems that the unsqueeze (-1) operation is in the function scatter_add_. But due to the absence of the  function scatter_ add_, I don’t know how to modify the Python code. What should I do.</p>
What does gard_input and grad_output mean

shakeel608: <blockquote>
<p>Can anyone please explain with a simple example what is the exact meaning of grad_input and grad_output. I tired to understand from documentations but I couldn’t grasp it</p>
</blockquote>

ptrblck: <p><code>grad_output</code> is the gradient coming from the output of the module during the backward pass while <code>grad_input</code> is the gradient which will be passed to the corresponding input of the module during the backward pass.<br>
Here is a small example using <code>nn.ReLU()</code>:</p>
<pre data-code-wrap="python"><code class="lang-python">m = nn.ReLU()
m.register_full_backward_hook(lambda module, grad_input, grad_output: print("grad_input: {}\ngrad_output: {}".format(grad_input, grad_output)))

x = torch.randn(2, 2, requires_grad=True)
print(x)
# tensor([[-0.5412, -0.2550],
#         [-1.5957,  0.2068]], requires_grad=True)

out = m(x)
print(out)
# tensor([[0.0000, 0.0000],
#         [0.0000, 0.2068]], grad_fn=&lt;BackwardHookFunctionBackward&gt;)

out.backward(gradient=torch.ones_like(out)*2.)
# grad_input: (tensor([[0., 0.],
#         [0., 2.]]),)
# grad_output: (tensor([[2., 2.],
#         [2., 2.]]),)

print(x.grad)
# tensor([[0., 0.],
#         [0., 2.]])
</code></pre>
<p>As you can see <code>grad_output</code> corresponds to the gradient I’m passing to <code>backward</code> while <code>grad_input</code> will be returned and assigned to <code>x</code> as it’s the input of the module.</p>

shakeel608: <p>Thanks a lot.<br>
This makes sense  to me  now.<br>
If I need these grad_out from the hooks in the main training loop?<br>
How should I get those after the backward call</p>

ptrblck: <p>You can use backward hooks to access these gradients as seen in my code snippet.</p>
How to login to Collibra from AWS EC2 instance?

sai_chinnu1852: <p>I am trying to use Collibra by installing it using AWS Marketplace AMI that they offer and launched the EC2 instance. However, when I try to login to Collibra using the IP Address of the EC2 instance and I am unable to connect. I tried adding the IP Address in the proxy settings but did notwork. Any idea what should I do?</p>
<p>I tried logging into Collibra website using the Public IPv4 Address and also tried connecting using the IPv4 DNS name as well. Neither of them worked. I have also tried launching it multiple times but with no success</p>

ptrblck: <p>Your post seems to be unrelated to PyTorch and you might want to ask the question in an AWS specific forum.</p>
Transforms v2 failing for real Coco Dataset

besterma: <p>I’ve recently tried to run the Transforms v2 tutorial on <a href="https://pytorch.org/vision/master/auto_examples/transforms/plot_transforms_e2e.html" rel="noopener nofollow ugc">end-to-end object detection</a>.<br>
When working on the real Coco 2017 train dataset, v2.SanitizeBoundingBoxes always fails due to some coco samples not containing any annotations.</p>
<p>Is it really not possible to use Transforms v2 with CocoDetection out-of-the-box, like described in the tutorial or am I missing something obvious?</p>
<p>Thanks in advance for your help,<br>
Benjamin</p>

besterma: <p>So it seems like there exists a utility function for coco to remove the unlabeled datapoints <a href="https://github.com/pytorch/vision/blob/13b35ffaa5167f3713ea7a53c43395d90b3a7cbc/references/detection/coco_utils.py#L106" rel="noopener nofollow ugc">here</a>. Unfortunately, this does not play together with the datasets.wrap_dataset_for_transforms_v2, but can be used in an own dataset class. I’m still perplexed as to why this is not really mentioned anywhere.</p>
Unable to convert output to PyTorch tensors format, PyTorch is not installed

acayci: <p>Hello,</p>
<p>I installed torch in MacOS in my Macbook Pro.  It is Apple M2 Pro.  ACAYCI-mac 23.2.0 Darwin Kernel Version 23.2.0: Wed Nov 15 21:55:06 PST 2023; root:xnu-10002.61.3~2/RELEASE_ARM64_T6020 arm64</p>
<p>I installed it using pip3.</p>
<pre><code class="lang-auto">(base) bash-3.2$ pip3 show torch
Name: torch
Version: 2.1.2
Summary: Tensors and Dynamic neural networks in Python with strong GPU acceleration
Home-page: https://pytorch.org/
Author: PyTorch Team
Author-email: packages@pytorch.org
License: BSD-3
Location: /Users/ACAYCI/anaconda3/lib/python3.11/site-packages
Requires: filelock, fsspec, jinja2, networkx, sympy, typing-extensions
Required-by: torchaudio, torchvision
</code></pre>
<p>In my Jupyter notebook,  I have the following code:</p>
<p>input_ids = tokenizer.encode(input_text, return_tensors=‘pt’)</p>
<p>During the execution,  it is giving</p>
<p>File ~/anaconda3/lib/python3.11/site-packages/transformers/tokenization_utils_base.py:700, in BatchEncoding.convert_to_tensors(self, tensor_type, prepend_batch_axis)<br>
698 elif tensor_type == TensorType.PYTORCH:<br>
699     if not is_torch_available():<br>
 → 700         raise ImportError(“Unable to convert output to PyTorch tensors format, PyTorch is not installed.”)<br>
701     import torch<br>
703     is_tensor = torch.is_tensor</p>
<p>ImportError: Unable to convert output to PyTorch tensors format, PyTorch is not installed.</p>
<p>Any help is much appreciated.</p>
<p>Mustafa</p>

acayci: <p>Hello,</p>
<p>I was able to fix this issue. I just restarted the Kernel.  It took care of it.</p>
<p>Mustafa</p>
Unexplained overhead when using custom CUDA kernel in torch.compile reduce-overhead mode

Vrushank_Desai: <p>Hi,</p>
<p>I am working on inference optimization of a diffusion model for pedagogical purposes. Since the model uses lots of small kernels and inference is with batch-size=1, pytorch overhead is huuuge. I am trying to use CUDA graphs to eliminate this overhead, as well as integrate a custom CUDA kernel for one of the blocks (1D Conv + group norm + mish). When using torch.compile in reduce-overhead mode, I am finding that the standard pytorch is able to more or less fully graph the network and all kernels run back to back in a single GPU stream. When using a custom kernel for the 1D conv + GN + M block, alignment of pytorch kernels and the kernels between streams seems off, and there is a huge amount of idle-time, where no activity is shown in trace on CPU or GPU. Initially I thought this was CUDA stream related, but even after passing the current CUDA stream with the ATen C++ API (getCurrentCUDAStream()) I see this behavior. The CUDA graph also seems to break into many pieces with the custom kernel.</p>
<p>Any advice on how to fit all pytorch + custom kernel execution for my network into a single kernel? I see my custom block is 3x faster than pytorch on its own, so if it could integrate into the network more organically I think I would get much better performance. Included trace and profiler stats for both custom &amp; pytorch launches to this post.</p>
<p>Thanks in advance!</p>
<p>Standard Pytorch Graphed Run-time, according to profiler:<br>
Self CPU time total: 1.953ms<br>
Self CUDA time total: 1.676ms<br>
<a href="https://i.imgur.com/UI4vMh7.png" class="onebox" target="_blank" rel="noopener nofollow ugc">https://i.imgur.com/UI4vMh7.png</a></p>
<p>Custom Kernel Graphed Run-time:<br>
Self CPU time total: 754.000us<br>
Self CUDA time total: 1.226ms<br>
<a href="https://imgur.com/a/ggT566o" class="onebox" target="_blank" rel="noopener nofollow ugc">https://imgur.com/a/ggT566o</a></p>
<p>Notice in the traces that despite lower CPU/CUDA time, overall run-time with custom kernel is longer due to idle time.</p>

Vrushank_Desai: <p>I was able to get a single graph using the standard stream capture API, I tried this a few days ago but had the torch.compile(mode=‘reduce-overhead’) decorator on so got some strange errors. So my original issue is fixed.</p>
<p>But if anyone has tips on making custom CUDA kernels play better with compile &amp; reduce overhead mode I would love to hear them!</p>
How can i remove for loops in this code that i can run it on GPU?

fatemeh.sepah: <p>hi<br>
how can i remove for loops in this code that i can run it on GPU?</p>
<p>class MyConv2d1(nn.Conv2d):<br>
def <strong>init</strong>(self,in_channels, out_channels, kernel_size, stride=1, padding=0):<br>
super(MyConv2d1, self).<strong>init</strong>(in_channels, out_channels, kernel_size, stride, padding)<br>
self.in_channels = in_channels<br>
self.out_channels = out_channels<br>
self.kernel_size = kernel_size<br>
self.stride = stride<br>
self.padding = padding</p>
<pre><code>    # Initialize weights and biases
    self.weights = layer_weights1#torch.randn(out_channels, in_channels, kernel_size, kernel_size)
    self.biases = layer_bias1#torch.zeros((out_channels, 1))

def forward(self, input):
    batch_size, in_channels, in_height, in_width = input.shape

    # Calculate output dimensions
    out_height = int((in_height + 2 * self.padding - self.kernel_size) / self.stride + 1)
    out_width = int((in_width + 2 * self.padding - self.kernel_size) / self.stride + 1)

    # Pad input
    input_padded = torch.nn.functional.pad(input, (self.padding, self.padding, self.padding, self.padding))

    # Initialize output
    output = torch.zeros((batch_size, self.out_channels, out_height, out_width))

    # Perform convolution

    unfoldedconv1=torch.nn.functional.unfold(input_padded,(self.kernel_size,self.kernel_size))
    inputfold=torch.tensor(unfoldedconv1)

    inputfold=unfoldedconv1.transpose(1,2)

    inputfold.apply_(FloatToBinary)

    outputfold=torch.zeros((batch_size,inputfold.shape[1],self.out_channels))

    weightfold=self.weights.view(self.weights.size(0),-1).t()
    weightfold2=torch.tensor(weightfold)
    weightfold2.apply_(FloatToBinaryThenInt)


    for b in range(batch_size):
        for outchannel in range(self.out_channels):
            for i in range(inputfold.shape[1]):
                sum=0
                for j in range(inputfold.shape[2]):
                    weightValue=abs(int(weightfold2[j][outchannel]))
                    SN=torch.zeros((1,weightValue))
                    SN=BISC(int(inputfold[b][i][j]))
                    partSN=torch.tensor(SN[0:weightValue])

                    if weightfold2[j][outchannel]&lt;0:
                        sum-=torch.count_nonzero(partSN)
                    else:
                        sum+=torch.count_nonzero(partSN)

                outputfold[b][i][outchannel]=sum*(2**length)
        outputfold/=2**(2*length)

    outputfold=outputfold.transpose(1,2)


    output=torch.nn.functional.fold(outputfold,(out_height,out_width),(1,1))
</code></pre>
How to modify the Network to combine CNN and LSTM Model

Leonardo_Bellizzi: <p>Hi, I’m trying to put together a CNN and a LSTM for a regression problem. I want to estimate BPM from video frame.<br>
I am quite a new gamer of torch so I will appriciate an explanation and a solution.<br>
Here is the code:</p>
<pre><code class="lang-auto">class CustomModel(nn.Module):
    def __init__(self):
        super(CustomModel, self).__init__()
        self.conv2D_1 = nn.Conv2d(3, 8, kernel_size=1, stride=1, padding=0)
        self.conv2D_2 = nn.Conv2d(8, 16, kernel_size=2, stride=2, padding=0)
        self.conv2D_3 = nn.Conv2d(16, 16, kernel_size=1, stride=1, padding=0)
        self.conv2D_4 = nn.Conv2d(16, 32, kernel_size=2, stride=2, padding=0)
        self.conv2D_5 = nn.Conv2d(32, 32, kernel_size=1, stride=1, padding=0)
        self.conv2D_6 = nn.Conv2d(32, 64, kernel_size=2, stride=2, padding=0)
        self.conv2D_7 = nn.Conv2d(64, 64, kernel_size=1, stride=1, padding=0)
        self.conv2D_8 = nn.Conv2d(64, 128, kernel_size=2, stride=2, padding=0)
        self.conv2D_9 = nn.Conv2d(128, 128, kernel_size=1, stride=1, padding=0)
        self.conv2D_10 = nn.Conv2d(128, 256, kernel_size=2, stride=2, padding=0)
        self.conv2D_11 = nn.Conv2d(256, 256, kernel_size=1, stride=1, padding=0)
        self.conv2D_12 = nn.Conv2d(256, 384, kernel_size=2, stride=2, padding=0)
        self.conv2D_13 = nn.Conv2d(384, 384, kernel_size=1, stride=1, padding=0)
        self.conv2D_14 = nn.Conv2d(384, 512, kernel_size=1, stride=2, padding=0)
        self.conv2D_15 = nn.Conv2d(512, 512, kernel_size=1, stride=2, padding=0)

        self.reshape_conv3D = nn.Flatten()
        self.lstm_1 = nn.LSTM(512, 128, batch_first=True)
        self.dropout_1 = nn.Dropout(0.3)
        self.lstm_2 = nn.LSTM(128, 32, batch_first=True)
        self.dropout_2 = nn.Dropout(0.3)
        self.lstm_3 = nn.LSTM(32, 1, batch_first=True)
        self.reshape_lstm = nn.Flatten()
        self.dense = nn.Linear(1, 1)

    def forward(self, x):
        x = self.conv2D_1(x)
        x = self.conv2D_2(x)
        x = self.conv2D_3(x)
        x = self.conv2D_4(x)
        x = self.conv2D_5(x)
        x = self.conv2D_6(x)
        x = self.conv2D_7(x)
        x = self.conv2D_8(x)
        x = self.conv2D_9(x)
        x = self.conv2D_10(x)
        x = self.conv2D_11(x)
        x = self.conv2D_12(x)
        x = self.conv2D_13(x)
        x = self.conv2D_14(x)
        x = self.conv2D_15(x)

        x = self.reshape_conv3D(x)
        x, _ = self.lstm_1(x)
        x = self.dropout_1(x)
        x, _ = self.lstm_2(x)
        x = self.dropout_2(x)
        x, _ = self.lstm_3(x)
        x = self.reshape_lstm(x)
        x = self.dense(x)

        return x
</code></pre>
<p>I know that LSTM has a different input shape with respect to CNN output but it’s not clear how to adapt the whole thing.<br>
Thank you for the answer.</p>
How to release the cuda memory occupied by creating variables?

zhang_yuj: <p>The function code used is as follows：</p>
<pre><code class="lang-auto">def batch_prototype(feature,mask):  #return B*C*feature_size
    batch_pro = torch.zeros(mask.shape[0], mask.shape[1], feature.shape[1]).to('cuda')
    for i in range(mask.shape[1]):
        classmask = mask[:,i,:,:]
        proclass = masked_average_pooling(feature,classmask.unsqueeze(1))
        batch_pro[:,i,:] = proclass
    return batch_pro
</code></pre>
<p>This piece of code is used in decode:</p>
<pre><code class="lang-auto">class Decoder_idea(nn.Module):
    def __init__(self, params):
        super(Decoder_idea, self).__init__()
        self.params = params
        self.in_chns = self.params['in_chns']
        self.ft_chns = self.params['feature_chns']
        self.n_class = self.params['class_num']
        self.bilinear = self.params['bilinear']
        assert (len(self.ft_chns) == 5)

        self.up1 = UpBlock(
            self.ft_chns[4], self.ft_chns[3], self.ft_chns[3], dropout_p=0.0)
        self.up2 = UpBlock(
            self.ft_chns[3], self.ft_chns[2], self.ft_chns[2], dropout_p=0.0)
        self.up3 = UpBlock(
            self.ft_chns[2], self.ft_chns[1], self.ft_chns[1], dropout_p=0.0)
        self.up4 = UpBlock(
            self.ft_chns[1], self.ft_chns[0], self.ft_chns[0], dropout_p=0.0)

        self.out_conv = nn.Conv2d(self.ft_chns[0], self.n_class,
                                  kernel_size=3, padding=1)

    def forward(self, feature):
        x0 = feature[0]
        x1 = feature[1]
        x2 = feature[2]
        x3 = feature[3]
        x4 = feature[4]

        x = self.up1(x4, x3)
        up1_out=x
        x = self.up2(x, x2)
        up2_out=x
        x = self.up3(x, x1)
        up3_out=x
        x = self.up4(x, x0)
        up4_out=x
        # print(x.shape,'feature_shape')
        output = self.out_conv(x)
        mask = torch.softmax(output,dim=1)
        **batch_pro = batch_prototype(x,mask)**
        torch.cuda.empty_cache()
        intermediate_out = {
            'up1': up1_out,
            'up2': up2_out,
            'up3': up3_out,
            'up4': up4_out,
        }
        return output, intermediate_out, batch_pro
</code></pre>
<p>Now my problem is that every time I use the batch_prototype function to calculate batch_pro, the occupied cuda memory cannot be released. Therefore, as time goes by, I will be prompted to exceed the memory that can be allocated by cuda. How can I solve this problem?</p>
Torch jit script

Tirthankar_Banerjee: <p>I have fine-tuned a Huggingface (HF) ASR (xlsr_1b model). How do I convert to a jit.script model?<br>
My finetuned HF model is saved as a .pt file as below:</p>
<p>model_id = “models/xlsr1b_torch_kas”<br>
model = Wav2Vec2ForCTC.from_pretrained(model_id)<br>
model_pt=“models/xlsr1b_torch_kas.pt”<br>
chkpt={“model”: model.state_dict(),}<br>
torch.save(chkpt, model_pt)</p>
<p>Now I want to convert this .pt model to jit. I load the saved .pt model first.</p>
<p>model_id = “models/xlsr1b_torch_kas”<br>
model = Wav2Vec2ForCTC.from_pretrained(model_id)<br>
checkpoint = torch.load(model_pt, map_location=“cpu”)<br>
model.load_state_dict(checkpoint[“model”], strict=False)<br>
model.eval()<br>
model.to(“cpu”)<br>
model.eval()</p>
<p>Now I want to convert to jit:</p>
<p>model.<strong>class</strong>.forward = torch.jit.ignore(model.<strong>class</strong>.forward)<br>
model = torch.jit.script(model)</p>
<p>This is throwing error-</p>
<p>UnsupportedNodeError: function definitions aren’t supported:<br>
File “/root/.virtualenvs/torchenv/lib/python3.8/site-packages/transformers/models/wav2vec2/modeling_wav2vec2.py”, line 455<br>
if self._requires_grad and self.gradient_checkpointing and self.training:</p>
<pre><code>            def create_custom_forward(module):
            ~~~ &lt;--- HERE
            
                def custom_forward(*inputs):
</code></pre>
<p>I have tried using <span class="mention">@torch.jit.ignore</span>(drop=True) and <span class="mention">@torch.jit.unused</span> before<br>
def create_custom_forward(module) but has not helped.</p>
<p>Where is the issue here as I think I am trying to ignore that part which is not required in the compiled version for inference.</p>
<p>Thanks for any pointer to resolution here.</p>
Pytorch C++ API, Conv2d crash

A_ElKader: <p>when running the following code :</p>
<pre><code>            torch::Tensor kernel = torch::full({ 1, 32, 1, 2 }, 0, torch::kInt32);
	// Define the bias
	torch::Tensor bias = torch::tensor({ 0 }, torch::kInt32);
	// Define the input image
	torch::Tensor image = torch::full({ 32, 6, 7 }, 0, torch::kInt);

	/// Define the convolution operation
	//torch::nn::Conv2d conv(torch::nn::Conv2dOptions(32, 1, {1,2}).bias(false));
	//depth_conv = Conv2d(in_channels = 10, out_channels = 10, kernel_size = 3, groups = 10)
	torch::nn::Conv2d depth_conv = torch::nn::Conv2d(
		torch::nn::Conv2dOptions(32, 1, {1,2}).groups(1).bias(false)
	);
	 
	

	// Set the filter for the convolution operation
	depth_conv-&gt;weight = kernel;

	// Set the bias for the convolution operation
	depth_conv-&gt;bias = bias;
	std::cout &lt;&lt; "Original Data Tensor:\n" &lt;&lt; image &lt;&lt; std::endl;
	std::cout &lt;&lt; "Original Wght Tensor:\n" &lt;&lt; depth_conv-&gt;weight &lt;&lt; std::endl;
	

	// Apply the convolution operation
	torch::Tensor output = depth_conv-&gt;forward(image);
	std::cout &lt;&lt; "Output:\n" &lt;&lt; output.dtype() &lt;&lt; std::endl;

	// Print the output shape and values
	std::cout &lt;&lt; "Output Shape: " &lt;&lt; output.sizes() &lt;&lt; std::endl;
	std::cout &lt;&lt; "Output:\n" &lt;&lt; output.dtype() &lt;&lt; std::endl;
</code></pre>
<p>the convolution is done and every thing is ok, when out of scope the code crash, any idea why is that ?</p>
Optimizing Neural Network Inputs: Ensuring Shared Gradients for Common Parameters Across Coordinates

NeuralNinjaCoder: <p>Hello,<br>
I try to optimize the input to a network where I aim to get the same parameters over a whole dimension of a tensor.</p>
<p>The input consists of process parameter and coordinates (input[coordinates,parameterfeatures]) and the output of the network gives back a quality for a object (output[coordinates,quality]).</p>
<p>Now I want to know the process parameters for the best quality of the object. But the process parameters should be for all coordinates be the same. Right now I get different process parameters for each coordinate.</p>
<p>Is there a way to “share” the gradient over a whole dimension of a tensor, so the process parameter get all updated the same over the coordinates?</p>
<p>My code looks like this:</p>
<pre><code class="lang-auto">input.requires_grad_(True)  # input.shape [1000,15]
optimizer = torch.optim.SGD([input], lr=0.1)

for iteration in range(num_iterations):
    
    result = net(input)    
    loss = result

    optimizer.zero_grad()
    loss.mean().backward()  
    
    input.grad[:,3:]=0 #only optimize the first three parameters
    optimizer.step()
</code></pre>
<p>Thanks!</p>

NeuralNinjaCoder: <p>I found the solution to what I wanted to do. To get the same values over the first dimension I created views with .expand(), now the parameter change is for all values the same.</p>
<pre><code class="lang-auto">optimizer = torch.optim.Adam([var], lr=0.1)
input = torch.cat((var.unsqueeze(0).expand(var_second.shape[0],-1),var_second),
                dim=1)

for iteration in range(num_iterations):
    optimizer.zero_grad()     
    
    loss = net(input)
    loss.mean().backward()

    optimizer.step()
    input = torch.cat((var.unsqueeze(0).expand(var_second.shape[0],-1),var_second),
                dim=1)
</code></pre>
Backprop Through sparse_dense_matmul

elanmart: <p>Hi,<br>
I’ve seen some recent progress on SparseTensors, and I have some questions:</p>
<ul>
<li>Do You have a roadmap for the whole sparse module? I’m curious if you’re planning to use <code>cuSPARSE</code> and if there is any way to contribute to this module.</li>
<li>I’d need a backward pass for <code>sparse_mat @ dense_mat</code>. What are my options? Can I write a Function using the available <code>spmm</code> for forward, and add some quick-and-dirty cython code to do the backward? This would mean returning a sparse grad, is this supported?</li>
</ul>
<p>Thank You for your time, I’m really enjoying <code>PyTorch</code> so far.</p>

ebetica: <p>See this thread: <a href="https://discuss.pytorch.org/t/exploiting-sparsity-in-batch-operations/1079/3">Exploiting sparsity in batch operations?</a></p>
<p>torch.smm impleemnts sparse * dense -&gt; sparse, and it assumes your sparsity is strong enough that it’ll help (you’ll need a <em>really</em> sparse tensor in the forward op). Sparse gradients are supported, and are implemented in the Embedding module.</p>

martinraison: <p>Hi,</p>
<ul>
<li>for now sparse tensor support basically evolves with the needs of our respective projects. Basic GPU support is being worked on - it will rely on <code>cuSPARSE</code> for some operations.</li>
<li>returning sparse gradients from <code>backward</code> should work</li>
<li>in addition to the thread <a class="mention" href="/u/ebetica">@ebetica</a> pointed to, since pytorch supports hybrid tensors (i.e. tensors with both sparse and dense dimensions), we may add a sparse * dense -&gt; hybrid function in the future (where “hybrid” here means one sparse dimension and one dense dimension). That would probably be more efficient in this case.</li>
</ul>

elanmart: <p><a class="mention" href="/u/martinraison">@martinraison</a> <a class="mention" href="/u/ebetica">@ebetica</a> thanks for the answers.</p>
<p>I’ve just noticed that there is also an implementation of <code>SparseLinear</code> <a href="https://github.com/pytorch/pytorch/blob/master/torch/lib/THNN/generic/SparseLinear.c" rel="nofollow noopener">here</a>. Can I use those<br>
<code>c</code> functions to create a <code>Function</code> which will simply call <code>self._backend.SparseLinear_accGradParameters</code>, just like <code>Embedding</code> <code>Function</code> does (<a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/_functions/thnn/sparse.py#L90" rel="nofollow noopener">link</a>)? Or there is some catch here?</p>
<p>I know that You guys probably have <code>SparseLinear</code> module on your roadmap, but I wanted to play around with it during the weekend, hence the question.</p>

Jiuhnny: <p>I hope to write the bp for my spmm kernel as well. ge-spmm presents some of it, but I doubt the correctness. <a href="https://github.com/hgyhungry/ge-spmm" class="inline-onebox" rel="noopener nofollow ugc">GitHub - hgyhungry/ge-spmm</a></p>
Negative training loss

promach: <p>I tried to implement <a href="https://gist.github.com/buttercutter/b3331ca1fd9e2f5871b0eded6b758f39#file-mamba-py" rel="noopener nofollow ugc">Mamba: Linear-Time Sequence Modeling with Selective State Spaces</a>, but I got some <a href="https://gist.github.com/buttercutter/b3331ca1fd9e2f5871b0eded6b758f39#file-negative_training_loss-log" rel="noopener nofollow ugc">negative training loss</a> , any ideas how to get around it ?</p>
LSTM for time series and non-time series data

Teslim: <p>I have a task where I wish to ML to model an output such that given a new set of time-independent parameters, I can predict this output over a given period of time.</p>
<p>For example, y = f(t, a, b) where a, b and c are time-independent variables</p>
<p>Let’s assume you have the following data:</p>
<p>a = 300, b = 30<br>
y(t) = [1, 2, 3, 4, 5, 6]</p>
<p>a = 330, b = 36<br>
y(t) = [1, 3, 4, 5, 7, 8]</p>
<p>In formulating my solution, I combined y(t) to form:<br>
data = [1, 2, 3, 4, 5, 6, 1, 3, 4, 5, 7, 8]<br>
a = [300, 330]<br>
b = [30, 36]</p>
<p>Since LSTM is the most commonly used for time-series data, I created a sequence of length 3 as follows:</p>
<p>X_time = [[1, 2, 3], [2, 3, 4], [3, 4, 5], [1, 3, 4], [3, 4, 5], [4, 5, 7]]</p>
<p>X_non_time = [[300, 30], [300, 30], [300, 30], [330, 36], [330, 36], [330, 36]]</p>
<p>y = [[4], [5], [6], [5], [7], [8]]</p>
<pre><code class="lang-auto">class LSTM(nn.Module):
    def __init__(self, lstm_input_size, fc_input_size, hidden_size, num_layers, output_size):
        super(LSTM, self).__init__()
        self.lstm = nn.LSTM(lstm_input_size, hidden_size, num_layers, batch_first=True)
        self.fc = nn.Linear(hidden_size+fc_input_size, hidden_size)
        self.fc2 = nn.Linear(hidden_size, output_size)

    def forward(self, xtime, xnon_time):
        out, _ = self.lstm(xtime)
        new_input = torch.cat((out[:, -1, :], xnon_time), dim=1)
        x = F.relu(self.fc(new_input))  # Take the output from the last time step
        out = self.fc2(x)
        return out
</code></pre>
<p>For the actual problem, i created a problem with the following information:</p>
<p>Shape of time dependent sequence (xtime): torch.Size([1764, 3, 1])<br>
Shape of time independent sequence (xnon_time): torch.Size([1764, 8])<br>
Shape of labels (output): torch.Size([1764, 1])</p>
<pre><code class="lang-auto">lstm_input_size = 1  # Number of time-dependent  features
fc_input_size = 8  # Number of time-independent  features
output_size = 1 #  Number of output

hidden_size = 50
num_layers = 5
output_size = 1
lstm_model = LSTM(lstm_input_size, fc_input_size, hidden_size, num_layers, output_size).to(device)
</code></pre>
<p>Please advise on whether this is right or not.</p>
Loaded model has higher loss in comparison with saved one

msdkarimi: <p>I’ve developed a basic transformer model for translation task. Model itself works well, train and validation loss decreases and BLEU score increases as epochs go on. My problem is related to loading the model itself, when I load the model after one epoch, loaded model’s loss is higher than version of model before saving. For instance, when epoch 20 has ended I got loss = 2.03, if I save the model and load it to start from epoch 21, loaded model has loss = 2.5.</p>
<p>The interesting thing is that if I set a seed when I load my dataset, it works well when I load the model, but model performance during the validation is way lower when I do not set seed.</p>
<p>here are my functions to save and load the model:</p>
<pre><code>def save_checkpoint(self, checkpoint_path, loss, epoch, type='regular'):
    torch.save({
        'model': self.model.state_dict(),
        'optimizer': self.optimizer.state_dict(),
        'scheduler': self.scheduler.state_dict(),
        'loss': loss,
        'epoch': epoch
    }, checkpoint_path)

 def load_checkpoint(self, checkpoint_path):
        if os.path.exists(checkpoint_path):
            checkpoint = torch.load(checkpoint_path)
            self.model.load_state_dict(checkpoint['model'])
            self.optimizer.load_state_dict(checkpoint['optimizer'])
            self.scheduler.load_state_dict(checkpoint['scheduler'])
            return checkpoint['loss'], checkpoint['epoch'] + 1
</code></pre>
<p>Many thanks in advance for your time and efforts.</p>

ptrblck: <p>Compare outputs of your model in <code>eval</code> mode using a static input tensor before and after reloading the <code>state_dict</code>. If these outputs are equal the <code>state_dict</code> loading works fine and the issue comes from another part of your training script.</p>

msdkarimi: <p>I’ve tested your idea but the inconsistency is still there. Do you any suggestion regarding data loader?</p>

ptrblck: <p>If the outputs in <code>eval</code> mode for the same static input are different, the data loading pipeline is not related and you would need to look into the model.<br>
You could print the intermediates directly in the <code>forward</code> of the model or you could use forward hooks to check them. Computing <code>tensor.abs().sum()</code> might be a good starter as it should indicate which layer causes the different results.</p>
Does yolov5 support deployment on ARM platforms? Currently

qiyuefun: <p>Great god, I need your help. The situation is as follows. Currently, YOLOv5 is being trained on the Ubuntu system Torch 2.0.1 Torchvision 0.15.2, and CUDA 117 version to convert the model best. pt to YOLOv5 Torchscript. I want to deploy it to the ARM platform. Is this idea feasible? Is it supported? If so, where can I download and compile using Libtorch</p>

ptrblck: <p>You could either build PyTorch from source on your ARM platform or use the <a href="https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch">NGC containers</a> which provide PyTorch builds for ARM with CUDA support.</p>
Custom RNN/GRU implementation in C++

a-carson: <p>I am interested in creating my own custom GRU implementation (for example changing the tanh activation to relu), but with the same training efficiency of the torch.nn.GRU class.</p>
<p>I believe I need to implement it as a C++ extension in order to avoid a time-stepping for-loop in Python.</p>
<p>Can anyone point me in the direction of where to start? Ideally, I would base it off the existing torch cpp GRU implementation, but I am struggling to find that in the source code.</p>
<p>Thanks in advance!</p>
Not able to install detectron2 and deepspeed libraries

Arun_Sandy: <p>While trying to install detectron2 and deepspeed I’m getting below errors :<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/b/bb93aa41c37e3645d35aeba53ad3636424a19bdd.png" data-download-href="https://discuss.pytorch.org/uploads/default/bb93aa41c37e3645d35aeba53ad3636424a19bdd" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/b/bb93aa41c37e3645d35aeba53ad3636424a19bdd.png" alt="image" data-base62-sha1="qLnEN2sh8saASK83KXfLbrLJSeV" width="596" height="500" data-dominant-color="121212"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1097×920 41.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I have followed the steps to install detectron2 from here (<a href="https://detectron2.readthedocs.io/en/latest/tutorials/install.html" class="inline-onebox" rel="noopener nofollow ugc">Installation — detectron2 0.6 documentation</a>),<br>
Although the above error says that I do not have a “torch” module, I have already installed it and it still doesn’t work.</p>
<p>This is the same case with installing deepspeed:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/6/26af14e7d0bf4baadea1b440de2e43c4afca99d2.png" data-download-href="https://discuss.pytorch.org/uploads/default/26af14e7d0bf4baadea1b440de2e43c4afca99d2" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/6/26af14e7d0bf4baadea1b440de2e43c4afca99d2.png" alt="image" data-base62-sha1="5wdfMIj1F4DTngDUjKekMXi3pfA" width="559" height="500" data-dominant-color="121211"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1109×991 44.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>My python version is 3.9 and torch version is :<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/f/a/faf924d87bde707d86fe62cc6f33366f059a1f81.png" alt="image" data-base62-sha1="zOd9FJaFALiOxw43dkt5qkP6O53" width="279" height="82"></p>
<p>Please let me know what might me the issue here.</p>
TorchVision cannot be used

XZRRRRRR: <p>I want to use some of the registered ops in the TorchVision API now, but when I run the test, I can’t find those OPSs with torch::jit output, e.g. torchvision::roi_align or torchvision::nms; I made sure that torchvision and libtorch versions correspond (torchvision 0.16.1 libtorch2.1.2 + cu121), cuda is 12.1 and I’m sure with_cuda compilation is turned on. Does anyone can help me?</p>
<p><span class="hashtag-raw">#include</span> “torch/script.h”<br>
<span class="hashtag-raw">#include</span> “torch/torch.h”<br>
<span class="hashtag-raw">#include</span> &lt;torchvision/csrc/ops/roi_align.h&gt;<br>
auto&amp; ops = torch::jit::getAllOperatorsFor(torch::jit::Symbol::fromQualString(“torchvision::roi_align”));<br>
ASSERT_EQ(ops.size(), 1);<br>
This result is<br>
Failure<br>
Expected equality of these values:<br>
ops.size()<br>
Which is: 0<br>
1</p>

XZRRRRRR: <p>My problem has been solved by adding "-Wl,–no-as-needed " in target_link_libraries in CmakeLists of my projection.</p>
nn.LogSoftmax(dim=1)

Pakpoom_Singkorapoom: <p>Hello</p>
<p>I am trying this exercise from pytorch tutorial.<br>
<a href="https://pytorch.org/tutorials/intermediate/char_rnn_classification_tutorial.html#creating-the-network" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/intermediate/char_rnn_classification_tutorial.html#creating-the-network</a></p>
<p>I have a question about <code>self.softmax = nn.LogSoftmax(dim=1)</code>.<br>
Could someone explain to me why dim=1???</p>
<p>Thanks</p>

ptrblck: <p>PyTorch layers accept batched inputs where often the dimensions represent <code>[batch_size, features, ...]</code>. <code>dim1</code> is therefore used to represent the number of classes in a classification use case. Applying a <code>log_softmax</code> on this dimension transforms logits to log probabilities and normalizes them over the class dimension.</p>

Pakpoom_Singkorapoom: <p>thank a lot for your answer, sir.</p>
If I use torch.compile to compile the whole graph，in the my own compiler, how to manage the memory in my own compiler?

mollon_mollon: <p>if I use torch.compile to compile the whole graph，in the my own compiler ，in forward stage，<br>
1.if I enable memory reuse in the forward pass，how the backwards get the activation to calcute the gradient？has there some example in pytorch？<br>
2.if i disable memory reuse，if i enable some op fusion，A op+B op fuison to one op, so the A output value is in sram or local memory or gloabal memory , torch can’t get the activation, in backwards how to calcute the gradient?has there some example in pytorch？<br>
3.how to manage the memory in my own compiler to use torch.compile to speed up the training?<br>
4.how the backwards (autogradient) get the activation from my own compiler ? the memory of every op in the graph must be in ddr?</p>
How to implement the exactly same softmax as F.softmax by pytorch

Junwu_Weng: <p>Hi everyone,</p>
<p>Recently I need to re-implement the softmax function to design my own softmax. I refer the codes on the Github and implemented one as shown below.</p>
<pre><code class="lang-auto">def own_softmax(self, x)
    
    maxes = torch.max(x, 1, keepdim=True)[0]
    x_exp = torch.exp(x-maxes)
    x_exp_sum = torch.sum(x_exp, 1, keepdim=True)

    return x_exp/x_exp_sum
</code></pre>
<p>However, after implementation I found that the results are not as good as the original one (F.softmax). So I am here to ask what is the difference between my implementation and the built-in function.</p>
<p>Thank you so much!</p>

Junwu_Weng: <p>Anyone can help here?</p>

Junwu_Weng: <p>I have tried some other implementation like the following one,</p>
<pre><code class="lang-auto">def own_softmax(self, x)
    
    means = torch.mean(x, 1, keepdim=True)[0]
    x_exp = torch.exp(x-means)
    x_exp_sum = torch.sum(x_exp, 1, keepdim=True)

    return x_exp/x_exp_sum
</code></pre>
<p>and found that this implementation can achieve better accuracy. However, it is still not as good as the F.softmax. Anyone can help?</p>

Junwu_Weng: <p><img src="https://discuss.pytorch.org/images/emoji/apple/sob.png?v=9" title=":sob:" class="emoji" alt=":sob:"> Anyone can help?</p>

Junwu_Weng: <p>so sad, so sad … <img src="https://discuss.pytorch.org/images/emoji/apple/pleading_face.png?v=9" title=":pleading_face:" class="emoji" alt=":pleading_face:"></p>

ptrblck: <p>Your custom function returns the same output as <code>F.softmax</code>:</p>
<pre><code class="lang-python">x = torch.randn(5, 10)
output = F.softmax(x, 1)

maxes = torch.max(x, 1, keepdim=True)[0]
x_exp = torch.exp(x-maxes)
x_exp_sum = torch.sum(x_exp, 1, keepdim=True)
output_custom = x_exp/x_exp_sum

print(torch.allclose(output, output_custom))
&gt; True
print(torch.sum(torch.abs(output-output_custom)))
&gt; tensor(2.3108e-7)
</code></pre>

nutszebra: <p>The output from your <code>own_softmax</code> is slightly different from <code>torch.nn.functional.softmax</code> .<br>
This may be the reason why your <code>own_softmax</code> degrades the performance.</p>
<pre><code class="lang-auto">x = torch.randn(2,10)
h_own = own_softmax(x)
h = torch.nn.functional.softmax(x, 1)
print(h - h_own)
</code></pre>

Junwu_Weng: <p>Thank you so much. It seems that different centralization method for the network output score influence the softmax output a lot. I have tested the centralization using max value and mean value, and their output are quite different. I am wondering whether the mean one is more stable？</p>

Junwu_Weng: <p>Thank you so much for your reply.</p>

ptrblck: <p>If I swap <code>torch.max</code> for <code>torch.mean</code>, I get approx. the same accuracy.<br>
Could you post the code you’ve used to check the behavior?</p>

omri123: <p>Hi Junwu!<br>
For some unique optimization I need to re-implement softmax.<br>
I am afraid to see accuracy degradation. Do you see performance degradation in training or do you use your softmax for inference only? I think training is more sensitive to numerical problems.</p>
NaN Gradient/Loss for Quadro RTX 8000 only

Minennick: <p>Hi all!</p>
<p>I am currently training different diffusion models by using the [Imagen-pytorch] repository from Phil Wang, which works super fine when trained on a Nvidia A6000 GPU of a colleague. When trained on my Quadro RTX 8000 I do get nan Losses caused by nan gradients.</p>
<p><strong>Setups I experimented with:</strong><br>
GPU: A6000<br>
Nvidia Driver Version: 525.125.06<br>
Cuda Version: 12.0<br>
torch: 1.13.1<br>
 → Everything works fine</p>
<p>GPU:  Quadro RTX 8000<br>
Nvidia Driver Version: 525.125.06<br>
Cuda Version: 12.0<br>
torch: 1.13.1<br>
 → NaN Losses/ Gradients</p>
<p>GPU:  Quadro RTX 8000<br>
Nvidia Driver Version: 470.223.02<br>
Cuda Version: 11.4<br>
torch: 1.12.1<br>
 → NaN Losses/ Gradients</p>
<p><strong>Other things I tried:</strong><br>
As stated in the [pytorch docs] I suspected to be something wrong with the AMP and GradScaler which I disabled both in the codebase. Still the NaN losses persist.</p>
<p>Using <code>torch.autograd.set_detect_anomaly(True)</code> the following error is thrown:</p>
<pre><code class="lang-auto">RuntimeError: Function 'MmBackward0' returned nan values in its 0th output.
</code></pre>
<p>At this point I am not sure why this is hardware specific and only the case for the Quadro RTX 8000 gpu. Maybe someone can shed some light <img src="https://discuss.pytorch.org/images/emoji/apple/sun_with_face.png?v=12" title=":sun_with_face:" class="emoji" alt=":sun_with_face:" loading="lazy" width="20" height="20"></p>
<p>(<a href="https://pytorch.org/tutorials/recipes/recipes/amp_recipe.html" class="inline-onebox" rel="noopener nofollow ugc">Automatic Mixed Precision — PyTorch Tutorials 2.2.0+cu121 documentation</a>)<br>
(<a href="https://github.com/lucidrains/imagen-pytorch" class="inline-onebox" rel="noopener nofollow ugc">GitHub - lucidrains/imagen-pytorch: Implementation of Imagen, Google's Text-to-Image Neural Network, in Pytorch</a>)</p>

ptrblck: <p>All used PyTorch binaries are old by now, so update to the latest stable or nightly release and rerun the tests.</p>

Minennick: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
I updated the complete setup to have:</p>
<p>GPU: Quadro RTX8000<br>
Nvidia Driver Version: 545.29.06<br>
Cuda Version: 12.3<br>
torch: 2.1.2</p>
<p>And now it does work. Never found out what the exact error was though. Still thanks for your help <img src="https://discuss.pytorch.org/images/emoji/apple/blush.png?v=12" title=":blush:" class="emoji" alt=":blush:" loading="lazy" width="20" height="20"></p>
Dynamic sequence length and recurrent layers

MarcoMeter: <p>Hello everyone,</p>
<p>I’m training a deep reinforcement learning agent that leverages GRU. In my training setup the sequence length is set to the length of the longest episode.</p>
<p>Now that I’m trying out torch.compile(), every epoch compiles a new backwards graph. After 40 epochs, 266 backwards graphs were compiled.</p>
<p>Is my assumption correct, that torch.compile (in the context of recurrent layers) should only be used if the sequence length is fixed?</p>

marksaroufim: <p>There’s an argument called <code>torch.compile(..., dynamic=True)</code> but you should try it out on master and expect some bugs</p>

MarcoMeter: <p>Thanks for the heads up! Setting the dynamic argument is not working on the current release. I’ll wait for newer versions.</p>

MarcoMeter: <p>I revisited this issue today and tried the recent nightly version of PyTorch. The rercurrent model successfully compiles under <code>dynamic=True</code> and <code>mode=max-autotune</code>.</p>
<p>Before compilation an A100 needed <code>6.84s</code> on average per PPO Update. After compilation, there is no notable speed up <code>6.87s</code>.</p>
<p>I conducted a training run without GRU Cell. Without compiling the model, I measured <code>5.56s</code> and with compilation <code>5.25s</code>.</p>
<p>My code also features TransformerXL training. If I compile this model, the forward pass outputs infinity. To debug this issue, I compiled every single submodule on its own and did not run into this exception. So far I did not measure any performance gains.</p>

marioo: <p>Please, does anyone have any updates on this? I am using PyTorch 2.1.0 and I am facing the same issue.</p>