akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to define the video inputs for pytorch with vulkan backend?

wangbokang97: <p>I have compiled the pytorch with vulkan backend in Android and successfully transferred a video action recoginzer model “Video Swin transformer” into .pt with vulkan backend.<br>
When I define the input like:</p>
<pre><code class="lang-auto">FloatBuffer buffer = Tensor.allocateFloatBuffer(1 * 3 * 224 * 224);
Tensor inputTensor = Tensor.fromBlob(buffer, new long[]{1, 3, 224, 224});
TensorImageUtils.TORCHVISION_NORM_STD_RGB, MemoryFormat.CHANNELS_LAST);
assert module != null;
final Tensor outputTensor = module.forward(IValue.from(inputTensor)).toTensor();
</code></pre>
<p>I got the error like:</p>
<pre><code class="lang-auto">com.facebook.jni.CppException: Dimension out of range (expected to be in range of [-4, 3], but got 4)
</code></pre>
<p>I also tried the input like:</p>
<pre><code class="lang-auto">FloatBuffer buffer = Tensor.allocateFloatBuffer(1 * 3 * 10 * 224 * 224);
Tensor inputTensor = Tensor.fromBlob(buffer, new long[]{1, 3, 10, 224, 224});
TensorImageUtils.TORCHVISION_NORM_STD_RGB, MemoryFormat.CHANNELS_LAST);
assert module != null;
final Tensor outputTensor = module.forward(IValue.from(inputTensor)).toTensor();
</code></pre>
<p>And the error is:</p>
<blockquote>
<p>Only Tensors with 1 &lt;= dim &lt;= 4 can be represented as a Vulkan Image!</p>
</blockquote>
<p>I really don’t what to do now. Could anyone do me a favor?</p>

wangbokang97: <p>I use pytorch1.13 with vulkanSDK==1.3.204.1</p>

wangbokang97: <p>I have figured out the problem here. And the right definition of video input should be the same as PC.<br>
However, as vulkan will transfer the input tensor as Vulkan Image, and Vulkan Image only supports 3d images, the video tensor with shape [batch, channels, frames, height, width] is not supported.</p>
Error converting .pth to .pt, Could not export Python function call 'Scatter'?

Dhirendra_Pratap: <p>i am trying to export a .pth model to .pt torchscript model, i got error while doing so.<br>
Could not export Python function call ‘Scatter’. Remove calls to Python functions before export. Did you forget to add <span class="mention">@script</span> or <span class="mention">@script_method</span> annotation? If this is a nn.ModuleList</p>
<pre><code class="lang-auto">            print("=&gt; loading checkpoint '{}'".format(args.evaluate))
            checkpoint = torch.load(args.evaluate)
            best_acc = checkpoint['best_acc']
            best_acc = best_acc.to()
            print(f'best_acc:{best_acc}')
            model.load_state_dict(checkpoint['state_dict'])
            print("=&gt; loaded checkpoint '{}' (epoch {})".format(args.evaluate, checkpoint['epoch']))
            rand_inp = torch.rand(1,3,224,224)
            # rand_inp = rand_inp.cuda()

            model.eval()
            model_traced = torch.jit.trace(model, rand_inp)
            model_traced.save("rab_db_poster_V2.pt")
            model_traced.eval()
</code></pre>
"torch.stack" is faster than preallocating?

YEp_d: <p>Using <code>torch.stack</code> on a list of tensors appear to be faster than preallocating the tensor and then writing into it.</p>
<pre><code class="lang-auto">import torch


def preallocate(*shape, seed=0):
    torch.manual_seed(seed)
    x = torch.empty(shape)
    for i in range(shape[0]):
        torch.rand(shape[1:], out=x[i])

    return x


def stacker(*shape, seed=0):
    torch.manual_seed(seed)
    x = []
    for _ in range(shape[0]):
        x.append(torch.rand(shape[1:]))

    return torch.stack(x)


if __name__ == "__main__":
    import timeit
    import random

    seed=random.randint(-0x8000_0000_0000_0000, 0xffff_ffff_ffff_ffff)
    print(timeit.timeit(f"preallocate(100,1000, seed={seed})",
          "from __main__ import preallocate", number=10000))
    print(timeit.timeit(f"stacker(100,1000, seed={seed})",
          "from __main__ import stacker", number=10000))
</code></pre>
<p>The function which uses <code>torch.stack</code> is 4–5 s faster on my machine after a few runs.</p>
<p>This is quite unexpected since I assume calling <code>torch.stack</code> would copy all the tensors into one big tensor, whereas preallocation would not have such an overhead. What would even make preallocation slower than <code>torch.stack</code>?</p>
Error with LSTM Layer on MPS

boguh: <p>Hi,</p>
<p>I am trying to implement an LSTM network and set it so calculations are on the GPU with .to(device) but I keep getting the same error:</p>
<blockquote>
<p>NotImplementedError: The operator ‘aten::mkldnn_rnn_layer’ is not currently implemented for the MPS device. If you want this op to be added in priority during the prototype phase of this feature, please comment on <a href="https://github.com/pytorch/pytorch/issues/77764" class="inline-onebox" rel="noopener nofollow ugc">General MPS op coverage tracking issue · Issue #77764 · pytorch/pytorch · GitHub</a>. As a temporary fix, you can set the environment variable <code>PYTORCH_ENABLE_MPS_FALLBACK=1</code> to use the CPU as a fallback for this op. WARNING: this will be slower than running natively on MPS.</p>
</blockquote>
<p>I thought that the ability to use MPS with an AMD GPU was already implemented? Does anyone know the issue?</p>
<p>Here is the code that produced the error:</p>
<pre><code class="lang-auto">x = torch.ones(1,1,10)

device = torch.device('mps')
x.to(device)

lstm_layer = torch.nn.LSTM(
    input_size=10,
    hidden_size=10,
    num_layers=1,
    batch_first=True
)

lstm_layer.to(device)

out, (h, c) = lstm_layer(x)
</code></pre>
<p><strong>Versions:</strong></p>
<ul>
<li>MacOS Ventura 13.4.1, Intel processor</li>
<li>AMD Radeon Pro 5300M</li>
<li>pytorch 2.0.1, installed using anaconda</li>
</ul>
Questions about RAFT API

shiba24: <p>Hi, I have been confused about the API of the torchvision implementation of RAFT (thank you for the convenient method, btw) and have a couple of questions regarding that.</p>
<ul>
<li>API: <code>http://pytorch.org/vision/main/models/raft.html</code></li>
<li>The example usage: <a href="https://pytorch.org/vision/stable/auto_examples/plot_optical_flow.html" class="inline-onebox" rel="noopener nofollow ugc">Optical Flow: Predicting movement with the RAFT model — Torchvision 0.15 documentation</a></li>
</ul>
<p>My first and biggest question is about the reference image of img1 and img2 in this example: <strong>which is the reference frame at which the predicted flow is?</strong> (attention the definition of “reference” might be different.)<br>
In other words, could you provide the documentation about the actual direction of the flow (img1 → img2 or img2 → img1; I <em>guess</em> the latter is the correct one).</p>
<p>Roughly speaking we get <code>flow = predict_raft(img1, img2)</code>. I conducted a sanity check by warping one of the input image with the flow and checking if the warped_image looks similar to the other input image. My warp function is as attached at the end of this post (in case my warp implementation is wrong).</p>
<p>And when I check both direction, comparing img1 with the <code>warped_img = warp(img2, flow)</code> looks more reasonable than comparing img2 with <code>warped_img = warp(img1, flow)</code>.</p>
<p>I found that some people have been confused too (e.g., <a href="https://github.com/princeton-vl/RAFT/issues/158#issuecomment-1512380857" class="inline-onebox" rel="noopener nofollow ugc">How to use it to calculate the optical flow · Issue #158 · princeton-vl/RAFT · GitHub</a> please use the browser translation).</p>
<p>And a related question is can you provide the color wheel (the legend of the optical flow) of <code>torchvision.utils.flow_to_image</code>?</p>
<pre><code class="lang-auto">def warp(x, flo):
    """
    warp an image tensor according to the optical flow
    x: [B, C, H, W] (image)
    flo: [B, 2, H, W] flow
    """
    B, C, H, W = x.size()
    # mesh grid
    xx = torch.arange(0, W).view(1, -1).repeat(H, 1)
    yy = torch.arange(0, H).view(-1, 1).repeat(1, W)
    xx = xx.view(1, 1, H, W).repeat(B, 1, 1, 1)
    yy = yy.view(1, 1, H, W).repeat(B, 1, 1, 1)
    grid = torch.cat((xx, yy), 1).float()
    
    if x.is_cuda:
        grid = grid.cuda()
    vgrid = grid + flo
    
    # scale grid to [-1,1] for grid_sample
    vgrid[:, 0, :, :] = 2.0 * vgrid[:, 0, :, :].clone() / max(W - 1, 1) - 1.0
    vgrid[:, 1, :, :] = 2.0 * vgrid[:, 1, :, :].clone() / max(H - 1, 1) - 1.0
    vgrid = vgrid.permute(0, 2, 3, 1)
    output = F.grid_sample(x, vgrid, align_corners=True)
    return output
</code></pre>
<p>Thanks!</p>
Accessing weights in custom layer

SantaTitular: <p>Hi,</p>
<p>I wrote a custom linear layer that allows the use of complex value-data as such:</p>
<details>
<summary>
Custom layer</summary>
<pre><code class="lang-auto">def apply_complex(fr, fi, input, dtype = torch.complex128):
    return (fr(input.real)-fi(input.imag)).type(dtype) \
                + 1j*(fr(input.imag)+fi(input.real)).type(dtype)
                
class ComplexLinear(nn.Module):

    def __init__(self, in_features, out_features):
       super(ComplexLinear, self).__init__()
       self.fc_r = nn.Linear(in_features, out_features, dtype=torch.float64)
       self.fc_i = nn.Linear(in_features, out_features, dtype=torch.float64)

    def forward(self, input):
       return apply_complex(self.fc_r, self.fc_i, input)```
</code></pre>
</details>
<p>However, at some point, I want to access the weights of the custom linear layer (<code>head</code>) using the code: <code>wm = head.weight.data</code> (it works for nn.Linear case). Is the equivalent just doing <code>wm = head.fc_r.weight.data + 1j * head.fc_i.weight.data</code> or I should define a weight parameter in the custom layer that does the same?</p>
"Boxed" functions

trusira: <p>Hi,<br>
I have been looking at graph lowering passes in Pytorch 2.x and came across “boxed” functions (through <code>make_boxed_func</code>) in AOTAutograd.<br>
Can someone explain boxed functions and what is their role in the compilation workflow?</p>
<p>Thanks!</p>

ptrblck: <p>Take a look at <a href="http://blog.ezyang.com/2020/09/lets-talk-about-the-pytorch-dispatcher/">this blog post</a> by <a class="mention" href="/u/ezyang">@ezyang</a> where he explains boxed and unboxed functions.</p>
Precision issue after converting to CoreML

bcxiao: <p>Hello, I am trying to convert my Pytorch model to CoreML format. However, the outputs of pytorch model and the converted CoreML model do not match well.</p>
<p>Here is the code of a minimal example:</p>
<pre><code class="lang-python">import numpy as np 
import torch
import torch.nn as nn

class Sample_model(torch.nn.Module) :
    def __init__(self, n_classes):
        super().__init__()
        self.fc1 = nn.Linear(2, n_classes)
        
    def forward(self, x):
        return self.fc1(x)
    

# initialize a model with 3 classes
model_fixed =  Sample_model(3)

# create a random input tensor of size (batch_size=2, feature_size=2) as an example
features = torch.tensor(np.array([[1, 2], [3, 4]], dtype=np.float32))
# use the first example as an example input to trace
example_input_trace = features[:1]
traced_model = torch.jit.trace(model_fixed, example_input_trace)


import coremltools as ct

model_ct = ct.convert(
    traced_model,
    convert_to="mlprogram",
    inputs=[ct.TensorType(shape=example_input_trace.shape, dtype=np.float32)]
 )

# use the second example as an example input to predict
test_input = features[1:]

coreml_pred = model_ct.predict({'x': test_input.numpy()})['linear_0']
original_torch_pred = model_fixed(test_input).detach().numpy()


print(original_torch_pred - coreml_pred)  
# output array I got for one run is ([[-0.00027072, -0.00030965, -0.00077653]], dtype=float32), which is not close to zero

np.allclose(original_torch_pred, coreml_pred)  
# return False, which is supposed to be True
</code></pre>
<p>My Pytorch version: 1.13.1<br>
coremltools version: 7.0<br>
My device: Apple M1 macbook<br>
OS: macOS Monterey 12.5.1</p>
<p>Is it normal to see the discrepancy of that scale even for this toy model? Or did I miss something in the code?</p>
<p>Really appreciate it if someone can help.</p>

bcxiao: <p>Got 1 solution. Set the precision when using coremltools.convert function:</p>
<pre><code class="lang-auto">model_ct = ct.convert(
    traced_model,
    convert_to="mlprogram",
    compute_precision=ct.precision.FLOAT32,
    inputs=[ct.TensorType(shape=example_input_trace.shape, dtype=np.float32)]
 )
</code></pre>
Issues with Bart model finetuning

dav_kor: <p>I am trying to fine-tune Bart for a classification problem. However, I am getting some weird plots of the gradients. My first impression was an issue with underflow or overflow, but I have tried to clip the norm of the gradients, but it is not helping.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/e/beb3c073e4eb3dbde72f87683c27d8d66cbde509.png" data-download-href="https://discuss.pytorch.org/uploads/default/beb3c073e4eb3dbde72f87683c27d8d66cbde509" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/e/beb3c073e4eb3dbde72f87683c27d8d66cbde509_2_690x228.png" alt="image" data-base62-sha1="rd1POI6x0bEJQOqoo0QM5O3v3jH" width="690" height="228" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/e/beb3c073e4eb3dbde72f87683c27d8d66cbde509_2_690x228.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/e/beb3c073e4eb3dbde72f87683c27d8d66cbde509_2_1035x342.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/e/beb3c073e4eb3dbde72f87683c27d8d66cbde509_2_1380x456.png 2x" data-dominant-color="F9FBFD"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1471×487 16.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Any suggestion?</p>
Derivative of NN in the loss function

xn2402: <p>Hi,</p>
<p>I want to approximate the Value function V(x) of a continuous-time system by a neural network.<br>
For the learning, the loss function have to contain the derivative of the model (V(x)) with respect to the input. How can I implement this in PyTorch?<br>
I added a standard LQR example in the following, but the loss is not decreasing.<br>
I think, grad_V_x no longer has a symbolic dependence on the network weights and thus steepest does not work.<br>
Any help is appreciated!</p>
<pre><code class="lang-auto">import numpy as np
import torch
import torch.nn as nn
import torch.optim as optim

# setting seed (for reproducible learning) 
seed = 0
torch.manual_seed(seed)
if torch.cuda.is_available():
    torch.cuda.manual_seed_all(seed)

# model dynamics
A = 1
B = 1

# cost function 
R = 1
Q = 1


# Architecture of the neural network 
model = nn.Sequential(
    nn.Linear(1, 10),
    nn.ReLU(),
    nn.Linear(10, 10),
    nn.ReLU(),
    nn.Linear(10, 1)
)

K=1.1 # stable initial policy 

# loss function and optimizer
loss_fn = nn.MSELoss()  # mean square error
optimizer = optim.Adam(model.parameters(), lr=0.015) 

l=1
while l&lt;10: # Policy iteration
    N = 11 # number of samples
    x_0 = np.linspace(-1,1,N)
    x_dot = np.zeros((N,1))   
    r = np.zeros((N,1))

    if l==1: # start with stable initial policy 
        for i in range(N):
            x_dot[i]=A*x_0[i]-B*K*x_0[i]
            r[i]=Q*x_0[i]**2 + R*(K*x_0[i])**2
        
    else: # use NN
        for i in range(N):
            grad_V_x = model(x_0_tensor[i]) # prediction
            grad_V_x = grad_V_x.detach().numpy() # convert to numpy
            u = -0.5*(1/R)*B*grad_V_x # calculate optimal input
            x_dot[i]=A*x_0[i]+B*u
            r[i]=Q*x_0[i]**2 + R*(u)**2 

    # train NN 
    epoch = 1
    tol = 0.0001
    loss = np.inf

    while loss&gt;tol and epoch&lt;3000: # terminal criterion 
        x_0_tensor = torch.from_numpy(np.expand_dims(x_0,axis=1)).float()
        x_0_tensor.requires_grad_()
        x_dot_tensor = torch.from_numpy(x_dot).float()

        r_tensor = torch.from_numpy(r).float()

        grad_V_x = torch.autograd.functional.jacobian(model,x_0_tensor)
        grad_V_x.requires_grad_()
        loss = loss_fn(r_tensor,-grad_V_x*x_dot_tensor ) # minimize Bellmann error 
        
        optimizer.zero_grad() # reset gradients
        loss.backward() # calculate gradient 
        optimizer.step() # update weights
        print(f"epoch {epoch} loss {loss}")
        epoch = epoch+1 # count epochs

    l=l+1
    print(l)
</code></pre>

AlphaBetaGamma96: <p>Hi <a class="mention" href="/u/xn2402">@xn2402</a>,</p>
<p>I’d recommend looking at the <code>torch.func</code> namespace as that allows for the composition of derivatives within the loss function, although there’s a <em>slight</em> change in the syntax. The documentation can be found here: <a href="https://pytorch.org/docs/stable/func.html" class="inline-onebox" rel="noopener nofollow ugc">torch.func — PyTorch 2.0 documentation</a></p>
Model size reduction

Kiana_k: <p>I did quantization on a model and then the model size reduced 50% .it was one hidden layer model. its weird because when I just increase the number of neurons in its hidden layer the model reduction was greater than before (it reduced 80%).what can be the reason?</p>

jerryzh168: <p>this is related to the percentage of the weight size on that layer over the sums of the weight size of the entire model I think, when you increase the number of neurons in the hidden layer, the percentage increases so the reduction is more significant.</p>
Most efficient way to get just the off-diagonal elements of a Tensor?

AlphaBetaGamma96: <p>Hi All,</p>
<p>I was wondering what’s the most efficient way to grab the off-diagonal elements of a batch of matrices?</p>
<p>Let’s assume I have some Tensor of shape <code>[B,N,N]</code> and wish to grab all off-diagonal elements (set all diagonal elements to 0) I’m currently using this,</p>
<pre><code class="lang-auto">def get_off_diagonal_elements(M):
  dim=len(M.shape)-2
  if(M.shape[-2] != M.shape[-1]):
    raise ValueError("Matrix error")
  mask = (1 - torch.eye(M.shape[-1], device=M.device)) 
  return M*mask

</code></pre>
<p>As I was typing out this question I had the idea of just copying the Tensor and filling the diagonal with 0s, however, I tried the following below and it failed with the following error</p>
<pre><code class="lang-auto">def get_off_diagonal_elements_fast(M):
  return M.clone().fill_diagonal_(0)
</code></pre>
<p>Also, do I need to call <code>.clone()</code> when returning the Tensor? The error is here,</p>
<pre><code class="lang-auto">RuntimeError: all dimensions of input must be of equal length
</code></pre>
<p>Any help would be appreciated!</p>

tom: <p>You need the clone if you want to keep M unchanged.</p>
<pre><code class="lang-auto">def get_off_diagonal_elements(M):
    res = M.clone()
    res.diagonal(dim1=-1, dim2=-2).zero_()
    return res
</code></pre>

AlphaBetaGamma96: <p>Perfect, thank you! Could I ask a quick follow-up? I assume the <code>.clone()</code> call is due to shallow copying issues?</p>

tom: <p>Yes,but the terminology here is views. Diagonal gets the diagonal as a view of the same memory blob as its Tensor instance  and methods ending with _ modify their Tensor instance.</p>
<p>Best regards</p>
<p>Thomas</p>

Siladittya_Manna: <pre><code class="lang-auto">def get_off_diagonal_elements(M):
    return M[~torch.eye(*M.shape,dtype = torch.bool)].view(M.shape[0], M.shape[1]-1)
</code></pre>
<p>OR</p>
<pre><code class="lang-auto">def get_off_diagonal_elements(M):
    return M[~torch.eye(*M.shape,dtype = torch.bool)]
</code></pre>
<p>should also work</p>
(Face recognition) Problem fine-tuning arcface with triplet loss

SEBASTIAN_PULGAR: <p>Hi!,</p>
<p>I am trying to do fine-tuning of the last layer of the arcface model of facial recognition, for this leaving frozen all the weights of the network, except those belonging to the last layers. The problem is that the loss does not decrease, instead if I do not leave any frozen weight the model learns in a good way, however, I want to use the knowledge learned by the network to be trained with arcface, so I want to keep the weights frozen.<br>
I would like to know if someone understands what happens, I have the impression that there is no compatibility between the embeddings, since arcface is trained with a classification logic and not with a contrastive logic as in triplet loss, but it does not make sense to me since in both cases the embeddings of the same person are close in the hyperplane, so there should not be problems.<br>
*I have the same problem with any model trained with classification losses, adaface, shpereface, etc. but not with contrastive models like facenet.</p>
<p>Regards</p>
Error too many indices for tensor of dimension 1

Reni_Setyaningsih: <p>Hallo everyone, i’m encountering a problem with my code…<br>
It said <code>IndexError: too many indices for tensor of dimension 1</code> while i executed</p>
<pre><code class="lang-auto">def plot_img_with_bbox(path: str) -&gt; Image:
    """Show image with all of the bounding box from JPG Path
    path: str=Path of jpg file

    Return
      Image=Image that already have been drawen with bounding box
    """
    path_xml, path_jpg = xml_and_jpg(path)
    sample_image = Image.open(path_jpg)
    sample_annotations, sample_labels = get_annotations_and_labels(path_xml)
    tensor_image = (transforms.ToTensor()(sample_image) * 255).to(torch.uint8)
    tensor_annotations = torch.tensor(sample_annotations)

    dict_color = {"other": "black", "LD": "yellow", "H": "green", "HD": "red"}
    colors = [dict_color[x] for x in sample_labels]
    torch.Size([1, 4])
    img = draw_bounding_boxes(
        tensor_image,
        boxes=tensor_annotations,
        labels=sample_labels,
        width=3,
        colors=colors,
    )
    view_image = transforms.ToPILImage()(img)
    return view_image

plot_img_with_bbox(list_unique_path[0])
</code></pre>
<p>How can i solve this problem?<br>
Thanks in advance guys</p>
Is it possible to use c++ dataloader to load dataset, but still use python for other parts of training?

coincheung: <p>Hi,</p>
<p>I just do not know what is the problem with my platform. It is very slow to process images with python. Thus I am thinking about using c++ to implement the part of loading and preprocessing samples, but still use pytorch for the other parts of the training loop.</p>
<p>Is there any examples of this? Or would you please provide some ideas on how to implement this?</p>

ptrblck: <p>Did you profile your machine and have verified that you are able to load images in C++ faster?</p>

coincheung: <p>Hi,</p>
<p>I think the problem is the multi-processing. I just found that launching two identical training script would make training much slower, I think this is because the machine is too weak to support launching multi-process. Given that the cpp frontend dataloader uses threads, it would make the machine run with less burden and thus training would be much faster.</p>
<p>By the way, recently, I was trying to deduplicate images in a dataset, I just found that launch a python process pool with 128 processes is almost 10x slower than a cpp program with 64 threads. I think this machine prefers cpp to python.</p>
A question about UNet Crop and nn.functional.interpolate

lint_weaver: <p>I’m trying to code the UNet paper, however I’ve come across the issue of implementing the crop when trying to code the decoder.</p>
<p>One person did this:</p>
<pre><code class="lang-auto">class Decoder(nn.Module):
    def __init__(self, chs=(1024, 512, 256, 128, 64)):
        super().__init__()
        self.chs         = chs
        self.upconvs    = nn.ModuleList([nn.ConvTranspose2d(chs[i], chs[i+1], 2, 2) for i in range(len(chs)-1)])
        self.dec_blocks = nn.ModuleList([Block(chs[i], chs[i+1]) for i in range(len(chs)-1)]) 
        
    def forward(self, x, encoder_features):
        for i in range(len(self.chs)-1):
            x        = self.upconvs[i](x)
            enc_ftrs = self.crop(encoder_features[i], x)
            x        = torch.cat([x, enc_ftrs], dim=1)
            x        = self.dec_blocks[i](x)
        return x
    
    def crop(self, enc_ftrs, x):
        _, _, H, W = x.shape
        enc_ftrs   = torchvision.transforms.CenterCrop([H, W])(enc_ftrs)
        return enc_ftrs
</code></pre>
<p>If I used nn.functional.interpolate() instead, would I get the same effect as the crop function in the above code statement?</p>

ptrblck: <aside class="quote no-group" data-username="lint_weaver" data-post="1" data-topic="188977">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/l/2acd7d/48.png" class="avatar"> lint_weaver:</div>
<blockquote>
<p>would I get the same effect as the crop function</p>
</blockquote>
</aside>
<p>I don’t know what you mean by “effect”, but an interpolation is not producing the same output as a cropping. Both manipulate the spatial size and you can use both to create a smaller output but the values won’t necessarily be equal.</p>
Out of Memory after 2 hours

bloos: <p>Hi!<br>
I’m developing a language classifier. In order to do that, I’ve downloaded Common Voice in 34 languages and a pretrained Wav2Vec2 Model that I want to finetune, to solve this task. The training procedure is parallelized with pytorch lightning to run on 8 RTX 3090. The System has 96GB of CPU RAM.</p>
<p>The Problem is, that my CPU memory consumption jumps to 70 GB initially and creeps up by several MB per step, until it crashes after an hour or two. So far, I’ve been unable to track down the memory error. I’ve tried several memory profilers and debugging techniques but with no success. Here is what I’ve tried:</p>
<ul>
<li>PDB with tracemalloc or objgraph: I like these tools but sadly the memory consumption didn’t show there, which makes me think that an external library keeps leaking</li>
<li>Valgrind: After 90 Minutes in the initialization phase i’ve stopped this attempt. Valgrind slows down python too much</li>
<li>I’ve compared tracemallocs measured memory with psutil.Process(os.getpid).memory_info . While tracemalloc did not show the memory consumption, psutil did</li>
<li>I’ve hypothesized that my high memory consumption is because torchaudio.datasets.COMMONVOICE loads the entire subdirectory as millions of short strings which produces a huge memory overhead. To reduce it I’ve forked torchaudio.datasets.COMMONVOICE and changed it so that it uses numpy arrays instead. Actually, this has reduced my memory consumption by 10 GB but the memory still leaked, so this didn’t solve the issue.</li>
</ul>
<p>My current guess is that torchaudio.load is the issue. I’ve trained a very similar approach that uses WAV instead of Commonvoice’s MP3 files and there was no leak.</p>
<p>Do you have any Ideas on how I could track the issue?</p>
<p>Update: Just for the case someone asks: The num_workers parameter of the dataloader is 0</p>

ptrblck: <p>You could replace the <code>torchaudio</code> dataset with a fake or custom one and check if the memory increase would still be visible. If not, you could then iterate the original dataset only and check if this might trigger the increase or if it’s only visible in a training loop.</p>

bloos: <p>thanks for the hint. I will try that tomorrow and report the results here <img src="https://discuss.pytorch.org/images/emoji/apple/+1.png?v=12" title=":+1:" class="emoji" alt=":+1:" loading="lazy" width="20" height="20"></p>
ONNX op support LpNormalization

escorciav: <p>Hi!</p>
<p>pytorch onnx (opset 9) unfolds L2Normalization as multiple operators instead of using LpNormalization operator of ONNX. Is there any clean workaround?</p>
<p>I found out the “Caffe2 &amp; ONNX implementations differ.” But, I don’t know how that relates to Pytorch.</p>
<p>Thanks in advance,<br>
Victor</p>

escorciav: <p>I just found out about <a href="https://github.com/microsoft/onnxconverter-common/blob/master/onnxconverter_common/onnx_ops.py#L631" rel="noopener nofollow ugc">onnxconverter-common.</a></p>
<p>Likewise, opened an issue as part of <a href="https://github.com/onnx/optimizer/issues/143" rel="noopener nofollow ugc">onnx-optimizer</a>.</p>

escorciav: <p>We ended up traversing the onnx graph in Python &amp; replacing the ops. Leverage Netron while doing so <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
Any tips for huge sequence length variation performance

apytorch: <p>I have a working forward model that takes sequence data and labels it. In looking for any possible performance improvements in my code I did run across the suggestion that sequence data could be preallocated to avoid memory fragmentation (an issue for me, that leads to OOM errors if I don’t clear the cache on every batch of the training loop).</p>
<p>The problem is that the max sequence is 3x the minimum sequence. And even if I preallocate the tensors, now the model for every batch has to process the longest possible sequence, which greatly slows down the overall training (completely wiping out the benefit of preallocation - and then some).</p>
<p>I’ve tried just sending the relevant portion of the data of each batch through the model, but that won’t work because all view calls complain that the shape of the “necessary” tensor isn’t compatible with the preallocated tensor.</p>
<p>Any other suggestions that I could try? Again, the model works. It’s just that I have to clear the cache every batch or I’ll get OOM errors from all the fragmentation, which I know is having some negative effect.</p>

yiftach: <p>Just for clarification - by OOM do you mean for the GPU memory or regular RAM?</p>

apytorch: <p>OOM on the GPU memory</p>

yiftach: <p>Can you show the structure of your training loop (over epochs and over iterations)? There might be room to restructure things to make memory release earlier.</p>
Source code of torch-pso

S_M: <p>Hi all I am trying to implement the PSO as an optimizer in PyTorch. I need to apply some modifications on it. How can I do that?<br>
Thank</p>

ptrblck: <p>Are you looking for <a href="https://github.com/qthequartermasterman/torch_pso">this repository</a>?</p>

S_M: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, yes I worked with this library. I tried to classify CIFAR-10. My code was from this <a href="https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html" rel="noopener nofollow ugc">reference</a> as you can see the loss function from this reference is CrossEntropy but training error in 10 epochs with default values of the PSO <a href="https://github.com/qthequartermasterman/torch_pso" rel="noopener nofollow ugc">repository</a> was approximately 3000!!! so I tried MSEloss and I saw the error was increased sharply. How can I handle this?</p>
Locating the bug operator of resnet18

aibear: <p>I am currently developing an operator for the AMD platform, and after comparing it with the operator on the CUDA platform, I found that they produce the same output. However, when I use this operator to construct ResNet18(load from torchvision.models.resnet18), I notice a significant difference in accuracy between the AMD and CUDA platforms after training. How can I quickly identify the problematic operator?</p>
<p>dataset: cifar10<br>
acc: 12% on AMD, 65% on CUDA.<br>
the settings and hyper parameters are all the same.</p>
Torch.data process the 1st batch extremely slow

dyin: <p>We are using torch.data lib to load a large # of TFRecord files, the code looks like this:</p>
<pre><code class="lang-auto">datapipes = []
for path in paths:
     datapipe = datapipe.open_files_by_fsspec(mode='rb')
    fsspec.asyn.iothread[0] = None
    fsspec.asyn.loop[0] = None
    datapipe = datapipe.decompress(file_type=compression_type)
    datapipe = datapipe.load_from_tfrecord()
    datapipe = datapipe.cycle(num_epochs)
    datapipes.append(datapipe)
    
pipes_to_weights_dict = dict(zip(datapipes, dir_weights))
datapipe = SampleMultiplexer(pipes_to_weights_dict)
datapipe = datapipe.map(_load_single_example)

rs = MultiProcessingReadingService(num_workers=num_parallel_calls, 
                                           worker_prefetch_cnt=prefetch_size // num_parallel_calls,
                                           main_prefetch_cnt=prefetch_size)

data_loader = DataLoader2(datapipe, reading_service=rs)


for i, e in enumerate(data_loader):
    print(time.time())
</code></pre>
<p>We found that it takes 10+mins to process the 1st batch. Spending some time adding some debug messages, and found that if I add a print before and inside the loop of <a href="https://github.com/pytorch/data/blob/d76d92c97ef04463c4c4c9c65b9ecf5343ea9d2c/torchdata/datapipes/iter/load/fsspec.py#L162" rel="noopener nofollow ugc">this</a> line,  it keeps printing the message before the loop, but not inside the loop.</p>
<p>I also tried a dataset with only a few TFRecords, the 1st batch can be processed quickly.</p>
<ol>
<li>Can I get some help to narrow down the root cause of long processing time for the 1st batch?</li>
<li>Seems torch.data is not actively developed, can you share more about the roadmap for Torch data loader? Should we use torch.data lib or something else?</li>
</ol>
<p>Thanks<br>
Dengpan</p>
Why PyTorch use Objective-C for MPS instead of C++?

redradist: <p>I am wondering why PyTorch team use Objective-C instead of C++ since all other operators in ATen are written in C++ ?</p>
<p>It is possible to use Metal API from C++ by using <code>metal-cpp</code> (see also <a href="https://github.com/LeeTeng2001/metal-cpp-cmake" class="inline-onebox" rel="noopener nofollow ugc">GitHub - LeeTeng2001/metal-cpp-cmake: Metal graphic api ported to C++ with CMake build system</a>)</p>
<p>Using one language for all low level operators could improve code reuse</p>
CUBLAS_STATUS_NOT_SUPPORTED when calling `cublasGemmStridedBatchedExFix

jgbor: <p>Hi,<br>
I 'm trying to peft a model on my GPU (MX110) but when the train function is called, I get this run time error.<br>
CUDA 11.8 is installed with the latest torch Python package. On CPU the training is working as intended.</p>
<p>The full error message:<br>
RuntimeError: CUDA error: CUBLAS_STATUS_NOT_SUPPORTED when calling <code>cublasGemmStridedBatchedExFix(handle, opa, opb, (int)m, (int)n, (int)k, (void*)&amp;falpha, a, CUDA_R_16BF, (int)lda, stridea, b, CUDA_R_16BF, (int)ldb, strideb, (void*)&amp;fbeta, c, CUDA_R_16BF, (int)ldc, stridec, (int)num_batches, CUDA_R_32F, CUBLAS_GEMM_DEFAULT_TENSOR_OP)</code></p>
<p>I already looked for solutions online but I could not find anything useful.<br>
Any advice would be appreciated</p>

ptrblck: <p>Could you post a minimal and executable code snippet reproducing the issue? Are other native kernels working fine in this setup?</p>

jgbor: <pre><code class="lang-python">from transformers import AutoModelForSeq2SeqLM, AutoTokenizer, GenerationConfig, TrainingArguments, Trainer
import torch
from datasets import load_dataset

dataset = load_dataset("knkarthick/dialogsum")
model_name='google/flan-t5-base'
original_model = AutoModelForSeq2SeqLM.from_pretrained(model_name, torch_dtype=torch.bfloat16, ignore_mismatched_sizes=True)
tokenizer = AutoTokenizer.from_pretrained(model_name)

def tokenize_function (example):
    start_prompt = 'Summarize the following conversation. \n\n'
    end_prompt = '\n\nSummary: '
    prompt = [start_prompt + dialogue + end_prompt for dialogue in example["dialogue"]]
    example['input_ids'] = tokenizer (prompt, padding="max_length", truncation =True, return_tensors="pt").input_ids

    example['labels'] = tokenizer (example["summary"], padding="max_length", truncation=True, return_tensors="pt").input_ids 
    
    return example

tokenized_datasets = dataset.map(tokenize_function, batched=True)
tokenized_datasets = tokenized_datasets.remove_columns(['id', 'topic', 'dialogue', 'summary', ])

from peft import LoraConfig, get_peft_model, TaskType

lora_config = LoraConfig(
    r=32, #rank
    lora_alpha = 32,
    target_modules = ["q", "v"],
    lora_dropout = 0.05,
    bias="none",
    task_type=TaskType.SEQ_2_SEQ_LM #FLAN-T5
)

peft_model = get_peft_model(original_model,
                            lora_config)
print(torch.cuda.is_available())
device = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")
peft_model.to(device)

peft_training_args = TrainingArguments(
    output_dir="./peft",
    auto_find_batch_size=True,
    learning_rate=1e-4,
    num_train_epochs=1,
    logging_steps=1,
    max_steps=1
)


peft_trainer = Trainer(
    model=peft_model,
    args=peft_training_args,
    train_dataset=tokenized_datasets["train"],
)

peft_trainer.train()
</code></pre>
Audio style transfer: combine a temporal and a spectral loss function

francescopapaleo: <p>Hi,</p>
<p>I am trying to implement two loss functions in my training script for convolutional models.<br>
I use a dataset of input+target pairs that are recorded at the input and output stage of an analog device.<br>
This training script is part of a <a href="https://github.com/francescopapaleo/neural-audio-spring-reverb/tree/main" rel="noopener nofollow ugc">larger project</a> to compare different model architectures on the same task.</p>
<pre><code class="lang-auto">criterion1: torch.nn.L1Loss()
criterion2: auraloss.freq.MultiResolutionSTFTLoss()
</code></pre>
<p>As optimizer, I am using Adam and as scheduler ReduceLROnPlateau.</p>
<pre><code class="lang-auto">[...]
pred = model(input)

loss1 = criterion1(pred, target)
if config['criterion2'] is not None:
loss2 = criterion2(pred, target)
loss = loss1 + loss2

loss.backward()                             
optimizer.step()

train_loss += loss.item()
[...]
</code></pre>
<p>Questions:</p>
<ul>
<li>Is this the right way to do it during training?</li>
<li>Do you have any suggestions on how to choose weights that would make both loss functions equally affecting the backward pass?</li>
</ul>
<p>Thank you for your help</p>

francescopapaleo: <p>Any suggestions on that?</p>

bloos: <p>It is one way of doing it. The Problem with this approach is that both gradients should be of similar size or one of the losses will be “ignored”<br>
Another possibility would be</p>
<pre><code class="lang-auto">loss = 0.5*loss1 + 0.5*loss2
</code></pre>
<p>Or even</p>
<pre><code class="lang-auto">loss = 0.7*loss1 + 0.3*loss2
</code></pre>
<p>This can become fiddly, especially when the losses are contradicting each other like in GANs, because you want them to reach an equilibrium</p>
<p>Another approach could be to use Multiple optimizers, but this becomes equally fiddly</p>
Are alpha and beta in torchrl's prioritized experience replay schedulable?

MadChuck: <p>Hi,</p>
<p>I’ve been thinking for a while to switch from openai baselines PER implementation to torchrl.data.PrioritizedReplayBuffer, since it should be faster, considering current implementation needs to load batch to torch format anyway.</p>
<p>In <a href="https://arxiv.org/abs/1511.05952" rel="noopener nofollow ugc">the original paper</a>, the parameters alpha and beta were changing over time. I also am aware that in some cases, fixed alpha and beta may be better (as reported in <a href="https://github.com/google/dopamine/blob/ce36aab6528b26a699f5f1cefd330fdaf23a5d72/dopamine/agents/rainbow/rainbow_agent.py#L269C5-L272C67" rel="noopener nofollow ugc">dopamine/rainbow</a>).</p>
<p>Since I don’t see an official way of scheduling alpha and beta in documentation, I’m curious whether that was done purposefully for simplicity with believe that constant alpha and beta are better or maybe it has to do with something else or is actually possible?</p>

vmoens: <p>This is not currently supported but I’d love to give it a look! Can you post a feature request on the GitHub repo?</p>

MadChuck: <p>Sure, thanks for the interest <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"><br>
<a href="https://github.com/pytorch/rl/issues/1575" rel="noopener nofollow ugc">https://github.com/pytorch/rl/issues/1575</a></p>
ONNX export failed on unsafe_chunk

tom-huntington: <blockquote>
<p>torch/onnx/symbolic_helper.py:258: UserWarning: ONNX export failed on unsafe_chunk because unknown dimension size not supported</p>
</blockquote>
<p>I was trying to export the Tacotron2 model provided by torchaudio:</p>
<pre><code class="lang-auto">import torch
import torchaudio
import onnx

bundle = torchaudio.pipelines.TACOTRON2_WAVERNN_PHONE_LJSPEECH
processor = bundle.get_text_processor()
tacotron2 = bundle.get_tacotron2().to(device)

text = "Hello world! Text to speech!"

with torch.inference_mode():
    processed, lengths = processor(text)
    processed = processed.to(device)
    lengths = lengths.to(device)
    spec, _, _ = tacotron2.infer(processed, lengths)


class OnnxWrapper(torch.nn.Module):
    def __init__(self, tacotron):
        super().__init__()
        self.tacotron = tacotron

    def forward(self, processed, lengths):
        spec, _, _ = self.tacotron.infer(processed, lengths)
        return spec


onnx_wrapper = OnnxWrapper(tacotron2).eval()

with torch.inference_mode():
    torch.onnx.export(to,
                      (processed, lengths),
                      "tacotron2.onnx",
                      training=torch.onnx.TrainingMode.EVAL,
                      export_params=True,
                      opset_version=11,
                      do_constant_folding=True,
                      input_names=['processed', 'lengths'],
                      output_names=['spec'])
</code></pre>
<p>From grepping pytorch’s source code, it seems that <code>unsafe_cunk</code> is only called in RNN.cpp for the LSTM and GRU modules.</p>
<p>However, there is no problem exporting LSTM and GRU modules to onnx, so I’m confused by what’s causing the error.</p>

ZHUORAN_GUO: <p>hello, I met the same problem, do you have solved it ?</p>
How to code joint training in my network

gpqls7662: <p>I want to use two network.<br>
And my network architecture is<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/7/77dc04b4b277032b37aa05c1cd33eb1d39fce408.png" data-download-href="https://discuss.pytorch.org/uploads/default/77dc04b4b277032b37aa05c1cd33eb1d39fce408" title="화면 캡처 2023-09-25"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/7/77dc04b4b277032b37aa05c1cd33eb1d39fce408_2_690x192.png" alt="화면 캡처 2023-09-25" data-base62-sha1="h6kcMBDTl9Ugwgt4yNmnC6QA0je" width="690" height="192" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/7/77dc04b4b277032b37aa05c1cd33eb1d39fce408_2_690x192.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/7/77dc04b4b277032b37aa05c1cd33eb1d39fce408_2_1035x288.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/7/7/77dc04b4b277032b37aa05c1cd33eb1d39fce408_2_1380x384.png 2x" data-dominant-color="F7F5F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">화면 캡처 2023-09-25</span><span class="informations">1972×551 10.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>My code is</p>
<pre><code class="lang-auto">self.optG.zero_grad()
self.optR.zero_grad()
refine = self.refine_network(self.cond)
loss1 = self.netG(self.gt, refine)
loss1 = loss1.sum()/int(b*c*h*w)
loss2 = self.criterion(refine, self.gt).sum() / int(b*c*h*w)
l_pix = loss1 + loss2
l_pix.backward()
self.optG.step()
self.optR.step()
</code></pre>
<p>Is it right code?<br>
I don’t know how to use two network that I designed.</p>
How does torch.hub.load_state_dict_from_url work

Divya_Velayudhan: <p>Hi</p>
<p>I am quite new to Pytorch, and hence this might be a silly question.</p>
<p>I am using Google Colab, and trying to load the pretrained weights of vit_small_patch16_384 from timm module. But I am facing some issues, as its .npz format.</p>
<p>!pip install timm==0.4.12 (timm version I used)</p>
<p>checkpoint = torch.hub.load_state_dict_from_url( ‘<a href="https://storage.googleapis.com/vit_models/augreg/S_16-i21k-300ep-lr_0.001-aug_light1-wd_0.03-do_0.0-sd_0.0--imagenet2012-steps_20k-lr_0.03-res_384.npz" rel="noopener nofollow ugc">https://storage.googleapis.com/vit_models/augreg/S_16-i21k-300ep-lr_0.001-aug_light1-wd_0.03-do_0.0-sd_0.0--imagenet2012-steps_20k-lr_0.03-res_384.npz</a>’, map_location=‘cpu’, check_hash=True)</p>
<p>I get the following error:</p>
<p>RuntimeError: Expected hasRecord(“version”) to be true, but got false.  (Could this error message be improved?  If so, please report an enhancement request to PyTorch.)</p>
<p>What am I doing wrong?<br>
Can anyone please help me to resolve this.<br>
Thanks in advance</p>

Zhiyuan_Chen: <p>Hi there, I have met the same error, have you fixed it now?</p>

marksaroufim: <p>Where did you get the url from? <a href="https://storage.googleapis.com/vit_models/augreg/S_16-i21k-300ep-lr_0.001-aug_light1-wd_0.03-do_0.0-sd_0.0--imagenet2012-steps_20k-lr_0.03-res_384.npz">https://storage.googleapis.com/vit_models/augreg/S_16-i21k-300ep-lr_0.001-aug_light1-wd_0.03-do_0.0-sd_0.0--imagenet2012-steps_20k-lr_0.03-res_384.npz</a></p>
<p>That’s a numpy file format and you’re trying to load a numpy checkpoint and not a torch one, fwiw the error message can be improved</p>
Question about equivariant regularization in classification task

Xiao_Bin: <p>Dear All,<br>
Recently, I met with a question, that is, whether I can introduce equivariant regularization (ER) in classification task? even I don’t make it clear why ER can be boldly used in segmentation task?<br>
Can anybody give some explainations, thanks.</p>

Xiao_Bin: <p>As above, assumed i have an image X, and its affine transformed version t(X), here affine transformation is denoted as A. After some conv. layers, the new features are Fx and Ftx, respectively, now can i compute the loss among A(Fx) and Ftx in classification task?<br>
help explain whether this possibility or not, be grateful!</p>
SAM Annotation Tool (SAMAT)

Divelix: <p>I just open-sourced (MIT License) semantic segmentation annotation powered by segment-anything model that I used for a while in my projects. Hopefully it will help someone.<br>
Link to my project: <a href="https://github.com/Divelix/samat" rel="noopener nofollow ugc">SAMAT</a> (any feedback in Discussions section on GitHub is appreciated)<br>
<strong>Features:</strong></p>
<ul>
<li>Brush annotation (opposed to polygons)</li>
<li>Magic Wand (like in Photoshop) powered by segment-anything masks (it is optional btw, if you don’t have cool GPU to prepare masks)</li>
</ul>
<p><strong>Why yet another annotation tool?</strong><br>
Before starting this project I tried supervisely, segments.ai, roboflow and several others, but found them not suitable enough for my projects.<br>
Their cons, I tried to fix with my tool:</p>
<ul>
<li>Latency: they are web based, hence burden with irritating latency during labeling (SAMAT is snappy because it is local desktop app)</li>
<li>Complicated: too much features, hence overloaded UI (SAMAT is just a colored brush)</li>
<li>Proprietary: they all follow SaaS model</li>
</ul>
<p>Btw, there is labeling tool called SALT on github which also uses segment-anything model, but have a different approach to UI/UX, so take a look at it too.</p>
Getting issues with torch.onnx.export

Amit0617: <p>I would keep it short. I have created a <a href="https://colab.research.google.com/drive/1BkrbAa7SUhALhd_70HPBp_-U_BVTBK9w?usp=sharing" rel="noopener nofollow ugc">colab</a> for your easy referencing. Model used is an open source model and I want to create computational graph of it using <code>torch.onnx.export</code> (currently getting Assertion error). I have crafted a dummy_data (<a href="https://colab.research.google.com/drive/1BkrbAa7SUhALhd_70HPBp_-U_BVTBK9w#scrollTo=tKBl3G-YSzyX" rel="noopener nofollow ugc">this cell</a>) for exporting that can be changed as you wish.</p>