akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Vanishes of gradients when I add some noise to the weights of neural networks

username: richardo_van
cooked: <p>There’s some problems when I achieve the differentially private federated learning. I added the noise to the weights of neural networks before the clients send them to the server.<br>
It is obvious that there are vanishes of gradients, but I don’t know how to handle this problem. If there are some people have been seen this problems?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/8/d8f87ae7bd495bb4c5d9c128f05d838481ccf5c1.png" data-download-href="https://discuss.pytorch.org/uploads/default/d8f87ae7bd495bb4c5d9c128f05d838481ccf5c1" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/8/d8f87ae7bd495bb4c5d9c128f05d838481ccf5c1_2_690x162.png" alt="image" data-base62-sha1="uXpuZ7SA5JnfsslGMArT5roXFNT" width="690" height="162" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/8/d8f87ae7bd495bb4c5d9c128f05d838481ccf5c1_2_690x162.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/8/d8f87ae7bd495bb4c5d9c128f05d838481ccf5c1_2_1035x243.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/8/d8f87ae7bd495bb4c5d9c128f05d838481ccf5c1_2_1380x324.png 2x" data-dominant-color="222327"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1726×406 35 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Changing embedding image size

username: ilhamsyahids
cooked: <p>I currently have a vector store that stores my large embedding of images. The vector length of these embeddings is 512. However, I would like to use a different featurizer that outputs embeddings with a vector length of 768. I want to use these new embeddings for searching without having to migrate or remove my existing vector store.</p>
<p>As someone who is new to this field, I have been considering a few options. One option I have thought about is taking the first 512 values of the 768-dimensional embeddings and discarding the rest. However, I am concerned that this approach may result in a loss of quality. Another option I have considered is using a projection method, as shown below:</p>
<pre><code class="lang-auto">import torch
from torch import nn

dim_in = 1024
dim_out = 768

projector = nn.Linear(dim_in, dim_out, bias=False)

inputs = torch.rand(dim_in)
inputs.shape # 1024

outputs = projector(dim_out)
outputs.shape # 768
</code></pre>
<p>However, I have noticed that when we initialize the projector, we always use random parameters, which means that the results may not be consistent.</p>
<p>Is there any other approach I could take? What is the correct way to convert embeddings with a 768 vector length into a 512 vector length, or in general, to convert larger embeddings into smaller embeddings and vice versa, without significant loss of quality?</p>
Question about tensor assign time

username: maqy1995
cooked: <p>Why does <code>unpack_time</code> in the code snippet below take so long?</p>
<pre><code class="lang-auto">import time
import torch


class MyDataloader:
    def __init__(self, shape):
        self.shape = shape
        self.tensor = torch.randn(shape, dtype=torch.float32)
        self.idx = 0
        self.num = 5
        self.batch_idxs = torch.randint(0, self.shape[0], size=(self.shape[0],), dtype=torch.int64)
        # self.batch_idxs = torch.arange(self.shape[0], dtype=torch.int64)
        self.results = []

    def __iter__(self):
        return self

    def __next__(self):
        if self.idx &lt; self.num:
            self.idx += 1
            result = [self.tensor[self.batch_idxs[i * 10000:(i + 1) * 10000]] for i in
                      range(5)] 
            # self.results.append(result) # if we keep the reference of result, the used time will decrease a lot
            return result
        else:
            raise StopIteration


if __name__ == "__main__":
    # torch.set_num_threads(1)
    process_start_time = time.time()
    my_dataloader = MyDataloader((100000, 10000))
    init_dataloader_end_time = time.time()
    loader_start_time = time.time()

    for x in my_dataloader:
        unpack_start = time.time()
        t = x  # cost time hear
        # a, b, c, d, e = x # similar to last line
        unpack_end = time.time()
        print("loader_batch_time {:.8f} s | unpack_time {:.8f} s".format(unpack_start - loader_start_time,
                                                                         unpack_end - unpack_start))
        loader_start_time = time.time()

    print("process dataloader time is {:.4f} s".format(time.time() - init_dataloader_end_time))
    print("process time is {:.4f} s".format(time.time() - process_start_time))
</code></pre>
<p>code print log:</p>
<pre><code class="lang-auto">loader_batch_time 0.40162539 s | unpack_time 0.00000095 s
loader_batch_time 0.34314251 s | unpack_time 0.12212968 s
loader_batch_time 0.31168938 s | unpack_time 0.11875153 s
loader_batch_time 0.30921507 s | unpack_time 0.11288309 s
loader_batch_time 0.28784609 s | unpack_time 0.12381434 s
process dataloader time is 2.1312 s
process time is 8.1604 s
</code></pre>
<p>As we can see, <code>unpack_time</code> takes a long time. Why is this?</p>
<p>In addition, the unpack_time of the first iteration is very short, and it seems that gc is performed in each subsequent cycle.<br>
If gc is performed, is this time cost normal? Are there any optimization methods to reduce this time?</p>
<p>Thanks for any reply!</p>
FSDP: how to compute grad norm?

username: Exception
cooked: <p>Hi all!<br>
I am using the following code to compute grad norm in FSDP</p>
<pre><code class="lang-python">def compute_grad_norm(model: torch.nn.Module):
    g_sqsum = 0.0
    for p in model.parameters():
        g = p.grad
        if g is not None:
            g_sqsum += (g ** 2).sum().item()
    return torch.tensor(g_sqsum, device=torch.cuda.current_device())

if __name__ == '__main__':
    dist.init_process_group("nccl")
    world_size = int(os.environ["WORLD_SIZE"])
    model = MyModel()
    model = FSDP(
        model,
        auto_wrap_policy=wrapping_policy,
        device_id=torch.cuda.current_device(),
        sharding_strategy=ShardingStrategy.FULL_SHARD,
        mixed_precision=bfSixteen_mixed,
        limit_all_gathers=True,
        sync_module_states=False,
        param_init_fn=None,
        use_orig_params=True,
    )
    
    # after loss backward
    grad_norm = compute_grad_norm(model)
    dist.all_reduce(grad_norm, op=dist.ReduceOp.SUM)
    grad_norm = grad_norm / world_size
</code></pre>
<p>I am wondering if it is correct to divide <code>grad_norm</code> by <code>world_size</code> after <code>all_reduce</code> in FSDP? To the best of my knowledge, in FSDP, each device only retains a portion of the parameters. Therefore, after the <code>all_reduce</code> operation, the total <code>grad_norm</code> should have already been obtained, and there is no need to divide it by <code>world_size</code>.</p>
Resuming training with different learning rate of the optimizer in libtorch

username: Jesus
cooked: <p>Hello everyone.</p>
<p>I am using libtorch-cxx11-abi-shared-with-deps-2.0.1+cpu</p>
<p>Suppose i trained a model with SGD and saved the model and the optimizer. Now i want to resume my training with the same optimizer but this time with different learning rate (say 0.002). Can i do that?</p>
<p>Another thing to clarify: while resuming the training we usually load the model and the optimizer from some saved checkpoints. But suppose this time i only load the model, and not the optimizers, and continue training what will be the problem?</p>
Load_state_dict not finishing in Celeryworker Task

username: Bennse
cooked: <p>Heyo,</p>
<p>I am running an EasyOCR docker on a kubernetescluster which itself is working totally fine.<br>
However the issue I am facing is that when running the easyocr.Reader function, the Celeryworker seems to never finish and finally time out after X Minutes.<br>
Through a lot of debug work I was able to localize the issue to be at the “load_state_dict” function in module.py of the Torch package.</p>
<p>I have not gotten a single error or anything, the Worker simply timeouts at some point.<br>
Locally without a Docker it works, locally and on Kubernetescluster inside a docker it doesn’t work.<br>
(It does work when running the code through an API Function. It only has that issue with the Celeryworker)</p>
<p>As I am not able to get any more detail out of it, I was wondering if anyone on here could help me.<br>
I have tried it with Torch versions 1.10.1 til 1.12.1 with Torchvision 0.11.2 til 0.13.1 and am not sure what to do next.</p>
<p>Cheers,<br>
Bennse</p>

username: jlewallen18
cooked: <p>This is a long shot…but were you able to figure this out? I’m dealing with the SAME exact thing, while running <code>load_state_dict</code>, it just straight up stops.</p>
<p>I’m using FastAPI and I am preloading my model, what’s strange is that when I log out which keys have been loaded, it stops about 75% of the way through, and when I restart my server, the other 25% are finally logged to the screen. So it’s like I’m hitting a max memory buffer issue. Absolutely no errors though.</p>
<p>Video demo:</p>
          <video title="Screen Recording 2023-11-08 at 11.16.38 PM" width="695" height="403" style="max-width:100%" poster="https://cdn-cf-east.streamable.com/image/2h5dem.jpg?Expires=1699858080&amp;Signature=fZoHFt4gbasMjHMHylm2~7u~qOwRkLWWaQnYOr-IJob4FQ4mqf0cpUuibjE5kTVNN0bIghKBCkRRqvOlGs20bCk9QzM5Ti-q3It06s4WtTo-Pa1uFF0OWPYr3CTIm9jfSKplpKi0qQQBO8fPaB22cv5BEEKuKfr8AcTf6D6lC2WmjHtAUHb-FBS3FNSRWlwSR-lakFBxtSwLsrfPiSodVvrWj3bOh1uApTbwq5vPprvrmD0tlU~YIa~jDoI~trkd8r2KNcf~dwZj~TAisbjR2qBkgc~mXBPyQW5s-5hqoE72lh5Cu3UP1vKFpCdv101ovQidwo-dY58EOSYeQ1h0bA__&amp;Key-Pair-Id=APKAIEYUVEN4EVB2OKEQ" controls="">
            <source src="https://cdn-cf-east.streamable.com/video/mp4/2h5dem.mp4?Expires=1699858080&amp;Signature=DDZLPnY17W47ZFL6lFDFbafx93nEc5MfMjr1gxJly4i9-ej20fj-R7RDUyNzQOTRE~itN9Y9Yai6na0eW6R8huJd-F3vDaNYFayp2gFRZxeImKJXQR3MwjFBOI9gngw84vef4YCDDIyXtuqZ8AO2fbdjDLV9XkmG-SfE74mY8V5XaJ7tsY-~e62oRenfja6THFlcUooP428cSB0wSU5kPXGBG1U4vOWpGgEV8OMLW9JfqwIDZCyBZ5EzSxl6JyJY6lnToaGJ-F2qcBnPLfngdUAOdeOKXqPBI1BJUd1M3ZFmKSzLvpZLzK5xOQc~uvQQQH6WT7q14t1LejvhF0s1lg__&amp;Key-Pair-Id=APKAIEYUVEN4EVB2OKEQ">
          </video>

<p>The final one that should be loaded is <code>audio_projection.2.bias</code> which you can see bring printed to the screen when the server restarts <img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=12" title=":thinking:" class="emoji" alt=":thinking:" loading="lazy" width="20" height="20"></p>
Torch Geometric: Node Classification on Large Graphs

username: mhgh2hadi
cooked: <p>Hi all,</p>
<p>I am a newbee to the GNN world. I wanted to run some node classification over my graph. The issue is that the graph itself doesn’t fit on one GPU. Is there a way that we can run the classification on subgraphs so instead of loading the entire graph on GPU, I process a subgraph at each learning iteration? Something like batch-wise processing of images.</p>
<p>Thanks.</p>
Problem with transform.RandomAffine: unexpected argument 'fillcolor'

username: ysun
cooked: <p>When I am trying to use the torchvision.transforms.RandomAffine, I always got such an error:TypeError: transform() got an unexpected keyword argument ‘fillcolor’. Here is my code:</p>
<p>class RandomAffineTform(object):<br>
“”“Online affine transformation on patches: randomly generated transform matrix.”""</p>
<pre><code>def __call__(self, sample):
    patch1, patch2 = sample['patch1'], sample['patch2']
    patch1, patch2 = transforms.ToPILImage()(patch1), transforms.ToPILImage()(patch2)

    aff_tf = transforms.RandomAffine(degrees=5, scale=(0.9, 1.1), shear=5)

    return {'patch1': transforms.ToTensor()(aff_tf(patch1)),
            'patch2': transforms.ToTensor()(aff_tf(patch2))}
</code></pre>
<p>I have no idea how to debug it.</p>

username: ptrblck
cooked: <p>Which <code>torchvision</code> version are you using?<br>
This code seems to work and basically does, what your code should do:</p>
<pre><code class="lang-python">img = transforms.ToPILImage()(torch.randn(3, 224, 224))
transform = transforms.RandomAffine(degrees=5, scale=(0.9, 1.1), shear=5)
x = transforms.ToTensor()(transform(img))
</code></pre>

username: ysun
cooked: <p>Hi,</p>
<p>I am using torchvision-0.2.1</p>

username: ysun
cooked: <p>Thank you <span class="mention">@ptrblck_de</span>, the problem was just solved.</p>
<p>As I did not uninstall the previous version of torchvision when I upgrade to the latest version, the code was still using 2.0 instead of 2.1. The problem was solved when I simply deleted the old version. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=6" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: Kiran_Roy
cooked: <p>new torch version has the attribute renamed to ‘fill’ instead of ‘fillcolor’</p>
Is it possible to do int16 qat?

username: wgt
cooked: <p>Hi, all:</p>
<p>I am trying to quantize a siam-style tracking model, and unfortunately I found that with int8 qat, the resulted model didn’t work at all. I am not sure whether is my qat setup goes wrong or int8 is not enough for such task.</p>
<p>Since I only want a quantized backbone, the qat setup is like:</p>
<pre><code class="lang-auto">1.replace skip-connection "+" with nn.quantized.FloatFunctional()
2.coding fuse model method for structure like conv-bn-relu, then:
    if cfg.TRAIN.QAT:
        model.eval()
        model.backbone.fuse_model()
        model.backbone.qconfig = quantization.QConfig(weight=wt_fake_quant_per_channel_8bit, activation=act_fake_quant_8bit,)
        model.backbone = quantization.prepare_qat(model.backbone, inplace=True)
        if cfg.ADJUST.ADJUST:
            model.neck.apply(quantization.disable_fake_quant)
        if cfg.BAN.BAN:
            model.ban_head.apply(quantization.disable_fake_quant)
        model.backbone.features[-1].skip_add.activation_post_process.apply(quantization.disable_fake_quant) 
    model.train()
    # start training
    train(dist_model, optimizer, lr_scheduler, tb_writer)
</code></pre>
<p><em><strong>Is the code above has something wrong? I disable the last activation quantization because the followed layers are working on float32, is this step necessary?</strong></em></p>
<p><em><strong>If such code looks good, I guess it is just int8 is not enough, So, is it possible to do int 16 qat, I try something like</strong></em>:</p>
<pre><code class="lang-auto">act_fake_quant_16bit = quantization.FakeQuantize.with_args(
    observer=MinMaxObserver_Clip6.with_args(
        quant_min=-32767,
        quant_max=32767,
        dtype=torch.qint32, 
        qscheme=torch.per_tensor_symmetric,  
        reduce_range=False
    ),
    quant_min=-32767,
    quant_max=32767,
    dtype=torch.qint32,
    qscheme=torch.per_tensor_symmetric,
    reduce_range=False
)

wt_fake_quant_per_channel_16bit = quantization.FakeQuantize.with_args(
    observer=quantization.PerChannelMinMaxObserver.with_args(
        quant_min=-32767,
        quant_max=32767,
        dtype=torch.qint32,
        qscheme=torch.per_channel_symmetric,
        reduce_range=False
    ),
    quant_min=-32767,
    quant_max=32767,
    dtype=torch.qint32,
    qscheme=torch.per_channel_symmetric,
    reduce_range=False
)
</code></pre>
<p>but get errors from torch says that only int8/uint8 dtype is acceptable.</p>

username: jerryzh168
cooked: <p>the way you are using it is fine, but eager mode quantization API does not have good customization support. maybe you can try our new flow? <a href="https://pytorch.org/docs/main/quantization.html#prototype-pytorch-2-export-quantization" class="inline-onebox" rel="noopener nofollow ugc">Quantization — PyTorch main documentation</a></p>
How to fix the issue of "RuntimeError: Inplace update to inference tensor outside InferenceMode is not allowed."

username: lroberts
cooked: <p>Hi, I quantized an llama2 13B model using AutoAWq and I’m trying to run the model through vllm which is using torch distributed’s <code>all_reduce</code>. The details of the environment (torch/ray/vllm version), are at the end of the stack trace screenshot.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/d/ed21908df46f9a5a6f8785aa8346083338da2b0e.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/ed21908df46f9a5a6f8785aa8346083338da2b0e" title="Screen Shot 2023-11-09 at 6.03.04 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/d/ed21908df46f9a5a6f8785aa8346083338da2b0e_2_520x500.jpeg" alt="Screen Shot 2023-11-09 at 6.03.04 PM" data-base62-sha1="xPL4uxFVU86JuKpUBg0NsCcymlM" width="520" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/d/ed21908df46f9a5a6f8785aa8346083338da2b0e_2_520x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/e/d/ed21908df46f9a5a6f8785aa8346083338da2b0e_2_780x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/d/ed21908df46f9a5a6f8785aa8346083338da2b0e_2_1040x1000.jpeg 2x" data-dominant-color="0D190A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2023-11-09 at 6.03.04 PM</span><span class="informations">1339×1287 357 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The exception appears to be raised by this line<br>
<a href="https://github.com/pytorch/pytorch/blame/4e2e0437ea483a02aebefeab60e8870658990a5b/c10/core/TensorImpl.h#L370" class="onebox" target="_blank" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blame/4e2e0437ea483a02aebefeab60e8870658990a5b/c10/core/TensorImpl.h#L370</a></p>
<p>but I’m not sure if this is incorrect behavior and if so what would be the recommended way to fix?</p>
<p>The reason why I guess this is purely a torch distributed issue is b/c the code executes perfectly fine when I set the number of gpus to 1. If I use 2 gpus then I receive this error. I am running this on an 80Gb  A100 with 8 gpus.</p>
Save and Load cudnn.benchmark results to resume with speed increase

username: jakethecake
cooked: <p>Hello!</p>
<p>As i understand it “torch.backends.cudnn.benchmark” benchmarks multiple convolution algorithms during the first epoch to then uses the fastest during subsequent epochs. If i checkpoint my model and then resume it, cudnn has to rerun the benchmark again for the first epoch of the resumed run. Is there away to save the results from the benchmark in epoch 1 and then load that result when resuming training?</p>
<p>Theoretically since it’s continuing on the same model, shouldn’t the same convolution algorithms be the most optimal when resuming training?</p>

username: ptrblck
cooked: <p>No, it’s currently not possible.</p>

username: jakethecake
cooked: <p>Thank you for the response! I had a feeling that this was the case.</p>
Real time inference - Is it possible to pass a GPU pointer to PyTorch model?

username: gedeon_barnabe
cooked: <p><strong>Context:</strong> Realtime acquisition of images, processed on a GPU. PyTorch model performs segmentation on latest image. I want to avoid copies from device to cpu (save processed image) and copy from cpu to device for segmentation using Pytorch model.</p>
<p><strong>Question:</strong> Is there any way to pass a GPU pointer to PyTorch model ? I’d like to avoid creating a separate process looking for any new image in a folder and processing it. Not really elegant and too slow.</p>
<p><strong>Alternate question:</strong> Is there anyway to create a video stream from processed images ?</p>

username: ptrblck
cooked: <p>You should be able to use <code>torch::from_blob</code> as long as the GPU array is kept alive (otherwise <code>clone()</code> the tensor if you are going out of scope).</p>

username: gedeon_barnabe
cooked: <p>Thanks a lot for your answer, I’ll definitely look into that !</p>

username: Lisa_Koenigsberg
cooked: <p>deleted since I started a new topic</p>
If we sell a product with PyTorch and it's dependencies, do we need to make our code GPL?

username: Max_Zheng
cooked: <p>Hi! This question is not related to programming. I can’t get any clear answer from anywhere else so I’m trying my luck here.</p>
<p>I’m trying to figure out if we can sell a hardware product with PyTorch and it’s dependencies pre-installed without making our code GPL. Although PyTorch itself is not GPL, some packages PyTorch depends on are under GPL. And since we are selling a proprietary device (instead of a service), we are required to make our code GPL if we are using GPL software. Does anyone know if this is accurate? If so, is there anyway to get around it? How do all the hardware startups deal with this matter?</p>

username: ptrblck
cooked: <p>CC <a class="mention" href="/u/jspisak">@jspisak</a> for licensing questions.</p>

username: dirard
cooked: <p>I encountered this question too.<br>
Is this question solved?</p>

username: jspisak
cooked: <p>let me check with our legal folks and get back to you… <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

username: Shane_Wong
cooked: <p><a class="mention" href="/u/jspisak">@jspisak</a> Any update on this?</p>

username: jspisak
cooked: <p>Ok pinged legal again. I should have at least some guidance by tomorrow…</p>

username: jspisak
cooked: <p>Sorry for the delay. I can’t give legal advice here but here is what I can say.</p>
<p>When distributing open source software on hardware, you may be required to comply with various third-party license requirements, that could include relicensing your code base. Please consult your legal department for specific guidance.</p>
Audio Classifier code throwing error due not adapting to variable sizes

username: aus10powell
cooked: <p>I want the model to accept input tensors of shape [B, C, F, T], where B represents the batch size, C represents the number of channels, F represents the FFT frequency, and T represents the dynamic time dimension. The model returns output tensors of shape [B, 10] representing logits.</p>
<p>I feel like I’m 95% there but having trouble-debugging where I’m going wrong in adaptability of my layers to the B,C, F and T params?</p>
<p>Here is my forward method and layer definitions</p>
<pre><code class="lang-python">        self.conv1 = torch.nn.Conv2d(in_channels=2, out_channels=32, kernel_size=(3, 3), stride=1, padding=1)
        self.relu = torch.nn.ReLU()
        self.maxpool = torch.nn.MaxPool2d(kernel_size=(2, 2), stride=2)
        self.conv2 = torch.nn.Conv2d(in_channels=32, out_channels=64, kernel_size=(3, 3), stride=1, padding=1)
        self.flatten = torch.nn.Flatten()
        self.fc1 = torch.nn.Linear(32 * (n_fft // 4) * (sample_rate // (2 * fft_hop_length)), 128)
        self.fc2 = torch.nn.Linear(128, 10)  # 10 classes for digits 0-9

    def forward(self, input: torch.Tensor) -&gt; torch.Tensor:
        print("*"*50)
        B, C, F, T = input.size()  # Extract dimensions from the input tensor

        print("Input shape",input.shape)
        print("")
        x = self.maxpool(self.relu(self.conv1(input)))
        print("x1 shape",x.shape)
        x = self.maxpool(self.relu(self.conv2(x)))
        print("x2 shape",x.shape)

        # Calculate the flattened size
        flattened_size = 32 * (self.hparams.n_fft // 4) * (self.hparams.sample_rate // (2 * self.hparams.fft_hop_length)) # NOTE: Method 1
        print("F//2:",(F // 2), " T // 2:",(T//2),"new flatted:",C * (F // 2) * (T // 2))

        print("flatten_size",flattened_size," sample rate:",self.hparams.sample_rate)
        print("*"*50)
        x = x.view(B, flattened_size)
        print("after flattening",x.size())
        
        x = self.fc1(x)
        print("After fc1",x.size())
        x = self.relu(x)
        x = self.fc2(x)
        print("Final shape:",x.size())
        return x
</code></pre>
<p>Here is an example of what is working:</p>
<pre><code class="lang-python">Input shape torch.Size([1, 2, 513, 102])

x1 shape torch.Size([1, 32, 256, 51])
x2 shape torch.Size([1, 64, 128, 25])
F//2: 256  T // 2: 51 new flatted: 26112
flatten_size 204800  sample rate: 8000
**************************************************
after flattening torch.Size([1, 204800])
After fc1 torch.Size([1, 128])
Final shape: torch.Size([1, 10])
</code></pre>
<p>I’m able to load some of the data, but here is an example of where I’m stuck:</p>
<pre><code class="lang-python">Input shape torch.Size([2, 2, 513, 72])

x1 shape torch.Size([2, 32, 256, 36])
x2 shape torch.Size([2, 64, 128, 18])
F//2: 256  T // 2: 36 new flatted: 18432
flatten_size 409600  sample rate: 8000

    x = x.view(B, flattened_size)
RuntimeError: shape '[2, 409600]' is invalid for input of size 294912
</code></pre>
Is this good convergence or overfitting?

username: Potasiuombun
cooked: <p>Hello I am currently training a  regression model. Here is how my training and validation errors currently look<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/2/72748fa0bf31a278ec513ebcf8c130e87a7ef0c1.png" data-download-href="https://discuss.pytorch.org/uploads/default/72748fa0bf31a278ec513ebcf8c130e87a7ef0c1" title="goodtraining"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/2/72748fa0bf31a278ec513ebcf8c130e87a7ef0c1_2_690x381.png" alt="goodtraining" data-base62-sha1="gkwa4Jt0rPpYz4LWBkfCZeIVws1" width="690" height="381" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/2/72748fa0bf31a278ec513ebcf8c130e87a7ef0c1_2_690x381.png, https://discuss.pytorch.org/uploads/default/original/3X/7/2/72748fa0bf31a278ec513ebcf8c130e87a7ef0c1.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/7/2/72748fa0bf31a278ec513ebcf8c130e87a7ef0c1.png 2x" data-dominant-color="FAFAFA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">goodtraining</span><span class="informations">715×395 19.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Where the current epoch is<code> Train Loss: 0.0011180895 | Val Loss: 0.0012685484</code>. My question is, is this a case of overfitting or vanishing gradient or something else? both the training and the validation are going down just very slowly.</p>
Complex numbers and backward()

username: pytorch_user13
cooked: <p>I have complex data which I would like to use to train the network in the torch. I have managed to create a net and all tensors.</p>
<p>However, as soon as I start training, I get the following error: <code>grad can be implicitly created only for real scalar outputs but got the torch.complex64</code></p>
<p>This happens during the call: loss.backward()</p>
<p>I wonder, is it possible to use the torch to perform training with complex data? If so, does anybody have experience with what has to be adjusted?</p>
<p>I wonder, is it possible to use the torch to perform training with complex data? If so, does anybody have experience with what has to be adjusted? Or how to reformulate the problem?</p>

username: J_Johnson
cooked: <p>Judging by your error, your loss might be a non real value. So this may be an issue with your loss function. At the end of the day, you need to find some real positive value for the loss in order to do backprop. This is the case whether you are using real, complex, quaternions, etc. So you may need to look at your loss function and consider how to arrange it such that you get a real loss output.</p>
<p>Below is just one example that works:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

dummy_inputs = torch.rand((10, 15), dtype = torch.cfloat)

model = nn.Linear(15, 1, dtype = torch.cfloat)
optimizer = torch.optim.SGD(model.parameters(), lr = 0.001)

def criterion(x, y):
    dist = torch.sqrt((x**2+y**2))
    return torch.abs(torch.mean(dist))

optimizer.zero_grad()
outputs = model(dummy_inputs)
loss = criterion(outputs, torch.rand_like(outputs))
loss.backward()
optimizer.step()
</code></pre>
<p>By no means am I saying the above loss function is the ideal one for your case. You may wish to read some papers on the subject to determine what loss functions have been proposed in the past with some success, and then tailor the loss function around your needs.</p>

username: pytorch_user13
cooked: <p>thanks! that was indeed a problem.</p>
Use a different class in main class

username: Vaibhav_Chahar
cooked: <p>I have a class of single layer (below) of neural network (Neural_Made) going from one layer to another layer. I want to construct my whole network using this class as a building block. For that I used another class Made to build my whole network. But I am unable to figure out how to do it. Please provide a solution for the query.</p>
<pre><code>class Neural_Made(nn.Linear):
    def __init__(self,n,m,exclusive):
        super(Neural_Made,self).__init__(n*m*m*m*m,n*m*m*m*m)
        self.n=n
        self.m=m
        self.exclusive=exclusive #exclusive is a boolean
        self.in_size=n*m*m*m*m
        self.register_buffer('Mask1',torch.ones([self.in_size]*2))
        if self.exclusive:
            self.Mask1=1-torch.triu(self.Mask1)
        else:
            self.Mask1=torch.tril(self.Mask1)
        nn.init.xavier_uniform_(self.weight.data)

    def forward(self,x):
        return nn.functional.linear(x,self.Mask1*self.weight,self.bias)

class Made(nn.Module):  #This is the main class which is using the above network
 def __init__(self,n,m):
    super(Made,self).__init__()
    self.n=n
    self.m=m
    Neural_Made(self.n,self.m,exclusive=True)
 def forward(self,x):
</code></pre>

username: ptrblck
cooked: <p>You can use your custom <code>nn.Module</code> just like any other layer in PyTorch. I.e. initialize it in the parent’s <code>__init__</code> method and call into it in the <code>forward</code> method.</p>

username: Vaibhav_Chahar
cooked: <p>I don’t get it. Can you please explain it in little detail? It will be much help.</p>

username: ptrblck
cooked: <p>Take a look at <a href="https://pytorch.org/tutorials/beginner/blitz/neural_networks_tutorial.html#define-the-network">this example</a> to see how a custom neural network is created.<br>
There you can see the initialization of each layer in the <code>Net.__init__</code> method and how to use them in the <code>forward</code>.</p>
Multiple separate distinct return statements in a forward pass

username: Stephen_Condon
cooked: <p>Is it possible to have multiple distinct return paths in a forward pass? I’ve seen in other posts whenever there is branching logic, they will return multiple values from the same return, kind of like</p>
<p>def forward():<br>
…<br>
return value1, value2</p>
<p>Instead, I want to be able to have something like</p>
<p>def forward():<br>
…<br>
if condition:<br>
return value1<br>
… (potentially more layers)<br>
if condition:<br>
return value2<br>
…<br>
return value</p>
<p>Can the autograd engine support this? How would you set up your losses to know where you exited from and what layers to update? Apologies if this is documented somewhere but I haven’t been able to find it.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Stephen_Condon" data-post="1" data-topic="191548">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/stephen_condon/48/62604_2.png" class="avatar"> Stephen_Condon:</div>
<blockquote>
<p>Can the autograd engine support this?</p>
</blockquote>
</aside>
<p>Yes, Autograd won’t have a problem since it tracks which ops and parameters were used in the forward pass.</p>
<aside class="quote no-group" data-username="Stephen_Condon" data-post="1" data-topic="191548">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/stephen_condon/48/62604_2.png" class="avatar"> Stephen_Condon:</div>
<blockquote>
<p>How would you set up your losses to know where you exited from and what layers to update?</p>
</blockquote>
</aside>
<p>This is use case specific and you might be able to reuse the same condition from the <code>forward</code>.</p>
Pre trained Model download

username: Sugata_Bhar
cooked: <p>Can anyone please give me the pre-trained model model_196000.pth required for this project,</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/phkhanhtrinh23/translation_layoutrecovery/">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/phkhanhtrinh23/translation_layoutrecovery/" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/344;"><img src="https://opengraph.githubassets.com/f741ea612669c173db8549b51d099231f7f5637b82484a05c302e5509f320bd3/phkhanhtrinh23/translation_layoutrecovery" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/phkhanhtrinh23/translation_layoutrecovery/" target="_blank" rel="noopener nofollow ugc">GitHub - phkhanhtrinh23/translation_layoutrecovery: This a project to...</a></h3>

  <p>This a project to translate a .pdf file, preserving the original layout of that .pdf file. [UPDATED] We have achieved the Second Prize of the Cinnamon AI Bootcamp 2023. - GitHub - phkhanhtrinh23/tr...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>i mean pdf translation project</p>

username: ptrblck
cooked: <p>You might want to ask the author of the project in their repository for it.</p>
Where is the C++ source code for PyTorch for Model Parallel?

username: jrt-20
cooked: <pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.optim as optim


class ToyModel(nn.Module):
    def __init__(self):
        super(ToyModel, self).__init__()
        self.net1 = torch.nn.Linear(10, 10).to('cuda:0')
        self.relu = torch.nn.ReLU()
        self.net2 = torch.nn.Linear(10, 5).to('cuda:1')

    def forward(self, x):
        x = self.relu(self.net1(x.to('cuda:0')))
        return self.net2(x.to('cuda:1'))
</code></pre>
<p>Where is the c++ source code about  " to.(‘cuda:0’) "</p>

username: ptrblck
cooked: <p>I’m unsure if you are looking for the actual kernel, which can be found <a href="https://github.com/pytorch/pytorch/blob/28f43c767c0de48885f8de285ca4af4424d28f29/aten/src/ATen/native/cuda/Copy.cu#L165">here</a>, or the <code>libtorch</code> equivalent op which would be <code>tensor.to(at::Device(at::kCUDA))</code>.</p>

username: jrt-20
cooked: <p>Thank you very much for your answer. I want to know where is the c++ source code about this function.</p>
<pre><code class="lang-auto">    def to(self, *args, **kwargs):
        device, dtype, non_blocking, convert_to_format = torch._C._nn._parse_to(*args, **kwargs)

        if dtype is not None:
            if not (dtype.is_floating_point or dtype.is_complex):
                raise TypeError('nn.Module.to only accepts floating point or complex '
                                f'dtypes, but got desired dtype={dtype}')
            if dtype.is_complex:
                warnings.warn(
                    "Complex modules are a new feature under active development whose design may change, "
                    "and some modules might not work as expected when using complex tensors as parameters or buffers. "
                    "Please file an issue at https://github.com/pytorch/pytorch/issues/new?template=bug-report.yml "
                    "if a complex module does not work as expected.")

        def convert(t):
            if convert_to_format is not None and t.dim() in (4, 5):
                return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None,
                            non_blocking, memory_format=convert_to_format)
            return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)

        return self._apply(convert)
</code></pre>

username: jrt-20
cooked: <p>this function is in nn.module.py</p>

username: ptrblck
cooked: <p>The previously linked CUDA kernel will be called in this case.</p>
Migrating a PyTorch 1.2.0 Model to 2.1.0 - AttributeError in GRU Module

username: qrm_dev
cooked: <p>Hello PyTorch Community,</p>
<p>I’m attempting to migrate a model from PyTorch 1.2.0 to 2.1.0 as part of replicating the results from a paper called <a href="http://138.197.118.157:8000/segbot/" rel="noopener nofollow ugc">SEGBOT</a>. I’ve found a repository that has reproduced the paper: <a href="https://github.com/neural-machine-translation/DE-NMT/tree/master/Reproduce_Paper/SEGBOT" rel="noopener nofollow ugc">SEGBOT Repository</a></p>
<p>However, when I try to run the code with PyTorch 2.1.0, I encounter the following error:</p>
<pre><code class="lang-auto">AttributeError: 'GRU' object has no attribute '_flat_weights'. Did you mean: '_all_weights'?
</code></pre>
<p>This error appears during the loading of a pre-trained model using torch.load. Here’s the snippet of the code where the error occurs:</p>
<pre><code class="lang-auto">mymodel = torch.load('trained_model.torchsave', map_location=lambda storage, loc: storage)
</code></pre>
<p>From my understanding, this seems to relate to a change in the GRU module between versions. As I’m new to the field, I would greatly appreciate any guidance on how to address this issue to make the model compatible with the latest PyTorch version.</p>
<p>Thank you for your assistance.</p>
Is it possible to handle image size processing and modeling that can only be done with a 16GB VRAM GPU using DDP on two 8GB VRAM GPUs?

username: Keiku
cooked: <p>Sorry for the newbie question.</p>
<p>16GB VRAM GPU is expensive, so I’m thinking of purchasing 8GB VRAM GPU x2, but is it possible to use 8GB VRAM GPU x2 to handle image size processing and modeling that can only be done with 16GB VRAM GPU? Will it be out of memory even if I process it with DDP?</p>

username: ptrblck
cooked: <p>Hard to tell without knowing anything about the use case so try to lease a node with multiple GPUs, execute your DDP code, and profile the memory usage.</p>

username: Keiku
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thank you for your reply. I haven’t purchased it, so it’s hard to verify. In general, for example, is it possible to migrate large image sizes that can barely be processed by a 16GB VRAM GPU to two 8GB VRAM GPUs with simple CNN classification?</p>
RocksDB Dataset class for fast I/O (training)

username: Foivos_Diakogiannis
cooked: <p>Dear all,</p>
<p>in my research on remote sensing semantic segmentation I developed a RocksDB based dataset (inspired by the great work on LMDB that was used in some <a href="https://caffe2.ai/docs/distributed-training.html" rel="noopener nofollow ugc">caffe2 tutorials</a>) as an alternative to HDF5 and have found great usage (very fast I/O for deep learning training). Especially given that usually I/O was the bottleneck in DL workflows I develop. In my workflows the data are usually of the format input1, input2, …, inputn, labels1, labels2, …, labelsn where each of these are image-like numpy arrays (or time series of these). So that is the intended use, but the RocksDB framework is quite flexible, no need to be constrained by that.</p>
<p>You can find a short <a href="https://github.com/feevos/ssg2/tree/master/data" rel="noopener nofollow ugc">tutorial</a> of the <a href="https://github.com/feevos/ssg2/blob/master/data/rocksdbutils.py" rel="noopener nofollow ugc">implementation</a> and a basic comparison <a href="https://github.com/feevos/ssg2/blob/master/Notebooks/HDF5_vs_RocksDB.ipynb" rel="noopener nofollow ugc">notebook</a>  with HDF5. I ended up with RocksDB given in all things I was reading it was the fastest option (and indeed it proved to be).</p>
<p>Container environment to test it:<br>
<code>docker pull fdiakogiannis/trchprosthesis_requirements:23.10-py3</code></p>
<p>I hope this prove helpful to some of you.<br>
Cheers</p>
Torch gpu util 0% if previous run fails by error or stopped

username: ken.dbc.public
cooked: <p>My local env. is like below,</p>
<ul>
<li>Ubuntu20.04</li>
<li>Nvidia driver version: 525.89.02</li>
</ul>
<p>and I run torch with my docker container with source code mounted to some dir.<br>
My container env. is like below,</p>
<ul>
<li>Ubuntu20.04</li>
<li>torch: 2.0.1</li>
<li>cuda: 11.7</li>
</ul>
<p>The problem is that, it works fine if I run&amp;exit by code 0. Also, the GPU util maintains 99% almost always.<br>
But whenever I stop run(by clicking pycharm stop button) or run stops by an error, I can’t use GPU for the next runs.<br>
Runs occupy GPU memory fully, but util goes to 0% for all the time, and the training requires bunch of time as much as CPU training.</p>
<p>I tried the commands below according to other forum issue, but didn’t work.<br>
$ sudo rmmod nvidia_uvm<br>
$ sudo modprobe nvidia_uvm</p>
<p>Is there any other way to solve problem like this?<br>
Now I’m keep rebooting my PC whenever the symtom occurs…</p>
Multiclass dice loss for scene labeling problems

username: amaxwel6
cooked: <p>I modified the code from the kornia library <a href="https://github.com/kornia/kornia/blob/master/kornia/losses/dice.py" rel="noopener nofollow ugc">link</a> for the dice loss metric. This library is designed for semantic segmentation tasks and expects tensors in the shape [Batch, Class/Logit, Height, Width]. For scene labeling tasks, the expected shape is [Batch, Class/Logit]. So, I made a simple modification to the code: I removed dimensions with IDs 2 and 3 from the dims argument. For a macro-average, the goal is to calculate the metric separately for each class then take an average. This is accomplished by not specifying a dimension argument in torch.sum(). To obtain a micro-average, where all classes are pooled to calculate the total count of TPs and errors (TPs and TNs), the dimension argument is set to 1. To obtain a single metric in the case of macro averaging, the results are averaged using torch.mean().</p>
<p>The math and logic are implemented in a function, which is then called within a subclass of nn_module().</p>
<p>I believe that the function is working correctly but would appreciate any input or a double-check of the implementation. Is the metric being calculated correctly and are macro/micro averaging correctly defined? Any input is appreciated.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.nn.functional as F

def dice_loss(pred, ref, nCls, average, eps: float = 1e-8):
    # Input tensors will have shape (Batch, Class)
    # Dimension 0 = batch
    # Dimension 1 = class code or predicted logit
    
    # compute softmax over the classes axis to convert logits to probabilities
    pred_soft = torch.softmax(pred, dim=1)

    # create reference one hot tensors
    ref_one_hot = F.one_hot(ref, num_classes = nCls)

    #Calculate the dice loss
    if average == "micro":
      #Use dim=1 to aggregate results across all classes
      intersection = torch.sum(pred_soft * ref_one_hot, dim=1)
      cardinality = torch.sum(pred_soft + ref_one_hot, dim=1)
    else:
      #With no dim argument, will be calculated separately for each class
      intersection = torch.sum(pred_soft * ref_one_hot)
      cardinality = torch.sum(pred_soft + ref_one_hot)

    dice_score = 2.0 * intersection / (cardinality + eps)
    dice_loss = -dice_score + 1.0

    # reduce the loss across samples (and classes in case of `macro` averaging)
    dice_loss = torch.mean(dice_loss)

    return dice_loss


class DiceLoss(nn.Module):
    def __init__(self, nCls, average, eps: float = 1e-8) -&gt; None:
        super().__init__()
        self.nCls = nCls
        self.average = average
        self.eps = eps

    def forward(self, pred, ref):
        return dice_loss(pred, ref, self.nCls, self.average, self.eps)
</code></pre>
Keras Code to Pytorch (LSTM)

username: Zain_Ahmed
cooked: <p>Hello,<br>
I am currently working on multi task learning problem (MTL). I have a Keras code and want to convert it to Pytorch code.<br>
I noticed that the number of learnable parameters for an LSTM block in Pytorch is different from number of learnable parameters in Keras code. My input size is 29 while output size is 32.<br>
Using the link below, the number of learnable parameters for Keras comes out as</p>
<aside class="onebox stackexchange" data-onebox-src="https://stackoverflow.com/questions/38080035/how-to-calculate-the-number-of-parameters-of-an-lstm-network">
  <header class="source">

      <a href="https://stackoverflow.com/questions/38080035/how-to-calculate-the-number-of-parameters-of-an-lstm-network" target="_blank" rel="noopener nofollow ugc">stackoverflow.com</a>
  </header>

  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/2527680/arsenal-fanatic" target="_blank" rel="noopener nofollow ugc">
    <img alt="Arsenal Fanatic" src="https://i.stack.imgur.com/Y3HRd.png?s=256&amp;g=1" class="thumbnail onebox-avatar" width="256" height="256">
  </a>

<h4>
  <a href="https://stackoverflow.com/questions/38080035/how-to-calculate-the-number-of-parameters-of-an-lstm-network" target="_blank" rel="noopener nofollow ugc">How to calculate the number of parameters of an LSTM network?</a>
</h4>

<div class="tags">
  <strong>machine-learning, neural-network, deep-learning, keras</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/2527680/arsenal-fanatic" target="_blank" rel="noopener nofollow ugc">
    Arsenal Fanatic
  </a>
  on <a href="https://stackoverflow.com/questions/38080035/how-to-calculate-the-number-of-parameters-of-an-lstm-network" target="_blank" rel="noopener nofollow ugc">03:13PM - 28 Jun 16 UTC</a>
</div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p><strong>params = 4 * ((size_of_input + 1) * size_of_output + size_of_output^2) = 7936</strong></p>
<p>While using the Pytorch documentation, the learnable parameters come out to be 8064. I got it by adding all the learnable weights.</p>
<p><a href="https://pytorch.org/docs/stable/generated/torch.nn.LSTM.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.nn.LSTM.html</a></p>
<p>Am I mistaken or am I missing something?</p>
<p>Finally, is equalizing the number of learnable parameters for the two models a good metric to ensure that the graph is same for both?</p>