akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to reproduce result

username: everlastin14
cooked: <p>I’m trying to run a easy deep learning model on embedding system without any framework</p>
<p>I have trained below model with my own dataset and quantize to int8</p>
<pre><code class="lang-auto">class Net(torch.nn.Module):
    def __init__(self, n_feature, n_hidden, n_output, quant=False):
        super(Net, self).__init__()
        self.fc1 = torch.nn.Linear(n_feature, n_hidden, bias=False)
        self.fc2 = torch.nn.Linear(n_hidden, n_output, bias=False)
        self.relu = torch.nn.ReLU()
        self.quant = quant
        if self.quant:
            self.quant = torch.quantization.QuantStub()
            self.dequant = torch.quantization.DeQuantStub()

    def forward(self, input):
        if self.quant:
            x = self.quant(input)
        else:
            x = input
        x = self.fc1(x)
        x = self.relu(x)
        x = self.fc2(x)
        if self.quant:
            x = self.dequant(x)

        return x
</code></pre>
<p>I can get each layer’s int8 weight as fc1.weight().int_repr()<br>
But how to use these parameter to reproduce result like net.forward()?</p>

username: jerryzh168
cooked: <p>Do you mean how to run the quantized pytorch model on your embedding system?</p>
Quantizaton of EfficientNet Models

username: anonymous1
cooked: <p>Hi,<br>
I performed the quantization technique on efficient net models by referring post-training static quantization method in PyTorch blogs. But I was only able to bring a reduction only by 5 MB.<br>
Also, I wasn’t able to perform the layer fusion step on the prebuilt layers of this model while quantizing using the existing PyTorch techniques. How do I approach this problem? Or is there an alternative method to bring down the size of the model without affecting its accuracy much?<br>
Can someone help me with this?</p>
<p>Thanks in advance!</p>

username: jerryzh168
cooked: <p>It might be related to fusion, why can’t you do fusion?</p>
Quantization of T5-small

username: zraeesy
cooked: <p>Hi all, I have applied dynamic quantization on my T5-small model (that I have built using AllenNLP framework). However, I see inconsistent latency numbers on different hardware on the same set/batch-size. On an m5 instance, I observe 10-15% speed up, while on c5.9xlarge, I actually see an increase in latency compared to the unquantized version. I chose dynamic quantization as I was expecting it to more or less work out-of-the-box.</p>
<p>Has anyone run into this issue before?</p>

username: jerryzh168
cooked: <p>on servers, we are using fbgemm, which is using intel SIMD intrinsics to speedup the runtime I think. cc <a class="mention" href="/u/dskhudia">@dskhudia</a> do we know if fbgemm always gives a speedup in aws machines?</p>
How different data types represent in Pytorch

username: Forceless
cooked: <p>Hi,<br>
I recently work on analyzing a model’s reliability and I wonder how qint8 data like quantized model’s weights are stored in a computer.<br>
Do these data types store in True form or Complement form<br>
Thank you</p>

username: Forceless
cooked: <p>I’m not good at English, so my word may have difference with what I want to present.<br>
But I’m glad and will soon reply if you remark<br>
When I say True form, It’s signed binary numbers like:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/7/6/76593cfcf84b8d2f68000e8cb7959bc05f533a97.png" alt="image" data-base62-sha1="gSXx4XQcO2Nqb58P4G4Viai3n1R" width="192" height="105"><br>
And does it save in this form or one’s complement or two’s complement</p>

username: jerryzh168
cooked: <p>qint8 is using (signed int8) underneath, so it will be two’s complement I think</p>
QAT: precision drop to 0

username: ynjiun_wang
cooked: <p>Hi,</p>
<p>I have a pytorch trained model that reached a testing precision of 90+ %.<br>
Then I run QAT for this model and converted it to quantizied_model. Then I tested the quantized_model with the same test dataset but the precision drop to 0%.</p>
<p>The framework for my QAT is as below:</p>
<pre><code class="lang-auto">	model = create_model(num_classes=num_classes)

    train_loader, test_loader = prepare_dataloader(config, num_workers=8, train_batch_size=128, eval_batch_size=256)

    # Load a pretrained model.
    model = load_model(model=model, model_filepath=model_filepath, device=cuda_device)

    # Move the model to CPU since static quantization does not support CUDA currently.
    model.to(cpu_device)

    # Make a copy of the model for layer fusion
    fused_model = copy.deepcopy(model)

    model.train()

    fused_model.train()

    # Fuse the model in place rather manually.
    
    for module_name, module in fused_model.named_children():
        if "extras" in module_name:
            for basic_block_name, basic_block in module.named_children():
                torch.quantization.fuse_modules(basic_block, [["0", "1", "2"]], inplace=True)
                for sub_block_name, sub_block in basic_block.named_children():
                    if sub_block_name == "Sequential":
                        torch.quantization.fuse_modules(sub_block, [["0", "1", "2"]], inplace=True)


    # Model and fused model should be equivalent.
    model.eval()
    fused_model.eval()
    assert model_equivalence(model_1=model, model_2=fused_model, device=cpu_device, rtol=1e-03, atol=1e-06, num_tests=100, input_size=(1,3,32,32)), "Fused model is not equivalent to the original model!"

    quantized_model = QuantizedRSSD(model_fp32=fused_model)
    
    quantization_config = torch.quantization.get_default_qconfig("fbgemm")
    
    quantized_model.qconfig = quantization_config
    
    prepared = torch.quantization.prepare_qat(quantized_model, inplace=True)

    prepared.apply(torch.quantization.enable_observer)
    # # Use training data for calibration.
    if not os.path.exists(quantized_model_filepath):
        print("Training QAT Model...")
        quantized_model.train()
        
        prepared.apply(torch.quantization.enable_fake_quant)
        
        train_model(model=prepared, config=config, model_filename=quantized_model_filename, 
                    train_loader=train_loader, test_loader=test_loader, device=cuda_device, 
                    learning_rate=1e-2, num_epochs=3)
    
    prepared.to(cpu_device)

    quantized_model = torch.quantization.convert(prepared, inplace=True)
    
    quantized_model.eval()
   
    if not os.path.exists(quantized_model_filepath):
        save_model(model=quantized_model, model_dir=model_dir, model_filename=quantized_model_filename)
</code></pre>
<p>What am I missing in my QAT causing the precision drop to 0? Please help. Thanks.</p>

username: jerryzh168
cooked: <p>QAT needs careful tuning, my guess is your learning rate might be too large here</p>
Quantization aware training lower than 8-bits?

username: Ricky_96
cooked: <p>Hello. I am not an expert of PyTorch, however I need to quantize my model to less than 8 bits (e.g. 4-bits, 2-bits etc.). I’ve seen that PyTorch actually does not officially support this “aggressive” quantization. Is there any way to do this? I’m asking you if there is some sort of documentation with steps to follow (or something like that) because as I’ve said I’m not an expert. Plus, I don’t need to only evaluate the accuracy of the quantized model, but also compress the model in a way that I will be able to deploy it on a controller or a mobile. Thanks in advance to whoever tries to help me.</p>

username: jerryzh168
cooked: <p><a class="mention" href="/u/supriyar">@supriyar</a> added quint4x2 datatype to quantization, some tests can be found in <a href="https://github.com/pytorch/pytorch/blob/master/test/quantization/test_quantize_fx.py#L3734" class="inline-onebox" rel="noopener nofollow ugc">pytorch/test_quantize_fx.py at master · pytorch/pytorch · GitHub</a>, but we do have have int4 datatype support for any ops except embedding_bag.</p>
<p>We are working on a design doc to support custom backends currently and I think it will be useful to support this as well, please keep an eye out for design docs in github issues, I may update here if I still remember this post.</p>

username: willdla
cooked: <p>its 2021-11-03 now<br>
any update for now?</p>

username: jerryzh168
cooked: <p>what dtype do you need? we only have quint4x2 currently but contributions are welcome. We don’t have other use cases right now so it might not make much sense for us to add other types currently.</p>
Slicing cuda tensor

username: MeiMei_Liu
cooked: <p>Hi everyone, I have a doubt about slicing on cuda tensor (Pytorch 1.10 - GPU RTX 3070). For example, following code cant learning</p>
<pre><code class="lang-auto"># labels for multi-outputs  
for epoch in range(epochs):
   for features, labels in train_loader:
      features = features.to(device)
      labels = labels.to(device)
      preds = model(x)
      loss = cross_entropy(preds, labels[:,0])
      optim.zero_grad()
      loss.backward()
      optim.step()
</code></pre>
<p>But, if I use :</p>
<pre><code class="lang-auto"> loss = cross_entropy(preds, labels[:,0].contiguous())
</code></pre>
<p>it works!, the model starts learning. What is the difference or its a bug?. (Sorry for my bad english).</p>
<p>Thanks in advance!</p>

username: ptrblck
cooked: <p>Could you post an executable code snippet, which would show the failure to learn in the first approach and the successful training in the second?<br>
Have you measured the mean +/- stddev accuracy for both approaches? If so, could you post them here as well, please?</p>

username: MeiMei_Liu
cooked: <p>I share the following link to Colab environment (pytorch1.10 Nvidia k80): <a href="https://colab.research.google.com/drive/15VNZZP1uPdrY4hH0cYbBYdVT_nxWbJVa?usp=sharing" rel="noopener nofollow ugc">https://colab.research.google.com/drive/15VNZZP1uPdrY4hH0cYbBYdVT_nxWbJVa?usp=sharing</a></p>
<p>you can see the logs with the first and second approach. In the first approach the accuracy is stagnant, however in the second approach, it got improved with the epochs. (in the same way I attach a script to run <a href="https://drive.google.com/file/d/1Fb4qc_ksMp-8jeWrb0H8SIDDdiFpwBQ5/view?usp=sharing" rel="noopener nofollow ugc">https://drive.google.com/file/d/1Fb4qc_ksMp-8jeWrb0H8SIDDdiFpwBQ5/view?usp=sharing</a>).</p>
<p>Thanks!</p>

username: ptrblck
cooked: <p>Thanks for the notebook! I wasn’t able to run it on Colab for unknown reasons (the runtime just stopped), but was able to reproduce the issue locally in PyTorch <code>1.10.0</code>. It seems to be fixed in the current master/nightly and works as expected with <code>1.11.0.dev20211101+cu113</code>.<br>
I couldn’t quickly find a related PR for the issue, but will check a bit more what might have been the issue in <code>1.10</code>.</p>
Inconsistency with how other scripted functions and forward handle member values

username: Prashant
cooked: <p>Hi I’m trying to understand why I’m allowed to use a member value like this in forward but not in a custom scripted function.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

class TestClass(nn.Module):
    def __init__(self, y): 
        self.y = y 
        super().__init__()

    def forward(self, x): 
        if self.y &gt; 0:
            return torch.zeros([1,1])
        else:
            return torch.ones([1,1])

#    @torch.jit.script
    def test(self, x): 
        if self.y &gt; 0:
            return torch.zeros([1,1])
        else:
            return torch.ones([1,1])

tester = TestClass(1)

torch.jit.script(tester)```</code></pre>
Multiprocess function locally with rpc?

username: ZimoNitrome
cooked: <p>I noticed that a post-processing step in my model takes more than twice the time of the actual NN part. This post-processing involves sorting the outputs in each sample (sample by sample in a batch of 32+).</p>
<p>The sorting package is provided as “torchsort” and is already optimized as a C++ torchscript.<br>
I thus tried to multiprocess each of the sample sortings in a batch using <code>torch.multiprocessing</code>, but of course receive the warning that: <code>[...] autograd does not support crossing process boundaries.</code>.</p>
<p>So I wonder if I can “fake” distributed training locally somehow with rpc?</p>
<p>Would that be a good choice to solve my speed problems?</p>
<p>Could it be done easily?</p>
Training Pre-Trained Retinanet for Different Image Sizes

username: Laya1
cooked: <p>I have a dataset of images that are of size 1000 x 1000 x 3 and I am interested in detecting only one class of images. The default input size of Resnet50-based Retinanet is much smaller than this. What changes should be done to the pre-trained model to adapt to the large size of images? I have tried making the following changes:</p>
<pre><code class="lang-auto">model = torchvision.models.detection.retinanet_resnet50_fpn(pretrained=False, pretrained_backbone=True, num_classes=2)
model.transform = torchvision.models.detection.transform.GeneralizedRCNNTransform(min_size=1000, max_size=1000, image_mean=[0.485, 0.456, 0.406], image_std=[0.229, 0.224, 0.225])
</code></pre>
<p>But the model provides empty bounding boxes before and after training.</p>
<p><strong>Update</strong>: I have tried different combinations and found that setting <code>pre_trained=False</code> or <code>num_classes=2</code> in the model results in empty bounding boxes in the output. Can anyone clarify why this is happening and what can be done to correct this?</p>
Yet another question on DistributedDataParallel and gradient averaging

username: vadimkantorov
cooked: <p>Do I understand correctly that DP will not average gradients of “replica” batches, while DDP will?</p>
<p>Is it always the case that gradient averaging is correct (assuming the loss averages over the “replica” batchsize)? If yes, is it the case because of the product operation  in the chain rule?</p>
<p>Thanks!</p>

username: Yanli_Zhao
cooked: <p>right, DDP averages gradients in default, as it assumes batch size is the same for each rank. If the batch size is not the same for each rank, or the loss function does not expect averaged gradients, you can call ’ <strong>register_comm_hook</strong>()’ on the DDP model, define your own communication strategy (sum, or other ops)</p>
How to do 1d interpolation, like np.interp?

username: pyscorcher
cooked: <p>So far I’ve only found references to <code>torch.nn.functional.interpolate</code> when searching for interpolation functions, but that is intended for interpolation of structured data. Is there something like <code>numpy.interp</code> or <code>scipy.interpolate.interp1d</code> available where I can specify <code>x,y</code> points and then given some <code>x</code> get a corresponding interpolated <code>y</code> value?</p>

username: pyscorcher
cooked: <p>It appears no such function is implemented as of yet: <a href="https://github.com/pytorch/pytorch/issues/50341" class="inline-onebox" rel="noopener nofollow ugc">Interpolation tracking issue · Issue #50341 · pytorch/pytorch · GitHub</a></p>
LSTM Time series prediction delayed (and other problems)

username: NSilviaS15
cooked: <p>Currently, I am doing a project on using a LSTM model to predict the price of cryptocurrencies for trading.<br>
After reading many of the online tutorials and looking through the forum as well, I couldn’t figure out how to produce better results. I am almost certain that I have made mistakes somewhere, but after referencing other people’s code and increasing / decreasing hidden size, layer count, tried to give the model more data / less data, normalized the data before splitting, splitting the data before normalizing, rewriting code for almost two months with no success, I have no clue on what to do next to do.</p>
<p>The latest copy of my code is in</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://colab.research.google.com/drive/1k84zmrlbA0o0pvpZtaAEf7shW57SYAHy">
  <header class="source">
      <img src="https://colab.research.google.com/img/favicon.ico?vrz=colab-20211103-060057-RC00_407299128" class="site-icon" width="128" height="128">

      <a href="https://colab.research.google.com/drive/1k84zmrlbA0o0pvpZtaAEf7shW57SYAHy" target="_blank" rel="noopener nofollow ugc">colab.research.google.com</a>
  </header>

  <article class="onebox-body">
    <img src="https://colab.research.google.com/img/colab_favicon_256px.png" class="thumbnail onebox-avatar" width="260" height="260">

<h3><a href="https://colab.research.google.com/drive/1k84zmrlbA0o0pvpZtaAEf7shW57SYAHy" target="_blank" rel="noopener nofollow ugc">Google Colaboratory</a></h3>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Right now, I have identified a few major problems with my code.</p>
<ol>
<li>No matter how far advanced I set my prediction, the LSTM outputs a similar curve delayed by the same amount. I have read that this is because the model decides to just copy the data over, but I can’t find a way to solve this.</li>
<li>If I increase the seq_len more, GPU memory runs out. I did try to use DataLoaders, but those seemed to be much slower and produce worse results for me than just inputting data directly. Is it because of my implementation of the code?</li>
</ol>
<p>Please help, and if there are any other resources I should look at please let me know as well, thanks a lot.</p>
How to extract the output at the end of each TransformerDecoderLayer layer in the TransformerDecoderer

username: seyeeet
cooked: <p>I am using a <code>TransformerDecoder</code> with 3 <code>TransformerDecoderLayer</code>. I was wondering if there is a way to get the output of each of the <code>TransformerDecoderLayer</code>?<br>
basically:<br>
<code>decoder = nn.TransformerDecoder(nn.TransformerDecoderLayer(d_model=64, nhead=8), num_layers=3)</code><br>
woudl it be possible to get the output of each intermediate layer (i.e. the output of first TransformerDecoderLayer, second TransformerDecoderLayer, and 3rd TransformerDecoderLayer which is equal to the output) and use it in the another model during the training?</p>

username: JuanFMontesinos
cooked: <p>You can use hooks:<br>
<a href="https://pytorch.org/docs/master/generated/torch.nn.Module.html?highlight=hook#torch.nn.Module.register_forward_hook" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/master/generated/torch.nn.Module.html?highlight=hook#torch.nn.Module.register_forward_hook</a><br>
(look for a good tutorial by googling)</p>
<p>Similarly, you ahve this new feature</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://pytorch.org/blog/FX-feature-extraction-torchvision/">
  <header class="source">
      <img src="https://pytorch.org/favicon.ico?" class="site-icon" width="32" height="32">

      <a href="https://pytorch.org/blog/FX-feature-extraction-torchvision/" target="_blank" rel="noopener nofollow ugc">pytorch.org</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/314;"><img src="https://pytorch.org/assets/images/fx-image2.png" class="thumbnail" width="690" height="314"></div>

<h3><a href="https://pytorch.org/blog/FX-feature-extraction-torchvision/" target="_blank" rel="noopener nofollow ugc">Feature Extraction in TorchVision using Torch FX</a></h3>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
amp_C fused kernels unavailable

username: divinho
cooked: <p>Geting this with pytorch 1.8. It also says</p>
<blockquote>
<p>you may get better performance by installing NVIDIA’s apex</p>
</blockquote>
<p>But this confuses me because I thought apex was integrated into pytorch nowadays?</p>
<p>Could I fix this problem just by upgrading pytorch?</p>

username: tom
cooked: <p>It would be good if you could give more context. This is from fairseq?</p>
<p>The mixed precision parts have been merged (though PyTorch 1.8 is so last year… <img src="https://discuss.pytorch.org/images/emoji/apple/stuck_out_tongue.png?v=10" title=":stuck_out_tongue:" class="emoji" alt=":stuck_out_tongue:"> ), but apex also has some unrelated optimizations, including kernels for operating on multiple tensors (I think).<br>
The question always is whether these optimizations are worth having in PyTorch (which has &gt;2000 functions last someone counted, which is quite a burden at times) or whether they’re only when you want to race implementations against each other.</p>
<p>You could install apex and compare performance before and after to find out. If it is a large difference, there might be an argument for getting these into PyTorch. If it is just to squeeze out the large percent of performance, maybe not as much.</p>
<p>Best regards</p>
<p>Thomas</p>

username: divinho
cooked: <p>Thanks for the reply!</p>
<p>Yeah it’s fairseq; wav2vec pretraining. Trying to speed it up.</p>
<p>Stupid question: If I install apex I presume I then have to change the code to use that and not pytorch’s amp package?</p>

username: tom
cooked: <p>As far as I understand, the AMP (now from PyTorch) and the fused kernels (still from apex) are largely independent even if the module is called amp_C, so you would only need to install apex.<br>
In general my impression was that fairseq is relatively well maintained and the people have good access to PyTorch know-how, so their code would do the right thing.</p>
<p>Best regards</p>
<p>Thomas</p>
How to create a torch.nn.Module model Class file?

username: iavinas
cooked: <p>I have a detecton2 model. For inference API we were using flask but now switching to Kfserving. Kfserving required a saved model file and the class name to load the model.</p>
<p>Previously I use this code snippet to load the model.</p>
<pre><code class="lang-auto">cfg = get_cfg()
cfg.merge_from_file(model_zoo.get_config_file("COCO-InstanceSegmentation/mask_rcnn_X_101_32x8d_FPN_3x.yaml"))
cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.8  # set threshold for this model
cfg.MODEL.ROI_HEADS.NMS_THRESH_TEST = 0.01
cfg.MODEL.ROI_HEADS.NUM_CLASSES = 1
from detectron2.modeling import build_model
from detectron2.checkpoint import DetectionCheckpointer
model = build_model(cfg)
DetectionCheckpointer(model).load("model_final.pth")
model.train(False)
</code></pre>
<p>but Now I require something like this.</p>
<pre><code class="lang-auto">class TwoLayerNet(torch.nn.Module):
    def __init__(self, D_in, H, D_out):
        
    def forward(self, x):
        
</code></pre>

username: ASharma
cooked: <p>Were you able to figure it out?</p>

username: theofpa
cooked: <p>KFServing (now named as KServe), has moved it’s support for PyTorch from the custom based model server to TorchServe. You can find more information about deploying a PyTorch model archive in Kserve <a href="https://kserve.github.io/website/modelserving/v1beta1/torchserve/" rel="noopener nofollow ugc">here</a>, and information about storing your model in MAR format <a href="https://github.com/pytorch/serve/blob/master/model-archiver/README.md" rel="noopener nofollow ugc">here</a>.</p>
Do I need to save the state_dict oof GradScaler?

username: pytorch2
cooked: <p>Do I need to save the state_dict of torch.cuda.amp.GradScaler and reload it to resume training? The docs say it dynamically estimates the scale factor each iteration, so I never saved it. So, will model.load_state_dict and optimizer.load_state_dict suffice?</p>

username: ptrblck
cooked: <p>If you want to restore the last scale factor (as well as the backoff and growth factor, if changed), then you should restore its <code>state_dict</code>.<br>
Your training should also work without restoring the gradient scaler, but will most likely not reproduce the same results as a run without interruptions, as the new gradient scaler could skip iterations at different steps.</p>

username: jefflee
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> Recently I encountered an interesting thing. When I trained my model, I didn’t save states of GradScaler, but saved optimizer states. Later I found that if I resumed training by loading the checkpoint, the performance was <strong>higher</strong> than continuing the training. The only difference I spot is that the GradScaler is newly created in the resumed session. So is it possible that if we recreate/reset GradScaler at each epoch, the performance would be actually higher than using the old GradScaler? Thanks.</p>

username: ptrblck
cooked: <p>I wouldn’t expect to see a difference in performance while using a new <code>GradScaler</code>.<br>
In case you are recreating a new <code>GradScaler</code>, the first iteration(s) might have been skipped due to a high scaling factor, so your runs might be a bit different. You could thus check, if resuming the training in FP32 (i.e. without the <code>GradScaler) and if resuming with the </code>GradScaler` + skipping the same number of update steps would also yield a different performance.</p>

username: jefflee
cooked: <p>Wow you replied so quickly. Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a> ! What if the first few iterations are not skipped, can I say that using a new scaler is equivalent to increasing the learning rate in the first few iterations? Seems there’s no warning message saying the gradients are scaled in the first few iterations…</p>

username: ptrblck
cooked: <p>No, the <code>GradScaler</code> would scale the loss, thus also the gradients, but the gradients would be unscaled before the <code>optimizer.step()</code> operation is performed. The loss scaling factor also has a <code>growth_factor</code> and <code>growth_interval</code> which would increase it after <code>growth_interval</code> steps were performed without a decrease by <code>growth_factor</code>. In case the scaling factor is low right before saving the checkpoint it could potentially benefit from a large factor again.</p>
Same functional.conv2d in for loop become extremely slow

username: 80Ultraman
cooked: <p>I use <strong>functional.conv2d</strong> to do cross-correlation operations. But I found the same function become unacceptably slow after calling it multiple times.</p>
<p>For test and simplicity, I created zero tensors the same size as my data. And run the exact same function in a loop.</p>
<p>The former hundred was done instantly. However, after 500-ish times it takes about 0.3s.</p>
<p>I noticed that the GPU is still 100% usage after the finish of the code and GPU memory wasn’t run out.</p>
<p>I tried to run this function in <strong>nn.Module</strong> according to <a href="https://discuss.pytorch.org/t/setting-custom-kernel-for-cnn-in-pytorch/27176">this post</a>. But the same problem stays.</p>
<p>Could anyone help me with this?</p>
<pre><code class="lang-python">import torch
import datetime

xa = torch.zeros(1, 1, 1785, 1785, device=torch.device('cuda:0'))
kernel = torch.zeros(1, 1, 129, 129, device=torch.device('cuda:0'))

for i in range(1000):
    start = datetime.datetime.now()
    
    torch.nn.functional.conv2d(xa, kerel, padding='same')

    end = datetime.datetime.now()
    print((end - start))
</code></pre>

username: tom
cooked: <p>Two quick comments:</p>
<ul>
<li>The code does not take the async nature of CUDA into account, this is also the cause of your observation about GPU utilization. You would need to add <code>torch.cuda.synchronize()</code> before every time taking to make sure you measure the time when all preceding GPU work has been done.</li>
<li>It is preferable to use a monotonic clock (<code>time.perf_counter()</code>) for performance measurements,</li>
</ul>
<p>Best regards</p>
<p>Thomas</p>
How to update network parameters for two sub networks

username: sz19980502
cooked: <p>we are setting by input=x<br>
we have two sub networks, net1, net2<br>
y1=net1(x)<br>
y2=net2(x)<br>
we have loss function L=|y1*y2-x|+|x/y2-y1|,where loss function is L1loss</p>
Dependent dll library 

username: feng1
cooked: <p>Libtorch is used for inference under C++, and both Release and Debug must be supported at the same time, but I found that the dlls that depend on the two modes have the same name. When I add the directory of the dependent library to the environment variable, the exe cannot distinguish between Release and Debug when looking for the dependent dll, which causes the program to crash. How can I solve it? The exe cannot be placed in the same directory as the dependent dll, it is best if it can be assigned to a fixed directory.</p>
Dataset with many large .npz file but limited memory

username: brianw0924
cooked: <p>I have many .npz dataset (images and labels), each is very large, so I can’t load all of them to a dataloader (only 32 GB memory)</p>
<p>since there are multiple npz files, and each have different length (maximum 5000)</p>
<p>what is the best implement way to dynamically load the data?</p>
<p>I only come up with the dumbest way, just store a list of .npz file name , and iterate through it, and then build the dataloader each iteration<br>
e.g.<br>
list = [file1, file2, file3, file4, …]<br>
for file in list:<br>
dataset = …<br>
dataloader = …<br>
training …<br>
del dataset, dataloader</p>
<p>but that result in another problem is that the npz file is so big (roughly 1~3GB), so every time I load the npz file will take huge amount of time.</p>
<p>is there a best solution for this scenario?</p>

username: Scott_Hoang
cooked: <p>So each image of your data is already very large?</p>

username: brianw0924
cooked: <p>Many labels are images too (segmentation)</p>
<p>so each of the .npz is very large</p>

username: Scott_Hoang
cooked: <p>Can you store your segmentation label at a different scale? Or pre-emptively resize all your data to a smaller resolution?</p>

username: brianw0924
cooked: <p>thanks, I would try this solution. but still figuring out the best solution :’(</p>
<p>I’m currently just iterate through the npz files and build dataloader dynamically while training</p>
Can help, I am new with pytorch. What's wrong here

username: Fa_Faj
cooked: <p>Traceback (most recent call last):<br>
File “train.py”, line 51, in <br>
for i, data in enumerate(train_dataset):<br>
File “/content/gdrive/My Drive/SNE-RoadSeg/data/<strong>init</strong>.py”, line 71, in <strong>iter</strong><br>
for i, data in enumerate(self.dataloader):<br>
File “/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py”, line 521, in <strong>next</strong><br>
data = self._next_data()<br>
File “/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py”, line 1203, in _next_data<br>
return self._process_data(data)<br>
File “/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py”, line 1229, in _process_data<br>
data.reraise()<br>
File “/usr/local/lib/python3.7/dist-packages/torch/_utils.py”, line 425, in reraise<br>
raise self.exc_type(msg)<br>
cv2.error: Caught error in DataLoader worker process 0.<br>
Original Traceback (most recent call last):<br>
File “/usr/local/lib/python3.7/dist-packages/torch/utils/data/_utils/worker.py”, line 287, in _worker_loop<br>
data = fetcher.fetch(index)<br>
File “/usr/local/lib/python3.7/dist-packages/torch/utils/data/_utils/fetch.py”, line 44, in fetch<br>
data = [self.dataset[idx] for idx in possibly_batched_index]<br>
File “/usr/local/lib/python3.7/dist-packages/torch/utils/data/_utils/fetch.py”, line 44, in <br>
data = [self.dataset[idx] for idx in possibly_batched_index]<br>
File “/content/gdrive/My Drive/SNE-RoadSeg/data/R2D_dataset.py”, line 38, in <strong>getitem</strong><br>
rgb_image = cv2.cvtColor(cv2.imread(os.path.join(useDir, ‘rgb’, name)), cv2.COLOR_BGR2RGB)<br>
cv2.error: OpenCV(4.1.2) /io/opencv/modules/imgproc/src/color.cpp:182: error: (-215:Assertion failed) !_src.empty() in function ‘cvtColor’</p>

username: ptrblck
cooked: <p>The error is raised by OpenCV and based on the error message it seems the <code>imread</code> is failing and might be returning an empty array so you should check the path and make sure <code>cv2</code> can properly read the image.</p>
How to freeze the input layer and output fc Layer when during training

username: watertreader
cooked: <p>Hi</p>
<p>I am pretty new to this forum and pyTorch. Basically I am trying to use EfficientNet as a classifier to detect emotion in FER2013 database. Basically I have needs to modify the input (need it to be 1 channel , 48x48 for image size)  and the output (need to be 7). Here is my modification</p>
<p>class Classifier(<a href="http://nn.Net" rel="noopener nofollow ugc">nn.Net</a>):<br>
def <strong>init</strong>(self,n_classes):</p>
<pre><code>    super(Classifier, self).__init__()
    
    self.effnet                  =  EfficientNet.from_pretrained('efficientnet-b0')
    self.effnet._fc.out_features = 7
    self.effnet._conv_stem = Conv2dStaticSamePadding(1,32,kernel_size=(3,3), stride=(2,2),
                                                      image_size=(48),bias=False)

def forward(self, input):
    x = self.effnet(input)
    return x
</code></pre>
<p>model = Classifier()</p>
<p>I am trying to test out the pretrained network model(ImageNet) so that means I will need to train the input and last FC layer as I replaced them</p>
<p>Read about how to freeze layers, I am thinking of omething along this line</p>
<pre><code class="lang-auto">for child in model.children():
     ct += 1
     if ct &lt; x and ct &gt;1:
           for param in child.parameters():
                 param.requires_grad = False

However is this the right method? Don no see the requires_grad param anywhere?

Regards
</code></pre>

username: ptrblck
cooked: <p>Yes, you can freeze parameters by setting their <code>requires_grad</code> attribute to <code>False</code> as also described in the <a href="https://pytorch.org/tutorials/beginner/finetuning_torchvision_models_tutorial.html">Fine Tuning Tutorial</a> (including more useful details about fine tuning a model <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=10" title=":wink:" class="emoji" alt=":wink:"> ).</p>
[libtorch][torch.jit.script][input dynamic shape]Error in loading ". Pt" file on C + +: index out of range

username: firewormovo
cooked: <p>Because it is the company’s internal code, I can’t disclose it.I can only briefly describe the phenomenon:<br>
I’m trying to deploy a model, but its input contains dynamic dimensions: it works normally when you use the python model script to perform information on the data; After I serialize the model JIT:: script(), it will report an abnormal error, such as index out of range.<br>
In Python mode (ptyhon code is used to directly load".cpkt" parameters for inferecne), I can inferecne normally when I load the input data without using onnx mode, and the code after loading parameters is added at the end<br>
model=toch.jit.script(model)<br>
model.save(“model.pt”)<br>
An error was reported when loading the pt file even if in python code model.</p>
<p>If you have encountered similar situations and have relevant solutions or skills, I hope you can recover. Thank you very much！</p>
Dataloader with custom sequential data

username: pytorchnewb
cooked: <p>I have data where each training example is a mix between sequential and non sequential.</p>
<p>For instance, my dataset currently returns something like:</p>
<pre><code class="lang-auto">inputs = (scalar_a, sequence_a, sequence_b)
targets = sequence_c

return inputs, targets
</code></pre>
<p>All sequences (a, b, c) are of same length but of course different examples in a batch may have different lengths.</p>
<p>My desired output is when I iterate through the dataloader I should get a tuple with the same structure except everything should be batched.</p>
<p>How do I go about doing this?</p>
How to add fully connected layer in pretrained RESNET model in torch

username: i_m_a_q
cooked: <p>I have a pretrained resnet152 model. It outputs 2048 dimensional feature vector. The code is given below.</p>
<pre><code class="lang-auto">import torch

import torch.nn as nn

import torchvision.models as models

import torchvision.transforms as transforms

from torch.autograd import Variable

from PIL import Image

# Load the pretrained model

model = models.resnet152(pretrained=True)

# Use the model object to select the desired layer

layer = model._modules.get('avgpool')

# Set model to evaluation mode

model.eval()

# Image transforms

scaler = transforms.Scale((224, 224))

normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])

to_tensor = transforms.ToTensor()

def get_vector(image_name):

    # 1. Load the image with Pillow library

    img = Image.open(image_name)

    # 2. Create a PyTorch Variable with the transformed image

    t_img = Variable(normalize(to_tensor(scaler(img))).unsqueeze(0))

    # 3. Create a vector of zeros that will hold our feature vector

    #    The 'avgpool' layer has an output size of 2048

    my_embedding = torch.zeros(2048)

    # 4. Define a function that will copy the output of a layer

    def copy_data(m, i, o):

            my_embedding.copy_(o.data.reshape(o.data.size(1)))

    # 5. Attach that function to our selected layer

    h = layer.register_forward_hook(copy_data)

    # 6. Run the model on our transformed image

    model(t_img)

    print(model)

    # 7. Detach our copy function from the layer

    h.remove()

    # 8. Return the feature vector

    return my_embedding.numpy()

a=get_vector('/content/drive/MyDrive/images/COCO_val2014_000000000042.jpg')

print(a)

print(a.size)
</code></pre>
<p>I want <strong>2048 dimensional feature vector</strong> that is returned by ResNet to be passed through a fully connected layer and reduce it to a 64 dimensional vector. How can I do that?</p>

username: tom
cooked: <p>when you print the model (print(model)) you should see that there is a model.fc layer. You can make your new nn.Linear and assign it to model.fc.</p>
Initial weights in Convolutional Neural Networks and Artificial Neural Networks

username: Sandeep_Kumar_GITAM
cooked: <p>Dear friends,<br>
Can anyone tell me how to visualize or print the initial weights assigned to the kernels while implementing Convolutional Neural Networks and Artificial Neural Networks using the Pytorch framework?</p>

username: ivan-jgr
cooked: <p>You can access to <code>.weight</code> field of each Module.</p>

username: CasellaJr
cooked: <ol>
<li>Create the class of your CNN</li>
<li>Istantiate it like: <code>model = YourCNN()</code>
</li>
<li><code>list(model.parameters())</code></li>
</ol>
Are there any plans for a fix for CUDA 11?

username: Pro100rus32
cooked: <p>There has been a problem for a very long time. CUDA 11 is slower than CUDA 10.</p>
<p>GitHub - <a href="https://github.com/pytorch/pytorch/issues/47908" rel="noopener nofollow ugc">issue</a></p>
<p>My 5-year-old laptop with an Nvidia 940M (CUDA 10) video card works better than the 3070TI (CUDA 11) - This is nonsense!</p>
<p>For such a long time, this has not been fixed, that I already regretted that I bought the 3070ti</p>

username: ptrblck
cooked: <p>The issue is most likely not caused by CUDA, but cuDNN as described in this and other issues.<br>
You could check the 1.9.0 binaries, which fixed some missing cutlass kernels while statically linking cuDNN, build from source with the latest release, or try the nightly binaries with cuDNN8.2.2.</p>

username: Pro100rus32
cooked: <p>I don’t really like doing these things. I’m just wondering when a ready-made library will be available (pip) with a fix for this problem.</p>
<p>Thanks for the answer!</p>

username: BengtGustafsson
cooked: <p>I’m experiencing the same slowdown on our own CUDA code which is not related to cuDNN. We do use cuDNN also but the slowdown is more for our own kernels. Unfortunately we have to use CUDA11 to support Ampere in cuDNN and then took a 30-40% performance hit on our own code.</p>

username: ptrblck
cooked: <p>If the slowdown is caused in your custom CUDA code by updating the CUDA toolkit, I would recommend to post this issue in the NVIDIA discussion board so that the compiler devs are aware of it.</p>
<p>EDIT: In case you are seeing the slowdown in 11.2, could you update to &gt;=11.2U2 and check the performance again? We were aware of an increase in local memory usage for 11.2 and 11.2U1, which was fixed in newer versions.</p>
Missing quantized dtype like torch.quint8 when building 1.10 with operator list

username: caraffi
cooked: <p>We have been able to build pytorch 1.9 lite static libs defining OP_LIST, and run it also with NNAPI models (See discussion at <a href="https://discuss.pytorch.org/t/about-build-android-sh-lite-and-nnapi/133581" class="inline-onebox">About build_android.sh, LITE and NNAPI</a>).<br>
Now we are in the process of switching from torch 1.9 to 1.10, with our patch (see previous link) we can build and run lite with NNAPI as long as we do not use the OP_LIST. But if we build pytorch with our OP_LIST (that worked fine for 1.9, and we regenerated with the same process for 1.10 using <code>torch.jit.export_opnames(model)</code>) we get the following error:</p>
<pre><code class="lang-auto">E/libc++abi: terminating with uncaught exception of type c10::Error: Creation of quantized tensor requires quantized dtype like torch.quint8 ()
    Exception raised from empty_affine_quantized_other_backends_stub at ../aten/src/ATen/native/quantized/TensorFactories.cpp:79 (most recent call first):
    (no backtrace available)    
</code></pre>
<p>Any hint on how to fix this? Maybe a missing operator in our OP_LIST yaml file?</p>

username: iseeyuan
cooked: <p>You may need to use the tracing based selective build in 1.10 to first trace the model and update the OP_LIST. cc <a class="mention" href="/u/cccclai">@cccclai</a></p>

username: caraffi
cooked: <p>How does tracing works for creating/integrating the OP_LIST?</p>

username: cccclai
cooked: <p>Maybe trying following this? <a href="https://pytorch.org/tutorials/prototype/tracing_based_selective_build.html" class="inline-onebox" rel="noopener nofollow ugc">Conclusion — PyTorch Tutorials 1.10.0+cu102 documentation</a></p>

username: cccclai
cooked: <p>I doubt there might be some issues to build tracer in master. Let me follow up on it</p>

username: caraffi
cooked: <p>Thanks! Just to understand, in which way the tracing based selective build is better than the static analysis performed by export_opnames? If the input coverage is extensive, is it expected to be more complete?</p>
Hadamard-Multiply every image in batch with every other

username: mzimmer
cooked: <p>So suppose I have two tensors of the following shape: <code>(batch_size, n, m)</code>, name them A and B.</p>
<p>What I want is to build a tensor of shape <code>(batch_size**2, n, m)</code> such that every row of A is entry-wise multiplicated with every row of B, i.e., the first <code>batch_size</code> rows should consistent of the first row of A times every row in B, the next <code>batch_size</code> rows should consequently be the second row of A times every row in B, etc.</p>
<p>How would one do this in Pytorch? Can this be solved by simple Broadcasting?<br>
Thanks a lot!</p>

username: mzimmer
cooked: <p>Found a solution myself using <code>torch.einsum</code>:</p>
<pre><code class="lang-auto">torch.einsum('aij,bij-&gt;abij', A, B)
</code></pre>
Sequence ordering with TimeSeriesDataSet.to_dataloader() from PyTorch Forecasting

username: pcc
cooked: <ul>
<li>PyTorch-Forecasting version: 0.9.0</li>
<li>PyTorch version: 1.9.0</li>
<li>Python version: 3.9.6</li>
<li>Operating System: Windows 10</li>
</ul>
<p>I’ve been using pytorch forecasting and trying to understand it’s internal function. I’ve runned into something that I find weird regarding the <strong>TimeSeriesDataSet</strong> class.</p>
<p>When using the <strong>TimeSeriesDataSet.to_dataloader()</strong> to transform it into a dataloader to train the model, the values for the categorical variables and continuous variables don’t seem to be ordered into a sequence.</p>
<h3>
<a name="actual-behavior-1" class="anchor" href="#actual-behavior-1"></a>Actual behavior</h3>
<p>For example, I get the following for the encoder_cat:<br>
‘encoder_cat’: tensor([[[ 0,  1,  7],<br>
[ 0,  2,  7],<br>
[ 0, 13,  7],<br>
[ 0, 18,  7],<br>
[ 0, 19,  7],<br>
[ 0, 20,  7],<br>
[ 0, 21,  7],<br>
[ 0, 22,  7],<br>
[ 0, 23,  7],<br>
[ 0, 24,  7],<br>
[ 0,  3,  7],<br>
[ 0,  4,  7],<br>
[ 0,  5,  7],<br>
[ 0,  6,  7],<br>
[ 0,  7,  7],<br>
[ 0,  8,  7],<br>
[ 0,  9,  7]], …</p>
<p>The first column represents the city group in the data, the second one is the hour and the last one is the day of week.</p>
<h3>
<a name="expected-behavior-2" class="anchor" href="#expected-behavior-2"></a>Expected behavior</h3>
<p>I was expecting to have something like this instead:<br>
‘encoder_cat’: tensor([[[ 0,  1,  7],<br>
[ 0, 2,  7],<br>
[ 0, 3,  7],<br>
[ 0, 4,  7],<br>
[ 0, 5,  7],<br>
[ 0, 6,  7],<br>
[ 0, 7,  7],<br>
[ 0, 8,  7],<br>
[ 0, 9,  7],<br>
[ 0, 10,  7],<br>
[ 0, 11,  7],<br>
[ 0, 12,  7],<br>
[ 0, 13,  7],<br>
[ 0, 14,  7],<br>
[ 0, 15,  7],<br>
[ 0, 16,  7],<br>
[ 0, 17,  7]],</p>
<h3>
<a name="questions-3" class="anchor" href="#questions-3"></a>Questions</h3>
<p>Based on this, can I assume that the observations are not fed as a sequence into the model? Can you elaborate on the reason why would that approach be desirable specifically when modelling Time Series? I did read that there is a stateful and stateless approach for timeseries forecasting using neural networks. Can I get a little more explanation on this?</p>
<p>Also, the same shuffling happens in the validation set and it might make sense, but in my opinion, it doesn’t since in production, I will feed an ordered sequence of data to my model in order to get the X following predictions.</p>
<p>I would greatly appreciate if someone can explain to me the internal function and the use of shuffled sequence or the reason why the categorical variables are stored in the wrong order and how it might affect the model in production?</p>
<p>Thank you</p>
Does Glow support Benchmark Models for Training and Inference?

username: lvng
cooked: <p>Hi,<br>
I want to run below benchmarking models using Glow, both training and inference.</p>
<p>ResNet - Image classification<br>
3D U-Net - Image segmentation<br>
SSD &amp; Mask R-CNN - Object detection<br>
RNN-T - Speech recognition<br>
BERT - NLP<br>
DLRM - Recommendation<br>
Minigo - ReinforcementLearning</p>
<p>Here, I would like to know which models can be trained and inferred using glow.</p>
<p>This helps me a lot creating some benchmarking examples using glow</p>
<p>Thanks.</p>