akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Which is the most adavance pytorch model for object detection implemented in pytorch

Filippo: <p>Hi all,</p>
<p>I need to deploy a CV model for object detection and I am not clear which models among the available is the “best performing”. in the HUB I can rank models base on the github stars but I am not sure this is a sufficient criteria (I guess older models have more starts).<br>
Which model should I use? and moving forward where can I check which model has the best performance and stay updated?</p>

Ab_Niazi: <p>I Think you can see at Mask R CNN  I heard many people use the Mask RCNN for the objet detection in industry.</p>
<p>This is my view and it can be wrong,</p>

pourmand1376: <p>Take a look at <a href="https://github.com/ultralytics/yolov5" rel="noopener nofollow ugc">Yolov5</a>. It is pretty well established and it does have active contributions (you can see that from last commit date).</p>
Pruning `torch.nn.MultiheadAttention` causes RuntimeError

MuYang: <p>I am running into the following RuntimeError when pruning parameters of <code>torch.nn.MultiheadAttention</code> module:</p>
<pre><code class="lang-auto">RuntimeError: Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed). Saved intermediate values of the graph are freed when you call .backward() or autograd.grad(). Specify retain_graph=True if you need to backward through the graph a second time or if you need to access saved tensors after calling backward.
</code></pre>
<p>Here is the small code I’m running.</p>
<pre><code class="lang-auto">import torch.nn as nn
import torch
import torch.nn.utils.prune as prune

class net(nn.Module):
    def __init__(self):
        super().__init__()

        self.att = nn.MultiheadAttention(
            embed_dim=100,
            num_heads=2,
            batch_first=True
        )
        self.proj = nn.Linear(100, 10)
        self.criterion = nn.CrossEntropyLoss()

    def forward(self, x):
        attn_output, attn_weights = self.att(x, x, x)
        logits = self.proj(attn_output)
        return logits

    def compute_loss(self, logits, target):
        loss = self.criterion(logits, target)
        return loss

model = net().cuda()
# prune here
params_to_prune = [
    (model.att, "in_proj_weight"),
    (model.att.out_proj, "weight"),
    (model.proj, "weight")
]
prune.global_unstructured(
    params_to_prune,
    pruning_method=prune.L1Unstructured,
    amount=0.05,
)

opt = torch.optim.SGD(model.parameters(), lr=0.01)
for step in range(0, 10):
    x = torch.rand((16, 50, 100)).cuda()  # data of batchsize=16, seqlength=50, featuredim=100
    y = torch.ones((16*50, )).long().cuda()
    opt.zero_grad()
    logits = model(x)
    loss = model.compute_loss(logits.reshape(16*50, 10), y)
    loss.backward()
    opt.step()
    print("step {} successful".format(step))
</code></pre>
<p>It seems that the error is brought by the pruning of this parameter in<code>torch.nn.MultiheadAttention</code>: <code>(model.att.out_proj, "weight")</code>. If I remove this line then there is no such RuntimeError. Am I missing anything here for pruning the MultiheadAttention module? How should I properly prune the parameters in <code>out_proj</code> (which I believe is just a Linear layer)? Thanks!</p>

mzimmer: <p>I am running into the same error. Is there any advice on how to prune such a module?</p>
Parameter selection recommendations for the DPSGD algorithm

JeffffFu: <p>Any DPSGD algorithm in FashionMnist and cifar10 parameter selection suggestions? Such as sigma, C, learning rete.</p>

pstock: <p>Hey JeffffFu,</p>
<p>We generally advise to start with <code>C=1</code> and the same learning rate as the vanilla, non-private baseline. Then, you can adapt sigma to the privacy/utility tradeoff you’re looking for.</p>
<p>Hope this helps,<br>
Pierre</p>
Restoring the original model

luc12: <p>Hi everyone,<br>
I’m using Opacus and I have a very specific question<br>
Basically, I want to alternate the training using differential privacy with the training without DP:</p>
<ol>
<li>I wrap the model with make_private and I train this “private model” for some epochs</li>
<li>Then I want to unwrap the model and restore the original one to train a “non private model” without differential privacy</li>
</ol>
<p>Is it possible to do this? Can I restore the original model after I wrapped it using make_private?<br>
I also have another question, assuming that it is possible to unwrap the private model, is it possible to alternate the training of the private model with the training of the original model without calling every time make_private?</p>
<p>Thank you and sorry for the “strange” question.</p>

pstock: <p>Hey luc12, thanks for reaching out!</p>
<p>To alternate between DP and vanilla training, you have two options:</p>
<ol>
<li>Keep the model wrapped in a <code>GradSampleModule</code> but change the noise multiplier to be <code>0</code> for vanilla training (you would still continue to clip per-sample gradients and you would still have Opacus training time versus faster vanilla training time).</li>
<li>Checkpoint the <code>model._module.state_dict()</code> and load it into a fresh, non-private model when you wish to switch. There may be other ways to remove the backward hooks with their handles but the checkpointing method seems to be the safest.</li>
</ol>
<p>Of course, the accounting has to be adapted. Hope this helps!</p>
<p>Pierre</p>

luc12: <p>Thank you, in the end I used the second solution!</p>
Semantic segmentation loss

Giuseppe: <p>Hi all, is there anyone which have implemented a Pixel-wise binary cross entropy loss? I need for the a semantic segmentation and I found a very little bit in pytorch documentation.<br>
Can anyone help me?</p>

Lelouch1: <p>Isn’t <a href="https://pytorch.org/docs/stable/generated/torch.nn.BCELoss.html" class="inline-onebox" rel="noopener nofollow ugc">BCELoss — PyTorch 1.12 documentation</a> what you need ? It is already working pixel-wise.</p>

ptrblck: <p><a class="mention" href="/u/lelouch1">@Lelouch1</a> is right and <code>nn.BCELoss</code> will also work pixel-wise. Small NIT: use <code>nn.BCEWithLogitsLoss</code> for better numerical stability.</p>

Giuseppe: <p>I didn’t know that BCE works as pixel-wise. I know that the one with logits has a numerical stability, Thanks a lot.</p>
How can I classify words spoken by one person?

YuhskeHujisaki: <p>How can I classify words spoken by one person?<br>
For example, subjects voice classify dogs and cats by voice.</p>

tom: <p>I would recommend the speech commands tutorial as a starting point:</p>
<p><a href="https://pytorch.org/tutorials/intermediate/speech_command_recognition_with_torchaudio.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/intermediate/speech_command_recognition_with_torchaudio.html</a></p>
<p>Best regards</p>
<p>Thomas</p>
Best way to deploy Pytorch models into production

arkh: <p>Hi!<br>
I trained an Image enhancer model, and now I want to deploy it on the web using Reactjs. Now there are a few different ways to do it. Here are a couple -</p>
<ol>
<li>Creating an API using FastAPI and then using that API from react.</li>
<li>Uploading the model on AWS sagemaker and creating a frontend using react.<br>
I wanted to know what would be the best way to do it if I want to put it into production and build a real startup out of that.<br>
Thanks!</li>
</ol>

tom: <p>I would say go with whatever is most convenient to you.<br>
When running the model from Python, you might consider JITing your model and using async plus a worker thread for running the JITed model.<br>
I am biased, but I think the example server in chapter 15 of our book is still very reasonable.</p>
<p>Best regards</p>
<p>Thomas</p>
Have autograd.grad in forward for torch.ONNX

malteal: <p>Hi,</p>
<p>I have a model in Pytorch, where the output of the forward is using autograd.grad(). I want to export this model to ONNX using <em>torch.onnx.export</em> but ONNX has some problems with handling autograd.grad() in the forward.</p>
<p><strong>Bug I found:</strong><br>
If I run the model on the GPU, I can export the model by setting the forward like this:</p>
<pre><code class="lang-auto">def forward(x,y):
  **some calculates resulting in z**
  z = autograd.grad(z, y,retain_graph=False,create_graph=False,
                                    grad_outputs=torch.ones_like(y))[0]
  return  0*(x+y)+z
</code></pre>
<p>if I just return <em>z</em> then I get this error:</p>
<blockquote>
<p>onnxruntime.capi.onnxruntime_pybind11_state.InvalidArgument: [ONNXRuntimeError] : 2 : INVALID_ARGUMENT : Invalid Feed Input Name:pT</p>
</blockquote>
<p>pT is the name of x. That is fine as add <em>0 * (x+y)</em> to the output removes this error and exports the model to ONNX.</p>
<p><strong>The issue that I have:</strong><br>
So I found a workaround when exporting the model to ONNX with autograd.grad as seen above. But my issue is now precision. When exporting to ONNX (when the model is on the GPU), the precision is VERY poor (sometimes 1000% off)…  If I export the exact same code/model but change the init device to the CPU, I get this error:</p>
<blockquote>
<p>RuntimeError: Cannot insert a Tensor that requires grad as a constant. Consider making it a parameter or input, or detaching the gradient<br>
Tensor:<br>
1.8547<br>
1.2683<br>
[ torch.FloatTensor{2,1} ]</p>
</blockquote>
<p>1.8547 1.2683 is just a dummy input.<br>
This error comes from the autograd.grad as if I remove it in the forward, the model is exported.</p>
<p><strong>My question/hope:</strong><br>
How should you properly export a model to ONNX, that calculates the gradient with respect to the input? Or if that isn’t recommend at the moment, could it be added to torch.onnx? <img src="https://discuss.pytorch.org/images/emoji/apple/crossed_fingers.png?v=12" title=":crossed_fingers:" class="emoji" alt=":crossed_fingers:" loading="lazy" width="20" height="20"></p>
<p><em>Thank you for your time and hope everything makes sense!</em></p>
How to translate pytorch training and validation loss tensorboard curves

ranaban: <p>I can not understand the x-axis values. I have used batch_size = 32 and 6153, 769 samples for training and validation respectively. Below is the training curve. The plotting is done using tensorboard.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/8/a/8a92ab18d89b3fe67854790be0acc0e874a3f87e.png" alt="train_loss" data-base62-sha1="jLS7PUzL3v8Qks8I34vBeQcZSGy" width="655" height="400"></p>
<p>Please tell me how to understand the plot. Thank you!</p>

ptrblck: <p>The x-axis is most likely representing the iterations or epochs  while the y-axis would represent the corresponding loss value for the training and validation datasets. Since no axis labels are used, I’m speculating, but based on your description and the plot it looks like a standard way to visualize losses.</p>
<p>The loss value should generally correspond to the performance metric of your model (and you have to make sure it does). E.g. if you are working on a multi-class classification, you should see that the loss and accuracy show an inverse relation: the lower the loss, the higher the accuracy (this is not a perfect representation as the model “confidence” in its predictions will also change the loss, but not necessarily the accuracy etc.).</p>
<p>Besides that you can also check the loss curves to see if your model still continues to train, if it’s overfitting (lage gap between the training and validation losses), how stable the overall training is, if a learning rate scheduler does its job, if the training is stuck etc.</p>

ranaban: <p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a>!<br>
Yes, Y-axis is showing loss values.<br>
I have used 10 epochs and was expecting to see the epochs in the X-axis like 0, 1, 2, …, 9. But it is showing 0, 200, 400, …, 1800, 2000. Which made me confused. I still couldn’t figure out, why those large values are there?</p>

ptrblck: <p>Maybe you are plotting the number of iterations on the x-axis and not the epochs?<br>
Could you check where the actual plot call is used and which arrays are passed to it?</p>

ranaban: <p>I wrote the below code to get the plot<br>
logger = TensorBoardLogger(‘model_logs’, name = ‘My-model’)</p>
<p>trainer = pl.Trainer(<strong>logger = logger</strong>, checkpoint_callback = checkpoint_callback, max_epochs = 10,<br>
gpus = 1, progress_bar_refresh_rate = 20)</p>
<p>%load_ext tensorboard<br>
%tensorboard --logdir ./<strong>model_logs</strong></p>
<p>Should I check the log files also?</p>
How to control one data update part model and another data update the whole model?

chener: <p>Hi, I have a specific question. In each training step, I have two pairs of input, (x1, y1) and (x2,y2), and the model is composed of two parts model_a and model_b. The loss is computed by |y - model_b( model_a( x ) ) | .</p>
<p>The problem is, that I want the loss computed from pair (x1, y1) to update both the parameters of model_a and model_b, however, the computed loss from (x2, y2) to update only model_a’s parameters. Is this possible?</p>
<p>I have tried the following, I used two optimizers optim_a and optim_b, optim_a updates the parameters of both model_a and model_b, while optim_b only updates the parameters of model_a.</p>
<pre><code class="lang-auto">optim_a = torch.optim.Adam( model_a.parameters()+model_b.parameters() )
optim_b = torch.optim.Adam( model_a.parameters() )

</code></pre>
<p>In the training phase, I compute both losses like,</p>
<pre><code class="lang-auto"> loss1 =  |y1 - model_b( model_a( x1 ) ) |  
 loss2 =  |y2 - model_b( model_a( x2) ) | . 
</code></pre>
<p>Then,</p>
<pre><code class="lang-auto">optim_a.zero_grad()
optim_b.zero_grad()
loss1.backward()
optim_a.step()

loss2.backward()
optim_b.step()
</code></pre>
<p>However, when conducting loss2.backward, I met “RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation”, that’s because the parameters of model_a and model_b are changed when conducting loss2.backward().</p>
<p>Any ideas to solve the problem? Nedd Help!!</p>

tom: <p>Would it work to do</p>
<ul>
<li>require gradients for everything</li>
<li>compute loss 1</li>
<li>set requires grad to false for model a</li>
<li>compute loss 2</li>
<li>zero grad, compute backward of total loss, take step</li>
</ul>
<p>As long as you detach anything from the loss1 computation that you use in computing loss2, it would seem worth a try.</p>
<p>Best regards</p>
<p>Thomas</p>
The samping of RDP

JeffffFu: <p>1、the uniform samping with Replacement ，uniform samping with Replacement and the Poisson samping . What is the relationship between the three of them?<br>
2、the paper《RDP of the sampled Gussian Mechanism》 use  Poisson samping?<br>
3、Opacus provide uniform samping with Replacement and Poisson samping methods, do they both correspond to the same privacy calculation formula？</p>

JeffffFu: <p>1、the uniform samping with Replacement ，uniform samping without Replacement and the Poisson samping . What is the relationship between the three of them?</p>

karthikprasad: <p>Hi,<br>
Re 1:<br>
We say Uniform sampling to mean Poisson sampling: Each sample is selected with a probability equal to <code>sample_rate</code><br>
The with replacement is due to the fact that during a batch construction, we consider the whole dataset (all the samples), and sample each data point with the probability <code>sample_rate</code></p>
<p>Re 2: Yes</p>
<p>Re 3: Yes, because the two are one and the same.</p>

JeffffFu: <p>Thank you for your answer, I have two more questions as follows.<br>
1, the uniform sampling with replacement , the uniform sampling without replacement and the Poisson sampling are differential. In uniform sampling, sample rate is the number of samples sampled In uniform sampling, the sample rate is the ratio of the number of samples sampled to the total number of samples, so the number of samples sampled is fixed and the same for each sample. In poisson samping, the sample rate is the probability of each sample being sampled, which means that the number of samples sampled is not necessarily the same each time. Am I understanding this correctly?<br>
2. In GDP privacy calculation, uniform sampling and poisson sampling have different privacy budgets for the same parameters, why are the privacy losses calculated the same for both in RDP?</p>

JeffffFu: <p>3、what is the differential of UniformWithReplacementSampler and DistributedUniformWithReplacementSampler?</p>

JeffffFu: <p>there are somebody can answer my question? thanks</p>

pstock: <p>Hey JeffffFu,</p>
<p>Thanks for your interest! To answer your questions:</p>
<ol>
<li>That’s correct, with Poisson sampling your batch size may vary (hence we only set the <em>expected</em> batch size).</li>
<li>Poisson sampling is necessary to obtain privacy budget computations under the paper you mentioned.</li>
<li>The <code>UniformWithReplacementSampler</code> is to be used when you train on a single process (generally a single GPU), the <code>DistributedUniformWithReplacementSampler</code> is when you train in a distributed way, generally on multiple GPUs.</li>
</ol>
<p>Hope this helps,<br>
Pierre</p>

JeffffFu: <p>thank you very much!</p>
Torch Distributed Data Parallel and bfloat16 support

lhoang: <p>Does torch distributed data parallel work on models that have bfloat16 parameters?</p>
<p>The documentation mentions that it works with fp16, so I was wondering if it extends to<br>
bfloat16.</p>
<p>This is an example of an error I get trying to use torch DDP on a model with bfloat16<br>
parameters:</p>
<pre><code class="lang-auto">packages/torch/autograd/__init__.py", line 154, in backward
    Variable._execution_engine.run_backward(
RuntimeError: Expected torch.FloatTensor, got CPUBFloat16Type
</code></pre>
<p>Any insight would be appreciated.</p>
<p>EDIT:<br>
This is for CPUs and not GPUs.</p>

ptrblck: <p>Based on <a href="https://github.com/pytorch/pytorch/issues/75215">this issue</a> it seems that <code>bfloat16</code> support is currently only implemented in the NCCL backend (so GPU). However, since the issue was opened in April, support might have been added for the CPU backends as well. Could you check the stacktrace and post where the error is coming from? I also assume you are seeing this error only if you are running a DDP setup?</p>

lhoang: <p>It occurs in the backward phase (my guess is that it’s preparing for gradient comms across processes).</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  .....
  File "bf16_profile.py", line 282, in train_model
    loss.backward()
  File "........./python3.8/site-packages/torch/_tensor.py", line 307, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "........./python3.8/site-packages/torch/autograd/__init__.py", line 154, in backward
    Variable._execution_engine.run_backward(
RuntimeError: Expected torch.FloatTensor, got CPUBFloat16Type
</code></pre>
<p>Yes, this only occurs if I wrap the model in DDP. The process group was initialized with MPI.<br>
From my investigation since this post I’m pretty sure CPU DDP still doesn’t have support for<br>
bfloat16.</p>
Train for more than the epochs specified in OneCycleLR

mfoglio: <p>I am training a CNN with <code>OneCycleLR</code>:</p>
<pre><code class="lang-auto">lr_scheduler = torch.optim.lr_scheduler.OneCycleLR(
    optimizer, max_lr=max_learning_rate, epochs=100, steps_per_epoch=1,
    pct_start=0.05, anneal_strategy='cos', cycle_momentum=True,
    base_momentum=0.85,
    max_momentum=0
)
</code></pre>
<p>This works well, except for the fact that I can’t train for more than 100 epochs. During my last training, I reached 100 epochs and I decided to train for 10 more. I couldn’t do it because of the following error:</p>
<pre><code class="lang-auto">ValueError: Tried to step 102 times. The specified number of total steps is 100
</code></pre>
<p>Is there a way I can keep training after 100 epoch using the last learning rate from the OneCycleLR?<br>
I could create a <code>OneCycleLR</code> with <code>110</code> but that would change the shape of the learning rate curve. Moreover, it would not be flexible enough if, for instance, I wanted to train for 115 epochs.</p>

ptrblck: <p>I would guess the error is raised in the <code>lr_scheduler.step()</code> call after the specified max epochs?<br>
If so, maybe add a condition and skip the <code>step()</code> call if you are training in epoch &gt; 100.</p>
Troubleshoot writer.add_graph

Simon_Watson: <p>Hi All,<br>
I have a model that I am converting from the original (produced by others in TFv1) to Torch.</p>
<p>I’m trying to use tensorboard renditions of computational graphs to compare to the original and Torch model. However, when I use the .add_graph() function as below:</p>
<p>writer.add_graph(sindy_autoencoder, torch_stacked_validation_data)</p>
<p>I get an error:</p>
<p>{RuntimeError}Encountering a dict at the output of the tracer might cause the trace to be incorrect, this is only valid if the container structure does not change based on the modules inputs. Consider using a constant container instead (e.g. for <code>list</code>, use <code>tuple</code> instead. for <code>dict</code>, use a <code>NamedTuple</code> instead). If you absolutely need this and know the side effect, pass strict=False to trace() to allow this behaviour.</p>
<p>I pass a torch.tensor in so I’m not sure what this means.</p>
<p>Can anyone advise on a possible resolution?</p>
<p>Simon</p>

ejguan: <p>You might want to open an issue to <a href="https://github.com/pytorch/pytorch/issues" rel="noopener nofollow ugc">pytorch</a> with a reproducable script.</p>
Any plans to (sort of) JIT the DataPipes into C++?

BarclayII: <p>The code for DataPipes looks like writing a new language where every statement is transforming a DataPipe into another one, or combining from or splitting into multiple DataPipes.  So in theory, if all the DataPipes are PyTorch builtin, then one can analyze the “graph” of the DataPipe code and essentially translate them into something similar to TorchScript, and execute it in C++ in a multithreaded manner.  I was wondering if PyTorch devs have already explored the idea or not.</p>

ejguan: <p>Thanks for asking. We do consider jittable DataPipe. But, for now, we haven’t explored the option to make DataPipe itself to C++.</p>
<p>We might be able to jit multiple map functions like <code>datapipe.map(fn1).map(fn2).map(fn3)</code> to a single function call → Those functions might also be jit together.</p>
<p>Also want to note that It’s kind hard to run data-pipeline in multithread manner with the deterministic result.</p>
<p>We are open to any new idea to make DataPipe more efficient.</p>
Getting Error (is invalid for input of size 102400)

Ab_Niazi: <p>I am trying to implement the GAN on CIFAR 10 data set.</p>
<p>I am following this <a href="https://jovian.ai/aakashns/06-mnist-gan" rel="noopener nofollow ugc">tutorial</a>  The actual tutorial was  on MNIST but I am doing this on CIFAR Data Set.</p>
<p>The code is below:</p>
<p>‘’'sample_vectors = torch.randn(batch_size, latent_size).to(device)</p>
<p>def save_fake_images(index):<br>
fake_images = G(sample_vectors)<br>
fake_images = fake_images.reshape(fake_images.size(0), 3, 32,32)<br>
fake_fname = ‘fake_images-{0:0=4d}.png’.format(index)<br>
print(‘Saving’, fake_fname)<br>
save_image(denorm(fake_images), os.path.join(sample_dir, fake_fname), nrow=10)</p>
<h1>
<a name="before-training-1" class="anchor" href="#before-training-1"></a>Before training</h1>
<p>save_fake_images(0)<br>
Image(os.path.join(sample_dir, ‘fake_images-0000.png’))‘’’</p>
<p><strong>I am getting this error "shape ‘[100, 3, 32, 32]’ is invalid for input of size 102400 "</strong><br>
100 is a batch size<br>
3 is numbe of channel<br>
32 is the size<br>
I have the idea that I had to reshape the tensor in to 3 dimension Since  I am new to Pytorch .</p>
<p>if I change the dimension from 3 to 1  this block of code will run but later in the training loop it shows the error and I am sure the error is because of the previous issue.</p>
<p>Error which is shows in trainging loop.</p>
<p><strong>mat1 and mat2 shapes cannot be multiplied (100x3072 and 1024x256)</strong></p>

ptrblck: <p>Based on the error:</p>
<pre data-code-wrap="python"><code class="lang-python">I am getting this error "shape ‘[100, 3, 32, 32]’ is invalid for input of size 102400 "
</code></pre>
<p>it seems your output might have a single channel and have the shape <code>[100, 1, 32, 32]</code> which would correspond to <code>102400</code> elements.</p>
<aside class="quote no-group" data-username="Ab_Niazi" data-post="1" data-topic="159870">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/50afbb/48.png" class="avatar"> Ab_Niazi:</div>
<blockquote>
<p>mat1 and mat2 shapes cannot be multiplied (100x3072 and 1024x256)</p>
</blockquote>
</aside>
<p>This error is raised in a linear layer, which expects <code>1024</code> input features, while your input activation has <code>3072</code> features. Change <code>in_features</code> to <code>3072</code> in this layer and it should work.</p>

Ab_Niazi: <p>Dear <a class="mention" href="/u/ptrblck">@ptrblck</a>  Thank you for the response.</p>
<p>since the code which I was using as the tutorial is has the data set of MNIST which has one channel and  cifar  images has 3 channels.</p>
<p>I was giving wrong input  I thought</p>
<p>32 * 32 = 1024 but I forgot that it has 3 channels<br>
so I need to multiply it by three</p>
<p>32 *32 * 3 = 3072</p>
<p>so now my code is working.</p>
<p>Thank you for pointing out at right direction</p>
Pattern for tunable number of layers (functorch friendly if possible)

jlperla: <p>In the process of hyperparameter tuning I want to be able to try the model with different numbers of layers.  Someone asked a similar question in <a href="https://discuss.pytorch.org/t/create-a-network-where-the-number-type-of-layers-are-given-as-parameter-input/61866" class="inline-onebox">Create a network where the number/type of layers are given as parameter input</a>  but it didn’t have an answer.</p>
<p>Is there a canonical way to do this which is the highest performance (and, hopefully, is setup to work well with functorch’s <code>make_functional</code> ?</p>
<p>For example, for a deep sets architecture we have things like</p>
<pre><code class="lang-python">n_in =  3# set for problem
n_out = 2 # set for problem
hidden_dim = 128 # TUNED!
activator = nn.RelU() # TUNED!  Maybe nn.Tanh/etc.
last_activator = nn.Softplus() # TUNED!  Maybe just identify/etc.
layers = 4 # TUNED!

model = nn.Sequential(
    nn.Linear(n_in, hidden_dim),
    self.activator,
    # Add in layers - 1
    *[
        nn.Sequential(
            nn.Linear(hidden_dim, hidden_dim),
            deepcopy(
                activator
            ),  
        )
        for i in range(layers - 1)
    ],
    nn.Linear(hidden_dim, n_out, bias=True),
    last_activator
)
</code></pre>
<p>Is there a pattern with the highest performance for this sort of thing with the flexible layers or is the performance identical due to tracing?  What about if we try to use <code>make_functional</code> and apply this across batches?  Or even try to with future incarnations of functorch’s <code>aot_function</code> ?</p>
Pytorch Optimization

Coder_Man: <p>I need to optimize my pytorch model. Is there any way to optimize the pytorch model by either updgrading versions (pytorch version or cuda version) or trying out any script ?</p>

ptrblck: <p>Updating to the latest stack is usually recommended (and let us know if you are seeing any regressions).<br>
Besides that the <a href="https://pytorch.org/tutorials/recipes/recipes/tuning_guide.html">performance guide</a> is a good starter to optimize your use case.</p>
All_reduce for TorchElastic

joohyunglee: <p><a href="https://github.com/pytorch/elastic/blob/master/examples/imagenet/main.py#L525" rel="noopener nofollow ugc">TorchElastic code</a> does <strong>NOT</strong> compute <code>all_reduce</code> in order to yield the performance throughout all gpus (please check the other <a href="https://github.com/pytorch/examples/blob/main/imagenet/main.py#L426" rel="noopener nofollow ugc">ImageNet example</a>.) How does TorchElastic gather values from all gpus? Thank you ahead!!</p>

d4l3k: <p>TorchElastic treats the model mostly as a black box. There’s no magic happening here–in that example there’s no sync of the metrics between gpus. If you did want that behavior you should follow do something like the example in the core pytorch repo</p>
Performance gap between single-card model and DDP/DP

zxhuang1698: <p>Hello,</p>
<p>In one of my recent project I have to leverage Distributed dataparallel/dataparallel due to the memory constraint. But simply wrapping up the model with DP/DDP and training it lead to a slightly worse performance than the original single card training (exact same setting).</p>
<p>Would it be possible to close this gap? Or, more specifically, what are the differences under the hood (e.g. Batchnorm should be replaced with SyncBatchnorm, etc.)</p>
<p>Thank you!</p>

wanchaol: <p><a class="mention" href="/u/zxhuang1698">@zxhuang1698</a> can you clarify a bit about your question, i’m not sure I get it, are you specifically referring to DistributedDataParallel?</p>
<p>In usual cases, Distributed training usually requires communication to sync model’s gradients and thus usually it is not linear scaling compared to local training.</p>

zxhuang1698: <p><a class="mention" href="/u/wanchaol">@wanchaol</a> sorry for the confusion. For performance, I do not mean the computational efficiency, rather I am talking about the performance of the task (e.g. loss, accuracy, IOU, etc.)</p>
<p>In the ideal case, I would assume training on multiple cards should lead to the same e.g. loss curve as training on a single card. I know randomness exists, but I do see multi-gpu models (both DP and DDP) perform worse than single-gpu models constantly in my setup, where they share the exact same training configs. So I am wondering whether there are any behaviors of DP/DDP that can cause such gap.</p>
<p>For example, I remember DP/DDP do not handle batchnorm in a synchronized way by default, where they calculate statistics on each card locally. This might cause performance to degrade when batchsize is small. Say we have a batchsize of 8 and four cards, then the batch statistics are calculated with an effective batchsize of 2 and can have high variance, compared to training the same model on a single card of batchsize 8.</p>
Transformer Not Training Properly

maxrivera: <p>I don’t believe I am training/using my transformer correctly. At the end of 20 epochs it’s giving output predictions that consist of only one or two types of tokens (shown below). This is my first time building a transformer and I’m really not sure if I’m performing the training loop correctly. I don’t know how to shift the output to the right or if that would make a difference. And I’m not feeding the previous predictions back into the decoder for to generate the next token in the sequence (I was told you only do that in inference), it’s just computing the whole prediction in one forward pass through the decoder. Could someone help me diagnose this issue?</p>
<pre><code class="lang-auto">target:  [['BOS', 'six', 'divided', 'by', 'two', 'equals', 'three', '.', 'EOS', 'PAD', 'PAD', 'PAD', 'PAD', 'PAD', 'PAD', 'PAD', 'PAD', 'PAD',...]]
predit:  [['BOS', 'EOS', 'EOS', 'EOS', 'BOS', 'EOS', 'EOS', 'EOS', 'EOS', 'EOS', 'EOS', 'EOS', 'EOS', 'EOS', 'EOS', 'EOS', 'EOS',...]]
target:  [['BOS', 'people', 'love', 'to', 'talk', '.', 'EOS', 'PAD', 'PAD', 'PAD', 'PAD', 'PAD', 'PAD', 'PAD', 'PAD', 'PAD', 'PAD',...]]
predit:  [['BOS', '.', 'BOS', '.', '.', '.', '.', '.', '.', '.', '.', '.', '.', '.', '.', '.', '.', '.', '.', '.', '.', '.', ....]]
target:  [['BOS', 'my', 'mother', 'is', 'writing', 'a', 'letter', 'now', '.', 'EOS', 'PAD', 'PAD', 'PAD', 'PAD', 'PAD', 'PAD',...]]
predit:  [['BOS', 'EOS', 'EOS', 'EOS', 'EOS', 'EOS', 'EOS', 'EOS', 'EOS', 'BOS', 'EOS', 'EOS', 'EOS', 'EOS', 'EOS', 'EOS', 'EOS',...]]
Epoch[20/20] train_loss: 4.4391326904296875 val_loss: 4.598245028791757
</code></pre>
<p>Here is my training loop:</p>
<pre><code class="lang-auto"># optimization loop 
best_loss = 1e5
best_epoch = 0
optimizer=torch.optim.Adam(params=model.parameters(),lr=1e-3)
loss_fn = torch.nn.CrossEntropyLoss(ignore_index=0) 
train_losses = []
val_losses = []
for epoch in range(1,EPOCHS+1):

    # train loop 
    for i, (src,trg) in enumerate(train_data):

        # place tensors to device 
        src = torch.Tensor(src).to(DEVICE).long()
        trg = torch.Tensor(trg).to(DEVICE).long()
        mask = torch.tril(torch.ones((MAX_LENGTH, MAX_LENGTH))).to(DEVICE)

        # forward pass 
        out = model(src,trg, mask)

        # print prediction vs target sentence 
        trg_sentence = id_to_word(trg,en_index_dict)
        print('target: ',trg_sentence)
        val, ind = torch.max(out, -1)
        pred_sentence = id_to_word(ind,en_index_dict)
        print('predit: ', pred_sentence)


        # compute loss 
        train_loss = loss_fn(out.view(-1, tgt_vocab), trg.view(-1))
        
        # backprop 
        optimizer.zero_grad()
        train_loss.backward()

        # update weights 
        optimizer.step()
</code></pre>
<p>I can provide more code if necessary! Thanks</p>
How can extract the Features map of ResNet 50

Giuseppe: <p>Hy guys, how can I extract the features in a resnet50 before the general average pooling? I need the image of 7x7x2048. If I put the FC in an nn.Identity in forward I only obtain the features vector. I need the image before the final pooling. How can I make?</p>

Megh_Bhalerao: <p>You can get the <code>output</code> of any layer of a model by doing <code>model.layer_name(input)</code>. This will give you the output of whatever layer you want, assuming that your <code>input</code> is correct.</p>

Giuseppe: <p>Do you know the <code>layer_name</code> that I need? <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=9" title=":sweat_smile:" class="emoji" alt=":sweat_smile:"></p>

Megh_Bhalerao: <p>I think you would have to look into your <code>resnet</code> architecture, right? <img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=9" title=":thinking:" class="emoji" alt=":thinking:"></p>

Giuseppe: <p>I think that will be the <code>self.layer4</code>, ResNet’s code here:</p>
<pre><code class="lang-auto">        self.bn1 = norm_layer(self.inplanes)
        self.relu = nn.ReLU(inplace=True)
        self.maxpool = nn.MaxPool2d(kernel_size=3, stride=2, padding=1)
        self.layer1 = self._make_layer(block, 64, layers[0])
        self.layer2 = self._make_layer(block, 128, layers[1], stride=2,
                                       dilate=replace_stride_with_dilation[0])
        self.layer3 = self._make_layer(block, 256, layers[2], stride=2,
                                       dilate=replace_stride_with_dilation[1])
        self.layer4 = self._make_layer(block, 512, layers[3], stride=2,
                                       dilate=replace_stride_with_dilation[2])
        self.avgpool = nn.AdaptiveAvgPool2d((1, 1))
        self.fc = nn.Linear(512 * block.expansion, num_classes)
</code></pre>
<p>Many thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Giuseppe: <p>I solved with:</p>
<pre><code class="lang-auto">model = torchvision.models.resnet50()
model.avgpool = nn.Identity()
model.fc = nn.Identity()
</code></pre>
<p>I abilitate model.eval(). it went only in forward to obtain what i wanted.<br>
it is solved</p>

Yichong_Wang: <p><a class="mention" href="/u/giuseppe">@Giuseppe</a>  I use your code to replace the avgpool layer and fc layer with nn.Indentity(), but the output still be a vector. <img src="https://discuss.pytorch.org/images/emoji/apple/dizzy_face.png?v=12" title=":dizzy_face:" class="emoji" alt=":dizzy_face:" loading="lazy" width="20" height="20"></p>

Giuseppe: <p>The vector is feature map. It is right!</p>
Does pruning improve inference speed/memory usage?

ajktym94: <p>I have been using the concept of <strong>Lottery Ticket Hypothesis</strong> to iteratively prune Resnet models with the final goal of improving inference speed and memory usage. LTH was implemented using the <code>nn.utils.prune</code> package. I have noticed that even with 80% of the weights removed, the memory usage did not improve. However, the inference speed improved by a small margin (1-1.5 seconds). I have tried this in several ways:</p>
<ol>
<li>Global pruning and keeping <code>weight_orig</code> and <code>weight_mask</code>
</li>
<li>Global pruning and removing <code>weight_orig</code> and <code>weight_mask</code> with <code>prune.remove</code>
</li>
<li>Removing the above parameters and converting the tensors to sparse format (which resulted in error while doing inference)</li>
</ol>
<p>There was no improvement in any of the above cases. However, compressing the models after removing the extra parameters helped in reducing the size of the model <strong>on disk</strong> by 75%.</p>
<p>Is this situation expected?</p>
Gaussian_blur compatible with Pytorch/TensorRT

Miguel_Campos: <p>Hello  guys.</p>
<p>I have a neural network that applies torchvision.transforms.GaussianBlur at the end of the forward method.</p>
<p>If I try to convert the network to TensorRT engine with <a href="https://github.com/pytorch/TensorRT" rel="noopener nofollow ugc">Pytorch/TensorRT</a> it does not work because it is not compatible with torchvision transformations.</p>
<p>I saw a interesting approach in the <a href="https://github.com/pytorch/vision/blob/6279089ad27d56ec29af5352d9249db66b1d38a9/torchvision/models/detection/transform.py#L74" rel="noopener nofollow ugc">GeneralizedRCNNTransformation</a> module, which seems to be compatible with Pytorch/TensorRT as is torch based and I would like to do something similar with the GaussianBlur.</p>
<p>Could you suggest me the way to do? Should I re-code the source code of GaussianBlur? Could be Kornia compatible?</p>
<p>Thanks you</p>