akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Usage of register_parameter

Alibay_Alili: <p>When I look through the codes, noticed that the main usage of <code>model.register_parameter(name, param)</code> is assigning <code>None</code> to the parameters. I have the following toy example:</p>
<pre><code class="lang-auto">class MyModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.register_parameter('t', None)
        
    def forward(self, x):
        pass

my_model = MyModel()
</code></pre>
<p>I thought that when I call <code>list(my_model.named_parameters())</code>, I will see my parameter named <code>t</code>. But the result is empty list. So 2 questions came to my mind:</p>
<ol>
<li>
<p>Instead of <code>register_parameter(...)</code>, what about just using <code>self.t = None</code>.<br>
PS: Ignoring that some sanity checks  <code>register_parameter(...)</code> is using. <a href="https://github.com/pytorch/pytorch/blob/7aa605ed92dbb726a2f285246369b0a17972ba12/torch/nn/modules/module.py#L138" class="inline-onebox" rel="noopener nofollow ugc">pytorch/module.py at 7aa605ed92dbb726a2f285246369b0a17972ba12 · pytorch/pytorch · GitHub</a></p>
</li>
<li>
<p>What about about using <code>self.t = nn.Parameter(None)</code>. In this case, even <code>list(my_model.named_parameters())</code> is not empty</p>
</li>
</ol>
<p>What is wrong with these approaches?</p>
transoforms.Normalize grayscale image

Ardeal: <p>Hi,</p>
<p>I normalize grayscale image by the following code. however, I got the following error:</p>
<pre><code class="lang-python">ValueError: Expected tensor to be a tensor image of size (..., C, H, W). Got tensor.size() = torch.Size([64, 64])

</code></pre>
<pre><code class="lang-python">image = cv2.imread(path, -1)
image = torch.from_numpy(image).type(torch.float)
image = transforms.Normalize([0.5, ], [0.5, ])(image)

</code></pre>

ptrblck: <p>Your <code>image</code> seems to be missing the channel dimension as given in the error message:</p>
<pre><code class="lang-python">image = torch.randn(64, 64)
image = transforms.Normalize([0.5, ], [0.5, ])(image)
# ValueError: Expected tensor to be a tensor image of size (..., C, H, W). Got tensor.size() = torch.Size([64, 64])

image = torch.randn(1, 64, 64)
image = transforms.Normalize([0.5, ], [0.5, ])(image) # works
</code></pre>
<p>so <code>unsqueeze</code> it before passing the input to the transformation.</p>

Ardeal: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
thank you!<br>
<code>unsqueeze</code> really works.</p>

Ardeal: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
I got one more weird issue:</p>
<pre data-code-wrap="Python"><code class="lang-nohighlight"># load binary image 64X64 with 0 and 255 value in the image.
self.data_transforms = transforms.Compose([
                transforms.Resize((28, 28)),
                transforms.Normalize([0.485,], [0.229])
            ])
image= torch.unsqueeze(torch.from_numpy(image).type(torch.float), 0)
image = self.data_transforms(image)
</code></pre>
<p>after the above code, the normalized data in image is very big, like:<br>
<code>[[[ 458.0684,  713.7275,  713.7274,  713.7275,  713.7275, 1111.4192,            969.3862,   -2.1179,   -2.1179,   -2.1179,   -2.1179,   -2.1179,            742.1346, 1111.4192, 1111.4192, 1111.4192, 1111.4192, 1111.4192,           1111.4192,  367.1666,   -2.1179,   -2.1179,   -2.1179,   -2.1179,            554.6549, 1111.4192, 1111.4192, 1111.4192], ......</code></p>
<p>Based the data normalized by the above code, the training accuracy keeps 10% unchanged during the training.</p>
<p>If I remove the transforms.Normalize, the training accuracy could be 98.50%.</p>
<p>The above 2 experiments prove that my code about how to use transforms.Normalize is not correct.</p>
<p>I would like to make it work, which may make the accuracy better.</p>

ptrblck: <p>The used stats in <code>Normalize</code> assume the input tensor has values in the range <code>[0, 1]</code>, which doesn’t seem to be the case.<br>
Check the <code>min</code> and <code>max</code> values of <code>image</code> before passing it to <code>self.data_transform</code> and make sure the <code>Normalize</code> stats fit its range.</p>
Understanding when and how PyTorch calculates the average gradient when averaging loss for batches

jkking: <p>I understand that a main motivation for updating our networks via batches (or mini-batches) is that PyTorch is able to update using the average gradient found from all losses in the batch. This then makes the distinction between “averaging the gradient” and “averaging the loss,” but leaves me a bit confused since generally we manipulate losses.</p>
<p>Is calling <code>.mean()</code> on a loss tensor and then <code>.backward()</code> then same as backpropagating on the average gradient, or is there more nuance? And is this always true?</p>
<p>I ask “is it always” as I’m thinking particularly of a RL application I’m toying with where I don’t have true batches. Rather I calculate losses per episode (zero-dimensional, just number with grad), and I could turn those losses into a pseudo-batch by appending them to a list. I would then wonder if doing <code>torch.tensor()</code> on that list, <code>tensor.mean()</code>, and <code>my_pseudo_batch_loss.backward()</code> provides the same “average gradient” functionality of typical batch updates.</p>
Extract c3d features, given video

Rafael_R: <p>Hi,</p>
<p>How to extract c3d features in Pytorch?</p>

ptrblck: <p>Could you explain, what c3d features are?</p>

Rafael_R: <p>its a 3d convolution model for videos <a href="https://github.com/facebookarchive/C3D" rel="nofollow noopener">https://github.com/facebookarchive/C3D</a></p>

ptrblck: <p>Would you like to extract the output activations from the 3d conv layers?<br>
If so, you could use <code>forward_hooks</code> as described <a href="https://discuss.pytorch.org/t/how-can-l-load-my-best-model-as-a-feature-extractor-evaluator/17254/6">here</a></p>

Gorgen: <p>Hi Rafael： could you please give me a hand to extract c3d features, given video?</p>
<p>Thanks<br>
best</p>
About loading weights to model

hitbuyi: <p>I want to know whether Pytorch gives any information if  the weights doesn’t fit the model</p>

ptrblck: <p>Yes, PyTorch will raise shape mismatch errors and will fail if additional or unexpected keys are found in the <code>state_dict</code> unless you specify <code>strict=False</code> while loading the <code>state_dict</code>.</p>
Gradients w.r.t. loss component

SofiaCP: <p>Hi,</p>
<p>Let’s assume I have a loss L with two components L= L1 + L2<br>
For each convolution in a CNN, I want to update some filters only with L1 and others with L2. How can this be achieved without having to back prop multiple times, which takes forever? Is there a way to decompose the gradient into the two parts corresponding to each term and select only one?</p>
<p>Thanks!</p>

ptrblck: <p>I think one valid approach would be to use <code>torch.autograd.grad</code> specifying the <code>inputs</code> for the desired filters and calculate the gradients using both losses separately. Alternatively, the <code>backward</code> method also accepts the <code>inputs</code> argument and could be used in a similar way.</p>

SofiaCP: <p>Thanks for your reply. I was using the <code>torch.autograd.grad</code> approach actually, but it is very slow. Isn’t there a way of doing this without having to backprop multiple times?</p>
<p>Thank you!</p>
How to print the weight after torch.quantization.convert?

davidhung.c: <p>I use the QAT method provided by Pytorch. After using the torch.quantization.convert, I can’t get the weight when print(model.conv.weight), I get the message bellow:<br>
“&lt;bound method Conv2d.weight of QuantizedConv2d(16, 16, kernel_size=(3, 3), stride=(1, 1), scale=1.0, zero_point=0, padding=(1, 1), groups=16, bias=False)&gt;”</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/0/00f6cacb2b7c708f81097c224797ac188d3ca756.png" data-download-href="https://discuss.pytorch.org/uploads/default/00f6cacb2b7c708f81097c224797ac188d3ca756" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/0/00f6cacb2b7c708f81097c224797ac188d3ca756_2_690x22.png" alt="image" data-base62-sha1="8wKu79m3zvOVCzkYdNL4N99vhk" width="690" height="22" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/0/00f6cacb2b7c708f81097c224797ac188d3ca756_2_690x22.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/0/00f6cacb2b7c708f81097c224797ac188d3ca756_2_1035x33.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/0/00f6cacb2b7c708f81097c224797ac188d3ca756.png 2x" data-dominant-color="363535"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1272×41 19.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>How to print the weight after torch.quantization.convert?<br>
Thanks!</p>

constant_learner: <p><a href="https://discuss.pytorch.org/t/how-to-extract-individual-weights-after-per-channel-static-quantization/67456/2"> this is the solution!</a></p>
<p>OK. It seems like we can use<br>
<code>layer[0].weight().int_repr().data</code><br>
to get the INT8 representation of the weight entries.</p>
<p>Also,<br>
<code>layer[0].weight().dequantize()</code><br>
gives the tensor of the weights in FP format to have element-wise access to its contents.</p>
Testing.assert_close gives different max difference than manual calculation

Vedant_Roy: <p>The following 2 snippets give different results for the same tensors:</p>
<pre><code class="lang-auto">testing.assert_close(a, b) # prints "Greatest absolute difference: 0.001220703125"
</code></pre>
<pre><code class="lang-auto">print(f"Max diff (flash): {(a - b).abs().max()}") # prints "Max diff (flash): 0.03125"
</code></pre>
<p>Can someone explain how this might be possible?</p>

ptrblck: <p>I get the same result:</p>
<pre><code class="lang-python">a = torch.randn(10, 10)
b = a.clone()
b[0, 0] *= 0.1
torch.testing.assert_close(a, b)
# Mismatched elements: 1 / 100 (1.0%)
# Greatest absolute difference: 0.8026720210909843 at index (0, 0) (up to 1e-05 allowed)
# Greatest relative difference: 8.999999832919992 at index (0, 0) (up to 1.3e-06 allowed)

print((a - b).abs().max())
# tensor(0.8027)
</code></pre>
<p>so could you post a minimal, executable code snippet reproducing the issue, please?</p>
Element-wise multiplication for model parameters and self-defined mask in traing workflow

terrytengli: <p>There is a stupid question from a PyTorch beginner.</p>
<ul>
<li>Here is a base model, I denote the model as $f(w_t, x)$.</li>
<li>There is also a mask that is the same size with the weight of $f(w_t, x)$. Let it to be denoted as $w_{m}$. For each element in $w_m$, they are sampled in the range of [0, 1].</li>
</ul>
<p>My question is, can we integrate the mask $w_m$ into the training workflow and only calculate the grade derivation on the parameters of model $f(w_t, x)$. In the training workflow, the new model will be like $f(w_t * w_m, x)$.</p>
<p>Just wanna update the $w_t$ partition of parameters, while keeping the untrainable property of $w_m$.</p>

ptrblck: <p>I don’t know how the mask is calculated etc. but wouldn’t masking the gradients after the <code>backward</code> call work?</p>

terrytengli: <p>Hi, <a class="mention" href="/u/ptrblck">@ptrblck</a>. Thanks for your reply and sorry for my imprecise declaration.</p>
<p>The details are as follows.</p>
<p>A DNN model is defined and we denote it as <code>m</code>. In PyTorch, it can be represented as,</p>
<pre><code class="lang-auto">m = Model()
</code></pre>
<p>I wanna split the parameters in <code>m</code> into <code>theta</code> and <code>m</code>. All of them are the same in shape. The relationship can be represented as $w = \theta \odot m$, where the symbol $w$ denotes the weight parameters in <code>m</code>.</p>
<p>In the training workflow, I update the parameters separately and step-by-step. Specificly, I obtain the <code>w</code> by combining the <code>theta</code> and <code>m</code> using $w = \theta \odot m$, and set <code>w</code> as the parameters of <code>m</code>. In the first round, I wanna only to update the <code>theta</code> and fix <code>m</code>. Different in the second round, I wanna only to update the <code>m</code> and fix <code>theta</code>. So back and forth.</p>
<p>What confused me was that I do not have any ideas on the implementation of separated parameters updating of <code>w</code>, <code>theta</code>, and <code>m</code>.</p>

terrytengli: <blockquote>
<p>I don’t know how the mask is calculated etc.</p>
</blockquote>
<p>Another view to understand the mask is that the mask <code>m</code> and the <code>theta</code> are two sets of parameters of the model <code>m</code>. The difference between <code>theta</code> and <code>m</code> is the value range, <code>m</code> is restricted to [0, 1] while <code>theta</code> is freedom.</p>
<blockquote>
<p>but wouldn’t masking the gradients after the <code>backward</code> call work?</p>
</blockquote>
<p>Thanks for your suggestions.</p>
<p>Masking the gradient after <code>backward</code> is not what we want, as we need all parameters of <code>m</code> or <code>theta</code> (depending on the round) to be updated, not the specific partition.</p>
Manually bias addition after convolution

Prakhar_Pradhan1: <p>I want to manually add the bias after convolution of input and weights. But output of convolution is 4-D tensor like torch.Size(1,3,30,30) and bias is of size torch.Size([3]), what reshape operation should I use to perform addition</p>

ptrblck: <p>You could <code>unsqueeze</code> the <code>bias</code> or add new dims by indexing it with <code>None</code>s:</p>
<pre><code class="lang-python">out = torch.randn(2, 3, 24, 24)
bias = torch.randn(3)

out = out + bias[None, :, None, None]
# or
out = out + bias.unsqueeze(0).unsqueeze(2).unsqueeze(3)
</code></pre>

Prakhar_Pradhan1: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> for your help.<br>
I am able to do this also using<br>
bias = bias.view(-1, 1, 1).expand_as(out)</p>
Data and model both sent to CUDA

Yoo: <p>Once we send the model to CUDA, should any input data to the model be in CUDA as well?<br>
What I found was that data in tensor did not need to be in CUDA as an input for a model which was sent to CUDA before.<br>
Is there any golden rule for this issue?</p>

ptrblck: <p>Yes, the input data also has to be on the GPU and PyTorch will fail otherwise:</p>
<pre><code class="lang-python">model = nn.Linear(10, 10).cuda()
x = torch.randn(1, 10)

out = model(x)
# RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument mat1 in method wrapper_addmm)

x = x.cuda()
out = model(x)
print(out.shape)
# torch.Size([1, 10])
</code></pre>
<p>I don’t know why it was not failing in your use case so could you describe a bit more what exactly you were running?</p>

Yoo: <p>So I defined a predict function in nn class which returns a scalar for class prediction</p>
<pre><code class="lang-auto">def predict(self, list):
      list = torch.LongTensor(list)
      return torch.argmax(self(list)).detach().numpy()
</code></pre>
<p>I called it in training loop after I sent my initialized model to CUDA. I thought self( ) would require something in CUDA but in this case I did not send the longtensor list to CUDA. And it worked even without warning</p>

ptrblck: <p>Could you post a minimal, executable code snippet reproducing this behavior, please?</p>
Can using only a portion of samples within a mini-batch be a problem?

joohyunglee: <p>I want to train an encoder, which contains batch-normalization layers, with two different losses. I use all the samples from the minibatch to compute <code>loss1</code>, whereas I can only use a portion of samples within a mini-batch to compute <code>loss2</code> (due to the heavy computation burden of <code>loss2</code>). *More specifically, the <code>loss2</code> is a variant of noise contrastive loss for which I want to use only a portion of samples for its negative sample. In summary:</p>
<ol>
<li>the encoder with (sync) batch normalization layer converts N samples into Nx2048 (2048 for arbitrary feature dimension)</li>
<li>
<code>loss1</code> uses all samples (Nx2048) to compute the loss</li>
<li>
<code>loss2</code> uses only a portion of minibatch, e.g. N/16x2048, to compute the loss</li>
</ol>
<pre><code class="lang-auto">class Model_with_losses(nn.Module):
def __init__(self, loss1, loss2):
   self.encoder = resnet50()
   self.loss1 = loss1
   self.loss2 = loss2
...
def forward(self, img):
   feature = self.encoder(img)
   l1 = self.loss1(feature)
   l2 = self.loss2(feature[:8, ::]) # feature.size(0) &gt;&gt; 8, e.g., 64
   loss = l1 + l2
   return loss # which will be backproped, e.g., loss.backward()
</code></pre>
<p>Will there be a problem?<br>
Thank you!!</p>

ptrblck: <p>I don’t think you should expect to see any problems besides the obvious limitations that the gradients from <code>loss2</code> would only be calculated based on the indexed features while the rest will be ignored, but this seems to fit your use case.</p>
Ensemble multiple models together

Z_Huang: <p>I am thinking of using two pretrained CNN models and stacking (soft voting for image classification) them together to generate test results.</p>
<p>In test (2 models are pretrained), I let 2 models to generate 2 sets of predictions. Then, I use softmax to pull out the predictive probabilities. According to the predictive probabilities, I let the predictive results with the highest probabilities represent the final output.</p>
<p>I found a tutorial online: <a href="https://medium.com/@alexppppp/how-to-train-an-ensemble-of-convolutional-neural-networks-for-image-classification-8fc69b087d3" class="inline-onebox" rel="noopener nofollow ugc">How to Train an Ensemble of Convolutional Neural Networks for Image Classification | Medium</a></p>
<p>Basically, their practice is to ensemble multiple models together and link them via a fully-connected layer. Therefore, only the fully-connected layer is trainable.</p>
<p>However, my method is to allow multiple models to predict by themselves and use the probabilities to decide the final result.</p>
<p>I am not sure which is an appropriate way to do ensemble learning. Any idea?</p>

ptrblck: <p>Using voting on different model predictions sounds valid and you could use <code>scikit-learn</code> for it which provides different voting approaches.<br>
A trainable multi-stage ensemble also sounds valid, but might be more complicated in its training.</p>
Is cuda_graph compatible with DistributedDataParallel

woolpeeker: <p>I test the code as follows, the mode is a DistributedDataParallel module</p>
<pre><code class="lang-auto">s = torch.cuda.Stream()
s.wait_stream(torch.cuda.current_stream())
with torch.cuda.stream(s):
    for i in range(3):
        train_step(model, inputs, targets, optimizer, scaler, update=True)
torch.cuda.current_stream().wait_stream(s)

g = torch.cuda.CUDAGraph()
optimizer.zero_grad(set_to_none=True)
with torch.cuda.graph(g):
    train_step(model, inputs, targets, optimizer, scaler, update=True)
</code></pre>
<p>It reports "<br>
File “/mnt/lustressd/luojiapeng/projects/deepspeed_tests/tools/run_ddp.py”, line 74, in train_step<br>
outputs = model(inputs)<br>
File “/mnt/cache/luojiapeng/miniconda3/envs/torch1.12/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 1130, in _call_impl<br>
return forward_call(*input, **kwargs)<br>
File “/mnt/cache/luojiapeng/miniconda3/envs/torch1.12/lib/python3.10/site-packages/torch/nn/parallel/distributed.py”, line 976, in forward<br>
self.logger.set_runtime_stats_and_log()<br>
RuntimeError: CUDA error: operation not permitted when stream is capturing<br>
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.<br>
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.<br>
"</p>

ptrblck: <p>Yes, CUDA Graphs are compatible with <code>DDP</code> as explained <a href="https://pytorch.org/docs/stable/notes/cuda.html#usage-with-distributeddataparallel">here</a>. As the doc explains, you need to run at least 11 warmup iterations in DDP-enabled eager mode before the capture is executed.</p>

woolpeeker: <p>Thanks for the quick reply. This information is very helpful. I’m still working on it, but not been successful yet.</p>
Best practices: A forked repo vs new repo

russellizadi: <p>I work with PyTorch and TorchAudio a lot and I create modules that I use in different projects. What are the following do you suggest?</p>
<ol>
<li>Add modules to the forked version of PyTorch and build from source in every project I am working.</li>
<li>Create another repo as an add-on to PyTorch and install that separately.</li>
</ol>

ptrblck: <p>I think it depends a bit on how you are planning to use these “modules” and if you are expecting other users to use them as well or if it’s mostly for your own use cases only.<br>
Writing an extension would allow other users to easily use your modules by just rebuilding/installing them, but would also need some maintenance effort on your side.</p>
RuntimeError: only batches of spatial targets supported (3D tensors) but got targets of size: : [32]

flummery: <p>I’m trying to break down the error I’m getting from the loss function. My train function is below:</p>
<pre><code class="lang-auto">model = model.to(device)
optimizer = torch.optim.SGD(model.parameters(), lr=LR)
criterion = nn.CrossEntropyLoss()

for epoch in range(2):
    for idx, (img, label) in enumerate(train_dl):
        print(img.shape, label.shape)
        
        img = img.to(device)
        label = label.to(device)
        
        optimizer.zero_grad()
        outputs = model(img)
        
        print(outputs.shape)
        loss = criterion(outputs, label)
        break
</code></pre>
<p>The out put is as follows:</p>
<pre><code class="lang-auto">img.shape: torch.Size([32, 3, 224, 224]), label.shape torch.Size([32])
outputs.shape: torch.Size([32, 64, 27, 5])
</code></pre>
<p>What am I doing wrong? Does it have something to do with my label shape?</p>

ptrblck: <p>Your output in the shape <code>[32, 64, 27, 5]</code> indicates a multi-class segmentation using 64 classes. For this use case the target should have a shape of <code>[32, 27 ,5]</code> containing class indices in the range <code>[0, 63]</code>.</p>
<p>If you are not working on a segmentation but a multi-class classification as the target shape of <code>[32]</code> indicates, make sure that your model output has the shape <code>[batch_size, nb_classes]</code>.</p>
Nested for and multiple tensorboard windows

papoo13: <p>I am working on continual learning and I train a model on multiple tasks. Each task is trained for a number of epochs. This means that I have a nested loop like the following:</p>
<pre><code class="lang-auto">for task_idx in range(num_tasks):
     train_model_on_task()
</code></pre>
<p>and the train function is:</p>
<pre><code class="lang-auto">def train_model_on_task():
      for epoch in range(num_epochs):
           avg_loss = TrainEpoch()
           avg_loss_eval = EvalEpoch()
</code></pre>
<p>I would like to see the acc anfd loss for the train and eval dataset of all the tasks but I don’t know how to give these commands to the tensoboard. Any help would be appreciated.</p>
How to parralelize scattered layer norm?

Michael_Moran: <p>I’m working with sets of different sizes with input dimensions [i,e] where i is the set element and e is the embdding. To avoid zero padding I batch along i and use the pytorch scatter library <a href="https://pytorch-scatter.readthedocs.io/en/latest/functions/scatter.html" class="inline-onebox" rel="noopener nofollow ugc">Scatter — pytorch_scatter 2.1.1 documentation</a> to to take the mean respecting the batching.</p>
<p>I’m having a lot more difficulty with getting layernorm to work.<br>
The input to my layer norm function is my batch with dimensions [i,e] and my “batch_mask” which provides the indicies where one batch ends and a new one begins. I take the mean of each individual batch, the std of each individual batch, then concatenate the results back together.</p>
<p>This is extremely slow and my gpu utilization is at 5% as a result. Is there a way I can make the following operation parralel given that the step size varies?</p>
<blockquote>
<p>new_x = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
for i,j in zip(Batch_Mask[“iterate”][:-1],Batch_Mask[“iterate”][1:]):<br>
_ = x[i:j]-torch.mean(x[i:j])<br>
new_x.append(_/torch.sqrt(torch.std(x[i:j])+eps))</p>
</blockquote>
Is using multiple (separate) QuantStubs useful?

yannbane: <p>Every quantization guide I read seems to gravitate towards using only one instance and call of <code>QuantStub.quant</code>: quantize the input, pass the quantized tensor through the model, and then dequantize the output.</p>
<p>However it makes sense for different layers/modules to use different quantization parameters (scale, zero-point).</p>
<ul>
<li>Am I wrong about this? Is there a reason no one mentions it? Is it that the de/quant ops add too much of an overhead and aren’t worth it for the accuracy gain?</li>
<li>In my initial experiments, using separate QuantStubs for <code>quant</code> operations <em>significantly</em> improves accuracy. Is there something I’m not seeing?</li>
</ul>
<p>At first I’ve been <em>forced</em> to use multiple <code>QuantStub</code>s because there are parts of my model (matmul/bmm primarily) that cannot be quantized. Now I’m wondering if there is an actual additional benefit to this.</p>

Vasiliy_Kuznetsov: <p>Hi <a class="mention" href="/u/yannbane">@yannbane</a>,</p>
<p>It does make sense for different layers to use different scales and zero_points. Observers are used to collect these statistics.  QuantStub is not the only place where observers are inserted - they are also often inserted at the output of quantizeable layers.  So, even though you might be only manually adding one QuantStub, the quantization framework will also add observers to other places.  For best accuracy, usually each point in the program which needs observation should use a separate observer.</p>
Could not run 'aten::int_repr' with arguments from the 'CPU' backend

wanghx: <p>I want to get the int value of each weight in the quantized model, using int_repr(), but I get a RuntimeError<br>
this is code</p>
<p>for weight in myModel.state_dict():<br>
if “weight” in weight:<br>
m = myModel.state_dict()[weight].int_repr()<br>
print(m)</p>
<p>RuntimeError: Could not run ‘aten::int_repr’ with arguments from the ‘CPU’ backend. ‘aten::int_repr’ is only available for these backends: [QuantizedCPU, Autograd, Profiler, Tracer].</p>

christophezei: <p>I have the same issue did you solve your problem ?</p>

Vasiliy_Kuznetsov: <p>This error message that you are trying to call <code>int_repr()</code> on a weight that is not quantized.  That function is only supported for quantized weights.  You could try checking if that function exists and only calling it if does.</p>
Is save_on_cpu() overlaps the computation and communication?

Cypher30: <p>I’m thinking of using <code>torch.autograd.graph.save_on_cpu()</code> to deal with some offload process, but I’m curious of whether the <code>saved_tensors_hooks</code> mechanism overlaps the computation and communication while executing, i.e. the communication operations such as offload could be done without blocking the computation. This could be helpful as the code with context manager will be more concise compare to the CUDA stream version.</p>

ptrblck: <p>I haven’t profiled this context manager, but based on <a href="https://github.com/pytorch/pytorch/blob/b136f3f310aa01a8b3c1e63dc0bfda8fd2234b06/torch/autograd/graph.py#L126">this code</a> the <code>packed</code> tensor is using pinned memory, so the copy should be non-blocking.<br>
The <code>copy_</code> call seems to be missing the <code>non_blocking=True</code> argument, so unsure if that’s on purpose or just missed.</p>
<p>CC <a class="mention" href="/u/varal7">@Varal7</a> as you seem to be the author: did you miss to use the <code>non_blocking=True</code> argument in the <a href="https://pytorch.org/docs/stable/generated/torch.Tensor.copy_.html#torch-tensor-copy"><code>copy_</code></a>?</p>

Cypher30: <p>Thanks a lot for your in time reply! Looking forward to the author’s reply!</p>

Varal7: <p>Hi, Thank you for the tag!</p>
<p>I vaguely remember that using <code>non_blocking=True</code> from GPU to CPU might be dangerous (<a href="https://discuss.pytorch.org/t/should-we-set-non-blocking-to-true/38234/18" class="inline-onebox">Should we set non_blocking to True? - #18 by sbelharbi</a>), so we only use when copying from CPU to GPU.</p>
<p>For reference, this is the PR that introduced the feature: <a href="https://github.com/pytorch/pytorch/pull/61928" class="inline-onebox" rel="noopener nofollow ugc">Add default hooks to save tensors on CPU by Varal7 · Pull Request #61928 · pytorch/pytorch · GitHub</a></p>
How to make vgg pytorch's ptl size smaller on android?

gclsoft: <pre><code class="lang-auto">import torch
# import joblib
from torch.utils.mobile_optimizer import optimize_for_mobile
from torchvision.models.vgg import vgg16
import torch, torchvision.models

# lb = joblib.load('lb.pkl')
device = torch.device('cuda:0')
#device = torch.device('cpu')#'cuda:0')
torch.backends.cudnn.benchmark = True
model = vgg16().to(device)



# model = torchvision.models.vgg16()
path = 'model-22222.pth'
torch.save(model.state_dict(), path) # nothing else here
model.load_state_dict(torch.load(path))

#model.load_state_dict(torch.load('./model-76-0.7754.pth'))

scripted_module = torch.jit.script(model)
optimized_scripted_module = optimize_for_mobile(scripted_module)
optimized_scripted_module._save_for_lite_interpreter("model-76-0.7754.ptl")
</code></pre>
<p>The <code>optimize_for_mobile</code> seems does not make the ptl file smaller, it’s about 527M, too large on Android. How to make it smaller?</p>

wi-sh: <p>You first need to ** Quantized** the model (This will reduce the size) and then convert it into torch script for it to run on android.</p>
<p><a href="https://pytorch.org/tutorials/intermediate/quantized_transfer_learning_tutorial.html#part-1-training-a-custom-classifier-based-on-a-quantized-feature-extractor" rel="noopener nofollow ugc">https://pytorch.org/tutorials/intermediate/quantized_transfer_learning_tutorial.html#part-1-training-a-custom-classifier-based-on-a-quantized-feature-extractor</a><br>
<a href="https://pytorch.org/tutorials/recipes/script_optimized.html" rel="noopener nofollow ugc">https://pytorch.org/tutorials/recipes/script_optimized.html</a></p>

gclsoft: <p>Is there any other solutions? For example using onnx , ncnn or transform into tensorflow lite? Thanks. Because the size only smaller 50%, how to reduce to about 5%. The size is too big yet. Not fit for the mobile phone users.</p>
Kernel dies in Jupyter Notebook when I run 'import torch'

Ron_Itelman: <p>My Jupyter Notebook kernel dies when I run ‘import torch’. I installed it through pip. Screenshots below.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/d/fd5e1d68766ff792ece05b08be7c479678d6c11b.png" data-download-href="https://discuss.pytorch.org/uploads/default/fd5e1d68766ff792ece05b08be7c479678d6c11b" title="install"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/d/fd5e1d68766ff792ece05b08be7c479678d6c11b_2_690x156.png" alt="install" data-base62-sha1="A9oribhlwJLprskG2MGX2lriB07" width="690" height="156" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/d/fd5e1d68766ff792ece05b08be7c479678d6c11b_2_690x156.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/d/fd5e1d68766ff792ece05b08be7c479678d6c11b_2_1035x234.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/d/fd5e1d68766ff792ece05b08be7c479678d6c11b_2_1380x312.png 2x" data-dominant-color="EAEAEA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">install</span><span class="informations">2264×512 143 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Ron_Itelman: <p>This was resolved when I uninstalled and reinstalled openmp:<br>
conda install -c intel openmp -n base</p>
Where is source code of Tensor.bmm?

suniuin: <p>hi, i can’t find source defination of Tensor.bmm which i think is a matrix operation method? can kindly tell which file has it  even written in c++ I am still interesting. thanks.</p>

Michael_Suo: <p>Look in <a href="https://github.com/pytorch/pytorch/tree/master/aten/src/ATen/native" rel="nofollow noopener">https://github.com/pytorch/pytorch/tree/master/aten/src/ATen/native</a> for the relevant implementations</p>

woolpeeker: <p>I still cannot find it?<br>
Does anyone has idea?</p>