akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How could I save state dict in int8 format and load back to float32?

username: coincheung
cooked: <p>Hi,</p>
<p>I would like to use quantization method to save my model state dict with int8 data type so that I could save my storage space. And I also need to load these int 8 states back to a float 32 model for inference. How could I do this please ?</p>

username: Vasiliy_Kuznetsov
cooked: <p>This isn’t a standard flow PyTorch quantization provides, but you could do something like this:</p>
<ol>
<li>for a Tensor, use <code>torch.quantize_per_tensor(x, ...)</code> to convert fp32 -&gt; int8, and <code>x.dequantize()</code> to convert from int8 to fp32.</li>
<li>override the <code>_save_to_state_dict</code> and <code>_load_from_state_dict</code> functions on the modules you’d like to do this on to use your custom logic.  You can call <code>quantize_per_tensor</code> when saving, and convert it back with <code>dequantize</code> when loading.</li>
</ol>

username: coincheung
cooked: <p>Hi,</p>
<p>Thanks for replying !!! There is a scale and bias term in the function <code>torch.quantize_per_tensor</code>. When I convert float32 parameter to int8 parameter, how could I determine these args? Is there any inner processing method, or should I figure out the max/min values of each tensor myself ?</p>

username: Vasiliy_Kuznetsov
cooked: <p>You can use <a href="https://github.com/pytorch/pytorch/blob/master/torch/quantization/observer.py" class="inline-onebox" rel="noopener nofollow ugc">pytorch/observer.py at master · pytorch/pytorch · GitHub</a>, this is also what the official quantization flows use.  For example,</p>
<pre><code class="lang-auto">obs = torch.quantization.MinMaxObserver(...)
# run data through the observer
obs(your_tensor)
# get scale and zp from the data observer has seen
scale, zp = obs.calculate_qparams()
</code></pre>
<p>One other thing to consider could be using fp16, that way you could just do <code>x.half()</code> without worrying about quantization params.</p>
Find the joint eigenvalues of two matrices

username: bsun0802
cooked: <p>Is there a function in pytorch which can find the joint eigenvalues of two matrices? MATLAB has an interface for it.</p><aside class="onebox stackexchange">
  <header class="source">
      <a href="https://stackoverflow.com/questions/36551182/how-can-i-find-the-joint-eigenvalues-of-two-matrices-in-matlab" target="_blank" rel="noopener nofollow ugc">stackoverflow.com</a>
  </header>
  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/2439763/michael" target="_blank" rel="noopener nofollow ugc">
    <img alt="Michael" src="https://www.gravatar.com/avatar/32ae69ccccd62a9691a4f1ccc70da7a4?s=128&amp;d=identicon&amp;r=PG" class="thumbnail onebox-avatar" width="60" height="60">
  </a>
<h4>
  <a href="https://stackoverflow.com/questions/36551182/how-can-i-find-the-joint-eigenvalues-of-two-matrices-in-matlab" target="_blank" rel="noopener nofollow ugc">How can I find the joint eigenvalues of two matrices in MATLAB?</a>
</h4>

<div class="tags">
  <strong>matlab, matrix, linear-algebra, eigenvalue</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/2439763/michael" target="_blank" rel="noopener nofollow ugc">
    Michael
  </a>
  on <a href="https://stackoverflow.com/questions/36551182/how-can-i-find-the-joint-eigenvalues-of-two-matrices-in-matlab" target="_blank" rel="noopener nofollow ugc">02:12PM - 11 Apr 16 UTC</a>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
Using GPU in dataloader with num_workers>0

username: rmahfuz
cooked: <p>I want to achieve the following in the dataloader (in the <strong>getitem</strong> method):<br>
Converting loaded data to a tensor on the GPU, and using the GPU to perform some processing on the tensor. When I set num_workers&gt;0, I get the following error:<br>
RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the ‘spawn’ start method</p>
<p>I am wondering if there is an established method to perform GPU processing in the dataloader with num_workers&gt;0</p>

username: ptrblck
cooked: <p>Have you tried using the mentioned <code>'spawn'</code> method and if so what kind of error are you seeing?</p>

username: rmahfuz
cooked: <p>No I have not tried using the ‘spawn’ method. I will try that and update you if I have a problem.</p>
How to get the logit value in semanatic segmentation for the assigned class?

username: maze09
cooked: <p>Hi everyone,</p>
<p>I got 4D logit value matrices with the shape (batchsize, classes (e.g. 9 logit values if number of classes ==9), height, width) and the assigned class matrices (label matrices) which are 3D and has the shape (batchsize, 1 (which are the indices of the true class), height, width). How can the logits values corresponding to the assigned class accessed easily.  Basically, use the label matrices as index for the logit matrices. The label matrix additionally contains default values which are 255 to exclude these pixels from gradient calculation. Hence, these pixels has to be filtered out first.</p>
<p>Thank you in advance!</p>
<p>Best,<br>
Matthias</p>

username: ptrblck
cooked: <p>You could use the <code>gather</code> operation and filter out the specific values afterwards:</p>
<pre><code class="lang-python">logits = torch.randn(2, 3, 4, 4)
target = torch.randint(0, 3, (2, 4, 4))
res = logits.gather(1, target.unsqueeze(1))
</code></pre>

username: maze09
cooked: <p>Hi Peter,<br>
thanks for your reply. I forgot to delete the question. I also used the gather operation and some masking to filter out values.</p>
<p>Best Matthias</p>
What's the difference between Variable.detach() and Variable.clone()

username: Gosicfly
cooked: <p>Is tha detach() share the same memory , but clone() creates a new memory?</p>

username: richard
cooked: <p>That’s correct, that is the main difference. <code>x</code> and <code>x.detach()</code> share the same memory, but <code>x.clone()</code> makes new memory.</p>
<p>There are also some subleties around <code>requires_grad</code> (and possibly other things). If <code>x</code> has <code>requires_grad=True</code>, then <code>x.clone()</code> also has <code>requires_grad=True</code> but <code>x.detach()</code> doesn’t.</p>

username: Gosicfly
cooked: <p>Thank you.<br>
I used detach() to prevent grads back,like stop_gradient() in Tensorflow. Is this correct? If not, what should be done to prevent the grad back ?</p>

username: richard
cooked: <p>To my understanding <code>stop_gradient()</code> in Tensorflow treats the thing as a constant. Sending in <code>x.detach()</code> to a nn layer will prevent gradients from being computed for <code>x</code>, so I believe the behavior is the same</p>

username: kush8singh
cooked: <p>Hi,<br>
I was playing around with <code>.clone</code> and had a little confusion with this piece of code</p>
<pre><code class="lang-auto">x = torch.tensor([3],requires_grad=True)
y = x.clone()
z = y**2
z.backward()

print y.grad # None
print x.grad # tensor([6])
</code></pre>
<p>From reading your reply , i expected a new leaf node, completely seperate from the first graph , but with values of <code>x</code> . But , here it seems like <code>y</code> is a part of the original graph .<br>
Could you please explain what is going on . Also , if possible could you explain where <code>.clone</code> can be useful using a simple example (possibly code).</p>
<p>Thanks ,<br>
Kush</p>

username: acgtyrant
cooked: <p>y is computed by the <code>clone</code> operation in <code>x</code>, so it is not a leaf variable that it will does not have its <code>grad</code>. But the backpropagation will though it to <code>x</code>, so <code>x</code> will get the grad.</p>
A question about saving the network mdoel

username: Ford
cooked: <p>there are two formats can be used when saving a trained network model, which are .pth and .mdl, respectively. I wonder what is the difference between these two formats? thanks.</p>

username: Jiajun_Zha
cooked: <p>I think the file extension doesn’t make a difference itself. The way you saved it actually matters.<br>
As long as you use the builtin torch.save(), the saved file is in a deterministic format not matter what file extension you give it.</p>

username: neuralpat
cooked: <p>That is my understanding as well.</p>
How to save dictionary with state_dict to json

username: nicofish
cooked: <p>I ran training on with all permutations of hyper parameters at multiple levels (batch_size, hidden_size, learn_rate). I saved the result of these training’s into a dictionary as follows:</p>
<pre><code class="lang-auto">modelList, accList, targList, predList = kFoldCross(model = model[str(params)], 
                                                    data = data, 
                                                    batch_size=batch_size, 
                                                    num_folds=5, 
                                                    device=device,
                                                    train_func = train_modelEdit,
                                                    learn_rate= learn_rate,
                                                    num_epoch= 200,
                                                    patience = 10)

 #make dictionary to store outputs
    bigDict[str(params)] = {'state_dict': modelList, 
                            'accuracy': accList, 
                            'target': targList, 
                            'prediction': predList}
</code></pre>
<p>where params is tuple of the hyper parameters i.e. (64, 32, .01)  (batch size 64, hidden_size 32, learn_rate .01). modelList is a list where each entry contains the state_dict of the model at a single run of the kfold validitation(5 state_dicts in total). The other lists (accList, targList, and predList) contain ouputs from each of the 5 runs of the kfold.</p>
<p>I tried to save bigDict to json as follows:</p>
<pre><code class="lang-auto">with open('bigDict_ConvLSTM_incomplete.json', 'w') as fp:
    json.dump(bigDict, fp)
</code></pre>
<p>but I get the following error:</p>
<pre><code class="lang-auto">TypeError: Object of type Tensor is not JSON serializable
</code></pre>
<p>which presumably refers to the torch.tensors in the state_dict.</p>
<p>Is there a way I can save the entire dictionary to json or do I have to save the model state_dict separately?</p>
<p>In the event that bigDict cannot be saved:<br>
I know I could save the state_dicts individually using torch.save(), but I do not want to have a bunch of different files. Is there at least a way to save them all together?</p>
<p>note: I am open to using any other format not necessarily json to save my files. Any advice is greatly appreciated.</p>
<p>Thank you</p>

username: ptrblck
cooked: <p>I would try to store everything in a binary format to avoid the serialization into the (human-readable) JSON format, e.g. by using <code>pickle</code> or <code>torch.save</code> on the <code>dict</code> directly.</p>
A question about finetuning a pre-trained Mask R-CNN model

username: ronghui
cooked: <p>what happened if we dont feed boxes into network? Can we still get mask from it?</p>

username: ptrblck
cooked: <p>During training <code>MaskRCNN</code> expects an input and target tensor containing <code>boxes</code>, <code>labels</code>, and <code>masks</code>, while only the input is required to return the aforementioned tensors as well as the <code>scores</code>.</p>
Add normalization layer in the beginning of a pretrained model

username: nakulp21
cooked: <p>I’m  using a pretrained UNet model whose first encoder has the following architecture</p>
<pre><code class="lang-auto">UNet(
  (encoder1): Sequential(
    (enc1conv1): Conv2d(3, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
    (enc1norm1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (enc1relu1): ReLU(inplace=True)
    (enc1conv2): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
    (enc1norm2): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (enc1relu2): ReLU(inplace=True)
  )
</code></pre>
<p>The model takes an input which has been normalized using min-max normalization. Instead, I want to add a batch/layer norm layer at the beginning so that I can feed the image as it is without normalization.</p>
<p>I don’t want to use torchvision.transforms to normalize the image, instead I want to add a layer at the beginning that does the same work for me.</p>
<p>Sorry, if this question has some flaws, I’m new to Pytorch.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="nakulp21" data-post="1" data-topic="107406">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nakulp21/48/32313_2.png" class="avatar"> nakulp21:</div>
<blockquote>
<p>I don’t want to use torchvision.transforms to normalize the image, instead I want to add a layer at the beginning that does the same work for me.</p>
</blockquote>
</aside>
<p>Batchnorm layers do not work in the same way as the <code>transforms.Normalize</code> transformation, since the latter uses static statistics while the batchnorm layer uses running estimates during validation and the batch statistics during training.</p>
<p>However, in case you still want to add this layer:<br>
based on the model description, I assume you are using a custom <code>nn.Module</code>, which uses <code>self.encoder1</code> as one submodule.<br>
If that’s the case, add <code>self.bn1 = nn.BatchNorm2d(3)</code> to the <code>__init__</code> method of <code>UNet</code> and call it in the <code>forward</code> method before executing <code>self.encoder1</code>.</p>
Will the gradients be collected correctly if I modify them manually with `tensor.register_hook` in DistributedDataParallel training?

username: feiyuhuahuo
cooked: <p>I want to modify gradients manually with function <code>tensor.register_hook()</code>.<br>
I’ve tested that if training on one GPU, gradients will be collected correctly.<br>
But I’m not sure if use DDP training, will the gradients also be collected correctly.</p>
<pre><code class="lang-auto">def modify_grad(self, g):
    return g * 3

feature.register_hook(self.modify_grad)
</code></pre>
<p>Say, if I changed the gradient on each GPU by multiplying 3, so 3 times of the original reduced gradient will be broadcast to each GPU?</p>

username: ptrblck
cooked: <p><a href="https://discuss.pytorch.org/t/distributeddataparallel-modify-gradient-before-averaging/59291">This topic</a> seems to be related to your question and also has a code example.</p>

username: feiyuhuahuo
cooked: <p>I did some test. <code>Tensor.register_hook()</code> does work in DDP. I first wrap my model with DDP, then I modify the grad manually with a custom function in the <code>forward</code> code where I construct my model:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/d/b/dbcd77f7cab66827035e4a3420e7686f145faf88.png" alt="image" data-base62-sha1="vmsMollJwpa5GfrlIqRCE6pwJNC" width="258" height="55"></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/5/b514c58e556b1af5fbf2b0607f2f22af0dbe9d9a.png" data-download-href="https://discuss.pytorch.org/uploads/default/b514c58e556b1af5fbf2b0607f2f22af0dbe9d9a" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/5/b514c58e556b1af5fbf2b0607f2f22af0dbe9d9a.png" alt="image" data-base62-sha1="pPUVmVDLxXrA5lzXKGbYsLaF6cy" width="690" height="21" data-dominant-color="42464A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">761×24 6.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Here’s the gradients after calling <code>loss.backward()</code>:<br>
If don’t modify the grad:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/d/6/d6bc543c8b9cdcfde91a06d2642731af650c9e1b.png" alt="image" data-base62-sha1="uDDFEpjtU5BQ77N02d0oSR8SN2H" width="595" height="35"><br>
And if modify the grad:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/e/4/e4f1713aab2c843ceee8fb466dda5eef0dcd497c.png" alt="image" data-base62-sha1="wFk9mtktssxcr9o9lWbpZ2iPWPO" width="601" height="37"><br>
You can see the gradients are just 3 times after being modified, which means <code>.register_hook()</code> works well in DDP.</p>
GPT2 TraceWarnings

username: MK2405
cooked: <p>Hi,</p>
<p>I have a GPT2 model fine-tuned for a question generation task. I am now trying to convert it into torchscript but I am facing the following issues. My results are not as good as the original model. It would be great if someone could help me figure out the issue. Following are the versions of the packages installed:<br>
torch: 1.7<br>
transformers: 4.1.1(tried with 3.0.1 and 2.0 as well)</p>
<p>/usr/local/lib/python3.6/dist-packages/transformers/models/gpt2/modeling_gpt2.py:168: TracerWarning: Converting a tensor to a Python float might cause the trace to be incorrect. We can’t record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!<br>
w = w / (float(v.size(-1)) ** 0.5)<br>
/usr/local/lib/python3.6/dist-packages/transformers/models/gpt2/modeling_gpt2.py:173: TracerWarning: Converting a tensor to a Python index might cause the trace to be incorrect. We can’t record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!<br>
mask = self.bias[:, :, ns - nd : ns, :ns]<br>
/usr/local/lib/python3.6/dist-packages/torch/jit/_trace.py:966: TracerWarning: Output nr 1. of the traced function does not match the corresponding output of the Python function. Detailed error:<br>
With rtol=1e-05 and atol=1e-05, found 254695 element(s) (out of 27897075) whose difference(s) exceeded the margin of error (including 0 nan comparisons). The greatest difference was 5.1975250244140625e-05 (2.250950336456299 vs. 2.251002311706543), which occurred at index (0, 69, 37541).<br>
_module_class,</p>
<p>Thanks</p>
ValueError: Classification metrics can't handle a mix of unknown and binary targets : 0 and 1 Predictions

username: Khawar_Islam
cooked: <p>I am constructing confusion matrix based on prediction and ground truth. Previously, it worked fine on Python 3.5 but now Python 3.5 is unavailable. So I have to install python 3.7 for better working.</p>
<p><strong>Code</strong></p>
<pre><code class="lang-auto">predictions = gbm.predict(x_test)
predictions_classes = []
for i in predictions:
    prediction_class = np.argmax(i)
    predictions_classes.append(prediction_class)

cm = confusion_matrix(y_test, predictions_classes, normalize=False)
print(cm)

tn, fp, fn, tp = cm.ravel()

recall = tp / (tp + fn)
precision = tp / (tp + fp)
accuracy = (tp + tn) / (tn + fp + fn + tp)
f1 = 2 * (precision * recall) / (precision + recall)

print("Precision: ", 100 * precision, "%")
print("Recall: ", 100 * recall, "%")
print("F1 score ", 100 * f1, "%")
print("Accuracy: ", 100 * accuracy, "%")
</code></pre>
<pre><code class="lang-auto">**Traceback (most recent call last):**
  File "/home/khawar/deepface/tests/Ensemble-Face-Recognition.py", line 846, in &lt;module&gt;
    cm = confusion_matrix(y_test, predictions_classes, normalize=False)
  File "/home/khawar/anaconda3/envs/deepface/lib/python3.7/site-packages/sklearn/utils/validation.py", line 72, in inner_f
    return f(**kwargs)
  File "/home/khawar/anaconda3/envs/deepface/lib/python3.7/site-packages/sklearn/metrics/_classification.py", line 276, in confusion_matrix
    y_type, y_true, y_pred = _check_targets(y_true, y_pred)
  File "/home/khawar/anaconda3/envs/deepface/lib/python3.7/site-packages/sklearn/metrics/_classification.py", line 91, in _check_targets
    "and {1} targets".format(type_true, type_pred))
ValueError: Classification metrics can't handle a mix of unknown and binary targets
</code></pre>

username: ptrblck
cooked: <p>I’m not sure if this question is PyTorch-specific or rather targets <code>sklearn</code> or Python.<br>
If your code is working fine in Python==3.7, I guess some underlying libraries, such as numpy or sklearn, were also updated?</p>
Performance of sampling from random distributions

username: Moustafa_Alzantot
cooked: <p>Hello,</p>
<p>I am running a training algorithm and in one step, I need to perform Sampling from a Gaussian distribution with a given standard deviation.</p>
<p>Each one epoch in my training takes around 5 seconds if I don’t perform the sampling step.<br>
However, If I do the sampling, it becomes too slow (<strong>1 epoch = 120 seconds)!</strong>!.</p>
<p>I am doing it using .<br>
Before starting the training, I create a normal distribution object.</p>
<pre><code class="lang-auto">import torch.distributions as tdist
normal_dist = tdist.Normal(0.0, sigma ) 
</code></pre>
<p>then within my training loop:</p>
<pre><code class="lang-auto">
noise = normal_dist.sample(w.size())
</code></pre>
<p>Any hints why is this happening and if there is a more efficient way?</p>
<p>EDIT:</p>
<ul>
<li>
<code>torch.randn</code> is also as slow.</li>
<li>I did comparison between tensorflow vs pytorch performance on random sampling, when the shape of the output noise small PyTorch tends to be faster, but if we are sampling big tensors, TensorFlow is way faster and Pytorch becomes too slow.</li>
<li>The size of the output in my epxeriment is <code>1024x128x128</code>.</li>
</ul>

username: ptrblck
cooked: <p>Could you try to time the noise creation?<br>
Are you using the CPU or GPU?<br>
If you’re using GPU, you should synchronize before starting and stopping the timer:</p>
<pre><code class="lang-python">torch.cuda.synchronize()
t0 = time.time()
noise = normal_dist.sample(w.size())
torch.cuda.synchronize()
t1 = time.time()
</code></pre>

username: Moustafa_Alzantot
cooked: <p>I did a small benchmark script to time the noise creation for PyTorch vs TensorFlow.</p>
<p><strong>Noise size = (1024, 128, 128) -</strong><br>
PyTorch Time = 0.09 Sec<br>
TF Time= 0.03 Sec</p>
<p><strong>Noise size = (1024, 256, 256) -</strong><br>
PyTorch Time =  0.398 Sec<br>
TF Time= 0.118 Sec</p>
<p>Since noise sampling <code>torch.randn</code> or <code>normal_dist.sample</code> happens on CPU, then we move the results to GPU. So I measured the time of generation of random noise without actually moving them to GPU.</p>

username: nionjo
cooked: <p>Similar issue for</p>
<blockquote>
<p>distrib = MultivariateNormal(loc=torch.zeros(input_dimensionality), covariance_matrix=torch.diag(torch.ones(input_dimensionality[0])))<br>
distrib.sample((N,)).cuda()</p>
</blockquote>
<p>for N=5 and input_dimensionality=16K</p>

username: nionjo
cooked: <p>are there any updates on this?</p>

username: Saurabh_Kataria
cooked: <p>Is there any update to this? I am also finding pytorch sampling slow. I am using <code>torch.distributions.Normal(0, 1).sample</code>.<br>
Edit: I found <code>torch.empty().normal_()</code> faster than <code>torch.distributions.Normal(0, 1).sample</code>, so just managing with this now.</p>
Sending messages to DataLoader workers

username: ayralth
cooked: <p>Hello,</p>
<p>I have a data.Dataset and a corresponding DataLoader with multiple workers. I would like the workers __getitem__ to have a specific behavior depending on some variable in my main training loop (e.g. training phases). How can I send a message from the main process to the workers ?</p>
<p>More details :<br>
I work with video data, and train with short video-clips extracted from long sequences of frames. The dataloader only returns the small clip. This sampling strategy needs to evolve throughout training (e.g. retrieve the clip at a specific position in the video) so I need the workers to receive a message from the main process (a general message for all workers).</p>
<p>Currently I am using files written in the main loop and read by the workers to send the message, but I dont think this is a good practice. In the opposite way, I send information from workers to main process within the __getitem__ return data.</p>
<p>Thanks for your help</p>
After training, how can all the weights that are dropped out be?

username: spring
cooked: <p>If dropout is applied during training, I know that dropout rate should be set to 0 through model.eval() when evaluating. Then, are the droppedout weights given random values ​​when evaluated? If so, I think the performance will be worse. Can’t we keep dropout even when we evaluate?</p>

username: albanD
cooked: <p>Hi,</p>
<p>Since dropout is random every time it is used, there are not a fixed set of weights that are droppedout. They are all used sometimes during training and so have been trained properly.</p>

username: spring
cooked: <p>Hi,</p>
<p>Thank you so much!</p>
"Masked" Batchnorm for variable length input?

username: Michael_Moran
cooked: <p>For my variable length input I have attempted to use a mask to avoid padding messing with the batch statistics.</p>
<p>x[~mask] = self.norm(x[~mask])</p>
<p>Here in this toy example x is of dimensionality (batch,embedding) and the mask is of dimensonality (batch) and is true where real data is and false where padding is. I have implemented this strategy and it seems extremely unstable. Is there something wrong with the way I’m going about this? Is there a better way to do a “partial” batch norm like this where padding is ignored?</p>

username: ptrblck
cooked: <p>The masked tensor should be a flattened one so I’m unsure how your normalization layer would work with this type of input:</p>
<pre><code class="lang-python">x = torch.randn(10, 10)
mask = torch.randint(0, 2, (10, 10)).bool()
print(x[mask].shape)
&gt; torch.Size([49])
</code></pre>

username: Michael_Moran
cooked: <p>That would explain why it wasnt working!</p>
<p>Is there an easy alternative? I’m trying to do standard normalization I just want the padding excluded from batch statistics.</p>

username: ptrblck
cooked: <p>I’m not sure if there is a better approach than to manually implement the (batch)norm layer and applying your custom logic to calculate the stats.<br>
For a tensor of <code>[batch_size, channels, height, width]</code>, you could split the tensor in the <code>channels</code> dimension, mask the spatial dimensions separately, and calculate the stats from it.</p>
Positional Encoder in transformer

username: kit_m
cooked: <p>My question is the <code>PositinalEncoding</code> class from <a href="https://pytorch.org/tutorials/beginner/transformer_tutorial.html" rel="noopener nofollow ugc">Transformer tutorial</a>.<br>
Where <code>self.pe</code> in the <code>forward</code> method is defined?<br>
I do not see it is defined in the <code>__init__</code> method. I see that <code>pe</code> is handled in <code>__init__</code> a few times; however, they are in the local scope as far as I read the code.</p>
<pre><code class="lang-auto">class PositionalEncoding(nn.Module):

    def __init__(self, d_model, dropout=0.1, max_len=5000):
        super(PositionalEncoding, self).__init__()
        self.dropout = nn.Dropout(p=dropout)

        pe = torch.zeros(max_len, d_model)
        position = torch.arange(0, max_len, dtype=torch.float).unsqueeze(1)
        div_term = torch.exp(torch.arange(0, d_model, 2).float() * (-math.log(10000.0) / d_model))
        pe[:, 0::2] = torch.sin(position * div_term)
        pe[:, 1::2] = torch.cos(position * div_term)
        pe = pe.unsqueeze(0).transpose(0, 1)
        self.register_buffer('pe', pe)

    def forward(self, x):
        x = x + self.pe[:x.size(0), :]
        return self.dropout(x)
</code></pre>

username: Abhilash_Srivastava
cooked: <aside class="quote no-group" data-username="kit_m" data-post="1" data-topic="108099">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kit_m/48/29740_2.png" class="avatar"> kit_m:</div>
<blockquote>
<p><code>self.register_buffer('pe', pe)</code></p>
</blockquote>
</aside>
<p><code>pe</code> is registered as a buffer. Buffers can be accessed as attributes using given names.</p>
PyTorch implementation of ELMo

username: yeedTorch
cooked: <p>Hey guys!</p>
<p>I am looking for an implementation of ELMo in PyTorch. We would like to train the model from scratch and<br>
the only implementation that allows this that I could find was the AllenAI TensorFlow one.</p>
<p>I was wondering if anybody knows of a PyTorch implementation I simply missed that would allow this?</p>
<p>Best</p>
<p>yeed</p>

username: lesscomfortable
cooked: <p>Didn’t find one either but <a href="https://github.com/flairNLP/flair" rel="noopener nofollow ugc">FLAIR </a> is a great option to generate embeddings with an LSTM based architecture.</p>
Node output shape from trace graph

username: waleeka
cooked: <p>I’m trying to traverse a graph that I got from tracing the model. I got most of the data I need, except for one: the shape of the output of each layer.</p>
<pre><code class="lang-python"># VGG16
model = torchvision.models.vgg16()
# Get trace graph
trace, out = torch.jit.get_trace_graph(model, torch.zeros([1, 3, 224, 224]))
torch.onnx._optimize_trace(trace, torch.onnx.OperatorExportTypes.ONNX)
graph = trace.graph()
# Print first node in the graph
node = list(graph.nodes())[0]
print(node)
</code></pre>
<p>Output:</p>
<pre><code class="lang-auto">%33 : Float(1, 64, 224, 224) = onnx::Conv[dilations=[1, 1], group=1, 
kernel_shape=[3, 3], pads=[1, 1, 1, 1], strides=[1, 1]](%0, %1, %2), 
scope: VGG/Sequential[features]/Conv2d[0]
</code></pre>
<p>See the <code>Float(1, 64, 224, 224)</code> in the output? That’s the value I want. Which method or property returns that?</p>

username: waleeka
cooked: <p>Bump!</p>
<p>Still searching for a solution. As a temporary hack, I used a regular expression to extract the shape from the string, but there must be a better way to do this. At least I hope there is.</p>

username: cjsoft
cooked: <p>Have you found any better way? I need to get output shape of a C node as well</p>

username: cjsoft
cooked: <p>Hi, I’ve found how to acquire output shape of a torch._C.Node.<br>
let <code>node</code> be a torch._C.Node. The output shape of this Node is <code>node.output().type().sizes()</code><br>
Hope this will help</p>

username: Antonio1
cooked: <p>I was trying to replicate the examples in <a href="https://github.com/waleedka/hiddenlayer/blob/master/demos/pytorch_graph.ipynb" rel="noopener nofollow ugc">here</a>, but my graphs were not showing any tensor shapes. I have then implemented the change <strong>cjsoft</strong> suggested by substituting the default</p>
<pre><code class="lang-auto">m = re.match(r".*Float\(([\d\s\,]+)\).*", str(next(torch_node.outputs())))
if m:
    shape = m.group(1)
    shape = shape.split(",")
    shape = tuple(map(int, shape))
else:
    shape = None
return shape
</code></pre>
<p>in <code>get_shape</code> within <code>pytorch_builder.py</code> with</p>
<pre><code class="lang-auto">shape = torch_node.output().type().sizes()
return shape
</code></pre>
<p>which solved the issue. However, when trying to plot my own network (a ResNet U-Net), it incurred this error:</p>
<p><code>RuntimeError: r INTERNAL ASSERT FAILED at "..\\aten\\src\\ATen/core/jit_type.h":171, please report a bug to PyTorch.</code></p>
<p>So I have wrapped everything into a <code>try/except</code> and it seems to work fine:</p>
<pre><code class="lang-auto">try:
    shape = torch_node.output().type().sizes()
except:
    shape = None
return shape
</code></pre>
<p>Hope it helps.</p>
Why do the generated samples have `require_grad = False`?

username: pytorchkiw
cooked: <p>This is my simple model:</p>
<pre><code class="lang-auto">class Generator(nn.Module):
    def __init__(self):
        super(Generator, self).__init__()
        self.latent_dim = 2
        self.model = nn.Linear(self.latent_dim, 2)

    def forward(self, z):
        return self.model(z)

    def generate_samples(self, N_samples):
        z = torch.tensor(np.random.normal(0, 1, (N_samples, self.latent_dim))).float()
        samples = self(z)
        return z
</code></pre>
<p>The following code returns <code>False</code>:</p>
<pre><code class="lang-auto">G = Generator()
G.generate_samples(20).requires_grad
</code></pre>
<p>What’s the problem here?</p>

username: mruberry
cooked: <p>Add requires_grad=True as a kwarg when creating the tensor. Tensors do not require gradient by default. See <a href="https://pytorch.org/docs/master/generated/torch.tensor.html#torch.tensor" class="inline-onebox" rel="noopener nofollow ugc">torch.tensor — PyTorch master documentation</a>.</p>

username: pytorchkiw
cooked: <p>I do not want gradients w.r.t. to the noise tensor <code>z</code> but w.r.t. to my linear layer <code>self.model</code>.</p>

username: Nikronic
cooked: <aside class="quote no-group" data-username="pytorchkiw" data-post="1" data-topic="108094">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/3be4f8/48.png" class="avatar"> pytorchkiw:</div>
<blockquote>
<p><code>return z</code></p>
</blockquote>
</aside>
<p>You are returning <code>z</code> which is your input and as <a class="mention" href="/u/mruberry">@mruberry</a> has mentioned, if you need grad for your inputs, then you should enable it. But you have mentioned that you only need grads w.r.t. model parameters (not input). In that case, you should return <code>samples</code> instead of <code>z</code> to see grads. I think you are returning the wrong tensor.</p>
<p>PS: You can use <code>torch.normal</code> instead of numpy.</p>
<p>Bests</p>
Basic Tensor Operation

username: Anakin
cooked: <p>Hello, here is a sequential process of LSTM implemented in Torch.<br>
As we know that, in each iteration we need to re-used previous hidden states and cell states. In torch, the equal operator synchronizes both tensors, say,</p>
<blockquote>
<p>h[{{}, t}] corresponds to self.output[{{}, t}],<br>
h is the current tensor working, self.output is actuyll a place holder.</p>
</blockquote>
<pre><code class="lang-auto">  local h, c = self.output, self.cell
  h:resize(N, T, H):zero()
  c:resize(N, T, H):zero()
  local prev_h, prev_c = h0, c0
  self.gates:resize(N, T, 4 * H):zero()
  for t = 1, T do
    local cur_x = x[{{}, t}]
    local next_h = h[{{}, t}]
    local next_c = c[{{}, t}]
    local cur_gates = self.gates[{{}, t}]
    cur_gates:addmm(bias_expand, cur_x, Wx)
    cur_gates:addmm(prev_h, Wh)
    cur_gates[{{}, {1, 3 * H}}]:sigmoid()
    cur_gates[{{}, {3 * H + 1, 4 * H}}]:tanh()
    local i = cur_gates[{{}, {1, H}}]
    local f = cur_gates[{{}, {H + 1, 2 * H}}]
    local o = cur_gates[{{}, {2 * H + 1, 3 * H}}]
    local g = cur_gates[{{}, {3 * H + 1, 4 * H}}]
    next_h:cmul(i, g)
    print(self.output[{{}, t}])
    next_c:cmul(f, prev_c):add(next_h)
    next_h:tanh(next_c):cmul(o)
    prev_h, prev_c = next_h, next_c
  end

  return self.output

</code></pre>
<p>In PyTorch, this does not work automatically after the initialization step. I know that I can manually explicitly re-assign the tensor by following codes. But it’s not simple as it does in Torch. I’m sure that Pytorh has definitely some similar mechanisms. Please give me some keywords that might help me out.</p>
<pre><code class="lang-auto">    # Initialize hidden stats
    h, c = self.output, self.cell
    ...

    h.resize_(N, T, H).zero_()
    c.resize_(N, T, H).zero_()
    for t in range(T):
        cur_x = x[:, t]
        next_h = h[:, t]
        next_c = c[:, t]
        # state update 
        # i, f, c, g
        .....

        # store the hidden states (is this replaceable with some inpace operations??)
        self.output[:, t] = next_h
        self.cell[:, t] = next_c
</code></pre>
<p>Thanks for any input.</p>
Segfault when using dataloader, custom dataset and from_blob

username: tsoj
cooked: <p>I have this (to the best of my knowledge) minimal example that segfaults when using a data_loader if I use from_blob to create a tensor for my custom dataset but it works fine if I just use something like torch::empty:</p>
<pre><code class="lang-auto">#include &lt;iostream&gt;
#include &lt;torch/torch.h&gt;
#include &lt;vector&gt;

#define USE_FROM_BLOB

struct MyDataset : public torch::data::datasets::Dataset&lt;MyDataset&gt;
{
    MyDataset()
    {

        std::vector&lt;std::array&lt;float, 60&gt;&gt; v1(1000);
        
#ifdef USE_FROM_BLOB
        m_data =torch::from_blob(v1.data(), {(int)v1.size(), (int)v1[0].size()});
#else
        m_data = torch::empty({(int)v1.size(), (int)v1[0].size()});
#endif
        std::cout &lt;&lt; "Here it works: " &lt;&lt; m_data[123][0].item&lt;float&gt;() &lt;&lt; std::endl;
    }

    torch::data::Example&lt;&gt; get([[maybe_unused]] size_t index) override
    { 

#ifdef USE_FROM_BLOB
        std::cout &lt;&lt; "Here it doesn't: " &lt;&lt; std::flush;
#else
        std::cout &lt;&lt; "Here it works as well: " &lt;&lt; std::flush;
#endif
        std::cout &lt;&lt; m_data[123][0].item&lt;float&gt;() &lt;&lt; ", see?" &lt;&lt; std::endl;

        return {m_data[123], m_data[123]};        
    }

    c10::optional&lt;size_t&gt; size() const override { return m_data.sizes()[0]; }

  private:
    torch::Tensor m_data;
};

int main()
{
    auto data_loader = torch::data::make_data_loader(MyDataset());

    for (auto&amp; batch : *data_loader)
    {
        std::cout &lt;&lt; batch.size() &lt;&lt; std::endl;
        break;
    }

    return 0;
}
</code></pre>
<p>Output when USE_FROM_BLOB is defined:</p>
<pre><code class="lang-auto">Here it works: 0
Here it doesn't: Segmentation fault (core dumped)
</code></pre>
<p>If it isn’t defined:</p>
<pre><code class="lang-auto">Here it works: 0
Here it works as well: 0, see?
1
</code></pre>
<p>Why is that and/or how can I fix this? I could just use torch::empty and then fill the tensor manually each element at a time, but it is really slow, compared to the from_blob.</p>

username: albanD
cooked: <p>Hi,</p>
<p>When you call <code>from_blob</code>, you tell the Tensor to use this content in memory but the Tensor does not “own” the memory!<br>
So it is your responsability to make sure that this memory is kept allocated as long as the Tensor lives.</p>
<p>In your case, when you exit the scope, the std::vector is destructed and the memory is freed, hence later access to the Tensor with segfault.<br>
Note that resizing the std::vector could also lead to similar issues if the underlying storage is re-allocated.</p>

username: tsoj
cooked: <p>Great, that works and makes sense!<br>
Is there also a similar function that copies the data into the tensor, so that I don’t have to keep the vector around?</p>

username: albanD
cooked: <p>I think the most efficient way is to do a <code>.clone()</code> of the temporary Tensor after you created it. This will return a new Tensor with its own memory and it will work just fine after the original vector goes out of scope.</p>
How To Flase This Block MkldnnConvolutionBackward

username: Muhammad_laghari
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/e/ded563ca66d8cc03885a4e73df87492924b64651.png" data-download-href="https://discuss.pytorch.org/uploads/default/ded563ca66d8cc03885a4e73df87492924b64651" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/e/ded563ca66d8cc03885a4e73df87492924b64651_2_672x500.png" alt="image" data-base62-sha1="vNhbpgGrdr73k858rufrfuaMWs1" width="672" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/e/ded563ca66d8cc03885a4e73df87492924b64651_2_672x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/e/ded563ca66d8cc03885a4e73df87492924b64651_2_1008x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/d/e/ded563ca66d8cc03885a4e73df87492924b64651.png 2x" data-dominant-color="EEEDEE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1129×839 56.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: albanD
cooked: <p>Hi,</p>
<p>I am not sure what your question is here? These Nodes correspond to the specific convolution that was used during the forward of the Conv2D.</p>
nn.Linear does not have `weight` any more after quantized

username: coincheung
cooked: <p>Hi,</p>
<p>I tried this code:</p>
<pre><code class="lang-auto">net = nn.Sequential(torch.nn.Linear(32, 32),)
net.qconfig = torch.quantization.get_default_qconfig('fbgemm')
net = torch.quantization.prepare(net)
inten = torch.randn(224, 32)
net(inten)

qnet = torch.quantization.convert(net)
print(qnet)
print(qnet.state_dict().keys())
print(qnet.state_dict()['weight'])

</code></pre>
<p>Only to find that there is not <code>weight</code> attribute in the <code>qnet</code>. How could I make the attribute same as original <code>nn.Linear</code> ?</p>

username: Vasiliy_Kuznetsov
cooked: <p>The weight for quantized Linear is packed.  You can access it by doing <code>linear_layer_instance._weight_bias()</code> (see <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/quantized/modules/linear.py#L31" class="inline-onebox" rel="noopener nofollow ugc">pytorch/linear.py at master · pytorch/pytorch · GitHub</a>).</p>
How to go about training YOLOv5 as an autoencoder?

username: bohare
cooked: <p>I am trying to train <a href="https://github.com/ultralytics/yolov5/blob/master/models/yolo.py" rel="noopener nofollow ugc">YOLOv5</a> on a custom dataset. Since the custom dataset is not too big, I would like to fine-tune an autoencoder based on YOLOv5 on an unlabeled dataset first and then fine-tune it further on the custom dataset so that it generalizes better.</p>
<p>My question is how can I approach this problem. Where can I make the changes to use a YOLOv5 model instance as an autoencoder?</p>
<p>Any guidance would be much appreciated. Thanks.</p>
Torchaudio: Wrong number of timesteps returned from Spectrogram?

username: divinho
cooked: <pre><code class="lang-auto">from torchaudio.transforms import Spectrogram
spec = Spectrogram(hop_length=160)
a = to.randn(16000)
f = spec(a)
f.shape  # (201, 101,)
</code></pre>
<p>Why are 101 frames? The padding argument is 0 by default so there should be 98. This is what fbank does</p>
<pre><code class="lang-auto">from torchaudio.compliance.kaldi import fbank
m = fbank(a.unsqueeze(0))
m.shape  # (98, 23,)
</code></pre>

username: krishna511
cooked: <p><a class="mention" href="/u/divinho">@divinho</a> I am not sure about this , but I think this extra 1 frame is to handle the zero hop case for the first frame.  atleast in case of librosa feature extraction of mel spectrogram they use center frames for hops.</p>

username: divinho
cooked: <p>Thank you for the quick reply, I don’t understand what you mean by zero hop case though?</p>

username: krishna511
cooked: <p>It means for the first frame the hop-length  is zero.  I wish someone experienced will comment soon <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: divinho
cooked: <p>I see, yeah it seems the culprit is <code>torch.stft</code>: <a href="https://github.com/pytorch/audio/blob/6b07bcf80fafd77cb8bee32c316ce8b55323b868/torchaudio/functional/functional.py#L33" class="inline-onebox" rel="noopener nofollow ugc">audio/functional.py at 6b07bcf80fafd77cb8bee32c316ce8b55323b868 · pytorch/audio · GitHub</a></p>
<p>Which has the <code>center</code> argumented hardcoded to True. That’s a bit annoying.</p>
Custom softmax yielding infs

username: Sam_Lerman
cooked: <p>I’m trying to use a custom <code>softmax</code> function that can yield 0 probabilities:</p>
<pre><code class="lang-auto">logits = torch.exp(values / temp) - 1
probas = logits / torch.sum(logits)
probas[torch.isnan(probas)] = 0
</code></pre>
<p>But the <code>exp</code> function here yields infinities for values greater than 15 or so.</p>
How to Plot accuracy and loss graph after training completion?

username: aatiibutt
cooked: <p>Hy!<br>
I am training a CNN through transfer learning, implemented the basic provided tutorial by pytorch (<a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Transfer Learning for Computer Vision Tutorial — PyTorch Tutorials 1.7.1 documentation</a>). I wanna plot accuracy and loss graph, but not getting getting any idea about it. I shall be grateful, if anybody help me out in this regard.</p>

username: caonv
cooked: <p>TensorboardX will do the work for you.</p>

username: 1chimaruGin
cooked: <p>Use Tensorboard or save each epoch’s loss and accuracy to a list and plot it with matplotlib.</p>

username: aatiibutt
cooked: <p>I used list method, it worked. Many Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
More stable softmax with temperature

username: haorannlp
cooked: <p>I wrote a seq2seq model and tried to implement minimum risk training (Eq. (13) in the paper: Minimum Risk Training for Neural Machine Translation)<br>
I added<br>
<code>torch.autograd.set_detect_anomaly(True)</code> at the beginning of the model.<br>
It outputed an error</p>
<p><code>RuntimeError: Function 'ExpBackward' returned nan values in its 0th output.</code></p>
<p>According to the tracceback, it has sth to do with the 2nd line of code below:<br>
<code> seq_nll = seq_nll-torch.max(seq_nll, dim=-1)[0].unsqueeze(1)</code><br>
<code>seq_probs = torch.pow(torch.exp(seq_nll), 0.005)  </code><br>
<code>normalizer = torch.sum(seq_probs, dim=-1).view(-1, 1)</code><br>
seq_nll is a tensor of shape (64,3) with very negative numbers like <code>[ -94.5122,  -50.0515,  -76.2685]</code>.<br>
These numbers are log likelihood of different sequences.<br>
The <code>exp</code> operation is to obtain the probability of those sequences.<br>
The <code>power</code> operation is to scale the probs.<br>
The <code>normalizer</code> is the sum of those re-scaled probs.</p>
<p>I guess the problem here is related to those very negative numbers.<br>
Is there a stable way to implemente the above code?</p>

username: Abhilash_Srivastava
cooked: <p>If I understand it correctly:<br>
<code>torch.pow(torch.exp(seq_nll), 0.005)</code> would simply imply <code>torch.exp(seq_nll * 0.005)</code></p>
<p>So, now you can directly use <code>torch.nn.functional.softmax</code>, with <code>seq_nll * 0.005</code> as the input.</p>