akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Why spyder shows：No moudle named 'torch'

TIAN: <p>File “C:/Users/CT/Desktop/TIAN/VasNet-pytorch-code/src/train-new_dataloader.py”, line 2, in <br>
import torch</p>
<p>ModuleNotFoundError: No module named ‘torch’</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/6/26e8a37692238692f4e7ffa2d3a2080341b0926e.png" alt="R0QFE{CQR3}3ESR$T56DYYA" data-base62-sha1="5yczjDufq9K2pVbLfLIPKgHDYRM" width="586" height="388"> !</p>

ptrblck: <p>Most likely you are starting <code>spyder</code> in another environment than the one which is used in your terminal.<br>
Make sure the env is correctly activated before starting the IDE.</p>

TIAN: <p>Thank you for your reply！<br>
It seemed that I didn’t run the spyder from the ‘pytorch_gpu’ in the Anaconda Navigator.<br>
Now i am installing the spyder in pytorch_gpu.<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/c/ace0c802b2921a9d72c49666d0f74ea6310d4eeb.png" data-download-href="https://discuss.pytorch.org/uploads/default/ace0c802b2921a9d72c49666d0f74ea6310d4eeb" title="@2FV}MMO(}QAOE%2~FWTOQB"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/c/ace0c802b2921a9d72c49666d0f74ea6310d4eeb_2_690x376.png" alt="@2FV}MMO(}QAOE%2~FWTOQB" data-base62-sha1="oFlIqEOGhUq8uuMMPGBuRvsBsv1" width="690" height="376" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/c/ace0c802b2921a9d72c49666d0f74ea6310d4eeb_2_690x376.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/c/ace0c802b2921a9d72c49666d0f74ea6310d4eeb_2_1035x564.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/a/c/ace0c802b2921a9d72c49666d0f74ea6310d4eeb_2_1380x752.png 2x" data-dominant-color="F0F0F0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">@2FV}MMO(}QAOE%2~FWTOQB</span><span class="informations">1920×1048 112 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
How to perform cross correlation for two 2d inputs and obtain same results as an operation in numpy/scipy?

sreeni5493: <pre><code>filt = np.zeros((3, 3))
filt[1, shift+1] = -1
filt[1, filt.shape[1] - 1] = 1
</code></pre>
<p>The above code generates a 3x3 filter that does a simple forward gradient. In order to do a simple correlation, I can do the following:</p>
<pre><code>from scipy.signal import correlate2d
from scipy import misc
im = misc.imread('grascale.png') # a grayscale image
imout = correlate2d(im,filt,mode='same',boundary='symm')
</code></pre>
<p>The above code ensures the output is the same size of input ‘im’ and also symmetrically reuses the values to fill the boundary pixels during convolution. How can I do the same using conv2d in pytorch? Also, is it enough to set bias to false and weight.data to filt value?</p>

qiangqiang_wu: <p>Hi, do you have any idea about that yet ? <a class="mention" href="/u/sreeni5493">@sreeni5493</a></p>

sreeni5493: <p>Yes. This is solvable. Conv2d is the same as correlate2d. Checked for random inputs and the results deviate in the order of power -8 (which means basically the results are the same). It does a valid convolution. So your best choice is to reflect boundaries after convolution or before convolution using torch nn functional pad. Do not do any interpolation either before or after convolution to change the size of the image. This leads to poor results. Best bet is to do reflect padding either post or pre convolution. Also, yes set weight.data to the filter that you wish to use. bias should be set to false or zero.</p>

qiangqiang_wu: <p><img src="//discuss.pytorch.org/images/emoji/apple/grinning.png?v=5" title=":grinning:" class="emoji" alt=":grinning:">Thanks for your reply. Could you show some codes? I am new to pytorch.<img src="//discuss.pytorch.org/images/emoji/apple/grinning.png?v=5" title=":grinning:" class="emoji" alt=":grinning:"></p>

Frida: <p>Hi, someone know to elaborate on the boundary conditions? Say, what the idea of wrap? symmetry?</p>
Efficient way for deterministic data augmentation

NightRain: <p>What is the most efficient way to have deterministic data augmentation (i.e. transformations every epoch are random, however they can be reproduced reliably for every data point)?</p>
<p>Currently I am thinking of creating a list with a numpy RandomState object for every datapoint. Even if the DataLoader uses multiple processes, every object is called once per epoch, so every datapoint is subject to the exact same random transformations when e.g. restarting training from scratch (assuming the RandomState objects are reinitialized with the same seed). One RandomState is not enough as there would be multiple processes (num_workers &gt; 0) accessing it and the datapoints are shuffled every epoch.</p>
<p>Is there a more efficient way to do this, considering multiple processes applying a random transformation every epoch to the objects where their order changes due to the shuffling?</p>

ptrblck: <aside class="quote no-group" data-username="NightRain" data-post="1" data-topic="100898">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/n/df705f/48.png" class="avatar"> NightRain:</div>
<blockquote>
<p>Is there a more efficient way to do this, considering multiple processes applying a random transformation every epoch to the objects where their order changes due to the shuffling?</p>
</blockquote>
</aside>
<p>I think one good approach would be to use the <code>worker_id</code> and seed all 3rd party libraries with this id.<br>
You can get the <code>id</code> via <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.get_worker_info"><code>torch.utils.data.get_worker_info()</code></a>.<br>
PyTorch itself should yield deterministic data samples, if you’ve properly set the seed before.<br>
However, if you are using e.g. the Python <code>random</code> library or numpy in the data loading pipeline, you would have to seed them as well using the aforementioned approach.</p>
How to change SyncBatchNorm

Mr.Crab: <p>i want to try the model on windows which is not supported to the distrubution. And i change the<br>
<code>net = torch.nn.SyncBatchNorm.convert_sync_batchnorm(net)</code>  into<br>
<code>net = torch.nn.BatchNorm2d(net)</code><br>
but i got a error which is “TypeError: new(): data must be a sequence (got DeepV3PlusHANet)”.<br>
And what should i do to run the code on windows, and how to change SyncBatchNorm with BatchNorm2d?</p>

ptrblck: <p>If you don’t want to use <code>SyncBatchNorm</code> you could just remove the <code>convert_sync_batchnorm</code> call and keep the original <code>nn.BatchNorm*D</code> layers.<br>
From the <a href="https://pytorch.org/docs/stable/generated/torch.nn.SyncBatchNorm.html?highlight=syncbatchnorm#torch.nn.SyncBatchNorm">docs</a>:</p>
<blockquote>
<p>Helper function to convert all <code>BatchNorm*D</code>  layers in the model to <a href="https://pytorch.org/docs/stable/generated/torch.nn.SyncBatchNorm.html?highlight=syncbatchnorm#torch.nn.SyncBatchNorm"><code>torch.nn.SyncBatchNorm</code></a> layers.</p>
</blockquote>

Mr.Crab: <p>did i need to remove other code?<br>
can SyncBatchNorm can replace the BatchNorm2D?</p>

ptrblck: <p>If you see other usages of any <code>SyncBatchNorm</code> calls, I would remove them as well.<br>
Yes, <code>convert_sync_batchnorm</code> converts the <code>nn.BatchNorm*D</code> layers to their sync-equivalent.<br>
If you don’t want to use this, just keep the model as it is without calling any <code>SyncBatchNorm</code> functions and it will use the standard <code>nn.BatchNorm*D</code> layers.</p>

Mr.Crab: <p>it sitll don’t work, i got the same error after removing the convert_sync_batchnorm.</p>

ptrblck: <p>Could you post an executable code snippet, so that we could have a look at this error?</p>

Mr.Crab: <p>Sorry the code is complex and i couldn’t give you a code snippet. You can get the rar file from this <a href="https://drive.google.com/file/d/1biA7CJnPibJAKS7Liw60JPuKFqpV9il8/view?usp=sharing" rel="noopener nofollow ugc">https://drive.google.com/file/d/1biA7CJnPibJAKS7Liw60JPuKFqpV9il8/view?usp=sharing</a>,  just run the train.py and may be you can get this error.<br>
i use windows 10 and pytorch 1.4.0. Thank you so much.</p>

ptrblck: <p>Could you try to slim down the code and post it here, please?<br>
Alternatively, could you create a gist on GitHub and post the link here?</p>
Deploy c++ trained model on PyTorch

Willem: <p>I have trained a model using GPU c++ frontend to Pytorch. Now I want to deploy the model (for inference only) on a CPU-only linux dist (actually, on a RaspBerry Pi).</p>
<p>I have a trained torch::nn::Module (created using c++ front-end) that I saved using torch::save. It was trained on a cluster with GPU support. Now I want to take the trained model, and deploy it on a RaspBerry Pi.</p>
<p>I have read a lot of tutorials on deployment, but many seem to assume that training happened in Python. Now I have trouble seeing the forest for the trees. Where to start?</p>

Albert_Christianto: <p>Well, the basic idea is:<br>
-&gt; load your network<br>
-&gt; load your trained weight into your loaded network<br>
-&gt; load your input data<br>
-&gt; pre-process it<br>
-&gt; feed the input data to your network<br>
-&gt; post-process it<br>
-&gt; show the results</p>
<p>here is an example that i can find<br>
<a href="https://github.com/weixu000/libtorch-yolov3-deepsort/tree/master/detection/src" rel="noopener nofollow ugc">https://github.com/weixu000/libtorch-yolov3-deepsort/tree/master/detection/src</a>.<br>
this example is using torch::nn. Note that to load the trained weight into the network, he write his own load function. So, to load the trained weight into the network, you can use torch::load function.</p>
<p>And I have some suggestions for you:</p>
<ol>
<li>Since you are using RaspBerry Pi, why don’t you use python (instead of using C++) to build your apps?</li>
<li>If you really need to build your apps in C++, maybe you can train your network in python and use TorchScript features from PyTorch. This features is very helpful on deployment.<br>
here are some examples on how to use torchscript:<br>
-&gt; <a href="https://github.com/BIGBALLON/PyTorch-CPP/blob/master/prediction.cpp" rel="noopener nofollow ugc">https://github.com/BIGBALLON/PyTorch-CPP/blob/master/prediction.cpp</a><br>
-&gt; <a href="https://www.programmersought.com/article/5464871369/" rel="noopener nofollow ugc">https://www.programmersought.com/article/5464871369/</a>
</li>
</ol>
<p>if this answer still doesn’t help, you can start by trying some examples in the documentation (<a href="https://pytorch.org/docs/stable/cpp_index.html" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/cpp_index.html</a>). these examples really help me in understanding how the pytorch c ++ api works<br>
regards,<br>
Albert Christianto</p>
Call backs based on validation loss using ray tune

nivesh_gadipudi: <p>I would like to request for any examples of hyperparameter tuning using PyTorch and ray tune with respect to validation loss to avoid over fitting my model.</p>
<p>Any link or example would work for me, thanks in advance.</p>

richardliaw: <p>Hi <a class="mention" href="/u/nivesh_gadipudi">@nivesh_gadipudi</a>, here are a <a href="https://docs.ray.io/en/latest/tune/examples/index.html#pytorch-examples" rel="noopener nofollow ugc">couple examples for using Ray Tune with Pytorch</a>.</p>
<p>An easy way to prevent overfitting is simply to return “done=True” when validation losses begins to diverge.</p>
<pre><code class="lang-auto">tune.report(done=True, ...)
# or if using class API:

def step(self):
   ...
   return {"done": True, ...}
</code></pre>
<p>Hope that helps!</p>
Issues building C++ installation of pytorch, Cannot Find CUDA

JamesDickens: <p>I am trying to build a test .cpp in CLION as in the tutorial <a href="https://pytorch.org/cppdocs/installing.html#minimal-example" rel="noopener nofollow ugc">https://pytorch.org/cppdocs/installing.html#minimal-example</a></p>
<p>I am using</p>
<ul>
<li>Cuda version 11.1</li>
<li>the nightly c++ release of pytorch (the one released today)</li>
<li>CUDNN version 8.0.4 for Cuda 11.1</li>
<li>The cygwin 64 compiler</li>
</ul>
<p>After reloading the CMakeLists.txt file I get the message:<br>
Caffe2: CUDA cannot be found.  Depending on whether you are building Caffe2<br>
or a Caffe2 dependent library, the next warning / error will give you more<br>
info.</p>
<p>Where in cuda.cmake there is a call to the function find_package(CUDA).</p>
<p>I have set my environment variables to be:<br>
CUDA_PATH = C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1</p>
<p>as well as added<br>
Toolkit\CUDA\v11.1\libnvvp;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\bin; to my path variable</p>
<p>Any insights to how to solve this problem appreciated.</p>
What's the correct way of using AMP with multiple losses?

Zequn_Qin: <p>As the doc says, " If your network has multiple losses, you must call scaler.scale on each of them individually." And this it looks like:</p>
<pre><code class="lang-auto">scaler = torch.cuda.amp.GradScaler()
with autocast():
    loss0 = some_loss
    loss1 = another_loss

scaler.scale(loss0).backward(retain_graph=True)
scaler.scale(loss1).backward()
</code></pre>
<p>It is relatively inconvenient especially when we want to have many weighted losses. What if we just sum all the losses to one and backward once? It looks like this:</p>
<pre><code class="lang-auto">scaler = torch.cuda.amp.GradScaler()
with autocast():
    loss0 = some_loss
    loss1 = another_loss
    loss_all = w0 * loss0 + w1 * loss1

scaler.scale(loss_all).backward()
</code></pre>
<p>Is this kind of implementation correct?</p>

mrartemev: <p>I have the same problem here, this use case is not very well explained in the docs. Can anyone provide an explanation?</p>

mcarilli: <aside class="quote no-group" data-username="Zequn_Qin" data-post="1" data-topic="93328">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zequn_qin/48/11801_2.png" class="avatar"> Zequn_Qin:</div>
<blockquote>
<pre><code class="lang-auto">scaler = torch.cuda.amp.GradScaler()
with autocast():
    loss0 = some_loss
    loss1 = another_loss
    loss_all = w0 * loss0 + w1 * loss1

scaler.scale(loss_all).backward()
</code></pre>
</blockquote>
</aside>
<p>yes that would work, and you could do the same without amp.</p>
<p>There’s no need for retain_graph=True on the first backward unless the two losses share backward graph.  scaler.scale(loss0).backward(retain_graph=True) in the <a href="https://pytorch.org/docs/stable/notes/amp_examples.html#working-with-multiple-models-losses-and-optimizers" rel="noopener nofollow ugc">example</a> only uses retain_graph=True because the two losses share some backward graph, it has nothing to do with amp.</p>
No reply on multi GPU question

pytorcher: <p>Hey, I never got a reply to my other question.  I’m now at a point where things are working well on 1 GPU and I want to run on a bigger model.  Can someone please help me get to the bottom of this  <a href="https://discuss.pytorch.org/t/multi-gpu-hook-not-correctly-filling-buffer/96046" class="inline-onebox">Multi GPU Hook not correctly filling buffer</a></p>
ETL integrated into PyTorch

mjnovice: <p>Hi all, I wanted to check the validity of an idea. I was thinking of having integrated support of ETL in PyTorch. Is this feasible ? Wanted to know if someone has done anything around this and if not, any particular reasons for not doing so ?</p>

ptrblck: <p>I’m not sure if ETL stand for “Extract, Transform, Load”, but if so how would PyTorch come into play combining data from multiple systems into a single database?<br>
Could you explain your idea a bit and correct me, if ETL stand for something else?</p>

mjnovice: <p>What I meant was that would it make sense to support different database bindings for Extract-Transform-Load, instead of having multiple frameworks like Spark, Hadoop etc. ?</p>

ptrblck: <p>I’m still unsure, if this is a PyTorch-specific question.<br>
Are you referring to some Spark/Hadoop-to-PyTorch bindings?</p>
Float16 dynamic quantization has no model size benefit

huoge: <p>Hello everyone. I recently use dynamic quantiztion to quant the model, when use torch.quantization.quantize_dynamic(model, dtype=torch.qint8) to quant the model, model from 39M to 30M, while use torch.quantization.quantize_dynamic(model, dtype=torch.float16) the model size has no changes.  Does anybody know why? or Do I do the wrong way to quantize model to float16?</p>
<p>I’d appreciate if anybody can help me! Thanks in advance!</p>

supriyar: <p>Hi <a class="mention" href="/u/huoge">@huoge</a> - for int8 dynamic quantization we quantize the weights to 8-bits so you see the expected size reduction.<br>
For fp16 quantization, the weight values are cast to fp16 (taking saturation into account), but the dtype is still set to float32. This has to do with the type expected by the FBGEMM backend when performing the gemm operation.</p>
Using Keypoints for time-series problems

ahmed: <p>What I would like to do if predict future actions of pedestrians over time. I am using the Keypoint R-CNN model to generate the keypoints, which I will be using to create a skeleton. Based on the evolution of the skeleton over time, I would like to predict what they may do after a number of frames.</p>
<p>Where I am stuck is how to use the images and keypoints together to do this. How would I feed a model the images and keypoints in a way that it will learn the evolution of skeleton over time? I believe I would be using 3D conv nets, but I am unsure of what the best to feed the data.</p>
<p>Do I extract the features of a number of concatenated images and fuse those features with concatenated keypoints for those images and then feed that into a classifier for learning the actions? Can I used <code>torchvision.ops.MultiScaleRoIAlign</code> to downsample the images with respect to predicted keypoints (I hope that is how <code>pooling</code> using <code>MultiScaleRoIAlign </code> works?)</p>
<p>Thanks in advance.</p>
SGE and distributed data parallel training

Behnam_Asefisaray: <p>Hi guys,</p>
<p>Is there any tutorial which shows how can we use distributed model training with SGE (Sun Grid Engine). In general I’m wondering how multiple nodes can communicate with each other in multiple node setup?</p>
<p>Cheers,</p>

pritamdamania87: <p>I’m not familiar with Sun Grid Engine, but if multiple nodes in the system can talk to each other over TCP, you can follow this tutorial: <a href="https://pytorch.org/tutorials/intermediate/dist_tuto.html" rel="noopener nofollow ugc">https://pytorch.org/tutorials/intermediate/dist_tuto.html</a>. You probably want to use the TCP initialization method as described here: <a href="https://pytorch.org/tutorials/intermediate/dist_tuto.html#initialization-methods" rel="noopener nofollow ugc">https://pytorch.org/tutorials/intermediate/dist_tuto.html#initialization-methods</a>.</p>
Audio data augmentation

stases: <p>Hello! I am quite new to PyTorch and training DNN models in general. I’m working on audio separation and I would like to augment my dataset by cropping random overlapping segments of audio, adding noise, etc. What bothers me is how in general data augmentation works, meaning will I augment my data, save it to HDD and then load it, or is it done “per batch”, stored temporarily? I’m not sure if I have explained this properly. If the latter is the answer to my question, then is there anything similar (for audio) to augmentation for images in PyTorch (random crops, etc.)? Thanks in advance!</p>

ptrblck: <aside class="quote no-group" data-username="stases" data-post="1" data-topic="100776">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/stases/48/30321_2.png" class="avatar"> stases:</div>
<blockquote>
<p>What bothers me is how in general data augmentation works, meaning will I augment my data, save it to HDD and then load it, or is it done “per batch”, stored temporarily?</p>
</blockquote>
</aside>
<p>Data augmentation is applied on the fly for each batch during training.<br>
If you are using a <code>Dataset</code> and pass it to a <code>DataLoader</code> with multiple workers, the data loading and processing would be executed in the background while the model is training.</p>
<p>I would have a look at <a href="https://pytorch.org/audio/transforms.html"><code>torchaudio</code></a> which ships with some transformation.</p>

stases: <p>Hey Peter! Thank you for your reply. I will try using torchaudio for my case. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Pytorch 1.6 torchvision.transforms.functional.rotate doens't accept tensor, only PIL

enterthevoidf22: <p>on github it seems the latest commit supports image of type tensor for rotation. also on documentation.<br>
still when i run the command i get an error saying image must be PIL object.</p>
<p>i can bypass it by simply transforming to PIL but it causes trouble debugging another error occuring further down the chain.</p>
<p>so do i need to switch to nightly in order to get the rotate function to work with tensors?</p>

Nikronic: <p>Hi,</p>
<p>Transform on tensors has been added to PyTorch 1.7, released few days ago, you need to update <code>torchvision</code> package to 0.8. Simply, update to latest stable build.</p>
<p>Bests</p>
How to make predictions with a model that uses BatchNorm1d?

ixombi: <p>I’m writing a reinforcement learning program and I’m using nn.BatchNorm1d for the first time. The code trains fine but when it goes into evaluating the policy I get the “Expected more than 1 value per channel when training” error because nn.BatchNorm1d expects a batch of observations but I’m sending a single observation. I know why I’m getting the error, what I don’t know is how to set up the call to the policy to avoid the error, that is, to use the running estimates to normalize the single observation. Any suggestions?</p>

ptrblck: <p>You would have to call <code>model.eval()</code> to use the running stats in batchnorm layers and disable dropout layers. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

ixombi: <p>Oh gee thanks. Did some search on the .eval() method. So it turns out that to evaluate we should</p>
<pre><code class="lang-auto"># Evaluation mode
model.eval()

with torch.no_grad():
    ...
    out_data = model(data)
    ...
</code></pre>
<p>and then turn the training method back on</p>
<pre><code class="lang-auto"># Training mode
...
model.train()
...
</code></pre>
<p>Again, first  time me doing this.</p>
Map function in Pytorch

monaj07: <p>Hi all,</p>
<p>Is there any map function in Pytorch? (something like <code>map</code> in python).</p>
<p>I need to map a 1xDxhxw tensor variable to a 1x(9D)xhxw tensor, to augment embedding of each pixel with its 8 neighbour embeddings. Is there any functionality in Pytorch that lets me do that efficiently?<br>
I tried using map in Python this way:</p>
<pre><code class="lang-auto">n, d, h, w = embedding.size()
padder = nn.ReflectionPad2d(padding=1)
embedding = padder(embedding)
embedding = map(lambda i, j, M: M[:, :, i-1:i+2, j-1:j+2], range(1, h), range(1, w), embedding)
</code></pre>
<p>But it does not work for <code>w &gt; 2</code> and <code>h &gt; 2</code>.</p>

smth: <p>we dont hav a map function in pytorch. However, you can use some clever tricks to do what you want more efficiently. <code>torch.unfold</code> might help.</p>

monaj07: <p>Many Thanks for your response.<br>
I will think on what can be done using <code>torch.unfold</code>.</p>

kennethtxytqw: <p>Thank you! <code>torch.unfold</code> really made things easier. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=6" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Juans: <p>Hi! what happens if we need to apply the same function to all elements of the unfolded tensor? Map would still be very useful…<br>
For example, when x is a tensor with an MNIST sample and f is an arbitrary function with input a tensor of size (M,) --&gt; (1,):</p>
<pre><code class="lang-auto">x=torch.unfold(x,kernel_size=2,stride=2) #would give for MNIST size [1,4,196]
x=torch.cat([f(x[..., indx]) for indx in range(x.shape[-1])], dim=1) # is very slow!
</code></pre>
<p>Is there any way to do this efficiently now?</p>
Explanation for shuffle in Dataloader

nima_rafiee: <p>Hi<br>
for the explanation of <em>shuffle</em> as an argument to Dataloader it’s mentioned that:<br>
<em>’ set to <code>True</code>  to have the data reshuffled at every <strong>epoch</strong>’</em><br>
shouldn’t it be at every <strong>iteration</strong> or I’m missing something?</p>

ptrblck: <p>The indices will be shuffled for the complete dataset once per epoch and then used to create batches.<br>
Each batch will thus contain random samples and I think the documentation explains it correctly.</p>
Mask RCNN slow evaluation

param_eter: <p>I’m training a Mask RCNN model in a distributed way over 2 GPUs. I’m using <a href="https://github.com/pytorch/vision/tree/master/references/detection" rel="noopener nofollow ugc">this</a> as a template.</p>
<p>I can get it working with the coco dataset, and am now repurposing it for my own dataset. I can get it to train but evaluation is extremely slow. I’m talking an hour to train and over 2 hours for evaluation.</p>
<p>When looking at the evaluate function in engine.py, I noticed this line:</p>
<p><code># FIXME remove this and make paste_masks_in_image run on the GPU</code></p>
<p>Is it so slow because it’s running on the CPU?</p>

ptrblck: <p>You could profile the code to get more information where the bottleneck of the evaluation run is.<br>
The <code>FIXME</code> message sounds suspicious, so could you link to the line of code?</p>
PyTorch 1.7 debug build is True when installed from binary

ydcjeff: <p>PyTorch 1.7 debug build returns True, run with python -m torch.utils.collect_env<br>
Installed with <code>pip install torch==1.7.0+cu101 torchvision==0.8.1+cu101 torchaudio==0.7.0 -f https://download.pytorch.org/whl/torch_stable.html</code> from <a href="http://pytorch.org" rel="noopener nofollow ugc">pytorch.org</a>.</p>
<pre><code class="lang-auto">Collecting environment information...
PyTorch version: 1.7.0+cu101
Is debug build: True
CUDA used to build PyTorch: 10.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.5 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: 6.0.0-1ubuntu2 (tags/RELEASE_600/final)
CMake version: version 3.12.0

Python version: 3.6 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: 10.1.243
GPU models and configuration: GPU 0: Tesla T4
Nvidia driver version: 418.67
cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.5
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.18.5
[pip3] torch==1.7.0+cu101
[pip3] torchaudio==0.7.0
[pip3] torchsummary==1.5.1
[pip3] torchtext==0.3.1
[pip3] torchvision==0.8.1+cu101
[conda] Could not collect
</code></pre>
<p>While in 1.6</p>
<pre><code class="lang-auto">Collecting environment information...
PyTorch version: 1.6.0+cu101
Is debug build: No
CUDA used to build PyTorch: 10.1

OS: Ubuntu 18.04.5 LTS
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
CMake version: version 3.12.0

Python version: 3.6
Is CUDA available: Yes
CUDA runtime version: 10.1.243
GPU models and configuration: GPU 0: Tesla T4
Nvidia driver version: 418.67
cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.5

Versions of relevant libraries:
[pip3] numpy==1.18.5
[pip3] torch==1.6.0+cu101
[pip3] torchsummary==1.5.1
[pip3] torchtext==0.3.1
[pip3] torchvision==0.7.0+cu101
[conda] Could not collect
</code></pre>
<p>Environment: Colab<br>
What does it mean? Is this a new changes in 1.7? I can’t see in the release notes.</p>
<p>Thank you.</p>

ptrblck: <p>Thanks for raising this issue. We’ll check, if this needs to be removed.</p>

ydcjeff: <p>Thank you for your answer <a class="mention" href="/u/ptrblck">@ptrblck</a><br>
I raised the GH issue here: <a href="https://github.com/pytorch/pytorch/issues/46973" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/issues/46973</a></p>
Computing invariant subspaces between two matrices

Gaurish_Thakkar: <p>With respect to the paper below.<br>
<a href="https://core.ac.uk/download/pdf/82384172.pdf" rel="noopener nofollow ugc">https://core.ac.uk/download/pdf/82384172.pdf</a>. I am interested in computing the invariant subspace of the two matrices.</p>
<p>Any help in this regard</p>
How to create trace matrix for im2col (unfold)

kevin-ssy: <p>Hi there,</p>
<p>I want to create a trace matrix for the classic im2col function. Given a 4D tensor [N, C, H, W] and an unfold function with kernel size (k*k) which can generate L patches. What I want is a trace matrix [N, L, H, W] where the value is set to 1 if the pixel lies in the current patch, otherwise it will be set to 0.</p>
<p>A rough instance:<br>
input (2D):</p>
<pre><code>[[1, 2, 3, 4],
 [5, 6, 7, 8]]
</code></pre>
<p>kernel size: 2x2, stride: 1</p>
<p>output matrix</p>
<pre><code>  1  2  3  4  5  6  7  8  (flatten idx)
[[1, 1, 0, 0, 1, 1, 0, 0],
 [0, 1, 1, 0, 0, 1, 1, 0],
 [0, 0, 1, 1, 0, 0, 1, 1]]
</code></pre>
<p>Is it possible to implement this without writing a CUDA module?</p>
How can I solve the problem?

Pann: <p>I want to know whether the plotted diagram is informal or not.Please answer how I solve the problem if it is wrong.Thanks in advance!<img src="https://discuss.pytorch.org/uploads/default/original/3X/c/0/c042fdcf12a22d51cfecafd280ad37cd2bb69cbd.png" alt="loss" data-base62-sha1="rqPczFdZ8naAVCLpj9kTtKB0fSt" width="640" height="476"></p>
How to tune recall, precision for neural network

satinder147: <p>Hi all,<br>
I am using a 3 dense layer architecture for a multi-classification problem. I am able to calculate the metrics(recall and precision), but my use case requires one of the metrics to be higher. I wanted to know how can we tune an MLP classifier for such use case?</p>

Abhilash_Srivastava: <p>It’s hard to suggest ways without knowing how your data looks like. A few things which have worked for me in the past:</p>
<ol>
<li>Ensure that train, valid, and test data come from the same distribution.</li>
<li>The model is trained correctly (ie no underfitting and no overfitting).</li>
<li>Improve precision: Analyse the confusion matrix. Look at the examples which are misclassified (FPs) and mine data with data similar to misclassified examples (FP mining). Retrain model.</li>
<li>Improve recall: Again look at the confusion matrix to look for FNs. It maybe the case that there are very few examples from one of the classes hence the model is not learning to classify that class. Add more data (or oversample) and train again.</li>
</ol>
<p>In most cases there’s a tradeoff between precision and recall.</p>
Pytorch 1.7.0, AttributeError: 'torch.Size' object has no attribute 'dtype' when using __torch_function__

ProGamerGov: <p>The following code worked before the 1.7.0 update, and now it fails.</p>
<pre><code class="lang-auto">    @classmethod
    def __torch_function__(self, func, types, args=(), kwargs=None):
        if kwargs is None:
            kwargs = {}
        args = [a.tensor() if hasattr(a, 'tensor') else a for a in args]
        test = func(*args, **kwargs)
        print(test.dtype)
        return TestTensor(test.float()) # Fails if the above print statement is commented out.
</code></pre>
<p>results in this error message:</p>
<pre><code class="lang-auto">
     25         test = func(*args, **kwargs)
---&gt; 26         print(test.dtype)
     27         return TestTensor(test.float())
     28 AttributeError: 'torch.Size' object has no attribute 'dtype'
</code></pre>
<p>How do I fix this?</p>

ProGamerGov: <p>Turns out it’s because of a change in that’s listed in the docs here: <a href="https://pytorch.org/docs/stable/notes/extending.html" class="inline-onebox" rel="noopener nofollow ugc">Extending PyTorch — PyTorch 2.1 documentation</a>, but not in the release notes.</p>
<blockquote>
<p>One should be careful within <code>__torch_function__</code> for subclasses to always call <code>super().__torch_function__(func, ...)</code> instead of <code>func</code> directly, as was the case before version 1.7.0. Failing to do this may cause <code>func</code> to recurse back into <code>__torch_function__</code> and therefore cause infinite recursion.</p>
</blockquote>
High Quantization-Aware training memory consumption

Georgios_Georgiadis: <p>I have noticed that once Quantization-Aware Training starts memory consumption increases significantly. E.g. for a network that consumes 2GB during training, I see memory consumption jumping to 4.5GB once QAT starts.<br>
While in part, this can be explained due to the various statistics that need to be stored, the difference in memory consumption is much larger than what these stats occupy. I am guessing that this might be due to additional intermediate activation maps that need to be kept around, but I was hoping that someone more informed could drop in and clarify if this is the reason.</p>

Vasiliy_Kuznetsov: <p>Hi <a class="mention" href="/u/georgios_georgiadis">@Georgios_Georgiadis</a>, one known problem is that fake_quantize modules are currently implemented as additional nodes in the computation graph, so their additional outputs (the fake_quantized versions weights and activations) contribute to the memory overhead during training.  We have plans to improve the memory overhead in the future by adding fused fake_quant kernels for common layers such as conv and linear.</p>
Can I perform inference with a quantized model on the GPU in PyTorch 1.6?

lcrew11: <p>Can I perform inference on the GPU with a quantized model? Articles on the PyTorch website mention that 'PyTorch 1.3 doesn’t provide quantized operator implementations on CUDA yet ’ (<a href="https://pytorch.org/docs/stable/quantization.html" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/quantization.html</a>) but there is no clarification if any additional GPU support has been added to PyTorch 1.6, or whether it is planned in any future releases.</p>

supriyar: <p>We don’t support quantized model inference on GPU currently. The docs have been updated to reflect that.</p>
Quantization of Reflection Padding?

nile649: <p>Hi, please enlighten me. How can I write a custom function to overcome this problem?<br>
Is it possible to replace padding in the pre-trained model with some Quantization supported operator?</p>

supriyar: <p>Quantization support for reflection_pad1d was added in <a href="https://github.com/pytorch/pytorch/pull/37452" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/pull/37452</a>. cc <a class="mention" href="/u/zafar">@Zafar</a></p>
<p>you can follow that PR to add support for additional reflection_pad operators if they aren’t supported.</p>
Question about how to load quantized model

zoeyyes: <p>hi, I quantized my model and I can save model succesfully, but when i  evaluta  model with torch.jit. there are some thing error:</p>
<pre><code class="lang-auto">        feature2 = self.L2normalize(feature2)
  File "/media/zst/8ec88aab-d885-4801-98ab-e3181c65261b/A/pro/re.py", line 20, in L2normalize
    def L2normalize(self, x):
        eps = 1e-6
        norm = x ** 2
               ~~~~~~ &lt;--- HERE
        norm = norm.sum(dim=1, keepdim=True) + eps
        #norm = torch.sum(norm,1) + eps
RuntimeError: Could not run 'aten::pow.Tensor_Scalar' with arguments from the 'QuantizedCPU' backend. 'aten::pow.Tensor_Scalar' is only available for these backends: [CPU, CUDA, SparseCPU, SparseCUDA, Named, Autograd, Profiler, Tracer, Autocast].
</code></pre>
<p>but when I test model with torch.load:</p>
<pre><code class="lang-auto">	Unexpected key(s) in state_dict: "conv1a.0.scale", "conv1a.0.zero_point", "conv1aa.0.scale", "conv1aa.0.zero_point", "conv1b.0.scale", "conv1b.0.zero_point", "conv2a.0.scale", "conv2a.0.zero_point", "conv2aa.0.scale", "conv2aa.0.zero_point", "conv2b.0.scale", "conv2b.0.zero_point", "conv3a.0.scale", "conv3a...
While copying the parameter named "conv1a.0.weight", whose dimensions in the model are torch.Size([16, 3, 3, 3]) and whose dimensions in the checkpoint are torch.Size([16, 3, 3, 3]), an exception occured : ('Copying from quantized Tensor to non-quantized Tensor is not allowed, please use dequantize to get a float Tensor from a quantized Tensor',)...
</code></pre>

supriyar: <p>We don;t support quantized op implementation of aten::pow currently. To get around this you can add a quant/dequant stub around your operator.</p>
<p>Please refer to the tutorial for how to do this - <a href="https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html" rel="noopener nofollow ugc">https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html</a></p>
PyTorch Extensions Object-Oriented Interface

leimao: <p>TensorFlow provided base classes for the users to inherit and virtual methods to override for their custom ops in C++. In PyTorch, however, this only happens in Python. Is there any reason why PyTorch designs its current extension interfaces like this? Any plans to provide extension base classes to the users to inherit in the near future? Thank you very much.</p>

alanzhai219: <p>Sure, pytorch has provides both python and C++ to extend custom op. Further more, cuda and other accelerate device have been supported. You can refer to pytorch tutorial <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html" rel="noopener nofollow ugc">https://pytorch.org/tutorials/advanced/cpp_extension.html</a>.<br>
Thanks,<br>
Alan</p>

leimao: <p><a class="mention" href="/u/alanzhai219">@alanzhai219</a> I checked Peter’s tutorial. Essentially we are creating a C++ function that is bonded to Python using Pybind11. There is no base class of the custom op to inherit in PyTorch C++, where TensorFlow provides a base class of the custom op for users to extend.</p>