akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Convolution arithmetic for cuda/cudnn

username: Levin
cooked: <p>I found convolution implementation in cuda <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/THCUNN/generic/SpatialConvolutionMM.cu" rel="noopener nofollow ugc">here</a>. I am not able to figure out what’s exactly happening there.</p>
<p>I guess it won’t be doing sliding window and then dot product and addition for each window. Can you please share the arithmetic that’s happening while doing convolution operation in cuda/cudnn?</p>
<p>Thank you!</p>

username: ptrblck
cooked: <p>The native convolution operation in PyTorch use a matrix multiplication after unfolding the input (often referred as <code>im2col</code>), while the cudnn implementations are closed source, but you might be able to get some information about the used algorithms in their docs.</p>
Efficient way to get ordered permuations

username: ADONAI_TZEVAOT
cooked: <p>I have integer numbers in ascending order from 1 to n with a difference of 1 like this :1,2,3,4,…, n . I want to get ordered permutation of the desired length of m. for example if n is 3 and m is 2 then the output should be (1,2),(2,3),(1,3).</p>

username: ptrblck
cooked: <p>I guess you could use <code>itertools.permutations</code> to permute your list.<br>
PS: this doesn’t seem to be PyTorch-specific, so note that you might get a faster and better answer e.g. on StackOverflow. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
Pytorch inference slow on cpu when i modified torchvision.models.mobilenet_v2

username: zihaog
cooked: <p>Hi,</p>
<p>I’m trying to do the inference on cpu with torchvision.models.mobilenet_v2<br>
If I load a pretrained mobilenet_v2, get rid of some layers of it and retrain, the inference time is very slow when I use my new weights. Here is the code:</p>
<pre><code class="lang-auto">device = "cpu"
parameters = "xxx.pth"
model = models.mobilenet_v2(pretrained=False)
model.features = nn.Sequential(*[model.features[i] for i in range(15)])
model.classifier = nn.Linear(160, 2, bias=True)
params = torch.load(parameters, map_location=device)
model.load_state_dict(params['state_dict'], strict=True)
model.to(device)
model.eval()

test_pic = os.listdir("xxx/test_pic")
for img_name in test_pic:
    img_path = os.path.join("xxx/test_pic", img_name)
    img = cv2.imread(img_path)
    with torch.no_grad():
        img = img.astype(np.float32)
        img = img[:, :, (2, 1, 0)]
        img = np.transpose(img, (2, 0, 1))
        img = torch.tensor(img)
        img = img.unsqueeze(dim=0)
        img = img.to(device)
        st = time.time()
        out = model(img)
        print(time.time() - st)
</code></pre>
<p>In this case, the time cost is about 1s per inference</p>
<p>And if I don’t change anything to the mobilenet, here is the code:</p>
<pre><code class="lang-auto">device = "cpu"
model = models.mobilenet_v2(pretrained=True)
model.to(device)
model.eval()

test_pic = os.listdir("xxx/test_pic")
for img_name in test_pic:
    img_path = os.path.join("xxx/test_pic", img_name)
    img = cv2.imread(img_path)
    with torch.no_grad():
        img = img.astype(np.float32)
        img = img[:, :, (2, 1, 0)]
        img = np.transpose(img, (2, 0, 1))
        img = torch.tensor(img)
        img = img.unsqueeze(dim=0)
        img = img.to(device)
        st = time.time()
        out = model(img)
        print(time.time() - st)
</code></pre>
<p>The time cost reduced to 150ms per inference.</p>
<p>I’ve checked my model is 2.9MB and the official pretrained model is 13.55MB.</p>
<p>Is it normal? Is there something wrong with my model? Any ideas will be appreciated.</p>
<p>thanks</p>

username: ptrblck
cooked: <p>Depending on your CPU you might be able to use <code>torch.set_flush_denormal(True)</code> to avoid a slower code path for denormal values (if that’s causing the slowdown, which could be the case based on your description).</p>
Division in batches of a 3D tensor

username: Navpreet_Singh
cooked: <p>I have a 3D tensor of size say 100x5x2 and mean of the tensor across axis=1 which gives shape 100x2.<br>
100 here is the batch size. Normally without batch, the division of tensor of shape 5x2 and 2 works perfectly but in the case of the 3D tensor with batch, I’m receiving error.</p>
<pre><code>a = torch.rand(100,5,2)
b = torch.rand(100,2)
z=a/b
</code></pre>
<p>The size of tensor a (5) must match the size of tensor b (100) at non-singleton dimension 1.<br>
How to divide these tensors such that my output is of shape 100x5x2 ? Something like bmm for division?</p>

username: pascal_notsawo
cooked: <p>I think you can unsqueeze <code>b</code> so that <code>a.dim() = b.dim()</code></p>
<pre><code class="lang-auto">
import torch

n, m, p = 100,5,2

a = torch.rand(n, m, p)

b = torch.rand(n, p)

b = b.unsqueeze(dim=1) # n x 1 x p

z=a/b

</code></pre>

username: Navpreet_Singh
cooked: <p>Yeah that works. Thanks alot.</p>
ONNX export error for "RoiAlign"

username: oliver_ss
cooked: <p>When I try to export the “roi_align” operator in torchvision to onnx, I meet the following error.</p>
<p>Do you know how to fix it? Thank you!</p>
<pre><code class="lang-auto">RuntimeError: ONNX export failed on an operator with unrecognized namespace torchvision::roi_align. If you are trying to export a custom operator, make sure you registered it with the right domain and version. Otherwise, please report a bug.
</code></pre>

username: ptrblck
cooked: <p>Based on the error message I guess you might need to register this custom operation as described <a href="https://pytorch.org/docs/stable/onnx.html#custom-operators">here</a>.</p>
Use distance between label and output for loss calculation

username: CoconutMilk
cooked: <p>Hello. I am currently in the process of creating a GRU model to guess the location of a user in a simulation. I have quantized the 100x100 2D location into a matrix of 0s and 1s.</p>
<p>Ex:<br>
Agent is in location x = 5 in a 1D map size of 10:<br>
encoded label tensor = [0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0]</p>
<p>Based on this, I am trying to write a differentiable loss that is based on the distance between the highest probable encoded output position and label:.</p>
<p>Ex:<br>
Agent is in location x = 5 in a 1D map size of 10:<br>
encoded label tensor    = [0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0] =&gt; argmax = 5th cell<br>
output tensor                = [0.5, 0.1, 0.2, 0.1, 0.2, 0.6, 0.3, 0.4, 0.4, 0.2] =&gt; argmax = 6th cell</p>
<p>Calculated distance = 5 - 6.</p>
<p>This is a simple example and I work on a 2D space so I have to calculate the Euclidean distance. I understand how the argmax argument isn’t differentiable but I am thinking of a way to incorporate this distance value into the loss so that my penalties make sense.</p>
<p>I tried using the Gumbel function to create the one-hot-encoded output tensor but not sure where to go from there since I still would need an argmax.</p>
<p>Sorry if the problem is confusing or if my explanation of it isn’t good enough.<br>
Any help is appreciated.</p>
ModuleNotFoundError: No module named 'engine'

username: netqyq
cooked: <p>When I executed the source <a href="https://pytorch.org/tutorials/_static/tv-training-code.py" rel="nofollow noopener">code</a>.</p>
<p>The code is from the tutorial <a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html</a></p>
<p>It said that:</p>
<pre><code class="lang-bash">(base) [yq@local pytorch-examples]$python3 tv-training-code.py
Traceback (most recent call last):
  File "tv-training-code.py", line 13, in &lt;module&gt;
    from engine import train_one_epoch, evaluate
ModuleNotFoundError: No module named 'engine'
</code></pre>
<p>I don’t know which package is called ‘engine’ here.</p>

username: ptrblck
cooked: <p>From the tutorial:</p>
<blockquote>
<p>In <code>references/detection/</code> , we have a number of helper functions to simplify training and evaluating detection models. Here, we will use <code>references/detection/engine.py</code> , <code>references/detection/utils.py</code> and <code>references/detection/transforms.py</code> . Just copy them to your folder and use them here.</p>
</blockquote>

username: Frank_Wang1
cooked: <p>So how do we get the references/detection/ folders? What should we download and install? I have installed the pytorch, torchvision in my environment, but I could not find those files. Thanks</p>

username: ptrblck
cooked: <p>You could copy/paste these files fro the <code>torchvision</code> repository or clone it locally.</p>

username: pvtien96
cooked: <aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicon.ico" class="site-icon" width="32" height="32">
      <a href="https://github.com/pytorch/vision" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars2.githubusercontent.com/u/21003710?s=400&amp;v=4" class="thumbnail onebox-avatar" width="400" height="400">

<h3><a href="https://github.com/pytorch/vision" target="_blank" rel="nofollow noopener">pytorch/vision</a></h3>

<p>Datasets, Transforms and Models specific to Computer Vision - pytorch/vision</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


username: pvtien96
cooked: <p>git clone <a href="https://github.com/pytorch/vision" rel="nofollow noopener">https://github.com/pytorch/vision</a><br>
git clone <a href="https://github.com/cocodataset/cocoapi" rel="nofollow noopener">https://github.com/cocodataset/cocoapi</a><br>
In vision/references/detection: copy all .py file to [tv-training-code.py] directory<br>
Copy cocoapi/PythonAPI/pycocotools to [tv-training-code.py] directory</p>
Plt.show and plt.plot difference in heatmap?

username: p_jung
cooked: <p>Hello</p>
<p>I have heatmap result as below<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/7/e/7eb81faed32dbf61f11d928e42a2fff711771716.png" alt="화면 캡처 2021-06-03 174133" data-base62-sha1="i50DXU2aTgpiisXGzKgCGbCJfMy" width="381" height="435"></p>
<p>landmark_heatmap.shape is (600,480)</p>
<p>left pic is<br>
plt.imshow(landmark_heatmap.detach().numpy())<br>
right pic is<br>
plt.plot(landmark_heatmap.detach().numpy())</p>
<p>why is it? can I change axis of plot? it dosent mention about it on doc…</p>
<p>also, I wonder if I can make gaussian fit to heatmap…to get confident</p>

username: ptrblck
cooked: <p><a href="https://matplotlib.org/stable/api/_as_gen/matplotlib.pyplot.imshow.html"><code>plt.imshow</code></a> is used to visualize image-like arrays and will plot the pixel intensity for each pixel location using a colormap (or would directly use the colors in case the image has 3 channels), while <a href="https://matplotlib.org/stable/api/_as_gen/matplotlib.pyplot.plot.html?highlight=plot#matplotlib.pyplot.plot"><code>plt.plot</code></a> would visualize each “row” of the input array as a new line and would use the pixel intensities as the y-value.</p>
How to apply early stopping in training code?

username: aatiibutt
cooked: <p>I am training deeplab but facing overfitting issue. To overcome this issue, i wanna add early stopping (i.e., patience) in my training block to avoid overfitting. However, I don’t know how and where to insert early stopping code in training code block. I have attached my training execution code below.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/a/2ae00a42c46f7686776730c082193eac472e1ce0.png" data-download-href="https://discuss.pytorch.org/uploads/default/2ae00a42c46f7686776730c082193eac472e1ce0" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/a/2ae00a42c46f7686776730c082193eac472e1ce0_2_690x479.png" alt="image" data-base62-sha1="67i3yICopKzFnk3rCLdz7dpFBPa" width="690" height="479" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/a/2ae00a42c46f7686776730c082193eac472e1ce0_2_690x479.png, https://discuss.pytorch.org/uploads/default/original/3X/2/a/2ae00a42c46f7686776730c082193eac472e1ce0.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/2/a/2ae00a42c46f7686776730c082193eac472e1ce0.png 2x" data-dominant-color="21221F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">714×496 102 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I shall be grateful, if anybody help me out.</p>
BCELossWithLogits Probability below 0.5

username: Jam
cooked: <p>I am working on a binary classification problem for which I am using the BCELosswithLogits and (since my data is imbalanced) the BinaryFocalLoss. To get the probability of the sample being positive, I apply the sigmoid function to the output of the model. However, even though I could improve my model from 68% AUC to 81% AUC, when I look at the predicted probabilities, they are always below 0.5. Am I doing something wrong?</p>

username: ptrblck
cooked: <p>If the probabilities are always below 0.5, which I assume is the used threshold to get the predictions, I would also guess that the model is only predicting class0, i.e. the negative class in a binary classification setup?<br>
If so, did you check the confusion matrix or other metrics?</p>

username: Jam
cooked: <p>Thanks for your reply! I will plot the confusion matrix but how do I know which is the correct threshold? So far, I assumed it is 0.5 but it could be different, couldn’t it?</p>

username: ptrblck
cooked: <p><code>0.5</code> could be a good starting point and you could check the ROC for different thresholds and pick the best suitable one for your use case.</p>

username: Jam
cooked: <p>Thanks, I will try do to that as well.<br>
So when using 0.5 as the threshold, I get the following confusion matrix:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/9/d/9d0fcd949f6bb17451696e57e965e98af3afeb32.png" alt="grafik" data-base62-sha1="mpqOSUAT721i08QSsx3DIdM7RQu" width="383" height="285"></p>
<p>That means, that my model actually predicts both classes but it only predicts FP. So it doesn’t get any of the TP, correct? I don’t understand how that is possible. Do you think the threshold could affect this?</p>

username: ptrblck
cooked: <p>Yes, it seems that no FP are returned. Also yes, you can check the ROC to select the threshold for the TPR vs. FPR value you want.</p>

username: Jam
cooked: <p>I adjusted the threshold accordingly (it is 0.25). Does it make sense that my lowest probability is only 0.126 (I’d have assumed something closer to 0) and my highest (best predictions for TP) is still below 0.5? Should I not expect something close to 1?</p>

username: ptrblck
cooked: <p>The smaller range of values might indicate that the model wasn’t trained for many epochs and thus the logits have a relative narrow range. You could thus try to train the model for more epochs, in case it’s not overfitting.</p>

username: Jam
cooked: <p>Thanks for your reply! So far it is overfitting after more than 30 epochs but I will try to make further adjustments.</p>
How to keep the outputs of different networks and then use it?

username: smani
cooked: <p>I need to aggregate the outputs from different networks and then do some calculations based on this aggregated outputs, and finally update individual networks. I am unclear about how to keep the outputs which are obtained in the first for loop (from individual networks) and use it in the second for loop to calculate the loss and update individual networks. I very much appreciate if you give me any suggestion.</p>
<pre><code class="lang-auto">outputs = []
for i in range(noOfModels):
    optimizer[i].zero_grad()
    tmp = net[i](img) # this will return an output of size 10x10 
    outputs.append(tmp)

outputs = torch.stack(outputs, 2)
x = do_somthing(outputs) # do some calculations with outputs

for i in range(noOfModels):
    loss = criterion(outputs[:,:,i], x)
    loss.backward()
    optimizer[i].step()
</code></pre>

username: tom
cooked: <p>I would probably just add the losses like this:</p>
<pre><code class="lang-auto">loss = 0
for i in range(noOfModels):
    loss = loss + criterion(outputs[:,:,i], x)
loss.backward()

for i in range(noOfModels):
    optimizer[i].step()
</code></pre>
<p>This should work if you didn’t break the computational graph somewhere.</p>
RuntimeError: Unknown IValue type for pickling: Device (pushIValueImpl at /opt/conda/conda-bld/pytorch_1579022030672/work/torch/csrc/jit/pickler.cpp:132)

username: Tekker_Tiger
cooked: <p>i get this error when i use torch.jit.script then torch.jit.save<br>
: frame <span class="hashtag">#0:</span> c10::Error::Error(c10::SourceLocation, std::string const&amp;) + 0x47 (0x7f37df3cb627 in /home/kencorp/miniconda3/lib/python3.7/site-packages/torch/lib/libc10.so)<br>
frame <span class="hashtag">#1:</span>  + 0x2f9d95d (0x7f37e258e95d in /home/kencorp/miniconda3/lib/python3.7/site-packages/torch/lib/libtorch.so)<br>
frame <span class="hashtag">#2:</span>  + 0x2f9dc3b (0x7f37e258ec3b in /home/kencorp/miniconda3/lib/python3.7/site-packages/torch/lib/libtorch.so)<br>
frame <span class="hashtag">#3:</span>  + 0x2f9d1ee (0x7f37e258e1ee in /home/kencorp/miniconda3/lib/python3.7/site-packages/torch/lib/libtorch.so)<br>
frame <span class="hashtag">#4:</span>  + 0x2f9dc5b (0x7f37e258ec5b in /home/kencorp/miniconda3/lib/python3.7/site-packages/torch/lib/libtorch.so)<br>
frame <span class="hashtag">#5:</span>  + 0x32bcdd9 (0x7f37e28addd9 in /home/kencorp/miniconda3/lib/python3.7/site-packages/torch/lib/libtorch.so)<br>
frame <span class="hashtag">#6:</span>  + 0x32c0bdd (0x7f37e28b1bdd in /home/kencorp/miniconda3/lib/python3.7/site-packages/torch/lib/libtorch.so)<br>
frame <span class="hashtag">#7:</span> torch::jit::ExportModule(torch::jit::script::Module const&amp;, std::string const&amp;, std::unordered_map&lt;std::string, std::string, std::hash<a>std::string</a>, std::equal_to<a>std::string</a>, std::allocator&lt;std::pair&lt;std::string const, std::string&gt; &gt; &gt; const&amp;, bool) + 0x19b (0x7f37e28aba5b in /home/kencorp/miniconda3/lib/python3.7/site-packages/torch/lib/libtorch.so)<br>
frame <span class="hashtag">#8:</span>  + 0x743dd2 (0x7f37e581cdd2 in /home/kencorp/miniconda3/lib/python3.7/site-packages/torch/lib/libtorch_python.so)<br>
frame <span class="hashtag">#9:</span>  + 0x26b876 (0x7f37e5344876 in /home/kencorp/miniconda3/lib/python3.7/site-packages/torch/lib/libtorch_python.so)<br>
<br>
frame <span class="hashtag">#29:</span> __libc_start_main + 0xf0 (0x7f37f0fb1830 in /lib/x86_64-linux-gnu/libc.so.6)</p>

username: driazati
cooked: <p>Support for <code>Device</code> was added somewhat recently, are you on the latest version of PyTorch? If so, can you open an issue on GitHub with a repro?</p>

username: Tekker_Tiger
cooked: <blockquote>
<blockquote>
<blockquote>
<p>import torch<br>
torch.<strong>version</strong><br>
‘1.4.0’</p>
</blockquote>
</blockquote>
</blockquote>

username: addisonklinke
cooked: <p>I found a similar issue in <code>1.4.0</code>, and upgrading to <code>1.8.1</code> fixes it. The release notes for <code>1.8.0</code> reference PR <a href="https://github.com/pytorch/pytorch/pull/46441" rel="noopener nofollow ugc">#46441</a> which</p>
<blockquote>
<p>adds hashing for many other types including <code>Tuple</code> , <code>bool</code> , <code>device</code> by implementing generic hashing on IValue</p>
</blockquote>
<p>I believe this is the improvement that resolved the root issue</p>
Creating a custom pass torch 1.8.0

username: Andre_Chang
cooked: <p>is there are new way to create a custom transformation pass in torchscript with torch1.8.0 ?</p>
<p>just like here: <a href="https://github.com/bwasti/pytorch_compiler_tutorial/blob/master/src/register.cpp#L26" class="inline-onebox" rel="noopener nofollow ugc">pytorch_compiler_tutorial/register.cpp at master · bwasti/pytorch_compiler_tutorial · GitHub</a></p>
<p>torch.jit.trace() doesnt call: RegisterPass pass anymore</p>

username: tince
cooked: <p>Hey Andre, I am curious, were you able to find a solution to this issue? I haven’t looked at 1.8 but I am interested in hearing what you found out.</p>

username: Andre_Chang
cooked: <p>No. I havent found a solution yet.<br>
Anyone can help out?</p>

username: tom
cooked: <p><code>RegisterPass</code> still is the right thing to do (or use <code>registerPostPass</code> if you want to call something - <code>RegisterPass</code> is just a trick to call <code>registerPostPass</code> at initialization, there also is a <code>registerPrePass</code> to get an earlier graph). Note that custom passes are not called during the tracing/scripting phase of the JIT but during the optimization phase in the graph executors.<br>
I once wrote a <a href="https://lernapparat.de/jit-runtime-overview/" rel="noopener nofollow ugc">overview blog post about the runtime phases</a>.</p>
<p>Best regards</p>
<p>Thomas</p>

username: Andre_Chang
cooked: <p>Thank you very much for the explanation</p>
Protecting Pytoch .pt model from decompilation in mobile app

username: mxf
cooked: <p>I wonder if there is a good practice available on how to protect trained Pytorch models compiled to <code>.pt</code> files and deployed to an Android app from being readable to anyone decompiling the app?<br>
Seems to be an important IP-protection issue.</p>
<p>I am writing an app that needs to be very performant and thus I move parts of the app logic into a C++ library that I include as an external package to my Android project.<br>
Would it be theoretically possible to include my Pytorch models directly as a dependency to the C++ library and include it together with the C++ library as a compiled artifact in the Android project?</p>
<p>Anyone has any experience with something like this?</p>

username: William.Tambellini
cooked: <p>Hi <a class="mention" href="/u/mxf">@mxf</a><br>
Be aware that from 1.6, .pt models are just zip files:<br>
<a href="https://pytorch.org/tutorials/beginner/saving_loading_models.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/saving_loading_models.html</a><br>
To reply to your question, yes, it would be possible to embed such file in a binary (sometimes called a ‘resource’).</p>
An't convert CUDA tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first

username: Skumarm
cooked: <p>def predict(image_path, model, top_k=top_num):</p>
<pre><code>model.to(power)
model.eval()

torch_image =torch.from_numpy(np.expand_dims(process_image(image_path),
                                        axis = 0)).type(torch.FloatTensor).to(power)

  
log_prob = model.forward(torch_image)

linear_probs = torch.exp(log_prob)   

top_probs, top_labs = linear_probs.topk(top_k, dim =1)

idx_to_class = {}
for key, value in model.class_to_idx.items():
    idx_to_class[value] = key

np_top_labs = top_labs[0].numpy()

top_labels = []
for label in np_top_labs:
    top_labels.append(int(idx_to_class[label]))

top_flowers = [cat_to_name[str(lab)] for lab in top_labels]

return top_probs, top_labels, top_flowers
</code></pre>
<p>image_path = path_image</p>
<p>prob, classes, flowers = predict(image_path,model)</p>

username: ptrblck
cooked: <p>As the error describes, you would have to transfer the <code>CUDATensor</code> to the CPU first, since numpy cannot use GPU tensors.<br>
Assuming this line of code raises the issue:</p>
<pre><code class="lang-python">np_top_labs = top_labs[0].numpy()
</code></pre>
<p>use this instead:</p>
<pre><code class="lang-python">np_top_labs = top_labs[0].cpu().numpy()
</code></pre>
<p>(and <code>.detach()</code>, if necessary).</p>

username: Skumarm
cooked: <p>Thank you for the support.</p>
<p>Regards</p>
Best way to encode latent variable with CNN?

username: Arthur_Zakirov
cooked: <p>Hello everyone,</p>
<p>after studying codes from different authors, I’ve seen different approaches to encode an image into a latent variable.</p>
<p>1 )  [N, 1,   28,   28] → Conv → [N, 32, 14, 14] → Conv → [N, 64,   7,   7] → Conv [N, 128, 1, 1]<br>
 → fc → [N, latent_dim]</p>
<p>2 )  [N, 1, 100, 100] → Conv → [N, 10, 48, 48] → Conv → [N, 20, 22, 22] → Conv [N, 1,   14, 14]<br>
 → flatten → fc → [N, latent_dim]</p>
<p>I’ve heard that the channels should increase, while the width should decrease. However in the 2) example the channels decrease at the end again back to 1. The use case for 2) is to encode a geographic map for vehicle trajectory prediction, in case you wonder. However it proved to be absolutely useless for my MNIST Variational Autoencoder.<br>
How do I know when to use this kind of approach?</p>
<p>Also regarding 1), are there any other (preferably better) ways to do this, when the goal is to build a VAE?</p>
<p>Thanks in advance</p>

username: tom
cooked: <p>(I think you might need a flatten in 1, too.)</p>
<p>We have a short discussion of the more general number of channels and width in our book (Stevens/Antiga/Viehmann: Deep Learning with PyTorch, Section “8.3.1 Our network as an nn.Module”).</p>
<p>So the idea behind “channels should increase while width should decrease” is mainly “total size should decrease but not too much”. That the channels increase is more a way to achieve “not too much” in general (so halving the spatial resolution + doubling channels halves the number of elements).</p>
<p>Aside from the size (196 vs 128), the key difference between 1 and 2 is that the first evicts all explicitly spatial information while the second keeps it but reduces the information “per pixel”. Which is more appropriate is likely task dependent. (And you could mix, too, if you feel like it.)</p>
<p>For the VAE, it might make sense to to keep some spatial information as might want to be able reconstruct flipped/rotated images (as a vague intuition), but the typical thing might be to keep more than one channel. But I would recommend to check out the common VAE architectures. It is still good to ask “why” they did it this way, but it’d give you a starting point.</p>
<p>Best regards</p>
<p>Thomas</p>
Optical flow dataset normalization

username: demonvferrari
cooked: <p>Hello EVeryone,</p>
<p>When using optical folow data to train an autoencoder,should we normalize the optical flow data? If so, can we use the imagenet normalization parameters(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])? Or, do we have to calculate the normalization parameters of the optical flow dataset?</p>
<p>Thank you</p>
Check if model is eval or train

username: Rafael_Valle
cooked: <p>How can one check is a model is in train or eval state?</p>

username: SimonW
cooked: <p><code>module.training</code> is the boolean you are looking for. <img src="//discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: Rafael_Valle
cooked: <p>That’s easy! Thank you!</p>

username: jliude
cooked: <p>great! thanks. just solve my issue.</p>

username: el_youssfi_azeddine
cooked: <p>me too, this just solved my issue, thanks Simon!</p>
nn.Relu but the dimension is not normal

username: SungmanHong
cooked: <p>I have batch x 4096 x 6(time samples) data in nn.<br>
I want to nn.Relu, so I want to reduce the data dimension to<br>
batch x 512 x 6(time samples)<br>
but the nn.Relu layer take the last dimension, if I understand correct.</p>
<p>How can I do a Relu, in the wanted dimension…?</p>

username: ptrblck
cooked: <p>The <code>nn.ReLU</code> layer won’t apply any reduction to the data, but would apply the relu activation on each element as seen here:</p>
<pre><code class="lang-python">batch_size = 2
x = torch.randn(batch_size, 4096, 6)
relu = nn.ReLU()
out = relu(x)
print(out.shape)
&gt; torch.Size([2, 4096, 6])
</code></pre>
<p>Depending on what <code>dim1</code> represents you could reduce it via e.g. <code>nn.Conv1d</code> layers assuming it’s used for the channel dimension.</p>
How to do multiple partial deriative of a loss funciton w.r.t. different parameters?

username: phantom90
cooked: <p>Hi there,</p>
<p>Suppose we have a loss function: loss = f(x, y), with x and y be two parameters. We want to do the following two operations:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/d/eda4ee4d339470abb63c201b1c51672110e97abc.png" alt="image" data-base62-sha1="xUiwrmbKacnlKdeQaT3vawdtJO4" width="27" height="45"></p>
<p>Are there any hints on how to implement this?</p>

username: soulitzer
cooked: <p>Hi,</p>
<p>If you are trying to compute the second order partial derivative, maybe something like:</p>
<pre><code class="lang-auto">import torch

x = torch.tensor(2., requires_grad=True)
y = torch.tensor(3., requires_grad=True)

def f(a, b):
  return a**2 * b

def df_da(a, b):
  return torch.autograd.grad(f(a, b), (a,), create_graph=True)[0]

print(torch.autograd.grad(df_da(x, y), (y,))) # d^2f/dydx = 4
</code></pre>

username: phantom90
cooked: <p>Thank you so much! Just one more question:</p>
<p>What if we want to use <code> loss.backward()</code> and <code>optimizer.step()</code> to update the parameter x? How to rewrite the expression: <code>torch.autograd.grad(df_da(x, y), (y,))</code>?</p>
<p>Formally, we want to do:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/4/64b83716fc427056da0ec79597bb976df838270c.png" alt="image" data-base62-sha1="en0qqCbj8i3XVnamrxlwiL0alic" width="127" height="45"></p>
<p>I know the question is a bit blur. If you need any clarification, please let me know.</p>
<p>Thanks again!</p>

username: soulitzer
cooked: <p>No problem!</p>
<p>The direct analog is to just have <code>torch.autograd.backward(df_da(x, y), inputs=(y,))</code>. Not specifying the inputs parameter should also be OK , though that would lead to the gradients also be accumulated for the other input (<code>x</code>), which is a bit of extra computation. Then assuming you have your optimizer configured to update <code>y</code>, <code>optimizer.step()</code> should just work, i.e., it does something like <code>y &lt;- y + lr * y.grad</code></p>
Model parameters to tensor and back to model

username: avish
cooked: <p>Hi, I have a model (nn.Module with multiple nested nn.Module as variable). I want to get all its parameter in a 1D vector and perform some operations on it, without changing length and put the result back into model as new parameters.</p>
<p>For getting parameter I am thinking of something like</p>
<pre><code class="lang-auto">                all_param = []
                for param in model.parameters():
                    all_param.append(param.view(-1))
                vec = torch.cat(all_param, dim=0)
                # do some operations on vec
                # ? put vec back into model.
</code></pre>
<p>I am looking into state dictionary to put back, but there can be nested module in model, so like parameters() just follows an ordered traversal over nesting, can I use it to update Module also or is there a shorter way to do so?</p>

username: pascal_notsawo
cooked: <p>Can you give an example of “some operation”? Because what you are trying to do can be very inefficient on large models, especially if you do it repeatedly.</p>
<p>But one way to do it as you ask is to keep the initial shapes of the parameters in the first loop, and their length after view(). Once your operations are finished, you separate the vector resulting from the concatenation according to the lengths stored before, then you give the initial shapes back to each parameter.</p>

username: avish
cooked: <p>Thank for reply.<br>
These are small toy models, use case is an experiment I am trying to do to generate models.<br>
Do you know, if I have weights of model in vector with correct shape how can I initialise a model with those weights.</p>

username: soulitzer
cooked: <p>You can use <code>copy_</code> under <code>no_grad</code> mode -</p>
<pre><code class="lang-python">model = torch.nn.Linear(10, 10)

def f(t):
  return t * t

params = [f(p) for p in model.parameters()]

with torch.no_grad():
  params = [p.copy_(q) for (p, q) in zip(model.parameters(), params)]
</code></pre>
Softmax only return close-to-zero values

username: Basile
cooked: <p>Hello,<br>
I was training a modified GAN (One Class Adversarial Network) for 2 class classification, and I encountered a weird bug :<br>
As I wanted to get probs from logits using softmax (I tried both using nn.functional.softmax and nn.Softmax), the output was just close to zero (and this was NOT caused by a wrong dim along which it was computed). The logits ranged from -20 to 20, which at first I thought could be a problem, but even for the small differences like -2 / 2 it output like 10^-5 and 10^-8. I then used a simple sigmoid on the first coordinate of the logits and it was all fine, but I wanted to discuss this issue and maybe why a robust function like softmax could have this problem. By the way, softmax was well executed at first (for a couple of hundred epochs), but then it suddenly output something wrong.</p>

username: KFrank
cooked: <p>Hi Basile!</p>
<aside class="quote no-group" data-username="Basile" data-post="1" data-topic="123169" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/basile/48/39038_2.png" class="avatar"> Basile:</div>
<blockquote>
<p>The logits ranged from -20 to 20, which at first I thought could be a problem, but even for the small differences like -2 / 2 it output like 10^-5 and 10^-8.</p>
</blockquote>
</aside>
<p>As you’ve described things, this can’t happen (unless I give a perverse<br>
interpretation to your description).</p>
<p>Could you capture the issue numerically, and post a complete, runnable<br>
script that reproduces the issue?  Please also let us know what version<br>
of pytorch you are running.</p>
<p>Best.</p>
<p>K. Frank</p>
Assigning values to Tensor without loops

username: KY_Chee
cooked: <p>There must be a better way to implement this function, but I couldn’t find any similar solution. Can somebody help with this? x is of shape (500,6,4).</p>
<p>def myFcn(x):</p>
<pre><code>cos_x2 = torch.cos(x[:,2])
sin_x2 = torch.sin(x[:,2])
cos_x5 = torch.cos(x[:,5])
sin_x5 = torch.sin(x[:,5])
out = torch.ones([x.shape[0], 6, 4])

for i in range(x.shape[0]):
  out[i,:,:] = torch.Tensor([
    [cos_x2[i], 0.0, 0.0, 0.0],
    [sin_x2[i], 0.0, 0.0, 0.0],
    [0.0, 1.0, 0.0, 0.0],
    [0.0, 0.0, cos_x5[i], 0.0],
    [0.0, 0.0, sin_x5[i], 0.0],
    [0.0, 0.0, 0.0, 1.0]])

return out</code></pre>

username: eqy
cooked: <p>This seems to be a faster implementation:</p>
<pre><code class="lang-auto">def myFcn(x):
    cos_x2 = torch.cos(x[:,2])
    sin_x2 = torch.sin(x[:,2])
    cos_x5 = torch.cos(x[:,5])
    sin_x5 = torch.sin(x[:,5])
    out = torch.ones([x.shape[0], 6, 4])

    for i in range(x.shape[0]):
      out[i,:,:] = torch.Tensor([
        [cos_x2[i], 0.0, 0.0, 0.0],
        [sin_x2[i], 0.0, 0.0, 0.0],
        [0.0, 1.0, 0.0, 0.0],
        [0.0, 0.0, cos_x5[i], 0.0],
        [0.0, 0.0, sin_x5[i], 0.0],
        [0.0, 0.0, 0.0, 1.0]])
    return out

def myFcn2(x):
    cos_x2 = torch.cos(x[:,2])
    sin_x2 = torch.sin(x[:,2])
    cos_x5 = torch.cos(x[:,5])
    sin_x5 = torch.sin(x[:,5])
    out = torch.zeros([x.shape[0], 6, 4])
    out[:, 0, 0] = cos_x2
    out[:, 1, 0] = sin_x2
    out[:, 3, 2] = cos_x5
    out[:, 4, 2] = sin_x5
    out[:, 2, 1] = 1.0
    out[:, 5, 3] = 1.0
    return out

x = torch.randn(512, 6)
t1 = time.time()
out1 = myFcn(x)
t2 = time.time()
out2 = myFcn2(x)
t3 = time.time()
print(torch.allclose(out1, out2))
print(t2-t1, t3-t2)
</code></pre>
<pre><code class="lang-auto">True
0.007179975509643555 0.00020051002502441406
</code></pre>
How to train with batches?

username: Navpreet_Singh
cooked: <p>I am experimenting with training word embeddings so here’s how it is going on. I have input nxn matrix which is n words of n lengths. So through my model, I get an output of nx2 where 2 is the dimension of each word embedding. My training runs perfectly fine with nll loss. So now if I want to do say training of 100 such nxn inputs, how do I implement it in Pytorch? I’ve read about custom dataset and dataloader, however my doubt is regarding the batches. Do I have to change the shape of my tensors to include batch size across all my implementation? Or do I just give 100xnxn as input to the dataset? What I want is that my model gets an input of n*n from the dataloader. How is that possible?</p>

username: ptrblck
cooked: <p>I’m unsure if the <code>n</code> dimension would already refer to the batch dimension or if it would be a seperate dimension (e.g. a temporal dimension).<br>
Since PyTorch models expect inputs already containing a batch dimension, I would guess that the first dimension is already treated as the batch dim.<br>
Depending on the used layers, you could use a <code>DataLoader</code>, specify a <code>batch_size</code>, and pass the data with the additional batch dimension to the model and check, if shape mismatch errors would be raised.</p>

username: Navpreet_Singh
cooked: <p>Thanks for your response. n is a separate dimension.<br>
I have created a custom model and I’m not using any pytorch’s defined model. I haven’t implemented batch size in my custom model. So my guess I would have to add it there, right?</p>
KL Divergence implementation

username: Arthur_Zakirov
cooked: <p>Hello everyone,</p>
<p>why do these two implementations of the KL divergence give me different results, can anybody find the error? The difference is about 5%</p>
<p>note: I’ve commented out the ’ .sum() ’ because if I don’t, the code colours in the forum text editor change, can anybody suggest the reason?</p>
<p>Version 1:</p>
<pre><code class="lang-auto">    def q_z_dist(self, mu, logvar):
        var = torch.exp(logvar)
        std = torch.sqrt(var)
        cov = torch.diag_embed(var)
        return td.MultivariateNormal(mu, cov)
    
    def p_z_dist(self, mu, logvar):
        mu_prior = torch.zeros_like(mu)
        var_prior = torch.ones_like(logvar)
        cov_prior = torch.diag_embed(var_prior)
        return td.MultivariateNormal(mu_prior, cov_prior)
    
    def KL_divergence(self, mu, logvar):
        p_dist = self.p_z_dist(mu, logvar)
        q_dist = self.q_z_dist(mu, logvar)

        KL = td.kl_divergence(p_dist, q_dist)
        KL_batch = KL #.sum()
        return KL_batch
</code></pre>
<p>Version 2:</p>
<pre><code class="lang-auto">    def KL_divergence(self, mu, logvar):
        KL = -0.5 * (1 + logvar - mu**2 -  torch.exp(logvar)).sum(dim = 1)
        KL_batch = KL.sum()
        return KL_batch
</code></pre>

username: tom
cooked: <p>So you have prior = p = N(0,1) and q = N(mu, diag(var))<br>
It seems that you compute D_KL(p || q) in the first and the more common D_KL(q || p) in the second. KL Divergence is not symmetric, so these will differ.</p>
<p>Best regards</p>
<p>Thomas</p>
Running multiple networks in succession crashes unit testing

username: pytorcher
cooked: <p>I have a system that is utilizing a few different open source networks for various applications.  I’m just starting to write unit tests with nose.  First plan is to test everything with one large test but that is causing crashes.  If I comment things out all 3 of my networks will pass their individual tests.  But if all called in succession in the same script the first one will pass and then the second one with crash on loss.backward() with either a  “RuntimeError: cuDNN error: CUDNN_STATUS_INTERNAL_ERROR” or a Segmentation Fault on different runs.  Any thoughts on what might be causing this?  Should I just write 3 separate tests?</p>

username: eqy
cooked: <p>Can you keep an eye on the memory usage between the tests (e.g., via <code>nvidia-smi</code>)? It might be that something is kept in memory from each of the tests causing an OOM when all three are run in successsion.</p>

username: pytorcher
cooked: <p>That could be it… beforehand is<br>
1502MiB /  4043MiB<br>
and if I stop in the middle its<br>
2209MiB /  4043MiB.<br>
Is there a way to clear that up?  I do already have a call to torch.cuda.empty_cache()<br>
.  I also remember reading that with pytorch when GPU memory is freed nvidia-smi will still show it as used even though it can be accessed by future pytorch code.</p>

username: eqy
cooked: <p>Can you check that nothing from the tests inadvertently keeps tensors/variables around when they aren’t needed (e.g., returned values or global variables)? A litmus test for this is to use trivially small inputs for the tests so that it is expected that everything should fit in memory even if it is all kept.</p>

username: pytorcher
cooked: <p>Hmm, global variables definitely could be it… I know there are a couple global things defined, if any globals ever have a pointer to the network itself I guess that would mean the whole network is saved.  Maybe just having multiple tests is a better idea.  Thanks for the help!</p>
How to preserve gradient using external libraries?

username: theexile
cooked: <p>I have a GAN network that returns a predicted <code>torch.tensor</code>. To guide this network, I have a loss function which is a summation of binary cross entropy loss and Wasserstein distance. However, in order to calculate Wasserstein distance, I am using <code>scipy.stats.wasserstein_distance</code> function from SciPy library. As you might know, this function requires two NumPy arrays as input. So, to use this function, I am converting my predicted tensor and ground-truth tensor to NumPy arrays as follows</p>
<pre><code class="lang-auto">pred_np = pred_tensor.detach().cpu().clone().numpy().ravel()
target_np = target_tensor.detach().cpu().clone().numpy().ravel()

W_loss = wasserstein_distance(pred_np, target_np)
</code></pre>
<p>Then, total loss is obtained by adding <code>W_loss</code> to BCELoss. I am now showing this part because it is a bit unnecessary and not related to my question.</p>
<p>My concern is I am detaching gradient so I suppose that while optimizing and updating model parameters it will not consider <code>W_loss</code>. I am a bit newbie so I hope my question is clear and appreciate for answers in advance.</p>

username: pascal_notsawo
cooked: <p>Why not look for implementations of the wasserstein distance in pytorch instead?</p>
Getting a tutorial into pytorch tutorials

username: ariG23498
cooked: <p>Hey Folks,</p>
<p>I have submitted a PR for a tutorial on self-supervised learning with SimCLR in the PyTorch tutorial repository. It has been dormant for a long time now. I just wanted criticism or advice on being able to contribute to the official tutorials.</p>
<p>The PR: <a href="https://github.com/pytorch/tutorials/pull/1503" class="inline-onebox" rel="noopener nofollow ugc">Adding Self-Supervised learning with SimCLR by ariG23498 · Pull Request #1503 · pytorch/tutorials · GitHub</a></p>
<blockquote>
<p>I am sorry if this is not the right channel to post my concern, but the community helps the best hence thought of sharing it here. Any help would be great <img src="https://discuss.pytorch.org/images/emoji/apple/hugs.png?v=12" title=":hugs:" class="emoji" alt=":hugs:" loading="lazy" width="20" height="20"></p>
</blockquote>
Different result from different saving image method

username: Florentino
cooked: <p>When I save the same value torch tensor or ndarray, save_image() from torchvision and save() from matplotlib respectively give me different image visualization, I know they both do some normalization method but can’t tell the detail algorithms they use.<br>
Does anyone know which causes the difference? Thanks</p>

username: ptrblck
cooked: <p>You could check both implementations and compare the used paths in them.<br>
<a href="https://github.com/matplotlib/matplotlib/blob/a3fbf1876830a89087842ea779515e0d099c885d/lib/matplotlib/image.py#L1548-L1657">matplotlib.pyplot.imsave</a> seems to deploy to different methods based on the specified format etc. while the <a href="https://github.com/pytorch/vision/blob/31d53367671d2deed42318a488a8d66158d7fbbe/torchvision/utils.py#L113-L136">torchvision.utils.save_image</a> method normalizes the image to <code>[0, 255]</code>.</p>

username: Florentino
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>  Thanks for answering, however I am wondering which function should I choose in what situation each?<br>
I saw these two function are used frequently in deep learning, but there must be different in output looking.</p>

username: ptrblck
cooked: <p>I think it depends on the tensor/array you are passing to these methods and what its values would mean.<br>
A normalization is often applied, if you provide data in a “non-image” format.<br>
E.g. floating point values in the range <code>[-1, 1]</code> wouldn’t directly represent pixel data (color values) and would thus be mapped to <code>uint8</code> data in the range <code>[0, 255]</code>, which is a standard image format.<br>
The best approach would be for you to create this mapping and pass the array in an expected format, so that neither <code>torchvision</code> nor <code>matplotlib</code> would change any values, as they are most likely trying to use their best guess on what the array might represent.</p>
How to multiply by a scalar with quantization observers

username: Alexander_Soare
cooked: <p>This simple script will illustrate what I’m trying to do</p>
<pre><code class="lang-auto">import torch
from torch import nn
from torch.quantization.quantize_fx import prepare_fx

class Model(nn.Module):
    def __init__(self):
        super().__init__()

    def forward(self, x):
        B = x.shape[0]
        B = B * 1
        return x

model = Model()
model.eval()

qconfig = torch.quantization.get_default_qconfig("fbgemm")
model = prepare_fx(model, {"": qconfig})

x = torch.arange(4)

print(model(x))
</code></pre>
<p>Right now it raises <code>AttributeError: 'int' object has no attribute 'numel'</code> because of the <code>B * 1</code> line.</p>
<p>By the way, the real example for what I’m trying to quantize is <a href="https://github.com/rwightman/pytorch-image-models/blob/d7bab8a6c52a72487d1bed0a28aad41e326d7622/timm/models/tnt.py#L145" rel="noopener nofollow ugc">here</a>.</p>

username: supriyar
cooked: <p>Hi <a class="mention" href="/u/alexander_soare">@Alexander_Soare</a>, can you try this code out with the latest pytorch nightly build? I tried to reproduce the issue but wasn’t able to</p>
<pre><code class="lang-auto">      class Model(nn.Module):
            def __init__(self):
                super().__init__()

            def forward(self, x):
                B = x.shape[0]
                B = B * 1
                return x

        model = Model()
        model.eval()

        qconfig = torch.quantization.get_default_qconfig("fbgemm")
        model = prepare_fx(model, {"": qconfig})

        x = torch.arange(4)

        print(model(x))
        print(model)
</code></pre>
<p>Outputs</p>
<pre><code class="lang-auto">tensor([0, 1, 2, 3])
GraphModule()



def forward(self, x):
    getattr_1 = x.shape
    getitem = getattr_1[0];  getattr_1 = None
    mul = getitem * 1;  getitem = None
    return x
</code></pre>

username: Alexander_Soare
cooked: <p><a class="mention" href="/u/supriyar">@supriyar</a> you’re right, nightly works. It’s 1.8.1 that doesn’t</p>