akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
OMP parallelism over pool of modules

fatvlad: <p>Hi!</p>
<p>I am trying to train bunch of small networks concurrently, which includes calls to <code>torch::jit::Module::forward</code> and subsequent call to <code>torch::autograd::grad</code> on results afterwards. To be more precise, here’s what I am doing:</p>
<ol>
<li>For each worker thread I create a copy of original module by calling <code>Module::clone</code>.</li>
<li>Within each worker thread I only use corresponding copy of a module to calculate value and gradient using thread-specific inputs.</li>
<li>After value and gradient are calculated, they are used to update state of the model stored separately.</li>
</ol>
<p>So each OMP worker has its own copy of a module which worker uses to process the payload.<br>
What I noticed is that with the increase of number of workers (amount of payload for each worker remains the same) performance becomes worse, approximately like that:</p>
<pre><code class="lang-auto">1 worker(s) - 11s
2 worker(s) - 16s
3 worker(s) - 20.5s
4 worker(s) - 25s
</code></pre>
<p>I removed the <code>autograd</code> part in the loop body and it seems that performance issue still persists with approximately same law, so it is not related to some synchronization within gradient calculation, although might be related to graph construction:</p>
<pre><code class="lang-auto">1 worker(s) - 3.56s
2 worker(s) - 4.61s
3 worker(s) - 5.91s
4 worker(s) - 7.58s
</code></pre>
<p>To avoid potential problems with conflicting parallelization libraries, before starting this test I’ve put <code>torch.set_num_threads(1)</code>. The whole test is done in Python, function with omp loop is loaded from pybind extension module.<br>
What bothers me most is that launching these tasks in separate processes does not have such dramatic performance drop. However it is much easier to work with threads.</p>
<p>Why is this happening and what are the workarounds?</p>
Initialize LSTM with sentence embedding

BramVanroy: <p>For a very specific task, I want to try out something that us basically an encoder-decoder architecture using LSTM without attention, but where we do not have an encoder. Instead, we get a sentence embedding of the input. Considering that in a full encoder-decoder architecture, we also just pass a single representation to the encoder (rather than all tokens as in transformer models), it seems that this should be possible.</p>
<p>The problem that I am having is that I am not sure <em>whether the sentence embedding should be passed as the hidden state or as the cell state to the decoder LSTM</em>, and how the other one should be initialized in such scenario. Fundamentally, it is not clear to me (even after reading tons of documentation) what the difference in <em>meaning</em> is between the cell state and hidden state. I know the difference in code, but <em>what do those differences represent?</em></p>
<p>Thanks in advance</p>

vdw: <p>Your right, since the output of the encoder is just a sentence embedding, you can use an existing encoding sentence embedding and use only the decoder part.</p>
<p>A couple of days ago, there was <a href="https://discuss.pytorch.org/t/combining-cnn-lstm-research-paper-implementation/94929">this post</a> about the implementation in a paper, where the embedding came from CNN and was fed into an LSTM decoder. In this paper – as far as I could understand – the authors set the intial hidden state <code>h_0</code> and zero-ed the initial cell state <code>c_0</code>. I would simply try this approach, as well as:</p>
<ul>
<li>use the embedding to set <code>h_0</code> and <code>c_0</code>
</li>
<li>use a <code>nn.GRU</code> which doesn’t have cell state <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:">
</li>
</ul>
<p>and just see how this effects the results.</p>

BramVanroy: <p>Thanks, very useful! I completely forget about GRU only having a hidden state, so I am going to try that for now - it must be a bit easier than fiddling with the LSTM in this case.</p>
Training on validation data before test set

bjr96571: <p>I’ve been training my model using a training/validation split, and I have something that I’m happy with. Before evaluating the model on the test set, though, I’m not sure if I should add in the validation data to the model’s training.</p>
<p>If I were to retrain the model including the validation data, I’m not sure how I would monitor overfitting either. That is, I normally monitor the loss and accuracy of the validation data, and if there was no longer any validation to be monitored, I don’t know when I would stop training.</p>
<p>Does anyone know what best practice here is?</p>

ptrblck: <aside class="quote no-group" data-username="bjr96571" data-post="1" data-topic="95475">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/e274bd/48.png" class="avatar"> bjr96571:</div>
<blockquote>
<p>If I were to retrain the model including the validation data, I’m not sure how I would monitor overfitting either. That is, I normally monitor the loss and accuracy of the validation data, and if there was no longer any validation to be monitored, I don’t know when I would stop training.</p>
</blockquote>
</aside>
<p>That is a valid concern and thus I wouldn’t train the model on the validation data.<br>
If you retrain it and use e.g. the test dataset to stop the training, you are leaking data and the test set was thus used during the training and is invalid.</p>
Cannot use MKL-DNN with PyTorch

tmannen: <p>Hi, I’m trying to use MKL-DNN backend with PyTorch, however I am unable to. I installed PyTorch with pip, and the version is 1.4.0. I try to move it to device “mkldnn”:</p>
<pre><code class="lang-auto">import torch
import torchvision
model = torchvision.models.resnet50(pretrained=True)
model.to("mkldnn")
</code></pre>
<p>But this gives an error:</p>
<pre><code class="lang-auto">RuntimeError: PyTorch is not linked with support for mkldnn devices
</code></pre>
<p>However I checked and PyTorch was linked with MKL-DNN. Torch config says:</p>
<pre><code class="lang-auto">PyTorch built with:
  - GCC 7.3
  - Intel(R) Math Kernel Library Version 2019.0.4 Product Build 20190411 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v0.21.1 (Git Hash 7d2fd500bc78936d1d648ca713b901012f470dbc)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - NNPACK is enabled
  - CUDA Runtime 10.1
  - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_37,code=compute_37
  - CuDNN 7.6.3
  - Magma 2.5.1
  - Build settings: BLAS=MKL, BUILD_NAMEDTENSOR=OFF, BUILD_TYPE=Release, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -fopenmp -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -O2 -fPIC -Wno-narrowing -Wall -Wextra -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Wno-stringop-overflow, DISABLE_NUMA=1, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, USE_CUDA=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_STATIC_DISPATCH=OFF,
</code></pre>
<p>So MKLDNN should be available. What could be wrong?</p>

albanD: <p>cc <a class="mention" href="/u/seemethere">@seemethere</a> do you know why this could be happening?</p>

ckmufeng: <p>Try like this:</p>
<pre><code class="lang-auto">from torch.utils import mkldnn as mkldnn_utils
# change input to be mkldnn layout
input_ = input.to_mkldnn()
# change weight to be mkldnn layout
model_ = mkldnn_utils.to_mkldnn(model)
# NB: output_ is mkldnn layout
output_ = model_(input_)
# change output_ back to default layout
output = output_.to_dense()
</code></pre>
DataParallel for Mixed GPUs

Marat: <p>Could pytorch effectively use setup in server with different GPUs, for example 2x 2080 Ti + 2x 3080 Ti for the one distributed learning process? Or some drawbacks may occur?</p>

ptrblck: <p>We had similar posts in this forum, which you could search for, and the main drawback would be that the slower GPUs would potentially create the bottleneck in your setup.</p>
AttributeError: module 'torch' has no attribute 'unique_dim'

hinson: <p>I have installed Pytorch, but have been geting the above error. I installed it but still did not work. Please, how can I solve this problem? Thank you.</p>
<p>some of my version information：<br>
python 3.7.8<br>
numpy 1.19.1<br>
torch       1.6.0+cpu<br>
torchvision 0.7.0+cpu</p>

ptrblck: <p><code>torch.unique_dim</code> doesn’t exist. Are you looking for <a href="https://pytorch.org/docs/stable/generated/torch.unique.html#torch.unique"><code>torch.unique</code></a> and would like to pass the <code>dim</code> argument to it?</p>

hinson: <p>Although when I use <code>from torch import *</code> , there shows this error , I can import part of torch.<br>
Thank U for reply.</p>

ptrblck: <p>You are right. I can reproduce it with a recent nightly binary. Since I’m not using star imports I have never seen it before.<br>
I’m unsure, why this error is raised, but could you create an issue on GitHub, so that we could track it?</p>
Could someone please elaborate on this sentence in description of nn.BCELoss()?

aynmeme: <p>So as part of the description of nn.BCELoss() on <a href="https://pytorch.org/docs/stable/generated/torch.nn.BCELoss.html" rel="nofollow noopener">https://pytorch.org/docs/stable/generated/torch.nn.BCELoss.html</a>  it is written :</p>
<p>“For one, if either yn=0y_n = 0yn​=0 or (1−yn)=0(1 - y_n) = 0(1−yn​)=0 , then we would be multipying 0 with infinity. Secondly, if we have an infinite loss value, then we would also have an infinite term in our gradient, since lim⁡x→0ddxlog⁡(x)=∞\lim_{x\to 0} \frac{d}{dx} \log (x) = \inftylimx→0​dxd​log(x)=∞ . This would make BCELoss’s backward method nonlinear with respect to xnx_nxn​ , and using it for things like linear regression would not be straight-forward.”</p>
<p>my question is on the last sentence; how does a loss term and its gradient going to infinity translate into to a nonlinear backward method and its incompatibility with linear regression?</p>

KFrank: <p>Hi Meme!</p>
<aside class="quote no-group quote-modified" data-username="aynmeme" data-post="1" data-topic="95652" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/aynmeme/48/16474_2.png" class="avatar"> aynmeme:</div>
<blockquote>
<p>This would make BCELoss’s backward method nonlinear with respect to xnx_nxn​ , and using it for things like linear regression would not be straight-forward."</p>
<p>my question is on the last sentence; how does a loss term and its gradient going to infinity translate into to a nonlinear backward method and its incompatibility with linear regression?</p>
</blockquote>
</aside>
<p>I’ve noticed that sentence too.  I think it’s pretty much nonsense, where<br>
either the author was just sloppy or didn’t understand what the words<br>
he was using meant.</p>
<p>The best I can come up with is that he wanted to say something like:</p>
<p>“When <code>BCELoss</code> becomes <code>inf</code>, its <code>backward()</code> method – and the<br>
whole backpropagation – will become polluted with <code>inf</code>s and <code>nan</code>s,<br>
causing training to fail.  We therefore clamp <code>BCELoss</code>’s internal <code>log()</code><br>
function at <code>-100</code> to save you from such an ignoble fate.”</p>
<p>Best.</p>
<p>K. Frank</p>

aynmeme: <p>Hi K. Frank,</p>
<p>Yes exactly. your translation of the author’s words do make all the sense and it’s also what i would have expected to be written there. Those words about nonlinearity of backprop just made me feel there’s whole other dimension of backward loss  propagation that i have no clue about. Relieved to know I’m not the only one.</p>
<p>Cheers<br>
M.</p>
Aggregate profiler output by module class?

leopd: <p>I’m trying to use the profiler, and I’m having trouble interpreting the output.  Is it possible to aggregate the profiling data by the python class(es) which invoked the operation?  I’d <em>love</em> to be able to see how much time/memory is going into MultiHeadAttention instead of just learning that “matmul” is the biggest hog.</p>
<p>If the profiling captures the call stack when operations are invoked I’d guess this wouldn’t be too hard.  If it’s not capturing the stack, I’d certainly be willing to pay whatever performance penalty is needed while running the profile.</p>

ptrblck: <p>If you are profiling your script on the GPU, you could use <a href="https://pytorch.org/docs/stable/cuda.html#torch.cuda.nvtx.range_push"><code>nvtx</code> ranges</a> and tag the multi head attention operation with it.<br>
I don’t know, if the built-in profiler provides options to print layer-specific outputs, but you could also take a look at <a href="https://github.com/NVIDIA/PyProf">pyprof</a>.</p>

leopd: <p>Thanks for the tip.  I’m trying to understand it, as the docs are pretty thin.  Would I do something like this:</p>
<pre><code class="lang-auto">    torch.cuda.nvtx.range_push("multihead_attention")
    out = self.attention(x)
    torch.cuda.nvtx.range_pop("multihead_attention")
</code></pre>
<p>And then hopefully this shows up in the profiler output?</p>

ptrblck: <p>You don’t have to specify the range name in <code>range_pop</code>, but besides that it should work.</p>
<p>My workflow would be:</p>
<pre><code class="lang-python"># Setup
...

# enable the profiling, such that nvprof or nsys doesn't capture operations before this point
torch.cuda.cudart().cudaProfilerStart()

torch.cuda.nvtx.range_push("forward")
output = model(input)
torch.cuda.nvtx.range_pop()

# don't profile this
optimizer.zero_grad()
loss = criterion(output, target)

torch.cuda.nvtx.range_push("backward")
loss.backward()
torch.cuda.nvtx.range_pop()

torch.cuda.nvtx.range_push("optimizer.step()")
optimizer.step()
torch.cuda.nvtx.range_pop()

torch.cuda.cudart().cudaProfilerStop()
</code></pre>
<p>You can of course use nested ranges, if needed.</p>
<p>Then run the script via:</p>
<pre><code class="lang-auto"># with nsys
nsys profile -w true -t cuda,nvtx,osrt,cudnn,cublas -s cpu -o my_report --capture-range=cudaProfilerApi --stop-on-range-end=true --cudabacktrace=true --cudabacktrace-threshold=10000 --osrt-threshold=10000 -x true python script.py

# or with nvprof
# single-process
nvprof --profile-from-start off -fo %p.nvprof python script.py

# multi-process
nvprof --profile-child-processes --profile-from-start off -fo %p.nvprof python -m torch.distributed.launch  --nproc_per_node=2 script.py
</code></pre>
<p>Note that my <code>nsys</code> cmd creates stack traces etc., so you might want to slim it down a bit.</p>
Pytorch dataloader reading order

Peter1: <p>HI,</p>
<p>I wonder the dataloader.</p>
<h2>Dataset Folder</h2>
<p>|<br>
|___class1<br>
|                |____File1<br>
|<br>
|___class2<br>
|               |_____File2<br>
|               |_____File3<br>
|</p>
<p><code>for inputs, labels in tqdm(test_dataloader):</code></p>
<p>The Files(1~3) will be loaded into inputs, labels.</p>
<p>How about the order?</p>
<p>The Files will be loaded, first order:File1, and second order: File2, and Third order: File3, … ?</p>
<p>So, can I map the loaded data to real file on memery by order?<br>
For example, This loaded data is from File3 becuase of third iteration</p>
<p>or any idea?</p>
YOLO algorithm - how it works?

Darek554: <p>Hi.<br>
I’m trying to learn about this algorithm, but any tutorial that i’ve read is like “YOLO is dividing  image on SxS grid, where each cell is responsible for detecting an object…” it explains nothing to me.<br>
How this algorithm is predicting boundry boxes which are larger than one grid cell?<br>
Does it concatenate somehow those grid cells that are adjacent and predicting same class object?</p>
<p>I’ll be thankfull for good learning resources <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

l00p: <p>Have you had a look at the papers?<br>
<a href="https://pjreddie.com/media/files/papers/YOLOv3.pdf" rel="nofollow noopener">Yolov3</a><br>
<a href="https://arxiv.org/pdf/2004.10934.pdf" rel="nofollow noopener">Yolov4</a></p>

Darek554: <p>I’ll try to read this, but i have no experience in reading papers so it may be impossible to understand this <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>

pfloat: <p>I have read a tutorial <a href="https://blog.paperspace.com/how-to-implement-a-yolo-object-detector-in-pytorch/" rel="nofollow noopener">[here fr example]</a> to implement yolov3 from scratch, it even has a working code</p>
<p>In complement of reading the original papers, the idea is just to search the web for a good tutorial with working code explained step by step. There must be plenty of them available online now.</p>
Conv2d(): argument 'input' (position 1) must be Tensor, not dict

Pracheta_Sahoo: <pre><code class="lang-auto">type or paste code here
model = MyModel()
    model_path='/home/pracheta2/semi-supervised/model_best_covid_original_500.pth.tar'
    model.load_state_dict(torch.load(model_path)['state_dict'])
    # load input images and prepare data
    datum_l_cc = utils.load_images(parameters['image_path'], 'L-CC')
    datum_r_cc = utils.load_images(parameters['image_path'], 'R-CC')
    datum_l_mlo = utils.load_images(parameters['image_path'], 'L-MLO')
    datum_r_mlo = utils.load_images(parameters['image_path'], 'R-MLO')
    x = {
        "L-CC": torch.Tensor(datum_l_cc).permute(0, 3, 1, 2).to(device),
        "L-MLO": torch.Tensor(datum_l_mlo).permute(0, 3, 1, 2).to(device),
        "R-CC": torch.Tensor(datum_r_cc).permute(0, 3, 1, 2).to(device),
        "R-MLO": torch.Tensor(datum_r_mlo).permute(0, 3, 1, 2).to(device),
    }
    transform = transforms.Compose([transforms.Resize(32),
                                      transforms.ToTensor(),
                                      transforms.Normalize(
                                          (.485, .456, .406), (.229, .224, .225))
                                      ])
    # run prediction:
    
    features, prediction_birads = model(x)
    print(prediction_birads)
</code></pre>
<p>Traceback (most recent call last):<br>
File “birads.py”, line 103, in <br>
inference(parameters_)<br>
File “birads.py”, line 80, in inference<br>
features, prediction_birads = model(x)<br>
File “/home/pracheta2/anaconda3/envs/oldpy/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 493, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “birads.py”, line 20, in forward<br>
x_feature = self.model_resnet(x)<br>
File “/home/pracheta2/anaconda3/envs/oldpy/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 493, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/home/pracheta2/anaconda3/envs/oldpy/lib/python3.6/site-packages/torchvision/models/resnet.py”, line 192, in forward<br>
x = self.conv1(x)<br>
File “/home/pracheta2/anaconda3/envs/oldpy/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 493, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/home/pracheta2/anaconda3/envs/oldpy/lib/python3.6/site-packages/torch/nn/modules/conv.py”, line 338, in forward<br>
self.padding, self.dilation, self.groups)</p>

Pranavan_Theivendira: <aside class="quote no-group" data-username="Pracheta_Sahoo" data-post="1" data-topic="95739">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pracheta_sahoo/48/22745_2.png" class="avatar"> Pracheta_Sahoo:</div>
<blockquote>
<p><code>prediction_birads</code></p>
</blockquote>
</aside>
<p>Hi,</p>
<p>The problem here is that the input <code>x</code> is a dictionary. It should be a tensor.</p>
<p>Thanks</p>
Torch.jit.trace works, but torch.jit.script does not

Vlad_A: <p>Is it expected that for some module torch.jit.trace works well, while torch.jit.script fails. In particular, I get an error, related to TorchScript treating all objects as Tensors by default, i.e.</p>
<pre><code class="lang-auto">RuntimeError: 
Tensor cannot be used as a tuple:
</code></pre>

SplitInfinity: <p>Yes, this is expected.</p>
<p><code>torch.jit.trace</code> constructs a JIT IR graph by observing the operations that are performed on <code>Tensors</code> by the <code>Module</code> being traced with little concern for the Python language constructs that are used to express these operations. This means that it can produce JIT graphs successfully for a wide variety of programs, but those graphs may not always be faithful representations of the corresponding programs. For example, tracing is unable to capture control flow because it observes tensor operations and runtime, and that means the graph it constructs will contain one branch of a given instance of if statement but not the other.</p>
<p><code>torch.jit.script</code> produces a JIT graph from a program by compiling it. However, it places a number of restrictions on the programs it can compile, such as requiring that they be statically typed (through heavy use of type annotations) and use only a specified subset of Python language features.</p>
<p>To answer your question, <code>torch.jit.script</code> assumes any argument without an annotated type is a <code>Tensor</code>. To be able to script your <code>Module</code>, you will need to add type annotations to all methods and attributes and to make sure you are not using any unsupported Python language features.</p>
Convert string to int/float in torch script

Dieblitzen: <p>Currently, I cannot convert simple python strings like “0” or “1” to an integer or float in torchscript. Trying <code>int('0')</code> will result in the error:</p>
<pre><code class="lang-auto">RuntimeError:
Arguments for call are not valid.
</code></pre>
<p>And it expects only a tensor, a bool, a float or a number to be converted to int.</p>
<p>With float, it only allows <code>"-inf"</code> or <code>"inf"</code> to be converted.</p>
<p>I tried encoding with bytes, but that isn’t allowed either. Do you have any suggestions for a JIT/Torchscript compatible way to convert strings to ints/floats?</p>

SplitInfinity: <p>Which version of <code>torch</code> are you using? I just tried this with <code>torch-1.6</code> (the latest release) and it works:<br>
<em>Code</em></p>
<pre><code class="lang-auto">import torch

def fn() -&gt; int:
    a = int('0')
    return a

s = torch.jit.script(fn)
print(s())
</code></pre>
<p><em>Output</em></p>
<pre><code class="lang-auto">0
</code></pre>
How to ensure the repeatability of the results of the Pytorch model across devices？

dirk: <p>I am a little bit confused about the randomness of the pytorch model.</p>
<p>I used the following code to fix the random seed so that the training results of the model can be repeated on the same device：</p>
<pre><code class="lang-auto">def seed_torch(seed=2020):
    random.seed(seed)
    os.environ["PYTHONHASHSEED"] = str(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)
    torch.cuda.manual_seed(seed)
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False
</code></pre>
<p>But what is confusing is that when the code is run on another device with the same hardware configuration and software environment, it will produce different results.</p>
<p>I reinstalled the virtual environment to ensure that the versions of the libraries are consistent, but this problem still puzzles me.</p>
<p>Feel free to ask if more code is needed to explain the problem.</p>

briankosw: <p>Refer to <a href="https://discuss.pytorch.org/t/reproducibility-over-different-machines/63047/2" class="inline-onebox">Reproducibility over Different Machines</a> for the same discussion!</p>
Draw lines on image and display in jupyterlab

l00p: <p>I’m converting a pytorch tensor representation of an image and series of points to numpy so that I can draw lines between the points and display the image in jupyter lab (using matplotlib)</p>
<p>If I comment out the <code>cv2.polylines</code> line, this code works as expected, showing me the image.</p>
<pre><code class="lang-auto">    # convert img tensor to np
    img = img / 2 + 0.5
    img = img.detach().cpu().numpy().astype(np.float32)
    img = np.transpose(img, (1, 2, 0))
    print(type(img))
    # prints:
    # &lt;class 'numpy.ndarray'&gt;
    print(img.shape)
    # prints:
    # (768, 256, 3)

    
    # convert label tensor to numpy ndarray
    pts = lbl.detach().cpu().numpy().reshape(-1, 1, 2)
    pts = np.rint(pts).astype(np.int32)
    print([pts])
    # prints:
    # [array([[[ 17, 153]],
    #        [[153, 154]],
    #        [[159, 692]],
    #        [[ 14, 691]]], dtype=int32)]
    
    # draw lines between the vertices in lbl
    cv2.polylines(img, [pts], True, (0, 1, 1))

    # show the image with matplotlib.pyplot
    plt.imshow(img)
    plt.show()
</code></pre>
<p>However polylines gives an error:</p>
<pre><code class="lang-auto">---&gt; 36         cv2.polylines(img, [pts], True, (0,255,255))
     37         plt.imshow(img)
     38         plt.show()

TypeError: Expected Ptr&lt;cv::UMat&gt; for argument 'img'
</code></pre>
<p>How can I draw lines on this image and then display it in jupyterlab?</p>
<p>python 3.7, opencv 4.4 (same behaviour in 4.2, 3.4)</p>
<p>Additional:</p>
<p>If I create a new blank numpy ndarray of matching size and type I can actually draw on it and display it, but I can only display, but not draw on the one I get from a tensor.</p>
<pre><code class="lang-auto">        blank_img = np.zeros((768, 256, 3), np.float32)
        print(type(blank_img))
        print(blank_img.shape)
        print(blank_img[:5, :5, 0])
</code></pre>

l00p: <p>The solution was to copy the image</p>
<pre><code>img = img.copy()
</code></pre>
<p>it seems that Tensor.numpy() actually gives you an immutable view of the underlying data structure so you can display it but not modify it. To modify it I had to create my own copy</p>

ptrblck: <p>Good to know this was a workaround.<br>
However, I’m able to use the numpy array directly without triggering a copy:</p>
<pre><code class="lang-python">pts = np.array([[[ 17, 153]],
                [[153, 154]],
                [[159, 692]],
                [[ 14, 691]]], dtype=np.int32)

x = torch.randn(768, 256, 3)
img = x.numpy()
cv2.polylines(img, [pts], True, (0, 1, 1))
</code></pre>

l00p: <p>maybe mine is immutable because it’s a view from a dataloader?</p>

ptrblck: <p>That might be the case, so good to know the <code>clone()</code> operation would fix it in some use cases.</p>
How to handle the OOM exception when training with DDP and syncBatchNorm?

hegc: <p>When training with the DDP and syncBatchNorm, one process runing on one GPU, When I catch the gpu OOM exception, the training is blocked. What should we do?<br>
My code is following, when OOM exception occurs in one process, I just ignore this batch, the training phase continue.</p>
<pre><code class="lang-auto">for i, (inputs, targets) in enumerate(train_loader):
    try:
        # do forward and backprop
    except RuntimeError as e:
        if 'out of memory' in str(e):
            print('| WARNING: ran out of memory, skipping this batch.')
            if hasattr(torch.cuda, 'empty_cache'):
                torch.cuda.empty_cache()
            optimizer.zero_grad()
        else:
            raise e

</code></pre>
<p>when one process catch the exception, the others get blocked.</p>
LM by biLSTM(confusing output(not state) order)

Roy_Paik: <p>Let me ask a quick question:<br>
In my understanding, when concatenating biLSTM output, it should be:</p>
<pre><code class="lang-auto">* fh: forward hidden state
* bh: backward hidden state
&lt;char level&gt;
"BAER"
[fh_B; bh_REAB], [fh_BA; bh_REA], [fh_BAE; bh_RE], [fh_BAER; bh_R]
</code></pre>
<p>but pytorch’s return look like:</p>
<pre><code class="lang-auto">[fh_B; bh_R], [fh_BA; bh_RE], [fh_BAE; bh_REA], [fh_BAER; bh_REAB]
</code></pre>
<p>Am I misunderstanding BiLSTM?</p>

vdw: <p>I had a <a href="https://discuss.pytorch.org/t/rnn-output-vs-hidden-state-dont-match-up-my-misunderstanding/43280">similar question</a>; maybe that helps.</p>
Tensorboard graph input/output naming

xli4217: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/0/20cd12baddff141444bc019319dba3a5d377ec72.png" data-download-href="https://discuss.pytorch.org/uploads/default/20cd12baddff141444bc019319dba3a5d377ec72" title="Screenshot from 2020-09-09 14-35-18"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/0/20cd12baddff141444bc019319dba3a5d377ec72_2_457x500.png" alt="Screenshot from 2020-09-09 14-35-18" data-base62-sha1="4GaE4j0hDCVnxhEJsCtRJOy8Mq6" width="457" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/0/20cd12baddff141444bc019319dba3a5d377ec72_2_457x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/0/20cd12baddff141444bc019319dba3a5d377ec72_2_685x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/2/0/20cd12baddff141444bc019319dba3a5d377ec72.png 2x" data-dominant-color="ECECEC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2020-09-09 14-35-18</span><span class="informations">751×821 38.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Hi, I’m curious why some of the input/output are named while others are numbers. Is there a way to make them all named? Thanks.</p>
How to add annotation to work with JIT for nn.Sequential

voqtuyen: <pre><code class="lang-auto">class Model(mm.Module):
    def __init__(self):
        # Init

    def forward(self, x):
        # type: (List[Tensor]) -&gt; List[Tensor]
        # Code
</code></pre>
<p>If we need to define type of input x that if different from Tensor, we could do some thing like above. However, i am confused about how to add annotation if i use</p>
<pre><code class="lang-auto">class Model(mm.Module):
    def __init__(self):
        # Add sequential module
        self.layer = nn.Sequential([...])

    def forward(self, x):
        # type: (List[Tensor]) -&gt; List[Tensor]
        # Code
</code></pre>
<p>If i define an attribute self.layer of type nn.Sequential and this nn.Sequential requires other data type for example List[Tensor], how can i annotate this to work with torch.jit?</p>

SplitInfinity: <p>This is not possible at the moment. What you can do is extend or make your own version of <code>Sequential</code> with the type annotation you need on its <code>forward</code> method.</p>
Trained model is not giving the always the same result for the same input

SandPhoenix: <p>Hello guys,<br>
So I trained a model and I trained it but after wards during inference time I do not get always the same result but this is just for some images. I find this strange and I did not find any logical explanation for this.<br>
Any idea would be appreciated.<br>
Thank you in advance.<br>
Peace</p>

Luca_Pamparana: <p>Do you have any dropout layers enabled during inference? Make sure the model is in <code>eval()</code> mode.</p>

SandPhoenix: <p>Hello,<br>
thank you for your answer but I do not have any dropout layer.<br>
I cannot seem to find the reason that I am getting that.</p>

Pranavan_Theivendira: <p>Hi,</p>
<p>You may have batch normalization layers. If you have, <code>model.eval()</code> followed by a context block of <code>torch.no_grad()</code> is the proper way of doing the evaluation. Otherwise, new batch statistics will be used in the evaluation phase(/ test phase) which will produce different results.</p>
<p>Thanks</p>
How to replace layers in a pretrained fasterrcnn_resnet50_fpn

peggs: <p>I have developed a customized <code>SpecialConv2d</code> layer that is mostly the same as <code>nn.Conv2d</code>, but does a couple extra things.</p>
<p>I would like to replace all <code>nn.Conv2d</code> layers in a pretrained Faster-RCNN instance with <code>SpecialConv2d</code>, but there doesn’t seem to be a <code>.features</code> property that I can access. How would I go about doing that?</p>
<pre><code class="lang-python">import torchvision
import torch
import specialconv

frcnn = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)

# I'd like to iterate over everything in frcnn
# and replace all torch.nn.Conv2d layers
# with specialconv.SpecialConv2d layers
</code></pre>
<p>Thanks so much!</p>

Pranavan_Theivendira: <aside class="quote no-group" data-username="peggs" data-post="1" data-topic="95489">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/43a26b/48.png" class="avatar"> peggs:</div>
<blockquote>
<pre><code class="lang-auto"># with specialconv.SpecialConv2d layers
</code></pre>
</blockquote>
</aside>
<p>Hi,</p>
<p>A naive way of copying can be done as follows. Assume we have to copy <code>model1</code> layer <code>l1</code> to <code>model2</code> layer <code>l1</code>.</p>
<pre><code class="lang-auto">model2.l1.weight = model1.l1.weight
model2.l1.bias = model1.l1.bias
</code></pre>
<p><code>l1</code> is a layer name. Use your layer names.</p>
<p>There may be better ways of doing. Let’s wait for others to reply.</p>
<p>Thanks<br>
Regards<br>
Pranavan</p>

peggs: <p>This is just what I wanted.</p>
<p>An example would be</p>
<pre><code class="lang-python">import torchvision
import torch
import specialconv

frcnn = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)
prev_weight = frcnn.backbone.body.conv1.weight
prev_bias = frcnn.backbone.body.conv1.bias
new_layer = SpecialConv2d(...) # make sure args match
new_layer.weight = prev_weight
new_bias = prev_bias
frcnn.backbone.body.conv1 = new_layer
</code></pre>
Indexing torch::nn::ModuleList in LibTorch

mih.net: <p>Hi!<br>
In segmentation <a href="https://github.com/HRNet/HRNet-Semantic-Segmentation/blob/master/lib/models/seg_hrnet.py" rel="nofollow noopener">HRNetV2 PyTorch realization</a> is used construction like<br>
<code>y = self.fuse_layers[i][j](x)</code><br>
Here fuse_layers is ModuleList of ModuleList of some Sequential NN and x,y just tensors.<br>
<strong>How is it possible to implement it in c++ with LibTorch v1.4?</strong> (Without torch.jit , please)<br>
May be I can get somehow ModuleList number i from fuse_layers, when get Sequential number j from it and call its ‘forward’? Please help. It’s very practical question. My boss want c++ implementation of HRNet.</p>

voqtuyen: <p>I end up changing to torch 1.5.0 and replace the normal for-loop</p>
<pre><code class="lang-auto">for i in range(self.num_channels):
</code></pre>
<p>to</p>
<pre><code class="lang-auto">for i, layer in enumerate(self.fuse_layers):
</code></pre>
<p>However, i encounter another problem.</p>
Setting grad inside with set_grad_enabled

eprox: <p>Hi,<br>
If you try set the grad to false for a layer inside with grad enabled, does this override the with clause?<br>
For example:</p>
<pre><code class="lang-auto">with torch.set_grad_enabled(mode = True):
    self.model.lin2.requires_grad = False # Will this work?
   for m in self.model.mlp_f: m.set_grad_enabled = False # Or this?
</code></pre>

tom: <p>Keep in mind that <code>set_grad_enabled</code> is a state of the program (“do you want to keep track of gradients for outputs when the inputs require gradients”) and applies to new tensors, while <code>someparam.requires_grad_(False)</code> (which is the suggested form to disable gradients <em>of tensors and parameters</em>) says “this thing, when used as input, doesn’t require gradients”.<br>
To then decide whether a given operation’s output requires gradients, the autograd engine checks if both gradient-mode is enabled and any inputs require gradients.<br>
As such these are two distinct knobs you can operate independently, but they have a combined effect.</p>
<aside class="quote no-group" data-username="eprox" data-post="1" data-topic="95497">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/e/b4bc9f/48.png" class="avatar"> eprox:</div>
<blockquote>
<pre><code class="lang-auto">with torch.set_grad_enabled(mode = True):
    self.model.lin2.requires_grad = False # Will this work?
   for m in self.model.mlp_f: m.set_grad_enabled = False # Or this?
</code></pre>
</blockquote>
</aside>
<p>This seems to try to work on Modules, which isn’t a thing (and using <code>requires_grad_</code> would tell you so, which is why it is preferred). If you wanted something like that, you should go for <code>for p in self.model.mlp_f.parameters(): p.requires_grad_(False)</code> which certainly works.</p>
<p>Best regards</p>
<p>Thomas</p>

eprox: <p>Ok Thank you that is useful.</p>
Understanding tensor.view(M, N)

udo: <p>Could someone kindly explain the logic behind the tensor reshape below? According to what kind of rule is the <code>b.view(3*2, 2)</code> tensor being re-organized?</p>
<pre><code class="lang-auto">import torch¬                                                                                                    
       a = torch.eye(2)¬                                                                                                
       b = a.repeat(1, 3)¬
</code></pre>
<blockquote>
<p>tensor([[1., 0., 1., 0., 1., 0.],¬<br>
[0., 1., 0., 1., 0., 1.]])¬</p>
</blockquote>
<p><code>b.view(3*2, 2)¬</code></p>
<blockquote>
<p>tensor([[1., 0.],¬<br>
[1., 0.],¬<br>
[1., 0.],¬<br>
[0., 1.],¬<br>
[0., 1.],¬<br>
[0., 1.]])¬</p>
</blockquote>

albanD: <p>Hi,</p>
<p>The view operation will just change the view you look at the memory.<br>
A contiguous Tensor is stored in a row major way, meaning that the rows contain elements that are next to each other in memory.<br>
When you do view, you just read these with different size, still with rows containing elements that are next to each other in memory.</p>
<p>In general, view are often used to collapse and uncollapse dimensions.<br>
In particular, if you want to work with a 1D Tensor in your case, you can do <code>b.view(3*2*2)</code>. Then work with this 1D Tensor and finally do <code>res.view(2, 6)</code> to get back to the original shape.</p>
I specified "retain_graph=True" but still get error "Trying to backward through the graph a second time, but the buffers have already been freed"

Yun_Lee: <p>I am looking at gradient-based adversarial attack code<br>
( setting data.requires_grad =True -&gt; and get the input(data) gradients that maximize loss function, not model gradients)</p>
<p>and get</p>
<pre><code class="lang-auto">RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time.
</code></pre>
<p>from</p>
<pre><code class="lang-auto">data_grad = torch.autograd.grad(loss, data, retain_graph=True, create_graph=True)[0]
</code></pre>
<p>when I searched the solution to that runtime error, the answer was to specify retain_graph = True<br>
but I already specified retain_graph=True in that code.</p>
<p>I guess the problem is that<br>
I used a custom model.</p>
<p>CustomModel is {{modelA. modelB), Last Classifier to combine Model A &amp; B (Working as Ensemble) }<br>
and when the data is forwarded, I changed the data differently on Model A and Model B. I mean, I used different functions on Model A, and Model B when I forwarded data in CustomModel.<br>
Those functions are not self parameters</p>
<p>Is it impossible to get gradient ( even though it is not model gradient )<br>
when I use a custom model in which functions that are not parameters are used in forward ?<br>
If there is a way, Can you tell me how to solve this error?</p>

albanD: <aside class="quote no-group" data-username="Yun_Lee" data-post="1" data-topic="95604">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/yun_lee/48/13932_2.png" class="avatar"> Yun_Lee:</div>
<blockquote>
<p>when I searched the solution to that runtime error, the answer was to specify retain_graph = True<br>
but I already specified retain_graph=True in that code.</p>
</blockquote>
</aside>
<p>You need to specify it the first time you called autograd.grad or .backward. Not the one that raises the error.</p>
<p>Also if you have your training loop, you should avoid doing differentiable computations outside of the loop. As it will also create part of the graph that will be re-used from one iteration to the next.</p>
Softmax giving nans and negative values as output

granth_jain: <p>Hi,</p>
<p>I am using softmax at the end of my model.</p>
<p>However after some training softmax is giving negative probability.In some situations I have encountered nans as probability as well.</p>
<p>one solution i found on searching is to use normalized softmax…however I can not find any pytorch imlpementaion for this.</p>
<p>Can someone please help to let know if there is a normalized softmax available or how to achieve this so that forward and backward propagations are smooth.</p>
<p>Please note that I am already using torch.nn.utils.clip_grad_norm_(model.parameters(), 40) to avoid exploding gradients</p>

hhaoao: <p>The focus of pytorch is not on data processing. All related issues must be resolved with the help of third-party toolkits. You can use the StandardScaler of <code>scikit-learn</code>. Of course, you can also learn about <code>skorch</code> (usually you don’t need this package, just <code>scikit-learn</code>).</p>
<p><a href="https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html?highlight=standardscaler#sklearn.preprocessing.StandardScaler" class="onebox" target="_blank" rel="nofollow noopener">https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html?highlight=standardscaler#sklearn.preprocessing.StandardScaler</a></p>

granth_jain: <p>Hi.</p>
<p>Thanks, I am trying to take a sigmoid before taking softmax…also can you please help to let me know what is the ideal value for clip in torch.nn.utils.clip_grad_norm_ while training an A3C</p>

hhaoao: <p>Sorry, I am not familiar with the a3c algorithm you mentioned, and I cannot provide you with more help.But I found a newbie guide, hope it helps you.<br>
<a href="https://pytorch.org/tutorials/beginner/transformer_tutorial.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/transformer_tutorial.html</a></p>

googlebot: <p>softmax is used normalized (max value subtracted), and negative softmax outputs are mathematically impossible. You should recheck what are you looking at. And maybe try LayerNorm(affine=true) or x.clamp_(-10.0,10.0) before softmax.</p>