akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to link a custom NCCL version

OasisArtisan: <p>So I’m working on a project where I had to modify NCCL a bit to serve my purpose.</p>
<p>Now my question is how would I force pytorch to use my version of NCCL?</p>
<p>To start with, Is NCCL dynamically linked so pytorch would automatically link to any version of NCCL available? or is it statically linked that I need to recompile Pytorch with my custom NCCL version?</p>
<p>Any pointers or tips are appreciated.</p>
<p>Cheers</p>

osalpekar: <p><a class="mention" href="/u/oasisartisan">@OasisArtisan</a> PyTorch has a specific version of NCCL as a submodule. If you want to use a different version of NCCL, you can rebuild PyTorch with the <code>USE_SYSTEM_NCCL</code> flag.</p>
<p>Here’s a similar forums question: <a href="https://discuss.pytorch.org/t/ncc-version-and-pytorch-nccl-version-mismatch/87771" class="inline-onebox">NCC version and Pytorch NCCL version mismatch</a></p>

OasisArtisan: <p>I see. And if i set the <code>USE_SYSTEM_NCCL</code> flag, then would NCCL be linked dynamically or statically to pytorch?</p>
<p>To illustrate my intention, I want to know that if I need to recompile pytorch everytime I change something in my custom NCCL version. If its linked dynamically, then as long as I keep the same NCCL interface I do not need to recompile PyTorch.</p>

osalpekar: <p>I believe it’s dynamically linked, but it seems that can be toggled with <code>USE_STATIC_NCCL</code>.</p>

OasisArtisan: <p>I see thanks a lot.</p>
<p>Cheers</p>

OasisArtisan: <p>If I can have some follow up questions…</p>
<p>First, I implicitly understood that if PyTorch was using its own NCCL submodule then it is linking to it statically. Is my understanding correct?</p>
<p>Second, is there a way to know what NCCL compilation flags were used to produce the PyTorch binaries installed by Conda?</p>
<p>Thanks again.</p>

ptrblck: <p>You can see <a href="https://github.com/pytorch/builder/blob/43e314f7d93b7e8f15a11739d7aa80d7c528202a/conda/pytorch-nightly/build.sh#L50">here</a> that NCCL is statically linked to the binaries and can take a look at the repository for more information about the build process. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
All parameters not updated in model.parameters

anesh117: <pre><code class="lang-auto">class NN(pl):
    def __init__(self,input_dim,num_units1,num_units2,output):
        
        super(NN,self).__init__()
        self.w=[]
        for i in range(input_dim):
            self.w.append(Variable(torch.rand((1,),requires_grad=True)))
        
        self.w=Variable(torch.reshape(torch.tensor(self.w,dtype=torch.float32),(1,input_dim)),requires_grad=True)
        

     def forward(self):
               .........
               .........
</code></pre>
<p>When I ran model.paramters() I can’t see the self.w in the learnable parameters. How do I make the self.w as learnable parameters?</p>

waterhorse1: <p>you need to set self.w as a nn.ParameterList rather than a List, which will not be counted when calling model.parameters()</p>
Visual explanation for zero_grad()

v-za: <p>Does anybody have or can point to a visual explanation for why we do zero_grad() in the training loop? What does it mean when we say the gradients accumulate?</p>

ptrblck: <p>I cannot explain it visually, but “accumulating gradients” means that they will be summed to the already calculated gradients.<br>
Each parameter contains a <code>.grad</code> attribute after the first <code>backward()</code> operation, which will be set to the current gradient. You would use these gradients to update the parameters and zero them out afterwards.<br>
However, some use cases (such as simulating a larger batch size) are accumulating the gradients for several iterations and perform the update step later.<br>
Here is a small example:</p>
<pre><code class="lang-python"># setup
model = nn.Linear(1, 1, bias=False)
x = torch.randn(1, 1)

# check gradients
print(model.weight.grad)

# fake training step to calculate gradients
out = model(x)
out.mean().backward()

# check gradient again
print(model.weight.grad)

# usually you would not update the model via optimizer.step()
# and zero out the gradients before the next backward() call

# accumulate gradients
for _ in range(10):
    out = model(x)
    out.mean().backward()

    # check gradient
    print(model.weight.grad)
</code></pre>
How to get learning rate for specific layer

xdwang0726: <p>I have set different learning rate for a specific layer, and I am wondering is there a way to check whether I have set correctly? (I want to print the learning rate for that specific layer) Thanks!</p>

ptrblck: <p>If you are using the recommended way of specifying a <code>dict</code> with the per-layer parameters, it should be visible which layers/parameters are using which learning rate.<br>
You could double check the <code>id</code>s of the parameters inside the optimizer and compare it to the parameters in the model.</p>
How to do gathering on a tensor with two-dim indexing

RunningLeon: <h3>Question</h3>
<p>Hi,<br>
Want to add symbolic func to a custom PyTorch op and export it to ONNX using existing ONNX ops. There is two-dim indexing operation. Have tried <code>index_select</code>, but not work. So could anyone take a look into this and help me with this?</p>
<h3>Further information</h3>
<p>Sample code</p>
<pre><code class="lang-auto">def my_custom_op(data, x_indices, y_indices):
    ## suppose this op is written in c++
    return data[x_indice, y_indices]

class MyCustomOp(torch.autograd.Function):
    
    @staticmethod
    def forward(ctx, data, x_indices, y_indices):
        return my_custom_op(data, x_indices, y_indices)

    @staticmethod
    def symbolic(g, data, x_indices, y_indices):
        from torch.onnx.symbolic_opset9 import index_select, transpose
        data_xs = index_select(g, data, 0, x_indices)
        ## don't know how to do this because index_select not work for this 
        # data_xs = transpose(g, data_xs, 0, 1)
        # data_ys = index_select(g, data_xs, 0, y_indices)
        return out

</code></pre>
<p>Thanks in advance.</p>
IsADirectoryError: [Errno 21] Is a directory

111448: <p>The data already readed ,any help</p>
<pre><code class="lang-auto">from google.colab import drive
drive.mount('/content/drive')
!unzip  "/content/drive/MyDrive/NADI2021_DEV.1.0.zip"
</code></pre>
<pre><code class="lang-auto">

data_path = 'Data/'
models_path = 'Models/'
if not os.path.exists(data_path):    
    os.makedirs(data_path)
if not os.path.exists(models_path):    
    os.makedirs(models_path)
</code></pre>
<p>.<br>
.<br>
.<br>
.<br>
.<br>
.<br>
.</p>
<pre><code class="lang-auto"># load model
embed_epoch = 1
model_embed = load_model(model_embed,embed_epoch,'embed')
model_embed.to(device)
</code></pre>
<pre><code class="lang-auto">Loading model from Models/saved_embed_epoch1.txt
---------------------------------------------------------------------------
IsADirectoryError                         Traceback (most recent call last)
&lt;ipython-input-145-da43c88e687e&gt; in &lt;module&gt;()
      1 # load model
      2 embed_epoch = 1
----&gt; 3 model_embed = load_model(model_embed,embed_epoch,'embed')
      4 model_embed.to(device)
      5 # load model

3 frames
/usr/local/lib/python3.6/dist-packages/torch/serialization.py in __init__(self, name, mode)
    209 class _open_file(_opener):
    210     def __init__(self, name, mode):
--&gt; 211         super(_open_file, self).__init__(open(name, mode))
    212 
    213     def __exit__(self, *args):

IsADirectoryError: [Errno 21] Is a directory: 'Models/'
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/f/bf7f1cf762774eb40dc65bd6666f4b576feb12eb.png" data-download-href="https://discuss.pytorch.org/uploads/default/bf7f1cf762774eb40dc65bd6666f4b576feb12eb" title="Screenshot (345)"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/f/bf7f1cf762774eb40dc65bd6666f4b576feb12eb_2_690x245.png" alt="Screenshot (345)" data-base62-sha1="rk3xcyasVUOWbfiMvI6x66150Ej" width="690" height="245" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/f/bf7f1cf762774eb40dc65bd6666f4b576feb12eb_2_690x245.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/f/bf7f1cf762774eb40dc65bd6666f4b576feb12eb_2_1035x367.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/f/bf7f1cf762774eb40dc65bd6666f4b576feb12eb_2_1380x490.png 2x" data-dominant-color="363636"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot (345)</span><span class="informations">1920×684 48.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>The error is raised, since you are passing a path to a directory instead of a model checkpoint.<br>
Make sure to pass the path to the file instead of the dir.</p>
Adding noise when using embedding layer

gg234: <p>Hi, I’m building a generator <code>g</code>,  that receives a latent-code (vector of shape 100) and outputs an image. Specifically, I have 1000 MNIST images, and I want the network to learn a latent code <code>z_i</code> for each image <code>x_i</code>, such that <code>g(z_i)=x_i</code> (this approach is known as Generative Latent Optimization). So I’ve used nn.Embedding(1000,embedding_dim=100) and standard generator architecture, that receives the code from embedding and outputs an image. As for loss, I combine reconstruction loss with regularization on the embedding-vector weights.<br>
<strong>My probelm is</strong>: I’d like to add noise to the latent-code vector before it is inserted to the generator (in order to make the latent-code compact). However I’m a beginner, and I don’t know whether I should call detach() when adding the noise or not. I’m not sure of my approach entirely. I don’t want to learn the scale of the noise or anything.<br>
Here’s my attempt:</p>
<pre><code class="lang-auto">class net(nn.Module):
  def __init__():
    self.embed = nn.Embedding(1000,embedding_dim=100)
    self.generator = nn.sequential( nn.Linear(100, 84), .... )
  def forward(batch_indices):
    batch_codes = self.embed(batch_indices)
    noise = torch.randn_like(batch_codes) * sigma
    noisy_batch_codes = batch_codes + noise # SHOULD THIS BE batch_codes.detach() + noise ??
    return self.generator(noisy_batch_codes)

g = net()
optim = SGD(g.parameters(), lr=0.01)
for epoch in range(num_epochs):
  for orig_images, orig_images_idx in trainloader:
    optim.zero_grad()
    output = g(orig_images_idx)
    reconstruction_loss = nn.MSELoss()(output, orig_images)
    embed_vector_weights = g.embed.weight[orig_images_idx]
    reg_loss = torch.norm(embed_vector_weights) * reg_coeff
    loss = reconstruction_loss + reg_loss
    loss.backward()
    optim.step()
</code></pre>

ptrblck: <p>I’m not familiar with this particular method, but if you call <code>batch_codes.detach()</code> while adding the <code>noise</code> tensor, <code>self.embed</code> won’t get any gradients and will thus not be updated.<br>
Does this fit the use case?</p>
openVino vs PyTorch Tracing

Dennis_Hugle: <p>Dear Community,</p>
<p>i followed the examples for PyTorch C++ export and Tracing, which works really nicely.<br>
However, there is also openVino, which claims to “support heterogeneous execution across an Intel® CPU, Intel® Integrated Graphics, Intel® Neural Compute Stick 2 […]”.</p>
<p>Does PyTorch Tracing also support CPU with Integrated Graphics/built-in AI Acceleration? If yes, are there additional steps needed from the dev side to make it run?</p>
<p>Cheers</p>
<p>PS: Does someone know a good source to compare openVino and PyTorch tracing?</p>
Can one load a pretrained model as a part of a bigger model?

meshghi: <p>Hi all,</p>
<p>I want to use a unet-vgg16 model with pre-traing weights on ImageNet in order to perform a semantic segmentation task on Cityscapes dataset. Since the input image in the Cityscapes dataset is 1024*2048, I was wondering if there is a way that I use the pretrained model and expand it to match the Cityscapes images?</p>
<p>P.S. I’m also thinking of using something like a sliding window approach to run unet-vgg16 pre-trained model on the fly on the Cityscapes images, patch by patch, and then construct the final segmentation image for each image. However, I don’t know how to do it.</p>
<p>I need help, and I really do appreciate your help in advance.</p>

user_123454321: <p>Yes this idea is already explored <a href="https://arxiv.org/abs/1711.08580" rel="noopener nofollow ugc">here</a>. You can find an implementation in <a href="https://github.com/Project-MONAI/MONAI/" rel="noopener nofollow ugc">MONAI</a>, the ‘ahnet’ network.</p>
Hyperparameters for CIFAR-10

tginart: <p>Hi all,</p>
<p>I am wondering if anyone has figured out any “standard” hyperparameter configurations for obtaining the best accuracy at a given epsilon privacy level on CIFAR-10?</p>
<p>If so, it might be a good idea to include them in the tutorial scripts?</p>
<p>Thanks in advance!</p>

ptrblck: <p>I don’t think these experiments are published yet, but am sure that any contribution is welcome.<br>
Would you be interested in working on such a tutorial? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

tginart: <p>I have done some hp sweeps and am planning to do some more over the holidays. I don’t expect them to be entirely exhaustive but they’d be a good starting place. I’ll share in early 2021. Thanks!</p>

Darktex: <p>Hi <a class="mention" href="/u/tginart">@tginart</a>! Sorry for the delay in getting back to you, we are still getting used to the forums ourselves and figuring out how to setup notifications <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>
<p>We are still in the process of figuring it out ourselves. Getting &gt;70 top-1 accuracy for reasonable privacy levels (let’s say under 10 epsilon at delta 1e-5) is quite hard and an area of active development.</p>
<p>One shortcut could be to pretrain without dp: in this case, you can get there quite easily. See this colab: <a>bit.ly/opacus-dev-day</a>.</p>
<p>In another experiment, we were able to get to 68 without any pretraining using a smaller model, but we are still in the process of cleaning up the results before we publish them <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>Looking forward to seeing more experiments and results! We would indeed welcome PRs, or tutorials. Even if you want to write it externally, we’d still be happy to link to it in our resources page <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

tginart: <p>Hello, <a class="mention" href="/u/darktex">@Darktex</a>!</p>
<p>Sorry for the late reply. Somehow this got buried in my notifications before the new year.</p>
<p>With regards to using smaller models — since the privacy budget can depend on the parameter count, it does stand to reason that a smaller model might give you more “bang for buck.” I actually tested this out a little bit by comparing MobileNetV2 to ResNet18 on a few runs but I wasn’t able to get a major improvement (granted, I could’ve tried much harder).</p>
<p>I’m looking forward to seeing your results on this! I’ll keep you posted on anything from my end as well. I think it would be useful/cool to maintain a sort of “state-of-the-art” leaderboard for provably DP models at a given (eps,delta). I could help kick this off with some of my experiments, although 68% is already significantly better than anything I’ve obtained. My best runs don’t even make it to 60%.</p>
torch.jit.frontend.UnsupportedNodeError: function definitions aren't supported

mkulariya: <p>I have a custom I3d model and want to convert to torchscript so that it can be used with Deepstream.<br>
I have tried jit.trace but got this error:</p>
<pre><code class="lang-auto">x = torch.ones((1, 3, 64, 224, 224)).cuda()
traced_script_module = torch.jit.trace(net, x)
</code></pre>
<p>where net is my custom I3D model.<br>
<code>Dictionary inputs to traced functions must have consistent type. Found Tensor and Dict[str, Tensor]</code></p>
<p>I have tried jit.script method as well and got this:<br>
<code>traced_script_module = torch.jit.script(net)</code></p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "convert_i3d.py", line 50, in &lt;module&gt;
    traced_script_module = torch.jit.script(net)
  File "/usr/local/lib/python3.6/dist-packages/torch/jit/__init__.py", line 1516, in script
    return torch.jit._recursive.create_script_module(obj, torch.jit._recursive.infer_methods_to_compile)
  File "/usr/local/lib/python3.6/dist-packages/torch/jit/_recursive.py", line 318, in create_script_module
    return create_script_module_impl(nn_module, concrete_type, stubs_fn)
  File "/usr/local/lib/python3.6/dist-packages/torch/jit/_recursive.py", line 330, in create_script_module_impl
    stubs = stubs_fn(nn_module)
  File "/usr/local/lib/python3.6/dist-packages/torch/jit/_recursive.py", line 535, in infer_methods_to_compile
    stubs.append(make_stub_from_method(nn_module, method))
  File "/usr/local/lib/python3.6/dist-packages/torch/jit/_recursive.py", line 49, in make_stub_from_method
    return make_stub(func, method_name)
  File "/usr/local/lib/python3.6/dist-packages/torch/jit/_recursive.py", line 34, in make_stub
    ast = torch.jit.get_jit_def(func, name, self_name="RecursiveScriptModule")
  File "/usr/local/lib/python3.6/dist-packages/torch/jit/frontend.py", line 185, in get_jit_def
    return build_def(ctx, fn_def, type_line, def_name, self_name=self_name)
  File "/usr/local/lib/python3.6/dist-packages/torch/jit/frontend.py", line 219, in build_def
    build_stmts(ctx, body))
  File "/usr/local/lib/python3.6/dist-packages/torch/jit/frontend.py", line 126, in build_stmts
    stmts = [build_stmt(ctx, s) for s in stmts]
  File "/usr/local/lib/python3.6/dist-packages/torch/jit/frontend.py", line 126, in &lt;listcomp&gt;
    stmts = [build_stmt(ctx, s) for s in stmts]
  File "/usr/local/lib/python3.6/dist-packages/torch/jit/frontend.py", line 192, in __call__
    raise UnsupportedNodeError(ctx, node)
torch.jit.frontend.UnsupportedNodeError: function definitions aren't supported:
  File "/usr/local/lib/python3.6/dist-packages/drishti/nnh/inferencers/i3d.py", line 390
        end_points["logits_pre_reshape"] = out_logits

        def squeeze_and_permute(tensor):
        ~~~ &lt;--- HERE
            # [N, 8k, 8, H, W] -&gt; [N, 8, H, W, 8k]
            tensor = tensor.permute([0, 2, 3, 4, 1])

</code></pre>
Interpreting in higher dimensions with maximal spacing constraint

learner47: <p>I do not know if this is the right forum to ask, but here it is. I have been using PyTorch with supervised learning. Now I am trying out something which (in my opinion) is not supervised and is outlined below. This is my first time attempting such a problem.</p>
<p>I have data that I would like to interpret in higher dimension (sort of the opposite of an auto-encoder). What I intent to achieve is, an <code>m</code> dimension data being represented in <code>n</code> dimensions (<code>n&gt;m</code>) with the constraint that the representation in the <code>n</code> dimensions is maximally separated.</p>
<p>For clarity, I will assume the binary field. My data has <code>m=20</code> dimensions. I would like to represent this data in a <code>n=50</code> dimensional space. There are <code>2^m</code> number of data combinations that are possible and <code>2^n</code> number of points in the larger dimension. My goal is to select <code>2^m</code> points from <code>2^n</code> points, which are as far away from each other as possible.</p>
<p>If anyone can help me with the type of learning (a basic outline) that I would need to do, it would be great. Thanks!</p>

googlebot: <p>that’s a bit too generic…</p>
<ol>
<li>do you really want to measure separation using distances in discrete space, like hamming distance? or you seek a solution for continuous numbers?</li>
<li>assuming continuous, will “autonomous” transformation suffice, or should it support gradient flow before/after? i.e. should separation measure form an auxiliary loss term for another task?</li>
</ol>

learner47: <p><a class="mention" href="/u/googlebot">@googlebot</a>, thank you for your reply.</p>
<ol>
<li>
<p>The ultimate goal is the hamming distance. But I think to reach there, through training, we might need to go through continuous numbers. What I mean is, suppose we start from random initialization, then we need to move either towards 0 or towards 1 to maximize the criterion. However, if it is possible to do it without having to go through the real numbers, that would also do.</p>
</li>
<li>
<p>I do not have another loss. The training concludes when I have found the points on the higher dimensional space.</p>
</li>
</ol>

googlebot: <p>Hm, check if <a href="https://en.wikipedia.org/wiki/Gray_code" rel="noopener nofollow ugc">gray code</a> won’t solve this problem; take n-digit gray codes and select 2^m regularly spaced elements as output space (using input as index)</p>
<p>In case that won’t work, I still feed that you need to seek a mathematical solution instead of “training”.</p>

learner47: <p><a class="mention" href="/u/googlebot">@googlebot</a> mathematical solutions work well for small dimensions. I would like to work with large dimensions, so I assumed training would help. I will take a look at gray code. Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:">.</p>
Speed up ensemble model by parallelizing the blocks

Mamdouh_Aljoud: <p>I am trying to speed up an algorithm responsible for producing 3d skeleton joints from 2D images. The algorithm (GAST-NET) consists of 4 main blocks running sequentially for every frame. I’m trying to parallelize the 4 blocks. I have some questions regarding the process.</p>
<ol>
<li>Will parallelization help speed up the algorithm? I am trying to parallelize on one GPU only.</li>
<li>What other ways can I look into that can help with speeding up the algorithm?</li>
<li>Slightly related question, Isn’t PyTorch already trying to use maximum GPU resources to produce output as fast as possible? I monitored the GPU utilization and it was between 38 and 50%. Is there a way I can ensure that the GPU is used to the fullest?</li>
</ol>

pritamdamania87: <aside class="quote no-group" data-username="Mamdouh_Aljoud" data-post="1" data-topic="108366">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mamdouh_aljoud/48/29609_2.png" class="avatar"> Mamdouh_Aljoud:</div>
<blockquote>
<ul>
<li>Will parallelization help speed up the algorithm? I am trying to parallelize on one GPU only.</li>
</ul>
</blockquote>
</aside>
<p>This really depends on the algorithm.</p>
<aside class="quote no-group" data-username="Mamdouh_Aljoud" data-post="1" data-topic="108366">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mamdouh_aljoud/48/29609_2.png" class="avatar"> Mamdouh_Aljoud:</div>
<blockquote>
<ul>
<li>What other ways can I look into that can help with speeding up the algorithm?</li>
<li>Slightly related question, Isn’t PyTorch already trying to use maximum GPU resources to produce output as fast as possible? I monitored the GPU utilization and it was between 38 and 50%. Is there a way I can ensure that the GPU is used to the fullest?</li>
</ul>
</blockquote>
</aside>
<p>If you could share some sample PyTorch code for the algorithm, I can look into it to see if there are some optimization opportunities.</p>
Time profile the model prediction on gpu in pytorch?

amitoz: <p>Is this the right way to time profile the model prediction on a test sample?</p>
<pre><code class="lang-python">import torch.autograd.profiler as profiler
x.to("cuda:0") # test sample
model.eval()
with profiler.profile(use_cuda=True) as prof:
    with torch.no_grad():
        pred = model(x)
print(prof)
</code></pre>

ptrblck: <p>I would recommend to add some warum iterations and also calculate the actual runtime as a mean over iterations. This should be automatically done for you by using <code>torch.utils.benchmark</code> from the current master branch or nightly binary.</p>

amitoz: <p>Thanks for your reply, I suppose you meant <strong>warmup</strong> iterations? But I got the general idea to take mean over number of iterations.</p>
<p>Also it prints this in the end:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/9/e/9ec004c6f946d846ae59f585d5f48b79b87172f7.png" alt="image" data-base62-sha1="mEmPYRLPG9JwfU0yk5968VdHBgH" width="252" height="50"></p>
<p>This might be silly question but please bear with me as I am curious here;  the cuda time corresponds to the time taken to compute the prediction</p>
<pre data-code-wrap="python"><code class="lang-python">pred = model(x)
</code></pre>
<p>What exactly does the CPU time corresponds here?</p>
'torch' has no attribute 'scatter'

oasjd7: <p>In torch version: 1.0.0, there is no ‘scatter’ function.<br>
How can I use this ‘scatter’ ?</p>

KFrank: <p>Hi oasjd7!</p>
<aside class="quote no-group" data-username="oasjd7" data-post="1" data-topic="108623" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/oasjd7/48/5053_2.png" class="avatar"> oasjd7:</div>
<blockquote>
<p>In torch version: 1.0.0, there is no ‘scatter’ function.<br>
How can I use this ‘scatter’ ?</p>
</blockquote>
</aside>
<p>I’ve never used version 1.0.0, so I’m just guessing …</p>
<p>According to the <a href="https://pytorch.org/docs/1.0.0/tensors.html" rel="noopener nofollow ugc">1.0.0 documentation</a> for <code>torch.Tensor</code> the <code>Tensor</code><br>
class has an in-place <code>.scatter_()</code> method.</p>
<p>That is, you would call the <code>.scatter_()</code> method on an instance of the<br>
<code>Tensor</code> class, making a copy of your <code>Tensor</code>, as necessary (because<br>
<code>.scatter_()</code> in an in-place operation).</p>
<p>(Or you could use a newer version of pytorch.)</p>
<p>Best.</p>
<p>K. Frank</p>
How does torch threshold values in Hardtanh function?

learner47: <p>I was going through the activation function <a href="https://pytorch.org/docs/stable/generated/torch.nn.Hardtanh.html" rel="noopener nofollow ugc">Hardtanh</a>. The definition involves clipping (or clamping) the values between <code>1</code> and <code>-1</code>. How does Torch threshold the values for this operation while still remaining differentiable?</p>

albanD: <p>Hi,</p>
<p>It does regular clamping like you would do with the clamp function. And the gradient for the constant part of the function is just 0.<br>
Does that answer your question?</p>

learner47: <p>you mean <code>torch.clamp</code>?</p>

albanD: <p>Yes.<br>
Just like when you have relu with a negative input. It is constant in that region and will return a 0 gradient.</p>
Loss function for objectiveness, classes, and one bounding box

PyMike: <p>I’m trying to correctly train a classification/localization (a single bounding box) network.</p>
<p>Input is an image and the output is 11 values:</p>
<pre><code class="lang-auto">&gt;  0: object_exists:  0 or 1
&gt;  1: is_bird:        0 or 1    want to ignored if truth-object_exists==0
&gt;  2: is_cat:         0 or 1    want to ignored if truth-object_exists==0
&gt;  3: is_dog:         0 or 1    want to ignored if truth-object_exists==0
&gt;  4: is_ladybug:     0 or 1    want to ignored if truth-object_exists==0
&gt;  5: has_wings:      0 or 1    want to ignored if truth-object_exists==0
&gt;  6: is_colorful:    0 or 1    want to ignored if truth-object_exists==0
&gt;  7: bbox_center_x: -1 .. 1    want to ignored if truth-object_exists==0
&gt;  8: bbox_center_y: -1 .. 1    want to ignored if truth-object_exists==0
&gt;  9: bbox_width:     0 .. 2    want to ignored if truth-object_exists==0
&gt; 10: bbox_height:    0 .. 2    want to ignored if truth-object_exists==0
</code></pre>
<p>When there is no object (i.e. truth-object_exists==0) all the classes and bbox values are set to 0.</p>
<p>I’m treating all values as regression with MSELoss(), and avoiding nn.CrossEntropyLoss() since multiple classes might be correct.</p>
<p>Here’s the training loop I use (the validation loop does the same):</p>
<pre><code class="lang-auto">loss_function = nn.MSELoss()
# Model training loop
for data, truth in training_loader:
    data, truth = data.cuda(), truth.cuda()
    optimizer.zero_grad()
    predictions = model(data)
    loss = loss_function(predictions, truth)
    loss.backward()
    optimizer.step()
</code></pre>
<p>This results of the model training cause the bounding box to heavily skew to the top left corner since the object is never detected with 100% accuracy.</p>
<p>First question, how do I change:<br>
<code>loss = loss_function(predictions, truth)</code><br>
to ignore the classes and bounding box if truth-object_exists==0?</p>
<p>Second question, how do I improve the loss function to weight the value of the bounding box to something different from the classes?</p>
<p>Last question, is nn.MSELoss() the most appropriate for all of the regression values?</p>
<p>Thank you for any guidance.</p>
DataLoader problem with Cityscapes dataset

meshghi: <p>Hi,<br>
I’m trying to load the Cityscapes dataset and format it in batches using DataLoader but I keep getting this error:</p>
<blockquote>
<p>TypeError: default_collate: batch must contain tensors, numpy arrays, numbers, dicts or lists; found &lt;class ‘PIL.PngImagePlugin.PngImageFile’&gt;</p>
</blockquote>
<p>my ode is this:</p>
<pre><code class="lang-auto">BATCH_SIZE = 128

train_ds = Cityscapes(DATA_DIR,
                      split='train',
                      mode='fine',
                      target_type='semantic',
                      transform=transforms.ToTensor()
                      )

train_dl = DataLoader(train_ds,
                      BATCH_SIZE,
                      shuffle=True,
                      num_workers=4,
                      pin_memory=True)


for batch_idx, (data, target) in enumerate(train_dl):
    print(batch_idx, batch_idx.shape)
    print(data, target)
    break
</code></pre>

user_123454321: <p>Can you try by setting the target_transform attribute as in</p>
<pre><code class="lang-auto">train_ds = Cityscapes(DATA_DIR,
                      split='train',
                      mode='fine',
                      target_type='semantic',
                      transform=transforms.ToTensor(),
                      target_transform=transforms.ToTensor()
                      )
</code></pre>

meshghi: <p>Thank you <a class="mention" href="/u/user_123454321">@user_123454321</a>, it worked.</p>
Distributed Data Parallel master memory overhead

anwu21: <p>I am training with a large amount of data per batch such that for batch_size = 1, I am observing the following:</p>
<ol>
<li>I cannot fit it on a single gpu.</li>
<li>I can fit it on a machine with 2 gpus when using DataParallel.  This is a little confusing to me since I have read that batch_size=1 cannot be used with DataParallel.  Does this mean half of a batch is on each gpu, and therefore “disconnected” when seen by the model?</li>
<li>I cannot fit it using DistributedDataParallel.</li>
</ol>
<p>I then modified my data so that each batch contains a smaller amount of data – I cannot go any smaller in terms of data per batch.  I observe that:</p>
<p>A) I can fit batch_size=2 on a single gpu, but not batch_size=3.<br>
B) I can fit batch_size=4 on a machine with 2 gpus using DataParallel, but not batch_size=5<br>
C) I can fit batch_size=1 using DistributedDataParallel, but not batch_size=2.</p>
<p>I was under the impression that DistributedDataParallel is more efficient than DataParallel from the tutorials I read, and thus somewhat counter to what I am observing.  I am wondering if DistributedDataParallel has memory overhead that DataParallel does not have and therefore explains what I am observing, as I have not been able to find any references regarding this situation.  It is also a bit odd to me that a single gpu using DistributedDataParallel fits less than a single gpu standalone.  Is it correct to assume that there is some type of workload imbalance afflicting DistributedDataParallel?  Even with workload imbalance, wouldn’t it still be better than that for DataParallel, which seems to be able to fit more data?</p>

Yanli_Zhao: <p>DistributedDataParallel in default uses extra buffer to synchronize gradients, thus consumes more memory (one extra copy of all grads) than what local training consumes.</p>
<p>In PyTorch 1.8, we added a new prototype flag “grad_as_bucket_view”, if you set it as True, it can save this extra copy of all grads, and should consume almost the same memory as local training.</p>
How many images are there after using transforms

acoder_acoder: <p>I have a special dataset and images in the dataset are with channels more than 3.</p>
<p>So I want to chage the code in torchvision.transforms. I do not know how many images will be produced after the transform.<br>
For example, I input one image with the shape 256   256   30, and I want to crop the input image to size 224   224   30. How many images will produced normally for RGB images?</p>
<p>I also try to write a new crop function for numpy array, then combine it with transforms.Lambda,  but when I use the transforms.ToTensor():<br>
ValueError: some of the strides of a given numpy array are negative. This is currently not supported, but will be added in future releases.</p>
<p>Thanks!!</p>

ptrblck: <p>Adding transformations to your <code>Dataset</code> does not increase the number of images, but applies these transformation on each sample.<br>
The number of samples is defined by the return value in <code>Dataset.__len__</code>.</p>
<p>E.g. if you are dealing with 1000 images, you’ll now get 1000 (randomly) transformed images.</p>
<p>There are a few exceptions, such as <code>transforms.FiveCrop</code> and <code>transforms.TenCrop</code>, which will yield 5 or 10 crops, respectively, and thus increase the number of samples.</p>
Gather values for multiple indices

itay1itzhak: <p>Hi all,<br>
I have a problem performing values gathering for multiple values and would appreciate it if anyone has an idea that can solve it.</p>
<p>So, I have 2 tensors:<br>
<strong>tgt</strong> - tokens (just numbers from a vocab)<br>
<strong>probs</strong> - the probs for each corresponded token</p>
<p>I want to know the probability for each unique token.<br>
But, each token can appear more than once in tgt, so we’ll have a few probs we need to aggregate. And we need this to happen for <strong>each</strong> token.</p>
<p>For example (with no batch dimension for simplicity):</p>
<pre><code class="lang-auto">tgt.shape (Batch_size X k)
probs.shape (Batch_size X k)

tgt = [3, 7, 3, 5, 11, 11, 11]
probs = [-0.2, -0.4, -0.5, -0.8, -0.1, -0.7, -0.2]
# We want the summed probs:
summed_probs = [-0.7, -0.4, -0.7, -0.8, -1.0, -1.0, -1.0]
# OR
summed_probs = [(3, -0.7),(7, -0.4), (5, -0.8), (11, -1.0)]
</code></pre>
<p>Right now I’m doing it with an iterative code using Numba, but I would like to get a tensor operation solution.</p>
<p>Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> !</p>

ptrblck: <p>If the order of the <code>tgt</code> values doesn’t matter, this would work:</p>
<pre><code class="lang-python">tgt = torch.tensor([3, 7, 3, 5, 11, 11, 11])
probs = torch.tensor([-0.2, -0.4, -0.5, -0.8, -0.1, -0.7, -0.2])

u, u_idx = tgt.unique(return_inverse=True)
ret = torch.zeros(len(u))
ret.scatter_add_(0, u_idx, probs)
print(ret)
</code></pre>
<p>and you could create the <code>tuples</code> if needed afterwards.</p>

itay1itzhak: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="107368">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<pre><code class="lang-auto">tgt = torch.tensor([3, 7, 3, 5, 11, 11, 11])
probs = torch.tensor([-0.2, -0.4, -0.5, -0.8, -0.1, -0.7, -0.2])
</code></pre>
</blockquote>
</aside>
<p>Thanks for your response!<br>
But if I’m not mistaken, this solution is only good for the 1-dimensional case. My example was with 1 dimension, but the problem is regarding the 2 dimensional case.<br>
For example:</p>
<pre><code class="lang-auto">tgt = torch.tensor([[3, 7, 3, 5, 11, 11, 11],[1, 1, 1, 4, 11, 15, 4]])
probs = torch.tensor([[-0.2, -0.4, -0.5, -0.8, -0.1, -0.7, -0.2],[-0.1, -0.9, -0.1, -0.4, -0.3, -0.3, -0.8]]) 

summed_probs = [[-0.7, -0.4, -0.7, -0.8, -1.0, -1.0, -1.0],[-1.1, -1.1, -1.1, -1.2, -0.3, -0.3, -1.2]]
# OR
summed_probs = [[(3, -0.7),(7, -0.4), (5, -0.8), (11, -1.0)],[(1, -1.1),(4, -1.2, (11, -0.3), (15, -0.3)]]
</code></pre>
Model fit on 1 of classes

nafise: <p>I have a 2-class categorization problem. Using the convolution network, I have created a model that I want to categorize. But in the training phase, the model predicts only one class.<br>
For example, it predicts output 1 for both class 0 and class 1.</p>
<p>Can anyone guide me?</p>

marvosyntactical: <p>Can you post your training loop?</p>
Lstm() received an invalid combination of arguments

David_FIM: <p>I use lstm from pytorch in my code to predict time series. while i write this code</p>
<pre><code class="lang-auto">class LSTM_model(nn.Module):
    def __init__(self, input_size, output_size, hidden_size,num_layers,dropout):
        super(LSTM_model,self).__init__()
        
        self.input_size = input_size
        self.hidden_size = hidden_size
        self.seq_len = seq_len
        self.num_layers = num_layers
        self.dropout = dropout
        self.output_size = output_size
        
#         self.lstm=nn.LSTM(self.input_dim, self.hidden_dim, self.num_layers)
        self.lstm = nn.LSTM(self.input_size, self.hidden_size,self.num_layers , self.dropout, batch_first=True)
        self.fc= nn.Linear(self.hidden_size , self.output_size)
        
    def forward(self, x , hidden):
        x, hidden= self.lstm(x,hidden)
        x = self.fc(x)
        return x,hidden
</code></pre>
<p>I got error from x,hidden = self.lstm(x.hidden) about the internal of that function.</p>
<p>the error message written below</p>
<p>i</p>
<pre><code class="lang-auto">python-input-59-c75cf1303610&gt; in forward(self, x, hidden)
     15 
     16     def forward(self, x , hidden):
---&gt; 17         x, hidden= self.lstm(x,hidden)
     18         x = self.fc(x)
     19         return x,hidden

D:\Anaconda\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--&gt; 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

D:\Anaconda\lib\site-packages\torch\nn\modules\rnn.py in forward(self, input, hx)
    579         self.check_forward_args(input, hx, batch_sizes)
    580         if batch_sizes is None:
--&gt; 581             result = _VF.lstm(input, hx, self._flat_weights, self.bias, self.num_layers,
    582                               self.dropout, self.training, self.bidirectional, self.batch_first)
    583         else:

TypeError: lstm() received an invalid combination of arguments - got (Tensor, tuple, list, int, int, float, bool, bool, bool), but expected one of:
 * (Tensor data, Tensor batch_sizes, tuple of Tensors hx, tuple of Tensors params, bool has_biases, int num_layers, float dropout, bool train, bool bidirectional)
      didn't match because some of the arguments have invalid types: (Tensor, !tuple!, !list!, !int!, !int!, !float!, !bool!, bool, bool)
 * (Tensor input, tuple of Tensors hx, tuple of Tensors params, bool has_biases, int num_layers, float dropout, bool train, bool bidirectional, bool batch_first)
      didn't match because some of the arguments have invalid types: (Tensor, !tuple!, !list!, !int!, int, float, bool, bool, bool)
</code></pre>
<p>Can someone help me please</p>

user_123454321: <p>What are you passing in hidden ?</p>

David_FIM: <p>i passed “None” as i want 0 in hidden</p>
How to save grad values after every batch?

divya: <p>(new to pytorch)<br>
Can anyone tell me how to save the gradients after every batch (and epoch) and also to load from the saved file?<br>
The model.dict() only save the weights and other parameters is what I could understand. But my requirement is to save the weight.grad</p>

CedricLy: <p>Try out:</p>
<pre><code class="lang-auto">grad_dic = {x[0]:x[1].data.grad for x in model.named_parameters()}
</code></pre>
<p>and save grad_dic somewhere.</p>
<pre><code class="lang-auto">torch.save(grad_dic,&lt;path&gt;)
...
grad_dic = torch.load(&lt;path&gt;)
</code></pre>

divya: <p>Thank you for the reply! I modified a bit and now i am able to save the gradient values!</p>
How to implement 4D tensor multiplication?

Ykk_L: <p>Given A: [B, N, K, K], B: [B, S, K, K].<br>
The shape of expected matrix multiplication result: [B, N, S, K, K].<br>
How to use <code>torch.matmul</code> to get this result?</p>

albanD: <p>Hi,</p>
<p>Assuming that you want to reduce dimension -1 of A and dimension -2 of B,<br>
you can do the following so that the batching works fine:</p>
<pre><code class="lang-auto">torch.matmul(A.unsqueeze(3), B.unsqueeze(2))
</code></pre>

Ykk_L: <aside class="quote no-group" data-username="albanD" data-post="2" data-topic="108476">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p>Assuming that you want to reduce dimension -1 of A and dimension -2 of B,</p>
</blockquote>
</aside>
<p>Hi,</p>
<p>I have tried your solution. But I met some errors. I use the code below.</p>
<pre><code class="lang-auto">a = torch.rand(2, 8, 3, 3)
b = torch.rand(2, 4, 3, 3)
ans = torch.matmul(a.unsqueeze(3), b.unsqueeze(2))
</code></pre>
<p>And I got the error:</p>
<pre><code class="lang-auto">ans = torch.matmul(a.unsqueeze(3), b.unsqueeze(2))
RuntimeError: The size of tensor a (8) must match the size of tensor b (4) at non-singleton dimension 1
</code></pre>

Ykk_L: <p>I know the problem. The following code works.</p>
<pre><code class="lang-auto">torch.matmul(A.unsqueeze(2), B.unsqueeze(1))
</code></pre>
<p>I’m curious about how it works. Could you please explain the reason?</p>
<p>Thanks.</p>

ccoding: <p>This function may be help you<br>
torch.einsum<br>
It multiplys two matrixs of any dimension.<br>
<a href="https://pytorch.org/docs/stable/generated/torch.einsum.html" rel="noopener nofollow ugc">torch.einsum</a></p>

Ykk_L: <p>Yeah, I have tried <code>torch.einsum</code> and got the same results.</p>
<pre><code class="lang-auto">torch.einsum('abik, ackj -&gt; abcij', A, B)
</code></pre>
<p>Thanks.</p>

albanD: <p>Ho my bad I miscounted the dimensions.</p>
<p>What the unsqueeze does is to make the sizes <code>2, 1, 8, 3, 3</code> and <code>2, 4, 1, 3, 3</code>. So that matmul can broadcast on these two dimensions of size 1 and do the matrix product you want.</p>

Ykk_L: <p>Got it. Thanks for your explanation.</p>
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn when training from examples

Gor_Matevosyan: <p>Hey everyone.<br>
I’m trying to train this exmaple:</p>
<pre><code class="lang-auto">
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print(device)

class TwoLayerNet(torch.nn.Module):
    def __init__(self, D_in, H, D_out):
        super(TwoLayerNet, self).__init__()
        self.linear1 = torch.nn.Linear(D_in, H)
        self.linear2 = torch.nn.Linear(H, D_out)

    def forward(self, x):
        h_relu = self.linear1(x).clamp(min=0)
        y_pred = self.linear2(h_relu)
        return y_pred

m = TwoLayerNet(1, 10, 2)
m.to(device)

criterion = torch.nn.MSELoss(reduction='sum')
optimizer = torch.optim.SGD(m.parameters(), lr=1e-4)

y = m(torch.Tensor([[1]]).cuda())
loss = criterion(y, torch.Tensor([[1, 1]]).cuda())
optimizer.zero_grad()
loss.backward()
</code></pre>
<p>And I got an error</p>
<pre><code class="lang-auto">RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
</code></pre>

ptrblck: <p>Your code snippet works fine if I add <code>loss.backward()</code>, so I guess you might be trying to call <code>.backward()</code> on another (detached) tensor?<br>
Could you compare the posted code snippet and update it to reproduce this error, please?</p>

Gor_Matevosyan: <p>I think I got the problem</p>
<p>I have this line of code in my notebook.</p>
<pre><code class="lang-auto">tts_model, tts_decoder, _ = torch.hub.load(repo_or_dir='snakers4/silero-models',
                                       model='silero_stt',
                                       language='en', # also available 'de', 'es'
                                       device=device)
</code></pre>
<p>When I restart the kernel and run the example provided with this line of code the problem occurs.<br>
To reproduce the error try to put this line before your model initialization.</p>

ptrblck: <p>Does the newly posted line of code raise this error or do you see the failure in the previously posted model, if the <code>silero_stt</code> model is just loaded before?</p>

Gor_Matevosyan: <p>the error raises when I call <code>loss.backward()</code> function. But it raises only when silero_stt model is loaded before. I tried to load both GPU and CPU versions.</p>

ptrblck: <p>I can reproduce the issue, which is raised by disabling the gradient calculation globally by the <code>silero-models</code> creation in <a href="https://github.com/snakers4/silero-models/blob/7cf51381e69aeab14da1dd4e324f98fa4ee7b14c/utils.py#L124">this line of code</a>.<br>
I don’t know, why it’s disabling the gradient calculation, but you can enable it again via:</p>
<pre><code class="lang-python">torch.set_grad_enabled(True)
</code></pre>
<p>after loading the model.</p>

Gor_Matevosyan: <p>Worked !! Thank you.<br>
Will ask them why they made it.</p>
Index Error in fine-tuning Faster-RCNN

Tupakula_Mallikarjun: <p>IndexError: too many indices for tensor of dimension 0 in Fast-RCNN implementation</p>

user_123454321: <p>This means that the data you are supplying to the model is not in the proper format. Can you show the image shapes and target shapes. Also, I think the faster-rcnn of torchvision requires a list of images and a list of targets (which is slightly different from other models).</p>

Tupakula_Mallikarjun: <p><a class="mention" href="/u/user_123454321">@user_123454321</a> Hey thanks for your reply. I had resolved it.</p>