akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
UserWarning: Argument fill/fillcolor is not supported for Tensor input

Eddie1997: <p>I got this user warning when I tried to implement the augmentation for my data, has anyone met this problem?</p>
<p>UserWarning: Argument fill/fillcolor is not supported for Tensor input. Fill value is zero<br>
warnings.warn(“Argument fill/fillcolor is not supported for Tensor input. Fill value is zero”)</p>

AlphaBetaGamma96: <p>Can you post a minimal reproducible example for this error?</p>

mMagmer: <p>check out this</p><aside class="onebox stackexchange" data-onebox-src="https://stackoverflow.com/questions/67393353/torchvision-transforms-randomrotation-fill-argument-not-working">
  <header class="source">

      <a href="https://stackoverflow.com/questions/67393353/torchvision-transforms-randomrotation-fill-argument-not-working" target="_blank" rel="noopener nofollow ugc">stackoverflow.com</a>
  </header>

  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/10288586/user930958" target="_blank" rel="noopener nofollow ugc">
    <img alt="user930958" src="https://www.gravatar.com/avatar/a8d444c9d408f9f6b40f0c08ed6cd240?s=256&amp;d=identicon&amp;r=PG&amp;f=y&amp;so-version=2" class="thumbnail onebox-avatar" width="256" height="256">
  </a>

<h4>
  <a href="https://stackoverflow.com/questions/67393353/torchvision-transforms-randomrotation-fill-argument-not-working" target="_blank" rel="noopener nofollow ugc">torchvision.transforms.RandomRotation fill argument not working</a>
</h4>

<div class="tags">
  <strong>python, pytorch, data-augmentation, torchvision</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/10288586/user930958" target="_blank" rel="noopener nofollow ugc">
    user930958
  </a>
  on <a href="https://stackoverflow.com/questions/67393353/torchvision-transforms-randomrotation-fill-argument-not-working" target="_blank" rel="noopener nofollow ugc">11:23PM - 04 May 21 UTC</a>
</div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<blockquote>
<p>You’re probably using torchvision v0.8.2 or older. This issue was fixed 5 months ago in the <a href="https://github.com/pytorch/vision/pull/2904/files#diff-497b983ae7c82237c4a0722bc6f6637525e0cdf8369dbd78ee19cf986bffe258R854-R862" rel="noopener nofollow ugc">PR #2904</a>. If you are not using <a href="https://github.com/pytorch/vision/releases/tag/v0.9.0" rel="noopener nofollow ugc">v0.9.0</a> or newer, you won’t be able to use <code>fill</code> in a <code>Tensor</code> input <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"><br>
So, the only solution is to upgrade your torchvision.</p>
</blockquote>

Eddie1997: <p>sorry guys I already fixed it, just update the pytorch and torchvision:)</p>

Eddie1997: <p>Just update the pytroch to latest version and problems solved:) haha</p>
Substract tensors with different sizes

Javier: <p>Hi all!</p>
<p>this is probably somewhere in this forum’s history but I can’t seem to find it.</p>
<p>My problem is the following: I have two tensors containing each a list of 3D points, one is S of size (81,3) and the other P of size (20000, 3). What I’d like to do is to compute the “displacement” vectors D = S-P between every 3D point in P to every 3D point in S. So the resulting D vector would have dimensions (20000, 81, 3), i.e for every point in P there are 81 displacements.</p>
<p>Does anyone know how to do this efficiently in pytorch?</p>
<p>Thanks in advance!</p>

ptrblck: <p>You could <code>unsqueeze</code> the tensors and let broadcasting do its job as seen here:</p>
<pre><code class="lang-python">S = torch.arange(0, 5*3).view(5, 3)
P = torch.arange(1, 10*3+1).view(10, 3)

D = S.unsqueeze(0) - P.unsqueeze(1)
print(D.shape)
# &gt; torch.Size([10, 5, 3])
print(D)
</code></pre>

Javier: <p>Thanks for the prompt reply <a class="mention" href="/u/ptrblck">@ptrblck</a> !</p>
<p>I was trying ‘repeat’ after the ‘unsqueeze’ to make them match in dimensions and then substract. It works but it looked unnecessary. This is more elegant.</p>
<p>Best</p>
How to parallarize Large Loss Computation

Bika: <p>Hello,<br>
I am facing an issue when back-propagating the loss due to large batch size due to the accumulation of the 8 GPU DataParallel. Is there is a way to divide the loss calculation into 8, then normalize the loss then use the loss.backward() but this time it should be smaller in size or how i can work this around ?</p>

cbalioglu: <p>I assume you are using DataParallel. I would suggest using DDP as it calculates the loss separately on each worker after syncing the gradients.</p>

Bika: <p><a class="mention" href="/u/cbalioglu">@cbalioglu</a> can you recommend some resources please to learn how to efficiently apply DDP to my large model using multiple gpus ?</p>

J_Johnson: <p>Check here:<br>
<a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/intermediate/ddp_tutorial.html</a></p>
Assert boxes.size(1) == 5, 'The boxes tensor shape is not correct as Tensor[K, 5]' AssertionError: The boxes tensor shape is not correct as Tensor[K, 5]

nbansal90: <p>Hello All,</p>
<p>I am getting the following error while using the <code>from torchvision.ops import roi_align</code> function in Pytorch, while using the function <code>roi_align()</code></p>
<pre><code class="lang-auto">assert boxes.size(1) == 5, 'The boxes tensor shape is not correct as 
Tensor[K, 5]' 
AssertionError: The boxes tensor shape is not correct as Tensor[K, 5]
</code></pre>
<p>I went through the Pytorch doc and see that it required for it to be of dimension 5, where the 1st column represents the <code>batch index</code>.</p>
<pre><code class="lang-auto">boxes (Tensor[K, 5] or List[Tensor[L, 4]]): the box coordinates in (x1, y1, 
x2, y2) format where the regions will be taken from. If a single Tensor is 
passed,then the first column should contain the batch index. If a list of 
Tensors is passed, then each Tensor will correspond to the boxes for 
an element in a batch
</code></pre>
<p>I am not quite sure what does the <code>batch index</code> mean here . In my training case, I have set the <code>Batch Size</code> to be 1 and size of boxes which is currently being passed is <code>torch.Size([66, 4])</code> (where 66 is something which would vary on each batch.)</p>
<p>Thank you!</p>

Apurv25: <p>Hi, did you find solution to your problem.</p>
Confusion about pytorch dropout

BruceDai003: <p>It seems that if I passed in a 2d tensor, say (32, 64). Here my batch_size = 32, input_channels = 64. And I pass it to the torch.nn.Dropout() API with dropout_probability = 0.5, each of the element will get a p=0.5 probability to be zeroed out.<br>
But could this be modified to let for example, along the 1st dimension or 2nd dimension, independently. For example, [0, :] will be either 0 or unchanged with p = 0.5?</p>
Optimize a transformed Embedding with sparse gradients

Georgios_Kopanas: <p>Hi,</p>
<p>I am trying to use SparseAdam. To be able to do that I am storing all my parameters in embedding modules. So far so good, but before I need to access some of the data using a LongTensor, I need to apply a transformation to my Embedding weights. In a more concrete and isolated algorithmic example I need to do something like this:</p>
<pre><code class="lang-auto">from torch import nn
import torch

a = nn.Embedding(num_embeddings=1000, embedding_dim=3, sparse=True)
b = nn.Embedding(num_embeddings=1000, embedding_dim=3, sparse=True)

gt = torch.ones((3, 3))

optimizer_xyz_diffuse = torch.optim.SparseAdam([a.weight] + [b.weight], lr=0.0001)


while True:
    idxa = torch.tensor([1, 2, 3])
    idxb = torch.tensor([4, 5, 6])
    all_idxs = torch.linspace(0, 999, 1000, dtype=torch.int32)
    a2 = 4*a(all_idxs)
    loss = (a2[:3]+b(idxb) - gt).mean()

    loss.backward()

    optimizer_xyz_diffuse.step()
    optimizer_xyz_diffuse.zero_grad()
</code></pre>
<p>As you probably guess from the example the gradients of a is using all the values. Is there a proper way to do this? I tried to access the weights directly but this immediately changes my gradient tensor to non-sparse.</p>
<p>In the real example, the transformation is more complicated than a scalar – it’s an MLP – and will influence the indices so it cannot be applied post-filtering.</p>
<p>All the best,<br>
Georgios Kopanas</p>
Pytorch float16-model failed in running

FengMu1995: <p>return torch._C._nn.upsample_bilinear2d(input, output_size, align_corners, scale_factors)<br>
RuntimeError: “compute_indices_weights_linear” not implemented for ‘Half’，<br>
pytorch1.9.1 did not support float16？</p>

ptrblck: <p>You could use <code>float16</code> on a GPU, but not all operations for <code>float16</code> are supported on the CPU as the performance wouldn’t benefit from it (if I’m not mistaken).</p>

FengMu1995: <p>Hi，<br>
I try running on gpu，Successfully. Thank you very much.</p>

FengMu1995: <p>Howerver,  I still have a problem with model-int8 on cpu and gpu，</p>
<p>“RuntimeError: “upsample_bilinear2d_out_frame” not implemented for ‘Char’ ”</p>

ptrblck: <p><code>int8</code> is not implemented in native operations and you would need to use the quantization util. for it.</p>

FengMu1995: <p>Could you explain it more specifically, please？ I haven’t understood the quantization util.</p>

ptrblck: <p>I would probably start with the <a href="https://pytorch.org/docs/stable/quantization.html">docs</a> and then take a look at <a href="https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html">this tutorial</a> and the <a href="https://pytorch.org/docs/stable/quantization-support.html">coverage</a> for more information.</p>

FengMu1995: <p>ok, I have known where the problem is. I should add the quant(x) into the forward headmost.</p>
Divergence after adding manual features before fully-connected layer

freshman: <p>Greetings,<br>
I’m training a ResNet, which could classify ECG signals.The input is 1-D signals containing 300 sample points. When I use just ResNet, the results are convergenece. Then I try to add manual features before FC, concatenanting 256 ResNet features and 14 manual features. The result is divergence.How can I solve this problem? Should I modify manual features’ order of magnitude?<br>
This my structure:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/7/07fc6088ecfb03ce9067ec4245dd4f6b52a9e6bf.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/07fc6088ecfb03ce9067ec4245dd4f6b52a9e6bf" title="structure"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/7/07fc6088ecfb03ce9067ec4245dd4f6b52a9e6bf_2_690x425.jpeg" alt="structure" data-base62-sha1="18E3zQGw2p57WpokfEX18qZtxxl" width="690" height="425" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/7/07fc6088ecfb03ce9067ec4245dd4f6b52a9e6bf_2_690x425.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/0/7/07fc6088ecfb03ce9067ec4245dd4f6b52a9e6bf_2_1035x637.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/0/7/07fc6088ecfb03ce9067ec4245dd4f6b52a9e6bf_2_1380x850.jpeg 2x" data-dominant-color="F9F8F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">structure</span><span class="informations">1869×1152 159 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Libtorch set default type

RuiZhangJerry: <p>hi，I’m constructing a neural network using libtorch, how can I set the default data type to the Double?</p>

ptrblck: <p>I think the proper approach would be to use the <code>AutoDefaultDtypeMode</code> RAII in <code>libtorch</code> via:</p>
<pre><code class="lang-python">AutoDefaultDtypeMode dtype_mode(default_dtype);

auto tensor = torch::tensor(1.f); // should be created in the default_dtype
</code></pre>
<p>where <code>default_dtype</code> is <code>c10::ScalarType</code>.</p>

RuiZhangJerry: <p>Thank you very much！I will try it right away，haha</p>
Is there a downside to loading the pre-training model？

MartinZhang: <p>Is there a downside to loading the pre-training model？</p>
<p>When we are training the model， Load the pre-training model can get the following advantage:</p>
<ol>
<li>Accelerated training, and can use less training epoch</li>
<li>Can avoid getting caught up in local optima or saddle points</li>
</ol>
<p>Is it possible to say that all training can depend on the pretraining model？</p>
<p>Is there a downside to loading the pre-training model？</p>
What is the reason for GPU inefficiency？

MartinZhang: <p>Today，I trained a model on the GPU.</p>
<p>When I check the GPU, the utilization rate is 20%-60%. And it’s not stable.</p>
<p>Please give me some advise that I can check them one by one.</p>
<p>Thanks a lot. <img src="https://discuss.pytorch.org/images/emoji/apple/handshake.png?v=10" title=":handshake:" class="emoji" alt=":handshake:"></p>

Andrea_Rosasco: <p>One reason might be that your mini-batch size is too small. If you increase the batch size the utilization should increase as well.</p>
<p>Mind that varying the mini-batch size can affect generalization and increasing it decreases the number of optimization steps you are doing in an epoch.</p>
<p>Another reason might be a slow preprocessing of the data. In that case, you can increase the data loader number of workers or try to speed up your preprocessing.</p>

MartinZhang: <p>Thank you for your adivce. I will try it. <a class="mention" href="/u/andrea_rosasco">@Andrea_Rosasco</a> <img src="https://discuss.pytorch.org/images/emoji/apple/handshake.png?v=10" title=":handshake:" class="emoji" alt=":handshake:"></p>
What is the difference between 1x1 con and avergaepool1d?

sid_1289: <p>what is the difference between 1x1 conv and avergaepool1d ?</p>

ptrblck: <p>A <code>1x1</code> conv layer will use a kernel size of <code>1x1</code> (i.e. a single pixel) and would act as a linear layer in the default setup. The <code>nn.AdaptiveAvgPool1d</code> layer works on 3D inputs in the shape <code>[batch_size, channels, seq_len]</code> and returns a desired output shape in the temporal dimension by applying an adaptive windowing approach to calculate the average of the input values.</p>
Normalization while fine tuning a trained model

bibekx: <p>Consider a scenario where I want to fine tune a trained model for a downstream task.</p>
<p>The trained model used [0, 1] normalization during its training. Now, is it okay if I do a z-score normalization [ input = (input - mean) / std ] while fine tuning?</p>
<p>I am not sure how to make sense of this scenario. Please help.</p>

ptrblck: <p>I would guess your initial loss might be a bit higher using this new normalization approach, as your model would get “new” samples. Fine tuning the model might work, but again I would guess that you might get better results if you fine tune the entire model (not only the last layer(s)). These are just my guesses, so I would recommend to run some experiments and compare both approaches (and please share the results if you plan on running these tests! <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=10" title=":wink:" class="emoji" alt=":wink:"> ).</p>
How To Choose Correct PyTorch Install in Python setup.py File?

Rohin_Dasari: <p>Hello, I am working on a Python package that uses Pytorch as a dependency. In the <code>setup.py</code> file for the package, <code>torch</code> is listed as a dependency. Currently, the code looks something like this:</p>
<pre><code class="lang-python">from distutils.core import setup

setup(
    name='cool package',
    description='A really cool package',
    author='me',
    url='https://my_cool_website.com',
    packages=['my_cool_package'],
    python_requires='&gt;=3.7.10',
    install_requires=[
            'torch'
        ]
)
</code></pre>
<p>Currently, this will install the CPU compatible version of Pytorch. In the installation instructions on the Pytorch homepage, however, the following command is listed as the installation instructions to install Pytorch with GPU support for Cuda 10.2:</p>
<pre><code class="lang-auto">pip3 install torch==1.10.1+cu102 torchvision==0.11.2+cu102 torchaudio===0.10.1+cu102 -f https://download.pytorch.org/whl/cu102/torch_stable.html
</code></pre>
<p>Is there a way to make sure that I install this version of Pytorch in the <code>setup.py</code> file if the user has GPU support on their machine?</p>
How can I resume training of a network that was amended by torch.fx

shengchunnan: <p>The newly introduced package <code>torch.fx</code> is quite useful for pruning and architectural search of networks. However, I have met problems resuming the training when the training job is stopped unexpectedly because the directed graph of this network has already changed. For example, a convolution layer was removed from the directed graph and it should get bypassed in the resumed job, however, this layer is still used in <code>nn.Module.forward()</code> in the python script. In other words, the model checkpoint saved by <code>torch.save()</code> may not match the original model definition.</p>
<p>The checkpoint does not include directed graph information so you have to continue to depend on the python definition, unless you save the checkpoint using <code>torch.jit.trace()</code> or <code>torch.jit.script()</code>. But torchscript would not support ‘torch.fx’ any longer because it seems graph of torchscript is not changeable.</p>
<p>Are there good suggestions to resume the training of a network whose graph is already changed?</p>
Conv1d results depend on length of input data

horsti371: <p>Hi there,</p>
<p>I get different results from Conv1d when using sub-length of data.<br>
A simple example:</p>
<pre><code class="lang-auto">conv = torch.nn.Conv1d(40, 30, 3, stride=1, padding=1, dilation=1)
test_data = torch.randn((1, 40, 705), dtype=torch.float32)
</code></pre>
<p>I expect to get the same results passing all data vs. sub-length of data (of course, only for the sub-length results).<br>
But when I compute the following results</p>
<pre><code class="lang-auto">r1 = conv(test_data)
r2 = conv(test_data[:,:,:405])
r3 = conv(test_data[:,:,:603])
</code></pre>
<p>they differ - I only compare a sub-length of 30 for the results:</p>
<pre><code class="lang-auto">(r1[:,:,:30] == r2[:,:,:30]).all()
&gt;&gt;&gt; tensor(False)
(r1[:,:,:30] == r3[:,:,:30]).all()
&gt;&gt;&gt; tensor(True)
</code></pre>
<p>What am I missing? I expect all results to be the same, but they differ for a sub-length of 405, but not for a larger sub-length of 603.</p>

ptrblck: <p>You are most likely seeing small abs. errors due to the limited floating point precision and can’t expect to see bitwise equal values if the order of operations is not fixed (or if determinism is set otherwise).<br>
You could see the same limitation by comparing the results of:</p>
<pre><code class="lang-python">x = torch.randn(100, 100, 100)
s1 = x.sum()
s2 = x.sum(0).sum(0).sum()
print((s1 - s2).abs().max())
&gt; tensor(0.0002)
</code></pre>

J_Johnson: <p>You can check the mean distance between the points via:</p>
<pre><code class="lang-auto">print(torch.mean((r1[:,:,:30]-r2[:,:,:30])**2)**0.5)
</code></pre>
<p>It’s fairly small. Something like e-06 to e-08.</p>
<p>The actual algorithm used to calculate convolutions is not from Pytorch but from 3rd party libraries, depending on cpu or gpu, etc.</p>
<p>Keep in mind that torch.float32 has only 5 points of precision. (i.e. 0.4322). So the mean difference falls below the rounding error threshold.</p>
<p>Additionally, in Machine Learning applications, you generally do not need such high precision. Even half precision is sufficient.</p>
Weird behavior when rying to export fp16 model to onnx

sc21: <p>Hi,</p>
<p>I am hoping to use a fp16 model for inference (from a model trained with fp32). I tried to use apex.amp to convert the pegasus-xum model from huggingface to fp16. At least the encoder part succeeds, but I am unable to convert it to ONNX. It prints out lots of numbers like the info below:</p>
<p>-0.3096  -0.0386  -0.2583   0.1227  -0.0636  -0.6738   0.1021  -0.6738<br>
-0.4084   1.1182  -0.4387  -0.6660  -0.3435   0.5137  -1.0996  -0.6328<br>
[ torch.cuda.HalfTensor{96103,1024} ]</p>
<p>But no onnx model was created in the path I specified.</p>
<p>However when I use the same code to convert the original model (fp32) to onnx it works. Any ideas why this happens?</p>
<p>Detailed code:</p>
<blockquote>
<p>model_name=“google/pegasus-xsum”<br>
pegasus_model = PegasusForConditionalGeneration.from_pretrained(model_name)<br>
pegasus_to_convert = copy.deepcopy(pegasus_model)<br>
device = torch.device(“cuda”) if torch.cuda.is_available() else “CPU” <span class="hashtag-raw">#device</span> = “cuda”<br>
pegasus_to_convert.to(device)<br>
optimizer = AdamW(pegasus_model.parameters())<br>
converted_o2, optimizer = apex.amp.initialize(pegasus_to_convert, optimizer, opt_level=“O2”)<br>
<span class="hashtag-raw">#for</span> name, param in converted_o2.named_parameters():<br>
#    print(name,param.data.dtype)<br>
tokenizer = PegasusTokenizer.from_pretrained(model_name)<br>
dummy_input = tokenizer(“This is an amazing sentence.”, return_tensors=‘pt’).to(device)<br>
converted_encoder = converted_o2.model.encoder<br>
ori_encoder = pegasus_model.model.encoder.to(device)<br>
output_converted_encoder_path = “xx”<br>
print(“Exporting to ONNX:”)<br>
//If we export ori_encoder this works<br>
export_encoder(ori_encoder, dummy_input[‘input_ids’], output_converted_encoder_path)<br>
// if we export the converted encoder, it does not work.<br>
export_encoder(converted_encoder, dummy_input[‘input_ids’], output_converted_encoder_path)</p>
</blockquote>
<p>The pytorch I use is version 1.10.1+cu113. The GPU is a10g with cuda version 11.2.</p>
Libtorch: reset dataloader result in segfault (trying to get fresh iterator)

jackshi: <p>In libtorch, I want to get a new iterator from the beginning of the DataLoader before consuming the previous one. I do this because I want to test on the same data over and over again so I don’t have to worry about test data sample variance.</p>
<p>If I simply call <code>current_iterator = data_loader_-&gt;begin();</code> I get this error:</p>
<pre><code class="lang-auto">Attempted to get a new DataLoader iterator while another iterator is not yet exhausted.

pytorch/torch/csrc/api/include/torch/data/dataloader/base.h: 60
</code></pre>
<p>If I call <code>data_loader_-&gt;reset()</code> before this reassignment I get a segfault.</p>
<p>What do I gotta do to get a fresh iterator?</p>

jackshi: <p>Right now the work around I use is to reconstruct the data loader every time I need a fresh iterator. Given that the DataLoader class as a reset method and getting a fresh iterator probably shouldn’t involve this much work, this is still an open issue for me.</p>
Is it possible to create these probability distributions using torch.distributions?

ProGamerGov: <p>The two distributions are as follows:</p>
<pre><code class="lang-auto">distribution_1 = [0.995**n for n in range(-5,80)] + [0.998**n for n in 2*list(range(20,40))]
# max value is: 1.0253794191470573 &amp; min is: 0.6730128848950394
</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/4/f44cd518eec6c08f3b2d83d912819dade6d9ee1c.png" alt="distribution_1" data-base62-sha1="yRb79SjvGdjO5jJ7XEWyXNzaC8k" width="378" height="248"></p>
<pre><code class="lang-auto">distribution_2 = list(range(-20,20))+list(range(-10,10))+list(range(-5,5))+5*[0]
# max value is: 19 and min value is: -20
</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/4/b44a523b747480a03142785db4d65abd065a511f.png" alt="distribution_2" data-base62-sha1="pIVaZ072nPKAIsPSn16lRRSG4k7" width="377" height="248"></p>
<p>I’m not that familiar with the PyTorch probability distributions: <a href="https://pytorch.org/docs/stable/distributions.html#exponential" class="inline-onebox" rel="noopener nofollow ugc">Probability distributions - torch.distributions — PyTorch 2.1 documentation</a>, so I don’t know if it’s possible get similar distributions using it.</p>
Public Art Sculpture Collaboration Opportunity

Twcochran: <p>Hello,</p>
<p>My name is Tyler Cochran, I’m a sculpture artist working in Seattle, WA USA, with the John Grade Sculpture organization, a collective of artists who make large scale public art installation sculptures in the United States, and abroad. Some of our work can be seen at our website: <a href="http://www.johngrade.com" rel="noopener nofollow ugc">www.johngrade.com</a>.</p>
<p>We are beginning the initial planning stage for a large (approximately 60 feet in height) project to be permanently installed in Facebook’s (Meta’s) new high rise construction in downtown Seattle, WA.</p>
<p>Our intention is to produce a sculpture in wood that represents a waterfall, whose visual aspects are hybridized with those of water streams affected by 40hz sound waves, similar to this: <a href="https://www.youtube.com/watch?v=uENITui5_jU" class="inline-onebox" rel="noopener nofollow ugc">Amazing Water &amp; Sound Experiment #2 - YouTube</a></p>
<p>We are also interested in exploring possibilities to utilize some of our digital assets as NFTs. Currently, our viewers and followers have no means by which to contribute or obtain ownership of our work, the scale and cost make that prohibitive. We believe there are important possibilities to explore in this area: to create greater impact for our viewers, broaden our audience, increase viewer involvement, and to generate content with less environmental impact. This also has implications for future VR and AR interactions.</p>
<p>We are hoping to utilize machine learning and PyTorch to help us create, combine, and then 3D model this sculpture, but we lack expertise in these areas. Our hope is that we may be able to find a collaborator with this kind of expertise, and possibly use it as an opportunity to deepen our partnership with Meta, helping one another to more fully realize some of our goals as they relate to this project. This is an opportunity for exposure to the realm of art and creativity, as well as to be a contributing member in a large public sculpture installation with a large audience, and meaningful presence.</p>
<p>I’m looking forward to seeing what can come of this opportunity, thanks for your time and consideration.</p>
<p>Tyler</p>
How to use different number of gpus on 2 nodes?

TheShadow29: <p>How do I train on two machines, one with 4 gpus and one with 8 gpus? I find that the documentation of torch elastic run (<a href="https://pytorch.org/docs/stable/elastic/run.html" class="inline-onebox" rel="noopener nofollow ugc">torchrun (Elastic Launch) — PyTorch 1.10.0 documentation</a>) suggests</p>
<p>" 4. This module only supports homogeneous <code>LOCAL_WORLD_SIZE</code>. That is, it is assumed that all nodes run the same number of local workers (per role)."</p>
<p>Thanks!</p>

cbalioglu: <p>cc <a class="mention" href="/u/kiuk_chung">@Kiuk_Chung</a></p>
<p>As of today DDP does not officially support running jobs with different number of GPUs on different machines. One workaround might be to use the <code>CUDA_VISIBLE_DEVICES</code> environment variable to split 8 GPUs into two sets and then launch two DDP processes.</p>
Encoder-decoder text summarisation

Leo2: <p>Hi<br>
Does anybody have experience with encoder decoder architectures?<br>
I am doing a project on abstractive text summarisation and i’ve run into an issue where the decoder <strong>always</strong> predicts the end of sequence token when using greedy selection (the highest probability is always the eos token). This is made seemingly worse when calculating attention.</p>
<p>I understand that beam search is generally used to construct the most likely sentence but should the highest probability ALWAYS be the same eos token? is this something that is common and has a common cause?</p>
<p>regards<br>
leo</p>
Shape of output becomes wrong with nn.DataParallel

jungminc88: <p>My model takes two batched sequences (along with their attention masks) and is supposed to return a tensor of shape (batch_size, batch_size) whose (i,j)-element should be cosine similarity of the i-th element of the first (b_que_iis[j]) and the j-th element of the second (b_art_iis[j])</p>
<p>I ran this with batch_size 16 on a machine with 4 GPUs, using nn.DataParallel, so the batch_size becomes 4 on each GPU.<br>
Each GPU returns a 4 by 4 tensor as a result.<br>
However, the final output of the model<br>
<code>sims = model(b_que_iis, b_que_ams, b_art_iis, b_art_ams)</code><br>
return a tensor of shape (16,4).</p>
<p>How should I solve this problem and get the desired output of (16, 16) tensor?<br>
I would greatly appreciate any comment.</p>
<p>Thank you!</p>
<pre><code class="lang-auto">class QA_match(nn.Module):
    def __init__(self, bert_model):
        super(QA_match, self).__init__()
        self.bert = bert_model
    def forward(self, b_que_iis, b_que_ams, b_art_iis, b_art_ams):
        query_cls = self.bert(input_ids=b_art_iis, 
                             attention_mask=b_art_ams).last_hidden_state[:,0,:]
        article_cls = self.bert(input_ids=b_que_iis, 
                             attention_mask=b_que_ams).last_hidden_state[:,0,:]
        query_cls = nn.functional.normalize(query_cls, dim=1)
        article_cls = nn.functional.normalize(article_cls, dim=1)

        sim = torch.einsum('ij, kj -&gt; ik', query_cls, article_cls) #take the dot product
        return sim
</code></pre>

ptrblck: <aside class="quote no-group" data-username="jungminc88" data-post="1" data-topic="139827">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jungminc88/48/37802_2.png" class="avatar"> jungminc88:</div>
<blockquote>
<p>Each GPU returns a 4 by 4 tensor as a result.<br>
However, the final output of the model<br>
<code>sims = model(b_que_iis, b_que_ams, b_art_iis, b_art_ams)</code><br>
return a tensor of shape (16,4).</p>
</blockquote>
</aside>
<p>This is expected, since the result will be concatenated in <code>dim0</code> again.<br>
Each device returns <code>4x4=16</code> values and thus the result of 4 devices is <code>4*4*4=16*4=64</code>.</p>
<aside class="quote no-group" data-username="jungminc88" data-post="1" data-topic="139827">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jungminc88/48/37802_2.png" class="avatar"> jungminc88:</div>
<blockquote>
<p>How should I solve this problem and get the desired output of (16, 16) tensor?</p>
</blockquote>
</aside>
<p>Assuming the additional values are created as function between the samples in a batch, you could try to calculate them after <code>nn.DataParallel</code> returned the output.</p>

jungminc88: <p>Ah I see. So I should do the einsum outside the model.<br>
Thank you very much!</p>
Is ignore_index necessary with the pack_padded_sequence

jpzhou: <p>When coping with variable lengths of the input sequence, if we use pack_padded_sequence, do we still need to set the <code>ignore_index</code> parameter of the loss function to get rid of calculating the gradient of the padding element?</p>
<p>For example, in the <a href="https://github.com/yunjey/pytorch-tutorial/tree/master/tutorials/03-advanced/image_captioning" rel="nofollow noopener">image captioning PyTorch tutorial</a>, the forward method of the DecoderRNN is</p>
<pre><code class="lang-auto">        embeddings = self.embed(captions)
        embeddings = torch.cat((features.unsqueeze(1), embeddings), 1)
        packed = pack_padded_sequence(embeddings, lengths, batch_first=True) 
        hiddens, _ = self.lstm(packed)
        outputs = self.linear(hiddens[0])
</code></pre>
<p>But during the training, it directly use the</p>
<pre><code class="lang-auto">            # Set mini-batch dataset
            images = images.to(device)
            captions = captions.to(device)
            targets = pack_padded_sequence(captions, lengths, batch_first=True)[0]
            
            # Forward, backward and optimize
            features = encoder(images)
            outputs = decoder(features, captions, lengths)
            loss = criterion(outputs, targets)
            decoder.zero_grad()
            encoder.zero_grad()
            loss.backward()
            optimizer.step()
</code></pre>
<p>My questions are:</p>
<p>Q1: If we use <code>pack_padded_sequence</code> to pack the sequence with padding, it seems that it is not necessary to set <code>ignore_index</code> in the loss function. However, if we then use <code>pad_packed_sequence</code> to unpack the result returned by RNN, do we need to set following <code>ignore_index</code> in the loss function?</p>
<p>Q2: How can I get the answer of Q1 by writing a toy program, such as printing the gradients of each element? Which element should I print out if I want to figure out whether the gradient of the padding element BP or not?</p>
<p>Thank you so much!</p>

jchillin: <p>Any updates on this?</p>

Ran: <p>Any updates on this?</p>
Libtorch loss.backward(); C10 Error

zheng_wang: <p>Hello, my environment is win10 vs2019. After I built the network structure using libtorch, forward got the output result and loss function. But when I lost.backward, there was a Microsoft C++ exception: c10::Error, located at memory location 0x000000F11C4FE260. How to solve this problem.</p>
<pre><code class="lang-auto">net-&gt;train();
	size_t index = 0;
	float Loss = 0.0;
	for (auto&amp; batch : loader) {
		torch::Tensor input = batch.data.to(device);
		torch::Tensor targets = batch.target.to(device);
		torch::Tensor output, loss;
		optimizer.zero_grad();
/*		std::cout &lt;&lt; targets;*/
		std::tie(output, loss) = net-&gt;forward(input, targets);
		std::cout &lt;&lt; output;
		std::cout &lt;&lt; loss;
// 		input.print();
// 		targets.print();
// 		output.print();
// 		loss.print();
		loss.backward();
		optimizer.step();
		Loss += loss.item&lt;float&gt;();
</code></pre>
<p>The error message is as follows</p>
<p><strong>0x00007FFA17833B29 处(位于 Project.exe 中)有未经处理的异常: Microsoft C++ 异常: c10::Error，位于内存位置 0x00000040FE0FE410 处。</strong></p>
<p>The print log is as follows</p>
<pre><code class="lang-auto">loading cfg &amp; creating modules ...
train start ...
torch::nn::Sequential(
  (0): torch::nn::Conv2d(3, 96, kernel_size=[11, 11], stride=[4, 4])
  (1): torch::nn::Functional()
) torch::nn::Sequential(
  (0): MaxPoolLayer2D
) torch::nn::Sequential(
  (0): torch::nn::Conv2d(96, 256, kernel_size=[5, 5], stride=[1, 1], padding=[2, 2])
  (1): torch::nn::Functional()
) torch::nn::Sequential(
  (0): MaxPoolLayer2D
) torch::nn::Sequential(
  (0): torch::nn::Conv2d(256, 384, kernel_size=[3, 3], stride=[1, 1], padding=[1, 1])
  (1): torch::nn::Functional()
) torch::nn::Sequential(
  (0): torch::nn::Conv2d(384, 384, kernel_size=[3, 3], stride=[1, 1], padding=[1, 1])
  (1): torch::nn::Functional()
) torch::nn::Sequential(
  (0): torch::nn::Conv2d(384, 256, kernel_size=[3, 3], stride=[1, 1], padding=[1, 1])
  (1): torch::nn::Functional()
) torch::nn::Sequential(
  (0): MaxPoolLayer2D
) torch::nn::Sequential(
  (0): Flatten
  (1): torch::nn::Linear(in_features=9216, out_features=4096, bias=true)
  (2): torch::nn::Functional()
) torch::nn::Sequential(
  (0): torch::nn::Dropout2d(p=0, inplace=false)
) torch::nn::Sequential(
  (0): torch::nn::Linear(in_features=4096, out_features=4096, bias=true)
  (1): torch::nn::Functional()
) torch::nn::Sequential(
  (0): torch::nn::Dropout2d(p=0, inplace=false)
) torch::nn::Sequential(
  (0): torch::nn::Linear(in_features=4096, out_features=2, bias=true)
  (1): EmptyLayer
) torch::nn::Sequential(
  (0): torch::nn::Softmax(dim=1)
) torch::nn::Sequential(
  (0): Loss_Function
) 0.4967  0.5033
[ CUDAFloatType{1,2} ]0.250011
[ CUDAFloatType{} ]
</code></pre>

Matej_Kompanek: <p>You could try wrapping your code with try catch block to see the exact message</p>
<pre><code class="lang-auto">try
{
... your code
}
catch (const c10::Error&amp; e)
{
	std::cout &lt;&lt; e.msg() &lt;&lt; std::endl;
}
</code></pre>

zheng_wang: <p>thank you very much!</p>

wwshu: <p>Thank u very much! Help me a lot!</p>
How to concatenate tensors in distributed multi-node setup?

JimW: <p>I am trying to implement something like this for 2 nodes (each node with 2 GPUs):</p>
<pre><code class="lang-auto">#### Parallel process initiated with torch.distributed.init_process_group()
### All GPUs work in parallel, and generate lists like :
    [20, 0, 1, 17] for GPU0 of node A 
    [1, 2, 3, 4] for GPU1 of node A
    [5, 6, 7, 8] for GPU0 of node B
    [0, 2, 4, 6] for GPU1 of node B
I tried torch.distributed.reduce() to get a sum of these 4:   [26,  10, 15, 35]
</code></pre>
<p>But what I want is a concatenated version like this [[20, 0, 1, 17], [1, 2, 3, 4] , [5, 6, 7, 8] , [0, 2, 4, 6]]<br>
Or [20, 0, 1, 17, 1, 2, 3, 4, 5, 6, 7, 8, 0, 2, 4, 6] is also OK with me.</p>
<p>Is it possible to achieve this from torch.distributed?</p>

JimW: <p>Thanks for checking this out. I believe this will be a very useful example for distributed operations of pytorch : )</p>
Am I using Buffers correctly?

lone-duck: <p>I am working on a project in Pyro, and am basing my code on their <a href="http://pyro.ai/examples/dmm.html" rel="noopener nofollow ugc">DMM example</a>. In their case they are doing model learning, so their initial hidden state is initialized as</p>
<pre><code class="lang-auto">self.z_0 = nn.Parameter(torch.zeros(z_dim))
</code></pre>
<p>which means that <code>z_0</code> is both learnable and will be transferred to GPU if the whole DMM is. In my case I am not doing model learning, so want a fixed <code>z_0</code>, but also want it to be transferred to the device. Some searching has led me to believe that initializing <code>z_0</code> as a <code>Buffer</code> achieves this, but I’m not sure if there are some side effects or another language feature which is more suited to my use case?</p>
<p>Thanks!</p>
How to reduce runtime?

kahroba: <p>Hi. I am training a neural network in the field of natural language processing.<br>
To reduce runtime, I’m going to change the type of tensors from torch.long to torch.int32. Do you think that might be helpful?<br>
Do you have another suggestion?</p>

ptrblck: <p>I wouldn’t think you will see a significant performance benefit assuming your code would work with <code>int32</code>. Usually integer tensors are used e.g. as the target values for <code>nn.CrossEntropyLoss</code>, for indexing etc. Since these targets or inputs are usually tiny compared to the model parameters as well as intermediate activations, the memory saving would also be in the same ballpark. To accelerate your model I would recommend to check <a href="https://pytorch.org/docs/stable/notes/amp_examples.html#amp-examples">mixed-precision training</a> as well as the <a href="https://pytorch.org/tutorials/recipes/recipes/tuning_guide.html">performance guide</a>.</p>
Module in a forloop gives an RuntimeError: Unable to find a valid cuDNN algorithm to run convolution

C_J: <p>I combined traditional UNET module + my own custom function in my forward model. For some reason, I can’t give data as a batch tensor, but I need to give one by one using forloop.</p>
<pre><code class="lang-auto">____________________________________________
optvars = [{'params': custom_var, 'lr':lr_custom}]
optimizer1 = optim.Adam(optvars)
optimizer2 = optim.Adam(UNET.parameters(), lr=lr_UNET)

for a in range(batch_size)
     input_data=batch[a,:,:]
     temp_data=custom_function(input_data, custom_var)
     output_data=UNET(temp_data)
     loss_total=loss_total + loss(output_data)

loss_total.backward()  #  &lt;-----where the error occur
custom_var.retain_grad() # &lt;----- I think I need something like this for  UNET parameters...
optimizer1.step()
optimizer2.step()
____________________________________
</code></pre>
<p>I get this error:<br>
RuntimeError: Unable to find a valid cuDNN algorithm to run convolution</p>
<p>It works when batch_size=1, but gives an error when batch_size&gt;=2.<br>
Also, when I don’t use UNET module in the forloop (only using custom_function), it works.<br>
I think it’s something to do with using module more than twice before backward propagation, but I’m not sure what’s the problem.<br>
Can anyone help please?</p>
<p>Thanks!</p>

ptrblck: <p>Which PyTorch version are you using with which CUDA/cuDNN releases?<br>
This error might be raised, if no workspace can be allocated due to a large memory usage.</p>

C_J: <p>I’m using 1.9.1 + cu111<br>
Now it does look like a memory shortage in allocation for sure, although the message doesn’t say so.</p>
UNET Module in a forloop gives an error when .backward() (RuntimeError: Unable to find a valid cuDNN algorithm to run convolution)

C_J: <p>I combined conventional unet module and a custom function in a forward model.<br>
This custom function cannot take input as a batch, so I use forloop to feed the data one by one.<br>
However, it gives me an error when batchsize &gt;=2 (works when =1).<br>
(It also works for &gt;=2 when I only use custom_func, without UNET.)<br>
Therefore I think I’m doing something wrong with evaluating the module multiple times in a single forward process.<br>
Can anyone help please? Thanks!!</p>
<pre><code class="lang-auto">______________________________
model_UNET = UNET()  # conventional unet
optvars = [{'params': custom_vars, 'lr':lr_custom}] # custom_vars used in custom_func
optimizer1 = optim.Adam(optvars)
optimizer2 = optim.Adam(UNET.parameters(), lr=lr_Unet)

# My forward model
for aa in range(batch_size):
     input=batch[aa,:,:]
     temp=custom_func(input, custom_vars)
     output=model_UNET(temp)
     loss_total=loss_total + loss(output)

# backward &amp; update
custom_vars.retain_grad() #&lt;-- Maybe I need something like this for parameters in UNET?
loss_total.backward() #&lt;-- This line gives the error
optimizer1.step()
optimizer2.step()
___________________________
</code></pre>

C_J: <p>Actually, when I changed the network size smaller, it worked.<br>
Maybe it was just a GPU memory problem?<br>
I still have a question:  Can I make my forward model using forloop like this, not batch tensor format?</p>
<p>Thanks</p>

ptrblck: <p>Is this a double post from <a href="https://discuss.pytorch.org/t/module-in-a-forloop-gives-an-runtimeerror-unable-to-find-a-valid-cudnn-algorithm-to-run-convolution/139797">here</a> or a new issue?</p>

C_J: <p>It’s a double post. I re-uploaded my question because the system blocked my original thread. Thanks</p>
Running_mean and running_var of batch_norm are not correctly exported to onnx

Ardeal: <p>Hi,<br>
I am using the following code to export resnet18 pth file to onnx file.<br>
I found that running_mean and running_var of batch_norm in onnx are different with those of pth state_dict.</p>
<p>Is there anything wrogn with my exporing code? or is there any reason why running_mean and running_var of batch_norm are not correctly exported?</p>
<pre><code class="lang-python">torch.onnx.export(resnet18, dummy_input, onnx_path, opset_version=11, training=torch.onnx.TrainingMode.TRAINING, do_constant_folding=False)
</code></pre>

ptrblck: <p>You might be ignoring a warning raised <a href="https://github.com/pytorch/pytorch/blob/9d3a6fa6233a228529205f08bf35fa6820942ff1/torch/onnx/utils.py#L58-L64">here</a>, which explains:</p>
<blockquote>
<p>UserWarning: You are exporting the model in training mode with onnx opset version 11. Opset versions lower than opset 12 will not be able to export nodes such as Dropout and BatchNorm correctly.</p>
</blockquote>
Document typo in the CrossEntropyLoss documentation page

son_of_Jesus: <p>Hi guys, I found a typo in the CrossEntropyLoss documentation website.<br>
<a href="https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html?highlight=cross%20entropy%20loss" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html?highlight=cross%20entropy%20loss</a><br>
the denominator component of the 3rd formula in the page should read Sum of exponents as opposed to Exponents of the sum. This is to match with the SoftMax.<br>
I verified it with some code, happy to provide it.<br>
Regards,<br>
Javier</p>

ptrblck: <p>Thanks for letting us know! Would you be interested in providing the fix for it? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

son_of_Jesus: <p>Hi ptrblck,</p>
<p>You are a legend in the forum and the mere fact that you are replying to me is the coolest thing this year!</p>
<p>I’m happy to provide a fix, the question is what’s inaccurate, the formula in the documents or the code?</p>
<p>If the problem is in the formula, then it’s a simple fix. But if it’s in the code, would you send me a link to the code where the Cross entropy is calculated, please?</p>
<p>Kind regards,</p>
<p>Javier</p>

ptrblck: <p>Ha, thanks for the kind words!<br>
I’ve took a quick look at the formula and think that the docs are wrong as it seems the <code>log_softmax</code> formula (<code>log((exp(x_i))/sum_j(exp(x_j)))</code>) mixed up the <code>exp</code> and <code>sum</code>.<br>
The docs are defined <a href="https://github.com/pytorch/pytorch/blob/a42345adee3eb32a6be3eb6863cac886ed8f795f/torch/nn/modules/loss.py#L1057-L1063">here</a> and you can check the used formula <a href="https://github.com/pytorch/pytorch/blob/a42345adee3eb32a6be3eb6863cac886ed8f795f/aten/src/ATen/native/LossNLL.cpp#L459-L503">here</a>, which is indeed using <code>log_softmax</code> (I hope I’m not missing something obvious <img src="https://discuss.pytorch.org/images/emoji/apple/stuck_out_tongue.png?v=10" title=":stuck_out_tongue:" class="emoji" alt=":stuck_out_tongue:"> ).</p>

son_of_Jesus: <p>Thanks ptrblck,</p>
<p>You are on the money! I agree with you, I believe the formula has a typo in the docs.</p>
<p>Do you want me to edit the docs or you guys do that?</p>
<p>Kind regards,</p>
<p>Javier</p>

ptrblck: <p>If you are interested in working on the PR, please go for it!</p>

son_of_Jesus: <p>Just did! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>let me know if there is a problem with it.</p>
<p>Thanks! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>