akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Failing tests after successful build. How to contribute?

PersonalFinance004: <p>I built PyTorch from source because I want to contribute to the library. Unfortunately, I can’t run</p>
<pre><code class="lang-auto">python test/run_test.py
</code></pre>
<p>without getting errors. I rebuilt three times inside different Anaconda environments with Python versions 3.7, 3.8 and 3.9, but there are always some errors being thrown, not necessarily the same between the builds.</p>
<p>My questions are these:</p>
<ol>
<li>Should I expect all the tests to succeed? If not, how do I go about fixing them? The builds succeed without errors.</li>
<li>If I ran the tests, and some failed, does this still mean that ALL tests were executed, or does the test suite stop at the first test that fails?</li>
<li>If the answer to 2. is affirmative, i.e. all tests are run independently of whether one of them failed, am I correct in my assumption that I could run the tests once, make notes of which ones failed, then implement my changes to the codebase (e.g. to fix some issue), and re-run the tests to see if anything ELSE fails. In case only the same tests fail as before, should I then make a pull request for my changes?</li>
</ol>
<p>I really would like to contribute, but this is the major roadblock.</p>

ptrblck: <ol>
<li>
<p>Ideally they should, but might be flaky or fail on unknown/new configs.</p>
</li>
<li>
<p>You could pass the <code>--continue-through-error</code> argument to <code>python run_test.py</code> so that all tests will be executed even if one fails.</p>
</li>
<li>
<p>See 2 for the first part. Are you interested in contributing to the tests directly or are you working on another feature/fix?</p>
</li>
</ol>

PersonalFinance004: <p>I’m working on an issue from the issue tracker. It’s a specific functionality, and there are specific tests for it (they are named after the function I’m working on). So I’m wondering whether it’s enough for these specific tests to pass, given that some of the tests in <code>run_test.py</code> fail and I’m not sure how to find out why they do. I googled to no avail.</p>

ptrblck: <p>Yes, you should take a look at related tests and could run them only via the <code>-k</code> flag.<br>
E.g. if you are working on an <code>torch.nn</code> fix, the test could be located in <code>test_nn.py</code> and you could filter it out via:</p>
<pre><code class="lang-python">python test_nn.py -v -k my_test_name
</code></pre>
<p>You could use <code>grep -r my_test_name</code> to find the related file and then execute this test in isolation.<br>
I would also highly recommend <a class="mention" href="/u/tom">@tom</a>’s video on <a href="https://lernapparat.de/fixing-your-first-pytorch-bug/">how to fix your first PyTorch bug</a>, which walks you through the necessary steps and shows some tips alone the way.</p>

PersonalFinance004: <p>Ok, I will do that. Also thank you for the recommendation.</p>
Which Loss Function to use?

Tejan_Mehndiratta: <p>Hi,<br>
I have an output: <code>(N, seq_len, C)</code> and target: <code>(N, C)</code>.<br>
where N is the batch_size and C is the number of classes<br>
It is a multilabel classification problem. Which Loss function should I use?</p>

pascal_notsawo: <p>Let me take a common example where we face this kind of situation.</p>
<p>In sequence classification with models like Bert, a special token (usually called classification token) is added in front of all sequences. And when the model returns the representation of the sequence (of dimension <code>N * seq_len * emb_dim</code> usually), instead of passing it to the classification layer directly to get the output logits of dimension <code>N * seq_len * C</code> (as in your case), we select only the representation of the classification token (of dimension <code>N * emb_dim</code>) to get an output logits of dimension <code>N * C</code> after projection.</p>
<p>But if you want to use an output of dimension <code>N * seq_len * C</code>, the defined loss function will depend on the task you want to achieve. So please give more information about what you want to do, and where the output and target come from.</p>

Tejan_Mehndiratta: <p>I am doing an emotion recognition task, where I have images, transcriptions and targets.<br>
The architecture is as follows:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/b/db19de3ff55d23d07d0a21b5f6a8684ac213f084.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/db19de3ff55d23d07d0a21b5f6a8684ac213f084" title="WhatsApp Image 2021-04-23 at 03.47.09"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/b/db19de3ff55d23d07d0a21b5f6a8684ac213f084_2_255x500.jpeg" alt="WhatsApp Image 2021-04-23 at 03.47.09" data-base62-sha1="vgfZh4DeKClEzMBD7vf3nGAcaz2" width="255" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/b/db19de3ff55d23d07d0a21b5f6a8684ac213f084_2_255x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/d/b/db19de3ff55d23d07d0a21b5f6a8684ac213f084_2_382x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/b/db19de3ff55d23d07d0a21b5f6a8684ac213f084_2_510x1000.jpeg 2x" data-dominant-color="7A7877"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">WhatsApp Image 2021-04-23 at 03.47.09</span><span class="informations">531×1038 37.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>where T is the cross attention transformer Layer. And since I can get the the value of L(text seq length) only at runtime from the train loader. I don’t know what to do?</p>

pascal_notsawo: <p>I don’t have much to offer you, but you can look at this (<a href="https://pytorch.org/docs/stable/generated/torch.nn.CTCLoss.html#torch.nn.CTCLoss" rel="noopener nofollow ugc">CTCLoss</a>)</p>
Questions about QAT v2

smivv: <p>Hello!</p>
<p>Previously I had <a href="https://discuss.pytorch.org/t/questions-about-qat/106417/3">problems</a> with quantizable version of segmentation pipeline but installing torch from master branch surprisingly made it working.</p>
<p>Now I am trying to perform QAT with regression model which basically has almost the same encoder architecture but different detection decoder (stacked linears, dropouts, relus, cat operations and sigmoid on top of it).</p>
<p>I am doing everything according to the manual, but nothing is working - after some steps my model stops training well (comparing with regular model).</p>
<p>I tried these hypotheses in order to find the problem:</p>
<ol>
<li>Different torch versions: 1.7.1, 1.8.0, 1.8.1, from master source;</li>
<li>Wrapped .cat operation in decoder with FloatFunctional;</li>
<li>Tried training with both fused and not fused modules;</li>
<li>Put QuantStub and DeQuantStub at the same hierarchy level in the model;</li>
</ol>
<p>Do you have maybe other hypotheses I have to try to test?<br>
Or is there any way I can debug quantization mode to find the reason?</p>
<p>Thanks in advance!</p>
PyTorch plugins installing in wrong directory?

Arthur_Conmy: <p>I am trying to run the StyleGAN2 model on a remote server. I am working in a directory with lots of storage space called <code>nfs/st01/...</code> but when I try to load in any StyleGAN model I get the error</p>
<blockquote>
<p>OSError: [Errno 122] Disk quota exceeded: ‘/home/asc70/.cache/torch_extensions/fused’</p>
</blockquote>
<p>Now, <code>home/asc70/</code> is my personal directory on the remote server, with little space available. How do I go about changing where these torch extensions are downloaded to?</p>

ptrblck: <p>Could you try to set <code>DEFAULT_CACHE_DIR</code> to another folder and rerun the script?<br>
These directories and other settings are specified in <a href="https://github.com/pytorch/pytorch/blob/75024e228ca441290b6a1c2e564300ad507d7af6/torch/hub.py#L55-L65"><code>hub.py</code></a> and can be changed via env vars.</p>
Why is only the second half of my neural net learning?

Dylan-Kerler: <p>Everything in the second half (in the green box) is learning reliable weights but everything in the first half (blue box) is learning seemingly random weights. The backprop updates the weights and gradients of the first half, yet it is learning seemingly random weights that do not improve the training. The second half is learning correctly though.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/5/5521b812c92c60476e1b0f46fb7ab0d709c17e1b.png" data-download-href="https://discuss.pytorch.org/uploads/default/5521b812c92c60476e1b0f46fb7ab0d709c17e1b" title="Screenshot 2021-04-27 at 20.34.50"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/5/5521b812c92c60476e1b0f46fb7ab0d709c17e1b_2_690x257.png" alt="Screenshot 2021-04-27 at 20.34.50" data-base62-sha1="c96PjDWtkOT4fMhZV0neeXiatvR" width="690" height="257" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/5/5521b812c92c60476e1b0f46fb7ab0d709c17e1b_2_690x257.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/5/5521b812c92c60476e1b0f46fb7ab0d709c17e1b_2_1035x385.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/5/5521b812c92c60476e1b0f46fb7ab0d709c17e1b.png 2x" data-dominant-color="E2E5E8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2021-04-27 at 20.34.50</span><span class="informations">1252×468 64.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>It seems the issue is something to do with the transpose/view function somehow causing the <code>grad_fn</code> function to be messed up. What <strong>exactly</strong> is the issue and how can it be fixed?</p>

ptrblck: <p>How do you check, that the parameter updates of the first half of the model do not improve the training, while the updates for the second half do improve it?</p>
How to load models if I do not know its structure?

Jingnan_Jia: <p>I know <code>torch.save(net.state_dict(), model_path)</code> can only save the weights, and <code>torch.save(net, model_path)</code> can save the weights <strong>and</strong> the structure of the network.<br>
I know ·torch.load(model_path)· can load the pretrained model.<br>
I know <code>net.load_state_dict(torch.load(model_path))</code> can map the pretrained weights to the current net.</p>
<p>But I found it is necessary to define the net before I load the net from disk. Otherwise, error would accur: <code>AttributeError: Can't get attribute 'VGG19' on &lt;module '__main__'&gt;</code>. I know this is because I know <code>torch.save()</code> is a wraper of <code>python</code>'s <code>pickle</code>, and the <a href="https://docs.python.org/3/library/pickle.html#what-can-be-pickled-and-unpickled" rel="noopener nofollow ugc">limitation is because of <code>pickle</code> itself</a> .</p>
<p>My question is:</p>
<p><strong>How to load models if I do not know its structure/code?</strong></p>

ptrblck: <p>This is currently not possible and</p>
<aside class="quote no-group" data-username="Jingnan_Jia" data-post="1" data-topic="119638">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jingnan_jia/48/31909_2.png" class="avatar"> Jingnan_Jia:</div>
<blockquote>
<p>and <code>torch.save(net, model_path)</code> can save the weights <strong>and</strong> the structure of the network.</p>
</blockquote>
</aside>
<p>will not directly save the structure of the model, but would still depend on the source files (and could thus break in various ways).<br>
You could check the experimental <a href="https://pytorch.org/docs/master/package.html#torch-package"><code>torch.package</code></a> utility, which might fit your use case, but note that it’s experimental and you might thus encounter issues with it.</p>
Need more examples for using `hessian` api

tengerye: <p>Thank you so much for providing high level APIs to compute <code>hessian</code> matrix and <code>jacobian</code> matrix (<a href="https://pytorch.org/docs/1.7.1/autograd.html?highlight=jacobian#torch.autograd.functional.jacobian" class="inline-onebox" rel="noopener nofollow ugc">Automatic differentiation package - torch.autograd — PyTorch 1.7.1 documentation</a>). I also noticed toy examples provided.</p>
<p>However, there are a lot of posts trying to figure it out how to use it on complicated networks’ parameters (<a href="https://discuss.pytorch.org/search?q=hessian" class="inline-onebox">Search results for 'hessian' - PyTorch Forums</a>, <a href="https://stackoverflow.com/questions/64024312/how-to-compute-hessian-matrix-for-all-parameters-in-a-network-in-pytorch" class="inline-onebox" rel="noopener nofollow ugc">machine learning - How to compute hessian matrix for all parameters in a network in pytorch? - Stack Overflow</a>, <a href="https://stackoverflow.com/questions/56480578/pytorch-most-efficient-jacobian-hessian-calculation" class="inline-onebox" rel="noopener nofollow ugc">python - PyTorch most efficient Jacobian/Hessian calculation - Stack Overflow</a>). If possible, would you please release some examples of using it with descent networks (with <code>forward</code> and <code>get_loss</code>)?</p>
Focal Loss + Label Smoothing

Alymostafa: <p>I’m trying to implement focal loss with label smoothing, I used this implementation <a href="https://kornia.readthedocs.io/en/v0.1.2/losses.html#torchgeometry.losses.FocalLoss" rel="noopener nofollow ugc">kornia</a> and tried to plugin the label smoothing based on this implementation with Cross-Entropy <a href="https://stackoverflow.com/questions/55681502/label-smoothing-in-pytorch" rel="noopener nofollow ugc">Cross entropy + label smoothing</a> but the loss yielded doesn’t make sense.</p>
<pre><code class="lang-auto">Focal loss + LS (My implementation): Train loss 2.9761913128770314 accuracy 0.40519300987212814
</code></pre>
<pre><code class="lang-auto">Focal loss(Kornia implementation): Train loss 0.0602325857395604 accuracy 0.8621959099036829</code></pre>
CUDA cannot found while building libtorch

Kwangeun_Byun: <p>Hi all<br>
I have problem while building using libtorch.<br>
I tried to solve this problem myself, But i couldn’t solve toe problem for a long time,<br>
so i wrote a post to get hlep.</p>
<p>I using Linux(centos7) CUDA 10 version libtorch zip.</p>
<p><a href="https://pytorch.org/cppdocs/installing.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/cppdocs/installing.html</a><br>
I tried to check whether the example code was executed by following the link above.</p>
<p>But, <code>cmake -DCMAKE_PREFIX_PATH=/absolute/path/to/libtorch ..</code> command error occur.</p>
<p>Below is the error message.</p>
<pre><code class="lang-auto">
-- Could NOT find CUDA (missing: CUDA_INCLUDE_DIRS CUDA_CUDART_LIBRARY) (found version "10.2")
CMake Warning at /disk1/puzzlebook/libtorch/share/cmake/Caffe2/public/cuda.cmake:31 (message):
  Caffe2: CUDA cannot be found.  Depending on whether you are building Caffe2
  or a Caffe2 dependent library, the next warning / error will give you more
  info.
Call Stack (most recent call first):
  /disk1/puzzlebook/libtorch/share/cmake/Caffe2/Caffe2Config.cmake:88 (include)
  /disk1/puzzlebook/libtorch/share/cmake/Torch/TorchConfig.cmake:68 (find_package)
  CMakeLists.txt:6 (find_package)


CMake Error at /disk1/puzzlebook/libtorch/share/cmake/Caffe2/Caffe2Config.cmake:90 (message):
  Your installed Caffe2 version uses CUDA but I cannot find the CUDA
  libraries.  Please set the proper CUDA prefixes and / or install CUDA.
Call Stack (most recent call first):
  /disk1/puzzlebook/libtorch/share/cmake/Torch/TorchConfig.cmake:68 (find_package)
  CMakeLists.txt:6 (find_package)


-- Configuring incomplete, errors occurred!
See also "/disk1/puzzlebook/example-app/build/CMakeFiles/CMakeOutput.log".
See also "/disk1/puzzlebook/example-app/build/CMakeFiles/CMakeError.log".

</code></pre>
<p>nvcc -V result is</p>
<pre><code class="lang-auto">nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Wed_Oct_23_19:24:38_PDT_2019
Cuda compilation tools, release 10.2, V10.2.89
</code></pre>
<p>and echo $CUDA_HOME result is<br>
<code>/usr/local/cuda</code></p>
<p>I made a lot of effort to solve the problem myself, but I couldn’t solve it, so I posted it with desperate heart.</p>
<p>If you have solved this problem or know how to do it, please let me know.</p>
<p>Thanks.<br>
Yours sincerely,</p>

ptrblck: <p>Did you build PyTorch/Caffe2 before from source and was it working correctly?<br>
If so, did you use the same CUDA version?<br>
Your setup looks correct, so could you maybe <code>clean</code> the build and rerun it?<br>
If that doesn’t help, could you check, if you would be able to build <code>libtorch</code> in a docker container?</p>
Compare Architectures

skywalkerisnull: <p>Hello,</p>
<p>We are looking at creating a number of models that are based on current models (i.e. make a variation on ResNet, U-Net, Mask-RCNN etc.), the variations may be in the form of different activation functions, additional layers, different number of input channels (single channel images vs multi-channel) as well as looking into NAS.</p>
<p>We are looking for a simple way to compare the architectures in order to do transfer learning. i.e. We want to compare the architecture of the model (without the weights and biases) to see if our internal pre-trained model we have, will be compatible with the one we are about to train, and we can bootstrap it with transfer learning.</p>
<p>Ideally, we are able to do something like generate a hash or finger print of the graph and that graph will be stable across PyTorch versions. Has anyone had any experience with trying to do this as well? Or any ideas on how this could be achieved?</p>
How to model a gaussian distribution using neural network in Pytorch?

Shantanu_Ghosh: <p>I want to model the mean and variance of the gaussian distribution of an encoder in VAE. Now it’s pretty simple the encoder will have two heads - for mean and variance vector and then we will calculate<br>
the loss = recons loss + KL Loss.<br>
Now I want to calculate the pdf of the encoder distribution q(z|x) as I have the access to the mean and variance of the distribution from the encoder. q(z|x) is a multivariate gaussian but from the encoder i have the mean and variance vector whereas I need a covariance matrix. For example if z is 64 dimesional vector then the encoder will output two 64 dimensional vectors - one for mean and another for variance and each of them has a dimension of 64. To calculate the pdf q(z|x) i need to have 64 x 64 covariance matrix. Can someone please help to estimate q(z|x)?</p>
<p>I am trying to model the p and q in the picture attached which are encoder and decoder respectively<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/4/74ac57d5b41566d2e240f69d6cb2849e80822d68.png" data-download-href="https://discuss.pytorch.org/uploads/default/74ac57d5b41566d2e240f69d6cb2849e80822d68" title="Screen Shot 2021-04-27 at 8.38.41 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/4/74ac57d5b41566d2e240f69d6cb2849e80822d68_2_690x70.png" alt="Screen Shot 2021-04-27 at 8.38.41 PM" data-base62-sha1="gE8D3BFR49Au30wdejjZTEoHwiA" width="690" height="70" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/4/74ac57d5b41566d2e240f69d6cb2849e80822d68_2_690x70.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/4/74ac57d5b41566d2e240f69d6cb2849e80822d68_2_1035x105.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/7/4/74ac57d5b41566d2e240f69d6cb2849e80822d68_2_1380x140.png 2x" data-dominant-color="EDEDED"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2021-04-27 at 8.38.41 PM</span><span class="informations">1687×172 18 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div> .</p>
Loading partial models with shared parameters

ap579: <p>Hi,</p>
<p>Let’s say I have two models, A and B that share 50% of their parameters. If Model A is already in GPU and I want to load Model B, is there a way to load only the parameters that are not shared with A? I’m trying to cut the load time of model B. It seems like just initializing the model will randomly assign weights, so the time to load will not reduce even if I can filter the state_dict.</p>
<p>Thanks!</p>
Seed function, I can not have same result

Ximing_Dong11104: <p>Hi, I have a question, I have set seed, but I still, have a different result.</p>
<p>thanks, I have no ideas a few days ;( <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"></p>
<p>seed=0</p>
<p>random.seed(seed)<br>
np.random.seed(seed)<br>
if torch.cuda.is_available():<br>
torch.cuda.manual_seed_all(seed)</p>
<p>torch.manual_seed(seed)<br>
torch.cuda.manual_seed(seed)<br>
torch.cuda.manual_seed_all(seed)</p>
<p>if seed == 0:<br>
torch.backends.cudnn.deterministic = True<br>
torch.backends.cudnn.benchmark = False<br>
device = torch.device(‘cuda’ if torch.cuda.is_available() else ‘cpu’)<br>
print(torch.cuda.get_device_name(0))<br>
batch_first = True<br>
TEXT.build_vocab(train_data,<br>
max_size = MAX_VOCAB_SIZE,<br>
vectors = “glove.6B.300d”,<br>
unk_init = torch.Tensor.normal_)</p>
<p>LABEL.build_vocab(train_data)<br>
train_iterator, valid_iterator, test_iterator = data.BucketIterator.splits(<br>
(train_data, valid_data, test_data),<br>
batch_size = BATCH_SIZE,<br>
sort_within_batch = True,<br>
device = device)<br>
class RNN(nn.Module):<br>
def <strong>init</strong>(self, vocab_size, embedding_dim, hidden_dim, output_dim, n_layers,<br>
bidirectional, dropout, pad_idx):</p>
<pre><code>    super().__init__()
    
    self.embedding = nn.Embedding(vocab_size, embedding_dim, padding_idx = pad_idx)
    
    
    self.rnn = nn.LSTM(embedding_dim, 
                       hidden_dim, 
                       num_layers=n_layers, 
                       bidirectional=True, 
                       dropout=dropout,
                       )
    
    self.fc = nn.Linear(hidden_dim * 2, output_dim)
    
    self.dropout = nn.Dropout(dropout)
    
def forward(self, text, text_lengths):
    
    embedded = self.embedding(text)
    packed_embedded = nn.utils.rnn.pack_padded_sequence(embedded, text_lengths.to('cpu'))
    
    packed_output, (hidden, cell) = self.rnn(packed_embedded)

    output, output_lengths = nn.utils.rnn.pad_packed_sequence(packed_output)

    hidden = self.dropout(torch.cat((hidden[-2,:,:], hidden[-1,:,:]), dim = 1))
    return self.fc( hidden)</code></pre>

ptrblck: <p>Check the <a href="https://pytorch.org/docs/stable/notes/randomness.html">Reproducibility docs</a>, which mention additional ops such as <code>torch.use_deterministic_algorithms()</code>. If a non-deterministic operation is found, it should raise an error.</p>
<p>PS: you can post code snippets by wrapping them into three backticks ```, which makes debugging easier. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

Ximing_Dong11104: <p>thank u so much I will try</p>
Init_process_group times out without an error (ProcessGroupGloo)

BenAAndrew: <p>Hi there,</p>
<p>I’m trying to add support to spread my batches across multiple NVIDIA GPU’s when training.<br>
To do this I’ve followed PyTorch documentation outlining that I must start a process group. This is done like so:</p>
<pre><code class="lang-auto">dist.init_process_group(
    backend="gloo", 
    init_method="file:///C:/Users/thefi/Voice-Cloning-App-MGPU/distributed_logging",
    world_size=2, 
    timeout=datetime.timedelta(0, 180),
    rank=0
)
</code></pre>
<p>The <code>distributed_logging</code> file is generated but after 3 minutes I get the following error:</p>
<pre><code class="lang-auto">dist.init_process_group(\n  File \"C:\\Users\\thefi\\MiniConda3\\envs\\vc\\lib\\site-packages\\torch\\distributed\\distributed_c10d.py\", line 439, in init_process_group\n    _default_pg = _new_process_group_helper(\n  File \"C:\\Users\\thefi\\MiniConda3\\envs\\vc\\lib\\site-packages\\torch\\distributed\\distributed_c10d.py\", line 517, in _new_process_group_helper\n    pg = ProcessGroupGloo(\nRuntimeError: Wait timeout\n"}
</code></pre>
<p>Unfortunately, it does not specify why the timeout occurs.<br>
Other things to note:</p>
<ol>
<li>This training process is being imported as a function in a Thread and not run from the command line</li>
<li>This is on windows, therefore I can only use <code>gloo</code> as the backend</li>
</ol>

pritamdamania87: <p>If you are using world_size=2, you need two processes (one with rank 0 and one with rank 1) otherwise with a single process you will see a timeout caused due to the system waiting for one missing process.</p>

BenAAndrew: <p>Could you elaborate on what you mean. Do you mean I need to running instances of this code in different processes?</p>

pritamdamania87: <blockquote>
<p>Do you mean I need to running instances of this code in different processes?</p>
</blockquote>
<p>Yes, you can refer to these docs for concrete examples: <a href="https://pytorch.org/tutorials/intermediate/dist_tuto.html" class="inline-onebox" rel="noopener nofollow ugc">Writing Distributed Applications with PyTorch — PyTorch Tutorials 2.1.1+cu121 documentation</a></p>
GAN Loss function

edshkim98: <p>Hi, I have a question about gradient vanishing problem in GAN.<br>
There are many articles saying gradient vanishing can occur when using BCE to train Generator if the discirminator network is too good. This is because the loss is small and there will be less gradient update. However, I don’t quite understand this phrase.<br>
During the training of Generator, we use BCE(D(G(z),1) where D is Discriminator, G(z) is Generator and 1 is true. From this, if the discriminator network performs really well, surely the loss for the above equation be very large as discriminator network knows that the input is fake rather than real, but the target is 1. Then surely, the gradient update will be bigger and there is no gradient vanishing problem here?<br>
Can somebody please explain to me whether I am correct and when is the case for gradient vanishing to happen during training generator?  Thank you!</p>
RuntimeError about pytorch1.7.1

kuaidiyuan: <p>RuntimeError: iter.device(arg).is_cuda() INTERNAL ASSERT FAILED at “/pytorch/aten/src/ATen/native/cuda/Loops.cuh”:94</p>

eqy: <p>Can you check if the inputs to the computation are all on the same device type (and if they are on GPU, the <em>same</em> GPU)?</p>
How to display image from pytorch dataloader

Dandi: <p>so i just did my training model, then i want to use the model that i trained back then to classify a single image in inside pytorch dataloader. is that possible?</p>

eqy: <p>Yes, you can either iterate through the dataloader or simply access a sample in the dataset via an index e.g., <code>mydataset[0]</code> as it should implement <code>__getitem__</code>.</p>
Cant load model in CPU DataParallel

jap8nted: <p>Hi,</p>
<p>I trained a model using DataParallel wrapper and then was trying to run inference in a machine without gpu and it does not work. Could somebody help me to check, here is the code and error:</p>
<pre><code class="lang-auto">import torch
import torchvision
import torch.nn as nn
import torch.nn.functional as F
import requests
from PIL import Image as PILImage

metadata = {
    "image_size": {"width": 512, "height": 512},
    "training_name": "resnet50_classifier",
    "mean": [0.4847615957260132, 0.4710499048233032, 0.36006495356559753],
    "std": [0.20003080368041992, 0.19453473389148712, 0.17458273470401764]
}
def get_image(image_url):

    image_response = requests.get(image_url, stream=True)
    image = PILImage.open(image_response.raw)
    image_channels = len(image.getbands())
    image_width, image_height = image.size

    return image, image_width, image_height

mean = metadata.get("mean", None)
stddev = metadata.get("std", None)
target_size = (512, 512)
pil_image, _, _ = get_image(url)
transform = torchvision.transforms.Compose(
            [torchvision.transforms.Resize(target_size), torchvision.transforms.ToTensor(), torchvision.transforms.Normalize(mean, stddev)]
        )

def _load_image( pil_image):
    image = transform(pil_image)
    image = image.float()
    image = image.unsqueeze_(0)
    image = image.to(device)
    return image

image = _load_image(pil_image)
model = torch.load(path_to_model, map_location=torch.device("cpu"))
model(image)
</code></pre>
<hr>
<p>RuntimeError                              Traceback (most recent call last)<br>
 in <br>
----&gt; 1 model(image)</p>
<p>~/.local/lib/python3.6/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)<br>
887             result = self._slow_forward(*input, **kwargs)<br>
888         else:<br>
 → 889             result = self.forward(*input, **kwargs)<br>
890         for hook in itertools.chain(<br>
891                 _global_forward_hooks.values(),</p>
<p>~/.local/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py in forward(self, *inputs, **kwargs)<br>
153                 raise RuntimeError("module must have its parameters and buffers "<br>
154                                    "on device {} (device_ids[0]) but found one of "<br>
 → 155                                    “them on device: {}”.format(self.src_device_obj, t.device))<br>
156<br>
157         inputs, kwargs = self.scatter(inputs, kwargs, self.device_ids)</p>
<p>RuntimeError: module must have its parameters and buffers on device cuda:0 (device_ids[0]) but found one of them on device: cpu</p>

eqy: <p>Have you tried instantiating the model separately first as in this [example]( <a href="https://pytorch.org/tutorials/beginner/saving_loading_models.html#saving-loading-model-across-devices" rel="noopener nofollow ugc">Saving and Loading Models — PyTorch Tutorials 1.8.1+cu102 documentation</a>).</p>
When using pytorch to save the model, torch.save(model.state_dict()) is used, how to continue training?

dachun: <p>As the title, I have used <code>torch.save(model.state_dict())</code> when try to save the model. I checked that only weights and biases should be saved. At that time, other parameters such as optimizer were not saved during training. How to continue training? Try to use <code>model.load_state_dict(torch.load(***.pkl)</code>, but I don’t know then. It’s best to use the relevant code for reference.</p>

ptrblck: <p><a href="https://pytorch.org/tutorials/beginner/saving_loading_models.html">This tutorial</a> explains how to store and load <code>state_dict</code>s and general checkpoints. Once you’ve loaded the model’s and optimizer’s <code>state_dict</code>s you could continue with your training as was you’ve done before.</p>

dachun: <p>Hi, but I haven’t save the optimizer’s state_dicts, only weights and bias. Could I continue to train with what I have saved?</p>

ptrblck: <p>Yes, you could continue training with it, but would most likely get a different loss curve compared to the “full” run without saving and restoring the training (if your optimizer uses internal buffers).<br>
If you load only the model’s <code>state_dict</code> your use case would be similar to the standard fine-tuning approach.</p>
I wonder if it is possible to multiply the output of the module by the weight of another module

11153: <p>Assuming that the model has two linear layers,</p>
<p>First, put the <strong>input x</strong> in the first layer.</p>
<p>And <strong>output</strong> from the first layer manipulates the <strong>weight</strong> of the second layer.</p>
<p>Finally, put the same <strong>input x</strong> into the second layer.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/b/6b565e44a7da777477a2be4d1604e88e71ef636f.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/6b565e44a7da777477a2be4d1604e88e71ef636f" title="sample"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/b/6b565e44a7da777477a2be4d1604e88e71ef636f_2_690x180.jpeg" alt="sample" data-base62-sha1="fjy8d3nJPdpbEq0j6SNG1LkJr3h" width="690" height="180" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/b/6b565e44a7da777477a2be4d1604e88e71ef636f_2_690x180.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/6/b/6b565e44a7da777477a2be4d1604e88e71ef636f_2_1035x270.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/b/6b565e44a7da777477a2be4d1604e88e71ef636f.jpeg 2x" data-dominant-color="EFEDEC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">sample</span><span class="informations">1096×287 24.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I would like to ask if both layers will be updated normally in the following situations.</p>
<p>Thank you.</p>

soulitzer: <p>Are you creating a custom nn module?</p>
<pre><code class="lang-auto">def forward(self, x):
    return torch.cat((x, x)) * self.w * x + self.b
</code></pre>
<p>If you do something like this, it should be OK. Trying to reuse Linear OTOH, and modifying its weight in-place is not recommended.</p>

11153: <p>Thanks for you kind reply!!</p>
<p>Originally, I tried to approach it second way, but I’ll try the first way.</p>
<p>Thank you.</p>

CloudFan: <p>Hi, did you solve this problem? I’m also trying to manipulate the weights in the second model with the outputs of the first model. The following is the foward function in second model:<br>
def forward(self, x, kernel, bias):<br>
self.conv[0].weight.data = self.conv[0].weight.data*kernel<br>
self.conv[0].bias.data = self.conv[0].bias.data +torch.squeeze(bias)<br>
out = self.conv(x)<br>
return out<br>
where kernel and bias are the output of the first model. But I got the error message<br>
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn<br>
Do you have any idea to solve this problem?</p>
RuntimeError: Error(s) in loading state_dict for DLASeg

prabhuiitdhn: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>  : Any help would be much appreciated.</p>
<p>I have setup the same environment, and I am trying to load pretrained model “model_mot.pth” to get the state_dict().<br>
But I am getting an error:</p>
<p>Unexpected key(s) in state_dict: “ltrb_amodal.0.weight”, “ltrb_amodal.0.bias”, “ltrb_amodal.2.weight”, “ltrb_amodal.2.bias”, “base.pre_hm_layer.0.weight”, “base.pre_hm_layer.1.weight”, “base.pre_hm_layer.1.bias”, “base.pre_hm_layer.1.running_mean”, “base.pre_hm_layer.1.running_var”, “base.pre_hm_layer.1.num_batches_tracked”.</p>
<p>The above keys are already available in state_dict_util. using pytorch 1.5.0</p>
<h1>
<a name="code-1" class="anchor" href="#code-1"></a>code</h1>
<pre><code class="lang-auto">model_path  = 'models/model_mot.pth'
model = create_model(opt.arch, opt.heads, opt.head_conv, opt=opt)
dummy_input = Variable(torch.randn(1, 1, 544, 960))
state_dict_util = torch.load(model_path, map_location="cpu")['state_dict']
model.eval()
new_state_dict = OrderedDict()
for k, v in state_dict_util.items():
    new_state_dict[k] = v
torch_model = model.load_state_dict(new_state_dict)
# torch_model = model.load_state_dict(torch.load(model_path)['state_dict'], strict=False)
torch.onnx.export(torch_model, dummy_input, "models/model_mot.onnx")
</code></pre>

ptrblck: <p>There seems to be a naming conflict in the loaded <code>state_dict</code> and your module names.<br>
You could check both via:</p>
<pre><code class="lang-python">for name, module in model.named_modules():
    print(name)

sd = model.state_dict()
for key in sd:
    print(key)
</code></pre>
<p>and see what the names should look like.</p>
<p>PS: please don’t tag specific people, as it could discourage others to post an answer <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
How to link LibTorch with a Gradle-based C++ project?

MathBunny: <p>Hello, I am trying to link LibTorch to a C++ codebase that uses Gradle as its build system. Unfortunately, it appears that the vast majority of documentation covers adding LibTorch to a CMake project, not Gradle, and I am running into issues linking the library.</p>
<p>My issue primarily stems from including the headers (even when using the pre-built variant). From the repository we have:</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/c371542efc31b1abfe6f388042aa3ab0cef935f2/cmake/TorchConfig.cmake.in#L58-L59" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/c371542efc31b1abfe6f388042aa3ab0cef935f2/cmake/TorchConfig.cmake.in#L58-L59" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/c371542efc31b1abfe6f388042aa3ab0cef935f2/cmake/TorchConfig.cmake.in#L58-L59</a></h4>
<pre class="onebox"><code class="lang-in"><ol class="start lines" start="58" style="counter-reset: li-counter 57 ;">
<li>${TORCH_INSTALL_PREFIX}/include</li>
<li>${TORCH_INSTALL_PREFIX}/include/torch/csrc/api/include)</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>It seems that there are <strong>two</strong> include directories for LibTorch, one of which is inside the other one. This poses issues for me when I try to provide the <code>headers.srcDir</code> because it appears to only accept one option for a folder path where it can look for the headers.</p>
<p>If I simply copy the headers over to <code>/usr/local/include</code> and provide that path to Gradle, then I am informed it cannot find the inner one. I tried several roundabout ways of doing this, but any time I run into issues where it either (a) cannot find the inner folder, (b) cannot find the outer folder, or (c) states that there are duplicate declarations for overlapping files.</p>
<p>Hence, I am wondering if anyone has an example of integrating it with a Gradle C++ project (or another build tool), possible workarounds, or any suggestions on how to deal with the two nested include folders. Thank you!</p>

MathBunny: <p>I managed to get progress on this front by leveraging <a href="https://docs.gradle.org/current/dsl/org.gradle.language.cpp.CppSourceSet.html#org.gradle.language.cpp.CppSourceSet:preCompiledHeader" rel="noopener nofollow ugc">CppSourceSet</a>, which is part of the ‘cpp’ plugin for Gradle. This allows for elegantly expressing multiple header file import locations.</p>
<p>Hence, we can simply do:</p>
<pre><code class="lang-auto">headers.srcDirs = ["/usr/local/include/torch/csrc/api/include/", "/usr/local/include/torch/"]
</code></pre>
<p>This manages to resolve the headers issue.</p>
<p>I cannot confirm if it completely works because I am since running into a C++11 ABI issue, but once I resolve that (which I believe is caused by another problem) I will look back and confirm that it is solved.</p>

MathBunny: <p>Can confirm that this works.</p>
Slicing tensor of different size

I_H_Yoo: <p>Hi. I’m trying to slice a tensor of different sizes for each batch. I have two tensors which are including start and end index respectively. For example, for a tensor of size [3, 1, 8], I want only specific tensors having different range:</p>
<pre><code class="lang-auto">start = torch.tensor([1, 3, 1])
end = torch.tensor([7, 7, 5])

# Do slicing

tensor([[[ 0,  1,  2,  3,  4,  5,  6,  7]], # [1, 7]

        [[ 8,  9, 10, 11, 12, 13, 14, 15]], # [3, 7]

        [[16, 17, 18, 19, 20, 21, 22, 23]]]) # [1, 5]
</code></pre>
<p>So, the first one will be <code>[1,  2,  3,  4,  5,  6]</code>, the second one will be <code>[11, 12, 13, 14]</code> and the last one will be <code>[17, 18, 19, 20]</code>. I used iteration, but it is too slow and not elegant. How can I get them more faster and more elegant than just iterative way? The below shows my implementation.</p>
<pre><code class="lang-python">context = torch.zeros(batch_size, 1, self.hid_dim).to(device)

for i in range(batch_size):
  local = torch.tensor([j for j in range(start[i], end[i])]).to(device) # [2D + 1]: for j
  batch_Pt = Pt[i]
  local = torch.exp(-1 * ((local-batch_Pt) ** 2)/((self.D ** 2) / 2)) # [2D + 1]
  score = align_score[i] * local  # [1, 2D + 1]
  context[i] = score.mm(H[i])
</code></pre>

harsha_g: <p>You can use <code>torch.gather</code> to accomplish this. This <a href="https://medium.com/analytics-vidhya/understanding-indexing-with-pytorch-gather-33717a84ebc4" rel="nofollow noopener">article</a> will help you get started.</p>

I_H_Yoo: <p>I don’t think it could be an answer. The point is I want to extract them, with different sizes. So the tensor of the first example will be <code>[1, 2, 3, 4, 5, 6]</code> and the second one will be <code>[11, 12, 13, 14]</code>. I guess that <code>torch.gather</code> only results tensors have the same size.</p>

XT_Li: <p>I’m facing the same issue here, have you found a solution?</p>

I_H_Yoo: <p>Sorry for thr late answer. I’m afraid but I can’t find thr solution. I just used for loop</p>

XT_Li: <p>Thanks so much for your reply! I worked out something like this <a href="https://discuss.pytorch.org/t/scatter-concatenation/118338" class="inline-onebox">Scatter concatenation?</a>, the speed is fine, but cannot be batched.</p>
Compute the distance between each point of the labels and the prediction

koopa31: <p>Hi everyone,<br>
I have a problem that is I think quite complicated to solve computationally speaking. I compute the segmentation from a label image, and I’d like to calculate the distance between the label’s image and each point of my prediction. Is there an efficient already implemented method in Pytorch to do that?<br>
The brute force method would be to go through all the labels pixels and keep the minimum distance with the given pixel of the prediction, for each pixel of the prediction. I do not know if I made myself clear enough. If not, don’t hesitate to ask for more details.<br>
Thank you very much!</p>
Does creating a new tensor break Autograd?

deepfailure: <p>Hello,</p>
<p>I am training a neural network that outputs a vector: <strong>output</strong> → with size (Batch size x N).</p>
<p>Then, I want to reduce the vector to (Batch size x M) (len(M)&lt;len(N)) by creating groups of size 10 of N and summing them.</p>
<p>My question is if I do:</p>
<p>new_output = torch.tensor([tensor.sum() for tensor in torch.split(output, 10, dim=1)])</p>
<p>I then have to compare new_output with another variable of size (Batch_size x M) which also required grad.</p>
<p>Will creating a new tensor break Autograd?</p>
<p>Also, how one can be sure that Autograd works correctly? For simple architectures is straightforward, however, I am not sure how to do it if I create more complex ones.</p>
<p>Thanks!</p>

AlphaBetaGamma96: <p>Hi,</p>
<p>I think using <code>torch.tensor</code> would break it because <code>torch.tensor</code> has no <code>grad_fn</code>.  What you could do is replace,</p>
<aside class="quote no-group" data-username="deepfailure" data-post="1" data-topic="119555">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/a3d4f5/48.png" class="avatar"> deepfailure:</div>
<blockquote>
<p>new_output = torch.tensor([tensor.sum() for tensor in torch.split(output, 10, dim=1)])</p>
</blockquote>
</aside>
<p>with,</p>
<pre><code class="lang-auto">new_output = torch.stack([tensor.sum() for tensor in torch.split(output, 10, dim=1)])
</code></pre>
<p>That should keep the gradients flowing as <code>torch.stack</code> has a <code>grad_fn</code> whereas <code>torch.tensor</code> doesn’t!</p>

deepfailure: <p>Thanks! So in order to check if autograd works I should be sure that the operations have grad_fn?</p>

AlphaBetaGamma96: <p>In terms of exact functionality of Autograd it’d best to get a Dev’s opinion (as they know what’s exactly going on), but what I would say from purely a <em>mathematical</em> perspective is that the gradient of a given operation is defined by the chain rule.</p>
<p>So it’s essentially a product of all intermediate gradients from a given layer (e.g. your <code>torch.tensor</code> function) up to the final layer (where you compute your loss value). So, if one of those intermediate gradients between these 2 layers is 0 (from not having a <code>grad_fn</code>) then all preceding layers (i.e. all layers <em>before</em> your <code>torch.tensor</code> layer) will have a zero-valued gradient by definition.</p>
[W NNPACK.cpp:80] Could not initialize NNPACK! Reason: Unsupported hardware

bloring: <p>Experiencing this message when running on multiple Intel based CPUs.</p>
<p>On one system I compiled torch from sources (Python 3.9) following “getting started” instructions. On the other I installed via pip (Python 3.8). So this seems to be an issue with multiple CPU architectures and multiple versions.</p>
<ol>
<li>What is the performance implication?</li>
<li>Do I need to fix this issue? Is it safe to ignore?</li>
<li>Is it possible to silence the warning?</li>
</ol>

rolyan_trauts: <p>I just installed pytorch via anaconda to test the speech commands tutorial.</p>
<p>I have got as far as</p>
<p>new_sample_rate = 8000<br>
transform = torchaudio.transforms.Resample(orig_freq=sample_rate, new_freq=new_sample_rate)<br>
transformed = transform(waveform)</p>
<p>ipd.Audio(transformed.numpy(), rate=new_sample_rate)</p>
<p>Then [W NNPACK.cpp:80] Could not initialize NNPACK! Reason: Unsupported hardware.</p>
<p>Ubuntu 20.04, I5-3570, GTX1050ti</p>

bloring: <p>OK, so after digging into NNPACK - requirements state avx2 and l3 cache are required.</p>
<p>one of my systems does not support avx2(Intel® Xeon® CPU E5-2690 v2 @ 3.00GHz) and the other does not have L3(Intel® Xeon Phi™ CPU 7250 @ 1.40GHz)</p>
<p>There’s nothing to be done, wish that this warning could be silenced.</p>

CracKCatZ: <p>Hey <a class="mention" href="/u/bloring">@bloring</a> so you mean there is no way to avoid this error because of no hardware support?<br>
(building from source could be a solution but it is crashing my pc)</p>