akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Upgrade to cuda in a conda environment

tomwagstaff-opml: <p>Hi everyone,</p>
<p>I had pytorch installed in a conda environment using the CPU. On a new machine I have a GPU available, so I want to upgrade to the CUDA version. I’ve removed pytorch from the environment and then run the recommended line, in my case:</p>
<pre><code class="lang-auto">conda install pytorch torchvision torchaudio pytorch-cuda=12.1 -c pytorch -c nvidia
</code></pre>
<p>However, I keep getting offered the CPU build:</p>
<pre><code class="lang-auto">pytorch            pytorch/win-64::pytorch-2.2.0-py3.9_cpu_0
</code></pre>
<p>How can I upgrade to the CUDA build?</p>
<p>I’ve checked similar questions <a href="https://discuss.pytorch.org/t/upgrading-from-cpu-only-to-gpu/64168">here</a> and <a href="https://discuss.pytorch.org/t/how-to-update-from-cpu-only-install-to-cuda-install/123885">here</a> but can’t find any answers there.</p>
Loss function for unordered sets

pirotex: <p>I have a problem with translating from one unordered set to another. My input unordered set consists of a list of drugs prescribed for a patient, and the output is the list of adverse effects the patient has experienced. This is similar to the classical seq2seq model, with the only difference being that the order of adverse effects does not matter. I can use a Transformer without positional encodings for the input. However, my question is, what type of loss function can I use for unordered sets?</p>
Output of Bidirectional RNNs and Attention

jupyter: <p>Hi,</p>
<p>Lately I’m working on Seq2Seq Architecture combine with Attention mechanism. I’m using Bidirectional GRU for both Encoder and Decoder. As much as I know about Attention: I use last hidden state of Decoder as query (which has shape (2<em>num_layers, N , H_out) and use Encoder outputs as keys (I think encoder outputs are actually hidden state of each time step t (h_t) which has shape (N, sequence_length, 2</em>H_out))</p>
<p>If I use 1 layer then the hidden state of decoder has shape: (2, N, H_out) → permute (N, 2, H_out)</p>
<p>So to calculate score between query and keys, I need to make it has the same feature size between keys and query.</p>
<p>I want to ask that how the vectors are laid out. I mean (left-&gt;right) then (right–&gt;left) of an layer then stack to another (left-&gt;right) then (right&gt;left) of another layer. or they are (left-&gt;right)*n_layer then stack to (right-&gt;left)*n_layer?</p>
<p>In the case I use n_layer = 1, then if I reshape the final hidden state of decoder like. b = b.reshape(batch_size, -1) then how it works? is it concatenate (left-&gt;right) first then (right-&gt;left) ?</p>
<p>And finally, Is it if the sequence length is 1 and n_layer is 1 so the output, and h_n of RNN is the same? they just have different shape.<br>
Thank you.</p>

vdw: <p>Maybe this <a href="https://github.com/chrisvdweth/nus-cs4248x/blob/master/3-neural-nlp/Section%203.2%20-%20RNN%20Machine%20Translation.ipynb" rel="noopener nofollow ugc">notebook</a> might help. It covers an Machine Translation training using an RNN-based encoder-decoder architecture using attention. The encoder can be configured to be bi-directional.</p>
<p>Let me know if you have any questions regarding the code.</p>

jupyter: <p>Sorry for late response. Thank you</p>
Multi-label classification: all predictions become zero

Zahra_Momeni: <p>I’m working on a multi-label classification. The number of labels are 122, and for each sample,  a maximum of 10 of these labels are one and the rest are zero. (so we have sparse multi-hot encoding)<br>
I used <code>BCEWithLogitsLoss</code> as a loss function in two ways:</p>
<p>1- without weight<br>
criterion = nn.BCEWithLogitsLoss()<br>
outputs = self.model(inputs)<br>
loss = criterion(outputs, targets)</p>
<p>2- with weight<br>
criterion = nn.BCEWithLogitsLoss(reduction=‘none’)<br>
outputs = self.model(inputs)<br>
loss = criterion(outputs, targets)<br>
loss = (loss * self.CLASS_WEIGHT).mean()</p>
<p>In both cases, after a few iterations of the first epoch, the f1-score value is zero and the reason is that the model predictions are zero. Since Target tensor is spars, what is the solution to improve the loss function to prevent this problem?</p>

Andrei_Cristea: <p>Hello! For simplicity, let’s focus on the “1- without weight” version. Even though the target is sparse, you shouldn’t be seeing predictions all go to zero. Can you share an executable snippet of code that has this issue, just so we can double-check that you’re not doing anything unusual in the model? For example, a common mistake is to pass the output through your own Sigmoid layer, which you don’t need to do since your loss function does it for you already.</p>

WoodySJR: <p>Hello. Did you solve the problem? I have the same problem…</p>
Install with pdm problem

Li_Li: <p>I want to install pytorch 2.2.0 by pdm with cache on because I have many projects using the same version of pytorch. I tried many python package manager such as pip, conda, venv, pipenv, poetry. But If I want to isolate each project with a environment, pytorch will be installed again and again. And I found PDM which can install package in a central cache and each project use symlink to share that package.<br>
I can use this method to install other package such as click. But when I install pytorch 2.2.0, it failed. I know it’s a problem related to PDM, but I can’t find solution there. So I post a question here. More information can be found <a href="https://github.com/pdm-project/pdm/issues/2614" rel="noopener nofollow ugc">here</a>.</p>
PyTorch ArrayRef invalid index problem with linear nn

Kyrylo_Lyskov: <p>I have a headache and following code, that doesn’t let me sleep at night:</p>
<pre><code class="lang-auto">  1 import torch
  2 import torch.nn as nn
  3 import torch.optim as optim
  4 import numpy as np
  6 samples = torch.linspace(0, 100,100) # GENERATE THE SET
  7 train_split = int(len(samples)*0.8)
  8 x_train, x_test = samples[:train_split], samples[train_split:]
  9 y_labels = 2*samples-4 # define the function
 10 y_labels += torch.tensor(np.random.normal(0, 5, len(samples))) # ADD NOISE
 13 class NeuralNetwork(nn.Module):
 14     def __init__(self):
 15         super().__init__()
 16         self.fc1 = nn.Linear(1, 1)
 17     def forward(self, x):
 18         return self.fc1(x)
 19 model = NeuralNetwork()
 20 loss_func = nn.MSELoss()
 21 optimizer = optim.Adam(model.parameters(), lr=0.001)
 22 num_epochs = 50
 23 for epoch in range(num_epochs):
 24     for inputs, labels in zip(x_train, y_labels[:train_split]):                                                                                                                                         
 25      
 26         y_pred = model(inputs) # File ".../torch/nn/modules/linear.py", line 116, in forward
    return F.linear(input, self.weight, self.bias)
RuntimeError: ArrayRef: invalid index Index = 18446744073709551615; Length = 0
 27         loss = loss_func(y_pred, labels)
 28         optimizer.zero_grad()
 29         loss.backward()
 30         optimizer.step()
 31         print("Epoch %d  - loss: %.4f%" % (epoch, loss))
</code></pre>
<p>It is a simple one layer thing, and I’m doing it intiuitevly so no need to call me a goof. I run it on couple machines, no difference…</p>
<p>P.S. ANY SUGGESTIONS ON FURTHER IMPROVEMENTS IN THE PROCESS OF WRITING THOSE BEASTS WOULD BE APPRECIATED!!!</p>

ptrblck: <p>Unsqueeze a dimension of the input tensor before passing it to the model:</p>
<pre data-code-wrap="python"><code class="lang-python">y_pred = model(inputs.unsqueeze(0))
</code></pre>
<p>and it should work.<br>
This seems like a nasty bug and I will create a GitHub issue for it.</p>
<p>EDIT: <a href="https://github.com/pytorch/pytorch/issues/119161">#119161</a></p>

Kyrylo_Lyskov: <p>I wrapped my data properly (…torch.Tensor([[x_train]])… ), and now it works stuck_out_tongue_winking_eye: but thank you!!! Also, I just wanted to express gratitude and my awe at how much you do for the Pytorch community by answering so many stupid and not-stupid questions. Thank you!</p>
Tf.image.extract_patches with "VALID" padding

mwk: <p>I’ve been able to reimplement the extract_patches and extract_image_patches in PyTorch for “SAME” padding but how would I implement it for “VALID” padding when the strides and kernel sizes are different in the height and width dimensions?</p>
<p>For example:</p>
<pre><code class="lang-auto">batch_size = 4
channels = 128
height, width = 1, 600
kernel_height, kernel_width = 1, 16
stride_height, stride_width = 1, 4

x = torch.arange(0, batch_size*width*height*channels).view(batch_size, channels, height, width) # shape [4, 128, 1, 600]

tf_x = x.permute(0,2,3,1).numpy()

patches = x.unfold(2, kernel_height, stride_height).unfold(3, kernel_width, stride_width) # [4, 1, 147, 128, 1, 16]

patches = patches.permute(0, 2, 3, 1, 4, 5).contiguous()  # [4, 1, 147, 128, 1, 16]

patches = patches.view(*patches.size()[:3], -1) # [4, 1, 147, 2048]
print(patches[0,0,0,:])
tf_patches = tf.image.extract_patches(images=tf_x,
                          sizes=[1, 1, kernel_width, 1],
                          strides=[1, 1, stride_width, 1],
                          rates=[1, 1, 1, 1],
                          padding='VALID') # [4, 1, 147, 2048]
print(tf_patches.numpy()[0,0,0,:])

assert torch.allclose(
    patches,
    torch.from_numpy(tf_patches.numpy()),
    atol=1e-3,
    rtol=1e-3
)
&gt; tensor([    0,     1,     2,  ..., 76213, 76214, 76215])
&gt; [    0   600  1200 ... 75015 75615 76215]
</code></pre>
<p>The sizes are the same but the order of the elements are wrong.</p>

mwk: <p>Nevermind, the unfolding has to be done first across the height because the shape is 1.</p>
<p>Fix:</p>
<pre><code class="lang-auto">  bs, ch, h, w = x.size()
  patches = x.unfold(3,kw,sw).unfold(2, kh, sh)
  patches = patches.permute(0,4,5,1,2,3).contiguous()
  patches = patches.view(patches.shape[0],-1,patches.shape[-2], patches.shape[-1])
</code></pre>
Apple M1Max PyTorch error: TypeError: Cannot convert a MPS Tensor to float64 dtype as the MPS framework doesn't support float64. Please use float32 instead

gkd720: <p>Previously searching online, and here, I didn’t see any solution yet, so just checking again.<br>
I’m running code that works using CUDA type GPUs. I wanted to modify it to run on M1 GPUs, so I added the “mps” option. Looks like everything’s OK until it gets to the training step, where it gets the Topic’s error message. Anything close to a possible solution seems to always mention Torch RL, but I’m not using that. Any thoughts on how to fix or get around, on what better version of something to use? Details below. Thanks.</p>
<p>Levels:</p>
<pre><code class="lang-auto">torch               2.1.2
torchaudio          2.1.2
torchvision         0.16.2
</code></pre>
<p>Code/error:</p>
<pre><code class="lang-auto">import torch
from torch.utils import data
. . .

  File "/Users/gkd/Programs/CancerRiskNet/scripts/main.py", line 55, in &lt;module&gt;
    epoch_stats, model = train.train_model(train_data, dev_data, model, args)
  File "/Users/gkd/Programs/CancerRiskNet/cancerrisknet/learn/train.py", line 43, in train_model
    run_epoch(data_loader, train=if_train, truncate_epoch=True, models=models,
  File "/Users/gkd/Programs/CancerRiskNet/cancerrisknet/learn/train.py", line 155, in run_epoch
    batch = prepare_batch(batch, args)
  File "/Users/gkd/Programs/CancerRiskNet/cancerrisknet/learn/train.py", line 203, in prepare_batch
    batch[key] = batch[key].to(args.device)
TypeError: Cannot convert a MPS Tensor to float64 dtype as the MPS framework doesn't support float64. Please use float32 instead.
</code></pre>
cuCollection integration with PyTorch

derrick: <p>Hello, I was wondering if there is an ongoing effort to integration cuCollections with PyTorch.  For instance, in this article <a href="https://medium.com/@Pinterest_Engineering/gpu-accelerated-ml-inference-at-pinterest-ad1b6a03a16d" class="inline-onebox" rel="noopener nofollow ugc">GPU-accelerated ML Inference at Pinterest | by Pinterest Engineering | Medium</a>, the authors implement a Tensorflow custom op [presumably] to use cuCollections hash maps for accelerating lookups during online inference.</p>
Computational Error with PyTorch

ebalogun01: <p>I noticed an issue with PyTorch computing exponents that are compositions of two numbers.<br>
For example say we want to compute $$x^{2 * 0.8}$$ for a torch tensor, this returns NaN for negative numbers even though its value is always positive. It seems like Torch always fails in this instance. However, if I rewrite as $$(x^{2})^{0.8}$$, this seems to work. This same operation does not fail on numpy array or when using python floats, it is unique to torch tensors.</p>
Memory Usage During Training Skyroket

Javier_Jay: <blockquote>
<p>accumulation_steps = 4<br>
epochs_times = 1</p>
</blockquote>
<blockquote>
<p>for epoch in range(epochs):<br>
total_loss = 0<br>
batch_times = 0<br>
for idx, batch in enumerate(train_loader):<br>
src = batch[‘source’].to(device)<br>
tgt = batch[‘target’].to(device)<br>
# Prepare target data for model input and loss calculation<br>
tgt_input = batch[‘target’][:, :-1].to(device)<br>
tgt_output = batch[‘target’].to(device)</p>
</blockquote>
<blockquote>
<pre><code>    sequence_length = tgt_input.size(1)
    tgt_mask = model.get_tgt_mask(sequence_length).to(device)
</code></pre>
</blockquote>
<blockquote>
<pre><code>    pred = model(src, tgt_input, tgt_mask)
    # Reset gradients
    optimizer.zero_grad()

    # Forward pass
    output = model(src, tgt)

    # Reshape output for loss calculation
    output=output.permute(1,2,0)

    # Calculate loss
    loss = criterion(output, tgt_output) / accumulation_steps
    total_loss += loss.item()

    # Backward pass and optimize
    loss.backward()
    
    del src, tgt,tgt_input,tgt_output,loss
    
    if (idx + 1) % accumulation_steps == 0:
        optimizer.step()    
        optimizer.zero_grad() 

        batch_times +=4
        print(f"{batch_times}th batch of {epochs_times}th epoch is done")
epochs_times +=1

avg_loss = total_loss / len(train_loader)
print(f'Epoch {epoch+1}, Loss: {avg_loss:.4f}')
</code></pre>
</blockquote>
<p>The codes above is my code to train my transformer model for translation.</p>
<p>While training the model, the memory usage suddenly increase like a step function.</p>
<p>And Out of Memory Errorr occurs</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/5/857c7d11abb4d3bf2ce7ff6ae16f53cc495780ee.png" alt="스크린샷 2024-02-05 194030" data-base62-sha1="j2SdA46TxMUIvjwxI3p7qjyX7ZA" width="640" height="134"></p>
<p>(The image above shows the usage graph of the VRAM)</p>
<p>At first, I tried to use other techniques like gradient accumulatio, but they did not work.</p>
<p>Would please help me to solve this problem?</p>
Trouble setting a manual seed

Matt_R: <p>Possibly a silly question as I’m very much a noob, but I’m trying hard to learn.  It’s my understanding that if you manually set a seed number, when printing the seed, it should not change.  What am I missing here?</p>
<pre><code class="lang-auto">import torch

print(f"Version: {torch.__version__}")

generator = torch.Generator(device="cuda").manual_seed(30)
print(f"Seed 1: {generator.seed()}")
print(f"Seed 2: {generator.seed()}")
</code></pre>
<p>Results:<br>
Version: 2.2.0+cu121<br>
Seed 1: 1289450348342013<br>
Seed 2: 2954870728471704</p>

ptrblck: <p>From the <a href="https://pytorch.org/docs/stable/generated/torch.Generator.html#torch.Generator.seed">docs</a>:</p>
<blockquote>
<p>seed() → <a href="https://docs.python.org/3/library/functions.html#int">int</a><br>
Gets a non-deterministic random number from std::random_device or the current time and uses it to seed a Generator.</p>
</blockquote>
<p>If you want to access the internal state, use <code>get_state()</code>.</p>

Matt_R: <p>You’re the man.  Thank you and sorry for the dumb question.</p>

ptrblck: <p>It’s not a dumb question and you are welcome <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
When using FSDP ssh disconnected

dahyeontsn: <p>i am using FSDP from pytorch distributed<br>
when i set batch size 80 (98% my GPU memory allocation), my remote ssh is disconnected and tmux server also killed.</p>
<p>but when i am reduce batch size a little, it works well.</p>
<p>and this is my version</p>
<p>Linux<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/6/a69127cbf9f877be04df0325a3720a2431c98194.png" data-download-href="https://discuss.pytorch.org/uploads/default/a69127cbf9f877be04df0325a3720a2431c98194" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/6/a69127cbf9f877be04df0325a3720a2431c98194.png" alt="image" data-base62-sha1="nLwfLeXTaGAriseMXaWoyrj6j8U" width="690" height="253" data-dominant-color="161616"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1122×413 12.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Pytorch<br>
python = “^3.10”<br>
torchvision = “^0.16.0”<br>
torch = {version = “^2.1.0”, extras = [“cu121,cp310”]}<br>
matplotlib = “^3.8.1”<br>
torchaudio = “^2.1.0”<br>
tensorboard = “^2.15.1”<br>
psutil = “^5.9.7”<br>
scikit-image = “^0.22.0”<br>
pyyaml = “^6.0.1”<br>
fastapi = {extras = [“standard”], version = “^0.106.0”}<br>
pytz = “^2023.3.post1”<br>
opencv-python = “^4.8.1.78”<br>
uvicorn = “^0.25.0”<br>
joblib = “^1.3.2”<br>
scipy = “^1.11.4”<br>
pandas = “^2.1.4”<br>
seaborn = “^0.13.0”<br>
thop = “^0.1.1.post2209072238”<br>
protobuf = “3.20.0”<br>
tqdm = “^4.66.1”<br>
python-multipart = “^0.0.6”<br>
librosa = “0.7.0”</p>
<p>Driver and CUDA<br>
Driver Version: 535.154.05   CUDA Version: 12.2</p>

ptrblck: <p>Did you check any logs to see why your session drops? E.g. are you running out of host RAM (if you are using CPU offloading)?</p>

dahyeontsn: <p>yes i am using CPU offloading like this.</p>
<pre><code>model = FSDP(model, auto_wrap_policy=functools.partial(
    size_based_auto_wrap_policy, min_num_params=1e-2),
             cpu_offload=CPUOffload(offload_params=True),
             mixed_precision=data_type,
             backward_prefetch=BackwardPrefetch.BACKWARD_PRE,
             sharding_strategy=ShardingStrategy.SHARD_GRAD_OP,
             device_id=rank)
</code></pre>
<p>But i don’t know why but it works well now.</p>
Building pytorch from sources, error import torch

ikorifey2002: <p>Linux Rocky 8.8<br>
I have done everything from official instruction.</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/pytorch/pytorch">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/pytorch/pytorch" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/344;"><img src="https://opengraph.githubassets.com/704d9b6cd919d6e6ec6f6e36eb58f5092a94b32ed059975bf2a5ad0beda9d3f9/pytorch/pytorch" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/pytorch/pytorch" target="_blank" rel="noopener nofollow ugc">GitHub - pytorch/pytorch: Tensors and Dynamic neural networks in Python with...</a></h3>

  <p>Tensors and Dynamic neural networks in Python with strong GPU acceleration - GitHub - pytorch/pytorch: Tensors and Dynamic neural networks in Python with strong GPU acceleration</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
using conda env, wrote python setup.py develop and evrething seems to be okey, but then i start pyrhon console<br>
write: import torch and get an error<br>
from torch._C import * # noqa: F403 ImportError: /home/user2/plotnikov/pytorch/torch/lib/libtorch_python.so: undefined symbol: PyObject_CallOneArg while import torch</p>
<p>where schould i look for problems</p>
Improve visual readability of the PyTorch documentation

itdevwu: <p>Documentation of PyTorch uses the Freight Sans font for text and IBM Plex Mono for code snippets.</p>
<p>While these are well-designed fonts, the PyTorch docs have terrible visual readability, particularly for users who:</p>
<ul>
<li>Have astigmatism (a type of nearsightedness) and eye fatigue.</li>
<li>Are using a high-resolution display without scaling.</li>
<li>Are using a low-resolution display.</li>
</ul>
<p>There are several possible causes:</p>
<ul>
<li><strong>Freight Sans with a lighter font weight at the size of 1rem (16px) is too slim</strong>. The small letter spacing makes it very hard to read, especially when the user is using a low-resolution display (&lt;= 1080p) or is not scaling a high-resolution display (4K) appropriately. Users with astigmatism may suffer severe eye fatigue because slim fonts at that size have blurred edges in their vision.</li>
<li><strong>Text colors are too light and pale, while the background is too bright</strong>. The text is set to <span class="hashtag-raw">#262626</span> (main text) and <span class="hashtag-raw">#6c6c6d</span> (inline mono, secondary TOC), while the background is set to pure white, which is the brightest color. Such a design distracts users and fails to emphasize the content.</li>
<li>Freight Sans is designed to be compact, and even the code font IBM Plex Mono has been shrunk to smaller than 1rem, Freight Sans is still relatively small, causing readability issues.</li>
</ul>
<p>There are several possible solutions:</p>
<ul>
<li><strong>Use the fallback font</strong> as the frontend engineers have assigned, as PyTorch Discuss has done. <strong>Helvetica</strong> and <strong>Arial</strong> are ordinary fonts, but they provide exceptional readability. Freight Sans is great, but it should be used in PyTorch brand design, not in documentation.</li>
<li><strong>Make the background or the text darker</strong>. It is common to use lighter colors for secondary content, but in the case of PyTorch, they are usually too light to be read clearly.</li>
<li>Redesign the documentation UI. I understand that it is almost impossible to stop using Sphinx, but more recent tools such as <a href="https://Docusaurus.io" rel="noopener nofollow ugc">Docusaurus</a> by Meta perform better in terms of readability. It is possible to improve the current UI, layout, and other designs of the PyTorch docs.</li>
</ul>

ptrblck: <p>Thanks for sharing your feedback.<br>
CC <a class="mention" href="/u/jspisak">@jspisak</a></p>
How to install torchaudio cpu version?

vxml12: <p>How to install torchaudio cpu version?<br>
I’ve tried<br>
pip install torchaudio-cpu==2.0.2<br>
pip install torchaudio==2.0.2=cpu*<br>
But without success.<br>
Thanks in advance!</p>

ptrblck: <p>You can find install instructions on our <a href="https://pytorch.org/get-started/locally/">homepage</a>:</p>
<pre><code class="lang-auto">pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu
</code></pre>
Wav2vec2 model quantization error

mi_gupta: <p>i have been trying to quantise the WAV2VEC2_ASR_BASE_960H model using static_fx mode but getting the example input size error. not sure what size to give (sample input size is (1,54400) and sampling rate is 16k) . also can anybody help me to get the steps to static quantize the model</p>
<pre><code class="lang-auto">import torch
import torchaudio
bundle = torchaudio.pipelines.WAV2VEC2_ASR_BASE_960H
model = bundle.get_model()
from torch.quantization import quantize_fx
import copy
m = copy.deepcopy(model)
m.eval()
qconfig_dict = {"": torch.quantization.get_default_qconfig("fbgemm")}
model_prepared = quantize_fx.prepare_fx(m, qconfig_dict,example_inputs=torch.rand(1,54400))
</code></pre>
Vmap: inplace arithmetic error

Luigi: <p>Hello everyone, I’m struggling to understand why vmap is not working, attached you can find a minimal code that reproduces the error:</p>
<pre><code class="lang-auto">import torch

def dummy_func(amount):
    test = torch.eye(4,4)
    test[2,3] = amount
    return test

if __name__ == "__main__":
    test_input = torch.tensor([1., 2., 3., 4., 5.])

    test_output = torch.vmap(dummy_func)(test_input)
</code></pre>
<p>The error you should get is the following:</p>
<p><code>RuntimeError: vmap: inplace arithmetic(self, *extra_args) is not possible because there exists a Tensor `other` in extra_args that has more elements than `self`. This happened due to `other` being vmapped over but `self` not being vmapped over in a vmap. Please try to use out-of-place operators instead of inplace arithmetic. If said operator is being called inside the PyTorch framework, please file a bug report instead.</code></p>
<p>Any help would be very appreciated! <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=12" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:" loading="lazy" width="20" height="20"></p>

marksaroufim: <p>The error you’re getting is complaining about <code>please try to use out-of-place operators</code></p>
<p>So you can’t have a write to an existing tensor</p>

Luigi: <p>Thanks for your reply! Do you have a fix to suggest?</p>
How to make a virtual step without truly updating weights and use the loss after the virtual step to update the original weights?

voldenuit-yanagi: <p>Assume that the original weight is w1 and the loss is L1. After a virtual step, the updated (not truly updated) weight is w2 and the according loss is L2. How to calculate d(L2)/d(w1) and use this to update w1? I think it might be something like this (d(L2)/d(w2)) * (d(w2)/d(w1)), but don’t know how to implement it.</p>
<p>Thanks a lot!</p>
Multiply two features in a convolutional neural network

KVV: <p>I am trying to build a convolutional neural network that takes two images (say, x and y) as input and predicts a numerical output (say, T). Intuitively, I know that the output T is a function of (some convolution of x)*(some convolution of y), * being element-wise multiplication.<br>
I know I can concatenate them but it is not working well enough. I want to element-wise multiply them instead of concatenate.My question is if I can multiply convolutional channels element wise without breaking auto-grad ? For example, can I do something like this without breaking auto-grad ? If not, how should I handle it ?</p>
<p>def forward(x,y):<br>
x = F.relu(self.convX1(x))<br>
x = F.relu(self.convY2(x))</p>
<p>y = F.relu(self.convB1(y))<br>
y = F.relu(self.convB2(y))</p>
<p>z = torch.mul(x,y)<br>
z = torch.flatten(z,1)<br>
z = F.relu(self.linear1(z)) <span class="hashtag-raw">#Maybe</span> a few more linear layers with the last layer giving one output<br>
return z</p>

J_Johnson: <p>Welcome to the forums!</p>
<p>Anytime you use a PyTorch operation, as long as the operation is differentiable, it will be tracked appropriately with autograd.</p>
<p>In fact, many standard Python arithmetic operators will be tracked with autograd, if the underlying variables are torch tensors. In other words you could also do this:</p>
<pre><code class="lang-auto">z = x * y #elementwise multiplication
z = x + y #elementwise addition
</code></pre>
Some questions about malloc Block in torch

jiangxiaobin96: <p>In Block* malloc function, when request size is less than 1MB, it will get block from small_blocks(BlockPool) in get_pool func. However, get_allocation_size will return 2MB.<br>
So here why get_allocation_size function needs to change size less than 1M to 2M, change size less than 10M to 20M?</p>
<pre><code class="lang-auto">Block* malloc(int device, size_t orig_size, cudaStream_t stream) {
...
    size_t size = round_size(orig_size); 
    auto&amp; pool = get_pool(size, stream);
    const size_t alloc_size = get_allocation_size(size);
    AllocParams params(device, size, stream, &amp;pool, alloc_size, stats);

    bool block_found =
        // Search pool
        get_free_block(params)
        // Trigger callbacks and retry search
        || (trigger_free_memory_callbacks(params) &amp;&amp; get_free_block(params));
...
}
</code></pre>

ptrblck: <p>The caching allocator uses a small and large pool creating blocks which are hopefully reusable. Creating custom allocations for each tensor with the exact size might limit the ability to reuse it and thus potentially increasing memory fragmentation. You can adapt the sizes via an env variable if needed.</p>

jiangxiaobin96: <p>Thanks for your explanation, it helps a lot.</p>
Processes get blocked though using async all-reduce

RoyChen: <p>Hi there, I am trying to use <strong>asynchronous all-reduce in torch.distributed</strong>, which is introduced in <a href="https://pytorch.org/docs/stable/distributed.html#synchronous-and-asynchronous-collective-operations" rel="noopener nofollow ugc">Pytorch Docs</a>. <strong>However, I found the processes still get blocked thought I set async_op=True.</strong> Can someone tell me where did I go wrong? <img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=12" title=":thinking:" class="emoji" alt=":thinking:" loading="lazy" width="20" height="20"></p>
<p>I copied the example code provided by Docs, adding some sleep and print commands to check if it is blocking.</p>
<pre><code class="lang-auto">import torch
import torch.distributed as dist
import os
import time

rank = int(os.getenv('RANK', '0'))
dist.init_process_group(
        backend='nccl',
        world_size=2,
        rank=rank,
        )

output = torch.tensor([rank]).cuda(rank)
if rank == 1:
       time.sleep(5)

s = torch.cuda.Stream()
print(f"Process {rank}: begin aysnc all-reduce", flush=True)
handle = dist.all_reduce(output, async_op=True)
# Wait ensures the operation is enqueued, but not necessarily complete.
handle.wait()
# Using result on non-default stream.
print(f"Process {rank}: async check")
with torch.cuda.stream(s):
    s.wait_stream(torch.cuda.default_stream())
    output.add_(100)
if rank == 0:
    # if the explicit call to wait_stream was omitted, the output below will be
    # non-deterministically 1 or 101, depending on whether the allreduce overwrote
    # the value after the add completed.
    print(output)
</code></pre>
<p>output:<br>
Process 0: begin aysnc all-reduce<br>
Process 1: begin aysnc all-reduce<br>
Process 1: async check<br>
Process 0: async check<br>
tensor([101], device=‘cuda:0’)</p>
<p>I expect ‘Process 0: async check’ should be printed before ‘Process 1: begin aysnc all-reduce’. Where did I go wrong?</p>
<p>p.s. It seems to be a nccl problem because I got expected output when using gloo.</p>

RoyChen: <p>I just found that I set CUDA_LAUNCH_BLOCKING=1 …<br>
Delete this line and everything works well.</p>
Accessing CTX from backward hook?

arushram14: <p>I would like to define a backward hook function that computes per-example gradients (to compute some other statistics). It looks like register_full_backward_hook has hook functions that have input args (module, grad_input, grad_output) but no ctx. Would there be any way to access the tensors saved in ctx from the hook so I can compute the gradients as desired?</p>

ptrblck: <p>I don’t believe this is possible but you could write a custom <code>autograd.Function</code> which would allow you to store intermediates and use them in the backward. Depending on what you want to store you might need to re-implement the forward computation, too.</p>

arushram14: <p>Okay got you. That was what I was thinking too but I just wanted to check if doing a separate forward pass each time would be necessary. Thanks!</p>
[Query] What is the use of different installation packages?

Srinjoy_Mukherjee: <p>Hello All, I want to know the use of different packages that are installed for pytorch?</p>
<pre><code class="lang-auto">conda install pytorch torchvision torchaudio pytorch-cuda=11.8 -c pytorch -c nvidia
</code></pre>
<p>I am currently working on a project which has following installation command for pytorch.</p>
<pre><code class="lang-auto">conda install pytorch==1.6.0 torchvision==0.7.0 cudatoolkit=10.2
</code></pre>
<p>What is the use of :</p>
<ol>
<li>pytorch-cuda=11.8</li>
<li>cudatoolkit=10.2</li>
</ol>
<p>and what is the difference between them?</p>
<p>Thanks and Apologies if the questions are too trivial <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_frowning_face.png?v=12" title=":slightly_frowning_face:" class="emoji" alt=":slightly_frowning_face:" loading="lazy" width="20" height="20"></p>

ptrblck: <p>The main difference is the used CUDA version and the corresponding packaging.<br>
In the past we relied on the <code>cudatoolkit</code> conda package and switched to <code>pytorch-cuda</code> metapackage defining all dependencies.</p>

Srinjoy_Mukherjee: <p>Thank you for the reply, <a class="mention" href="/u/ptrblck">@ptrblck</a></p>