akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Do we have plan to add an `all_to_all_object_list` in torch.distributed?

Fei_Liu: <p>Currently the <code>torch.distributed</code> package only has <code>all_to_all_single</code> and <code>all_to_all</code> that support tensor communications rather than for general (picklable) objects. Are we planning to add the general object support, just as the cases for scatter and gather? It can be useful.</p>
R2 score return -inf

Viktor_Todosijevic: <p>So I want to evaluate my regression model that has predictions and targets of the shape (batch_size, 236) and I pass this into torchevals r2_score</p>
<pre><code class="lang-auto">from torcheval.metrics.functional import r2_score
score = r2_score(predictions, targets).item()
</code></pre>
<p>And I get -inf which is definitely not what I was expecting. I was using this same data in sklearn and when calling <code>reg.score(predictions, targets)</code> which should calculate the same thing I think, I had no problems, sure enough.</p>
<p>Am I doing something wrong? Is the high dimensionality screwing it up?</p>

ptrblck: <p><code>-inf</code> would be expected if the target is constant for imperfect predictions.<br>
From the <code>sklearn.metrics.r2_score</code> <a href="https://scikit-learn.org/stable/modules/generated/sklearn.metrics.r2_score.html">docs</a>:</p>
<blockquote>
<p>In the particular case when <code>y_true</code> is constant, the score is not finite: it is either <code>NaN</code> (perfect predictions) or <code>-Inf</code> (imperfect predictions). To prevent such non-finite numbers to pollute higher-level experiments such as a grid search cross-validation, by default these cases are replaced with 1.0 (perfect predictions) or 0.0 (imperfect predictions) respectively. You can set <code>force_finite</code> to <code>False</code> to prevent this fix from happening.</p>
</blockquote>
<p>Here is a code snippet showing this result:</p>
<pre data-code-wrap="python"><code class="lang-python">y_true = np.full((100,), fill_value=1.)
y_pred = np.random.randn(100)

print(sklearn.metrics.r2_score(y_true, y_pred))
# 0.0

print(sklearn.metrics.r2_score(y_true, y_pred, force_finite=False))
# -inf
</code></pre>
<p>I don’t see a similar argument in <a href="https://pytorch.org/torcheval/main/generated/torcheval.metrics.R2Score.html#torcheval.metrics.R2Score"><code>torcheval.metrics.R2Score</code></a> so check if your target could be constant.</p>
Could not convert string to float: 'tensor(0.3494,'

nima_pw: <p>Hi,</p>
<p>I’ve saved my model train and validation accuracy using <code>with open( )...  as f , f.close()</code> in <code>.txt</code> format. The output is like this:</p>
<pre><code class="lang-auto">tensor(0.3494, device='cuda:0', dtype=torch.float64)
tensor(0.4721, device='cuda:0', dtype=torch.float64)
tensor(0.5966, device='cuda:0', dtype=torch.float64)
tensor(0.7131, device='cuda:0', dtype=torch.float64)
tensor(0.8192, device='cuda:0', dtype=torch.float64)
tensor(0.9051, device='cuda:0', dtype=torch.float64)
tensor(0.9604, device='cuda:0', dtype=torch.float64)
.
.
.
</code></pre>
<p>Now I want to plot the results:</p>
<pre><code class="lang-auto">accs_trn = np.loadtxt('Test/accs_trn.txt', dtype=float)
accs_val = np.loadtxt('Test/accs_val.txt', dtype=float)
plt.figure(figsize=(12, 4))
plt.plot(accs_trn[0][0])
plt.plot(accs_val[0][0])
plt.xlabel('Iteration')
plt.ylabel('Accuracy')
plt.title('model')
plt.show()
</code></pre>
<p>But I get this error:<br>
<code>ValueError: could not convert string to float: 'tensor(0.3494,'</code></p>
<p>I know what this error means but I don’t know how to handle it.<br>
Can you please help me with how to use these <code>.txt</code> files to plot?</p>

ptrblck: <p>Since you’ve stored the data as strings you could use regex to match the actual floating point value and transform it into a <code>float</code> literal.</p>

nima_pw: <p>Dear <a class="mention" href="/u/ptrblck">@ptrblck</a>,<br>
Thanks for your reply.</p>
<p>Can you help me with how I should save the accuracy of the model so I can be able to reload it and plot it easily?</p>

ptrblck: <p>You could save the binary data directly in numpy or PyTorch.<br>
Here is a simple example:</p>
<pre><code class="lang-python">accuracy = torch.tensor([0.876])
print(accuracy)
# tensor([0.8760])

torch.save(accuracy, "acc.pt")

loaded = torch.load("acc.pt")
print(loaded)
# tensor([0.8760])

# same for numpy
acc_np = accuracy.numpy()
with open("acc_np.npy", "wb") as f:
    np.save(f, acc_np)

np_loaded = np.load("acc_np.npy")
print(np_loaded)
# [0.876]
</code></pre>
Inserting a Function and Obtaining Pre-Shuffle datapipe

Jay2: <p>Hi,</p>
<p>I’m currently working with PyTorch’s DataPipe module and I have a specific requirement for my data preprocessing pipeline. I’m using the following DataPipe: <code>IterableWrapper(range(100 * 100)).shuffle().sharding_filter().map(lambda x: np.array([x, x + 1])).batch(20)</code>.</p>
<p>However, I need to make two modifications to this pipeline. Firstly, I would like to obtain the data before the <code>shuffle()</code> operation is applied. Secondly, I want to insert a custom function, <code>fn3</code>, between the <code>sharding_filter()</code> and <code>map(lambda x: np.array([x, x + 1]))</code> operations. The desired result would be:</p>
<p><code>IterableWrapper(range(100 * 100)).shuffle().sharding_filter().filter(fn3).map(lambda x: np.array([x, x + 1])).batch(20)</code></p>
<p>Could you please guide me on how to achieve these modifications? Any help would be greatly appreciated.</p>
CUDA getting out of memory

Abhishek_Gupta: <p>After training the model, I was going to test the model by inputting one image. But I don’t know why Google Colab GPU is getting out of memory. Below is the code<br>
<code>model = Yolov1(split_size = 7,num_boxes = 2,num_classes = 2).to(device)</code><br>
<code>model.load_state_dict(torch.load("/content/model.pth"))</code><br>
<code>model.eval()</code></p>
<blockquote>
<p><code>img = cv2.imread("/content/1589731926566.jpg",cv2.COLOR_BGR2RGB)</code></p>
</blockquote>
<blockquote>
<p><code>image = Image.open("/content/1589731926566.jpg")</code><br>
<code>image.resize((448,448))</code><br>
<code> l = ToTensor()</code><br>
<code>image = l(image)</code><br>
<code> image = image.unsqueeze(0)</code><br>
<code>image = image.to(device)</code></p>
</blockquote>
<blockquote>
<p><code>img = cv2.resize(img,(448,448))</code></p>
</blockquote>
<p>Here <code>img</code> is for plotting bounding boxes and <code>image</code> is for inputting in model</p>
<p>GPU crashes after this line<br>
<code>y = model(image)</code></p>
<p>Please suggest anything</p>
<p>P.S. Please tell any shortcut so that whole paragraph converts each line in this format `` at once</p>
<p>Thank You</p>

eqy: <p>Do you know how much memory your instance has? It could be that there isn’t enough memory to run the model while keeping intermediate activations in memory.</p>
<p>If you are just trying inference for now, you could try wrapping the model execution e.g, <code>torch.no_grad()</code> e.g.,</p>
<pre><code class="lang-auto">with torch.no_grad():
    y = model(image)
</code></pre>
<p>which should hopefully save some memory as intermediate activations will no longer be kept in memory.</p>
<p>You can use three backticks “```” at the beginning and end of a code block to format it.</p>

Abhishek_Gupta: <p>Thank you very much! It worked</p>
<p>Thank for the help <img src="https://discuss.pytorch.org/images/emoji/apple/innocent.png?v=12" title=":innocent:" class="emoji" alt=":innocent:" loading="lazy" width="20" height="20"> <img src="https://discuss.pytorch.org/images/emoji/apple/innocent.png?v=12" title=":innocent:" class="emoji" alt=":innocent:" loading="lazy" width="20" height="20"></p>
Understanding pin_memory performance boosts

Fei_Liu: <p>I have my training data on CPU memory. I tested datasets and dataloaders with the following three scenarios:</p>
<ol>
<li>Do not use pin_memory at all</li>
<li>Use pin_memory in dataloader, along with async CPU=&gt;GPU tensor copy before model forward call. But the original tensor in the dataset (on CPU) is not pinned.</li>
<li>Pin the whole tensor in advance before feeding it to dataset (then it doesn’t matter whether to set the pin_memory in dataloader), and also use async copy as in 2.</li>
</ol>
<p>I noticed that 2 &amp; 3 gives significant time savings ( &gt; 30% in this test case), but 2 and 3 have almost idential time cost. I want to understand why.</p>
<p>My understanding of the pin_memory is that it moves the CPU tensor to non-pageable CPU memory, which a. does the extra pageable=&gt;non-pageable copy in advance, b. enables async copy to GPU mem. If that’s the case, wouldn’t 3 above give even more savings by avoiding pinning tensors again and again in the dataloader iterations?</p>
<p>Since I’m not seeing any improvements of 2=&gt;3, is it fair to say that the “pin” operation itself is not of great cost (so it’s OK to repetitively do it in dataloader iterations) ? The savings come from the async/batch operations after the pin, and the pay (the pin itself) is almost free. Is this true in general? And are there any cases that it might be beneficial to pre-pin the whole dataset to avoid repetitive pins in the dataloader (as in 3 above) ?</p>

Fei_Liu: <p>Or, it could be the other way around?:<br>
pre-pin the whole CPU dataset is always a bad idea, you should just do the on-the-fly pin as in the standard dataloader (which will give almost identical performance and avoid other headaches from pre-pin a huge dataset) ?</p>
Weights for Bidirectional LSTM

yushu: <p>Hello!</p>
<p>I am implementing the LSTM since I want to modify its weights during the training. The naive LSTM works pretty well, but when I come to the bidirectional LSTM, its weights confuse me.</p>
<p>As the <a href="https://pytorch.org/docs/stable/generated/torch.nn.LSTM.html" rel="noopener nofollow ugc">docs</a> introduced, for the bidirectional version, its <strong>weight_ih_l[k]</strong> already has the shape <code>(4*hidden_size, num_directions * hidden_size)</code> which I suppose it contains the weights for the forward direction and backward direction since the <code>num_directions * hidden_size</code>.  But it also has <strong>weight_ih_l[k]_reverse</strong>. I am wondering if this weight is redundant.</p>
<p>Since the official LSTM is fully written in C++, it is not easy for me to understand the source code (or find it <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"> ).</p>
<p>Thanks!</p>
Delivering PyTorch Inference acceleration as plug-in to PyTorch

kumardeepakamd: <p>Instead of modifying PyTorch code to incorporate acceleration for a specific target hardware (say AMD CPU) and building PyTorch code and delivering modified PyTorch (or upstreaming later on), is there a way to deliver acceleration as a plug-in where a user can install a plug-in on top of an already installed PyTorch version? Any pointer to the right way of doing so would be great. Thank you!</p>

marksaroufim: <p>Tensor subclasses should be what you’re looking for <a href="https://github.com/albanD/subclass_zoo/blob/main/new_device.py" class="inline-onebox" rel="noopener nofollow ugc">subclass_zoo/new_device.py at main · albanD/subclass_zoo · GitHub</a></p>
Model Sometimes Takes Significantly Longer to Iterate Through

tempaccountforissue: <p>I’m pretty new to PyTorch, so please excuse my lack of knowledge on this subject. I’m working with <a href="https://github.com/openai/shap-e" rel="noopener nofollow ugc">OpenAI’s Shape-E</a>. It runs on CUDA, and is usually able to run pretty fast (about 15-30 seconds). However, sometimes it randomly takes significantly longer, (about 6-7 minutes) and I have no idea what is causing this. I’ve tried restarting my PC, but this hasn’t fixed the issue. Are there common causes for issues like this? How could I go about debugging it?</p>
Weighted MSE behaving opposite of expectation

ray-l: <p>Hi,</p>
<p>I’m pretty new to ML in general, and I’m wondering if there’s something wrong (either conceptually or practically) with my code</p>
<p>I’m feeding input through some CNNs, flattening layers, some hidden layers and eventually output to a 1D vector of some dimension (all strictly positive).</p>
<p>I want the model to fit to a particular index of the output vector, therefore implemented the following weighted MSE:</p>
<pre><code class="lang-auto">class Weighted_MSE(nn.Module):

    def __init__(self, weights):
        self.weights = weights
        super(Weighted_MSE, self).__init__()
 
    def forward(self, inputs, targets):   
        MSEW = torch.mean(self.weights * (inputs - targets) ** 2)
        return MSEW
</code></pre>
<p>If I want the model to fit to output[0] then the weights will look like:</p>
<pre><code class="lang-auto">tensor([4., 1., 1., 1., 1., ....])
</code></pre>
<p>But in practice, what I found out is that if the weights are small, the model behaves like a regular MSE,  Once the weight increases beyond certain threshold, the model no longer fits to the output index, and instead just always output ‘0’ on that index.<br>
This is exactly the opposite of what I expected to happen. Because all the training output values are strictly greater than 0, this seems to amplify the training errors, rather than reducing them.</p>
<p>Any help is appreciated!</p>
What is the sampling rate of the image classifier with DP in the tutorial examples?

Bo_Chen1: <p>Hi, I am running the tutorial example: <a href="https://github.com/pytorch/opacus/blob/main/tutorials/building_image_classifier.ipynb" class="inline-onebox" rel="noopener nofollow ugc">opacus/building_image_classifier.ipynb at main · pytorch/opacus · GitHub</a>. I noticed that for the privacy engine we don’t need to specify the sampling rate since it is computed by batch size automatically. So my question is how does batch size decide the sampling rate?</p>

Bo_Chen1: <p>I came across this post <a href="https://discuss.pytorch.org/t/the-opacus-example-train-batch-size-vs-sampling-rate/158771/3" class="inline-onebox">The Opacus example, train batch size vs sampling rate - #3 by ashkan_software</a>, where it seems like sampling rate = batch size / len(dataset). But when I check the UniformWithReplacementSampler <a href="https://github.com/pytorch/opacus/blob/main/opacus/utils/uniform_sampler.py" class="inline-onebox" rel="noopener nofollow ugc">opacus/uniform_sampler.py at main · pytorch/opacus · GitHub</a>, it seems like sampling rate = 1 / batch size there. I am a newbie in this area so please correct me if I am wrong.</p>

Bo_Chen1: <p>After carefully reviewing I think in both places sampling rate = batch size / len(dataset).</p>
Issue with Running D2Go Model on Android Using PyTorch Android Lite

Rachel_Bergman: <p>Hello,</p>
<p>I trained and exported a model using Detectron2’s D2Go with the following environment:</p>
<ul>
<li>PyTorch version: 2.0.1+cu118</li>
<li>CUDA version (used by PyTorch): 11.8</li>
<li>torchvision version: 0.15.2+cu118</li>
<li>OpenCV version: 4.7.0</li>
<li>NumPy version: 1.24.1</li>
<li>detectron2 version: 0.6</li>
<li>Python version: 3.10</li>
</ul>
<p>The model performed excellently in my tests, including when I uploaded and tested the exported mobile model.</p>
<p>However, I’m encountering an issue when trying to use the model on Android. My Android environment uses:</p>
<ul>
<li>PyTorch Android Lite version 1.13.1</li>
<li>TorchVision Ops version 0.14.0</li>
</ul>
<p>I was able to load the model successfully, but when I try to perform a forward pass for detection, I consistently receive a tuple of size 4, with each element of the tuple being empty.</p>
<p>I suspect that the issue might be due to version discrepancy, where the versions used to create the model are more advanced than the versions I’m using on Android.</p>
<p>Is there a light version of PyTorch Android Lite compatible with PyTorch version 2.0.1+cu118? Or is there any other potential solution or workaround to this issue?</p>
<p>Thank you in advance for any help or guidance.</p>
What does pin_memory_device do in DataLoader

Fei_Liu: <p>(The documentation looks way too high level and I didn’t find much useful by search engine.)</p>
<p>Is it that pin to a specific device (e.g. <code>'cuda:0'</code>) is faster (for later data transfers to device) than a general pin if I know which device it’s going to? Also Is this equivalent to <code>Tensor.pin_memory(device)</code> ? (It appears that <code>Tensor.pin_memory</code> accepts an optional <code>device</code> argument, although it’s not in the official documentation.)</p>

ptrblck: <p>From <a href="https://discuss.pytorch.org/t/attributeerror-dataloader-object-has-no-attribute-pin-memory-device/170129/2">this topic</a> it’s a placeholder for other backends which might support pinned memory.</p>
Grad through frozen weights

Joshua_Clancy: <p>I am a little confused, please help enlighten me.</p>
<p>Imagine an image model with cats and dogs data. I have seen models where there is a <em>trainable</em> autoencoder recreating the image, and then the output is passed through a <em>frozen</em> resnet encoder, with a <em>trainable</em> top linear layer categorizing cat and dog.</p>
<p>That is, we have the input travel through trainable layers, frozen layers, and then trainable layers again. <strong>How can that possibly train? What are the general rules around training through frozen weights?</strong> I have tested training an autoencoder model through a frozen decoder… that seems to work. But adding a trainable layer afterward seems to me to break the chain.</p>
<p>Follow up related question, I have seen situations where the input variable/tensor has a grad, while the weights/bias don’t have a grad (are frozen). Is this what allows differentiation through frozen layers to work?</p>
<p>Thanks!!!</p>

ptrblck: <p>Autograd will make sure to continue the backpropagation to the first parameter, which needs gradients even if frozen layers are between other trainable layers.<br>
You could set the <code>requires_grad</code> attribute to <code>True</code> for the input, but that would force Autograd to backpropagate to the input, whether or not some layers are frozen.<br>
If you don’t need the gradient in your input tensor, I would recommend not to set <code>requires_grad=True</code> for it.</p>

kmsd: <p>In this case, is the backpropagation through the frozen layers the same as through the trainable layers?  Or does autograd not take the derivative with respect to the frozen weights (only taking it with respect to the activations), in which case the computational load on frozen layers is half(?) that of the trainable layers?</p>
<p>In my case, I have a Transformer encoder (RoBERTa) with a trainable embedding layer, and everything subsequent is frozen.  I want to know whether the backwards pass is the same as if everything were trainable, being that the embeddings need to be updated, or whether significant computation is saved on the backwards pass since all the transformer layers (and everything except the embeddings) are frozen.</p>

ptrblck: <aside class="quote no-group" data-username="kmsd" data-post="3" data-topic="84672">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kmsd/48/60992_2.png" class="avatar"> kmsd:</div>
<blockquote>
<p>Or does autograd not take the derivative with respect to the frozen weights (only taking it with respect to the activations)</p>
</blockquote>
</aside>
<p>This is the case as seen in <a href="https://discuss.pytorch.org/t/for-freezing-certain-layers-why-do-i-need-a-two-step-process/175289/2">this post</a> where the <code>wgrad</code> kernels are missing if I freeze an intermediate layer while the <code>dgrad</code> kernels are still visible.</p>
What are the appropriate torch versions to use when using torch-tensorrt?

chrisuz: <p>When I download torch==2.0.1, I am missing certain libtorch files to import torch_tensorrt. Namely:</p>
<p>from torch_tensorrt._C import dtype, DeviceType, EngineCapability, TensorFormat<br>
ImportError: libtorch_cuda_cu.so: cannot open shared object file: No such file or directory</p>
<p>but when I revert to torch 1.13 as pip instructs me to it either automatically redownloads the latest torch, putting me back at square 1, or the boilerplate code for running tensorrt does not work. Would anyone be able to advise?</p>

ptrblck: <p>Double post from <a href="https://discuss.pytorch.org/t/no-libtorch-cuda-cpp-so-available-when-build-pytorch-from-source/159864/8">here</a>.</p>
<p>Based on the <code>Torch-TensorRT</code> <a href="https://github.com/pytorch/TensorRT/releases">release notes</a> it seems <code>torch==1.13</code> is the latest supported version which is also being downloaded if I install <code>torch_tensorrt==1.3.0</code>:</p>
<pre><code class="lang-python">Processing ./torch_tensorrt-1.3.0-cp310-cp310-linux_x86_64.whl
Collecting tensorrt&lt;8.6.0,&gt;=8.5.1.7
  Downloading tensorrt-8.5.3.1-cp310-none-manylinux_2_17_x86_64.whl (549.5 MB)
     |████████████████████████████████| 549.5 MB 5.0 MB/s 
Collecting torch&lt;1.14.0,&gt;=1.13.0
  Downloading torch-1.13.1-cp310-cp310-manylinux1_x86_64.whl (887.5 MB)
...
</code></pre>
<p>CC <a class="mention" href="/u/narendasan">@narendasan</a> to correct me in case the support was extended.</p>

chrisuz: <p>thanks, this worked, with torchvision==0.14.0, still getting issue with the boilerplate code itself but this worked</p>
Autoencoder with conventional layers with pytorch

drdata: <p>hello guys ,<br>
i’m trying to use autoencoder with conventional layers with pytorch<br>
I got this error<br>
RuntimeError: Given groups=1, weight of size [16, 1, 3, 3], expected input[1, 53400, 122, 1] to have 1 channels, but got 53400 channels instead</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/4/a41cabff3a25c03efe88f67353b7ba7c15ed1318.png" data-download-href="https://discuss.pytorch.org/uploads/default/a41cabff3a25c03efe88f67353b7ba7c15ed1318" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/4/a41cabff3a25c03efe88f67353b7ba7c15ed1318_2_690x389.png" alt="image" data-base62-sha1="npNJwN83jWDr1cnRgYo24E1zSlO" width="690" height="389" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/4/a41cabff3a25c03efe88f67353b7ba7c15ed1318_2_690x389.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/4/a41cabff3a25c03efe88f67353b7ba7c15ed1318_2_1035x583.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/a/4/a41cabff3a25c03efe88f67353b7ba7c15ed1318_2_1380x778.png 2x" data-dominant-color="F3F3F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2204×1244 189 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Based on your code and error message it seems you might be using a channels-last memory layout while channels-first is expected.<br>
If so, <code>permute</code> the inputs and pass the input as <code>[batch_size, channels, height, width]</code> to the model.</p>
How can I install with pytorch bugfix without compile

FFengIll: <p>For example, 2.0.1 update with bugfix on mps, but it is hard to use pip / conda to install the fixed version (since no update on registry)</p>
<blockquote>
<p><a href="https://github.com/pytorch/pytorch/releases/tag/v2.0.1" class="inline-onebox" rel="noopener nofollow ugc">Release PyTorch 2.0.1 Release, bug fix release · pytorch/pytorch · GitHub</a></p>
</blockquote>

ptrblck: <p>If the fix touches purely Python files you could directly manipulate them in the installed folder of your binaries. To do so you can search for the corresponding files in <code>torch.__path__</code>.<br>
However, if source files were changes which need to be compiled, such as <code>.cpp</code> files, you would need to rebuild PyTorch from source.</p>
SequenceTaggingDataset equivalent with the new torchtext version

giturra: <p>Hello everyone,</p>
<p>I am preparing a dataset for a sequence labeling task, but I noticed that torchtext API was updated, depreciating the <code> legacy</code> package. So my question is, is there a way of getting the separation dataset (train, valid, and test) similar to the <code>legacy.datasets.SequenceTaggingDataset.splits</code> function?</p>
<p>I was checking out the documentation, but I couldn’t find something similar to this.</p>
<p>Thank you for any advice or suggestion.</p>
<pre><code class="lang-auto">train_data, valid_data, test_data = legacy.datasets.SequenceTaggingDataset.splits(
    path="./",
    train="train.txt",
    validation="dev.txt",
    test="test.txt",
    fields=fields,
    encoding="utf-8",
    separator=" "
)
</code></pre>
Captum with BERT on AWS inferentia

Ajay_Nair: <p>Hi All,</p>
<p>I have to compute gradient on BERT model on inferentia. For this I guess I also need access to the hidden layers. Im currently not able to proceed because of not finding literature on the net for inferentia. There seems to be some non inferentia links but those do not work for this case.</p>
<p>Amongst the things which are missing are how to get positional embeddings.</p>
<p>Or maybe using captum there may be another way.<br>
Please provide some code snippets to help me out.</p>
<p>Thanks in advance Ajay</p>
Simple custom loss with exception on backward

adm: <p>Hi,</p>
<p>I am writing a custom loss function that minimizes the values based on some intermediate matrix. Simply model returns 3 values. The index of the highest value is the index of the value that should be used from an intermediate matrix (mm), so in short, loss = sum([<strong>mm[argmax(pred)]</strong> for pred in output]). I wrote such a custom function, and the backward steps raise an exception.</p>
<p>MWE:</p>
<pre><code class="lang-auto">def my_loss(inp, target):
    loss = F.softmax(inp, dim=1)
    loss = torch.argmax(loss, dim=1)
    loss = torch.gather(target, 1, loss.view(-1, 1))  # Gather the corresponding values from target
    loss = loss.sum()
    return loss

# values used to calculate loss
mm = torch.tensor([
    [0.3, 0.8, 1.3],
    [0.1, 0.2, 0.4]
])

# input
inp = torch.rand((2, 5)) # 2 sameples x 5 features

# model
model = nn.Linear(5, 3)

# returns 3 values where the argmax is the index of value in mm that should be used to calculate the loss
pred = model(inp)

# calculate loss
loss = my_loss(pred, mm)

loss.backward()
</code></pre>
<p>Error:</p>
<pre><code class="lang-auto">RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
</code></pre>
<p>The issue is probably with the <code>gather</code> function, as it takes the target for gradient calculation. Any idea how to solve it?</p>

KFrank: <p>Hi adm!</p>
<aside class="quote no-group" data-username="adm" data-post="1" data-topic="181270" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/adm/48/12127_2.png" class="avatar"> adm:</div>
<blockquote>
<p>loss = sum([<strong>mm[argmax(pred)]</strong></p>
</blockquote>
</aside>
<p>Your problem is that <code>argmax()</code> returns an  integer, so it isn’t differentiable and<br>
using it “breaks the computation graph.”</p>
<p>The following may or may not make sense for your specific use case:</p>
<p><code>softmax()</code>, conceptually, might be better named “softargmax()” (or even<br>
“soft-one-hot-encoded-argmax()”).  You could consider using:</p>
<pre data-code-wrap="python"><code class="lang-python">loss = (F.softmax (inp, dim = 1) * target).sum()
</code></pre>
<p>as a differentiable “soft” version of your loss function.  You could introduce a<br>
parameter, <code>loss = (F.softmax (alpha * inp, dim = 1) * target).sum()</code>,<br>
where larger values of <code>alpha</code> sharpen (make “harder”) the “softargmax()” in<br>
your loss function.</p>
<p>Best.</p>
<p>K. Frank</p>
I am seeing an error, when trying to access camera with pyroch in google co lab?

Teddy_Salas: <p>Hai i am trying to detect objects through laptop camera using PyTorch in google co lab, when i run  cv2 code, i am seeing an error, can anyone help me to solve the problem, here is the code:-</p>
<p>import cv2<br>
import torch<br>
import torchvision.transforms as transforms<br>
from torchvision.models import resnet50</p>
<p>model = model_x_5<br>
model.eval()<br>
transform = transforms.Compose([<br>
transforms.ToPILImage(),<br>
transforms.Resize(size=(64,64)),<br>
transforms.ToTensor(),<br>
transforms.Normalize(mean=[0.485,0.456,0.406], std=[0.229,0.224,0.225])<br>
])<br>
cap =  cv2.VideoCapture(0)</p>
<p>while True:<br>
ret,frame = cap.read()<br>
frame_rgb = cv2.cvtColor(frame,cv2.COLOR_BGR2RGB)<br>
input_tensor = transform(frame_rgb)<br>
input_tensor = input_tensor.unsqueeze(0)<br>
with torch.no_grad():<br>
output = model(input_tensor)<br>
predicted_idx = torch.max(output,1)<br>
labels = [“Toothbrush”]<br>
predicted_label = labels[predicted_idx.item()]<br>
cv2.putText(frame,predicted_label,(10,30),cv2.FONT_HERSHEY_SIMPLEX,1,(0,255,0),2)<br>
cv2.imshow(“object detection”,frame)<br>
if cv2.waitKey(1) &amp; 0xFF == ord(‘q’):<br>
break<br>
cap.release()<br>
cv2.destroyAllWindows()</p>
<p>error:----------------------------------------------------------------------------<br>
error                                     Traceback (most recent call last)<br>
 in &lt;cell line: 17&gt;()<br>
17 while True:<br>
18   ret,frame = cap.read()<br>
—&gt; 19   frame_rgb = cv2.cvtColor(frame,cv2.COLOR_BGR2RGB)<br>
20   input_tensor = transform(frame_rgb)<br>
21   input_tensor = input_tensor.unsqueeze(0)</p>
<p>error: OpenCV(4.7.0) /io/opencv/modules/imgproc/src/color.cpp:182: error: (-215:Assertion failed) !_src.empty() in function ‘cvtColor’</p>

ptrblck: <pre><code class="lang-python">(-215:Assertion failed) !_src.empty() 
</code></pre>
<p>indicates the image is empty and thus OpenCV fails.<br>
I’m unsure how Google Colab should be able to access your local webcam.</p>

brighteningeyes: <p>as the code runs on colab, it tries to access the camera there, which is not possible.<br>
if you want to get access to your camera, you need to work with opencv.js (and run javascript in your browser) and then send the images to colab.</p>

Teddy_Salas: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> is there anyway to google co lab can access cam or webcam using pytorch?</p>

ptrblck: <p>I don’t know but it seems <a class="mention" href="/u/brighteningeyes">@brighteningeyes</a> suggested an approach.</p>

Teddy_Salas: <p><a class="mention" href="/u/brighteningeyes">@brighteningeyes</a> i can create   opencv.js in google co lab ?, &amp; i can add model in pytorch to opencv.js like above code?</p>

brighteningeyes: <p>javascript executes on your device. you need to send it to colab for processing.<br>
if you use opencv.js, and try to execute code with javascript, it will be executed on your machine</p>
What is the correct way to grab weights for before and after comparison after backpropagation?

inkplay: <p>I am trying to grab an entire model’s weight before training and also after training to check their differences but for some reason pytorch is telling me the old stored weight is the same as the newly stored weights after my optimizer.step()?</p>
<p>My code is below, one thing to notice is I only trained the input with layer1, layer2 should not update it’s weight because it is not used. After 1 forward/backward propagation I should have different weights for layer1 and the activation, layer2’s weight should be untouched.</p>
<pre><code class="lang-auto">
class mymodel(nn.Module):
    def __init__(self):
        super().__init__() 
        self.layer1 = nn.Linear(10, 5)
        self.layer2 = nn.Conv2d(1, 5, 4, 2, 1)
        self.act = nn.Sigmoid()
    def forward(self, x):
        x = self.layer1(x) #only layer1 and act are used layer 2 is ignored so only layer1 and act's weight should be updated
        x = self.act(x)
        return x
model = mymodel()

weights = []

for param in model.parameters(): # loop the weights in the model before updating and store them
    print(param.size())
    weights.append(param)

critertion = nn.BCELoss() #criterion and optimizer setup
optimizer = optim.Adam(model.parameters(), lr = 0.001)

foo = torch.randn(3, 10) #fake input
target = torch.randn(3, 5) #fake target

result = model(foo) #predictions and comparison and backprop
loss = criterion(result, target)
optimizer.zero_grad()
loss.backward()
optimizer.step()


weights_after_backprop = [] # weights after backprop
for param in model.parameters():
    weights_after_backprop.append(param) # only layer1's weight should update, layer2 is not used

for i in zip(weights, weights_after_backprop):
    print(torch.equal(i[0], i[1]))

# **prints all Trues when "layer1" and "act" should be different, I have also tried to call param.detach in the loop but I got the same result.

</code></pre>

albanD: <p>Hi,</p>
<p>The thing is to reduce memory usage, all optimizer modify the weights inplace.<br>
Which means that the content of your list called <code>weights</code> contains the same tensors that are modified inplace by the optimizer.<br>
You can add a <code>weights.append(param.clone())</code> to make sure your store a different copy of the weights.</p>

Sumit: <p>This still isn’t working for me. The parameters in the model before and after training are the same.</p>

AlphaBetaGamma96: <p>Hi <a class="mention" href="/u/sumit">@Sumit</a>,</p>
<p>You could always do a deepcopy before your optimizing step.</p>
<pre><code class="lang-auto">from copy import deepcopy
params_before = deepcopy(net.parameters())

loss = loss_fn(...)
optim.zero_grad()
loss.backward()

params_after = deepcopy(net.parameters())

</code></pre>
Fast Neural Style Transfer

Sart: <p>I’m facing this error while training my model :<br>
AttributeError: ‘list’ object has no attribute ‘relu2_2’</p>
<p>Code :<br>
for epoch in range(1, epochs+1):<br>
transformer.train()<br>
print(‘No of Epochs:’, epoch)<br>
for batch_i, (images, _) in enumerate(train_loader):<br>
optimizer.zero_grad()<br>
images_original = images.to(device)<br>
generated = transformer(images_original)</p>
<pre><code>        features_original = vgg(images_original)
        features_transformed = vgg(generated)

        content_loss = content_weight*l2_loss(features_transformed.relu2_2, features_original.relu2_2)

        style_loss = 0
        for ft_y, gm_s in zip(features_transformed, gram_style):
            gm_y = gram_matrix(ft_y)
            style_loss += l2_loss(gm_y, gm_s[: images.size(0), :, :])
        style_loss *= style_weight

        total_loss = content_loss + style_loss
        total_loss.backward(retain_graph=True)
        optimizer.step()

        train_metrics['content'] += [content_loss.item()]
        train_metrics['style'] += [style_loss.item()]
        train_metrics['total'] += [total_loss.item()]

        print('Train Metrics Total:',train_metrics['total'])
</code></pre>
<p>Can anyone figure out why this error is coming as when I checked with other github repos on fastnst they worked perfectly with somewhat similar code. Thanks in advance for help!</p>

ptrblck: <p>Based on the error message it seems:</p>
<pre><code class="lang-python">features_transformed = vgg(generated)
</code></pre>
<p>returns a <code>list</code> which then raises the error in:</p>
<pre><code class="lang-python">l2_loss(features_transformed.relu2_2, ...)
</code></pre>
<p>so check the <code>forward</code> method of your <code>vgg</code> definition and make sure an object is returned which contains the <code>relu2_2</code> attribute.</p>

Sart: <p>Thank you for the help !!</p>
PyTorch equivalent of scipy.sparse.linalg.gmres

AlphaBetaGamma96: <p>Hi all,</p>
<p>Is there an equivalent of <code>scipy.sparse.linalg.gmres</code> within the PyTorch framework? In JAX, they seem to have a direct equivalent with <code>jax.scipy.sparse.linalg.gmres</code>, and I was wondering if there’s a way of doing this within PyTorch?</p>
<p>I’ve had a look and can’t find anything, but I may have missed it! If there’s a way of doing this in PyTorch, please do let me know!</p>
Combining two feature Vectors

Mayank: <p>Hello Everyone,</p>
<p>I am looking for a way to combine two feature vectors of same size and in result get the same size.</p>
<p>Size of Feature Map1 = ( 256, 32, 32)<br>
Size of Feature Map2 = ( 256, 32, 32)</p>
<p>In the excisting model the authors are concatenating them and hence getting the result = ( 512, 32, 32)</p>
<p>I want the result in size (256, 32, 32).<br>
But the conditions are, I can not divide them by 2 , or do Dot or Cross multiplication.</p>
<p>Can you please help me suggest any other possible ideas?</p>

ptrblck: <aside class="quote no-group" data-username="Mayank" data-post="1" data-topic="181263">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mayank/48/47503_2.png" class="avatar"> Mayank:</div>
<blockquote>
<p>But the conditions are, I can not divide them by 2 , or do Dot or Cross multiplication.</p>
</blockquote>
</aside>
<p>You could just sum these tensors together if other elementwise operations are disallowed.</p>

Mayank: <p>Thankyou <a class="mention" href="/u/ptrblck">@ptrblck</a> for your quick reply.<br>
I have been asked to some other way besides these operations ( like adding them, dividing them, multiplying them). That’s why i am little confused if there is any other possible way ?</p>
Customized loss for LSTM with variable input length

irvingf7: <p>Hi PyTorch forum! I am not very familiar with LSTM, so please bear with me on this.</p>
<p>Assuming that we are working with sequential data with variable input length and a batch size of two.</p>
<p>e.g.<br>
[1, 1]<br>
[1, 1, 1, 1, 1]</p>
<p>Now I pad these inputs to be of the same length, resulting in<br>
[1, 1, 0, 0, 0]<br>
[1, 1, 1, 1, 1]</p>
<p>I need to write a customized loss function, and the way I wrote it, the loss function would produce a loss of 0 whenever it encounters elements that are padding elements (0 in this case). So the loss for one batch would look like this:<br>
[2, 3, 0, 0, 0]<br>
[4, 1, 2, 6, 7] (all non 0 numbers are just made-up numbers)</p>
<p>I then sum them together and take the average by dividing the batch size, which is 2. The model is updated using this loss value, so the model only gets updated when all the sequences are processed entirely in this batch.</p>
<p>It suddenly occurs to me that now input with a shorter length will drag down the loss because it’s much shorter and will naturally have a smaller loss in most cases. So the average loss of this batch may not be very reflective of the performance</p>
<p>Is this a theoretically sound way to do it? Is there a better approach to handle this variable input scenario? Thank you so much!</p>

vdw: <p>You could use masking. The easiest solution would probably to simply the the loss function that you want to ignore certain values (e.g., you padding index 0).</p>
<p>For example, have a look at the <a href="https://pytorch.org/docs/master/generated/torch.nn.CrossEntropyLoss.html#torch.nn.CrossEntropyLoss" rel="noopener nofollow ugc">docs for the CrossEntropyLoss</a>. It features a parameter <code>ignore_index</code> that <em>“Specifies a target value that is ignored and does not contribute to the input gradient.”</em></p>
<p>It sounds like this is what you’re looking for.</p>
Style transfer for my own unique style transfer

GeorgeIrwin: <p>I am looking for creating a project for my own style done by Deep Learning.<br>
I have a large set of photos I manually processed by Photoshop and to automate and accelerate my work I’d like to teach neural style transfer model to learn my own style.</p>
<p>I was wondering which model is considered both modern and capable and relatively easy to tweak with my own examples.<br>
Specifically a model which can handle multiple persons within the same image.</p>
<p>To make things more concrete, I found <a href="https://github.com/menyifang/DCT-Net" rel="noopener nofollow ugc">DCT-NET</a>. Does anyone have experience with such models?</p>
How to use "break" in DistributedDataParallel training

jetcai1900: <p>I am using DistributedDataParallel to train the model on multiple GPUs. If I would like to stop the process early, how could I achieve it? Thanks.</p>

mrshenli: <p>Is this about uneven inputs on different processes? See:</p>
<ol>
<li><a href="https://github.com/pytorch/pytorch/issues/33148" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/33148</a></li>
<li><a href="https://github.com/pytorch/pytorch/issues/38174" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/38174</a></li>
</ol>
<p>If all processes know when to exit, simply break the loop would work. The tricky case is when one processes breaks the loop but other processes proceed as mentioned in the above two issues.</p>

jetcai1900: <aside class="quote no-group" data-username="mrshenli" data-post="2" data-topic="88296">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mrshenli/48/12220_2.png" class="avatar"> mrshenli:</div>
<blockquote>
<p>If all processes know when to exit, simply break the loop would work. The tricky case is when one processes breaks the loop but other processes proceed as mentioned in the above two issues.</p>
</blockquote>
</aside>
<p>Indeed this is what I meet. one process breaks the loop while others continue. The condition when the process breaks is the loss in eval dataset increases (overfitting). Do you have any ideas? Thanks.</p>

mrshenli: <p>Ideally, we should address this in DDP and close <a href="https://github.com/pytorch/pytorch/issues/38174" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/38174</a>. Before that takes place, you can use <code>all_reduce</code> synchronize some signal across all processes. See <a href="https://discuss.pytorch.org/t/multiprocessing-barrier-blocks-all-processes/80345/10" class="inline-onebox">Multiprocessing - Barrier Blocks all Processes?</a></p>
<p>One thing to note is that, this might have perf impacts, especially when the model is light and its forward pass runs faster than communicating the signal.</p>

jetcai1900: <p>Thanks for your help. Probably I would set a fixed epoch number to address this, which is simple thought is not optimal.</p>

Rakshith_V: <p>flag_tensor = torch.zeros(1).to(device)<br>
if local_rank  = 0:<br>
----------<span class="hashtag">#Conditions</span> for breaking the loop :<br>
-------------------flag_tensor += 1<br>
dist.all_reduce(flag_tensor,op=ReduceOp.SUM)<br>
if flag_tensor == 1:<br>
----------print(“Training stopped”)<br>
----------break</p>

gianlucarloni: <p>Thank you for your simple yet elegant solution. It worked. I wanted my training script to end when meeting an early stopping criterion on the validation loss, and now all the processes are correctly ended.</p>
TypeError: (): incompatible function arguments

big_old: <p>Hello，I am customizing process group backends using cpp extensions according to PyTorch Tutorials，<a href="https://pytorch.org/tutorials/intermediate/process_group_cpp_extension_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Customize Process Group Backends Using Cpp Extensions — PyTorch Tutorials 2.1.1+cu121 documentation</a> . But an error occurred . It seems to be type error, but I find the type is right. So how can I fix it? The following is the detailed error:</p>
<blockquote>
<p>Blockquote</p>
</blockquote>
<p>File “/opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py”, line 895, in init_process_group<br>
default_pg = _new_process_group_helper(<br>
File “/opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py”, line 1029, in _new_process_group_helper<br>
backend_class = creator_fn(backend_prefix_store, group_rank, group_size, timeout)<br>
TypeError: (): incompatible function arguments. The following argument types are supported:<br>
1. (arg0: c10d::Store, arg1: int, arg2: int, arg3: datetime.timedelta) → c10d::Backend<br>
Invoked with: &lt;torch.distributed.distributed_c10d.PrefixStore object at 0x7fe1ead6def0&gt;, 2, 4, datetime.timedelta(seconds=1800)</p>
CNN is overfitting - Is the standardization wrong?

J_Nic: <p>Dear all,</p>
<p>for my master thesis I want to predict three variables (Arousal, Valence, Dominance) from so called salience maps. I am running a multiple output CNN on the salience maps. However, the model is pretty much overfitting, regardless of the tuning of learning rate, batch size or complexity.</p>
<p>I am actually not sure if I did one of the more simple things wrong - the standarization.</p>
<p>The descriptive statistics for my X look like this:<br>
Mean = 0.05043895889652882,<br>
SD = 0.055531537078160255,<br>
min= 0.0,<br>
max 1.0).</p>
<p>Do I still need to standardize the variable? I am a bit confused here, because I am kind of new to Deep Learning and am getting lost in the jungle of solutions for my overfitting model.</p>
<p>Please let me know if I can provide additional information.</p>
<p>Kind regards</p>
Error when Exporting Model to Onnx format

Fox: <p>Hey there PyTorch community its my first time posting and I’m very new to ML but I’ve come across a problem which i haven’t been able to find a solution for so i decided to ask here.</p>
<p>So I’ve been testing out LSTM to learn how it works and wanted to export a model using it to onnx to test out, but whenever I try to export I get the following message with the program closing:</p>
<blockquote>
<p>“UserWarning: Exporting a model to ONNX with a batch_size other than 1, with a variable length with LSTM can cause an error when running the ONNX model with a different batch size. Make sure to save the model with a batch size of 1, or define the initial states (h0/c0) as inputs of the model.”</p>
</blockquote>
<p>upon research i haven’t found anything which has helped with this issue and honestly I don’t even know what to try considering I cant find any information online and i don’t fully understand the error message.</p>
<p>here is my code for exporting:</p>
<pre><code class="lang-auto">def export_model(model):
    with torch.inference_mode():
        x = torch.randn(1, 1, 3)
        h0 = torch.zeros(4, 1, 8)
        c0 = torch.zeros(4, 1, 8)
        torch_out, hidden = model(x, (h0, c0))
        input_names = ["input", "h0", "c0"]
        output_names = ["output", "hn", "cn"]

        torch.onnx.export(model, (x, (h0, c0)), 'lstm.onnx',
                    export_params=True,
                    input_names=['input', 'h0', 'c0'],
                    output_names=['output', 'hn', 'cn'])
    print(f"Exporting Model: lstm.onnx")
</code></pre>
<p>If anyone has any insights that would be greatly appreciated.<br>
Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>