akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
ValueError: Expected input batch_size (8) to match target batch_size (280)

LohTC: <p>Hi folks,</p>
<p>I’m newbie and got stuck by this error msg “ValueError: Expected input batch_size (8) to match target batch_size (280)”.</p>
<p>Appreciate any advice, please… Thank you!</p>
<pre><code class="lang-auto">from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased")

def preprocess_function(examples):
    model_inputs = tokenizer(
        text = examples["text"],
        max_length=max_text_length,
        truncation=True,
        padding = True,
    )
    label = tokenizer(examples["label"], max_length=max_label_length, truncation=True, padding = True)
    model_inputs["label"] = label["input_ids"]
    return model_inputs

tokenized_aicgp = aicgp.map(preprocess_function, batched=True)

from transformers import DataCollatorWithPadding
data_collator = DataCollatorWithPadding(tokenizer=tokenizer)

from transformers import AutoModelForSequenceClassification, TrainingArguments, Trainer
model = AutoModelForSequenceClassification.from_pretrained("distilbert-base-uncased", num_labels=no_of_labels, id2label=id2label, label2id=label2id)


training_args = TrainingArguments(
    output_dir="my_model",
    learning_rate=2e-5,
    per_device_train_batch_size=8, #16
    per_device_eval_batch_size=8, #16
    num_train_epochs=2,
    weight_decay=0.01,
    evaluation_strategy="epoch",
    save_strategy="epoch",
    load_best_model_at_end=True,
    push_to_hub=False,
)
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_aicgp["train"],
    eval_dataset=tokenized_aicgp["test"],
    tokenizer=tokenizer,
    data_collator=data_collator,
    compute_metrics=compute_metrics,
)
trainer.train()


Note:

The program aborted at trainer.train() with the below error message:
ValueError: Expected input batch_size (8) to match target batch_size (280).

tokenized_aicgp
DatasetDict({
    train: Dataset({
        features: ['label', 'text', 'lblId', 'input_ids', 'attention_mask'],
        num_rows: 56
    })
    test: Dataset({
        features: ['label', 'text', 'lblId', 'input_ids', 'attention_mask'],
        num_rows: 14
    })
})

length label 56
length text 56
length lblId 56
length input_ids 56
length attention_mask 56

</code></pre>

ptrblck: <p>You might need to ask in the HuggingFace discussion board as your code uses their API and doesn’t show how all tensor shapes are handled.</p>
Vectorising/Optimising Grid Image Reassembly

u7122029: <p>So I have some code that takes a tensor of shape <code>(4,3,16,16)</code> and reassembles this tensor into a shape <code>(3,32,32)</code> by grabbing the elements in dimension 0 and placing them in this order:</p>
<pre><code class="lang-auto">[[0, 1],
 [2, 3]]
</code></pre>
<p>which is essentially like grabbing a bunch of stacked jigsaw pieces and putting them back together. The code looks like this right now</p>
<pre><code class="lang-py">def reassemble_jigsaw(patches):
    """
    Reassemble an image given the jigsaw patches.
    :param patches: The jigsaw image patches (not shuffled)
    :return: Image with all the image patches put together in the order they are given.
    """
    grid_length = int(np.sqrt(len(patches)))
    patch_row_length = patches[0].shape[1]
    patch_col_length = patches[0].shape[2]
    reassembled = torch.zeros((3, patch_row_length * grid_length, patch_col_length * grid_length))
    for row in range(grid_length):
        for col in range(grid_length):
            reassembled[:,
            patch_row_length * row:patch_row_length * (row + 1),
            patch_col_length * col:patch_col_length * (col + 1)] = patches[row * grid_length + col]

    return reassembled
</code></pre>
<p>Unfortunately, I know for a fact that this is slow because it uses loops. Of course, there might not be many iterations, but the slow speed is very noticeable when reassembling many images back-to-back.</p>
<p>I tried looking everywhere for a solution and I found this <a href="https://stackoverflow.com/questions/65327458/how-can-i-fold-a-tensor-that-i-unfolded-with-pytorch-that-has-overlap" class="inline-onebox" rel="noopener nofollow ugc">python - How can I fold a Tensor that I unfolded with PyTorch that has overlap? - Stack Overflow</a>, but I couldn’t figure out how to use it in my application, and I am not sure if it is any quicker. Could I please know if there is a way to perform the same task but faster?</p>

u7122029: <p>Bump, still waiting for a response.</p>

ptrblck: <p>Using <code>(un)fold</code> is the right approach and you will also find a few examples in this forum posted by me and others. You didn’t explain where you are stuck using this approach, unfortunately.</p>
[DDP] Train different layers of one model separately

robhen: <p>Hi all,</p>
<p>I have a model that I want to train using DDP in a way where I have two sets of workers <code>W_1</code> and <code>W_2</code>.  Now a worker from <code>W_1</code> shall update only the layers <code>L_1</code>, while a worker from <code>W_1</code> shall update only the layers <code>W_2</code>. The set of layers <code>L_1</code> and <code>L_2</code> are disjoint. How can I achieve this?</p>
<p>Let see an example:</p>
<p>I have a model</p>
<pre><code class="lang-python">class DummyModel(nn.Module):

    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 1, 1)
        self.conv2 = nn.Conv2d(1, 1, 1)

    def forward(self, x):
        return self.conv2(self.conv1(x))
</code></pre>
<p>Now each worker from <code>W_1</code> shall optimise the weights of <code>MyModel.conv1</code> and each worker from <code>W_2</code> shall optimise the weights of <code>MyModel.conv2</code>.</p>
<p>Consider the following pseudo-code for two workers (with rank = 0 and rank = 1):</p>
<pre><code class="lang-python">my_model = MyModel()
opt_w1 = torch.optim.Adam(my_model.conv1.parameters())
opt_w2 = torch.optim.Adam(my_model.conv2.parameters())
if rank == 0:
    opt_w1.zero_grad()
    opt_w2.zero_grad()
    loss = my_model(input_batch, target)
    loss.backward()
    # sync and update only conv1
    opt_w1.step()
elif rank == 1:
    opt_w2.zero_grad()
    opt_w1.zero_grad()
    loss = my_model(input_batch, target)
    loss.backward()
    # sync and update only conv2
    opt_w2.step()
</code></pre>
<p>The pseudo-code is not correct however. The <code>backward()</code> computes the gradient for all weights. It would sum up all gradients and divide by the number <code>|W_1|+|W_2|</code>. To be correct according to the task, we need to sum up only the gradients from the workers <code>W_1</code> and divide by <code>|W_1|</code>  and broadcast, likewise sum up only the gradients from the workers <code>W_2</code> and divide by <code>|W_2|</code>  and broadcast.</p>
<p>How can I do this?</p>
<p>(to give you some motivation, I have a network that works with different types of input, e.g. RGB images and grayscale images, and I know for each type of input, which layers I should optimise).</p>

ptrblck: <p>Your code won’t work assuming you are using <code>DDP</code> since you are diverging the models. Model parameters are only initially shared and <code>DDP</code> depends on the gradient synchronization as well as the same parameter update to keep all models equal. In your example you are explicitly updating different parts of the model depending on the rank and will thus create different parameter sets, outputs, losses, gradients.</p>

robhen: <p>Thank you for the detailed answer!</p>
RuntimeError: Tensors of the same index must be on the same device and the same dtype except `step` tensors that can be CPU and float32 notwithstanding

ganesh1102: <p>I’ve posted this question on StackOverflow already, so I’d like to point you to the StackOverflow webpage for a fix. Thank you very much.</p>
<p><a href="https://stackoverflow.com/questions/77339169/pytorch-getting-runtimeerror-even-though-all-tensors-are-on-the-same-device" rel="noopener nofollow ugc">Link</a></p>
Reducing docker size with PyTorch model

Dawid_S: <p>I’d like to deploy four of my models with a total size of ~100mb when the state saved on disk. It would be great if the docker could take as small space as possible, no more than 700 mb.</p>
<p>Now I’m creating docker and install a few dependencies. Suddenly it takes 2.8 GB on disk, where PyTorch and related libraries take at least 800 MB in conda.</p>
<p>Therefore I’m looking for a simple way to deploy these models. As far as I understand I could use jit and e able to run models with small library libtorch. However, I’d like to run the model in Python, not C++. Is it possible?</p>
<p>And the second thing I don’t understand is whether I should use onnx or jit? What is the difference?</p>
<p>I’m looking for a simple guide with the steps to do that: export with jit and load with some lib I don’t know in Python? Or export to ONNX and then to TensorFlow? Or maybe get rid of conda and just install some PyTorch version and it should be no more than 80 MB on disk?</p>
<p>Any help appreciated!</p>

smth: <p>if you are deploying to a CPU inference, instead of GPU-based, then you can save a lot of space by installing PyTorch with CPU-only capabilities. That significantly reduces the docker image size (the pytorch component is ~128MB compressed.</p>
<p>To install CPU-only, go to <a href="https://pytorch.org">https://pytorch.org</a> and in the Install selector, select the option <code>CUDA</code> to be <code>None</code> and you will get the right set of commands.</p>

Dawid_S: <p><a class="mention" href="/u/smth">@smth</a> hank you for the help!<br>
unfortunately, having empty <code>virtualenv</code> of size 4.8 mb, after the command:</p>
<p><code>pip install https://download.pytorch.org/whl/cpu/torch-1.5.0-cp37-none-macosx_10_9_x86_64.whl</code><br>
or command<br>
<code>pip install torch</code></p>
<p>venv grows to 375 mb.</p>
<p>I’m using Python 3.7 and macOS Catalina 10.15.4</p>
<p>My lib catalog in venv looks like this:</p>
<blockquote>
<p>16M caffe2<br>
4,0K easy-install.pth<br>
3,0M future<br>
56K future-0.18.2.dist-info<br>
292K libfuturize<br>
236K libpasteurize<br>
86M numpy<br>
132K numpy-1.18.3.dist-info<br>
116K past<br>
7,3M pip-19.0.3-py3.7.egg<br>
560K setuptools-40.8.0-py3.7.egg<br>
4,0K setuptools.pth<br>
256M torch<br>
352K torch-1.5.0.dist-info</p>
</blockquote>

smth: <p>correct, that is uncompressed. when you compress it, it should be about half of that.</p>

Dawid_S: <p>I’m not familiar with any way of using it as compressed package. I mean, when I want to infer, I need to uncompress it anyway?</p>

Dawid_S: <p>I assume the best way is to convert model and use it with torchlib or export it to onnx and try there.</p>

raghad_sayed: <p>how to add it in my my requirments file ?<br>
<strong>torch torchvision torchaudio --index-url <a href="https://download.pytorch.org/whl/cpu" rel="noopener nofollow ugc">https://download.pytorch.org/whl/cpu</a></strong><br>
when add this line in the requirments file i got this error<br>
<strong><span class="hashtag-raw">#0</span> 1.175 ERROR: Invalid requirement: ‘torch torchvision torchaudio’ (from line 3 of ./requirements.txt)</strong></p>
Poisson noise to the CT image using Pytorch Tensor

MayThet: <p>I have applied Poisson noise to the CT image using the following code. However, I’ve found that the noise is fewer when I added the noise_sigma value of 50 ( lambda parameter in Poisson noise set to 50). I would appreciate your guidance and suggestions on another methods for incorporating Poisson noise into the neural network using PyTorch tensors.</p>
<blockquote>
<p>imorig = torch.FloatTensor(imorig)<br>
if args[‘add_noise’]:<br>
noise = torch.poisson(torch.ones(imorig.size()) * args[‘noise_sigma’])<br>
imnoisy = imorig + noise<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/d/9d614093262fbe0ba188355a3d13c6d0b56e62fc.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/9d614093262fbe0ba188355a3d13c6d0b56e62fc" title="poissionNoiseCT.PNG"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/d/9d614093262fbe0ba188355a3d13c6d0b56e62fc_2_500x500.jpeg" alt="poissionNoiseCT.PNG" data-base62-sha1="msfk8FjaUeRqSm4SIwGxlJGDP6Q" width="500" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/d/9d614093262fbe0ba188355a3d13c6d0b56e62fc_2_500x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/9/d/9d614093262fbe0ba188355a3d13c6d0b56e62fc_2_750x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/9/d/9d614093262fbe0ba188355a3d13c6d0b56e62fc_2_1000x1000.jpeg 2x" data-dominant-color="1E1E1E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">poissionNoiseCT.PNG</span><span class="informations">1532×1529 161 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
</blockquote>

MayThet: <p>I have edited the code as follows. The following image is noise sigma 50. (Poisson noise lambda parameter = 50).</p>
<p>Could you please confirm if it’s correct or suggest any improvements?</p>
<blockquote>
<p>if args[‘add_noise’]:<br>
imnoisy = add_poisson_noise(imorig, args[‘noise_sigma’])</p>
<p>def add_poisson_noise(image_tensor, noise_factor):<br>
noisy_tensor = torch.poisson(image_tensor *<br>
noise_factor) / noise_factor<br>
return noisy_tensor</p>
</blockquote>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/a/aadb973214b7e8b0092e3ce97a7f7dc9bd9ff675.png" alt="CT50" data-base62-sha1="ontDIMWucmyNjDg6OYQ8fQq9ruJ" width="640" height="480"></p>
Linker errors when trying to use C++ API from my own project

Aviais: <p>Hi,<br>
I’m trying to use the C++ API from my own CPP Android project.<br>
I used your pytorch-android-cpp-demo and successfully built Android libtorch.a and libc10.a<br>
However, While I’m trying to add these static libraries to my project I’m getting the following errors:</p>
<p>/mnt/c/repos/pytorch-android-cpp-demo/pytorch/aten/src/ATen/core/ivalue.cpp:390: error: undefined reference to ‘c10::operator&lt;&lt;(std::__ndk1::basic_ostream&lt;char, std::__ndk1::char_traits &gt;&amp;, c10::Device const&amp;)’<br>
/mnt/c/repos/pytorch-android-cpp-demo/pytorch/aten/src/ATen/core/ivalue.cpp:449: error: undefined reference to ‘c10::operator&lt;&lt;(std::__ndk1::basic_ostream&lt;char, std::__ndk1::char_traits &gt;&amp;, c10::Device const&amp;)’<br>
/mnt/c/repos/pytorch-android-cpp-demo/pytorch/build_android/aten/src/ATen/Functions.cpp:145: error: undefined reference to ‘c10::toString(c10::DispatchKey)’<br>
/mnt/c/repos/pytorch-android-cpp-demo/pytorch/build_android/aten/src/ATen/Functions.cpp:165: error: undefined reference to ‘c10::toString(c10::DispatchKey)’<br>
/mnt/c/repos/pytorch-android-cpp-demo/pytorch/c10/core/TensorOptions.h:252: error: undefined reference to ‘c10::get_default_dtype()’<br>
/mnt/c/repos/pytorch-android-cpp-demo/pytorch/c10/core/TensorOptions.h:252: error: undefined reference to ‘c10::get_default_dtype()’<br>
/mnt/c/repos/pytorch-android-cpp-demo/pytorch/build_android/aten/src/ATen/Functions.cpp:289: error: undefined reference to ‘c10::toString(c10::DispatchKey)’<br>
/mnt/c/repos/pytorch-android-cpp-demo/pytorch/build_android/aten/src/ATen/Functions.cpp:827: error: undefined reference to ‘c10::toString(c10::DispatchKey)’</p>
<p>Can you help me figure out what am I missing?</p>

Bilahari_Akkiraju: <p>did you figure it out ? seeing similar issues.</p>
Where can I find the code of "ops.quantized.Conv1d"?

sw-kim: <p>The torch quantized conv1d code can be seen in the following link: <a href="https://pytorch.org/docs/stable/_modules/torch/ao/nn/quantized/modules/conv.html#Conv1d" class="inline-onebox" rel="noopener nofollow ugc">torch.ao.nn.quantized.modules.conv — PyTorch 2.1 documentation</a></p>
<p>However, it is difficult to find the code of “ops.quantized.conv1d”<br>
: return ops.quantized.conv1d(input, self._packed_params, self.scale, self.zero_point)</p>
<p>When I see the code, the ops.quantized.conv1d is implemented in the form of “C code”.<br>
Where is the location of code of “ops.quantized.conv1d”?</p>
<p>thanks.</p>
How to obtain the last convolucional layer of a visiontorch model?

zaratruta: <p>I’m training an efficientnet b0 architecture on cifar 10. I want to apply grad cam in the last convolutional layer of this model. However i’m not sure about how to identify this layer and How to obtain It. Can you help me?</p>

ptrblck: <p>You could print the model and check what the last layer is or you could also check the model’s source code so see which layer is used last.</p>
Use tensorflow and pytorch in same code caused error

xiaoyulun: <p>I have a project that need to use tensorflow model and pytorch model at the same time. When tf model do inference, it caused</p>
<pre><code class="lang-auto">2019-01-13 13:48:27.819434: E tensorflow/stream_executor/cuda/cuda_dnn.cc:378] Loaded runtime CuDNN library: 7102 (compatibility version 7100) but source was compiled with 6021 (co mpatibility version 6000).  If using a binary install, upgrade your CuDNN library to match.  If building from sources, make sure the library loaded at runtime matches a compatibleversion specified during compile configuration.
2019-01-13 13:48:27.819986: F tensorflow/core/kernels/conv_ops.cc:667] Check failed: stream-&gt;parent()-&gt;GetConvolveAlgorithms(conv_parameters.ShouldIncludeWinogradNonfusedAlgo&lt;T&gt;() , &amp;algorithms)
</code></pre>
<p>But the cudnn version on my local version is:</p>
<pre><code class="lang-auto">#define CUDNN_MAJOR      5
#define CUDNN_MINOR      1
#define CUDNN_PATCHLEVEL 10
···
#define CUDNN_VERSION    (CUDNN_MAJOR * 1000 + CUDNN_MINOR * 100 + CUDNN_PATCHLEVEL)

#include "driver_types.h"
</code></pre>
<p>my pytorch version is 0.4.1. tensorflow version is 1.4.0<br>
It may caused by pytorch autoload cudnn in its version.<br>
How to avoid this problem ?</p>

albanD: <p>Hi,</p>
<p>Yes pytorch ships with the latest cudnn binary for best performances.<br>
You will need to get both installs using the same version. Either by finding a version of Tensorflow that works with the latest cudnn or by compiling one of the two framework from source using the cudnn version expected by the other one such that they match.</p>

xiaoyulun: <p>Thanks for your reply. I just found that if import tensorflow first and import torch later. It seems to be worked.<img src="https://discuss.pytorch.org/images/emoji/apple/joy.png?v=6" title=":joy:" class="emoji" alt=":joy:"></p>

BoyuanChen: <p>OMG yes indeed! <img src="https://discuss.pytorch.org/images/emoji/apple/laughing.png?v=12" title=":laughing:" class="emoji" alt=":laughing:" loading="lazy" width="20" height="20"> <img src="https://discuss.pytorch.org/images/emoji/apple/laughing.png?v=12" title=":laughing:" class="emoji" alt=":laughing:" loading="lazy" width="20" height="20"> <img src="https://discuss.pytorch.org/images/emoji/apple/+1.png?v=12" title=":+1:" class="emoji" alt=":+1:" loading="lazy" width="20" height="20"></p>
Why is memory bandwidth peaked at 250 GB / sec where as Nvidia A100 has peak of 1.9 TB/sec

Amey_Naik: <p>Here for the example mentioned in the blog <a href="https://horace.io/brrr_intro.html" class="inline-onebox" rel="noopener nofollow ugc">Making Deep Learning go Brrrr From First Principles</a>. Example <a href="https://colab.research.google.com/drive/1hEtorT5y9mcXHR0gpensD7oZfuyyxtu7?usp=sharing" rel="noopener nofollow ugc">here</a>.</p>
<p>Why is memory bw capped at 250 GB/sec, where as Nvidia A100 has peak of 1.9 TB/sec?</p>
<p>Why is it memory bound at 250 GB/sec?</p>
Does Pytorch compute per sample gradients

zainsarwar865: <p>I’m confused about how Pytorch works in a batch setting. For the longest time, I had assumed that you compute individual losses for each sample in a batch, average them and compute the gradients with respect to that loss and use that to update your parameters.</p>
<p>However, I’ve recently some things which make me unsure of my assumption. What I’ve read is that you compute the loss for each individual sample, and for each sample, you compute different gradients. So if your batch size is 128, you essentially compute a set of 128 gradients for each parameter and before updating the parameter, you average/sum them.</p>
<p>Which one of these is closer to what actually happens in Pytorch?</p>
<p>Thank you!</p>
Why is memory bandwidth peaked at 250 GB / sec where as Nvidia A100 has peak of 1.9 TB/sec

constant_learner: <p>Here for the example mentioned in the blog <a href="https://horace.io/brrr_intro.html" class="inline-onebox" rel="noopener nofollow ugc">Making Deep Learning go Brrrr From First Principles</a>. Example <a href="https://colab.research.google.com/drive/1hEtorT5y9mcXHR0gpensD7oZfuyyxtu7?usp=sharing" rel="noopener nofollow ugc">here</a>.</p>
<p>Why is memory bw capped at 250 GB/sec, where as Nvidia A100 has peak of 1.9 TB/sec?</p>
<p>Why is it memory bound at 250 GB/sec?</p>
Get the trace for a batch of matrices

AlphaBetaGamma96: <p>Hi All,</p>
<p>I was wondering if it’s at all possible to take the trace of matrix for a batch of matrices? For example, let’s say I have some Tensor of shape <code>[B, N, N]</code> and wish to find the trace along <code>B</code> for each <code>[N, N]</code> matrix?</p>
<p>If I use the <code>torch.trace</code> command, I get an error saying that it expected a matrix. I understand I could just iterate over dim=0, and take the trace of each matrix then add them to a vector. But is there a way to do this with one command? Which takes <code>[B, N, N]</code> and returns <code>[B]</code>?</p>
<p>Thanks in advance! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

richard: <p>You can do this via a combination of taking a (batch) diagonal and then summing each diagonal.</p>
<p>So:</p>
<pre><code class="lang-auto">B, N = 2, 3
x = torch.randn(B, N, N)
x.diagonal(offset=0, dim1=-1, dim2=-2).sum(-1)
</code></pre>
<p>If you’re on a nightly build of PyTorch, this can be accomplished in one shot via <a href="https://pytorch.org/docs/master/generated/torch.vmap.html">torch.vmap</a>. <code>vmap</code> essentially “adds a batch dimension to your code”:</p>
<pre><code class="lang-auto">B, N = 2, 3
x = torch.randn(B, N, N)
batch_trace = torch.vmap(torch.trace)(x)
</code></pre>

AlphaBetaGamma96: <p>Thank you for the extensive answer! It works as intended now! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>Also, if I could ask a quick question about <code>torch.vmap</code>, would such a function work with non-batchable functions like <code>torch.autograd.functional.hessian</code>? I’m currently using that function, and it’s a major bottleneck in the runtime of my code!</p>
<p>Thank you!</p>

eigenvivek: <p>Extending an <a href="https://pytorch.org/docs/stable/generated/torch.einsum.html" rel="noopener nofollow ugc">example from the docs</a>, you can also use <code>torch.einsum</code></p>
<pre><code class="lang-auto">b, n = 2, 3
torch.einsum("...ii", torch.randn(b, n, n))
</code></pre>
MultiProcessingReadingService problem with tar files

corentin_r: <p>Hi,</p>
<p>I have a webdataset that is composed of tar files. I created a pipeline to use the dataset. Below is the code I use, decode is a function to do some preprocessing on the data (images and their caption in my case).</p>
<pre><code class="lang-auto">dp = FileOpener(list(braceexpand(data_path + "/{00000..05000}.tar")), mode="b") 
dp = dp.load_from_tar(length=datasetLength).webdataset()
dp = dp.shuffle().sharding_filter()
dp.apply_sharding(num_processes, process_index, sharding_group=SHARDING_PRIORITIES.DISTRIBUTED)
dp = dp.map(decode)
dp = dp.batch(batch_size=batch_size, drop_last=True)

trainLoader = DataLoader2(dp)
</code></pre>
<p>It works fine but then I tried to use the MultiProcessingReadingService to make data loading faster. By doing that, I run into a pickle error.</p>
<pre><code class="lang-auto">Process ForkProcess-1:
Traceback (most recent call last):
  File "/azureml-envs/azureml_99407ef20b35f1d5e9103d8f1bfac59a/lib/python3.8/site-packages/torch/utils/data/graph.py", line 67, in _list_connected_datapipes
    p.dump(scan_obj)
TypeError: cannot pickle 'ExFileObject' object
</code></pre>
<p>I have dill installed but it doesn’t change anything.<br>
Does anyone know what I am doing wrong ?</p>
<p>Thanks in advance,<br>
Corentin</p>

cinjon: <p>Hitting ~ the same problem. Did you solve this?</p>

corentin_r: <p>Hey,</p>
<p>I ended up decompressing my webdataset and creating a regular pytorch dataset.<br>
I used the regular DataLoader with multiple workers to speed it up.<br>
Since my dataset was on the same server as my compute cluster I didn’t have too much trouble with latency.</p>
<p>Here is what I used:</p>
<pre><code class="lang-auto">def __getitem__(self, index):
    line = self.captionKey.iloc[index]
    caption, key = line["caption"], line["key"]

    key = str(key).zfill(9)

    stream = BytesIO()
    numberAttempts = 0
    while True:
        try:
            self.containerClient.download_blob(key[:5] + key).readinto(stream)
            break
        except Exception:
            numberAttempts += 1
            time.sleep(1)
            if numberAttempts &gt; 10:
                raise Exception(f"Impossible to download the image {key[:5] + key}")

    with Image.open(stream) as image:
        image = self.preprocess(image)

    return image, caption
</code></pre>
How to implement an architecture which has 2 different branches and losses?

theairbend3r: <p>Hi all,</p>
<p>I’m trying to implement the paper =&gt; <a href="https://arxiv.org/pdf/1409.7495.pdf" rel="noopener nofollow ugc">Unsupervised Domain Adaptation by Backpropagation</a> by Gannin et. al.</p>
<p>The architecture is as follows -</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/8/087b346de1968d1463705588b6fa2773ad556af1.png" data-download-href="https://discuss.pytorch.org/uploads/default/087b346de1968d1463705588b6fa2773ad556af1" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/8/087b346de1968d1463705588b6fa2773ad556af1_2_690x318.png" alt="image" data-base62-sha1="1d1METR5aBlDYJJlg5W4jchcgjD" width="690" height="318" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/8/087b346de1968d1463705588b6fa2773ad556af1_2_690x318.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/8/087b346de1968d1463705588b6fa2773ad556af1_2_1035x477.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/8/087b346de1968d1463705588b6fa2773ad556af1.png 2x" data-dominant-color="E9EAE6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1180×545 78.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>How  do we implement something like this in PyTorch. More specifically:</p>
<ol>
<li>An architecture which has 2 different outputs that are trying to classify different things.</li>
<li>Run back-propagation taking into account the individual losses from both “branches”.</li>
<li>In this paper, the authors multiply the backprop values by <code>-1</code> in the gradient reverse layer. How do we do something like that in PyTorch?</li>
</ol>

JuanFMontesinos: <p>It’s a bit abstract but simple.</p>
<p>Gradients are accumulative in pytorch. This means unless you call optim.zero_grad() to set gradients to zero, gradients will keep accumulating.</p>
<p>You need to run both networks, backward pink using retain_graph=True. Then iterate through the gradients (for p in model.parameters(): p.grad *=-1)<br>
Then backward blue loss and then use optimizer step and zero grad.</p>
<p>Note: I do highlight the ordering as if you backprop blue first you would be multiplying them by -1 too.</p>

0tist: <p>Has there been more progress in the field of multi-branch models?</p>
DeepAR and torch.jit.trace

moberc: <p>I have created a simple deepAR model (from gluonts.torch.model.deepar.DeepAREstimator).  The model is based on 3 time series with real target values only.  I cannot get torch.jit.trace to work with this model.  The input is always wrong.<br>
I need to convert my model to Torch Script to load it to java djl.<br>
I an able to do that with simpler models.<br>
Has anybody got deepAR trace to work?  An example would be much appreciated.<br>
Thanks!</p>

marksaroufim: <p><code>torch.jit.trace</code> can be silently wrong <a href="https://pytorch.org/docs/stable/generated/torch.jit.trace.html" class="inline-onebox">torch.jit.trace — PyTorch 2.1 documentation</a></p>
<p>Would suggest you take a look at <code>torch.export.export()</code> from the latest release</p>

moberc: <aside class="quote no-group" data-username="marksaroufim" data-post="2" data-topic="189699">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/marksaroufim/48/36747_2.png" class="avatar"> marksaroufim:</div>
<blockquote>
<p>torch.export.export()</p>
</blockquote>
</aside>
<p>I have tried export with latest version of torch (2.1.0). I am getting<br>
RuntimeError: Windows not yet supported for torch.compile.<br>
Thanks!</p>

marksaroufim: <p>Yeah windows support is unlikely to happen anytime soon, I even know some windows developers that are using WSL nowadays</p>

moberc: <p>I was able to solve my problem by getting the inputs from the model itself:</p>
<p>predictor = torch.load(‘models/deepAR.pth’)<br>
feat_static_cat = predictor.prediction_net.example_input_array[‘feat_static_cat’]<br>
feat_static_real = predictor.prediction_net.example_input_array[‘feat_static_real’]<br>
past_time_feat = predictor.prediction_net.example_input_array[‘past_time_feat’]<br>
past_target = predictor.prediction_net.example_input_array[‘past_target’]<br>
past_observed_values = predictor.prediction_net.example_input_array[‘past_observed_values’]<br>
future_time_feat = predictor.prediction_net.example_input_array[‘future_time_feat’]</p>
<p>example_inputs = (feat_static_cat, feat_static_real, past_time_feat,<br>
past_target, past_observed_values, future_time_feat)<br>
model = predictor.prediction_net.model<br>
traced_script_module = torch.jit.trace(model, example_inputs)</p>
Image Segmentation models don't converge

imane.A: <p>Hello everyone,<br>
I’m running an experiment of image segmentation, I have used multiple algorithms such as Unet, Unet++, MAnet, DeepLabV3 but none of them converge, the error is fluctuating but staying at the same high level after many epochs (I tried different number of epoch, as well as different batch sizes and learning rates) but I can’t seem to find an answer as to why it’s not working. Here are snippets of my code if anybody could help me out I would be very greatfull.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/1/810ee23a275a74df3e4215a2a2369cb4d0b680b4.png" data-download-href="https://discuss.pytorch.org/uploads/default/810ee23a275a74df3e4215a2a2369cb4d0b680b4" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/1/810ee23a275a74df3e4215a2a2369cb4d0b680b4.png" alt="image" data-base62-sha1="ipHtSiAxtu8nV6uwi8FlWJDaubi" width="690" height="413" data-dominant-color="F2F2F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">769×461 14.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

imane.A: <p>Here’s my model<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/b/6b58795a6b2eaec2d0088057e55dc884db93ffbf.png" data-download-href="https://discuss.pytorch.org/uploads/default/6b58795a6b2eaec2d0088057e55dc884db93ffbf" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/b/6b58795a6b2eaec2d0088057e55dc884db93ffbf.png" alt="image" data-base62-sha1="fjCDVRGlMnt5ajaLy5M8pfDKcpx" width="690" height="260" data-dominant-color="F2F2F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">788×298 8.31 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

imane.A: <p>and here’s how I created the dataset before putting it through the data loaders, my inputs are images and their masks<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/9/5/95c84749d543e10a4e0449b8a97d4f4f42f8d007.png" alt="image" data-base62-sha1="ln2eJSg80G35OLRqprQKAd6QuPl" width="599" height="413"></p>

imane.A: <p>As decoder I used efficient net, resnet, VGG but none of them work as well !</p>
Labels for padding

Gears_Gears: <p>Hi!</p>
<p>I need a proper <strong>dummy</strong> label value for cross-entropy-loss calculation? Could I get some guidance please?</p>
<p>I have a convolutional network that takes input of shape (number of images in batch=B, channels=3, width, height). This input is a batch of images that comes from 5 classes.</p>
<p>I must simulate ‘imbalance’ so that sometimes, I sample B’ &lt; B number of images. This means that I need to ‘pad’ my input with B - B’ number of all-zero images so that the network still works.<br>
In the TensorFlow I’m trying to emulate, they give one-hot encoded labels for the B’ ‘real’ images and all-0s for the dummy all-zero images.</p>
<p>The problem is, PyTorch’s cross-entropy loss doesn’t take one-hot. My B’ images have labels will come from [0, 1, 2, 3, 4], so if I label the dummy images 0, it seems to confuse the model and I get very low accuracy (since both the dummy and some of actual input gets the same label).</p>
<p>Many Thanks!</p>

ptrblck: <p>I’m not sure if I understand your use case but are you passing a <code>zeros</code> tensor into your TF implementation? If so, try to do the same in PyTorch by passing a floating point tensor as newer PyTorch releases accept a one-hot encoded tensor as the target in <code>nn.CrossEntropyLoss</code>.</p>

Gears_Gears: <p>Dear ptrblck,</p>
<p>Thanks very much for your help! After reading the documentation more closely, I found that ignore-index might be a good way to go about it! I’m trying it out now!</p>
<p>Use case &amp; some more information: I’m still working on the <a href="https://github.com/haebeom-lee/l2b" rel="noopener nofollow ugc">TAML</a> algorithm but getting very bad (1.3 cross-entropy loss, 0.3 accuracy after 1000 iters using the same settings as the paper) results. Naively using 0 as the dummy label (when in fact it is a ‘real’ label) is what I’m fixing right now.</p>
<p>The algorithm tries to simulate the imbalanced situation where the number of samples in each classification task differs. So usually, for a 5-way classification task, we are getting B images from each of the 5 classes. The ‘imbalance’ here would be that sometimes we get less than B (so B is the upper bound here) number of images from some classes. TAML’s goal is to use an encoder to know based on the imbalance to reuse more or less information from previous training ‘experiences.’ The way the authors does it in their TF implementation is to one-hot labels like <a href="https://github.com/haebeom-lee/l2b/blob/master/data.py" rel="noopener nofollow ugc">[0 1 0 0 0] for images and [0 0 0 0 0] for the padded images</a> is needed to keep the total number of images in each task constant.</p>
Should the requires_grad=False of model weights when using torch.inference_mode

lisyuan: <p>Hi,</p>
<p>As far as I understanding, when using the <code>torch.inference_mode()</code> or <code>torch.no_grad()</code>,<br>
the model will not caculating the gradients and the <code>requires_grad</code> will be <code>False</code>.<br>
I write a simple code to reproduce my question</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

class Network(nn.Module):
    def __init__(
        self,
    ):
        super().__init__()
        self.conv = nn.Conv2d(2, 32, (3, 3))

    def forward(self, x):

        with torch.inference_mode():
            self.eval()
            x = self.conv(x)
            print(x.requires_grad)
            print(self.conv.weight.requires_grad)

        return x

x = torch.randn(4, 2, 257, 513)
backbone = Network()
backbone(x)
</code></pre>
<p>The corresponding outputs:</p>
<pre><code class="lang-auto">False
True
</code></pre>
<p>The output <code>requires_grad</code> of <code>x</code> is <code>False</code>, but the <code>requires_grad</code> of <code>self.conv.weight</code> is <code>True</code>.<br>
Is the behavior of the latter correct or there might be something that I haven’t noticed?<br>
Thank you for your reply in advance.</p>

ptrblck: <p>This is expected since the <code>inference_mode</code> context won’t change the <code>.requires_grad</code> attributes of parameters or inputs. The output won’t be attached to a computation graph and no intermediate tensors will be stored, as expected.</p>

lisyuan: <p>Ok, I got it. Thanks for your reply!</p>

akh: <p>Quick clarifying question from a newbie. Does it mean that <code>inference_mode</code> would ignore all <code>.requires_grad</code> attributes on parameters or inputs? Would setting <code>.requires_grad</code> of parameters and inputs to <code>False</code> improve performance within <code>inference_mode</code>?</p>

ptrblck: <p>No, it won’t improve the performance as the gradient calculation is already disabled in <code>inference_mode</code>.</p>
Is torch.masked_select differentiable?

saul_santos: <p>The question says it all. Thank you in advance.</p>

albanD: <p>Hi,</p>
<p>If you don’t get an error when you backprop, then it is <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>A quick check shows that it is:</p>
<pre><code class="lang-python">import torch

a = torch.rand(10, 10, requires_grad=True)
mask = torch.randn(10, 10).clamp(min=0).bool() # Generate random mask

out = torch.masked_select(a, mask)

out.sum().backward()
print(a.grad) # A Tensor of 0s and 1s
</code></pre>

Oliver1: <p>Hi, what about torch.sort(), which does not give an error, but the sort operation should be Non-differentiable.</p>

albanD: <p>It depends how you look at it <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"><br>
It is differentiable if you consider it is only shuffling the elements around: the for each output propagates to exactly the input it came from.<br>
There are also research papers discussing taking the “getting the indices for shuffling” part in the gradient computation (I don’t have the link on hand but you can find it only when looking for differentiable sorting).</p>

Oliver1: <p>Thank you so much! I will search for the related papers you mentioned.</p>
AttributeError: module 'torch.backends' has no attribute 'opt_einsum'

PeaBrane: <p>I updated my pytorch version from 2.0.1 to 2.1.0 today and noticed that <code>torch.backends</code> no longer has opt_einsum as an attribute. The error can be simply reproduced by running</p>
<pre><code class="lang-auto">torch.backends.opt_einsum
</code></pre>
<p>When I go into the source code of <code>torch.backends</code>, I found the following block of code</p>
<pre><code class="lang-auto">from torch.backends import (
    cpu as cpu,
    cuda as cuda,
    cudnn as cudnn,
    mkl as mkl,
    mkldnn as mkldnn,
    mps as mps,
    openmp as openmp,
    quantized as quantized,
)
</code></pre>
<p>which seems to suggest that opt_einsum has removed from the imports. However, the directory <code>opt_einsum</code> still exists in <code>torch/backends</code>.</p>
<p>I’m just curious if this change is intentional, and if so what is the reason?</p>

soulitzer: <aside class="quote no-group" data-username="PeaBrane" data-post="1" data-topic="190331">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/peabrane/48/40023_2.png" class="avatar"> PeaBrane:</div>
<blockquote>
<pre><code class="lang-auto">torch.backends.opt_einsum
</code></pre>
</blockquote>
</aside>
<p>As long as it exists in the torch/backends dir, shouldn’t it return you the still opt_einsum module?</p>

PeaBrane: <p>Yes I think you’re right, it appears that</p>
<pre><code class="lang-auto">import torch.backends import opt_einsum
</code></pre>
<p>still works, and all the functions of opt_einsum can still be accessed normally.</p>
<p>I am still a bit curious if this is a sign of deprecation or some incoming refactoring. Couldn’t find anything in the release notes.</p>

soulitzer: <blockquote>
<p>I am still a bit curious if this is a sign of deprecation or some incoming refactoring. Couldn’t find anything in the release notes.</p>
</blockquote>
<p>That block of imports was actually added later, so opt_einsum was never actually included there.</p>
Multi-sequence multi-step time series task

Wayne_127: <p>I am a college student who is new to deep learning and I have a challenging multi-sequence multi-step time series task. Currently, I have 90 days of time series dataframe, which is 90 rows, with 20 columns of features and two columns of target values, and I want to use 90 days of data to predict two columns of target values for the next four days. I would like to use the cnn network model or lstm network model in the pytorch framework, who can help me construct a baseline. I would appreciate it. If possible, please leave a message or contact my email.</p>

srishti-git1110: <p><a href="https://pytorch.org/tutorials/beginner/nlp/sequence_models_tutorial.html" rel="noopener nofollow ugc">This</a>, <a href="https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html" rel="noopener nofollow ugc">this</a> and <a href="https://pytorch.org/tutorials/intermediate/char_rnn_classification_tutorial.html" rel="noopener nofollow ugc">this</a> might help.</p>
ValueError: Using a target size (torch.Size([491])) that is different to the input size (torch.Size([1, 491])) is deprecated. Please ensure they have the same size

neelam19051: <p>Hi, I have a script that i am trying to run, but it gives the following error -</p>
<p>``<br>
batch_size_16,learning_rate_0.001,epoch_times_45<br>
selected_9606_protein_scores.csv<br>
selected_9606_protein_scores.csv<br>
Traceback (most recent call last):<br>
File “/home/bvsbic/Downloads/MLtest/Validation1.py”, line 1084, in <br>
Terms = [‘BP’, ‘MF’, ‘CC’]<br>
File “/home/bvsbic/Downloads/MLtest/Validation1.py”, line 1065, in validation<br>
test_set = benchmark[test_index].tolist()<br>
File “/home/bvsbic/Downloads/MLtest/Validation1.py”, line 903, in Main<br>
domain_train_out, domain_test_out, domain_t = Domain_train(0.001, 32, train_benchmark, test_benchmark, 45,<br>
File “/home/bvsbic/Downloads/MLtest/Validation1.py”, line 639, in Domain_train<br>
loss = loss_function(out, GO_annotiations)<br>
File “/home/bvsbic/Downloads/MLtest/okenv/lib/python3.9/site-packages/torch/nn/modules/module.py”, line 727, in _call_impl<br>
result = self.forward(*input, **kwargs)<br>
File “/home/bvsbic/Downloads/MLtest/okenv/lib/python3.9/site-packages/torch/nn/modules/loss.py”, line 530, in forward<br>
return F.binary_cross_entropy(input, target, weight=self.weight, reduction=self.reduction)<br>
File “/home/bvsbic/Downloads/MLtest/okenv/lib/python3.9/site-packages/torch/nn/functional.py”, line 2518, in binary_cross_entropy<br>
raise ValueError("Using a target size ({}) that is different to the input size ({}) is deprecated. "<br>
<em>ValueError: Using a target size (torch.Size([491])) that is different to the input size (torch.Size([1, 491])) is deprecated. Please ensure they have the same size.</em></p>
<pre><code class="lang-auto">Actually, I am new here, and I don't know how to post the whole code; plz help me.</code></pre>

srishti-git1110: <aside class="quote no-group" data-username="neelam19051" data-post="1" data-topic="190345">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/neelam19051/48/64794_2.png" class="avatar"> neelam19051:</div>
<blockquote>
<p>Please ensure they have the same size.</p>
</blockquote>
</aside>
<p>You need to ensure that the two tensors you calculate the (cross entropy) loss between have the same sizes.<br>
See -</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
import torch.nn as nn

target = torch.rand(491)
print(target.size())

out = torch.rand(1, 491)
print(out.size())

loss_fn = nn.BCELoss()
loss = loss_fn(out, target) # reproduces your error

loss = loss_fn(out, target.unsqueeze(0)) # the fix
print(loss) # runs error free
</code></pre>
DDP with gloo is stuck at initialization

ravichandrakolluri: <p>I am trying to use gloo for distributed training. but the script is stuck at initializing the process group.</p>
<p>dist.init_process_group(backend=“gloo”, init_method=“tcp://10.139.64.218:7077”,rank=1, world_size=3)</p>
<p>I am running the scripts on a 3 node cpu cluster on Azure Databricks</p>
<p>please help.</p>

H-Huang: <p>Can you include more information? Can you include your script and also the logs? You can run with <code>TORCH_CPP_LOG_LEVEL=INFO</code> to also output more detailed logs</p>
Run pytorch code on Kaggle TPUs

Schroter: <p>Hi All,</p>
<p>I have some code. Which I borrowed from <a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html" rel="noopener nofollow ugc">here</a>. Just wondering how to edit it so that I would be able to utilise the TPUs in Kaggle notebooks.</p>
<p>Would anyone be able to help me in this regards.</p>
<p>Thanks &amp; Best Regards</p>
<p>Michael</p>