akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
ValueError: No checkpoint is available for model type shufflenetv2_x2.0

username: ANS_CENTER
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/d/cd969c0e8e2a462caec22fe97e0f9bef2ca008bf.png" data-download-href="https://discuss.pytorch.org/uploads/default/cd969c0e8e2a462caec22fe97e0f9bef2ca008bf" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/d/cd969c0e8e2a462caec22fe97e0f9bef2ca008bf.png" alt="image" data-base62-sha1="tkIyMxJNlmotmbzDmqgN4mMe0a3" width="690" height="237" data-dominant-color="F0F5F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">978×337 26.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<h1><a name="i-have-error-about-checkpoint-valueerror-no-checkpoint-is-available-for-model-type-shufflenetv2_x20-1" class="anchor" href="#i-have-error-about-checkpoint-valueerror-no-checkpoint-is-available-for-model-type-shufflenetv2_x20-1"></a>I have error about checkpoint: ValueError: No checkpoint is available for model type shufflenetv2_x2.0</h1>

username: ptrblck
cooked: <p>The pretrained weights were added in <a href="https://github.com/pytorch/vision/pull/5906">this PR</a> last year so you might need to update <code>torchvision</code>.<br>
This code works fine in the current nightly release:</p>
<pre><code class="lang-python">model = models.shufflenet_v2_x2_0(pretrained=True)
</code></pre>
Sampling image patches based on object densities

username: AmrMustafa
cooked: <p>Hello everyone, I am working on an image segmentation project where I handle high-resolution images with relatively small objects. So, my approach was to first tile the image such that I deal with smaller patches without loosing the details. However, instead of tiling the images as an offline step, I want to implement a sampling transformation that takes random crops from my image but I do not want empty crops, instead I want the sampling to be based on the number of objects within the crop. Is this doable? Can you guys point me in the right direction? I hope I made my point clear, and thanks in advance!</p>
Teacher-Student learning Framework: using EMA to distil knowledge from one model to the other

username: Iyad_alshami
cooked: <p>Hello there!<br>
I have two models that are identical and I’m trying to update one of them using the other in an EMA manner.<br>
like this:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/5/f52c18a4b2fc71b7b8636b178abdd5fd323719c6.png" data-download-href="https://discuss.pytorch.org/uploads/default/f52c18a4b2fc71b7b8636b178abdd5fd323719c6" title="EMA"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/5/f52c18a4b2fc71b7b8636b178abdd5fd323719c6_2_345x20.png" alt="EMA" data-base62-sha1="yYTsh7MciTiCPZoNzxYrUHgPrsa" width="345" height="20" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/5/f52c18a4b2fc71b7b8636b178abdd5fd323719c6_2_345x20.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/5/f52c18a4b2fc71b7b8636b178abdd5fd323719c6_2_517x30.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/5/f52c18a4b2fc71b7b8636b178abdd5fd323719c6_2_690x40.png 2x" data-dominant-color="000000"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">EMA</span><span class="informations">890×52 5.95 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I’ve attempted to do this using the following code:</p>
<pre><code class="lang-auto">    with torch.no_grad():
        for teacher_param, student_param in zip(teacher.parameters(), student.parameters()):
            teacher_param.data *= tau
            teacher_param.data += (1 - tau) * student_param.data
</code></pre>
<p>Is this the correct approach to achieve my goal?<br>
Thank you so much for your help!</p>

username: Iyad_alshami
cooked: <p>I found a paper called <strong>Unbiased Teacher for Semi-Supervised Object Detection</strong> that implemented this technique. They have publicly available code, and here is a quick url to see how they implemented it:</p>
<p><a href="https://github.com/facebookresearch/unbiased-teacher/blob/226f8c4b7b9c714e52f9bf640da7c75343803c52/ubteacher/engine/trainer.py#L632" rel="noopener nofollow ugc">https://github.com/facebookresearch/unbiased-teacher/blob/226f8c4b7b9c714e52f9bf640da7c75343803c52/ubteacher/engine/trainer.py#L632</a></p>
How to convert audio (e.g. wav) to tensor and back?

username: YK11
cooked: <p>Hello!<br>
It is easy to convert audio (wav) to tensor using .load() command. But I am interested in reverse operation. Can someone help me?</p>
<p>Thanks!</p>

username: alexis-jacq
cooked: <p>Isn’t it <code>torchaudio.save('amazing_sound.wav', sound, sample_rate)</code> ?</p>
<p>By the way, you can find the doc here: <a href="http://pytorch.org/audio/">http://pytorch.org/audio/</a></p>

username: Tilak_raj_Choubey
cooked: <p>I assume there isn’t any way to convert the tensor back to byte format without saving. That’s a feature worth adding I guess.</p>
How to find a list of available backends of torch.compile?

username: sc21
cooked: <p>I’m using Pytorch nightly (version 2.1.0.dev20230304+cu117)<br>
I was trying to use <code>torch._dynamo.list_backends()</code> to list the available backends (from <a href="https://pytorch.org/blog/Accelerating-Hugging-Face-and-TIMM-models/" rel="noopener nofollow ugc">this article</a>), but it does not seem to work.</p>

username: sc21
cooked: <p>Found the answer. You need to install torchdynamo and call<code> torchdynamo.list_backends()</code> from <a href="https://pytorch.org/get-started/pytorch-2.0/" rel="noopener nofollow ugc">this article</a>.</p>
Using a single optimizer object for multiple models

username: kvtzn
cooked: <p>Hi,</p>
<p>I was wondering if using a single optimizer object to train multiple models is plausible. Particularly, I would like to use a <code>torch.optim.SGD</code> object on models deriving from <code>torch.nn.Module</code>.</p>
<p>I would like to make sure that the models are optimized independently. Is passing the parameters of each model as a separate parameter group correct? Will the gradient be calculated for each model separately, or for common minimum points of all models’ parameters? I checked the code for SGD but I’m not entirely sure.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="kvtzn" data-post="1" data-topic="174039">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/f14d63/48.png" class="avatar"> kvtzn:</div>
<blockquote>
<p>Is passing the parameters of each model as a separate parameter group correct?</p>
</blockquote>
</aside>
<p>Yes, passing the parameters from each model into a new <code>param_group</code> allows you to specify the optimizer parameters separately for each group, such as the learning rate. If you don’t want to use different arguments in your optimizer for each model you can also pass all parameters into a single <code>param_group</code>.</p>
<aside class="quote no-group" data-username="kvtzn" data-post="1" data-topic="174039">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/f14d63/48.png" class="avatar"> kvtzn:</div>
<blockquote>
<p>Will the gradient be calculated for each model separately, or for common minimum points of all models’ parameters?</p>
</blockquote>
</aside>
<p>The optimizer will not calculate the gradients, but will use the already calculated <code>.grad</code> attribute to update the parameters. The gradients are calculated during the <code>backward</code> call and which gradients are computed depends on the forward pass (in particular which parameters were used to calculate the <code>loss</code> or model output).<br>
Note that some optimizers use internal running stats which will update parameters even with a zero gradient. If this behavior is not desired, you could delete the gradients via <code>optimizer.zero_grad(set_to_none=True)</code>.</p>

username: kvtzn
cooked: <p>I see, thank you for the detailed explanation!</p>
KL distance between Gaussian Mixture Model and Normal

username: ZSheikhb
cooked: <p>I am wondering whether there is any similar class in pytorch as <a href="https://www.tensorflow.org/probability/api_docs/python/tfp/monte_carlo/expectation" rel="noopener nofollow ugc">expectation(tensorflow)</a> which can be used for computing KL divergence between the probabilities which do not have builtin KL divergences modules inside  the <code>torch.distribution</code> class and possibily they can be computed with Monte Carlo sampling ( <code>KL(p||q) = \int p(x) log(p(x) / q(x)) dx = E_p[ log(p(x) / q(x))</code>). Some of the KL divergences also don’t have closed form formulations such as the KL divergence between two Gaussian mixtures (<code>MixtureSameFamily</code>) or a Gaussian mixture and a normal distribution (<code>Normal</code>)?<br>
Thanks</p>
How to create/train a binary classification model for checking candidate phrases

username: vdw
cooked: <p>Let’s say I have sentences like “He called me a silly sausage when I made a stupid joke”, and I want to identify/extract all swear words and the like (here: “silly sausage”).</p>
<p>I could, of course, set this up as a sequence labeling task. However, can this also be done as a binary classification task where the model gets the sentence as well as a candidate phrase, and the model predicts whether this candidate phrase is a swear word or not?</p>
<p>I know how to do this with traditional models (e.g., Naive Bayes, SVM, Logistic Regression):</p>
<ul>
<li>Extract word/phrase features for the candidate given the sentence</li>
<li>Train a model on the predicted features, or use extracted features for the prediction.</li>
</ul>
<p>But how can I do this in an end-to-end fashion using a CNN or RNN model, for example?</p>

username: brighteningeyes
cooked: <p>hello,<br>
you can extract features from sentence with cnn (by using embedding and conv1d) and then use attention to extract weather the given phrase is swear or not. (you need to use the phrase as your query and the given sentence as the key).</p>

username: vdw
cooked: <p>Ah, right…I think I get the idea. Thanks!</p>
<p>Say I would encode the sentence and the candidate using an CNN or RNN, would I need to use separate models, or could I use the same, say, RNN to encode both the sentence and candidate?</p>

username: brighteningeyes
cooked: <p>you can use something like bert to extract features for both of them.<br>
you also can use different networks for feature extraction.<br>
in my idea, try a pre-trained network like bert for feature extraction, and train the attention based on that.</p>
How to set bias a tensor of (C_out, C_out, kernel_size, kernel_size)?

username: xiaowk5516
cooked: <p>I have a trained model. I want to chenge bias of some convolutions in the model for better inference performance. for example, the shape of kernel is (C_out, C_in, 3, 3) and the shape of bias is (C_out, ). I want to set new value to bias. the shape of new bias is (C_out, C_in, 3, 3) and it can set different bias for every point in kernel.  how can i achieve this?</p>
<p>Docs in <a href="https://pytorch.org/docs/stable/generated/torch.nn.Conv2d.html#torch.nn.Conv2d" class="inline-onebox" rel="noopener nofollow ugc">Conv2d — PyTorch 2.1 documentation</a> that told shape of kernel and bias.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/7/270c7d8c85b0a1595931e654e43f34d4ca6ee4eb.png" data-download-href="https://discuss.pytorch.org/uploads/default/270c7d8c85b0a1595931e654e43f34d4ca6ee4eb" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/7/270c7d8c85b0a1595931e654e43f34d4ca6ee4eb_2_690x192.png" alt="image" data-base62-sha1="5zrnGfUrxUr2ltwKNEl8rOb7kUj" width="690" height="192" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/7/270c7d8c85b0a1595931e654e43f34d4ca6ee4eb_2_690x192.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/7/270c7d8c85b0a1595931e654e43f34d4ca6ee4eb_2_1035x288.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/2/7/270c7d8c85b0a1595931e654e43f34d4ca6ee4eb_2_1380x384.png 2x" data-dominant-color="F4F3F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1493×417 37.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: eqy
cooked: <p>Do you mean you want a bias of shape e.g., (k, k, C_out), or of (output_y, output_x, C_out)? For the former, note that it should basically be equivalent to the existing case as the values would simply be added up. For the latter, you would need to need to disable the bias in the convolution and create a new layer for the bias as it would have different semantics than native Conv2d.</p>

username: xiaowk5516
cooked: <p>thank you. I misunderstood my needs. I calculate a new bias of shape (C_out, C_in, k, k)，and just add the bias in (C_in, k, k).</p>
Why the accuracy rate to 100% using its examples' dataset in epoch1 by VIT?

username: zhexianrenx
cooked: <p>I run this example’ code in my computer ,but the accuracy rate is 100% in epoch1 .I can’t understand why my result is quite different.</p>
<pre><code class="lang-auto">efficient_transformer = Linformer(
    dim=128,
    seq_len=49+1,  # 7x7 patches + 1 cls-token
    depth=12,
    heads=8,
    k=64
)
model = ViT(
    dim=128,
    image_size=224,
    patch_size=32,
    num_classes=2,
    transformer=efficient_transformer,
    channels=3,
).to(device)

# loss function
criterion = nn.CrossEntropyLoss()
# optimizer
optimizer = optim.Adam(model.parameters(), lr=lr)
# scheduler
scheduler = StepLR(optimizer, step_size=1, gamma=gamma)
for epoch in range(epochs):
    epoch_loss = 0
    epoch_accuracy = 0

    for data, label in tqdm(train_loader):
        data = data.to(device)
        label = label.to(device)

        output = model(data)
        loss = criterion(output, label)

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        acc = (output.argmax(dim=1) == label).float().mean()
        epoch_accuracy += acc / len(train_loader)
        epoch_loss += loss / len(train_loader)

    with torch.no_grad():
        epoch_val_accuracy = 0
        epoch_val_loss = 0
        for data, label in valid_loader:
            data = data.to(device)
            label = label.to(device)

            val_output = model(data)
            val_loss = criterion(val_output, label)

            acc = (val_output.argmax(dim=1) == label).float().mean()
            epoch_val_accuracy += acc / len(valid_loader)
            epoch_val_loss += val_loss / len(valid_loader)


</code></pre>
<p>Most of my coding above is following the snippet below in code block<br>
(<a href="https://github.com/lucidrains/vit-pytorch/blob/main/examples/cats_and_dogs.ipynb" class="inline-onebox" rel="noopener nofollow ugc">vit-pytorch/cats_and_dogs.ipynb at main · lucidrains/vit-pytorch · GitHub</a>)<br>
Thanks for advance.</p>
How to allocate different memories to multiple gpus while training?

username: medvisioner
cooked: <p>Suppose I have two GPUs, GPU-0 and GPU-1 (they are the same type). I hope to train a simple classification network (e.g. ResNet) on them. For some special reasons, I hope GPU-0 can take more memories.</p>
<p>For example, consider the batch size set to 64, I hope about 40 batches of data are allocated on GPU-0 and the rest 24 batches on GPU-1.</p>
<p>I am guessing this can not be done via nn.DataParallel or nn.DistributedDataParallel, right? To do this, I think I need to copy the model and data <em>manually</em>  to GPU-0 and GPU-1, then merge the computed loss together.</p>
<p>I am pretty unfamiliar with distributional training in PyTorch and fail to find a proper tutorial. A related question is raised <a href="https://discuss.pytorch.org/t/multi-gpu-training-different-memory-usage/27981">here</a>, however the objective is quite different.</p>
<p>Could anyone illustrate this problem with an example? Thanks ahead.</p>

username: ptrblck
cooked: <p>I would guess <code>DistributedDataParallel</code> might work for such a use case since each process is defining it’s own data loading pipeline.<br>
You might not be able to use the <code>DistributedSampler</code>, but could try to load the desired dataset subsets manually with different batch sizes. Let me know, if this would work for you.</p>
Script or Trace save() to target prior release versions

username: Rahul-Singhal-Crafty
cooked: <p>Hi, as the title says, is it possible to target a prior release.</p>
<p>In our current scenario, we have a tool that supports Torch 1.6 but we have 3rd party models that were created using Torch 1.9 and Torch 1.11</p>
<p>It’s not clear that the models utilize the later features.  One option we’ve considered is setting up multiple environments, but checking if this is a supported feature.</p>
<p>2nd question, is there a documentation that discusses torchscript/jit feature additions and breaking changes from Torch 1.6 to Torch 1.13 anywhere online. My initial searching has not yielded much.</p>

username: ptrblck
cooked: <p>No, PyTorch is not forward compatible. Storing a model in the current release might not allow older releases to properly execute it.<br>
Backwards compatibility should work, i.e. storing a model in older releases would allow you to still load and execute it.</p>
<p>I’m not aware of a doc explicitly discussing this, but note that deprecated features are usually raising warning for at least 2 minor releases before breaking changes are introduced.<br>
If you did not see any warnings, you might want to take a look at the release notes and search for some changes which might be related to the issues you are seeing.</p>

username: Rahul-Singhal-Crafty
cooked: <p>I don’t have the full error in front of me, but one was due to not finding .sub_()</p>
<p>Got it, I’ll check the release notes.</p>

username: ptrblck
cooked: <p>That’s a bit weird, as <code>Tensor.sub_</code> was already available for a long time and also supported on <code>1.9.0</code> as seen in the <a href="https://pytorch.org/docs/1.9.0/generated/torch.Tensor.sub_.html?highlight=sub_#torch.Tensor.sub_">docs</a>. Let me know once you have the warning and get stuck upgrading the code.</p>
Issue of using JIT with Python dataclass

username: melonise
cooked: <p>Hi,</p>
<p>I am trying to use Python dataclass with PyTorch JIT. And it works for a simple data class. For example,</p>
<pre><code class="lang-auto">import torch
from dataclasses import dataclass

@torch.jit.script
@dataclass
class SimpleView:
    id: int
    name: str

def test_simple_view():
    view = SimpleView(
        id=1,
        name='foo',
    )
    assert view.name == 'foo'
    print(view)
</code></pre>
<p>But when I tried to use more complex dataclasses, for example, one dataclass is embedded inside another dataclass, it does not work because JIT fails to compile such dataclass. Here is the code sample,</p>
<pre><code class="lang-auto">import torch
from dataclasses import dataclass

@torch.jit.script
@dataclass
class ValueStruct:
    label: str
    score: float

@torch.jit.script
@dataclass
class EmbeddedView:
    id: int
    name: str
    value: ValueStruct

def test_embedded_view():
    view = EmbeddedView(
        id=2,
        name='bar',
        value=ValueStruct(
            label='baz',
            score=9.9,
        )
    )
    assert view.value.label == 'baz'
    print(view)
</code></pre>
<p>And this is the error I got when running the above test case,</p>
<pre><code class="lang-auto">============================= test session starts ==============================
collecting ... 
main.py:None (main.py)
main.py:48: in &lt;module&gt;
    class EmbeddedView:
../../python/py39_1/lib/python3.9/site-packages/torch/jit/_script.py:1323: in script
    _compile_and_register_class(obj, _rcb, qualified_name)
../../python/py39_1/lib/python3.9/site-packages/torch/jit/_recursive.py:47: in _compile_and_register_class
    script_class = torch._C._jit_script_class_compile(qualified_name, ast, defaults, rcb)
E   RuntimeError: 
E   Unknown type name 'main.ValueStruct':
E     File "__torch_jit_dataclass/EmbeddedView/__init__", line 0
E   def __init__(self, id: int, name: str, value: main.ValueStruct) -&gt; None:
E                                                 ~~~~~~~~~~~~~~~~ &lt;--- HERE
E     self.id = id
E     self.name = name
ERROR: not found: /Users/dan.xu/dev/projects/ts_dataclass/main.py::test_embedded_view
collected 0 items / 1 error
(no name '/Users/dan.xu/dev/projects/ts_dataclass/main.py::test_embedded_view' in any of [&lt;Module main.py&gt;])


=============================== 1 error in 1.90s ===============================

Process finished with exit code 4
</code></pre>
<p>I found the dataclasses support in TorchScript was added last year in these PRs(<a href="https://github.com/pytorch/pytorch/issues/72901" rel="noopener nofollow ugc">#72901</a>, <a href="https://github.com/pytorch/pytorch/pull/76771" rel="noopener nofollow ugc">#76771</a>), and the version of PyTorch I am using is 1.13.1, which should contain the dataclasses support. I am wondering if there is a way to support nested dataclasses, and if this is a problem for the dataclass TorchScript support. Please help provide your advice, thanks!</p>

username: melonise
cooked: <p>Hi, does anyone have some idea on this dataclass issue?</p>
Updating weight for a list of model

username: SjZ
cooked: <p>for model in clinet_models:<br>
optimizer = torch.optim.SGD(model.parameters(),lr = 0.1)<br>
for batch_idx,(inputs,target) in enumerate(test_loader):<br>
outputs = model(inputs)<br>
loss_fn = nn.CrossEntropyLoss()<br>
loss_ = loss_fn(outputs,target.squeeze())<br>
optimizer.zero_grad()<br>
loss_.backward()<br>
optimizer.step()</p>
<p>Hello, I am trying to update model’s parameters for a list of model, but it seems it keeps the same after this, how can I properly update the parameters?</p>

username: soulitzer
cooked: <p>The loop looks correct to me, did you make sure your parameters require_grad=True? Otherwise a small reproducible code snippet would probably be helpful others to look more into this.</p>

username: SjZ
cooked: <p>Thanks for the help,I’m just started using PyTorch not sure how to set it, I want to mimic federated learning in one machine. Here is my complete code for this task</p>
<p>def train_local_model(model,dataset,loss_fn,optimizer):<br>
model.train()<br>
dataloader = DataLoader(dataset,batch_size = batch_size,shuffle = True)<br>
for batch_idx,(inputs,targets) in enumerate(dataloader):<br>
outputs = model(inputs)<br>
loss = loss_fn(outputs,targets.squeeze())<br>
optimizer.zero_grad()<br>
loss.backward()<br>
optimizer.step()</p>
<p>clinet_models = [copy.deepcopy(global_model) for i in range(num_clients)]</p>
<p>for epoch in range(num_epochs):<br>
for client_model,client_datset in zip(clinet_models,client_datasets):<br>
train_local_model(client_model,client_datset,loss_fn,optimizer) It turns out this does not update the parameters</p>

username: SjZ
cooked: <p>Well I checked, the parameters of the models have require_grad = True</p>
removing weights from resnet110

username: nadji_001
cooked: <p>I am pruning resnet110, but I could not do any weight pruning. I am not looking to set the value of the weights to zero since it only affects the sparsity of the model, but I want to delete/remove the weight.  I used torch_pruning but it only allows to prune out channel.</p>

username: ptrblck
cooked: <p>I’m unsure what your use case is, since deleting a <code>weight</code> would cause failures unless you also manipulate the actual forward pass and exclude the deleted <code>weight</code> from it.</p>
<p>Here is a small example:</p>
<pre><code class="lang-python">model = models.resnet18()
# delete random weight
model.fc.weight = None

x = torch.randn(1, 3, 224, 224)
out = model(x)
# TypeError: linear(): argument 'weight' (position 2) must be Tensor, not NoneType
</code></pre>
<p>Deleting specific values inside a tensor is also not possible unless you remove an entire slice from a dimension:</p>
<pre><code class="lang-python">print(model.conv1.weight.shape)
# torch.Size([64, 3, 7, 7])

# you cannot delete one value only
model.conv1.weight[3, 2, 1, 4] = None
# TypeError: can't assign a NoneType to a torch.FloatTensor

# but e.g. a complete channel
model.conv1.weight = nn.Parameter(model.conv1.weight[:63])
print(model.conv1.weight.shape)
# torch.Size([63, 3, 7, 7])
</code></pre>
AvgPool2d and non-deterministic results

username: papoo13
cooked: <p>Is AvgPool2d is non-deterministic?<br>
Only using this causes reproducibility issues in my model.<br>
o have set all the seeds and determinism=True as well. when I run the code, I don’t get any errors regarding having a non-deterministic function but the results are affected. Does anyone have any idea about this? if it is,  why? and is there a way to efficiently implement it?</p>

username: ptrblck
cooked: <p><code>nn.AvgPool2d</code> should be deterministic as no error is raised in this code:</p>
<pre><code class="lang-python">import torch
import torch.nn as nn

torch.use_deterministic_algorithms(True)

pool = nn.AvgPool2d(2).cuda()
x = torch.randn(1, 3, 224, 224, device="cuda", requires_grad=True)

for _ in range(10):
    out = pool(x)
    out.mean().backward()
    print(out.double().abs().sum())
    print(x.grad.double().abs().sum())
    x.grad = None
</code></pre>
<p>and the values also do not diverge.<br>
Some non-deterministic layers are mentioned in the <a href="https://pytorch.org/docs/stable/generated/torch.use_deterministic_algorithms.html#torch.use_deterministic_algorithms">docs of <code>torch.use_deterministic_algorithms</code></a>.</p>

username: papoo13
cooked: <p>Thank you for the reply. I do not face the reproducibility issue on the same GPU model but when I run my code on two different machines ( I have the same environment, seeds are set up and also determinism is True and I do not get any warning regarding non-deterministic function), I get different results. I have a Bayesian model used in continual set up and I use the learned posterior as the next prior (= part of the ELBO loss) and even tiny changes in the first posterior causes the differences to grow larger and larger in such a set up. Any idea why running the code on two different machines doesn’t lead to reproducible results even when everything has been set up?</p>

username: ptrblck
cooked: <p>Bitwise-identical results between different setups are not guaranteed since e.g. different code paths could be used based on the GPU capability.<br>
Using deterministic settings should make sure to yield deterministic outputs on the same setup.<br>
You should also note that both setups will run into the same limit of the floating point precision and none of them is “more correct” than the other assuming both are using the same <code>dtype</code>.</p>
MIG compatibility with PyTorch

username: bergentruckung
cooked: <p>Hello,</p>
<p>Are there any known issues with memory allocation for PyTorch on Nvidia’s MIG instances (GPU is A100 80GB and the variant is 2g.20gb)? We’re seeing some crashes that worked fine on non-MIG enabled cards (like, T4).</p>
<p>We’re on PyTorch 1.11.0.</p>
<p>Thanks,<br>
Bergen</p>

username: ptrblck
cooked: <p>There are no known issues. What kind of crashes are you seeing?</p>
Why can't I do so?

username: Nino_Shnitcel
cooked: <p>May be my question is naive and it has an answer in the python documentation.  But so<br>
I am trting to build such a network structure</p>
<pre><code class="lang-auto">N = 5
INPUT_SIZE = 10
class Net(nn.Module):
    def __init__(self, input_size):
        super(Net, self).__init__()
        self.input_size = input_size
        self.in_out_size = [input_size + N - i for i in range(input_size)]
        self.line = [nn.Linear(self.in_out_size[i], self.in_out_size[i+1]) for i in range(input_size-1)]

 def forward(self, x):
        for i in range(self.input_size):
            x = self.line[i](x)
        return x
</code></pre>
<p>But I can’t do it because I have empty the parameters list.</p>
<pre><code class="lang-auto">if __name__ == '__main__':
    model = Net(INPUT_SIZE)
    print(model.parameters())
    []
</code></pre>
<p>Can I use a cycles of kind <code>self.line = [nn.Linear/nn.GRU/nn.CNN for i in range()]</code> for bilding of network?</p>

username: J_Johnson
cooked: <p>It is possible but you’ll need to changes two lines of code:</p>
<pre><code class="lang-auto">self.line = nn.ModuleList([nn.Linear(self.in_out_size[i], self.in_out_size[i+1]) for i in range(input_size-1)])
</code></pre>
<p>You need to wrap the list in a ModuleList wrapper. You can also apply other types of list operations after you’ve done so, such as <code>self.line.append(...)</code>.</p>
<p>And in your forward pass, should match the same range as in the ModuleList, or you’ll get an error at run time:</p>
<pre><code class="lang-auto">for i in range(self.input_size - 1):
            x = self.line[i](x)
</code></pre>

username: Nino_Shnitcel
cooked: <p>Thank you! It really works!</p>
Load my labels into GPU

username: D_ilo
cooked: <p>Hi I’m sure this topic is well known and people already asked this question. But I couldn’t solve my issue, which is loading my labels into GPU.</p>
<p>I’m Using a deit_base_patch16_224 model with timm library and I would like to train my model on GPU. Here is a quick look at my code :</p>
<pre><code class="lang-auto"># Load the pre-trained ViT model from timm
model = timm.create_model("deit_base_patch16_224", pretrained=True)
next(model.parameters()).is_cuda
model = model.to(device)
model.reset_classifier(20)

# Define the loss function and optimizer
criterion = nn.BCEWithLogitsLoss()
optimizer = optim.Adam(model.parameters(), lr=1e-4)


# Train the model
for epoch in tqdm(range(num_epochs)):
    # Set the model to training mode
    model.train()

    # Loop over the training dataset in batches
    for i, (inputs, labels) in enumerate(train_loader):
        # Zero the gradients
        labels_ = labels.to(device)
        inputs = inputs.to(device)
        print(labels.is_cuda)
        optimizer.zero_grad()
        # Forward pass
        outputs = model(inputs)
        
        loss = criterion(outputs, labels.to(device))

        # Backward pass and optimization step
        loss.backward()
        optimizer.step()

    # Set the model to evaluation mode
    model.eval()

    # Compute the validation accuracy
    correct = 0
    total = 0
    with torch.no_grad():
        for inputs, labels in val_loader:
            outputs = model(inputs)
            predicted = (torch.sigmoid(outputs) &gt; 0.5).int()
            total += labels.size(0)
            correct += (predicted == labels).all(dim=1).sum().item()
    val_accuracy = 100 * correct / total

    # Print the epoch number and validation accuracy
    print("Epoch {}/{}: Validation Accuracy = {:.2f}%".format(epoch+1, num_epochs, val_accuracy))

</code></pre>
<p>My inputs and my model already beeing on GPU I’m having this error :</p>
<pre><code class="lang-auto">RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument mat1 in method wrapper_addmm)
</code></pre>

username: D_ilo
cooked: <p>I solved it by noticing that I needed to first run this line :</p>
<pre><code class="lang-auto">model.reset_classifier(20)
</code></pre>
<p>before putting the model into GPU :</p>
<pre><code class="lang-auto">model = model.to(device)
</code></pre>
<p>A cool line of code to use to check if all of your parameters are running with GPU is :</p>
<pre><code class="lang-auto">for name, param in model.named_parameters(): 
   print(name, param.device)
</code></pre>
Expected more than 1 spatial element when training

username: ljeonjko
cooked: <p>Hi everyone!</p>
<p>I have a network designed for 64x64 images (height/width), I’m trying to reform it to take input of 8x8. I’ve managed to fix the generator, but I’m stuck with the discriminator:</p>
<pre><code class="lang-auto">class Discriminator(nn.Module):
    def __init__(self, latent_vector_size, features_d, num_channels):
        super(Discriminator, self).__init__()
        
        self.latent_vector_size = latent_vector_size
        self.features_d = features_d
        self.num_channels = 2 #num_channels # OVO JE PROMINJENO DA GLEDA SAMO JEDAN KANAL
        self.optimizer = None
        self.main = None
    
    @staticmethod    
    def discriminator_block(in_filters, out_filters, stride, normalize):
        
        layers = [
            nn.Conv2d(
                in_channels = in_filters,
                out_channels = out_filters,
                kernel_size = 3,
                stride = stride,
                padding = 1                
                )
            ]
        if normalize:
            layers.append(
                nn.InstanceNorm2d(out_filters)
                )
        
        layers.append(
            nn.LeakyReLU(0.2, inplace = True)
            )
        
        return layers
        
        
    def build(self):
        
        layers = []
        in_filters = self.num_channels
        
        for out_filters, stride, normalize in [(self.features_d, 2, False), (self.features_d * 2, 2, True),
                                               (self.features_d * 4, 2, True), (self.features_d * 8, 1, True)]:
        
            layers.extend(self.discriminator_block(
                in_filters,
                out_filters,
                stride,
                normalize
                ))
            
            in_filters = out_filters
        
        layers.append(
            nn.Conv2d(
                in_channels = out_filters,
                out_channels = 1,
                kernel_size = 3,
                stride = 1,
                padding = 1
                )
            )
        self.main = nn.Sequential(*layers)
        
    def forward(self, input):
        return self.main(input)
    
    def define_optim(self, learning_rate, beta1):
        self.optimizer = optim.Adam(self.parameters(), lr = learning_rate, betas = (beta1, 0.999))
    
    @staticmethod   
    def init_weights(layers):
        classname = layers.__class__.__name__
        if classname.find('Conv') != -1:
            nn.init.normal_(layers.weight.data, 0.0, 0.02)
            
        elif classname.find('BatchNorm') != -1:
            nn.init.normal_(layers.weight.data, 1.0, 0.02)
            nn.init.constant_(layers.bias.data, 0)
</code></pre>
<p>Input is <code>[20, 2, 8, 8]</code>, but the discriminator gets to <code>torch.Size([20, 16, 1, 1])</code>. How can I adjust the discriminator so that <code>nn.InstanceNorm2d</code> gets the proper input size?</p>
<p>Thanks!</p>
Variable number of input images into CNN

username: Johnny_Cash
cooked: <p>Is it possible to input a variable number of images into a CNN? I would like to pool information from a stack of images. I suppose this is possible for a fixed number of images - for 5 images for example input a tensor with depth 15 because of RBG and then maybe use Conv3D and MaxPool3D to reduce dimension. Is this somehow possible for non static number of images or does the number of images that has to be put in fixed?</p>
<p>I suppose this should work somehow because for example you could write a network - a fully convolutional one - where you can input random size of images, so you remove the limitation along the 2d axis basically.</p>

username: J_Johnson
cooked: <p>You can always alter the batch_size. Otherwise, you can use Conv3d layers and stack your images so they are of shape batch_size, channels, num_images, dim1, dim2. Then just alter the kernel_size, stride, dilation, etc. to keep that dim size the same. But you would not be able to stack batches of images of different number of images in this method. Also, you would have learned relationships between the the images stacked, and so order of the images would become relevant.</p>

username: Johnny_Cash
cooked: <p>Ok, so one batch would have sets with each same number of images. And the next batch could just have sets also with the same number of images within the batch.</p>

username: J_Johnson
cooked: <p>You can send into a model any batch_size your memory can handle, other than that, there is no limitation on the batch_size between back-propagation. Granted, your learning rate might need an appropriate adjustment to match the batch_size and avoid overfitting to smaller batches.</p>

username: Johnny_Cash
cooked: <p>That’s clear. I meant something different, but nevermind I figured it out. Thanks!</p>
Should I detach loss target

username: Vladislav_Korecky
cooked: <p>When calculating a loss (with the <code>mse_loss</code> function), should I call <code>.detach()</code> on the <strong>target</strong> tensor if it has grad_fn on it? And does it even matter?</p>

username: ptrblck
cooked: <p>I depends on your use case and if you want to “train” the target tensor. Could you explain why the target has a valid <code>grad_fn</code>, how it was created, and if you want to calculate gradient for this operation?</p>

username: Vladislav_Korecky
cooked: <p>I am making a DQN (Deep Q Network), a reinforcement learning strategy, in which the network approximates a reward for actions in a given state. When training, the target for a single action is the reward from the environment + the predicted future reward ( <code>target = reward + t.max(net.forward(next_state))</code> ). This is why my target has an autograd because part of it is a prediction. So no, I don’t want the target to be trained, but I also don’t want to waste compute time calling <code>.detach()</code> on everything, that’s why I’m asking if it matters.</p>

username: ptrblck
cooked: <p>I would assume the gradient calculation would be more expensive than calling <code>.detach()</code> on a tensor.<br>
In any case, you could also directly wrap the target computation into a <code>with torch.no_grad()</code> guard to avoid creating the computation graph in the first place.</p>

username: Vladislav_Korecky
cooked: <p>Thanks, helps a lot!</p>