akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Ddp distributed training card 0 occupied more space

username: xjm
cooked: <p>here is my test code as follows, it seems no errors</p>
<pre><code class="lang-auto">import os
os.environ["CUDA_VISIBLE_DEVICES"] = "1, 2,3,4"

import torch
import torch.distributed as dist
import torch.multiprocessing as mp
import torch.nn as nn
import torch.optim as optim
from torch.nn.parallel import DistributedDataParallel as DDP

def example(rank, world_size):
    # create default process group
    dist.init_process_group("gloo", init_method='tcp://127.0.0.1:6666', rank=rank, world_size=world_size)
    # create local model
    model = nn.Linear(10, 10).to(rank)
    # construct DDP model
    ddp_model = DDP(model, device_ids=[rank])
    # define loss function and optimizer
    loss_fn = nn.MSELoss()
    optimizer = optim.SGD(ddp_model.parameters(), lr=0.001)

    # forward pass
    outputs = ddp_model(torch.randn(20, 10).to(rank))
    labels = torch.randn(20, 10).to(rank)
    # backward pass
    loss_fn(outputs, labels).backward()
    # update parameters
    optimizer.step()
    print("finished rank: {}".format(rank))

def main():
    world_size = torch.cuda.device_count()
    mp.spawn(example,
        args=(world_size,),
        nprocs=world_size,
        join=True)

if __name__=="__main__":
    main()
</code></pre>
<p>but the result is very strange, as we can see, the card 0 occupied more space<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/9/8/9820a3faf4ee273176e28541a0311116873c6957.jpeg" alt="b3da9abf3e2fbae4e83160570d3de6c" data-base62-sha1="lHMvt3JWImqFBZZDhWBvlc5vrX9" width="562" height="237"></p>
<p>I don’t know where the error occurred</p>

username: ptrblck
cooked: <p>Double post from <a href="https://discuss.pytorch.org/t/torch-not-able-to-utilize-gpu-ram-properly/83245/14">here</a> with follow-up.</p>
Do I need to implement my own collective communication primitive?

username: zxgx
cooked: <p>Hi, I’m wondering if I should (and how to) implement a weird collective primitive to fulfill my experiment.</p>
<p>Basically, in my experiment, each process has two process groups, and each call of this weird primitive would invoke one all-gather in each group.</p>
<p>Would it be more communication-effective if I compact these two all-gather into a single new collective primitive?</p>
<p>If so, how can I implement this primitive? I notice that there is an example in <code>test/cpp_extensions/cpp_c10d_extension.cpp</code>, but the problem is that I know nothing about the <code>c10d</code> code organization. Where should I start to understand the source code of <code>torch.distributed</code> to implement my own primitive?</p>
Torchserve support for model parallel / model pipelining?

username: vadimkantorov
cooked: <p>I saw some support for DAGs in torchserve docs: <a href="https://pytorch.org/serve/workflows.html" class="inline-onebox" rel="noopener nofollow ugc">19. TorchServe Workflows — PyTorch/Serve master documentation</a></p>
<p>I wonder if torchserve supports model pipelining? e.g. preprocessing and postprocessing should run on their own pools of CPU threads to minimize waiting time of the main GPU model thread</p>
<p>For that, preprocessing and postprocessing stages should somehow have their own input/output queues and their own pools of CPU threads (maybe even pinned to actually different CPU cores), and then preprocessing / main model and postprocessing  should be connected in some sort of pipeline</p>
<p>Is this scenario supported by torchserve?</p>
<p>In the ideal world, even multi-node deployments like this are usefulю e.g. some video file decoding done one one powerful CPU machine which sends the results to the lightCPU+strongGPU machine. If pipelines/queues like this can be natively orchestrated by torchserve, it’d be really great</p>

username: marksaroufim
cooked: <p>It’s something we’ve talked about quite a bit since preprocessing is often the real bottleneck we’ve found in user code - this is an RFC, feel free to ping me and Matthias on the thread and we can chat more about this <a href="https://github.com/pytorch/serve/pull/2189" class="inline-onebox" rel="noopener nofollow ugc">[RFC] Composable handler by mreso · Pull Request #2189 · pytorch/serve · GitHub</a></p>

username: vadimkantorov
cooked: <p>Hmm, as an outsider - this RFC does not make it clear if these handle parallelization/separate queues/threads of these preproc and postproc and how to handle scheduling / CPU pinning / inter-node pipeline communication</p>
<p>By pipelining I mean something like the instruction handling pipeline that is found in CPUs: separating some parts of the data processing flow into queues and handling these queues on different sets of CPU cores</p>
Basketball 3 points line recognition

username: Ariel_Su
cooked: <p>I ’m doing a project and I’m trying to detect the 3 points line in a basketball court. The frames are tv frames, taken from mid height and lateral view.</p>
<p>I’ve calculated an homography. Over the image after applying the homographic transformation I’ve run canny and circular hough, and tried to find an ellipse that will catch the upper part of the court, but I can’t get anything that is good enough. Does anyone have a suggestion to what could I do?</p>

username: DoctorPolygon
cooked: <p>To build training data? Or are you looking for an architecture or network idea to implement?</p>
Loading model and optimizer and then fine-tuning only last layer

username: davidfeijoo
cooked: <p>I am trying to train a model with SGD(lr = 0.001, momentum = 0.9) and save it after training in order to then adapt it to a subset of the training speakers by freezing all the weights and only training the ones in the last layer. How should I manage the loading of the optimizer and changing its parameters to only the last layer? I know how to freeze the layers but then the optimizer size doesn’t match the loaded one:</p>
<pre><code class="lang-auto"># Load the model and optimizer state_dicts
    checkpoint = torch.load(modelPath,map_location=device)

    # Apply pretrained model weights
    model.load_state_dict(checkpoint['model_state_dict'])

    # Freeze the weights of the first layers
    for param in model.parameters():
        param.requires_grad = False

    # Enable weight updates in the last layer
    for param in model.output_layer.parameters():
        param.requires_grad = True

    # Define the optimizer
    optimizer = optim.SGD(filter(lambda p: p.requires_grad, model.parameters()), learning_rate, momentum=0.9)
    optimizer.load_state_dict(checkpoint['optimizer_state_dict'])  #doesn't match de size of the new optimizer
</code></pre>

username: KFrank
cooked: <p>Hi David!</p>
<aside class="quote no-group" data-username="davidfeijoo" data-post="1" data-topic="181901" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/davidfeijoo/48/61089_2.png" class="avatar"> davidfeijoo:</div>
<blockquote>
<pre><code class="lang-auto">    # Define the optimizer
    optimizer = optim.SGD(filter(lambda p: p.requires_grad, model.parameters()), learning_rate, momentum=0.9)
    optimizer.load_state_dict(checkpoint['optimizer_state_dict'])  #doesn't match de size of the new optimizer
</code></pre>
</blockquote>
</aside>
<p>It seems to me that the simplest solution would be not to load the stored<br>
state of the optimizer.  Just create the new optimizer and use it.</p>
<p>It is true that because its <code>momentum</code> is non-zero, <code>optimizer</code> does carry<br>
non-trivial state – the accumulated momenta from the gradients of prior<br>
(pre-saving) optimization steps.  But I don’t think that zeroing out those<br>
momenta and then starting to reaccumulate them as you take optimization<br>
steps for for the weights in the last <code>output_layer</code> that you are fine tuning<br>
is likely to matter much.</p>
<p>(Hypothetically, you could try to dig the final-layer momenta out of the stored<br>
<code>optimizer_state_dict</code>, but I wouldn’t bother with it.  I’m not aware of any<br>
pytorch feature that would do this for you at a conveniently high level.)</p>
<p>Best.</p>
<p>K. Frank</p>
Deterministic Algorithms yield an error

username: fabikonsti
cooked: <p>Hi,<br>
I try to ensure reproducibility in my code by setting the random seeds and using only deterministic algorithms.<br>
However, the following simple code yields an error in my implementation:</p>
<pre><code class="lang-auto">import torch
torch.use_deterministic_algorithms(True, warn_only=True)
device = torch.device("cuda")
tensor = torch.ones(size=(10, 2), device=device, dtype=torch.long)
indices = torch.arange(4, 8, device=device)
value = 5.

tensor[indices] = int(value)
</code></pre>
<p>The error I obtain is:<br>
linearIndex.numel()<em>sliceSize</em>nElemBefore == expandedValue.numel() INTERNAL ASSERT FAILED at “…/aten/src/ATen/native/cuda/Indexing.cu”:389, please report a bug to PyTorch. number of flattened indices did not match number of elements in the value tensor: 8 vs 4<br>
File “debug_determinism.py”, line 14, in <br>
tensor[indices] = int(value)</p>
<p>The same code runs totally fine on the CPU, but yields the error message when executing the code on the GPU.<br>
Is this a bug in the current release?</p>

username: ptrblck
cooked: <p>The error should be related to <a href="https://github.com/pytorch/pytorch/issues/79987">this issue</a>.<br>
CC <a class="mention" href="/u/eqy">@eqy</a> as I’m unsure what the current status is.</p>

username: fabikonsti
cooked: <p>In the referenced post, index_put is used, which expects an tensor of the same shape. However, it should also be possible to assign the same value to all indexed entries (as I tried in the example above). Thus, I am not quite sure if the problems are related.</p>
<p>Strangely, it works with a boolean value, but not with a numerical value.</p>
Why is dist.destroy_process_group not mentioned in the official documentation?

username: Fei_Liu
cooked: <p><a href="https://pytorch.org/docs/stable/distributed.html" rel="noopener nofollow ugc">Official torch.distributed web doc</a>.</p>
<p>Are there any special reasons to not call <code>dist.destory_process_group</code> ?</p>

username: ptrblck
cooked: <p>I don’t think there is a specific reason for it and it might just have been forgotten. Would you be interested in contributing the missing docs?</p>

username: Fei_Liu
cooked: <p>Got it. Thanks for the reply! I’d like to leave it to the official pytorch team.</p>
Setting and Accessing Attributes in PyTorch Models

username: blade
cooked: <ol>
<li>Is it possible to set attributes to model parameters by looping over <code>model.named_children()</code> and accessing them later by looping over ‘self.model.named_parameters()’? One example is below:</li>
</ol>
<pre><code class="lang-auto">for key, val in self.model.named_children():
    val.classname = val.__class__.__name__

for key, val in self.model.named_parameters():
    print(val.classname)
</code></pre>
<ol start="2">
<li>More generally, what is a good read to understand how <code>named_children()</code>, <code>named_modules()</code>, <code>named_parameters()</code>, and <code>state_dict()</code> differ from each other and what are their application?</li>
</ol>
Parameters to pass to clip_grad_norm

username: nntoan209
cooked: <p>Hi, after using model = DataParallel(model), when using clip_grad_norm_ function, how should I pass the parameters, model.parameters() or model.module.parameters() ?<br>
I’m training a ViT model, when I set max epoch to 100 epochs, everything was fine, but when I set max epoch to 200, the loss became [nan] after 55 epochs, and I’m afraid it’s because of the gradient clipping step</p>
Issue with running Tactron2 with Pytorch2

username: rawanessam
cooked: <p>Hi all,<br>
trying to use torch compile with Tacotron model, but I always get that error (‘lengths’ argument should be a 1D CPU int64 tensor, but got 1D meta Long tensor), that fired in <a href="https://github.com/NVIDIA/tacotron2/blob/master/model.py#L173" rel="noopener nofollow ugc">forward function</a> when calling x = nn.utils.rnn.pack_padded_sequence()<br>
even when converting input_lengths to CPU.</p>
How to try except GPU runtime error when OOM

username: FreyWang
cooked: <p>I try to catch the except so that model can change to next gpu automatically, but it do not work</p>

username: ptrblck
cooked: <p>Have a look at the <a href="https://github.com/pytorch/fairseq/blob/d3795d6cd1c66ac05dc0f4861ce69ab4680bff3d/fairseq/trainer.py#L139-L152">FairSeq example</a> on how to recover from OOM errors.<br>
They just skip the batch and try to continue the training. You could try to adapt this example to move your model.</p>

username: Todd_Morrill
cooked: <p>This seems like a really useful trick to have handy for single GPU training. How should this interact with Torch distributed? My first idea was to use the <a href="https://pytorch.org/tutorials/advanced/generic_join.html" rel="noopener nofollow ugc">Join context manager</a>, which handles a variable number ranks participating in all-reduce operations, but it seems like Join is a one way door, once you <strong>exit</strong>() the Join context, the placeholder ops won’t allow a rank to continue training. So it seems like you need to be able to dynamically participate in placeholder ops, which may not be trivial to implement. Maybe the solution is <code>torch.distributed.elastic</code>, where you just let the rank crash and spin it back up to join the process group?</p>

username: ptrblck
cooked: <p>Letting the rank crash might work but I would also assume restoring the training might not be trivial.<br>
I would also expect to see the same memory usage in <code>DDP</code> so do you know why one rank runs OOM?</p>

username: Todd_Morrill
cooked: <p>I’m primarily focused on large language models at the moment and long sequence lengths cause OOM issues. I could probably be more rigorous about finding the maximum sequence length (and batch size) that will fit in GPU memory though.</p>
Value Error visualizing Network using networkx

username: cvillanue
cooked: <p>Hello all!</p>
<p>I was told that using the NetworkX package was helpful in creating complex graphs, so I decided to go ahead and use NetworkX to visualize the logic behind forward propagation where each layer of the neural network is represented as a subset of nodes. Unfortunately, I ran into this issue:</p>
<pre><code class="lang-auto">ValueError: all nodes must have subset_key (default='subset') as data
</code></pre>
<p>however I have already stated this in my code. I’m attaching the graph creation of the code below. Any modification or suggestions would be greatly appreciated!</p>
<pre><code class="lang-auto"># Create a graph object
G = nx.DiGraph()

# Adding nodes &amp; edges
G.add_nodes_from(['Input Layer'] + ['Hidden Layer ' + str(i) for i in range(1, hidden_size+1)] + ['Output Layer'])

for i in range(input_size):
    G.add_edge('Input Layer', 'Hidden Layer 1', weight=weights_hidden[i][0])
for i in range(hidden_size):
    if i &lt; hidden_size-1:
        G.add_edge('Hidden Layer ' + str(i+1), 'Hidden Layer ' + str(i+2), weight=weights_hidden[:, i+1])
    G.add_edge('Hidden Layer ' + str(i+1), 'Output Layer', weight=weights_output[i])

# Set positions for nodes
pos = nx.multipartite_layout(G, subset_key='subset')
nx.draw_networkx_nodes(G, pos, node_color='lightblue', node_size=500, alpha=0.8)

edge_labels = {(u, v): round(d['weight'], 2) for u, v, d in G.edges(data=True)}
nx.draw_networkx_edges(G, pos, width=2, alpha=0.8, arrows=True)
nx.draw_networkx_edge_labels(G, pos, edge_labels=edge_labels, font_size=8)

node_labels = {node: node for node in G.nodes}
nx.draw_networkx_labels(G, pos, node_labels, font_size=10, font_weight='bold')

plt.axis('off')

# plotting
plt.title('Feed-forward Neural Network Architecture')
plt.tight_layout()
plt.show()
</code></pre>
Cannot run DDP on 2 GPUS

username: srivassid
cooked: <p>Hi</p>
<p>I am trying out a gpt-like tool here</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/karpathy/nanoGPT">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/karpathy/nanoGPT" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/345;"><img src="https://opengraph.githubassets.com/40733d4d22686500b61d434612442460e8604b78179e99df88a86f126c567b0f/karpathy/nanoGPT" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/karpathy/nanoGPT" target="_blank" rel="noopener nofollow ugc">GitHub - karpathy/nanoGPT: The simplest, fastest repository for...</a></h3>

  <p>The simplest, fastest repository for training/finetuning medium-sized GPTs. - GitHub - karpathy/nanoGPT: The simplest, fastest repository for training/finetuning medium-sized GPTs.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>But i cannot get it to run on multiple gpus. It has the funcitonality to run on multiple gpus (and nodes even) but with the command <code>torchrun --standalone --nproc_per_node=2 train.py</code> i get an error</p>
<p><code>RuntimeError: Default process group has not been initialized, please make sure to call init_process_group</code></p>
<p>I can run it on a single gpu though.<br>
What can i do? I am new to pytorch. The code that implements ddp is like the following</p>
<pre><code class="lang-auto">ddp = int(os.environ.get('RANK', -1)) != -1 # is this a ddp run?
# ddp = True
if ddp:
    # init_process_group(backend=backend)
    ddp_rank = int(os.environ['RANK'])
    ddp_rank = 0
    ddp_local_rank = int(os.environ['LOCAL_RANK'])
    ddp_local_rank = 1
    ddp_world_size = int(os.environ['WORLD_SIZE'])
    ddp_world_size = 2
    device = f'cuda:{ddp_local_rank}'
    torch.cuda.set_device(device)
    master_process = ddp_rank == 0 # this process will do logging, checkpointing etc.
    seed_offset = ddp_rank # each process gets a different seed
    #assert gradient_accumulation_steps % torch.cuda.device_count() == 0
    gradient_accumulation_steps //= torch.cuda.device_count()
else:
    # if not ddp, we are running on a single gpu, and one process
    master_process = True
    seed_offset = 0
    ddp_world_size = 1
</code></pre>
<p>Btw i have commented out the assert part. Also, as i understand the value are passed as environment variables?</p>
<p>Thanks</p>

username: ptrblck
cooked: <p>The <a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html">DDP tutorial</a> might be a good starter.</p>

username: srivassid
cooked: <p>Ok I’ll look into it<br>
Thanks</p>
Crossentropy error for binary classification

username: Navya
cooked: <p>RuntimeError: Expected target size [128, 2], got [128]</p>
<p>The output and target shapes are torch.Size([128, 256, 2]) and torch.Size([128]). Trying a binary classification problem.<br>
Appreciate any help on this.<br>
Thank you,</p>

username: ptrblck
cooked: <p>The shapes are wrong as they don’t correspond to each other for a multi-class classification using <code>nn.CrossEntropyLoss</code>.<br>
As described in the <a href="https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html#torch.nn.CrossEntropyLoss">docs</a> the output should have the shape <code>[batch_size, nb_classes, *]</code> containing logits and the target should have the shape <code>[batch_size, *]</code> containing class indices in the range <code>[0, 1]</code> or the same shape if probabilities are used.</p>
<p>In particular the model output should be permuted and you would then have to either reduce the dimemension with the size <code>256</code> or provide <code>256</code> targets for each sample.</p>
How to pass gamma and beta to batch_norm in torch?

username: Snowy
cooked: <p>Hello,</p>
<p>I am trying to convert some code that involves conditional batch normalization from Tensorflow to Pytorch. In TF, you can call tf.nn.batch_normalization() which accepts the input, mean, variance, scale, and shift (gamma and beta). However, in Pytorch if I call torch.nn.functional.batch_norm, it does not have any parameters for mean, variance, gamma, and beta. Or have I got this wrong?</p>
<p>Here is the parameter list:</p>
<pre><code class="lang-auto">return F.batch_norm(
            input,
            # If buffers are not to be tracked, ensure that they won't be updated
            self.running_mean
            if not self.training or self.track_running_stats
            else None,
            self.running_var if not self.training or self.track_running_stats else None,
            self.weight,
            self.bias,
            bn_training,
            exponential_average_factor,
            self.eps,
        )
</code></pre>
<p>BTW, I don’t want to use BatchNorm2 because it does not have an option to pass the above-mentioned values.</p>

username: ptrblck
cooked: <p>The <code>mean</code> and <code>var</code> will be calculated from the input activation and I don’t know why you want to calculate it beforehand.<br>
Every other parameter is accepted as an argument where <code>weight</code> and <code>bias</code> correspond to <code>gamma</code> and <code>beta</code>, respectively.</p>

username: Snowy
cooked: <p>I see, I thought I should calculate them myself. Thank you for your reply <a class="mention" href="/u/ptrblck">@ptrblck</a>!</p>
AttributeError: module 'torchvision' has no attribute 'io'

username: njionou
cooked: <p>Dear all,</p>
<p>I would like to prepare an image dataset with torch and while typing</p>
<p><em>import torchvision</em></p>
<p><em>tsr_img =  torchvision.io.read_image(‘/content/PyTorch_Load_Image_FromRepository/data/image-dog/bobby.jpg’)</em></p>
<p>I receive as error</p>
<p>AttributeError: module ‘torchvision’ has no attribute ‘io’</p>
<p>I have tried to update torchvision but the same error arises. Is there any solution?</p>
<p>With regards<br>
Patrick</p>

username: ptrblck
cooked: <p>It works for me using e.g. <code>0.16.0.dev20230524+cu118</code> and it’s unclear which <code>torchvision</code> version you are using so could you add the missing information for us to reproduce the issue?</p>

username: njionou
cooked: <p>I have now updated my version or torch and it seems to work. Thank you for your suggestion.</p>
<p>With regards.<br>
Patrick</p>
Converting PyTorch model to TorchScript, ValueError: You have to specify either decoder_input_ids or decoder_inputs_embeds

username: rhwang
cooked: <p>I’m converting the SwitchTransformer model from HuggingFace to TorchScript.</p>
<p>But I encountered the following error message</p>
<p><code>ValueError: You have to specify either decoder_input_ids or decoder_inputs_embeds</code>.</p>
<p>In some other QnAs in StackOverflow, it said that Enc-Dec model cannot be converted at once and it should be divided into encoder and decoder, then it can be converted, but I still cannot understand. The model itself is integrated at the function level.</p>
<p>Does that mean that I should divide it making a new function?</p>
<p>Here are the two links that I referred to.</p>
<ul>
<li><a href="https://stackoverflow.com/questions/65140400/valueerror-you-have-to-specify-either-decoder-input-ids-or-decoder-inputs-embed" class="inline-onebox" rel="noopener nofollow ugc">python - ValueError: You have to specify either decoder_input_ids or decoder_inputs_embeds - Stack Overflow</a></li>
<li><a href="https://stackoverflow.com/questions/66109084/how-to-convert-huggingfaces-seq2seq-models-to-onnx-format/66117248#66117248" class="inline-onebox" rel="noopener nofollow ugc">python - how to convert HuggingFace's Seq2seq models to onnx format - Stack Overflow</a></li>
</ul>
<p>Also, in the answer in the above link, someone said that the converted model doesn’t have <code>generate</code> function, then how can I make an inference?</p>
<p>Here’s the code that I’m using now.</p>
<pre><code class="lang-auto">from transformers import AutoTokenizer, SwitchTransformersForConditionalGeneration
from transformers import AutoTokenizer, SwitchTransformersConfig
import torch

torch.set_printoptions(threshold=10_000)

# Tokenizer
tokenizer = AutoTokenizer.from_pretrained(
    "google/switch-base-8", resume_download=True, torchscript=True)

input_text = "A &lt;extra_id_0&gt; walks into a bar a orders a &lt;extra_id_1&gt; with &lt;extra_id_2&gt; pinch of &lt;extra_id_3&gt;."
input_ids = tokenizer(input_text, return_tensors="pt").input_ids.to(0)
# To use GPU, add .to(0)
input_ids = tokenizer(input_text, return_tensors="pt", padding=True).input_ids.to(0)

# To use GPU, add args device_map="auto"
model = SwitchTransformersForConditionalGeneration.from_pretrained(
    "google/switch-base-8",
    device_map="auto",
    resume_download=True, torch_dtype=torch.bfloat16,
    torchscript = True,
    )

# This is for TorchScript
model.eval()

model = torch.jit.trace(model, (input_ids))

outputs = model.generate(
    input_ids,
    decoder_start_token_id=0,
    bos_token_id=2,
    )
</code></pre>
Updated to PyTorch 2.0 BUT

username: Aeychpge
cooked: <p>So earlier today, I got tired of my User.bat telling me I was running Torch 1.13+cu117 and that I should update to 2.0. So I did.</p>
<p>Only, I just ran my user.bat and its still telling me I’m running 1.13+cu117.<br>
When I updated Torch, it said something about uninstalling the previous version, or maybe I hallucinated that, but regardless, any and all help with this conundrum would be greatly appreciated because I can’t find anything on this and reddit has been less than helpful.</p>

username: vdw
cooked: <p>It seems you’re on Windows, so I cannot truly help.</p>
<p>But incidentally I’ve updated to 2.0 yesterday, and during the update using pip it definitely said it removed the previous version. So this part I can confirm :).</p>
<p>What’s the output of</p>
<pre><code class="lang-auto">import torch

print(torch.__version__)
</code></pre>

username: Aeychpge
cooked: <p>Someone reached out to me on Discord and helped me manually update it so I am now all good!</p>
<p>Am now running Torch 2.0</p>

username: aifartist
cooked: <p>A possibility:<br>
You were in the directory where you start the app.<br>
But you hadn’t set the venv so the install went to the system level location.<br>
Then you started the app which might source venv/bin/activate to use the local venv.<br>
But that isn’t where you installed Torch 2.0.</p>
How to output different size than embed_dim with MultiheadAttention?

username: JohnCwok
cooked: <p>I am trying to use self-attention to filter the content of an image, for that I actually provide different tensors for Q, V and K respectively (making sure that the dimensions are compatible with the matrix multiplications). My issue is that the final output has the dimension of Q (i.e. embed_dim) when I would like it to be of the dimension of V (vdim). Looking at the doc, I think it is not possible, but I feel this feature should be there somehow.<br>
Before I start trying to write an implementation of the self-attention, does anyone know how to do what I described?</p>
Distributed training question

username: Jaeseon
cooked: <p>Greetings,<br>
I have been working on a project to implement distributed training and data parallelization on a deep learning object detection model. I’ve been using PySpark, PyTorch, and SparkTorch, although unfortunately it was not working properly.<br>
Could anyone provide resources or insights into these questions?</p>
<ol>
<li>Is the usage of SparTorch (or similar tools such as TensorFlowonSpark or Horovod) necessary in this task? (Distributed Training?)</li>
<li>I have successfully loaded the necessary files from HDFS in PySpark.Dataframe format. Could I potentially bypass the use of SparkTorch and feed this dataset directly into a PyTorch deep learning model?</li>
<li>If the above is feasible, could you guide me on the process of converting PySpark.dataframe into a format that PyTorch can utilize?</li>
<li>If we can indeed bypass SparkTorch, would this mean that the data used in training is still parallelized? And it is trained distributed?</li>
<li>In this scenario, what role would SparkTorch(or similar tools) typically play?</li>
</ol>
<p>For additional context, I have attempted to use the lates version of PySpark to take advantage of the built-in torch.distributor module. However, due to certain constraints, I’m currently working with PySpark 2.4 and Torch.</p>
Dropout internal implementation

username: Seo
cooked: <p>Hi there, I am studying the Dropout implementation in PyTorch. The original paper says that</p>
<blockquote>
<p>" If a unit is retained with probability p during training, the outgoing weights of that unit are multiplied by p at test time"</p>
</blockquote>
<p>this in order to balance the greater number of active connections between the layers. Anyway I have just found that pytorch does not seems to implement dropout in this way, when training boolean is false, it simply returns the output without rescaling.<br>
Furthermore, Dropout does not seem to multiply the activations by a binary mask, I put an example here:</p>
<p>1- During training:</p>
<pre><code class="lang-auto">dr = torch.nn.Dropout(p=.5)
t = torch.tensor((2, 3), dtype=torch.float32)
print(t)
out = dr(t)
print(out)
</code></pre>
<blockquote>
<p>tensor([2., 3.])<br>
tensor([0., 6.])</p>
</blockquote>
<p>I expected to have one 0 entry and the other one unchanged, instead the other one seems to be multiplied by 2 (different runs give me the other case: tensor([4., 0.]))</p>
<p>2-During validation:<br>
The output remains unchanged tensor([2., 3.]) instead of being rescaled by p=.5. In fact in the C++ module it simply returns the input when train is False.</p>
<pre><code class="lang-auto">template&lt;bool feature_dropout, bool alpha_dropout, bool inplace, typename T&gt;
Ctype&lt;inplace&gt; _dropout_impl(T&amp; input, double p, bool train) {
  TORCH_CHECK(p &gt;= 0 &amp;&amp; p &lt;= 1, "dropout probability has to be between 0 and 1, but got ", p);
  if (p == 0 || !train || input.numel() == 0) {
    return input;
  }
...............
</code></pre>
<p>the source code I linked is in <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/Dropout.cpp" rel="noopener nofollow ugc">pytorch/aten/src/ATen/native/Dropout.cpp</a></p>

username: ptrblck
cooked: <p>PyTorch uses the inverse scaling during training, to avoid the operation during inference, which would yield the same overall result, and is mentioned in the dropout paper in section 10:</p>
<blockquote>
<p>Another way to achieve the same effect is to scale up the retained activations by multiplying<br>
by 1/p at training time and not modifying the weights at test time.</p>
</blockquote>

username: ado_sar
cooked: <p>During training with inverted dropout and keep probability <code>p</code>, the output of a neuron is (on average) <code>p*x + (1-p)*0 = p*x</code>. Now if we divided by <code>1/p</code> the output of a neuron is on average <code>x</code>. Why then keeping the <code>1/p</code> factor during inference? Shouldn’t <code>1/p</code> become <code>1</code>?</p>

username: ptrblck
cooked: <p>During inference the drop probability is not used if inverse scaling was already used during training as explained in my previous post.</p>
What's the aten::ScalarImplicit?

username: MartinZhang
cooked: <p>what’s the aten::ScalarImplicit?</p>
<p>where can I find the reference documentation about it.</p>
<p>thanks.</p>

username: ptrblck
cooked: <p>I don’t think this class is publicly documented but you could check its implementation <a href="https://github.com/pytorch/pytorch/blob/2e21cb095a04dccb1900367623b4ca59ddcf26a2/torch/csrc/jit/runtime/register_prim_ops.cpp#L219-L227">here</a> which calls into <a href="https://github.com/pytorch/pytorch/blob/2e21cb095a04dccb1900367623b4ca59ddcf26a2/torch/csrc/jit/runtime/register_ops_utils.cpp#L106-L121"><code>checkImplictTensorToNum</code></a> and then returns the <code>.item()</code> of the passed tensor, so its scalar value.</p>
Empty tensors, argmax and ONNX exporting

username: rbvh
cooked: <p>I’m trying to export a model to ONNX format. One component of the model is that, for an <code>n</code> by <code>n</code> boolean matrix <code>M</code>, I need to determine the index of the first <code>True</code> in every row. The easy way to do this is just <code>torch.argmax(M, dim=-1)</code>. However, <code>M</code> is sometimes empty, and <code>torch.argmax</code> throws an error. It seems like it would be easy to fix this by just checking <code>M.numel() == 0</code>, but I get the following warning:</p>
<pre><code class="lang-auto">TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
</code></pre>
<p>I think this means that the ONNX model will always set the result of <code>M.numel() == 0</code> to either <code>True</code> or <code>False</code>, which is definitely not what I want. Is there some way to avoid this issue? I’ve been trying to think of ways to find the first nonzero element without using any <code>min</code> or <code>max</code> functions, but it does not seem obvious.</p>

username: rbvh
cooked: <p>Turns out that one thing that seems to work is <code>torch.equal(M, torch.empty(0,0))</code> rather than <code>M.numel() == 0</code></p>
What can be the reasons of BatchNorm working and Dropout not working in YoloV1 Pytorch implementation?

username: KOHTPOJIEP
cooked: <p>Hi. I tried to implement YoloV1 in PyTorch using <a href="https://arxiv.org/abs/1506.02640" rel="noopener nofollow ugc">their paper</a>. Eventually, I got in a situation, when model trained for some time decreasing the loss, but was stuck in middle values before starting point and 0.</p>
<p>I then found <a href="https://github.com/aladdinpersson/Machine-Learning-Collection/tree/master/ML/Pytorch/object_detection/YOLO" rel="noopener nofollow ugc">Aladdin Persson implementation</a> (which he described in <a href="https://youtu.be/n9_XyCGr-MI" rel="noopener nofollow ugc">YouTube video</a>, specific moment regarding BatchNorm <a href="https://youtu.be/n9_XyCGr-MI?t=1017" rel="noopener nofollow ugc">here</a>). He said that original paper used Dropout, because BatchNorm was not invented at the time, and he wants to use BatchNorm instead. I thought there is no critical difference between these two, and decided to stick up with paper for the sake of learning to implement such things.</p>
<p>But my model still couldn’t even overfit. After some investigation, I figured out that only real difference between my implementation and his is using BatchNorm: if I replace BatchNorm with any Dropout bigger than 0.0 in <a href="https://github.com/aladdinpersson/Machine-Learning-Collection/blob/master/ML/Pytorch/object_detection/YOLO/model.py#L38" rel="noopener nofollow ugc">CNNBlock</a> - model starts learning worse. It converges later then normal if Dropout P is 0.1-0.2 and stops converging at all if this P is bigger than 0.2.</p>
<p>Could someone please explain how this works? Should I not use Dropout ever, because BatchNorm is better, or both options have pros and cons?</p>
<p>Specifically to YoloV1 - may be Dropout worked only in their specific learning pipeline with a lot of data and pre-training?</p>