akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
[model.parallelize()] RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:1!

Katerina_Margatina: <p>Hello, I’m really struggling with setting up model parallelism for a large T5 model. I have 2 GPUs and I want to evaluate (inference only) a <code>flan-t5-large</code> model (as it does not fit in one GPU).</p>
<p>I load the model:</p>
<blockquote>
<p><code>device = torch.device("cuda" if torch.cuda.is_available() else "cpu")</code><br>
<code>model = T5ForConditionalGeneration.from_pretrained('google/flan-t5-large',torch_dtype=torch.float16, device_map="auto")</code><br>
<code>model.parallelize()</code><br>
<code>model.to(device)</code></p>
</blockquote>
<p>I put the inputs in the gpu:</p>
<blockquote>
<p><code>input_ids=batch[0].to(device)</code><br>
<code>attention_mask=batch[1].to(device) </code><br>
<code>labels=batch[3].to(device)</code></p>
</blockquote>
<p>But during the forward function</p>
<blockquote>
<p><code>outputs = self.model(input_ids=input_ids, attention_mask=attention_mask, labels=labels)</code></p>
</blockquote>
<p>I get <code>RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:1!</code>.</p>
<p>I check that all input tensors are in <code>device='cuda:0'</code> and as for the model:</p>
<blockquote>
<p><code>model.device</code><br>
<code>device(type='cuda', index=0)</code></p>
</blockquote>
<blockquote>
<p><code>model.device_map</code><br>
<code>{0: [0, 1, 2, 3, 4, 5, 6, 7, 8, ...], 1: [12, 13, 14, 15, 16, 17, 18, 19, 20, ...]}</code></p>
</blockquote>
<p>The error is specifically in <a href="https://github.com/huggingface/transformers/blob/9d1116e9951686f937d17697820117636bfc05a5/src/transformers/models/t5/modeling_t5.py#L260" rel="noopener nofollow ugc">line 260 in modeling_t5.py</a> where I see that <code>self.weight</code> is in <code>'cuda:0'</code> and <code>hidden_states</code> in <code>'cuda:1'</code>.</p>
<p>I would really appreciate some help with this! Thanks in advance.</p>

eqy: <p>Calling <code>model.parallelize()</code> and then <code>model.to(device)</code> immediately after looks suspicious as you would be trampling over the <code>parallelize</code> function’s own <code>.to()</code> <a href="https://github.com/huggingface/transformers/blob/ae54e3c3b18bac0832ad62ea9b896dfd52a09850/src/transformers/models/t5/modeling_t5.py#L889" rel="noopener nofollow ugc">call</a> so I would consider removing the <code>model.to(device)</code> call in your script.</p>

Katerina_Margatina: <p>I removed it and still got the error. But, after trying everything I found that it worked if I also removed the argument <code>device_map="auto"</code> when I load the model!</p>
<p>So the above should be:</p>
<blockquote>
<p><code>model = T5ForConditionalGeneration.from_pretrained('google/flan-t5-large',torch_dtype=torch.float16)</code><br>
<code>model.parallelize()</code></p>
</blockquote>
<p>Thanks <a class="mention" href="/u/eqy">@eqy</a> !</p>

yunshiuan: <p>Thank you <a class="mention" href="/u/katerina_margatina">@Katerina_Margatina</a> ! I ran into the same problem and your solution works mysteriously like a charm!</p>
Image Colorization Tutorial - Using U-Net and GANs

Moein_Shariatnia: <p>I’ve done an in depth <strong>Tutorial</strong> on <strong>Image Colorization</strong> task using <strong>U-Net and Conditional GAN</strong> with <strong>PyTorch</strong>.<br>
I’ve written a blog post about it on <strong>TowardsDataScience</strong>: <a href="https://towardsdatascience.com/colorizing-black-white-images-with-u-net-and-conditional-gan-a-tutorial-81b2df111cd8?source=friends_link&amp;sk=e9d275985a6e00ada31e48ddc903fc9d" rel="noopener nofollow ugc">Link</a><br>
Also, all the project as a notebook along with the blog post explanations are available on my <strong>GitHub</strong> repo: <a href="https://github.com/moein-shariatnia/Deep-Learning/tree/main/Image%20Colorization%20Tutorial" rel="noopener nofollow ugc">Link</a></p>
<p>You can open the whole project directly on <strong>Google Colab</strong> and using the pretrianed weights, start colorizing your black and white images and learn a lot about the task and how it get solved using deep learning.<br>
<a href="https://colab.research.google.com/github/moein-shariatnia/Deep-Learning/blob/main/Image%20Colorization%20Tutorial/Image%20Colorization%20with%20U-Net%20and%20GAN%20Tutorial.ipynb" rel="noopener nofollow ugc"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" width="117" height="20"></a></p>
<p>Please star the GitHub repo if you liked the tutorial and the project. Thanks!</p>

ptrblck: <p>Cool, thanks for sharing it here! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
CPU memory keeps increasing when load data

hou9612: <p>I’m doing a video vlassification task, to load frames of videos faster, I first write frames to hdf5 files, the core code is like this:</p>
<pre><code class="lang-auto">frame_path = 'frame_path'
with frame_path.open('rb') as fi:
    data = fi.read() 
    np.frombuffer(data, dtype='uint8')
    h5_file[i]= np.frombuffer(data, dtype='uint8') # h5_file is the h5py file that stores frames.
</code></pre>
<p>Then in the dataloader, I try to read frames like this:</p>
<pre><code class="lang-auto">import cv2 as cv
....
data = h5_file['rgb'][i]
data = cv.imdecode(data,cv.IMREAD_COLOR)  
data = cv.cvtColor(data, cv.COLOR_BGR2RGB)
</code></pre>
<p>or</p>
<pre><code class="lang-auto">from PIL import Image
...
with Image.open(io.BytesIO(data),'r') as data:
    data = np.array(data)
</code></pre>
<p>For both of these two methods (use opencv or Pillow):<br>
when the python version is 3.8.8, the CPU memory keeps increasing<br>
when the python version is 3.9.16, the CPU memory will not keep incereasing when do not use distributeddataparallel (single processing), but still keeps increasing when use distributeddataparallel (multiprocessing).</p>

ptrblck: <p>I don’t know which <code>libhdf5</code> version you are using but based on <a href="https://github.com/h5py/h5py/issues/2010">this issue</a> it might be a known issue in <code>1.21.0</code>.</p>

hou9612: <p>It seems that the suggestion in the issue you metioned solves my problem, thanks very much!!!</p>
Changing where optim.step() writes to

Christopher_Brissett: <p>Hello!</p>
<p>I am working on a niche project that requires storage of network weights for k iterations. It is easy enough to get the weights at a given iteration in PyTorch, however there is an issue. It seems as though optim.step() updates the weights of PyTorch tensors in place, meaning an expensive copy has to be performed for each of the k iterations. This drastically alters the runtime of the network.</p>
<p>To cut down on this, I would like to pre-allocate a block of memory on device with size k*|W|, and have optim.step() update a chunk of this at each iteration. To do this efficiently I would like to stride the output pointer of optim.step() along this block of memory, and write to each chunk of size |W|.</p>
<p>Is something like this possible in PyTorch?</p>
<p>Thanks for taking the time to think about this!</p>
Pytorch older version for torch.rfft and torch.irfft

asifkhan: <p>I am currently using PyTorch version 1.11.0 with CUDA version 11.3. I am trying to integrate a Wiener filter (the new version supports torch.fft.rfft and torch.fft.irfft) into my deep learning model, but the code for the filter is written in an older version of PyTorch that supports <code>torch.rfft</code> and <code>torch.irfft</code>. To use this code, I downgraded my PyTorch and CUDA versions to 1.6.0 and 10.2, respectively.</p>
<p>However, when I run the code or try to debug it, the execution gets stuck and does not proceed. I am looking for help to identify and resolve the issue. Any assistance would be greatly appreciated. Thank you.</p>

albanD: <p>Hi,</p>
<p>These functions were simply migrated to <code>torch.fft.*</code> namespace.<br>
If you can update the library code to use these new <code>torch.fft.rfft()</code> function that should work.<br>
If you can’t, you can also add the function back by hand by doing something like <code>torch.rfft = torch.fft.rfft</code> as a workaround.</p>

asifkhan: <p>Thank you for the suggestion. I will attempt to follow your advice. <img src="https://discuss.pytorch.org/images/emoji/apple/+1.png?v=12" title=":+1:" class="emoji" alt=":+1:" loading="lazy" width="20" height="20"></p>
Algorithm Suggestions

NHarris: <p>Hello,</p>
<p>I am attempting to apply RL to networking which would have a continuous state space, consisting of dynamic network statistics, and discrete action space, a list of integers, 0-9, to be chosen from; any input or suggestions for algorithms to apply as I’m seeing most algorithms only deal with both being continuous or discrete, not a combination of the two?</p>

J_Johnson: <p>You can structure a model to give two outputs, one discrete and one continuous. Then run the appropriate loss functions on both and add the two losses before backprop.</p>
<p>It might look something like:</p>
<pre><code class="lang-auto">...
forward(self, x):
    x = layer1(x)
    x = layer2(x)
    ...
    y = x.clone()
    x = cont_layer(x)
    y = disc_layer(y)
    return x, y
...

loss1 = criterion1(x)
loss2 = criterion2(y)

total_loss = loss1 + loss2
...
</code></pre>
<p>But, keep in mind, some loss functions may need to be scaled by multiplying by a constant before adding. For example, l1 or l2 loss maybe give you very large numbers when compared to cross entropy. So multiplying by some constant may help the model learn better from both and not only focus on optimizing for one.</p>
How to plot latent space of my VAE?

Mario_Tantillo: <p>hi!<br>
I have a VAE . From my model i got the output of encoder , and it is a tensor like this [32,192,4,4] (32 is batchsize(the last 32 imgs from dataset), 192 is nchannels after conv, 4 and 4 pixels dim).<br>
How i can plot using PCA or TSNE ?<br>
Help me please</p>

sat: <p>Hi! I’m not super sure by what you mean by ‘plot the latent space’. Could you clarify what you mean?</p>
<aside class="quote no-group" data-username="Mario_Tantillo" data-post="1" data-topic="157258">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mario_tantillo/48/46521_2.png" class="avatar"> Mario_Tantillo:</div>
<blockquote>
<p>a tensor like this [32,192,4,4] (32 is batchsize(the last 32 imgs from dataset), 192 is nchannels after conv, 4 and 4 pixels dim).</p>
</blockquote>
</aside>
<p>Generally, the latent space of a VAE is represented as some probability distribution (e.g. a vector representing each the mean and variance of each latent dimension). If I unerstand your case correctly, the output of the final convolutional layer should be flattened into a vector and passed through one or more fully connected layers to produce the mean and variance of the probability distribution in the latent space. Then of course you can plot the distribution of your latent variables.</p>
<p>Another way of exploring the data manifold/latent space visually is to sample from the latent space in regular intervals and examine the corresponding outputs that the decoder produces (for example, similar to figure 4 in <a href="https://arxiv.org/abs/1312.6114" rel="noopener nofollow ugc">Kingma and Welling (2013)</a>)</p>
<p>Hope this helps!</p>
DDP sometimes "disables" my keyboard?

seankala: <p>I’m not exactly sure how to put this. Sometimes when I run DDP and the process terminates prematurely, I’d notice that I can’t type anything into my console anymore. I usually kill the Tmux session and start a new one.</p>
<p>What is this phenomenon called and why does it happen? Thanks.</p>

Jamie_Donnelly: <p>Are you running DDP locally and it’s temporarily using up ~all available resources before crashing? I’ve crashed my laptop a few times in a similar manner trying to be clever with MPI.</p>
Single attention head raises error in evaluation procedure of transformer

gzx: <p><strong>RuntimeError: Only support when num_heads is even in transformer</strong></p>
<p>But there is only one head in my transformer layer<br>
I met the this error when i am using v2.0. Unfortunately, my system strictly needs num_heads = 1</p>
<ul>
<li>It is strange, because when nn.Module.training is set to True(i’m training my model), <strong>no error occurred</strong>.</li>
<li>When I apply .eval() for <strong>non-dropout</strong> model evaluation, error returns : <em>RuntimeError: Only support when num_heads is even in transformer</em>.</li>
<li>More strangely, when I abort padding_mask input, this error disappears and model can work. However, for attention model, I really need mask!</li>
</ul>
<pre><code class="lang-python">#this would avoid error, but is not a wise choice.
if self.training:
    encode_X = self.transformer_encoder(X,src_key_padding_mask=mask)
else:
    encode_X = self.transformer_encoder(X)
return encode_X
</code></pre>
<p>My current torch version is latest 2.0, i found similar error report for 1.12 (maybe training has been repaired but eval hasn’t , I guess)</p>
Training loop getting slower and slower

EvanZ: <p>I have a very basic training loop that is getting much slower over time. According to some other help I saw online I added <code>torch.cuda.empty_cache()</code> but that is not fixing it. Any suggestions?</p>
<pre><code class="lang-auto">device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')
print(f'Device used for training: {device}')
model = DistilBertForSequenceClassification.from_pretrained('distilbert-base-uncased', num_labels=1)
model = torch.nn.DataParallel(model)
model.to(device)
model.train()

optim = AdamW(model.parameters(), lr=5e-5)

batch_iter = 0
training_loss = 0
num_batches = len(train_dataloader)
sns.set_style('darkgrid')
for epoch in range(3):
    for batch in tqdm(train_dataloader):
        batch_iter += 1
        optim.zero_grad()
        input_ids = batch['input_ids'].to(device)
        attention_mask = batch['attention_mask'].to(device)
        labels = torch.logit(batch['label']).to(device)
        outputs = model(input_ids=input_ids, attention_mask=attention_mask,
                        labels=labels)
        loss = outputs[0]
        loss = loss.mean()
        training_loss += loss.item()/(num_batches/100)
        if batch_iter % round(num_batches/100) == 0:
            writer.add_scalar('Loss/train',training_loss,batch_iter)
            training_loss = 0
        loss.backward()
        optim.step()
        torch.cuda.empty_cache()
    model.module.save_pretrained(f"checkpoints/model_epoch_{epoch}")
</code></pre>

ptrblck: <aside class="quote no-group" data-username="EvanZ" data-post="1" data-topic="110982">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/evanz/48/372_2.png" class="avatar"> EvanZ:</div>
<blockquote>
<p>According to some other help I saw online I added <code>torch.cuda.empty_cache()</code> but that is not fixing it.</p>
</blockquote>
</aside>
<p>Calling <code>empty_cache()</code> could potentially decrease the performance, since the cached memory will be freed and is thus not reusable. PyTorch would need to use synchronizing <code>cudaMalloc</code> operations in order to allocate new memory, which is the reason for the potential slowdown.</p>
<p>If you are not using other processes, which need to use GPU memory, I would not recommend to use <code>empty_cache</code> in your code.</p>
<p>That being said, if you are seeing a significant slowdown, you might (accidentally) store some tensors, which are still attached to the computation graph or your systems might be throttled e.g. because the cooling isn’t sufficient for the current workload (you could check the clocks of your system).</p>

EvanZ: <p>I posted the entirety of the training loop above. I don’t think I’m accidentally storing any tensors, but please let me know if I am!</p>

Ogbogu_Kalu: <p><a class="mention" href="/u/evanz">@EvanZ</a><br>
Hi. did you ever resolve this ? facing a similar issue</p>

MGHSTAR: <p>I set pin_memory=True and solved my problem.</p>
How to reduce the times of object loading in pytorch distributed data parrallel?

Bebo: <p>I am working on a PyTorch project built on <a href="https://mmdetection.readthedocs.io/en/latest/" rel="noopener nofollow ugc">mmdetection</a>. In this project, the ground truths are generated from a huge file that should be loaded into memory before the training process. Illustrate in the following code:</p>
<p>In <a href="https://github.com/open-mmlab/mmdetection/blob/ecac3a77becc63f23d9f6980b2a36f86acd00a8a/tools/train.py#L129" rel="noopener nofollow ugc">tools/train.py</a></p>
<pre><code class="lang-python">from annotation_handler import preload_annotaiton

# ...

assert os.path.exists(cfg.data.annotation_file)
preload_annotation(cfg.data.annotation_file)

runner.trian()
</code></pre>
<p>In <code>annotation_handler.py</code></p>
<pre><code class="lang-python">
ANNOS = dict()

def preload_annotation(path):
    ANNOS.update({path: load(path)})

def get_annotaion(query, anno_path):
    global ANNOS
    if anno_path not in ANNOS:
        ANNOS.update({path: load(path)})  # NOTE: the laod() is costly in time
    
    gts = generate_proper_groundtruth(query, ANNOS[anno_path])
    return gts
</code></pre>
<p>In dataset</p>
<pre><code class="lang-python">from annotation_handler import get_annotation

class MyDataset(Dataset):
    def __getitem__(self, idx):
        source = load_source(idx)
        query = querys[idx]
        target = get_annotation(query, self.anno_path)
        return source, target
</code></pre>
<p>My implementation expects to load the large file only once. However, I found the large annotation reloaded at each epoch’s beginning when using distributed training. And the reload time is the same as <code>the number of workers</code> * <code>the number of GPUs</code>. The beginning of each epoch, the data loading time increase dramatically.</p>
<ol>
<li>How to ensure the large annotation file is only loaded at the beginning.</li>
<li>How to share the large annotation across multiple subprocesses in <code>torch.multiprocessing</code>?</li>
</ol>
nn.BatchNorm2d forward pass calculate in training mode

pepper8362: <p>I try to re-produce the forward pass of <code>nn.BatchNorm2d</code> in training mode as follows:</p>
<pre><code class="lang-auto">bn = nn.BatchNorm2d(16)
data = 2.2+torch.randn(32,16,10,10).float()      
out_bn = bn(data)
my_out_bn = (data - data.mean([0,2,3],keepdim=True)) / torch.sqrt(data.var([0,2,3],keepdim=True) + bn.eps)
my_out_bn = bn.weight.reshape(1,-1,1,1) * my_out_bn + bn.bias.reshape(1,-1,1,1)
print((out_bn - my_out_bn).abs().max())
</code></pre>
<p>I find the maximum absolute error could go as high as 7e-4, as it seems not reduce with double precision. Is there anyhting wrong in my own implementation? Thanks!</p>

ptrblck: <p>Use the biased variance <code>data.var([0,2,3], unbiased=False, keepdim=True)</code> as explained in the <a href="https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm2d.html#torch.nn.BatchNorm2d">docs</a> and you should see a lower error.</p>

pepper8362: <p>Thanks, the error is indeed reduced. Just a follow up question, does the layer use unbiased variance to accumulate <code>running_var</code>?</p>

ptrblck: <p>Yes, the unbiased variance is used to update the <code>running_var</code>.<br>
A manual implementation can be seen <a href="https://github.com/ptrblck/pytorch_misc/blob/master/batch_norm_manual.py">here</a> and should match PyTorch’s layer.</p>
Warnings about torch.compile, dose it matter?

ToughMind: <p>I reciver warnings as below:</p>
<pre><code class="lang-auto">[2023-04-12 20:39:39,709] torch._inductor.sizevars: [WARNING] failed on: -524288*ModularIndexing(d1, 2, 8) - 128*ModularIndexing(64*d1, 1, 128) + 128*ModularIndexing(8192*d1 + 1, 128, 128) + 524288*ModularIndexing(8192*d1 + 1, 16384, 8) + 1
[2023-04-12 20:39:39,791] torch._inductor.sizevars: [WARNING] failed on: -524288*(q2//16384) + 524288*(((q2 + 8192)//16384)) - 128*ModularIndexing(q2, 128, 128) + 128*ModularIndexing(q2 + 8192, 128, 128)
[2023-04-12 20:39:39,797] torch._inductor.sizevars: [WARNING] failed on: -524288*(q1//2) + 524288*(((8192*q1 + 1)//16384)) - 128*ModularIndexing(64*q1, 1, 128) + 128*ModularIndexing(8192*q1 + 1, 128, 128) + 1
[2023-04-12 20:39:40,039] torch._inductor.sizevars: [WARNING] failed on: -262144*ModularIndexing(2*d1, 1, 8) + 262144*ModularIndexing(8192*d1 + 1, 4096, 8) + 1
[2023-04-12 20:39:41,253] torch._inductor.sizevars: [WARNING] failed on: -262144*ModularIndexing(2*d1, 1, 8) + 262144*ModularIndexing(8192*d1 + 1, 4096, 8) + 1
[2023-04-12 20:39:44,305] torch._inductor.sizevars: [WARNING] failed on: -262144*ModularIndexing(2*d1, 1, 8) + 262144*ModularIndexing(8192*d1 + 1, 4096, 8) + 1
[2023-04-12 20:39:44,811] torch._inductor.sizevars: [WARNING] failed on: -262144*ModularIndexing(2*d1, 1, 8) + 262144*ModularIndexing(8192*d1 + 1, 4096, 8) + 1
[2023-04-12 20:39:44,907] torch._inductor.sizevars: [WARNING] failed on: -524288*ModularIndexing(d1, 2, 8) - 128*ModularIndexing(64*d1, 1, 128) + 128*ModularIndexing(8192*d1 + 1, 128, 128) + 524288*ModularIndexing(8192*d1 + 1, 16384, 8) + 1
[2023-04-12 20:39:44,924] torch._inductor.sizevars: [WARNING] failed on: -524288*(q2//16384) + 524288*(((q2 + 8192)//16384)) - 128*ModularIndexing(q2, 128, 128) + 128*ModularIndexing(q2 + 8192, 128, 128)
[2023-04-12 20:39:44,926] torch._inductor.sizevars: [WARNING] failed on: -524288*(q1//2) + 524288*(((8192*q1 + 1)//16384)) - 128*ModularIndexing(64*q1, 1, 128) + 128*ModularIndexing(8192*q1 + 1, 128, 128) + 1
[2023-04-12 20:39:46,479] torch._inductor.sizevars: [WARNING] failed on: -262144*ModularIndexing(2*d1, 1, 8) + 262144*ModularIndexing(8192*d1 + 1, 4096, 8) + 1
</code></pre>
<p>Dose those warnings mean that there are errors I need to care?</p>

marksaroufim: <p>Are you on the official release as in 2.0? We had a bug where we didn’t disable developer warnings there so those iissues shouldn’t be seen in future releases</p>

ToughMind: <p>I am on the <code>2.1.0.dev20230409</code> version. Thanks for your answer!</p>

marksaroufim: <p>So dev version will print these warnings but they shouldn’t a major concern to you</p>
Backend gloo started a random port number for distributed communication

Christina4Jojo: <p>hi,i got a problem here<br>
when i use pytorch to  start a distributed job<br>
i use parameters below for <strong>init_process_group</strong> method</p>
<blockquote>
<h1>os.environ[‘WORLD_SIZE’] = “2”</h1>
<h1>os.environ[‘RENDEZVOUS’] = “tcp://192.168.118.***:2346”</h1>
<h1>os.environ[‘RANK’] = “1”</h1>
<h1>os.environ[‘DISTRIBUTED_BACKEND’] = “gloo”</h1>
<h1>os.environ[‘GLOO_SOCKET_IFNAME’] = “ens192”</h1>
<h1>os.environ[‘TP_SOCKET_IFNAME’] = “ens192”</h1>
<h1>os.environ[‘GLOO_SOCKET_TCP_PORT’] = “20000”</h1>
</blockquote>
<p>it works normally and i can get the result what i what</p>
<p>but when i started my firewall service</p>
<p>it reported an error message like below</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/f/bf8fad263eb1d367aca2033fe08d278bc7f72c49.png" data-download-href="https://discuss.pytorch.org/uploads/default/bf8fad263eb1d367aca2033fe08d278bc7f72c49" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/f/bf8fad263eb1d367aca2033fe08d278bc7f72c49_2_690x201.png" alt="image" data-base62-sha1="rkD1mKAz9YHRc3a0wPTWgFEn8qR" width="690" height="201" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/f/bf8fad263eb1d367aca2033fe08d278bc7f72c49_2_690x201.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/f/bf8fad263eb1d367aca2033fe08d278bc7f72c49_2_1035x301.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/f/bf8fad263eb1d367aca2033fe08d278bc7f72c49_2_1380x402.png 2x" data-dominant-color="302F33"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1671×489 71.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>looks like a random port was generated and connection was refused</p>
<p>how can i choose a certain port instead a random one so i can allow it in my firewall iptables</p>
<p>i have tried share file-system initialization instead of TCP initialization hoped it will use a certaion port to do the communication work</p>
<blockquote>
<p>os.environ[‘WORLD_SIZE’] = “2”<br>
os.environ[‘RANK’] = “1”<br>
os.environ[‘RENDEZVOUS’] = "file:///opt/share_with/party_share_12345</p>
</blockquote>
<p>but the error was still there</p>
<p>i wanna ask if i can choose a certain port when i use  gloo backend<br>
or if mpi backend has this problem also<br>
nccl is out of my consideration because i have not GPU in my server</p>
<p>it will be very apprecaite if you can offer any help or message.</p>
TransformerEncoder with src_key_padding_mask on eval can produce different-sized outputs

alper: <p><code>TransformerEncoder</code> produces different-sized outputs on eval and train modes, while a single <code>TransformerEncoderLayer</code> or <code>MultiheadAttention</code> produce the same-sized tensor on both modes. Namely, if <code>src_key_padding_mask</code> is redundant (if the sequence length of all samples in the batch is shorter than the length of the sequence dimension, i.e., maximum possible length), the output is reduced to the maximum sized sequence length in the batch, instead of maximum possible length. Below is a sample code:</p>
<pre><code class="lang-auto">import torch

mh = torch.nn.MultiheadAttention(128, 8, batch_first=True)
layer = torch.nn.TransformerEncoderLayer(128, 8, batch_first=True)
encoder = torch.nn.TransformerEncoder(layer, 2)

# (batch, seq_len, embed_dim)
x = torch.randn(7, 5, 128)
attn_mask = torch.tensor([[False, False, False, True, True],
                          [False, False, False, True, True],
                          [False, False, True, True, True],
                          [False, False, True, True, True],
                          [False, False, False, True, True],
                          [False, False, True, True, True],
                          [False, False, False, True, True]], dtype=torch.bool)
# the tensor could have been (7, 3, 128), but for some reason, it is (7, 5, 128)

mh_train_w_grad, _ = mh(x, x, x, key_padding_mask=attn_mask)
enc_train_w_grad = encoder(x, src_key_padding_mask=attn_mask)

with torch.no_grad():
    mh_train_wo_grad, _ = mh(x, x, x, key_padding_mask=attn_mask)
    enc_train_wo_grad = encoder(x, src_key_padding_mask=attn_mask)

mh.eval()
encoder.eval()

mh_eval_w_grad, _ = mh(x, x, x, key_padding_mask=attn_mask)
enc_eval_w_grad = encoder(x, src_key_padding_mask=attn_mask)

with torch.no_grad():
    mh_eval_wo_grad, _ = mh(x, x, x, key_padding_mask=attn_mask)
    enc_eval_wo_grad = encoder(x, src_key_padding_mask=attn_mask)
print(f"Train mode multi-head attention: w/grad {mh_train_w_grad.shape} w/o {mh_train_wo_grad.shape}")
print(f"Train mode transformer encoder: w/grad {enc_train_w_grad.shape} w/o {enc_train_wo_grad.shape}")
print(f"Eval mode multi-head attention: w/grad {mh_eval_w_grad.shape} w/o {mh_eval_wo_grad.shape}")
print(f"Eval mode transformer encoder: w/grad {enc_eval_w_grad.shape} w/o {enc_eval_wo_grad.shape}")
</code></pre>
<p>Is this an expected behavior? Of course, I could have ensured the sizes before supplying the input, but this was not the behavior I was actually expecting, especially since they behave differently on train and eval modes. I will open an issue if this is indeed an unexpected behavior. Is there a catch I am missing?</p>
<p>ps: This is the case for <code>torch==2.0.0</code> and <code>torch==1.13.0</code> on macos. When I try the same script with <code>torch==1.12.0</code> it produces the output that <em>I expect</em>.</p>

ptrblck: <p>I guess the discrepancy might be caused by the usage of the <code>fast_path</code> implementation.<br>
CC <a class="mention" href="/u/mikekgfb">@mikekgfb</a> as one of the code owners.</p>
Effect of Changing batchsize (N) on Pre-trained Inception-V3 Model Predictions in torchvision

a2108870: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/4/d4bd211208d2219fb55fdd59c904be409c049d57.png" data-download-href="https://discuss.pytorch.org/uploads/default/d4bd211208d2219fb55fdd59c904be409c049d57" title="屏幕截图 2023-04-12 113825"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/4/d4bd211208d2219fb55fdd59c904be409c049d57_2_690x200.png" alt="屏幕截图 2023-04-12 113825" data-base62-sha1="ulYqJiapmHqy26iKXIqkC7eK8d1" width="690" height="200" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/4/d4bd211208d2219fb55fdd59c904be409c049d57_2_690x200.png, https://discuss.pytorch.org/uploads/default/original/3X/d/4/d4bd211208d2219fb55fdd59c904be409c049d57.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/d/4/d4bd211208d2219fb55fdd59c904be409c049d57.png 2x" data-dominant-color="F2F2F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">屏幕截图 2023-04-12 113825</span><span class="informations">1014×295 44.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I’m using the pre-trained Inception-V3 model from torchvision, and my input tensor to the model has a shape of N,3,299,299. However, I’ve noticed that changing the value of N in the input tensor affects the model’s predictions. I’ve already switched the model to evaluation mode before making predictions. For example, in the screenshot, when I set N to 4 and compare the predictions from the first row of the model’s output with when N is set to 1, they are not the same."</p>

ptrblck: <p>How large is the actual error as you might observe the expected small errors caused by the limited floating point precision?</p>

a2108870: <p>Thank you for your response. I have observed that when N changes, most of the predicted values from the model show some variation starting from the third decimal place. Could this be caused by floating-point precision? Both the model and the input tensor are of data type float32.</p>

ptrblck: <p>Yes, this might be indeed the case and <a href="https://discuss.pytorch.org/t/shuffling-the-input-before-the-model-and-shuffling-the-output-after-the-model-being-not-consistent-on-cuda/177103/5">this topic</a> discusses a similar effect.</p>
PyTorch 2.0 compile problem in mac

mohit8: <p>Hi I am trying to run the following code in my mac</p>
<pre><code class="lang-auto">import torch
import torchvision.models as models

model = models.resnet18()
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)
compiled_model = torch.compile(model)

x = torch.randn(16, 3, 224, 224)
optimizer.zero_grad()
out = compiled_model(x)
out.sum().backward()
optimizer.step()
</code></pre>
<p>It is throwing this error</p>
<p>BackendCompilerFailed: debug_wrapper raised CppCompileError: C++ compile error</p>
<p>Command:<br>
g++ /var/folders/0d/syd94gdj2fvgzt42r00zyh840000gn/T/torchinductor_mohit/eh/cehz4qmb25b7f22uecphqlnronbtybx2buc2d3lhquxtt7zs3d7j.cpp -shared -fPIC -Wall -std=c++17 -Wno-unused-variable -I/Users/mohit/opt/anaconda3/envs/pytorch_2/lib/python3.10/site-packages/torch/include -I/Users/mohit/opt/anaconda3/envs/pytorch_2/lib/python3.10/site-packages/torch/include/torch/csrc/api/include -I/Users/mohit/opt/anaconda3/envs/pytorch_2/lib/python3.10/site-packages/torch/include/TH -I/Users/mohit/opt/anaconda3/envs/pytorch_2/lib/python3.10/site-packages/torch/include/THC -I/Users/mohit/opt/anaconda3/envs/pytorch_2/include/python3.10 -lomp -O3 -ffast-math -fno-finite-math-only -Xclang -fopenmp -D C10_USING_CUSTOM_GENERATED_MACROS -o/var/folders/0d/syd94gdj2fvgzt42r00zyh840000gn/T/torchinductor_mohit/eh/cehz4qmb25b7f22uecphqlnronbtybx2buc2d3lhquxtt7zs3d7j.so</p>
<p>Output:<br>
In file included from /var/folders/0d/syd94gdj2fvgzt42r00zyh840000gn/T/torchinductor_mohit/eh/cehz4qmb25b7f22uecphqlnronbtybx2buc2d3lhquxtt7zs3d7j.cpp:2:<br>
/var/folders/0d/syd94gdj2fvgzt42r00zyh840000gn/T/torchinductor_mohit/zt/cztcl2vp5yqlnhofzpqfficjcxgyict6e3xhfdd7sdbkipp4p44x.h:6:10: fatal error: ‘omp.h’ file not found<br>
<span class="hashtag">#include</span> &lt;omp.h&gt;<br>
^~~~~~~<br>
1 error generated.</p>
<p>Set torch._dynamo.config.verbose=True for more information</p>
<p>You can suppress this exception and fall back to eager by setting:<br>
torch._dynamo.config.suppress_errors = True</p>
<p>Please help me with this issue</p>

marksaroufim: <p>This was fixed in nightlies where now the default compiler defaults to aot_eager so you can fix your code if you run <code>torch.compile(m, backend="aot_eager")</code></p>