akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Memory use increased during training

goodboy-hub: <p>I met a problem about memory when i was training a model. Because the memory of my computer is small, so i monitor the memory use during training. I found that as i began to train, there will be a big increase in memory use. (about 4G increased), but the parameters were only ~0.5G(estimated by torchsummary). So , where does the extra 3.5G come from? I am sure it is not caused by the data loader, so it confused me a lot.</p>

goodboy-hub: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/a/9a764322b59325577a4309139a3268c0848f2942.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/9a764322b59325577a4309139a3268c0848f2942" title="屏幕截图 2023-02-22 225400"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/a/9a764322b59325577a4309139a3268c0848f2942_2_690x283.jpeg" alt="屏幕截图 2023-02-22 225400" data-base62-sha1="m2qUbennIPC4qn3ZOYwKM6SFenw" width="690" height="283" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/a/9a764322b59325577a4309139a3268c0848f2942_2_690x283.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/9/a/9a764322b59325577a4309139a3268c0848f2942.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/a/9a764322b59325577a4309139a3268c0848f2942.jpeg 2x" data-dominant-color="0A090A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">屏幕截图 2023-02-22 225400</span><span class="informations">997×409 56.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <aside class="quote no-group" data-username="goodboy-hub" data-post="1" data-topic="173248">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/goodboy-hub/48/57223_2.png" class="avatar"> goodboy-hub:</div>
<blockquote>
<p>So , where does the extra 3.5G come from?</p>
</blockquote>
</aside>
<p>I guess this memory increase could come from the intermediate forward activations which are needed for the gradient computation. You could take a look at <a href="https://discuss.pytorch.org/t/resnet-50-takes-10-13gb-to-run-with-batch-size-of-96/117402/2">this post</a> to get an estimation of the memory usage.</p>
Loading a file in chunks using `torch.load`

tanay_mehta: <p>I am working on a problem where I need to have my entire dataset as one big <code>.pt</code> file on the disk. But since that file might exceed the size of either my GPU or RAM, I want a possible way to load it in chunks using <code>torch.load</code>.</p>
<p>Is there a way to do that? If not, is it a good potential feature request?</p>

ptrblck: <p>You could store the tensor as a numpy array and use <code>numpy.memmap</code> to access chunks of the array.</p>
RuntimeError: Expected target size [4, 256], got [4, 256, 256, 1]

patatessever: <p>Hi everyone, I am aware that this sort of questions are already discussed but I couldnt make it work, so I have 3 channel 256x256 images and 7 classes for a semantic segmentation task.<br>
RuntimeError: Expected target size [4, 256], got [4, 256, 256, 1] I am getting this error write now.</p>
<pre><code class="lang-auto">class CustomImageDataset(Dataset):
    def __init__(self, mask, img, transform=None):
        self.mask = mask
        self.img = img
        self.transform = transform

    def __len__(self):
        return np.unique(mask).size

    def __getitem__(self, idx):
        image = self.img[idx]
        mask = torch.from_numpy(self.mask[idx])
        mask = mask.type(torch.LongTensor)

        if self.transform:
            image = self.transform()(image)
 
        return image, mask
</code></pre>
<p>Here is my dataset creator code. My inputs to this dataset creator is  (num_img, 256,256,3) and (num_masks, 256,256,1) .</p>
<p>Edit : I was also getting  “RuntimeError: only batches of spatial targets supported (3D tensors) but got targets of dimension: 4” this error and I handled it with argmax operation (<a href="https://discuss.pytorch.org/t/runtimeerror-only-batches-of-spatial-targets-supported-3d-tensors-but-got-targets-of-dimension-4/82098" class="inline-onebox">RuntimeError: only batches of spatial targets supported (3D tensors) but got targets of dimension: 4</a>) however it is causing 0 gradient error too. I feel quite lost…</p>

ptrblck: <aside class="quote no-group" data-username="patatessever" data-post="1" data-topic="173230">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/patatessever/48/57624_2.png" class="avatar"> patatessever:</div>
<blockquote>
<p>My inputs to this dataset creator is (num_img, 256,256,3) and (num_masks, 256,256,1) .</p>
</blockquote>
</aside>
<p>The input data shape looks wrong as PyTorch uses the channels-first memory layout by default. The inputs are thus expected to have the shape <code>[batch_size, channels, height, width]</code> and you might need tom <code>.permute</code> the input tensor.</p>
<p>In a multi-class segmentation use case using <code>nn.CrossEntropyLoss</code> the model outputs are expected to have the shape <code>[batch_size, nb_classes, height, width]</code> while the targets should have the shape <code>[batch_size, height, width]</code> containing class indices in the range <code>[0, nb_classes-1]</code>.</p>
Concatenated aranges from start and end tensors without loop

Noki_Lee: <p><code>starts</code> and <code>ends</code> have the same size.<br>
<code>starts</code> is variable.<br>
<code>ends</code> is variable, but each end is larger than each start.</p>
<pre><code class="lang-auto">starts = torch.tensor([0, 1, 2, 3])
ends = torch.tensor([2, 4, 6, 8])
aranges = []
for s, e in zip(starts, ends):
    aranges.append(torch.arange(s, e))
result = torch.cat(aranges, dim=0)
result 
</code></pre>
<p>Can we get the same <code>result</code> without loop?</p>

ptrblck: <p>I don’t think this would be possible since each intermediate tensor has a different shape, so appending them to a <code>list</code> sounds like a valid approach.<br>
At least I’m not aware of a loop-free approach, but let’s wait for others in case they can come up with an idea.</p>
Block_diag for tensors of more than 2 dimensions

Najwa_Laabid: <p>I am representing an adjacency matrix as a tensor of 4 dimensions as follows: (batch_size, n_nodes, n_nodes, n_edge_features). Say I have 2 adjacency matrices of subgraphs  <code>t1.shape = (1, 46, 46, 8)</code> and <code>t2.shape = (1, 4, 4, 8)</code>. I want to ‘merge’ the two graphs to get an adjacency matrix <code>t3 = (1, 50, 50, 8)</code>. In other words, I want to stack the two tensors on the diagonal (using smthg like block_diag) with off-diagonal entries = 0.</p>
<p>Unfortunately <code>torch.block_diag</code> only works for 2D tensors, and I am not sure how to use other concatenation/stacking function (e.g. <code>torch.stack</code>, <code>torch.cat</code>) to achieve this.</p>

Najwa_Laabid: <p>I came up with this, any improvements are welcome:</p>
<pre><code class="lang-auto">      # t1.shape = (bs, n_nodes, n_nodes, feature_size) = (1, 2, 2, 2)
      t1 = torch.Tensor([[[1, 1], [2, 2]], [[3, 3], [4, 4]]])\
                .unsqueeze(dim=0) 
      # t1.shape = (bs, n_nodes, n_nodes, feature_size) = (1, 3, 3, 2)
      t2 = torch.Tensor([[[5, 5], [6, 6], [7, 7]], 
                         [[8, 8], [9, 9], [10, 10]], 
                         [[11, 11], [12, 12], [13, 13]]])\
                .unsqueeze(dim=0)

      # t3.shape = (1, 5, 5, 2)
      t3 = torch.zeros(t1.shape[0], t1.shape[1]+t2.shape[1], 
                       t1.shape[2]+t2.shape[2], t1.shape[-1])
 
      t3[:,:t1.shape[1],:,:][:,:,:t1.shape[2],:] = t1
      t3[:,t1.shape[1]:t1.shape[1]+t2.shape[1],:,:]\
        [:,:,t1.shape[2]:t1.shape[2]+t1.shape[2],:] = t2

</code></pre>
Modify torch::jit::script::Module Parameters

astroesteban: <p>Hey y’all!</p>
<p>I am using LibTorch <strong>v1.12.1</strong> and I have been, unsuccessfully, trying to modify the parameters of a TorchScript module that I have exported from Python and then loaded into C++.</p>
<p>First what I have in one section of the app is:</p>
<pre><code class="lang-cpp">    auto m_module = torch::jit::load(modelPath);

    auto parameters = std::vector&lt;torch::Tensor&gt;{};

    for (const auto &amp;param : m_module.parameters()) {
       parameters.push_bach(param.detach().clone());
    }
</code></pre>
<p>The app then modifies the parameters in the std::vector in some way say:</p>
<pre><code class="lang-cpp">for (auto p : parameters) {
    p += 10;
}
</code></pre>
<p>Then in another section of the code I have:</p>
<pre><code class="lang-cpp">    auto idx= std::size_t{0};
    for (auto lhsParam : m_module.named_parameters()) {
        m_module.register_parameter(lhsParam.name, parameters.at(idx), true);
        ++i;
    }
</code></pre>
<p>It seems like the register_parameter function isn’t actually updating the parameters of the m_module.</p>
<p>Is there something I’m missing here? Or does the register_parameter function not work that way? Is there another way of updating the parameters of m_module?</p>
<p>Any tips would be appreciated.</p>
<p>Thanks!</p>

astroesteban: <p>Found a good solution:</p>
<pre><code class="lang-cpp">     // ! Disables autograd which makes parameter copying possible
    torch::autograd::GradMode::set_enabled(false);

    auto idx= std::size_t{0};
    for (auto lhsParam : m_module.named_parameters()) {
        lhsParam.value.copy_(parameters.at(idx));
        ++i;
    }
    
    torch::autograd::GradMode::set_enabled(true);
</code></pre>

Nick95: <p>Hi Esteban, did you also managed to extract weights from such jit module and import them into a torch module?</p>
About torch.qint32 support

PuNeal: <p>Why the upper_bound of qrange_len is set to 2**31 when dtype is torch.qint32?<br>
<a href="https://github.com/pytorch/pytorch/blob/2628901033e7df5b87a9b7137ddffb450d4a7c42/torch/ao/quantization/utils.py#L356" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/2628901033e7df5b87a9b7137ddffb450d4a7c42/torch/ao/quantization/utils.py#L356</a></p>
<p>This looks like mismatch with the following code:<br>
<a href="https://github.com/pytorch/pytorch/blob/2628901033e7df5b87a9b7137ddffb450d4a7c42/torch/ao/quantization/utils.py#L374" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/2628901033e7df5b87a9b7137ddffb450d4a7c42/torch/ao/quantization/utils.py#L374</a></p>

jcaip: <p>Hi PuNeal,</p>
<p>Thanks for flagging this! this looks like a typo and it should be 2**32.<br>
I’ll open a PR to fix this or if you want to contribute please feel free.</p>

PuNeal: <p>Hi Jesse,<br>
Thanks for your reply, but I don’t know how to contribute. Can you send me the commit link after you fix this bug? Thank you.</p>

jcaip: <p>Sure, no worries. I’ve opened a PR for this fix here: <a href="https://github.com/pytorch/pytorch/pull/95297" class="inline-onebox" rel="noopener nofollow ugc">[quant][bug fix] Fix qrange_len in `torch.ao.quantization.utils.py` by jcaip · Pull Request #95297 · pytorch/pytorch · GitHub</a> that should be merged soon.</p>
Missing Pytorch version

stanleygeorge: <p>Hi All<br>
I am unable to install Pytorch for version 1.12.1 + cuda 10.2 even though the previous versions webpage(<a href="https://pytorch.org/get-started/previous-versions/" class="inline-onebox" rel="noopener nofollow ugc">Previous PyTorch Versions | PyTorch</a>) says that I should be able to download it…</p>
<pre><code class="lang-auto">(xxx_venv) C:\Users\George\crack_inspect\patchcore-inspection&gt;pip install torch==1.12.1+cu102 torchvision==0.13.1+cu102 --extra-index-url https://download.pytorch.org/whl/cu102                    
Looking in indexes: https://pypi.org/simple, https://download.pytorch.org/whl/cu102
ERROR: Could not find a version that satisfies the requirement torch==1.12.1+cu102 (from versions: 1.11.0, 1.12.0, 1.12.1, 1.13.0, 1.13.1)
ERROR: No matching distribution found for torch==1.12.1+cu102

</code></pre>
<p>Any idea whats going wrong?</p>
How to get Kernel, Stride,Padding details from a model object?

p_Raj: <p>I want to print the Kernel ,Padding, And Stide only from a model object for Conv layers, Maxpool layers .</p>
<p>For example:</p>
<p>model = models.resnet18(pretrainded = True)</p>
<p>output:<br>
layer1 : kernel = 7,Stride= 2,Padding  = 3</p>

Kapil_Rana: <p>You can access anything using the argument name that you have used during model creation.<br>
For Example<br>
Model = net()<br>
model.conv1.kernal_size will give kernel size</p>
<p>so in your case it would be print(“layer1:”, model.conv1.kernel_size, model.conv1.stride, model.conv1.padding)</p>

p_Raj: <p>Thanks you.</p>
<p>My requirement here is Just to get the Kernel,Stride , and padding information without knowing the model archicature details. So that i dont have to mention the layer details.<br>
Looking for some iterative way to extract all the info  at one go.</p>

ptrblck: <p>You could use <a class="mention" href="/u/kapil_rana">@Kapil_Rana</a> 's approach and combine it with iterating all modules:</p>
<pre><code class="lang-python">for name, module in model.named_modules():
    if isinstance(module, nn.Conv2d):
        print(name, module.kernel_size)
        ...
</code></pre>

Kapil_Rana: <p>Than you can print whole model<br>
print(model)</p>
<p>And one more you can iterate over all model modules and print kernel, stride and padding only.</p>

Vasquez1: <p>It assists us with keeping a greater amount of the data at the boundary of a picture. Without cushioning, not many qualities at the following layer would be influenced by pixels as the edges of a picture.</p>

zzeitt: <p>I just print the module:</p>
<pre><code class="lang-python">print(self.conv1)
</code></pre>
<p>and it works…somthing like followings are printed:</p>
<pre><code class="lang-python">self.conv1=Conv2d(25, 25, kernel_size=(4, 4), stride=(1, 1), padding=(2, 2), bias=False)
</code></pre>
<p>Thanks pytorch!</p>
Onnx export format

PuNeal: <p>Hi, I have a question about the format when exporting model to onnx.<br>
Pytorch version: 1.13.1<br>
I use the example at [<a href="https://pytorch.org/docs/master/onnx.html#example-alexnet-from-pytorch-to-onnx" rel="noopener nofollow ugc">torch.onnx — PyTorch master documentation</a>](<a href="https://pytorch.org/docs/master/onnx.html#example-alexnet-from-pytorch-to-onnx" rel="noopener nofollow ugc">torch.onnx — PyTorch master documentation</a>)(I use resnet18 instead of alexnet). Then I visualize the resnet18.onnx file. Besides, I test this example using torch==1.10.0 and get the following result:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/6/76ffae97f0356e680f1c31f66aaa5dc55d6744f8.png" data-download-href="https://discuss.pytorch.org/uploads/default/76ffae97f0356e680f1c31f66aaa5dc55d6744f8" title="resnet18_compare"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/6/76ffae97f0356e680f1c31f66aaa5dc55d6744f8_2_520x500.png" alt="resnet18_compare" data-base62-sha1="gYI8xg088FDMS633uLDp7LzTL5K" width="520" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/6/76ffae97f0356e680f1c31f66aaa5dc55d6744f8_2_520x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/6/76ffae97f0356e680f1c31f66aaa5dc55d6744f8_2_780x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/7/6/76ffae97f0356e680f1c31f66aaa5dc55d6744f8.png 2x" data-dominant-color="E1E1E3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">resnet18_compare</span><span class="informations">803×771 27.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
The left is the result of torch 1.13.1 and the right is torch 1.10.0.<br>
This looks so strange, why the bias of Conv2d is not contained in Conv node?<br>
It looks like that the result when using torch 1.10.0 is normal. What is the difference between torch1.10.0 and 1.13.1?</p>
NeighborLoader with time_attr

abhi_khoyani: <p>I’m initializing a graph <code>data</code> with <code>time_attr</code> set and wants a <code>NeighborLoader</code> that return all the nodes that has smaller <code>time_attr</code> than the sampled node and below is my code.</p>
<pre><code class="lang-auto">import torch
import numpy as np
from torch_geometric.data import Data

data = Data()
data.node_id = torch.arange(10)
data.x = torch.tensor([[x, x**2] for x in range(10)])
temp = np.tril(np.ones((data.num_nodes, data.num_nodes)), k = -1)
u, v = np.where(temp!=0)
data.edge_index = torch.tensor(np.array([u,v]), dtype = torch.long)  #setting edges so that it'll be following time constraint. but in future I want to remove this to all connection between all the nodes
data.time_attr = torch.arange(10)


from torch_geometric.loader import NeighborLoader
loader = NeighborLoader(data, num_neighbors=[10], batch_size=1,
                        time_attr='time_attr', shuffle = False)
for batch in loader:
     print(b.x, b.edge_index, b.input_id)
&gt;&gt;tensor([[0, 0]]) tensor([], size=(2, 0), dtype=torch.int64) tensor([0])
tensor([[1, 1]]) tensor([], size=(2, 0), dtype=torch.int64) tensor([1])
tensor([[2, 4]]) tensor([], size=(2, 0), dtype=torch.int64) tensor([2])
tensor([[3, 9]]) tensor([], size=(2, 0), dtype=torch.int64) tensor([3])
tensor([[ 4, 16]]) tensor([], size=(2, 0), dtype=torch.int64) tensor([4])
tensor([[ 5, 25]]) tensor([], size=(2, 0), dtype=torch.int64) tensor([5])
tensor([[ 6, 36]]) tensor([], size=(2, 0), dtype=torch.int64) tensor([6])
tensor([[ 7, 49]]) tensor([], size=(2, 0), dtype=torch.int64) tensor([7])
tensor([[ 8, 64]]) tensor([], size=(2, 0), dtype=torch.int64) tensor([8])
tensor([[ 9, 81]]) tensor([], size=(2, 0), dtype=torch.int64) tensor([9])

</code></pre>
<p>can anybody help me how to accomplish this as my code is not giving proper output or what I’m missing!!?</p>
<p>And another thing is there’s two sampler in docs:</p>
<ol>
<li><code>torch_geometric.sampler.NeighborSampler</code></li>
<li>
<code>torch_geometric.loader.NeighborSampler</code>  so what is the difference between these two and why there’s mentioned that 2nd <code>NeighborSampler</code> will be deprecated, use <code>NeighborLoader</code>?</li>
</ol>
How to double layer GAT

aleksandrpikul134: <p>I try to double layer GAT for this code:</p>
<pre><code class="lang-auto">class AntispoofModel(nn.Module):
    def __init__(self, device="cpu", **kwargs):
        super().__init__()
        resnet = torch.hub.load('pytorch/vision:v0.10.0', 'resnet18', pretrained=True)
        self.resnet = nn.Sequential(*[i for i in list(resnet.children())[:-2]]).to(device)
        for ch in self.resnet.children():
            for param in ch.parameters():
                param.requires_grad = False
        self.gat = GAT(**kwargs).to(device)
        self.device = device
        self.adj = torch.tensor(grid_to_graph(3, 3, return_as=np.ndarray)).to(device)
        
    def forward(self, x):
        x = self.resnet(x.to(self.device))
        x = nn.functional.avg_pool2d(x, 2)
        x = x.view(-1, 9, 512)
        #adj = torch.stack([self.adj for i in range(x.shape[0])]).to(self.device)
        x = self.gat(x, self.adj)
        return torch.sigmoid(x)
</code></pre>
<p>As I understand I just need to add line <code>x = self.gat(x, self.adj)</code> somewhere else. But I don’t understand where (start/middle/end). Can someone help, please?<br>
I use ’ <a href="https://github.com/Diego999/pyGAT" class="inline-onebox" rel="noopener nofollow ugc">GitHub - Diego999/pyGAT: Pytorch implementation of the Graph Attention Network model by Veličković et. al (2017, https://arxiv.org/abs/1710.10903)</a>’</p>
Intermediate Fusion

Ajay_krishan_gairola: <p>What is CNN intermediate fusion? How can I develop intermediate fusion in a pretrained AlexNet model in PyTourch?</p>

ptrblck: <p>I don’t know what “intermediate fusion” refers to, but in case you are interested in fusing multiple operations into one, you might want to script the model via <code>torch.jit.script</code> or use the newly introduced <code>torch.compile</code> operation in the nightly binaries.</p>

Ajay_krishan_gairola: <p>Thank you for your reply, ptrblck. Feature-level fusion, also known as intermediate fusion. In intermediate fusion, the features, respective to each modality, are concatenated before classification; and intermediate fusion protocols operate on the feature map.</p>
<p>So, I’m looking for feature-level fusion of two CNN models (like Alexnet + Googlenet).</p>

Ajay_krishan_gairola: <p>class MyEnsemble(nn.Module):<br>
def <strong>init</strong>(self, modelA, modelB, nb_classes=4):<br>
super(MyEnsemble, self).<strong>init</strong>()<br>
self.modelA = modelA<br>
self.modelB = modelB<br>
# Remove last linear layer</p>
<pre><code>    self.modelA.classifier = nn.Identity() #densenet201
    self.modelB.fc= nn.Identity() #resnet101
    
    # Create new classifier

    self.classifier = nn.Linear(1920+2048, nb_classes)
    #  mat1 and mat2 shapes cannot be multiplied (16x514 and 512x4096)
</code></pre>
<p>========<br>
As in the above code, Removed last linear layer and Create new classifier. Now I want to remove middle layer and want to Create new classifier in this code???how??</p>

ptrblck: <p>It depends on the model and might not be possible as explained in your <a href="https://discuss.pytorch.org/t/how-to-delete-layer-in-pretrained-model/17648/63">cross-post</a>.</p>
Custom augmentation for time series: significant increase in training time

witl: <p>Hello everyone,</p>
<p>I’m currently working on a project that involves training a neural network on time series data. To improve the network’s performance, I’m using custom data augmentation techniques that modify the time series in various ways.</p>
<p>While the augmentations have improved the network’s performance, I’m noticing a significant increase in training time when using this augmentation technique. Is there a more efficient way to run this function on the GPU, or is there a better way to implement this augmentation technique altogether? Moreover, is there a way to use torchvision transforms on time series? I’d appreciate any advice or guidance on this issue.</p>
<p>I’ll share the simpler augmentation I implemented and how I’m applying it on my <code>__getitem__</code> function within my custom dataset. While using <code>random_crop_resize</code> below, my mean epoch time goes from 0.2 to 0.31 seconds while using a batch size of 256.</p>
<p><strong>Custom augmentation</strong></p>
<pre><code class="lang-auto">def random_crop_resize(o: torch.tensor, size: int, scale: tuple = (0.1, 1.0), mode: str = 'linear'):
    seq_len = o.shape[-1]
    lambd = np.random.uniform(scale[0], scale[1])
    win_len = int(round(seq_len * lambd))
    if win_len == seq_len:
        if size == seq_len: return o
        _slice = slice(None)
    else:
        start = np.random.randint(0, seq_len - win_len)
        _slice = slice(start, start + win_len)
    return F.interpolate(o[..., _slice], size=size, mode=mode, align_corners=False)
</code></pre>
<p><strong>Custom dataset <code>__getitem__</code></strong></p>
<pre><code class="lang-auto"> def __getitem__(self, index):
        serie = self.raw[index]
        shape = serie.shape

        # Augmentation
        aug = serie.view(1, shape[0], shape[1])
        aug = random_crop_resize(aug, size=shape[1], scale=(0.1, 1.0))

        aug = aug.view(shape[0], shape[1])
        return aug, self.bin_target[index], self.target[index]
</code></pre>
When I run the tutorial, there is an error

zhenkun: <p>in text_sentiment_ngrams_tutorial.ipynb</p>
<pre><code class="lang-auto">import torch
from torchtext.datasets import AG_NEWS
train_iter = iter(AG_NEWS(split='train'))
</code></pre>
<p>----&gt; 3 train_iter = iter(AG_NEWS(split=‘train’))</p>
<p>File ~/opt/anaconda3/lib/python3.9/site-packages/torchtext/data/datasets_utils.py:248, in _create_dataset_directory..decorator..wrapper(root, *args, **kwargs)<br>
246 if not os.path.exists(new_root):<br>
247     os.makedirs(new_root)<br>
 → 248 return fn(root=new_root, *args, **kwargs)</p>
<p>File ~/opt/anaconda3/lib/python3.9/site-packages/torchtext/data/datasets_utils.py:212, in _wrap_split_argument_with_fn..new_fn(root, split, **kwargs)<br>
210 result = []<br>
211 for item in _check_default_set(split, splits, fn.<strong>name</strong>):<br>
 → 212     result.append(fn(root, item, **kwargs))<br>
213 return _wrap_datasets(tuple(result), split)</p>
<p>File ~/opt/anaconda3/lib/python3.9/site-packages/torchtext/datasets/ag_news.py:55, in AG_NEWS(root, split)<br>
50 if not is_module_available(“torchdata”):<br>
51     raise ModuleNotFoundError(<br>
52         "Package <code>torchdata</code> not found. Please install following instructions at <code>https://github.com/pytorch/data</code>"<br>
53     )<br>
—&gt; 55 url_dp = IterableWrapper([URL[split]])<br>
56 cache_dp = url_dp.on_disk_cache(<br>
57     filepath_fn=lambda x: os.path.join(root, split + “.csv”),<br>
58     hash_dict={os.path.join(root, split + “.csv”): MD5[split]},<br>
59     hash_type=“md5”,<br>
60 )<br>
61 cache_dp = HttpReader(cache_dp)</p>
<p>NameError: name ‘IterableWrapper’ is not defined</p>

zhenkun: <p>conda install -c pytorch torchtext</p>
<p>The problem is solved!</p>

AbdulsalamBande: <p>Restart jupyter notebook and it should work</p>

IneedMrmeeseeks: <p>I’ve got a different error on the same code lines.</p>
<pre><code class="lang-auto">import torch
from torchtext.datasets import AG_NEWS
train_iter = iter(AG_NEWS(split='train'))
</code></pre>
<hr>
<p>TypeError                                 Traceback (most recent call last)<br>
Input In [5], in &lt;cell line: 1&gt;()<br>
----&gt; 1 train_iter = iter(AG_NEWS(split=‘train’))</p>
<p>File ~/anaconda3/envs/mitre_gan/lib/python3.10/site-packages/torchtext/datasets/text_classification.py:170, in AG_NEWS(*args, **kwargs)<br>
145 def AG_NEWS(<em>args, **kwargs):<br>
146     “”" Defines AG_NEWS datasets.<br>
147         The labels includes:<br>
148             - 1 : World<br>
(…)<br>
167<br>
168     “”"<br>
 → 170     return _setup_datasets(</em>((“AG_NEWS”,) + args), **kwargs)</p>
<p>TypeError: _setup_datasets() got an unexpected keyword argument ‘split’</p>
<p>I don’t know what the problem is.<br>
Thx.</p>

vdw: <p>Not sure if this related, but it seems that torchtext 0.13m made quite some changes; see also <a href="https://discuss.pytorch.org/t/cannot-import-name-field-from-torchtext-data/166885">this post</a>.</p>
Operations to undo repeat_interleave

tgangs: <p>Are there Pytorch operations that could reverse the effect of <code>repeat_interleave</code> in the following statement:</p>
<pre><code class="lang-auto">y = x.repeat_interleave(x)
</code></pre>
<p>For example, if y = tensor([1, 2, 2, 2, 2, 3, 3, 3, 1]), could we recover x = tensor([1, 2, 2, 3, 1])?</p>

ptrblck: <p>I think <code>unique_consecutive</code> can be used as seen here:</p>
<pre><code class="lang-python">x = torch.tensor([1, 2, 2, 3, 1])
y = x.repeat_interleave(x)

u, c = y.unique_consecutive(return_counts=True)
repeat = c / u
out = u.repeat_interleave(repeat.long())
print(out)
# tensor([1, 2, 2, 3, 1])
</code></pre>

tgangs: <p><code>unique_consecutive</code> works nicely here. Thank you! <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=12" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:" loading="lazy" width="20" height="20"></p>
Force 2d convolution to slide only along one direction

Machupicchu: <p>Hello,</p>
<p>i have a convolution layer as such:</p>
<pre><code class="lang-auto">
self.conv1 = nn.Conv2d(1, nb_kernels[0], kernel_size = (2,7))
</code></pre>
<p>for an input of shape 2x256</p>
<p>and i want my 2d conv kernel to slide ONLY in the columns direction, that is to keep the dimension where it has 2 rows and slide along the columns (I would also like to keep the columns dimension but that is ok it seems with a padding of 3 in the col dir)</p>
<p>however i dont seem to be able to do it with the kernel size of 2 rows ?</p>
<p>I tried to put a stride <code>stride=(0, 1)</code> but that is not allowed by pytorch it seems. … I suddenly don’t understand in fact why the kernels ‘wants’ to slide along the rows direction ?</p>
<p>how can i do it?<br>
(i tried padding=(1,3) → but this does not work either because it produces an output of shape [3, 256] … this confuses me … I guess the 3 comes from padding top and bottom so the kernel slides over an input of 4x256 and can therefore slide 3 times?)</p>
<p>P.S.<br>
ideally i don’t want to use 1d convolutions and channels which i imagine would be a solution</p>

Machupicchu: <p>sorry found out myself… too tired… of course its valid conv by default<br>
and the 2x7 just fits in the rows direction<br>
my mistake</p>

ptrblck: <p>Double post from <a href="https://discuss.pytorch.org/t/how-can-i-force-a-2d-convolution-to-only-slide-along-one-direction/173190">here</a>. I’m glad you have found the answer, but please stick to one account. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
How to Find the gradient for following case

Dhruv_Shah: <p>Hello Thank you for viewing my post,<br>
I have error in finding dy/dx for the following case:<br>
y = [2,3,4] ( This values are determined by function f(x) this may be different every time they are called)<br>
and I have x = [3,4,5] …<br>
What I mean  to say is y is not directly a function of x (All the Examples I found on Youtube/Google involve declaring x and setting y as a function of x with mathematical functions)</p>
<p>How can i find dy/dx using tensor.?<br>
Thank you for reading my question I would mean i lot if you could help me</p>
<p>Example of what I want:<br>
Lets say I have following data:<br>
y = [2,3,4]<br>
x = [1,2,3]<br>
then dy/dx for first element is <strong>Δ</strong>y/ <strong>Δ</strong>x = (3-2)/(2-1) = 1<br>
(Euler Backward)<br>
similarly for other elements</p>

ptrblck: <aside class="quote no-group" data-username="Dhruv_Shah" data-post="1" data-topic="173151">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dhruv_shah/48/57151_2.png" class="avatar"> Dhruv_Shah:</div>
<blockquote>
<p>What I mean to say is y is not directly a function of x</p>
</blockquote>
</aside>
<p>If <code>y</code> is not depending on <code>x</code>, it seems you won’t be able to use Autograd and the chain rule to calculate the gradients. Could you describe in more detail how <code>y</code> is calculated?</p>

Dhruv_Shah: <p>Thank you very much.<br>
y is taken from a prestored pandas dataframe (Solution of a Differential equation calculated using FEM methods) and x is the spatial domain.<br>
Thank you once again</p>

ptrblck: <p>In this case you might either need to implement the FEM in a differentiable way or try to calculate the gradinets somehow analytically. I don’t know how PyTorch or rather Autograd would be able to compute the backward pass without the actual computation graph.</p>

Dhruv_Shah: <p>Thank you <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"><br>
The tutorials on youtube don’t give a detailed behind the scene structure of pytorch I found only this video <a href="https://www.youtube.com/watch?v=MswxJw-8PvE&amp;t=281s" rel="noopener nofollow ugc">https://www.youtube.com/watch?v=MswxJw-8PvE&amp;t=281s</a><br>
Can you suggest some other resource or material …</p>
The different in nn.Dropout and nn,Dropout1d

Peterwisu: <p>Hi, I just wonder what is different between nn.Dropout and Dropout1d?  which one should I use for normal Linear layers? Thank you</p>

ptrblck: <p><code>nn.Dropout1d</code> will zero out entire “channels”, i.e. values in <code>dim1</code>. These activations are usually the provided by <code>nn.Conv1d</code> modules (or other “temporal” layers).<br>
For standard <code>nn.Linear</code> layers working on an input in the shape <code>[batch_size, in_features]</code> (i.e. without a “temporal” dimension), <code>nn.Dropout</code> would be the common choice.</p>
Does iterating over unshuffled Dataloader change random state?

hubsiiiii: <p>Hello,</p>
<p>I am a little bit confused about the reproducibility and random state of PyTorch. I set seeds in my code, and everything is deterministic and reproducible. If I start my experiments with the same input, I get the same output. This is how I set the seeds:</p>
<pre><code class="lang-auto">    if args.deterministic:
        cudnn.benchmark = False
        cudnn.deterministic = True
        random.seed(args.seed)
        np.random.seed(args.seed)
        torch.manual_seed(args.seed)
        torch.cuda.manual_seed(args.seed)
</code></pre>
<p>However, I just found out that when I load a pre-trained model and evaluate it on my test dataloader <strong>BEFORE</strong> I continue training, I get different accuracy results compared to not evaluating it before continuing training! The shuffled argument in my test dataloader is set to <strong>False</strong>, the number of workers is set to <strong>0</strong> and I don’t have any transformations in my test dataset that require randomness.<br>
I even tried to simply iterating over my test dataloader without passing the data through my model before continuing training. I still get different results, but they are identical to what I would get if I evaluate my model on the test dataloader (before training).</p>
<p>Therefore, my question is, does simply iterating over a dataloader change the random state?</p>
<p>Thank you, I would really appreciate your help <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p><strong>EDIT</strong>: I implemented a small example:<br>
If I execute the following code:</p>
<pre><code class="lang-auto">import torch
torch.manual_seed(42)
print(torch.rand(1))
</code></pre>
<p>I <strong>always</strong> get the following output:</p>
<blockquote>
<p>tensor([0.8823])</p>
</blockquote>
<p>However, if I execute the following code:</p>
<pre><code class="lang-auto">import torch
from torchvision import datasets
from torchvision.transforms import ToTensor
from torch.utils.data import DataLoader

torch.manual_seed(42)

test_data = datasets.FashionMNIST(root="data", train=False, download=True, transform=ToTensor())
test_dataloader = DataLoader(test_data, batch_size=64, shuffle=False)
for test_features, test_labels in test_dataloader:
    pass

print(torch.rand(1))
</code></pre>
<p>I get the following output:</p>
<blockquote>
<p>tensor([0.3829])</p>
</blockquote>
<p>I think the outputs should be the same since I am not performing any operations that require randomness.</p>

ptrblck: <p>That’s expected since the <code>_BaseDataLoaderIter</code> sets its <code>_base_seed</code> <a href="https://github.com/pytorch/pytorch/blob/a4b02a15d33cdd946d95af51f48bb0640a6abcb5/torch/utils/data/dataloader.py#L605">here</a> via:</p>
<pre><code class="lang-python">self._base_seed = torch.empty((), dtype=torch.int64).random_(generator=loader.generator).item()
</code></pre>
Write torch tensor into a JSON file

ilovetorch: <p>I have some data like this:</p>
<pre><code class="lang-auto">tensor([[0.9938+0.j, 0.1109+0.j],
        [1.0000+0.j, 0.0000+0.j],
        [0.9450+0.j, 0.3272+0.j],
        [0.9253+0.j, 0.3792+0.j],
        [0.9450+0.j, 0.3272+0.j],
        [0.9028+0.j, 0.4300+0.j],
        [0.8776+0.j, 0.4794+0.j],
        [0.9253+0.j, 0.3792+0.j],
        [0.9253+0.j, 0.3792+0.j],
        [0.9985+0.j, 0.0555+0.j]]
</code></pre>
<p>and I would like to save it to a JSON file.<br>
I have tried to convert it to string by doing <code>data.numpy().tostring()</code> but it gave me an error saying <code>TypeError: Object of type bytes is not JSON serializable</code>.<br>
What is the correct way to do this? Any suggestion will be appreciated.</p>
Can't torch multiprocessing pass model parameters correctly?

zhishen_nie1: <p>In order to improve the utilization of the graphics card, I decided to run multiple models at the same time. I find that torch.multiprocessing could finish it.<br>
Here is my code.</p>
<p>import torch.multiprocessing as mp<br>
pool = []<br>
for i in range(num_workers):<br>
model = torch.load(…).eval()<br>
p = mp.Process(<br>
target=target_function,<br>
args=(arg1, arg2, model)<br>
)<br>
p.start()<br>
pool.append(p)<br>
for p in pool:<br>
p.join()<br>
However, all the params of model in target_function is 0. I guess there is some problem in passing the model parameters. So how to slove this problem.</p>
Add l1-regularization term to loss

jsh0212: <p>I want to add l1-regularization term to loss, but I am little bit confused.</p>
<p>When I print the parameters’ shape of CNN, each of them has different dimension.</p>
<pre><code class="lang-auto">&gt;&gt;[print(p.shape) for p in model.parameters()]
torch.Size([64, 3, 3, 3])
torch.Size([64])
torch.Size([64])
torch.Size([64, 64, 3, 3])
torch.Size([64])
torch.Size([64])
torch.Size([64, 64, 3, 3])
torch.Size([64])
torch.Size([64])
torch.Size([64, 64, 3, 3])
torch.Size([64])
torch.Size([64])
torch.Size([64, 64, 3, 3])
torch.Size([64])
torch.Size([64])
torch.Size([128, 64, 3, 3])
torch.Size([128])
torch.Size([128])
torch.Size([128, 128, 3, 3])
torch.Size([128])
torch.Size([128])
torch.Size([128, 64, 1, 1])
torch.Size([128])
torch.Size([128])
torch.Size([128, 128, 3, 3])
...
</code></pre>
<p><strong>1) Use torch.linalg.norm</strong></p>
<p>cost += lambda * sum(torch.linalg.norm(p,ord=1) for p in model.parameters())</p>
<p>When I use torch.linalg.norm I have to choose dim.</p>
<pre><code class="lang-auto">&gt;&gt;sum(torch.linalg.norm(p,1) for p in model.parameters())
Traceback (most recent call last):
  File "&lt;string&gt;", line 1, in &lt;module&gt;
  File "&lt;string&gt;", line 1, in &lt;genexpr&gt;
RuntimeError: 'dim' must specify 1 or 2 dimensions when order is numerical and input is not 1-D or 2-D
</code></pre>
<p>But when I choose dimension as 0, each output has different shape so summation is impossible.</p>
<pre><code class="lang-auto">&gt;&gt;sum(torch.linalg.norm(p,1,0) for p in model.parameters())
Traceback (most recent call last):
  File "&lt;string&gt;", line 1, in &lt;module&gt;
RuntimeError: The size of tensor a (3) must match the size of tensor b (64) at non-singleton dimension 0
</code></pre>
<p>How can I add l1-regularization term to loss using torch.linalg.norm?</p>
<p><strong>2) Use torch.tensor.norm</strong></p>
<p>cost += lambda * sum(p.norm(1) for p in model.parameters())</p>
<p>It works but its result value is very large. Is it right way to add l1-regularization term to loss?</p>
<pre><code class="lang-auto">&gt;&gt;sum(p.norm(1) for p in model.parameters())
tensor(111630.8516, device='cuda:0', grad_fn=&lt;AddBackward0&gt;)
</code></pre>
<p><strong>3) What’s the difference between torch.linalg.norm and torch.tensor.norm?</strong></p>

UCDuan: <p>should it be sum(p.norm(1)) or mean(p.norm(1))?</p>
How can i force a 2d convolution to only slide along one direction?

AlanTuring: <p>Hi I have an input of shape 2x256 and a convolution layer of shape:</p>
<p><code>self.conv1 = nn.Conv2d(1, nb_kernels[0], kernel_size = (2,7))</code></p>
<p>my problem is that i want the kernel to only slide along the columns (the long) dimension and i want therefore the output to be 2x256 again (unchanged)</p>
<p>So usually i would pad, but it does not work for the rows and pyTorch does not allow a stride of 0 in the rows (or columns) direction… how can i achieve this?<br>
I don’t understand why the conv kernel “wants” to slide along the rows since it is 2 and rows are 2… shouldn’t it stay there?<br>
and strangely padding like so : padding=(0, 3) does not work: the output is 1x256 not 2x256 ! I dont understand that…</p>
<p>(P.S: stride=(1, 1), padding=(1, 3) do not work for me since they produce an output of 3x256)</p>

AlanTuring: <p>Oh no I m sorry … i think i got my mistake … of course the valid convolution by default produces a 1x256 with a kernel of 2xn … because it just fits in so to speak…<br>
i am tired …sorry for this post.</p>
File.npy write as mode 'a'

goodboy-hub: <p>Does a .npy file allow the write mode ‘a’? For example ,i have stored an array ‘<em>data1</em>’ in 0.npy, could i also store another array ‘<em>data2</em>’ in ‘0.npy’? If i just use np.load() then the <em>data1</em> will be lost</p>

ptrblck: <p>Based on <a href="https://stackoverflow.com/questions/30376581/save-numpy-array-in-append-mode">this topic</a> it seems <code>np.save</code> does not directly support it but some users have posted potential workarounds.</p>
RuntimeError: The size of tensor a (52) must match the size of tensor b (19) at non-singleton dimension 3

imran: <p>I am using pytorch implementation of yolov4 and I wanted to train custom object detection model with image resolution 416x416 with following repo</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/Tianxiaomo/pytorch-YOLOv4">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/Tianxiaomo/pytorch-YOLOv4" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/345;"><img src="https://opengraph.githubassets.com/8a401c3f2a09b957df5a43a748640f7935d7a40478c9f83f0826b975d1201261/Tianxiaomo/pytorch-YOLOv4" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/Tianxiaomo/pytorch-YOLOv4" target="_blank" rel="noopener nofollow ugc">GitHub - Tianxiaomo/pytorch-YOLOv4: PyTorch ,ONNX and TensorRT implementation...</a></h3>

  <p>PyTorch ,ONNX and TensorRT implementation of YOLOv4 - GitHub - Tianxiaomo/pytorch-YOLOv4: PyTorch ,ONNX and TensorRT implementation of YOLOv4</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Is this implementation is not compatible with changing input image size.</p>
<p>In configuration file I changed image size form 608 to 416 as shown below</p>
<p>Cfg.width = 416<br>
Cfg.height = 416</p>
<p>I got the following error.</p>
<p>Traceback (most recent call last):<br>
File “./pytorch-YOLOv4/train.py”, line 445, in<br>
train(model=model,<br>
File “./pytorch-YOLOv4/train.py”, line 321, in train<br>
loss, loss_xy, loss_wh, loss_obj, loss_cls, loss_l2 = criterion(bboxes_pred, bboxes)<br>
File “/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py”, line 532, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “./pytorch-YOLOv4/train.py”, line 205, in forward<br>
pred[…, 0] += self.grid_x[output_id]<br>
RuntimeError: The size of tensor a (52) must match the size of tensor b (19) at non-singleton dimension 3</p>
<p>Any help is appreciated.</p>
Running multiple instances of training slows down dataloader

saandeep_aathreya: <p>I have written a custom dataloader to fetch the data and labels. I am training my deep learning model with different hyperparameter values. But one thing I notice is that if I run multiple programs, each with different hyperparameter values, on the same GPU (with same GPU usage, batch size etc…), both the program slows down by atleast 2X. Is there any particular reason for this? I am thinking it might be because of multiple loader instances? Here is a very general template of how my loader looks like.</p>
<pre><code class="lang-auto">class CustomLoader:
    def __init__(self) -&gt; None:
        super().__init__()
        self.all_files = []
        self.all_labels = []

    def get_all_files(self):
        //returns all image file paths and its corresponding labels

    def __len__(self):
        return len(self.all_files)

    def __getitem__(self, idx):
        image = Image.open(self.all_files[idx])
        label = self.all_labels[idx]
        return image, label



</code></pre>

eqy: <p>Are you only comparing dataloading speed or end-to-end benchmarking (e.g., including training/inference) on the same GPU? It would be expected that increased contention from running two programs that are calling kernels on the same GPU would cause a slowdown. If you are only benchmarking dataloading, I would check if the underlying storage is the bottleneck (are you saturating the IOPs or bandwidth) or if CPU is the bottleneck (e.g., that image decoding/preprocessing/transforms are too slow).</p>

saandeep_aathreya: <p>Thanks for the reply.<br>
I am comparing end-to-end i.e. the time taken to run through one epoch during training. My GPU capacity is 24GB and each program occupies ~3GB of memory.</p>
Runtime error using Distributed with gloo

CynthiaMY: <p>Hello everyone,</p>
<p>I am trying to use the PyTorch distributed package and gloo backend.</p>
<p>But, I get the following error.</p>
<p>Master:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "Distributed.py", line 169, in &lt;module&gt;
    init_processes(args.rank, size, run)
  File "Distributed.py", line 80, in init_processes
    dist.init_process_group(backend=backend, rank=rank, world_size=size)
  File "/usr/local/lib/python3.5/dist-packages/torch/distributed/__init__.py", line 49, in init_process_group
    group_name, rank)
RuntimeError: [enforce fail at /pytorch/torch/lib/gloo/gloo/transport/tcp/device.cc:127] rp != nullptr. Unable to find address for: &lt;My specified Ip&gt; at /pytorch/torch/lib/THD/process_group/General.cpp:17
</code></pre>
<p>And another worker gets:</p>
<p><code>RuntimeError: [enforce fail at /pytorch/torch/lib/gloo/gloo/transport/tcp/device.cc:127] rp != nullptr. Unable to find address for: 10.37.0.1 at /pytorch/torch/lib/THD/process_group/General.cpp:17</code></p>
<p>Below is how I initialize:</p>
<pre><code class="lang-auto">def init_processes(rank, size, fn, backend='gloo'):
    """ Initialize the distributed environment. """

    os.environ['MASTER_ADDR'] = &lt;My specified Ip&gt;
    os.environ['MASTER_PORT'] = '8888' if rank == 0 else '31566'

    print("Init Processes -&gt;", 'backend:', backend, 'rank:', rank, 'MASTER_ADDR:', os.environ['MASTER_ADDR'],
          'MASTER_PORT:', os.environ['MASTER_PORT'])
    dist.init_process_group(backend=backend, rank=rank, world_size=size)
    fn(rank, size)
</code></pre>
<p>Any thoughts on what may be causing this or how I can fix it ? Thx<br>
ps. I use docker env with python 3.5 and PyTorch 0.3.1 with cuda9.0.</p>

ifgovh: <p>By default, both NCCL and Gloo backends will try to find the network interface to use for communication. However, this is not always guaranteed to be successful from our experiences. Therefore, if you encounter any problem on either backend not being able to find the correct network interface. You can try to set the following environment variables (each one applicable to its respective backend):</p>
<ul>
<li>
<strong>NCCL_SOCKET_IFNAME</strong> , for example  <code>export NCCL_SOCKET_IFNAME=eth0</code>
</li>
<li>
<strong>GLOO_SOCKET_IFNAME</strong> , for example  <code>export GLOO_SOCKET_IFNAME=eth0</code><br>
<a href="https://pytorch.org/docs/stable/distributed.html#environment-variable-initialization" rel="nofollow noopener">https://pytorch.org/docs/stable/distributed.html#environment-variable-initialization</a>
</li>
</ul>
<p>BTW, use ifconfig to find your first Ethernet interface.</p>

platero: <p>Thanks! This saves my day.</p>