akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Is it possible for using 3rd party lib when define loss function?

username: liuchangf
cooked: <p>Hello There?</p>
<p>I want to use 3rd party lib when define loss function , because pytorch can not fulfill all requirments for users ,pseudo code as following:</p>
<pre><code>from shapely.geometry import Polygon  # 3rd party pacakage

p1=Polygon(polygon_from_model_inference)
p2=Polygon(polygon_ground_truth)
intersection_area=p1.intersection(p2).area
iou=intersection_area/(p1.area+p2.area-intersection_area)
loss= iou
</code></pre>
<p>Whether pytoch will automatically trace the gradient of “loss” or not ? Thank you in advance</p>

username: ptrblck
cooked: <p>Yes, it’s possible, but Autograd won’t be able to track these operations, so you would need to write the <code>backward</code> pass manually via a custom <code>autograd.Function</code> as described <a href="https://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_custom_function.html">here</a>.</p>
Need help with implementing my own model based on a pretrained model

username: dhy
cooked: <p>Hello,</p>
<p>I am trying to implement my own model based on RoBERTa. My Python code looks like below.</p>
<pre><code class="lang-auto">class My_RoBERTa(nn.Module):
    def __init__(self, some_config, some_version):
        super().__init__()
        self.roberta = RobertaModel.from_pretrained(some_version)
    def forward(self, input_ids, attn_masks, labels=None):
        print(1)
        print(input_ids.size())
        print('* *')
        print(attn_masks.size())
        print('* *')
        outputs = self.roberta(input_ids, attn_masks)
        print(3)
        print(4)
        print(5)
</code></pre>
<p>If I instantiate this model and pass 3 examples with a length = 25, then my expectation of output is below</p>
<pre><code class="lang-auto">1
torch.Size([3,25])
* *
torch.Size([3,25])
* *
3
4
5
</code></pre>
<p>However, when I run it, I actually get something like below.</p>
<pre><code class="lang-auto">1
torch.Size([3,25])
* *
torch.Size([3,25])
* *
1
torch.Size([3,25])
* *
torch.Size([3,25])
* *
3
3
4
4
5
5
</code></pre>
<p>I think this is some kind of overwriting issue caused by the “forward” method, but I do not know how to resolve this problem. Could anyone please help me out?</p>
<p>Thank you so much in advance!</p>

username: ptrblck
cooked: <p>Are you using a data parallel approach (e.g. via <code>nn.DataParallel</code> or <code>DistributedDataParallel</code>)?<br>
If so, then the output would be expected as the model is executed on each device.</p>
Can't find source of memory leak

username: sohaib_attaiki
cooked: <p>Hi All,</p>
<p>I have a small model (2M params), and I’m using batch = 1. The size of every batch varies, but in average, I’m using 5 gb per iteration for the first epoch.</p>
<p>My problem is that after the first epoch, my memory consumption increases until it hits an OOM. I’m suspecting this due to a memory leak, so I tried the following fixes:</p>
<ul>
<li>add <code>torch.cuda.empty_cache()</code> after each memory heavy operation</li>
<li>delete my mini batch data and loss after each iteration</li>
<li>detach and convert to cpu all log data</li>
</ul>
<p>but this didn’t seem to solve my issue!</p>
<p>Is there something I’m missing? can you help me solve this issue?<br>
Thank you in advance!</p>

username: tom
cooked: <p>I would check if it is roughly linear per iteration or something else by printing the PyTorch allocated memory after every iteration.<br>
If it is a leak, it should likely be linear.</p>

username: sohaib_attaiki
cooked: <p>I <a class="mention" href="/u/tom">@tom</a><br>
Thanks for your response.<br>
It’s not linear, and this is because the size of every mini-batch is different, I’m using graphs, and every graph has a different number of nodes!</p>

username: tom
cooked: <p>Well, if you have to, you could run the same graph over and over again.</p>
How can i predict in trained model

username: Yasar_Niyazoglu
cooked: <p>I’ve trained my own model in pytorch and I want to test it with an unused image for my own training and testing, but it doesn’t work.</p>
<pre><code class="lang-auto">from torchvision import transforms
img=(Image.open(r"C:\Users\yasar\Desktop\Pytorch\ferrari.jpg"))
transform = transforms.Compose([        # Defining a variable transforms
 transforms.Resize(256),                # Resize the image to 256×256 pixels
 transforms.CenterCrop(224),            # Crop the image to 224×224 pixels about the center
 transforms.ToTensor(),                 # Convert the image to PyTorch Tensor data type
 transforms.Normalize(                  # Normalize the image
 mean=[0.485, 0.456, 0.406],            # Mean and std of image as also used when training the network
 std=[0.229, 0.224, 0.225]      
 )])

img_t = transform(img)
img_t.shape

batch_t = torch.unsqueeze(img_t, 0)
model.eval()

out = model(batch_t)
out.shape
</code></pre>
<p>RuntimeError: shape ‘[1, 519840]’ is invalid for input of size 20000</p>

username: ptrblck
cooked: <p>What were the input shapes during training for this model?<br>
Based on the error message you’ve posted, I assume you are running into a shape mismatch inside the model’s <code>forward</code> (maybe in a linear layer)?</p>

username: Yasar_Niyazoglu
cooked: <pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self):
        super(Net,self).__init__()
        
        self.conv1=nn.Conv2d(in_channels=3,out_channels=4,kernel_size=(5,5))
        self.conv2=nn.Conv2d(in_channels=4,out_channels=8,kernel_size=(3,3))
        self.conv3=nn.Conv2d(in_channels=8,out_channels=16,kernel_size=(2,2))#22
        self.conv4=nn.Conv2d(in_channels=16,out_channels=32,kernel_size=(2,2))#10
        
        self.max=nn.MaxPool2d(kernel_size=(2,2))
        self.func=nn.ELU()
        
        self.fc1=nn.Linear(in_features=519840,out_features=50)
        #self.fc1=nn.Linear(in_features=32*100*100,out_features=50)
        self.fc2=nn.Linear(in_features=50,out_features=50)
        self.fc3=nn.Linear(in_features=50,out_features=100)
        self.fc4=nn.Linear(in_features=100,out_features=4)
        
    def forward(self,x):
        
        x=self.conv1(x)
        x=self.func(x)
        
        x=self.max(x)
        
        x=self.conv2(x)
        x=self.func(x)
        
        x=self.max(x)
        
        x=self.conv3(x)
        x=self.func(x)
        
        x=self.max(x)
        
        x=self.conv4(x)
        x=self.func(x)

        #x=x.view(x.size(0), -1)
        x = x.view(x.size(0),519840) 
        
        x=self.fc1(x)
        x=self.func(x)
        x=self.fc2(x)
        x=self.func(x)
        x=self.fc3(x)
        x=self.func(x)
        x=self.fc4(x)
        return x       ```
28*28 size</code></pre>
Eigenvalues of a tridiagonal block matrix in Pytorch

username: Amine08
cooked: <p>Hi,</p>
<p>what is the fastest way to calculate the eigenvalues of a tridiagonal block matrix in pytorch?</p>
<p>Ór</p>
<p>what is the fastest way to check whether a tridiagonal block matrix is postive semi-definite and calculate log det of it in pytorch?</p>
<p>is there an equivalent function (for tridiagonal block matrix) in Pytorch to <a href="https://docs.scipy.org/doc/scipy/reference/generated/scipy.linalg.eigvalsh_tridiagonal.html#scipy.linalg.eigvalsh_tridiagonal" rel="noopener nofollow ugc">scipy.linalg.eigvalsh_tridiagonal</a> ?</p>
<p>Thanks!</p>
Data-parallel solution comparisons. Which would be the data-parallel solution? nn.DataParallel() vs DistributedDataParallel vs PyTorch Lightning Horovod vs any other

username: dakeoffer
cooked: <p>What would be the best data-parallel solution regarding the model’s maintaining the same performance or even better compared with training on one GPU?</p>
<p>nn.DataParallel() vs DistributedDataParallel vs PyTorch Lightning Horovod vs any other available methods</p>

username: rvarm1
cooked: <p>We recommend to use <code>DistributedDataParallel</code> over <code>nn.DataParallel</code> as the latter relies on python threading, which is slow due to the GIL.</p>
<p>Regarding comparisons to PyTorch lightning, lightning offers DDP as a plugin and calls into DDP under the hood, so the performance should be comparable. I’m not aware of any performance comparisions between DDP and Horovod, unfortunately.</p>

username: rvarm1
cooked: <p>A couple of papers actually have comparisons between PT, Horovod, and other frameworks: <a href="https://assets.amazon.science/ba/69/0a396bd3459294ad940a705ad7f5/herring-rethinking-the-parameter-server-at-scale-for-the-cloud.pdf" rel="noopener nofollow ugc">AWS Herring paper</a> and a <a href="https://medium.com/distributed-computing-with-ray/faster-and-cheaper-pytorch-with-raysgd-a5a44d4fd220" rel="noopener nofollow ugc">Ray blog post</a> which does a similar comparison.</p>
Reconciling difference in fft output between pytorch 1.5.0 and pytorch 1.9.0

username: nbarron
cooked: <aside class="onebox stackexchange" data-onebox-src="https://stackoverflow.com/questions/68368458/reconciling-difference-in-fft-output-between-pytorch-1-5-0-and-pytorch-1-9-0">
  <header class="source">

      <a href="https://stackoverflow.com/questions/68368458/reconciling-difference-in-fft-output-between-pytorch-1-5-0-and-pytorch-1-9-0" target="_blank" rel="noopener nofollow ugc">stackoverflow.com</a>
  </header>

  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/16442060/nbarron" target="_blank" rel="noopener nofollow ugc">
    <img alt="nbarron" src="https://lh3.googleusercontent.com/a/AATXAJxS_1UFgIoFeDQdEHBMn-3Pad1vRaSHi6n1ua9E9Q=k-s128" class="thumbnail onebox-avatar" width="128" height="128">
  </a>

<h4>
  <a href="https://stackoverflow.com/questions/68368458/reconciling-difference-in-fft-output-between-pytorch-1-5-0-and-pytorch-1-9-0" target="_blank" rel="noopener nofollow ugc">Reconciling difference in fft output between pytorch 1.5.0 and pytorch 1.9.0</a>
</h4>

<div class="tags">
  <strong>python, arrays, numpy, pytorch</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/16442060/nbarron" target="_blank" rel="noopener nofollow ugc">
    nbarron
  </a>
  on <a href="https://stackoverflow.com/questions/68368458/reconciling-difference-in-fft-output-between-pytorch-1-5-0-and-pytorch-1-9-0" target="_blank" rel="noopener nofollow ugc">07:38PM - 13 Jul 21 UTC</a>
</div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
Triplet Loss Returning Zero

username: ASMIftekhar
cooked: <p>Hello,<br>
I was trying to test with tiplet loss and got something very weird. So lets say my</p>
<pre><code class="lang-auto">anchors, pos_embeddings, neg_embeddings are:
(tensor([[8.0368e+16, 4.2619e+16]]), tensor([[0.2196, 0.2067]]), tensor([[0.1873, 0.9207]]))
</code></pre>
<p>Now,</p>
<p><code>F.triplet_margin_loss(anchors.long(),pos_embeddings.long(),neg_embeddings.long(),margin=1,p=2,reduction='none') is returning 0</code></p>
<p>I know the anchors are extremely high, but I want to understand triplet loss behavior.</p>

username: ptrblck
cooked: <p>In your code snippet you are transforming the inputs to <code>LongTensors</code>, so <code>pos_embeddings</code> and <code>neg_embeddings</code> will be equal:</p>
<pre><code class="lang-python">anchors, pos_embeddings, neg_embeddings = torch.tensor([[8.0368e+16, 4.2619e+16]]), torch.tensor([[0.2196, 0.2067]]), torch.tensor([[0.1873, 0.9207]])
print(pos_embeddings.long())
&gt; tensor([[0, 0]])
print(neg_embeddings.long())
&gt; tensor([[0, 0]])
</code></pre>
<p>Given that, the output of the loss function should then be <code>margin</code>, which is set to <code>1</code>.<br>
However, the intermediate distances would be large:</p>
<pre><code class="lang-python">pos_embeddings = torch.tensor([[0, 0]])
neg_embeddings = pos_embeddings
dist_pos = torch.norm(anchor - pos_embeddings, p=2);
print(dist_pos)
&gt; tensor(1.2923e+10, grad_fn=&lt;NormBackward1&gt;)
dist_neg = torch.norm(anchor - neg_embeddings, p=2);
print(dist_neg)
&gt; tensor(1.2923e+10, grad_fn=&lt;NormBackward1&gt;)
</code></pre>
<p>The next operation would be the addition of the <code>margin</code> to the distance and the <code>clamp</code> operation:</p>
<pre><code class="lang-python">output = torch.clamp_min(1. + dist_pos - dist_neg, 0)
print(output)
&gt; tensor(0., grad_fn=&lt;ClampMinBackward0&gt;)
</code></pre>
<p>Here you can see, that <code>1. + dist_pos - dist_neg</code> underflows. The reason is the decimal step size, which is <code>&gt;1</code> for values <code>&gt;2**24</code> as described in <a href="https://en.wikipedia.org/wiki/Single-precision_floating-point_format">this Wikipedia article</a>.<br>
A potential fix would be to use:</p>
<pre><code class="lang-python">output = torch.clamp_min(1. + (dist_pos - dist_neg), 0)
</code></pre>
<p>or <code>float64</code>.</p>
<p>CC <a class="mention" href="/u/tom">@tom</a> what do you think about subtracting the distances first? Could this yield any other (unwanted) issues?</p>

username: tom
cooked: <p>It might work, but TBH neither does the cast to long make sense to me nor do the large numbers.<br>
The original intention of the triple loss was to <a href="http://www.bmva.org/bmvc/2016/papers/paper119/paper119.pdf" rel="noopener nofollow ugc">have feature vectors from three examples where a and p are of the same class and n is of a different one</a>. I don’t think having anchors live on a completely different scale than p and n makes much sense for that.</p>
Why F.unfold() and F.fold() use?

username: PeterKim1
cooked: <p>Hello. I have a question about F.unfold() and F.fold()</p>
<p>I understand how to operate F.unfold() and F.fold(), but i don’t understand why they need.</p>
<p>I wonder what kind of situation it is usually used in. (especially, in vision context.)</p>
<p>Thanks.</p>

username: ptrblck
cooked: <p>You could <code>unfold</code> an image tensor, apply a convolution via a matrix multiplication, and <code>fold</code> it back to the desired output in order to e.g. write a custom conv (where you could add additional operations), for custom pooling layers etc.<br>
Also, creating “windows” from e.g. time signals can use the same approach.</p>
Find predicted image ID and box from SSD

username: rebeen
cooked: <p>Could you please help me to find  predicted image id and Box from SSD,  I am using this <a href="https://github.com/qfgaohao/pytorch-ssd/blob/master/train_ssd.py" rel="nofollow noopener">GitHub link</a> here is the  test function which I want to save the image id and box</p>
<p>def test(loader, net, criterion, device):<br>
net.eval()<br>
running_loss = 0.0<br>
running_regression_loss = 0.0<br>
running_classification_loss = 0.0<br>
num = 0<br>
for _, data in enumerate(loader):<br>
images, boxes, labels = data<br>
images = images.to(device)<br>
boxes = boxes.to(device)<br>
labels = labels.to(device)<br>
num += 1</p>
<pre><code>    with torch.no_grad():
        confidence, locations = net(images)
        regression_loss, classification_loss = criterion(confidence, locations, labels, boxes)
        loss = regression_loss + classification_loss

    running_loss += loss.item()
    running_regression_loss += regression_loss.item()
    running_classification_loss += classification_loss.item()
return running_loss / num, running_regression_loss / num, running_classification_loss / num</code></pre>

username: anujd9
cooked: <p>Did you check out <a href="https://github.com/qfgaohao/pytorch-ssd/blob/master/run_ssd_example.py" rel="noopener nofollow ugc">this</a> sample provided in the repo to run your model predictions?</p>
What is the equivalent of 'tf.keras.layers.experimental.preprocessing.Normalization' in Pytorch

username: Arian_Jafari
cooked: <p>I am trying to switch my model from tensorflow-keras to pytorch but I faced a problem. I have “tf.keras.layers.experimental.preprocessing.Normalization” in my model as a layer, so the model will save the mean and std as model parameters. Therefore, when it comes to testing, I can use these data to normalize my test_set. I do that because I want my trained model be usable with different users who don’t have access to the whole training data. How can I do the same thing in Pytorch?</p>

username: tom
cooked: <p>The PyTorch way™ is to precompute these and provide the result (for use in torchvision.transforms.Normalize).</p>
<p>I was going to link to the forums for an example for how to do the precomputation, but I got confused with what people iterating lots of versions supposedly less buggy than the previous and so here is a quick and dirty example to recompute the dataset mean and std for CIFAR (you wouldn’t need that, because you can just compute mean and std in the dataset in one go, but this lets you check). The algorithm basically is the Welford-generalization to more than one thing in the update.</p>
<pre><code class="lang-auto">import torchvision
import torch.utils.data

ds = torchvision.datasets.CIFAR10('/mnt/data/vision/cifar10', transform=torchvision.transforms.ToTensor())
dl = torch.utils.data.DataLoader(ds, batch_size=1024)

count = 0
mean = torch.zeros((3,), dtype=torch.double)
M2 = torch.zeros((3,), dtype=torch.double)
for im, lab in dl:
    # https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#Parallel_algorithm
    im = im.to(dtype=torch.double)
    batch_count = im.size(0)
    batch_mean = im.mean(dim=(0, 2, 3))
    batch_M2 = im.var(dim=(0, 2, 3)) * batch_count
    delta = batch_mean - mean
    mean += delta * batch_count / (count + batch_count)
    M2 += batch_M2 + delta ** 2 * count * batch_count / (count + batch_count)
    count += batch_count
    
std = (M2 / count)**0.5

# for CIFAR, we can check
assert (torch.from_numpy(ds.data/255).mean(dim=(0, 1, 2)) - mean).abs().max() &lt; 1e-6
assert (torch.from_numpy(ds.data/255).std(dim=(0, 1, 2)) - std).abs().max() &lt; 1e-6
</code></pre>
<p>Best regards</p>
<p>Thomas</p>

username: Arian_Jafari
cooked: <p>Thanks Thomas. I also thought of logging the mean and std when I train my model and provide them as meta data to the user who is going to use the trained model for his/her test set, so s/he can use the transofrm to normalzie before feeding them to the pre trained model.</p>
Use sklearn function with CUDA (GPU) tensors

username: ptrch_c_m
cooked: <p>I would like to know if I can use <code>sklearn</code> <code>precision_recall_fscore_support()</code> with CUDA tensors.</p>
<p>Normally, this function gets two arrays (specifically vectors), one with labels and another with predictions.</p>
<p>Can I give it multiple labels and prediction vectors in parallel, utilizing the GPUs?</p>
<p>Thank you in advance.</p>

username: ptrblck
cooked: <p>You won’t be able to directly pass CUDATensors from PyTorch to this method, since <code>sklearn</code> uses numpy as the backend, which cannot use the GPU.<br>
However, you could check the implementation and rewrite this method using <code>torch</code> functions, which could then use CUDATensors.</p>

username: ptrch_c_m
cooked: <p>This is unfortunate, however thank you <a class="mention" href="/u/ptrblck">@ptrblck</a> for your fast reply.</p>
Custom mean of tensor partitions

username: mamadpierre
cooked: <p>I have a tensor of size <code>(n1, n2, n3)</code> and a <code>lst</code> of size <code>n2</code> with integer values in <code>[0, 1, ..., n2-1]</code> as partition (or cluster) assignments. What I want is an output of size <code>(n1, len(set(lst)), n3)</code> where the second elements are the mean of those belong to the same partition (cluster). For instance, imagine we have tensor <code>a</code> of size <code>(50, 5, 16)</code> and we have <code>lst = [0, 1, 1, 1, 2]</code>. I want an output <code>b</code> of size <code>(50,3,16)</code> where <code>b[:, 0, :] = a[:, 0, :]</code>, <code>b[:, 1, :] = mean of (a[:, 1, :], a[:, 2, :], a[:, 3, :])</code> and <code>b[:, 2, :] = a[:, 4, :]</code>.</p>
<p>What would be an efficient way of implementing this in PyTorch? I appreciate your time.</p>

username: tom
cooked: <p>I’d look at the third-party package <a href="https://pytorch-scatter.readthedocs.io/en/latest/functions/scatter.html" rel="noopener nofollow ugc">PyTorch scatter</a>. It has a reduction=mean mode.<br>
You need to convert lst to a tensor and possibly use broadcasting.<br>
Now, the scatter implementation uses atomics, which is problematic e.g. in terms of performance. If the partitions are ordered (as your example suggests), you might compare to just doing a for loop and taking means over the slices.</p>
<p>Best regards</p>
<p>Thomas</p>
Setting seed in torch DDP

username: ptrmcl
cooked: <p>Hi all,</p>
<p>I’m confused about how to set the seed in torch DDP. I know that the models must be initialized to the same parameter values across processes, so the seed must be the same.</p>
<p>Looking at the <a href="https://github.com/facebookresearch/deit/blob/main/main.py#L182" rel="noopener nofollow ugc">code from the DeiT repo</a> I see the line <code>seed = args.seed + utils.get_rank()</code>.</p>
<p>Doesn’t this mean we have a different seed for each process? So wouldn’t the models in each process be initialized differently?</p>

username: rvarm1
cooked: <p>Yes, in that case models on each rank would be initialized with different values.</p>
<p>However at startup time DDP broadcasts model parameters from rank 0 to ensure all ranks start training with the same model params, so setting a seed is not needed, unless you want determinism across different training runs with regard to the model params.</p>
Sequential batched matmul on GPU

username: kkm99
cooked: <p>I want to compute <code>C = torch.mm(A, B)</code> where <code>A</code> is a sparse tensor of size <code>(10,000,000, 100,000)</code> and <code>B</code> is a dense matrix of size <code>(100,000, 50)</code> on GPU. However, the operation runs out of memory on GPU.</p>
<p>I am thinking about computing the product between severals rows of <code>A</code> and <code>B</code> (something like <code>torch.mm(A[start_row:end_row, :], B)</code>) on GPU a time, and clear the GPU memory and repeat the operation on next few rows.  After all operation is done, stack all the result to recover the matrix multiplication.</p>
<p>For this approach, I am concerned about the overhead of transferring rows and the sequential nature.  Is there any efficient way to do the operation using built-in functions?</p>
<p>Thanks for any advice!</p>
Softmax for multiclass segmentation

username: drydenwiebe
cooked: <p>Hello,</p>
<p>I have a tensor representing multi class semantic segmentation that is the output of my network. It is of the shape  [B, C, H, W]</p>
<p>Where B is the batch size, C is the number of classes, H is the image height and W is the image width. Right now I want these to be a probability distribution over the number of classes per pixel (for each image in the batch), how would I accomplish this with torch.nn.Softmax.</p>
<p>Thanks!</p>

username: ptrblck
cooked: <p>You could use:</p>
<pre><code class="lang-python">B, C, H, W = 2, 3, 4, 4
x = torch.randn(B, C, H, W)

act = nn.Softmax(dim=1)
out = act(x)
print(out.sum(1))

out = F.softmax(x, dim=1)
print(out.sum(1))
</code></pre>

username: drydenwiebe
cooked: <p>That makes sense, thank you!</p>
speed up unparallelizable for loop

username: Amir90
cooked: <p>I built a network in pytorch, and upon profiling, saw that ~90% of the work is done in a for loop in one of my blocks. The problem is that this loop is not parallelizable, due to dependency on the previous values that were masked by mask1 (see MWE bellow).</p>
<p>I tried compiling it using <span class="mention">@torch.jit.script</span> and the speedup was negligible at ~0.5s. I am doing the minimal amount of work in the loop, anything else was vectorized. Attached is a MWE with the sizes of tensors that I am working with.</p>
<p>Will writing the code in C++ be better than what torchscript does? Is there any other way to considerably improve the runtime of the loop?<br>
Thanks you.</p>
<pre><code class="lang-auto">import torch
import time

n = 11300
f = 7800
batch_size = 10
device = "cuda:3" if torch.cuda.is_available() else "cpu"

inp1 = torch.randint(0, n, size=[batch_size, n, 4], device=device)
inp2 = torch.randint(0, n, size=[batch_size, f, 3], device=device)
inp3 = torch.randint(0, f, size=[batch_size, n, 2], device=device)
inp4 = torch.randint(0, n, size=[batch_size, n, 2], device=device)
inp5 = torch.randint(0, n, size=[batch_size, n, 2], device=device)


batch_list = torch.arange(batch_size, device=device)

mask1 = torch.ones([batch_size, n], dtype=torch.bool, device=device)
mask2 = torch.ones([batch_size, n], dtype=torch.bool, device=device)
mask3 = torch.ones([batch_size, n], dtype=torch.bool, device=device)

ind=0
start_time = time.time()
for i in range(n):
    if torch.all(~mask1[:, i]):
        continue
    batch_list_tmp = batch_list[mask1[:, i]]
    mask2[mask1[:, i], i] = False
    mask3[batch_list_tmp.unsqueeze(-1), inp1[:, i][:, ::2][batch_list_tmp]] = 0
    mask1[batch_list_tmp, i] = False
    concat_list_1 = inp5[
        batch_list.unsqueeze(-1), inp4[batch_list.unsqueeze(-1), inp1[batch_list, i]].view([batch_size, -1])]
    concat_list_2 = inp5[batch_list.unsqueeze(-1).unsqueeze(-1), inp4[
        batch_list.unsqueeze(-1).unsqueeze(-1), concat_list_1[batch_list]].view([batch_size, 8, -1])[
        batch_list]].view([batch_size, 8, -1])
    closure = inp2[batch_list.unsqueeze(-1).unsqueeze(-1),
                           inp3[batch_list.unsqueeze(-1).unsqueeze(-1),
                              concat_list_2[batch_list]].view(batch_size, 8, -1)[batch_list]].view([batch_size, 8, -1])
    mask1[batch_list_tmp.unsqueeze(-1), closure.view([batch_size, -1])[batch_list_tmp]] = False
end_time = time.time()

print(end_time-start_time) # takes ~5-7 seconds on server
</code></pre>
Dataloader using python thread (as opposed to multiprocess)

username: isaacg
cooked: <p>Is there a way to make Dataloader use python threads instead of multiprocess?</p>

username: ptrblck
cooked: <p>No, there is currently no way to use multi-threads. As far as I know, a multi-threaded loader would run into the Python GIL, which would yield a bad performance, thus multi-processing is used.</p>

username: isaacg
cooked: <p>I think there’s a couple catches to the GIL where you can still get good parallel performance using python threads.  I could be wrong, but here is what I believe to be true:</p>
<ol>
<li>The GIL simulates compute bound tasks of multiple threads through preemptive multitasking. So if you have N processors, you will <em>not</em> get an N-fold speedup.</li>
<li>The GIL will yield threads if they get put into the wait state from I/O bound operations. This is handy if you are reading off of a network drive with large latency. This is my situation which is why I asked.</li>
<li>The GIL can be released by lower level libraries effectively removing the issue with item 1. I believe OpenCV does this.  So, my thought is, if my understanding is correct, using OpenCV to do your data augmentation should yield the GIL and thus using multiple threads should give you a speedup.</li>
</ol>
<p>I am developing in windows and have found multiprocess to be a bit of a headache.  For example, different versions of python copy over different parts of the context when making a new process.  In generally, process making on Windows isnt as slick as it is on Linux.</p>
<p>Anyone out there reading this, please let me know if something I stated above is incorrect.</p>
<p>Thanks,<br>
Isaac</p>

username: Django
cooked: <p>I am interested in the same topic, if my dataloader is IO bound (for example because I am reading a lot of small arrays from a slow hard drive) I believe that IO performances would greatly increase using more workers then avialable cpu cores, thus having a thread based data loader could make sense.</p>
How to speed up training on ImageNet

username: jS5t3r
cooked: <p>I try to figure out to train a classifier on ImageNet… because I want to train classifiers with different image sizes…<br>
To do so, I am using this <a href="https://github.com/pytorch/examples/tree/master/imagenet" class="inline-onebox" rel="noopener nofollow ugc">examples/imagenet at master · pytorch/examples · GitHub</a> as a framework.<br>
When I train it on the ImageNet, it takes around 16 hours per epoch on an A100, which is rather slow.</p>
<p>How to improve training speed besides adjusting the numbers of workers? I am also upgrading to Python 3.9…<br>
Do I have to go through all samples from the Validation set? Can I just take subset?</p>

username: hkz
cooked: <p>The post below is all you need…</p><aside class="quote quote-modified" data-post="19" data-topic="548">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/b9e5f3/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-to-prefetch-data-when-processing-with-gpu/548/19">How to prefetch data when processing with GPU?</a> 
  </div>
  <blockquote>
    I’ve spent some time working on this problem over a variety of projects. I’ve cut and pasted some past thoughts, bullets, etc from previous discussions. My background involves architecting systems that move large volumes of data from network cards to storage and then back again on request, with processing between the steps. A very similar set of concerns. 
The two main constraints that usually dominate your PyTorch training performance and ability to saturate the shiny GPUs are your total CPU IP…
  </blockquote>
</aside>
How to install pytorch 1.3.0 or above with cuda 8?

username: Peixiang_Zhong
cooked: <p>I have an old server with NVIDIA driver 375.26, which only supports cuda8.0 as indicated in <a href="https://docs.nvidia.com/deploy/cuda-compatibility/index.html" class="inline-onebox" rel="noopener nofollow ugc">CUDA Compatibility :: GPU Deployment and Management Documentation</a></p>
<p>The server has a P100 card, and I cannot upgrade the driver for various reasons.</p>
<p>How could I install PyTorch 1.3.0 or above with this setup?</p>

username: ptrblck
cooked: <p>The current min. CUDA version is 9.2 and CUDA8.0 is not tested, so I don’t know if you could be lucky in building it from source (have you tried it and was it failing?).</p>

username: Peixiang_Zhong
cooked: <p>Thank you for your reply.</p>
<p>I haven’t tested building it from source. I decided to use the cpu version for now.</p>
Plotting individual hidden nodes using Pyplot

username: jordanjsg
cooked: <p>Hi, I want to create a function graph_hidden(net, layer, node), that takes a neural network, a layer (either 1 or 2) and a node (number from 0 to num. nodes) and plot the activation of the node.</p>
<p>I have this code so far but it keeps encountering errors, any advice would help greatly!</p>
<pre><code class="lang-auto">xrange = torch.arange(start=-8,end=8.1,step=0.01,dtype=torch.float32)
yrange = torch.arange(start=-8,end=8.1,step=0.01,dtype=torch.float32)
xcoord = xrange.repeat(yrange.size()[0])
ycoord = torch.repeat_interleave(yrange, xrange.size()[0], dim=0)
grid = torch.cat((xcoord.unsqueeze(1),ycoord.unsqueeze(1)),1)

with torch.no_grad(): # suppress updating of gradients
net.eval()        # toggle batch norm, dropout    
if (layer == 1):
    output = net.hid1[node](grid)
elif (layer == 2):
    output = net.hid2[node](grid)
net.train() # toggle batch norm, dropout back again

pred = (output &gt;= 0.5).float()

# plot function computed by model
plt.clf()
plt.pcolormesh(xrange,yrange,pred.cpu().view(yrange.size()[0],xrange.size()[0]),
               cmap='Wistia', shading='auto')
</code></pre>
Input data != target data for LSTM

username: Laura_Montalvo
cooked: <p>Hello there!<br>
I have some doubts about LSTMs and how to preserve their temporality. Let me explain a little bit:</p>
<p>I have as input a trajectory of 12 positions with different information: my input data is (batch_size, 12, 110) where 110 are 110 columns with different information about that position.<br>
As output/target I have the lat/lon coordinate: target/output data is (batch_size, 12, 2) where 2 represents the latitude and longitude of the position.</p>
<p>Once trained the network I represent the coordenates into the map and what I see is that the sequential benefit of LSTM is not being taken advantage of.<br>
I’ve thought about teacher forcing but before that I wanted to know if there’s a way of exploiting sequential information this way in LSTM architecture… Any idea?</p>
<p>Thanks!!</p>
Padding_mode in nn.Unfold

username: decadenza
cooked: <p>This is possibly a feature request.</p>
<p>I’m using <code>nn.Unfold</code> for a custom <code>nn.Conv2d</code> layer. Found the following small issue.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

a = torch.arange(9, dtype=float).reshape(1,1,3,3)
unfold = nn.Unfold(kernel_size=2)
unfold(a)
</code></pre>
<p>Output is</p>
<pre><code class="lang-auto">tensor([[[0., 1., 3., 4.],
         [1., 2., 4., 5.],
         [3., 4., 6., 7.],
         [4., 5., 7., 8.]]], dtype=torch.float64)
</code></pre>
<p>doing instead</p>
<pre><code class="lang-auto">unfold = nn.Unfold(kernel_size=2, padding=1)
unfold(a)
</code></pre>
<p>Output then is</p>
<pre><code class="lang-auto">tensor([[[0., 0., 0., 0., 0., 0., 1., 2., 0., 3., 4., 5., 0., 6., 7., 8.],
         [0., 0., 0., 0., 0., 1., 2., 0., 3., 4., 5., 0., 6., 7., 8., 0.],
         [0., 0., 1., 2., 0., 3., 4., 5., 0., 6., 7., 8., 0., 0., 0., 0.],
         [0., 1., 2., 0., 3., 4., 5., 0., 6., 7., 8., 0., 0., 0., 0., 0.]]],
       dtype=torch.float64)
</code></pre>
<p>As you can see, nn.Unfold automatically considers padding as <code>padding_mode='zeros'</code>.<br>
It does not accept parameters other than <code>kernel_size</code>, <code>dilation</code>, <code>padding</code> and <code>stride</code>.</p>
<p>How should I implement a <code>replicate</code> mode?</p>
<p>Regards</p>
<p>EDIT<br>
Of course I can solve the above using <code>torch.nn.functional.pad</code>. Doing so it will probably create an extra copy though…</p>
Multi-GPU training using multiple Inputs

username: blueeagle
cooked: <p>Hi everyone,</p>
<p>i implemented an architecture that handles multiple inputs, each being processed by its own encoder. In order to speed things up, I want to train my model on multiple gpus. This is my code:</p>
<pre><code class="lang-auto">def forward(self, x):
''' x: list of input tensors '''
h = list()
        for i, x_i in enumerate(x):
            h_i = self.encoders[i](x_i)
            h.append(h_i)
z = torch.cat(h, dim=0)
y_pred = self.classifier(z)
</code></pre>
<p>If I would simply use data_parallel class here, each encoder would get copied to each GPU, however I think it would be faster, if each encoder would be trained on its own GPU. Is there any possibility to achieve this?</p>
<p>Thank you!</p>

username: ptrblck
cooked: <p>Yes, you can push each model to a specific GPU via <code>self.encoders[i].to('cuda:{}'.format(i))</code> (where <code>i</code> would be the GPU id) as well as the input, transfer the outputs back, and concatenate the final output tensor.</p>

username: blueeagle
cooked: <p>Thank you! I think this is exactly what I was looking for.</p>
Multiple workers for single batch

username: lorenz-gorini
cooked: <p>Hi everyone!<br>
I am working on a project with histopathological images (called Whole-Slide Image). Each of these images is ~1 GB, so they are really hard to handle.<br>
Particularly I struggle when I use <code>DataLoader(num_worker=N)</code> (where N&gt;1) because PyTorch starts loading and preprocessing (slight data augmentation in our case) multiple batches in RAM and then the RAM fills up fast.<br>
I wanted to know if there are other people working on implementing an alternative <code>DataLoader</code> mechanism that could allow us to have multiple workers working on the same batch.<br>
I would also like to know if you have any suggestions regarding this topic.<br>
Since I never opened a Pytorch PR and since I noticed that worker shutdown/handling is a very delicate matter, do you think I could open a draft and then someone could provide some suggestions and support?</p>

username: ptrblck
cooked: <p>This feature request is tracked <a href="https://github.com/pytorch/pytorch/issues/24985">here</a> and I’m sure contributions are welcome, so please feel free to post your interest in the issue and code owners will follow up with you. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>