akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Detach tensors from computation graph

jingyu_han: <p>I have a network that returns two different outputs.</p>
<pre><code class="lang-auto">class Deeplab_v3plus(nn.Module):
    def __init__(self, cfg, *args, **kwargs):
        super(Deeplab_v3plus, self).__init__()
        self.backbone = Resnet101(stride=16)
        self.aspp = ASPP(in_chan=2048, out_chan=256, with_gp=cfg.aspp_global_feature)
        self.decoder = Decoder(cfg.n_classes, low_chan=256)
        #  self.backbone = Darknet53(stride=16)
        #  self.aspp = ASPP(in_chan=1024, out_chan=256, with_gp=False)
        #  self.decoder = Decoder(cfg.n_classes, low_chan=128)

        self.init_weight()

    def forward(self, x):
        H, W = x.size()[2:]
        feat4, _, _, feat32 = self.backbone(x)
        
        feat_aspp,features = self.aspp(feat32)
        
        logits = self.decoder(feat4, feat_aspp)
        logits = F.interpolate(logits, (H, W), mode='bilinear', align_corners=True)
        
        for f in features:
            f.detach_()
            
        return logits, features
</code></pre>
<p>Now I only need the first one to participate in calculating our loss.</p>
<pre><code class="lang-auto">
        optim.zero_grad()
        lodits,_= net(im)
        
        loss = criteria(logits, lb)
        loss.backward()
        optim.step()
</code></pre>
<p>the second output(<code>features</code>) will be used in another network that running simultaneously, so at this moment, it’s unnecessary to join in calculating my loss.</p>
<p><strong>Question:</strong><br>
when launching our program, I always get such an error as follows:</p>
<p>`</p>
<blockquote>
<p>Expected to have finished reduction in the prior iteration before starting a new one. This error indicates that your module has parameters that were not used in producing loss. You can enable unused parameter detection by (1) passing the keyword argument <code>find_unused_parameters=True</code> to <code>torch.nn.parallel.DistributedDataParallel</code>; (2) making sure all <code>forward</code> function outputs participate in calculating loss. If you already have done the above two steps, then the distributed data parallel module wasn’t able to locate the output tensors in the return value of your module’s <code>forward</code> function</p>
</blockquote>
<p>`</p>
<p>Then I tried to detach “features” from my computation graph, however, it’s failed. Any cues would be highly appreciated!!! thank you in advance.</p>
Non Max Suppression and Object Detection

Moe_Kaung_Kin: <p>My understanding on how non max suppression work is suppress all overlapping boxes that are over  jaccard overlap threshold (may be 0.5). The boxes to be considered are on confident score (may be 0.2 or something). My knowing is if there is boxes that got score over 0.2 (may be score is 0.3 and overlap is 0.4) the boxes won’t suppress. By this way one objects will be predicted by many boxes,one high score box and many low confident score boxes but I found that the model predict only one box for one object. Can someone enlighten me?<br>
I currently viewing the ssd from  <a href="https://github.com/sgrvinod/a-PyTorch-Tutorial-to-Object-Detection" rel="nofollow noopener">https://github.com/sgrvinod/a-PyTorch-Tutorial-to-Object-Detection</a></p>
<p>Here is the coding…</p>
<pre><code>class_scores, sort_ind = class_scores.sort(dim=0, descending=True)  # (n_qualified), (n_min_score)
class_decoded_locs = class_decoded_locs[sort_ind]  # (n_min_score, 4)
overlap = find_jaccard_overlap(class_decoded_locs, class_decoded_locs)  # (n_qualified, 
n_min_score)
suppress = torch.zeros((n_above_min_score), dtype=torch.uint8).to(device)  # (n_qualified)

for box in range(class_decoded_locs.size(0)):
# If this box is already marked for suppression
       if suppress[box] == 1:
       continue
       suppress = torch.max(suppress, overlap[box] &gt; max_overlap)
       suppress[box] = 0</code></pre>
From torchvision import _C as C

lyh: <p>When i import torchvision._C,the machine report “DLL load failed”.<br>
My computer is Windows 10-Home<br>
Anaconda<br>
torch-1.3.1<br>
torchvision-0.4.1<br>
Thanks a lot!</p>

lyh: <p>Its done.Just install according to the official website.<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://pytorch.org/favicon.ico?" class="site-icon" width="16" height="16">
      <a href="https://pytorch.org/get-started/locally/" target="_blank" rel="nofollow noopener">pytorch.org</a>
  </header>
  <article class="onebox-body">
    <img src="https://pytorch.org/assets/images/pytorch-logo.png" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://pytorch.org/get-started/locally/" target="_blank" rel="nofollow noopener">PyTorch</a></h3>

<p>An open source deep learning platform that provides a seamless path from research prototyping to production deployment.</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
Implementing sech

G.M: <p>I’m trying to implement <em>sech</em> function, but Pytorch doesn’t have an internal implmentation for it as <em>tanh</em> does. I came up with two solutions:</p>
<pre><code class="lang-auto">import torch as tc
import  math
</code></pre>
<ol>
<li>
</li>
</ol>
<pre><code class="lang-auto">tanh_y = tc.tanh(input)
sech_y = tc.sqrt(1-a.pow(2))
#if it's in autograd.Function, I can then use the tanh_y and sech_y for backward.
</code></pre>
<ol start="2">
<li>
</li>
</ol>
<pre><code class="lang-auto">a, b = tc.pow(math.e, input), tc.pow(math.e, -input)
#or:
#a = input.exp()
#b = a.reciprocal()
sech_y = 2/(a+b)
#tanh_y = (a-b)/(a+b)
</code></pre>
<p>I’m just curious about which one is better(perhaps faster), or does it not quite make any difference.<br>
Thanks for your time.</p>

albanD: <p>Hi,</p>
<p>Given all the formulas I found online, <code>out = 1/torch.cosh(z)</code> should be the best (less total number of operations).</p>

G.M: <p>thx. I didn’t know there was a cosh function in Pytorch <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"></p>
Jit enabling requires grad

vishalagarwal: <p>using traced model after jit compilation to do forward pass, <code>torch.clamp()</code> seems to introduce <code>requires_grad</code> to be True to a forward. In normal mode, it is fine but when I use the traced model this happens. To confirm <code>torch.clamp()</code> is introducing this I print <code>x.requires_grad</code> before and after the clamp operation.<br>
This bug seems weird. Any fix?</p>

ptrblck: <p>Do you have a minimal code snippet, as I cannot reproduce this issue?</p>
What is the difference between opt.zero_grad() and model.zero_grad()?

Wong: <p>Hi everyone,</p>
<p>We have a question. What is the difference between optimizer.zero_grad() and model.zero_grad()? Do they need to be defined together when programming?</p>
<p>Thanks.</p>

ptrblck: <p>While creating an optimizer, you could pass parameters to it.<br>
In a lot of cases, you would just pass all model parameters to a single optimizer, so both calls will yield the same result (zeroing out the gradients of all parameters).</p>
<p>However, you could also pass the first half of the model parameters to <code>optimizer1</code> and the second to <code>optimizer2</code>.<br>
In this use case, <code>model.zero_grad()</code> would zero out the gradients of all model parameters, while <code>optimizerX.zero_grad()</code> would only zero out the gradients of its passed parameters.</p>
Aws s3 and SQL support

Gourav_Sengupta: <p>Hi,<br>
I find it pretty scary that Pytorch does not have loaders for SQL or aws S3 or Parquet formats, does this mean no one is actually using it for large scale data sets?</p>
<p>Is any one using Pytorch for Kafka streams or other streaming systems?</p>
<p>Theoretically we should be able to build these components on our own, so can we write our own operating system, and develop our own programming language. But should we not expect to see a certain level of maturity from a tool that is competing with Tensorflow?</p>
<p>Regards,<br>
Gourav Sengupta</p>

sailfish009: <p>i think there is <a href="https://github.com/dmmiller612/sparktorch" rel="nofollow noopener">sparktorch</a>.</p>
The T/F judgement of GAN

stillbetter: <p>In StarGAN, the D model has 2 different output, judging the image from dataset or G and predicting the domain vector. My question is that since we have the predicted domain vector to direct the G, why we need the D to judging Ture or False? Or in other words, how, in StarGAN, T/F judgment can help the whole model capture more feature information than the n dimension domain vector?</p>
<p>I guess the T/F judgment is the essence of GAN model which build a bridge between D and G and keep the dynamic game. And discrete domain vector will make D concentrate on the specific image features while the T/F judgment score gives D the maximum degree of ‘freedom’ to capture the feature information.</p>
Torchaudio Spectrogram returning some data points as 0 and Log2 inf

Giuseppe_Sarno: <p>Hello,<br>
I followed the tutorial at <a href="https://pytorch.org/tutorials/beginner/audio_preprocessing_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/audio_preprocessing_tutorial.html</a><br>
and applied log2 to the spectrogram. However one of my mp3 files is as such the Spectrogram returns data = 0. This makes the log2 to return inf.<br>
Is this expected ? or is there something it can be done to avoid this scenario?</p>
<p>Thanks.</p>

vincentqb: <p>This is expected. Do you need to apply the log transformation? If so, you can also add a little value before taking log:</p>
<pre><code class="lang-auto">epsilon = 1e-6
log2(epsilon + specgram)
</code></pre>

Giuseppe_Sarno: <aside class="quote no-group" data-username="vincentqb" data-post="2" data-topic="66566">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vincentqb/48/20169_2.png" class="avatar"> vincentqb:</div>
<blockquote>
<p>epsilon = 1e-6</p>
</blockquote>
</aside>
<p>Thank you, will need to do something like this.</p>
<p>The odd problem I have is the following:</p>
<p>I am converting / comparing 2 sound files. One WAV and the other the MP3 of the same file then create the spectrogram and convert to image.<br>
I get the issue with the MP3 version of the file while everything is ok for WAV (noticed this consistently with other audios).<br>
The other key difference is that because the first data points for each row are very small compared to the others when I normalize the pictures to RGB (0 -255) the image is very faint. While the pictures generated with a Wav file is good. I normalize the waveform, spectrum and the image.<br>
Are there techniques to avoid this problem ? any particular reason why the MP3 version has this different behavior ?</p>
<p>Thanks,</p>

vincentqb: <p>Can you provide a minimal code and files to reproduce? Is the mp3 converted from the wav or vice-versa?</p>

Giuseppe_Sarno: <p>Hi vincentqb,<br>
I created a new topic as the problem is now different to the question on this topic <a href="https://discuss.pytorch.org/t/spectrogram-to-rgb-pictures-for-resnet-faint-image-with-mp3-files/66909">spectrogram-to-rgb-pictures-for-resnet-faint-image-with-mp3-files</a></p>
I there anyway to just get the sorted value without indices?

Kevin96: <p>When I sort a huge matrix, using torch.sort(), I encountered cuda out of memory error.</p>
<p>In my case, I only need the sorted value, i.e. torch.sort(tensor)[0]. While torch.sort returns two tensors, which might be memory consuming for me.</p>
<p>I am wondering if there is other functions or options to use to get rid of that. Thanks!</p>
How to implement TD(λ)

Tyler_Walker: <p>i have a vanilla NN that looks like this</p>
<pre><code class="lang-auto">self.model = torch.nn.Sequential(
		 	torch.nn.Linear(self.INPUT_SIZE, self.HIDDEN_SIZE),
		 	torch.nn.Sigmoid(),
		 	torch.nn.Linear(self.HIDDEN_SIZE, self.OUTPUT_SIZE),
		 	torch.nn.Sigmoid())
</code></pre>
<p>i want to implement TD(λ) between steps.  λ is a constant btw 0, 1 that sets lifespan of a gradient trace.<br>
psuedo-code for this is</p>
<pre><code class="lang-auto">loss.backward()
model.gradients = model.gradients + λ * model.previous_gradients
optimizer.step()
model.previous_gradients = model.gradients
</code></pre>
<p>i think i might be able to accomplish this using gradient hooks, but i’m unsure how to do that.  i can’t shake the feeling that it might be easier than that.  SGD’s momentum is mathematically similar, but i can’t tell if it’s identical.</p>

alexis-jacq: <p>As far as I understand, using gradient hook should only allow you to modify the new gradient. But you want to modify the previous one, so I don’t see a simpler way than:</p>
<pre><code class="lang-auto"># multiply previous gradient by lambda:
for p in model.parameters():
    p.grad *= lambda
loss.backward() # add the new gradient
optimizer.step()
</code></pre>

Tyler_Walker: <p>oh, that’s interesting!  then never call optimizer.zero_grad()<br>
thanks.</p>

Dhawgupta: <p>I am pretty confused on how to implement TD(lambda), can you tell me what is your loss function, i thougth that there were no losses for which you get the TD Lambda gradient ?</p>
How to handle BPTT with mini-batches

Mohamed_Nassar: <p>Hello All,<br>
I have the following training loop</p>
<pre><code class="lang-auto">for epoch in range(num_epochs): 
  hidden = model.init_hidden()
  for startidx in range(0, num_batches, batch_size):
    endidx = startidx + batch_size
    step = startidx//batch_size
    xbatch = xhot_seq[startidx:endidx]
    ybatch = yhot_seq[startidx:endidx]

    # Forward pass

    # Clear stored gradient
    model.zero_grad()
    
    y_pred, hidden = model(xbatch, hidden)
    target = torch.argmax(ybatch.long(),dim=1)
    loss = loss_fn(y_pred, target )
    loss_hist[epoch] = loss.item()
    # Backward pass
    loss.backward()
    # Update parameters
    optimiser.step()
</code></pre>
<pre><code class="lang-auto">  def forward(self, input, hiddenState):
    (h_state,c_state) = hiddenState

    lstm_out, (h_state,c_state) = self.lstm(input, (h_state,c_state))

    out = self.linear(lstm_out[:,-1,:])
    y_pred = self.softmax(out)

    return y_pred, (h_state,c_state)
</code></pre>
<p>my input has the shape of [batch size, seq length, num of features]</p>
<p>Now does backpropagation consider hidden state between sequence to sequence inside the mini-batch, i.e if my batch size is 16, in the last input sequence will the model backpropagate through sequence length only (in my case i set seq length = 40) or through my sequence length and the previous 15 sequences.<br>
Note that i’m restting the hidden state each epoch. Also what should i do to make my model statefull ?<br>
should i never reset the hidden states ?</p>
Load 300GB across 200 `.npy` files with Dataset and DataLoader

Michael_Guarino1: <p>Hi I have a large dataset (~300GB) stored across ~200 numpy files (<code>.npy</code>). I wrote my own custom Dataset class to load a numpy file and batch it dynamically. I would like to do this because I don’t want to load all ~200 numpy files at once as RAM is limited. Is this is right tool for this use case and if so what do you recommend I do to make this work. Thank you in advance!</p>
<pre><code class="lang-auto">from torch.utils.data import Dataset, DataLoader

class MYDS(Dataset):
    def __init__(self, x_paths, y_paths):
        self.xs = None
        self.ys = None

        for x_path, y_path in list(zip(x_paths, y_paths)):
            print(f'loaded: {x_path}, {y_path}')
            self.xs = torch.from_numpy(np.load(x_path))
            self.ys = torch.from_numpy(np.load(y_path))

    def __len__(self):
        return len(self.xs)

    def __getitem__(self, idx):
        return [self.xs[idx], self.ys[idx]]

if __name__=='__main__':
    
    x_paths = ['2018-05_part-00000_train_x.npy', '2018-05_part-00001_train_x.npy', '2018-05_part-00002_train_x.npy'] # small subset of actual dataset
    y_paths = ['2018-05_part-00000_train_y.npy', '2018-05_part-00001_train_y.npy', '2018-05_part-00002_train_y.npy'] # small subset of actual dataset

    ds = MYDS(x_paths, y_paths)
    dl = DataLoader(ds, batch_size=128, shuffle=False, num_workers=2)

    for batch_idx, (x, y) in enumerate(dl):
        print(batch_idx, x[0], y[0]) # loading batches only from x_paths[-1] and y_paths[-1] numpy files
</code></pre>

JuanFMontesinos: <p>I would go forward numpy memory map. You can create a dataset per numpy file and then just use a concat_dataset to mix them all. That would be the easiest way.</p>

Michael_Guarino1: <p>Ok so I could create a Dataset per numpy file then use np.memmap  to partially load each numpy file to ensure I don’t load everything into ram all at once. Would also need a DataLoader per Dataset?</p>

JuanFMontesinos: <p>Not really, there is a class called dataset_concat already written. It is instantiated passing all the datasets and it will manage choose them as if a Dataset class were so that len_total = sum(len_i) and so on.</p>
<p><a href="https://pytorch.org/docs/stable/data.html?highlight=concat#torch.utils.data.ConcatDataset" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/data.html?highlight=concat#torch.utils.data.ConcatDataset</a></p>
Are combined views of multiple tensors possible?

cdhernandez: <p>I need to add an arbitrary penalization to the weights of my network but the normal way of doing this, i.e.</p>
<blockquote>
<p>pen=0<br>
for p in net.parameters():<br>
pen+=penaltyfunc ( p )</p>
</blockquote>
<p>is a bit too slow at high layer counts</p>
<p>to solve this I want to create a view(-1) of all the weight/bias matrices and then combine them into a new tensor. From there I can the whole penalization in one step. Normally with a view, the value of the view changes along with whatever you are viewing. However I’m not sure how to combine all these views into a single view. if I use the cat command, it returns a snapshot of the values of the views, i.e. it won’t change as the actual weights change.</p>
<p>Is there a way to do this or do i just need to execute the cat command each time I want to calculate the penalty?</p>
<p>I’ve tried searching for view multiple tensors, multiview, view+cat, but nothing really relevant comes up.</p>

albanD: <p>Hi,</p>
<p>We used to do this in <code>torch7</code>: you had a specific function on the model that was making all the weights a view into a single big buffer. The short version is that this is a nightmare to maintain properly.<br>
Anything you will do that copy a Tensor will break that link and then your optimizer (or penalty function here) will look at different versions of your weights and everything silently break.</p>
<p>In general, doing this for-loop should not be too slow. Is you model very different from general CNN?</p>

cdhernandez: <p>No its a combination of convolutions, pools and fully connected layers so fairly straightforward, but its likely to get weirder once we get the base working and we try new things. The issue is this:</p>
<p>My benchmarks:<br>
slowpen<br>
1000 loops, best of 3: 1.88 ms per loop<br>
100 loops, best of 3: 1.82 ms per loop<br>
fastpen<br>
1000 loops, best of 3: 258 µs per loop<br>
1000 loops, best of 3: 443 µs per loop<br>
create<br>
10000 loops, best of 3: 132 µs per loop</p>
<p>So currently I’m currently creating a list of views and then using the stack command to combine them and take a snapshot each time i want to calculate the penalty. The speed for the forward/backward steps is listed above. Its already about 7 times faster than the for loop method and this only gets worse as the network gets deeper.<br>
More importantly, that cat step takes like half the calculation time (‘create’) for the forward pass.</p>
<p>Further, this is only for a &lt;10 layer network, the time only gets worse when I added more layers (making them wider doesn’t make a huge difference though).</p>
<p>All I need is a way to combine 2 view objects into a new view object without breaking their respective links. If that command exists thats all I need but thus far haven’t been able to find it (if it exists). I don’t need to copy the whole buffer, just do a series of straightforward calculations on it like various norms or means.</p>
Dist.init_process_group works but TCPStore failed

forhonourlx: <p>Hi,<br>
I am trying init dist and get stuck.<br>
I have 2 nodes: master and slave, both pytorch 1.3.1 installed by anaconda<br>
It works on both when:</p>
<pre><code class="lang-auto">    dist.init_process_group(
            backend ="NCCL",
            world_size = 2,
            rank = 0,# 0 for master and 1 for slave
            init_method="tcp://192.168.1.102:23458"#master addr and port
            )
</code></pre>
<p>It is hung up on both when:</p>
<pre><code class="lang-auto">store = dist.TCPStore("192.168.1.102", 23458, 2, 0)
</code></pre>
<p>Could somebody help?<br>
Thanks in advance and Happy New Year.</p>

mrshenli: <p><a href="https://github.com/pytorch/pytorch/blob/20e5c90d8246aac61f70e84c78f0dd9670b76987/torch/distributed/rendezvous.py#L125" rel="nofollow noopener">This</a> is how TCPStore is initialized when you call <code>init_process_group</code>. You can print out the args on both process to check which one went wrong.</p>
<p>For <code>start_daemon</code>, did you pass in 1 for rank0 and 0 for rank1?</p>
Deterministic problem when infer same input

xinzheshen: <p>When i infer the same input, the output is not deterministic sometimes, and the code is as below.<br>
Debug found that the posterior is same, but the sample is different for same input sometimes.</p>
<pre><code class="lang-auto">                    posterior = F.softmax(logits, dim=1)                  
                    distrib = torch.distributions.Categorical(posterior)
                    sample = distrib.sample().float()
                    
</code></pre>
<p>I execute model.eval() and have set the seed at the begining. Do you have any suggestions? Thank you.</p>
<pre><code class="lang-auto">seed = 1234
torch.manual_seed(seed)
torch.cuda.manual_seed(seed)
torch.cuda.manual_seed_all(seed)
np.random.seed(seed)
random.seed(seed)
torch.backends.cudnn.benchmark = False
torch.backends.cudnn.deterministic = True
</code></pre>

mrshenli: <p>Can you ask this question with Uncategorized label. It seems there is some randomness somewhere, but <code>torch.distributed</code> (not <code>torch.distributions</code>) is not involved here.</p>
Using custom method in distributed model

bigyeet: <p>Not sure if this is the right way to do it. I’m wrapping my model in nn.DataParallel for multi gpu training. There is an LSTM module as part of this model. This LSTM module has a custom method that resets the hidden states, called after each time a forward pass is done during training. This is the only custom method that’s used.</p>
<p>To access this reset method in the parallel model, I do<br>
<code>model.module.lstm.reset_hidden_state()</code><br>
Whereas if my model is <strong>not</strong> wrapped in DataParallel, it would just be<br>
<code>model.lstm.reset_hidden_state()</code></p>
<p>Is this right, or do I have to write a custom DataParallel wrapper that has scatter, gather, etc methods? If so, how would I do it?</p>
<p>This is the lstm module:</p>
<pre><code class="lang-auto">class LSTM(nn.Module):
    def __init__(self, latent_dim, num_layers, hidden_dim):
        super().__init__()
        self.lstm = nn.LSTM(input_size=latent_dim, num_layers=num_layers, hidden_size=hidden_dim, batch_first=True, dropout=0.0)
        self.hidden_state = None
    def reset_hidden_state(self):
        self.hidden_state = None
    def forward(self,X):
        self.lstm.flatten_parameters()
        X, self.hidden_state = self.lstm(X, self.hidden_state)
        return X
</code></pre>

mrshenli: <aside class="quote no-group" data-username="bigyeet" data-post="1" data-topic="65410">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/dfb087/48.png" class="avatar"> bigyeet:</div>
<blockquote>
<p>Is this right, or do I have to write a custom DataParallel wrapper that has scatter, gather, etc methods? If so, how would I do it?</p>
</blockquote>
</aside>
<p>It depends on what you expected <code>reset_hidden_state</code> to achieve. Below is what happens in EVERY forward pass when you use DataParallel.</p>
<ol>
<li>split input data</li>
<li>replicate model to all devices</li>
<li>feed input data splits to all model replicas</li>
<li>gather outputs from all replicas</li>
<li>done with forward</li>
</ol>
<p>After the forward pass, the autograd graph actually contains multiple model replicas. It looks sth like</p>
<p>original model ← scatter ← model replicas ← replica output ← gather ← final output.</p>
<p>So in your above use case, if <code>reset_hidden_state</code> has any side effect that you would like to apply to the backward pass, it will only apply to the original model, not to model replicas. But if you are only trying to clear some states for the next forward pass, it should work.</p>
Can you control conv algorithm using CUDNN API?

Shubhankar: <p>I would like to know if you can choose the algorithm viz. FFT, Winograd etc. used for convolution in C++ API for Pytorch using CUDNN API as shown <a href="http://www.goldsborough.me/cuda/ml/cudnn/c++/2017/10/01/14-37-23-convolutions_with_cudnn/" rel="nofollow noopener">here</a>.</p>

albanD: <p>Hi,</p>
<p>This is not possible at the moment. There are a couple of issues open for this: <a href="https://github.com/pytorch/pytorch/issues/3667">https://github.com/pytorch/pytorch/issues/3667</a> and <a href="https://github.com/pytorch/pytorch/issues/18536">https://github.com/pytorch/pytorch/issues/18536</a></p>

Shubhankar: <p>Thanks I came across those and thought may be its possible to have fine grained control from C++ API instead of python. But it seems that C++ API just exposes the C++ backend of the same python frontend.</p>
Data splitting in DistributedDataParallel

juniper: <p>Hi,</p>
<p>I’m trying to use DistributedDataParallel on a CPU-only machine with multiple cores.</p>
<p>The documentation for DDP (<a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/parallel/distributed.py" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/torch/nn/parallel/distributed.py</a>) states: "For multi-device modules and CPU modules, device_ids must be None or an empty list, and <strong>input data for the forward pass must be placed on the correct device</strong>. (default: all devices for single-device modules).</p>
<p>I want to parallelize training across CPU processes in a single machine. My dataset is an in-memory numpy array.</p>
<p>Would I have to manually separate this dataset into different subsets, and load each subset for each CPU process? Does splitting the input along the batch dimension work for CPU modules as well? I am using torch’s multiprocessing module to spawn processes to use with my DDP model.</p>
<p>Thank you.</p>
<p>PS. What’s the best practice for sharing an in-memory array across torch processes? That would be helpful as well.</p>
<hr>
<p>Some additional observations:</p>
<p>When I print the loss in each process, the loss value is the same. If the data was being split properly by DDP, wouldn’t each process have a different loss value?</p>

juniper: <p>From my experiments, it appears that for DDP using CPU processes, there is no splitting of data across the batch dimension across processes.</p>
<p>In the source code as well, if the model’s device_ids is None, then scattering is not performed in the forward() pass of the model.</p>
<p>Can someone more authoritative confirm this behavior?</p>

mrshenli: <blockquote>
<p>In the source code as well, if the model’s device_ids is None, then scattering is not performed in the forward() pass of the model.</p>
</blockquote>
<p>Yes, this is correct.</p>
<p>Input data split only occurs in two situations:</p>
<ol>
<li>When using DataParallel (single-process multi-thread)</li>
<li>Using DistributedDataParallel (DDP), and provide a <code>device_ids</code> list of multiple CUDA devices. In this case, each DDP process will operate on multiple devices and multiple model replicas, and hence need to split the input data. (This is not recommended, as this could be slow)</li>
</ol>
<p>For the recommended use case of DDP (one device/replica per DDP process), DDP will NOT split input or distributed them into multiple processes. Each DDP process needs to read its own input data independently. You could try manually splitting those data (say on rank0) and pass them across processes though, if they are on the same machine. Or, I also saw many people using the <a href="https://pytorch.org/cppdocs/api/classtorch_1_1data_1_1samplers_1_1_distributed_sampler.html" rel="noopener nofollow ugc">DistributedSampler</a> to load input data</p>
Merging outputs of semantic segmentation model

adrjj: <p>HI guys,</p>
<p>In the context of semantic segmentation (19 classes) my model output a tensor of size :<br>
<code>(b, 19, h, w)</code>.</p>
<p>I need to merge some classes to get a tensor of size <code>(b, 10, h, w)</code> with respect to a mapping dictionary:</p>
<p><code>dic = {9:[0,1], 8:[3,4], 7:[9,10,11], 6:[6,7,8], 5:[2,5], 4:[18,17,16], 3:[4,5,6], 2:[12,13], 1:[15], 0:[14]}</code></p>
<p>What is the most efficient way to do that ?</p>
<p>Thanks !</p>

ptrblck: <p>I’m not sure, if you could use an alternative to a simple for loop <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>
<pre><code class="lang-python">b, h, w = 2, 4, 4
x = torch.randn(b, 19, h, w)
z = torch.zeros(b, 10, h, w)

dic = {9:[0,1], 8:[3,4], 7:[9,10,11], 6:[6,7,8], 5:[2,5], 4:[18,17,16], 3:[4,5,6], 2:[12,13], 1:[15], 0:[14]}

for key in dic:
    d = dic[key]
    z[:, key] = x[:, d].sum()
</code></pre>

adrjj: <p>Hi !</p>
<p>Thank you for your answer, yes I don’t see any other alternative I will go for the for loop then.</p>
<p>Thanks</p>
Zero_grad placement

jwillette: <p>I came across some code on github where there was a strange placement of the call to zero_grad and I couldn’t explain why it was there or if there was any difference in placing it a few lines earlier.</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/tristandeleu/pytorch-meta/blob/master/examples/maml/train.py#L45" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/tristandeleu/pytorch-meta/blob/master/examples/maml/train.py#L45" target="_blank" rel="nofollow noopener">tristandeleu/pytorch-meta/blob/master/examples/maml/train.py#L45</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="35" style="counter-reset: li-counter 34 ;">
<li>test_targets = test_targets.to(device=args.device)</li>
<li>
</li>
<li>outer_loss = torch.tensor(0., device=args.device)</li>
<li>accuracy = torch.tensor(0., device=args.device)</li>
<li>for task_idx, (train_input, train_target, test_input,</li>
<li>        test_target) in enumerate(zip(train_inputs, train_targets,</li>
<li>        test_inputs, test_targets)):</li>
<li>    train_logit = model(train_input)</li>
<li>    inner_loss = F.cross_entropy(train_logit, train_target)</li>
<li>
</li>
<li class="selected">    model.zero_grad()</li>
<li>    params = update_parameters(model, inner_loss,</li>
<li>        step_size=args.step_size, first_order=args.first_order)</li>
<li>
</li>
<li>    test_logit = model(test_input, params=params)</li>
<li>    outer_loss += F.cross_entropy(test_logit, test_target)</li>
<li>
</li>
<li>    with torch.no_grad():</li>
<li>        accuracy += get_accuracy(test_logit, test_target)</li>
<li>
</li>
<li>outer_loss.div_(args.batch_size)</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>I would expect this call to come at the beginning of the for loop before the model makes a prediction. Is there any different effect by putting it where it is in the link?</p>

Eta_C: <p>I think all is right, when <code>zero_grad</code> is executed before <code>backward</code></p>

jwillette: <p>so just being anywhere before backward is the important part?</p>

albanD: <p>If you code does not do weird stuff where it changes the gradients outside of the backward, you can put it anywhere in between two backward. You could even put it just after the backward if you want <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
How to run libtorch on Visual Studio

RuFAI: <p>Hi, does anyone know here, how to run libtorch on Visual Studio? I get linker errors although I fed the libtorch file into C++ and linker directories. I followed the tutorial, where Cmake is used but is it really necessary? I just want to hit the play button and get things done. No release build, no other build. Just run the code.</p>

EsdeathYZH: <p>I think the link error probably is because that official libtorch is built using gcc 4.x for compatibility and some dependencies of your project is built using gcc 5.x or higher version.</p>
<p>Try to build libtorch from source?</p>

RuFAI: <p>Hi, thanks for your advice. How do I do this? I already read, that there is an opportunity to build from libtorch source, but to be honest I don’t have a clue what it means. Could you explain it briefly please <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> ?</p>

EsdeathYZH: <p>If you project is linked with some other dependencies? The ABI issue won’t lead to link error(undefined reference to …) if you are just running C++ API example given in tutorials.</p>
<p>I only link libtorch on linux and don’t know much about how to link in windows.</p>

RuFAI: <p>There are no other dependencies there <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"></p>

EsdeathYZH: <p>If that I think you need to check you VS configuration or search other tutorials about linking libtorch on Windows…</p>

js_Du: <p>Hi, I am new to pytorch and I got confused about libtorch. After running build_libtorch.sh, how can I organize files like the one downloaded from homepage? Thanks.</p>

EsdeathYZH: <p>You may use these commands:</p>
<pre><code class="lang-auto">cd [PYTORCH_ROOT_DIR]
mkdir -p libtorch/include &amp;&amp; mkdir -p libtorch/share &amp;&amp; mkdir -p libtorch/bin
cp -r build/lib libtorch
cp -r torch/share/cmake libtorch/share

for dir in ATen c10 caffe2 torch
do 
    cp -r torch/include/$dir libtorch/include
done
</code></pre>
<p>But I can’t make sure the official libtorch is built as above.</p>

js_Du: <p>Thank you! It works.</p>
How to tile a tensor without occupying new GPU memory

harry1herry: <p>If the size of tensor is wery large, such as 40000*64. I want to tile the 0 dim with 100 times, if I use repeat, the menory of GPU will grow a lot. How can I do it without occupying new GPU memory, just use expand?</p>
<pre><code class="lang-auto">import torch
a=torch.rand(40000,64).cuda()
a=a.unsqueeze(1).repeat(1,100,1).view(-1,64)
</code></pre>
<p>The the memory will up to 1500M, it is to higher!</p>
Convert list to torch.tensor

1414b35e42c77e0a57dd: <pre><code class="lang-auto">list_x = []
x = torch.Size([4,3,32,32])
y = torch.Size([6,3,32,32])
list_x.append(x)
list_x.append(y)
list_x = torch.tensor(list_x)
</code></pre>
<p>x and y have different shape of dimension 0.<br>
In this case how to convert list to torch.tensor?</p>

Tahir: <p>I dont think you can convert it like this .</p>
<p>what result (i.e. tensor shape) you expect to see after adding those two tensors?</p>

1414b35e42c77e0a57dd: <p><a class="mention" href="/u/tahir">@Tahir</a><br>
Thanks for you answer.<br>
I want to add only inputs that satisfy the conditions.<br>
So the size(dim==0) depends on whether the inputs are satisfied or not.</p>
<pre><code class="lang-auto">list_x = []
for batch_idx, input in enumerate(data_loader):
   list_x.append(input) # save input images under some conditions

new_data_loader = DataLoader(list_x, ...)
   
</code></pre>
<p>In this purpose, Is there any soultions ?</p>

Eta_C: <p>Maybe… <code>torch.cat</code> ?</p>

1414b35e42c77e0a57dd: <p><a class="mention" href="/u/eta_c">@Eta_C</a><br>
Thanks for you reply.<br>
I’m using <code>torch.cat</code> but I think it is too slow.</p>

Eta_C: <p>Actually, if your memory is large enough, you can try this</p>
<pre><code class="lang-python">dataset = torch.empty_like(data_loader.dataset)
cur_ind = 0
for batch_idx, input in enumerate(data_loader):
    dataset[cur_ind:cur_ind+len(input)] = input
    cur_ind += len(input)
new_data_loader = DataLoader(dataset[:cur_ind, ...], ...)
</code></pre>

1414b35e42c77e0a57dd: <aside class="quote no-group" data-username="Eta_C" data-post="6" data-topic="66843">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/eta_c/48/17667_2.png" class="avatar"> Eta_C:</div>
<blockquote>
<p>data_loader.dataset</p>
</blockquote>
</aside>
<p>What is <code>data_loader.dataset</code> ?</p>
Inconsistency in accuracy between different versions of pytorch/fairseq

jinamshah: <p>Hi all,<br>
I am posting this question here as fairseq is a part of the pytorch organization and is broadly used in nlp applications.</p>
<p>I have implemented a translation task using an old version of fairseq and received an accuracy of ‘x’. I then upgraded the pytorch (from 1.1 to 1.3) and fairseq versions to the latest stable versions, and then tried rerunning my training job, keeping the parameters the same. The accuracy that I received from this experiment is ‘x/3’. Can someone please help me understand what has changed in particular with the translation task?</p>
<p>Also please note:</p>
<p>the setup remains the same (multi-gpu)</p>
<p>the model has stopped converging like before</p>
<p>Please let me know if any further information is required!</p>

ptrblck: <p>It seems the issue is being tracked <a href="https://github.com/pytorch/fairseq/issues/1589">here</a>.</p>

jinamshah: <p>yes. I have created that issue <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Pytorch's Resnet 50 accuracy

barakb: <p>Hey, Pytorch’s (torchvision) Resnet 50 accuracy is declared to be 76.15.<br>
But  when I’m using the training script from PyTorch’s repo, which is mentioned in the official torchvision website(<a href="https://pytorch.org/docs/stable/torchvision/models.html#classification" rel="nofollow noopener">https://pytorch.org/docs/stable/torchvision/models.html#classification</a>):<br>
<a href="https://github.com/pytorch/examples/blob/master/imagenet/main.py" rel="nofollow noopener">https://github.com/pytorch/examples/blob/master/imagenet/main.py</a><br>
and the Resnet50 from torchvision:<br>
[<a href="https://github.com/pytorch/vision/blob/master/torchvision/models/resnet.py" rel="nofollow noopener">https://github.com/pytorch/vision/blob/master/torchvision/models/resnet.py</a>]<br>
When training it, after one epoch I’m getting an accuracy of 76.6, how can it be? isn’t the models fully trained?</p>
<p>Thanks!</p>
C++ runtime exception with implement of MNIST

focus: <p>I copied the code on the official site: <a href="https://pytorch.org/cppdocs/frontend.html" rel="nofollow noopener">https://pytorch.org/cppdocs/frontend.html</a> and have the same path problem in this line：<br>
auto data_loader = torch::data::make_data_loader(torch::data::datasets::MNIST("./data").map(torch::data::transforms::Stack&lt;&gt;()),/<em>batch_size=</em>/64);<br>
it is similar to the following problem:<br>
<a href="https://discuss.pytorch.org/t/running-error-with-libtorch-c-front/62977">Running error with libtorch C++ front</a><br>
Does anybody have the same problem? How to solve it?</p>
GPU memory usage problem?

raymongL: <p>Here is my code:</p>
<pre><code class="lang-auto">gpu1 = 0
gpu2 = 1
net1 = models.resnet50(pretrained=True)
net1.cuda(gpu1)
net1.load_state_dict(...)
net1.eval()
myLoader = Dataloader(...)
net2 = models.resnet50(pretrained=True)
net2.cuda(gpu2)
</code></pre>
<p>So i got a 4 GPUs server (id = 0, 1, 2, 3 respectively), and i designated two GPUs for net1 and net2. But somewhere after net1.cuda(gpu1) and before net2.cuda(gpu2), a few memory of (gpu_id = 2) is used. In my case, (gpu_id = 2) has 881M memory occupied.</p>
<p>Can i get a clue on what’s going on? This did not happen when i run the same codes on another dataset.</p>

ptrblck: <p>Was the <code>state_dict</code>, which is loaded in <code>net1</code> saved on GPU2?<br>
Do you see the same memory usage, if you comment out the <code>net1.load_state_dict()</code> and <code>DataLoader</code> creation lines of code?</p>

raymongL: <p>You are right. I comment ‘net1.load_state_dict()’, and the memory usage is gone.<br>
So i guess pytorch use GPU2 for load_state_dict() by default?<br>
How can i do the load_state_dict() without usage of irrelevant GPUs?<br>
But why this did not happen on other datasets?</p>

ptrblck: <p>Could you check the device of the parameters in side this <code>state_dict</code> and then try to load the <code>state_dict</code> via <code>map_location='cpu'</code>?<br>
I don’t think this behavior is related to the dataset, but more likely to the previous training run and storage of the <code>state_dict</code>, but I’m also guessing at the moment. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

raymongL: <p>Yeah, i print the parameters of state_dict, and they are on “device:2”.<br>
I use map_location=‘cpu’ and it solved my problem, thanks!<br>
I think if load_state_dict() is by default load to cpu, sounds more reasonable to me.<br>
Thanks a lot!</p>
Question about changing optimizer

chunchun: <p>I get the error ‘nan or inf for input tensor’ when I change SGD to RMS,Why?</p>

ptrblck: <p>Could you post the error message, please?<br>
Do you get this error immediately after changing the optimizer?<br>
Is the output of your model NaN or Inf?</p>

chunchun: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/3/438ef3ff22584d3e4ff698c60a3b486cbd77da1a.png" alt="image" data-base62-sha1="9DEgv4ToI2LIWx9WMP40ONwHtQ6" width="379" height="212"><br>
I just change the optimizer,there will be the error like this.</p>

chunchun: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="66371" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Could you post the error message, please?<br>
Do you get this error immediately after changing the optimizer?<br>
Is the output of your model NaN or Inf?</p>
</blockquote>
</aside>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/6/469eccc65a7977fce81761207b478c9b1c28ebb9.png" alt="image" data-base62-sha1="a4JEjbk47hhoCJcu9mcsAepJLn3" width="646" height="122"><br>
and the loss becomes nan.<br>
Why?</p>

ptrblck: <p>Once the parameters become NaN, e.g. due to a high learning rate, your output will also be NaN.<br>
Could you try to initialize the new optimizer with a smaller learning rate and retry the code again?</p>

chunchun: <p>Yes,I have tried 0.00001 but it doesn’t work.<br>
I try to print the gradient ,it’s sometimes very small(about 0),sometimes very large(about 100000).<br>
Could you please tell me why?</p>

ptrblck: <p>I’m not sure, but my guess would be the internal functionality of RMSProp.<br>
This optimizer divides the gradient by a running average of its recent magnitude.<br>
If your gradients are quite small, since you’ve already trained your model for a few epochs, I assume this division might blow up.</p>

chunchun: <p>Thanks a lot for your answer.</p>
Running vgg16 on birds data

gebrahimi: <p>I am trying to train vgg-16 model on birds-200 data. the input is size [3, 448, 448] and there are 200 classes. My gpu is empty but when i try to run the code, i get the following error.</p>
<pre><code>RuntimeError: CUDA out of memory. Tried to allocate 392.00 MiB (GPU 0; 7.93 GiB total capacity; 7.28 GiB already allocated; 111.25 MiB free; 21.83 MiB cached)
</code></pre>
<p>I am currently using batch size 32 an decreasing the batch size did not help. If i change the data to cifar and use vgg16, the code runs without any problem. Any idea what can be wrong?</p>

ptrblck: <p>Could you try to lower the batch size even further or resize the spatial size of your input images to e.g. <code>224x224</code>?</p>
C++ Run torch_script model on cv::Mat

elduderino15: <p>How do I run a python trained, dumped torch_script model on a cv::Mat of the right input size in C++?<br>
My tries (doing this for the first time) fail with weirdest errors…<br>
and I seem not to find a proper solution. Thank you, EL</p>

ptrblck: <p><a href="https://discuss.pytorch.org/t/rgb-cv-mat-to-tensor-with-different-width-and-height/49325">This topic</a> should have some valid approaches to convert a <code>cv::Mat</code> to a tensor.<br>
You could try to resize the <code>cv::Mat</code> to the right spatial size in OpenCV and add a batch dimension in libtorch (or create a batch of image tensors).</p>

elduderino15: <p>Thank you. I am asking because my build system / linker fails (below). Not sure what the proper way is.</p>
<aside class="quote quote-modified" data-post="1" data-topic="66786">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/e/8e8cbc/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/c10-error-when-converting-cv-mat-to-tensor-with-from-blob/66786">C10 error when converting cv::Mat to tensor with from_blob</a> <a class="badge-category__wrapper " href="/c/c/11"><span data-category-id="11" data-drop-close="true" class="badge-category " title="Topics related to the C++ Frontend, C++ API or C++ Extensions"><span class="badge-category__name">C++</span></span></a>
  </div>
  <blockquote>
    When I add from_blob to my code the project fails to compile. I want to create a tensor out of a cv::Mat but it keeps failing with the error below. Again, without from_blob my project compiles fine. 
cv::Mat amplitudePatch = amplitude(roi);
torch::from_blob(amplitudePatch.data, 
{1, amplitudePatch.size().height, amplitudePatch.size().width, amplitudePatch.channels()},
at::TensorOptions(at::kFloat));

SunlightNoiseClassifier/libSunlightNoiseClassifier.a(SunlightNoiseClassifier.cpp.o): In function…
  </blockquote>
</aside>
Best/clean way to define your own loss function

Abdelpakey: <p>Hi All,</p>
<p>I saw so many questions/answers here about custom loss function. However, didn’t find what I’m looking for.</p>
<p>my simple question is: From Pytorch’s perspective what is the best/clean way to write your own loss function- class-based or function-based?</p>

aauker: <p>PyTorch is rather modular. Any scalar value can be a loss, so any function of tensors, class labels, or parameters that returns a singular value is a loss function.</p>
<p>The only thing to it beyond coding style is to consider the convexity of the function and it’s limiting behavior (how does it look for extreme values?) and numerical stability (would you run into precision issues with float 32 numbers?).</p>