akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Sparsing the elements of tensors

MRRP: <p>Hi Pytorch Community<br>
I asked once before but no one answered to my question so now i will ask it again and i will appreciate if you can provide me with any help.<br>
I have a dictionary which contains a few tensors(Fisher matrix of a CNN) like below:<br>
Fisher = {conv1.weight = 'a tensor of shape(6,3,5,5), conv1.bias = …,}<br>
now i want to keep the 50 percent biggest elements in the fisher matrix of the entire CNN(whole dictionary) and set the rest of them to 0 so i will have a sparse version of it.<br>
I will appreciate if anyone can help me with that</p>

ptrblck: <p>I’m not sure I understand the use case completely, so please add more information in case I’m missing something.<br>
Wouldn’t indexing the tensor(s) based on a mask to select the lagest elements, and zeroing out the others work or are you processing the entire parameter set somehow first?</p>

MRRP: <p>Thank you for replying<br>
Well, the use case is that i want to put a penalty in my loss function based on those tensors which are the diagonal of the fisher information matrix of another model so i can keep the models parameters close to each other. now i want those tensors to be sparse.<br>
and i want to sparse the entire parameter set.<br>
so if i have a fisher information like below:<br>
fisher = {conv1.weight = a tensor of shape(6,3,5,5),…, fc1.weight = a tensor of shape(200,400),…}<br>
i want to keep the 50 percent largest elements of the entire ‘fisher’</p>

ptrblck: <p>Should the sparsity be “learned” by the model to keep the “50% largest elements”? If so, I would unfortunately not be experienced enough in this area and wouldn’t know how to achieve this.</p>

MRRP: <p>No we don’t need the model here. i was trying to say how the background works and where that dictionary came from. now we have the dictionary and that dictionary has some keys and those keys have tensor values assigned to them.<br>
I want a function that takes that dictionary as input and keep 50 percent largest elements in the whole dictionary and make the rest 0 and output the sparsed dictionary.<br>
I don’t know if i was able to explain what i want properly or not. sorry for my bad explanation</p>

ptrblck: <p>In that case masking should work:</p>
<pre><code class="lang-python">model = models.resnet18()
sd = model.state_dict()

for key in sd:
    print(key)
    print('abs.sum before {}'.format(sd[key].abs().sum()))
    mask = torch.empty_like(sd[key])
    mask.random_(0, 2)
    print('mask, 0s: {}, nelement: {}'.format(mask.sum(), mask.nelement()))
    sd[key].mul_(mask)
    print('abs.sum after {}'.format(sd[key].abs().sum()))
</code></pre>

MRRP: <p>Thank you so much<br>
That helped me alot</p>
Use multiple GPUs in cross validation?

betanken: <p>Hi everyone,</p>
<p>I am wondering how to set up a multiple-GPUs-cross validation in Pytorch.<br>
In the cross-validation part, my code(only the epoch part) looks like this:</p>
<pre><code class="lang-auto">for epoch in range(num_epochs):
            loss = 0
            train_total_loss=0
            model.to(device)
            model.train()
            for batch_index, (x_batch, y_batch) in enumerate(train_loader):
                x_batch,y_batch=x_batch.to(device),y_batch.to(device)
                optimizer.zero_grad()
                out = model(x_batch)
                y_batch=y_batch.view(out.shape)
                loss = torch.nn.MSELoss(y_batch,out)
                loss.backward()
                optimizer.step()
                train_total_loss += loss.item()
            train_total_loss =  train_total_loss/train_loader.__len__() # loss of each epoch 
</code></pre>
<p>Should I add <strong>torch.nn.DataParallel(model)</strong> before <strong>model.to(device)</strong> to use multiple GPUs? And should I change how the loss is calculated?</p>
<p>Thanks a lot for help!</p>

H-Huang: <p>It is recommended to use <code>torch.nn.parallel.DistributedDataParallel</code> see pointers here (<a href="https://pytorch.org/docs/master/notes/ddp.html?highlight=distributed%20data%20parallel" class="inline-onebox" rel="noopener nofollow ugc">Distributed Data Parallel — PyTorch master documentation</a>) since DataParallel is not actively being worked on and will eventually be deprecated.</p>
<p>If you do want to use <code>torch.nn.DataParallel</code> (<a href="https://pytorch.org/docs/master/generated/torch.nn.DataParallel.html#torch.nn.DataParallel" class="inline-onebox" rel="noopener nofollow ugc">DataParallel — PyTorch master documentation</a>) then you also need to specify the device IDs for each GPU that you want to use. For example, for two GPUs you would specify <code>torch.nn.DataParallel(model, device_ids=[0, 1])</code> for <code>cuda:0</code> and <code>cuda:1</code>, the <code>model.to(device)</code> is not necessary.</p>
Torch_geometric.data - something like pop function?

natgolovach: <p>Hi everyone!</p>
<p>I have my own torch_geometric.data Dataset, that includes targets as well. What is the better way to “pop” the targets from it so I can have everything but targets in Dataset?</p>

ptrblck: <p>I’m not sure which <code>Dataset</code> you are using, but e.g. <a href="https://pytorch-geometric.readthedocs.io/en/latest/modules/data.html#torch_geometric.data.Data"><code>torch_geometric.data.Data</code></a> uses an optional target attribute so I would assume you don’t need to pass it to the dataset while creating it.</p>

natgolovach: <p>I have something similar to <a href="https://github.com/drorlab/gvp-pytorch/blob/main/gvp/data.py#L192" class="inline-onebox" rel="noopener nofollow ugc">gvp-pytorch/data.py at main · drorlab/gvp-pytorch · GitHub</a>, but with targets</p>
<pre><code class="lang-auto">        data = torch_geometric.data.Data(x=X_ca, seq=seq, name=name,
                                         node_s=node_s, node_v=node_v,
                                         edge_s=edge_s, edge_v=edge_v,
                                         edge_index=edge_index, mask=mask, 
                                         targets=targets)
</code></pre>
<p>my question is I need to split this to data and targets, what is the best practice?<br>
to get targets only i can use data[“targets”], but how to get everything else but targets?</p>
<pre><code class="lang-auto">        torch_geometric.data.Data(x=X_ca, seq=seq, name=name,
                                 node_s=node_s, node_v=node_v,
                                 edge_s=edge_s, edge_v=edge_v,
                                 edge_index=edge_index, mask=mask)
</code></pre>

natgolovach: <p>Solved it that way</p>
<pre><code class="lang-auto">targets = data["target"]
delattr(data,"target")
x = data
</code></pre>
For actor-critic type method? Should the calculation of target value be involved in the backpropagation

AlanSouthwark: <p>I am confused about computation for target value in Q learning and actor critic method. So during the backpropagation, does the value network being involved for two times since we call it two times for calculating current value and next state value. So should I put with torch.no_grad()  before calculate the next state value?</p>
<pre><code class="lang-auto">        action, log_prob = actor.forward(current_state) 
        current_value = critic.forward(current_state)

        action = action.detach().cpu().numpy()

        new_state, reward, done, info = env.step(action)

        reward = torch.from_numpy(np.array(reward)).type(torch.FloatTensor).to(device)
        new_state = torch.from_numpy(new_state).type(torch.FloatTensor).to(device)

        # should I put with torch.no_grad() at here?
        with torch.no_grad() 
            next_value = self.critic.forward(new_state)
       
        target_value = reward + gamma * next_value * (1 - int(done))
        advantage = target_value - current_value

        actor_loss = -1 * log_prob * advantage
        critic_loss = advantage ** 2
        loss = (actor_loss + critic_loss)

        actor_optimizer.zero_grad()
        critic_optimizer.zero_grad()
        loss.backward()
        actor_optimizer.step()
        critic_optimizer.step()
</code></pre>
Transformer does not recognize emojis despite adding them as additional tokens

the_coder: <p>My research interest is effect of emojis in text. I am trying to classify sarcastic tweets in text. A month ago I have used a dataset where I added the tokens using:</p>
<blockquote>
<p>tokenizer.add_tokens(‘List of Emojis’).</p>
</blockquote>
<p>So when I tested the BERT model had successfully added the tokens. But 2 days ago when I did the same thing for another dataset, BERT model has categorized then as ‘UNK’ tokens. My question is, is there a recent change in the BERT model? I have tried it with the following tokenizer,</p>
<blockquote>
<p>BertTokenizer.from_pretrained(‘bert-base-uncased’)</p>
</blockquote>
<p>This is same for distilbert. It does not recognize the emojis despite explicitly adding them. At first I read somewhere there is no need to add them in the tokenizer because BERT or distilbert has already those emojis in the 30000 tokens but I tried both. By adding and without adding. For both cases it does not recognize the emojis.</p>
<p>What can I do to solve this issue. Your thoughts on this would be appreciated.</p>
Get index info from torch.where

James_Lee: <p>Hello. I found <code>torch.</code> <code>where</code> (<em>condition</em> , <em>x</em> , <em>y</em> ) [<a href="https://pytorch.org/docs/stable/generated/torch.where.html" class="inline-onebox" rel="noopener nofollow ugc">torch.where — PyTorch 1.9.1 documentation</a>] seemed doesn’t support getting the index location of each value found under a specific condition, as <code>torch.</code> <code>max</code>  [<a href="https://pytorch.org/docs/stable/generated/torch.max.html" class="inline-onebox" rel="noopener nofollow ugc">torch.max — PyTorch 1.9.1 documentation</a>] and <code>torch.</code> <code>min</code>  [<a href="https://pytorch.org/docs/stable/generated/torch.min.html" class="inline-onebox" rel="noopener nofollow ugc">torch.min — PyTorch 1.9.1 documentation</a>]has supported. Is there a way to get the index when calling <code>torch.</code> <code>where</code> or are there any alternatives? Thanks.</p>

googlebot: <p>try torch.nonzero(condition), perhaps with as_tuple argument</p>
Difference in implementation of SGD with Momentum/Nesterov

Sam-gege: <p>from the optim.SGD <a href="https://pytorch.org/docs/stable/generated/torch.optim.SGD.html" rel="noopener nofollow ugc">doc</a> it says: “The implementation of SGD with Momentum/Nesterov subtly differs from Sutskever et. al. and implementations in some other frameworks.” Does anyone know why? I’ve read some other sources and they all implement different than pytorch. Thanks!</p>
Error when resuming training: No 'step' key in optimizer state (PyTorch 1.7)

miel: <p>Hello,</p>
<p>I would like to ask a question please.</p>
<p>The question concerns a strange error that I encountered when using <a href="https://github.com/pytorch/fairseq" rel="noopener nofollow ugc">fairseq</a>, but I believe the issue concerns PyTorch in general.</p>
<p>After upgrading to PyTorch 1.7, I frequently had a memory issue with my trainings:</p>
<pre><code class="lang-auto">File "/home/user/.local/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 943, in all_reduce
            work = group.allreduce([tensor], opts)work = group.allreduce([tensor], opts)work = group.allreduce([tensor], opts)
MemoryErrorMemoryErrorMemoryError: : : std::bad_allocstd::bad_allocstd::bad_alloc
</code></pre>
<p><strong>This issue happened randomly</strong>: for an exact same configuration (same model, training data, and everything else) across different runs: sometimes the job runs well and sometimes there is a memory error or socket timeout error.</p>
<p><strong>After downgrading back to 1.6, I no longer observed this issue. However, I was not able to resume trainings that I had started on 1.7</strong> because of the following error:</p>
<pre><code class="lang-auto">File "/home/user/code/fairseq/fairseq/optim/adam.py", line 199, in step
    self.optimizer.step(closure)
  File "/home/user/code/fairseq/fairseq/optim/adam.py", line 199, in step
    state["step"] += 1
        state["step"] += 1KeyErrorstate["step"] += 1
: 
'step'KeyError
: KeyError'step': 
'step'
    state["step"] += 1
KeyError: 'step'
</code></pre>
<p>(The error happened at <a href="https://github.com/pytorch/fairseq/blob/48a607527a8c2435c63795cca9a05348ef6e1f9d/fairseq/optim/adam.py#L199" rel="noopener nofollow ugc">this line</a> in the code.)</p>
<p>I have manually checked the checkpoints saved by 1.7 and found that their optimizer states indeed do not have the key <code>step</code>, while the 1.6 checkpoints do.</p>
<p><strong>Could you please tell me how I can continue my trainings on PyTorch 1.6 by resuming from the checkpoints saved by PyTorch 1.7?</strong></p>
<p>Thank you so much in advance for your help!</p>

ptrblck: <p>You could most likely add the missing <code>step</code> state manually to the <code>optimizer</code>.</p>
<p>However, I would generally recommend to stick to the latest version so we would need to look into the memory issue. Do you have a code snippet to reproduce this issue and could you post your current setup (PyTorch, CUDA, cudnn, NCCL, versions etc.)?</p>

miel: <p>Thank you for your reply! I used PyTorch 1.7 to resume the trainings that I needed and just used the latest version for all later trainings.</p>
Question about tensors conversion

Allelujah-Haptism: <p>Hi, I am trying to understand how the conversion of tensors happens in PyTorch.</p>
<pre><code class="lang-auto">x = torch.tensor([1, 2, 45896213, 5584700])
y  = x.type(torch.int8)
print(y)
tensor([ 1,  2, 21, 60], dtype=torch.int8)
</code></pre>
<p>so, my questions are:</p>
<ol>
<li>why this piece of code does not raise any conversion errors.</li>
<li>why I am getting 21 and 60<br>
Thanks.</li>
</ol>

googlebot: <p>45896213 % 256 = 21</p>
<p>can’t comment about error, it is debatable. note that there may be some valid uses of truncation and size compression, but check for out of range values is relatively expensive.</p>
Saving Quantized model

flash87c: <p>I have converted a fp32 model to 8bit model using post training static quantization. I tried to save the model using torch.save() and torch.jit.save() but both methods are not working. And then i tried to just save the state_dict, but then when i load it, the results are not consistent. Is there any other way to save a quantized model?</p>
<p>If you need any more info please let me know.</p>
<p>Thanks in advance.</p>

dskhudia: <p>loading/saving state_dict is the preferred method. Please save state_dict and then before loading it to a quantized model, make sure to follow the quantization steps, e.g., fusion. Also see <a href="https://discuss.pytorch.org/t/loading-of-quantized-model/64213" class="inline-onebox">Loading of Quantized Model</a></p>

flash87c: <p>I did exactly what you told, but the results are different. I tried with fusing and without fusing, but it’s just not working. I can see that all the zero point and scales are same, all the weights are same, but the results are not same.</p>

hx89: <p><a class="mention" href="/u/flash87c">@flash87c</a> could you share a small repro of what you did so that we can take a look?</p>

crane: <p>hello,  i meet the same question , have u solve it ?</p>

Parameswar: <p>Hello,<br>
I am facing this same issue, did you find solution for that?<br>
Thanks.</p>

jerryzh168: <p>cc <a class="mention" href="/u/vasiliy_kuznetsov">@Vasiliy_Kuznetsov</a> have we solve the serialization issue? maybe we can make a post here if that is the case</p>

yyl-github-1896: <p>You can use API torch.jit.save() to save quantized models. Just as what has been done in PyTorch Quantization Tutorial. <a href="https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">(beta) Static Quantization with Eager Mode in PyTorch — PyTorch Tutorials 1.9.1+cu102 documentation</a></p>
How can I update the gradient manually once per epoch?

alex_rachkov: <p>I am working on a regression problem involving a very sparse dataset (&lt;1% non-zero examples). One of the strategies I would like to try is to accumulate the gradient for each batch over an entire epoch, and only then update the model (since a batch has many zero examples and the gradients quickly tend to zero). I am thinking of a pseudocode similar to the following:</p>
<pre><code class="lang-auto">for e in epochs:
   grad = 0
   for batch in dataloader:
      grad += model.backward(batch)
   grad /= len(dataloader)
   model.update(grad)
</code></pre>
<p>My current code that updates the gradient after every batch is:</p>
<pre><code class="lang-auto">    for e in epochs:
       for batch in dataloader:

            # Forward pass
            optimizer.zero_grad()

            x: torch.tensor = batch[0].float().to(device)
            y: torch.tensor = batch[1].float().to(device)

            y_pred = model_reg(x).squeeze()

            loss = criterion(y_pred, y)

            # Backward pass
            loss.backward()
            optimizer.step()
</code></pre>
<p>Is there a way I could do what I am thinking of in Pytorch? I know autograd is a bit special, so I wanted to seek your wisdom in these matters. My current idea was to have the optimizer.zero_grad() and optimizer.step() being done outside of the batch loops (so done only once per epoch). However, I am not sure how I could divide the accumulated gradient at the end of an epoch by the number of batches.</p>
<p>I have been looking through the documentation and past topics and couldn’t find an answer. Any advice or help are greatly appreciated!</p>
<p>Thanks,<br>
Alex</p>

ptrblck: <p><a href="https://discuss.pytorch.org/t/why-do-we-need-to-set-the-gradients-manually-to-zero-in-pytorch/4903/20">This post</a> gives you a few examples of gradient accumulation approaches and their advantages or shortcomings.<br>
Based on your current code snippet it seems you would like to use <code>torch.autograd,grad</code> instead, which I think should work in a similar manner (you might need to perform the gradient accumulation manually though).</p>
Error when loading on another device

mfoglio: <p>I am getting the following error on a <code>torch.script.load("file.pth", "cuda")</code>:</p>
<pre><code class="lang-auto">RuntimeError:
Unknown type name 'NoneType':
Serialized   File "code/__torch__/classification/models/multi_output/model.py", line 9
  channels_fc : int
  learning_rate : float
  scheduler : NoneType
              ~~~~~~~~ &lt;--- HERE
  _torchscript_attributes : Dict[str, List[str]]
  features : __torch__.torch.nn.modules.container.Sequential
</code></pre>
<p>What does it mean? Is it looking for python code? I thought that torchscript models would not require the original python code to be run.</p>

ptrblck: <p>Could you explain your use case a bit more, i.e. how did you save the model and what is the model definition? Also, where does <code>torch.script.load</code> come from? Are you using <code>torch.jit.load</code>?</p>
Exception ignored in: <function _MultiProcessingDataLoaderIter.__del__ at

bingqing_liu: <pre><code class="lang-auto">Exception ignored in: &lt;function _MultiProcessingDataLoaderIter.__del__ at 0x7f231e2bd8b0&gt;
Traceback (most recent call last):
  File "/home/a09/software/anaconda3/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1328, in __del__
    self._shutdown_workers()
  File "/home/a09/software/anaconda3/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1320, in _shutdown_workers
    if w.is_alive():
  File "/home/a09/software/anaconda3/lib/python3.8/multiprocessing/process.py", line 160, in is_alive
    assert self._parent_pid == os.getpid(), 'can only test a child process'
AssertionError: can only test a child process
</code></pre>
<p>mycode is here:</p>
<pre><code class="lang-auto">batch_size=512
trainset=mydataset(text,win_size,neg_per_pos,freqs,word2idx)
trainloader=tud.DataLoader(trainset,batch_size=batch_size,shuffle=True,num_workers=1)
it=iter(trainloader)
centers,poss,negs=next(it)
print(centers.shape)
print(poss.shape)
print(negs.shape)
</code></pre>
<p>and then the warning occurs,i guess it’s about the parameter “num_workers”,but i really want to use that.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/0/a09d39bafc211ba1abed462afa9083305886f63f.png" data-download-href="https://discuss.pytorch.org/uploads/default/a09d39bafc211ba1abed462afa9083305886f63f" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/0/a09d39bafc211ba1abed462afa9083305886f63f_2_690x158.png" alt="image" data-base62-sha1="mURfsAnzV6CklY15JCuGlywkq1V" width="690" height="158" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/0/a09d39bafc211ba1abed462afa9083305886f63f_2_690x158.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/0/a09d39bafc211ba1abed462afa9083305886f63f_2_1035x237.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/a/0/a09d39bafc211ba1abed462afa9083305886f63f_2_1380x316.png 2x" data-dominant-color="F8DFDF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1489×343 53.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>You might be running into a similar issue as described <a href="https://discuss.pytorch.org/t/error-while-multiprocessing-in-dataloader/46845/9">here</a> which seems to point to a bad interaction between <code>tqdm</code> and Python’s <code>multiprocessing</code> module.</p>
Combining two trained yolo model weights

Dhirendra_Jha: <p>I’m currently working on image detection models that train on separate (but related) types of data. My problem is every week I got multiple new classes so combining all images and retrain every time it takes lots of time, so I like two train new data separately and after training we combined both weights  I’d like to make a combined model weight, to create one model weights.</p>

hatim_shabbir: <p>Hi, did you find the solution to this problem? I have been working on something similar to this your help would be appreciated.</p>
RuntimeError: Expected 4-dimensional input for 4-dimensional weight [8, 4, 3, 3], but got 1-dimensional input of size [16] instead

amahoro12: <pre><code>return F.conv2d(input, weight, bias, self.stride,
</code></pre>
<p>RuntimeError: Expected 4-dimensional input for 4-dimensional weight [8, 4, 3, 3], but got 1-dimensional input of size [16] instead</p>
<p>class QNetwork(nn.Module):<br>
def <strong>init</strong>(self, in_channels=4, num_actions=200):<br>
super(QNetwork, self).<strong>init</strong>()<br>
self.conv1 = nn.Conv2d(in_channels, 8, kernel_size=3, stride=1, padding=1)<br>
self.conv2 = nn.Conv2d(8, 16, kernel_size=4, stride=2)<br>
self.conv3 = nn.Conv2d(64, 64, kernel_size=3, stride=1)<br>
self.fc4 = nn.Linear(7 * 7 * 64, 512)<br>
self.fc5 = nn.Linear(512, num_actions)</p>
<pre><code>def forward(self, x):
    x = F.relu(self.conv1(x))
    x = F.relu(self.conv2(x))
    x = F.relu(self.conv3(x))
    x = F.relu(self.fc4(x.view(x.size(0), -1)))
    return self.fc5(x)

def select_action(self, x):
    with torch.no_grad():
        fc5 = self.forward(x)
        action_index = torch.argmax(fc5, dim=1)
    return action_index.item()</code></pre>

KFrank: <p>Hi Anne!</p>
<aside class="quote no-group quote-modified" data-username="amahoro12" data-post="1" data-topic="133514" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/c37758/48.png" class="avatar"> amahoro12:</div>
<blockquote>
<p>RuntimeError: Expected 4-dimensional input for 4-dimensional weight [8, 4, 3, 3], but got 1-dimensional input of size [16] instead<br>
…<br>
self.conv1 = nn.Conv2d(in_channels, 8, kernel_size=3, stride=1, padding=1)</p>
</blockquote>
</aside>
<p>You are passing a 1D tensor of shape <code>[16]</code> into your model as its<br>
input.  But the first layer of your model is a <code>Conv2d</code> (with<br>
<code>in_channels = 4</code>).</p>
<p>Your <code>Conv2d</code> layer, and hence your model, requires an input of shape<br>
<code>[nBatch, in_channels = 4, height, width]</code>.  (<code>nBatch</code> can be <code>1</code>,<br>
but it has to be there.)</p>
<p>You would typically be passing into your model a batch of multi-channel<br>
2D images.  (Because of your choice of having <code>in_channels = 4</code> for<br>
<code>conv1</code>, in this specific case you need to pass in a batch of four-channel<br>
images.)</p>
<p>Best.</p>
<p>K. Frank</p>

amahoro12: <p>Thank you a lot for your response. let me correct it according to your guidance. I will give you the feedback</p>
QAT: AttributeError: 'Sequential' object has no attribute 'Conv2d'

ynjiun_wang: <p>Hi,</p>
<p>I am performing QAT and encounter the error:<br>
<code>AttributeError: 'Sequential' object has no attribute 'Conv2d'</code><br>
When running the line:<br>
<code>torch.quantization.fuse_modules(basic_block, [["Conv2d", "BatchNorm2d", "ReLU"]], inplace=True)</code></p>
<p>where the basic_block is:</p>
<pre><code class="lang-auto">Sequential(
  (0): Conv2d(512, 256, kernel_size=(1, 1), stride=(1, 1))
  (1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (2): ReLU()
  (3): Sequential(
    (0): Conv2d(256, 256, kernel_size=(3, 3), stride=(2, 2), padding=(2, 2), groups=256)
    (1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (2): ReLU()
    (3): Conv2d(256, 512, kernel_size=(1, 1), stride=(1, 1))
  )
)
</code></pre>
<p>Any suggestion what should I modify to correct the error?</p>
<p>Thanks a lot.</p>

ynjiun_wang: <p>change<br>
<code>torch.quantization.fuse_modules(basic_block, [["Conv2d", "BatchNorm2d", "ReLU"]], inplace=True)</code></p>
<p>to<br>
<code>torch.quantization.fuse_modules(basic_block, [["0", "1", "2"]], inplace=True)</code></p>
<p>Then it works.</p>
How is model size determined for quantize aware training

pyfeast07: <p>I am trying quantize aware training on a GAN model and am quantizing the generator part. I use two approaches:</p>
<ol>
<li>I quantize the entire network including all the layers in my network</li>
<li>I quantize only the first conv layer (conv2d, relu) and leave the rest of the layers as is. (I do set qconfig accordingly during training, None for the ones I’m not quantizing and qnnpack for the first conv layer)</li>
</ol>
<p>However, my model size for both the cases are same. Could anyone tell me how can that be possible?</p>
C++ class python wrapper

yjy765: <p>Hi, I am trying to make python wrapper for model execution in C++</p>
<p>I defined a class which loads torchscript model in the constructor and run a model by calling its method. code is as follows<br>
– scripttest.cpp –</p>
<pre><code class="lang-auto">#include &lt;torch/script.h&gt;
#include &lt;pybind11/pybind11.h&gt;
#include &lt;iostream&gt;
#include &lt;vector&gt;
#include &lt;torch/extension.h&gt;
using namespace std;
namespace py = pybind11;

class Model{
    private:
        torch::jit::script::Module module;
        torch::Tensor t1;
    public:
        Model(){
            t1 = torch::randn({1,3,11,11}, torch::dtype(at::kFloat).device(c10::kCUDA));
            module = torch::jit::load("test.pt");
        }
        at::Tensor get(){
            at::GradMode::set_enabled(false);
            at::Tensor t2 = module.forward({t1}).toTensor();
            return t2;
        }
};

PYBIND11_MODULE(scripttest, m){
    py::class_&lt;Model&gt;(m, "Model")
        .def(py::init&lt;&gt;())
        .def("get", &amp;Model::get);
}
</code></pre>
<p>— CMakeLists.txt —</p>
<pre><code class="lang-auto">cmake_minimum_required(VERSION 3.4...3.18)
set(CMAKE_CXX_STANDARD 14)
project(scripttest LANGUAGES CXX)

#include_directories("&lt;libtorch/to/path&gt;")

find_package(Torch REQUIRED)
find_package(pybind11 REQUIRED)
#add_library(scripttest)
pybind11_add_module(scripttest scripttest.cpp)
//find_library(TORCH_PYTHON_LIBRARY torch_python PATHS "{TORCH_INSTALL_PREFIX}/lib")
target_link_libraries(scripttest PRIVATE "${TORCH_LIBRARIES}" )//${TORCH_PYTHON_LIBRARY})
</code></pre>
<p>– python script –</p>
<pre><code class="lang-auto">from scripttest import Model
m = Model()
out = m.get()
</code></pre>
<p>It gives <strong>"ImportError: undefined symbol: _Z16THPVariable_WrapN2at6TensorE"</strong></p>
<p>When I removed &lt;torch/extension.h&gt; in the scripttest.cpp, it says <strong>"TypeError: Unable to convert function return value to a Python type! The signature was (self: build.scripttest.Model) → at::Tensor"</strong></p>
<p>When I uncommented CMakeLists.txt to use TORCH_PYTHON_LIBRARY, this returns <strong>Segmentation fault (core dumped)</strong> at m.get().</p>
<p>I downloaded 1.9.1+cu11.1 libtorch for ubuntu and 1.7.1 torch in anaconda.</p>
<p>Actually if I change get function not to return anything, then it does not raise any problem. But, I have to return torch result and also I should add torch tensor argument for get method later to replace random input tensor defined in get method.</p>
<p>Does anyone know how to solve it?<br>
I read this thread(<a href="https://discuss.pytorch.org/t/can-libtorch-return-torch-tensor-to-pytorch-with-pybind11/125412/3" class="inline-onebox">Can libtorch return torch::Tensor to pytorch with pybind11？ - #3 by tom</a>) and this tutorial(<a href="https://pytorch.org/tutorials/advanced/cpp_extension.html" class="inline-onebox" rel="noopener nofollow ugc">Custom C++ and CUDA Extensions — PyTorch Tutorials 1.9.1+cu102 documentation</a>) but I want to use C++ class.</p>
How to get outputs in the same order as inputs with multiple spawned processes running on multiple GPUs and batches of data processed by each?

kaleemiqb: <p>I am using Pytorch Distributed Data Parallel approach and spawning multiple processes from parent process, each running on separate GPU.I am using Pytorch Distributed Data Sampler along with Data Loader for loading batches of input data to each process.<br>
My questions:</p>
<ol>
<li>Under the hood, how does Pytorch Distributed Data Sampler, Data Loader make slices of input data? Just for simplicity say we have 4 GPUs, and 400 input samples and batch size of say 50, then will Pytorch Distributed Data Sampler (together with Data Loader) make first 50 samples go to GPU-0, next 50 to GPU-1., next 50 to GPU-2, then GPU-3 and then again next 50 to GPU-0 i.e. <strong>in the order of GPU device number?</strong> or the order of GPU to select for next batch of input <strong>is random based on which GPU has finished its previous batch first</strong>? or is it like 400 samples get divided into 4 parts first and then GPU-0 would get first 100 samples of input data (50 at a time ),  GPU-1 will get next 100 samples ( 50 at a time) and so on…and in this case no matter if say GPU-3 gets its second batch started earlier than GPU-0, but still with respect to input data, GPU-0 would still have first 100 samples and GPU-3 would have last 100?</li>
</ol>
<p>2). My Second question is how to retrieve output data <strong>in same order as input data</strong> so that final consolidated output ( having outputs from all processes combined in one data structure) is in same order as original inputs and each output corresponds to the right input</p>

wanchaol: <p>Thanks for posting <a class="mention" href="/u/kaleemiqb">@kaleemiqb</a> If using the recommended DistributedDataParallel (DDP) mode, where there is a dedicated process for each GPU, DDP does not split input data. Each process will have its own data loader and its own DDP instance. DDP only help to automatically compute the global averaged gradient in the backward pass. So it really depend on the dataload next batch is loaded, which I think it’s random.</p>
<p>for the second question, you can record the input batch, and the output of the model in its own process in a map, and if you want to concat them together do a all_gather manually, but input_batch across process might not rank properly.</p>

wanchaol: <p>Also, if you want a detailed control on how the data generate and be consumed by processes, consider using custom dataloader <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Writing Custom Datasets, DataLoaders and Transforms — PyTorch Tutorials 1.9.1+cu102 documentation</a></p>
DDP multiple models, share subset of common params

Blake_Camp: <p>Hello all,</p>
<p>I would like to train multiple models, each on a separate GPU, using DDP.  The challenge is that each model has some unique params (for each model), and some common params which are shared by all models. Any tips on how this can be done?  Thanks very much in advance.</p>

wanchaol: <p>Hi <a class="mention" href="/u/blake_camp">@Blake_Camp</a> Thanks for posting the question. You can try mixing DDP and RPC together, use RPC to hold a shared part of your model in one rank, and use DDP for the remaining part. see <a href="https://pytorch.org/tutorials/advanced/rpc_ddp_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Combining Distributed DataParallel with Distributed RPC Framework — PyTorch Tutorials 1.9.1+cu102 documentation</a></p>

wanchaol: <p>Also, if you could provide more details on your use case, it should be helpful for us to see if there’re existing solutions <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Sync Gradient for two tiny MLP model training on the same GPU

DanielWang: <p>How to effectively sync the gradients/parameters between two tiny MLP models (3-layer each, with 256 hidden dimensions), Note that these two MLP models are trained separately by two parallel processes.<br>
I currently come up with two potential solutions.</p>
<blockquote>
<ol>
<li>using the <code>torch.distributed library with </code>gloo` backend for CPU-based parameter sync. it seems to be very slow due to copying tensor back-and-forth between CPU and GPU.</li>
<li>using shared global memory for GPU-based parameter sync. However, it seems hard to achieve for the current Pytorch version.</li>
</ol>
</blockquote>

wanchaol: <p><a class="mention" href="/u/danielwang">@DanielWang</a> Thanks for posting the question. Could you elaborate more on what’s your use case? from the description I see you have two MLP models with exactly the same architecture, and trained separately by two parallel process, it looks to me you can just use DDP with the MLP model, and it will periodically (full_sync or async) sync the model parameters gradients by DDP under the hood? <a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Getting Started with Distributed Data Parallel — PyTorch Tutorials 1.9.1+cu102 documentation</a></p>

DanielWang: <p>DDP is usually for model running on two GPUs, while for models share the same GPU it probably not work</p>

wanchaol: <p>I see, so you are using one GPU but run two training in parallel. We couldn’t run all_reduce collective manually since there’s only one GPU. If we are on CPU we can leverage <code>tensor.share_memory_()</code>, but since we are on GPU, this is not an option. Yeah so the only option I could think of is the first option you mentioned, you can periodically move the two model back to CPU, sync params/grads, then move them back and resume training.</p>
torch.dist.All_reduce

DanielWang: <p>I found a problem when use <code>torch.dist.allreduce</code>. I want to manually reduce and sum all model parameter gradients.<br>
This is the first solution, which can give me the correct reduced_and_sum results.</p>
<pre><code class="lang-auto">               for p in params:
                    dist.all_reduce(p.grad, op=dist.ReduceOp.SUM)
</code></pre>
<p>However, the below second solution does not do any reduce at all after running and returns me the same value before the reduction.</p>
<pre><code class="lang-auto">            def sync_para(x):
               dist.all_reduce(x.grad, op=dist.ReduceOp.SUM)
               x.grad = grad

             map(lambda x: sync_para(x), params)
</code></pre>
<p>Why <code>map</code> function cannot be used at here?</p>

wanchaol: <p><a class="mention" href="/u/danielwang">@DanielWang</a> Thanks for posting the question, in the second case, did you found there’re any errors when using it? My guess is that since map() is written in C and did some optimizations, its implied loop can be more efficient than a regular Python for loop, so the order might be different, and since all_reduce is happening in SPMD fashion, a mis-order might incur some issues.</p>

DanielWang: <p>another solution is to use <code>async_op=True</code> instead of this sync operator</p>

wanchaol: <p>I see, curious what makes <code>async_op=True</code> work? Another thought: does the <code>map</code> really running? It seems like <code>map()</code> will return a generator instead of really looping over the list. Did you try put print statements inside and see if it really got invoked? or wrap it with a sth like <code>list(map(lambda x: sync_para(x), params))</code></p>
Torch.index_select does not behave as expected

seyeeet: <p>I am trying to use <code>torch.index_select</code> but it does not seems to work okay for dimension more than 2.<br>
for the following example I expect the output to be in shape of <code>3x1x5x5</code> but it gives me <code>3x3x5x5</code>.</p>
<pre><code class="lang-auto">x = torch.randn(3,3,5, 5)
indices = torch.tensor([1,1, 2])
print(torch.index_select(x, 1, indices).shape)
torch.Size([3, 3, 5, 5])
</code></pre>
<p>I want to get the <code>1st</code> channel or <code>x</code> in <code>batch 1 </code> and <code>1st</code> channel of <code>x</code> in  <code>batch 2</code> and <code>2nd</code> channel of <code>x</code> in <code>batch 3</code></p>

isalirezag: <p>now sure how the <code>index_select</code> should work, but I think for your question you can do it like this:</p>
<pre><code class="lang-auto">b,c,w,h = x.shape
new_ind = (indices.view(-1) +  (torch.arange(0,b)*c).view(-1) ).tolist()
out = x.view(b*c,w,h)[new_ind]
out.unsqueeze_(1)
print((out[0,0]==x[0,1]).all(),(out[1,0]==x[1,1]).all(),(out[2,0]==x[2,2]).all())

</code></pre>
Transfer weights from classification to maskrcnn

azer: <p>Hi,<br>
I am working on a Maskrcnn project for an specific a limited dataset where my number of mask are limited and  the area of instance are very small. I have tried to trained the model but I cannot improve the AP to more than 0.34.</p>
<p>It occurred to me that I could use transfer learning from the resnet50 model I run for classification from which I got pretty good F1 score and macc (and plenty of samples).</p>
<p>I have no idea how to transfer the weights and I used as a base the<br>
maskrcnn_50fpn method. Please below:</p>
<p>I would be very grateful if you could help with a piece of advice:</p>
<pre><code class="lang-auto">def maskrcnn_50fpn_transfer(num_classes=2, pretrained_backbone=False, **kwargs):
    """
    buils MaskRCNN_50_fnp 
    """
    resnet50 = torchvision.models.resnet50(pretrained = False)
    # 3 classes from claassification
    resnet50.fc = nn.Linear(in_features=2048, out_features= 3)
    
    checkpoint = torch.load(
        '/content/drive/MyDrive/INM373_classification/weights/resnet50/nwce/model_6.pth',
                        map_location=torch.device('cpu'))
    
    resnet50.load_state_dict(checkpoint['model'], strict = False)

    # pretrained_backbone = resnet50

    backbone = resnet_fpn_backbone('resnet50', resnet50) 
    model = MaskRCNN(backbone, num_classes, **kwargs)
    return model

</code></pre>
Replace several values in tensor

Alex07: <p>I have a tensor like</p>
<pre><code class="lang-auto">a = torch.Tensor([3,2,7,9,2,9, 9])
</code></pre>
<p>However, I need the values to start from 0 and go to <code>n-1</code> where <code>n</code> is the number of unique elements in <code>a</code>. One possible way to compute this seems to be:</p>
<pre><code class="lang-auto">r = torch.zeros_like(a)
for i,el in enumerate(torch.unique(a)):
    r[a == el] = i
</code></pre>
<p>giving</p>
<pre><code class="lang-auto">tensor([1., 0., 2., 3., 0., 3., 3.])
</code></pre>
<p>Is there a more efficient way to compute this (possibly avoiding the <code>unique</code> and loop which takes a bit of time as I am having a quite big number of different elements ~10k and a very large <code>a</code>) ?</p>
<p>Thanks!</p>

KFrank: <p>Hi Alex!</p>
<aside class="quote no-group" data-username="Alex07" data-post="1" data-topic="133531" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/a698b9/48.png" class="avatar"> Alex07:</div>
<blockquote>
<p>possibly avoiding the <code>unique</code> and loop</p>
</blockquote>
</aside>
<p>There’s no way you can avoid <code>unique()</code> (or its equivalent).  That’s<br>
an inherent bit of processing required by your problem.</p>
<p>You can avoid the loop – depending on the details of your use case,<br>
such as having the values in <code>a</code> being non-negative integers that are<br>
not ridiculously large – by building a look-up table and indexing into<br>
it to translate the values in <code>a</code>:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.9.0'
&gt;&gt;&gt; a = torch.tensor ([3, 2, 7, 9, 2, 9, 9])
&gt;&gt;&gt; a_unique = torch.unique (a)
&gt;&gt;&gt; lt = torch.full ((a.max() + 1, ), -1)
&gt;&gt;&gt; lt[a_unique] = torch.arange (len (a_unique))
&gt;&gt;&gt; r = lt[a]
&gt;&gt;&gt; r
tensor([1, 0, 2, 3, 0, 3, 3])
</code></pre>
<p>(If the elements of <code>a</code> are integers but some are negative, you can<br>
still use the look-up-table approach by shifting the values of <code>a</code> up<br>
so that they become non-negative.)</p>
<p>Best.</p>
<p>K. Frank</p>
RuntimeError: Expected object of scalar type Long but got scalar type Float for argument #2 'target' in call to _thnn_nll_loss2d_forward

nikiguo93: <p>hi. I got this error “RuntimeError: Expected object of scalar type Long but got scalar type Float for argument <span class="hashtag">#2</span> ‘target’ in call to _thnn_nll_loss2d_forward”  when I executed the following code:<br>
model = torchvision.models.resnet50(pretrained=False)<br>
criterion=MSELoss().cuda()<br>
train_loss = 0<br>
for i_batch, sample_batched in enumerate(TraindataLoader):<br>
local_X = sample_batched[‘input’].cuda()<br>
local_Y = sample_batched[‘EEGResponse’].cuda()<br>
can anbody tell me how to solve this problem?</p>
<pre><code>    # compute the model output
    yhat = model(local_X)
    yhat= torch.reshape(yhat,[size_batches,1,17,100])
    # calculate loss
    loss = criterion(yhat, local_Y)
    train_loss += loss.item()
    losses.update(loss.item(), size_batches)
        
    # compute gradient and do SGD step
    optimizer.zero_grad()

        
    # credit assignment
    loss.backward()
    # update model weights
    optimizer.step()</code></pre>

KFrank: <p>Hi Nikiguo!</p>
<aside class="quote no-group quote-modified" data-username="nikiguo93" data-post="1" data-topic="133513" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nikiguo93/48/30791_2.png" class="avatar"> nikiguo93:</div>
<blockquote>
<p>“RuntimeError: Expected object of scalar type Long but got scalar type Float for argument <span class="hashtag-raw">#2</span> ‘target’ in call to _thnn_nll_loss2d_forward”<br>
…<br>
criterion=MSELoss().cuda()</p>
</blockquote>
</aside>
<p>Somewhere in your code you are not actually using <code>MSELoss</code><br>
as your <code>criterion</code> (but rather, most likely, <code>CrossEntropyLoss</code>).</p>
<aside class="quote no-group">
<blockquote>
<pre><code>    # calculate loss
    loss = criterion(yhat, local_Y)
</code></pre>
</blockquote>
</aside>
<p>At this point print out <code>type (criterion)</code> before calling<br>
<code>loss = criterion(yhat, local_Y)</code> to see what <code>criterion</code><br>
actually is.</p>
<p>Best.</p>
<p>K. Frank</p>

nikiguo93: <p>Hi. Thank you for your suggestion. I printed the type of criterion and it is crossentropyloss. But I define it as MSELoss. Previously I used to defined it as CrossEntropyLoss but now I change it. It is weird. Could you please tell me how to fix this problem?</p>
DDP Training with network of dynamic depth

YutongZheng: <p>Hi, I’m training a self-pruning network on DDP, i.e. the input of the network can go through different paths towards the output. However, in DDP, this seems to result in an error in my case:</p>
<pre><code class="lang-auto">RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one. This error indicates that your module has parameters that were not used in producing loss. You can enable unused parameter detection by (1) passing the keyword argument `find_unused_parameters=True` to `torch.nn.parallel.DistributedDataParallel`; (2) making sure all `forward` function outputs participate in calculating loss. If you already have done the above two steps, then the distributed data parallel module wasn't able to locate the output tensors in the return value of your module's `forward` function. Please include the loss function and the structure of the return value of `forward` of your module when reporting this issue (e.g. list, dict, iterable).
</code></pre>
<p>After I set <code>find_unused_parameters=True</code>, the above error goes away, but it seems like some of the parameters in prunable layers fail to synchronize (parameters in different GPUs are not the same one) after the update of one training step.</p>
<p>The desired behavior of the training is: as long as some of the processes in GPUs used the parameter, it always performs the update for this parameter across processes in ALL GPUs. Is there any way to make the training behavior possible with DDP? Thanks!</p>