akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Packing list of model into one model

Toby: <p>Hi,</p>
<p>I have a problem while using multiple models in a loop inside a <code>nn.Module</code>, an example script below shows my problem:</p>
<pre><code class="lang-python">from torch import nn 
import torch

class test_module(nn.Module):
    def __init__(self):
        super().__init__()
        self.model = nn.Sequential(*[nn.Linear(10,10), nn.ReLU()])
    def forward(self, x):
        return self.model(x)
        
class test_model(nn.Module):
    def __init__(self, num = 3):
        super().__init__()
        self.layers = list()
        for i in range(num):
            self.layers.append(test_module())
    def forward(self, x):
        for l in self.layers:
            out = l(x)
        return out
    
model = test_model()
print(f'Model architecture: {model}')

input = torch.rand(3,10)
out = model(input)
print(f'Output shape: {out.shape}')
</code></pre>
<p>The output is:</p>
<pre><code class="lang-auto">Model architecture: test_model()
Output shape: torch.Size([3, 10])
</code></pre>
<p>While I expect the model should print the combination of three <code>test_module</code>. This problem also affects when I tried to move the model to CUDA by:</p>
<pre><code class="lang-python">model.cuda()
input= input.cuda()
out = model(input)
</code></pre>
<p>Then the bug raises:</p>
<pre><code class="lang-auto">RuntimeError: Tensor for 'out' is on CPU, Tensor for argument #1 'self' is on CPU, but expected them to be on GPU (while checking arguments for addmm)
</code></pre>
<p>Can someone show me where am I wrong and how to fix it? Thank you.</p>

Toby: <p>Sorry, for those who don’t know, we have to use <code>nn.ModuleList()</code> instead of <code>list()</code></p>
One-hot encoding multi-label predicting all zeros

Krys: <p>Hi,<br>
I’m trying to implement a multi-label classification model starting from resnet18,<br>
In my dataset I have 188 classes and each element can belong to one, two or three classes, I’m using one hot encoding.<br>
So about 99% of attributes are zeros and my model predict only vectors with zeros and without 1.<br>
Which metrics and loss functions can I use to measure my model correctly?<br>
I’m trying to use</p>
<p>criterion = nn.BCEWithLogitsLoss(pos_weight = torch.ones([188]).to(device))</p>
<p>optimizer = optim.SGD(model.parameters(), lr=0.001)</p>
<p>Am I doing it right? ‘cause it seems doesn’t solve my problem.<br>
Thank you for help</p>

arya47: <p>BCE stands for Binary Cross Entropy and is for binary classification only. Try using cross entropy loss</p>

ptrblck: <p>You are right that <code>nn.BCEWithLogitsLoss</code> is used for binary classification use cases, but can also be used for multi-label classifications, while <code>nn.CrossEntropyLoss</code> is used for multi-class classifications (each sample has a single label only, which isn’t the case here).</p>
<p><a class="mention" href="/u/krys">@Krys</a> try to overfit a small dataset (e.g. just 10 samples) and make sure your model is able to do so by playing around with some hyperparamters. Once this is done you could try to scale up the use case again.</p>
No module found error

sandeep1: <p>Getting error, ModuleNotFoundError: No module named ‘cat’<br>
The file cat is in parent of the current file in which from is written.( as shown in the figure)</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/7/972de4c38ff30343195b4281249c124e97bc361d.png" data-download-href="https://discuss.pytorch.org/uploads/default/972de4c38ff30343195b4281249c124e97bc361d" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/7/972de4c38ff30343195b4281249c124e97bc361d_2_690x367.png" alt="image" data-base62-sha1="lzoqkk0wP9LFB0T3tzjpBnWyGx7" width="690" height="367" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/7/972de4c38ff30343195b4281249c124e97bc361d_2_690x367.png, https://discuss.pytorch.org/uploads/default/original/3X/9/7/972de4c38ff30343195b4281249c124e97bc361d.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/7/972de4c38ff30343195b4281249c124e97bc361d.png 2x" data-dominant-color="EFEFF0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">998×532 51.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>This issue seems to be related to the Python import system, so I would recommend to take a look at the <a href="https://docs.python.org/3/reference/import.html">docs</a>.<br>
Based on your description I guess you might need to append the <code>cat</code> path to <code>sys.path</code>.</p>
Get values return by Tuple Object in Maskcrnn libtorch

namduc: <p>Hi<br>
I’m a newbie in C++ and libtorch, I try load model by torchscript and execute inference and result like this:</p>
<pre><code class="lang-auto">Load model successful!
[W mask_rcnn.py:86] Warning: RCNN always returns a (Losses, Detections) tuple in scripting (function )
Inference time: 2321 ms
({}, [{boxes: [ CPUFloatType{0,4} ], labels: [ CPULongType{0} ], scores: [ CPUFloatType{0} ], masks: [ CPUFloatType{0,1,800,800} ]}])
</code></pre>
<p>How do I get value boxes, labels, scores and masks from return output object using c++ ?<br>
I tried many ways but compile always error with “c10::IValue” error thrown<br>
One more question, why is the time inference when I convert the model to torchscript, executed by C++ is slower than python ?<br>
Many thanks</p>
Retrieving multiple inputs / targets from data loader

Alan_Tessier: <p>Is there a way to return multiple values instead of two?<br>
In the example below, I can only return 1 input and 1 target.</p>
<pre><code class="lang-cpp">torch::data::Example&lt;&gt; KittiSet ::get(size_t index) {
  auto sample = data[index];
  // Get Images
  auto cv_img = cv::imread(sample["image"]);
  auto image = CVtoTensor(cv_img, WIDTH, HEIGHT);
  auto cv_depth = cv::imread(sample["depth"]);
  auto depth = CVtoTensor(cv_depth, WIDTH, HEIGHT);
  // Get Cam and Pose
  auto cam_data = txtToTensor(sample["cam"]);
  auto cam = torch::from_blob(cam_data.data(), {3, 3});
  auto pose_data = txtToTensor(sample["pose"]);
  auto len = (int)pose_data.size() / 12;
  auto pose = torch::from_blob(pose_data.data(), {len, 3, 4});

  return {image, depth};
}
</code></pre>
<p>Instead I would like to return 3 inputs and 1 target.<br>
In sum, I would like to return all the data instead of just a single input and target.<br>
Does anyone know how I can achieve this?</p>
<pre><code class="lang-cpp">  return {image,cam, pose depth}
</code></pre>
<p>Code: <a href="https://github.com/alantess/slam/blob/main/cpp/include/dataset.h" class="inline-onebox" rel="noopener nofollow ugc">slam/dataset.h at main · alantess/slam · GitHub</a></p>
PyTorch ddp with fork

amirhf: <p>The multiprocessing best practices in the documentations states:</p>
<p>“The CUDA runtime does not support the <code>fork</code> start method; either the <code>spawn</code> or <code>forkserver</code> start method are required to use CUDA in subprocesses”</p>
<p>Does this mean that I can’t write a ddp training script that works on gpus with ‘fork’?</p>
<p>I haven’t found a clear answer for this and I’m not sure what CUDA runtime means in the docs. In my specific use case, I kinda have to use ‘fork’ so I can pass object like data with shared memory.</p>
<p>If so, what are the limitations of using mp.Process with fork method?</p>

mrshenli: <p>Hey <a class="mention" href="/u/amirhf">@amirhf</a>,</p>
<p>It’s OK to fork a process as long as the parent process has not yet created a CUDA runtime/context. The CUDA context will be created lazily when the process creates CUDA tensors or run CUDA operations.</p>
<blockquote>
<p>In my specific use case, I kinda have to use ‘fork’ so I can pass object like data with shared memory.</p>
</blockquote>
<p>I would expect PyTorch Tensors <code>shared_memory</code> also works in spawn mode. Did you hit any error when doing that?</p>
How to kill entire training job if one GPU fails when using DDP

EricWiener: <p>Hi,</p>
<p>I’m using DDP to train on multiple GPUs and I’m using a shared server. I sometimes get errors where one GPU process will run out of memory for the dataloader and error out, but the other GPUs will keep going. Eventually, this results in the entire job freezing, but not erroring out (and resulting in large server costs without any progress being made). Is there any way to specify with DDP to kill all GPUs if one fails?</p>
<p>Thanks!</p>

mrshenli: <p>Hey <a class="mention" href="/u/ericwiener">@EricWiener</a>,</p>
<p>TorchElastic is designed to recover from such errors, see: <a href="https://pytorch.org/docs/stable/distributed.elastic.html" class="inline-onebox" rel="noopener nofollow ugc">Torch Distributed Elastic — PyTorch 1.9.0 documentation</a></p>
<p>If you just need the non-failing processes to crash when any peer hit OOM, you can set a timeout when calling <code>init_process_group</code>, and set <code>NCCL_ASYNC_ERROR_HANDLING</code> env var when using NCCL backend. see the <code>timeout</code> argument docstring for <code>init_process_group</code> at <a href="https://pytorch.org/docs/master/distributed.html#torch.distributed.init_process_group" class="inline-onebox" rel="noopener nofollow ugc">Distributed communication package - torch.distributed — PyTorch master documentation</a></p>
Can we input confusion matrix data to sklearn just for plotting confusion matrix?

hema_rathore: <p>I have a model in matlab , which I am not able to regenerate in Python. My question is can I just plot the confusion matrix in python, using any lib like sklearn or so by just inputting the confusion matrix.<br>
Plz help</p>

pascal_notsawo: <p>You just need predicted values and expected values to have your confusion matrix, with <a href="https://scikit-learn.org/stable/modules/generated/sklearn.metrics.confusion_matrix.html" rel="noopener nofollow ugc">sklearn.metrics.confusion_matrix</a> for example.</p>
<pre><code class="lang-auto">from sklearn.metrics import confusion_matrix
import seaborn as sns
import matplotlib.pyplot as plt

def plot_conf(y_true, y_pred, label : str = "", figsize=(7,4)) :
    cm = confusion_matrix(y_true, y_pred)
    fig, ax = plt.subplots(figsize=figsize)         # Sample figsize in inches
    fig.suptitle("confusion matrix %s"%label)
    f = sns.heatmap(cm, annot=True, linewidths=.5, ax=ax)
</code></pre>
<p>So you can for example find a way to store, from matlab, the latter in a file that you will read afterwards in python for your computations : for example for a vector <code>y</code>, a text file containing on each line an element of <code>y</code>, so that the reconstruction in python will be easy.</p>
<p>Sorry, I don’t master matlab, this is just the idea that comes directly to my mind.</p>

hema_rathore: <p>Sir I am not able to reproduce the predicted and true values here in python. Let me see, if i can do anything for that. But thanks for reply in my other code here in python I got this<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/5/3/53f6147b2ba087b7e46bef510b61efe084b5d8cb.png" alt="image" data-base62-sha1="bYKQWS7OZmPvDQxeEu2S8yU7Skj" width="394" height="252"><br>
Sir can I normalize it to find accuracy in percentage, like not in number.</p>
<p>Regards</p>

krishna511: <p>You can just follow this <a href="https://stackoverflow.com/questions/20927368/how-to-normalize-a-confusion-matrix" rel="noopener nofollow ugc">link</a>. You need to add this line</p>
<pre><code class="lang-auto">cmn = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]
</code></pre>

hema_rathore: <p>Thanks mam <a class="mention" href="/u/krishna511">@krishna511</a></p>
Group normalization with BN-pretrained model for object detection

TaojiannanYang: <p>Is it ok to directly replace the BN with GN(group normalization) in a BN-pretrained model(I don’t have time to train a GN-based model on ImageNet), and finetune the model for object detection? e.g. finetune on COCO, will there be a big difference between the performance?</p>

duddal: <p><a class="mention" href="/u/taojiannanyang">@TaojiannanYang</a> Did you find the answer to this question?</p>
Does the pytorch provide a pre-training model for the resnet with group normalization?

WBS_WHU: <p>Does the pytorch provide a pre-training model for the resnet with group normalization? the paper only provide pretrained model of resnet50\101, where can i find the resnet18 and others? Does the pytorch provide a pre-training model for the resnet with group normalization?</p>

duddal: <p><a class="mention" href="/u/wbs_whu">@WBS_WHU</a> Did you find the pretrained model?</p>
Using pretrained weights for ResNet18 with GroupNormalization and Standardized Weights

duddal: <p>Hello,</p>
<p>I have a custom Encoder-Decoder network. The Encoder I am using is a Resnet18. However, I can only use <code>batch_size=1</code> because of the memory constraints.<br>
Therefore, I replaced the BatchNorm layers with GroupNorm+Standardized_Weights as mentioned in the GroupNorm paper.</p>
<p>However, my question is whether I can use the pre-trained weights (pre-trained model) provided by PyTorch? If not, Can you please let me know whether we have a pre-trained ResNet model with GroupNorm + Standardized_Weights layers?</p>
<p>Thank You.</p>
Cross entropy for multi-dimensional implementation

Chris_XU: <p>Hi, I would like to understand the cross entropy loss for multi-dimensional input by implementing it by myself. Currently I am able to get the close result by iterating using <code>np.ndindex</code>,</p>
<pre><code class="lang-auto">K = 10
X = torch.randn(32, K, 20, 30)
t = torch.randint(K, (32, 20, 30)).long()
w = torch.randn(K).abs()

CrossEntropyLoss(weight=w, reduction="none")(X, t)

# this is same as

XX = X.movedim(1, -1)
Ls = torch.zeros_like(t, dtype=torch.float)
for ind in np.ndindex(32, 20, 30):
    label = t[ind]
    loss = -w[label] * torch.log(XX[ind][label].exp() / XX[ind].exp().sum())
    Ls[ind] = loss
Ls
</code></pre>
<p>But this involves looping which I assume would be slower than a direct vectorized computation. Therefore I am wondering if there are some tricks I can do to implement in plain torch. Thanks!</p>

Chris_XU: <p>Any ideas? I am looking for a more clean and better way to replace the loop. Thanks!</p>

KFrank: <p>Hi Chris!</p>
<aside class="quote no-group" data-username="Chris_XU" data-post="1" data-topic="128100" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/chris_xu/48/40766_2.png" class="avatar"> Chris_XU:</div>
<blockquote>
<p>But this involves looping which I assume would be slower than a direct vectorized computation.</p>
</blockquote>
</aside>
<p>You can get a tensor of weights corresponding to the class labels<br>
in <code>t</code> by indexing into <code>w</code>, and you can get a tensor of (negative)<br>
log-probabilities by calling <code>.take_along_dim()</code> with <code>t</code> as the<br>
argument:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.9.0'
&gt;&gt;&gt; _ = torch.manual_seed (2021)
&gt;&gt;&gt; K = 10
&gt;&gt;&gt; X = torch.randn(32, K, 20, 30)
&gt;&gt;&gt; t = torch.randint(K, (32, 20, 30)).long()
&gt;&gt;&gt; w = torch.randn(K).abs()
&gt;&gt;&gt; xeA = torch.nn.CrossEntropyLoss (weight = w, reduction = 'none') (X, t)
&gt;&gt;&gt; xeB = w[t] * -X.log_softmax (1).take_along_dim (t.unsqueeze (1), 1).squeeze (1)
&gt;&gt;&gt; torch.allclose (xeA, xeB)
True
&gt;&gt;&gt; torch.equal (xeA, xeB)
True
</code></pre>
<p>As an aside:</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">    loss = -w[label] * torch.log(XX[ind][label].exp() / XX[ind].exp().sum())
</code></pre>
</blockquote>
</aside>
<p>you are computing log-softmax() “by hand” in the straightforward way,<br>
but it is numerically more stable to use the log-sum-exp trick, either<br>
by implementing it explicitly or by using pytorch’s <code>log_softmax()</code><br>
function (as I have done above).</p>
<p>Best.</p>
<p>K. Frank</p>

Chris_XU: <p>Thanks! This is a clever way! I didn’t know there was a <code>take_along_dim</code> function beforehand. Thanks a lot for pointing out the log_softmax trick!</p>
Pytorch installation with AMD Ryzen CPU and Nvidia GPU

braindotai: <p>Hello there, I’m about to buy a laptop with AMD Ryzen 7 Octa Core 4800H and GPU NVIDIA GeForce GTX 1660 Ti.<br>
I’m just wanna make sure that I wouldn’t face any issue with Torch cuda installation on this system (windows). Are there any issues that I could face later?</p>

arya47: <p>Just something i want to add from personal experience, keep in mind gpu memory when buying a 1660Ti, 4gb / 6gb is reasonable for simple models but you might face a lot of hurdles in working with projects like medical imaging where you’ll be limited to using a batch size of 1 or 2.</p>

braindotai: <p>Yeah I’m totally fine with 6gb vram, all I wanna do is some basic GPU testing with the models for inference, that’s all. No long-run training. So all I’m worried about is about the torch+cuda (and tf-gpu) installation on the windows system with AMD Ryzen CPU and Nvidia GPU.</p>
RuntimeError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 10.92 GiB total capacity; 10.28 GiB already allocated; 11.00 MiB free; 33.95 MiB cached)

asha97: <p>I tried to decrease the dataset but error remain same.</p>
<pre><code class="lang-auto">Creating data loaders
Traceback (most recent call last):
  File "main_gdt.py", line 504, in &lt;module&gt;
    main(args)
  File "main_gdt.py", line 225, in main
    writer=writer,
  File "main_gdt.py", line 274, in train_one_epoch
    feats1 = compute_feats(model, video, audio)
  File "/DATA/rani1/GDT-main/gdt_helper.py", line 18, in compute_feats
    feat_v, feat_a = model(video1, audio1) if feats1 is None else feats1
  File "/DATA/rani1/miniconda3/envs/GDT/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/DATA/rani1/GDT-main/model.py", line 259, in forward
    img_features = self.video_network(img).squeeze()
  File "/DATA/rani1/miniconda3/envs/GDT/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/DATA/rani1/GDT-main/model.py", line 187, in forward
    x = self.base(x).squeeze()
  File "/DATA/rani1/miniconda3/envs/GDT/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/DATA/rani1/GDT-main/src/vmz.py", line 237, in forward
    x = self.layer3(x)
  File "/DATA/rani1/miniconda3/envs/GDT/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/DATA/rani1/miniconda3/envs/GDT/lib/python3.7/site-packages/torch/nn/modules/container.py", line 92, in forward
    input = module(input)
  File "/DATA/rani1/miniconda3/envs/GDT/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/DATA/rani1/GDT-main/src/vmz.py", line 113, in forward
    out = self.conv2(out)
  File "/DATA/rani1/miniconda3/envs/GDT/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/DATA/rani1/miniconda3/envs/GDT/lib/python3.7/site-packages/torch/nn/modules/container.py", line 92, in forward
    input = module(input)
  File "/DATA/rani1/miniconda3/envs/GDT/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/DATA/rani1/miniconda3/envs/GDT/lib/python3.7/site-packages/torch/nn/modules/container.py", line 92, in forward
    input = module(input)
  File "/DATA/rani1/miniconda3/envs/GDT/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/DATA/rani1/miniconda3/envs/GDT/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 480, in forward
    self.padding, self.dilation, self.groups)

</code></pre>

sandeep1: <p>please share your forward function. It’s hard to follow this.</p>
ConvTranspose2d on one (1) pixel

Mwni: <p>The goal: transform a 1d vector of length 512 into an 3x299x299 image.</p>
<p>The attempt is to transform each value of the vector into a 1x1 channel, then deconvolve/upsample from there. Sadly, trying to run ConvTranspose2d over a tensor of size (512, 1, 1) throws</p>
<pre><code class="lang-auto">Kernel size can not be greater than actual input size
</code></pre>
<p>Below is an illustration of the desired transformation.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/5/f50d8f313dd1d2f6dc55043d3d524c9c5faa45d3.png" data-download-href="https://discuss.pytorch.org/uploads/default/f50d8f313dd1d2f6dc55043d3d524c9c5faa45d3" title="pixelupsample"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/5/f50d8f313dd1d2f6dc55043d3d524c9c5faa45d3_2_690x265.png" alt="pixelupsample" data-base62-sha1="yXQ1VfDuj2dSKaRZ73mhqYwvcBB" width="690" height="265" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/5/f50d8f313dd1d2f6dc55043d3d524c9c5faa45d3_2_690x265.png, https://discuss.pytorch.org/uploads/default/original/3X/f/5/f50d8f313dd1d2f6dc55043d3d524c9c5faa45d3.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/f/5/f50d8f313dd1d2f6dc55043d3d524c9c5faa45d3.png 2x" data-dominant-color="D1E1F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">pixelupsample</span><span class="informations">831×320 25.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I can’t be the first one to try this, how is this usually done?</p>

ptrblck: <p>It works for me trying to create an output of <code>2x2</code> from a single pixel:</p>
<pre><code class="lang-python">x = torch.randn(1, 1, 1, 1)
conv_trans = nn.ConvTranspose2d(1, 1, kernel_size=2)
out = conv_trans(x)
print(out.shape)
&gt; torch.Size([1, 1, 2, 2])
</code></pre>
<p>so I guess your setup might be different?</p>

Mwni: <p>You are right. Since I had built the convolutions in a loop, I had set <code>stride=2</code>. This was the reason why it wasn’t working.</p>
<p>Thank you!</p>
How to add 2 tensors of unequal shapes in the last dimension?

Navpreet_Singh: <p>I’m trying to add 2 tensors in the forward() function.</p>
<p>Eg Tensor 1 shape = (1,365,12) and Tensor 2 shape = (1,365,10) . I want the result such that Result Tensor shape = (1,365,12) but we only add the first 10 dim on Tensor1 with Tensor2 and the last 2 dim remain unchanged.  (Changing the shape of Tensor 2 to add 2 zero columns in last dim is not possible).</p>
<p>I tried doing it using this approach : Tensor1[:, :, :10] = Tensor1[:, :, :10] + Tensor2. Also, fyi I got Tensor1 and Tensor2 using torch.split(Tensor3,[12,10],dim=2) where Tensor3 shape = (1,365,22) .<br>
It works, although I get the following warning :</p>
<p>Output 0 of SliceBackward is a view and is being modified inplace. This view is an output of a function that returns multiple views. Inplace operators on such views are being deprecated and will be forbidden starting from version 1.8. Consider using <code>unsafe_</code> version of the function that produced this view or don’t modify this view inplace. (Triggered internally at  …\torch\csrc\autograd\variable.cpp:547.)</p>
<p>Is there any alternate way I can approach this?</p>

googlebot: <p>try</p>
<pre><code class="lang-auto">result = Tensor3[...,:12].clone()
result[...,:10] += Tensor3[...,12:22]
</code></pre>
<p>this may work without clone, but output will be non-contiguous, and it may get cloned by autograd anyway</p>
How to find input corresponding to argmin of output?

samarendra109: <p>I have a basic scenario like this,</p>
<pre><code class="lang-auto">import torch

def someFn(w):
    
    return w.sum(dim=1)

w = torch.rand(10,3,2)
fW = someFn(w)

fMinW, fIndex = fW.min(dim=1)

# Select wmin of size 10,3
</code></pre>
<p>Now I want to select the 10 (1x3) vectors that correspond to the fMinW output. How can I achieve this ?</p>

ptrblck: <p>Assuming the desired output can be created via:</p>
<pre><code class="lang-python">res2 = []
for idx in range(w.size(0)):
    res2.append(w[idx, :, fIndex[idx]])
res2 = torch.stack(res2)
</code></pre>
<p>then this should work:</p>
<pre><code class="lang-python">res1 = w[torch.arange(w.size(0)), :, fIndex]

# compare
print((res2 == res1).all())
&gt; tensor(True)
</code></pre>
Activation gradient penalty

michaelklachko: <p>I’m trying to penalize the norm of activation gradients:</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 32, kernel_size=5)
        self.conv2 = nn.Conv2d(32, 64, kernel_size=5)
        self.pool = nn.MaxPool2d(2, 2)
        self.relu = nn.ReLU()
        self.linear = nn.Linear(64 * 5 * 5, 10)

    def forward(self, input):
        conv1 = self.conv1(input)
        pool1 = self.pool(conv1)
        self.relu1 = self.relu(pool1)
        self.relu1.retain_grad()
        conv2 = self.conv2(relu1)
        pool2 = self.pool(conv2)
        relu2 = self.relu(pool2)
        self.relu2 = relu2.view(relu2.size(0), -1)
        self.relu2.retain_grad()
        return self.linear(relu2)

model = Net()
optimizer = torch.optim.SGD(model.parameters(), lr=0.001)

for i in range(1000):
    output = model(input)
    loss = nn.CrossEntropyLoss()(output, label)
    optimizer.zero_grad()
    loss.backward()

    grads = torch.autograd.grad(loss, [model.relu1, model.relu2], create_graph=True)

    grad_norm = 0
    for grad in grads:
	grad_norm += grad.pow(2).sum()

    grad_norm.backward(retain_graph=True)

    optimizer.step()
</code></pre>
<p>However, it does not produce the desired regularization effect. If I do the same thing for weight gradients, it works well. Am I doing this right? Specifically, what happens in <code>grad_norm.backward()</code> step? I just want to make sure the weight gradients are updated, and not activation gradients. Currently, when I print out gradients for weights and activations immediately before and after that line, both change - so I’m not sure what’s going on.</p>

harshays: <p>Hi <a class="mention" href="/u/michaelklachko">@michaelklachko</a>, if you were able to resolve this issue, can you take a look at <a href="https://discuss.pytorch.org/t/optimizing-a-loss-based-on-autograd-grad-output/129360" class="inline-onebox">Optimizing a loss based on autograd.grad output</a>? I think I have a similar problem.</p>
RuntimeError: Function MmBackward returned an invalid gradient at index 1 - got [128, 16] but expected shape compatible with [4, 16]

moonsheep: <p>Hello there,<br>
I’m trying to implement (copypaste from different sources) the DDQN algorithm in PyTorch. I get the following error when calling <code>loss.backward()</code>: <code>RuntimeError: Function MmBackward returned an invalid gradient at index 1 - got [128, 16] but expected shape compatible with [4, 16]</code>.<br>
I believe this is the relevant code:</p>
<pre><code class="lang-auto">        transitions = self.memory.sample()
        batch_state, batch_next_state, batch_action, batch_reward = zip(*transitions)

        batch_state = torch.cat(batch_state)
        batch_next_state = torch.cat(batch_next_state)
        batch_action = torch.cat(batch_action).squeeze(1)
        batch_reward = torch.cat(batch_reward)

        # current Q values are estimated by NN for all actions
        current_q_values = self.policy_net(batch_state).gather(0, batch_action)

        # expected Q values are estimated from actions which gives maximum Q value
        max_next_q_values, _ = self.target_net(batch_next_state).detach().max(0)
        expected_q_values = batch_reward + (self.gamma * max_next_q_values)

        # Compute loss
        loss = self.criterion(current_q_values, expected_q_values)

        # Zero the gradients and backpropagate
        self.optimizer.zero_grad()
        loss.backward()
        for param in self.policy_net.parameters():
            param.grad.data.clamp_(-1, 1)

        # Optimizer
        self.optimizer.step()
</code></pre>
<p>Both <code>policy_net</code> and <code>target_net</code> are just two linear layers with ReLU activation.<br>
The error makes me believe that the graph is messed up somewhere, but I’m not sure how such a situation can be reached. I assume MmBackward refers to a <code>torch.mm</code> computation, but I don’t think I ever call torch.mm.<br>
I am incredible lost and any help would be much appreciated. Thanks in advance!</p>
Why could imshow a phote with (x,x,3)?

wukong: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/4/04438dcddda7666c4e13955dfc259566db9a2f11.png" data-download-href="https://discuss.pytorch.org/uploads/default/04438dcddda7666c4e13955dfc259566db9a2f11" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/4/04438dcddda7666c4e13955dfc259566db9a2f11_2_581x500.png" alt="image" data-base62-sha1="BIDTAsS4PFtxz5ClYbBkMQRHlT" width="581" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/4/04438dcddda7666c4e13955dfc259566db9a2f11_2_581x500.png, https://discuss.pytorch.org/uploads/default/original/3X/0/4/04438dcddda7666c4e13955dfc259566db9a2f11.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/4/04438dcddda7666c4e13955dfc259566db9a2f11.png 2x" data-dominant-color="DDDBDC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">760×654 76.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I know a pic usually has 3 channels, so I think grid in plt.imshow() is a photo. But is still seems stange for me. Maybe I should read more blog.</p>

eqy: <p>It looks like this is a channels-last image format (which is common among image processing libraries like PIL).</p>

wukong: <p>thank you I read the code angin and angin. Your answer help me a lot for the understand of this code^_^</p>
Why cpp_extension.load not create a .so file in temp dir?

LuoXin-s: <p>I have this strange build process, First load system detect I have no needed file and create a dir, then another detect like description is printed and following <strong>“No modifications detected for re-loaded extension module fused, skipping build step…”.</strong></p>
<p><strong>I didn’t even see the compilation process.</strong><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/5/051ac1e024a475537344f45e3aa4691e99c1e821.png" data-download-href="https://discuss.pytorch.org/uploads/default/051ac1e024a475537344f45e3aa4691e99c1e821" title="VUWPU_JG402@KKO$A4TRNE"><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/5/051ac1e024a475537344f45e3aa4691e99c1e821.png" alt="VUWPU_JG402@KKO$A4TRNE" data-base62-sha1="J9Ij9prKIrBKByBldkaRo9aiyt" width="690" height="403" data-dominant-color="121213"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">VUWPU_JG402@KKO$A4TRNE</span><span class="informations">865×506 18 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Usage of a calculated loss of different shape than model output for backpropagation

johannesgensheimer: <p>Hello everyone,</p>
<p>I’m building a CNN model in PyTorch for Image Super Resolution. After applying the model to the low resolution image I want to validate only a small part of the image against a high resolution image, meaning that I have to post process the model output (shape 100x100) so that it is only an array (shape 50).<br>
So basically here are my steps:</p>
<blockquote>
<p>mse_loss = nn.MSELoss()<br>
optimizer = optim.Adam(model.parameters(), lr=params[“learning_rate”], weight_decay=params[“weight_decay”])<br>
pred = model(input) <span class="hashtag-raw">#pred</span>.shape: (1,1,100,100)<br>
pred_array = CalculatePredArray(pred) <span class="hashtag-raw">#pred_array</span>.shape: (1,1,50), CalculatePredArray own function<br>
loss = mse_loss(pred_array, target_array) <span class="hashtag-raw">#target</span> array has also shape (1,1,50)<br>
loss.backward()<br>
optimizer.step()</p>
</blockquote>
<p>Error message: RuntimeError: The size of tensor a (100) must match the size of tensor b (50) at non-singleton dimension 3</p>
<p>My model only consists of nn.Conv2d and nn.ReLU layers, so it is independent of any input or output shape. I’m also not using any flattening or similar.</p>
<p>It should be possible to train the model with a loss of different shape, right? The shape shouldn’t matter, as the model only consists of kernels. But I don’t know how to write the according code for that and didn’t find anything that solved my problem online.</p>
<p>Does anyone know how to implement that?</p>
<p>Thank you very much in advance!</p>
<p>Best regards,<br>
Johannes</p>

ptrblck: <p>Your code should work as seen here:</p>
<pre><code class="lang-python">model = nn.Conv2d(3, 1, 3, 1, 1)
input = torch.randn(1, 3, 100, 100)

pred = model(input)
pred_array = pred[:, :, :50, 0]
print(pred_array.shape)
&gt; torch.Size([1, 1, 50])
loss = F.mse_loss(pred_array, torch.rand_like(pred_array))
loss.backward()
</code></pre>
<p>and the error message indicates that <code>pred_array</code> does not have the posted shape but has a shpe of 100 in <code>dim3</code>.</p>

johannesgensheimer: <p>It’s working! Thank you A LOT <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
My mistake was to use nn.MSELoss instead of F.mse_loss. That was very helpful!</p>
Pytorch with Windows 7 dose not support training in multi-GPUs?

Miraitowa: <p>When I used “model = nn.DataParaller(model)” to use 3 GPUS(RTX2080TI)  to training my model, there were a warning showing that “PyTorch is not compiled with NCCL support”.<br>
Dose  not  Pytorch with Windows 7 support training in multi-GPUs ?</p>

ptrblck: <p>Based on <a href="https://github.com/NVIDIA/nccl/issues/442#issuecomment-877714987">this comment</a> it seems that the latest NCCL release (2.10.3) is supporting WSL2 partially, but I don’t think that a native Windows NCCL version exits, so you might need to check other distributed backends.</p>

Miraitowa: <p>Thanks for your reply. But I maybe not describe clearly. The system of my computer is Windows 7 and I do not use WSL2. The version of PyTorch I used is for Winodws, and I use Acaconda3 to manage my environment.</p>

ptrblck: <p>Yes, I understand your setup and don’t think there is a Windows-supported NCCL version out (at least no official one; in the past there might have been forks etc.).</p>
Pytorch support int16 quantization

kwgqjj: <p>Hello, my friends</p>
<pre><code>I have a question about int16 quantization of pytorch. Does pytorch have the plan to support int16 quantization in the future?  I'm looking forward to your reply. Thanks.</code></pre>

jerryzh168: <p>maybe, it depends on the trend in the hardwares we want to support, we do not have concrete plans to do that currently though. Please let us know if you have a prevailing use case for that.</p>
Dynamic quantization

Abdulaziz_Gaybulayev: <p>I have tried  <strong>post training dynamic quantization</strong> with YOLOv5 model. Model file is available here (<a href="https://github.com/ultralytics/yolov5/releases/download/v5.0/yolov5s.pt" rel="noopener nofollow ugc">https://github.com/ultralytics/yolov5/releases/download/v5.0/yolov5s.pt</a>)</p>
<p>When I tried to quantize, model size is increasing twice and inferencing time is same with FP32 model.</p>
<p>Based on pytorch tutorial (<a href="https://pytorch.org/tutorials/recipes/recipes/dynamic_quantization.html" class="inline-onebox" rel="noopener nofollow ugc">Dynamic Quantization — PyTorch Tutorials 1.9.0+cu102 documentation</a>).</p>
<pre><code class="lang-auto">model = attempt_load(weights, map_location=device)
quantized_model = torch.quantization.quantize_dynamic(model, {torch.nn.Conv2d, torch.nn.Linear}, dtype=torch.qint8)  #int8 
# model = torch.quantization.quantize_dynamic(model, dtype=torch.qint8)  #int8 
</code></pre>
<p>I am not sure <code>{torch.nn.Conv2d, torch.nn.Linear}</code> ( list of submodule names in model to apply quantization to) is correctly using or not when applying it to YOLO model</p>
<p>I even checked the dtype using<br>
<code>for param in quantized_model.parameters(): print(param.dtype)</code></p>
<p>It is still fp32</p>
<p>Thank you for advances beforehand</p>
<p>Thank you for advances beforehand</p>

jerryzh168: <aside class="quote no-group" data-username="Abdulaziz_Gaybulayev" data-post="1" data-topic="129257">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/abdulaziz_gaybulayev/48/41224_2.png" class="avatar"> Abdulaziz_Gaybulayev:</div>
<blockquote>
<p>When I tried to quantize, model size is increasing twice and inferencing time is same with FP32 model.</p>
</blockquote>
</aside>
<p>can you print the model after quantize_dynamic? also I think we do not support dynamic quantization for torch.nn.Conv2d right now.</p>
Should I explicitly define mask for transformer decoder or it is already there by default?

seyeeet: <p>Hi<br>
I like to train a language model and want to use transformer for it. I have heard that masks in the dcoders are very importnat while training, for example some mask like this is used in training models like bert in decoder.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/d/5/d5f318eaa9bb3797ee370d310fad4d5ec50843ee.png" alt="image" data-base62-sha1="uwGx9U37tEOJ3FR2klz6xfzD8my" width="440" height="340"><br>
I was wondering if I have to define them or there are already in the model by default?</p>
Why the output is different when batching?

Astariul: <p>I thought that for same input, we should receive same output, no matter the batch size.</p>
<p>But I tried a small experiment, and it’s not confirming my intuition…</p>
<hr>
<p>I define a simple network :</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

smol = nn.Sequential(
    nn.Linear(256, 256),
    nn.ReLU(),
    nn.Linear(256, 256),
    nn.ReLU()
)
</code></pre>
<p>Then I try 2 inputs : a batch with only sample x, and a batch with sample x and y. I expect the output of sample x to be the same in both case, but it’s not :</p>
<pre><code class="lang-auto">x = torch.rand([1,  256])                           # Batch with only x 
xy = torch.cat([x, torch.rand([1,  256])], dim=0)   # Batch with x and another sample

out_alone = smol(x)[0]
out_batch = smol(xy)[0]

assert torch.equal(out_alone, out_batch), f"\n{out_alone[:15]}\n{out_alone[:15]}"
</code></pre>
<blockquote>
<p>AssertionError:<br>
tensor([0.0870, 0.0757, 0.0076, 0.0000, 0.0000, 0.0000, 0.0032, 0.0976, 0.0000,<br>
0.0648, 0.2508, 0.0000, 0.1737, 0.0546, 0.0043],<br>
grad_fn=)<br>
tensor([0.0870, 0.0757, 0.0076, 0.0000, 0.0000, 0.0000, 0.0032, 0.0976, 0.0000,<br>
0.0648, 0.2508, 0.0000, 0.1737, 0.0546, 0.0043],<br>
grad_fn=)</p>
</blockquote>
<p><a href="https://colab.research.google.com/drive/1DL4nH_weVexteBW7wV_6Y_R8ta3L_AZW?usp=sharing" rel="noopener nofollow ugc">Colab notebook to reproduce it</a></p>
<hr>
<p>Even if it seems to be only a precision difference, why is the output not exactly the same ?</p>
<p><strong>Why the output is dependent on the batch size, when the input is same ?</strong></p>

ptrblck: <p>Different algorithms could be used for different input shapes, which would thus create the small errors due to the limited numerical precision.<br>
E.g. a different order of operations would also create these numerical differences as seen here:</p>
<pre><code class="lang-python">x = torch.randn(10, 10, 10)
s1 = x.sum()
s2 = x.sum(0).sum(0).sum(0)
print((s1 - s2).abs().max())
&gt; tensor(9.5367e-07)
</code></pre>
<p>and you cannot assume to get bitwise-accurate results using different algorithms.</p>
Attention type 'block_sparse' is not possible if sequence_length: 458 <= num global tokens:

Krishna_Garg: <p>I am using the pre-trained <code>google/bigbird-pegasus-large-arxiv</code> model.</p>
<p>But I receive the following update during the forward pass.</p>
<pre><code class="lang-auto">Attention type 'block_sparse' is not possible if sequence_length: 458 &lt;= num global tokens: 2 * config.block_size + min. num sliding tokens: 3 * config.block_size + config.num_random_blocks * config.block_size + additional buffer: config.num_random_blocks * config.block_size = 704 with config.block_size = 64, config.num_random_blocks = 3.Changing attention type to 'original_full'...
</code></pre>
<p>I understand the update and I am aware of benefit of time and memory it saves while using <code>block_sparse</code> than <code>original_full</code>.</p>
<p>So, how should I go about selecting the suitable <code>block_size</code> and  <code>num_random_blocks</code> when I know that there is a lot of variation in the sequence length of my inputs?</p>
Append two datasets (MNIST and SVHN)

CasellaJr: <p>I have two datasets, MNIST and SVHN.<br>
It is known that the num. test samples of MNIST is 1000 and the num. test samples of SVHN is 26032.<br>
Now, i want to append the test set of SVHN to MNIST, in the sense that test_set_append[9999] is the last element of MNIST, and test_set_append[10000] is the first element of SVHN.<br>
I have already tried with the ConcatDataset:</p>
<pre><code class="lang-auto">class ConcatDataset(torch.utils.data.Dataset):
    def __init__(self, *datasets):
        self.datasets = datasets

    def __getitem__(self, i):
        return tuple(d[i] for d in self.datasets)

    def __len__(self):
        return min(len(d) for d in self.datasets)
</code></pre>
<p>But the result is not what I want:</p>
<pre><code class="lang-auto">test_set = ConcatDataset(test_set_a, test_set_b)
test_set[1]
((tensor([[[0., 0., 0.,  ..., 0., 0., 0.],
           [0., 0., 0.,  ..., 0., 0., 0.],
           [0., 0., 0.,  ..., 0., 0., 0.],
           ...,
           [0., 0., 0.,  ..., 0., 0., 0.],
           [0., 0., 0.,  ..., 0., 0., 0.],
           [0., 0., 0.,  ..., 0., 0., 0.]]]), 2),
 (tensor([[[0.5451, 0.5451, 0.5451,  ..., 0.5176, 0.5216, 0.5255],
           [0.5725, 0.5686, 0.5686,  ..., 0.5373, 0.5333, 0.5333],
           [0.6039, 0.6000, 0.6000,  ..., 0.5686, 0.5569, 0.5490],
           ...,
           [0.6196, 0.6000, 0.5804,  ..., 0.3882, 0.4000, 0.4353],
           [0.5843, 0.5529, 0.5216,  ..., 0.3961, 0.4235, 0.4667],
           [0.5569, 0.5216, 0.4824,  ..., 0.4275, 0.4667, 0.5137]]]), 2))
</code></pre>

ptrblck: <p>In your custom <code>ConcatDataset</code> you are returning a <code>tuple</code> containing samples of both datasets, but I think <code>torch.utils.data.ConcatDataset</code> would directly work.</p>

CasellaJr: <p>So, you mean this?<br>
<code>test_set=torch.utils.data.ConcatDataset([test_set_a, test_set_b])</code></p>

ptrblck: <p>Yes, I believe this should work, but let us know if that’s not the case.</p>

CasellaJr: <p>Yes, it worked. Thank you!</p>
Pytorch version in torch.cat

Mohamed_Nabih: <p>I am trying to make a concatenation operation using this line of code:<br>
‘’’<br>
mixture = torch.cat([mixture, torch.zeros(1, 1, padded_length, device=device2)], dim=-1)<br>
‘’’’<br>
Actually, it works well with PyTorch version 1.4.0 but in PyTorch version 1.7.1 it complains can anyone explain Why?</p>

Mohamed_Nabih: <p><a class="mention" href="/u/alband">@albanD</a>  Do you have an explanation?</p>

ptrblck: <p>Could you describe the issue a bit more, please, and explain what the issue is exactly? Also, an executable code snippet to reproduce the issue in the latest release (assuming you are also seeing it in <code>1.9.0</code>) would be helpful.</p>
<p>PS: I would not recommend to tag specific users, as it could discourage others to take a look at your question.</p>

Mohamed_Nabih: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>  Sorry for specific tagging I will not do this again. I solved this problem.<br>
I have another question I have a tensor of shape [1, 1, 64000] where the first 1 is the batch size can I make it of shape [1, 64000] i.e remove the first dim.</p>

ptrblck: <p>Yes, you can remove dimensions with a size of <code>1</code> by using <code>tensor = tensor.squeeze(dim)</code>, where <code>dim=0</code> in your case.</p>

Mohamed_Nabih: <p>So to if the tensor (x) shape for example [2, 1, 64000] and want to reshape it to anothe tensor (y) of shape [1, 64000] I will use<br>
y = x.squeeze(dim=0)</p>

ptrblck: <p>No, squeezing a dimension with <code>size&gt;1</code> won’t work, since you would need to reduce the number of elements and could index the tensor instead e.g. via <code>y = x[0]</code>.</p>
Split a tensor into two groups using a mask matrix

amir_veyseh: <p>Given the tensor <code>A</code> with shape <code>(b,n,d)</code> and another matrix <code>M</code> with shape <code>(b,n)</code>, what is the most efficient way to split <code>A</code> into two tensors <code>B</code> and <code>C</code> with shapes <code>(b,l,d)</code> and <code>(b,m,d)</code>, respecitvely, where <code>n=l+m</code> and the row <code>A[k,i,:]</code> should be included in tensor <code>B</code> if <code>M[k,i] == 1</code>, otherwise it should be included  in <code>C</code>.</p>
<p>The trivial solution is to have a for loop and selects the row in <code>A</code> based on the corresponding values in <code>M</code>. Is there any way to do this splitting in parallel?</p>

ptrblck: <p>I guess you could use <code>scatter_</code> for this operation. Could you post your loop approach, so that we could check how exactly you are currently indexing the tensors?</p>

amir_veyseh: <p>Thanks for your response. Here’s the loop approach:</p>
<pre><code class="lang-auto">        B = []
        C = []
        for i in range(M.shape[0]):
            b = []
            c = []
            for j in range(M.shape[1]):
                if M[i,j] == 1:
                    c += [A[i,j]]
                else:
                    b += [A[i,j]]
            B += [torch.stack(b, dim=0)]
            C += [torch.stack(c, dim=0)]
</code></pre>
<p>There might be some padding required to convert B and C to tensors, but this is the general idea of the loop approach.</p>

ptrblck: <p>Assuming <code>M</code> only contains zeros and ones (or you could <code>clamp</code> it to these values), you could use it as a mask as seen here:</p>
<pre><code class="lang-python"># setup
b, n, d = 2, 10, 4
A = torch.randn(b, n, d)
M = torch.randint(0, 2, (b, n))

# your approach
B = []
C = []
for i in range(M.shape[0]):
    b = []
    c = []
    for j in range(M.shape[1]):
        if M[i,j] == 1:
            c += [A[i,j]]
        else:
            b += [A[i,j]]
    B += [torch.stack(b, dim=0)]
    C += [torch.stack(c, dim=0)]

B = torch.cat(B)
C = torch.cat(C)

# alternative
B_ = A[~M.bool()]
C_ = A[M.bool()]

print((B == B_).all())
&gt; tensor(True)
print((C == C_).all())
&gt; tensor(True)
</code></pre>

FruitVinegar: <p>In this case, how to re-concatenate again with B, C, M to A?<br>
Like below:</p>
<pre><code class="lang-auto">A_again = mask_concatenate(B, C, M)
assert (A == A_again).all() == True
</code></pre>