akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Jit runs 3x slower than PyTorch

passion3394: <pre><code>for j in range(100):
    start = time.time()
    torch.cuda.synchronize()

    _ = model(inputs)
    torch.cuda.synchronize()

    end = time.time()
    print('torch infer time: ', (end-start))

script_module = torch.jit.script(model.cuda().eval(),inputs)

for j in range(100):
    start = time.time()
    torch.cuda.synchronize()

    _ = script_module(inputs)
    torch.cuda.synchronize()

    end = time.time()
    print('torch jit infer time: ', (end-start))
</code></pre>
<p>pytorch forward time is about 100ms, but jit forward time is about 330 ms, any suggestions?</p>
<p>torch infer time:  0.10617709159851074<br>
torch infer time:  0.10543274879455566<br>
torch infer time:  0.10464048385620117<br>
torch infer time:  0.10502433776855469<br>
torch infer time:  0.10710620880126953<br>
torch infer time:  0.10780525207519531<br>
torch infer time:  0.11376476287841797<br>
torch infer time:  0.10918307304382324<br>
torch infer time:  0.10780143737792969<br>
torch infer time:  0.10900402069091797<br>
torch infer time:  0.10415053367614746<br>
torch infer time:  0.10154843330383301<br>
torch infer time:  0.10197162628173828<br>
torch infer time:  0.10551595687866211<br>
torch infer time:  0.10509729385375977<br>
torch infer time:  0.10562729835510254<br>
torch infer time:  0.1086127758026123<br>
torch jit infer time:  0.5637478828430176<br>
torch jit infer time:  0.3401811122894287<br>
torch jit infer time:  0.3404266834259033<br>
torch jit infer time:  0.322681188583374<br>
torch jit infer time:  0.325390100479126<br>
torch jit infer time:  0.3332507610321045<br>
torch jit infer time:  0.33599114418029785<br>
torch jit infer time:  0.33664798736572266<br>
torch jit infer time:  0.3400590419769287<br>
torch jit infer time:  0.33101773262023926<br>
torch jit infer time:  0.3290140628814697<br>
torch jit infer time:  0.3344461917877197<br>
torch jit infer time:  0.3417356014251709<br>
torch jit infer time:  0.33629822731018066<br>
torch jit infer time:  0.3459322452545166</p>

passion3394: <p>The model has many tensor creation ops, is it the reason?</p>
Matmul in transformers

my3bikaht: <p>Sorry, maybe it’s a stupid question and it’s already late, how grad computed for matrix multiplied by itself from the calculus standpoint? I mean if we have matrix m x m and multiply it by itself just like in self attention in transformers, isn’t it already non-linearity without activation function?</p>

KFrank: <p>Hi Sergey!</p>
<aside class="quote no-group" data-username="my3bikaht" data-post="1" data-topic="154917" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/50afbb/48.png" class="avatar"> my3bikaht:</div>
<blockquote>
<p>how grad computed for matrix multiplied by itself from the calculus standpoint?</p>
</blockquote>
</aside>
<p>It’s just regular calculus.  Let’s say that <code>A</code> is an <code>m x m</code> matrix.  Then<br>
<code>(A @ A)[i, j] = sum_k (A[i, k] * A[k, j])</code>.  To compute the full<br>
Jacobian of <code>A @ A</code> with respect to <code>A</code>, you need to evaluate the derivatives<br>
<code>d (A @ A)[i, j] / d A[k, l]</code> for all values of the indices <code>i</code>, <code>j</code>, <code>k</code>, and <code>l</code>.<br>
You have a simple sum of products, so doing so is straightforward, although<br>
it gets a bit fussy keeping track of the indices.</p>
<aside class="quote no-group">
<blockquote>
<p>isn’t it already non-linearity without activation function?</p>
</blockquote>
</aside>
<p>Yes, it is non-linear, and you don’t have a separate activation function.  But<br>
that’s okay – there’s nothing problematic with autograd computing gradients<br>
of non-linear expressions:</p>
<pre data-code-wrap="python"><code class="lang-python">x = torch.tensor ([2.0], requires_grad = True)
(x**3).backward()
</code></pre>
<p>works just fine.</p>
<p>Best.</p>
<p>K. Frank</p>
RuntimeError: mat1 and mat2 shapes cannot be multiplied (1x1 and 2048x1000)

donparker1: <p>I am new to pytorch</p>
<p>class RN50_End(torch.nn.Module):<br>
def <strong>init</strong>(self, rn50):<br>
super(RN50_End, self).<strong>init</strong>()<br>
self.avgpool = rn50.avgpool<br>
self.fc = rn50.fc</p>
<pre><code>def forward(self, x):
    x = self.avgpool(x)
    x = torch.flatten(x, 1)
    x = self.fc(x)
    return x
</code></pre>
<p>end_x = torch.rand([112, 2048, 7, 7 ], dtype=torch.float32).contiguous()</p>
<p>r50end = RN50_End(rn50)<br>
for i, (images, target) in enumerate(cal_loader):<br>
output = r50end(images)</p>

ptrblck: <p>I don’t know how <code>rn50</code> is defined but the error is raised as a shape mismatch in <code>self.fc</code> as this layer seems to expect an activation input with <code>2048</code> features while <code>x</code> has a single feature only.</p>
Extract elements from matrix given indices from max_pool2d

Samue1: <p>In the code below, I would like to extract a submatrix from <code>S</code> according to the indices stored in <code>idx</code>.<br>
For the example code, the resulting submatrix <code>s</code> would have shape (8 x 8).</p>
<pre><code class="lang-auto">import torch

def main():
    batch_size = 2
    channels = 2
    height = 4
    width = 4

    X = torch.arange(start=1, end=batch_size*channels*height*width+1).float()
    X = X.reshape(batch_size, channels, height, width)

    _, idx = torch.nn.functional.max_pool2d_with_indices(X, kernel_size=(2, 2))

    S = torch.arange(start=1, end=(channels*height*width)**2+1).float()
    S = S.reshape(channels*height*width, channels*height*width)

    # s = ?

if __name__ == "__main__":
    main()
</code></pre>
<p>Here is a simple example that illustrates the problem better</p>
<p>For the following tensor</p>
<pre><code class="lang-auto">X = [[[[1, 0],
       [0, 0]],

      [[0, 0],
       [1, 0]]]]
</code></pre>
<p><code>max_pool2d_with_indices()</code> returns the following indices</p>
<pre><code class="lang-auto">idx = [[[[0]],
        [[2]]]]
</code></pre>
<p>(For the extraction of the submatrix from <code>S</code> it would be better treating <code>X</code> as a vector so that <code>idx</code> becomes <code>idx = [[[[0]],[[6]]]]</code>. However, at this point, I don’t know how to compute these indices.)</p>
<p>that I want to use to extract the following submatrix <code>s</code></p>
<pre><code class="lang-auto">s = [[0, 6],
     [48, 54]]
</code></pre>
<p>from</p>
<pre><code class="lang-auto">S = [[0,  1,  2,  3,  4,  5,  6,  7],
     [8,  9, 10, 11, 12, 13, 14, 15],
     [16, 17, 18, 19, 20, 21, 22, 23],
     [24, 25, 26, 27, 28, 29, 30, 31],
     [32, 33, 34, 35, 36, 37, 38, 39],
     [40, 41, 42, 43, 44, 45, 46, 47],
     [48, 49, 50, 51, 52, 53, 54, 55],
     [56, 57, 58, 59, 60, 61, 62, 63]]
</code></pre>
<p>I would be very happy, if someone could give me a hint how to do that.</p>

kshitij: <p>Probably <a href="https://discuss.pytorch.org/t/extract-values-from-tensor-by-indices-of-max-pooling/36602">this</a> might help</p>
How to investigate a not on the same device error

alexn: <p>I have an error:</p>
<blockquote>
<p>RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument weight in method wrapper__native_layer_norm_backward)</p>
</blockquote>
<p>I’m trying to find what’s not on the GPU. The forward pass runs with no error, this message appear only at the backward pass. I’ve checked that all my data &amp; the model are on the GPU. Also that I have initialised the optimiser with the parameters on the device. I don’t see what could be that is not on the GPU. I’ve tried to check all the device of all the tensors in the grad_fn tree with <code>[getBack() function](https://stackoverflow.com/questions/52988876/how-can-i-visualize-what-happens-during-loss-backward)</code> but this is way too slow (too many steps). What else can I do to find where the issue is?</p>

ptrblck: <p>Based on the stacktrace it seems the <code>weight</code> of a <code>LayerNorm</code> layer is not on the same device as the gradient input to it. I would probably try to isolate this layer and check where the inputs/outputs are coming from. A minimal code snippet to reproduce the issue would be great for further debugging.</p>

alexn: <p>It’s a layer in a pretrained transformer model. Possibly the last layer.</p>
<p>I’ve checked the layers one by one and all have been loaded in the GPU. I will try to cobble up some code snippet with the error.</p>
<p>Thanks!</p>

ptrblck: <p>Once you’ve narrowed down the layers, add print statements to check the <code>.device</code> attribute of the input activation as well as the layer parameters. Forward hooks might work for this check.</p>
Parameter server based rpc in tutorial test accuracy is 0.1

aonrdbit: <p>Hello!<br>
I implemented the rpc based parameter server framework according to <a href="https://pytorch.org/tutorials/intermediate/rpc_param_server_tutorial.html" rel="noopener nofollow ugc">this link</a>, but encountered some problems.</p>
<p>If the neural network being trained does not contain batchnorm layers, then both training and testing can yield ideal results. However, if the neural network contains batchnorm layers, such as resnet, it is normal during the training phase, but during the testing phase, i.e. after model.eval(), the test accuracy of the model is always 0.1. My test code is as follows.</p>
<pre><code class="lang-auto">def get_accuracy(test_loader, model):
    model.eval()
    correct_sum = 0
    device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
    with torch.no_grad():
        for i, (data, target) in enumerate(test_loader):
            data, target = data.to(device), target.to(device)
            out = model(data)
            pred = out.argmax(dim=1, keepdim=True)
            pred = pred.to(device)
            correct = pred.eq(target.view_as(pred)).sum().item()
            correct_sum += correct
    print(f"Accuracy {correct_sum / len(test_loader.dataset)}")
</code></pre>
<p>After encountering this problem, I checked some information. The reason might be the running_mean and running_var of the batchnorm layer, but I don’t know what to do with them. I am very confused, can you give me some advice, thank you!</p>

rvarm1: <p>Thanks for your question! It appears that <code>model.eval()</code> with batchnorm has historically had some confusion, see for example: <a href="https://discuss.pytorch.org/t/model-eval-gives-incorrect-loss-for-model-with-batchnorm-layers/7561/11" class="inline-onebox">Model.eval() gives incorrect loss for model with batchnorm layers - #11 by meetshah1995</a>.</p>
<p>Could you try to follow the advice in that thread to see if it improves the eval accuracy? Also, what sort of accuracy do you get if you disable model.eval()?</p>
How to train attention network?

mmdbrdrn: <p>Hello,<br>
I am trying to add a spatial attention mechanism to my semantic segmentation network (e.g., U-net), however, I have two questions?</p>
<ol>
<li>I want to add attention to multiple layers, should I design one network and just resample the output wrt the dimensions of that specific layer? or should I design different attention networks for each layer?</li>
<li>should I train attention network(s) at the same time I train the main network? or should they be trained sequebtially?</li>
</ol>
<p>Thank you so much</p>
C++ Frontend: avoid spawning extra threads in already multithreaded code

lodo1995: <p>I am writing a multithreaded C++ application, in which each thread needs to perform some PyTorch computations.</p>
<p>Unfortunately, this seems to be quite hard. I found that:</p>
<ul>
<li>if I do not do anything special, PyTorch will (via OpenMP) create a process pool spanning all available cores, and all my threads will dispatch operations to this pool</li>
<li>if I try to use <code>at::set_num_threads(1)</code> (either globally at the beginning, or within each thread at spawn), a single OpenMP process is spawned by PyTorch, which will then receive the operations dispatched from all my threads.</li>
</ul>
<p>In both cases, this does not give me the control I need in my scenario. I need that the PyTorch operations are executed by the thread that dispatched them. So if one of my threads wants to add two tensors together, the addition should be performed by that thread, without dispatching via OpenMP.</p>
<p>What are the options available to achieve this? Until now, the only idea I have is to manually build libtorch from source, using the environment variable <code>MKL_THREADING=SEQ</code>. However, this might take quite a lot of time and effort to set up correctly, plus I’m not sure it will work.</p>
<p>So, did anyone find themselves in this situation? Any other idea or workaround?</p>
Torch.save FileNotFoundError: [Errno 2] No such file or directory

tianle-BigRice: <p>when i try to call torch.save to save my model in a name, it rises a FileNotFoundError. the trace back is as follow:</p>
<pre><code class="lang-auto">Traceback (most recent call last):

  File "D:\python\RDANET\models\RDAnet.py", line 236, in &lt;module&gt;
    model.save()

  File "D:\python\RDANET\models\RDAnet.py", line 173, in save
    t.save(self.state_dict(), name)

  File "d:\anaconda3\lib\site-packages\torch\serialization.py", line 369, in save
    with _open_file_like(f, 'wb') as opened_file:

  File "d:\anaconda3\lib\site-packages\torch\serialization.py", line 234, in _open_file_like
    return _open_file(name_or_buffer, mode)

  File "d:\anaconda3\lib\site-packages\torch\serialization.py", line 215, in __init__
    super(_open_file, self).__init__(open(name, mode))

FileNotFoundError: [Errno 2] No such file or directory: 'D:/python/RDANET/checkpoints/RDAnet_0718_16_30_44.pkl'
</code></pre>
<p>I’m running code on a Windows remote desktop，the code：</p>
<pre><code class="lang-auto">    def save(self, name=None):

        if name is None:
            prefix = 'D:/python/RDANET/checkpoints/' + self.model_name + '_'
            name = time.strftime(prefix + '%m%d_%H_%M_%S.pkl')
        t.save(self.state_dict(), name)
        return name
        model.save()
</code></pre>

tianle-BigRice: <p>I solved the problem，This works when I use prefix = self.model_name + ‘_’ .So I created the folder <em>checkpoints</em>. And then it didn’t go wrong. This is so strange</p>

akaysh: <p>One or more of your intermediate directories do not exist in your path (D:/python/RDANET/checkpoints/) where you are trying to save the model. Torch expects the directories to exist in the path of saving the model.</p>

baizx98: <p>You must create the directory manually.</p>
Only preserve the top k channels in channel attention layer

RYY0722: <p>Hi there!<br>
I wish to implement a customized channel attention layer. Concretely, for a (B,C,H,W) tensor, the channel attention layer will return an attention tensor of shape (B,C,1,1), and the input tensor is multiplied with this vector-like tensor.<br>
What I wish to add is selecting the most important k layers according to the attention vector, and the resulting tensor will be of shape (B,k,H,W).<br>
For the implementation, the only thing in my mind is that</p>
<ol>
<li>sorting the attention</li>
<li>selecting the top k by operation like [:,:k,:,:] something</li>
</ol>
<p>But my concern is that will autograd still work? Because I can’t think about a way that any differentialable function can acheive the “[:k]” operation.<br>
Can my naive implementation work? If yes, can anyone give me some help about how it works, like how the gradient pass, etc. If not, can anyone give me some suggestions?<br>
Thank you so much!!!<br>
My idea is based on implementation here:<br>
<a href="https://github.com/BangguWu/ECANet/blob/master/models/eca_module.py" rel="noopener nofollow ugc">ECANet-ECA_module</a></p>
Expand different number of times each index of a tensor dimension

Nuno-Mota: <p>Hey</p>
<p>Let’s say I have a tensor like:</p>
<pre><code class="lang-auto">a =  [[1, 1, 1],
      [2, 2, 2],
      [3, 3, 3]]
</code></pre>
<p>and that I want to expand each row a certain amount of times, without actually allocating memory. If I want each row to be repeated the same number of times I could do something like:</p>
<pre><code class="lang-auto">a = a.unsqueeze(1).expand(-1, 2, -1).reshape(-1, 3)
</code></pre>
<p>Which gives:</p>
<pre><code class="lang-auto">a =  [[1, 1, 1],
      [1, 1, 1],
      [2, 2, 2],
      [2, 2, 2],
      [3, 3, 3],
      [3, 3, 3]]
</code></pre>
<p>However, how would I do it efficiently (mostly without allocating unnecessary memory) if I wanted to expand each row a different amount of times? Something like:</p>
<pre><code class="lang-auto">a = a.unsqueeze(1).expand(-1, (2,3,4), -1).reshape(-1, 3)
</code></pre>
<p>Which would yield:</p>
<pre><code class="lang-auto">a =  [[1, 1, 1],
      [1, 1, 1],
      [2, 2, 2],
      [2, 2, 2],
      [2, 2, 2],
      [3, 3, 3],
      [3, 3, 3],
      [3, 3, 3],
      [3, 3, 3]]
</code></pre>

XZK9: <p>You can use torch.index_select</p>
Invalid Device id Error when I assign more device_ids

KanZa: <p>When I assign more than two devices to device_ids, it gives me an Invalid Device id Error. Though I am using almost the same code for different models for many gpus.</p>
<p>torch.backends.cudnn.benchmark = True<br>
models = torch.nn.DataParallel(models, device_ids=[0, 1])</p>
How do we add custom backend for PyTorch profiler?

Muthu_Annamalai: <p>I see torch.profiler.ProfilerActivity objects for CPU and CUDA; how can I add more for custom accelerator like TPU etc. ?<br>
Please advise any good starting points to add events into the PyTorch profiler infrastructure from custom hardware.</p>
<p>TIA</p>

tom: <p>I’m not sure this part of PyTorch is very flexibly extensible right now. Quite likely, I would start by looking into <a href="https://github.com/pytorch/kineto/" rel="noopener nofollow ugc">kineto</a> to see either how to extend it to support your hardware or what kind of API PyTorch uses. Then look at the PyTorch use of kineto.</p>
<p>Best regards</p>
<p>Thomas</p>

Muthu_Annamalai: <p>thanks Mr. Thomas. Appreciate your insight.</p>

Raf: <p>Hi guys!<br>
How we may measure time of some operation on Google Colab for TPU ?<br>
<a href="https://pytorch.org/tutorials/recipes/recipes/profiler_recipe.html" class="inline-onebox" rel="noopener nofollow ugc">PyTorch Profiler — PyTorch Tutorials 1.11.0+cu102 documentation</a> - pytorch profiler support only CPU and CUDA devices, Libkineto support NVIDIA GPUs.<br>
Is exist ready method for measure consume time of some operation on Google Colab for TPU?</p>
<p>To use time.time_ns() measure only consume time of CPU on VM with TPU - how i understood.</p>
<p>Thanks!</p>
Training segmentation error : __call__() missing 1 required positional argument: 'target' 

Xionfire: <p>Hi, I try to train a FCN-Resnet50 with cityscapes gtFine datasets but i encounter this error and i don’t know any way to solve it, do you have idea to solve it ?</p>
<p>python3 train.py --aux-loss --data-path=/home/manip/vision/references/segmentation/cityscapes -b 240<br>
pytorch-segmentation/datasets/<strong>init</strong>.py<br>
Not using distributed mode<br>
Namespace(amp=False, aux_loss=True, batch_size=240, data_path=’/home/manip/vision/references/segmentation/cityscapes’, dataset=‘cityscapes’, device=‘cuda’, dist_url=‘env://’, distributed=False, epochs=30, lr=0.01, lr_warmup_decay=0.01, lr_warmup_epochs=0, lr_warmup_method=‘linear’, model=‘fcn_resnet50’, momentum=0.9, output_dir=’.’, print_freq=10, resume=’’, start_epoch=0, test_only=False, use_deterministic_algorithms=False, weight_decay=0.0001, weights=None, weights_backbone=None, workers=0, world_size=1)<br>
Traceback (most recent call last):<br>
File “train.py”, line 313, in <br>
main(args)<br>
File “train.py”, line 233, in main<br>
train_one_epoch(model, criterion, optimizer, data_loader, lr_scheduler, device, epoch, args.print_freq, scaler)<br>
File “train.py”, line 102, in train_one_epoch<br>
for image, target in metric_logger.log_every(data_loader, print_freq, header):<br>
File “/home/manip/vision/references/segmentation/utils.py”, line 164, in log_every<br>
for obj in iterable:<br>
File “/home/manip/.local/lib/python3.8/site-packages/torch/utils/data/dataloader.py”, line 530, in <strong>next</strong><br>
data = self._next_data()<br>
File “/home/manip/.local/lib/python3.8/site-packages/torch/utils/data/dataloader.py”, line 570, in _next_data<br>
data = self._dataset_fetcher.fetch(index)  # may raise StopIteration<br>
File “/home/manip/.local/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py”, line 49, in fetch<br>
data = [self.dataset[idx] for idx in possibly_batched_index]<br>
File “/home/manip/.local/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py”, line 49, in <br>
data = [self.dataset[idx] for idx in possibly_batched_index]<br>
File “/home/manip/.local/lib/python3.8/site-packages/torchvision/datasets/cityscapes.py”, line 197, in <strong>getitem</strong><br>
image, target = self.transforms(image, target)<br>
File “/home/manip/.local/lib/python3.8/site-packages/torchvision/datasets/vision.py”, line 95, in <strong>call</strong><br>
input = self.transform(input)<br>
TypeError: <strong>call</strong>() missing 1 required positional argument: ‘target’</p>

ptrblck: <p>Based on the stacktrace the code fails <a href="https://github.com/pytorch/vision/blob/59c4de9123eb1d39bb700f7ae7780fb9c7217910/torchvision/datasets/vision.py#L95">here</a> so check your used transformations. I guess you might have passed to wrong transformations to <code>StandardTransform</code> via <code>transform</code> and <code>target_transform</code> as the former seems to expect two inputs in your script.</p>
Is the loss value useful for validation and test parts?

SouthTorch: <p>In my code, I print out the loss for training part, and loss and accuracy (correct/total) for the validation and test parts.<br>
1: I find that loss in validation and test are usually 2 magnitude larger than training loss.<br>
2: validation loss looks not useful at all, all I need right now is to use validation accuracy history and training loss history to decide whether the mode is suitable and whether the traning is overfitting or not.<br>
Do you guys care about the validation loss value? Thanks.</p>
Robust PCA does not work on GPU with PyTorch as a backend

mahmood: <p>Hello there,<br>
I tried a library called tensorly to use one of its functions Robust PCA. It works fine when using it on CPU it does not work on GPU with PyTorch as a backend with tensorly-0.7.0 and PyTorch 1.11.0+cu113 I also tried  tensorly-0.7.0 and pytorch 1.7+cu102</p>
<p>Here is my CODE</p>
<pre><code class="lang-auto">pip install tensorly
print(torch.__version__)
import torch
import tensorly as tl

tl.set_backend('pytorch')
cuda = torch.device('cuda')
fake_data = torch.randn(2500, 9000, device=cuda)
low_rank_part, sparse_part = tl.decomposition.robust_pca(fake_data, reg_E=0.04, learning_rate=1.2, n_iter_max=20)
</code></pre>
<p>and the error</p>
<pre><code class="lang-auto">
/usr/local/lib/python3.7/dist-packages/tensorly/backend/core.py:1106: UserWarning: In partial_svd: converting to NumPy. Check SVD_FUNS for available alternatives if you want to avoid this.
  warnings.warn('In partial_svd: converting to NumPy.
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
[&lt;ipython-input-7-e31773fb0320&gt;](https://localhost:8080/#) in &lt;module&gt;()
      1 cuda = torch.device('cuda')
      2 fake_data = torch.randn(2500, 9000, device=cuda)
----&gt; 3 low_rank_part, sparse_part = tl.decomposition.robust_pca(fake_data, reg_E=0.04, learning_rate=1.2, n_iter_max=20)

3 frames
&lt;__array_function__ internals&gt; in amax(*args, **kwargs)

[/usr/local/lib/python3.7/dist-packages/torch/_tensor.py](https://localhost:8080/#) in __array__(self, dtype)
    730             return handle_torch_function(Tensor.__array__, (self,), self, dtype=dtype)
    731         if dtype is None:
--&gt; 732             return self.numpy()
    733         else:
    734             return self.numpy().astype(dtype, copy=False)

TypeError: can't convert cuda:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.
</code></pre>
<p>Dependencies versions<br>
Linux-5.4.188±x86_64-with-Ubuntu-18.04-bionic<br>
Python 3.7.13 (default, Apr 24 2022, 01:04:09)<br>
[GCC 7.5.0]<br>
NumPy 1.21.6<br>
SciPy 1.4.1<br>
TensorLy 0.7.0<br>
Pytorch 1.11.0+cu113</p>

ptrblck: <p>Apparently the method expects a CPUTensor, as internally <code>.numpy()</code> is called as given in the error message:</p>
<pre><code class="lang-python">TypeError: can't convert cuda:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.
</code></pre>
<p>EDIT: also cross-post from <a href="https://github.com/tensorly/tensorly/issues/414">here</a></p>
Creating bigram embeddings by concatenating adjacent word embeddings

sds: <p>Hi,<br>
I am trying to create bigram vectors by concatenating adjacent word vectors along the sequence dimension. I have gotten that to work, but my current implementation (see toy example below) mixes pytorch tensor operations with python list comprehensions, which is likely suboptimal and inefficient. Is there a better “pure pytorch” implementation of this, or am I just optimizing prematurely? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"><br>
Thanks!</p>
<p>In [22]: t<br>
Out[22]:<br>
tensor([[1.1000, 1.2000, 1.3000],<br>
[2.1000, 2.2000, 2.3000],<br>
[3.1000, 3.2000, 3.3000]])</p>
<p>In [23]: torch.stack(list(torch.cat((a, b)) for a, b in zip(t, t[1:])))<br>
Out[23]:<br>
tensor([[1.1000, 1.2000, 1.3000, 2.1000, 2.2000, 2.3000],<br>
[2.1000, 2.2000, 2.3000, 3.1000, 3.2000, 3.3000]])</p>

ptrblck: <aside class="quote no-group" data-username="sds" data-post="1" data-topic="154767">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/54ee81/48.png" class="avatar"> sds:</div>
<blockquote>
<p>or am I just optimizing prematurely? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
</blockquote>
</aside>
<p>Every optimization if premature unless it’s the bottleneck <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>
<p>Depending on your use case, <code>unfold</code> might be faster but would also need to create a copy:</p>
<pre data-code-wrap="python"><code class="lang-python">t = torch.tensor([[1.1000, 1.2000, 1.3000],
                  [2.1000, 2.2000, 2.3000],
                  [3.1000, 3.2000, 3.3000]])

out = torch.stack(list(torch.cat((a, b)) for a, b in zip(t, t[1:])))

size = 2
stride =1
ret = t.unfold(0, size, stride).permute(2, 0, 1).contiguous().view(-1, t.size(1)*size)

print((out == ret).all())
# tensor(True)
</code></pre>
How to understand this '*' in this transfered learning model?

SouthTorch: <pre><code class="lang-auto">trained_model=torchvision.models.resnet18(pretrained=True)
model=nn.Sequential(*list(trained_model.children())[:-1],
                                   nn.Flatten(),
                                   nn.Linear(512, 5)
                                  )
</code></pre>
<p>I guess I understand how the transfered learning work here:<br>
in the new <code>model</code>:<br>
<code>*list(trained_model.children())[:-1]</code> must be (I guess) working as a fixed function without introducing any new parameters. So when you train <code>model</code>, you will only update the parameters introduced by <code>nn.Linear(512, 5)</code>.<br>
In this way, the transfered learning is done, am I right?</p>
<p>What I don’t understand is this <code>*</code>, what it does?<br>
I can understand I used the first 17 children layers of the <code>trained _model</code> by using <code>list(trained_model.children())[:-1]</code>, but isn’t this already enough? why add a <code>*</code> here?<br>
Thanks.</p>

willdone1337: <p>1 - In your way, in training process, you update all params in the network. If you wanna just train last 512,5 layer you have to freeze all layers before last.<br>
2 - * operator used for unpacking the all layers from renset because nn.Sequential’s input must be nn.Modules not list(nn.Module)</p>

SouthTorch: <p>Thanks, willdone!</p>
<p>1: Really? Then how to freeze those layers?<br>
2: is there a way to get 17 layers without using <code>*</code> and <code>list()</code>?</p>
<p>Thanks.</p>

willdone1337: <p>1 -  for i, (n,p) in enumerate(model.named_parameters()):<br>
condition for freezing:<br>
p.requires_grad = False</p>
<p>2 - you can use adding layers to nn.ModuleList but then you have to go forward in the cycle but how I know nn.sequential is working faster, and *list is easy way to handle it</p>

SouthTorch: <p>Thanks, willdone1337!</p>
Found at least two devices, cuda:-2 and cuda:0!

fcvm: <pre><code class="lang-auto">terminate called after throwing an instance of 'std::runtime_error'
  what():  The following operation failed in the TorchScript interpreter.

...

RuntimeError: Expected all tensors to be on the same device, 
but found at least two devices, cuda:-2 and cuda:0! 
(when checking argument for argument weight in method wrapper__cudnn_convolution)
</code></pre>
<p>Does anyone know how to deal with this error?<br>
I only have one GPU and made sure to send both the model and input tensors to the same device.<br>
What does a negative device (-2) even mean?</p>

hyhuang00: <p>I got a similar error as yours. Have you figured this out?</p>
Transformers backpropagation

andreasceid: <p>How is backpropagation applied in transformers? It is my understanding that in an encoder cell there are 2 different fully connected NNs:</p>
<ul>
<li>One is for the query embedding</li>
<li>And the other is for the value embedding</li>
</ul>
<p>Those 2 NNs output an augmented version of the query and value embeddings. Those 2 vectors are then multiplied (using the dot product) to compute a similarity score between them. My question is:</p>
<p>How is the backpropagation algorithm expanded in this case? Do both the query NN and the value NN have the same loss backpropagated? If the same loss is backpropagated, then why use 2 different neural networks for that?</p>
Multiple models torchserve

turso: <p>What is the best way to do inference with multiple models inside torchserve?</p>
<p>TL;TR &gt; i have to predict a value with a model1, and predict another value at model2, which its inputs is model1’s output</p>
<p>It is my first time using torchserve, so I don’t know if I should load two models at same time, and use both inside Inference function…</p>
<p>What is the best practice?</p>
<p>Thanks!!</p>
EfficientNet optimizer in PyTorch

mfoglio: <p>In the official EfficientNet paper, I read:</p>
<blockquote>
<p>We train our EfﬁcientNet models on ImageNet using similar settings as (Tan et al., 2019): RMSProp optimizer with decay 0.9 and momentum 0.9; batch norm momentum 0.99; weight decay 1e-5; initial learning rate 0.256 that decays by 0.97 every 2.4 epochs</p>
</blockquote>
<p>Leaving aside the scheduler optimizer, how can I create the same optimizer in PyTorch? Is <code>alpha</code> the decay? I am confused by the name of the parameters of <code>RMSprop</code> in PyTorch.</p>
<p>Is this correct?</p>
<pre><code class="lang-auto">torch.optim.RMSprop(
    self.parameters(), lr=self.learning_rate, alpha=0.9, momentum=0.9, weight_decay=1e-5
)
</code></pre>
Stratified dataloader for imbalanced data

jasperhyp: <p>Hi,</p>
<p>I reviewed previous posts on this topic and found that most answers seem to aim for building a balanced batch instead of keeping the original class distribution, e.g. <a href="https://discuss.pytorch.org/t/how-to-handle-imbalanced-classes/11264">this</a> one and <a href="https://discuss.pytorch.org/t/balanced-sampling-between-classes-with-torchvision-dataloader/2703">that</a> one.  It seems this might not be a very good practice because oversampling the innately imbalanced distributions might create a bias.  I was just wondering if there is a functionality similar to the “stratify” option in <code>sklearn.model_selection.train_test_split</code>, where the class distribution is kept in each stratum, i.e. batch.</p>
<p>Please feel free to comment on this argument &amp; question!</p>

nivek: <p>If I am understanding correctly, you want to preserve the original class distribution. If you want each batch to approximately preserve the original distribution, <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.RandomSampler" rel="noopener nofollow ugc"><code>RandomSampler</code></a> should be helpful (or <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.random_split" rel="noopener nofollow ugc"><code>random_split</code></a> to split a dataset into two). Since those are random, the distribution will not be exact across batches/datasets but should approximately be.</p>
<p>If you want each batch to have the <strong>exact</strong> distribution, you can consider writing your own <code>Sampler</code> or use the <code>sklearn</code> function.</p>
<p>Let me know if that is helpful. If not, you can provide some minimal code snippets (or sample data distribution) that you are working with, and we can have a look.</p>

jasperhyp: <p>Thanks!  I think that what I want is just ensuring each batch having the random probability and at least 1 minority sample.  Would write a simple class to realize this.</p>
What is a time-efficient way to find min-distance element of one tensor for each element of another tensor?

mkarikom: <h3>
<a name="goal-1" class="anchor" href="#goal-1"></a>Goal:</h3>
<p>How to compute min-distance column of B for each column of A<br>
In the following attempt: for all b_j in B, find argmin(i) norm(a_i,b_j)</p>
<p>Where:<br>
a_i = column i of matrix A (M x N)<br>
b_j = column j of matrix B (K x N)</p>
<h5>
<a name="should-return-m-in-im-lenm-k-2" class="anchor" href="#should-return-m-in-im-lenm-k-2"></a>Should return: m \in I^M, len(m) = K</h5>
<h3>
<a name="attempt-3" class="anchor" href="#attempt-3"></a>Attempt:</h3>
<pre><code class="lang-auto">A = torch.rand(20,200)
B = torch.rand(10,200)
torch.norm(A - B.transpose(1,0).unsqueeze(0).repeat(20,1,1),dim=2).argmin()

RuntimeError: The size of tensor a (200) must match the size of tensor b (10) at non-singleton dimension 2
</code></pre>