akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Tensor.to(device) not working (no error)

jybb: <p><strong>I’m having an issue of slow .to(device) transfer.</strong> However, the same code worked well with the colab. The systems that I am using are as follows:</p>
<ul>
<li>Ubuntu 18.04</li>
<li>torch. version : 1.4.0</li>
<li>torch.version.cuda : 10.1 (nvidia-smi shows CUDA version 11.3)</li>
<li>NVIDIA - GeForce RTX 3090 Ti<br>
(Interestingly, I manually installed CUDNN 8.6, but nvcc is not working)</li>
</ul>
<p>I’m testing simple VGG19 model.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn


cfg = {
    'VGG19': [64, 64, 'M', 128, 128, 'M', 256, 256, 256, 256, 'M', 512, 512, 512, 512, 'M', 512, 512, 512, 512, 'M']
}


class VGG(nn.Module):
    def __init__(self, vgg_name):
        super(VGG, self).__init__()
        self.features = self._make_layers(cfg[vgg_name])
        self.classifier = nn.Linear(512, 10)

    def _make_layers(self, cfg):
        layers = []
        in_channels = 3
        for x in cfg:
            if x == 'M':
                layers += [nn.MaxPool2d(kernel_size=2, stride=2)]
            else:
                layers += [nn.Conv2d(in_channels, x, kernel_size=3, padding=1),
                           nn.ReLU(inplace=True)]
                in_channels = x
        layers += [nn.AvgPool2d(kernel_size=1, stride=1)]
        return nn.Sequential(*layers)
    
    def forward(self, x):
        out = self.features(x)
        out = out.reshape(out.size(0), -1)
        out = self.classifier(out)
        return out
</code></pre>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.optim as optim
import torch.nn.functional as F
import torch.backends.cudnn as cudnn

import torchvision
import torchvision.transforms as transforms

import os
import argparse
from models import VGG

device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')

print('==&gt; Preparing data..')
transform = transforms.Compose(
    [transforms.ToTensor(), 
     transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])

trainset = torchvision.datasets.CIFAR10(
    root='./data', train=True, download=True, transform=transform)
trainloader = torch.utils.data.DataLoader(
    trainset, batch_size=128, shuffle=True, num_workers=2)

testset = torchvision.datasets.CIFAR10(
    root='./data', train=False, download=True, transform=transform)
testloader = torch.utils.data.DataLoader(
    testset, batch_size=100, shuffle=False, num_workers=2)

classes = ('plane', 'car', 'bird', 'cat', 'deer',
           'dog', 'frog', 'horse', 'ship', 'truck')

print('==&gt; Building model..')
net = VGG('VGG19')
net = net.to(device)
criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(net.parameters(), lr=args.lr,
                      momentum=0.9, weight_decay=5e-4)
scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=200)
</code></pre>

ptrblck: <p>Your Ampere GPU needs CUDA 11, so update PyTorch to the latest release with any CUDA 11 runtime.</p>
Pos_weigh for multiclassification task

Anita: <p>Hi. I am using a CNN for grading breast cancer and it this problem falls into the category of image multiclassification. I have highly imbalanced classes on my dataset, due to high-grade cancer cases being scarce.</p>
<p>The output from the model is basically logits from the last Linear layer as follows:<br>
self.classifiers = nn.Linear(size[1], n_classes)<br>
where I extract the logits<br>
logits = self.classifiers(M)</p>
<p>then I feed the logits to the error loss function on the training loop as:<br>
loss_fn = nn.CrossEntropyLoss()<br>
loss = loss_fn(logits, label)</p>
<p>How can I implement some sort of weight in this case, to penalize classes that have low representation in the dataset? I have in total 6 classes [0,1,2,3,4,5] and the ones with the lower representation are 3,4,5.</p>

KFrank: <p>Hi Anita!</p>
<aside class="quote no-group" data-username="Anita" data-post="1" data-topic="163299" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/anita/48/56469_2.png" class="avatar"> Anita:</div>
<blockquote>
<p>then I feed the logits to the error loss function on the training loop as:<br>
loss_fn = nn.CrossEntropyLoss()<br>
loss = loss_fn(logits, label)</p>
<p>How can I implement some sort of weight in this case, to penalize classes that have low representation in the dataset?</p>
</blockquote>
</aside>
<p><code>CrossEntropyLoss</code>’s <code>weight</code> constructor argument reweights your classes<br>
in the loss function.  (<code>CrossEntropyLoss</code>’s <code>weight</code> is quite analogous to<br>
<code>BCEWithLogitsLoss</code>’s <code>pos_weight</code> constructor argument.)</p>
<p>I’m not sure I understand what you mean by “penalize classes that have low representation.”  Typically you would want to weight more heavily the classes<br>
that occur less frequently in your dataset, and the reciprocal of the frequency<br>
with which each class occurs is a common choice for the class weight.</p>
<p>Thus, if classes 0, 1, and 2 occur tens times as often as classes 3 and 4,<br>
and twenty times as often as class 5, you could use;</p>
<pre data-code-wrap="python"><code class="lang-python">class_weights = 1.0 / torch.tensor ([20.0, 20.0, 20.0, 2.0, 2.0, 1.0])
loss_fn = torch.nn.CrossEntropyLoss (weight = class_weights)
</code></pre>
<p>as your weighted loss criterion.</p>
<p>(Alternatively, you could sample your less-common classes more frequently,<br>
for example, by using <code>WeightedRandomSampler</code>.  I tend to think that there<br>
are some theoretical reasons to prefer weighted sampling over class weights<br>
in the loss function for the use case you describe, but both approaches are<br>
sensible and both are commonly used.)</p>
<p>Best.</p>
<p>K. Frank</p>
Error while running executable file

Sreekanth: <p>Hi,<br>
We built executable file for Yolo v5 without any error using pyinstaller, however, when we try to run the exe we are getting Moduleerror: torch module not found.</p>
<p>While building the exe we have excluded few libraries to reduce the exe size but have provided the path for the library.</p>
<p>When we run the model using terminal the model runs without any issue/error.</p>
<p>Any idea why the exe is not able to locate the torch.</p>
<p>Thanks,<br>
Sreekanth</p>
Get arguments for torch.nn.functional methods

collinmccarthy: <p>Hi,</p>
<p>Is there any way to extract the argument names / order for methods in <code>torch.nn.functional</code>? If I try using the <code>inspect</code> module I get a <code>TypeError('unsupported callable')</code>, e.g.</p>
<pre><code class="lang-python">inspect.getargspec(torch.nn.functional.conv2d)  # TypeError: unsupported callable
inspect.signature(torch.nn.functional.conv2d)  # ValueError: no signature found for builtin &lt;built-in method conv2d of type object at 0x7f7a3ce19780&gt;
</code></pre>
<p>I’m trying to save the kernel size, stride, etc. without having to hard-code which arguments they come from.</p>
<p>Thanks in advance. -Collin</p>
Dataset Structure for Graph Convolutional Networks

Austin_Lutterbach: <p>I am attempting to construct a GCN on my graph data using the code found <a href="https://github.com/praxidike97/GraphNeuralNet/blob/master/main.py" rel="noopener nofollow ugc">here</a> as a template.</p>
<p>When this code constructs Network with GCN layers:</p>
<pre><code class="lang-python">class Net(torch.nn.Module):
    def __init__(self, dataset):
        super(Net, self).__init__()
        self.conv1 = GCNConv(dataset.num_node_features, 16)
        self.conv2 = GCNConv(16, dataset.num_classes)

    def forward(self, data):
        # HERE
        x, edge_index = data.x, data.edge_index

        x = self.conv1(x, edge_index)
        x = F.relu(x)
        x = F.dropout(x, training=self.training)
        x = self.conv2(x, edge_index)

        return F.log_softmax(x, dim=1)
</code></pre>
<p>The dataset this particular guide uses has all features stored in an m x n tensor called <code>x</code>, which is stored as an attribute of the dataset (<code>dataset.x</code>). I presume this corresponds to m examples and n features.</p>
<p>However, for my dataset, I have stored my features as separate tensor attributes rather than being aggregated into one attribute <code>x</code>. So e.g. <code>dataset.get_all_tensor_attrs()</code> will return all my features <code>f1,f2,f3,...</code>  as well as my response <code>y</code> as their own separate attributes. Is this a valid way to store the features in my graph dataset? Or should I combine them all under one attribute (say <code>x</code>), and continue on with my journey?</p>
How to import torchdata?

jiapei100: <p>I built and installed <strong>torchdata</strong> from source. Did anybody meet the same problem??</p>
<pre data-code-wrap="console"><code class="lang-nohighlight">➜  ~ python
Python 3.10.6 (main, Aug 10 2022, 11:40:04) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
&gt;&gt;&gt; import torch
&gt;&gt;&gt; import torchdata as td
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "~/.local/lib/python3.10/site-packages/torchdata/__init__.py", line 9, in &lt;module&gt;
    from . import datapipes
  File "~/.local/lib/python3.10/site-packages/torchdata/datapipes/__init__.py", line 9, in &lt;module&gt;
    from . import iter, map, utils
  File "~/.local/lib/python3.10/site-packages/torchdata/datapipes/iter/__init__.py", line 32, in &lt;module&gt;
    from torchdata.datapipes.iter.load.aisio import (
  File "~/.local/lib/python3.10/site-packages/torchdata/datapipes/iter/load/aisio.py", line 11, in &lt;module&gt;
    from torchdata.datapipes.utils import StreamWrapper
  File "~/.local/lib/python3.10/site-packages/torchdata/datapipes/utils/__init__.py", line 9, in &lt;module&gt;
    from ._visualization import to_graph
  File "~/.local/lib/python3.10/site-packages/torchdata/datapipes/utils/_visualization.py", line 13, in &lt;module&gt;
    from torch.utils.data.graph import traverse_dps
ImportError: cannot import name 'traverse_dps' from 'torch.utils.data.graph' (~/.local/lib/python3.10/site-packages/torch/utils/data/graph.py)
&gt;&gt;&gt; 
</code></pre>
<p>Cheers</p>

nivek: <aside class="quote no-group" data-username="jiapei100" data-post="1" data-topic="163158">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jiapei100/48/79_2.png" class="avatar"> jiapei100:</div>
<blockquote>
<pre><code class="lang-auto">ImportError: cannot import name 'traverse_dps' from 'torch.utils.data.graph' (~/.local/lib/python3.10/site-packages/torch/utils/data/graph.py)
</code></pre>
</blockquote>
</aside>
<p>Hi, if you are installing TorchData from source. Please install PyTorch nightly or from source as well. You can find the instruction to <a href="https://pytorch.org/" rel="noopener nofollow ugc">install nightly on our website’s main page</a>.</p>

akuysal: <p>I am using PyCharm IDE. You can easily download and install packages via its Python Packages tool window. I tried your code block and there is no error.</p>
How to work with two dataset for regression model

shuttii: <p>How can I implement or create a NN model with two tabulated datasets<br>
Dataset A - shape [11000,68] - Details of Large organisations  [target - score]<br>
Dataset B - shape [6200, 24] - Details of small scale companies [target - score]</p>
<p>The two datasets have only few similar features like size ,no of employees, etc . I am trying something like transfer learning. Train on larger companies and predict a score on small companies.</p>
<p>My test data might have any features from 92 (68 + 24) trained features, and can also have missing values.</p>
<p>Can someone please suggest me a way to implement this?</p>
Custom operation - problem with usage of CPUBlas functionality

damian_sg: <p>Hi,<br>
I’m working on some custom operation, which is a part of a shared library (let’s call it <em>libxxx.so</em>).<br>
After browsing the torch code, I found <code>gemm_batched</code> function, which is perfect for my implementation.<br>
However, I faced a couple of issues with that function.</p>
<ol>
<li>Signature from <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/CPUBlas.h#L103" rel="noopener nofollow ugc">CPUBlas.h</a> differs from <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/CPUBlas.cpp#L411" rel="noopener nofollow ugc">CPUBlas.cpp</a>.<br>
This results in undefined symbols (either at linking time when building Cpp test binary or at loading time in Python code) which <strong>cannot be found anywhere</strong>. After aligning signatures in the torch code I got myself to the 2nd issue (I guess I can also remove CPUBlas.h include and mark <code>gemm_batched</code> as extern).</li>
<li>When <em>libxxx.so</em> gets loaded, I get the following error:</li>
</ol>
<pre><code class="lang-auto">OSError: project_root/libxxx.so: undefined symbol: _ZN2at6native7cpublas12gemm_batchedIfEEvNS0_13TransposeTypeES3_llllT_PPKS4_lS7_lS4_PPS4_l
</code></pre>
<p>which corresponds to:</p>
<pre><code class="lang-auto">void at::native::cpublas::gemm_batched&lt;float&gt;(at::native::TransposeType, at::native::TransposeType, long, long, long, long, float, float const**, long, float const**, long, float, float**, long)
</code></pre>
<p>This symbol indeed is marked as undefined in <em>libxxx.so</em>. <em>ldd</em> shows that <em>libxxx.so</em> requires <em>libtorch_cpu.so</em>, where mentioned symbol is present - though not found in my case.<br>
When building Cpp test binary I get:</p>
<pre><code class="lang-auto">/usr/bin/c++ (...) libxxx.so  lib/libgtest_main.a  ../.venv/lib/python3.8/site-packages/torch/lib/libtorch.so  -Wl,--no-as-needed,"project_root/.venv/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so" (...)
/usr/bin/ld: libxxx.so: undefined reference to `void at::native::cpublas::gemm_batched&lt;float&gt;(at::native::TransposeType, at::native::TransposeType, long, long, long, long, float, float const**, long, float const**, long, float, float**, long)'
collect2: error: ld returned 1 exit status
</code></pre>
<p>so similar issue, despite <em>libtorch_cpu.so</em> containing the required symbol is in fact linked.</p>
<p>More about the building:</p>
<pre><code class="lang-auto">find_package(Torch REQUIRED)
target_link_libraries(${PROJECT_NAME} PRIVATE ${TORCH_LIBRARIES})
</code></pre>
<p>where <em>Torch_DIR</em> is set to “project_root/.venv/lib/python3.8/site-packages/torch/share/cmake”</p>
<p>There are no issue with any other functionality from at/torch. Any ideas on how to overcome this problem?</p>
How to correctly resolve a imbalance classification problem?

evilroach: <p>It’s a binary classification problem of a time series.</p>
<p>I just made a vanilla GRU network, to confirm the importance of the features.</p>
<p>Because the samples are very much imbalanced, i.e. the major class samples are far more than the minor class samples(about 250:1), I had to find out a method to resolve it.</p>
<p>At first, I just simply abandon randomly some samples in Class0(major), to make the number of samples of both classes being equal, and then I used them to train my GRU model.<br>
Every epoch, I always shuffle the training set and the validating set, and re-do abandon, and re-select samples of Class0 to build a new training set and a new validating set.</p>
<p>Things went well when training/validating, the metric score(F0.5score) looks like not bad. I guess that the features I constructed are effective.</p>
<p>But when I use the trained model to predict with real data, the model has been flooded with too many samples of Class0(major), and it goes badly with a poor score.</p>
<p>Hence I turned to use <code>WeightedRandomSampler</code>  , let as much as possible samples of Class0 to be fed to the model. Approximately after 10 epochs, every samples(regardless Class0 or Class1) have been seen by the model at least one time.</p>
<p>But in this way, it is difficult for the model to learn with the data.</p>
<p>I’m a freshman with deep learning, would some good man please help me?</p>

WANGSSSSSSS: <p>maybe，you can reweight the loss from those unbalanced classes.</p>

evilroach: <p>Thanks!</p>
<p>How to do it?<br>
Is there some examples?</p>

Bihy: <p>You can use Focal loss. But I think the best way to mitigate the class imbalance is to oversample your dataset.</p>
Deduce model input c++ data structure

Francescott: <p>Hello, I have a .pt that represent a serialized trained model.<br>
I am trying to do inference with it in c++ (loading and then inference using a random input).<br>
How can I deduce what is the correct format of the input?<br>
Which c++ data structure should I use?<br>
Which size are correct for the tensors?</p>
<p>The compiler doesn’t give me useful hints since it just show up  error like this:</p>
<pre><code class="lang-auto">C:\Users\me\AppData\Local\conda\conda\envs\laneChangePy38\lib\runpy.py(87): _run_code
C:\Users\me\AppData\Local\conda\conda\envs\laneChangePy38\lib\runpy.py(194): _run_module_as_main
RuntimeError: Sizes of tensors must match except in dimension 1. Expected size 0 but got size 285 for tensor number 1 in the list
</code></pre>
<p>But for me it’s unclear where is the problem (what is tensor number 1 in the list?) , the code shown for debugging is really low level pytorch code (see below).</p>
<pre><code class="lang-auto">  File "code/__torch__/trajectory_model/___torch_mangle_272.py", line 19, in forward
    id_embedding = data["identifier"]
    batch_size = data["num_graphs"]
    _4 = (_backbone).forward(_3, _0, _2, _1, id_embedding, int(batch_size), valid_lens, )
          ~~~~~~~~~~~~~~~~~~ &lt;--- HERE
    _5 = torch.slice(_4, 0, 0, 9223372036854775807)
    input = torch.select(_5, 1, 0)
  File "code/__torch__/layers/vectornet/___torch_mangle_261.py", line 20, in forward
    _subgraph = self._subgraph
    _0 = (_subgraph).forward(argument_2, argument_3, argument_4, argument_1, )
    input = torch.view(torch.cat([_0, id_embedding], 1), [argument_6, -1, 66])
                       ~~~~~~~~~ &lt;--- HERE
    _1 = (_global_graph).forward(input, valid_lens, )
    return _1
</code></pre>
Using shared memory to share model across multiprocess leads to memory exploded

hiha3456: <p>Hello, I am a newbie of Pytorch, currently having a reinforcement learning task, and I want to share the model among N processes on a single machine.</p>
<p>One of the processes is responsible for updating the weights of the model, and the other N-1 processes use the model for inference (i.e. the actor in actor-critic).</p>
<p>This design is adopted because the inference process needs to perform some operations on the CPU (env.step()) at the same time, so I hope to improve efficiency by detouring Python’s GIL lock using multi-processes.</p>
<p>After I found out that whether I put the model on CPU or GPU, as long as I use the shared memory mechanism, the consumption of  RAM’s and GPU’s memory increases linearly with the number of processes. It looks like each child process makes a copy of the model.<br>
And when using a shared model, for each child process, deploying the model on cuda takes more memory than deploying it on the CPU.</p>
<p>In my example, if I use spawn to create child processes, when the model is deployed on the CPU, the rss usage per process is about 1.24 GB; when deployed on the GPU, the rss usage per process is about 3.40 GB.</p>
<p>In addition, when deploying the model on the GPU, each child process will spawn an additional child process occupying 0.94 GB rss.</p>
<p>All of these leads to the poor scalability of our project.</p>
<p>If I use the message passing mechanism instead of shared memory, i.e. the main process not only performs update, but also performs inference for other sub-processes,and synchronizes with other sub-processes through message passing, it will bring a lot of I/O consumption, and the bottleneck of the whole system becomes the CPU of main process.</p>
<p>So here I have two main questions:</p>
<ol>
<li>Why does the total memory usage increase linearly with the number of processes when using shared memory? Is my code wrong or there exist other problems?</li>
<li>As seen above, shared memory and message passing each have their own issues, is there a way to bypass these issues to achieve the result I want?</li>
</ol>
<p>I put the code of my experiments <a href="https://github.com/hiha3456/mnist_hogwild_share_memory_exps" rel="noopener nofollow ugc">here</a>,<br>
and the results of exps are in the <a href="https://github.com/hiha3456/mnist_hogwild_share_memory_exps/blob/master/exp_results.md" rel="noopener nofollow ugc">file</a><br>
Theses exps based on <a href="https://pytorch.org/docs/stable/notes/multiprocessing.html" rel="noopener nofollow ugc">the example mnist_hogwild of Pytorch</a></p>
<p>I will release more details(e.g. screenshots ) if needed</p>
<p>Thanks a lot.</p>

mostafa_said: <p>hi,</p>
<p>did you got a solution for your problem</p>
Voxel-wise classification training pipeline

neurospacer: <p>I want to train simple neural network two layer classifier. Input data is 3-dimensional volumes of feature maps extracted from Brain MRI. I have manually extracted 6 feature maps each size of [128,128,128]. I want to feed each voxel into neural network, so input layer will have 6 neurons (for each map). The labels are the masks that manually segmented in region of brain malformation. I got following question regarding implementation<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/a/fa12bb05446732df39c1013c1a8b460cc836c5b0.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/fa12bb05446732df39c1013c1a8b460cc836c5b0" title="Screenshot 2022-10-11 at 14.33.26"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/a/fa12bb05446732df39c1013c1a8b460cc836c5b0_2_690x452.jpeg" alt="Screenshot 2022-10-11 at 14.33.26" data-base62-sha1="zGfuQSQxBhhB8MXC9gyX79B9FN6" width="690" height="452" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/a/fa12bb05446732df39c1013c1a8b460cc836c5b0_2_690x452.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/f/a/fa12bb05446732df39c1013c1a8b460cc836c5b0_2_1035x678.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/a/fa12bb05446732df39c1013c1a8b460cc836c5b0_2_1380x904.jpeg 2x" data-dominant-color="D9D9D9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2022-10-11 at 14.33.26</span><span class="informations">1584×1038 135 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div>:</p>
<ol>
<li>
<p>The issue is I don’t quite understand how to create a training pipeline and dataloader for this type of input. I assume to create PatchDataset with size 1x1x1  <a href="https://docs.monai.io/en/stable/data.html?highlight=patchdataset#patchdataset" class="inline-onebox" rel="noopener nofollow ugc">Data — MONAI 1.3.0 Documentation</a>. Are there examples of voxel-wise training example</p>
</li>
<li>
<p>I assume that iterating on each voxel is not right strategy to implement, how should class imbalance be regulated?</p>
</li>
</ol>
Mean shift function for gray image

Kamin: <p>The <strong>meanshift function</strong> given below adds or subtracts channel-wise mean (3-channel) from images. If the code needs to be re-written for 1 channel (gray images), is it necessary to apply this function? If so, how to change the code and how to calculate the mean value for gray images?</p>
<pre><code class="lang-auto">class MeanShift(nn.Module):
    def __init__(self, mean_rgb, sub):
        super(MeanShift, self).__init__()

        sign = -1 if sub else 1
        r = mean_rgb[0] * sign
        g = mean_rgb[1] * sign
        b = mean_rgb[2] * sign

        self.shifter = nn.Conv2d(3, 3, 1, 1, 0) #3 is size of output, 3 is size of input, 1 is kernel 1 is padding, 0 is group 
        self.shifter.weight.data = torch.eye(3).view(3, 3, 1, 1) # view(3,3,1,1) convert a shape into (3,3,1,1) eye(3) is a 3x3 matrix and diagonal is 1.
        self.shifter.bias.data   = torch.Tensor([r, g, b])
        #in_channels, out_channels,ksize=3, stride=1, pad=1
        # Freeze the mean shift layer
        for params in self.shifter.parameters():
            params.requires_grad = False

    def forward(self, x):
        x = self.shifter(x)
        return
</code></pre>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self, **kwargs):
        super(Net, self).__init__()
        self.sub_mean = MeanShift((0.4488, 0.4371, 0.4040), sub=True)
        self.add_mean = MeanShift((0.4488, 0.4371, 0.4040), sub=False)
</code></pre>
<p>If the dataset contains the gray images,</p>
What the warning is?

JeffffFu: <p>10/11/2022 17:28:42 -  Despite set_to_none is set to False, opacus will set p.grad_sample and p.summed_grad to None due to non-trivial gradient accumulation behaviour</p>
<p>I ran almost the same two pieces of code, why does this warning  appear in one, under what circumstances does this warning appear, does this warning have a big impact?</p>
<p>thanks!</p>

alexandresablayrolle: <p>TL;DR: This warning has basically no impact.</p>
<p>It is about the behavior of <code>optimizer.zero_grad()</code>. The <code>set_to_none</code> argument only affects <code>p.grad</code>; <code>p.grad_sample</code> and <code>p.summed_grad</code> are never zeroed out and always set to None.</p>
<p>We switched the level from warning to debug, hence the potential difference between the two codebases?</p>

JeffffFu: <p>yes, thank you very much</p>
Implementation of Improved SemHash

ycjing: <p>Hi!</p>
<p>Since there is already an implementation for Gumbel Softmax in the official PyTorch, I wonder if PyTorch also has the implementation of Improved SemHash (<a href="https://arxiv.org/abs/1801.09797" class="inline-onebox" rel="noopener nofollow ugc">[1801.09797] Discrete Autoencoders for Sequence Models</a>), like the semhash function in TensorFlow (<a href="https://github.com/tensorflow/tensor2tensor/blob/master/tensor2tensor/layers/discretization.py" class="inline-onebox" rel="noopener nofollow ugc">tensor2tensor/discretization.py at master · tensorflow/tensor2tensor · GitHub</a>), which is an alternative of Gumbel Softmax. I’d truly appreciate it if anyone could help. Thanks so much!</p>
<p>Cheers,</p>
How to operate torch.dot() in matrix consist of vectors in pytorch?

ellipsis: <p>I have tensor like this:</p>
<pre><code class="lang-auto">arr1 = np.array([[ 1.6194, -0.6058, -0.8012], [ 1.1483,  1.6538, -0.8062]])
arr2 = np.array([[-0.3180, -1.8249,  0.0499], [-0.4184,  0.6495, -0.4911]])
X = torch.Tensor(arr1)
Y = torch.Tensor(arr2)
</code></pre>
<p>I want to do torch.dot on every tensor 1D (2 vectors) inside my 2D tensor</p>
<pre><code class="lang-auto">torch.dot(X, Y)   
</code></pre>
<p>I want to get the result like this tensor([dotResult1, dotResult2]).<br>
But I got the error like this:</p>
<pre><code>RuntimeError: 1D tensors expected, but got 2D and 2D tensors
</code></pre>
<p>My main purpose is to do “something” operation on every vector inside my matrix but I don’t want to use looping here, does anyone know how to do that?</p>

KFrank: <p>Hi Ellipsis …</p>
<aside class="quote no-group" data-username="ellipsis" data-post="1" data-topic="163134" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/e/ed8c4c/48.png" class="avatar"> ellipsis:</div>
<blockquote>
<p>My main purpose is to do “something” operation on every vector inside my matrix but I don’t want to use looping here</p>
</blockquote>
</aside>
<p>This will depend on what “something” is.  In general, you would want to<br>
figure out how to do “something” using pytorch tensor operations that act<br>
on your 2D tensors all at once.</p>
<p>For your particular example of computing the dot products of the vectors<br>
in your 2D tensors you can use element-wise multiplication and then sum<br>
the rows or, equivalently, use the general purpose <a href="https://pytorch.org/docs/1.12/generated/torch.einsum.html" rel="noopener nofollow ugc">einsum()</a> to do the same<br>
thing:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.12.0'
&gt;&gt;&gt; X = torch.tensor ([[ 1.6194, -0.6058, -0.8012], [ 1.1483,  1.6538, -0.8062]])
&gt;&gt;&gt; Y = torch.tensor ([[-0.3180, -1.8249,  0.0499], [-0.4184,  0.6495, -0.4911]])
&gt;&gt;&gt; resultA = (X * Y).sum (dim = 1)
&gt;&gt;&gt; resultA
tensor([0.5506, 0.9896])
&gt;&gt;&gt; resultB = torch.einsum ('ij, ij -&gt; i', X, Y)
&gt;&gt;&gt; torch.equal (resultA, resultB)
True
</code></pre>
<p>(As you’ve noticed, pytorch does not offer a specific “batch-vector-dot-product”<br>
function – not that it needs one.)</p>
<p>Best.</p>
<p>K. Frank</p>

ellipsis: <p>Thank you it’s work!</p>
Find indices of a tensor satisfying a condition

Abueidda: <p>Does torch have a function that helps in finding the indices satisfying a condition?</p>
<p>For instance,</p>
<pre><code class="lang-auto">F = torch.randn(10)
b = torch.index(F &lt;= 0)
print(b)
</code></pre>
<p>Thanks!</p>

Nikronic: <p>HI,</p>
<p>torch uses same convention as numpy such for finding values or indices of particular tensor regarding specific condition. <a href="https://pytorch.org/docs/master/generated/torch.where.html" rel="nofollow noopener">torch.where</a> and <a href="https://pytorch.org/docs/master/generated/torch.nonzero.html#torch.nonzero" rel="nofollow noopener">torch.nonzero</a></p>
<p>What you can do is to apply your condition and get a binary mask of indices that match the condition and find the indices using <code>torch.nonzero()</code>.</p>
<pre><code class="lang-auto">import torch

a = torch.randn(10)
b = a &lt;= 0
indices = b.nonzero()
</code></pre>
<p>bests</p>

Abueidda: <p>Thanks, <a class="mention" href="/u/nikronic">@Nikronic</a>.  This really solves it.</p>

deshwal.mahesh: <p>How can you do it for <code>2-D</code>   <code>MxM</code> tensor?</p>

harpone: <aside class="quote no-group" data-username="Nikronic" data-post="2" data-topic="80802">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nikronic/48/71873_2.png" class="avatar"> Nikronic:</div>
<blockquote>
<p><code>.nonzero()</code></p>
</blockquote>
</aside>
<p>The same way, i.e. <code>.nonzero()</code></p>

Ojasv_Kamal: <p>Hi <a class="mention" href="/u/harpone">@harpone</a><br>
Is it possible to find the index of an N-D tensor? For example, can I find the index of <code>torch.tensor([1, 2])</code> in <code>torch.tensor([[1,2], [3, 4], [5, 6]])</code>?</p>

harpone: <p>yeah your latter tensor here is 2D tensor and .nonzero() works fine with that. In general if you have an ND tensor, you will get an index tensor of shape <code>[num_matches, N]</code>.</p>

Sirojbek_Safarov: <aside class="quote no-group quote-modified" data-username="Nikronic" data-post="2" data-topic="80802">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nikronic/48/71873_2.png" class="avatar"> Nikronic:</div>
<blockquote>
<pre><code class="lang-auto">
</code></pre>
</blockquote>
</aside>
<p>Hello. Is there any way without torch.nonzero or torch.where. These functions make a NoNZero node in onnx and this nonzero node cannot be converted to tensorrt</p>
How to convert this code to pytorch from tensorflow

Thang_Nguyen_Viet: <p>import tensorflow as tf</p>
<p>def model_mobilenetv2(input_tensor, train_lambda=0.1):<br>
model_input = tf.keras.layers.Input(tensor=input_tensor)<br>
base_model = tf.keras.applications.MobileNetV2(include_top=False, weights=‘imagenet’, input_tensor=model_input, pooling=‘avg’)</p>
<pre><code># for parking space type
_ = tf.keras.layers.Dense(units=128, activation='relu')(base_model.output)
type_output = tf.keras.layers.Dense(units=4, activation='softmax', name='type_output')(_)

# for parking line angle
_ = tf.keras.layers.Dense(units=128, activation='relu')(base_model.output)
angle_output = tf.keras.layers.Dense(units=1, activation='sigmoid', name='angle_output')(_)

model = tf.keras.models.Model(inputs=model_input, outputs=[type_output, angle_output])

for layer in model.layers:
    layer.trainable = True

opt = tf.keras.optimizers.Adam()

model.compile(optimizer=opt,
              loss={'type_output': 'sparse_categorical_crossentropy', 'angle_output': 'mse'},
              loss_weights={'type_output': train_lambda, 'angle_output': (1-train_lambda)},
              metrics={'type_output': 'accuracy', 'angle_output': 'mae'})

return model
</code></pre>
How to contribute to pytorch documentation in translation?

nunenuh: <p>Hi, I’ve been using pytorch since version 0.3. I see pytorch only available in english language.<br>
Does pytorch have any plan to make the translation available to other languages as tensorflow did?<br>
If there is a plan for translate pytorch documentation, how to contribute to pytorch documentation for the Indonesian Language?</p>

ptrblck: <p>I think Soumith might have some information regarding the translation of the docs and might also know, if someone is already working on it.</p>
<p>CC <a class="mention" href="/u/smth">@smth</a></p>

gusti: <p>Hi, I am also interested in contributing to the Pytorch documentation for Bahasa Indonesia.</p>
Make_fx and vjp

awf: <p>The <a href="https://github.com/pytorch/functorch/blob/main/README.md" rel="noopener nofollow ugc">functorch README</a> has a nice example of source-to-source AD for grad (which also works nicely for jacrev)</p>
<pre><code class="lang-auto">from functorch import make_fx, jacrev
def f(x):
    return torch.sin(x).sum()
x = torch.randn(100)
grad_f = make_fx(jacrev(f))(x)
print(grad_f.code)
</code></pre>
<p>I’m trying to make it work for VJP, but can’t quite get the incantation.  I would have thought this should work</p>
<pre><code class="lang-auto">from functorch import make_fx, grad, vjp, jacrev
def f(x):
    return torch.sin(x)
x = torch.randn(10)
dret = torch.randn(10)
grad_f = make_fx(vjp(f,x)[1])(dret)
print(grad_f.code)
</code></pre>
<p>But it throws</p>
<pre><code class="lang-auto">RuntimeError: Tracing expected 3 arguments but got 1 concrete arguments
</code></pre>
<p>I guess I can see why it might not work, but wonder if there’s a workaround.</p>
Way to register custom datapipe functional method to pyi?

Kiuk_Chung: <p>If I implemented a custom DataPipe (iter or map) as:</p>
<pre><code class="lang-auto">@functional_datapipe("foo")
class FooPipe(IterDataPipe[str]):
    ...
</code></pre>
<p>Is there a way I can add the functional datapipe method <code>foo</code> <code>torchdata.datapipes.iter.IterDataPipe</code> so that auto-complete would work in an IDE?</p>
<p>I see that the functional methods for pipes supported natively by torchdata are added in <code>torchdata.datapipes.iter|map.__init__.pyi</code> and hence I get nice TAB-completion support in an IDE/interpreter.</p>
<p>Additionally for datapipes that are supposed to be used at the head of the chain (e.g. those that don’t take a source pipe). Is there a way to access them functionally (see example below):</p>
<pre><code class="lang-auto">@functional_datapipe("s3_list")
class CustomS3ListDatapipe(IterDataPipe[str]):
    def __init__(bucket: str, key: str):
         # no source data pipe required
         ...

# any way I can use CustomS3ListDatapipe as
import torchdata.datapipes.iter as pipe

pipe.s3_list(bucket="foo", key="bar").map(...).load_files_by_s3()

# instead of

CustomS3ListDatapipe(bucket="foo", key="bar").map(...).load_files_by_s3()
</code></pre>

nivek: <ol>
<li>Adding custom DataPipe functional method to .pyi</li>
</ol>
<p>It is currently not possible because we generate the .pyi ahead of time. If you are building from source, you can potentially add your custom DataPipe to <code>torchdata.datapipes.iter</code> and run <code>python setup.py develop</code> to update the relevant .pyi file . I understand that this is cumbersome and unlikely to be useful for you.</p>
<p>The better approach that we have considered is to update the .pyi file when <a href="https://github.com/pytorch/pytorch/blob/master/torch/utils/data/datapipes/datapipe.py#L134" rel="noopener nofollow ugc"><code>register_datapipe_as_function</code></a> is called, but we have not found the time to do that yet. Since that is called by the decorator <code>@functional_datapipe</code>, it will ensure that user defined DataPipes can be added for IDE auto completion as well.</p>
<p>We are happy to accept PR/suggestion related to this issue. See the related <a href="https://github.com/pytorch/data/issues/726" rel="noopener nofollow ugc">open issue here</a>.</p>
<ol start="2">
<li>Using <code>pipe.s3_list</code> instead of the class name</li>
</ol>
<p>We likely will not provide that option since <code>@functional_datapipe</code> is meant to be applied onto DataPipe and using the class name may only use a few more characters. Let us know if there is any reason why that may be needed.</p>

Kiuk_Chung: <p>Thanks for pointing to the related issue! As for <span class="hashtag">#2</span>, mostly just wondering whether its possible for consistency sake - for those datapipes that do not accept source pipes as ctor args, if there was a way to register them as functions to <code>torchdata.datapipes.iter</code> (or <code>map</code>) then we can go purely functional. But not a big deal if not supported.</p>
How to do logical indexing of a torch tensor?

copyrightly: <p>I have a torch tensor <code>x</code> with size <code>torch.Size([10, 1, 28, 28])</code> and a logical np array <code>a</code>:</p>
<pre><code class="lang-auto">array([False, False, False, False, False, False,  True,  True,  True, True])
</code></pre>
<p>I want to select the first dimension of <code>x</code> according to logical values in <code>a</code>. But I got error when I try <code> x[a]</code>. What should I do?</p>

Eta_C: <p><code>x[torch.from_numpy(a)]</code></p>

copyrightly: <p>I got the error <code>can't convert np.ndarray of type numpy.bool_.</code><br>
But<code> x[torch.from_numpy(np.array(a, dtype=np.uint8))]</code> works. Thank you!</p>

JuyiLin: <pre><code> dim 1 can be like `out = x[:, a]`
</code></pre>
Vastly different behaviour FasterRCNN v1 and v2

GeorgePearse: <p><a href="https://pytorch.org/vision/master/_modules/torchvision/models/detection/faster_rcnn.html" class="inline-onebox" rel="noopener nofollow ugc">torchvision.models.detection.faster_rcnn — Torchvision main documentation</a> on this page there are two different implementations:</p>
<ul>
<li>
<p>fasterrcnn_resnet50_fpn (I get clutter overlapping predictions and low map50 performance), but importantly no NMS is applied which is really what I’m after, I want control of that step.</p>
</li>
<li>
<p>fasterrcnn_resnet50_fpn_v2. NMS seems to be applied internally, and I get beautiful clean bounding boxes, but cluttered scenes fall apart and I get one big bounding box for the whole frame (it’s actually more of a counting problem, so that’s about the worst possible). Much higher map50s though.</p>
</li>
</ul>
<p>Basically I’m after the performance of the latter with the behaviour of the first.<br>
Lots of the parameters of both should be responsible for tweaking the internal NMS, but so far I’ve had very little luck with them.</p>
<p>nms_thresh=1 should turn off NMS right?</p>
TypeError: expected sequence object with len >= 0 or a single integer

Nishanth_Gowda: <p>Hi All,</p>
<p>I have 4 GPUs in my machine and I am using Dataparallel. I have set the device as:<br>
<code>torch.cuda.set_device(0)</code> However I am getting this error when I have batch size of 1. Could you please suggest what might be the problem?</p>
<p><code>Traceback (most recent call last):   File "single_evaluate.py", line 105, in &lt;module&gt;     test(model, manager)   File "single_evaluate.py", line 44, in test     output_batch = model(data_batch)   File "/home/nishanth/anaconda3/envs/try_midas/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl     result = self.forward(*input, **kwargs)   File "/home/nishanth/anaconda3/envs/try_midas/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 162, in forward     return self.gather(outputs, self.output_device)   File "/home/nishanth/anaconda3/envs/try_midas/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 174, in gather     return gather(outputs, output_device, dim=self.dim)   File "/home/nishanth/anaconda3/envs/try_midas/lib/python3.7/site-packages/torch/nn/parallel/scatter_gather.py", line 68, in gather     res = gather_map(outputs)   File "/home/nishanth/anaconda3/envs/try_midas/lib/python3.7/site-packages/torch/nn/parallel/scatter_gather.py", line 62, in gather_map     for k in out))   File "/home/nishanth/anaconda3/envs/try_midas/lib/python3.7/site-packages/torch/nn/parallel/scatter_gather.py", line 62, in &lt;genexpr&gt;     for k in out))   File "/home/nishanth/anaconda3/envs/try_midas/lib/python3.7/site-packages/torch/nn/parallel/scatter_gather.py", line 63, in gather_map     return type(out)(map(gather_map, zip(*outputs)))   File "/home/nishanth/anaconda3/envs/try_midas/lib/python3.7/site-packages/torch/nn/parallel/scatter_gather.py", line 63, in gather_map     return type(out)(map(gather_map, zip(*outputs)))   File "/home/nishanth/anaconda3/envs/try_midas/lib/python3.7/site-packages/torch/nn/parallel/scatter_gather.py", line 63, in gather_map     return type(out)(map(gather_map, zip(*outputs))) TypeError: expected sequence object with len &gt;= 0 or a single integer</code></p>

agu: <p>We recommend <code>DistributedDataParallel</code> over <code>DataParallel</code>. Perhaps, you can try <code>DistributedDataParallel</code> out first instead of trying to debug your current <code>DataParallel</code> script?</p>