akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Dictionary to tensor in cpp

patel_shahrukh: <p>Hello <span class="mention">@all</span></p>
<p>i am facing the issue in converting dictionary to tenor in iOS</p>

hanton: <p>Hi <a class="mention" href="/u/patel_shahrukh">@patel_shahrukh</a>, could you elaborate more about your use case, do you want to convert iOS Dictionary(NSDictionary) to PyTorch Tensor, or IValue? What issue are you facing?</p>
Welcome to Pytorch Live

dmitriylyan: <p>Hello community! We are psyched to launch PyTorch Live in public beta. It’s a set of tools to help you build on-device ML prototypes on iOS and Android in matter minutes. You can find tutorials on how to get started on our <a href="https://www.pytorch.org/live" rel="noopener nofollow ugc">website</a> and on <a href="https://github.com/pytorch/live" rel="noopener nofollow ugc">github</a>.</p>
<p>This is our first public release, and we are planning to release updates regularly, based on community feedback. So let us know about your experience building with PyTorch Live. Happy to engage and help you along!</p>
CrossEntropyLoss vs per-class-probabilities target

my3bikaht: <p>Documentation mentions that it is possible to pass per class probabilities as a target.</p>
<blockquote>
<p>The target that this criterion expects should contain either:<br>
…<br>
Probabilities for each class;<br>
…<br>
Target: … If containing class probabilities, same shape as the input.</p>
</blockquote>
<p>which also comes with example:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; # Example of target with class probabilities
&gt;&gt;&gt; input = torch.randn(3, 5, requires_grad=True)
&gt;&gt;&gt; target = torch.randn(3, 5).softmax(dim=1)
&gt;&gt;&gt; output = loss(input, target)
&gt;&gt;&gt; output.backward()
</code></pre>
<p>Example fails with</p>
<pre><code class="lang-auto">  File "../torch/nn/functional.py", line 2824, in cross_entropy
    return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
RuntimeError: 1D target tensor expected, multi-target not supported
</code></pre>
<p>Am I missing something?</p>

KFrank: <p>Hi Sergey!</p>
<aside class="quote no-group" data-username="my3bikaht" data-post="1" data-topic="138331" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/50afbb/48.png" class="avatar"> my3bikaht:</div>
<blockquote>
<pre><code class="lang-auto">  File "../torch/nn/functional.py", line 2824, in cross_entropy
    return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
RuntimeError: 1D target tensor expected, multi-target not supported
</code></pre>
</blockquote>
</aside>
<p>Support for “soft,” probabilistic targets for <code>CrossEntropyLoss</code> is new<br>
as of (I believe) the current stable version, 1.10.0.</p>
<p>Your best bet will be to upgrade to the current stable release.</p>
<p>Best.</p>
<p>K. Frank</p>
Libtorch build does not work in MFC environment

HwanUng_Lee: <p>Hello.<br>
I took the model from the console to the Libtorch and tested it using VS2015.</p>
<p>However, in the MFC environment, the inclusion of “torch\script” in the header results in an error.</p>
<p>I tested it in the console environment again for comparison, but I found it working properly on the console.</p>
<p>Is Libtorch not available in MFC environments?</p>
<ul>
<li>Visual Studio 2015</li>
<li>CUDA = 10.0</li>
<li>Libtorch = <a href="https://download.pytorch.org/libtorch/cu101/libtorch-win-shared-with-deps-1.4.0.zip" rel="noopener nofollow ugc">https://download.pytorch.org/libtorch/cu101/libtorch-win-shared-with-deps-1.4.0.zip</a></li>
</ul>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/c/7cf885b5ebbd6fc8bf41b8e79ada1b3621c2fea0.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/7cf885b5ebbd6fc8bf41b8e79ada1b3621c2fea0" title="캡처"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/c/7cf885b5ebbd6fc8bf41b8e79ada1b3621c2fea0_2_690x425.jpeg" alt="캡처" data-base62-sha1="hPxF6h82ligfgiZhc4k7ZXSd6py" width="690" height="425" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/c/7cf885b5ebbd6fc8bf41b8e79ada1b3621c2fea0_2_690x425.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/7/c/7cf885b5ebbd6fc8bf41b8e79ada1b3621c2fea0.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/7/c/7cf885b5ebbd6fc8bf41b8e79ada1b3621c2fea0.jpeg 2x" data-dominant-color="2E3136"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">캡처</span><span class="informations">834×514 77.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Valerio_1: <p>Hi.<br>
I have the same problem, I need to use PyTorch C++ API in an existing MFC application. No problem in a console application, but somehow MFC seems to be incompatible with the PyTorch C++ API.<br>
Did you manage to solve the problem? Any idea?<br>
Many thanks and kind regards,<br>
VALERIO</p>
Dataloader: for csv files

Kanchon-Kanti-Podder: <p>Hello Everyone. I am new and only basic knowledge on PyTorch. I need a custom Dataloader. The directory of my dataset will be<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/7/e/7e893a906f0e1baa586da81782204c607b111d59.png" alt="aaa" data-base62-sha1="i3oaHi77DGgOycV8AF2Hwdh0qBb" width="566" height="413"></p>
<p>In return I need batch of csv files and class names (Ex:Class 1, Class 2).</p>
<p>Thank you in advance.</p>
Correct GPU memory allocation

dr.interned: <p>Hello</p>
<p>I have a server equipped with 3 Quadro P2000 and one GTX 1050. Quadros have 5GB of video memory and GTX has 4GB. I use DataParallel for multi gpu processing. But i notised, that batch size limited to 4gb and when i try to increase batch size I catch OOM on GTX.<br>
For example if i use batch size 50 GTX memory is full, but each Quadro use only 4GBs instead of 5GB. So I got 3GB of unused GPU Ram in total. And when I increase batch size to 56 I catch OOM beacuse the GTX memory is full.</p>
<p>So the question is: What is the correct way to use all available gpu memory</p>
<p>I use this code to parallalize training</p>
<pre><code class="lang-python">net = model()
    if torch.cuda.device_count() &gt; 1:
        net = torch.nn.DataParallel(net, device_ids=list(range(torch.cuda.device_count()))) 
    net.to('cuda')
</code></pre>
<p>Ps. GTX index is 0</p>
<p>Thank you for your support !!!</p>

eqy: <p>I believe <code>DataParallel</code> assumes identical or very similar GPUs give the <code>_check_balance</code> warning here:<br>
<a href="https://pytorch.org/docs/stable/_modules/torch/nn/parallel/data_parallel.html#DataParallel" rel="noopener nofollow ugc">torch.nn.parallel.data_parallel — PyTorch 1.10.0 documentation</a></p>
<p>However, you may want to check that using the 1050 is beneficial from a load-balancing perspective to begin with as it may be bottlenecking training compared to the P2000s even when accounting for the difference in available memory.</p>

my3bikaht: <p>Also take a look at this post:</p><aside class="quote" data-post="1" data-topic="77891">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/deeply/48/3840_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/dataparallel-vs-distributeddataparallel/77891">DataParallel vs DistributedDataParallel</a> <a class="badge-category__wrapper " href="/c/distributed/12"><span data-category-id="12" data-drop-close="true" class="badge-category "><span class="badge-category__name">distributed</span></span></a>
  </div>
  <blockquote>
    Hi, 
what is the difference between 
model = nn.parallel.DistributedDataParallel(model, device_ids=[args.gpu]) 
and 
model = nn.DataParallel(model, device_ids=[args.gpu]) 
?
  </blockquote>
</aside>


dr.interned: <p>Seems like Distributed Data Parallel is the solution. Thanks a lot!!! I’ll try this</p>

dr.interned: <p>Thank you for your reply! It looks like regular dataParallel is not the best way for multi gpu training. Thank you!</p>
nn.ModuleList.apply()?

squirrel: <p>It’s weight initialization code below:</p>
<pre><code>import torch.nn.init as init

def xavier(param):
    init.xavier_uniform(param)
def weights_init(m):
    if isinstance(m, nn.Conv2d):
        xavier(m.weight.data)
        m.bias.data.zero_()
</code></pre>
<p>if VGG is a nn.ModuleList, and:</p>
<pre><code>    VGG.apply(weights_init)
</code></pre>
<p>dosen’t throw an error.</p>
<p>But I just can’t find any documentation or statement about this .apply() function. Where could I find it?<br>
Thanks a lot.</p>

smth: <p>The documentation seems to be missing (i’ve made a note to add it), but <code>apply</code> loops over each child Module and applies a function to it.</p>

smth: <p>just for closure of this thread, documentation is here: <a href="http://pytorch.org/docs/master/nn.html?highlight=modulelist#torch.nn.Module.apply">http://pytorch.org/docs/master/nn.html?highlight=modulelist#torch.nn.Module.apply</a></p>

blade: <p>It seems the above link is outdated. As of Dec '21, the documentation is here:<br>
<a href="https://pytorch.org/docs/master/generated/torch.nn.Module.html?highlight=apply#torch.nn.Module.apply" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/master/generated/torch.nn.Module.html?highlight=apply#torch.nn.Module.apply</a></p>
Unable to load weights created in CUDA into CPU

Timangar: <p>I have a .pt file of a model I trained in the libtorch CUDA backend. I now tried to load this model on another machine which does not support CUDA using</p>
<blockquote>
<p>torch::load(model, “model.pt”);</p>
</blockquote>
<p>throws the c10::Error</p>
<blockquote>
<p>Could not run ‘aten::empty_strided’ with arguments from the ‘CUDA’ backend.</p>
</blockquote>
<p>Any ideas?</p>

mbullo: <p>Hi Timangar. I’m not familiar with c++ but looking at the docs <a href="https://pytorch.org/cppdocs/api/function_namespacetorch_1a894930db6e4c535bd28eb9e329837361.html#_CPPv4I0DpEN5torch4loadEvR5ValueDpRR12LoadFromArgs" rel="noopener nofollow ugc"><code>torch::load()</code></a> should call <a href="https://pytorch.org/cppdocs/api/classtorch_1_1serialize_1_1_input_archive.html#_CPPv4N5torch9serialize12InputArchive9load_fromERNSt7istreamEN3c108optionalIN5torch6DeviceEEE" rel="noopener nofollow ugc"><code>load_from</code></a> which allows to specify the device through the parameter <code>device</code>. Maybe this can help!</p>
Change Mask RCNN Config

Zret3ck: <p>Hi I am currently following this <a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html" rel="noopener nofollow ugc">guide</a>  to create a RCNN that is able to detect ears. I have trained it with my training dataset for 50 epochs and the final result is either really accurate or has a lot of noise</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/d/1d47368df881a4a1875bf72b34e683fbde350a92.jpeg" alt="0019" data-base62-sha1="4b0q2x9LZU81lZwPSDtt0YlQvPc" width="480" height="360"></p>
<p>Would it be possible to change the config for the RCNN to specify the maximum amount of ears in an image, or specify a higher minimum confidence? Maybe some other parameters which might help in increasing the accuracy too</p>
<p>Get Model Function:</p>
<pre><code>def get_model_instance_segmentation(num_classes):

# load an instance segmentation model pre-trained pre-trained on COCO

model = torchvision.models.detection.maskrcnn_resnet50_fpn(pretrained=True)

# get number of input features for the classifier

in_features = model.roi_heads.box_predictor.cls_score.in_features

# replace the pre-trained head with a new one

model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)

# now get the number of input features for the mask classifier

in_features_mask = model.roi_heads.mask_predictor.conv5_mask.in_channels

hidden_layer = 256

# and replace the mask predictor with a new one

model.roi_heads.mask_predictor = MaskRCNNPredictor(in_features_mask,

                                                hidden_layer,

                                                num_classes)

#FIXME doesn't work, check how to save/load the trained model

#if os.path.exists(model_PATH):

#    model.load_state_dict(torch.load(model_PATH))

return model
</code></pre>
<p>Evaluate Test Function (Probably Unrelated):</p>
<pre><code>@torch.inference_mode()

def evaluate(model, data_loader, device):

#n_threads = torch.get_num_threads()

# FIXME remove this and make paste_masks_in_image run on the GPU

#torch.set_num_threads(1)

#cpu_device = torch.device("cpu")

model.eval()

with torch.no_grad():

    for idx, (images, targets) in enumerate(data_loader):

        images = list(img.to(device) for img in images)

        prediction = model(images)

        image_mask = np.zeros((360, 480, 1), dtype="uint8")

        for i in range(len(prediction[0]['masks'])):

            # iterate over masks

            mask = prediction[0]['masks'][i, 0]

            mask = mask.mul(255).byte().cpu().numpy()

            contours, _ = cv2.findContours(

                    mask.copy(), cv2.RETR_CCOMP, cv2.CHAIN_APPROX_NONE)

            cv2.drawContours(image_mask, contours, -1, (255, 0, 0), 2, cv2.LINE_AA)

        cv2.imwrite("segmentation/results/" + format(idx + 1, "04") + '.jpeg', image_mask)
</code></pre>
How to find out if layer is trained or not

MohamedA95: <p>Hello everyone,<br>
I have a situation where a model would have different topologies each with a different forward function, during training I  choose the topology which defines the forward function implementation. Later I need to be able to traverse through the trained layer without knowing which topology was used to train the model. Currently, when I load the model using <code>net.load_state_dict(checkpoint)</code> I get all the layers, those who are used in the chosen forward function &amp; those who are not (trained &amp; untrained layers). I have an idea but Idk if there is a better way or not. After doing <code>torch.load</code> I would loop over <code>checkpoint.keys()</code> and split the name over the ‘.’ to extract the first part i.e <code>encoder_1.conv1.bias</code> would be <code>encoder_1</code> then I would collect those names in a list and use it to extract the data from <code>net.load_state_dict(checkpoint)</code>. But this looks too hacky is not it?? is there is a more straightforward way??</p>
What version of GCC do you build libtorch with?

KevNull: <p>I’ve seen people say they’ve successfully build with GCC 5.4.0, and so have I, but has anyone built with anything more recent on Linux?</p>
<p>I’m having problems integrating into a larger project that I think is due to using such different compilers, and the larger project does not compile with 5.4, hence why I’m asking. Any 6.somethings maybe? Using CUDA 9.2</p>

tom: <p>I’ve been one of the people who used GCC 5 for a long time.<br>
With recent cuda (I’m currently on 9.2) supporting newer GCC I had switched to 6 and this week (because I had trouble with MKLDNN’s use  of <code>-werror</code>) to gcc 7.<br>
For my (CPU only) laptop, I always use the default compiler of Debian/unstable, currently gcc 8.2.0.</p>
<p>Best regards</p>
<p>Thomas</p>

KevNull: <p>Thanks for the info. I think I ran into the same problem with 6 last week.</p>

tom: <p>As long as nvcc doesn’t complain, I’d not be too concerned these days. 4.8 (or whatever it was) vs. later was a huge pain for this C++11 ABI issue, but I haven’t had any trouble past that.</p>
<p>Best regards</p>
<p>Thomas</p>

KevNull: <p>Thanks. I’ve build Pytorch in GCC 7 just now. Things have been coming together nicely in the past six weeks.</p>

jose: <p>Is there any step by step guide on how to build Pytorch from source using gcc / g++ &gt;5? I need to get libtorch from source on my macOS, but I keep getting error after error.</p>

driazati: <p>The <a href="https://github.com/pytorch/pytorch#from-source" rel="nofollow noopener">README.md</a> has instructions to build from source and <a href="https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md#developing-pytorch" rel="nofollow noopener">CONTRIBUTING.md</a> may also have some more details. Feel free to file an issue on GitHub with the error output if the build isn’t working.</p>

surak: <p>In <a href="https://easybuild.io" rel="noopener nofollow ugc">EasyBuild</a>, we got as far as GCC 10, and I’m struggling with GCC 11. There’s a number of small things failing with it. Check Easybuild’s github repo for the patches applied in order to get it to work with GCC10…</p>
Parallelism with PyTorch. Where to get an overview?

pascalm: <p>Hello,</p>
<p>so I’m new to pyTorch and very unsure about parallelism in Pytorch. I’ve written my own parallel programs with openMP and MPI. So far only using CPU (or distributed systems with CPU nodes), never GPUs/CUDA. I’m just mentioning that to say: I think I’m familiar with the basics.</p>
<p>I know that pyTorch supports several devices but I’m very unsure about how it exactly does that. I think one big philosophy of pyTorch is:</p>
<blockquote>
<p>Use torch.tensor() for data.</p>
</blockquote>
<p>Of couse, we also need some logic i.e. a neural network. For which we use e.g. <code>nn.module</code>.</p>
<p>So we have “logic and data”.</p>
<p>Now I know that I could e.g. set up my neural network, use <code>to(device)</code>, set up some data and do <code>tensor(device=device)</code> and it works. But I could also done one hundred other things.</p>
<p>What is a good way of doing things? I don’t have any fancy setup yet and i don’t do any fancy work/load balance or whatever. I simply have my neural network, 1 cpu, 1 gpu and I want it to be parallelized.</p>
<p>What I do currently is:</p>
<ol>
<li>Get my neural network.</li>
<li>Use to(device=device) to send my model to the device</li>
<li>Use torch.utils.data.Dataset(device=device)  and in <strong>getitem</strong> send the torch.tensors() to the device</li>
</ol>
<p>Is that okay? Can I somehow globally define a device s.t. everything is just send to that device? Do I have to model my data in a specific way to enable parallelism? I can imagine, that I somehow have to add extra care here because how else would pyTorch know which data depends on another.</p>
Using loss.backward() once and still facing RuntimeError: Trying to backward through the graph a second time. Specify retain_graph=True

Aman_Pathak: <p>Below is the model definition and the training loop.<br>
I referred a lot of solutions and tried <code>retain_graph = True</code>, no luck.</p>
<p>Model:</p>
<pre><code class="lang-auto">class modelClassifier (nn.Module):
    def __init__(self, bottleneck_dim):
        super(modelClassifier, self).__init__()

        self.bottleneck_dim = bottleneck_dim
        self.fc01 = nn.Linear(self.bottleneck_dim, 200)
        self.fc02 = nn.Linear(200,10)

    def forward(self,x):
        
        x = F.relu(self.fc01(x))
        x = F.relu(self.fc02(x))
        return x
</code></pre>
<p>Model initailization:</p>
<pre><code class="lang-auto">model_clf = modelClassifier(bottleneck_dim=60)
optimizer_clf = torch.optim.SGD(model_clf.parameters(), lr = 0.001)
loss_function_clf = nn.CrossEntropyLoss()
</code></pre>
<p>Training loop:</p>
<pre><code class="lang-auto">EPOCHS = 10 
for epoch in range(EPOCHS):
    for batch_num, (data, target) in enumerate(dataloader_clf):
        model_clf.train()      
        outputs = model_clf(data)
        loss_clf = loss_function_clf(outputs, target)
        optimizer_clf.zero_grad()
        loss_clf.backward(retain_graph = True)
        optimizer_clf.step()
</code></pre>
<p>Error Traceback:</p>
<pre><code class="lang-auto">RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-149-07e5978e3c22&gt; in &lt;module&gt;
     10 
     11         optimizer_clf.zero_grad()
---&gt; 12         loss_clf.backward(retain_graph = True)
     13 
     14         optimizer_clf.step()


RuntimeError: Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed). Saved intermediate values of the graph are freed when you call .backward() or autograd.grad(). Specify retain_graph=True if you need to backward through the graph a second time or if you need to access saved tensors after calling backward.

</code></pre>

ptrblck: <p>Your code works fine without <code>retain_graph=True</code>:</p>
<pre><code class="lang-python">model_clf = modelClassifier(bottleneck_dim=60)
optimizer_clf = torch.optim.SGD(model_clf.parameters(), lr = 0.001)
loss_function_clf = nn.CrossEntropyLoss()

for _ in range(10):
    data = torch.randn(1, 60)
    target = torch.randint(0, 10, (1,))
    model_clf.train()      
    outputs = model_clf(data)
    loss_clf = loss_function_clf(outputs, target)
    optimizer_clf.zero_grad()
    loss_clf.backward()
    optimizer_clf.step()
</code></pre>
<p>I would generally not recommend to use <code>retain_graph=True</code> unless you have a valid reason to do so.</p>

Aman_Pathak: <p>Thank you for the quick response.</p>
<p>I’ll try it out and revert back.</p>
How to create a "class object" in C++?

Vadim: <p>I have a model that takes a Python class as a parameter to forward():</p>
<pre><code class="lang-auto">class Inputs:
    def __init__(self, a:Tensor, b:Tensor, c:Tensor):
        self.a = a
        self.b = b
        self.c = c

class Model(nn.Module):
    def forward(self, inputs: Inputs):
        ...
</code></pre>
<p>After converting it to a ScriptedModule and saving to a file, I would like to load and use this model in a C++ program.<br>
However, I am not sure what would be the equivalent of <code>Inputs</code> class on the C++ side…  Any pointers to docs/examples that show how to do this?</p>
Passing Tensor pointers to CUDA code

A_A: <p>Hi,</p>
<p>I have a CUDA application which I want to interface with PyTorch with following conditions:</p>
<ol>
<li>
<p>I do not want to create additional dependency of PyTorch in the C++ application for multiple reasons (e.g., CUDA version conflicts).</p>
</li>
<li>
<p>I want to send/receive GPU data from PyTorch in python to this C++ code. I am thinking of passing tensor.data_ptr() (after making it contiguous) to the C++ code and also pass a pointer to the pre-initialized output array which will be filled by C++ code. Basically following the approach from: <a href="https://discuss.pytorch.org/t/constructing-pytorchs-cuda-tensor-from-c-with-image-data-already-on-gpu/3124" class="inline-onebox">Constructing PyTorch's CUDA tensor from C++ with image data already on GPU</a></p>
</li>
</ol>
<p>Does 2. seem to be the best way to achieve 1? Any pitfalls I should take into consideration?</p>
<p>Thanks,<br>
Ahmed</p>

ptrblck: <p>Your approach sounds valid and you could take a look at <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html">this tutorial</a> to see how a custom CUDA extension can be written (and in particular the CUDA kernel).</p>

A_A: <p>I have written following code (for reference to future visitors) and it seems to be running fine.</p>
<pre><code class="lang-auto">m.def("my_func_with_gpu_pointers", [](const long i_ptr, const long j_ptr, const long edge_costs_ptr, const long sol_out_ptr, const int num_edges, const int gpuDeviceID) 
{
  cudaSetDevice(gpuDeviceID);
  const int* const i = reinterpret_cast&lt;const int* const&gt;(i_ptr);
  const int* const j = reinterpret_cast&lt;const int* const&gt;(j_ptr);
  const float* const edge_costs = reinterpret_cast&lt;const float* const&gt;(edge_costs_ptr);
  int* const sol_out_ptr_int = reinterpret_cast&lt;int* const&gt;(sol_out_ptr);
  thrust::device_vector&lt;int&gt; i_thrust(i, i + num_edges);
  thrust::device_vector&lt;int&gt; j_thrust(j, j + num_edges);
  thrust::device_vector&lt;float&gt; costs_thrust(edge_costs, edge_costs + num_edges);
  thrust::device_vector&lt;int&gt; sol;
  sol = my_func(std::move(i_thrust), std::move(j_thrust), std::move(costs_thrust)); 
  thrust::copy(sol.begin(), sol .end(), sol_out_ptr_int);
}
</code></pre>
[Solved] Confusing classification loss results

ratchet: <p>Hello, I am attempting classification loss results. The examples in the documentation only show only random value inputs so it’s hard to gain perspective on the results I’m seeing. I would expect these results apart from probably the first nll_loss to be 0 since I am using the same tensor as an input expect encoded as one hot, yet I’m seeing 0.5514 for what should be 0 loss?</p>
<p>I am probably missing something obvious but in the past I’ve only dealt with regression tasks and these classification result just seem wrong? I’ve been trying to look this up but from all searches this should work as I expect yet it does not?</p>
<pre><code class="lang-auto">a = torch.tensor([0, 1, 2])
b = F.one_hot(a).float()

F.cross_entropy(b, a)            # returns 0.5514
F.nll_loss(b, a)                 # returns -1
F.nll_loss(F.log_softmax(b), a)  # returns 0.5514
nn.BCELoss()(b, b)               # returns 0
</code></pre>

ptrblck: <p><code>F.cross_entropy</code> expects logits as the inputs and will apply <code>F.log_softmax</code> and <code>F.nll_loss</code> internally on <code>b</code> which is why line1 and line3 are giving equal results.<br>
<code>F.nll_loss</code> expects log probabilities and gives an invalid loss value since you are passing one-hot encoded inputs to it.<br>
<code>nn.BCELoss</code> expects probabilities for both inputs and thus returns a zero loss.</p>

ratchet: <p>Yes, thank you. After some time I realized that onehot is a probability and to convert it to logit I need to apply a soft version of log(x / (1 - x)) which gave the close to zero results. I’ll changed the title to solved.</p>
Native Ditributed Data Parallel with Mixed Precision

mostafaelaraby: <p>I was wondering if gradients are scaled from fp16 to fp32  before all reducing as in Apex Distributed data parallel default flag <strong>allreduce_always_fp32</strong> .</p>
<p>Also what is the equivalent of <strong>delay_allreduce</strong> in torch.nn.distributed.DistributedDataParallel ?</p>

ptrblck: <p>I wouldn’t assume to see any automatic transformation of gradients if no ddp hooks etc. are used.<br>
The native mixed-precision training util. via <code>torch.cuda.amp</code> uses <code>FP32</code> parameters and thus also FP32 gradients.</p>
<p>You could try to increase the <code>bucket_cap_mb</code> to kick off the gradient sync later, if needed (unsure if there is a cleaner way).</p>
C++ indexing error no matching function for call to ‘at::Tensor::index(at::Tensor&)’

justanhduc: <p>Hi. I am trying to index a tensor using the following syntax</p>
<pre><code class="lang-auto">tensor = tensor.index(booleanIndexTensor);
</code></pre>
<p>This works before for some previous version of Torch c++. However, when I try with Pytorch 1.8, it throws the following error</p>
<pre><code class="lang-auto">error: no matching function for call to ‘at::Tensor::index(at::Tensor&amp;)’
</code></pre>
<p>It seems like the API has changed, but I couldn’t find the new syntax anywhere else. Could you please let me know how I can adapt it for newer versions. Thanks a lot!</p>

justanhduc: <p>FYI, the last version I can compile is 1.7.</p>

tom: <p>As far as I can tell, we have <code>tensor[index]</code> and <code>tensor.index({index})</code> as indexing with a single tensor in PyTorch.<br>
Edit: In the <a href="https://github.com/pytorch/pytorch/releases/tag/v1.8.0" rel="noopener nofollow ugc">1.8.0 release notes</a>, your case is in the backward compatibility notes here (from PR <a href="https://github.com/pytorch/pytorch/pull/49138" class="inline-onebox" rel="noopener nofollow ugc">Making ops c10-full: list of optional tensors by smessmer · Pull Request #49138 · pytorch/pytorch · GitHub</a> ) and will need to change into <code>tensor.index({index})</code>.</p>
<p>Best regards</p>
<p>Thomas</p>

justanhduc: <p>Thank you very much <a class="mention" href="/u/tom">@tom</a>. Indeed, it compiled fine when I changed to <code>tensor.index({index})</code>.</p>
<p>I copied the related note from the link you sent here, just for anyone having the same problem.</p>
<blockquote>
<p>This will break backwards compatibility for the C++ API. There is no implicit conversion from ArrayRef (which was the old argument type) to List&lt;optional&gt;. One common call pattern is tensor.index({indices_tensor}), where indices_tensor is another Tensor, and that will continue working because the {} initializer_list constructor for List&lt;optional&gt; can take Tensor elements that are implicitly converted to optional, but another common call pattern was tensor.index(indices_tensor), where previously, the Tensor got implicitly converted to an ArrayRef, and to implicitly convert Tensor → optional → List&lt;optional&gt; would be two implicit conversions. C++ doesn’t allow chaining. two implicit conversions. So those call sites have to be rewritten to tensor.index({indices_tensor}).</p>
</blockquote>
Layer_norm sync during distributed training

Ardeal: <p>Hi,</p>
<p>my understanding is:<br>
batchnorm could be synced during distribued training by API <code>torch.nn.SyncBatchNorm.convert_sync_batchnorm()</code>.<br>
Could layer_norm be synced during distributed training by API <code>torch.nn.SyncBatchNorm.convert_sync_batchnorm()</code>?</p>
<pre><code class="lang-python">model_sync = torch.nn.SyncBatchNorm.convert_sync_batchnorm(model)
</code></pre>

tom: <p>The reason to sync BatchNorm is because it collects statistics across samples (i.e. elements of a minibatch) which will be on different GPUs.<br>
LayerNorm does not merge statistics between elements of a minibatch but only computes statistics within a sample, which will be on a given GPU. So there is nothing to sync.</p>
<p>Best regards</p>
<p>Thomas</p>

Ardeal: <p><a class="mention" href="/u/tom">@tom</a> ,<br>
LayerNorm does not collect statistics, so we don’t need to sync LayerNorm, right?</p>
<p>if we sync LayerNorm, what will happen?</p>

tom: <p>Sorry, the initial answer was off, so I edited it. Thank you for pointing out that the first wasn’t a working explanation.</p>

Ardeal: <p><a class="mention" href="/u/tom">@tom</a> ,<br>
Thank you!<br>
we don’t need to sync LayerNorm on different GPUs.</p>
How can I cache elements of a element-wise vectorised function?

tomroth1001: <p>I have a function <code>fn</code> that takes a number <code>a</code> and returns a new number <code>b</code>. I have vectorised it into a second function <code>some_fn</code> that takes as input a tensor, performs <code>fn</code> element-wise on each element in the tensor, and returns a tensor of the same size.</p>
<p>As an example:</p>
<pre><code class="lang-auto">def some_fn(x): 
    # some vectorised element-wise transformation on the tensor `x` here that takes a while
    # and returns a tensor `result` of the same size. Each element of `x` is independent of 
    # each other. 
    return result 

tensor1 = torch.tensor([1,2,3,4,5])
tensor2 = torch.tensor([3,4,5,6,7])

result1 = some_fn(tensor1)  
result2 = some_fn(tensor2)

print(result1)  # e.g. 15, 23, 39, 42, 54
print(result2)  # e.g. 39, 42, 54, 60, 77 
</code></pre>
<p>In the above example there is some overlap in <code>tensor1</code> and <code>tensor2</code> of [3,4,5]. These particular numbers will give the same result each time when passed through <code>some_fn</code> and I would like to cache these results so they don’t have to be calculated again (because they are expensive to calculate).</p>
<p>In addition, they may be stored on the GPU (although I’m not sure if that makes a difference or not).</p>
<p>Any help appreciated.</p>

tom: <p>Well, if the indices are “small numbers”, you could make a caching tensor.<br>
But if your code is parallelized and you need to compute some indices anyways, the speed gains might be limited and it’ll be a lot of bookkeeping to get this right.</p>
<p>Best regards</p>
<p>Thomas</p>
How to clip the gradient?

ADONAI_TZEVAOT: <p>How to clip the gradient in a model during training alone ? Please help</p>

tom: <p>Gradient clipping is always only done in training (because you ordinarily don’t do backprop  in evaluation).<br>
There are two ways:</p>
<ul>
<li>backpropagate, then clip gradients using the utility functions,</li>
<li>add backward hooks doing the clipping in flight.</li>
</ul>
<p>You might see this somewhat old thread for a discussion of the details:</p>
<aside class="quote quote-modified" data-post="16" data-topic="191">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tom/48/3162_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/proper-way-to-do-gradient-clipping/191/16">Proper way to do gradient clipping?</a> 
  </div>
  <blockquote>
    Note that clip_grad_norm_ modifies the gradient after the entire backpropagation has taken place. In the RNN context it is common to restrict the gradient that is being backpropagated during the calculation. This is described e.g. in <a href="https://arxiv.org/abs/1308.0850" rel="noopener nofollow ugc">Alex Graves’ famous RNN paper</a>. 
To do the latter, you typically use register_hook on the inputs or outputs of certain operations, e.g. with lambda x: x.clamp(-10,10) to do element-wise clipping. 
For a practical example, you could search for register_hook in <a href="https://github.com/t-vi/pytorch-tvmisc/blob/master/misc/graves_handwriting_generation.ipynb" rel="noopener nofollow ugc">my Grav…</a>
  </blockquote>
</aside>

<p>Best regards</p>
<p>Thomas</p>
Cuda isn't Available

julianolm: <p>Hello. I am using a machine on cloud with two Nvidia Tesla A100 and have alredy installed pytorch, but it seems to still does not recognize cuda afterall.</p>
<p>I have installed the latest pytorch version with</p>
<p><code>pip3 install torch==1.10.0+cu113 torchvision==0.11.1+cu113 torchaudio==0.10.0+cu113 -f https://download.pytorch.org/whl/cu113/torch_stable.html</code></p>
<p>Importing torch and asking for torch.version.cuda I get ‘11.3’.</p>
<p>Since it is a system with multiple users, I gess paths may be different from default, but what procedure should I take to resolve this?</p>
<p>System has the following software installed</p>
<p>CUDA Toolkit 11.4<br>
nvhpc-21.7 - NVIDIA HPC SDK<br>
NVIDIA cuDNN</p>

ptrblck: <p>To use the pip wheels you would only need to install an appropriate NVIDIA driver on your workstation so I guess that your driver installation is either not working or your environment is not supporting GPUs (e.g. if you are launching a docker container without the <code>--gpus all</code> flag).</p>
Confused by torch.roll

frkmac1: <p>Could anyone tell me how the “torch.roll” works in backward process? I was trying to check how gradients changes in backward process, but it seems that there is no gradient information for the layer of “torch.roll”.</p>

ptrblck: <p>The backward operation of <code>roll</code> would just undo the <code>roll</code> operation on the tensor:</p>
<pre><code class="lang-python">x = torch.randn(10, requires_grad=True)

y = x * torch.arange(10)
y.mean().backward()
print(x.grad)
# &gt; tensor([0.0000, 0.1000, 0.2000, 0.3000, 0.4000, 0.5000, 0.6000, 0.7000, 0.8000, 0.9000])

x.grad = None
y = x.roll(1) * torch.arange(10)
y.mean().backward()
print(x.grad)
# &gt; tensor([0.1000, 0.2000, 0.3000, 0.4000, 0.5000, 0.6000, 0.7000, 0.8000, 0.9000, 0.0000])
</code></pre>
Tensor dynamic range

Honnang: <p>usually a int8 tensor ranges from 0 to 255<br>
but in float, the value range is not fixed but it’s dynamic<br>
meaning that the levels between the min and max values are what determines floating point</p>
<p>can I make a fixed range between eg. 0 and 1<br>
so that it is not dynamic</p>

Eta_C: <p>If you want to generate a fixed range tensor</p>
<pre><code class="lang-python">size = [1, 2, 3]
x = torch.rand(size)
x = torch.randint(0, 255, size) / 255
</code></pre>
<p>If you have a tensor</p>
<pre><code class="lang-python">x = torch.clamp(x, 0, 1)
</code></pre>
Solution for slow backward propagation

yx-chan131: <p>I’m trying to train a StyleGAN2 model using <a href="https://github.com/xinntao/BasicSR" rel="noopener nofollow ugc">BasicSR library</a>.<br>
My setup is torch1.8 with 4 NVIDIA GeForce 3090.</p>
<p>For every iteration it takes about 0.5s (including data loading, forward propagation and backward propagation). However, when comes to regularization for generator which happens every 4 iterations, the backward propagation becomes extremely slow. It takes average 25s to do the backward propagation.</p>
<p>Below is the code for generator regularization:</p>
<pre><code class="lang-auto">if current_iter % self.net_g_reg_every == 0:
            path_batch_size = max(1, batch // self.opt['train']['path_batch_shrink'])
            noise = self.mixing_noise(path_batch_size, self.mixing_prob)
            fake_img, latents = self.net_g(noise, return_latents=True)
            l_g_path, path_lengths, self.mean_path_length = g_path_regularize(fake_img, latents, self.mean_path_length)

            l_g_path = (self.path_reg_weight * self.net_g_reg_every * l_g_path + 0 * fake_img[0, 0, 0, 0])
            # TODO:  why do we need to add 0 * fake_img[0, 0, 0, 0]
            l_g_path.backward()
            loss_dict['l_g_path'] = l_g_path.detach().mean()
            loss_dict['path_length'] = path_lengths
</code></pre>
<p>The loss function “l_g_path”:</p>
<pre><code class="lang-auto">def g_path_regularize(fake_img, latents, mean_path_length, decay=0.01):
    noise = torch.randn_like(fake_img) / math.sqrt(fake_img.shape[2] * fake_img.shape[3])
    grad = autograd.grad(outputs=(fake_img * noise).sum(), inputs=latents, create_graph=True)[0]
    path_lengths = torch.sqrt(grad.pow(2).sum(2).mean(1))

    path_mean = mean_path_length + decay * (path_lengths.mean() - mean_path_length)

    path_penalty = (path_lengths - path_mean).pow(2).mean()

    return path_penalty, path_lengths.detach().mean(), path_mean.detach()
</code></pre>
Reshape with padding

stsouko: <p>How to reshape 2d array with size [(A+B+C+…n),M] to 3d array with shape [N,X,M]<br>
where X is max(A, B, C, …n) which contains A or B or C elements with padding.</p>
<p>[[1, 2],<br>
[3, 4],<br>
[5, 6],<br>
[7, 8]]</p>
<p>A = 2, B = C = 1</p>
<p>to</p>
<p>[[[1, 2],<br>
[3, 4]],<br>
[[5, 6],<br>
[0, 0]],<br>
[[7, 8],<br>
[0, 0]]]</p>
<p>gradients should be keeped.</p>
Pre-built segmentation model training is slow

KKKcat: <p>Hi,</p>
<p>I’m training a segmentation model, basically <code>torchvision.models.segmentation.deeplabv3_resnet50</code><br>
It takes ~45 minutes for one epoch. I feel this is too slow. I hope someone can help me find redundancy/mistakes (if any) in my codes. Or any suggestions on how to speed up each epoch are appreciated.</p>
<ul>
<li>Image shape: 512x512</li>
<li>Training size: 20k</li>
<li>Batch size: 8</li>
<li>GPU: NVIDIA Tesla V100</li>
<li>Memory-Usage: 11805MiB / 16160MiB</li>
<li>GPU-Util: 85%-100%</li>
</ul>
<p>My codes almost followed this <a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html#training-the-model" rel="noopener nofollow ugc">tutorial</a></p>
<pre><code class="lang-auto">
model = models.segmentation.deeplabv3_resnet50(pretrained=1, progress=0, num_classes=10)
model.to(device)
optimizer = optim.Adam(model.parameters(), lr=0.0001)
lr_scheduler = optim.lr_scheduler.ExponentialLR(optimizer=optimizer, gamma=0.8)

for epoch in range(num_epochs):

    for phase in ['train', 'test']:
        if phase == 'train':
            model.train()
        else:
            model.eval()

        for batch in dataloader_dict[phase]:
                
            image, label = batch['image'].to(device), batch['label'].to(device)
                
            optimizer.zero_grad()
                
            # forward pass
            with torch.set_grad_enabled(phase == 'train'): 

                output = model(image)['out']  
                loss = criterion(output, label)
                prediction = output.argmax(dim=1)  
                    
                if phase == 'train':
                    loss.backward()
                    optimizer.step()

        if phase == 'train':
            lr_scheduler.step()
</code></pre>
Model outputs a single prediction instead of predictions for whole batch

Oliver: <p>I defined a simple convolutional model to try something.<br>
Unfortunately, I can’t figure out why the following only outputs a single prediction instead of a vector of predictions with size batch_size.</p>
<pre><code class="lang-auto">class ShapeNet(LightningModule):

  def __init__(self, in_channels=1, hidden_dim=16, n_classes=3, lr = 0.0001):

    self.lr = lr

    super(ShapeNet, self).__init__()

    self.convolution = nn.Sequential(OrderedDict([

        ("conv1", self.conv_block(in_channels, hidden_dim)),

        ("conv2",self.conv_block(hidden_dim, hidden_dim * 2)),

        ("final_conv",self.conv_block(hidden_dim * 2, 1, final_layer=True)),

    ]))

    self.fc = nn.Sequential(OrderedDict([

        ("fc", nn.Linear(in_features = 3136, out_features = n_classes)),

        ("softmax", nn.Sigmoid())                        

    ]))

   

  def conv_block(self, in_channels, out_channels, kernel_size = 3, stride = 2, final_layer=False):

    if not final_layer:

            return nn.Sequential(OrderedDict([

                ("Conv2d", nn.Conv2d(in_channels, out_channels, kernel_size, stride)),

                ("BatchNorm", nn.BatchNorm2d(out_channels)),

                ("MaxPool", nn.MaxPool2d(kernel_size=(2, 2), stride=(2, 2))),

                ("LeakyRelu", nn.LeakyReLU(0.2, inplace=True))

            ])

            )

    else: 

        return nn.Sequential(OrderedDict([

            ("final", nn.Conv2d(in_channels, out_channels, kernel_size, stride))

        ])

        )

  def forward(self, x):

    # pass the input through our first set of CONV =&gt; RELU =&gt;

    # POOL layers

    x = self.convolution(x.float())

    x = flatten(x)

    x = self.fc(x)

    return x
</code></pre>
<p>Would someone be able to tell me where I’m going wrong here?</p>
<p>when I call model(x) with x being of shape(batch_size, 1, width, height)<br>
I’d expect a vector of predictions of shape (batch_size, n_classes).<br>
But I only get (n_classes)</p>
<p>thanks in advance!</p>

Oliver: <p>I figured it out, coming from tensorflow I had thought that flatten will keep batch_size, but I needed to pass the argument start_dim = 1.</p>
Training a model to distinguish licenses from non-licenses

mephamah: <p>Hello folks!</p>
<p>I am building an app that requires users to submit a picture of a license as part of their registration.  This is actually a form of racing license, not a drivers license, but they appear somewhat similar.  Eventually I’m hoping to train a model that identifies what state the license is from, but I don’t yet have nearly enough images of these specific licenses to do so.  As a first attempt, I instead am planning to train a model that simply distinguishes license-like images from other images.  I am familiar with the theory underpinning machine learning, and am reasonably comfortable with the syntax of pytorch.  However, there are some more practical questions that I don’t have answers to:</p>
<ol>
<li>What would be a good model architecture for this</li>
<li>Will transfer learning be helpful or should I train a model from scratch</li>
<li>Roughly how many images of license-like and not-license-like things should I gather</li>
<li>How much will image augmentation help reduce the required number of images</li>
<li>What ratio of license and not-license should be in the training data</li>
<li>What should not-license images look like? I know that this depends on what phony images users are going to submit, but should I restrict not-licenses to be document-like images, or include images of anything and everything?</li>
</ol>
<p>I greatly appreciate any help on this, it is going to be a fun project!</p>
How can I load the same number of data/images per class from the pascal dataset for the segmentation task?

joey.s: <p>I was wondering if there is a way to load a specific number of data per class for a multi-classes segmentation task?</p>
<p>can I use BatchSampler in this regard? if so, can you help me with an example? I have found these two topics <a href="https://discuss.pytorch.org/t/how-to-enable-the-dataloader-to-sample-from-each-class-with-equal-probability/911/2">here</a> and <a href="https://discuss.pytorch.org/t/load-the-same-number-of-data-per-class/65198/2">here</a> but I couldn’t implement them for multi-class labeling. Thanks in advance!</p>
Version 1.10 don't install torch python files

Belisko_Marek: <p>I’m adding pytorch support to yocto and I can compile it using cmake with cuda and python option set to on. When compiled in deploy directory I cannot see python files from torch subdirectory installed. Any ideas? I debug it a little and in caffee2 is added subdirectory for torch but it build only code and install and not python bindings. Thanks.</p>
Vectorized autograd.grad for class activation maps

vadimkantorov: <p>I have a neural model <code>f</code> that produces a score per class. I’d like to compute something like Class Activation Maps by computing gradient of each output wrt input (and I don’t need to compute any gradients with respect to weights of <code>f</code>.</p>
<p>Is there a batched way to compute gradients wrt to every output value? If found <a href="https://pytorch.org/docs/stable/generated/torch.autograd.functional.jacobian.html#torch.autograd.functional.jacobian" class="inline-onebox" rel="noopener nofollow ugc">torch.autograd.functional.jacobian — PyTorch 1.10.0 documentation</a>, but it seems that it will do forward many times as well. Is it true? Does calling it like below make sense?</p>
<pre><code class="lang-python">import torch
import torch.nn as nn

model = nn.Linear(10, 20).requires_grad_(False)

x = torch.zeros(5, 10).requires_grad_(True)

y = model(x)

print(x.shape)
# torch.Size([5, 10])

print(y.shape)
# torch.Size([5, 20])

grads = torch.autograd.functional.jacobian(lambda x: y, x, vectorize = True)
print(grads.shape)
# torch.Size([5, 20, 5, 10])
</code></pre>

vadimkantorov: <p>Why does the <code>grads.shape</code> above contains batch dimension twice? Ideally, I’d like to get something like <code>[5, 10, 20]</code> where 5 is the batch dimension.</p>
<p>I understand it may make sense when the outputs depend on other elements of the batch (e.g. with a batchnorm), but when I don’t need this dependence, is there a way to break it? Should I sum over one of these batch dimensions?</p>
<p><a class="mention" href="/u/alband">@albanD</a></p>

AlphaBetaGamma96: <p>This is because <code>torch.autograd.functional.jacobian</code> differentiates with respect to the whole Tensor, i.e. it includes the batch dim as a differentiable dimension of the input array (hence why you get the batch dim twice). You can then just take the diagonal over the pair of batch dimensions to get the per-sample grads you want! This can be done via <code>torch.einsum</code></p>
<pre><code class="lang-auto">grads = torch.einsum("bibj-&gt;bij",grads)
</code></pre>

albanD: <blockquote>
<p>Should I sum over one of these batch dimensions?</p>
</blockquote>
<p>Not really. If you function is actually independent across batches, then all the non “diagonal” elements will be 0s.<br>
But if you have batchnorm or something like that, they won’t be 0 and you may not want to use them?</p>
<p>If you’re happy with summing, then you can get that result faster by expanding your weights to have a batch size and compute the gradients wrt to these expanded weights.</p>