akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Should create_graph always be True when accumulating gradients in autograd.grad()

Atom-101: <p>I am trying to modify the following implementation of Meta-SGD training to support minibatches through gradient accumulation:</p>
<pre><code class="lang-auto">    Y_pred = model(X)
    loss = loss_fn(Y_pred, Y)
    model.zero_grad()

    grads = torch.autograd.grad(loss, model.parameters(), create_graph=True)

    adapted_state_dict = model.cloned_state_dict()
    adapted_params = OrderedDict()
    for (key, val), grad in zip(model.named_parameters(), grads):
        task_lr = model.task_lr[key]
        adapted_params[key] = val - task_lr * grad
        adapted_state_dict[key] = adapted_params[key]

    return adapted_state_dict
</code></pre>
<p>Every parameter in the model has its own learning rate, and this learning rate needs to be tuned, so a gradient graph is required. To simulate a larger batch size using gradient accumulation, I used the following code, but I think it is wrong because the memory requirements are extremely high:</p>
<pre><code class="lang-auto">for xb,yb in data.train_dl:
            ypred = model(xb)
            loss = loss_fn(ypred,yb)
            if not grads:    
                grads = torch.autograd.grad(loss, model.parameters(), create_graph=True)
            else:
                grads = [grads[i] + torch.autograd.grad(loss, model.parameters(), create_graph=True)[i] for i in range(len(grads))]
</code></pre>
<p>I think this code is creating a separate gradient graph for every minibatch. If I pass create_graph=True for the first call to autograd.grad and only retain_graph=True for the subsequent calls, will it work or is there a better way?</p>
<p>Also, the behaviour of loss.backward() and autograd.grad seems to be different wrt to retain_graph. The backward() function can be called multiple times even without retain_graph=True, but autograd.grad can only be called once. For example, this code works when executed multiple times:</p>
<pre><code class="lang-auto">outputs = model(inputs)
loss = loss_fn(outputs,labels)
loss.backward()
</code></pre>
<p>But this code works only once:</p>
<pre><code class="lang-auto">outputs = model(inputs)
loss = loss_fn(outputs,labels)
grads = torch.autograd.grad(loss,model.parameters())
</code></pre>
<p>backward() is also much faster than autograd.grad()</p>

Hobo92: <p>Hi, I also met the same issue. Setting create_graph=True is usually used for calculating second derivative. I found that if I set create_graph=True for each batch, the memory usage will soon expire. This indicates that this will create separate graphs for each batch. If you want to prevent this and only set create_graph=True for the first batch but set retrain_graph=True for the following batches (in this way, create_graph=False), the second derivative will be wrong because it only calculates the second derivative using the first batch of data.</p>
Weird artifacts with Super Resolution

Greatmiralis: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/4/44dbcc66fc9b29396a525acaba0ac03714032c8f.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/44dbcc66fc9b29396a525acaba0ac03714032c8f" title="gen62"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/4/44dbcc66fc9b29396a525acaba0ac03714032c8f_2_500x500.jpeg" alt="gen62" data-base62-sha1="9P9nO3CvxW1TPa0oNIVXKvl77Gf" width="500" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/4/44dbcc66fc9b29396a525acaba0ac03714032c8f_2_500x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/4/4/44dbcc66fc9b29396a525acaba0ac03714032c8f_2_750x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/4/44dbcc66fc9b29396a525acaba0ac03714032c8f_2_1000x1000.jpeg 2x" data-dominant-color="A29F98"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">gen62</span><span class="informations">1000×1000 138 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I have made a Super Resolution model and it is learning pretty well but for some reason these artifacts appear. It seems like some pixels that should be white are mostly yellow and some pixels that should be black or dark are mostly blue. Does someone know what can cause these artifacts to appear?</p>

Grumgog: <p>It little bit late for answert, but for next seeker of solution for same problem. Today similar problem appear for me too. After investigation I found that model’s output tensor is not clamped for range <code>[0; 1]</code> some values have little bit more or less values out of that range. I use PIL for tranforming input/output images. Obviously PIL (and<code> torchvision.transforms.ToPILImage()</code>) demands clamped values of tensor. In that case you can use <code>torch.clamp(tensor, 0, 1)</code> for clamping tensor.</p>
While using FSDP, how to unshard single modules parameters outside forward/backward?

Ilias_Dahi: <p>Hi,</p>
<p>When wrapping a model like:</p>
<pre><code class="lang-auto">fsdp_model = FullyShardedDataParallel(
   model(),
   fsdp_auto_wrap_policy=default_auto_wrap_policy,
   cpu_offload=CPUOffload(offload_params=True),
)
</code></pre>
<p>Using summon_full_params(model) will unshard all parameters for all wrapped modules which will result in the full model in each RANK, causing OOM in case of a large model. I know that I can wrap modules individually like this:</p>
<pre><code class="lang-auto">class model(nn.Module):
   def __init__(self):
       super().__init__()
       self.layer1 = wrap(nn.Linear(8, 4))
       self.layer2 = nn.Linear(4, 16)
       self.layer3 = wrap(nn.Linear(16, 4))
 
wrapper_kwargs = Dict(cpu_offload=CPUOffload(offload_params=True))
with enable_wrap(wrapper_cls=FullyShardedDataParallel, **wrapper_kwargs):
   fsdp_model = wrap(model())
</code></pre>
<p>But still, the model will be the root_module, and <code>summon_full_params()</code> expects the root_module to be able to unshard the parameters.</p>
<p>My question is,  is it correct, safe and optimal to do:</p>
<pre><code class="lang-auto">with enable_wrap(wrapper_cls=FullyShardedDataParallel, **wrapper_kwargs):
   fsdp_layer1 = wrap(model.layer1)

with enable_wrap(wrapper_cls=FullyShardedDataParallel, **wrapper_kwargs):
   fsdp_layer3 = wrap(model.layer3)
</code></pre>
<p>Then apply summon_full_params on those individual modules?<br>
Thanks!</p>
Torch.load, should I just always use mmap?

AlbertZeyer: <p>I don’t exactly understand what would be a good reason <em>not</em> to use <code>mmap</code>. It rather sounds to me that I just always should set this to True?</p>

JuanFMontesinos: <p>mmap loads the information from disk when you slice the tensor (or you apply it in a relevant operation)<br>
It’s good or bad depending on the context.</p>
<p>Good if bounded by memory (you can lazily initiallize them). Obviously, you cannot overwrite its values.</p>
<p>Bad if you have memory to host the whole data you need.</p>

AlbertZeyer: <aside class="quote no-group" data-username="JuanFMontesinos" data-post="2" data-topic="191305">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/juanfmontesinos/48/76115_2.png" class="avatar"> JuanFMontesinos:</div>
<blockquote>
<p>Obviously, you cannot overwrite its values.</p>
</blockquote>
</aside>
<p>You mean, I cannot use <code>load_state_dict(..., assign=True)</code> and train that model afterwards on CPU, as the values would be readonly this way? I did not consider this.</p>
<p>With the default (<code>assign=False</code>), this is not a problem though, right?</p>
<p>And also, if they are then to be moved to GPU anyway (<code>model.to("cuda")</code>), this also would not be a problem?</p>
<p>And also, if I did <code>torch.load(..., map_location="cuda", mmap=True)</code>, this would also not apply?</p>
<aside class="quote no-group" data-username="JuanFMontesinos" data-post="2" data-topic="191305">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/juanfmontesinos/48/76115_2.png" class="avatar"> JuanFMontesinos:</div>
<blockquote>
<p>Bad if you have memory to host the whole data you need.</p>
</blockquote>
</aside>
<p>Why? Mmap would also load them into memory once they are needed.</p>
Applying Hyperparameters to complex networks

souvikg544: <p>I have a network that has the following components -</p>
<ol>
<li>1 3d convolutional Block</li>
<li>Resnet Like architecture</li>
<li>Dense Temporal Convolutional Networks.<br>
I want to implement Hypernwtwork in this type of a network.<br>
Will be helpful if I can receive guidance from the amazing community here.</li>
</ol>
About supporting devices and processors

Earth-M: <p>Hi,</p>
<p>I’m wondering about what devices and processors ExecuTorch supports.</p>
<ol>
<li>Can we run ExecuTorch on Nvidia Jetson devices like Jetson Xavier NX?</li>
<li>Also can ExecuTorch use Xavier NX’s GPU?</li>
<li>Is there any list of supporting devices and processors? It’s nice to have an upcoming list too.</li>
<li>Is there any benchmark list?</li>
</ol>
<p>Thanks</p>
Asynchronous loading for data

y_h: <p>Dear Community,</p>
<p>There is a CPU-intensive data preparation task in the customized dataloader and I don’t want this to block the training procedure. The natural way to do this is to introduce a buffer in the background so that workers can get the data from the buffer and retrieve them on time. It seems that the build-in dataloader multithreading doesn’t hold this task very well, as one worker running out of buffer will block other workers from feeding data  (<a href="https://discuss.pytorch.org/t/in-what-order-do-dataloader-workers-do-their-job/88288">reference</a>). Is there a good way to do this?</p>
Torch_geometric Dataset

SKTheLearner: <p>Hi</p>
<p>I am experimenting with GNN and am sort of stuck at creating a dataset/loader with torch_geometric for training in mini batches. Domain is molecules/drugs…</p>
<p>Here is where am at…</p>
<ol>
<li>I have created a python list of<br>
a. Nodes(a list of arrays of various shapes)  -    train_N … eg…(shapes.).[(42,10),(42,10),(65,10)…]<br>
b. and a list of corresponding adjacent matrices - train_adj…eg…[(42,42),(42,42)…(65,65)…]<br>
c. y values shape (200,5000)</li>
</ol>
<p>How do I convert all these into a torch_geometric DataLoader and use it in my model training in batches ?</p>
<p>How do I</p>
Demonstrating Latency-boundedness of an operation

taehyunzzz: <p>Let’s say that I have a linear layer = nn.Linear(1024,4096).<br>
If the input data for this layer is skinny, i.e. for example, a torch.rand((1,1024)) tensor,<br>
one may predict that the operation would be latency-bound rather than memory-bound,<br>
as the operation time itself will be short, and the runtime will mainly be used on reading<br>
data from memory rather than computation.<br>
In this case, what sets apart the memory-boundness and latency-boundness is the<br>
length of total runtime excluding the head/tail memory read/write times.</p>
<p>But I am curious if there is a quantative way of demonstrating the latency-boundness<br>
if Pytorch operations. Any suggestions?</p>

tom: <p>In general, this is hard because measuring these things can be very delicate and performance characteristics of operations can be quite intricate (see e.g. this account of memory alignment impact <a href="https://nitter.net/karpathy/status/1621629552333328384" class="inline-onebox" rel="noopener nofollow ugc">Andrej Karpathy (@karpathy): "the latency of the entire training loop, the whole network. yes it's that bad." | nitter</a> )<br>
That said, if we imagine to latency (kernel launch overhead etc.) to be fixed (independent of size) and memory to be (at least) proportional to input size, you would expect that, say, doubling the size does not double the runtime. The other part is that you essentially know how much time it takes to launch a kernel and sync, so that might be the latency(?).</p>
<p>Best regards</p>
<p>Thomas</p>
Hai every one, how to deploy a PyTorch model in playstore?

Teddy_Salas: <p>Respected every one , i wanted know how to deploy a CNN model in android play store , can any one please help?</p>

ptrblck: <p>You could check the <a href="https://pytorch.org/mobile/ios/">legacy workflow </a> or the new <a href="https://pytorch.org/executorch/stable/demo-apps-ios.html">ExecuTorch workflow </a> (which is still in development if I’m not mistaken) to create your application and could then try to publish it as any other application.</p>
ModuleNotFoundError: No module named 'torch._C._distributed_c10d'; 'torch._C' is not a package

Haitao_Xiao: <p>1.module ‘torch.distributed’ has no attribute ‘init_process_group’<br>
I am trying to source compile pytorch on macos, I have installed libuv and pkg-config but I am getting the above 2 errors, is there a solution?</p>
Training on edge device: iPhone

Neuperc: <p>Hi,</p>
<p>I need to fine-tune a model on the iPhone after deployment (in an online manner). However, it seems like both Torch Mobile, and the recently released ExecuTorch are for inference purposes only. My questions are:</p>
<p>1- Is it possible to train a model on edge device (e.g. with LibTorch Lite)?<br>
2- If yes, what is the most straightforward way to do it?</p>
<p>Thank you!</p>

Jin_Pyo_Jun: <p>I recently succeeded in building android pytorch to enable learning. There should be a USE_DISTRIBUTE setting, but it was forcibly turned off in case of mobile. That part needs to be corrected. In addition, many source modifications such as sleef and protobuf must be made. In the future, I plan to distribute it to GitHub using a shell script. The iPhone is expected to be similar.</p>
Vision transformer application for regression task

cbd: <p>I seen most of the vision transformer architecture for classification task. Why it is not used for the regression application where output is also image such as deblurring or image translation?<br>
If vision transformer is used in any such application, request you to share the link.</p>
<p>Just need to confirm like FCN, is it possible to use different image size for training and inference in ViT?</p>

tom: <p>I think there have been pretraining methods that output completions for masked parts of the input. Is that like what you have in mind? (I cannot find the reference I was looking for, but <a href="https://openreview.net/forum?id=y_OmkmCH9w" class="inline-onebox" rel="noopener nofollow ugc">MST: Masked Self-Supervised Transformer for Visual Representation | OpenReview</a> seems somewhat similar.)<br>
My impression is that it’s relatively resource-intense, but can be done.</p>
<p>I don’t think you can use varying image sizes with ViT easily as typically the patches are of fixed size and fixed number.</p>
<p>Best regards</p>
<p>Thomas</p>

cbd: <p>In my application, I need output as image. So I am bit confuse with the decoder part to generate the image from the output of the vision transformer encoder. In the mentioned reference paper, author used the CNN architecture in decoder part. My confusion is, Is there only CNN architecture in decoder part or some other architecture may help to regenerate desired output as image from vision transformer encoder output. Any reference may helpful.</p>

tom: <p>An obvious one might be imagegpt: <a href="https://github.com/openai/image-gpt" class="inline-onebox" rel="noopener nofollow ugc">GitHub - openai/image-gpt</a> , but in general, you if you have 16x16 pixel blocks and want c output channels, you could just make the last layer c * 16*16 and build your image from that by reshaping.</p>
<p>Best regards</p>
<p>Thomas</p>
Is the higher level library for meta-learning compatible with pytorch's distributed libraries?

Brando_Miranda: <p>I want to speed up my code with distributed training for MAML. I am using <a href="https://github.com/facebookresearch/higher" rel="noopener nofollow ugc">higher</a> and was wondering if their library even works with things like DDP, distirbuted rpc, distibuted autograd etc.</p>
<p>Note: this is being cross posted to increased visibility (no response in the gitissues after a week so perhaps someone in the forum knows <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> ):</p>
<ul>
<li><a href="https://github.com/facebookresearch/higher/issues/99" class="inline-onebox" rel="noopener nofollow ugc">Is the higher library compatible with pytorch's distributed RPC? · Issue #99 · facebookresearch/higher · GitHub</a></li>
<li><a href="https://github.com/facebookresearch/higher/issues/98" class="inline-onebox" rel="noopener nofollow ugc">Is higher compatibale with distributed data parallel DDP ? · Issue #98 · facebookresearch/higher · GitHub</a></li>
</ul>

Shivang_Patel1: <p>Any update on this? Thanks!</p>
Inference on iOS and GPU

cmarschner: <p>Hi,<br>
I’ve spent a couple of days to get MobileSAM (Segment Anything based on TinyVIT) working on iOS on GPU, but so far I’ve only managed to get a CPU version running, with a latency of 2 seconds.</p>
<p>What would be the options for speeding this up…?</p>
<ul>
<li>pytorch mobile with metal backend: pytorch lite doesn’t support a couple of operators I had to work around, but in the end the lack of support for strided convolution for the metal backend became a blocker.</li>
<li>pytorch MPS backend: There is no information about whether this even targets mobile</li>
<li>Apple CoreMLTools: Convert via ONNX - only a subset of operators supported</li>
<li>ONNX runtime?</li>
<li>Convert to TFLite via ONNX? I got some weird metadata there that made this a blocker (operator names where operator type names should be)</li>
<li>pytorch ExecuTorch: In development, earliest end of 2024</li>
</ul>
<p>It seems this has not been a priority for either Apple or Pytorch to make this work and support it end-to-end… Is there a way out?</p>
Can I build LibTorch with MinGW?

chacham15: <p>I’m trying to add a production module for inferencing. The problem is that all of our build machines are running in docker (without docker windows) and so in order to produce windows builds we use x86_64-w64-mingw32-g++ which wont link against MSVC. So, I’ve been wondering if there is a way to build LibTorch w/CPU only using mingw. My trial has yielded some problems with the build so far (i.e. std::mutex not being defined) which I can try and figure out a solution for, but I wanted to know before I sink a lot of time into this, if people think that this approach is unlikely to yield results without an enormous amount of effort.</p>

FEMmaster: <p>My advice is to give up, minGW conflicts with Libtorch, because Libtorch is compiled by MSVC, unless you can find a version of Libtorch compiled by minGW, which I hear exists</p>
Select the inter-GPU interconnect

Epsilon_1: <p>Is there a way to select one of the two inter-GPU interconnects available (PCIe and Nvlinks) during model training? Can I do this programmatically? I know Nvlinks are faster than PCIe but I’d like to have the flexibility of selecting one or the other, hopefully on the fly.</p>

ptrblck: <p>I don’t believe this is possible and NVLinks will be used if available. You could disable p2p in e.g. NCCL, which should then communicate through the host.</p>
Torch compile and onnx.export

yoko: <p>is there any performance gain if I apply torch.compile before onnx.export?</p>
<pre><code class="lang-auto">model = torch.compile(model, **compile_cfg)
torch.onnx.export(
            model,
            input,
            "model.onnx",
            **onnx_cfg,
        )
</code></pre>
Device list for torch mps

APaul: <p>Is there an equivalent for <code>torch.cuda.device.count()</code> for mps. I only found that there is <code>torch.backends.mps.is_available()</code> and <code>torch.backends.mps.is_built()</code></p>
<p>I am trying to get running rllib on Apple M1 GPUs and I see that they do this check to retrieve the name of GPU devices <a href="https://github.com/ray-project/ray/blob/b67258a86dbdca329d6a2a6d0756233130594909/rllib/evaluation/rollout_worker.py#L499" rel="noopener nofollow ugc">https://github.com/ray-project/ray/blob/b67258a86dbdca329d6a2a6d0756233130594909/rllib/evaluation/rollout_worker.py#L499</a> and trying to find an alternative for it in mps</p>

dhruvbird: <p>I believe all the cores on M1/M2 Mac are exposed as a single <code>"mps"</code> device, so there’s no notion of a device count.</p>
MultiHeadAttention Attention Scores all the same after `model.eval()`

_Sandeep: <p>I have a relatively simple architecture for a graph neural net where I first process an input to a hidden dimension and then pass it and other inputs preprocessed in the same way into a MultiheadAttention layer. However, I noticed that at inference time all of my attention scores are equal, which isn’t the case during training.</p>
<p>Digging into this its cause by all the dot products between projected features being 0. Is there a reason this might be happening?</p>
<p>Edit:<br>
It seems like if I don’t call <code>model.eval()</code> the attention scores become more similar to what I see during training. Maybe this has to do with the fast path optimizations torch is doing?</p>
Clear all the GPU memory used by pytorch in current python code without exiting python

NagabhushanSN95: <p>I am running a modified version of a third-party code which uses pytorch and GPU. I run the same model multiple times by varying the configs, which I am doing within python i.e. I have a wrapper python file which calls the model with different configs. But I am getting <code>out-of-memory</code> errors while running the second or third model. That is to say, the model can run once properly without any memory issues. So, if I end the code after running the first model and then start the second model afresh, the code works fine. However, if I chain the models within python, I’m running into <code>out-of-memory</code> issues.</p>
<p>I suspect there are some memory leaks within the third-party code. On googling, I found two suggestions. One is to call <code>torch.cuda.empty_cache()</code>, and the other is to delete the tensors explicitly using <code>del tensor_name</code>. However, <code>empty_cache()</code> command isn’t helping free the entire memory, and the third-party code has too many tensors for me to delete all the tensors individually. Is there any way to clear the entire GPU memory used by the current python program within the python code itself?</p>
Does any one here owns an NLP project/piece/product?

Syed: <p>Does any one here owns an NLP project/piece/product?  like if they have (project/product) in:<br>
1.Research and Development<br>
2. Open-Source Projects<br>
3. Hackathons and Competitions<br>
4. Contributing to Datasets<br>
5. Developing Applications<br>
6. Contributing to NLP Communities in the form of articles.<br>
7. Educational Resources<br>
8. has to offer Consulting and Services etc…?<br>
Please help us out… we just launched a beta platform where one can showcase/list products/projects/algorithms for other to use or purchase.<br>
the resources it covers are Nlp, ML, Ai bots, robotics, Neural Network and many other categories.</p>
Is there a way to access a vectorized representation of all model parameters?

Stao: <p>Say I have a model, I can iterate through all the parameters using model.parameters(), but this is quite slow if I just want to do a consistent operation across all parameters, for example, adding a random noise to all parameters. If I have this vectorized representation of all the model parameters, I can just sample a random tensor and directly add to the representation.</p>

ptrblck: <p>You could pass the parameter list to <code>torch._foreach_...</code> methods, but note that these APIs are internal and their interface could change. Besides that you could also check e.g. <a href="https://github.com/PhilJd/contiguous_pytorch_params">this repository</a>, which doesn’t seem to be maintained anymore but might still be helpful as a template in case you want to use a custom implementation.</p>

Stao: <p>Thank you very much, will look into that</p>
Dataloading with shuffle=True very slow

Nikos_Ellinas: <p>I have a dataset with multiple .h5 files, I have seen all the pipeline tricks that should be used, i.e. open files in get_item and keep them open etc.<br>
Each .h5 file corresponds to one speaker, and I have made a custom sampler that shuffles the indices so that in each batch there are samples from different speakers.<br>
However, when I train with this sampler (note that the data are loaded from an HDD) the training is very slow, but when I remove the ‘sampler’ argument from the dataloader, so it performs essentially sequential sampling, the training is fast.<br>
Does this have to do with the data that are stored in consecutive positions in memory? I load each sample separately so the memory access is different in each sample. Why when doing shuffling the training goes slow?</p>

ptrblck: <aside class="quote no-group" data-username="Nikos_Ellinas" data-post="1" data-topic="191184">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nikos_ellinas/48/65209_2.png" class="avatar"> Nikos_Ellinas:</div>
<blockquote>
<p>Does this have to do with the data that are stored in consecutive positions in memory? I load each sample separately so the memory access is different in each sample. Why when doing shuffling the training goes slow?</p>
</blockquote>
</aside>
<p>Yes, this is expected behavior using an HDD as random accesses are more expensive due to its nature of using a spinning magnetic disk.</p>

Nikos_Ellinas: <p>I tried loading the same data when copied on an SSD and I noticed the speed is about the same as when I loaded all of the data in the RAM.<br>
Does this mean, that the dataloading pipeline has reached its full speed and then the final training speed depends only on the model?</p>

ptrblck: <p>This could be the case, in particular if you are using multiple workers which preload the next batches in the background while the model is being trained. You could check the data loading time as seen in <a href="https://github.com/pytorch/examples/blob/c67bbaba018d438447036c114133c6ad6c4f78b1/imagenet/main.py#L318">the ImageNet example</a>, which should converge towards 0 if the workers are fast enough to preload the next batches in time.</p>

Nikos_Ellinas: <p>Thanks I did not know I can check this, I will test it!</p>
GPUassert: no kernel image is available for execution on the device

David_YYD: <p>Hello, I’d like to use torch to run a package with a neural network on my notebook but meet a problem: no kernel image is available for execution on the device. Information about the system and GPU is described below.</p>
<p>System: window 10<br>
python version: 3.11.5<br>
GPU: NVIDIA GeForce MX250<br>
architecture: PASCAL GPU Architecture Compatibility<br>
compute capability: 6.1<br>
CUDA toolkit: 11.8<br>
torch version: 2.0.0+cu118<br>
torch.cuda.get_arch_list() :<br>
[‘sm_37’, ‘sm_50’, ‘sm_60’, ‘sm_61’, ‘sm_70’, ‘sm_75’, ‘sm_80’, ‘sm_86’, ‘sm_90’, ‘compute_37’]</p>
<p>when I run print(torch.zeros(1).cuda()), get the output: tensor([0.], device=‘cuda:0’)</p>
<p>error when I run the package of PyroNN:<br>
GPUassert: no kernel image is available for execution on the device C:\Users\sun\AppData\Local\Temp\build-via-sdist-igv1vm0r\pyronn-0.3.1\pyronn\ct_reconstruction\cores\kernels\cone_projector_3D_CudaKernel.cu 252</p>
<p>what should I do to address this problem?<br>
thanks very much!</p>

ptrblck: <p>As your output confirms PyTorch itself supports your GPU and is able to use it. However, <code>PyroNN</code> doesn’t support you GPU architecture and raises the error. I don’t know their support matrix but you might need to build <code>PyroNN</code> from source for your device.</p>

David_YYD: <p>OK, thanks for your suggestion!</p>
Why libtorch cannot inference after cudaDeviceReset

Assassintears: <p>Hi, I use Libtorch to deploy my project. Sometimes, I want to reload model to detect objects, then I call <code>cudaDeviceReset</code> to release resources. My code like this:</p>
<pre><code class="lang-cpp">// 
/****************
some preprocess
****************/
#region load model and detect
// load model
m_device = torch::Device(torch::kCUDA);
m_model = std::make_unique&lt;torch::jit::Module&gt;(torch::jit::load(modelFile));
m_model-&gt;to(m_device);


// read images with opencv
cv::Mat mat = cv::imread("xxxx.bmp");
cvtColor(mat , mat , cv::COLOR_GRAY2RGB);
// some other process


torch::Tensor inputs = torch::from_blob(mat.data,
				{ 1,imgRow,imgCol,3 }, torch::kByte);

			inputs = inputs.permute({ 0, 3, 1, 2 });
			inputs = inputs.toType(torch::kFloat32);
			inputs = inputs.div(255);
			// tensor to device(GPU)
			inputs = inputs.to(m_device);


			if (nullptr == m_model)
			{
				return -1;
			}
// this works well
auto out = m_model-&gt;forward({ inputs }).toTuple();

#endregion



#region release model
// release model
if (nullptr != m_model)
{
	m_model-&gt;to(kCPU);
	m_model.release();
	m_model = nullptr;

	c10::cuda::CUDACachingAllocator::CUDAAllocator* pCudaAllocator = c10::cuda::CUDACachingAllocator::get();

	if (nullptr != pCudaAllocator)
	{
		c10::cuda::CUDACachingAllocator::emptyCache();
	}
}
	cudaDeviceReset();
	cudaDeviceSynchronize();
#endregion

// after release model, recall the code `load model and detect`
</code></pre>
<p>Well, the above code shows the main procedure, after call the code above 3 times, the <code>m_model-&gt;forward({ inputs }).toTuple();</code> will throw an exception, if I donnot call <code>cudaDeviceReset(); 	cudaDeviceSynchronize();</code>, it will works well.</p>
<p>I want to know why calling <code>cudaDeviceReset</code> multitimes will raise an inference problem, please help me to resolve this problem, thank you very much.</p>

ptrblck: <p>Why do you want to reset the GPU and which error are you seeing? Are you also resetting libtorch and initialize it again afterwards?</p>