akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Stacked vs. eponymous torchrun cli options

RobS: <p>Docs here: <a href="https://pytorch.org/docs/stable/elastic/run.html#single-node-multi-worker" class="inline-onebox" rel="noopener nofollow ugc">torchrun (Elastic Launch) — PyTorch 2.0 documentation</a></p>
<p>In the Pytorch docs for torchrun, it lists two options for single-node multi-worker training: “Single-node multi-worker” and “Stacked single-node multi-worker”.</p>
<p>For me the “single-node multi-worker” did not work as intended but the “Stacked single-node multi-worker” training worked exactly as expected. “single-node multi-worker” training only processed on one GPU while the “stacked” version of the command engaged all available GPUs. What are the intended differences and use cases of these two options?</p>
Looking for a deep learning performance tips and tricks

dazzle-me: <p>Hi,<br>
I’m a happy owner of a 4090 and I’m about to train simple classification network on 2 million 224x224 images (vision-only), I’m using a fairly huge model and I want to speed-up my training setup.</p>
<p>I’ve been in deep learning for a while, but I still haven’t found any comprehensive guide on how to train models real fast, so here is what I know:</p>
<ol>
<li>use fp16 (GradScaler), 2x memory consumption reduction and some speed-up dependent on the architecture</li>
<li>how to user tensor cores? I have no idea if they’re being used at all, how to enable them, how to monitor them? do they become automatically enabled when I use fp16?</li>
<li>I’ve also tried torch.compile, it brought ~10% improvement in speed since pytorch doesn’t support flash attention for cards aside from A100/H100 as I understand (is it true though?)</li>
</ol>
<p>thats it really, also I have some assumptions on what might work,<br>
and here I want you to share your personal experience</p>
<ol>
<li>Is there any benefit using bfloat16 instead of float16?</li>
<li>Did you try using memory_format.channel_last?</li>
</ol>
<p>and maybe you know some other tricks and tips on how to speed-up training?<br>
worth to mention, that I don’t have any CPU bottleneck, thats for sure (I’m using Ryzen 9 5950x with Samsung 980 pro 2TB NVME)</p>

ptrblck: <p>I would recommend taking a look at our <a href="https://pytorch.org/tutorials/recipes/recipes/tuning_guide.html">performance guide</a> for general tips to speed up your model training.<br>
E.g. <code>channels-last</code> should be beneficial for mixed-precision training, so you might want to enable it. Also, <code>torch.backends.cudnn.benchmark = True</code> could yield another speedup (assuming you are using static shapes or a limited range of variable input shapes). These points are explained in a bit more detail in the linked docs.</p>
How to set dtype for NN layers?

Dominik: <p>I have training data available as <code>dtype = torch.float64</code> and I want to use this data type to train my model. Now it seems that the <code>nn.Layer</code> (e.g. <a href="https://pytorch.org/docs/stable/nn.html#linear-layers" rel="nofollow noopener"><code>nn.Linear</code></a>) interface doesn’t offer the possibility of specifying a data type. Right now I get <code>RuntimeError: Expected object of type torch.DoubleTensor but found type torch.FloatTensor</code> because the dtypes are incompatible. How can I set the dtype of my NN layers to <code>torch.float64</code>? Do I have to assign the weights manually?</p>

ptrblck: <p>You can cast the whole layer to a new <code>dtype</code> using:</p>
<pre><code class="lang-python">lin = nn.Linear(10, 10).double()
</code></pre>

Dominik: <p>Thanks, that works! Is there a way to set it as the default for all layers or do I have to cast each layer manually? Something like a context manager would be handy; does it exist?</p>

ptrblck: <p>You can call it on the whole model which will cast all layers and parameters to the type.<br>
Alternatively, you could set the default type at the beginning of your script using <code>torch.set_default_dtype()</code>.</p>

cruzas: <p>How can this be done in C++?</p>
<p>Also, in general, how can the .double() be achieved for a torch::Tensor object in C++?</p>
<p>Thank you.</p>

ptrblck: <p>You could use <code>tensor = tensor.to(at::kDouble);</code> to transform the tensor to an FP64 one and</p>
<pre><code class="lang-python">AutoDefaultDtypeMode dtype_mode(default_dtype);
</code></pre>
<p>should work to define the default type.</p>

kevinchahine: <p>How can the dtype of a layer be changed in c++?</p>
<p>ex: <code>auto layer = torch::nn::Linear(4, 5); // creates a linear layer that works on float32 tensors.</code></p>
<p>How can I make a Linear layer with works on int8 tensors?</p>

constant_learner: <aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="31274">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p><code>torch.set_default_dtype()</code>.</p>
</blockquote>
</aside>
<p>looks like torch.float16 doesn’t work?</p>

ptrblck: <p>It seems to work for me as seen here:</p>
<pre><code class="lang-python">lin = nn.Linear(10, 10)
print(lin.weight.dtype)
# torch.float32

torch.set_default_dtype(torch.float16)
lin = nn.Linear(10, 10)
print(lin.weight.dtype)
# torch.float16
</code></pre>
<p>However, I don’t know if <em>all</em> operations support a change in the default <code>dtype</code> as I think it can be risky if e.g. integer types are expected. Could you post more information about your errors, please?</p>
Pytorch 2.0 suppress logging

miraodasilva: <p>Hi, I’m currently trying torch.compile in Pytorch 2.0 and it works well but absolutely floods my terminal with logs such as</p>
<blockquote>
<p>[2023-03-17 20:04:31,840] torch._dynamo.output_graph: [INFO] Step 2: done compiler function debug_wrapper</p>
</blockquote>
<p>I was wondering if there is a way to suppress these logs? Warnings are okay but for me the INFO logs are too much. I tried to find a way to torch.compile less verbose but couldn’t find anything.</p>
<p>Thanks!</p>

ptrblck: <p>This issue seems to be related to <a href="https://discuss.pytorch.org/t/the-pytorch-compiler-hijacks-my-logger-and-floods-it-with-debug-messages/173208/2">this discussion</a> and <a href="https://github.com/pytorch/pytorch/issues/94788#issuecomment-1439471999">this feature request</a>.</p>
Transfer learning from checkpoint - reset Optimizer exp_avg

bdzyubak: <p>I am trying to load a checkpoint and do transfer learning for image sets with different inputs. The original model was trained in 2 channels, but I need to train it on images with one. I was able to change the actual layer as follows:</p>
<pre><code class="lang-auto">net.load_state_dict(torch.load(model_weights), strict=match_required)
input_conv_layer = torch.nn.Conv3d(1, 32, kernel_size=(3, 3, 3), stride=(1, 1, 1), padding=(1, 1, 1))
input = torch.nn.init.xavier_uniform_(input_conv_layer.weight)
net['state_dict']['Fpn.C0.0.0.0.weight'] = input
</code></pre>
<p>However, when I begin training, the optimizer is trying to use the old state:</p>
<blockquote>
<p>net[‘optimizer’][‘state’][0][‘exp_avg’] and net[‘optimizer’][‘state’][0][‘exp_avg_sq’]</p>
</blockquote>
<p>both having 2 input channels. What is the correct way to change the optimizer to work with the new input_conv_layer?</p>
<p>I don’t mind resetting the optimizer, but would be interesting to know optimize w/ without reset. If I simply define a new optimizer:<br>
<code>optimizer = torch.optim.Adam(arch.parameters(), lr=1e-5)</code><br>
It is not a dictionary, and doesn’t match what is in</p>
<blockquote>
<p>net[‘optimizer’]</p>
</blockquote>
<p>P.S.: What is the proper terminology for net = pytorch.load(path) vs other_thing.load_state_dict(net)</p>
Documentation saving/loading quantized models

escorciav: <p>Hi guys!</p>
<p>Thanks for providing support for quantization in PyTorch.</p>
<p>Consider updating the docs regarding <a href="https://pytorch.org/docs/stable/quantization.html#saving-and-loading-quantized-models" rel="noopener nofollow ugc">saving/loading quantized models</a> till eager-mode quantization (manual tagging)  is supported.</p>
<p>Notice that you do have</p>
<p>I’m using the eager mode; the example didn’t work off the shelf. It took me a while to realize that the absence of <code>De</code>/[]-<code>QuantStub</code> was killing me.</p>
<p>Notice that you were OK with adding <code>De</code>/[]-<code>QuantStub</code> boilerplate <a href="https://pytorch.org/docs/stable/quantization.html#passing-a-quantized-tensor-into-a-non-quantized-kernel" rel="noopener nofollow ugc">here</a></p>
<p>Cheers,<br>
Victor</p>

supriyar: <p>hi, this is documented in the section on Eager mode quantization - see <a href="https://pytorch.org/docs/stable/quantization.html#model-preparation-for-eager-mode-static-quantization" class="inline-onebox" rel="noopener nofollow ugc">Quantization — PyTorch 1.13 documentation</a></p>

escorciav: <p>You are right!</p>
<p>Perhaps, for completeness, it might be great to reference that link in the section that I mentioned <img src="https://discuss.pytorch.org/images/emoji/apple/blush.png?v=12" title=":blush:" class="emoji" alt=":blush:" loading="lazy" width="20" height="20"> .</p>
Accuracy calculation for inbalanced data

Nyakov: <p>Hi.<br>
I have single-label, three-class classification problem.<br>
First two classes is say dog and cat, and third one is neither of that (it can by bird fox or just blank picture).<br>
I am after cat/dog detection when the third class dominates the data.<br>
And so I try to calculate accuracy for my model.<br>
My intuition tells me to count only first two classes, because this is what important to me.<br>
It this approach correct or it does not matter?</p>
<p>I also counting only samples with probability more than 0.6 because I want to be biased against false positive.</p>
<p>My code:</p>
<pre><code class="lang-auto">def batch_accuracy(pred: torch.Tensor, test: torch.Tensor) -&gt; tuple[float, float]:
    pred_prob = torch.nn.functional.softmax(pred, dim=1).to("cpu")
    test_cpu = test.to("cpu")

    correct_count = 0.0
    total_count = 0.0

    batch_size = pred_prob.size(0)
    for i in range(batch_size):
        class_index = test_cpu[i].item()
        if class_index == 0 or class_index == 1:
            total_count += 1.0
            if pred_prob[i][class_index] &gt; 0.6:
                correct_count += 1.0

    return correct_count, total_count
</code></pre>
Larger batch size in HF Trainer vs PyTorch

mmg: <p>Using the ‘bert-base-uncased’ model, I was able to use a batch size of 64 for the train dataset using the code from NLP with Transformers book by HF Team (<a href="https://github.com/nlp-with-transformers/notebooks" class="inline-onebox" rel="noopener nofollow ugc">GitHub - nlp-with-transformers/notebooks: Jupyter notebooks for the Natural Language Processing with Transformers book</a>). No special steps like grad accumulation and mixed precision were done.<br>
However, with my custom pytorch training loop using the same model and dataset, I was only able to train on a  batch size of 16 - increasing this would result in OOM error. Has anyone faced a similar situation?</p>

mmg: <p>Never mind. I figured it out after ‘extensive’ debugging. It turns out that i was padding to max_length and hf datasets was padding to max_length of the entire dataset. Once i tweaked it, both could be trained on bs of 64</p>
/homes/jl007/.conda/envs/video/lib/python3.10/site-packages/playwright/driver/node: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by /homes/jl007/.conda/envs/video/lib/python3.10/site-packages/playwright/driver/node)

Gorgen: <p>the code<br>
"<br>
from TikTokApi import TikTokApi<br>
import os<br>
import uuid<br>
import time</p>
<h1>
<a name="create-the-directory-to-store-the-videos-1" class="anchor" href="#create-the-directory-to-store-the-videos-1"></a>Create the directory to store the videos</h1>
<p>if not os.path.exists(‘tiktok_videos’):<br>
os.mkdir(‘tiktok_videos’)</p>
<h1>
<a name="instantiate-the-tiktokapi-class-2" class="anchor" href="#instantiate-the-tiktokapi-class-2"></a>Instantiate the TikTokApi class</h1>
<p>api = TikTokApi()</p>
<h1>
<a name="set-the-number-of-videos-to-crawl-3" class="anchor" href="#set-the-number-of-videos-to-crawl-3"></a>Set the number of videos to crawl</h1>
<p>num_videos = 1000</p>
<h1>
<a name="set-the-tiktok-username-or-hashtag-to-crawl-4" class="anchor" href="#set-the-tiktok-username-or-hashtag-to-crawl-4"></a>Set the TikTok username or hashtag to crawl</h1>
<p>query = ‘funny’</p>
<h1>
<a name="crawl-the-videos-and-save-them-to-the-local-directory-5" class="anchor" href="#crawl-the-videos-and-save-them-to-the-local-directory-5"></a>Crawl the videos and save them to the local directory</h1>
<p>for i in range(num_videos):<br>
# Get the next batch of videos<br>
videos = api.byUsername(query, count=100)</p>
<pre><code># Loop through the videos and download them
for video in videos:
    # Generate a unique filename for the video
    filename = f'{uuid.uuid4()}.mp4'

    # Download the video and save it to a file in the specified directory
    with open(f'tiktok_videos/{filename}', 'wb') as f:
        response = api.get_Video_By_DownloadURL(video['video']['downloadAddr'])
        f.write(response.content)

    # Print a message to show progress
    print(f'Downloaded video {i+1}/{num_videos}')

    # If we have downloaded the desired number of videos, break out of the loop
    if i+1 == num_videos:
        break

# Add a delay to avoid hitting the rate limit
time. Sleep(2)
</code></pre>
<p>"</p>
<p>issue “/homes/jl007/.conda/envs/video/lib/python3.10/site-packages/playwright/driver/node: /lib64/libm.so.6: version `GLIBC_2.27’ not found (required by /homes/jl007/.conda/envs/video/lib/python3.10/site-packages/playwright/driver/node)”</p>
<p>how to solve it? Thanks, best wishes</p>

ptrblck: <p>Your code doesn’t seem to use PyTorch (or I’m missing it) and the error seems to be raised by <code>playwright</code>, so I would recommend asking the authors of this library or to check <a href="https://github.com/microsoft/playwright/issues/20704">this issue</a> which seems to be related.</p>

Gorgen: <p>ok. many thanks, best wishes</p>
Dictionary of dictionaries of tensors

Benja: <p>Hi!. I want to create a dictionary of dictionaries in which I save some tensors. Imagine I have this dictionary:</p>
<p>data = {‘1’:[], ‘2’:[], ‘3’:[]}</p>
<p>and on each of these instances I have the dictionary {‘x’:[], ‘y’:[], ‘z’:[]}, that is to say</p>
<p>data = {‘1’:{‘x’:[], ‘y’:[], ‘z’:[]}, ‘2’:{‘x’:[], ‘y’:[], ‘z’:[]}, ‘3’:{‘x’:[], ‘y’:[], ‘z’:[]}}</p>
<p>In my case, I have more than just three instances, and inside each of them I have more instances, that’s why I would like to know if there is a better way to define the dictionary data. Thanks in advance!!</p>
Will the official Docker image be deleted?

mzur: <p>Given the <a href="https://blog.alexellis.io/docker-is-deleting-open-source-images/" rel="noopener nofollow ugc">recent announcement</a> that many Docker Hub organizations will be deleted, will <code>pytorch/pytorch</code> be affected? It doesn’t show the “Sponsored OSS” badge.</p>

ptrblck: <p>Thanks for forwarding this information.<br>
CC <a class="mention" href="/u/malfet">@malfet</a> <a class="mention" href="/u/seemethere">@seemethere</a></p>
<p>Interesting part from the post:</p>
<blockquote>
<p>If the project you maintain is owned by a foundation like the CNCF or Apache Foundation, you may simply be able to apply to Docker’s program.</p>
</blockquote>
<p>I don’t know if the PyTorch foundation would meet these requirements.</p>

malfet: <p>Please note, that for a while images has been published both both to <a href="http://docker.com" rel="noopener nofollow ugc">docker.com</a> as well as on <a href="https://ghcr.io/pytorch/pytorch" rel="noopener nofollow ugc">https://ghcr.io/pytorch/pytorch</a></p>

seemethere: <p>We applied for and got accepted to the open source program for the <em>pytorch</em> organization on Docker Hub a while ago. As well the pytorch organization on Docker is now covered under Meta’s enterprise license so that may be why we are not showing the Sponsored OSS badge anymore.</p>
<p>We stay committed to publishing on all container registry platforms we currently support.</p>

mzur: <p>That’s good to know, thanks!</p>

mzur: <aside class="quote no-group" data-username="malfet" data-post="3" data-topic="174996" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/malfet/48/27485_2.png" class="avatar"> malfet:</div>
<blockquote>
<p>Please note, that for a while images has been published both both to docker .com as well as on <a href="https://ghcr.io/pytorch/pytorch" class="inline-onebox-loading" rel="noopener nofollow ugc">https://ghcr.io/pytorch/pytorch</a></p>
</blockquote>
</aside>
<p>Really? The link gives me a 404 and a <code>docker pull ghcr.io/pytorch/pytorch</code> a “denied” (I have authentication to <a href="http://ghcr.io" rel="noopener nofollow ugc">ghcr.io</a> set up).</p>
Pytorch 2 FlashAttention and Memory Efficient Attention

BillyGun27: <p>I read that pytorch added memory-optimized algorithms like FlashAttention and Memory Efficient Attention</p>
<p><a href="https://pytorch.org/docs/master/generated/torch.nn.functional.scaled_dot_product_attention.html?highlight=scaled_dot_product#torch.nn.functional.scaled_dot_product_attention" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/master/generated/torch.nn.functional.scaled_dot_product_attention.html?highlight=scaled_dot_product#torch.nn.functional.scaled_dot_product_attention</a></p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://pytorch.org/blog/pytorch-2.0-release/#beta-scaled-dot-product-attention-20">
  <header class="source">
      <img src="https://pytorch.org/favicon.ico?" class="site-icon" width="32" height="32">

      <a href="https://pytorch.org/blog/pytorch-2.0-release/#beta-scaled-dot-product-attention-20" target="_blank" rel="noopener nofollow ugc">pytorch.org</a>
  </header>

  <article class="onebox-body">
    <img src="https://pytorch.org/assets/images/pytorch-logo.png" class="thumbnail onebox-avatar" width="500" height="500">

<h3><a href="https://pytorch.org/blog/pytorch-2.0-release/#beta-scaled-dot-product-attention-20" target="_blank" rel="noopener nofollow ugc">PyTorch</a></h3>

  <p>An open source machine learning framework that accelerates the path from research prototyping to production deployment.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Does anyone know if pytorch will support Flash Attention or other memory-optimized algorithms in PyTorch Mobile later? maybe there will also be mobile GPU backend support compatibility?</p>
Print(torch.cuda.is_available()) closes the Python command prompt

Metaldandy: <p>I’m trying to enter this to check if they’re compatible with my system but upon entering the third line it just closes the prompt without giving an output or error.</p>
<p>import torch<br>
print(torch.<strong>version</strong>)<br>
print(torch.cuda.is_available())</p>
<p>The only error info I can find is from the Event Viewer which are these two</p>
<p>Event 1000, Application Error<br>
Faulting application name: python.exe, version: 3.10.9150.1013, time stamp: 0x638fa05d<br>
Faulting module name: nvcuda64.dll, version: 31.0.15.3129, time stamp: 0x640826be<br>
Exception code: 0xc0000409<br>
Fault offset: 0x000000000053c834<br>
Faulting process id: 0x2af8<br>
Faulting application start time: 0x01d9587b425c6f40<br>
Faulting application path: C:\Users\Tampa\AppData\Local\Programs\Python\Python310\python.exe<br>
Faulting module path: C:\WINDOWS\system32\DriverStore\FileRepository\nv_dispi.inf_amd64_059948e396d205d5\nvcuda64.dll<br>
Report Id: 07e17557-8856-46b1-884c-14add9ca30ea<br>
Faulting package full name:<br>
Faulting package-relative application ID:</p>
<p>Event 1001, Windows Error Reporting<br>
Fault bucket 1843125454459915623, type 5<br>
Event Name: BEX64<br>
Response: Not available<br>
Cab Id: 0</p>
<p>Problem signature:<br>
P1: python.exe<br>
P2: 3.10.9150.1013<br>
P3: 638fa05d<br>
P4: nvcuda64.dll<br>
P5: 31.0.15.3129<br>
P6: 640826be<br>
P7: 000000000053c834<br>
P8: c0000409<br>
P9: 0000000000000005<br>
P10:</p>
<p>Attached files:<br>
\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WERAEA.tmp.dmp<br>
\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WERB68.tmp.WERInternalMetadata.xml<br>
\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WERB79.tmp.xml<br>
\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WERB77.tmp.csv<br>
\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WERB97.tmp.txt</p>
<p>These files may be available here:<br>
\?\C:\ProgramData\Microsoft\Windows\WER\ReportArchive\AppCrash_python.exe_71b197582739d23e413f38df2bba3b7d65e86_86ac2e64_099d44c8-e087-45bb-96b2-71a267320715</p>
<p>Analysis symbol:<br>
Rechecking for solution: 0<br>
Report Id: 07e17557-8856-46b1-884c-14add9ca30ea<br>
Report Status: 268435456<br>
Hashed bucket: 6ff48c6f696031cd999418d5e5904d67<br>
Cab Guid: 0</p>
<p>I used Python 3.10.9 and 3.10.6, Torch 1.13.1+cu117, Windows 10, GTX 1660 Super. Sorry if this is the wrong place to be asking for help, I’m still very new to all of this.</p>

ptrblck: <p>Could you try to update to the current <code>2.0.0</code> release using the CUDA 11.7 or 11.8 runtime and check if you would see the same error?<br>
Also, was PyTorch ever working on your setup or is this a new machine?</p>
PyTorch and Torch framework same

hizen: <p>Is my question about PyTorch and Torch frameworks the same? Because I confuse these two thing.<br>
Two different documentation<br>
Can anyone please explain these?</p><aside class="onebox allowlistedgeneric" data-onebox-src="http://torch.ch/">
  <header class="source">
      <img src="http://torch.ch/static/favicon.png" class="site-icon" width="16" height="16">

      <a href="http://torch.ch/" target="_blank" rel="noopener nofollow ugc">torch.ch</a>
  </header>

  <article class="onebox-body">
    <img src="http://torch.ch/static/torch_logo.png" class="thumbnail" width="" height="">

<h3><a href="http://torch.ch/" target="_blank" rel="noopener nofollow ugc">Torch | Scientific computing for LuaJIT.</a></h3>

  <p>Torch is a scientific computing framework for LuaJIT.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


ptrblck: <p>Torch7 is deprecated, was using <code>Lua</code>, and was the predecessor to PyTorch.<br>
It’s not the same and you should not use it anymore, as it’s not maintained.<br>
You might still see some common concepts that were adapted in PyTorch from it.</p>
I’m having some problems. Can you help me？

1135451439: <p>Traceback (most recent call last):<br>
File “D:\pythonProject\MatchFormer-main\test.py”, line 57, in<br>
trainer.test(model, datamodule=data_module, verbose=False)<br>
File “D:\anaconda\anaconda4\envs\pytorch-gpu\lib\site-packages\pytorch_lightning\trainer\trainer.py”, line 794, in test<br>
return call._call_and_handle_interrupt(<br>
File “D:\anaconda\anaconda4\envs\pytorch-gpu\lib\site-packages\pytorch_lightning\trainer\call.py”, line 38, in _call_and_handle_interrupt<br>
return trainer_fn(*args, **kwargs)<br>
File “D:\anaconda\anaconda4\envs\pytorch-gpu\lib\site-packages\pytorch_lightning\trainer\trainer.py”, line 842, in _test_impl<br>
results = self._run(model, ckpt_path=self.ckpt_path)<br>
File “D:\anaconda\anaconda4\envs\pytorch-gpu\lib\site-packages\pytorch_lightning\trainer\trainer.py”, line 1051, in _run<br>
self._call_setup_hook() # allow user to setup lightning_module in accelerator environment<br>
File “D:\anaconda\anaconda4\envs\pytorch-gpu\lib\site-packages\pytorch_lightning\trainer\trainer.py”, line 1298, in _call_setup_hook<br>
self._call_lightning_datamodule_hook(“setup”, stage=fn)<br>
File “D:\anaconda\anaconda4\envs\pytorch-gpu\lib\site-packages\pytorch_lightning\trainer\trainer.py”, line 1375, in _call_lightning_datamodule_hook<br>
return fn(*args, **kwargs)<br>
File “D:\pythonProject\MatchFormer-main\model\data.py”, line 116, in setup<br>
self.world_size = dist.get_world_size()<br>
File “D:\anaconda\anaconda4\envs\pytorch-gpu\lib\site-packages\torch\distributed\distributed_c10d.py”, line 1067, in get_world_size<br>
return _get_group_size(group)<br>
File “D:\anaconda\anaconda4\envs\pytorch-gpu\lib\site-packages\torch\distributed\distributed_c10d.py”, line 453, in _get_group_size<br>
default_pg = _get_default_group()<br>
File “D:\anaconda\anaconda4\envs\pytorch-gpu\lib\site-packages\torch\distributed\distributed_c10d.py”, line 584, in _get_default_group<br>
raise RuntimeError(<br>
RuntimeError: Default process group has not been initialized, please make sure to call init_process_group.</p>

ptrblck: <p>Did you initialize the process group as explained in the error message?<br>
I would also recommend changing the topic title to a give more information about your error so that more experienced users might want to help out.</p>
Question about graph break related to RETURN_VALUE and INLINE function call

James_Yan: <p>Hi folks,</p>
<p>I am testing the performance of dynamo+inductor on a pyg model and found that there are many graph breaks that greatly decrease the performance.<br>
I turned <code>_dynamo.config.log_level</code> to DEBUG, and found that most graph-breaks’ <code>GraphCompileReason</code> is <code>return_value</code>.<br>
Such as</p>
<pre><code class="lang-python">GraphCompileReason(reason='return_value', user_stack=[&lt;FrameSummary file /home/X/trienv/lib/python3.8/site-packages/torch_geometric/nn/conv/arma_conv.py, line 143 in message&gt;])
</code></pre>
<p>and the corresponding code line is</p>
<pre><code class="lang-python">        return edge_weight.view(-1, 1) * x_j # this line
</code></pre>
<p>It makes sense for a function call to trigger a frame thus resulting in a graph break. But what puzzles me is that there are many function calls containing <code>RETURN_VALUE</code> that are inlined without graph break as the log shows, such as</p>
<pre><code class="lang-python">[2023-03-14 14:23:10,376] torch._dynamo.symbolic_convert: [DEBUG] INLINING &lt;code object scatter_add at 0x7f7e2d852660, file "/home/X/trienv/lib/python3.8/site-packages/torch_scatter-2.1.0-py3.8-linux-x86_64.egg/torch_scatter/scatter.py", line 26&gt; 
  29           0 LOAD_GLOBAL              0 (scatter_sum)
              2 LOAD_FAST                0 (src)
              4 LOAD_FAST                1 (index)
              6 LOAD_FAST                2 (dim)
              8 LOAD_FAST                3 (out)
             10 LOAD_FAST                4 (dim_size)
             12 CALL_FUNCTION            5
             14 RETURN_VALUE
</code></pre>
<p>I have two questions:</p>
<ol>
<li>Is Dynamo inline the function calls greedily, and only break the graph if it is not possible?</li>
<li>Is there any principle or guideline for python code writing to make a function with <code>RETURN_VALUE</code> inlined and avoid a graph break in Dynamo?</li>
</ol>

James_Yan: <p>After carefully reading the log and I found that the <code>return_value</code> GraphCompileReason comes from a normal compilation of function frames. As for the failure of the function inlining, I discovered that it is due to some inline-failure function calls.<br>
For example, if a function fails in inlining, then the whole call stack related to this function will refuse to inline, this makes the log not entirely intuitive.</p>
<p>To avoid graph break, after diagnosing several cases in GNN scenario, I personally think the core factor is to avoid comparison on tensor value, especially on some frequently used helper functions, that will always result in graph fragments.</p>
How to implement an irregular mean-pooling？

xlxw: <p>Suppose I have 2-D tensor as shown below:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/5/4/5412079dcfd56a432b3660d805593ade8fcd315f.jpeg" alt="origin_mat" data-base62-sha1="bZIJDv7JOP3PUD7olLsNv8AguRh" width="363" height="452"></p>
<p>I wanna reserve the green blocks and compute mean vector over each yellow block with indexes (e.g., [[2,3], [6,7,8]])</p>
<p>How to implement these operations with pytorch? pls help me</p>

ptrblck: <p>I’m unsure if there is a cleaner and better approach, but you might use <code>tensor.scatter_reduce_</code> with the <code>reduce="mean"</code> argument and slice the output. This would waste compute since you would be moving the unwanted values to the end of the result, but might be faster than iterating the indices.<br>
This code should work:</p>
<pre><code class="lang-python"># setup
x = torch.randn(10, 8)

# create indices
idx = [[2,3], [6,7,8]]
indices = torch.full((x.size(0),), len(idx))
for i, l in enumerate(idx):
    indices[l] = i
print(indices)
# tensor([2, 2, 0, 0, 2, 2, 1, 1, 1, 2])

indices = indices.unsqueeze(1).expand(-1, 8)
out = torch.zeros(3, 8).scatter_reduce_(0, indices, x, reduce="mean", include_self=False).mean(dim=1)[:2]

reference = []
for i in idx:
    reference.append(x[i, :].mean())
reference = torch.tensor(reference)

print((reference - out).abs().max())
# tensor(1.4901e-08)
</code></pre>

xlxw: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="175033">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p><code>scatter_reduce_</code></p>
</blockquote>
</aside>
<p>Thx, it seems cool !</p>
Is there a pytorch profiler integration with huggingface trainer?

sajaldash: <p>I am trying to profile various resource utilization during training of transformer models using HuggingFace Trainer. Since the HF Trainer abstracts away the training steps, I could not find a way to use pytorch trainer as shown in <a href="https://pytorch.org/blog/introducing-pytorch-profiler-the-new-and-improved-performance-tool/" rel="noopener nofollow ugc">here</a>. I can extend the HF Trainer class and overwrite the train() function to integrate the profiler.step() instruction, but the train() function is a lengthy and complex one. Is there an easy way to using PyTorch Profiler with HuggingFace Trainer class?</p>

Data-drone: <p>It looks like this an example for using profiler with huggingface:</p><aside class="onebox githubblob" data-onebox-src="https://github.com/yqhu/profiler-workshop/blob/c8d4a7c30a61cc7b909d89f88f5fd36b70c55769/hf_training_trainer_prof.py">
  <header class="source">

      <a href="https://github.com/yqhu/profiler-workshop/blob/c8d4a7c30a61cc7b909d89f88f5fd36b70c55769/hf_training_trainer_prof.py" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/yqhu/profiler-workshop/blob/c8d4a7c30a61cc7b909d89f88f5fd36b70c55769/hf_training_trainer_prof.py" target="_blank" rel="noopener nofollow ugc">yqhu/profiler-workshop/blob/c8d4a7c30a61cc7b909d89f88f5fd36b70c55769/hf_training_trainer_prof.py</a></h4>


      <pre><code class="lang-py">from datasets import load_dataset, load_metric
from transformers import AutoTokenizer, DataCollatorWithPadding, AutoModelForSequenceClassification
from transformers import Trainer, TrainingArguments, TrainerCallback
import torch
import numpy as np
import time


raw_datasets = load_dataset("glue", "mrpc")
checkpoint = "bert-base-uncased"
tokenizer = AutoTokenizer.from_pretrained(checkpoint)

def tokenize_function(example):
    return tokenizer(example["sentence1"], example["sentence2"], truncation=True)

tokenized_datasets = raw_datasets.map(tokenize_function, batched=True)
data_collator = DataCollatorWithPadding(tokenizer=tokenizer)

model = AutoModelForSequenceClassification.from_pretrained(checkpoint, num_labels=2)

</code></pre>



  This file has been truncated. <a href="https://github.com/yqhu/profiler-workshop/blob/c8d4a7c30a61cc7b909d89f88f5fd36b70c55769/hf_training_trainer_prof.py" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
What does it mean? mistake?

Wendale_Yang: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/0/00c324bdb0776cf3a3fd07ab81b6c31cb9708384.png" data-download-href="https://discuss.pytorch.org/uploads/default/00c324bdb0776cf3a3fd07ab81b6c31cb9708384" title="pytorch2_1"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/0/00c324bdb0776cf3a3fd07ab81b6c31cb9708384_2_690x298.png" alt="pytorch2_1" data-base62-sha1="6K5LJjDFTIxsBFECKSskPu3OKM" width="690" height="298" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/0/00c324bdb0776cf3a3fd07ab81b6c31cb9708384_2_690x298.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/0/00c324bdb0776cf3a3fd07ab81b6c31cb9708384_2_1035x447.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/0/0/00c324bdb0776cf3a3fd07ab81b6c31cb9708384_2_1380x596.png 2x" data-dominant-color="F9F9F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">pytorch2_1</span><span class="informations">2490×1078 209 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Yes, this was a typo and is already corrected to <code>3.7</code>. Thanks for reporting! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
Warning on MPS backends

benhour: <p>Dear All,<br>
This is my first post, so please accept my apology if my question is silly.<br>
I am running PyTorch on MacBook Pro M2 Max and I am getting the following warning:<br>
“”"<br>
UserWarning: The operator ‘aten::nonzero’ is not currently supported on the MPS backend and will fall back to run on the CPU. This may have performance implications. (Triggered internally at /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/mps/MPSFallback.mm:11.)<br>
nonzero_finite_vals = torch.masked_select(<br>
“”"<br>
I am afraid that I am not able to use the full potential of MPS on my system.<br>
I appreciate your help regarding this issue.</p>
<p>Bests,<br>
Benhour</p>

ptrblck: <p><code>nonzero</code> support was added in <a href="https://github.com/pytorch/pytorch/pull/91616">this PR</a> and seems to be supported for <code>macOS&gt;=13.0</code> so you might need to update your PyTorch installation.</p>
CUDA version discrepancies

dmather: <p>Hi!</p>
<p>Im pretty new to PyTorch and using GPUs in my data science projects and I’m trying to learn as much I can!</p>
<p>Im confused why when I check my GPU’s information with nvidia-smi it shows a CUDA version of 11.4 but when I run my model for training it gives me an error saying my Quandro K4200 GPU only has a CUDA capability of 3.0 which is too old for PyTorch to run.  Any idea on why I see this discrepancy? And is it possible for me to access the CUDA 11.4 that my system says it has?</p>
<p>Thanks for your help!</p>
<p>(the error I get)<br>
Found GPU1 Quadro K4200 which is of cuda capability 3.0.<br>
PyTorch no longer supports this GPU because it is too old.<br>
The minimum cuda capability supported by this library is 3.7.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/7/1706b8b43c0b9cfefa5dbf5f330da47ebf4e21d5.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/1706b8b43c0b9cfefa5dbf5f330da47ebf4e21d5" title="Screen Shot 2023-03-16 at 5.48.26 PM"><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/7/1706b8b43c0b9cfefa5dbf5f330da47ebf4e21d5.jpeg" alt="Screen Shot 2023-03-16 at 5.48.26 PM" data-base62-sha1="3hHnIAFqhyIpr0pfw8eNOdZU0e1" width="690" height="349" data-dominant-color="2B2B2B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2023-03-16 at 5.48.26 PM</span><span class="informations">1118×566 64.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>The CUDA version reported in <code>nvidia-smi</code> is related to the NVIDIA driver you have installed.<br>
Note that the PyTorch binaries ship with their own CUDA runtime, cuDNN etc. and your local CUDA toolkit (if installed) will only be used to build PyTorch from source or custom CUDA extensions.<br>
As the error message explains, your GPU is too old since compute capability 3.0 is not supported anymore. The current binaries support all compute capabilities from <code>3.7</code> to <code>9.0</code> (if the CUDA 11.8 runtime is used).</p>
Logging file from the Trainer.train()

cardcounter: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/c/cca184aeb1d7066bdc074940cf63b3ec01eacaaa.png" data-download-href="https://discuss.pytorch.org/uploads/default/cca184aeb1d7066bdc074940cf63b3ec01eacaaa" title="Screenshot from 2023-03-16 19-42-35"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/c/cca184aeb1d7066bdc074940cf63b3ec01eacaaa_2_690x108.png" alt="Screenshot from 2023-03-16 19-42-35" data-base62-sha1="tcfsedUqPI6ZWRaR3YgyB8LfJOy" width="690" height="108" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/c/cca184aeb1d7066bdc074940cf63b3ec01eacaaa_2_690x108.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/c/cca184aeb1d7066bdc074940cf63b3ec01eacaaa_2_1035x162.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/c/c/cca184aeb1d7066bdc074940cf63b3ec01eacaaa_2_1380x216.png 2x" data-dominant-color="1D1D1D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2023-03-16 19-42-35</span><span class="informations">1901×299 41.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<pre><code class="lang-auto">[INFO|trainer.py:2813] 2023-03-16 19:41:58,395 &gt;&gt; Saving model checkpoint to ./log                                                                
[INFO|configuration_utils.py:457] 2023-03-16 19:41:58,396 &gt;&gt; Configuration saved in ./log/config.json                                             
[INFO|modeling_utils.py:1762] 2023-03-16 19:41:59,383 &gt;&gt; Model weights saved in ./log/pytorch_model.bin                                           
[INFO|tokenization_utils_base.py:2163] 2023-03-16 19:41:59,384 &gt;&gt; tokenizer config file saved in ./log/tokenizer_config.json                      
[INFO|tokenization_utils_base.py:2170] 2023-03-16 19:41:59,385 &gt;&gt; Special tokens file saved in ./log/special_tokens_map.json                      
</code></pre>
<ul>
<li>In the <strong>training_args.py</strong> there is a way to change the output directory while the<br>
<code>[INFO]</code>form the std.output is not <em>logged</em> anywhere, are there any way log this text <code>[INFO] [DEBUG] [Warning]</code> in a text file?<br>
<a href="https://github.com/huggingface/transformers/blob/main/src/transformers/training_args.py" rel="noopener nofollow ugc">https://github.com/huggingface/transformers/blob/main/src/transformers/training_args.py</a></li>
<li>where is this log text file located by default, change we change the location of this log text file/</li>
</ul>
Equivalent batch using DataParallel or DistributedDataParallel

marco_c: <p>Hi! If I, for example, use 25 for the batch size using only one GPU, if I use DataParallel or DistributedDataParallel on 2 GPUs, for me to have the equivalent batch size I should just multiply my batch size by 2, that is, 50?</p>
<p>If so, does this fit even in situations that I configure to use 2 GPUs, but it was not necessary to use more than 1 GPU (it only fit in GPU)?</p>
<p>Thanks in advance!</p>
How to replace modules like TorchFX for compiled model

cokespace2: <p>In TorchFX we can replace modules after symbolic trace returning a fx.GraphModule, how can we do that for a compiled module from torch.compile?</p>

marksaroufim: <p>It’s quite similar this notebook should give you more ideas as how this would work <a href="https://colab.research.google.com/drive/1Zh-Uo3TcTH8yYJF-LLo5rjlHVMtqvMdf#scrollTo=zaMCj87J_JLU" class="inline-onebox" rel="noopener nofollow ugc">Google Colab</a></p>
Understanding decompositions done in primtorch and autograd

GLZhu: <p>Hi community, I am developing a hardware backend ported to torchdynamo.<br>
From engineering perspective, we care more about a minal subset of prims ir/aten ir that need to be supported. Hence how the decomposition is triggered is key to us.</p>
<ol>
<li>We all know that in some case pytorch will fallback to eager and that makes aotautograd or primtorch unavailable. In this senerio, how are we supposed to do decomposition on our own. From my point of view is a copy of inductor’s decomposition list enough for us?</li>
<li>I searched through my log with env TORCH_COMPILE_DEBUG turned on and I want to debug decompositions happened, what is the suggested way to do this?</li>
<li>Is it possible that if we want to do as GPUs are doing, we port our codegen with triton IR rather than aten/prim ops, in this case, is there a minimal opset that we have to implement?</li>
</ol>

marksaroufim: <p>cc <a class="mention" href="/u/sherlocknomad">@SherlockNoMad</a> who might have more context</p>
Why no output for vit pretrained model?

Goldname: <p>I am trying to use torchvision.models.vit_b_32(). However, when I pass in some arbitrary data I get all zeros in the output</p>
<pre><code class="lang-auto">import torch
import torchvision
import torchvision.transforms as transforms

device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')

transform = transforms.Compose(
    [transforms.ToTensor(),
     transforms.Resize(224),
     transforms.Normalize(mean=[0.485, 0.456, 0.406],
                                 std=[0.229, 0.224, 0.225])])

batch_size = 4

testset = torchvision.datasets.CIFAR10(root='./data', train=False,
                                       download=True, transform=transform)
testloader = torch.utils.data.DataLoader(testset, batch_size=batch_size,
                                         shuffle=False, num_workers=2)

newmodel = torchvision.models.vit_b_32().to(device)
images, labels = next(iter(testloader))
images = images.to(device)
labels = labels.to(device)
print(torch.sum(newmodel(images)[0]))
</code></pre>
<p>Why is the last print statement printing zero?</p>
BCEWithLogitsLoss training predication

snj00u: <p>Hello</p>
<p>I am using BCEWithLogitsLoss to train a UNet model, the input images are in a RGB format and target images are binary (0 to 1)</p>
<p>When training the model I looked at the shape of inputs in the batch, are they are returning:</p>
<p>torch.Size([8, 3, 250, 250])  - input RGB image<br>
torch.Size([8, 1, 250, 250])  - target binary image</p>
<p>over training the predicated outcome has a shape of:<br>
torch.Size([8, 1, 250, 250])</p>
<p>pred = model(images)<br>
print(pred.shape)<br>
loss = torch.nn.BCEWithLogitsLoss()(pred, true_masks)</p>
<p>I just wondered that normal for the predicated outcome to include the batch size, being 8 in this case?</p>
<p>Thank you</p>

ptrblck: <p>Yes, the model is expected to return an output for each sample. Since you have passed 8 samples as a batch to the model also 8 outputs are expected in the output batch.</p>

snj00u: <p>Thank you</p>
<p>Within a batch for-loop it will be calling each function line 8 times behind the scenes? For example:</p>
<p>loss.backward()<br>
optimizer.step()</p>
<p>If we look at the output of: pred = model(images), will this show the logits numbers? Rather than the sigmoid outcomes?</p>
<p>Thank you</p>

ptrblck: <aside class="quote no-group" data-username="snj00u" data-post="3" data-topic="174928">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/439d5e/48.png" class="avatar"> snj00u:</div>
<blockquote>
<p>Within a batch for-loop it will be calling each function line 8 times behind the scenes?</p>
</blockquote>
</aside>
<p>No, since PyTorch layers accept batched inputs and will launch single kernels for each operation.</p>
<aside class="quote no-group" data-username="snj00u" data-post="3" data-topic="174928">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/439d5e/48.png" class="avatar"> snj00u:</div>
<blockquote>
<p>If we look at the output of: pred = model(images), will this show the logits numbers? Rather than the sigmoid outcomes?</p>
</blockquote>
</aside>
<p>It depends on your model and if it’s returning raw logits from e.g. a linear layer of if a <code>torch.sigmoid</code> is applied on the output before returning it.</p>

snj00u: <p>Sorry, I meant over training if we have:</p>
<p>pred = model(images),<br>
loss = torch.nn.BCEWithLogitsLoss()(pred, true_masks)</p>
<p>would the output of pred be the logits numbers, so much wider range of numbers than 0 to 1? I just noticed the output over training has minus values, so just trying to understand what it represents. I have check the true_masks and they are binary with a max of 1.</p>
<p>Thanks</p>
No kernel image is available for execution on the device

tcl: <p>Hello, I’ve been trying to install Pytorch CUDA on Windows 10 but I’ve had some problems.</p>
<p>I have an NVIDIA GeForce 920m with 425.31 Controller Version and CUDA 10.1.131 Drivers and therefore I installed CUDA 10.1 Toolkit</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/6/76ee2efd1a5ea267ab643d24f16e45e8912ef418.png" alt="image" data-base62-sha1="gY6E8EYsMVjNTsCR1Ttkt6SPpa0" width="667" height="148"></p>
<p>Using Python 3.7.9, I runned this line:</p>
<p><code>pip install torch==1.8.1+cu101 torchaudio==0.8.1 torchvision==0.9.1+cu101 -f https://download.pytorch.org/whl/cu101/torch_stable.html</code></p>
<p>And after all the setup, I only run on the code this:</p>
<p><code>device = torch.device("cuda:0") </code><br>
<code>torch.cuda.is_available()</code> which returns <code>True</code></p>
<p>And breaks in this line:</p>
<p><code>redent=torchtensor(entradas,dtype=tipo,device)</code></p>
<p>Returning: <code>RuntimeError: CUDA error: no kernel image is available for execution on the device</code></p>
<p>I don’t know If I did something wrong in the setup or somewhere else</p>

ptrblck: <p>Your <code>920m</code> should be a Kepler GPU with a compute capability of 3.5, which was dropped a few releases ago (I don’t know which one exactly, but it seems <code>1.8.1</code> already did not support it).<br>
You could either downgrade to an even older PyTorch release, or try to build from source and add compute capability 3.5 explicitly.</p>

tcl: <p>I managed to install PyTorch 1.2.0 for CUDA 10 downloading the .whl file, thanks!</p>
Tensorboard add_histogram

Rakshith_V: <p>In the bin option of tensor-board’s add_histogram method. I see the options such as “tensorflow,auto,fd”. Can someone how these options work ? (Except auto)</p>

ptrblck: <p>The <code>bins</code> arguments is passed in <a href="https://github.com/pytorch/pytorch/blob/d16c2c36ad0a78419db9b95622bf62274aa692f9/torch/utils/tensorboard/writer.py#L484-L485">this line of code</a> to <code>np.histogram</code> if I’m not mistaken, which then uses this input argument in <a href="https://numpy.org/doc/stable/reference/generated/numpy.histogram_bin_edges.html#numpy.histogram_bin_edges"><code>np.histogram_bin_edges</code></a>. If <code>tensorboard</code> is used the bins will be explicitly created <a href="https://github.com/pytorch/pytorch/blob/d16c2c36ad0a78419db9b95622bf62274aa692f9/torch/utils/tensorboard/writer.py#L249-L257">here</a> and assigned <a href="https://github.com/pytorch/pytorch/blob/d16c2c36ad0a78419db9b95622bf62274aa692f9/torch/utils/tensorboard/writer.py#L482-L483">here</a>.</p>