akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Export model with custom CUDA op in Pytorch 2.0 raises AttributeError: 'function' object has no attribute 'to_function_proto'

GPhilo: <p>Disclaimer: I’m not sure if this is a bug or if I’m missing something.</p>
<p>I’m trying to export to ONNX a Torch model that uses some custom CUDA ops. Following the torch.onnx <a href="https://pytorch.org/docs/stable/onnx.html" rel="noopener nofollow ugc">guide</a>, I define a symbolic function similar to:</p>
<pre><code class="lang-python">def my_symbolic_forward(g, arg0, arg1, int_arg2):
            arg2_i = sym_help._maybe_get_const(int_arg2, "i")
            # [omitted code where I calculate shape information]
            return g.op(
                "my_namespace::my_forward",
                arg0,
                arg1,
                arg2_i=arg2_i,
            )
</code></pre>
<p>The function is then registered via:</p>
<pre><code class="lang-python">torch.onnx.register_custom_op_symbolic("my_namespace::my_forward", my_symbolic_forward, 1)
</code></pre>
<p>This was tested and works in Torch 1.9 and 1.13, however in Torch 2.0.1 I get an AttributeError that looks related to onnxscript. The error is triggeret at <a href="https://github.com/pytorch/pytorch/blob/e9d2d74f0abe4b0e0f238e11b537c64041b3f9a7/torch/onnx/_internal/onnx_proto_utils.py#L281" rel="noopener nofollow ugc">this line</a>, in function <code>_find_onnxscript_op</code>, which looks odd, considering I did not go the onnxscript-way to export the model with my custom op.</p>
<p>Does this look like a bug (in which case I’ll open an issue on GitHub), or did I miss something telling pytorch that it should not treat this as onnxscript?</p>

ptrblck: <p>Based on your description the issue sounds like a real regression so I would recommend creating an issue on GitHub so that the code owners are aware of it.<br>
With that being said, I assume you are using TorchScript under the hood to export to ONNX, which is in maintenance now.</p>

GPhilo: <p>Thanks, I will open a github issue as soon as I have a chance and I’ll post a link to it here as well for bookkeeping.</p>
<p>I am indeed using TorchScript under the hood, but I am not aware of any other way to export to ONNX… are there alternatives?</p>

ptrblck: <p>Not yet, but the export path is in development as described <a href="https://discuss.pytorch.org/t/pytorch-c-deployment-story-2019/58199/23">here</a>.</p>

carsonswope: <p>I have the same issue on 2.0.1, and can confirm that it is fixed by reverting to 1.13.1</p>

GPhilo: <p>I managed to reproduce the behaviour and opened a Github issue <a href="https://github.com/pytorch/pytorch/issues/104700" rel="noopener nofollow ugc">here</a>.</p>

GPhilo: <p>Just updating here as well: the GitHub issue has been resolved, so I guess the fix will be in the next version update (hopefully)</p>
Yolox multinode training not able to run

purvang_lapsiwala: <p>Hi, I do understand this issue should be in YOLOX repo but I think project is no longer active and no one is answering so decided to try my luck here.</p>
<p>I am trying to train YOLOX on 2 nodes, each with 8 gpus. both servers can be can be connected with ssh. I am trying to run multi node training as shown in <a href="https://github.com/Megvii-BaseDetection/YOLOX#:~:text=Multi%20Machine%20Training" rel="noopener nofollow ugc">yolox</a> but  after starting multinode script, it initializes gpus and then it hangs and doesn’t move (pytorch DDP). using horovod for distributed training working fine.</p>
<p>environment</p>
<p>python 3.8.10<br>
torch 1.13.1+cu116<br>
torchvision 0.14.1+cu116<br>
cuda 11.6<br>
libcudnn8 8.4.1.50-1</p>
<p>Thanks</p>
Torchscript realtime: running multiple inferences from the same model

elgiano: <p>Hello!<br>
I’m writing some software (in cpp) to load torchscripts and run their methods in realtime for audio applications (<a href="https://github.com/elgiano/nn.ar" rel="noopener nofollow ugc">code on github</a>).</p>
<p>My question is: is it safe to run multiple independent inferences concurrently from the same loaded Module? I would guess it’s not, also because methods could have side effects…</p>
<p>But then what would be the way to go? I’ve tried to use <code>clone</code> so far, but I would like to hear from this community what would you do?</p>
Representation Learning MSc course + pytorch exercises

black0017: <p>[<img src="https://discuss.pytorch.org/images/emoji/apple/movie_camera.png?v=12" title=":movie_camera:" class="emoji" alt=":movie_camera:" loading="lazy" width="20" height="20"><img src="https://discuss.pytorch.org/images/emoji/apple/books.png?v=12" title=":books:" class="emoji" alt=":books:" loading="lazy" width="20" height="20"> Exciting NEWS, everyone! <img src="https://discuss.pytorch.org/images/emoji/apple/star2.png?v=12" title=":star2:" class="emoji" alt=":star2:" loading="lazy" width="20" height="20"> ] - I’m ultra thrilled to announce that the first ever online <span class="hashtag-raw">#RepresentationLearniing</span> MSc course is now publicly available.  Ready to learn how to learn informative representations from images, proteins, and natural language? <img src="https://discuss.pytorch.org/images/emoji/apple/mortar_board.png?v=12" title=":mortar_board:" class="emoji" alt=":mortar_board:" loading="lazy" width="20" height="20"><img src="https://discuss.pytorch.org/images/emoji/apple/video_camera.png?v=12" title=":video_camera:" class="emoji" alt=":video_camera:" loading="lazy" width="20" height="20"></p>
<p><img src="https://discuss.pytorch.org/images/emoji/apple/one.png?v=12" title=":one:" class="emoji" alt=":one:" loading="lazy" width="20" height="20"> Visit <a href="https://www.youtube.com/playlist?list=PL3mKiGE4zNJJ83K4c3IBka6eYfe6v71dS" class="inline-onebox" rel="noopener nofollow ugc">Representation Learning MSc course SS2023 HHU Dusseldorf - YouTube</a> to access the full playlist of recorded lectures.  <img src="https://discuss.pytorch.org/images/emoji/apple/open_book.png?v=12" title=":open_book:" class="emoji" alt=":open_book:" loading="lazy" width="20" height="20"><img src="https://discuss.pytorch.org/images/emoji/apple/bulb.png?v=12" title=":bulb:" class="emoji" alt=":bulb:" loading="lazy" width="20" height="20"></p>
<p><img src="https://discuss.pytorch.org/images/emoji/apple/two.png?v=12" title=":two:" class="emoji" alt=":two:" loading="lazy" width="20" height="20"> Feel free to tag anyone who might be interested. <img src="https://discuss.pytorch.org/images/emoji/apple/speech_balloon.png?v=12" title=":speech_balloon:" class="emoji" alt=":speech_balloon:" loading="lazy" width="20" height="20"><img src="https://discuss.pytorch.org/images/emoji/apple/handshake.png?v=12" title=":handshake:" class="emoji" alt=":handshake:" loading="lazy" width="20" height="20"></p>
<p><img src="https://discuss.pytorch.org/images/emoji/apple/three.png?v=12" title=":three:" class="emoji" alt=":three:" loading="lazy" width="20" height="20"> Spread the word and help us out! <img src="https://discuss.pytorch.org/images/emoji/apple/earth_africa.png?v=12" title=":earth_africa:" class="emoji" alt=":earth_africa:" loading="lazy" width="20" height="20"><img src="https://discuss.pytorch.org/images/emoji/apple/speaking_head.png?v=12" title=":speaking_head:" class="emoji" alt=":speaking_head:" loading="lazy" width="20" height="20"></p>
<p>It’s our (together with Felix Michels, and Tim Kaiser) first attempt, thus be prepared for some mistakes along the way.</p>
ConnectionResetError: [WinError 10054] while downloading pre-trained models

heloproc: <p>The download starts fine but later on I get the following error-</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "D:\pyai\j_v2.py", line 31, in &lt;module&gt;
    train_data = datasets.MNIST(root="./data", train=True, download=True, transform=transforms.ToTensor())
  File "D:\py\lib\site-packages\torchvision\datasets\mnist.py", line 99, in __init__
    self.download()
  File "D:\py\lib\site-packages\torchvision\datasets\mnist.py", line 187, in download
    download_and_extract_archive(url, download_root=self.raw_folder, filename=filename, md5=md5)
  File "D:\py\lib\site-packages\torchvision\datasets\utils.py", line 433, in download_and_extract_archive
    download_url(url, download_root, filename, md5)
  File "D:\py\lib\site-packages\torchvision\datasets\utils.py", line 151, in download_url
    raise e
  File "D:\py\lib\site-packages\torchvision\datasets\utils.py", line 144, in download_url
    _urlretrieve(url, fpath)
  File "D:\py\lib\site-packages\torchvision\datasets\utils.py", line 48, in _urlretrieve
    _save_response_content(iter(lambda: response.read(chunk_size), b""), filename, length=response.length)
  File "D:\py\lib\site-packages\torchvision\datasets\utils.py", line 37, in _save_response_content
    for chunk in content:
  File "D:\py\lib\site-packages\torchvision\datasets\utils.py", line 48, in &lt;lambda&gt;
    _save_response_content(iter(lambda: response.read(chunk_size), b""), filename, length=response.length)
  File "D:\py\lib\http\client.py", line 465, in read
    s = self.fp.read(amt)
  File "D:\py\lib\socket.py", line 705, in readinto
    return self._sock.recv_into(b)
ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host

</code></pre>
<p>any idea how to fix this</p>

ptrblck: <p>Your connection is closed so check if your firewall or any other networking setting could be responsible for it.</p>
Pretrained MaskRCNN returning normal box but weird mask predictions?

johnny69: <p>Training a pretrained <a href="https://pytorch.org/vision/main/models/generated/torchvision.models.detection.maskrcnn_resnet50_fpn.html#torchvision.models.detection.maskrcnn_resnet50_fpn" rel="noopener nofollow ugc"><code>maskrcnn_resnet50_fpn</code></a> to detect cells from an image. I’ve trained it for 200 epochs. The box predictions it makes are sensible, e.g.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/0/6053d4c83454b1fd8935eb6681187afe0e12795f.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/6053d4c83454b1fd8935eb6681187afe0e12795f" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/0/6053d4c83454b1fd8935eb6681187afe0e12795f_2_690x342.jpeg" alt="image" data-base62-sha1="dK9rzbBnOUw2Y1x8rz87xoVtwuP" width="690" height="342" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/0/6053d4c83454b1fd8935eb6681187afe0e12795f_2_690x342.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/6/0/6053d4c83454b1fd8935eb6681187afe0e12795f_2_1035x513.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/6/0/6053d4c83454b1fd8935eb6681187afe0e12795f_2_1380x684.jpeg 2x" data-dominant-color="EBECEE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1384×686 90.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The mask predictions however, are not sensible, e.g.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/a/7af1da26025eb84eaec0074b4c1062fa0e03c1d1.png" data-download-href="https://discuss.pytorch.org/uploads/default/7af1da26025eb84eaec0074b4c1062fa0e03c1d1" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/a/7af1da26025eb84eaec0074b4c1062fa0e03c1d1_2_246x500.png" alt="image" data-base62-sha1="hxCpRNh4ZSandWog8gYyaVJ8FSF" width="246" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/a/7af1da26025eb84eaec0074b4c1062fa0e03c1d1_2_246x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/a/7af1da26025eb84eaec0074b4c1062fa0e03c1d1_2_369x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/7/a/7af1da26025eb84eaec0074b4c1062fa0e03c1d1.png 2x" data-dominant-color="775180"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">488×990 28.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I’m not sure why this is occurring. This occurs even after I’ve trained the model for 200 epochs. In addition, this occurs on both the train and the test sets.</p>
<p>Any idea of how to fix this?</p>
What is the framework of pytorch from python api to linux underlay?

borninfreedom: <p>A schematic similar to the <a href="https://developer.android.com/guide/platform" rel="noopener nofollow ugc">Android framework</a></p>

ptrblck: <p>Could you describe your question in more detail as I don’t understand the reference to Android?</p>

borninfreedom: <p>Thanks for the reply. Specifically, for example, in the android framework, the java api layer can call the native c/c++ layer, the native c/c++ layer continues to call the HAL layer down, the HAL layer continues to call the linux kernel, and the whole calling process is finished. For pytorch, I only vaguely know that python api can call libcuda.so, and then libcuda.so can call libc.so, but I don’t clearly understand the whole calling framework.</p>
How to release GPU memory for a sequence of predictions?

JimW: <p>I am trying to use one pretrained model to generate a sequence of predictions for long video. The model is supposed to be frozen during the inference.<br>
The pseudo code is like this:</p>
<pre><code class="lang-auto">model = torch.load("pretrained_model.pyth")

for one_video in MyDataloader:
    all_my_pred_list = []
    for vid_seg in one_video:
        pred = model(vid_seg)
        all_my_pred_list.append(pred)
        del vid_seg

    # Other routine that consumes all_my_pred_list:  for example  video_level_pred =  average(all_my_pred_list)

    for ele in all_my_pred_list:
        del ele
</code></pre>
<p>The problem with the above code is that,  when the video is very long, it will run out of gpu memory.<br>
What is the best way to release GPU memory during the above inference without destroying the loaded model?</p>

ptrblck: <p>Warp the forward pass into a <code>with torch.no_grad()</code> or <code>with torch.inference_mode()</code> guard which avoids storing the intermediate forward activations, which would be needed for the gradient computation.<br>
Currently you are not only storing <code>pred</code> but the entire computation graph including all intermediate tensors unless you have disabled gradient calculation globally.</p>

JimW: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, thank you so much for your response.  Yes, I forgot to perform model.eval() in the above exemplary code. My situation was that,  even if I did model.eval() earlier,  the GPU memory is still large because I am doing multiple inference (a sequence of predictions generated for one single input video). So what can I do to further release GPU memories?  I am imagining that I should “free something” inside the inner most for-loop: Now I am doing “del vid_seg”, but I am not sure how to safely remove some temporary memories introduced by the model.</p>

ptrblck: <p><code>model.eval()</code> won’t reduce the memory usage but will change the behavior of some modules: e.g. dropout will be disabled and batchnorm layers will use their running stats to normalize the input activation.<br>
Use the already mentioned guards instead.</p>
CUDA initialization error with `torch.zeros([1]).cuda()`

caronzh03: <p>Hi,</p>
<p>I have 4x A100 GPUs on a single machine.<br>
With PyTorch <code>2.0.0+cu117</code> and nvidia driver version of <code>515.105.01</code> installed, the following is working as expected:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; torch.cuda.is_available()
True
</code></pre>
<p>However, when I was trying to run a simple <code>torch.zeros([1]).cuda()</code> command, it kept throwing Runtime errors at cuda init stage:</p>
<pre><code class="lang-auto">File "~/.conda/envs/torchtest/lib/python3.8/site-packages/torch/cuda/__init__.py", line 375, in get_device_properties
    return _get_device_properties(device)  # type: ignore[name-defined]
RuntimeError: device &gt;= 0 &amp;&amp; device &lt; num_gpus INTERNAL ASSERT FAILED at "../aten/src/ATen/cuda/CUDAContext.cpp":50, please report a bug to PyTorch.
</code></pre>
<p>When I set <code>export CUDA_VISIBLE_DEVICES=0</code>, the above <code>.cuda()</code> command worked, but unsetting this env var broke it again.</p>
<p>Where in my set up could have gone wrong?</p>

ptrblck: <p>I guess the env variable might be set to an invalid value since directly exporting it seems to work.<br>
If that’s not the case, do you have multiple GPUs installed where some are dead or inactive (we had recently a similar issue in this forum where 2 GPUs were installed in the system while one of them was not plugged in and caused issues during the initialization).</p>

caronzh03: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>.</p>
<p>I think it has to do with the A100 GPU setup - it has MIG devices configured, and my guess now is that those were not configured correctly for CUDA to enumerate all the devices:</p>
<pre><code class="lang-auto">GPU 0: NVIDIA A100-SXM4-80GB (UUID: GPU-6a0256bd-cc1d-7e2a-d8ba-b6d5deefb5ff)
  MIG 7g.80gb     Device  0: (UUID: MIG-f659fe52-a79b-5941-807c-2366430ee70e)
GPU 1: NVIDIA A100-SXM4-80GB (UUID: GPU-bc5e96fd-a778-84ea-83a8-bb24a49c2d91)
  MIG 7g.80gb     Device  0: (UUID: MIG-7cc70377-7e53-5450-807d-45f954c77439)
GPU 2: NVIDIA A100-SXM4-80GB (UUID: GPU-a556b2d7-6245-0820-54d7-537606e233e6)
  MIG 7g.80gb     Device  0: (UUID: MIG-20b4d0ee-83cf-5f08-930a-153b788d38cd)
GPU 3: NVIDIA A100-SXM4-80GB (UUID: GPU-450d7302-f972-c964-b90a-bcfc396949ab)
  MIG 7g.80gb     Device  0: (UUID: MIG-bd9d4fd8-b734-5cff-95e6-2d7dd1e0272f)
</code></pre>
<p>Will update once I have more findings and a fix.</p>

ptrblck: <p>Thanks for the update. In that case disable MIG or specify the UUID in <code>CUDA_VISIBLE_DEVICES</code>.</p>

caronzh03: <p>Yep! Disabling the MIG config worked!</p>
Should I update my nvidia drivers on Ubuntu if I don't want to have any issues with Pytorch?

tss: <p>I have recently set up environment for pytorch on fresh Ubuntu 22.04.2 LTS system.</p>
<p>My current package versions:</p>
<blockquote>
<p>nvidia drivers: 520.61.05<br>
CUDA version: 11.8<br>
pytorch: 2.0.1+cu118</p>
</blockquote>
<p>It seems that there are some compatibility issues between latest pytorch version and latest CUDA Toolkit 12. Furthermore, it seems that CUDA 12.8 (latest) is incompatible with nvidia drivers v520.61.05 and CUDA 11.8 is incompatible with nvidia drivers v535.54.03 (latest) according to Table 3 from <a href="https://docs.nvidia.com/deploy/cuda-compatibility/index.html#:%7E:text=The%20CUDA%20driver%20maintains%20backward,within%20the%20corresponding%20major%20release." rel="noopener nofollow ugc">this</a> page.</p>
<p>Now I want to run <code>sudo apt-get update</code> but it seems that it would update my nvidia drivers and (or) CUDA toolkit and pytorch won’t work properly with CUDA.</p>
<p>Should I add nvidia drivers and CUDA toolkit to the blacklist for updates? If so, how should I do that? What is a good approach to manage the versions of these packages?</p>

ptrblck: <p>Table 3 explains CUDA’s forward compatibility for data center GPUs, which need additional setup steps.<br>
The PyTorch binaries with CUDA 11.8 are compatible to both drivers according to table 1.</p>

tss: <p>I’ve just run the command <code>sudo apt-get update</code> and it didn’t update neither the nvidia drivers nor CUDA toolkit so I think that the issue is resolved</p>
Best way to merge dimensions in pytorch

Spencerfonbuena: <p>I am reading through the paper “Crossformer: Transformer Utilizing Cross-Dimension Dependency for Multivariate Time Series Forecasting”</p>
<p>As part of their architecture, they merge together time series through the encoder, so as to attend to more  coarse levels of time data. In their github, they do it in the following manner:</p>
<pre><code class="lang-auto"># [Dummy Input]
seg_to_merge = []
win_size = 2
x = torch.arange(40).reshape(1,2,10,2)

for i in range(win_size):
    seg_to_merge.append(x[:, :, i::win_size, :])
x = torch.cat(seg_to_merge, -1)
</code></pre>
<p>However, this gives the same output as this line:</p>
<pre><code class="lang-auto">x = x.reshape(1,2,5,4)
</code></pre>
<p>Is there a difference in the two? I try to keep it as simple as possible, and the reshape just seems to be a lot better. The only thing I can think of is that the reshape has some effect on the backprop, or something behind the scenes. If anyone has any better ideas for the name of this question I am also open to changing it. I just am not sure what a tag line for this question would be.</p>

ptrblck: <p>I’m also unsure why the loop with the <code>cat</code> is used as <code>reshape</code> will not break the computation graph so should be fine to use.</p>
`fork` and `wait` within LibTorch's `torch::jit::compile`

dfalbel: <p>Hi,</p>
<p>I’m seeing the following when trying to <code>fork</code> and <code>wait</code> inside <code>torch::jit::compile</code> from LibTorch. For example, something like:</p>
<pre><code class="lang-auto">#include &lt;torch/torch.h&gt;
#include &lt;iostream&gt;

int main() {
  auto compiled = torch::jit::compile(R"|(
def foo (x: Tensor):
    return torch.neg(x)

def hello (x: Tensor):
  future = torch.jit.fork(foo, x)
  x_normal = foo(x)
  x_parallel = torch.jit.wait(future)
  return x_normal, x_parallel
)|");
    return 0;
}
</code></pre>
<p>Errors with:</p>
<pre><code class="lang-auto">libc++abi: terminating with uncaught exception of type torch::jit::ErrorReport: 
attribute lookup is not defined on builtin:
  File "&lt;string&gt;", line 6

def hello (x: Tensor):
  future = torch.jit.fork(foo, x)
           ~~~~~~~~~~~~~~ &lt;--- HERE
  x_normal = foo(x)
  x_parallel = torch.jit.wait(future)
</code></pre>
<p>Is this supposed to work? If not, as I don’t really understand jit internals, it would be nice to understand why this couldn’t work.</p>
<p>Thank you very much</p>
Does torch.ops.aten.convolution_backward use cudnn if available?

arushram14: <p>I understand this is a generated function so I’m having a hard time seeing where it is implemented to make sure if it would use cudnn or not. In general what is a good way to navigate the source code, and how can I find the answer to whether it does use cudnn or not?</p>

ptrblck: <aside class="quote no-group" data-username="arushram14" data-post="1" data-topic="183715">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/arushram14/48/61996_2.png" class="avatar"> arushram14:</div>
<blockquote>
<p>In general what is a good way to navigate the source code</p>
</blockquote>
</aside>
<p>A plain GitHub search would point you to <a href="https://github.com/pytorch/pytorch/blob/5da4745c24859d501c9555dbe67a20dfcd199942/aten/src/ATen/native/Convolution.cpp#L1977">this code</a> which is used to dispatch to different backends.<br>
<code>grep -r convolution_backward</code> could also be a good approach to search for the funciton definition.</p>
<p>A proper way to verify if cuDNN is used or not is to profile the code and to check the used kernel.</p>

arushram14: <p>Thanks, this is very helpful! Are there any similar implementations for pooling gradients utilizing CuDNN?</p>
<p>Edit: Upon using grep it seems the only mention of cudnnPoolingBackward is in the caffe library. Is there any reason it isn’t used (ie is PyTorch’s native CUDA implementation as performant)?</p>

ptrblck: <p>cuDNN provides pooling layers, which are not used in PyTorch in the quantization backend if I’m not mistaken. PyTorch eager uses a native pooling layer.</p>
DataPipe batch vs Dataloader batch

David_Samy: <p>Hi guys,</p>
<p>I am trying to refactor my code with DataPipes instead of Datasets + Dataloader. One of the problems I am running into is when I tried to handle batching with DataPipes instead of the Dataloader, the results are different:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/6/b6a660257588233099d3ff7fa9a4e12956be6df6.png" alt="image" data-base62-sha1="q3NmaLAV8iB51FJLyRB3e0TuncO" width="528" height="290"></p>
<p>As you can see the example batch is different depending on the method used. Could someone explain how to achieve the same functionality that DataLoader(batch_size = n) but with DataPipes?</p>
<p>Thanks.</p>
"torch._dynamo hit config.cache_size_limit (64)"

vgoklani: <p>Hey there,</p>
<p>I keep hitting this warning, followed by a long set of compiler messages:</p>
<pre><code>[2023-07-10 15:24:52,962] torch._dynamo.convert_frame: [WARNING] torch._dynamo hit config.cache_size_limit (64) function: 'forward' (/usr/local/lib/python3.10/dist-packages/transformers/models/t5/modeling_t5.py:452) reasons:  ___check_obj_id(L['self'], 139679724322528) to diagnose recompilation issues, see https://pytorch.org/docs/master/compile/troubleshooting.html.
</code></pre>
<p>I checked the pytorch docs:</p>
<pre><code>https://pytorch.org/docs/stable/dynamo/troubleshooting.html#excessive-recompilation
</code></pre>
<p>and saw this:</p>
<pre><code>torch._dynamo.config.cache_size_limit = &lt;your desired cache limit&gt;
</code></pre>
<p>Is there any guidance for how to choose the cache limit? Also, not sure how to set this attribute as it’s unable to reference _dynamo directly (?)</p>
<p>Not sure if it matters, but the model being trained was a <code>t5-base</code> via huggingface’s trainer on an NVIDIA A6000 Ada card (48GB). The installation is straight from this docker-image: <code>nvcr.io/nvidia/pytorch:23.06-py3</code></p>
<p>Thanks!</p>

marksaroufim: <p>It’s likely that your model is just graph breaking because of some changes per iteration, try nighlies, try dynamic=True, try making sure your inputs are padded to the exact same shape and finally try the recompilation profiler from teh troubleshooting guide if neither of those work. And if that doesnt help either it’s probably something model specific and it might be then helpful to open up an issue with repro on github</p>
Torch compile shapes

aifartist: <p>AUTOTUNE convolution(2x2560x16x16, 1280x2560x1x1)</p>
<p>How does optimizing for so many obscure shapes help someone doing 1x4x64x64 stable diffusion inferencing?  The initial “1” is the batch size so might differ.  The 64x64 is what is used for 512x512 images in the UNet.  The VAE then takes the 64x64 to 512x512.</p>
<p>Wouldn’t there be a huge compile speed up in not exploring the huge shape space?</p>

marksaroufim: <p>Yeah I’m not sure how we’d know a priori but perhaps <a class="mention" href="/u/eellison">@eellison</a> knows</p>
Take a subset of a DataLoader

Moe: <p>I want to create a small subset of the IMDB dataset, so I can test if my program as a whole works before running training on the whole dataset. I tried using a Sampler, but this doesn’t seem to work since the dataset is already loaded as an iterator as follows:</p>
<pre><code>train_iter, test_iter = IMDB(root = ".")
</code></pre>
<p>Is there a quick way I can do this?</p>

ptrblck: <p>If you are preloading the entire dataset in its <code>__init__</code> method you would need to change this logic there.<br>
On the other hand, if you are lazily loading each sample in the <code>__getitem__</code> you can use a <code>Subset</code> or <code>SubsetRandomSampler</code> to only load the desired indices.</p>