jiebi/CodeConvo huggingface dataset:
Code change:
// TODO: gradOutput shape check // Resize and initialize result tensor. THCTensor_(resizeAs)(state, gradInput, input); <del> THCTensor_(newContiguous)(state, gradInput); </del> THCTensor_(zero)(state, gradInput); int batchSize;

Related discussion:
text: Since reshape() cannot have an autograd formula (it is sometimes a view and sometimes not), then you should register your kernel as a CompositeImplicitAutograd kernel (this will properly register onto AutogradXLA and remove this warning)
Thanks ! This make sense! I have a follow up question, we registered reshape and reshape symint in and This file will then get codegen into and . What's the correct way of marking to be ?
Ho I'm not familiar with how your codegen works... But in principles, I guess you want to do the same as NestedTensor here which uses for the dispatch key in do you have more context on this by any chance?
it's probably enough for you to just register that kernel to both the and dispatch keys (this is similar to what CompositeImplicitAutograd does, but it does it for all backends). I'm a bit confused though - Why do you need a custom reshape impl? And how are you going to get training support for it? (do you have a custom implementation of your reshape's backward)
Haha this is actually by you in for functionization. Do you think we still need this trick for functionization?
Yes - I think we should be able to remove it. Do you mind trying that? That should just run the existing kernel defined . And that should be identical to the current behavior, since the "registration" for reshape in XLA just calls back into ATen ()
I merged , let's wait and see if tmr's nightly will resolve this issue.
Thank you. I will test it and give feedback. Till now, I have run my experiments with the warning. Do I need to rerun the experiments?
OK, with the recent nighly versions of Pytorch and Pytorch-XLA I do not observe the warning anymore while running my experiments. Thank you. Should I rerun the experiments that were run with past Pytorch versions throwing the warning? Does this update only remove the warning or does it also change the calculations that I ran with the previous Pytorch versions? Can I rely on my previous experiments?
Sorry we have to revert the pr since it creates regression on nightly.
Hello I am using nightly without any problem. What should I do? Wait for new update?
if you don't see any regression in performance, you can likely just keep using today's nightly.
Code change:
// TODO: gradOutput shape check // Resize and initialize result tensor. THCTensor_(resizeAs)(state, gradInput, input); <del> THCTensor_(newContiguous)(state, gradInput); </del> THCTensor_(zero)(state, gradInput); int batchSize;

Related discussion:
text: I am not even sure if that warning has any real impact on the model yet.
Thank you for the answer Reverting will cause problem for me. But reverting will only regenerate the warning. In my experiments, till now, I have not observed any difference with or without the warning. But I am not 100% sure. I'm looking forward to any more information you can give me.
Seeing the same error. Any solutions?
Yea I need to find someone to look into this issue. It seems like removing the reshape lowering will introduce a regression in performance.
I am facing the same issue, I don't even have a "reshape" call in my code anywhere.
reopened , will try to fix this before the 2.3 release branch cut.
will lead the investigation on the regression that blocks merging Regression only happens in dynamo so my guess is it has something to do with functionization pass..
Running through a small example with on HEAD and on , the snippet HLOo generated for in both cases are: Now will try to experiment with functionalizaton explicitly on/off to see if there are some unexpected decompositions.
I am also experiencing the same problem. I have also seen significant reduction in the learning capabilities of my model. My model has more than 80% accuracy when I trained it in another device without xla. However, its accuracy has been reduced to less than 10% compared to when I trained on a GCP TPU VM with pytorch-xla. It is also apparent that loss values do not change between consecutive epochs during training. Here is a portion of the output including the error message: {'loss': 0.0288, 'gradnorm': 0., 'learningrate': 4.-05, 'epoch': 6.0} {'evalloss': 0., 'evalruntime': 5.4579, 'evalsamplespersecond': 615.621, 'evalstepspersecond': 19.238, 'epoch': 6.0} 20% 2502/ [28:36&lt;1:08:07, 2.45it/s/home/egecitepred/miniconda3/envs/ege/lib/python3.8/site- UserWarning: aten::reshape: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it.
Code change:
// TODO: gradOutput shape check // Resize and initialize result tensor. THCTensor_(resizeAs)(state, gradInput, input); <del> THCTensor_(newContiguous)(state, gradInput); </del> THCTensor_(zero)(state, gradInput); int batchSize;

Related discussion:
text: This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd. (Triggered internally at ) Variable.executionengine.runbackward( # Calls into the C++ engine to run the backward pass {'loss': 0.0289, 'gradnorm': 0., 'learningrate': 3.-05, 'epoch': 7.0} {'evalloss': 0., 'evalruntime': 5.4422, 'evalsamplespersecond': 617.398, 'evalstepspersecond': 19.294, 'epoch': 7.0} 23% 2919/ [31:44&lt;1:05:13, 2.45it/s/home/egecitepred/miniconda3/envs/ege/lib/python3.8/site- UserWarning: aten::reshape: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it. This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd.
Code change:
// TODO: gradOutput shape check // Resize and initialize result tensor. THCTensor_(resizeAs)(state, gradInput, input); <del> THCTensor_(newContiguous)(state, gradInput); </del> THCTensor_(zero)(state, gradInput); int batchSize;

Related discussion:
text: (Triggered internally at ) Variable.executionengine.runbackward( # Calls into the C++ engine to run the backward pass {'loss': 0.0288, 'gradnorm': 0., 'learningrate': 3.-05, 'epoch': 8.0} {'evalloss': 0., 'evalruntime': 5.4432, 'evalsamplespersecond': 617.286, 'evalstepspersecond': 19.29, 'epoch': 8.0} I am also receiving this error message at every epoch. Is it possible to register CompositeImplicitAutograd key to autograd kernel like the error message says? If someone could provide me instructions on registering this key, I could try to see if this fixes the error or at least the reduction of the performance. I don't have enough experience in this area yet. So, I couldn't find a way of registering the autograd key.
Hey thanks for reporting the issue. Just to confirm, the warning messages in the original issue should not have an impact in the performance. We tried to remove this warning message and merged , which caused a performance regression (which is now reverted in head). Which nightlies did you observe the performance regression in your case?
Running a quick experiment on resnet18, the performance is more or so similar with HEAD and reshape op removed. With HEAD, timing some runs of the resnet18 dynamo unit test: And timing some runs of the resnet18 dynamo unit test with removed: Before we continue further with the investigation, and I discussed that if simply registering this kernel to both the AutogradXLA and XLA dispatch keys would remove the warning message, we might just do that.. though I'm not too familiar this. do you know how can we register the kernel?
Hello This message seems to slightly reduced the performance of my model unlike others. I am not absolutely sure about this but I made some additional observations that lead me to believe that. I have tried the same model on a GPU without XLA. It has somewhat higher performance compared to TPU VM with PytorchXLA. However, my main problem is the reduction of learning capabilities after training. This error message reduced my result metrics after training from ~80% to ~20% under the same conditions. Also, my output for each epoch shows the same evalloss value.
Code change:
// TODO: gradOutput shape check // Resize and initialize result tensor. THCTensor_(resizeAs)(state, gradInput, input); <del> THCTensor_(newContiguous)(state, gradInput); </del> THCTensor_(zero)(state, gradInput); int batchSize;

Related discussion:
text: My other experiments on a GPU had the same eval loss value on epoch 1. It seems the model (with TPU XLA) is learning only for the first epoch and fails to properly backprop. Thus, it keeps repeating the first epoch again and again. Although I am not sure about this, it is the only explanation I could come up. I am not sure which nightly version I am using. However, I know that I downloaded this library on February 29th using this command: "pip install torch~=2.2.0 torch_xla[tpu]~=2.2.0 -f "
Hey I wonder if that's a different issue -- could you open a separate issue in PyTorch/XLA? Since Brian is OOO, do you have any idea on the kernel registration mentioned in
From the warning, you are hitting the AutogradXLA key since the fallback at that key is triggering this warning. So the problem is that your differentiable reshape implementation should be registered at that key. If you say above that "ResigisterAutogradXLA.h/cpp." gets generated then I would check if reshare is there. If it is, I would double check that this file is properly included in your binary and properly registers the kernels. You can also check to see all the kernels registered for that op and from where
Hey thanks for the pointers! I tried to register the op to with With this PR, I could see it get registered: However, now I'm getting the error: I assume something is wrong with our new lowering, which is just a redispatch to functionalization -- However, this seems to be what we're doing for some of similar ops -- Maybe I'm just confused here lol, do you see anything obvious that is wrong? Thanks for the help, again!
Code change:
// TODO: gradOutput shape check // Resize and initialize result tensor. THCTensor_(resizeAs)(state, gradInput, input); <del> THCTensor_(newContiguous)(state, gradInput); </del> THCTensor_(zero)(state, gradInput); int batchSize;

Related discussion:
text: The vision pinned hash is from a long time ago (2020?) and attempting to update it to a newer hash fails. See for examples of CI failing when the hash is updated, and for a possible reason why. When fixed, we should add vision back into . n/a cc
Self-assigning myself to have a look at it
to TorchVision, which changes how operates when given a source folder: instead of execution it extracts the requirements from above-mentioned toml file. So, one can either pass to continue installing torchvision in the old way, or update to contain proper dependencies
fixed by
Code change:
// TODO: gradOutput shape check // Resize and initialize result tensor. THCTensor_(resizeAs)(state, gradInput, input); <del> THCTensor_(newContiguous)(state, gradInput); </del> THCTensor_(zero)(state, gradInput); int batchSize;

Related discussion:
text: When I try to train my model which contains MaxPool3d , It always end up with 'out of memory' error. my environment info is here: 16.04 PyTorch version: 0.4.0a0+ installed PyTorch from source python version is 3.6.2 CUDA/cuDNN version: 9.1/7.1.2 GCC version (if compiling from source):GCC 5.4 Build command you used (if compiling from source): as default I can reproduce this bug by the following script:
Thanks for the report, I can reproduce this and am looking into it.
Code change:
Once these are installed, you can use the backend for Caffe2:: # ...continuing from above <del> import onnx.backend.caffe2 as backend </del> <ins> import onnx_caffe2.backend as backend </ins> import numpy as np rep = backend.prepare(graph, device="CUDA:0") # or "CPU" # For the Caffe2 backend: #     rep.predict_net is the Caffe2 protobuf for the network #     rep.workspace is the Caffe2 workspace for the network <del> #       (see the class onnx.backend.c2.Workspace) </del> <ins> #       (see the class onnx_caffe2.backend.Workspace) </ins> outputs = rep.run(np.random.randn(10, 3, 224, 224).astype(np.float32)) # To run networks with more than one input, pass a tuple # rather than a single numpy ndarray.

Related discussion:
text: Functions and does not work correctly when called in DataPrallel mode The reason is while the first thread is budling the extension, the second one asks for a current version, and the returns the version of the first thread build. Second thread then skips lock file and tries to load non existed extension, failing with an error: I would like to submit a PR with a fix: PyTorch version: 2.4.0a0+git8046de3 Is debug build: False CUDA used to build PyTorch: 11.8 ROCM used to build PyTorch: N/A OS: Ubuntu 20.04.6 LTS (x8664) GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0 Clang version: Could not collect CMake version: version 3.26.4 Libc version: glibc-2.31 Python version: 3.12.2 (main, Feb 27 2024, 17:35:02) [GCC 11.2.0] (64-bit runtime) Python platform: Linux-5.15.0-105-generic-x8664-with-glibc2.31 Is CUDA available: True CUDA runtime version: 11.8.89 CUDAMODULELOADING set to: LAZY GPU models and configuration: GPU 0: NVIDIA GeForce RTX 3090 GPU 1: NVIDIA GeForce RTX 3090 GPU 2: NVIDIA GeForce RTX 3090 Nvidia driver version: 525.147.05 cuDNN version: Could not collect HIP runtime version: N/A MIOpen runtime version: N/A Is XNNPACK available: True CPU: Architecture: x8664 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian Address sizes: 46 bits physical, 48 bits virtual CPU(s): 36 On-line CPU(s) list: 0-35 Thread(s) per core: 2 Core(s) per socket: 18 Socket(s): 1 NUMA node(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 85 Model name: Intel(R) Core(TM) i9-10980XE CPU @ 3.00GHz Stepping: 7 CPU MHz: 3000.000 CPU max MHz: 4800,
Code change:
Once these are installed, you can use the backend for Caffe2:: # ...continuing from above <del> import onnx.backend.caffe2 as backend </del> <ins> import onnx_caffe2.backend as backend </ins> import numpy as np rep = backend.prepare(graph, device="CUDA:0") # or "CPU" # For the Caffe2 backend: #     rep.predict_net is the Caffe2 protobuf for the network #     rep.workspace is the Caffe2 workspace for the network <del> #       (see the class onnx.backend.c2.Workspace) </del> <ins> #       (see the class onnx_caffe2.backend.Workspace) </ins> outputs = rep.run(np.random.randn(10, 3, 224, 224).astype(np.float32)) # To run networks with more than one input, pass a tuple # rather than a single numpy ndarray.

Related discussion:
text: 0000 CPU min MHz: 1200,0000 BogoMIPS: 6000.00 Virtualization: VT-x L1d cache: 576 KiB L1i cache: 576 KiB L2 cache: 18 MiB L3 cache: 24,8 MiB NUMA node0 CPU(s): 0-35 Vulnerability Gather data sampling: Mitigation; Microcode Vulnerability Itlb multihit: KVM: Mitigation: VMX disabled Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Mitigation; Clear CPU buffers; SMT vulnerable Vulnerability Retbleed: Mitigation; Enhanced IBRS Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and user pointer sanitization Vulnerability Spectre v2: Mitigation; Enhanced IBRS, IBPB conditional, RSB filling, PBRSB-eIBRS SW sequence Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Mitigation;
Code change:
Once these are installed, you can use the backend for Caffe2:: # ...continuing from above <del> import onnx.backend.caffe2 as backend </del> <ins> import onnx_caffe2.backend as backend </ins> import numpy as np rep = backend.prepare(graph, device="CUDA:0") # or "CPU" # For the Caffe2 backend: #     rep.predict_net is the Caffe2 protobuf for the network #     rep.workspace is the Caffe2 workspace for the network <del> #       (see the class onnx.backend.c2.Workspace) </del> <ins> #       (see the class onnx_caffe2.backend.Workspace) </ins> outputs = rep.run(np.random.randn(10, 3, 224, 224).astype(np.float32)) # To run networks with more than one input, pass a tuple # rather than a single numpy ndarray.

Related discussion:
text: TSX disabled Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constanttsc art archperfmon pebs bts repgood nopl xtopology nonstoptsc cpuid aperfmperf pni pclmulqdq dtes64 monitor dscpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse41 sse42 x2apic movbe popcnt tscdeadlinetimer aes xsave avx f16c rdrand lahflm abm 3dnowprefetch cpuidfault epb catl3 cdpl3 invpcidsingle ssbd mba ibrs ibpb stibp ibrsenhanced tprshadow vnmi flexpriority ept vpid eptad fsgsbase tscadjust bmi1 avx2 smep bmi2 erms invpcid cqm mpx rdta avx512f avx512dq rdseed adx smap clflushopt clwb intelpt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqmllc cqmoccupllc cqmmbmtotal cqmmbmlocal dtherm ida arat pln pts hwp hwpactwindow hwpepp hwppkgreq avx512vnni mdclear flushl1d archcapabilities Versions of relevant libraries: [pip3] pytorch-triton==3.0.0+ [pip3] torch==2.4.0a0+git8046de3 [conda] magma-cuda110 2.5.2 1 pytorch [conda] mkl-include 2024.1.0 intel691 intel [conda] mkl-static 2024.1.0 intel691 intel [conda] pytorch-triton 3.0.0+ pypi0 pypi [conda] torch 2.4.0a0+git8046de3 dev_0 <developcc
Code change:
import contextlib import platform import ctypes <ins> import os </ins> import torch _initialized = False

Related discussion:
text: right now if is not in LDLIBRARYPATH, though it was found at compile-time, these lines will fail: Avoid this, by idk doing something... I think i have a few good ideas.
We need to add /usr/local/cuda/lib64 (or that's mac dir) to the rpath of _C
the problem i dont think is to add it to rpath, it's because we
Right. We could try with the default cuda installation path if it's not found from LDLIBRARYPATH.
Actually, do we really want to use the compile path? We should try to detect it at runtime, right? Or are we going to ship CUDA libs in our binaries?
we first load using runtime paths, and if it fails, fallback to path known at compile time.
I fear this is going to get brittle, especially if you are shipping CUDA libs. e.g. a user swaps HW and picks up a new driver and/or toolkit. As says, I'm pretty sure you want to use the runtime paths. Similarly, CUDA 7.5 won't run on Pascal and we have a cudnn built for 7.5 that will not do the right thing on CUDA 8, so those need to go together. If a user installs the CUDA bits via deb/rpm the right things are supposed to happen. If you push bits, we need to figure out how to make sure things line up. But this is perhaps a much larger packaging conversation.
Yeah, I've implemented it to first try loading it without any path and then try and . I'll push the commit tomorrow.
Code change:
#ifdef TH_REAL_IS_INT #define NUMPY_TYPE_ENUM NPY_INT32 #endif <ins> #ifdef TH_REAL_IS_SHORT #define NUMPY_TYPE_ENUM NPY_INT16 #endif </ins> #ifdef TH_REAL_IS_BYTE #define NUMPY_TYPE_ENUM NPY_UINT8 #endif

Related discussion:
text: Should we add a ShortTensor Type, or just convert int16 to IntTensor ?
We already have a type available in pytorch. We have to enable conversion in the relevant function here:
Auto travis-ci failed at python2.7 :(
Do you have any plan to support int8 numpy conversion while you have too?
Char tensor uses char which is not guaranteed to be signed by the C standard. We'd need to change our C code to use a
I try , it complains , any idea?
The error message is quite self explanatory. PyTorch doesn't support tensors at the moment.
Code change:
{"_cuda_initialSeed", (PyCFunction)THCPModule_initialSeed, METH_NOARGS, NULL}, {"_cuda_cudaHostAllocator", (PyCFunction)THCPModule_cudaHostAllocator, METH_NOARGS, NULL}, {"_cuda_synchronize", (PyCFunction)THCPModule_cudaSynchronize, METH_NOARGS, NULL}, <ins> {"_cuda_getLibPath", (PyCFunction)THCPModule_getLibPath, METH_NOARGS, NULL}, </ins> #endif {"_safe_call",      (PyCFunction)THPModule_safeCall,          METH_VARARGS | METH_KEYWORDS, NULL}, {"_sendfd",         (PyCFunction)THPModule_sendfd,            METH_VARARGS, NULL},

Related discussion:
text: right now if is not in LDLIBRARYPATH, though it was found at compile-time, these lines will fail: Avoid this, by idk doing something... I think i have a few good ideas.
We need to add /usr/local/cuda/lib64 (or that's mac dir) to the rpath of _C
the problem i dont think is to add it to rpath, it's because we
Right. We could try with the default cuda installation path if it's not found from LDLIBRARYPATH.
Actually, do we really want to use the compile path? We should try to detect it at runtime, right? Or are we going to ship CUDA libs in our binaries?
we first load using runtime paths, and if it fails, fallback to path known at compile time.
I fear this is going to get brittle, especially if you are shipping CUDA libs. e.g. a user swaps HW and picks up a new driver and/or toolkit. As says, I'm pretty sure you want to use the runtime paths. Similarly, CUDA 7.5 won't run on Pascal and we have a cudnn built for 7.5 that will not do the right thing on CUDA 8, so those need to go together. If a user installs the CUDA bits via deb/rpm the right things are supposed to happen. If you push bits, we need to figure out how to make sure things line up. But this is perhaps a much larger packaging conversation.
Yeah, I've implemented it to first try loading it without any path and then try and . I'll push the commit tomorrow.
Code change:
def align_tensors(*tensors): <del> if not torch._C._BUILD_NAMEDTENSOR: raise RuntimeError('NYI: torch.align_tensors is experimental and a part ' 'of our named tensors project.') return torch._C._VariableFunctions.align_tensors(tensors) </del> <ins> raise RuntimeError('`align_tensors` not yet implemented.') </ins>

Related discussion:
text: only supports fully named inputs; all input dimensions must have a name. When passing it an unnamed input, it errors out with the following message: It should really say "Found unnamed dim at index 0 of Tensor[None, None]".
fixed in
Code change:
def align_tensors(*tensors): <del> if not torch._C._BUILD_NAMEDTENSOR: raise RuntimeError('NYI: torch.align_tensors is experimental and a part ' 'of our named tensors project.') return torch._C._VariableFunctions.align_tensors(tensors) </del> <ins> raise RuntimeError('`align_tensors` not yet implemented.') </ins>

Related discussion:
text: This is an older implementation that I think doesn't make any sense anymore. We should throw a NYI exception for it. The expected behavior for this should be:
fixed in