jiebi/CodeConvo huggingface dataset:
I am not even sure if that warning has any real impact on the model yet.
Thank you for the answer Reverting will cause problem for me. But reverting will only regenerate the warning. In my experiments, till now, I have not observed any difference with or without the warning. But I am not 100% sure. I'm looking forward to any more information you can give me.
Seeing the same error. Any solutions?
Yea I need to find someone to look into this issue. It seems like removing the reshape lowering will introduce a regression in performance.
I am facing the same issue, I don't even have a "reshape" call in my code anywhere.
reopened , will try to fix this before the 2.3 release branch cut.
will lead the investigation on the regression that blocks merging Regression only happens in dynamo so my guess is it has something to do with functionization pass..
Running through a small example with on HEAD and on , the snippet HLOo generated for in both cases are: Now will try to experiment with functionalizaton explicitly on/off to see if there are some unexpected decompositions.
I am also experiencing the same problem. I have also seen significant reduction in the learning capabilities of my model. My model has more than 80% accuracy when I trained it in another device without xla. However, its accuracy has been reduced to less than 10% compared to when I trained on a GCP TPU VM with pytorch-xla. It is also apparent that loss values do not change between consecutive epochs during training. Here is a portion of the output including the error message: {'loss': 0.0288, 'gradnorm': 0., 'learningrate': 4.-05, 'epoch': 6.0} {'evalloss': 0., 'evalruntime': 5.4579, 'evalsamplespersecond': 615.621, 'evalstepspersecond': 19.238, 'epoch': 6.0} 20% 2502/ [28:36&lt;1:08:07, 2.45it/s/home/egecitepred/miniconda3/envs/ege/lib/python3.8/site- UserWarning: aten::reshape: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it.
This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd. (Triggered internally at ) Variable.executionengine.runbackward( # Calls into the C++ engine to run the backward pass {'loss': 0.0289, 'gradnorm': 0., 'learningrate': 3.-05, 'epoch': 7.0} {'evalloss': 0., 'evalruntime': 5.4422, 'evalsamplespersecond': 617.398, 'evalstepspersecond': 19.294, 'epoch': 7.0} 23% 2919/ [31:44&lt;1:05:13, 2.45it/s/home/egecitepred/miniconda3/envs/ege/lib/python3.8/site- UserWarning: aten::reshape: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it. This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd.
(Triggered internally at ) Variable.executionengine.runbackward( # Calls into the C++ engine to run the backward pass {'loss': 0.0288, 'gradnorm': 0., 'learningrate': 3.-05, 'epoch': 8.0} {'evalloss': 0., 'evalruntime': 5.4432, 'evalsamplespersecond': 617.286, 'evalstepspersecond': 19.29, 'epoch': 8.0} I am also receiving this error message at every epoch. Is it possible to register CompositeImplicitAutograd key to autograd kernel like the error message says? If someone could provide me instructions on registering this key, I could try to see if this fixes the error or at least the reduction of the performance. I don't have enough experience in this area yet. So, I couldn't find a way of registering the autograd key.
Hey thanks for reporting the issue. Just to confirm, the warning messages in the original issue should not have an impact in the performance. We tried to remove this warning message and merged , which caused a performance regression (which is now reverted in head). Which nightlies did you observe the performance regression in your case?
Running a quick experiment on resnet18, the performance is more or so similar with HEAD and reshape op removed. With HEAD, timing some runs of the resnet18 dynamo unit test: And timing some runs of the resnet18 dynamo unit test with removed: Before we continue further with the investigation, and I discussed that if simply registering this kernel to both the AutogradXLA and XLA dispatch keys would remove the warning message, we might just do that.. though I'm not too familiar this. do you know how can we register the kernel?
Hello This message seems to slightly reduced the performance of my model unlike others. I am not absolutely sure about this but I made some additional observations that lead me to believe that. I have tried the same model on a GPU without XLA. It has somewhat higher performance compared to TPU VM with PytorchXLA. However, my main problem is the reduction of learning capabilities after training. This error message reduced my result metrics after training from ~80% to ~20% under the same conditions. Also, my output for each epoch shows the same evalloss value.
My other experiments on a GPU had the same eval loss value on epoch 1. It seems the model (with TPU XLA) is learning only for the first epoch and fails to properly backprop. Thus, it keeps repeating the first epoch again and again. Although I am not sure about this, it is the only explanation I could come up. I am not sure which nightly version I am using. However, I know that I downloaded this library on February 29th using this command: "pip install torch~=2.2.0 torch_xla[tpu]~=2.2.0 -f "
Hey I wonder if that's a different issue -- could you open a separate issue in PyTorch/XLA? Since Brian is OOO, do you have any idea on the kernel registration mentioned in
From the warning, you are hitting the AutogradXLA key since the fallback at that key is triggering this warning. So the problem is that your differentiable reshape implementation should be registered at that key. If you say above that "ResigisterAutogradXLA.h/cpp." gets generated then I would check if reshare is there. If it is, I would double check that this file is properly included in your binary and properly registers the kernels. You can also check to see all the kernels registered for that op and from where
Hey thanks for the pointers! I tried to register the op to with With this PR, I could see it get registered: However, now I'm getting the error: I assume something is wrong with our new lowering, which is just a redispatch to functionalization -- However, this seems to be what we're doing for some of similar ops -- Maybe I'm just confused here lol, do you see anything obvious that is wrong? Thanks for the help, again!
The vision pinned hash is from a long time ago (2020?) and attempting to update it to a newer hash fails. See for examples of CI failing when the hash is updated, and for a possible reason why. When fixed, we should add vision back into . n/a cc
Self-assigning myself to have a look at it
to TorchVision, which changes how operates when given a source folder: instead of execution it extracts the requirements from above-mentioned toml file. So, one can either pass to continue installing torchvision in the old way, or update to contain proper dependencies
fixed by
When I try to train my model which contains MaxPool3d , It always end up with 'out of memory' error. my environment info is here: 16.04 PyTorch version: 0.4.0a0+ installed PyTorch from source python version is 3.6.2 CUDA/cuDNN version: 9.1/7.1.2 GCC version (if compiling from source):GCC 5.4 Build command you used (if compiling from source): as default I can reproduce this bug by the following script:
Thanks for the report, I can reproduce this and am looking into it.
Functions and does not work correctly when called in DataPrallel mode The reason is while the first thread is budling the extension, the second one asks for a current version, and the returns the version of the first thread build. Second thread then skips lock file and tries to load non existed extension, failing with an error: I would like to submit a PR with a fix: PyTorch version: 2.4.0a0+git8046de3 Is debug build: False CUDA used to build PyTorch: 11.8 ROCM used to build PyTorch: N/A OS: Ubuntu 20.04.6 LTS (x8664) GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0 Clang version: Could not collect CMake version: version 3.26.4 Libc version: glibc-2.31 Python version: 3.12.2 (main, Feb 27 2024, 17:35:02) [GCC 11.2.0] (64-bit runtime) Python platform: Linux-5.15.0-105-generic-x8664-with-glibc2.31 Is CUDA available: True CUDA runtime version: 11.8.89 CUDAMODULELOADING set to: LAZY GPU models and configuration: GPU 0: NVIDIA GeForce RTX 3090 GPU 1: NVIDIA GeForce RTX 3090 GPU 2: NVIDIA GeForce RTX 3090 Nvidia driver version: 525.147.05 cuDNN version: Could not collect HIP runtime version: N/A MIOpen runtime version: N/A Is XNNPACK available: True CPU: Architecture: x8664 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian Address sizes: 46 bits physical, 48 bits virtual CPU(s): 36 On-line CPU(s) list: 0-35 Thread(s) per core: 2 Core(s) per socket: 18 Socket(s): 1 NUMA node(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 85 Model name: Intel(R) Core(TM) i9-10980XE CPU @ 3.00GHz Stepping: 7 CPU MHz: 3000.000 CPU max MHz: 4800,
0000 CPU min MHz: 1200,0000 BogoMIPS: 6000.00 Virtualization: VT-x L1d cache: 576 KiB L1i cache: 576 KiB L2 cache: 18 MiB L3 cache: 24,8 MiB NUMA node0 CPU(s): 0-35 Vulnerability Gather data sampling: Mitigation; Microcode Vulnerability Itlb multihit: KVM: Mitigation: VMX disabled Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Mitigation; Clear CPU buffers; SMT vulnerable Vulnerability Retbleed: Mitigation; Enhanced IBRS Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and user pointer sanitization Vulnerability Spectre v2: Mitigation; Enhanced IBRS, IBPB conditional, RSB filling, PBRSB-eIBRS SW sequence Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Mitigation;
TSX disabled Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constanttsc art archperfmon pebs bts repgood nopl xtopology nonstoptsc cpuid aperfmperf pni pclmulqdq dtes64 monitor dscpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse41 sse42 x2apic movbe popcnt tscdeadlinetimer aes xsave avx f16c rdrand lahflm abm 3dnowprefetch cpuidfault epb catl3 cdpl3 invpcidsingle ssbd mba ibrs ibpb stibp ibrsenhanced tprshadow vnmi flexpriority ept vpid eptad fsgsbase tscadjust bmi1 avx2 smep bmi2 erms invpcid cqm mpx rdta avx512f avx512dq rdseed adx smap clflushopt clwb intelpt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqmllc cqmoccupllc cqmmbmtotal cqmmbmlocal dtherm ida arat pln pts hwp hwpactwindow hwpepp hwppkgreq avx512vnni mdclear flushl1d archcapabilities Versions of relevant libraries: [pip3] pytorch-triton==3.0.0+ [pip3] torch==2.4.0a0+git8046de3 [conda] magma-cuda110 2.5.2 1 pytorch [conda] mkl-include 2024.1.0 intel691 intel [conda] mkl-static 2024.1.0 intel691 intel [conda] pytorch-triton 3.0.0+ pypi0 pypi [conda] torch 2.4.0a0+git8046de3 dev_0 <developcc