jiebi/CodeConvo huggingface dataset:
Traceback (most recent call last): File "", line 322, in &lt;moduleroislabel = fasterRCNN(imdata, iminfo, gtboxes, numboxes) File "/usr/local/lib/python2.7/dist-", line 491, in call result = self.forward(input, kwargs) File "/home/faster-", line 50, in forward rois, rpnlosscls, rpnlossbbox = self.RCNNrpn(basefeat, iminfo, gtboxes, numboxes) File "/usr/local/lib/python2.7/dist-", line 491, in call result = self.forward(input, kwargs) File "/home/faster-", line 87, in forward rpndata = self.RPNanchortarget((, gtboxes, iminfo, numboxes)) File "/usr/local/lib/python2.7/dist-", line 491, in call result = self.forward(input, kwargs) File "/home/faster-", line 157, in forward positiveweights = 1.0 / numexamples File "/usr/local/lib/python2.7/dist-", line 320, in rdiv return self.reciprocal() other RuntimeError: reciprocal is not implemented for type Exception NameError: "global name 'FileNotFoundError' is not defined" in &lt;bound method of object at 0x7fb842e6e3d0ignored This error is coming while implementing pytorch faster-rcnn repository. Any solutions please??
can you use the issue template, we need more information to help you.
Using Pytorch 0.3.0 solves the problem.
are you using python 2? AFAIK FileNotFoundError only exists in python 3, so this should be fixed Also, this error is hiding the real bug in the code, which seems to be the following: Did you mean to take the reciprocal of a LongTensor?
does the shutdown workers FileNotFoundError happen on python 2.7 as well?
I haven't tested on py2. But I believe it should be an with errno.
Yup I am using Python 2. To overcome this error of FileNotFoundError I have downgraded Pytorch from 0.4.0 version to 0.3.0. Use Pytorch 0.3.0 with Python 2, it will work fine.
alternatively, upgrading to Python 3 will fix the issue (and Python 3 is nicer than Python 2).
We'll fix this though.
For a quick fix you can make python2 compatible by defining FileNotFoundError somewhere
This error still exist in pytorch 0.4
Yes it is fixed after 0.4
I am still getting it...
did you build pytorch from source? It was fixed after 0.4 is released, so it will be in the next pytorch release.
I updated it via
Can anyone help me with my problem?
I was also working with faster r-cnn and got the error. Downgrading to 0.3.0 solved FileNotFoundError but will produce another error: self.ratiolistbatch[leftidx:(rightidx+1)] = ((np.float64)) # trainset ratio list ,each batch is same number TypeError: 'module' object is not callable Where this error is related to in the above error message. Upgrading can solve this error but will produce the previous one. How do you cope with the second error?
A = (5, 4) B = (0, 9).view(3, 3) C = (0, 15).view(3, 5) idxs = torch.LongTensor([0, 2, 4]) A.indexadd(0, idxs, B) # RuntimeError: inconsistent tensor size, expected r [4], t [4] and src [3] to have the same number of elements, but got 4, 4 and 3 elements respectively at (...)/aten/src/TH/generic/THTensorMath.c:1008 A.indexadd(0, idxs, C) # RuntimeError: inconsistent tensor size, expected r [4], t [4] and src [5] to have the same number of elements, but got 4, 4 and 5 elements respectively at (...)/aten/src/TH/generic/THTensorMath.c:1008 So far so good. But if we use CUDA... A = (5, 4).cuda() B = (0, 9).view(3, 3).cuda() C = (0, 15).view(3, 5).cuda() idxs = torch.LongTensor([0, 2, 4]).cuda() A.indexadd(0, idxs, B) print(A) # 0 1 2 0 # 0 0 0 0 # 3 4 5 0 # 0 0 0 0 # 6 7 8 0 # [ of size 5x4 (GPU 0)] OK, this looks wrong... A.zero() A.indexadd(0, idxs, C) print(A) # 0 1 2 3 # 4 0 0 0 # 5 6 7 8 # 9 0 0 0 # 10 11 12 13 # [ of size 5x4 (GPU 0)] Now this looks definitely wrong. Increase C's dimension to something like (3, 500), and it overwrites other tensors or triggers asserts. Same thing happens with indexcopy_.
I'll take it if no one's looking at it yet.
I am trying to build PyTorch from source on macOS High Sierra Version 10.13.3 with an NVIDIA GeForce GT 750M. I'm following the instructions at prints prints prints prints When I enter , as per the instructions, I receive the following error message:
Carlos - this may be minor but I eventually found problems with ananconda for this an other CUDA-enabled builds. Couple of things that may help: a - ensure that your activated conda environment is env actually the env you want. I was checking the path with and it was showing the incorrect path to python and pip half of the time. I ended up creating new conda environments (switching to miniconda by the way) and passing the python=3.6 flag to ensure a full environment was deployed. Maybe somehow reverts back to an incorrect path b - in your activated environment, c - make sure a recent OS update didn't bump your CommandLineTools to v 9.0 (I found 8.0 was the only that worked, not 8.1) I built this on 10.12, CUDA 9, cudNN 7.0.
Could you see if can build for you? I don't have CUDA 9 on mac, so it'd be great if you can let me know if it works or not. :)
I'm no longer getting an error at installation, but still returns for some reason.
right now if is not in LDLIBRARYPATH, though it was found at compile-time, these lines will fail: Avoid this, by idk doing something... I think i have a few good ideas.
We need to add /usr/local/cuda/lib64 (or that's mac dir) to the rpath of _C
the problem i dont think is to add it to rpath, it's because we
Right. We could try with the default cuda installation path if it's not found from LDLIBRARYPATH.
Actually, do we really want to use the compile path? We should try to detect it at runtime, right? Or are we going to ship CUDA libs in our binaries?
we first load using runtime paths, and if it fails, fallback to path known at compile time.
I fear this is going to get brittle, especially if you are shipping CUDA libs. e.g. a user swaps HW and picks up a new driver and/or toolkit. As says, I'm pretty sure you want to use the runtime paths. Similarly, CUDA 7.5 won't run on Pascal and we have a cudnn built for 7.5 that will not do the right thing on CUDA 8, so those need to go together. If a user installs the CUDA bits via deb/rpm the right things are supposed to happen. If you push bits, we need to figure out how to make sure things line up. But this is perhaps a much larger packaging conversation.
Yeah, I've implemented it to first try loading it without any path and then try and . I'll push the commit tomorrow.
My compiler (Apple LLVM version 8.1.0 (clang-802.0.42) complained about this, but I think it's reasonable. In the specialization THCNumerics<uint8tthe abs-function is defined in a selfrecursive way ie uint8t abs(uint8t a) { return abs(a); } Because an uint8t has no sign it should imo defined as uint8t abs(uint8t a) { return a; }
Yup that's an error. Can you please send a PR?
I am sorry. I am inexperienced with that and have not currently time. But I will soon invest some time in learning it.
I'm volunteer to push a patch for this issue.
I followed the guide The command was working just fine until a couple days ago, but now it returns I manually checked and there is no cu100/[...] What should I do? I have CUDA 10.0 I would like to use pytorch...
You could try as a temporary workground. cc
This is most likely related to this: I've re-uploaded the stable html to re-include the cu100 binaries, and submitted a PR to make sure this doesn't happen again
When converting a tensor to dlpack, the strides of single-value tensors () should be normalized to . This is the fallout conclusion from . Here is a demo of what is wrong: Collecting environment information... PyTorch version: 1.10.0a0+git2aa19f3 Is debug build: False CUDA used to build PyTorch: Could not collect ROCM used to build PyTorch: N/A OS: Ubuntu 20.04.4 LTS (x8664) GCC version: (GCC) 9.4.0 Clang version: 12.0.1 ( ) CMake version: version 3.21.1 Libc version: glibc-2.31 Python version: 3.9.6 (default, Jul 11 2021, 03:39:48) [GCC 9.3.0] (64-bit runtime) Python platform: Linux-5.4.0-122-generic-x8664-with-glibc2.31 Is CUDA available: False CUDA runtime version: 11.2.152 GPU models and configuration: GPU 0: NVIDIA GeForce GTX 1660 SUPER Nvidia driver version: 470.141.03 cuDNN version: /usr/lib/x8664-linux- HIP runtime version: N/A MIOpen runtime version: N/A Is XNNPACK available: False Versions of relevant libraries: [pip3] mypy==0.812 [pip3] mypy-extensions==0.4.3 [pip3] numpy==1.21.2 [pip3] torch==1.10.0a0+git2aa19f3 [pip3] torchaudio==0.10.0a0+ [conda] magma-cuda112 2.5.2 1 pytorch [conda] mkl 2021.3.0 h726a3e6557 conda-forge [conda] mkl-include 2021.3.0 h726a3e6557 conda-forge [conda] numpy 1.21.2 py39hdbf815f0 conda-forge [conda] torch 1.12.0a0+git1f29b31 dev0 <develop[conda] torchaudio 0.10.0a0+ pypi0 pypi
does this capture the problem?
As shown in below. Is this something expected? This seems most closely related to , but it is closed without any specific solutions.
Importing after also causes segv:
Happens with sklearn as well. . Reversing the order of imports seems to fix things for me though.
Can you reinstall sklearn or scipy? I think I've seen a case where that helped.
I've tried reinstalling things. Doesn't fix it. Could it be because scikit seems to build using openblas and not MKL?
I don't think so, it's a problem with numpy initialization. AFAIK the same was happening with TF, not sure if it's solved now.
same. run import scipy/ import sklearn after torch will cause error. but it's all right if load these two first.
Hi, I'm trying to create a module that, given an n-dimensional returns an n+2 dimensional arranged correspondingly from slices of a 3-dimensional . In theory this can be accomplished with plus a reshape, except that I need the matrix corresponding to the to be the identity matrix instead of a zero matrix. This could be accomplished by allowing the user to specify what default vector is returned corresponding to . Would that be possible? Thanks, Shawn
this is certainly possible to implement, we have to change our C backends slightly to accommodate this. will put it on a list of things to do
Closing this because it is possible to specify a custom value to use for the vector at . An example of doing so is in the .
Gradient changes, though inputs, outputs, and internal state stay same. Even when I run forward in evaluation mode and backward in training I still get the gradient changed. It works fine with cudnn disabled.
cudnn does not support backprop in evaluate mode, we should add a fallback to nn for this case
Hi, thanks for the report. It turns out that the cudnn library (at least up to v5) doesn't support backward in evaluation mode. We should assert that in the code.
Hi, thanks for the responses. What about another case when you run forward in evaluation mode and backward in training? It produces same issue, but seems to work fine in lua torch. Here is the code with changes: If there is another way to implement virtual batch normalization - please share.
this should be fixed in master. It will be incorporated into the next release that will have binaries late today.
Out of curiosity, why do backward in evaluate mode? (Or rather, what is virtual batch normalization?)
a lot of research involves this. For example neural style transfer, or etc.
Steps to reproduce: a CI CUDA docker image, e.g.,  PyTorch, and run Expected result: It works Actual result: Workaround: set forces only one to be passed.
CC is this the nccl+sccache problem you were seeing?
cc has the exact error. CI log doesn't actually post the error.
sccache probably doesn't allow passing multiple right now, I can look into it if needed.
I am having this same error. How can you figure out which gencode to pass? I'm trying to build from source from a Dockerfile as well, as seen in this gist:
this is super helpful, thanks.
just wanted to make sure you saw that this is pulling from a ppc64le (IBM Power system) docker image, so you'll run into different issues if trying to build on an x64 machine.
I modified it very slightly for x64: It seems to build pytorch successfully for me (on latest master I suppose).
looking into this
Should we add a ShortTensor Type, or just convert int16 to IntTensor ?
We already have a type available in pytorch. We have to enable conversion in the relevant function here:
Auto travis-ci failed at python2.7 :(
Do you have any plan to support int8 numpy conversion while you have too?
Char tensor uses char which is not guaranteed to be signed by the C standard. We'd need to change our C code to use a
I try , it complains , any idea?
The error message is quite self explanatory. PyTorch doesn't support tensors at the moment.
The SDPA tutorial is failing for in Google Colab when run with "CPU" only as the h/w accelerator. The tutorial has a note "If you don’t have a GPU and are running on CPU then the context manager will have no effect and all three runs should return similar timings." however things fail when run on CPU only. Code: Error: Collecting environment information. PyTorch version: 2.1.0+cu118 Is debug build: False CUDA used to build PyTorch: 11.8 ROCM used to build PyTorch: N/A OS: Ubuntu 22.04.2 LTS (x8664) GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 Clang version: 14.0.0-1ubuntu1.1 CMake version: version 3.27.7 Libc version: glibc-2.35 Python version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime) Python platform: Linux-5.15.120+-x8664-with-glibc2.35 Is CUDA available: False CUDA runtime version: 11.8.89 CUDAMODULELOADING set to: N/A GPU models and configuration: Could not collect Nvidia driver version: Could not collect cuDNN version: Probably one of the following: /usr/lib/x8664-linux- /usr/lib/x8664-linux- /usr/lib/x8664-linux- /usr/lib/x8664-linux- /usr/lib/x8664-linux- /usr/lib/x8664-linux- /usr/lib/x8664-linux- HIP runtime version: N/A MIOpen runtime version: N/A Is XNNPACK available: True CPU: Architecture: x8664 CPU op-mode(s): 32-bit, 64-bit Address sizes: 46 bits physical, 48 bits virtual Byte Order: Little Endian CPU(s): 2 On-line CPU(s) list: 0,1 Vendor ID: GenuineIntel Model name: Intel(R) Xeon(R) CPU @ 2.
20GHz CPU family: 6 Model: 79 Thread(s) per core: 2 Core(s) per socket: 1 Socket(s): 1 Stepping: 0 BogoMIPS: 4399.99 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constanttsc repgood nopl xtopology nonstoptsc cpuid tscknownfreq pni pclmulqdq ssse3 fma cx16 pcid sse41 sse42 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahflm abm 3dnowprefetch invpcidsingle ssbd ibrs ibpb stibp fsgsbase tscadjust bmi1 hle avx2 smep bmi2 erms invpcid rtm rdseed adx smap xsaveopt arat mdclear archcapabilities Hypervisor vendor: KVM Virtualization type: full L1d cache: 32 KiB (1 instance) L1i cache: 32 KiB (1 instance) L2 cache: 256 KiB (1 instance) L3 cache: 55 MiB (1 instance) NUMA node(s): 1 NUMA node0 CPU(s): 0,1 Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Mitigation; PTE Inversion Vulnerability Mds: Vulnerable; SMT Host state unknown Vulnerability Meltdown: Vulnerable Vulnerability Mmio stale data: Vulnerable Vulnerability Retbleed: Vulnerable Vulnerability Spec store bypass: Vulnerable Vulnerability Spectre v1: Vulnerable: __user pointer sanitization and usercopy barriers only; no swapgs barriers Vulnerability Spectre v2: Vulnerable, IBPB: disabled, STIBP: disabled, PBRSB-eIBRS: Not affected Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Vulnerable Versions of relevant libraries: [pip3] numpy==1.23.5 [pip3] torch==2.1.0+cu118 [pip3] torchaudio==2.1.
0+cu118 [pip3] torchdata==0.7.0 [pip3] torchsummary==1.5.1 [pip3] torchtext==0.16.0 [pip3] torchvision==0.16.0+cu118 [pip3] triton==2.1.0 [conda] Could not collect cc: cc
This operator has not been implemented for . We will push a PR to update the tutorial.
Hello I have a training loop running without any errors or warnings on GPUs using Pytorch 2.0.1. When I try to modify my learning loop to run on TPUs I get the error Pytorch XLA developers solved the problem. But after the problem is solved I get the warning below. /home/mfatih/env38/lib/python3.8/site- UserWarning: aten::reshape: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it. This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd. (Triggered internally at ) Variable.executionengine.runbackward( # Calls into the C++ engine to run the backward pass Pytorch XLA developers told me that the warning is related to Pytorch, not Pytorch-XLA. Is this warning an indication of the wrong backward computations? You can regenerate the error using the public . Please change the working directory on in the config file. Run the for debugging. After a successful short train and val cycle we get the warning. The warning occurs in operations placed in the functions in during backward pass. Here is my environment PyTorch version: 2.2.0.dev20231213+cpu Is debug build: False CUDA used to build PyTorch: None ROCM used to build PyTorch: N/A OS: Ubuntu 20.04.4 LTS (x8664) GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0 Clang version: Could not collect CMake version: version 3.26.0 Libc version: glibc-2.31 Python version: 3.8.10 (default, Nov 22 2023, 10:22:35) [GCC 9.4.0] (64-bit runtime) Python platform: Linux-5.13.
0-1027-gcp-x8664-with-glibc2.29 Is CUDA available: False CUDA runtime version: No CUDA CUDAMODULELOADING set to: N/A GPU models and configuration: No CUDA Nvidia driver version: No CUDA cuDNN version: No CUDA HIP runtime version: N/A MIOpen runtime version: N/A Is XNNPACK available: True CPU: Architecture: x8664 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian Address sizes: 46 bits physical, 48 bits virtual CPU(s): 96 On-line CPU(s) list: 0-95 Thread(s) per core: 2 Core(s) per socket: 24 Socket(s): 2 NUMA node(s): 2 Vendor ID: GenuineIntel CPU family: 6 Model: 85 Model name: Intel(R) Xeon(R) CPU @ 2.00GHz Stepping: 3 CPU MHz: 2000.186 BogoMIPS: 4000.37 Hypervisor vendor: KVM Virtualization type: full L1d cache: 1.5 MiB L1i cache: 1.5 MiB L2 cache: 48 MiB L3 cache: 77 MiB NUMA node0 CPU(s): 0-23,48-71 NUMA node1 CPU(s): 24-47,72-95 Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Mitigation; PTE Inversion Vulnerability Mds: Mitigation; Clear CPU buffers; SMT Host state unknown Vulnerability Meltdown: Mitigation; PTI Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and _user pointer sanitization Vulnerability Spectre v2: Mitigation; Retpolines, IBPB conditional, IBRSFW, STIBP conditional, RSB filling Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Mitigation; Clear CPU buffers;
SMT Host state unknown Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constanttsc repgood nopl xtopology nonstoptsc cpuid tscknownfreq pni pclmulqdq ssse3 fma cx16 pcid sse41 sse42 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahflm abm 3dnowprefetch invpcidsingle pti ssbd ibrs ibpb stibp fsgsbase tscadjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves arat mdclear archcapabilities Versions of relevant libraries: [pip3] numpy==1.24.1 [pip3] torch==2.2.0.dev20231213+cpu [pip3] torch-xla==2.2.0+git5577dd7 [pip3] torchaudio==2.2.0.dev20231213+cpu [pip3] torchmetrics==1.2.1 [pip3] torchsummary==1.5.1 [pip3] torchvision==0.18.0.dev20231213+cpu [conda] Could not collect cc
From the log it seems like it suggese XLA need to manually register the backward for reshape which is a bit weird. This is the first time I see this error, if you can shed some light on this issue that would be great!
Hey! this warning recently as a set of more testing to ensure custom ops are properly registered. This is expected to trigger if you register a backend kernel (XLA key) but no autograd imple (AutogradXLA) for a given op.
Since reshape() cannot have an autograd formula (it is sometimes a view and sometimes not), then you should register your kernel as a CompositeImplicitAutograd kernel (this will properly register onto AutogradXLA and remove this warning)
Thanks ! This make sense! I have a follow up question, we registered reshape and reshape symint in and This file will then get codegen into and . What's the correct way of marking to be ?
Ho I'm not familiar with how your codegen works... But in principles, I guess you want to do the same as NestedTensor here which uses for the dispatch key in do you have more context on this by any chance?
it's probably enough for you to just register that kernel to both the and dispatch keys (this is similar to what CompositeImplicitAutograd does, but it does it for all backends). I'm a bit confused though - Why do you need a custom reshape impl? And how are you going to get training support for it? (do you have a custom implementation of your reshape's backward)
Haha this is actually by you in for functionization. Do you think we still need this trick for functionization?
Yes - I think we should be able to remove it. Do you mind trying that? That should just run the existing kernel defined . And that should be identical to the current behavior, since the "registration" for reshape in XLA just calls back into ATen ()
I merged , let's wait and see if tmr's nightly will resolve this issue.
Thank you. I will test it and give feedback. Till now, I have run my experiments with the warning. Do I need to rerun the experiments?
OK, with the recent nighly versions of Pytorch and Pytorch-XLA I do not observe the warning anymore while running my experiments. Thank you. Should I rerun the experiments that were run with past Pytorch versions throwing the warning? Does this update only remove the warning or does it also change the calculations that I ran with the previous Pytorch versions? Can I rely on my previous experiments?
Sorry we have to revert the pr since it creates regression on nightly.
Hello I am using nightly without any problem. What should I do? Wait for new update?
if you don't see any regression in performance, you can likely just keep using today's nightly.