jiebi/CodeConvo huggingface dataset:
(old title) When build C extension, the error: FileNotFoundError was got. OS: Windows 10 pro PyTorch version: 0.4.0a0+ How you installed PyTorch (conda, pip, source): source Python version: 3.6.4 CUDA/cuDNN version: CUDA 9.0 GCC version (if compiling from source): msvc 14/15 (then compiling with CUDA I use VS2015, but when build extension, the program automatically use vs2017) When building c extension on Windows, I got the error: (The Chinese above means compiler success compile the library, and generated .lib and .exp) And the same error was got on a Linux Work Station. (gcc is 5.4.0)
In , copy the linked file from , for example , but in my Windows, and a Linux work station, the linked one is in: ,for example . So, there must be a bug, or error in pytorch's ffi or python's ffi. (pyhton 3.6.4, cffi 1.11.4), If it broke down because of change in cffi, I think I can create a PR. I do think this is because of the change of cffi api,
CC who enabled extension build for Windows on
Is this change documented in somewhere like Python SDK?
max pooling functions are not consistent with max functions. Below an example, every max pooling (be it 1d, 2d or 3d, adaptive or not) acts the same, on cpu or on cuda. Essentially, there are two fondamental differences : max pooling of all values is while for it's max pooling of nan and valid values is valid values, which means s get ignored, while for , as soon as there is a value, the result is . More generally, choosing explicetely how to deal with as in numpy () could be a solution, but maybe this is related to CuDNN's max pooling ? Built from latest sources (as of 05/17) PyTorch version: 0.5.0a0+ Is debug build: No CUDA used to build PyTorch: 9.1.85 OS: Ubuntu 16.04.4 LTS GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.9) 5.4.0 CMake version: version 3.9.4 Python version: 3.6 Is CUDA available: Yes CUDA runtime version: 9.1.85 GPU models and configuration: GPU 0: Quadro M1000M Nvidia driver version: 390.30 cuDNN version: Probably one of the following: /usr/lib/x8664-linux- /usr/lib/x8664-linux-gnu/libcudnnstaticv7.a Versions of relevant libraries: [conda] magma-cuda91 2.3.0 1 pytorch [conda] torch 0.5.0a0+ <pip[conda] torch 0.3.1b0+ <pip[conda] torch 0.5.0a0+ <pip
Hi, . Thank you for providing the example. I'll take a look.
Hmhm. I seem to get a double free in the ...
Another question: Do we really want to provide an option to ignore NaN? My view is that "if you have nan in your net, you're screwed", so I would just return NaN there, personally.
My practical usecase is for KITTI groundtruth depthmaps and FlowMaps which are 2D sparse arrays.
For an algorithm that use FlowNet-like architecture that outputs predictions at multiple scale levels, we can either compare predictions to downscaled GT or upscaled predictions to GT. The first being obviously less computationally expensive, the ignore NaN would help downscaling such sparse 2D maps. For the moment we do something I find ugly, you can see it Essentially it zeroes the s, takes the map and construct two maps of positive and negative values which are then maxpooled and back together. I am actually open for a strict "no-nan" policy on pooling functions, but in that case better enforce it before someone writes a code that tries to benefit from maxpooling ignore s feature/bug and if you have a clever way of pooling sparse 2D tensors, I'm open to it, but I guess it's a topic for pytorch forums ;)
Personally, I think it is more sane have NaN -NaN in the pooling and offer a parametrizable (where you get to pick the values) that does or so.
So the proposed fix does NaN -NaN similar to max. I didn't try to fix gradients for the nan case. This would involve keeping the values and I don't think that is worth it (in particular because I would not expect the pooling layer to be last, and otherwise we'd probably get NaN as grad_out). If you are reasonably happy with it, I'd move it to a PR.
+1 for NaN -NaN, "abyssus abyssum invocat" We could also add an optional mask which would be a ByteTensor of the same size, specifying whether or not the considered pixel is used for the pooling, that could be used for any kind of pulling. An functionality would then be to provide the mask
I could work with that. to continue to be picky, the potential drawback is that it won't work for other pooling methods such as average pooling or median pooling. The problem here is that the max operation inherently ignores nonmax values which can be leveraged for "ignore some pixels" operation, but it has a "non universal" feel to it since it woun't be as easy for other kinds of 2D operations.
To be clear, the fix is good for me, but I figured a related discussion on selective pooling (whether to ditch NaN values or anything you want to ignore) could happen (maybe not on this issue ?)
This issue tracks the components / tests that are not working on Windows: [ ] : currently disabled on Windows because Windows doesn't have and we need to look for substitutes [ ] : Fuser is disabled on Windows because the current implementation uses symbols from Linux-specific headers and . We will need to find alternatives for Windows. [ ] : some parts of and are disabled because Windows doesn't support opening an already opened file (see discussion at and ) [ ] : currently doesn't work with Windows ( is the porting diff) [ ] : in causes intermittent CUDA out-of-memory error on Windows [ ] , : DataLoader with multiple workers causes intermittent CUDA out-of-memory error on Windows. [x] [x] (done in ) [x] [x] - - [x] [x] [x] - For more discussions, also see: cc
The first one is solved by but I think DataLoader can be further improved when is fininshed.
Cool I will mark it as resolved :)
The has been in .
We can try to revert and , because the memory leak in the CPU side could also cause CUDA errors.
Are they fixed by I think we can revert them after we merge the PR.
I think this may be related. Since once the memory of the CPU side is low, the will fail with too.
I think it could be. For what it's worth, when I tried to inspect the CUDA OOM error, showed no process that was taking memory, but running CUDA tests on the machine would still fail.
Now that is merged into master, could you please try to revert the changes on and ?
Awesome! Just to understand it better: does fix both numworker=1 and numworker1 cases?
Yes, they are both solved.
I guess we should mark the cpp_extension test as completed, for it's now enabled in CI.
Closing this issue due to age and because its references are long out of date. For example, distributed tests now have their own jobs.
got the following err msg with conv2D and ver 0.4: RuntimeError: expected stride to be a single integer value or a list of 1 values to match the convolution dimensions, but got stride=[1, 1]. I was not able to see the problem so I uninstalled 0.4 and installed 0.3.1 Still got an err but this time it said that the expected input should be a 4d tensor and it got a 3D tensor. This helped understand the issue and I fixed it (just add a dimension). reinstalled 0.4 and its working (no surprise...). I think 0.4 should have the same err feedback otherwise its really hard to undrstand the problem How you installed PyTorch (conda, pip, source): conda Build command you used (if compiling from source): OS: macos PyTorch version: 0.4 Python version: 3.5 CUDA/cuDNN version: no GPU models and configuration: GCC version (if compiling from source): CMake version: Versions of any other relevant libraries:
Can you please post a small self-contained code snippet that would let us reproduce the problem?
The strange error messge was also mentioned a few times in the forum. Here is a code snippet for PyTorch :
This is because we direct all conv ops to and infer dim + throw error message there.
Hmm that's not great. We might want to pass the expected dimensionality of the convolution to the generic implementation, so that we can improve the error messages.
Thx for the responses. Just a reminder, for the same err the provided err msg in version 0.3.1 was very informative
Yes, this should definitely be fixed. Sorry about this!
The error message of seems to be a bit misleading, too. Code: Should I create a new issue or is it related to the current one?
that appears fixed on master! :D
gdb points that the error might be in trying to get a size from an empty tensor: I'm using PyTorch version 0.4.0a0+
There's a check that should exclude zero-dim tensors from ( should be 0 in this case), so I'm wondering why that's not happening right now... edit: Nevermind, I was running an old build. I pulled the latest master and with being an empty tensor (with shape (0,)), cat crashes.
Related: We should probably rewrite to better handle these cases
Okay, I found the bug. The CUDA version of check doesn't check the case where the input contains all empty tensors, while the CPU version does. I'll put up a fix soon.
The following code, which repeatedly exports a model with , has a memory leak. During the export, every tensor parameter in is cloned once and then immediately leaked forever, without ever being collected by the GC. It's not the underlying buffer that's cloned, it's the lightweight wrapper object itself. Still, for long running processes that often export networks in this manner this is a unbounded memory leak that eventually results in OOM errors. I've reproduced this issue on both Linux and Windows, with pytorch versions and respectively. The final five lines inside the for loop are to debug what happens, they are not neccesary to reproduce the issue. forces a gc collection cycle, ensuring we're not accidentally counting dead objects show the total amount of objects that exist for each type for all objects whose amount has increased. From this we can see that we're leaking 2 additional tensors per that the tensors we're leaking have shapes and , so they're just the weight and bias of the linear that the underlying buffer is always the same, so only the shallow class instance is being that nothing is pointing to these newly created objects, so they should be collected. Example output after running for a while: seems closely related but is more about a temporary doubling in memory, this issue is about a permanent memory leak. was closed as a duplicate of the previous issue, but better matches this issue. Collecting environment information. PyTorch version: 1.10.0 Is debug build: False CUDA used to build PyTorch: 11.3 ROCM used to build PyTorch: N/A OS: Ubuntu 18.04.5 LTS (x8664) GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0 Clang version: 6.0.0-1ubuntu2 (tags/RELEASE600/final) CMake version: version 3.10.2 Libc version: glibc-2.17 Python version: 3.7.11 (default, Jul 27 2021, 14:32:16) [GCC 7.5.0] (64-bit runtime) Python platform: Linux-5.15.0-41-generic-x8664-with-debian-buster-sid Is CUDA available: True CUDA runtime version: 11.3.
109 GPU models and configuration: GPU 0: NVIDIA GeForce RTX 3080 Ti GPU 1: NVIDIA GeForce RTX 3080 Ti Nvidia driver version: 515.48.07 cuDNN version: Probably one of the following: /usr/lib/x8664-linux- /usr/lib/x8664-linux- /usr/lib/x8664-linux- /usr/lib/x8664-linux- /usr/lib/x8664-linux- /usr/lib/x8664-linux- /usr/lib/x8664-linux- HIP runtime version: N/A MIOpen runtime version: N/A Is XNNPACK available: True Versions of relevant libraries: [pip3] numpy==1.21.2 [pip3] torch==1.10.0 [pip3] torchelastic==0.2.0 [pip3] torchtext==0.11.0 [pip3] torchvision==0.11.0 [conda] blas 1.0 mkl [conda] cudatoolkit 11.3.1 ha36c4319 nvidia [conda] ffmpeg 4.3 hf484d3e0 pytorch [conda] mkl 2021.3.0 h06a4308520 [conda] mkl-service 2.4.0 py37h7f8727e0 [conda] mklfft 1.3.1 py37hd3c417c0 [conda] mklrandom 1.2.2 py37h51133e40 [conda] numpy 1.21.2 py37h20f2e390 [conda] numpy-base 1.21.2 py37h79a11010 [conda] pytorch 1.10.0 py3.7cuda11.3cudnn8.2.00 pytorch [conda] pytorch-mutex 1.0 cuda pytorch [conda] torchelastic 0.2.0 pypi0 pypi [conda] torchtext 0.11.0 py37 pytorch [conda] torchvision 0.11.0 py37cu113 pytorch
I'm also observing something like this without JIT, although I'm not sure it's the same issue. The program works fine until I add ONNX export every time a checkpoint is saved. Once I do that, the GPU memory usage grows until it OOMs.
It seems that can change memory outside x, when x is a cuda tensor. If x is non-cuda tensor, we get: In contrast, when x is cuda tensor, does not make any error It's hard to share the whole code, but I have noticed that such operation outside a tensor did affect the performance of existing network, so I'm afraid that this op can change arbitrary memory on GPU which can be dangerous. Could you check this out?
This snippet is fine - it's enough for us to reproduce the problem. It appears we're missing some out-of-bounds checks (we have them for other indexing functions). Thanks for reporting.
working on this
I am new to pythonwhen i solve the promblem with the help below I find some confusion in the code I set ‘dimension=1self.dimension = dimension’it seem ok for mebut i don’t kown how the value of ’dimension‘ was initialled. Thank you !
I already Konw it comes from 'module = JoinTable(dimension, nInputDims)' But when I convert the model to pytorch , error appears: Traceback (most recent call last): File "", line 173, in <moduleGnetf =generator.forward(input) File "/usr/local/lib/python2.7/dist-", line 33, in forward return self.updateOutput(input) File "/usr/local/lib/python2.7/dist-", line 36, in updateOutput currentOutput = module.updateOutput(currentOutput) File "/usr/local/lib/python2.7/dist-", line 37, in updateOutput (dim, offset, (dim)).copy_(currentOutput) RuntimeError: inconsistent tensor size at /home/lxl/pytorch-master/torch/lib/TH/generic/THTensorCopy.c:51
I Use "generator.modules[0] = nn.JoinTable(1)",it was fine ,but error again: Traceback (most recent call last): File "", line 171, in <moduleGnetf =generator.forward(input) File "/usr/local/lib/python2.7/dist-", line 33, in forward return self.updateOutput(input) File "/usr/local/lib/python2.7/dist-", line 36, in updateOutput currentOutput = module.updateOutput(currentOutput) File "/usr/local/lib/python2.7/dist-", line 96, in updateOutput if is None: AttributeError: 'SpatialFullConvolution' object has no attribute 'finput'
How old is the Lua model file you're trying to import? Can you please try to load it in Lua, save again, and load it in PyTorch? Also, please update PyTorch to the newest version.
The model is convert from the cudnn model trained by myseft the code below is the convert code BTWmy torch was installed on 17th Dec,2016 the pytorch version i use Metadata-Version: 1.0 Name: torch Version: 0.1.10+ I Build it from source today
Hi There, I'm trying to install Pytorch as a module on my university's computing cluster. The node uses CentOS (6.4) x8664. I install running these commands: module load gcclib/5.2.0 cmake/3.8.2 module load gcc/5.2.0 module load anaconda3/4.0.0 export CMAKEPREFIXPATH="(which conda))/../" export NOCUDA=1 git clone --recursive cd pytorch/ python install --prefix=${HOME} Things look fine as it installs but then I run into the following errors, ending in the process terminating with error relating to GCC? I've tried with version GCC 6.2.0 and the same result occurs. Not sure what to even try to fix this! Thanks for any help you can provide!
Same here, with gcc 5.4.0
I don't think building from source works very well with gcc 5.4. Could you install gcc 4.9 and try compiling?
I get this error instead when using gcc 4.9.0, doesn't make it very far at all.
Hi I think this is broken after / due to
I have the same problem ( etc.) with gcc 4.8.5 on Linux with the current head ( ) What works for me is to add at the beginning of the following four files: (see also -- this can be turned into a pull request very easily) This fix is the same as but for different files.
This worked for me Thank you.
fixed in latest master, thanks to
Recently, we are testing PyTorch 1.7 as we need the with-statement support, so that our algorithm which contains a custom module using can be deployed to the production environment. During the testing, we encountered the following assertion failure: Steps to reproduce the behavior: the following Python code to create a serialized PyTorch module. a C++ test program. the test program with the following command: in the current directory. should terminate with exit code 0 instead of throw an exception. Collecting environment information... PyTorch version: 1.7.0.dev20200922+cpu Is debug build: True CUDA used to build PyTorch: None ROCM used to build PyTorch: N/A OS: CentOS Linux 7 (Core) (x8664) GCC version: (GCC) 4.8.5 (Red Hat 4.8.5-39) Clang version: Could not collect CMake version: Could not collect Python version: 3.7 (64-bit runtime) Is CUDA available: False CUDA runtime version: No CUDA GPU models and configuration: No CUDA Nvidia driver version: No CUDA cuDNN version: No CUDA HIP runtime version: N/A MIOpen runtime version: N/A Versions of relevant libraries: [pip3] numpy==1.19.2 [pip3] torch==1.7.0.dev20200922+cpu [pip3] torchvision==0.7.0+cpu [conda] Could not collect cc
This is a test. Please ignore it. Edited.
<!-- validation-comment-start --<bodyHello there! From the DISABLED prefix in this issue title, it looks like you are attempting to disable a job in PyTorch CI. The information I have parsed is below: Job name: Credential: Within ~15 minutes, and all of its dependants will be disabled in PyTorch CI. Please verify that the job name looks correct. With great power comes great responsibility. </body<!-- validation-comment-end --
When trying to install Pytorch on my Mac by following the instructions I get What I did: ` I also tried Both approaches gave the same error. System: xcode-select version 2395. Version: macOS Monterey 12.3.1 (21E258) MacBook Pro (16-inch, 2019) Processor: 2,6 GHz 6-Core Intel Core i7 memory: 16 GB 2667 MHz DDR4 PyTorch version: N/A Is debug build: N/A CUDA used to build PyTorch: N/A ROCM used to build PyTorch: N/A OS: macOS 12.3.1 (x8664) GCC version: Could not collect Clang version: 13.1.6 (clang-1316.0.21.2.3) CMake version: version 3.22.1 Libc version: N/A Python version: 3.9.12 (main, Apr 5 2022, 01:53:17) [Clang 12.0.0 ] (64-bit runtime) Python platform: macOS-10.16-x8664-i386-64bit Is CUDA available: N/A CUDA runtime version: Could not collect GPU models and configuration: Could not collect Nvidia driver version: Could not collect cuDNN version: Could not collect HIP runtime version: N/A MIOpen runtime version: N/A Is XNNPACK available: N/A Versions of relevant libraries: [pip3] numpy==1.21.5 [conda] mkl 2022.0.0 hecd8cb5105 [conda] mkl-include 2022.0.0 hecd8cb5105 [conda] numpy 1.21.5 py39h9c3cb841 [conda] numpy-base 1.21.5 py39he782bc11 cc
I am running the same environment and get the same issue. Any insight would be very appreciated
Update: In another repo I get the same error when trying to link to pytorch etc. There I made a minimal case and managed to build when I removed linking to . I can see that we have in the script. Maybe that is the cause of the error?
Looks like they are built with the correct architecture.
More progress, from local minimal case: fails. is just a hello world program.
If in is removed, then it builds.
The same story with PyTorch 1.10.0. The error appears when I'm trying to build with Apple clang 13.1.6 (Xcode Command Line Tools 13.3). But all works correctly if I build it with Apple clang 13.0 (Xcode Command Line Tools 13.2.1)
Nice, worked for me as well. Is this a bug somewhere or what is the exact problem? I drawback is that XCode needs to be up to date with new iOS versions.
Which version of Apple Clang worked? 13.0.0 or 13.0.1? Are you on Monterey 12.4?
Filed an issue: cc:
This issue has been fixed in PeachPy a while back by but pinned version of PeachPy that PyTorch is using has not been updated in a very long time