jiebi/CodeConvo huggingface dataset:
Change:
}; struct PythonEngine : public Engine { <del> virtual void thread_main(std::shared_ptr<ReadyQueue> queue) override { </del> <ins> virtual void thread_main(std::shared_ptr<ReadyQueue> queue, int device) override { </ins> // Create a PyThreadState, but release the GIL. This lets AutoGIL calls // inside thread_main acquire the GIL without having to create a new // PyThreadState each time. AutoGIL gil; AutoNoGIL no_gil; <del> Engine::thread_main(queue); </del> <ins> Engine::thread_main(queue, device); </ins> } virtual void thread_on_exception(FunctionTask& task, std::exception& e) override {

Discussion:
Consider following code, and run it with multi gpus (e.g. 4): It will output: That is, the default device would always be zero even in DataParallel. My pytorch version is 0.1.122. I think this is not the desired behavior. It would cause some troubles. I tried to insert my own cuda kernel into backward to calculate the gradients, it become very slow, and I fixed it by (grad.getdevice()). Anyway, I think currentdevice in forward and backward should be the same, or could anyone explain to me why they are different?
this seems like something we should fix: ?
<!-- drci-comment-start --:pagefacingup: Preview :pagefacingup: Preview :question: Need help or want to give feedback on the CI? Visit our Note: Links to docs will display an error until the docs builds have been completed. This comment was automatically generated by Dr. CI and updates every 15 minutes. <!-- drci-comment-end --
Change:
print('-- Detected CUDA at ' + CUDA_HOME) else: print('-- Not using CUDA') <del> if WITH_NCCL and SYSTEM_NCCL: print('-- Using system provided NCCL library') </del> <ins> if WITH_NCCL and WITH_SYSTEM_NCCL: print('-- Using system provided NCCL library at ' + NCCL_LIB_DIR + ', ' + NCCL_INCLUDE_DIR) </ins> elif WITH_NCCL: print('-- Building NCCL library') else:

Discussion:
Currently, you can have a libnccl on your system, and pytorch setup favors that instead of compiling one itself from source. The code that takes care of this is: However, this breaks gloo build because gloo gets a path to NCCL via:
I got the error during the installation: Is it related to this problem?
that's a dupe of
no, that's a consequence of , now you have to do before compiling. I think you don't want to do recursive submodule update, because that will pull in gtest and another copy of nccl.
Alternatively we can pass new flags to gloo to prevent it from using its own NCCL
Thanks works for me.
You can prevent it from using its own nccl, but its cmake still has to be able to find some nccl.
I think should be sufficient. I'm testing.
fixes it when we dont have an external NCCL and pytorch compiles it's own NCCL?
I don't see how NCCLEXTERNAL fixes anything, looks like in gloo NCCLEXTERNAL is used just to set dependency if gloo is compiling its own nccl (which we don't want in any case).
The only thing I tested when I was that the vendored nccl was not being built (even if its submodule was checked out); I tested this by adding an to one of the headers. I'll test with the other configurations.
Actually, there are a few bugs here. First, even though GLOOFLAGS looks like it's setting NCCLROOTDIR: it actually isn't because the gloo invocation is which expands to but only is passed to cmake. Suppose you fix this. nccl is still not going to be picked up, because INSTALLDIR is PyTorch's local install dir, but if you disabled nccl builds, there will never actually be a nccl here. So cmake is just going to do it's standard search to find a nccl somewhere. Inversely, if you ever built nccl, and then turn it off, we will keep picking up that locally built version until you delete it. So we shouldn't do that. Suppose you fix this.

Then I think gloo will (continue to) correctly pick up system nccl and be built with it, and the final thing to fix is that build of PyTorch csrc doesn't know how to search for nccl headers somewhere. This is probably related to . I wonder, maybe we shouldn't do any autodetection at all, and force the user to pass us the include and library paths of nccl if they want to build with system.
FYI, I committed a change to gloo to stop bundling NCCL itself. It will now happily pick up whatever NCCL you give it through NCCLROOTDIR or by trying to find it in the standard systems paths. See facebookincubator/gloo You do have to set to make Gloo try and find it and compile in support.
So to fix this, you should be able to not specify anything aside from if you want to use system NCCL, and add if you have some custom location (either a PyTorch bundled version or a custom install altogether).
Change:
going to be volatile. Volatility spreads accross the graph much easier than non-requiring gradient - you only need a **single** volatile leaf to have a volatile output, while you need **all** leaves to not require gradient to <del> have an output the doesn't require gradient. Using volatile flag you don't </del> <ins> have an output that doesn't require gradient. Using volatile flag you don't </ins> need to change any settings of your model parameters to use it for inference. It's enough to create a volatile input, and this will ensure that no intermediate states are saved.

Discussion:
Platforms: mac, macos This test was disabled because it is failing on master (). This fails again , so I disable this test pending the discussion in on how best to clean up these runners. cc
<!-- validation-comment-start --<bodyHello there! From the DISABLED prefix in this issue title, it looks like you are attempting to disable a test in PyTorch CI. The information I have parsed is below: Test name: Platforms for which to skip the test: mac, macos Within ~15 minutes, will be disabled in PyTorch CI for these platforms: mac, macos. Please verify that your test name looks correct, e.g., . To modify the platforms list, please include a line in the issue body, like below. The default action will disable the test for all platforms if no platforms list is specified. We currently support the following platforms: asan, linux, mac, macos, rocm, win, windows.</body<!-- validation-comment-end --
I don't think this is really sustainable (nor productive) as all tests will eventually fail with no space left on device error... Instead, we should be reclaiming space on those nodes...
Yeah, this is only a temporary measure while we continue the investigation in This test writes a 2GB file to disk I think. I'll bring up the issue again tomorrow to get help from the team. At the moment, I have no idea which one is responsible for filling up the disk.
mac M1 runners has been cleaned up, resolving this one
failing again
Got a fix here to increase the minimum space required on MacOS to be 6GB. The current threshold of 4GB is not big enough
Change:
IF ($ENV{PYTORCH_BINARY_BUILD}) MESSAGE(STATUS "PYTORCH_BINARY_BUILD detected. Statically linking libstdc++") SET(CMAKE_CXX_FLAGS "-static-libstdc++ ${CMAKE_CXX_FLAGS}") <ins> IF (UNIX AND NOT APPLE) # hiding statically linked library symbols, this flag is not available for the linker under MACOSX SET(CMAKE_CXX_FLAGS "-Wl,--exclude-libs,ALL ${CMAKE_CXX_FLAGS}") ENDIF(UNIX AND NOT APPLE) </ins> ENDIF() ADD_LIBRARY(shm SHARED core.cpp)

Discussion:
see the commit comment: To reproduce install ros-kinetic-tf 1.11.8-0xenial---0800 package from the deb xenial main repository. my system is: lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 16.04.1 LTS Release: 16.04 Codename: xenial cuda version is 8.0
Change:
self.assertRaisesRegex(RuntimeError, 'modified by an inplace operation', lambda: z.backward()) <ins> def test_no_grad_input(self): class MyFunction(Function): @staticmethod def forward(self, x): return x @staticmethod def backward(self, grad_output): return grad_output x = torch.randn(5, requires_grad=True) with torch.no_grad(): y = MyFunction.apply(x) self.assertTrue(x.requires_grad) self.assertIsNone(y.grad_fn) </ins> def test_backward_copy(self): # This tests checks backward engine for a very subtle bug that appreared # in one of the initial versions of autograd. Gradients tensors were

Discussion:
OS: ubuntu 16.04 PyTorch version: 0.4.0a0+ How you installed PyTorch (conda, pip, source): source Python version: 3.6.3 CUDA/cuDNN version: v9.1, cudnn-9.1-linux-x64-v7 GPU models and configuration: 4x Tesla M60 GCC version (if compiling from source): 5.4.0 When using context () on a model with dataparallel operation (same with ) on 1 GPU the flag requires_grad does not come back to True. If only one GPU is used then everything works fine. Script: On 2 GPUs, this outputs which is wrong in the last line. On 1 GPU (set ) the output is correct.
is looking into it
Change:
dim_args_idx = test[4] if len(test) == 5 else [] <ins> skipTestIf = test[5] if len(test) == 6 else [] </ins> for dim_perm in product([-1, 1], repeat=len(dim_args_idx)): test_name = basic_test_name new_constructor_args = [arg * dim_perm[dim_args_idx.index(i)] if i in dim_args_idx else arg

Discussion:
Command to install the latest nightly binary: installs : However, PyTorch itself reports : I guess this version tag might be defined during the build process from e.g. and then forwarded from , but also source builds should use based on the tag. Unsure, if this is just a reporting issue or if we are building the nightly binaries with and executing them with , so creating this issue to dig a bit further into it. CC cc
This is really weird, I've put up measure in place to validate against it... Will investigate today (and moving to pytorch repo)
can you please check that 2.2.0 reports (and is build) against the correct version of nccl library?
Confirmed for nightlies : Same issue for RC:
Looking at the build log Looks like we are compiling with: USESYSTEMNCCL
On Windows I see following exception: We compiling with should we return None rather then exception ?
In this case we are indeed using 2.18.1: Do you know why we are using the system-wide NCCL installation instead of the one? Should we delete the system NCCL and use by default?
Looks like its set here: Looks like it was by: Let me propose the PR to unset this flag : USESYSTEM_NCCL
I guess we are using the system NCCL installation to be able to during the runtime. If we would use the module I assume we would link NCCL into the PyTorch binaries. If so, we should make sure to update the NCCL version whenever is updated. This could break our dependency on the NCCL pip package.
Yes. Also when we update this nccl here:
We used following path  What would be good path for nccl 2.19.3 and cuda 12.1 ? As per this issue: Will be using 12.0 ncck 2.19.3
Installing system NCCL into the nightly build docker container feels great, but we need to figure out, how to keep it in sync with PyTorch. I wonder if one can, for example, just check nccl version from this requirements file and build from source?
yes probably we should build and upload nccl for our supported cuda versions, same way as we build docker containers, by following details here:
Verified with final rc
Change:
{ char transpose_r, transpose_m1, transpose_m2; THTensor *r__, *m1_, *m2_; <ins> int free_m1 = 0; int free_m2 = 0; </ins> if( (m1->nDimension != 2) || (m2->nDimension != 2)) THError("matrices expected, got %dD, %dD tensors", m1->nDimension, m2->nDimension);

Discussion:
Here is an example of what seems to be a memory leak in pytorch, although it could be numpy? The leak was only occurring when using the or array unsqueezing methods in numpy. Also worth noting that the issue is resolved by removing the from and casting the torch tensor to type. An easy workaround is to reshape the array after turning it into a torch Tensor.
Change:
raise assert not hasattr(TestAutograd, test_name), 'Two tests have the same name: ' + test_name <ins> for skip in skipTestIf: do_test = skip(do_test) </ins> setattr(TestAutograd, test_name, do_test)

Discussion:
Command to install the latest nightly binary: installs : However, PyTorch itself reports : I guess this version tag might be defined during the build process from e.g. and then forwarded from , but also source builds should use based on the tag. Unsure, if this is just a reporting issue or if we are building the nightly binaries with and executing them with , so creating this issue to dig a bit further into it. CC cc
This is really weird, I've put up measure in place to validate against it... Will investigate today (and moving to pytorch repo)
can you please check that 2.2.0 reports (and is build) against the correct version of nccl library?
Confirmed for nightlies : Same issue for RC:
Looking at the build log Looks like we are compiling with: USESYSTEMNCCL
On Windows I see following exception: We compiling with should we return None rather then exception ?
In this case we are indeed using 2.18.1: Do you know why we are using the system-wide NCCL installation instead of the one? Should we delete the system NCCL and use by default?
Looks like its set here: Looks like it was by: Let me propose the PR to unset this flag : USESYSTEM_NCCL
I guess we are using the system NCCL installation to be able to during the runtime. If we would use the module I assume we would link NCCL into the PyTorch binaries. If so, we should make sure to update the NCCL version whenever is updated. This could break our dependency on the NCCL pip package.
Yes. Also when we update this nccl here:
We used following path  What would be good path for nccl 2.19.3 and cuda 12.1 ? As per this issue: Will be using 12.0 ncck 2.19.3
Installing system NCCL into the nightly build docker container feels great, but we need to figure out, how to keep it in sync with PyTorch. I wonder if one can, for example, just check nccl version from this requirements file and build from source?
yes probably we should build and upload nccl for our supported cuda versions, same way as we build docker containers, by following details here:
Verified with final rc
Change:
np.int64, np.int32, np.int16, <del> np.uint8 </del> <ins> np.uint8, np.longlong, </ins> ] for dtype in dtypes: array = np.array([1, 2, 3, 4], dtype=dtype)

Discussion:
Minimal example: Torch initializes the tensor with values which are different from the numpy array. I've seen issues for similar kinds of problems (see ). Maybe the easiest solution to all these initialization problems might be, to check for the same values of the original and initialized array/tensor instead of initializing tensors silently with wrong numbers. What do you think? Furthermore, I've opened an issue which is also related this one in terms of initializing a numpy array with a number of type and then initializing a torch tensor from that array, see Feel free to merge them if you think that both issues can be solved together.
Could you please provide some information about what version of pytorch and numpy you're using? ! It works as expected for me on master
I'm using torch 0.3.0.post4 (latest one on website of PyTorch) and numpy 1.13.3.
I'm running the same thing, this is strange. In the first line , I assumed that came from . Is this the case?
Nope, it's indeed the python built-in type. Actually this issue came up when I tried to get the shape of (i.e. , , returns ).
Thanks for the clarification, I'm now seeing your results. Looks like the tensor constructor doesn't work well with python2.7 long types in this case
Sure, no problem. Thanks for looking into it!
The problem seems to be that np.longlong and np.int64 are the same thing (at least on a 64bit system, I think), but the NPYLONGLONG and NPYINT64 macros are different... I'll put up a pr for this soon.

A minimal example: Although has and this type is listed as supported, the array can't be converted. Consider that is initialized with an upper bound of type . I'm not completely sure, whether this issue is related to torch or numpy, but numpy initializes the array with values as expected, whereas torch throws an inconsistent error message.
Edit: nevermind, I didn't run the same exact thing ! Works for me on master and 0.3. Seems like it could be related to
Change:
(IndexCopy,     (0,),               ((S, S), index_variable(2, S), (2, S))      ), (IndexFill,     (0, 2),             ((S, S), index_variable(2, S))              ), (IndexSelect,   (0,),               ((S, S), index_variable(2, S))              ), <ins> (Gather,        (0,),               ((M, S), gather_variable((S, S), 1, M))     ), (Gather,        (1,),               ((M, S), gather_variable((M, S//2), 0, S)), 'dim1'), (Scatter,       (0,),               ((M, S), gather_variable((S, S), 1, M), (S, S))), (Scatter,       (1,),               ((M, S), gather_variable((M, S//2), 0, S), (M, S//2)), 'dim1'), </ins> (Concat,        (0,),               ((1, S, S), (2, S, S), (3, S, S))           ), (Resize,        (S*S, S),           ((S, S, S),)                                ), (Diag,          (),                 ((S, S),),                  '2d'            ),

Discussion:
<!-- A clear and concise description of what content in is an issue. If this has to do with the general website, please file an issue at instead. If this has to do with , please file an issue at --In the getting started section section the following quote appears: I guess this is from previous versions where CUDA support was not built-in and this can be removed, correct? cc
This seems to be easy fixable docs issue
If gets merged this is fixed.
Thanks for the PR Just merged it.
Change:
lib_path = os.path.join(cwd, "torch", "lib") <del> # Check if you remembered to check out submodules def check_file(f): if not os.path.exists(f): print("Could not find {}".format(f)) print("Did you run 'git submodule update --init'?") sys.exit(1) check_file(os.path.join(lib_path, "gloo", "CMakeLists.txt")) check_file(os.path.join(lib_path, "nanopb", "CMakeLists.txt")) check_file(os.path.join(lib_path, "pybind11", "CMakeLists.txt")) </del> tmp_install_path = lib_path + "/tmp_install" include_dirs += [ cwd,

Discussion:
It seems like running has a dependency on first running . This command also seems to also error out when installing via a wheel. This creates a problem for tools like  which rely on the output of to examine dependencies. Is there anyway to allow this command to not error out when installing via a wheel? Full stack trace: I understand the preferred package manager for this project is conda but being officially endorsed from , pipenv has a fairly large audience:
thanks do you know when's the next planned release?
Hopefully within a month
Change:
THCDeviceTensor<Dtype, 1> gradOutput, THCDeviceTensor<Dtype, 2> gradInput, Dtype *weights, <ins> int n_classes, </ins> int ignore_index) { CUDA_KERNEL_LOOP(index, batch_size) {

Discussion:
Claim: When I use an invalid label with F.crossentropy with reduce=False, I don't receive an error message. I do receive an error message when reduce=True. Code: Result: No error message received. Expected error message: Context: I was using , accidentally used an invalid label, and got extremely large losses and extremely small losses, and inconsistently so. Environment: OS: Ubuntu 16.04 on Docker and host PyTorch version: 0.3.0.post4 How you installed PyTorch: conda Python version: 3.6.3
Change:
if (is_padding_neg()) throw std::runtime_error("negative padding is not supported"); if (is_output_padding_neg()) throw std::runtime_error("negative output_padding is not supported"); <del> AutoGPU guard(input_.data->getDevice()); auto input = input_.unpack_data()->contiguous(); </del> <ins> auto input = input_.unpack_data(); AutoGPU guard(input->getDevice()); input = input->contiguous(); </ins> std::unique_ptr<Tensor> weight(weight_.unpack_data()->clone_shallow()); auto bias = bias_.unpack_data(); auto grad_output = grad_outputs[0]->data->contiguous();

Discussion:
Usually calling backwards twice gives this message: But conv2d just segfaults. $ pt A B Segmentation fault (core dumped)
Change:
{"_cuda_getDevice",   (PyCFunction)THCPModule_getDevice_wrap,   METH_NOARGS,  NULL}, {"_cuda_getDeviceCount", (PyCFunction)THCPModule_getDeviceCount_wrap, METH_NOARGS, NULL}, {"_cuda_getDeviceName", (PyCFunction)THCPModule_getDeviceName_wrap, METH_O,   NULL}, <ins> {"_cuda_getDeviceCapability", (PyCFunction)THCPModule_getDeviceCapability_wrap, METH_O,   NULL}, </ins> {"_cuda_getCurrentStream", (PyCFunction)THCPModule_getCurrentStream_wrap, METH_NOARGS, NULL}, {"_cuda_getCurrentBlasHandle", (PyCFunction)THCPModule_getCurrentBlasHandle_wrap, METH_NOARGS, NULL}, {"_cuda_setStream",    (PyCFunction)THCPModule_setStream_wrap,  METH_O, NULL}, {"_cuda_isDriverSufficient", (PyCFunction)THCPModule_isDriverSufficient, METH_NOARGS, NULL}, {"_cuda_getDriverVersion", (PyCFunction)THCPModule_getDriverVersion, METH_NOARGS, NULL}, <ins> {"_cuda_getCompiledVersion", (PyCFunction)THCPModule_getCompiledVersion, METH_NOARGS, NULL}, </ins> {"_cuda_getRNGState", (PyCFunction)THCPModule_getRNGState,      METH_NOARGS,  NULL}, {"_cuda_setRNGState", (PyCFunction)THCPModule_setRNGState,      METH_O,       NULL}, {"_cuda_emptyCache", (PyCFunction) THCPModule_emptyCache,       METH_NOARGS,  NULL},

Discussion:
There's quite a few user-reports of installing the CUDA 7.5 binary of pytorch and then running on GTX 1080 / 1070, and there will be reports of installing 7.5 / 8.0 binaries and running on V100. Raising a user-warning in such situations and letting them know will fix the problem.
The version of CUDA used to compile the binary should be available thanks to this PR:
I think is a more robust way of getting the CUDA used to compile PyTorch, as currently if there are multiple CUDA installed in the machine, the heuristics used in might not match those used by CMake FindCUDA. I think that we should modify the to return the cuda version used during compilation. I have the impression that it will not require changing much because doesn't require cuda to execute. What do you think?
Change:
extra_link_args = [] if IS_WINDOWS: <del> extra_compile_args = ['/Z7', '/EHa', '/DNOMINMAX' </del> <ins> extra_compile_args = ['/Z7', '/EHa', '/DNOMINMAX', '/wd4267', '/wd4251', '/wd4522', '/wd4522', '/wd4838', '/wd4305', '/wd4244', '/wd4190', '/wd4101', '/wd4996', '/wd4275' </ins> # /Z7 turns on symbolic debugging information in .obj files # /EHa is about native C++ catch support for asynchronous # structured exception handling (SEH) # /DNOMINMAX removes builtin min/max functions <ins> # /wdXXXX disables warning no. XXXX </ins> ] if sys.version_info[0] == 2: # /bigobj increases number of sections in .obj file, which is needed to link

Discussion:
That's pretty big! In contrast, an OS X build log is only 1.3M. Is there a reason they have to be this big? CC
My guess is that it's because of warnings. I remember having to get through thousands of them when looking for the error that failed the build.
Yes, the major part of them are warnings. And some of them is not fixable. We can hide them using some flags on the compilers.
And this build configuration actually contains both the CPU and GPU build, which may also be a possible reason.
The reduced log size after PR is about 1.2M.
Change:
0.0000  0.0000 -1.6250 [torch.FloatTensor of size 3x3] <del> >>> torch.triu(a, k=1) </del> <ins> >>> torch.triu(a, diagonal=1) </ins> 0.0000  1.7304  1.4573 0.0000  0.0000 -0.1809 0.0000  0.0000  0.0000 [torch.FloatTensor of size 3x3] <del> >>> torch.triu(a, k=-1) </del> <ins> >>> torch.triu(a, diagonal=-1) </ins> 1.3225  1.7304  1.4573 -0.3052 -0.3111 -0.1809

Discussion:
The states that the argument controls which diagonal to consider. However, this argument is actually named in pytorch 0.2.0.
Change:
torch.mm(a, b, out=self.output) else: if self.transA: <del> a = a.transpose(2, 3) </del> <ins> a = a.transpose(1, 2) </ins> if self.transB: <del> b = b.transpose(2, 3) </del> <ins> b = b.transpose(1, 2) </ins> self.output.resize_(a.size(0), a.size(1), b.size(2)) torch.bmm(a, b, out=self.output)

Discussion:
Hi I think that in lines 28 and 30: if self.transA: a = a.transpose(2, 3) if self.transB: b = b.transpose(2, 3) should be: if self.transA: a = a.transpose(1, 2) if self.transB: b = b.transpose(1, 2) Indeed in that branch the tensor has 3 dimension and thus the code crashes. Maybe an indexing error when translating from lua ? Thank you very much,
Yes, you're right. Could you send a PR with a fix? Thanks!
Yes of course, just done, thank you very much.