jiebi/CodeConvo huggingface dataset:
Change:
void evaluate_function(FunctionTask& task); ReadyQueue& ready_queue(int device); void start_threads(); <del> virtual void thread_main(std::shared_ptr<ReadyQueue> queue); </del> <ins> virtual void thread_main(std::shared_ptr<ReadyQueue> queue, int device); </ins> virtual void thread_on_exception(FunctionTask& task, std::exception& e); std::once_flag start_threads_flag;

Discussion:
Consider following code, and run it with multi gpus (e.g. 4): It will output: That is, the default device would always be zero even in DataParallel. My pytorch version is 0.1.122. I think this is not the desired behavior. It would cause some troubles. I tried to insert my own cuda kernel into backward to calculate the gradients, it become very slow, and I fixed it by (grad.getdevice()). Anyway, I think currentdevice in forward and backward should be the same, or could anyone explain to me why they are different?
this seems like something we should fix: ?
<!-- drci-comment-start --:pagefacingup: Preview :pagefacingup: Preview :question: Need help or want to give feedback on the CI? Visit our Note: Links to docs will display an error until the docs builds have been completed. This comment was automatically generated by Dr. CI and updates every 15 minutes. <!-- drci-comment-end --
Change:
GCC version: {gcc_version} Clang version: {clang_version} CMake version: {cmake_version} <ins> Libc version: {libc_version} </ins> Python version: {python_version} <ins> Python platform: {python_platform} </ins> Is CUDA available: {is_cuda_available} CUDA runtime version: {cuda_runtime_version} GPU models and configuration: {nvidia_gpu_models}

Discussion:
TensorPipe uses some advanced features of the Linux kernel. It does so on a "best effort" basis (i.e., use them if available, fall back on something else if not). This logic is unfortunately hard to test because it would require specific kernel versions on our CI machines. So it occasionally happens that we find out about issues when users report them. To help investigate these reports we've found it useful to know the kernel version and the glibc version used by the user (both when building and when running, if they're different). It would be great if these could be to the report produced by collect_env.
CC
Change:
DESTINATION "${TH_INSTALL_BIN_SUBDIR}") ENDMACRO(Install_Required_Library libname) <del> IF (BLAS_FOUND AND BLAS_INSTALL_LIBRARIES) IF (BLAS_goto2_LIBRARY) Install_Required_Library(${BLAS_goto2_LIBRARY}) Install_Required_Library("${libpath}/libgfortran") Install_Required_Library("${libpath}/libquadmath") Install_Required_Library("${libpath}/libgcc") </del> <ins> FIND_PACKAGE(BLAS) IF(BLAS_FOUND) SET(USE_BLAS 1) IF(BLAS_INFO STREQUAL "mkl") ADD_DEFINITIONS(-DTH_BLAS_MKL) </ins> ENDIF() <del> IF (BLAS_openblas_LIBRARY) Install_Required_Library(${BLAS_openblas_LIBRARY}) Install_Required_Library("${libpath}/libquadmath") Install_Required_Library("${libpath}/libgfortran") Install_Required_Library("${libpath}/libgcc") ENDIF() ENDIF() </del> <ins> ENDIF(BLAS_FOUND) </ins> FIND_PACKAGE(LAPACK) IF(LAPACK_FOUND)

Discussion:
currently compiled master does not use any blas for any ops and uses our lame fallback C code. I'm looking into it. Basically USE_BLAS is not being set in
Change:
else: self.gradInput = self.gradInput if not wasTable else input.clone() for i, module in enumerate(self.modules): <del> currentGradInput = getattr(module, method)(input, gradOutput[i], scale) </del> <ins> if method == 'updateGradInput': currentGradInput = module.updateGradInput(input, gradOutput[i]) elif method == 'backward': currentGradInput = module.backward(input, gradOutput[i], scale) </ins> if i == 0: self.gradInput.resize_as_(currentGradInput).copy_(currentGradInput) else:

Discussion:
Subclassing torch.Tensor with fails due to a on the latest nightly builds, whereas it previously worked without issue. Run this snippet of code: And you'll recieve an error message like this: Changing the to results in the exact same error message. There should be no error message as it should be working like it was previously in Torch 1.8.1 and below. I think that the issue started on Torch version: , and I can confirm that it still exists as of , . The issue started on the nightly builds at least 2 days ago, if that helps resolve things. It's been causing some of Captum's CircleCI tests to fail, and the tests show the first occurrence of the error on . cc
This is due to either or
I think the problem is I'm not supposed to delete because that means the super() call doesn't work. Instead, I should just verify that the passed in class is a subclass, then this should work.
Change:
from contextlib import contextmanager import torch <ins> import torch.cuda </ins> import torch.distributed as dist from common import TestCase

Discussion:
is causing a crash when we try to build pytorch. Crash looks like this: (all times in US/West) Our 7AM, 8AM builds and earlier were all fine but 9AM, 10AM, 11AM builds all crashed with that same traceback. The new import was in which I think went in between 8AM and 9AM today See command below: See here if needed for : <!-- A clear and concise description of what you expected to happen. --Please copy and paste the output from our (or fill out the checklist below manually). You can get the script and run it with: PyTorch Version (e.g., 1.0): building from master OS (e.g., Linux): linux How you installed PyTorch (, , source): source Build command you used (if compiling from source): see above under 'reproduce' Python version: 3.6
maybe easier to read for build command used:
please try in the CircleCi environment?
in our build script seems to have fixed this. Thanks !
Change:
[13, 14, 15, 16], [17, 18, 19, 20], ]) <del> exp_v = self.ValueTensor([7, 14, 14, 3, 20]) </del> <ins> exp_v = self.ValueTensor([7, 14, 3, 20]) </ins> res = dense._sparse_mask(x) expected = self.SparseTensor(i, exp_v, torch.Size([5, 4])) self.assertEqual(res, expected)

Discussion:
and I talked about the intended semantics of sparsemask when given a sparse tensor as an argument, and we came to the following conclusions: with an uncoalesced sparse tensor does not make sense, full stop. Unlike values, where your update might be linear and work correctly on an uncoalesced sparse tensor, there isn't any good reason why you want to pluck out values from the dense tensor multiple times, it is basically always a bug. the (eventual) documentation for sparsemask, we will recommend calling coalesce before doing any operations involving sparsemask (or nonlinear updates to indices/values). should add a test to sparsemask which checks if the flag of the input tensor is true, and raise an error if it is not. This is a cheap safety check which will prevent people from forgetting to call coalesce (as in (2)). There is a performance consequence to calling coalesce every iteration of training. If the gradient is producing coalesced vectors, we can alleviate this problem by adding a preprocessing step to coalesce() which tests if the input is already coalesced, and bugs out in that case. However, I think this is out of scope for this particular ticket for now.
Change:
return new_var; } <ins> const char* ERR_BACKWARD_TWICE = "Trying to backward through the graph a second time, but the buffers have " "already been freed. Specify retain_variables=True when calling backward " "the first time."; </ins> }} // namespace torch::autograd

Discussion:
Usually calling backwards twice gives this message: But conv2d just segfaults. $ pt A B Segmentation fault (core dumped)
Change:
} // Replace with nonzeros auto nonzero = index.nonzero(); <del> for (int64_t j = 0; j < nonzero.size(1); j++) { result.emplace_back(nonzero.select(1, j)); </del> <ins> auto is_empty = nonzero.numel() == 0; for (int64_t j = 0; j < index.dim(); j++) { if (is_empty) { // We can't call select on an empty tensor so we just create an empty // tensor. result.emplace_back(nonzero.type().tensor()); } else { result.emplace_back(nonzero.select(1, j)); } </ins> } } else { result.emplace_back(index);

Discussion:
In PyTorch master 0.4 , variables cannot be indexed. This used to work in PyTorch 0.2 :
This is breaking some of Pyro's inference algorithms on PyTorch master:
Change:
idx = Variable(torch.LongTensor()) self.assertEqual(x[idx].numel(), 0) <ins> # empty assignment should have no effect but not throw an exception y = x.clone() y[idx] = -1 self.assertEqual(x, y) mask = torch.zeros(4, 3).byte() y[mask] = -1 self.assertEqual(x, y) </ins> def test_basic_advanced_combined(self): # From the NumPy indexing example x = Variable(torch.arange(0, 12).view(4, 3))

Discussion:
odd difference between cuda Tensor and Variable: encountered in torch.distributions.Categorical, built yesterday from master
related to hmm, here's the issue is for variables, so probably not (immediately) related
different code path but similar issue
This is a regression from the latest 0.3 binaries, which don't have this error.
Change:
assert_strict_equal(zc, z) <del> def index_variable(num_indices, max_indices): index = torch.randperm(max_indices)[:num_indices].long() </del> <ins> def index_variable(shape, max_indices): if not isinstance(shape, tuple): shape = (shape,) index = torch.rand(*shape).mul_(max_indices).floor_().long() return Variable(index, requires_grad=False) def gather_variable(shape, index_dim, max_indices): assert len(shape) == 2 assert index_dim < 2 batch_dim = 1 - index_dim index = torch.LongTensor(*shape) for i in range(shape[index_dim]): index.select(index_dim, i).copy_( torch.randperm(max_indices)[:shape[batch_dim]]) </ins> return Variable(index, requires_grad=False)

Discussion:
<!-- A clear and concise description of what content in is an issue. If this has to do with the general website, please file an issue at instead. If this has to do with , please file an issue at --In the getting started section section the following quote appears: I guess this is from previous versions where CUDA support was not built-in and this can be removed, correct? cc
This seems to be easy fixable docs issue
If gets merged this is fixed.
Thanks for the PR Just merged it.
Change:
result = x[rows[:, None], columns] self.assertEqual(result.data.tolist(), [[0, 2], [9, 11]]) <ins> def test_empty_index(self): x = Variable(torch.arange(0, 12).view(4, 3)) idx = Variable(torch.LongTensor()) self.assertEqual(x[idx].numel(), 0) </ins> def test_basic_advanced_combined(self): # From the NumPy indexing example x = Variable(torch.arange(0, 12).view(4, 3))

Discussion:
In PyTorch master 0.4 , variables cannot be indexed. This used to work in PyTorch 0.2 :
This is breaking some of Pyro's inference algorithms on PyTorch master:
Change:
def forward(ctx, input, grid, padding_mode='zeros'): ctx.save_for_backward(input, grid) <ins> if input.device != grid.device: raise RuntimeError(("input (device {}) and grid (device {}) must be on the same device" + "for grid_sampler").format(input.device, grid.device)) </ins> if padding_mode == 'zeros': ctx.padding_mode = MODE_ZEROS elif padding_mode == 'border':

Discussion:
yields irrelevant errors when passing wrong arguments First one: Second one: PyTorch or Caffe2: pytorch 0.4 How you installed PyTorch (conda, pip, source): pip Build command you used (if compiling from source): OS: ubuntu16.04 PyTorch version: 0.4 Python version: 3.6 CUDA/cuDNN version: GPU models and configuration: 1080 GCC version (if compiling from source): CMake version: Versions of any other relevant libraries:
Thank you for the report! I'll move them to the new check for same GPU (which includes reasonable CPU vs. GPU reporting).
In PyTorch master, the error message has already improved for the second case:
I am getting exactly the first error reported: and I don't understand what is the problem. Can someone help on this?
if you use PyTorch v0.4.1, you should get a better error message helping you on what's wrong.
Thanks for your reply I found out that the problem was the hidden state in my recurrent net, I did not use the .cuda()
Change:
IF(MSVC) # we want to respect the standard, and we are bored of those **** . ADD_DEFINITIONS(-D_CRT_SECURE_NO_DEPRECATE=1) <del> LIST(APPEND CUDA_NVCC_FLAGS "-Xcompiler /wd4819 -Xcompiler /wd4503") </del> <ins> LIST(APPEND CUDA_NVCC_FLAGS "-Xcompiler /wd4819 -Xcompiler /wd4503 -Xcompiler /wd4190 -Xcompiler /wd4244 -Xcompiler /wd4251 -Xcompiler /wd4275 -Xcompiler /wd4522") </ins> ADD_DEFINITIONS(-DTH_EXPORTS) IF (NOT NO_CUDA) ADD_DEFINITIONS(-DTHC_EXPORTS)

Discussion:
That's pretty big! In contrast, an OS X build log is only 1.3M. Is there a reason they have to be this big? CC
My guess is that it's because of warnings. I remember having to get through thousands of them when looking for the error that failed the build.
Yes, the major part of them are warnings. And some of them is not fixable. We can hide them using some flags on the compilers.
And this build configuration actually contains both the CPU and GPU build, which may also be a possible reason.
The reduced log size after PR is about 1.2M.
Change:
} if (k == 3) { <del> if (should_compute_output(0)) { grad_input = view3d(*grad_input); </del> <ins> if (grad_input) { grad_input = view3d(*grad_input); } if (grad_weight) { grad_weight = view3d(*grad_weight); </ins> } <del> grad_weight = view3d(*grad_weight); </del> } auto outputs =  as_tensor_list(std::move(grad_input),

Discussion:
I have a network defined as: --. In this particular case, I want to keep the two conv layer weights fixed, and only adapt the middle layer weights. In the code below I have three ways of defining the conv layers parameters, one with buffers, one with parameters that have , and one with regular parameters (which doesn't keep the weights fixed, but I'm using it as a reference). If I use the first two definitions (commented out in the code below), when I call it results in a kernel crash. If I use the third definition, it works fine albeit without maintaining fixed weights. In order to apply the linear layer I reshape the output and inputs, and I believe the crash stems from somewhere there (on a side note: Is this the right way to apply a linear layer on a time sequence?)
Looks like the problem is with . If I reimplement this network using 2D convolutions with unit widths it works fine.
Change:
CUDA_INC ?= $(CUDA_HOME)/include NVCC ?= $(CUDA_HOME)/bin/nvcc <del> NVCC_GENCODE ?= -gencode=arch=compute_35,code=sm_35  </del> <ins> NVCC_GENCODE ?= -gencode=arch=compute_30,code=sm_30  -gencode=arch=compute_35,code=sm_35  </ins> -gencode=arch=compute_50,code=sm_50  -gencode=arch=compute_52,code=sm_52  -gencode=arch=compute_52,code=compute_52

Discussion:
I'm running PyTorch on a system with 4 Nvidia K10 GPUs and can't get the Multi-GPU code to work. I attempted to modify the first neural net in the 60-minute tutorial to run on all my GPUs, by adding this line to the cell defining the network: If I then attempt to use the net as follows , Jupyter waits a few seconds and reports the kernel has died. Following along in I see all GPUs encountering an at the same moment. Am I using the API wrong, could there be a configuration issue in my system, or is this a bug in the DataParallel implementation? I'm using CUDA 8 and PyTorch 0.1.6. Edit: my Nvidia driver version is 367.48 I also forgot to add, executing the above code on CUDA without parallelization (i.e. ) works fine.
Is there a stack-trace that you see appear on the command-line of the Jupyter process? Is this a cloud provider box? (I ask to see if I can reproduce it myself)
This is the only command-line output Jupyter gives related to the error: I'm running this on an in-house compute node.
thanks. this is helpful, I'll look further. does nccl have cuda code that we need to build for specific archs? K10 is 3.0 and i build 3.0 code for thc and thcunn
Yes, that's the problem: The README says "Kepler or newer". does NCCL work on K10s?
It should (once you add the necessary compilation flags).
Change:
'clang_version', 'cmake_version', 'os', <ins> 'libc_version', </ins> 'python_version', <ins> 'python_platform', </ins> 'is_cuda_available', 'cuda_runtime_version', 'nvidia_driver_version',

Discussion:
TensorPipe uses some advanced features of the Linux kernel. It does so on a "best effort" basis (i.e., use them if available, fall back on something else if not). This logic is unfortunately hard to test because it would require specific kernel versions on our CI machines. So it occasionally happens that we find out about issues when users report them. To help investigate these reports we've found it useful to know the kernel version and the glibc version used by the user (both when building and when running, if they're different). It would be great if these could be to the report produced by collect_env.
CC
Change:
namespace torch { namespace autograd { <ins> struct FunctionTask { BackwardTask* base; std::shared_ptr<Function> fn; GradBuffer grad; FunctionTask(BackwardTask* base, std::shared_ptr<Function> fn, GradBuffer grad) : base(base) , fn(fn) , grad(std::move(grad)) {} }; </ins> struct ReadyQueue { std::deque<FunctionTask> queue; std::condition_variable not_empty;

Discussion:
Version include <stringin some cpp files: define order in The pre-defined struct is not passed by clang in macOS, so we should reorder them.
Simple fix
Change:
if fn.batch_first and not is_input_packed: input = input.transpose(0, 1) <del> if (not is_input_packed and input.dim() != 3) or (is_input_packed and input.dim() != 2): raise RuntimeError( 'input must have 3 dimensions, got {}'.format(input.dim())) if fn.input_size != input.size(-1): raise RuntimeError('input.size(-1) must be equal to input_size. Expected {}, got {}'.format( fn.input_size, input.size(-1) )) </del> if fn.dropout != 0 and cudnn.version() < 5103: raise RuntimeError('dropout supported only in cudnn v5.1 and above')

Discussion:
I run RNN sucessfully with the code. and I run embedding example. Then, got RuntimeError. I set to use nothing seems wrong. And the error info seems not the dim problem? The error is so strange.
What exactly do you mean by "embedding example"? Could you provide a code snippet that causes the RuntimeError?
To better reproduce the error, I use the official code example of The origin runs well: Due to use , I have to make batch first. And then , it raises error.
Thank you for the example! The say that setting means that the first dim of input is the batch, ie, input has size . however is unchanged, so the following should work: I do agree the error message could be better, though
Change:
dim_args_idx = test[4] if len(test) == 5 else [] <ins> skipTestIf = test[5] if len(test) == 6 else [] </ins> for dim_perm in product([-1, 1], repeat=len(dim_args_idx)): test_name = basic_test_name new_args = [arg * dim_perm[dim_args_idx.index(i)] if i in dim_args_idx else arg for i, arg in enumerate(args)]

Discussion:
Command to install the latest nightly binary: installs : However, PyTorch itself reports : I guess this version tag might be defined during the build process from e.g. and then forwarded from , but also source builds should use based on the tag. Unsure, if this is just a reporting issue or if we are building the nightly binaries with and executing them with , so creating this issue to dig a bit further into it. CC cc
This is really weird, I've put up measure in place to validate against it... Will investigate today (and moving to pytorch repo)
can you please check that 2.2.0 reports (and is build) against the correct version of nccl library?
Confirmed for nightlies : Same issue for RC:
Looking at the build log Looks like we are compiling with: USESYSTEMNCCL
On Windows I see following exception: We compiling with should we return None rather then exception ?
In this case we are indeed using 2.18.1: Do you know why we are using the system-wide NCCL installation instead of the one? Should we delete the system NCCL and use by default?
Looks like its set here: Looks like it was by: Let me propose the PR to unset this flag : USESYSTEM_NCCL
I guess we are using the system NCCL installation to be able to during the runtime. If we would use the module I assume we would link NCCL into the PyTorch binaries. If so, we should make sure to update the NCCL version whenever is updated. This could break our dependency on the NCCL pip package.
Yes. Also when we update this nccl here:
We used following path  What would be good path for nccl 2.19.3 and cuda 12.1 ? As per this issue: Will be using 12.0 ncck 2.19.3
Installing system NCCL into the nightly build docker container feels great, but we need to figure out, how to keep it in sync with PyTorch. I wonder if one can, for example, just check nccl version from this requirements file and build from source?
yes probably we should build and upload nccl for our supported cuda versions, same way as we build docker containers, by following details here:
Verified with final rc
Change:
grad_input = torch.cross(other, grad_output, self.dim) grad_other = torch.cross(grad_output, input, self.dim) return grad_input, grad_other <ins> class Inverse(Function): @staticmethod def forward(ctx, input): inverse = torch.inverse(input) ctx.save_for_backward(inverse) return inverse @staticmethod def backward(ctx, grad_output): inverse, = ctx.saved_variables return -torch.mm(inverse.t(), torch.mm(grad_output, inverse.t())) </ins>

Discussion:
Command to install the latest nightly binary: installs : However, PyTorch itself reports : I guess this version tag might be defined during the build process from e.g. and then forwarded from , but also source builds should use based on the tag. Unsure, if this is just a reporting issue or if we are building the nightly binaries with and executing them with , so creating this issue to dig a bit further into it. CC cc
This is really weird, I've put up measure in place to validate against it... Will investigate today (and moving to pytorch repo)
can you please check that 2.2.0 reports (and is build) against the correct version of nccl library?
Confirmed for nightlies : Same issue for RC:
Looking at the build log Looks like we are compiling with: USESYSTEMNCCL
On Windows I see following exception: We compiling with should we return None rather then exception ?
In this case we are indeed using 2.18.1: Do you know why we are using the system-wide NCCL installation instead of the one? Should we delete the system NCCL and use by default?
Looks like its set here: Looks like it was by: Let me propose the PR to unset this flag : USESYSTEM_NCCL
I guess we are using the system NCCL installation to be able to during the runtime. If we would use the module I assume we would link NCCL into the PyTorch binaries. If so, we should make sure to update the NCCL version whenever is updated. This could break our dependency on the NCCL pip package.
Yes. Also when we update this nccl here:
We used following path  What would be good path for nccl 2.19.3 and cuda 12.1 ? As per this issue: Will be using 12.0 ncck 2.19.3
Installing system NCCL into the nightly build docker container feels great, but we need to figure out, how to keep it in sync with PyTorch. I wonder if one can, for example, just check nccl version from this requirements file and build from source?
yes probably we should build and upload nccl for our supported cuda versions, same way as we build docker containers, by following details here:
Verified with final rc