jiebi/CodeConvo huggingface dataset:
Change:
- THTensor* tensor2 ]] <del> #ifndef THP_LAPACK_CONSTANTS #define THP_LAPACK_CONSTANTS </del> <ins> #if defined(TH_REAL_IS_FLOAT) || defined(TH_REAL_IS_DOUBLE) || CUDA_FLOAT || CUDA_DOUBLE </ins> // We need to pass pointers to chars to tensor lapack functions... static const char __U = 'U'; static const char __L = 'L';

Discussion:
Trying to run on github actions via AWS instance gives me: I will of course set up now, but it shouldn't crash like that. May be it should use as a fallback? pt-2.0 cc
Is this a multitenant setup? Also curious which distro are you running, I was under the impression that most linux distros set home by default
this is AWS image. one default user. I think it's user, but it wants you to run as root, which is weird. I have manually now set to overcome this.
We should update triton to use pathlib or something
Change:
.. autofunction:: einsum .. autofunction:: flip .. autofunction:: histc <ins> .. autofunction:: meshgrid </ins> .. autofunction:: renorm .. autofunction:: trace .. autofunction:: tril

Discussion:
is not available in the docs page, but the doc string has been in . This was mentioned earlier in

Oh, bilinear, I missed that last time I looked at it. I'll add a docstring.

returns nothing. Might be useful to have some linter in the build process to find Python functions missing

Pytorch 0.4.0 gives a segmentation fault with dilated 2D convolutions over a specific size. import torch import as nn model = nn.Conv2d(inchannels=64, outchannels=64, kernelsize=5, dilation=2) model(([1, 64, 1166, 1166])) # This runs model(([1, 64, 1167, 1167])) # This crashes PyTorch version: 0.4.0 Is debug build: No CUDA used to build PyTorch: 9.1.85 OS: CentOS Linux 7 (Core) GCC version: (GCC) 4.8.5 (Red Hat 4.8.5-28) CMake version: version 2.8.12.2 Python version: 3.6 Is CUDA available: Yes CUDA runtime version: 8.0.61 GPU models and configuration: GPU 0: GeForce GTX 1080 Ti GPU 1: TITAN X (Pascal) GPU 2: Quadro K2200 Nvidia driver version: 390.59 cuDNN version: Probably one of the following: /usr/local/cuda- /usr/local/cuda- /usr/local/cuda-7.5/lib64/libcudnnstatic.a /usr/local/cuda- /usr/local/cuda-8.0/lib64/libcudnn_static.a
do you think this is the same as the segfault you showed me a week ago on convolution?
ah yes. int overflow. I only fixed conv3d because I didn’t want to hurt perf. I’ll fix all of them.

This should be an easy one.. The function is implemented, just no sign of it in the docs.
Change:
} } <del> if (use_cudnn) { </del> <ins> if (use_cudnn && eps >= CUDNN_BN_MIN_EPSILON) { </ins> #ifdef WITH_CUDNN torch::cudnn::cudnn_batch_norm_backward( state,

Discussion:
Hi, I have two linux servers. My code works in one machine but not the other. In the one that it doesn't work, the error message is: occured during batch normalization( convolution is fine). Also, the same code works in CPU but not in GPU. I guess this problem is mainly related to GPU libraries like cuda or cudnn. The one that doesn't work has the following configurations: Nvidia driver 375.26 cuda 8.0.61 cudnn 5.1.10 While in the one it does work, I have the following configurations: Nvidia driver 367.48 cuda 8.0.44 cudnn 5.1.5 TensorFlow can run in the above environment flawless though. So I am wondering if this is caused by the incompatibility between PyTorch 0.1.12 and cuda 8.0.61/cudnn 5.1.10 ?
you use the pytorch binaries, we ship cudnn6 in them, use it it's great. 5.1.10 maybe has a bug. cc: . do you have a small test case to reproduce this? does not use CuDNN batchnorm by default, so maybe that's why?
Small test case would help, thanks.
After some experimentation, I have pin pointed where exactly the error comes from. It seems batch norm in cudnn 5.1.10 doesn't support eps as small as 1e-6. A small test case would be:
Yes, cudnn.h for 5.1.10 defines CUDNNBNEPSILON to be 1e-5 (minimum allowable value). I can't check header for 5.1.5. Even with 1e-5 epsilon for many input data distributions the results become pretty unstable. Is there a case for this restriction to be lifted, does using very small epsilon provide benefits?
I see. Not sure if small epsilon is necessary. I am using 1e-6 just to replicate others' configurations. Should PyTorch provide more informative error message about this restriction?
Ideally, CUDNN should provide more informative error messages (pytorch can not know why cudnn is returning an error code in each particular case) , but that's a long standing discussion.
It just hits on me.

As mentioned by , TensorFlow does not use CuDNN batchnorm by default. So is this possible in Pytorch, to not use default CuDNN batchnorm?
Never mind. I found the switch in .
Change:
return ret; } <del> Variable VariableType::as_variable(Tensor tensor) const { </del> <ins> static Variable as_variable(Tensor tensor) { </ins> return make_variable(std::move(tensor)); } <del> std::tuple<Variable, Variable> VariableType::as_variable(std::tuple<Tensor, Tensor> tensors) const { </del> <ins> static std::tuple<Variable, Variable> as_variable(std::tuple<Tensor, Tensor> tensors) { </ins> return std::make_tuple<>( make_variable(std::move(std::get<0>(tensors))), make_variable(std::move(std::get<1>(tensors)))); } <del> std::tuple<Variable, Variable, Variable> VariableType::as_variable(std::tuple<Tensor, Tensor, Tensor> tensors) const { </del> <ins> static std::tuple<Variable, Variable, Variable> as_variable(std::tuple<Tensor, Tensor, Tensor> tensors) { </ins> return std::make_tuple<>( make_variable(std::move(std::get<0>(tensors))), make_variable(std::move(std::get<1>(tensors))), make_variable(std::move(std::get<2>(tensors)))); } <del> std::tuple<Variable, Variable, Variable, Variable> VariableType::as_variable(std::tuple<Tensor, Tensor, Tensor, Tensor> tensors) const { </del> <ins> static std::tuple<Variable, Variable, Variable, Variable> as_variable(std::tuple<Tensor, Tensor, Tensor, Tensor> tensors) { </ins> return std::make_tuple<>( make_variable(std::move(std::get<0>(tensors))), make_variable(std::move(std::get<1>(tensors))),

Discussion:
See Docker run fails with , which is due to the fact that is specified in docker container: Which results in: CI cc
Check that adding to manywheels image results in a failure:
Tentatively marking hi-pri since nightly testing are failing
This is resolved, here is the successful run:
Change:
self.assertRaises(TypeError, lambda: reference[0.0, ..., 0.0:2.0]) self.assertRaises(TypeError, lambda: reference[0.0, :, 0.0]) <ins> def test_index(self): self._test_index(self, lambda x: x) </ins> @staticmethod def _test_advancedindex(self, conv_fn): # Tests for Integer Array Indexing, Part I - Purely integer array

Discussion:
Arguably, indexing a Tensor by a LongTensor with no dimension should return a LongTensor with no dimension. Consider the following case: This breaks indexing by indices when the list of indices is empty, which can be hard to detect. In my use-case, I use such a masking to select rows of a tensor corresponding to "valid" entries in a vector; I had not considered that it would not work when there are no valid entries.
Thanks! I'll take a look.
Change:
""" # TODO: try to find an input on another gpu, copy it, # and accumulate into the copy <ins> if destination is None: destination = torch.cuda.current_device() </ins> input_size = inputs[0].size() <ins> nccl_root = None </ins> for i, inp in enumerate(inputs): assert inp.is_cuda, "reduce_add expects all inputs to be on GPUs" <ins> if inp.get_device() == destination: nccl_root = i </ins> if inp.size() != input_size: got = 'x'.join(str(x) for x in inp.size()) expected = 'x'.join(str(x) for x in input_size) raise ValueError("input {} has invalid size: got {}, but expected " "{}".format(i, got, expected)) <del> if destination is None: destination = torch.cuda.current_device() </del> <ins> assert nccl_root is not None, "reduce_add expects destination to be on the same GPU with one of the tensors" </ins> with torch.cuda.device(destination): result = type(inp)(input_size).zero_() if nccl.is_available(inputs) and inputs[0].get_device() == destination: outputs = [result] + [t.new(t.size()) for t in inputs[1:]] <del> nccl.reduce(inputs, outputs, root=destination) </del> <ins> nccl.reduce(inputs, outputs, root=nccl_root) </ins> return result for inp in inputs:

Discussion:
The loss will never decrease when I train a model using multi-gpus but without gpu0. This is an example I modified from the : If the result after one epoch is If the result after one epoch is
should always contain . If you want to specify which GPUs to use, you can set the environment variable , either by running your script like this: or set it programmatically in your script like this: So that deviceids 0, 1 will map to GPUs 1, 2, respectively. (Note that for the second method, you should put that line before any cuda relevant function call.)
Thank you.
<!-- drci-comment-start --:pagefacingup: Preview :pagefacingup: Preview :question: Need help or want to give feedback on the CI? Visit our Note: Links to docs will display an error until the docs builds have been completed. This comment was automatically generated by Dr. CI and updates every 15 minutes. <!-- drci-comment-end --
Change:
:members: <ins> Padding Layers -------------- :hidden:`ReflectionPad2d` ~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: ReflectionPad2d :members: :hidden:`ReplicationPad2d` ~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: ReplicationPad2d :members: :hidden:`ReplicationPad3d` ~~~~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: ReplicationPad3d :members: :hidden:`ZeroPad2d` ~~~~~~~~~~~~~~~~~~~ .. autoclass:: ZeroPad2d :members: :hidden:`ConstantPad2d` ~~~~~~~~~~~~~~~~~~~~~~~ .. autoclass:: ConstantPad2d :members: </ins> Non-linear Activations ----------------------------------

Discussion:
After merging cuda 12.4 for nightly. I observe following failures: libtorch-cuda124-shared-with-deps-cxx11-abi-build: Build Official Docker Images 124: We need to resolve these before continuing adding CUDA 12.4 CI workflows 0 cc
attempt 3 worked for libtorch build issue:
For Docker image, related discussion:
I will instance a new bug for the docker job failure.

Is there a way to use reflection padding instead of zero padding with nn.Conv2d ? I see there is a but the option is not in nn.Conv2d. Thanks,
are in , but they're not exposed in the docs.
Thanks!
Change:
self.assertEqual(inp_i.grad, i.grad) assert not hasattr(TestAutograd, test_name), 'Two tests have the same name: ' + test_name <ins> for skip in skipTestIf: do_test = skip(do_test) </ins> setattr(TestAutograd, test_name, do_test)

Discussion:
Command to install the latest nightly binary: installs : However, PyTorch itself reports : I guess this version tag might be defined during the build process from e.g. and then forwarded from , but also source builds should use based on the tag. Unsure, if this is just a reporting issue or if we are building the nightly binaries with and executing them with , so creating this issue to dig a bit further into it. CC cc
This is really weird, I've put up measure in place to validate against it... Will investigate today (and moving to pytorch repo)
can you please check that 2.2.0 reports (and is build) against the correct version of nccl library?
Confirmed for nightlies : Same issue for RC:
Looking at the build log Looks like we are compiling with: USESYSTEMNCCL
On Windows I see following exception: We compiling with should we return None rather then exception ?
In this case we are indeed using 2.18.1: Do you know why we are using the system-wide NCCL installation instead of the one? Should we delete the system NCCL and use by default?
Looks like its set here: Looks like it was by: Let me propose the PR to unset this flag : USESYSTEM_NCCL
I guess we are using the system NCCL installation to be able to during the runtime. If we would use the module I assume we would link NCCL into the PyTorch binaries. If so, we should make sure to update the NCCL version whenever is updated. This could break our dependency on the NCCL pip package.
Yes. Also when we update this nccl here:
We used following path  What would be good path for nccl 2.19.3 and cuda 12.1 ? As per this issue: Will be using 12.0 ncck 2.19.3
Installing system NCCL into the nightly build docker container feels great, but we need to figure out, how to keep it in sync with PyTorch. I wonder if one can, for example, just check nccl version from this requirements file and build from source?
yes probably we should build and upload nccl for our supported cuda versions, same way as we build docker containers, by following details here:
Verified with final rc
Change:
return r; } <del> variable_list flatten(const Tensor& x, const TensorList& y, const Tensor& z) { </del> <ins> static variable_list flatten(const Tensor& x, const TensorList& y, const Tensor& z) { </ins> std::vector<Variable> r; r.reserve(2 + y.size()); r.emplace_back(x);

Discussion:
See Docker run fails with , which is due to the fact that is specified in docker container: Which results in: CI cc
Check that adding to manywheels image results in a failure:
Tentatively marking hi-pri since nightly testing are failing
This is resolved, here is the successful run:
Change:
arguments(), is_variadic(is_variadic), has_out(has_out) {}; Option(const Option&) = delete; Option(Option&& other): <del> arguments(std::move(other.arguments)), is_variadic(other.is_variadic) {}; </del> <ins> arguments(std::move(other.arguments)), is_variadic(other.is_variadic), has_out(other.has_out) {}; </ins> std::vector<Argument> arguments; bool is_variadic;

Discussion:
! Maybe we should change it to `? No response cc
are these old installation commands also auto-tested with CI?
Looks no longer relevant.
Change:
auto m = self.size(0); auto n = self.size(1); auto k = sigma.size(0); <ins> auto gsigma = grads[1]; auto u = raw_u; auto v = raw_v; auto gu = grads[0]; auto gv = grads[2]; </ins> <del> Tensor u, v; </del> if (!some) { <del> // ignore the free subspace </del> <ins> // We ignore the free subspace here because possible base vectors cancel // each other, e.g., both -v and +v are valid base for a dimension. // Don't assume behavior of any particular implementation of svd. </ins> u = raw_u.narrow(1, 0, k); v = raw_v.narrow(1, 0, k); <del> } else { u = raw_u; v = raw_v; </del> <ins> if (gu.defined()) { gu = gu.narrow(1, 0, k); } if (gv.defined()) { gv = gv.narrow(1, 0, k); } </ins> } <del> auto gu = grads[0]; auto gsigma = grads[1]; auto gv = grads[2]; </del> auto vt = v.t(); Tensor sigma_term;

Discussion:
I am having problem in running the backward propagation with svd. In particular, it seems working when the first dimension of the matrix x (below) is greater than the second. But I get the following error if the first is smaller than the second. PyTorch version: 2018.04.20 Is debug build: No CUDA used to build PyTorch: 8.0.61 OS: Ubuntu 16.04.4 LTS GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.9) 5.4.0 CMake version: Could not collect Python version: 3.5 Is CUDA available: No CUDA runtime version: No CUDA GPU models and configuration: No CUDA Nvidia driver version: No CUDA cuDNN version: No CUDA Versions of relevant libraries: [pip3] numpy (1.13.3) [pip3] torch (2018.4.20) [pip3] torchvision (0.2.0) [conda] pytorch 0.3.1 py35cuda8.0.61cudnn7.0.52 pytorch [conda] pytorch-nightly 2018.04.20 py35cuda8.0.61cudnn7.1.21 pytorch [conda] torchvision 0.2.0 py35heaa392f_1 pytorch
Repro'ed on 0.4 branch. works btw.
What does "Repro'ed" means? Should I install from that branch? I need to compute the kernel of a matrix, therefore I need the last columns of the matrix V (which is why I have to pass ).
It's a shorthand for "reproduced", which means that the bug still hasn't been fixed. Sorry, we'll try to get to it soon.
Change:
raise ValueError("output_size should be a sequence containing " "2 or 4 elements, but it has a length of {}".format( len(output_size))) <del> out_sizew, out_sizeh = output_size sizew = ((input.size(3) - 1) * self.dw - 2 * self.padw + self.kw) </del> <ins> out_sizeh, out_sizew = output_size </ins> sizeh = ((input.size(2) - 1) * self.dh - 2 * self.padh + self.kh) <del> out_padw = out_sizew - sizew </del> <ins> sizew = ((input.size(3) - 1) * self.dw - 2 * self.padw + self.kw) </ins> out_padh = out_sizeh - sizeh <del> out_padw_ok = 0 <= out_padw < self.dw </del> <ins> out_padw = out_sizew - sizew </ins> out_padh_ok = 0 <= out_padh < self.dh <del> if not out_padw_ok or not out_padh_ok: </del> <ins> out_padw_ok = 0 <= out_padw < self.dw if not out_padh_ok or not out_padw_ok: </ins> raise ValueError(("requested an output size of {}x{}, but " "valid sizes range from {}x{} to {}x{} (for an input of " "{}x{})").format(out_sizeh, out_sizew, sizeh, sizew,

Discussion:
Repro is torchtrain w/ it happens more often if in my experience but it still happens either way. Logically its clear we have a race: rank0 is free to exit and shut down TCPStore before or after other ranks shut themselves down. we have to consider either a) rank 0 waits for all other ranks to exit before it exists b) we make other ranks tolerant of tcpstore going away (A) has potential downsides in efficiency even in the best case. But also practically, we could use tcpstore barrier, but it wont work reliably at large scales. we'd have to consider something like 'scalable coordinator' to go this route (B) we could probably implement w/ a try/catch in the heartbeat monitor for the connection error, similar to how we catch cuda driver exit. This feels a little hacky. Its important we somehow distinguish between the cases when we are actually shutting down and want to swallow the error, from cases where the is a tcpstore/network error during runtime and we actually want to raise the error. This points to (a) as a more correct direction. cc
do you have any update on "make torchrun share TCPStore instance with pytorch"? I think that is our "proper" fix for this problem, and in the meantime people can work around it by 1) sleeping on rank0 for a sufficiently long time (between 5-10s seemed plenty for 128rank jobs) simplest possible to not sleep long enough in edge cases 2) using a barrier at job shutdown such that rank0 (and tcsptore) are gauranteed alive.
was landed as a fix
Change:
self.assertIn('buf', l.state_dict()) self.assertIs(l.state_dict()['buf'], buf) <ins> def test_Conv2d_inconsistent_types(self): inputs = Variable(torch.randn(4, 1, 7, 7).float()) weights = Variable(torch.randn(1, 1, 3, 3).double()) # inconsistent types should raise an exception self.assertRaises(RuntimeError, lambda: nn.functional.conv2d(inputs, weights)) # but it should work with the same type nn.functional.conv2d(inputs.float(), weights.float()) </ins> @unittest.skipIf(not TEST_CUDA, 'CUDA not available') def test_Conv2d_large_workspace(self): # These sizes require huge cuDNN workspaces. Make sure we choose a

Discussion:
the error only happens in very specific conditions. (change the view of input tensor and use weight converted from numpy array.) The output of functional conv2d is far off in scale (e12 or so.)
this happens because generates double precision tensor by default, whereas torch generates single precision. We need to add guards for tensor types in THNN and THCUNN.
It's a problem with function, that only . Should be a simple fix.
Whoops! Yeah it just ignores the "thpp::Type type" parameter
what type is expected here?
a is a FloatTensor but weights is a DoubleTensor. Change: to: (or change a to (...).double())
cool thanks!
Change:
THArgCheckWithCleanup(n_sample > 0, THCleanup(if (start_dim == 1) THTensor_(resize1d)(prob_dist, n_categories);), 2, <del> "cannot sample n_sample < 0 samples"); </del> <ins> "cannot sample n_sample <= 0 samples"); </ins> if (!with_replacement) {

Discussion:
Looking at the values returned from tn.multinomial(1) and counting them, it seems like it returns the values of the multinomial distribution taken from t, i.e. that it has ignored the signs of the values. However, it seem like always returns a tensor filled with zeros.
looking into this
Change:
def in_cooldown(self): return self.cooldown_counter > 0 <del> def _init_is_better(self, mode, threshold, threshold_mode): if mode not in {'min', 'max'}: raise ValueError('mode ' + mode + ' is unknown!') if threshold_mode not in {'rel', 'abs'}: raise ValueError('threshold mode ' + threshold_mode + ' is unknown!') </del> <ins> def _cmp(self, mode, threshold_mode, threshold, a, best): </ins> if mode == 'min' and threshold_mode == 'rel': rel_epsilon = 1. - threshold <del> self.is_better = lambda a, best: a < best * rel_epsilon self.mode_worse = float('Inf') </del> <ins> return a < best * rel_epsilon </ins> elif mode == 'min' and threshold_mode == 'abs': <del> self.is_better = lambda a, best: a < best - threshold self.mode_worse = float('Inf') </del> <ins> return a < best - threshold </ins> elif mode == 'max' and threshold_mode == 'rel': rel_epsilon = threshold + 1. <del> self.is_better = lambda a, best: a > best * rel_epsilon self.mode_worse = -float('Inf') </del> <ins> return a > best * rel_epsilon </ins> else:  # mode == 'max' and epsilon_mode == 'abs': <del> self.is_better = lambda a, best: a > best + threshold self.mode_worse = -float('Inf') </del> <ins> return a > best + threshold def _init_is_better(self, mode, threshold, threshold_mode): if mode not in {'min', 'max'}: raise ValueError('mode ' + mode + ' is unknown!') if threshold_mode not in {'rel', 'abs'}: raise ValueError('threshold mode ' + threshold_mode + ' is unknown!') if mode == 'min': self.mode_worse = float('inf') else:  # mode == 'max': self.mode_worse = (-float('inf')) self.is_better = partial(self._cmp, mode, threshold_mode, threshold) </ins>

Discussion:
I'm working on checkpointing in our nn trainer. When saving the state objects, I get an error: PicklingError: Can't pickle <type 'function': attribute lookup builtin.function failed I tracked it down to trying to save the scheduler ReduceLROnPlateau. I think the issue is due to the scheduler referencing the optimizer, which has to be saved with statedict(). I presume the same error will occur with other schedulers too. It seems like the problem could be solved by providing a statedict() and loadstatedict() in the scheduler classes. pytorch master rev (Jan 22) installed from source x86_64 on centos 7.4 cuda 9.0 cudnn 7 k80 gcc 5.4 python 2.7.14
The following script reproduces the crash: import torch import from import ReduceLROnPlateau if name == 'main': model = (10,10) optim = (model.parameters()) sched = ReduceLROnPlateau(optim, mode='max', factor=0.5) with open('','wb') as f: # Works # ((model.statedict(), optim.statedict()), f) # Crashes ((model.statedict(), optim.state_dict(), sched), f)
Change:
check_file(os.path.join(lib_path, "pybind11", "CMakeLists.txt")) check_file(os.path.join('aten', 'src', 'ATen', 'cpu', 'cpuinfo', 'CMakeLists.txt')) check_file(os.path.join('aten', 'src', 'ATen', 'cpu', 'tbb', 'tbb_remote', 'Makefile')) <ins> check_file(os.path.join('aten', 'src', 'ATen', 'utils', 'catch', 'CMakeLists.txt')) </ins> check_pydep('yaml', 'pyyaml') check_pydep('typing', 'typing')

Discussion:
Trying to build master (), I'm seeing I tried in to no avail.
NM, also needed a .
Need a check to see if this is folder is checked out in , like the other submodules. Should be a simple patch.
Change:
r__->stride[(transpose_r == 'n' ? 1 : 0)]); /* free intermediate variables */ <del> if(m1_ != m1) </del> <ins> if(free_m1) </ins> THTensor_(free)(m1_); <del> if(m2_ != m2) </del> <ins> if(free_m2) </ins> THTensor_(free)(m2_); if(r__ != r_)

Discussion:
Here is an example of what seems to be a memory leak in pytorch, although it could be numpy? The leak was only occurring when using the or array unsqueezing methods in numpy. Also worth noting that the issue is resolved by removing the from and casting the torch tensor to type. An easy workaround is to reshape the array after turning it into a torch Tensor.