jiebi/CodeConvo huggingface dataset:
Change:
If :attr:`compute_uv` is ``False``, the returned `U` and `V` matrices will be zero matrices of shape :math:`(m times m)` and :math:`(n times n)` respectively. :attr:`some` will be ignored here. <ins> .. note:: The singular values are returned in descending order. If :attr:`input` is a batch of matrices, then the singular values of each matrix in the batch is returned in descending order. </ins> .. note:: The implementation of SVD on CPU uses the LAPACK routine `?gesdd` (a divide-and-conquer algorithm) instead of `?gesvd` for speed. Analogously, the SVD on GPU uses the MAGMA routine `gesdd` as well.

Discussion:
Documentation for does not indicate whether or not singular values are returned in descending order. My suspicion is that they are since both LAPACK and MAGMA's ?gesdd return them in descending order. It seems like this should be clarified in the docs to avoid unnecessary sorting, or mistaken assumptions. <!-- A clear and concise description of what content in is an issue. If this has to do with the general website, please file an issue at instead. If this has to do with , please file an issue at --
Change:
auto set_history = [&](Variable& var, uint32_t output_nr, bool is_input, bool is_modified, bool is_differentiable) { if (!is_differentiable) { <del> if (!var.requires_grad()) return; </del> <ins> if (!var.requires_grad()) { return; } </ins> // NB: we don't support returning non-differentiable views that could require grad <del> // (this could happen if someone were to return an input to the function). </del> if (var.is_view()) { throw std::runtime_error("Returning Variables sharing storage with other Variables " "that require grad is not supported in Python functions. " "Please submit a feature request if you hit this error."); } <del> var.detach_(); </del> <ins> // Return detached aliases of inputs, instead of changing their requires_grad // property. if (is_input) { var = var.detach(); } else { var.detach_(); } </ins> } else if (is_modified) { if (var.is_leaf() && var.requires_grad()) { throw std::runtime_error("a leaf Variable that requires grad has been used in an in-place operation.");

Discussion:
OS: ubuntu 16.04 PyTorch version: 0.4.0a0+ How you installed PyTorch (conda, pip, source): source Python version: 3.6.3 CUDA/cuDNN version: v9.1, cudnn-9.1-linux-x64-v7 GPU models and configuration: 4x Tesla M60 GCC version (if compiling from source): 5.4.0 When using context () on a model with dataparallel operation (same with ) on 1 GPU the flag requires_grad does not come back to True. If only one GPU is used then everything works fine. Script: On 2 GPUs, this outputs which is wrong in the last line. On 1 GPU (set ) the output is correct.
is looking into it
Change:
Otherwise, :attr:`dim` is squeezed (see :func:`torch.squeeze`), resulting in the output tensor having 1 fewer dimension than :attr:`input`."""}) factory_common_args = merge_dicts(common_args, parse_kwargs(""" dtype (:class:`torch.dtype`, optional): the desired data type of returned tensor. Default: if ``None``, uses a global default (see :func:`torch.set_default_tensor_type`).

Discussion:
with , both max and min are broken <img width="880" alt="Screen Shot 2020-03-23 at 1 10 07 PM" src="https://user-"on , min is broken <img width="875" alt="Screen Shot 2020-03-23 at 1 13 23 PM" src="https://user-"
Fixed by and
Change:
"""Fills the input Tensor or Variable with values according to the method described in "Understanding the difficulty of training deep feedforward neural networks" - Glorot, X. & Bengio, Y. (2010), using a normal distribution. The resulting tensor will have values sampled from :math:`N(0, std)` where <del> :math:`std = gain times sqrt{2 / (fan_in + fan_out)}`. Also known as Glorot initialisation. </del> <ins> :math:`std = gain times sqrt{2 / (fan_in + fan_out)}`. Also known as Glorot initialisation. </ins> Args: tensor: an n-dimensional torch.Tensor or autograd.Variable

Discussion:
why 1.21.6? because that's the oldest supported numpy on windows/python3.10: all cc
Here is the test for pinning numpy to 1.21.6 The reason its was not pinned to this version is that conda can't properly resolve all the dependencies at this step here:
It looks like you are correct, I will put up a PR to pin lower then already pinned:
Here is the PR:
Change:
#include "torch/csrc/autograd/engine.h" #include <set> <ins> #include <string> </ins> #include <THPP/THPP.h> using thpp::Tensor;

Discussion:
Not sure what is the reason for these errors, any suggestions? I suspect the clang version is not supported? Here is the
clang have more strict checking , pull request will fixed it.
fixed, thanks to
Change:
""" return self._apply(lambda t: t.cuda(device_id)) <del> def cpu(self, device_id=None): </del> <ins> def cpu(self): </ins> """Moves all model parameters and buffers to the CPU.""" return self._apply(lambda t: t.cpu())

Discussion:
as shown in source code, this parameters is not used. Any reason that we keep it?
this can be cleaned up.
Change:
uniforms = clamp_probs(self.logits.new(self._extended_shape(sample_shape)).uniform_()) gumbels = -((-(uniforms.log())).log()) scores = (self.logits + gumbels) / self.temperature <del> return scores - log_sum_exp(scores) </del> <ins> return scores - _log_sum_exp(scores) </ins> def log_prob(self, value): K = self._categorical._num_events

Discussion:
Leaving this issue here for higher visibility. Originally reported by here: Also mentioned by Piggybacking off of this issue, has strange behavior (reported here: )
I have fixed this locally. Could I send a PR?
Yes, go ahead!
Change:
def make_grad_input(fn): def call_grad_input(self, bufs, input, weight, grad_output): backend = type2backend[type(input)] <del> grad_input = input.new().resize_as_(input).zero_() </del> <ins> grad_input = input.new().resize_as_(input) </ins> kernel_size = weight.size()[2:] args = parse_arguments(self, fn.arguments[5:], bufs, kernel_size) getattr(backend, fn.name)(backend.library_state, input, grad_output,

Discussion:
THNN convolutions are slower due to unnecessary .zero() calls on gradients: is not needed same for bias, prob worth adding a constant in C-side that controls whether to override grad or not
Same for batch_norm and auto thnn functions, not sure if all of them though. Prob the speedups will be quite significant.
I think they are needed. THNN functions accumulate the gradient instead of overwriting the buffer, don't they?
only grad wrt weight, inputs are overriden
We should reopen this, still many functions have unnecessary .zero() calls on gradinput and gradparam, e.g.: : : most of the functions don't need it
I think the ones are currently necessary.
for nn yes, but cudnn has: and on cudnnConvolutionBackwardFilter and on cudnnBatchNormalizationBackward which control addition of gradients and should be set to 1 and 0 in pytorch cudnn (it is and 1 in )
Yeah, we can remove it on BatchNorm. If you look at Conv, it already only zeros out grad_weight for cunn.
this is now fixed in master:
Change:
} auto SavedVariable::unpack() -> std::shared_ptr<Variable> { <del> if (!data) return nullptr; </del> <ins> if (!data) { if (version) { throw std::runtime_error(ERR_BACKWARD_TWICE); } return nullptr; } </ins> int current_version = **version; if (expected_version != current_version) {

Discussion:
Usually calling backwards twice gives this message: But conv2d just segfaults. $ pt A B Segmentation fault (core dumped)
Change:
('trace', (M, M), ()), ('cross', (S, 3), ((S, 3),)), ('cross', (S, 3, S), ((S, 3, S), 1), 'dim'), <ins> ('inverse', (S, S), (), '', (), [skipIfNoLapack]), </ins> ('clone', (S, M, S), ()), ('eq', (S, S, S), ((S, S, S),)), ('ne', (S, S, S), ((S, S, S),)),

Discussion:
Command to install the latest nightly binary: installs : However, PyTorch itself reports : I guess this version tag might be defined during the build process from e.g. and then forwarded from , but also source builds should use based on the tag. Unsure, if this is just a reporting issue or if we are building the nightly binaries with and executing them with , so creating this issue to dig a bit further into it. CC cc
This is really weird, I've put up measure in place to validate against it... Will investigate today (and moving to pytorch repo)
can you please check that 2.2.0 reports (and is build) against the correct version of nccl library?
Confirmed for nightlies : Same issue for RC:
Looking at the build log Looks like we are compiling with: USESYSTEMNCCL
On Windows I see following exception: We compiling with should we return None rather then exception ?
In this case we are indeed using 2.18.1: Do you know why we are using the system-wide NCCL installation instead of the one? Should we delete the system NCCL and use by default?
Looks like its set here: Looks like it was by: Let me propose the PR to unset this flag : USESYSTEM_NCCL
I guess we are using the system NCCL installation to be able to during the runtime. If we would use the module I assume we would link NCCL into the PyTorch binaries. If so, we should make sure to update the NCCL version whenever is updated. This could break our dependency on the NCCL pip package.
Yes. Also when we update this nccl here:
We used following path  What would be good path for nccl 2.19.3 and cuda 12.1 ? As per this issue: Will be using 12.0 ncck 2.19.3
Installing system NCCL into the nightly build docker container feels great, but we need to figure out, how to keep it in sync with PyTorch. I wonder if one can, for example, just check nccl version from this requirements file and build from source?
yes probably we should build and upload nccl for our supported cuda versions, same way as we build docker containers, by following details here:
Verified with final rc
Change:
case NPY_DOUBLE: return kDouble; case NPY_FLOAT: return kFloat; case NPY_HALF: return kHalf; <ins> case NPY_LONGLONG: </ins> case NPY_INT64: return kLong; case NPY_INT32: return kInt; case NPY_INT16: return kShort;

Discussion:
Minimal example: Torch initializes the tensor with values which are different from the numpy array. I've seen issues for similar kinds of problems (see ). Maybe the easiest solution to all these initialization problems might be, to check for the same values of the original and initialized array/tensor instead of initializing tensors silently with wrong numbers. What do you think? Furthermore, I've opened an issue which is also related this one in terms of initializing a numpy array with a number of type and then initializing a torch tensor from that array, see Feel free to merge them if you think that both issues can be solved together.
Could you please provide some information about what version of pytorch and numpy you're using? ! It works as expected for me on master
I'm using torch 0.3.0.post4 (latest one on website of PyTorch) and numpy 1.13.3.
I'm running the same thing, this is strange. In the first line , I assumed that came from . Is this the case?
Nope, it's indeed the python built-in type. Actually this issue came up when I tried to get the shape of (i.e. , , returns ).
Thanks for the clarification, I'm now seeing your results. Looks like the tensor constructor doesn't work well with python2.7 long types in this case
Sure, no problem. Thanks for looking into it!
The problem seems to be that np.longlong and np.int64 are the same thing (at least on a 64bit system, I think), but the NPYLONGLONG and NPYINT64 macros are different... I'll put up a pr for this soon.

A minimal example: Although has and this type is listed as supported, the array can't be converted. Consider that is initialized with an upper bound of type . I'm not completely sure, whether this issue is related to torch or numpy, but numpy initializes the array with values as expected, whereas torch throws an inconsistent error message.
Edit: nevermind, I didn't run the same exact thing ! Works for me on master and 0.3. Seems like it could be related to
Change:
torch_version=version_str, is_debug_build=debug_mode_str, python_version='{}.{} ({}-bit runtime)'.format(sys.version_info[0], sys.version_info[1], sys.maxsize.bit_length() + 1), <ins> python_platform=get_python_platform(), </ins> is_cuda_available=cuda_available_str, cuda_compiled_version=cuda_version_str, cuda_runtime_version=get_running_cuda_version(run_lambda),

Discussion:
TensorPipe uses some advanced features of the Linux kernel. It does so on a "best effort" basis (i.e., use them if available, fall back on something else if not). This logic is unfortunately hard to test because it would require specific kernel versions on our CI machines. So it occasionally happens that we find out about issues when users report them. To help investigate these reports we've found it useful to know the kernel version and the glibc version used by the user (both when building and when running, if they're different). It would be great if these could be to the report produced by collect_env.
CC
Change:
self.assertEqual(v[mask].shape, (3, 7, 3)) self.assertEqual(v[mask], torch.stack([v[0], v[2], v[3]])) <ins> v = Variable(torch.Tensor([1])) self.assertEqual(v[v == 0], Variable(torch.Tensor())) </ins> def test_multiple_byte_mask(self): v = Variable(torch.randn(5, 7, 3)) # note: these broadcast together and are transposed to the first dim

Discussion:
In PyTorch master 0.4 , variables cannot be indexed. This used to work in PyTorch 0.2 :
This is breaking some of Pyro's inference algorithms on PyTorch master:
Change:
if self.is_cuda: raise TypeError("cannot pin '{0}' only CPU memory can be pinned" .format(self.type())) <del> storage = self.storage() </del> <ins> storage = self.contiguous().storage() </ins> if storage is None: storage = (self.storage_type())() return type(self)().set_(storage.pin_memory()).view_as(self)

Discussion:
Hello, I was experimenting with a custom which receives a batch of different sized integer sequences and pads them to form 2D tensors. I noticed a weird issue for which I created the following MWE: It's probably because of the different memory ordering on Python's side and CUDA's allocator. Tested on K40 and GTX1080 with driver 375.39, CUDA8.
Pin memory is missing a call inside. Should be an easy fix
fixed via
Change:
build_libs_cmd = ['bash', 'torch/lib/build_libs.sh'] my_env = os.environ.copy() my_env["PYTORCH_PYTHON"] = sys.executable <ins> if WITH_SYSTEM_NCCL: my_env["NCCL_ROOT_DIR"] = NCCL_ROOT_DIR </ins> if WITH_CUDA: build_libs_cmd += ['--with-cuda'] if subprocess.call(build_libs_cmd + libs, env=my_env) != 0:

Discussion:
Currently, you can have a libnccl on your system, and pytorch setup favors that instead of compiling one itself from source. The code that takes care of this is: However, this breaks gloo build because gloo gets a path to NCCL via:
I got the error during the installation: Is it related to this problem?
that's a dupe of
no, that's a consequence of , now you have to do before compiling. I think you don't want to do recursive submodule update, because that will pull in gtest and another copy of nccl.
Alternatively we can pass new flags to gloo to prevent it from using its own NCCL
Thanks works for me.
You can prevent it from using its own nccl, but its cmake still has to be able to find some nccl.
I think should be sufficient. I'm testing.
fixes it when we dont have an external NCCL and pytorch compiles it's own NCCL?
I don't see how NCCLEXTERNAL fixes anything, looks like in gloo NCCLEXTERNAL is used just to set dependency if gloo is compiling its own nccl (which we don't want in any case).
The only thing I tested when I was that the vendored nccl was not being built (even if its submodule was checked out); I tested this by adding an to one of the headers. I'll test with the other configurations.
Actually, there are a few bugs here. First, even though GLOOFLAGS looks like it's setting NCCLROOTDIR: it actually isn't because the gloo invocation is which expands to but only is passed to cmake. Suppose you fix this. nccl is still not going to be picked up, because INSTALLDIR is PyTorch's local install dir, but if you disabled nccl builds, there will never actually be a nccl here. So cmake is just going to do it's standard search to find a nccl somewhere. Inversely, if you ever built nccl, and then turn it off, we will keep picking up that locally built version until you delete it. So we shouldn't do that. Suppose you fix this.

Then I think gloo will (continue to) correctly pick up system nccl and be built with it, and the final thing to fix is that build of PyTorch csrc doesn't know how to search for nccl headers somewhere. This is probably related to . I wonder, maybe we shouldn't do any autodetection at all, and force the user to pass us the include and library paths of nccl if they want to build with system.
FYI, I committed a change to gloo to stop bundling NCCL itself. It will now happily pick up whatever NCCL you give it through NCCLROOTDIR or by trying to find it in the standard systems paths. See facebookincubator/gloo You do have to set to make Gloo try and find it and compile in support.
So to fix this, you should be able to not specify anything aside from if you want to use system NCCL, and add if you have some custom location (either a PyTorch bundled version or a custom install altogether).
Change:
} void THTensor_(sparseMask)(THSTensor *r_, THTensor *t, THSTensor *mask) { <ins> THArgCheck(mask->coalesced, 2, "mask is uncoalesced"); </ins> THSTensor_(resizeAs)(r_, mask); if (mask->nnz == 0) { THSTensor_(zero)(r_);

Discussion:
and I talked about the intended semantics of sparsemask when given a sparse tensor as an argument, and we came to the following conclusions: with an uncoalesced sparse tensor does not make sense, full stop. Unlike values, where your update might be linear and work correctly on an uncoalesced sparse tensor, there isn't any good reason why you want to pluck out values from the dense tensor multiple times, it is basically always a bug. the (eventual) documentation for sparsemask, we will recommend calling coalesce before doing any operations involving sparsemask (or nonlinear updates to indices/values). should add a test to sparsemask which checks if the flag of the input tensor is true, and raise an error if it is not. This is a cheap safety check which will prevent people from forgetting to call coalesce (as in (2)). There is a performance consequence to calling coalesce every iteration of training. If the gradient is producing coalesced vectors, we can alleviate this problem by adding a preprocessing step to coalesce() which tests if the input is already coalesced, and bugs out in that case. However, I think this is out of scope for this particular ticket for now.
Change:
auto num_threads = leaked->get_thread_count(); // NOLINTNEXTLINE(modernize-make-unique) threadpool.reset(new PThreadPool(num_threads)); <del> TORCH_WARN("Leaking Caffe2 thread-pool after fork."); </del> } } return threadpool.get();

Discussion:
When numworker 0 (in DataLoader), I call setnum_threads function and get some warnings. Execute the following script: got log: there should be no any warnings. PyTorch Version (e.g., 1.0): torch1.9 OS (e.g., Linux): Centos7 How you installed PyTorch (, , source): pip Build command you used (if compiling from source): no Python version: 3.8 CUDA/cuDNN version: 11.1/ GPU models and configuration: TeslaV100 Any other relevant information:
Duplicate of .
closing as dupplicate
Change:
from numbers import Integral import warnings import math <ins> from operator import mul from functools import reduce </ins> import torch from torch._C import _infer_size

Discussion:
Right now it makes everything .
To be clear, for a tensor, the problem is only when all of N, H, and W are 1. (So BatchNorm2d on batch size of 1 is OK as long as you don't have a 1x1 image). What's the desired behavior? The only reasonable behavior I can think of is: Raise an exception when the dimensions of which you are normalizing are one or Output zero (+ optional affine transform) I'm not sure the outputing zero is a good idea. I can't think of a case where that's what you want.
Change:
#include <TH/TH.h> #include <libshm.h> #include "THP.h" <ins> #include "byte_order.h" </ins> #include "generic/Storage.cpp" #include <TH/THGenerateAllTypes.h> #include "generic/StorageCopy.cpp" #include <TH/THGenerateAllTypes.h>

Discussion:
<!-- A clear and concise description of what the bug is. --The target name is used by a test executable in mkldnn. Steps to reproduce the behavior: BUILDTEST=1 build <!-- If you have a code sample, error messages, stack traces, please provide it here as well --<!-- A clear and concise description of what you expected to happen. --CMake configure passes. Please copy and paste the output from our (or fill out the checklist below manually). You can get the script and run it with: PyTorch Version (e.g., 1.0): v1.5.0-rc2 OS (e.g., Linux): Windows How you installed PyTorch (, , source): source Build command you used (if compiling from source): python build Python version: 3.x CUDA/cuDNN version: 9.2 / 10.1 / 10.2 / NA GPU models and configuration: Tesla M60 Any other relevant information: <!-- Add any other context about the problem here. --We should not build tests for mkldnn.
I have encountered the same error, how do I resolve it?
It is actually mkldnn that is out of sync. So running the following script should help.
Thanks a lot for this I posted this on the PyTorch Discussions but to no avail, and now I've finally understood the reason of this error. Thanks a lot. Surprisingly, for me the build works out if I build from the master repo, but changing to a different version always fails.
Change:
upstream="$1" pr="$2" git diff --name-only "$upstream" "$pr" <del> git diff --name-only "$upstream" "$pr" | grep -Eq '^(CMakeLists.txt|Makefile|.gitmodules|.jenkins/caffe2|binaries|caffe|caffe2|cmake|conda|docker|docs/caffe2|modules|scripts|third_party)' </del> <ins> # For safety, unconditionally trigger for any changes. #git diff --name-only "$upstream" "$pr" | grep -Eq '^(CMakeLists.txt|Makefile|.gitmodules|.jenkins/caffe2|binaries|caffe|caffe2|cmake|conda|docker|docs/caffe2|modules|scripts|third_party)' </ins>

Discussion:
This is most obvious by observing binary sizes, for example on pypi: (6.0 MB ) (33.8 MB ) Also, from the build log it's clear that 3.8 for some reason build with flag: CI cc
Which is a good old

Einsum currently modifies variables in-place (without an explicit indication that it does so), which prevents pytorch from automatically backpropagating. Results in the following runtime error: Demonstration and discussion can also be found . PyTorch or Caffe2: PyTorch How you installed PyTorch (conda, pip, source): pip OS: PyTorch version: 0.4.0 Python version: 3.6.1
Oops. Thank you for reporting and the minimal example. I'll see to get that fixed.
Saw the same issue over here. Thank you !
Is cloning the tensor before passing to einsum a valid workaround?
Yes, but a somewhat expensive one. If you have the ability to recompile, you could also apply the PR fixing this. Somehow it seems to be stuck in the review queue because it hit an unrelated CI problem...
Okay so how to use it then?
Just get a recent enough master and it will work.