jiebi/CodeConvo huggingface dataset:
Change:
template<typename FnType, FnType fn, typename ...Args> static PyObject* wrap_tuple_fn(Args ... args) { <del> PyObject *result = (*fn)(std::forward<Args>(args)...); </del> <ins> THPObjectPtr result((*fn)(std::forward<Args>(args)...)); </ins> if (!result) return NULL; <del> if (PyTuple_Check(result)) { return PyObject_CallFunctionObjArgs((PyObject*)&THPSizeType, result, NULL); </del> <ins> if (PyTuple_Check(result.get())) { return PyObject_CallFunctionObjArgs((PyObject*)&THPSizeType, result.get(), NULL); </ins> } <del> Py_INCREF(result); return result; </del> <ins> return result.release(); </ins> } static auto sq_concat = PyTuple_Type.tp_as_sequence->sq_concat;

Discussion:
I think I encountered a memory leak situation potentially due to . I create a to reproduce my observations. Basically, for three different (meaningless) models, once the function is used, the memory usage will keep increasing. To run the code, where the flag controls whether is used. Please run it long enough. I'm on Ubuntu 14.04 with python3.6. I tried two versions of pytorch, 0.1.122 and 0.1.12+. Side Note: Even without using , the memory usauge can also increase for a while, but it will finally stablize at a level. What's more, when (i.e. ) is used, it can take very long before the memory usage stablizes. I'm not sure whether this is a problem or not.
I think I'm sure there is a memory leak due to , and I found a weird way to do a temporary fix under pytorch version "0.1.122". The simplest example is as follows: Basically, if we replace with , and pass in an unpacked list of numbers, (i.e., instead of ), the memory leak is gone. Using will also have the leak issue. Unfortunately, I don't know why this is working. Finally, since there are so many in pytorch code, to do a quick hacky fix, one can change the function in the file as follows:
After some more research, I think we have a much bigger problem here. My current conjecture is that: whenever we use ( object) as an argument to a function, there will be a memory leak. An example is the function in , where of the code is which uses . Again, a minimal example for reproduction is as follows: As in the case, once I change the code to , memory leak is gone. So, in short, the problem is on the usage of .
The same issue. +1
Thanks for tracking this down and for the repro script!
Thanks for the quick fix! BTW, will the fix be included in the release version (0.1.12_?)?
it'll be released in
I see. Thanks again.
Change:
if momentum != 0: param_state = self.state[p] if 'momentum_buffer' not in param_state: <del> buf = param_state['momentum_buffer'] = d_p.clone() </del> <ins> buf = param_state['momentum_buffer'] = p.data.new().resize_as_(p.data).zero_() buf.mul_(momentum).add_(d_p) </ins> else: buf = param_state['momentum_buffer'] buf.mul_(momentum).add_(1 - dampening, d_p)

Discussion:
When using classic SGD optimizer with momentum with sparse embeddings the memory keeps garbage collecting / allocating leading to slow down and out of memory error eventually. ! ! The issue dissapears when momentum is not used ! or when embeddings are not sparse ! I'm using the last pytorch version on conda:
I tried out your script with momentum 0.1 on master, it takes roughly 10800mb gpu memory max. This is caused by using sparse buffer. I'm sending out a PR for this.
Change:
input_size=(1, 2, 4, 4, 4)), OldModuleTest(nn.VolumetricMaxPooling, (2, 2, 2), <del> input_size=(2, 3, 5, 5, 5)), </del> <ins> input=(torch.randn(2, 3, 5, 5, 5) * 1000)), </ins> OldModuleTest(nn.VolumetricMaxPooling, (2, 2, 2, 2, 2, 2), input_size=(2, 3, 5, 5, 5),

Discussion:
A few times, VolumetricMaxPooling (in ) kept failing on the continuous builds, and needed to investigate that. So it turns out, there were these one set of indices that were in the same Pooling window that differed by less than epsilon. So, the numeric gradient was hitting boundary conditions (max-pooling is discontinuous of course). ! ! The obvious fix is to modify the test to not have the input tensor have these boundary conditions. However, as an aside, I am not sure why the boundary condition is only hit sometimes and not always, because the same random numbers are generated always, and division is unstable but it is deterministic (right? or have i always got this wrong)
Repeat running will produce error occasional , more detail:
Change:
above the diagonal. The argument :attr:`diagonal` controls which diagonal to consider. If <del> :attr:`diagonal` = 0, all elements on and below the main diagonal are </del> <ins> :attr:`diagonal` = 0, all elements on and above the main diagonal are </ins> retained. A positive value excludes just as many diagonals above the main diagonal, and similarly a negative value includes just as many diagonals below the main diagonal. The main diagonal are the set of indices

Discussion:
! It seems that the highlighted "below" should be "upper“. <!-- A clear and concise description of what content in is an issue. If this has to do with the general website, please file an issue at instead. If this has to do with , please file an issue at --
Hi, thank you for bringing this up. I'll send in a patch to fix it.
Change:
unset(CUDA_ARCH_PTX CACHE) endif() <del> if(DEFINED ENV{TORCH_CUDA_ARCH_LIST}) </del> <ins> if($ENV{TORCH_CUDA_ARCH_LIST}) </ins> # Pass CUDA architecture directly set(__cuda_arch_bin $ENV{TORCH_CUDA_ARCH_LIST}) message(STATUS "Set CUDA arch from TORCH_CUDA_ARCH_LIST: ${__cuda_arch_bin}")

Discussion:
This is most obvious by observing binary sizes, for example on pypi: (6.0 MB ) (33.8 MB ) Also, from the build log it's clear that 3.8 for some reason build with flag: CI cc
Which is a good old

Einsum currently modifies variables in-place (without an explicit indication that it does so), which prevents pytorch from automatically backpropagating. Results in the following runtime error: Demonstration and discussion can also be found . PyTorch or Caffe2: PyTorch How you installed PyTorch (conda, pip, source): pip OS: PyTorch version: 0.4.0 Python version: 3.6.1
Oops. Thank you for reporting and the minimal example. I'll see to get that fixed.
Saw the same issue over here. Thank you !
Is cloning the tensor before passing to einsum a valid workaround?
Yes, but a somewhat expensive one. If you have the ability to recompile, you could also apply the PR fixing this. Somehow it seems to be stuck in the review queue because it hit an unrelated CI problem...
Okay so how to use it then?
Just get a recent enough master and it will work.
Change:
'Cumprod', 'Gather', 'MaskedScatter', <del> 'Sort', 'Topk', </del> 'Norm', 'Prod', ))

Discussion:
Just a very small grammatical issue. ! Suggested fix: "mutates" to "mutate" <!-- A clear and concise description of what content in is an issue. If this has to do with the general website, please file an issue at instead. If this has to do with , please file an issue at --
Thanks for pointing this out. Please feel free to open a pull request for this.
Thanks for adding labels. Forgot to do that. I will make a pull request
Change:
self += parameters def __getitem__(self, idx): <del> if not (-len(self) <= idx < len(self)): raise IndexError('index {} is out of range'.format(idx)) if idx < 0: idx += len(self) return self._parameters[str(idx)] </del> <ins> if isinstance(idx, slice): return ParameterList(list(self._parameters.values())[idx]) else: if not (-len(self) <= idx < len(self)): raise IndexError('index {} is out of range'.format(idx)) if idx < 0: idx += len(self) return self._parameters[str(idx)] </ins> def __setitem__(self, idx, param): return self.register_parameter(str(idx), param)

Discussion:
it'd be great to pick out elements of of a sequential with array slicing:
I agree this idea!
However,
this has come up again (last in ). Shall we reconsider?
I attempted getting a to work. Is this fine? If this is fine, I can issue a PR too. Let me know what you think. I believe this modification needs to be done in the class.
cc: This issue has been open for quite sometime.
Change:
upstream="$1" pr="$2" git diff --name-only "$upstream" "$pr" <del> git diff --name-only "$upstream" "$pr" | grep -Eq '^(aten/|caffe2/|.jenkins/pytorch|docs/(make.bat|Makefile|requirements.txt|source)|mypy|requirements.txt|setup.py|test/|third_party/|tools/|.gitmodules|torch/)' </del> <ins> # Now that PyTorch build depends on Caffe2, unconditionally trigger # for any changes. # TODO: Replace this with a NEGATIVE regex that allows us to blacklist # files (letting us skip builds when they are unnecessary) #git diff --name-only "$upstream" "$pr" | grep -Eq '^(aten/|caffe2/|.jenkins/pytorch|docs/(make.bat|Makefile|requirements.txt|source)|mypy|requirements.txt|setup.py|test/|third_party/|tools/|.gitmodules|torch/)' </ins>

Discussion:
This is most obvious by observing binary sizes, for example on pypi: (6.0 MB ) (33.8 MB ) Also, from the build log it's clear that 3.8 for some reason build with flag: CI cc
Which is a good old

Einsum currently modifies variables in-place (without an explicit indication that it does so), which prevents pytorch from automatically backpropagating. Results in the following runtime error: Demonstration and discussion can also be found . PyTorch or Caffe2: PyTorch How you installed PyTorch (conda, pip, source): pip OS: PyTorch version: 0.4.0 Python version: 3.6.1
Oops. Thank you for reporting and the minimal example. I'll see to get that fixed.
Saw the same issue over here. Thank you !
Is cloning the tensor before passing to einsum a valid workaround?
Yes, but a somewhat expensive one. If you have the ability to recompile, you could also apply the PR fixing this. Somehow it seems to be stuck in the review queue because it hit an unrelated CI problem...
Okay so how to use it then?
Just get a recent enough master and it will work.
Change:
- **c_n** (num_layers * num_directions, batch, hidden_size): tensor containing the cell state for t=seq_len Attributes: <del> weight_ih_l[k] : the learnable input-hidden weights of the k-th layer `(W_ir|W_ii|W_in)`, of shape `(input_size x 3*hidden_size)` weight_hh_l[k] : the learnable hidden-hidden weights of the k-th layer `(W_hr|W_hi|W_hn)`, of shape `(hidden_size x 3*hidden_size)` bias_ih_l[k] : the learnable input-hidden bias of the k-th layer `(b_ir|b_ii|b_in)`, of shape `(3*hidden_size)` bias_hh_l[k] : the learnable hidden-hidden bias of the k-th layer `(W_hr|W_hi|W_hn)`, of shape `(3*hidden_size)` </del> <ins> weight_ih_l[k] : the learnable input-hidden weights of the k-th layer `(W_ii|W_if|W_ig|W_io)`, of shape `(input_size x 4*hidden_size)` weight_hh_l[k] : the learnable hidden-hidden weights of the k-th layer `(W_hi|W_hf|W_hg|W_ho)`, of shape `(hidden_size x 4*hidden_size)` bias_ih_l[k] : the learnable input-hidden bias of the k-th layer `(b_ii|b_if|b_ig|b_io)`, of shape `(4*hidden_size)` bias_hh_l[k] : the learnable hidden-hidden bias of the k-th layer `(W_hi|W_hf|W_hg|b_ho)`, of shape `(4*hidden_size)` </ins> Examples::

Discussion:
From : Happened because of some change in the environment as it is also affecting historical builds. cc
This issue seems to come from within libuv, a dependency of TensorPipe. First guess: there's two conflicting versions of libuv (e.g., a preinstalled one and the one shipped as a submodule of TensorPipe), and we build the one shipped by TensorPipe but somehow it ends up including the headers of the other one? In particular, the constant it's complaining about was in , which became part of libuv 1.41.0, which is the version that TensorPipe bundles. But it seems those jobs are installing version 1.40.0 via conda: So the question is why are the conda headers picked up and prioritized? I feel this will require some CMake digging, and it's possible that the fix will have to land in libuv's upstream CMake files. Ugh.
BTW, if you want to validate this theory you can try setting this flag to : Which will cause TensorPipe to use the preinstalled libuv instead of building its own one. But please don't land this "workaround", as it will only lead to users coming to complain that they couldn't build PyTorch because their system has a libuv version that is too old for what TensorPipe needs.
this is definitely what is happening: libuv-1.40 is installed from conda, which does not have in the header: But I'm still confused why it started to happen just now [Edit] Can it be update that happened 4 hours ago?
Finally found a PR that caused a regression: (and testing were clearly regressed in CI: )
Change:
// a test for Py_None here; instead, you need to mark the argument // as *allowing none*; you can do this by writing 'Tensor?' instead // of 'Tensor' in the ATen metadata. <del> throw TypeError("expected Variable as argument %d, but got %s", i, </del> <ins> throw TypeError("expected Tensor as argument %d, but got %s", i, </ins> Py_TYPE(args[i])->tp_name); } return reinterpret_cast<THPVariable*>(args[i])->cdata;

Discussion:
Considering the merge of and , and the "removal" of from the API, there are still some messages that reference . This might prove to be confusing for new users starting off from , as they probably might not have heard of . I did a code search to find out multiple instances of this in . PyTorch or Caffe2: PyTorch How you installed PyTorch (conda, pip, source): source (@ ) Build command you used (if compiling from source): OS: Ubuntu 16.04.2 PyTorch version: Python version: 3.5 CUDA/cuDNN version: 8.0.44 GPU models and configuration: GeForce 940M GCC version (if compiling from source): 5.4.0 CMake version: 3.5.1
Change:
self.assertEqual(torch.mm(flattened_tensor, flattened_tensor.t()), torch.eye(rows) * gain ** 2, prec=1e-6) <ins> # Generates rand tensor with non-equal values. This ensures that duplicate # values won't be causing test failure for modules like MaxPooling. # size should be small, otherwise randperm fails / long overflows. def _rand_tensor_non_equal(*size): total = reduce(mul, size, 1) return torch.randperm(total).view(*size).double() </ins> def add_test(test): test_name = test.get_name()

Discussion:
Intermittent test failures occurring on AdaptiveMaxPool3d. and I have seen this in the past where Max pooling has exactly the same values in the same window, then it's differently resolved on CPU/GPU. For example [0 2.5 2.5 3] can give a max index of or on CUDA depending on the runtime. So generating input values that are within epsilon of each other is important (especially at half precision). fix that one.
Will do. Thanks for identifying the issue.
Change:
@skipIfNoLapack def test_gels(self): <del> def _test(a, b, expectedNorm): </del> <ins> def _test_underdetermined(a, b, expectedNorm): m = a.size()[0] n = a.size()[1] assert(m <= n) </ins> a_copy = a.clone() b_copy = b.clone() res1 = torch.gels(b, a)[0]

Discussion:
b = (600,1) A = (600,2) X, = (b,A) X.size() #torch.Size([2, 1]) Y, = (b.cuda(),A.cuda()) Y.size() #torch.Size([600, 1])
that is interesting. says that the first rows returned ( here) include the solution of the least-squares problem. I've verified that the first 2 rows of X and Y are the same so you can probably use that for now while we figure out what the rest of the output is.
Change:
self.assertEqual(reference[2, 2, 2, ...], 27, 0) self.assertEqual(reference[...], reference, 0) <del> reference_5d = self._consecutive((3, 3, 3, 3, 3)) </del> <ins> reference_5d = conv_fn(consec((3, 3, 3, 3, 3))) </ins> self.assertEqual(reference_5d[..., 1, 0], reference_5d[:, :, :, 1, 0], 0) self.assertEqual(reference_5d[2, ..., 1, 0], reference_5d[2, :, :, 1, 0], 0) self.assertEqual(reference_5d[2, 1, 0, ..., 1], reference_5d[2, 1, 0, :, 1], 0) self.assertEqual(reference_5d[...], reference_5d, 0) # LongTensor indexing <del> reference = self._consecutive((5, 5, 5)) idx = torch.LongTensor([2, 4]) </del> <ins> reference = conv_fn(consec((5, 5, 5))) idx = conv_fn(torch.LongTensor([2, 4])) </ins> self.assertEqual(reference[idx], torch.stack([reference[2], reference[4]])) # TODO: enable one indexing is implemented like in numpy # self.assertEqual(reference[2, idx], torch.stack([reference[2, 2], reference[2, 4]]))

Discussion:
Arguably, indexing a Tensor by a LongTensor with no dimension should return a LongTensor with no dimension. Consider the following case: This breaks indexing by indices when the list of indices is empty, which can be hard to detect. In my use-case, I use such a masking to select rows of a tensor corresponding to "valid" entries in a vector; I had not considered that it would not work when there are no valid entries.
Thanks! I'll take a look.
Change:
return linearIndex; } <ins> static bool hasEmptyTensor(TensorList tensors) { for (auto& tensor : tensors) { if (tensor.defined() && tensor.numel() == 0) { return true; } } return false; } </ins> static std::tuple<Tensor, Tensor> makeLinearIndex(Tensor self, TensorList orig) { <ins> checkIndexTensorTypes(orig); </ins> // first expand ByteTensor (boolean masks) into 1 or more LongTensors auto indices = expandByteTensors(self, orig); <ins> if (hasEmptyTensor(indices)) { return std::make_tuple(self, self.type().toScalarType(kLong).tensor()); } </ins> // next broadcast all index tensors together indices = expand_outplace(indices); // add missing null Tensors so that it matches self.dim()

Discussion:
In PyTorch master 0.4 , variables cannot be indexed. This used to work in PyTorch 0.2 :
This is breaking some of Pyro's inference algorithms on PyTorch master:
Change:
return() endif() set(CAFFE2_FOUND_CUDA TRUE) <ins> message(STATUS "Caffe2: CUDA detected: " ${CUDA_VERSION}) message(STATUS "Caffe2: CUDA nvcc is: " ${CUDA_NVCC_EXECUTABLE}) message(STATUS "Caffe2: CUDA toolkit directory: " ${CUDA_TOOLKIT_ROOT_DIR}) if(CUDA_FOUND) # Sometimes, we may mismatch nvcc with the CUDA headers we are # compiling with, e.g., if a ccache nvcc is fed to us by CUDA_NVCC_EXECUTABLE # but the PATH is not consistent with CUDA_HOME.  It's better safe # than sorry: make sure everything is consistent. set(file "${PROJECT_BINARY_DIR}/detect_cuda_version.c") file(WRITE ${file} "" "#include <cuda.h>n" "#include <stdio.h>n" "int main() {n" "  printf("%d.%d", CUDA_VERSION / 1000, (CUDA_VERSION / 10) % 100);n" "  return 0;n" "}n" ) try_run(run_result compile_result ${PROJECT_BINARY_DIR} ${file} CMAKE_FLAGS "-DINCLUDE_DIRECTORIES=${CUDA_INCLUDE_DIRS}" LINK_LIBRARIES ${CUDA_LIBRARIES} RUN_OUTPUT_VARIABLE cuda_version_from_header COMPILE_OUTPUT_VARIABLE output_var ) if(NOT compile_result) message(FATAL_ERROR "Caffe2: Couldn't determine version from header: " ${output_var}) endif() message(STATUS "Caffe2: Header version is: " ${cuda_version_from_header}) if(NOT ${cuda_version_from_header} STREQUAL ${CUDA_VERSION}) # Force CUDA to be processed for again next time # TODO: I'm not sure if this counts as an implementation detail of # FindCUDA set(${cuda_version_from_findcuda} ${CUDA_VERSION}) unset(CUDA_TOOLKIT_ROOT_DIR_INTERNAL CACHE) # Not strictly necessary, but for good luck. unset(CUDA_VERSION CACHE) # Error out message(FATAL_ERROR "FindCUDA says CUDA version is ${cuda_version_from_findcuda} (usually determined by nvcc), " "but the CUDA headers say the version is ${cuda_version_from_header}.  This often occurs " "when you set both CUDA_HOME and CUDA_NVCC_EXECUTABLE to " "non-standard locations, without also setting PATH to point to the correct nvcc.  " "Perhaps, try re-running this command again with PATH=${CUDA_TOOLKIT_ROOT_DIR}/bin:$PATH.  " "See above log messages for more diagnostics, and see https://github.com/pytorch/pytorch/issues/8092 for more details.") endif() endif() </ins> # Find cuDNN. if(CAFFE2_STATIC_LINK_CUDA)

Discussion:
Here's the situation: Ideally, you'd say something like to make a build of PyTorch with a non-standard version of CUDA. When this happens, needs to find the correct copy of nvcc to use. If you are using ccache as per the instructions in you'll have set to your ccache binary, so that ccache happens. cmake will correctly respect this variable. However, here's the problem: ccache's nvcc will use your path to determine what the real nvcc to use is, this means that you'll pick up whatever nvcc is in your PATH, which is probably not the one you want, because you're trying to use a different nvcc. So in sum, you have to do the non-intuitive to get the whole shebang to work. The worst part is that it won't tell you if you've got it wrong, it will happily use the wrong nvcc. I haven't gone through user submitted issues to see if anyone has succumbed to this particular bug.
A long-term solution might be getting to work
works now.
Change:
(4.53, 3.83, -6.64))).t() b = torch.Tensor(((8.58, 8.26, 8.48), (9.35, -4.43, -0.70))).t() <del> _test(a, b, expectedNorm) </del> <ins> _test_underdetermined(a, b, expectedNorm) </ins> # test reuse expectedNorm = 0

Discussion:
b = (600,1) A = (600,2) X, = (b,A) X.size() #torch.Size([2, 1]) Y, = (b.cuda(),A.cuda()) Y.size() #torch.Size([600, 1])
that is interesting. says that the first rows returned ( here) include the solution of the least-squares problem. I've verified that the first 2 rows of X and Y are the same so you can probably use that for now while we figure out what the rest of the output is.
Change:
n_dist = THTensor_(size)(prob_dist, 0); n_categories = THTensor_(size)(prob_dist, 1); <del> THArgCheck(n_sample > 0, 2, "cannot sample n_sample < 0 samples"); </del> <ins> THArgCheckWithCleanup(n_sample > 0, THCleanup(if (start_dim == 1) THTensor_(resize1d)(prob_dist, n_categories);), 2, "cannot sample n_sample < 0 samples"); </ins> if (!with_replacement) { <del> THArgCheck((!with_replacement) && (n_sample <= n_categories), 2,  "cannot sample n_sample > prob_dist:size(1) samples without replacement"); </del> <ins> THArgCheckWithCleanup((!with_replacement) && (n_sample <= n_categories), THCleanup(if (start_dim == 1) THTensor_(resize1d)(prob_dist, n_categories);), 2, "cannot sample n_sample > prob_dist:size(1) samples without replacement"); </ins> } /* cumulative probability distribution vector */

Discussion:
I was reading the documentation of method in the . I tried this in IPython environment. Of course, error occurs because 5 is larger than the size of weights and I didn't set . The error message is like this: But after this, the dimensions of weights changed. If I run as following without runtime exception, then the dimensions of weights do not change. This is caused by not cleaning up after exception raised in this . Maybe the solution is adding cleaning up code in the .
The problem emerged in two other situations as well, here are the examples: I'm sending a pull request with the fix. Thanks for reporting the problem. :)
Change:
return r; } <del> std::vector<Tensor> as_tensor_list(std::vector<Variable> &vars) { </del> <ins> static std::vector<Tensor> as_tensor_list(std::vector<Variable> &vars) { </ins> std::vector<Tensor> tensors; for (auto& v : vars) { tensors.emplace_back(std::move(v));

Discussion:
See Docker run fails with , which is due to the fact that is specified in docker container: Which results in: CI cc
Check that adding to manywheels image results in a failure:
Tentatively marking hi-pri since nightly testing are failing
This is resolved, here is the successful run: