jiebi/CodeConvo huggingface dataset:
Change:
r""" expm1_() -> Tensor <del> In-place version of :meth:`~Tensor.exp` </del> <ins> In-place version of :meth:`~Tensor.expm1` </ins> """) add_docstr_all('exponential_',

Discussion:
Build error: According to , the should be modified as:
I the same problem when installing pytorch from source. You mean the file is wrong?
Although the PyYAML documentation site suggests that is possible, I could not perform it. That is a reason for the failing. This has been reported as well: I believe that the should have this instead: In fact when I did this: This indicates that the import didn't happen in PyYAML itself.
you guys are right, the case has to be modified to: . The original issue (my guess) is that your pyyaml had some compile error and the CLoader wasn't successfully installed.
Change:
if modules is None: raise AttributeError( "cannot assign module before Module.__init__() call") <ins> remove_from(self.__dict__, self._parameters, self._buffers) </ins> modules[name] = value elif modules is not None and name in modules: if value is not None:

Discussion:
They do one pair of these calls for each slicing operation.
Does this actually have a performance impact? These calls tend to be very quick
Yes, it's a major reason why small unrolled LSTMs (which involve in addition to two GEMMs: a chunk into four gates, for 4 pairs of calls, five unary pointwise ops, for 5 more, and five binary pointwise ops, for another 5) are much slower than they are in Theano and TensorFlow.
Also, there are excessive cudaGetDevice calls for each pointwise operation (9 cudaGetDevice) even without chunking, as in the following repro code: As for small unrolled LSTMs it does show up.
This should no longer be an issue, now that is implemented as a native ATen function. I think we're still using the Python version for tensors, but this will be resolved soon when we stop using tensors altogether.
Change:
fi if [[ "$BUILD_ENVIRONMENT" == *pytorch-linux-xenial-cuda9-cudnn7-py3 ]] ||  <del> [[ "$BUILD_ENVIRONMENT" == *pytorch-linux-trusty-py3.6-gcc7.2 ]]; then </del> <ins> [[ "$BUILD_ENVIRONMENT" == *pytorch-linux-trusty-py3.6-gcc7* ]]; then </ins> BUILD_TEST_LIBTORCH=1 else BUILD_TEST_LIBTORCH=0

Discussion:
We're not testing against CUDA 9.2 and we should. C.f. We should test against GCC 7 (because this is how Anaconda is building the packages), AND we should use test with this in OS X, because CUDA 9.2 now supports the default xcode shipped on OS X.
Building PyTorch with CUDA 9.2 for 7.0 compute compatibility devices produces quite worrisome warnings: warning : Instruction 'vote' without '.sync' may produce unpredictable results on sm70 and later architectures warning : Instruction 'vote' without '.sync' is deprecated since PTX ISA version 6.0 and will be discontinued in a future PTX ISA version warning : Instruction 'shfl' without '.sync' is deprecated since PTX ISA version 6.0 and will be discontinued in a future PTX ISA version warning : Instruction 'shfl' without '.sync' may produce unpredictable results on sm70 and later architectures
CUDA 9.2 CI on macOS is done in
Change:
This is the reverse operation of the manner described in :meth:`~Tensor.gather`. <del> :attr:`self`, :attr:`index` and :attr:`src` should have same number of dimensions. It is also required that `index.size(d) <= src.size(d)` for all dimensions `d`, and that `index.size(d) <= self.size(d)` for all dimensions `d != dim`. </del> <ins> :attr:`self`, :attr:`index` and :attr:`src` (if it is a Tensor) should have same number of dimensions. It is also required that ``index.size(d) <= src.size(d)`` for all dimensions ``d``, and that ``index.size(d) <= self.size(d)`` for all dimensions ``d != dim``. </ins> Moreover, as for :meth:`~Tensor.gather`, the values of :attr:`index` must be <del> between `0` and `(self.size(dim) -1)` inclusive, and all values in a row along the specified dimension :attr:`dim` must be unique. </del> <ins> between ``0`` and ``self.size(dim) - 1`` inclusive, and all values in a row along the specified dimension :attr:`dim` must be unique. </ins> Args: dim (int): the axis along which to index

Discussion:
is not available in the docs page, but the doc string has been in . This was mentioned earlier in

Oh, bilinear, I missed that last time I looked at it. I'll add a docstring.

returns nothing. Might be useful to have some linter in the build process to find Python functions missing

Pytorch 0.4.0 gives a segmentation fault with dilated 2D convolutions over a specific size. import torch import as nn model = nn.Conv2d(inchannels=64, outchannels=64, kernelsize=5, dilation=2) model(([1, 64, 1166, 1166])) # This runs model(([1, 64, 1167, 1167])) # This crashes PyTorch version: 0.4.0 Is debug build: No CUDA used to build PyTorch: 9.1.85 OS: CentOS Linux 7 (Core) GCC version: (GCC) 4.8.5 (Red Hat 4.8.5-28) CMake version: version 2.8.12.2 Python version: 3.6 Is CUDA available: Yes CUDA runtime version: 8.0.61 GPU models and configuration: GPU 0: GeForce GTX 1080 Ti GPU 1: TITAN X (Pascal) GPU 2: Quadro K2200 Nvidia driver version: 390.59 cuDNN version: Probably one of the following: /usr/local/cuda- /usr/local/cuda- /usr/local/cuda-7.5/lib64/libcudnnstatic.a /usr/local/cuda- /usr/local/cuda-8.0/lib64/libcudnn_static.a
do you think this is the same as the segfault you showed me a week ago on convolution?
ah yes. int overflow. I only fixed conv3d because I didn’t want to hurt perf. I’ll fix all of them.

This should be an easy one.. The function is implemented, just no sign of it in the docs.
Change:
*tempValues__data = *t_data; *tempIndices__data = *tempIndices__dimOffset; }); <ins> THTensor_(free)(tempValues_); THLongTensor_free(tempIndices_); </ins> } if (!keepdim) {

Discussion:
There is a possible memory leak in () This causes memory to blow up: When using () memory usage is ok There is no leak for unitary dimensions
Change:
import ctypes import warnings import torch.cuda <ins> import sys </ins> import os.path as path lib = None <del> # TODO: fix libname for OSX / Windows </del> <ins> # TODO: fix libname for Windows </ins> # TODO: dynamic version checks via cudnnGetVersion # TODO: load 5.1.3 if using CUDA 7.5 and 5.1.5 if using CUDA 8.0 thisdir = path.dirname(__file__) libpaths = ['', path.join(thisdir, '../../lib')] <del> libnames = ['libcudnn.so.5.1.5', 'libcudnn.so.5.1.3'] </del> <ins> if sys.platform.startswith('linux'): libnames = ['libcudnn.so.5.1.5', 'libcudnn.so.5.1.3'] elif sys.platform == 'darwin': libnames = ['libcudnn.5.dylib'] else: libnames = [] </ins> def _loadlib(): global lib

Discussion:
The official website does not provide the torchvison version installation method. Here is the error message when torchvison 0.3.0 is used. torchvision/C.cpython-36m-x8664-linux- undefined symbol: ZN2at7getTypeERKNS6TensorE That should be caused by the mismatch version between pytorch-nightly and torchvision 0.3.0.
It should be that matches with , but they are not provided currently.
I tried installing it with pip as per , but now I get when importing . I did a similar thing on June 25th. It worked then. Any known fix for this?
do we document what to do here?
I finally solved this by building from source with Before running that make sure to upgrade and to use . Note: this will not work if you have on your system when building since you'd get version mismatch between and . Tested only on .
you build pytorch from source or just torchvision? or both of them.
I installed from conda with: I then cloned the repo and ran inside the repo. I guess this would also work if you do , but I haven't tried it.
Just a reminder (what I encountered just now): one should uninstall previous ` from source, or they can mix up and mess up everything.
There are nightlies for torchvision under Windows now.
I have problems using Pytorch with my Geforce RTX 3060, (sm86). I tried to install with help of : Create a new venv, based on python 3.7.10. Activate it. conda install pytorch-nightly cudatoolkit=10.0 -c pytorch (I also did it without the "=10.0") After this, Pytorch detects the sm86 with no errors, problem sm86 solved with pytorch-nightly Now, I have a problem with torchvision. tried installing torchvision with File "C:repovisiontorchvisionmodels", line 195, in Inception3 AttributeError: module '' has no attribute 'unused' tried installing torchvision version 0.3.0, from , but an error rose: tried installing torchvision at origin/master, from , but another error rose: Here is the list of packages installed:
Change:
def _test_sparse_mask_fixed(self): i = self.IndexTensor([ <del> [1, 3, 3, 0, 4], [2, 1, 1, 2, 3], </del> <ins> [1, 3, 0, 4], [2, 1, 2, 3], </ins> ]) <del> v = self.ValueTensor([1, 2, 3, 4, 5]) x = self.SparseTensor(i, v, torch.Size([5, 4])) </del> <ins> v = self.ValueTensor([1, 2, 3, 4]) x = self.SparseTensor(i, v, torch.Size([5, 4])).coalesce() </ins> dense = self.ValueTensor([ [1, 2, 3, 4], [5, 6, 7, 8],

Discussion:
and I talked about the intended semantics of sparsemask when given a sparse tensor as an argument, and we came to the following conclusions: with an uncoalesced sparse tensor does not make sense, full stop. Unlike values, where your update might be linear and work correctly on an uncoalesced sparse tensor, there isn't any good reason why you want to pluck out values from the dense tensor multiple times, it is basically always a bug. the (eventual) documentation for sparsemask, we will recommend calling coalesce before doing any operations involving sparsemask (or nonlinear updates to indices/values). should add a test to sparsemask which checks if the flag of the input tensor is true, and raise an error if it is not. This is a cheap safety check which will prevent people from forgetting to call coalesce (as in (2)). There is a performance consequence to calling coalesce every iteration of training. If the gradient is producing coalesced vectors, we can alleviate this problem by adding a preprocessing step to coalesce() which tests if the input is already coalesced, and bugs out in that case. However, I think this is out of scope for this particular ticket for now.
Change:
from torch.autograd import Variable from torch.nn import Parameter from common_nn import NNTestCase, ModuleTest, CriterionTest, TestBase,  <del> module_tests, criterion_tests, TEST_CUDA, TEST_MULTIGPU, TEST_CUDNN, PRECISION </del> <ins> module_tests, criterion_tests, TEST_CUDA, TEST_MULTIGPU, TEST_CUDNN,  TEST_CUDNN_VERSION, PRECISION </ins> from common import freeze_rng_state, run_tests

Discussion:
cuDNN backend for RNN stabilizes at large mem usage. The following code stabilizes for me at 14+GiB reported by nvidia-smi. The same RNN without cuDNN uses <2.5GiB.
Oops... any idea what's going on with the garbage collector in this case?
are you running this on a P100 GPU?
Yes, I observe similar behavior on Titan X Pascal
hmmm, do cudnn RNNs have weird memory-oversubscribing properties? If it wasn't for CuDNN RNN, then it's not unreasonable. we use a custom CUDA allocator that will oversubscribe if needed, but will free the memory blocks it's holding onto if it hits OOM / failed allocations.
Tried the same RNN in torch and am getting ~2.6GiB on nvidia-smi.
With cuDNN I'm getting a constant usage of 2241MB on my machine using PyTorch (M40).
are you using cudnnv5, or v6?
v5
no, cudnn does not do memory oversubscribing, and already tried the same network in lua torch with cudnn, where it gets expected memory usage. Also, he is getting OOMs on slightly bigger networks, where apparently caching allocator should have tried to free previous blocks. Interesting that can not repro this.
Just to make sure did you set when you ran? I left it as False in the repro.
I can repro in the docker image built from the current version on P100, and I go OOM after 8 batches with the same image on 1080, which generally should not happen - if there is enough memory to go through 1 batch, memory usage should not increase afterwards.
No, I just copied your script and ran it. Weird thing is that it goes OOM after 18 batches once I enable . Without it memory remains stable.
A lot of cudaMalloc allocations go not through THCudaMalloc, and never get freed:
So not going through THCudaMalloc is actually expected for storages allocated with THCStorage_(newWithSize), but what's not expected is to every time cudaMalloc bytes of rnn workspace, instead of reusing earlier allocations. (~784 MB allocations are for workspace, ~156MB allocations happen somewhere in criterion).
My guess is that the bug is at the Python level.

Either: Something is holding on to a Tensor longer than it should Something is creating a circular reference which prevents quick clean-up
The issue is that in creates cyclic references. It's made worse by the fact that it holds on to more memory than it should (like the workspace). You can see this by adding a as the first line in your for loop. With the explicit call to the GC every iteration, the memory usage doesn't explode.
how can these cyclic references be fixed?
Not sure yet, but I will fix it tomorrow with
Change:
} std::string FunctionSignature::toString() const { <ins> // TODO: consider printing more proper schema strings with defaults, optionals, etc. </ins> std::ostringstream ss; <ins> bool keyword_already = false; </ins> ss << "("; int i = 0; for (auto& param : params) { if (i != 0) { ss << ", "; } <ins> if (param.keyword_only && !keyword_already) { ss << "*, "; keyword_already = true; } </ins> ss << param.type_name() << " " << param.name; i++; }

Discussion:
Consider the following: A reasonable reading of this message is that the order of the parameters should be reversed, but this isn't equivalent: The problem is that the second signature has a keyword-only argument but it's printed. It also doesn't include defaults and such, but that's probably secondary. In this case the problem is at least flagged, but it's not clear to me whether there are other deprecated signatures that silently suggest the wrong thing. Please copy and paste the output from our (or fill out the checklist below manually). You can get the script and run it with: Collecting environment information... PyTorch version: 1.5.0a0+ Is debug build: Yes CUDA used to build PyTorch: Could not collect OS: CentOS Linux 7 (Core) GCC version: (GCC) 7.4.0 CMake version: version 3.14.0 Python version: 3.7 Is CUDA available: No CUDA runtime version: 9.2.88 GPU models and configuration: GPU 0: Tesla M40 GPU 1: Tesla M40 Nvidia driver version: 396.69 cuDNN version: /usr/local/cuda-9.2/targets/x8664- Versions of relevant libraries: [pip] numpy==1.17.2 [pip] torch==1.5.0a0+ [conda] blas 1.0 mkl [conda] mkl 2019.4 243 [conda] mkl-include 2019.4 243 [conda] mkl-service 2.3.0 py37he904b0f0 [conda] mklfft 1.0.14 py37ha843d7b0 [conda] mklrandom 1.1.0 py37hd6b4f25_0 [conda] torch 1.5.0a0+ <pip<!-- Add any other context about the problem here. --
Change:
gI = apply_fn<Transpose>(0, 1)(gIt); } } <ins> if (should_compute_output(0) && !ggO.defined()) ggO = at::zeros_like(gO); if (should_compute_output(1) && !gI.defined()) gI = at::zeros_like(input); if (should_compute_output(2) && !gW.defined()) gW = at::zeros_like(weight); </ins> return {ggO, gI, gW}; }

Discussion:
The following code: produces this error Also, removing the ReLU from the network: causes the error to change to: However, when the second convolution is removed as well: then the code runs error free I'm on a ppc64le architecture with a P100 GPU, CUDA 8.0 and PyTorch version '0.4.0a0+'
I'm also having this problem using branch v0.3.0
i could reproduce this as well. we'll look into this.
I'm having the same error :/
same error with gradient penalty for WGAN
When you execute , are you cause memory leak
no there shouldn't be a memory leak.
I am having an out of memory error as well, only when applying backward to the output of this method. I am using 3 titans GPUs so memory is not a problem I would assume. def computeGP(netD, realdata, realembed, fakedata, LAMBDA): BATCHSIZE = (0) alpha = (BATCHSIZE, 1) alpha = (BATCHSIZE, int(realdata.nelement() / BATCHSIZE)).contiguous().view(BATCHSIZE, 3, 64, 64) alpha = () interpolates = alpha realdata + ((1 - alpha) fakedata) interpolates = () interpolates = autograd.Variable(interpolates, requiresgrad=True) discinterpolates, = netD(interpolates, realembed) gradients = (outputs=discinterpolates, inputs=interpolates, (()).cuda(), creategraph=True, retaingraph=True, onlyinputs=True)[0] gradientpenalty = (((2, dim=1) - 1) * 2).mean() LAMBDA return gradient_penalty `
I am also using it with gradient penalty for WGAN, But When I use gradient_penalty.backward(), the memory leak is happen, My pytorch is 0.4.0a0+.
I can reproduce the memory leak with this: However the leak disappears when only one convolution is used: Also, if nn.Linear is used, there is no leak. Torch version: 0.4.0a0+
I've opened a separate issue for that memory leak:
Change:
results = {} def _worker(i, module, input, kwargs, results, lock): <del> var_input = input while not isinstance(var_input, Variable): var_input = var_input[0] </del> <ins> var_input = get_a_var(input) </ins> try: with torch.cuda.device_of(var_input): output = module(*input, **kwargs)

Discussion:
is the culprit. Assumes that the first input is a variable or a sequence containing a variable in index 0 (recursively indexed). Workaround is to pass a dummy variable as first input, as suggested in
Could you provide a short reproducer for testing? Think this shouldn't be too hard to fix.
Sure, here's a gist along with commands/outputs:
fixed in master via
Change:
PyObject *THPFunction_saved_tensors(THPFunction *self, void *_unused) { <del> THPUtils_assert(!self->has_freed_buffers, "Trying to backward through the " "graph second time, but the buffers have already been freed. Please " "specify retain_variables=True when calling backward for the first time."); </del> <ins> THPUtils_assert(!self->has_freed_buffers, ERR_BACKWARD_TWICE); </ins> if (!self->saved_variables) return PyTuple_New(0);

Discussion:
Usually calling backwards twice gives this message: But conv2d just segfaults. $ pt A B Segmentation fault (core dumped)
Change:
import torch.optim as optim import torch.legacy.optim as old_optim from torch.autograd import Variable <ins> from torch import sparse </ins> from common import TestCase, run_tests

Discussion:
and I talked about the intended semantics of sparsemask when given a sparse tensor as an argument, and we came to the following conclusions: with an uncoalesced sparse tensor does not make sense, full stop. Unlike values, where your update might be linear and work correctly on an uncoalesced sparse tensor, there isn't any good reason why you want to pluck out values from the dense tensor multiple times, it is basically always a bug. the (eventual) documentation for sparsemask, we will recommend calling coalesce before doing any operations involving sparsemask (or nonlinear updates to indices/values). should add a test to sparsemask which checks if the flag of the input tensor is true, and raise an error if it is not. This is a cheap safety check which will prevent people from forgetting to call coalesce (as in (2)). There is a performance consequence to calling coalesce every iteration of training. If the gradient is producing coalesced vectors, we can alleviate this problem by adding a preprocessing step to coalesce() which tests if the input is already coalesced, and bugs out in that case. However, I think this is out of scope for this particular ticket for now.
Change:
PyArrayObject *array_ptr = (PyArrayObject*)object.get(); if (!array_ptr || ptr != PyArray_DATA(array_ptr)) throw std::logic_error("invalid call to NumpyArrayAllocator::free()"); <del> object = nullptr; </del> <ins> { AutoGIL gil; object = nullptr; } </ins> delete this; } #endif

Discussion:
When both PyTorch and Pandas are imported, calling results in a segfault. The issue was originally reported by It looks like calling returns a null pointer here: Examining in GDB: I suspect it's both PyTorch and Pandas doing threading magic and somehow getting in each other's way. I haven't yet been able to provide a minimal example that triggers this behaviour and does not use my package; it's likely that I'm doing something odd that causes this. More documentation here: Possibly related issues: and Python version: 3.6 PyTorch version: v0.2.0 Pandas version: 0.20.1
Change:
using TensorRef = std::reference_wrapper<const Tensor>; using TensorRefList = std::initializer_list<TensorRef>; <ins> // ArrayRef is not covariant, which means there is no // implicit conversion between TensorList (aka ArrayRef<Tensor>) // and ArrayRef<Variable>.  What we do instead is manually // construct a variable_list, which itself is implicitly convertible // into an ArrayRef<Variable> (but don't return an ArrayRef<Variable>; // ArrayRef is non-owning!) static variable_list cast_tensor_list(const TensorList& tensors) { // TODO: Eliminate the intermediate vector allocation return variable_list(tensors.begin(), tensors.end()); } </ins> static VariableFlags compute_flags(const TensorRefList& tensors) { return compute_flags_tmpl(tensors); }

Discussion:
when executing:
I'm seeing this as well as of last night. Its preceded by many warnings about : But the first error is the one about . I went back, and I get a good install as late as commit
Change:
The lower triangular part of the matrix is defined as the elements on and below the diagonal. <del> The argument :attr:`k` controls which diagonal to consider. </del> <ins> The argument :attr:`diagonal` controls which diagonal to consider. </ins> <del> - :attr:`k` = 0, is the main diagonal. - :attr:`k` > 0, is above the main diagonal. - :attr:`k` < 0, is below the main diagonal. </del> <ins> - :attr:`diagonal` = 0, is the main diagonal. - :attr:`diagonal` > 0, is above the main diagonal. - :attr:`diagonal` < 0, is below the main diagonal. </ins> Args: input (Tensor): the input `Tensor` <del> k (int, optional): the diagonal to consider </del> <ins> diagonal (int, optional): the diagonal to consider </ins> out (Tensor, optional): The result `Tensor` Example::

Discussion:
The states that the argument controls which diagonal to consider. However, this argument is actually named in pytorch 0.2.0.
Change:
#ifdef USE_MAGMA THArgCheck(a_->nDimension == 2, 1, "A should be 2 dimensional"); THArgCheck(b_->nDimension == 2, 1, "b should be 2 dimensional"); <del> THArgCheck(a_->size[0] == b_->size[0], 2, "size incompatible A,b"); THArgCheck(a_->size[0] >= a_->size[1], 2, "A should have m >= n"); </del> <ins> THArgCheck(a_->size[0] == b_->size[0], 2, "Expected A and b to have same size " "at dim 0, but they have incompatible sizes"); THArgCheck(a_->size[0] >= a_->size[1], 2, "Expected A with shape (m x n) to have " "m >= n. The case for m < n is not implemented yet."); </ins> THCTensor *a = THCTensor_(newColumnMajor)(state, ra_, a_); THCTensor *b = THCTensor_(newColumnMajor)(state, rb_, b_);

Discussion:
I am currently using PyTorch 0.3.0 and I am noticing some (possibly strange) differences between the behavior of when operating on CPU tensors versus GPU ones, in particular relating to whether the number of rows of A is allowed to be larger than the number of columns or not. Here is some minimal code that reproduces my problem: The first solve on the CPU works fine; however, the second solve gives me the error: Is this a limitation of the CUDA implementation of that PyTorch uses, or otherwise intended? If so, any guidance about a possible work-around would be greatly appreciated. Our only thought on a work-around was to manually QR decompose A with and then use , but doesn't support CUDA tensors.
MAGMA gels doesn't implement the m < n case.