jiebi/CodeConvo huggingface dataset:
Change:
<ins> #pragma once #include <Python.h> template<class T> class THPPointer { public: THPPointer(): ptr(nullptr) {}; THPPointer(T *ptr): ptr(ptr) {}; THPPointer(THPPointer &&p) { free(); ptr = p.ptr; p.ptr = nullptr; }; ~THPPointer() { free(); }; T * get() { return ptr; } const T * get() const { return ptr; } T * release() { T *tmp = ptr; ptr = NULL; return tmp; } operator T*() { return ptr; } THPPointer& operator =(T *new_ptr) { free(); ptr = new_ptr; return *this; } THPPointer& operator =(THPPointer &&p) { free(); ptr = p.ptr; p.ptr = nullptr; return *this; } T * operator ->() { return ptr; } operator bool() { return ptr != nullptr; } private: void free(); T *ptr = nullptr; }; typedef THPPointer<PyObject> THPObjectPtr; </ins>

Discussion:
Go to Look at the html. It contains the following: ! This problem manifests in google search. Searching for 1.8.1 docs brings up the "1.8.0 docs": ! We can probably fix this by doing a quick sed on the website but we should develop a solution for future releases. I think this has happened before in the past and am not sure what causes it. cc
Change:
add_docstr(torch._C.tril, """ <del> tril(input, k=0, out=None) -> Tensor </del> <ins> tril(input, diagonal=0, out=None) -> Tensor </ins> Returns the lower triangular part of the matrix (2D Tensor) :attr:`input`, the other elements of the result Tensor :attr:`out` are set to 0.

Discussion:
The states that the argument controls which diagonal to consider. However, this argument is actually named in pytorch 0.2.0.
Change:
fi # sccache will fail for CUDA builds if all cores are used for compiling <del> # gcc 7.2 with sccache seems to have intermittent OOM issue if all cores are used if ([[ "$BUILD_ENVIRONMENT" == *cuda* ]] || [[ "$BUILD_ENVIRONMENT" == *gcc7.2* ]]) && which sccache > /dev/null; then </del> <ins> # gcc 7 with sccache seems to have intermittent OOM issue if all cores are used if ([[ "$BUILD_ENVIRONMENT" == *cuda* ]] || [[ "$BUILD_ENVIRONMENT" == *gcc7* ]]) && which sccache > /dev/null; then </ins> export MAX_JOBS=`expr $(nproc) - 1` fi

Discussion:
We're not testing against CUDA 9.2 and we should. C.f. We should test against GCC 7 (because this is how Anaconda is building the packages), AND we should use test with this in OS X, because CUDA 9.2 now supports the default xcode shipped on OS X.
Building PyTorch with CUDA 9.2 for 7.0 compute compatibility devices produces quite worrisome warnings: warning : Instruction 'vote' without '.sync' may produce unpredictable results on sm70 and later architectures warning : Instruction 'vote' without '.sync' is deprecated since PTX ISA version 6.0 and will be discontinued in a future PTX ISA version warning : Instruction 'shfl' without '.sync' is deprecated since PTX ISA version 6.0 and will be discontinued in a future PTX ISA version warning : Instruction 'shfl' without '.sync' may produce unpredictable results on sm70 and later architectures
CUDA 9.2 CI on macOS is done in
Change:
from torch.autograd import gradcheck from torch.autograd.function import once_differentiable <del> from common import TestCase, run_tests </del> <ins> from common import TestCase, run_tests, skipIfNoLapack </ins> from torch.autograd._functions import * from torch.autograd import Variable, Function

Discussion:
Command to install the latest nightly binary: installs : However, PyTorch itself reports : I guess this version tag might be defined during the build process from e.g. and then forwarded from , but also source builds should use based on the tag. Unsure, if this is just a reporting issue or if we are building the nightly binaries with and executing them with , so creating this issue to dig a bit further into it. CC cc
This is really weird, I've put up measure in place to validate against it... Will investigate today (and moving to pytorch repo)
can you please check that 2.2.0 reports (and is build) against the correct version of nccl library?
Confirmed for nightlies : Same issue for RC:
Looking at the build log Looks like we are compiling with: USESYSTEMNCCL
On Windows I see following exception: We compiling with should we return None rather then exception ?
In this case we are indeed using 2.18.1: Do you know why we are using the system-wide NCCL installation instead of the one? Should we delete the system NCCL and use by default?
Looks like its set here: Looks like it was by: Let me propose the PR to unset this flag : USESYSTEM_NCCL
I guess we are using the system NCCL installation to be able to during the runtime. If we would use the module I assume we would link NCCL into the PyTorch binaries. If so, we should make sure to update the NCCL version whenever is updated. This could break our dependency on the NCCL pip package.
Yes. Also when we update this nccl here:
We used following path  What would be good path for nccl 2.19.3 and cuda 12.1 ? As per this issue: Will be using 12.0 ncck 2.19.3
Installing system NCCL into the nightly build docker container feels great, but we need to figure out, how to keep it in sync with PyTorch. I wonder if one can, for example, just check nccl version from this requirements file and build from source?
yes probably we should build and upload nccl for our supported cuda versions, same way as we build docker containers, by following details here:
Verified with final rc
Change:
// Each block handles one distribution // First pass, find the total sum of the distribution AccT sum = accZero; <ins> T val; </ins> for (int cat = threadIdx.x; cat < categories; cat += blockDim.x) { <del> sum = THCNumerics<AccT>::add( sum, ScalarConvert<T, AccT>::to(dist[curDist * categories + cat])); </del> <ins> val = dist[curDist * categories + cat]; assert(THCNumerics<T>::ge(val, zero)); sum = THCNumerics<AccT>::add(sum, ScalarConvert<T, AccT>::to(val)); </ins> } // threadIdx.x == 0 has the sum value from this

Discussion:
Looking at the values returned from tn.multinomial(1) and counting them, it seems like it returns the values of the multinomial distribution taken from t, i.e. that it has ignored the signs of the values. However, it seem like always returns a tensor filled with zeros.
looking into this
Change:
self.assertEqual(hy.data[0][0][0], 10) self.assertEqual(hy.data[1][0][0], output_val) <del> @unittest.skipIf(not TEST_CUDNN, "needs cudnn") </del> <ins> @unittest.skipIf(not (TEST_CUDNN and TEST_CUDNN_VERSION >= 5103), "needs cudnn >= 5.1") </ins> def test_RNN_dropout_state(self): import sys if sys.version_info[0] == 2:

Discussion:
cuDNN backend for RNN stabilizes at large mem usage. The following code stabilizes for me at 14+GiB reported by nvidia-smi. The same RNN without cuDNN uses <2.5GiB.
Oops... any idea what's going on with the garbage collector in this case?
are you running this on a P100 GPU?
Yes, I observe similar behavior on Titan X Pascal
hmmm, do cudnn RNNs have weird memory-oversubscribing properties? If it wasn't for CuDNN RNN, then it's not unreasonable. we use a custom CUDA allocator that will oversubscribe if needed, but will free the memory blocks it's holding onto if it hits OOM / failed allocations.
Tried the same RNN in torch and am getting ~2.6GiB on nvidia-smi.
With cuDNN I'm getting a constant usage of 2241MB on my machine using PyTorch (M40).
are you using cudnnv5, or v6?
v5
no, cudnn does not do memory oversubscribing, and already tried the same network in lua torch with cudnn, where it gets expected memory usage. Also, he is getting OOMs on slightly bigger networks, where apparently caching allocator should have tried to free previous blocks. Interesting that can not repro this.
Just to make sure did you set when you ran? I left it as False in the repro.
I can repro in the docker image built from the current version on P100, and I go OOM after 8 batches with the same image on 1080, which generally should not happen - if there is enough memory to go through 1 batch, memory usage should not increase afterwards.
No, I just copied your script and ran it. Weird thing is that it goes OOM after 18 batches once I enable . Without it memory remains stable.
A lot of cudaMalloc allocations go not through THCudaMalloc, and never get freed:
So not going through THCudaMalloc is actually expected for storages allocated with THCStorage_(newWithSize), but what's not expected is to every time cudaMalloc bytes of rnn workspace, instead of reusing earlier allocations. (~784 MB allocations are for workspace, ~156MB allocations happen somewhere in criterion).
My guess is that the bug is at the Python level.

Either: Something is holding on to a Tensor longer than it should Something is creating a circular reference which prevents quick clean-up
The issue is that in creates cyclic references. It's made worse by the fact that it holds on to more memory than it should (like the workspace). You can see this by adding a as the first line in your for loop. With the explicit call to the GC every iteration, the memory usage doesn't explode.
how can these cyclic references be fixed?
Not sure yet, but I will fix it tomorrow with
Change:
modules = list(self._modules.keys()) buffers = list(self._buffers.keys()) keys = module_attrs + attrs + parameters + modules + buffers <ins> # Eliminate attrs that are not legal Python variable names keys = [key for key in keys if not key[0].isdigit()] </ins> return sorted(keys)

Discussion:
Hello, In the case where these classes sequentially append the received elements with , the modules become inaccessible since a variable name starting with a number is forbidden in Python. I attached a patch that names the modules as instead of just numbers.
Use syntax for Sequential and ModueList.
possible but the attributes are there and they are obviously shown in autocompletion suggestions in IPython for example and when you try to access them you got an error because their naming does not conform to Python rules.
Perhaps we should exclude them from
Change:
<del> FROM nvidia/cuda:8.0-devel-ubuntu16.04 </del> <ins> FROM nvidia/cuda:8.0-cudnn6-devel-ubuntu16.04 </ins> RUN echo "deb http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64 /" > /etc/apt/sources.list.d/nvidia-ml.list <del> ENV CUDNN_VERSION 6.0.20 </del> RUN apt-get update && apt-get install -y --no-install-recommends  build-essential  cmake  git  curl  <ins> vim  </ins> ca-certificates  libjpeg-dev  <del> libpng-dev  libcudnn6=$CUDNN_VERSION-1+cuda8.0  libcudnn6-dev=$CUDNN_VERSION-1+cuda8.0 &&  </del> <ins> libpng-dev && </ins> rm -rf /var/lib/apt/lists/* RUN curl -o ~/miniconda.sh -O  https://repo.continuum.io/miniconda/Miniconda3-4.2.12-Linux-x86_64.sh  && 

Discussion:
Torchvision is considered a basic requirement of the tutorials. Perhaps it makes sense to include it in the docker build.
It's easy to install it by command ;)
It's easy to install torch itself using a similar command? :). I was thinking a docker image should have torchvision included.
Runtime docker file already has torchvision.
Change:
[[17, 7], [18, 2], [19, 7], [20, 1]], ]) res = dense._sparse_mask(x) <del> exp_v = self.ValueTensor([[7, 9], [14, 1], [14, 1], [3, 3], [20, 1]]) </del> <ins> exp_v = self.ValueTensor([[7, 9], [14, 1], [3, 3], [20, 1]]) </ins> expected = self.SparseTensor(i, exp_v, torch.Size([5, 4, 2])) self.assertEqual(res, expected)

Discussion:
and I talked about the intended semantics of sparsemask when given a sparse tensor as an argument, and we came to the following conclusions: with an uncoalesced sparse tensor does not make sense, full stop. Unlike values, where your update might be linear and work correctly on an uncoalesced sparse tensor, there isn't any good reason why you want to pluck out values from the dense tensor multiple times, it is basically always a bug. the (eventual) documentation for sparsemask, we will recommend calling coalesce before doing any operations involving sparsemask (or nonlinear updates to indices/values). should add a test to sparsemask which checks if the flag of the input tensor is true, and raise an error if it is not. This is a cheap safety check which will prevent people from forgetting to call coalesce (as in (2)). There is a performance consequence to calling coalesce every iteration of training. If the gradient is producing coalesced vectors, we can alleviate this problem by adding a preprocessing step to coalesce() which tests if the input is already coalesced, and bugs out in that case. However, I think this is out of scope for this particular ticket for now.
Change:
pytorch-linux-xenial-cuda9-cudnn7-py2-test pytorch-linux-xenial-cuda9-cudnn7-py3-build pytorch-linux-xenial-cuda9-cudnn7-py3-test <ins> pytorch-linux-xenial-cuda9.2-cudnn7-py3-gcc7-build pytorch-linux-xenial-cuda9.2-cudnn7-py3-gcc7-test </ins> pytorch-linux-xenial-py3-clang5-asan-build pytorch-linux-xenial-py3-clang5-asan-test pytorch-linux-trusty-py2.7.9-build

Discussion:
We're not testing against CUDA 9.2 and we should. C.f. We should test against GCC 7 (because this is how Anaconda is building the packages), AND we should use test with this in OS X, because CUDA 9.2 now supports the default xcode shipped on OS X.
Building PyTorch with CUDA 9.2 for 7.0 compute compatibility devices produces quite worrisome warnings: warning : Instruction 'vote' without '.sync' may produce unpredictable results on sm70 and later architectures warning : Instruction 'vote' without '.sync' is deprecated since PTX ISA version 6.0 and will be discontinued in a future PTX ISA version warning : Instruction 'shfl' without '.sync' is deprecated since PTX ISA version 6.0 and will be discontinued in a future PTX ISA version warning : Instruction 'shfl' without '.sync' may produce unpredictable results on sm70 and later architectures
CUDA 9.2 CI on macOS is done in
Change:
x_clone ^= y self.assertEqual(x_clone, xor_result) <ins> def test_apply(self): x = torch.range(1, 5) res = x.clone().apply_(lambda k: k + k) self.assertEqual(res, x * 2) self.assertRaises(RuntimeError, lambda: x.apply_(lambda k: "str")) </ins> if __name__ == '__main__': unittest.main()

Discussion:
On Python 3.5.2 errors out with . map and map2 seem to be work just fine.
Change:
def test_broadcast_batched_matmul(self): TestTorch._test_broadcast_batched_matmul(self, lambda t: t.cuda()) <ins> def test_index(self): TestTorch._test_index(self, lambda t: t.cuda()) </ins> def test_advancedindex(self): TestTorch._test_advancedindex(self, lambda t: t.cuda())

Discussion:
Arguably, indexing a Tensor by a LongTensor with no dimension should return a LongTensor with no dimension. Consider the following case: This breaks indexing by indices when the list of indices is empty, which can be hard to detect. In my use-case, I use such a masking to select rows of a tensor corresponding to "valid" entries in a vector; I had not considered that it would not work when there are no valid entries.
Thanks! I'll take a look.
Change:
if probs is not None: self.probs = probs / probs.sum(-1, keepdim=True) else: <del> self.logits = logits - log_sum_exp(logits) </del> <ins> self.logits = logits - _log_sum_exp(logits) </ins> self._param = self.probs if probs is not None else self.logits self._num_events = self._param.size()[-1] batch_shape = self._param.size()[:-1] if self._param.ndimension() > 1 else torch.Size()

Discussion:
Leaving this issue here for higher visibility. Originally reported by here: Also mentioned by Piggybacking off of this issue, has strange behavior (reported here: )
I have fixed this locally. Could I send a PR?
Yes, go ahead!
Change:
y._backward_hooks['test'] = error b.backward(torch.ones(5, 5)) <ins> def test_requires_grad_inplace(self): a = Variable(torch.randn(5, 5)) b = Variable(torch.randn(5, 5), requires_grad=True) a += b self.assertTrue(a.requires_grad) # non-leaf Variable a = Variable(torch.randn(5, 5)) + 0 b = Variable(torch.randn(5, 5), requires_grad=True) a += b self.assertTrue(a.requires_grad) </ins> def test_previous_functions(self): x = Variable(torch.randn(5, 5), requires_grad=True) y = Variable(torch.randn(5, 5), requires_grad=True)

Discussion:
import torch as th from torch.autograd import Variable x = Variable(th.Tensor([1 , 2 , 3]) , requiresgrad = False) y = Variable(th.Tensor([100]) , requiresgrad = True) x = x.cuda(0) y = y.cuda(0) x[0] = y print(x.requiresgrad) If ((0) (0)) is executed then the output is False, True otherwise. I think the result that x.requiresgrad = True is more natural.
I further checked some in-place operator in GPU mode(say, x = Variable(...).cuda(0) ), and find out that after the operation is done, the requiresgrad of the Variable doesn't change. For example: import torch as th from torch.autograd import Variable x = Variable(torch.Tensor([1]) , requiresgrad = False) y = Variable(torch.Tensor([2]) , requiresgrad = True) x = x.cuda(0) y = y.cuda(0) x.add(y) print(x.requires_grad) the output is False
It doesn't have to do with Variable holding a CUDA tensor, but rather with the fact that it's no longer a leaf. I've sent a fix in .
Thanks for the report!
Change:
SET_PROPERTY(TARGET ATen PROPERTY CXX_STANDARD 11) endif(NOT ${CMAKE_VERSION} VERSION_LESS "3.1") <del> FIND_PACKAGE(BLAS) </del> IF(BLAS_FOUND) <del> SET(USE_BLAS 1) </del> IF ($ENV{TH_BINARY_BUILD}) MESSAGE(STATUS "TH_BINARY_BUILD detected. Enabling special linkage.") TARGET_LINK_LIBRARIES(ATen "${BLAS_LIBRARIES};${BLAS_LIBRARIES};${BLAS_LIBRARIES}") ELSE ($ENV{TH_BINARY_BUILD}) TARGET_LINK_LIBRARIES(ATen ${BLAS_LIBRARIES}) ENDIF ($ENV{TH_BINARY_BUILD}) <del> IF(BLAS_INFO STREQUAL "mkl") ADD_DEFINITIONS(-DTH_BLAS_MKL) ENDIF() </del> ENDIF(BLAS_FOUND) IF(LAPACK_FOUND)

Discussion:
currently compiled master does not use any blas for any ops and uses our lame fallback C code. I'm looking into it. Basically USE_BLAS is not being set in
Change:
Args: input: Variable :math:`(N, C)` where `C = number of classes` target: Variable :math:`(N)` where each value is `0 <= targets[i] <= C-1` <del> weight (Variable, optional): a manual rescaling weight given to each class. If given, has to be a Variable of size "nclasses" </del> <ins> weight (Tensor, optional): a manual rescaling weight given to each class. If given, has to be a Tensor of size "nclasses" </ins> size_average (bool, optional): By default, the losses are averaged over observations for each minibatch. However, if the field sizeAverage is set to False, the losses are instead summed

Discussion:
The of nn.functional.crossentropy seems wrong. The parameter weight should be a tensor, rather than a Variable.
Change:
fn.w_desc = init_weight_descriptor(fn, fn.weight_buf) w = fn.weight_buf <del> if tuple(hx.size()) != hidden_size: raise RuntimeError('Expected hidden size {}, got {}'.format( hidden_size, tuple(hx.size()))) </del> if cx is not None and tuple(cx.size()) != hidden_size: raise RuntimeError('Expected cell size {}, got {}'.format( hidden_size, tuple(cx.size())))

Discussion:
I run RNN sucessfully with the code. and I run embedding example. Then, got RuntimeError. I set to use nothing seems wrong. And the error info seems not the dim problem? The error is so strange.
What exactly do you mean by "embedding example"? Could you provide a code snippet that causes the RuntimeError?
To better reproduce the error, I use the official code example of The origin runs well: Due to use , I have to make batch first. And then , it raises error.
Thank you for the example! The say that setting means that the first dim of input is the batch, ie, input has size . however is unchanged, so the following should work: I do agree the error message could be better, though
Change:
return storage <ins> def rebuild_storage_empty(cls): return cls() </ins> def reduce_storage(storage): from . import get_sharing_strategy if storage.is_cuda:

Discussion:
PyTorch version: 0.4.0a0+ Repro: Result:
Emm. I fail to reproduce this bug in Python 3.5.0 and Mac OS 10.13.3. What's your system environment?
The bug is probably specific to the sharing mode. Mac OS uses by default.
Change:
pass def run(self): <ins> # Check if you remembered to check out submodules def check_file(f): if not os.path.exists(f): print("Could not find {}".format(f)) print("Did you run 'git submodule update --init'?") sys.exit(1) check_file(os.path.join(lib_path, "gloo", "CMakeLists.txt")) check_file(os.path.join(lib_path, "nanopb", "CMakeLists.txt")) check_file(os.path.join(lib_path, "pybind11", "CMakeLists.txt")) </ins> libs = [] if WITH_NCCL and not WITH_SYSTEM_NCCL: libs += ['nccl']

Discussion:
It seems like running has a dependency on first running . This command also seems to also error out when installing via a wheel. This creates a problem for tools like  which rely on the output of to examine dependencies. Is there anyway to allow this command to not error out when installing via a wheel? Full stack trace: I understand the preferred package manager for this project is conda but being officially endorsed from , pipenv has a fairly large audience:
thanks do you know when's the next planned release?
Hopefully within a month
Change:
torch.stack([reference[:, 2, 1], reference[:, 2, 3], reference[:, 2, 5]], 1)) lst = [list(range(i, i + 10)) for i in range(0, 100, 10)] <del> tensor = torch.DoubleTensor(lst) </del> <ins> tensor = conv_fn(torch.DoubleTensor(lst)) </ins> for _i in range(100): idx1_start = random.randrange(10) idx1_end = idx1_start + random.randrange(1, 10 - idx1_start + 1)

Discussion:
Arguably, indexing a Tensor by a LongTensor with no dimension should return a LongTensor with no dimension. Consider the following case: This breaks indexing by indices when the list of indices is empty, which can be hard to detect. In my use-case, I use such a masking to select rows of a tensor corresponding to "valid" entries in a vector; I had not considered that it would not work when there are no valid entries.
Thanks! I'll take a look.
Change:
<del> From PyTorch: Copyright (c) 2016-     Facebook, Inc            (Adam Paszke) Copyright (c) 2014-     Facebook, Inc            (Soumith Chintala) Copyright (c) 2011-2014 Idiap Research Institute (Ronan Collobert) Copyright (c) 2012-2014 Deepmind Technologies    (Koray Kavukcuoglu) Copyright (c) 2011-2012 NEC Laboratories America (Koray Kavukcuoglu) Copyright (c) 2011-2013 NYU                      (Clement Farabet) Copyright (c) 2006-2010 NEC Laboratories America (Ronan Collobert, Leon Bottou, Iain Melvin, Jason Weston) Copyright (c) 2006      Idiap Research Institute (Samy Bengio) Copyright (c) 2001-2004 Idiap Research Institute (Ronan Collobert, Samy Bengio, Johnny Mariethoz) From Caffe2: Copyright (c) 2016-present, Facebook Inc. All rights reserved. All contributions by Facebook: Copyright (c) 2016 Facebook Inc. All contributions by Google: Copyright (c) 2015 Google Inc. All rights reserved. All contributions by Yangqing Jia: Copyright (c) 2015 Yangqing Jia All rights reserved. All contributions from Caffe: Copyright(c) 2013, 2014, 2015, the respective contributors All rights reserved. All other contributions: Copyright(c) 2015, 2016 the respective contributors All rights reserved. Caffe2 uses a copyright model similar to Caffe: each contributor holds copyright over their contributions to Caffe2. The project versioning records all such contribution and copyright details. If a contributor wants to further mark their specific copyright on a particular contribution, they should indicate their copyright solely in the commit message of the change when it is committed. </del> ======================================================================= Software under third_party =======================================================================

Discussion:
The three-clause BSD license in file LICENSE says in clause 1 "Redistributions of source code must retain the above copyright notice..." However, there is no longer a copyright notice in the file; it appears to have been removed in commit (see ).
cc