jiebi/CodeConvo huggingface dataset:
Change:
Hi, I have two linux servers. My code works in one machine but not the other. In the one that it doesn't work, the error message is: occured during batch normalization( convolution is fine). Also, the same code works in CPU but not in GPU. I guess this problem is mainly related to GPU libraries like cuda or cudnn. The one that doesn't work has the following configurations: Nvidia driver 375.26 cuda 8.0.61 cudnn 5.1.10 While in the one it does work, I have the following configurations: Nvidia driver 367.48 cuda 8.0.44 cudnn 5.1.5 TensorFlow can run in the above environment flawless though. So I am wondering if this is caused by the incompatibility between PyTorch 0.1.12 and cuda 8.0.61/cudnn 5.1.10 ?
you use the pytorch binaries, we ship cudnn6 in them, use it it's great. 5.1.10 maybe has a bug. cc: . do you have a small test case to reproduce this? does not use CuDNN batchnorm by default, so maybe that's why?
Small test case would help, thanks.
After some experimentation, I have pin pointed where exactly the error comes from. It seems batch norm in cudnn 5.1.10 doesn't support eps as small as 1e-6. A small test case would be:
Yes, cudnn.h for 5.1.10 defines CUDNNBNEPSILON to be 1e-5 (minimum allowable value). I can't check header for 5.1.5. Even with 1e-5 epsilon for many input data distributions the results become pretty unstable. Is there a case for this restriction to be lifted, does using very small epsilon provide benefits?
I see. Not sure if small epsilon is necessary. I am using 1e-6 just to replicate others' configurations. Should PyTorch provide more informative error message about this restriction?
Ideally, CUDNN should provide more informative error messages (pytorch can not know why cudnn is returning an error code in each particular case) , but that's a long standing discussion.
It just hits on me. As mentioned by , TensorFlow does not use CuDNN batchnorm by default. So is this possible in Pytorch, to not use default CuDNN batchnorm?
Never mind. I found the switch in .

Discussion:
using thpp::Tensor; <ins> #ifndef CUDNN_BN_MIN_EPSILON #define CUDNN_BN_MIN_EPSILON 0 #endif </ins> auto BatchNormForward::apply(const variable_list& inputs) -> variable_list { check_input_variables("BatchNorm", inputs, 3, 1);

std::unique_ptr<Tensor> save_std(output->newTensor()); save_std->resizeAs(*running_var); <del> if (use_cudnn) { </del> <ins> if (use_cudnn && eps >= CUDNN_BN_MIN_EPSILON) { </ins> #ifdef WITH_CUDNN torch::cudnn::cudnn_batch_norm_forward( state,

} } <del> if (use_cudnn) { </del> <ins> if (use_cudnn && eps >= CUDNN_BN_MIN_EPSILON) { </ins> #ifdef WITH_CUDNN torch::cudnn::cudnn_batch_norm_backward( state,
Change:
Are there any plans to implement something like tensorflow's and ? exists but it doesn't have the backward step implemented.
One workaround for the moment would be to write as , which should support backprop since . Adding support for should be trivial, the gradient for it can be found in
I'll add backward support for , which should be equivalent to . I'll try to do it later today.
How difficult would it be to add in batched versions (where only the final two dimensions are square) of GESV and inverse? Are there any plans for this?
If we have batched version of , then adapting the autograd part is very simple. There is an open issue already that discuss implementing batched gesv
Actually, there is an open PR for that, so it might be coming soon
I'll rebase the batched version of gesv. It'll come in soon
May I ask why gesv only support . As I saw from , it support that A is a MxN matrix, and B is a NxK matrix, then solve a matrix X (NxK). If I want to use a non-square matrix A in gesv, would you mind to let me know how to implement in PyTorch?
this doesn't exactly answer your question but you can add zero-rows to the A matrix until it becomes square.
thanks a lot, I find the solution, can solve the problem.
Any plan to add gradient calculation for gels?
I am also wondering if will have differentiation support. My current workaround is to use instead
I actually implemented overdetermined case following tensorflow implementation. For backward pass I only did basic grad test. On Monday, March 18, 2019, Johnson Zhong <:
Thanks, seems to work fine!

Discussion:
(Cross, (), ((S, 3), (S, 3))), (Cross, (), ((S, 3, S), (S, 3, S), 1), 'dim'), (Inverse, (), ((S, S),), '', (), [skipIfNoLapack]), <ins> (Gesv, (), ((S, S), (S, S)), '', (), [skipIfNoLapack]), </ins> (Clone, (), ((S, M, S),)), (Squeeze, (), ((S, 1, M, 1),)), # TODO: enable neg dim checks

('cross', (S, 3), ((S, 3),)), ('cross', (S, 3, S), ((S, 3, S), 1), 'dim'), ('inverse', (S, S), (), '', (), [skipIfNoLapack]), <ins> ('gesv', (S, S), ((S, S),), '', (), [skipIfNoLapack]), </ins> ('clone', (S, M, S), ()), ('eq', (S, S, S), ((S, S, S),)), ('ne', (S, S, S), ((S, S, S),)),

def backward(ctx, grad_output): inverse, = ctx.saved_variables return -torch.mm(inverse.t(), torch.mm(grad_output, inverse.t())) <ins> class Gesv(Function): @staticmethod def forward(ctx, b, a): # TODO see if one can backprop through LU X, LU = torch.gesv(b, a) ctx.save_for_backward(X, a) ctx.mark_non_differentiable(LU) return X, LU @staticmethod def backward(ctx, grad_output, grad_LU=None): X, a = ctx.saved_variables grad_b, _ = torch.gesv(grad_output, a.t()) grad_a = -torch.mm(grad_b, X.t()) return grad_b, grad_a </ins>

def inverse(self): return Inverse.apply(self) <ins> def gesv(self, a): return Gesv.apply(self, a) </ins> def multinomial(self, num_samples=1, with_replacement=False): return Multinomial(num_samples, with_replacement)(self)
Change:
is the culprit. Assumes that the first input is a variable or a sequence containing a variable in index 0 (recursively indexed). Workaround is to pass a dummy variable as first input, as suggested in
Could you provide a short reproducer for testing? Think this shouldn't be too hard to fix.
Sure, here's a gist along with commands/outputs:
fixed in master via

Discussion:
from torch.autograd import Variable <ins> def get_a_var(obj): if isinstance(obj, Variable): return obj if isinstance(obj, list) or isinstance(obj, tuple): results = map(get_a_var, obj) for result in results: if isinstance(result, Variable): return result if isinstance(obj, dict): results = map(get_a_var, obj.items()) for result in results: if isinstance(result, Variable): return result return None </ins> def parallel_apply(modules, inputs, kwargs_tup=None): assert len(modules) == len(inputs) if kwargs_tup:

results = {} def _worker(i, module, input, kwargs, results, lock): <del> var_input = input while not isinstance(var_input, Variable): var_input = var_input[0] </del> <ins> var_input = get_a_var(input) </ins> try: with torch.cuda.device_of(var_input): output = module(*input, **kwargs)
Change:
This is an ipython session. Note that the doesn't remain the same for /= even though it works for div_
Just to be clear, the reason this is an issue is that it means that functions can't do inplace operations on function arguments.
Also, this works fine for +=

Discussion:
t2 = torch.from_numpy(t.numpy().transpose()) self.assertEqual(t1, t2) <ins> def test_inplace_division(self): t = torch.rand(5, 5) id_before = id(t) t /= 2 id_after = id(t) self.assertEqual(id_before, id_after) </ins> # Functions to test negative dimension wrapping METHOD = 1 INPLACE_METHOD = 2

def __idiv__(self, other): return self.div_(other) <ins> __itruediv__ = __idiv__ </ins> def __mod__(self, other): return self.remainder(other)
Change:
Hi.. I think this is a bug: So, it doesn't matter if descending is True or False, just specifying it explicitly requires dimension to be specified.. It would be better (I think) if descending could be specified alone..
Same issue for if is used:
sort was fixed in topk is still a problem.

Discussion:
- THTensor* self - long k - CONSTANT __last_dim <del> - CONSTANT true - CONSTANT true </del> <ins> - arg: bool largest default: "true" kwarg_only: True - arg: bool sorted default: "true" kwarg_only: True </ins> - arguments: - arg: THTensor* values output: True
Change:
has broken DataParallel tests. Now that deviceids are explicitly sent to parallelapply, this assert gets triggered if inputs are not big enough to be on all devices (e.g. batch size of 20 on 8 GPUs gets chunked into 6*3+2, so that 8-th GPU is idle, and assert gets triggered).
your latest fix in doesn't fix this completely. Here's a failing stack-trace:
How is it possible that the tests didn't catch that in the PR?
it's because our build box which only has 2 GPUs doesn't hit this case. It's failing on 4-GPU setting (maybe batch size becomes too small then)

Discussion:
inputs, module_kwargs = scatter_kwargs(inputs, module_kwargs, device_ids, dim) if len(device_ids) == 1: return module(*inputs[0], **module_kwargs[0]) <del> replicas = replicate(module, device_ids[:len(inputs)]) outputs = parallel_apply(replicas, inputs, module_kwargs, device_ids) </del> <ins> used_device_ids = device_ids[:len(inputs)] replicas = replicate(module, used_device_ids) outputs = parallel_apply(replicas, inputs, module_kwargs, used_device_ids) </ins> return gather(outputs, output_device, dim)
Change:
Currently due to the way cwrap generates wrapping code for the c source, passing an argument to .numpy() would not result in any error. I think probably this should not be the desired behaviour for the API?
fixed via

Discussion:
std::unique_ptr<npy_intp[]> sizes; std::unique_ptr<npy_intp[]> strides; <ins> THPUtils_assert(PyTuple_Size(args) == 0, "numpy() takes no positional arguments"); </ins> // Numpy and Torch disagree on empty tensors. In Torch, an empty tensor // is a tensor with zero dimensions. In Numpy, a tensor with zero dimensions // is a scalar (with one element). So we'll convert an empty Torch tensor
Change:
Hello, on Debian (and possibly Ubuntu and other systems), is in and not in non-root users' search paths. Thus this line: at does not do what it should when compiling not as root. Either writing or adding a second check with the directory would help building pytorch on these systems. Would such a change be acceptable? Thank you & best regards Thomas
Or fix this by exporting ?
such a chance would be absolutely acceptable.
fixed in PR

Discussion:
THD_LIB = os.path.join(lib_path, 'libTHD.1.dylib') NCCL_LIB = os.path.join(lib_path, 'libnccl.1.dylib') <del> if WITH_NCCL and subprocess.call('ldconfig -p | grep libnccl >/dev/null', shell=True) == 0: </del> <ins> if WITH_NCCL and (subprocess.call('ldconfig -p | grep libnccl >/dev/null', shell=True) == 0 or subprocess.call('/sbin/ldconfig -p | grep libnccl >/dev/null', shell=True) == 0): </ins> SYSTEM_NCCL = True main_compile_args = ['-D_THP_CORE']
Change:
As mentioned in title, I want to make sure if pytorch can deal with the situation that each input in a batch has different shapes and thus different computation graphs.
Please don't open duplicated issue for discussion. Context will be missed. Try to discuss about it here:

Discussion:
# (3) initialize mean square values and square gradient storage if not 'm' in state: <del> state['m'] = x.new().resize_as_(dfdx).fill_(1) </del> <ins> state['m'] = x.new().resize_as_(dfdx).zero_() </ins> state['tmp'] = x.new().resize_as_(dfdx)

# State initialization if len(state) == 0: state['step'] = 0 <del> state['square_avg'] = grad.new().resize_as_(grad).fill_(1) </del> <ins> state['square_avg'] = grad.new().resize_as_(grad).zero_() </ins> square_avg = state['square_avg'] alpha = group['alpha']
Change:
I haven't looked into this in depth but this looks fishy: More here: cc
is this still an issue in 1.3.1?
I can repro this on master:
The problem is: removes the parameters from the module's parameter list and makes them normal class attributes: applies only to parameters, not to the weight attributes.  (which is called from acts on which includes the value of the weight attributes and the bias parameters. So, it contains a mix of CUDA and CPU tensors.
This is my workaround for right now: This applies weight norm to an rnn module and overrides the function to deal with the error.
This appears to be fixed. do you want to verify?
I tested this on nightly and it worked for me, so closing.

Discussion:
# (3) initialize mean square values and square gradient storage if not 'm' in state: <del> state['m'] = x.new().resize_as_(dfdx).fill_(1) </del> <ins> state['m'] = x.new().resize_as_(dfdx).zero_() </ins> state['tmp'] = x.new().resize_as_(dfdx)

# State initialization if len(state) == 0: state['step'] = 0 <del> state['square_avg'] = grad.new().resize_as_(grad).fill_(1) </del> <ins> state['square_avg'] = grad.new().resize_as_(grad).zero_() </ins> square_avg = state['square_avg'] alpha = group['alpha']
Change:
When both PyTorch and Pandas are imported, calling results in a segfault. The issue was originally reported by It looks like calling returns a null pointer here: Examining in GDB: I suspect it's both PyTorch and Pandas doing threading magic and somehow getting in each other's way. I haven't yet been able to provide a minimal example that triggers this behaviour and does not use my package; it's likely that I'm doing something odd that causes this. More documentation here: Possibly related issues: and Python version: 3.6 PyTorch version: v0.2.0 Pandas version: 0.20.1

Discussion:
#include <Python.h> #include "THP.h" <ins> #include "torch/csrc/utils/auto_gil.h" </ins> // Adapted from fblualib void* ObjectPtrAllocator::malloc(ptrdiff_t size) {

} void ObjectPtrAllocator::free(void* ptr) { <del> object = nullptr; </del> <ins> { AutoGIL gil; object = nullptr; } </ins> allocator->free(allocatorContext, ptr); delete this; } void StorageWeakRefAllocator::free(void* ptr) { <del> PyGILState_STATE gstate = PyGILState_Ensure(); PyObject_SetAttrString(object.get(), "cdata", Py_None); object = nullptr; PyGILState_Release(gstate); </del> <ins> { AutoGIL gil; PyObject_SetAttrString(object.get(), "cdata", Py_None); object = nullptr; } </ins> allocator->free(allocatorContext, ptr); delete this; }

// See Note [Numpy memory management] void* NumpyArrayAllocator::realloc(void* ptr, ptrdiff_t size) { <del> PyArrayObject *array_ptr = (PyArrayObject*)object.get(); if (array_ptr && ptr == PyArray_DATA(array_ptr)) { void* newPtr = this->malloc(size); memcpy(newPtr, ptr, std::min((size_t) size, (size_t) PyArray_NBYTES(array_ptr))); // Whee! We're done! object = nullptr; return newPtr; } return allocator->realloc(allocatorContext, ptr, size); </del> <ins> throw std::logic_error("NumpyArrayAllocator::realloc() not supported"); </ins> } // See Note [Numpy memory management]

PyArrayObject *array_ptr = (PyArrayObject*)object.get(); if (!array_ptr || ptr != PyArray_DATA(array_ptr)) throw std::logic_error("invalid call to NumpyArrayAllocator::free()"); <del> object = nullptr; </del> <ins> { AutoGIL gil; object = nullptr; } </ins> delete this; } #endif

} cudaError_t CudaStorageWeakRefAllocator::free(void* ptr) { <del> PyGILState_STATE gstate = PyGILState_Ensure(); PyObject_SetAttrString(object.get(), "cdata", Py_None); object = nullptr; PyGILState_Release(gstate); </del> <ins> { AutoGIL gil; PyObject_SetAttrString(object.get(), "cdata", Py_None); object = nullptr; } </ins> cudaError_t err = allocator->free(allocatorContext, ptr); delete this; return err;

// See Note [Numpy memory management] &THNumpyArrayAllocator, new NumpyArrayAllocator(numpy_array))); <ins> THStorage_(clearFlag)(storage.get(), TH_STORAGE_RESIZABLE); </ins> result = THTensor_(newWithStorage)(LIBRARY_STATE storage, 0, sizes, strides); } else

return: self cname: resize cpu_half: True <ins> before_call: THPUtils_assert(arg_self->storage->flag & TH_STORAGE_RESIZABLE, "calling resize_ on a tensor that has non-resizable storage. Clone it first " "or create a new tensor instead."); </ins> arguments: - THTensor* self - arg: THSize* size
Change:
The output is also incorrect. It's the output from the sorted indices, instead of the user specified indices. Reported by

Discussion:
self.assertEqual(output[0][0].sum().data[0], 0) self.assertEqual(output[1][2].sum().data[0], 0) <ins> def test_embedding_max_norm(self): embedding = nn.Embedding(22, 5, max_norm=1.0) input = Variable(torch.LongTensor([2, 8, 8, 6])) output = embedding(input) self.assertEqual(output[1], output[2]) self.assertTrue(output.data.norm(p=2, dim=1).le(1).all()) @unittest.skipIf(not TEST_CUDA, "CUDA unavailable") def test_embedding_max_norm_cuda(self): embedding = nn.Embedding(22, 5, max_norm=1.0).cuda() input = Variable(torch.LongTensor([2, 8, 8, 6])).cuda() output = embedding(input) self.assertEqual(output[1], output[2]) self.assertTrue(output.data.norm(p=2, dim=1).le(1).all()) </ins> def test_embedding_functional(self): a = Variable(torch.LongTensor([ [1, 3, 2],

@staticmethod def _renorm(ctx, indices, weight, max_norm, norm_type): <del> if indices.dim() == 2: indices = indices.clone().view(-1) </del> <ins> # clone indices since LookupTable_renorm modifies it in-place </ins> ctx._backend.LookupTable_renorm( ctx._backend.library_state, <del> indices, </del> <ins> indices.clone().view(-1), </ins> weight, max_norm, norm_type
Change:
The indices don't align with the values. In the example given on the website, by this logic, seems like it's indexing the columns first and that the indices are in reverse order to their values. Seems like corresponds to 4 and corresponds to 3. i = torch.LongTensor([[0, 1], [2, 0]]) v = torch.FloatTensor([3, 4]) (i, v, torch.Size([2,3])).todense() 0 0 3 4 0 0 [torch.FloatTensor of size 2x3] If I change the indices to and , I get i = torch.LongTensor([[0, 1], [0,1]]) v = torch.FloatTensor([3, 4]) (i, v, torch.Size([2,3])).todense() 3 0 0 0 4 0 [torch.FloatTensor of size 2x3] seems like unexpected behavior. Also if I try - should be intuitive - putting a value in position i = torch.LongTensor([[0, 1], [2, 0], [1,1]]) v = torch.FloatTensor([3, 4, 5]) (i, v, torch.Size([2,3])).to_dense() Traceback (most recent call last): File "<stdin", line 1, in <moduleRuntimeError: invalid argument 2: number of dimensions must be nDimI + nDimV at /pytorch/torch/lib/THS/generic/THSTensor.c:169 Help please?
This is a doc problem, thank you. The docs are misleading here, because it is not that is a coordinate, but 0 from the first tensor, and 2 from the second tensor. I'll try to rewrite these docs so they are clearer.
Basically, our constructor expects to be (transpose of what you gave it).
We should consider changing that

Discussion:
are zeros. A sparse tensor is represented as a pair of dense tensors: a tensor <del> of values and a tensor of indices.  A sparse tensor can be constructed </del> <ins> of values and a 2D tensor of indices.  A sparse tensor can be constructed </ins> by providing these two tensors, as well as the size of the sparse tensor <del> (which cannot be inferred from these tensors!) </del> <ins> (which cannot be inferred from these tensors!)  Suppose we want to define a sparse tensor with the entry 3 at location (0, 2), entry 4 at location (1, 0), and entry 5 at location (1, 2).  We would then write: </ins> <del> >>> i = torch.LongTensor([[0, 1], [2, 0]]) >>> v = torch.FloatTensor([3, 4]) </del> <ins> >>> i = torch.LongTensor([[0, 1, 1], [2, 0, 2]]) >>> v = torch.FloatTensor([3, 4, 5]) </ins> >>> torch.sparse.FloatTensor(i, v, torch.Size([2,3])).to_dense() 0  0  3 <del> 4  0  0 </del> <ins> 4  0  5 [torch.FloatTensor of size 2x3] Note that the input to LongTensor is NOT a list of index tuples.  If you want to write your indices this way, you should transpose before passing them to the sparse constructor: >>> i = torch.LongTensor([[0, 2], [1, 0], [1, 2]]) >>> v = torch.FloatTensor([3,      4,      5    ]) >>> torch.sparse.FloatTensor(i.t(), v, torch.Size([2,3])).to_dense() 0  0  3 4  0  5 </ins> [torch.FloatTensor of size 2x3] You can also construct hybrid sparse tensors, where only the first n
Change:
see the commit comment: To reproduce install ros-kinetic-tf 1.11.8-0xenial---0800 package from the deb xenial main repository. my system is: lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 16.04.1 LTS Release: 16.04 Codename: xenial cuda version is 8.0

Discussion:
IF ($ENV{PYTORCH_BINARY_BUILD}) MESSAGE(STATUS "PYTORCH_BINARY_BUILD detected. Statically linking libstdc++") SET(CMAKE_CXX_FLAGS "-static-libstdc++ ${CMAKE_CXX_FLAGS}") <ins> IF (UNIX AND NOT APPLE) # hiding statically linked library symbols, this flag is not available for the linker under MACOSX SET(CMAKE_CXX_FLAGS "-Wl,--exclude-libs,ALL ${CMAKE_CXX_FLAGS}") ENDIF(UNIX AND NOT APPLE) </ins> ENDIF() ADD_LIBRARY(shm SHARED core.cpp)
Change:
The loss will never decrease when I train a model using multi-gpus but without gpu0. This is an example I modified from the : If the result after one epoch is If the result after one epoch is
should always contain . If you want to specify which GPUs to use, you can set the environment variable , either by running your script like this: or set it programmatically in your script like this: So that deviceids 0, 1 will map to GPUs 1, 2, respectively. (Note that for the second method, you should put that line before any cuda relevant function call.)
Thank you.
<!-- drci-comment-start --:pagefacingup: Preview :pagefacingup: Preview :question: Need help or want to give feedback on the CI? Visit our Note: Links to docs will display an error until the docs builds have been completed. This comment was automatically generated by Dr. CI and updates every 15 minutes. <!-- drci-comment-end --

Discussion:
l = nn.Linear(10, 5).float().cuda() i = Variable(torch.randn(20, 10).float().cuda(1)) l.cuda(1) <del> expected_out = l(i).data l.cuda(0) out = dp.data_parallel(l, i, (0, 1)) self.assertEqual(out.get_device(), 0) self.assertEqual(out.data, expected_out) </del> <ins> expected_out = l(i) loss = expected_out.sum() loss.backward() expected_grads = [] for param in l.parameters(): expected_grads.append(param.grad.clone()) dev_ids_list = [(0, 1), (1, 0)] for dev_id in dev_ids_list: with torch.cuda.device(dev_id[0]): l.cuda() l.zero_grad() out = dp.data_parallel(l, i, dev_id) loss = out.sum() loss.backward() self.assertEqual(out.get_device(), dev_id[0]) self.assertEqual(out.data, expected_out.data) for expected, param in zip(expected_grads, l.parameters()): self.assertEqual(param.grad.data, expected.data) </ins> # Check for None device_ids <ins> l = l.cuda() </ins> out = dp.data_parallel(l, i) @unittest.skipIf(not TEST_MULTIGPU, "multi-GPU not supported")

""" # TODO: try to find an input on another gpu, copy it, # and accumulate into the copy <ins> if destination is None: destination = torch.cuda.current_device() </ins> input_size = inputs[0].size() <ins> nccl_root = None </ins> for i, inp in enumerate(inputs): assert inp.is_cuda, "reduce_add expects all inputs to be on GPUs" <ins> if inp.get_device() == destination: nccl_root = i </ins> if inp.size() != input_size: got = 'x'.join(str(x) for x in inp.size()) expected = 'x'.join(str(x) for x in input_size) raise ValueError("input {} has invalid size: got {}, but expected " "{}".format(i, got, expected)) <del> if destination is None: destination = torch.cuda.current_device() </del> <ins> assert nccl_root is not None, "reduce_add expects destination to be on the same GPU with one of the tensors" </ins> with torch.cuda.device(destination): result = type(inp)(input_size).zero_() if nccl.is_available(inputs) and inputs[0].get_device() == destination: outputs = [result] + [t.new(t.size()) for t in inputs[1:]] <del> nccl.reduce(inputs, outputs, root=destination) </del> <ins> nccl.reduce(inputs, outputs, root=nccl_root) </ins> return result for inp in inputs: