jiebi/CodeConvo huggingface dataset:
Code change:
This happens when the index tensor contains duplicate elements. If this is not allowed indexcopy should raise an exception when this happens and we should fix the test. Otherwise we need to fix the autograd op. Here's an example: cc
Why would you with repeated indices? Sounds for me like UB is a reasonable. We should check that in C
I ran into this in and fixed the test for now. I'm not sure how to interpret last comment: did you mean UB behavior is "unreasonable" rather than "a reasonable"? The last sentence reads that way.
I'd be ok with saying that it's UB in such case. There's no natural choice, and it can decrease performance.
Is this issue still worth working on? If adding the additional check can decrease performance, can we add another argument specifying that indexes contain duplicate elements?

Related discussion:
text: (Addcdiv,       (),                 ((S, S), (S, S), torch.rand(S, S) + 1e-2)   ), (Addcdiv,       (0.6,),             ((S, S), (S, S), torch.rand(S, S) + 1e-2), 'scale'), (IndexAdd,      (0,),               ((S, S), index_variable(2, S), (2, S))      ), <del> (IndexCopy,     (0,),               ((S, S), index_variable(2, S), (2, S))      ), </del> <ins> # (IndexCopy,     (0,),               ((S, S), index_variable(2, S), (2, S))      ), </ins> (IndexFill,     (0, 2),             ((S, S), index_variable(2, S))              ), (IndexSelect,   (0,),               ((S, S), index_variable(2, S))              ), (Gather,        (0,),               ((M, S), gather_variable((S, S), 1, M))     ),
Code change:
As Arthur Szlam reports, fb-internal cudnn is still lagging behind, and giving batch-size 1024 with batchnorm is raising an error from cudnn. Need to check for compile-time version and disable this codepath
The 1024 limitation was removed in 5.1.10

Related discussion:
text: <del> import torch from torch.autograd.function import Function from torch._thnn import type2backend import torch.backends.cudnn as cudnn class BatchNorm(Function): def __init__(self, running_mean, running_var, training, momentum, eps): super(BatchNorm, self).__init__() self.running_mean = running_mean self.running_var = running_var self.training = training self.momentum = momentum self.eps = eps def forward(self, input, weight=None, bias=None): self.save_for_backward(input, weight, bias) # don't use cuDNN for half inputs because cuDNN requires the weight and # bias tensors to be floats, unlike THCUNN which requires half tensors. self.use_cudnn = (cudnn.is_acceptable(input) and cudnn.version() > 5110 and weight is not None and bias is not None and not isinstance(input, torch.cuda.HalfTensor)) # temporary buffers used in forward and backward num_features = input.size(1) _save_mean = input.new(num_features) _save_std = input.new(num_features) output = input.new(input.size()) if self.use_cudnn: torch._C._cudnn_batch_norm_forward( input, output, weight, bias, self.running_mean, self.running_var, _save_mean, _save_std, self.training, self.momentum, self.eps) else: backend = type2backend[type(input)] backend.BatchNormalization_updateOutput( backend.library_state, input, output, weight, bias, self.running_mean, self.running_var, _save_mean, _save_std, self.training, self.momentum, self.eps) if self.requires_grad: self._save_mean = _save_mean self._save_std = _save_std return output def backward(self, grad_output): input, weight, bias = self.saved_tensors grad_input, grad_weight, grad_bias = None, None, None if self.needs_input_grad[0] or self.use_cudnn: grad_input = input.new(input.size()) if (len(self.needs_input_grad) > 1 and self.needs_input_grad[1]) or self.use_cudnn: grad_weight = weight.new(weight.size()).zero_() if (len(self.needs_input_grad) > 1 and self.needs_input_grad[2]) or self.use_cudnn: grad_bias = bias.new(bias.size()).zero_() if self.use_cudnn and self.training: # cudnn does not support backward in evaluate mode torch._C._cudnn_batch_norm_backward( input, grad_output, grad_input, grad_weight, grad_bias, weight, self.running_mean, self.running_var, self._save_mean, self._save_std, self.training, self.eps) else: grad_output = grad_output.contiguous() backend = type2backend[type(input)] backend.BatchNormalization_backward( backend.library_state, input, grad_output, grad_input, grad_weight, grad_bias, weight, self.running_mean, self.running_var, self._save_mean, self._save_std, self.training, 1.0, self.eps) return grad_input, grad_weight, grad_bias </del>
Code change:
As Arthur Szlam reports, fb-internal cudnn is still lagging behind, and giving batch-size 1024 with batchnorm is raising an error from cudnn. Need to check for compile-time version and disable this codepath
The 1024 limitation was removed in 5.1.10

Related discussion:
text: def _initialize_backend(): from .._functions.thnn import _all_functions as _thnn_functions from .._functions.linear import Linear <del> from .._functions.batchnorm import BatchNorm </del> from .._functions.conv import ConvNd from .._functions.rnn import RNN,  RNNTanhCell, RNNReLUCell, GRUCell, LSTMCell
Code change:
As Arthur Szlam reports, fb-internal cudnn is still lagging behind, and giving batch-size 1024 with batchnorm is raising an error from cudnn. Need to check for compile-time version and disable this codepath
The 1024 limitation was removed in 5.1.10

Related discussion:
text: HingeEmbeddingLoss, MarginRankingLoss backend.register_function('Linear', Linear) <del> backend.register_function('BatchNorm', BatchNorm) </del> backend.register_function('ConvNd', ConvNd) backend.register_function('RNN', RNN) backend.register_function('RNNTanhCell', RNNTanhCell)
Code change:
Hi, I need to pass input through one nn.Module, then argmax the output, and then pass that output to a second nn.Module. I want to backprop through the argmax back to the weights of the first module. That's impossible right now, though, it seems. When I loop through the gradient of the weights of my first module (with the following code): firstmodel.zerograd() secondmodel.zerograd() loss.backward() # retainvariables=True for param in secondmodel.parameters(): print( 'data: ', ) print( 'grad data: ', ) for param in firstmodel.parameters(): print( 'data: ', ) print( 'grad data: ', ) I get the following error, stating that those gradients can no longer be viewed/exist: -main()
argmax is not differentiable (and even if you try to think about it as a function differentiable almost everywhere, its derivative is always 0, so you won't get anything meaningful).

Related discussion:
text: | :attr:`stride` controls the stride for the cross-correlation. | If :attr:`padding` is non-zero, then the input is implicitly zero-padded on both sides <del> for :attr:`padding` number of points | :attr:`dilation` controls the spacing between the kernel points. It is harder to describe, but this `link`_ has a nice visualization of what :attr:`dilation` does. | :attr:`groups` controls the connections between inputs and outputs. </del> <ins> for :attr:`padding` number of points. | :attr:`dilation` controls the spacing between the kernel points; also known as the à trous algorithm. It is harder to describe, but this `link`_ has a nice visualization of what :attr:`dilation` does. | :attr:`groups` controls the connections between inputs and outputs. `in_channels` and `out_channels` must both be divisible by `groups`. </ins> |       At groups=1, all inputs are convolved to all outputs. |       At groups=2, the operation becomes equivalent to having two conv layers side by side, each seeing half the input channels, and producing half the output channels, and both subsequently concatenated. <ins> At groups=`in_channels`, each input channel is convolved with its own set of filters (of size `out_channels // in_channels`). </ins> .. note::
Code change:
Hi, I need to pass input through one nn.Module, then argmax the output, and then pass that output to a second nn.Module. I want to backprop through the argmax back to the weights of the first module. That's impossible right now, though, it seems. When I loop through the gradient of the weights of my first module (with the following code): firstmodel.zerograd() secondmodel.zerograd() loss.backward() # retainvariables=True for param in secondmodel.parameters(): print( 'data: ', ) print( 'grad data: ', ) for param in firstmodel.parameters(): print( 'data: ', ) print( 'grad data: ', ) I get the following error, stating that those gradients can no longer be viewed/exist: -main()
argmax is not differentiable (and even if you try to think about it as a function differentiable almost everywhere, its derivative is always 0, so you won't get anything meaningful).

Related discussion:
text: | :attr:`stride` controls the stride for the cross-correlation. | If :attr:`padding` is non-zero, then the input is implicitly zero-padded on both sides <del> for :attr:`padding` number of points | :attr:`dilation` controls the spacing between the kernel points. It is harder to describe, but this `link`_ has a nice visualization of what :attr:`dilation` does. | :attr:`groups` controls the connections between inputs and outputs. </del> <ins> for :attr:`padding` number of points. | :attr:`dilation` controls the spacing between the kernel points; also known as the à trous algorithm. It is harder to describe, but this `link`_ has a nice visualization of what :attr:`dilation` does. | :attr:`groups` controls the connections between inputs and outputs. `in_channels` and `out_channels` must both be divisible by `groups`. </ins> |       At groups=1, all inputs are convolved to all outputs. |       At groups=2, the operation becomes equivalent to having two conv layers side by side, each seeing half the input channels, and producing half the output channels, and both subsequently concatenated. <ins> At groups=`in_channels`, each input channel is convolved with its own set of filters (of size `out_channels // in_channels`). </ins> The parameters :attr:`kernel_size`, :attr:`stride`, :attr:`padding`, :attr:`dilation` can either be:
Code change:
Hi, I need to pass input through one nn.Module, then argmax the output, and then pass that output to a second nn.Module. I want to backprop through the argmax back to the weights of the first module. That's impossible right now, though, it seems. When I loop through the gradient of the weights of my first module (with the following code): firstmodel.zerograd() secondmodel.zerograd() loss.backward() # retainvariables=True for param in secondmodel.parameters(): print( 'data: ', ) print( 'grad data: ', ) for param in firstmodel.parameters(): print( 'data: ', ) print( 'grad data: ', ) I get the following error, stating that those gradients can no longer be viewed/exist: -main()
argmax is not differentiable (and even if you try to think about it as a function differentiable almost everywhere, its derivative is always 0, so you won't get anything meaningful).

Related discussion:
text: composed of several input planes. This module can be seen as the gradient of Conv1d with respect to its input. <del> It is sometimes (but incorrectly) refered to as a deconvolutional operation. </del> <ins> It is also known as a fractionally-strided convolution or a deconvolution (although it is not an actual deconvolution operation). | :attr:`stride` controls the stride for the cross-correlation. | If :attr:`padding` is non-zero, then the input is implicitly zero-padded on both sides for :attr:`padding` number of points. | If :attr:`output_padding` is non-zero, then the output is implicitly zero-padded on one side for :attr:`output_padding` number of points. | :attr:`dilation` controls the spacing between the kernel points; also known as the à trous algorithm. It is harder to describe, but this `link`_ has a nice visualization of what :attr:`dilation` does. | :attr:`groups` controls the connections between inputs and outputs. `in_channels` and `out_channels` must both be divisible by `groups`. |       At groups=1, all inputs are convolved to all outputs. |       At groups=2, the operation becomes equivalent to having two conv layers side by side, each seeing half the input channels, and producing half the output channels, and both subsequently concatenated. At groups=`in_channels`, each input channel is convolved with its own set of filters (of size `out_channels // in_channels`). </ins> .. note::
Code change:
Hi, I need to pass input through one nn.Module, then argmax the output, and then pass that output to a second nn.Module. I want to backprop through the argmax back to the weights of the first module. That's impossible right now, though, it seems. When I loop through the gradient of the weights of my first module (with the following code): firstmodel.zerograd() secondmodel.zerograd() loss.backward() # retainvariables=True for param in secondmodel.parameters(): print( 'data: ', ) print( 'grad data: ', ) for param in firstmodel.parameters(): print( 'data: ', ) print( 'grad data: ', ) I get the following error, stating that those gradients can no longer be viewed/exist: -main()
argmax is not differentiable (and even if you try to think about it as a function differentiable almost everywhere, its derivative is always 0, so you won't get anything meaningful).

Related discussion:
text: output_padding (int or tuple, optional): Zero-padding added to one side of the output groups (int, optional): Number of blocked connections from input channels to output channels bias (bool, optional): If True, adds a learnable bias to the output <ins> dilation (int or tuple, optional): Spacing between kernel elements </ins> Shape: - Input: :math:`(N, C_{in}, L_{in})`
Code change:
Hi, I need to pass input through one nn.Module, then argmax the output, and then pass that output to a second nn.Module. I want to backprop through the argmax back to the weights of the first module. That's impossible right now, though, it seems. When I loop through the gradient of the weights of my first module (with the following code): firstmodel.zerograd() secondmodel.zerograd() loss.backward() # retainvariables=True for param in secondmodel.parameters(): print( 'data: ', ) print( 'grad data: ', ) for param in firstmodel.parameters(): print( 'data: ', ) print( 'grad data: ', ) I get the following error, stating that those gradients can no longer be viewed/exist: -main()
argmax is not differentiable (and even if you try to think about it as a function differentiable almost everywhere, its derivative is always 0, so you won't get anything meaningful).

Related discussion:
text: composed of several input planes. This module can be seen as the gradient of Conv2d with respect to its input. <del> It is sometimes (but incorrectly) refered to as a deconvolutional operation. </del> <ins> It is also known as a fractionally-strided convolution or a deconvolution (although it is not an actual deconvolution operation). </ins> | :attr:`stride` controls the stride for the cross-correlation. | If :attr:`padding` is non-zero, then the input is implicitly zero-padded on both sides <del> for :attr:`padding` number of points </del> <ins> for :attr:`padding` number of points. </ins> | If :attr:`output_padding` is non-zero, then the output is implicitly zero-padded on one side <del> for :attr:`output_padding` number of points | :attr:`dilation` controls the spacing between the kernel points. It is harder to describe, but this `link`_ has a nice visualization of what :attr:`dilation` does. | :attr:`groups` controls the connections between inputs and outputs. </del> <ins> for :attr:`output_padding` number of points. | :attr:`dilation` controls the spacing between the kernel points; also known as the à trous algorithm. It is harder to describe, but this `link`_ has a nice visualization of what :attr:`dilation` does. | :attr:`groups` controls the connections between inputs and outputs. `in_channels` and `out_channels` must both be divisible by `groups`. </ins> |       At groups=1, all inputs are convolved to all outputs. |       At groups=2, the operation becomes equivalent to having two conv layers side by side, each seeing half the input channels, and producing half the output channels, and both subsequently concatenated. <ins> At groups=`in_channels`, each input channel is convolved with its own set of filters (of size `out_channels // in_channels`). </ins> The parameters :attr:`kernel_size`, :attr:`stride`, :attr:`padding`, :attr:`output_padding` can either be: <del> - a single ``int`` -- in which case the same value is used for the height and width dimension </del> <ins> - a single ``int`` -- in which case the same value is used for the height and width dimensions </ins> - a ``tuple`` of two ints -- in which case, the first `int` is used for the height dimension, and the second `int` for the width dimension
Code change:
This is how I implement the decoder of a sequence to sequence model I'm not sure about the new autograd mechanics but this worked in the previous version. If I didn't make the unrolling codes a function it will work. It will also work with CPU. I compiled from source with Python 2.7, CUDA 8.0 and Cudnn 6
Thanks for the minimal reproduction! It broke OpenNMT and a big internal model but we weren't looking forward to isolating what parts of those had the issue.
That's great, I'll take a look today. Thanks!
I've pushed a fix to .
How does the inference work in this case? I mean, once you train it how do you test it in the test phase?

Related discussion:
text: { PyObject_GC_UnTrack(self); THPFunction_clear(self); <ins> self->cdata_ptr.~weak_ptr(); </ins> self->cdata.~PyFunction(); Py_TYPE(self)->tp_free((PyObject*)self); }
Code change:
This is how I implement the decoder of a sequence to sequence model I'm not sure about the new autograd mechanics but this worked in the previous version. If I didn't make the unrolling codes a function it will work. It will also work with CPU. I compiled from source with Python 2.7, CUDA 8.0 and Cudnn 6
Thanks for the minimal reproduction! It broke OpenNMT and a big internal model but we weren't looking forward to isolating what parts of those had the issue.
That's great, I'll take a look today. Thanks!
I've pushed a fix to .
How does the inference work in this case? I mean, once you train it how do you test it in the test phase?

Related discussion:
text: // most fields THPFunction* self = (THPFunction*)obj; new (&self->cdata) torch::autograd::PyFunction(obj); <ins> new (&self->cdata_ptr) std::weak_ptr<torch::autograd::PyFunction>(); </ins> self->cdata.num_inputs = -1; self->cdata.is_stochastic = PyObject_IsInstance(obj, THPStochasticFunctionClass); return obj;
Code change:
This is how I implement the decoder of a sequence to sequence model I'm not sure about the new autograd mechanics but this worked in the previous version. If I didn't make the unrolling codes a function it will work. It will also work with CPU. I compiled from source with Python 2.7, CUDA 8.0 and Cudnn 6
Thanks for the minimal reproduction! It broke OpenNMT and a big internal model but we weren't looking forward to isolating what parts of those had the issue.
That's great, I'll take a look today. Thanks!
I've pushed a fix to .
How does the inference work in this case? I mean, once you train it how do you test it in the test phase?

Related discussion:
text: } }; <ins> // Similar to shared_from_this. There's a problem that the Python object // and its cdata depend on each other being alive, so we can't keep // shared_ptrs as members, but we'd like to be able to manage the lifetime of // the objects using shared_ptrs in the C++ graph. The only way to get a new // shared_ptr that references them is through THPFunction_asFunction. When // called for the first time it will allocate a new shared_ptr and save a // weak_ptr in cdata_ptr attr. Later, when we try to take another reference, // we'll try to lock cdata_ptr and return its value if successful. Otherwise it // means that all shared_ptrs returned previously have been freed, so we can // create a new one. This ensures that this object is managed by at most one // shared_ptr control block at any time - a guarantee we depend on in other places // (e.g. we use weak_ptrs in SavedVariable because we know it won't go out of scope). </ins> std::shared_ptr<PyFunction> THPFunction_asFunction(THPFunction* self) { if (!self) { return std::shared_ptr<PyFunction>(); } Py_INCREF((PyObject*)self); <del> return std::shared_ptr<PyFunction>(&self->cdata, Decref()); </del> <ins> auto ptr = self->cdata_ptr.lock(); if (ptr) return ptr; ptr = std::shared_ptr<PyFunction>(&self->cdata, Decref()); self->cdata_ptr = ptr; return ptr; </ins> }
Code change:
This is how I implement the decoder of a sequence to sequence model I'm not sure about the new autograd mechanics but this worked in the previous version. If I didn't make the unrolling codes a function it will work. It will also work with CPU. I compiled from source with Python 2.7, CUDA 8.0 and Cudnn 6
Thanks for the minimal reproduction! It broke OpenNMT and a big internal model but we weren't looking forward to isolating what parts of those had the issue.
That's great, I'll take a look today. Thanks!
I've pushed a fix to .
How does the inference work in this case? I mean, once you train it how do you test it in the test phase?

Related discussion:
text: std::vector<bool> *is_variable_input; char has_freed_buffers; <ins> // See a comment in THPFucntion_asFunction for details about this field. std::weak_ptr<torch::autograd::PyFunction> cdata_ptr; </ins> torch::autograd::PyFunction cdata; };
Code change:
This is how I implement the decoder of a sequence to sequence model I'm not sure about the new autograd mechanics but this worked in the previous version. If I didn't make the unrolling codes a function it will work. It will also work with CPU. I compiled from source with Python 2.7, CUDA 8.0 and Cudnn 6
Thanks for the minimal reproduction! It broke OpenNMT and a big internal model but we weren't looking forward to isolating what parts of those had the issue.
That's great, I'll take a look today. Thanks!
I've pushed a fix to .
How does the inference work in this case? I mean, once you train it how do you test it in the test phase?

Related discussion:
text: // should have saved the grad accumulator. Even if the Variable no longer // alive, the accumulator should be kept alive by the references in the graph). if (requires_grad && !grad_fn && weak_grad_fn.expired() && grad_accumulator.expired()) <del> throw std::logic_error("No grad accumulator for a saved leaf!"); </del> <ins> throw std::logic_error("No grad accumulator for a saved leaf!"); </ins> new_var->grad_accumulator = grad_accumulator; return new_var;
Code change:
This is how I implement the decoder of a sequence to sequence model I'm not sure about the new autograd mechanics but this worked in the previous version. If I didn't make the unrolling codes a function it will work. It will also work with CPU. I compiled from source with Python 2.7, CUDA 8.0 and Cudnn 6
Thanks for the minimal reproduction! It broke OpenNMT and a big internal model but we weren't looking forward to isolating what parts of those had the issue.
That's great, I'll take a look today. Thanks!
I've pushed a fix to .
How does the inference work in this case? I mean, once you train it how do you test it in the test phase?

Related discussion:
text: LDFLAGS="-L$INSTALL_DIR/lib " LD_POSTFIX=".so.1" LD_POSTFIX_UNVERSIONED=".so" <del> if [[ $(uname) == 'Darwin' ]]; then </del> <ins> if [[ $(uname) == 'Darwin' ]]; then </ins> LDFLAGS="$LDFLAGS -Qunused-arguments -Wl,-rpath,@loader_path" LD_POSTFIX=".1.dylib" LD_POSTFIX_UNVERSIONED=".dylib"