jiebi/CodeConvo huggingface dataset:
Change:
The last line here fails on CPU or when CUDNN is otherwise unavailable: This is because accidentally assumes 's in-place semantics rather than the functional semantics of ( gets it right): I can push a PR that fixes this, or one of the devs can put it in the next bugfix PR:
If you have a ready solution for your issues feel free to send the PR straight away. Contributions are welcome!
Thanks

Discussion:
def forward(input, weight, hidden): if batch_first: <del> input.transpose(0, 1) </del> <ins> input = input.transpose(0, 1) </ins> nexth, output = func(input, hidden, weight) if batch_first: <del> output.transpose(0, 1) </del> <ins> output = output.transpose(0, 1) </ins> return output, nexth
Change:
THNN convolutions are slower due to unnecessary .zero() calls on gradients: is not needed same for bias, prob worth adding a constant in C-side that controls whether to override grad or not
Same for batch_norm and auto thnn functions, not sure if all of them though. Prob the speedups will be quite significant.
I think they are needed. THNN functions accumulate the gradient instead of overwriting the buffer, don't they?
only grad wrt weight, inputs are overriden
We should reopen this, still many functions have unnecessary .zero() calls on gradinput and gradparam, e.g.: : : most of the functions don't need it
I think the ones are currently necessary.
for nn yes, but cudnn has: and on cudnnConvolutionBackwardFilter and on cudnnBatchNormalizationBackward which control addition of gradients and should be set to 1 and 0 in pytorch cudnn (it is and 1 in )
Yeah, we can remove it on BatchNorm. If you look at Conv, it already only zeros out grad_weight for cunn.
this is now fixed in master:

Discussion:
def make_grad_input(fn): def call_grad_input(self, bufs, input, weight, grad_output): backend = type2backend[type(input)] <del> grad_input = input.new().resize_as_(input).zero_() </del> <ins> grad_input = input.new().resize_as_(input) </ins> kernel_size = weight.size()[2:] args = parse_arguments(self, fn.arguments[5:], bufs, kernel_size) getattr(backend, fn.name)(backend.library_state, input, grad_output,

def backward(self, grad_output): tensors = self.saved_tensors input, params = tensors[0], tensors[1:] <del> grad_input = (input.new().resize_as_(input).zero_() </del> <ins> grad_input = (input.new().resize_as_(input) </ins> if self.needs_input_grad[0] else None,) grad_param = tuple(p.new().resize_as_(p).zero_() if self.needs_input_grad[i+1] else None for i, p in enumerate(params))
Change:
On Python 3.5.2 errors out with . map and map2 seem to be work just fine.

Discussion:
x_clone ^= y self.assertEqual(x_clone, xor_result) <ins> def test_apply(self): x = torch.range(1, 5) res = x.clone().apply_(lambda k: k + k) self.assertEqual(res, x * 2) self.assertRaises(RuntimeError, lambda: x.apply_(lambda k: "str")) </ins> if __name__ == '__main__': unittest.main()

python_name: apply_ defined_if: "!IS_CUDA" only_register: True <ins> override_method_flags: METH_O </ins> ]] static PyObject * THPTensor_(apply)(THPTensor *self, PyObject *arg) {

#endif /* !IS_CUDA */ #undef BUILD_REAL_FMT
Change:
Repro: Strides seem to be copied correctly, and the first row is also ok. All other rows are garbage.

Discussion:
array = np.array([1, 2, 3, 4], dtype=dtype) self.assertEqual(torch.from_numpy(array), torch.Tensor([1, 2, 3, 4])) <ins> # check storage offset x = np.linspace(1, 125, 125) x.shape = (5, 5, 5) x = x[1] expected = torch.range(1, 125).view(5, 5, 5)[1] self.assertEqual(torch.from_numpy(x), expected) # check noncontiguous x = np.linspace(1, 25, 25) x.shape = (5, 5) expected = torch.range(1, 25).view(5, 5).t() self.assertEqual(torch.from_numpy(x.T), expected) # check noncontiguous with holes x = np.linspace(1, 125, 125) x.shape = (5, 5, 5) x = x[:, 1] expected = torch.range(1, 125).view(5, 5, 5)[:, 1] self.assertEqual(torch.from_numpy(x), expected) </ins> @unittest.skipIf(not TEST_NUMPY, "Numpy not found") def test_numpy_index(self): i = np.int32([0, 1, 2])

#ifdef NUMPY_TYPE_ENUM THTensor* THPTensor_(fromNumpy)(PyObject *numpy_array) { PyArrayObject *array = (PyArrayObject*)numpy_array; <del> THStoragePtr storage = THStorage_(newWithDataAndAllocator)( (real*)PyArray_DATA(array), PyArray_NBYTES(array) / sizeof(real), &THNumpyArrayAllocator, new NumpyArrayAllocator(numpy_array)); </del> // Numpy and Torch disagree on empty tensors. In Torch, an empty // tensor is a tensor with zero dimensions. In Numpy, an empty tensor

// So we'll convert all Numpy tensors of 0 elements to empty Torch tensors. if (PyArray_SIZE(array) != 0) { auto ndim = PyArray_NDIM(array); <ins> size_t storage_size = 1; </ins> THLongStoragePtr sizes = THLongStorage_newWithSize(ndim); long *sizes_data = sizes->data; for (int i = 0; i < ndim; ++i) {

"future releases."); return NULL; } <ins> // XXX: this won't work for negative strides storage_size += strides_data[i] * (sizes_data[i] - 1); </ins> } <ins> THStoragePtr storage = THStorage_(newWithDataAndAllocator)( (real*)PyArray_DATA(array), storage_size, &THNumpyArrayAllocator, new NumpyArrayAllocator(numpy_array)); </ins> THTensor *result = THTensor_(newWithStorage)(storage, 0, sizes, strides); return result; } else { <ins> THStoragePtr storage = THStorage_(new)(); </ins> THTensor *result = THTensor_(newWithStorage)(storage, 0, NULL, NULL); return result; }
Change:
to reproduce Will result in:
What's your GPU and CUDA version?
TitanX Maxwell, cuda 8.0
In cudnn bindings convolution math type is inferred from input data type (and inferred to be equal to input data type): Math type for convolutions with half inputs and weights should be CUDNNDATAFLOAT because 1) computations with half math are not supported anywhere except Pascal and TX1 2) Even on Pascal, 6.1 cards have very low performance fp16 math 3) fp16 math comes with a set of accuracy problems
Thanks. Seems like batchnorm fp16 support is still missing. e.g: will result in
Ah, yes. fp16 support was removed from cudnn bn backend (because it still requires fp32 for parameters), and probably it is not properly routed to or not supported in nn backend. BTW, be careful with truncating bn params to 16 bits, in our experiments on wide resnets/cifar it cost ~ half a percent of accuracy, though there may be ways to overcome this.
Is there an update regarding this? Could you describe how to truncate the bn params?
did you update PyTorch to a recent version? The snippet no longer raises an error for me
Thanks for the fast response. I installed the newest version via conda. Just retried the above code snippet, it works indeed. Shouldn't it be possible to "half" a entire model like this: For this i get: CUDNNSTATUSBAD_PARAM
Your input size is wrong, models in torchvision expect specific input sizes. You can play around with the batch size, but number of channels and image size are fixed. When I change input size to (64,3,224,224) the snippet above does not return an error.

Discussion:
cudnnDestroyConvolutionDescriptor(desc); } void set(cudnnDataType_t dataType, int dim, int* pad, int* stride, int * upscale) { <ins> cudnnDataType_t mathType = dataType; if (dataType == CUDNN_DATA_HALF) mathType = CUDNN_DATA_FLOAT; </ins> CHECK(cudnnSetConvolutionNdDescriptor(desc, dim, pad, stride, upscale, <del> CUDNN_CROSS_CORRELATION, dataType)); </del> <ins> CUDNN_CROSS_CORRELATION, mathType)); </ins> } };
Change:
x = Variable(((16, 64, 8, 8))) res = nn.Conv2d(16, 16, kernelsize=3, groups=2, bias=False)(x) results in: return ('updateoutput', input, weight, bias) File "", line 158, in thnn groupedargs += [group(t) for t in args] File "", line 153, in group dim = 0 if () == 1 else 1 AttributeError: 'NoneType' object has no attribute 'dim' The in line 158 is because there is no bias. I am running on CPU so this is in the code.

Discussion:
cudnn=True, ), dict( <ins> fullname='Conv2d_groups_thnn', constructor=lambda: nn.Conv2d(4, 6, (3, 2), groups=2), input_size=(2, 4, 6, 5), ), dict( </ins> module_name='ConvTranspose2d', constructor_args=(3, 4, 3, (3, 2), 1, (1, 1)), cudnn=True,

res = [] for g in range(self.groups): def group(tensor, dim=None): <ins> if tensor is None: return None </ins> if dim is None: dim = 0 if tensor.dim() == 1 else 1 n = tensor.size(dim) // self.groups

grouped_args += [group(t) for t in args] res.append(impl[fn_name](self, self._bufs[g], *grouped_args)) if fn_name == 'grad_params': <del> return [torch.cat(t, 0) for t in zip(*res)] </del> <ins> return [torch.cat(t, 0) if t[0] is not None else None for t in zip(*res)] </ins> else: return torch.cat(res, 1)
Change:
I'm running PyTorch on a system with 4 Nvidia K10 GPUs and can't get the Multi-GPU code to work. I attempted to modify the first neural net in the 60-minute tutorial to run on all my GPUs, by adding this line to the cell defining the network: If I then attempt to use the net as follows , Jupyter waits a few seconds and reports the kernel has died. Following along in I see all GPUs encountering an at the same moment. Am I using the API wrong, could there be a configuration issue in my system, or is this a bug in the DataParallel implementation? I'm using CUDA 8 and PyTorch 0.1.6. Edit: my Nvidia driver version is 367.48 I also forgot to add, executing the above code on CUDA without parallelization (i.e. ) works fine.
Is there a stack-trace that you see appear on the command-line of the Jupyter process? Is this a cloud provider box? (I ask to see if I can reproduce it myself)
This is the only command-line output Jupyter gives related to the error: I'm running this on an in-house compute node.
thanks. this is helpful, I'll look further. does nccl have cuda code that we need to build for specific archs? K10 is 3.0 and i build 3.0 code for thc and thcunn
Yes, that's the problem: The README says "Kepler or newer". does NCCL work on K10s?
It should (once you add the necessary compilation flags).

Discussion:
CUDA_INC ?= $(CUDA_HOME)/include NVCC ?= $(CUDA_HOME)/bin/nvcc <del> NVCC_GENCODE ?= -gencode=arch=compute_35,code=sm_35  </del> <ins> NVCC_GENCODE ?= -gencode=arch=compute_30,code=sm_30  -gencode=arch=compute_35,code=sm_35  </ins> -gencode=arch=compute_50,code=sm_50  -gencode=arch=compute_52,code=sm_52  -gencode=arch=compute_52,code=compute_52
Change:
Currently, CMake 3.0 or higher is required for THPP. This is annoying for building from source on Ubuntu 14.04 systems, as default CMake is 2.8, and updating it requires either compiling from source or adding some random repos:
it should work fine if you change it to 2.8. I'm testing this out in a sec, and will push it into master
Nope, it does not for me.

Discussion:
<del> CMAKE_MINIMUM_REQUIRED(VERSION 3.0) </del> <ins> CMAKE_MINIMUM_REQUIRED(VERSION 2.8) </ins> SET(CMAKE_MODULE_PATH ${CMAKE_CURRENT_SOURCE_DIR}/cmake ${CMAKE_MODULE_PATH}) ################################################################################

ENDFUNCTION() ################################################################################ <ins> if(${CMAKE_VERSION} VERSION_LESS "2.8.12") set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=c++11") else(${CMAKE_VERSION} VERSION_LESS "2.8.12") if(${CMAKE_VERSION} VERSION_LESS "3.1") add_compile_options(-std=c++11) # CMake 2.8.12 to 3.1 endif(${CMAKE_VERSION} VERSION_LESS "3.1") endif(${CMAKE_VERSION} VERSION_LESS "2.8.12") </ins> IF(NOT Torch_FOUND) FIND_PACKAGE(Torch REQUIRED)

INCLUDE_DIRECTORIES(${CMAKE_CURRENT_SOURCE_DIR}) ADD_LIBRARY(THPP SHARED ${all_cpp}) <del> SET_PROPERTY(TARGET THPP PROPERTY CXX_STANDARD 11) </del> SET_TARGET_PROPERTIES(THPP PROPERTIES VERSION 1 SOVERSION 1) <ins> if(NOT ${CMAKE_VERSION} VERSION_LESS "3.1") SET_PROPERTY(TARGET THPP PROPERTY CXX_STANDARD 11) endif(NOT ${CMAKE_VERSION} VERSION_LESS "3.1") </ins> TARGET_LINK_LIBRARIES(THPP PRIVATE ${TH_LIBRARIES}) INSTALL(TARGETS THPP
Change:
Gradient changes, though inputs, outputs, and internal state stay same. Even when I run forward in evaluation mode and backward in training I still get the gradient changed. It works fine with cudnn disabled.
cudnn does not support backprop in evaluate mode, we should add a fallback to nn for this case
Hi, thanks for the report. It turns out that the cudnn library (at least up to v5) doesn't support backward in evaluation mode. We should assert that in the code.
Hi, thanks for the responses. What about another case when you run forward in evaluation mode and backward in training? It produces same issue, but seems to work fine in lua torch. Here is the code with changes: If there is another way to implement virtual batch normalization - please share.
this should be fixed in master. It will be incorporated into the next release that will have binaries late today.
Out of curiosity, why do backward in evaluate mode? (Or rather, what is virtual batch normalization?)
a lot of research involves this. For example neural style transfer, or etc.

Discussion:
if (len(self.needs_input_grad) > 1 and self.needs_input_grad[2]) or self.use_cudnn: grad_bias = bias.new(bias.size()).zero_() <del> if self.use_cudnn: </del> <ins> if self.use_cudnn and self.training: # cudnn does not support backward in evaluate mode </ins> torch._C._cudnn_batch_norm_backward( input, grad_output, grad_input, grad_weight, grad_bias, weight,
Change:
TensorDataset adds casts 1D target tensor to 2D which is not acceptable by NLL loss layer.
Fixed in .

Discussion:
l = torch.randn(15) source = TensorDataset(t, l) for i in range(15): <del> self.assertEqual(t[i:i + 1], source[i][0]) self.assertEqual(l[i:i + 1], source[i][1]) </del> <ins> self.assertEqual(t[i], source[i][0]) self.assertEqual(l[i], source[i][1]) </ins> class ErrorDataset(Dataset):

for i, (sample, target) in enumerate(loader): idx = i * batch_size self.assertEqual(sample, self.data[idx:idx + batch_size]) <del> self.assertEqual(target, self.labels[idx:idx + batch_size].view(-1, 1)) </del> <ins> self.assertEqual(target, self.labels[idx:idx + batch_size]) </ins> self.assertEqual(i, math.floor((len(self.dataset) - 1) / batch_size)) def _test_shuffle(self, loader):

self.assertFalse(found_data[data_point_idx]) found_data[data_point_idx] += 1 break <del> self.assertEqual(target, self.labels.narrow(0, data_point_idx, 1)) </del> <ins> self.assertEqual(target, self.labels[data_point_idx]) </ins> found_labels[data_point_idx] += 1 self.assertEqual(sum(found_data.values()), (i + 1) * batch_size) self.assertEqual(sum(found_labels.values()), (i + 1) * batch_size)

assert data_tensor.size(0) == target_tensor.size(0) self.data_tensor = data_tensor self.target_tensor = target_tensor <del> if self.data_tensor.dim() == 1: self.data_tensor = self.data_tensor.view(-1, 1) if self.target_tensor.dim() == 1: self.target_tensor = self.target_tensor.view(-1, 1) </del> def __getitem__(self, index): return self.data_tensor[index], self.target_tensor[index]
Change:
Caffe and TensorFlow supports both CPUONLY and CUDA mode. The meaning of CPUONLY mode is to enable users to run pytorch on architectures except for AMD64 and PPC64EL (Seems that CUDA is only available on AMD64 and PPC64EL). For example, the CPU_ONLY version of is available for (not only) amd64, armel, i386, ppc64el and s390x, while the CUDA version only supports amd64 and ppc64el.
I think we don't plan to invest too much time in supporting many different architectures. The vast majority of the users is on x86 and this will remain our main focus. If someone is willing to spend their time supporting other archs, we'll happily accept PRs, but we have a lot of other priorities. We also support CPU only and CUDA mode. You can use or build pytorch without any GPU. We also have CPU-only binaries (see the website). You can force building without CUDA with env variable, as I pointed out in another PR (it's broken because of THPP at the moment, but I'll fix it soon). The only limitation for supporting different archs that I see is what our backends are limited to, and the fact that you need a C++11 compiler to build the extension. Apart from this, I don't think we depend on x86 anywhere. Can you please elaborate on what kind of guidance would you like to see? More detailed instructions on how to build pytorch somewhere in the docs?
As reported in [1], Pytorch's backend library (libTH) is now healthy in portability. I suggest that the environment variable should be referred in . [1]

Discussion:
- [NVIDIA CUDA](https://developer.nvidia.com/cuda-downloads) 7.5 or above - [NVIDIA CuDNN](https://developer.nvidia.com/cudnn) v5.x <ins> If you want to disable CUDA support, export environment variable `NO_CUDA=1`. </ins> #### Install optional dependencies On Linux
Change:
Hi, I'm trying to create a module that, given an n-dimensional returns an n+2 dimensional arranged correspondingly from slices of a 3-dimensional . In theory this can be accomplished with plus a reshape, except that I need the matrix corresponding to the to be the identity matrix instead of a zero matrix. This could be accomplished by allowing the user to specify what default vector is returned corresponding to . Would that be possible? Thanks, Shawn
this is certainly possible to implement, we have to change our C backends slightly to accommodate this. will put it on a list of things to do
Closing this because it is possible to specify a custom value to use for the vector at . An example of doing so is in the .

Discussion:
), dict( module_name='Conv3d', <ins> constructor_args=(3, 4, (2, 3, 4), 1, 0, 1, 1, False), input_size=(2, 3, 3, 4, 5), cudnn=True, desc='no_bias' ), dict( module_name='Conv3d', </ins> constructor_args=(3, 4, 2, 2), input_size=(2, 3, 5, 5, 5), cudnn=True,
Change:
The problem is here: You can't write an arbitrary number of bytes. See . On my system the limit seems to be 2GB, YMMV. To be safe, you probably want to fix the read call as well at , because there's an SSIZE_MAX limit.

Discussion:
SYSCHECK(write(fd, &self->size, sizeof(long))); // fast track for bytes and little endian if (sizeof(real) == 1 || THP_nativeByteOrder() == THPByteOrder::THP_LITTLE_ENDIAN) { <del> SYSCHECK(write(fd, data, sizeof(real) * self->size)); </del> <ins> char *bytes = (char *) data; uint64_t remaining = sizeof(real) * self->size; while (remaining > 0) { ssize_t result = write(fd, bytes, remaining); if (result < 0) throw std::system_error(result, std::system_category()); bytes += result; remaining -= result; } </ins> } else { long buffer_size = std::min(self->size, (long)5000); std::unique_ptr<uint8_t[]> le_buffer(new uint8_t[buffer_size * sizeof(real)]); <del> for (long i = 0; i < self->size; i += buffer_size) { </del> <ins> for (int64_t i = 0; i < self->size; i += buffer_size) { </ins> size_t to_convert = std::min(self->size - i, buffer_size); if (sizeof(real) == 2) { THP_encodeInt16Buffer((uint8_t*)le_buffer.get(),

THPByteOrder::THP_LITTLE_ENDIAN, to_convert); } <del> SYSCHECK(write(fd, data, to_convert * sizeof(real))); </del> <ins> SYSCHECK(write(fd, le_buffer.get(), to_convert * sizeof(real))); </ins> } } }

// fast track for bytes and little endian if (sizeof(real) == 1 || THP_nativeByteOrder() == THPByteOrder::THP_LITTLE_ENDIAN) { <del> SYSCHECK(read(fd, data, sizeof(real) * storage->size)); </del> <ins> char *bytes = (char *) data; uint64_t remaining = sizeof(real) * storage->size; while (remaining > 0) { ssize_t result = read(fd, bytes, remaining); if (result < 0) throw std::system_error(result, std::system_category()); bytes += result; remaining -= result; } </ins> } else { long buffer_size = std::min(size, (long)5000); std::unique_ptr<uint8_t[]> le_buffer(new uint8_t[buffer_size * sizeof(real)]); <del> for (long i = 0; i < size; i += buffer_size) { </del> <ins> for (int64_t i = 0; i < size; i += buffer_size) { </ins> size_t to_convert = std::min(size - i, buffer_size); SYSCHECK(read(fd, le_buffer.get(), sizeof(real) * to_convert)); if (sizeof(real) == 2) {
Change:
Not sure what is the reason for these errors, any suggestions? I suspect the clang version is not supported? Here is the
clang have more strict checking , pull request will fixed it.
fixed, thanks to

Discussion:
#include "torch/csrc/autograd/engine.h" #include <set> <ins> #include <string> </ins> #include <THPP/THPP.h> using thpp::Tensor;
Change:
As shown in below. Is this something expected? This seems most closely related to , but it is closed without any specific solutions.
Importing after also causes segv:
Happens with sklearn as well. . Reversing the order of imports seems to fix things for me though.
Can you reinstall sklearn or scipy? I think I've seen a case where that helped.
I've tried reinstalling things. Doesn't fix it. Could it be because scikit seems to build using openblas and not MKL?
I don't think so, it's a problem with numpy initialization. AFAIK the same was happening with TF, not sure if it's solved now.
same. run import scipy/ import sklearn after torch will cause error. but it's all right if load these two first.

Discussion:
# automatically filled by the dynamic loader. import os as _dl_flags <ins> # if we have numpy, it *must* be imported before the call to setdlopenflags() # or there is risk that later c modules will segfault when importing numpy try: import numpy as np except: pass </ins> # first check if the os package has the required flags if not hasattr(_dl_flags, 'RTLD_GLOBAL') or not hasattr(_dl_flags, 'RTLD_NOW'): try:
Change:
The of nn.functional.crossentropy seems wrong. The parameter weight should be a tensor, rather than a Variable.

Discussion:
Args: input: Variable :math:`(N, C)` where `C = number of classes` target: Variable :math:`(N)` where each value is `0 <= targets[i] <= C-1` <del> weight (Variable, optional): a manual rescaling weight given to each class. If given, has to be a Variable of size "nclasses" </del> <ins> weight (Tensor, optional): a manual rescaling weight given to each class. If given, has to be a Tensor of size "nclasses" </ins> size_average (bool, optional): By default, the losses are averaged over observations for each minibatch. However, if the field sizeAverage is set to False, the losses are instead summed