jiebi/CodeConvo huggingface dataset:
Code change:
This is how I implement the decoder of a sequence to sequence model I'm not sure about the new autograd mechanics but this worked in the previous version. If I didn't make the unrolling codes a function it will work. It will also work with CPU. I compiled from source with Python 2.7, CUDA 8.0 and Cudnn 6
Thanks for the minimal reproduction! It broke OpenNMT and a big internal model but we weren't looking forward to isolating what parts of those had the issue.
That's great, I'll take a look today. Thanks!
I've pushed a fix to .
How does the inference work in this case? I mean, once you train it how do you test it in the test phase?

Related discussion:
text: -DCMAKE_CXX_FLAGS="$C_FLAGS $CPP_FLAGS" make install cp "lib/libnccl.so.1" "${INSTALL_DIR}/lib/libnccl.so.1" <del> ln -s "${INSTALL_DIR}/lib/libnccl.so.1" "${INSTALL_DIR}/lib/libnccl.so" </del> <ins> if [ ! -f "${INSTALL_DIR}/lib/libnccl.so" ]; then ln -s "${INSTALL_DIR}/lib/libnccl.so.1" "${INSTALL_DIR}/lib/libnccl.so" fi </ins> cd ../.. }
Code change:
We should check that the version from cudnn.h matches the version from the loaded shared library. Really weird things happen if they don't match.

Related discussion:
text: if arg['name'] in ['self', 'state', 'dataType', 'handle']: arg['ignore_check'] = True declaration['options'] = self.filter_unique_options(declaration['options']) <del> return declarations </del> <ins> return [d for d in declarations if not d.get('only_register', False)] </ins> def filter_unique_options(self, options): def signature(option):
Code change:
We should check that the version from cudnn.h matches the version from the loaded shared library. Really weird things happen if they don't match.

Related discussion:
text: if hasattr(lib, 'cudnnGetErrorString'): lib.cudnnGetErrorString.restype = ctypes.c_char_p __cudnn_version = lib.cudnnGetVersion() <ins> compile_version = torch._C._cudnn_version() # Check that cuDNN major and minor versions match if (__cudnn_version // 100) != (compile_version // 100): raise RuntimeError( 'cuDNN version mismatch: PyTorch was compiled against {} ' 'but linked against {}'.format(compile_version, __cudnn_version)) </ins> else: lib = None return lib
Code change:
We should check that the version from cudnn.h matches the version from the loaded shared library. Really weird things happen if they don't match.

Related discussion:
text: - THTensor* output - std::vector<int> pad - std::vector<int> stride <del> - std::vector<int> dilation </del> <ins> - std::vector<int> dilation </ins> - int groups - bool benchmark ]]
Code change:
We should check that the version from cudnn.h matches the version from the loaded shared library. Really weird things happen if they don't match.

Related discussion:
text: - bool training - double epsilon ]] <ins> [[ name: cudnn_version only_register: True ]] static PyObject * THCUDNN_cudnn_version(PyObject *self, PyObject *args) { return PyLong_FromLong(CUDNN_VERSION); } </ins>
Code change:
Hi I think that in lines 28 and 30: if self.transA: a = a.transpose(2, 3) if self.transB: b = b.transpose(2, 3) should be: if self.transA: a = a.transpose(1, 2) if self.transB: b = b.transpose(1, 2) Indeed in that branch the tensor has 3 dimension and thus the code crashes. Maybe an indexing error when translating from lua ? Thank you very much,
Yes, you're right. Could you send a PR with a fix? Thanks!
Yes of course, just done, thank you very much.

Related discussion:
text: torch.mm(a, b, out=self.output) else: if self.transA: <del> a = a.transpose(2, 3) </del> <ins> a = a.transpose(1, 2) </ins> if self.transB: <del> b = b.transpose(2, 3) </del> <ins> b = b.transpose(1, 2) </ins> self.output.resize_(a.size(0), a.size(1), b.size(2)) torch.bmm(a, b, out=self.output)
Code change:
as shown in source code, this parameters is not used. Any reason that we keep it?
this can be cleaned up.

Related discussion:
text: """ return self._apply(lambda t: t.cuda(device_id)) <del> def cpu(self, device_id=None): </del> <ins> def cpu(self): </ins> """Moves all model parameters and buffers to the CPU.""" return self._apply(lambda t: t.cpu())
Code change:
as shown in source code, this parameters is not used. Any reason that we keep it?
this can be cleaned up.

Related discussion:
text: optimizer (Optimizer): Wrapped optimizer. step_size (int): Period of learning rate decay. gamma (float): Multiplicative factor of learning rate decay. <del> Default: -0.1. </del> <ins> Default: 0.1. </ins> last_epoch (int): The index of last epoch. Default: -1. Example:
Code change:
as shown in source code, this parameters is not used. Any reason that we keep it?
this can be cleaned up.

Related discussion:
text: optimizer (Optimizer): Wrapped optimizer. milestones (list): List of epoch indices. Must be increasing. gamma (float): Multiplicative factor of learning rate decay. <del> Default: -0.1. </del> <ins> Default: 0.1. </ins> last_epoch (int): The index of last epoch. Default: -1. Example:
Code change:
Consider following code, and run it with multi gpus (e.g. 4): It will output: That is, the default device would always be zero even in DataParallel. My pytorch version is 0.1.122. I think this is not the desired behavior. It would cause some troubles. I tried to insert my own cuda kernel into backward to calculate the gradients, it become very slow, and I fixed it by (grad.getdevice()). Anyway, I think currentdevice in forward and backward should be the same, or could anyone explain to me why they are different?
this seems like something we should fix: ?
<!-- drci-comment-start --:pagefacingup: Preview :pagefacingup: Preview :question: Need help or want to give feedback on the CI? Visit our Note: Links to docs will display an error until the docs builds have been completed. This comment was automatically generated by Dr. CI and updates every 15 minutes. <!-- drci-comment-end --

Related discussion:
text: for i in range(10): Variable(torch.randn(10, 10), _grad_fn=CollectOnDelete()) <del> @unittest.skipIf(not torch.cuda.is_available() or torch.cuda.device_count() < 2, "CUDA not available or <2 GPUs detected") </del> <ins> @unittest.skipIf(torch.cuda.device_count() < 2, "no multi-GPU") </ins> def test_unused_output_gpu(self): from torch.nn.parallel._functions import Broadcast x = Variable(torch.randn(5, 5).float().cuda(), requires_grad=True)
Code change:
Consider following code, and run it with multi gpus (e.g. 4): It will output: That is, the default device would always be zero even in DataParallel. My pytorch version is 0.1.122. I think this is not the desired behavior. It would cause some troubles. I tried to insert my own cuda kernel into backward to calculate the gradients, it become very slow, and I fixed it by (grad.getdevice()). Anyway, I think currentdevice in forward and backward should be the same, or could anyone explain to me why they are different?
this seems like something we should fix: ?
<!-- drci-comment-start --:pagefacingup: Preview :pagefacingup: Preview :question: Need help or want to give feedback on the CI? Visit our Note: Links to docs will display an error until the docs builds have been completed. This comment was automatically generated by Dr. CI and updates every 15 minutes. <!-- drci-comment-end --

Related discussion:
text: y.sum().backward() self.assertEqual(x.grad.data, torch.ones(5, 5) * 2) <ins> @unittest.skipIf(torch.cuda.device_count() < 2, "no multi-GPU") def test_backward_device(self): # check that current device matches the variable's device device = [None] class Identity(torch.autograd.Function): @staticmethod def forward(ctx, x): return x.clone() @staticmethod def backward(ctx, grad_output): device[0] = torch.cuda.current_device() return grad_output.clone() v = Variable(torch.randn(1).cuda(1), requires_grad=True) Identity.apply(v).backward() self.assertEqual(device[0], 1) </ins> def test_detach(self): x = Variable(torch.randn(10, 10), requires_grad=True) y = x + 2
Code change:
Consider following code, and run it with multi gpus (e.g. 4): It will output: That is, the default device would always be zero even in DataParallel. My pytorch version is 0.1.122. I think this is not the desired behavior. It would cause some troubles. I tried to insert my own cuda kernel into backward to calculate the gradients, it become very slow, and I fixed it by (grad.getdevice()). Anyway, I think currentdevice in forward and backward should be the same, or could anyone explain to me why they are different?
this seems like something we should fix: ?
<!-- drci-comment-start --:pagefacingup: Preview :pagefacingup: Preview :question: Need help or want to give feedback on the CI? Visit our Note: Links to docs will display an error until the docs builds have been completed. This comment was automatically generated by Dr. CI and updates every 15 minutes. <!-- drci-comment-end --

Related discussion:
text: #include "torch/csrc/autograd/engine.h" #include "torch/csrc/autograd/functions/basic_ops.h" <ins> #include "torch/csrc/utils/auto_gpu.h" </ins> #include <atomic> #include <condition_variable>
Code change:
Consider following code, and run it with multi gpus (e.g. 4): It will output: That is, the default device would always be zero even in DataParallel. My pytorch version is 0.1.122. I think this is not the desired behavior. It would cause some troubles. I tried to insert my own cuda kernel into backward to calculate the gradients, it become very slow, and I fixed it by (grad.getdevice()). Anyway, I think currentdevice in forward and backward should be the same, or could anyone explain to me why they are different?
this seems like something we should fix: ?
<!-- drci-comment-start --:pagefacingup: Preview :pagefacingup: Preview :question: Need help or want to give feedback on the CI? Visit our Note: Links to docs will display an error until the docs builds have been completed. This comment was automatically generated by Dr. CI and updates every 15 minutes. <!-- drci-comment-end --

Related discussion:
text: // This Engine's ReadyQueues and their corresponding threads are leaked here Engine::~Engine() = default; <del> auto Engine::thread_main(std::shared_ptr<ReadyQueue> queue) -> void { </del> <ins> auto Engine::thread_main(std::shared_ptr<ReadyQueue> queue, int device) -> void { </ins> THInferNumThreads(); <ins> AutoGPU guard(device); </ins> while (1) { FunctionTask task = queue->pop_back(); if (!task.base->has_error.load()) {
Code change:
Consider following code, and run it with multi gpus (e.g. 4): It will output: That is, the default device would always be zero even in DataParallel. My pytorch version is 0.1.122. I think this is not the desired behavior. It would cause some troubles. I tried to insert my own cuda kernel into backward to calculate the gradients, it become very slow, and I fixed it by (grad.getdevice()). Anyway, I think currentdevice in forward and backward should be the same, or could anyone explain to me why they are different?
this seems like something we should fix: ?
<!-- drci-comment-start --:pagefacingup: Preview :pagefacingup: Preview :question: Need help or want to give feedback on the CI? Visit our Note: Links to docs will display an error until the docs builds have been completed. This comment was automatically generated by Dr. CI and updates every 15 minutes. <!-- drci-comment-end --

Related discussion:
text: num_devices = 0; } #endif <del> ready_queues = std::vector<std::shared_ptr<ReadyQueue>>(num_devices + 1); for (auto& queue : ready_queues) { </del> <ins> int num_threads = num_devices + 1; ready_queues = std::vector<std::shared_ptr<ReadyQueue>>(num_threads); for (int i = 0; i < num_threads; ++i) { auto& queue = ready_queues[i]; </ins> queue.reset(new ReadyQueue()); <del> std::thread t(&Engine::thread_main, this, queue); </del> <ins> std::thread t(&Engine::thread_main, this, queue, i - 1); </ins> t.detach(); } }
Code change:
Consider following code, and run it with multi gpus (e.g. 4): It will output: That is, the default device would always be zero even in DataParallel. My pytorch version is 0.1.122. I think this is not the desired behavior. It would cause some troubles. I tried to insert my own cuda kernel into backward to calculate the gradients, it become very slow, and I fixed it by (grad.getdevice()). Anyway, I think currentdevice in forward and backward should be the same, or could anyone explain to me why they are different?
this seems like something we should fix: ?
<!-- drci-comment-start --:pagefacingup: Preview :pagefacingup: Preview :question: Need help or want to give feedback on the CI? Visit our Note: Links to docs will display an error until the docs builds have been completed. This comment was automatically generated by Dr. CI and updates every 15 minutes. <!-- drci-comment-end --

Related discussion:
text: void evaluate_function(FunctionTask& task); ReadyQueue& ready_queue(int device); void start_threads(); <del> virtual void thread_main(std::shared_ptr<ReadyQueue> queue); </del> <ins> virtual void thread_main(std::shared_ptr<ReadyQueue> queue, int device); </ins> virtual void thread_on_exception(FunctionTask& task, std::exception& e); std::once_flag start_threads_flag;
Code change:
Consider following code, and run it with multi gpus (e.g. 4): It will output: That is, the default device would always be zero even in DataParallel. My pytorch version is 0.1.122. I think this is not the desired behavior. It would cause some troubles. I tried to insert my own cuda kernel into backward to calculate the gradients, it become very slow, and I fixed it by (grad.getdevice()). Anyway, I think currentdevice in forward and backward should be the same, or could anyone explain to me why they are different?
this seems like something we should fix: ?
<!-- drci-comment-start --:pagefacingup: Preview :pagefacingup: Preview :question: Need help or want to give feedback on the CI? Visit our Note: Links to docs will display an error until the docs builds have been completed. This comment was automatically generated by Dr. CI and updates every 15 minutes. <!-- drci-comment-end --

Related discussion:
text: }; struct PythonEngine : public Engine { <del> virtual void thread_main(std::shared_ptr<ReadyQueue> queue) override { </del> <ins> virtual void thread_main(std::shared_ptr<ReadyQueue> queue, int device) override { </ins> // Create a PyThreadState, but release the GIL. This lets AutoGIL calls // inside thread_main acquire the GIL without having to create a new // PyThreadState each time. AutoGIL gil; AutoNoGIL no_gil; <del> Engine::thread_main(queue); </del> <ins> Engine::thread_main(queue, device); </ins> } virtual void thread_on_exception(FunctionTask& task, std::exception& e) override {
Code change:
In we define a number of macros which are subsequently used in if statements: This is undefined. On clang 3.4.2, these warnings don't seem to actually get printed out () but on clang 4.0.0 they seem to always get emitted.

Related discussion:
text: #define IS_CUDA false #define CUDA_FLOAT false #else #define IS_CUDA true <del> #define CUDA_BYTE defined(THC_REAL_IS_BYTE) #define CUDA_CHAR defined(THC_REAL_IS_CHAR) #define CUDA_SHORT defined(THC_REAL_IS_SHORT) #define CUDA_INT defined(THC_REAL_IS_INT) #define CUDA_LONG defined(THC_REAL_IS_LONG) #define CUDA_FLOAT defined(THC_REAL_IS_FLOAT) #define CUDA_DOUBLE defined(THC_REAL_IS_DOUBLE) #define CUDA_HALF defined(THC_REAL_IS_HALF) </del> <ins> #if defined(THC_REAL_IS_BYTE) #define CUDA_BYTE 1 #else #define CUDA_BYTE 0 </ins> #endif <ins> #if defined(THC_REAL_IS_CHAR) #define CUDA_CHAR 1 #else #define CUDA_CHAR 0 #endif #if defined(THC_REAL_IS_SHORT) #define CUDA_SHORT 1 #else #define CUDA_SHORT 0 #endif #if defined(THC_REAL_IS_INT) #define CUDA_INT 1 #else #define CUDA_INT 0 #endif #if defined(THC_REAL_IS_LONG) #define CUDA_LONG 1 #else #define CUDA_LONG 0 #endif #if defined(THC_REAL_IS_FLOAT) #define CUDA_FLOAT 1 #else #define CUDA_FLOAT 0 #endif #if defined(THC_REAL_IS_DOUBLE) #define CUDA_DOUBLE 1 #else #define CUDA_DOUBLE 0 #endif #if defined(THC_REAL_IS_HALF) #define CUDA_HALF 1 #else #define CUDA_HALF 0 #endif #endif // ifndef THC_GENERIC_FILE </ins> #if IS_CUDA #define THIndexTensor THCudaLongTensor #define THIndexTensor_(NAME) TH_CONCAT_2(THCudaLongTensor_,NAME)
Code change:
There is a possible memory leak in () This causes memory to blow up: When using () memory usage is ok There is no leak for unitary dimensions

Related discussion:
text: *tempValues__data = *t_data; *tempIndices__data = *tempIndices__dimOffset; }); <ins> THTensor_(free)(tempValues_); THLongTensor_free(tempIndices_); </ins> } if (!keepdim) {
Code change:
A = (5, 4) B = (0, 9).view(3, 3) C = (0, 15).view(3, 5) idxs = torch.LongTensor([0, 2, 4]) A.indexadd(0, idxs, B) # RuntimeError: inconsistent tensor size, expected r [4], t [4] and src [3] to have the same number of elements, but got 4, 4 and 3 elements respectively at (...)/aten/src/TH/generic/THTensorMath.c:1008 A.indexadd(0, idxs, C) # RuntimeError: inconsistent tensor size, expected r [4], t [4] and src [5] to have the same number of elements, but got 4, 4 and 5 elements respectively at (...)/aten/src/TH/generic/THTensorMath.c:1008 So far so good. But if we use CUDA... A = (5, 4).cuda() B = (0, 9).view(3, 3).cuda() C = (0, 15).view(3, 5).cuda() idxs = torch.LongTensor([0, 2, 4]).cuda() A.indexadd(0, idxs, B) print(A) # 0 1 2 0 # 0 0 0 0 # 3 4 5 0 # 0 0 0 0 # 6 7 8 0 # [ of size 5x4 (GPU 0)] OK, this looks wrong... A.zero() A.indexadd(0, idxs, C) print(A) # 0 1 2 3 # 4 0 0 0 # 5 6 7 8 # 9 0 0 0 # 10 11 12 13 # [ of size 5x4 (GPU 0)] Now this looks definitely wrong. Increase C's dimension to something like (3, 500), and it overwrites other tensors or triggers asserts. Same thing happens with indexcopy_.
I'll take it if no one's looking at it yet.

Related discussion:
text: #define THC_GENERIC_FILE "generic/THCTensorIndex.cu" #else <ins> // Check tensor dimensions for index operations, and return the slice size. // src can be nullptr in case of indexFill: in that case it is ignored. static ptrdiff_t THCTensor_(getSliceSize)(THCState *state, THCTensor *dst, int dim, THCudaLongTensor *index, THCTensor *src) { int dstDims = THCTensor_(nDimension)(state, dst); int srcDims = (src == nullptr) ? dstDims : THCTensor_(nDimension)(state, src); THArgCheck(THCudaLongTensor_nDimension(state, index) == 1, 4, "expecting vector of indices"); THArgCheck(dim >= 0 && dim < dstDims, 2, "Indexing dim is out of bounds"); ptrdiff_t dstSliceSize = 1; for (int d = 0; d < dstDims; d++) { if (d != dim) { dstSliceSize *= dst->size[d]; } } if (src == nullptr) return dstSliceSize; THArgCheck(dim < srcDims, 3, "Indexing dim is out of bounds"); THArgCheck(THCudaLongTensor_nElement(state, index) == src->size[dim], 4, "length of src.size[dim] is not equal to length of indices"); ptrdiff_t srcSliceSize = 1; bool mismatch = false; if (dstDims != srcDims) mismatch = true; for (int d = 0; d < srcDims; d++) { if (d != dim) { srcSliceSize *= src->size[d]; if (!mismatch && dst->size[d] != src->size[d]) mismatch = true; } } THArgCheck(dstSliceSize == srcSliceSize, 2, "Source/destination tensor have different slice sizes (%ld vs %ld)", dstSliceSize, srcSliceSize); if (mismatch) { static bool warningShown = false; if (!warningShown) { warningShown = true; fprintf(stderr, "Warning: source/destination slices have same size but different " "shape for an index operation.  This behavior is deprecated.n"); } } return dstSliceSize; } </ins> void THCTensor_(indexCopy_long)(THCState *state, THCTensor *dst, int dim, THLongTensor *indices, THCTensor *src) { THCAssertSameGPU(THCTensor_(checkGPU)(state, 2, dst, src));
Code change:
A = (5, 4) B = (0, 9).view(3, 3) C = (0, 15).view(3, 5) idxs = torch.LongTensor([0, 2, 4]) A.indexadd(0, idxs, B) # RuntimeError: inconsistent tensor size, expected r [4], t [4] and src [3] to have the same number of elements, but got 4, 4 and 3 elements respectively at (...)/aten/src/TH/generic/THTensorMath.c:1008 A.indexadd(0, idxs, C) # RuntimeError: inconsistent tensor size, expected r [4], t [4] and src [5] to have the same number of elements, but got 4, 4 and 5 elements respectively at (...)/aten/src/TH/generic/THTensorMath.c:1008 So far so good. But if we use CUDA... A = (5, 4).cuda() B = (0, 9).view(3, 3).cuda() C = (0, 15).view(3, 5).cuda() idxs = torch.LongTensor([0, 2, 4]).cuda() A.indexadd(0, idxs, B) print(A) # 0 1 2 0 # 0 0 0 0 # 3 4 5 0 # 0 0 0 0 # 6 7 8 0 # [ of size 5x4 (GPU 0)] OK, this looks wrong... A.zero() A.indexadd(0, idxs, C) print(A) # 0 1 2 3 # 4 0 0 0 # 5 6 7 8 # 9 0 0 0 # 10 11 12 13 # [ of size 5x4 (GPU 0)] Now this looks definitely wrong. Increase C's dimension to something like (3, 500), and it overwrites other tensors or triggers asserts. Same thing happens with indexcopy_.
I'll take it if no one's looking at it yet.

Related discussion:
text: dims = THCudaLongTensor_nDimension(state, indices); THArgCheck(dims <= MAX_CUTORCH_DIMS, 4, CUTORCH_DIM_WARNING); <del> ptrdiff_t numIndices = THCudaLongTensor_nElement(state, indices); int srcDims = THCTensor_(nDimension)(state, src); cudaStream_t stream = THCState_getCurrentStream(state); THArgCheck(THCudaLongTensor_nDimension(state, indices) == 1, 3, "expecting vector of indices"); THArgCheck(dim < srcDims, 4, "Indexing dim is out of bounds"); THArgCheck(srcDims > 0, 2, "Source tensor is empty"); THArgCheck(numIndices == src->size[dim], 4, "length of src.size[dim] is not equal to length of indices"); int indContig = THCudaLongTensor_isContiguous(state, indices); </del> // The `src` is partitioned into two parts: // -the size of each slice we are indexing, which is the // total size of the tensor ignoring dimension `dim`; // -the number of indices we are choosing, which is the total size // of the tensor `indices`. <ins> ptrdiff_t sliceSize = THCTensor_(getSliceSize)(state, dst, dim, indices, src); </ins> ptrdiff_t srcTotalSize = THCTensor_(nElement)(state, src); int64_t dstCopyDimSize = THCTensor_(size)(state, dst, dim); <del> ptrdiff_t sliceSize = srcTotalSize / numIndices; </del> <ins> ptrdiff_t numIndices = THCudaLongTensor_nElement(state, indices); cudaStream_t stream = THCState_getCurrentStream(state); int indContig = THCudaLongTensor_isContiguous(state, indices); </ins> int mpc = THCState_getCurrentDeviceProperties(state)->multiProcessorCount;