jiebi/CodeConvo huggingface dataset:
Change:
#include "torch/csrc/autograd/engine.h" #include "torch/csrc/autograd/functions/basic_ops.h" <ins> #include "torch/csrc/utils/auto_gpu.h" </ins> #include <atomic> #include <condition_variable>

Discussion:
Consider following code, and run it with multi gpus (e.g. 4): It will output: That is, the default device would always be zero even in DataParallel. My pytorch version is 0.1.122. I think this is not the desired behavior. It would cause some troubles. I tried to insert my own cuda kernel into backward to calculate the gradients, it become very slow, and I fixed it by (grad.getdevice()). Anyway, I think currentdevice in forward and backward should be the same, or could anyone explain to me why they are different?
this seems like something we should fix: ?
<!-- drci-comment-start --:pagefacingup: Preview :pagefacingup: Preview :question: Need help or want to give feedback on the CI? Visit our Note: Links to docs will display an error until the docs builds have been completed. This comment was automatically generated by Dr. CI and updates every 15 minutes. <!-- drci-comment-end --
Change:
modules[2] = nn.Conv2d(5, 3, 2) module_list[2] = modules[2] check() <ins> self.assertEqual(module_list[1:], nn.ModuleList(modules[1:])) self.assertEqual(module_list[3:], nn.ModuleList(modules[3:])) self.assertEqual(module_list[:-1], nn.ModuleList(modules[:-1])) self.assertEqual(module_list[:-3], nn.ModuleList(modules[:-3])) self.assertEqual(module_list[::-1], nn.ModuleList(modules[::-1])) </ins> with self.assertRaises(TypeError): module_list += nn.ReLU()

Discussion:
it'd be great to pick out elements of of a sequential with array slicing:
I agree this idea!
However,
this has come up again (last in ). Shall we reconsider?
I attempted getting a to work. Is this fine? If this is fine, I can issue a PR too. Let me know what you think. I believe this modification needs to be done in the class.
cc: This issue has been open for quite sometime.
Change:
-DNO_CUDA=%NO_CUDA% ^ -DNO_NNPACK=%NO_NNPACK% ^ -Dnanopb_BUILD_GENERATOR=0 ^ <del> -DCMAKE_BUILD_TYPE=Release </del> <ins> -DCMAKE_BUILD_TYPE=%BUILD_TYPE% </ins> %MAKE_COMMAND% cd ../..

Discussion:
The current master triggers the following error messages from Visual Studio 2015 (Debug build): 2------ Build started: Project: ATen, Configuration: Debug x64 ------ 2cl : Command line error D8016: '/Ox' and '/RTC1' command-line options are incompatible ... 14LINK : fatal error LNK1104: cannot open file '....srcATenDebug' It appears that they are triggered by *.c files in the ATen project, like "srcTHvectorAVX.c"
Why do you need Debug build?
Isn't ATen meant to support both Debug and Release builds? FYI I'm involved with the project at which does support both build configurations, and we would like to be able to link our project with ATen.
Well, you can remove the flags in as a workground.
We should patch this file to ignore this flag when building the library in debug mode. It shouldn't be too hard
I catched similar errors , when I Tryed to build Debug Version I try to build Release and catched next errors e:pytorchatensrcatencpuvec256vec256int.h(184): note: see reference to function template instantiation 'at::vec256::anonymous-namespace'::Vec256<int16t::blend<1(at::vec256::anonymous-namespace'::Vec256<int16t)' being compiled 28e:pytorchatensrcatencpuvec256vec256int.h(147): error C3861: 'mm256extract_epi16': identifier not found What I should do?
There're two issues in your description. The first question is easy. You'll also have to specify the flag to 1. The second question is about your compiler. I guess you are using VS2015 or earlier versions of VS2017. Make sure the version in the output is 19.11 after your enter in CMD.
Yes I am using VS2015. My Compiler is 19.0.2. Must I use VS 2017 in any case?
Sorry, but the answer is yes. The AVX2 functions was in VS2017 and they are used in PyTorch. So VS2017 19.11 is a must.
No problem:) Thank you.
Change:
.. autofunction:: cumprod .. autofunction:: cumsum .. autofunction:: dist <ins> .. autofunction:: logsumexp </ins> .. autofunction:: mean .. autofunction:: median .. autofunction:: mode

Discussion:
Leaving this issue here for higher visibility. Originally reported by here: Also mentioned by Piggybacking off of this issue, has strange behavior (reported here: )
I have fixed this locally. Could I send a PR?
Yes, go ahead!
Change:
{ PyObject_GC_UnTrack(self); THPFunction_clear(self); <ins> self->cdata_ptr.~weak_ptr(); </ins> self->cdata.~PyFunction(); Py_TYPE(self)->tp_free((PyObject*)self); }

Discussion:
there is a bug. the code is from The code for installing cuda10.2 should be corrected to The code for installing cuda10.2 should be corrected to cc
Thank you for reporting the typo

This is how I implement the decoder of a sequence to sequence model I'm not sure about the new autograd mechanics but this worked in the previous version. If I didn't make the unrolling codes a function it will work. It will also work with CPU. I compiled from source with Python 2.7, CUDA 8.0 and Cudnn 6
Thanks for the minimal reproduction! It broke OpenNMT and a big internal model but we weren't looking forward to isolating what parts of those had the issue.
That's great, I'll take a look today. Thanks!
I've pushed a fix to .
How does the inference work in this case? I mean, once you train it how do you test it in the test phase?
Change:
<ins> #pragma once // RAII structs to acquire and release Python's global interpreter lock (GIL) #include <Python.h> // Acquires the GIL on construction struct AutoGIL { AutoGIL() : gstate(PyGILState_Ensure()) { } ~AutoGIL() { PyGILState_Release(gstate); } PyGILState_STATE gstate; }; // Releases the GIL on construction struct AutoNoGIL { AutoNoGIL() : save(PyEval_SaveThread()) { } ~AutoNoGIL() { PyEval_RestoreThread(save); } PyThreadState* save; }; </ins>

Discussion:
Go to Look at the html. It contains the following: ! This problem manifests in google search. Searching for 1.8.1 docs brings up the "1.8.0 docs": ! We can probably fix this by doing a quick sed on the website but we should develop a solution for future releases. I think this has happened before in the past and am not sure what causes it. cc
Change:
c = nn.Conv2d(3, 3, 3) self.assertRaises(RuntimeError, lambda: c(None)) <ins> def test_Conv2d_backward_twice(self): input = Variable(torch.randn(2, 3, 5, 5)) c = nn.Conv2d(3, 3, 3) o1 = c(input) o1.sum().backward() self.assertRaisesRegex(RuntimeError, 'Specify retain_variables=True', lambda: o1.sum().backward()) </ins> @unittest.skipIf(not TEST_CUDA, 'CUDA not available') def test_Conv2d_large_workspace(self): # These sizes require huge cuDNN workspaces. Make sure we choose a

Discussion:
Usually calling backwards twice gives this message: But conv2d just segfaults. $ pt A B Segmentation fault (core dumped)
Change:
// For Convolution strategies that don't implicitly handle grad_bias, we add a helper // function here to perform it using simple Tensor operators static at::Tensor compute_grad_bias(const at::Tensor& grad_output) { <del> // grad_output is in N, C, H, W, we re-shape and reduce over spatial dims and batches </del> <ins> // grad_output is in N, C, H, W, we re-shape and reduce over spatial dims and batches </ins> return grad_output.contiguous().view({grad_output.size(0), grad_output.size(1), -1}).sum(0).sum(1); }

Discussion:
The following code: produces this error Also, removing the ReLU from the network: causes the error to change to: However, when the second convolution is removed as well: then the code runs error free I'm on a ppc64le architecture with a P100 GPU, CUDA 8.0 and PyTorch version '0.4.0a0+'
I'm also having this problem using branch v0.3.0
i could reproduce this as well. we'll look into this.
I'm having the same error :/
same error with gradient penalty for WGAN
When you execute , are you cause memory leak
no there shouldn't be a memory leak.
I am having an out of memory error as well, only when applying backward to the output of this method. I am using 3 titans GPUs so memory is not a problem I would assume. def computeGP(netD, realdata, realembed, fakedata, LAMBDA): BATCHSIZE = (0) alpha = (BATCHSIZE, 1) alpha = (BATCHSIZE, int(realdata.nelement() / BATCHSIZE)).contiguous().view(BATCHSIZE, 3, 64, 64) alpha = () interpolates = alpha realdata + ((1 - alpha) fakedata) interpolates = () interpolates = autograd.Variable(interpolates, requiresgrad=True) discinterpolates, = netD(interpolates, realembed) gradients = (outputs=discinterpolates, inputs=interpolates, (()).cuda(), creategraph=True, retaingraph=True, onlyinputs=True)[0] gradientpenalty = (((2, dim=1) - 1) * 2).mean() LAMBDA return gradient_penalty `
I am also using it with gradient penalty for WGAN, But When I use gradient_penalty.backward(), the memory leak is happen, My pytorch is 0.4.0a0+.
I can reproduce the memory leak with this: However the leak disappears when only one convolution is used: Also, if nn.Linear is used, there is no leak. Torch version: 0.4.0a0+
I've opened a separate issue for that memory leak:
Change:
#  binary_linux_libtorch_3.6m_cpu_test: #    environment: #      BUILD_ENVIRONMENT: "libtorch 3.6m cpu" <del> #    resource_class: gpu.medium </del> <ins> #    resource_class: gpu.nvidia.small </ins> #    <<: *binary_linux_test # #  binary_linux_libtorch_3.6m_cu90_test: #    environment: #      BUILD_ENVIRONMENT: "libtorch 3.6m cu90" <del> #    resource_class: gpu.medium </del> <ins> #    resource_class: gpu.nvidia.small </ins> #    <<: *binary_linux_test # docker_build_job:

Discussion:
As it still using deprecated resource class, see See example of the failure in
Followup: we need to plan better --we knew it was gonna happen and none of us fixed it early enough. Action item: go through tasks about deprecation in sprint planning
Change:
} } <del> return F::detail::batch_norm( </del> <ins> return torch::nn::functional::detail::batch_norm( </ins> input, this->running_mean, this->running_var,

Discussion:
As part of pull request , F is redefined globally in nn/modules/batchnorm.h F is then used once in batchnorm.h and again in instancenorm.h not a bug, but a problem for anyone using F in some other way. it's solvable, but I don't see this pattern anywhere else in libtorch, was wondering if this use is an anomaly, and if we could use thanks cc
Thanks a lot for reporting! Yes this is an anomaly, and will be fixed by
Change:
(Trace, (), ((S, S),)), (Cross, (), ((S, 3), (S, 3))), (Cross, (1,), ((S, 3, S), (S, 3, S)), 'dim'), <ins> (Inverse, (), ((S, S),), '', (), [skipIfNoLapack]), </ins> (Clone, (), ((S, M, S),)), (Squeeze, (), ((S, 1, M, 1),)), # TODO: enable neg dim checks

Discussion:
Command to install the latest nightly binary: installs : However, PyTorch itself reports : I guess this version tag might be defined during the build process from e.g. and then forwarded from , but also source builds should use based on the tag. Unsure, if this is just a reporting issue or if we are building the nightly binaries with and executing them with , so creating this issue to dig a bit further into it. CC cc
This is really weird, I've put up measure in place to validate against it... Will investigate today (and moving to pytorch repo)
can you please check that 2.2.0 reports (and is build) against the correct version of nccl library?
Confirmed for nightlies : Same issue for RC:
Looking at the build log Looks like we are compiling with: USESYSTEMNCCL
On Windows I see following exception: We compiling with should we return None rather then exception ?
In this case we are indeed using 2.18.1: Do you know why we are using the system-wide NCCL installation instead of the one? Should we delete the system NCCL and use by default?
Looks like its set here: Looks like it was by: Let me propose the PR to unset this flag : USESYSTEM_NCCL
I guess we are using the system NCCL installation to be able to during the runtime. If we would use the module I assume we would link NCCL into the PyTorch binaries. If so, we should make sure to update the NCCL version whenever is updated. This could break our dependency on the NCCL pip package.
Yes. Also when we update this nccl here:
We used following path  What would be good path for nccl 2.19.3 and cuda 12.1 ? As per this issue: Will be using 12.0 ncck 2.19.3
Installing system NCCL into the nightly build docker container feels great, but we need to figure out, how to keep it in sync with PyTorch. I wonder if one can, for example, just check nccl version from this requirements file and build from source?
yes probably we should build and upload nccl for our supported cuda versions, same way as we build docker containers, by following details here:
Verified with final rc
Change:
gate_size = hidden_size self._all_weights = [] <del> self._param_buf_size = 0 </del> for layer in range(num_layers): for direction in range(num_directions): layer_input_size = input_size if layer == 0 else hidden_size * num_directions

Discussion:
Since the new pytorch version, I have a warning when I that says: When I do as suggested, and then try to encode again, I get the following error though: Any idea?
do you have code the reproduces the issue?
What was the fix? I still see the error on Pytorch Version: 0.2.03 AttributeError: 'LSTM' object has no attribute 'parambufsize'
Change:
def backward(self, grad_output): tensors = self.saved_tensors input, params = tensors[0], tensors[1:] <del> grad_input = (input.new().resize_as_(input).zero_() </del> <ins> grad_input = (input.new().resize_as_(input) </ins> if self.needs_input_grad[0] else None,) grad_param = tuple(p.new().resize_as_(p).zero_() if self.needs_input_grad[i+1] else None for i, p in enumerate(params))

Discussion:
THNN convolutions are slower due to unnecessary .zero() calls on gradients: is not needed same for bias, prob worth adding a constant in C-side that controls whether to override grad or not
Same for batch_norm and auto thnn functions, not sure if all of them though. Prob the speedups will be quite significant.
I think they are needed. THNN functions accumulate the gradient instead of overwriting the buffer, don't they?
only grad wrt weight, inputs are overriden
We should reopen this, still many functions have unnecessary .zero() calls on gradinput and gradparam, e.g.: : : most of the functions don't need it
I think the ones are currently necessary.
for nn yes, but cudnn has: and on cudnnConvolutionBackwardFilter and on cudnnBatchNormalizationBackward which control addition of gradients and should be set to 1 and 0 in pytorch cudnn (it is and 1 in )
Yeah, we can remove it on BatchNorm. If you look at Conv, it already only zeros out grad_weight for cunn.
this is now fixed in master:
Change:
} void ObjectPtrAllocator::free(void* ptr) { <del> object = nullptr; </del> <ins> { AutoGIL gil; object = nullptr; } </ins> allocator->free(allocatorContext, ptr); delete this; } void StorageWeakRefAllocator::free(void* ptr) { <del> PyGILState_STATE gstate = PyGILState_Ensure(); PyObject_SetAttrString(object.get(), "cdata", Py_None); object = nullptr; PyGILState_Release(gstate); </del> <ins> { AutoGIL gil; PyObject_SetAttrString(object.get(), "cdata", Py_None); object = nullptr; } </ins> allocator->free(allocatorContext, ptr); delete this; }

Discussion:
When both PyTorch and Pandas are imported, calling results in a segfault. The issue was originally reported by It looks like calling returns a null pointer here: Examining in GDB: I suspect it's both PyTorch and Pandas doing threading magic and somehow getting in each other's way. I haven't yet been able to provide a minimal example that triggers this behaviour and does not use my package; it's likely that I'm doing something odd that causes this. More documentation here: Possibly related issues: and Python version: 3.6 PyTorch version: v0.2.0 Pandas version: 0.20.1
Change:
global Storage Tensor = _import_dotted_name(t) Storage = _import_dotted_name(t.replace('Tensor', 'Storage')) <ins> if 'cuda' in t: import torch.cuda torch.cuda.init() </ins> _C._set_default_tensor_type(Tensor)

Discussion:
When I was working on Vision and Audio CI, I often find that the dowloaded nightly PyTorch is CPU version instead of expected CUDA version. It will break the domain library CIs. It's often caused by nightly build failed or that day's nightly build isn't finished. There are 2 open issues for it. If this day's nightly PyTorch cuda build isn't available, I think it's better to provide last day's nightly builds rather than CPU version. cc
It looks that all nightly builds would be finished before 4 AM at PST, so vision CI failure probability might be higher in the morning of France.

Hi, I've updated pytorch to the lastest master () from github, but got an error when I use : What to do for workaround? Thanks! Jin
Here's a workaround. I think the problem is that pytorch hasn't initialized CUDA yet. At any rate, the following works for me:
Thanks but your workaround doesn't work for me. The error message still shows. Jin
Sorry, I meant the following: Could you please try if that works for you?
Yep. Just what I thought and was going to add comment here :smile: It works. Thank you! Jin
Change:
<del> CMAKE_MINIMUM_REQUIRED(VERSION 3.0) </del> <ins> CMAKE_MINIMUM_REQUIRED(VERSION 2.8) </ins> SET(CMAKE_MODULE_PATH ${CMAKE_CURRENT_SOURCE_DIR}/cmake ${CMAKE_MODULE_PATH}) ################################################################################

Discussion:
Currently, CMake 3.0 or higher is required for THPP. This is annoying for building from source on Ubuntu 14.04 systems, as default CMake is 2.8, and updating it requires either compiling from source or adding some random repos:
it should work fine if you change it to 2.8. I'm testing this out in a sec, and will push it into master
Nope, it does not for me.
Change:
} } <del> PyObject * THSPTensor_(NewEmpty)() { auto r = THSTensor_(new)(LIBRARY_STATE_NOARGS); THSPTensor_(initStorage)(r); return THSPTensor_(New)(r); } </del> PyObject * THSPTensor_(New)(THSTensor *tensor) { THSTensorPtr ptr(tensor);

Discussion:
This causes to not work properly on newly created empty sparse tensors. In the example below, both should print 1 instead.
Change:
if hasattr(lib, 'cudnnGetErrorString'): lib.cudnnGetErrorString.restype = ctypes.c_char_p __cudnn_version = lib.cudnnGetVersion() <ins> compile_version = torch._C._cudnn_version() # Check that cuDNN major and minor versions match if (__cudnn_version // 100) != (compile_version // 100): raise RuntimeError( 'cuDNN version mismatch: PyTorch was compiled against {} ' 'but linked against {}'.format(compile_version, __cudnn_version)) </ins> else: lib = None return lib

Discussion:
We should check that the version from cudnn.h matches the version from the loaded shared library. Really weird things happen if they don't match.
Change:
THPUtils_setError(error.c_str()); } <del> #if defined(NUMPY_TYPE_ENUM) || defined(THC_GENERIC_FILE) </del> <ins> #if defined(NUMPY_TYPE_ENUM) || (defined(WITH_NUMPY) && defined(THC_GENERIC_FILE)) </ins> #ifndef THC_REAL_IS_HALF #define load_real real

Discussion:
When I don't have a working numpy dependency detected (due to unrelated shenanigans), the build fails with this: And then a big pile of related errors. CC
This is most likely the problem:
CC
Ok, should probably read
Should I go ahead and make the change? I'll also remove the at line 193. The rest looks ok to me.
go for it.
Actually, the change I proposed is incorrect (I misread the second defined), but I'm on it.
Change:
SparseTensor = getattr(torch.cuda.sparse, tensor_type) else: SparseTensor = getattr(torch.sparse, tensor_type) <ins> padding_idx = ctx.padding_idx indices = indices.view(1, -1) grad_output = grad_output.view(-1, ctx._weight_size[1]) if padding_idx is not None: nonpadding_indices_indices = (indices.view(-1) != padding_idx).nonzero().view(-1) indices = indices.index_select(1, nonpadding_indices_indices) grad_output = grad_output.index_select(0, nonpadding_indices_indices) </ins> grad_weight = SparseTensor( <del> indices.view(1, -1), grad_output.view(-1, ctx._weight_size[1]), </del> <ins> indices, grad_output, </ins> ctx._weight_size, ) return None, grad_weight, None, None, None, None, None

Discussion:
As reported here with repro. When sparse=True, paddingidx should probably error out (as the easiest solution). Alternatively, the gradients for paddingidx indices should be zeroed out explicitly in the sparse=True backward codepath.
Change:
ensure_attr('ClassNLLCriterion', 'weights') ensure_attr('ParallelCriterion', 'repeatTarget') ensure_attr('MultiMarginCriterion', 'weights') <del> ensure_attr('SpatialConvolution', 'bias', 'finput', 'fgradInput', 'gradWeight', 'gradBias') </del> <ins> ensure_attr('SpatialConvolution', 'bias', 'finput', 'fgradInput', 'gradWeight', 'gradBias', '_gradOutput') </ins> attr_map('ReLU', {'val': 'value'}) attr_map('Threshold', {'val': 'value'}) attr_map('Unsqueeze', {'pos': 'dim'})

Discussion:
On Windows, torch.slogdet reproducibly fails when computing torch.slogdet on larger tensors. Here is the error: Steps to reproduce the behavior: Expected this op to succeed on Windows. This does not fail on Linux. cc
Which GPU card are you using on Windows?
I have an RTX 3080 and an RTX 3090 and see the issue on both. If this is a CUDA issue, could you point me to the right place to report it?
I can reproduce this with PyTorch 1.7.0 + RTX 3080.
It seems that the current implementation of this function is based on MAGMA. Will try to rebuild with newer versions.
It seems that the current windows MAGMA binaries are built without cuda arch 8.0 support. I will fix it.
Awesome! Thank you so much for the quick turnaround on this. Please let me know if you want me to test a fix.
Wait for the merge of
Change:
return: self cname: resize cpu_half: True <ins> before_call: THPUtils_assert(arg_self->storage->flag & TH_STORAGE_RESIZABLE, "calling resize_ on a tensor that has non-resizable storage. Clone it first " "or create a new tensor instead."); </ins> arguments: - THTensor* self - arg: THSize* size

Discussion:
When both PyTorch and Pandas are imported, calling results in a segfault. The issue was originally reported by It looks like calling returns a null pointer here: Examining in GDB: I suspect it's both PyTorch and Pandas doing threading magic and somehow getting in each other's way. I haven't yet been able to provide a minimal example that triggers this behaviour and does not use my package; it's likely that I'm doing something odd that causes this. More documentation here: Possibly related issues: and Python version: 3.6 PyTorch version: v0.2.0 Pandas version: 0.20.1
Change:
return type(self), (self.tolist(),) def __repr__(self): <del> return str(self) </del> <ins> return repr(str(self)) </ins> def __str__(self): <del> return _tensor_str._str(self) </del> <ins> # All strings are unicode in Python 3, while we have to encode unicode # strings in Python2. If we can't, let python decide the best # characters to replace unicode characters with. if sys.version_info > (3,): return _tensor_str._str(self) else: return _tensor_str._str(self).encode( sys.stdout.encoding or 'UTF-8', 'replace') </ins> def __iter__(self): return iter(map(lambda i: self.select(0, i), _range(self.size(0))))

Discussion:
It seems that the following link that can be reached from here (check the screenshot) is broken: Page: Broken link: Screenshot: ! cc
this should be transferred to , but I don't have permissions to do that. Fix is available at pytorch/hub.