jiebi/CodeConvo huggingface dataset:
Change:
y = x[2:] self.assertEqual(int(y), 3) <del> @unittest.skipIf(torch.set_flush_denormal(False), "flush_denormal not supported") </del> <ins> # skip this test for now as it affects all tests @unittest.skipIf(True, "flush_denormal not supported") </ins> def test_set_flush_denormal(self): tiny_float = 1e-42 tiny_double = 1e-320

Discussion:
hi, I just saw this error in the last few days. both for GPU and CPU only environment. Is the test precision assertion correct ?
On PPC64 it looks like denormal flags aren't being set correctly. This PR from 2 days ago denormals support, but likely assumed only AMD64 / x86
currently the setflushdenormal call will always return False if on ppc64le. The unittest has a skipIf as below. I believe the intention is to skip this test if not running on x86. But since setflushdenormal returns a False on ppc64le , then it actually runs the test and not run on x86 "flushdenormal not supported") I believe is you just add a not as below in the torch.setflushdenormal(False), "flush_denormal not supported") the test should then run on x86 but skipped for ppc64le or other platforms.
Doing calls that have side effects in skipIf is a bad idea. We should write a context manager that always resets the previous state + skips the test if the call returned False
what can we do in the meantime to skip over the denormal test, which is invalid, on ppc64 ?
Skip this test everywhere. These checks will affect all tests and we don’t want that
closed via
Change:
THD_LIB = os.path.join(lib_path, 'libTHD.1.dylib') NCCL_LIB = os.path.join(lib_path, 'libnccl.1.dylib') <del> if WITH_NCCL and subprocess.call('ldconfig -p | grep libnccl >/dev/null', shell=True) == 0: </del> <ins> if WITH_NCCL and (subprocess.call('ldconfig -p | grep libnccl >/dev/null', shell=True) == 0 or subprocess.call('/sbin/ldconfig -p | grep libnccl >/dev/null', shell=True) == 0): </ins> SYSTEM_NCCL = True main_compile_args = ['-D_THP_CORE']

Discussion:
Hello, on Debian (and possibly Ubuntu and other systems), is in and not in non-root users' search paths. Thus this line: at does not do what it should when compiling not as root. Either writing or adding a second check with the directory would help building pytorch on these systems. Would such a change be acceptable? Thank you & best regards Thomas
Or fix this by exporting ?
such a chance would be absolutely acceptable.
fixed in PR
Change:
THCAssertSameGPU(THCTensor_(checkGPU)(state, 2, self_, src_)); THArgCheck(src_->nDimension == 2, 1, "expected a matrix"); <del> THCTensor *src = src_; if (self_ == src_) src = THCTensor_(newContiguous)(state, src_); </del> <ins> if (self_ != src_) THCTensor_(resizeAs)(state, self_, src_); </ins> <del> int64_t stride0 = src->stride[0]; int64_t stride1 = src->stride[1]; real *start = THCTensor_(data)(state, src); </del> <ins> int64_t stride0 = self_->stride[0]; int64_t stride1 = self_->stride[1]; real *start = THCTensor_(data)(state, self_); </ins> TensorTriOp<real, 1> op(start, stride0, stride1, k); if (self_ == src_) { <del> if (!THC_pointwiseApply1(state, src, op)) { </del> <ins> if (!THC_pointwiseApply1(state, src_, op)) { </ins> THArgCheck(false, 2, CUTORCH_DIM_WARNING); } } else { <del> THCTensor_(resizeAs)(state, self_, src); if (!THC_pointwiseApply2(state, self_, src, op)) { </del> <ins> if (!THC_pointwiseApply2(state, self_, src_, op)) { </ins> THArgCheck(false, 2, CUTORCH_DIM_WARNING); } } <del> if (self_ == src_) THCTensor_(freeCopyTo)(state, src, src_); </del> THCudaCheck(cudaGetLastError()); }

Discussion:
On recent PyTorch master (built from source; 0.4.0a0+) on a P100, CUDA 9:
I think the results on CPU meets the defination of triu (upper triangular part of the matrix). It is really hard to handle with a tensor from expand operation with multi-thread not only on GPU but also on CPU. The unexpected result will also happen on CPU if we apply openmp to it. I think it is necessary to eleminate the difference between the implementation and defination.
It doesn’t matter if it’s cpu or not. triu doesn’t have an underscore which means it doesn’t operate in-place and should always give the same result (in new memory)
Change:
other types will be a shallow copy and can be corrupted if written to in the model's forward pass. <ins> .. warning:: Forward and backwrad hooks defined on :attr:`module` and its submodules won't be invoked anymore, unless the hooks are initialized in the :meth:`forward` method. </ins> Args: module: module to be parallelized device_ids: CUDA devices (default: all devices)

Discussion:
PyTorch version: 0.3.0 Python version: 3.5 GPU models and configuration: 8x NVIDIA Titan I found that requiresgrad setting in module is ignored when module is wrapped with nn.DataParallel. For example, This code should be executed because very small part of network activation (only fc layer) is stored for backward computation, but it results to run-time error: Internally, I found a crack in function which is in . In replicate function, it copies all parameter in module (# of replica times) with . In broadcasting code, it just defines new with default constructor parameter, which is always set to True. I think there can be some choices to fix this issue. is intended behavior for DataParallel, so we should use volatile (or ) to implement transfer learning in DataParallel model should be fixed to handle requires_grad when copying parameters. should be synchronizing in I think any of solution doesn't need lots of effort to fix it.
Change:
if (weight_dim != k) { std::stringstream ss; <del> ss << "Expected " << weight_dim << "-dimensional input for " << weight_dim << "-dimensional weight " << weight.sizes() << ", but got input of size " << input.sizes() << " instead"; </del> <ins> ss << "Expected " << k << "-dimensional weight for " << k << "-dimensional input " << input.sizes() << ", but got weight of size " << weight.sizes() << " instead"; </ins> throw std::runtime_error(ss.str()); } if (weight.size(0) < groups) {

Discussion:
OS: Ubuntu 16.04 PyTorch version: 0.3.0.post4 How you installed PyTorch (conda, pip, source): pip Python version: 3.5.2 CUDA/cuDNN version: CUDA 8 The below script produces an incorrect error message () The below script produces an incorrect error message ()
Change:
(4.53, 3.83, -6.64, 2.06))).t() b = torch.Tensor(((8.58, 8.26, 8.48, -5.28), (9.35, -4.43, -0.70, -0.26))).t() <del> _test(a, b, expectedNorm) </del> <ins> _test_underdetermined(a, b, expectedNorm) </ins> # test overderemined expectedNorm = 17.390200628863

Discussion:
b = (600,1) A = (600,2) X, = (b,A) X.size() #torch.Size([2, 1]) Y, = (b.cuda(),A.cuda()) Y.size() #torch.Size([600, 1])
that is interesting. says that the first rows returned ( here) include the solution of the least-squares problem. I've verified that the first 2 rows of X and Y are the same so you can probably use that for now while we figure out what the rest of the output is.
Change:
print('Distributed not available, skipping tests') sys.exit(0) <ins> SKIP_IF_NO_CUDA_EXIT_CODE = 75 def skip_if_no_cuda_distributed(func): func.skip_if_no_cuda_distributed = True @wraps(func) def wrapper(*args, **kwargs): if not torch.cuda.is_available(): sys.exit(SKIP_IF_NO_CUDA_EXIT_CODE) return func(*args, **kwargs) return wrapper </ins> @contextmanager def _lock():

Discussion:
is causing a crash when we try to build pytorch. Crash looks like this: (all times in US/West) Our 7AM, 8AM builds and earlier were all fine but 9AM, 10AM, 11AM builds all crashed with that same traceback. The new import was in which I think went in between 8AM and 9AM today See command below: See here if needed for : <!-- A clear and concise description of what you expected to happen. --Please copy and paste the output from our (or fill out the checklist below manually). You can get the script and run it with: PyTorch Version (e.g., 1.0): building from master OS (e.g., Linux): linux How you installed PyTorch (, , source): source Build command you used (if compiling from source): see above under 'reproduce' Python version: 3.6
maybe easier to read for build command used:
please try in the CircleCi environment?
in our build script seems to have fixed this. Thanks !
Change:
log_scale = (self.temperature.new(self.temperature.shape).fill_(K).lgamma() - self.temperature.log().mul(-(K - 1))) score = logits - value.mul(self.temperature) <del> score = (score - log_sum_exp(score)).sum(-1) </del> <ins> score = (score - _log_sum_exp(score)).sum(-1) </ins> return score + log_scale

Discussion:
Leaving this issue here for higher visibility. Originally reported by here: Also mentioned by Piggybacking off of this issue, has strange behavior (reported here: )
I have fixed this locally. Could I send a PR?
Yes, go ahead!
Change:
toDeviceTensor<real, 1>(state, gradOutput), toDeviceTensor<real, 2>(state, gradInput), weights ? THCTensor_(data)(state, weights) : NULL, <ins> n_classes, </ins> ignore_index); <ins> THCudaCheck(cudaGetLastError()); </ins> if (weights) { THCTensor_(free)(state, weights); }

Discussion:
Claim: When I use an invalid label with F.crossentropy with reduce=False, I don't receive an error message. I do receive an error message when reduce=True. Code: Result: No error message received. Expected error message: Context: I was using , accidentally used an invalid label, and got extremely large losses and extremely small losses, and inconsistently so. Environment: OS: Ubuntu 16.04 on Docker and host PyTorch version: 0.3.0.post4 How you installed PyTorch: conda Python version: 3.6.3
Change:
if hasattr(torch, name) and name not in EXCLUDE_FUNCTIONAL: def fn(*inputs): output = getattr(torch, name)(*inputs) <del> if test_output_indices is None: return output else: return tuple(output[i] for i in test_output_indices) </del> <ins> return output_process_fn(output) </ins> f_args_variable = (self_variable,) + args_variable f_args_tensor = (self_tensor,) + args_tensor

Discussion:
I am having problem in running the backward propagation with svd. In particular, it seems working when the first dimension of the matrix x (below) is greater than the second. But I get the following error if the first is smaller than the second. PyTorch version: 2018.04.20 Is debug build: No CUDA used to build PyTorch: 8.0.61 OS: Ubuntu 16.04.4 LTS GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.9) 5.4.0 CMake version: Could not collect Python version: 3.5 Is CUDA available: No CUDA runtime version: No CUDA GPU models and configuration: No CUDA Nvidia driver version: No CUDA cuDNN version: No CUDA Versions of relevant libraries: [pip3] numpy (1.13.3) [pip3] torch (2018.4.20) [pip3] torchvision (0.2.0) [conda] pytorch 0.3.1 py35cuda8.0.61cudnn7.0.52 pytorch [conda] pytorch-nightly 2018.04.20 py35cuda8.0.61cudnn7.1.21 pytorch [conda] torchvision 0.2.0 py35heaa392f_1 pytorch
Repro'ed on 0.4 branch. works btw.
What does "Repro'ed" means? Should I install from that branch? I need to compute the kernel of a matrix, therefore I need the last columns of the matrix V (which is why I have to pass ).
It's a shorthand for "reproduced", which means that the bug still hasn't been fixed. Sorry, we'll try to get to it soon.
Change:
IF(C_AVX2_FOUND) IF(MSVC) <del> SET_SOURCE_FILES_PROPERTIES(${PROJECT_SOURCE_DIR}/src/TH/vector/AVX2.c PROPERTIES COMPILE_FLAGS "/Ox /arch:AVX2 ${C_AVX2_FLAGS}") </del> <ins> SET_SOURCE_FILES_PROPERTIES(${PROJECT_SOURCE_DIR}/src/TH/vector/AVX2.c PROPERTIES COMPILE_FLAGS "${MSVC_OPT_FLAG}/arch:AVX2 ${C_AVX2_FLAGS}") </ins> ELSE(MSVC) SET_SOURCE_FILES_PROPERTIES(${PROJECT_SOURCE_DIR}/src/TH/vector/AVX2.c PROPERTIES COMPILE_FLAGS "-O3 ${C_AVX2_FLAGS}") ENDIF(MSVC)

Discussion:
The current master triggers the following error messages from Visual Studio 2015 (Debug build): 2------ Build started: Project: ATen, Configuration: Debug x64 ------ 2cl : Command line error D8016: '/Ox' and '/RTC1' command-line options are incompatible ... 14LINK : fatal error LNK1104: cannot open file '....srcATenDebug' It appears that they are triggered by *.c files in the ATen project, like "srcTHvectorAVX.c"
Why do you need Debug build?
Isn't ATen meant to support both Debug and Release builds? FYI I'm involved with the project at which does support both build configurations, and we would like to be able to link our project with ATen.
Well, you can remove the flags in as a workground.
We should patch this file to ignore this flag when building the library in debug mode. It shouldn't be too hard
I catched similar errors , when I Tryed to build Debug Version I try to build Release and catched next errors e:pytorchatensrcatencpuvec256vec256int.h(184): note: see reference to function template instantiation 'at::vec256::anonymous-namespace'::Vec256<int16t::blend<1(at::vec256::anonymous-namespace'::Vec256<int16t)' being compiled 28e:pytorchatensrcatencpuvec256vec256int.h(147): error C3861: 'mm256extract_epi16': identifier not found What I should do?
There're two issues in your description. The first question is easy. You'll also have to specify the flag to 1. The second question is about your compiler. I guess you are using VS2015 or earlier versions of VS2017. Make sure the version in the output is 19.11 after your enter in CMD.
Yes I am using VS2015. My Compiler is 19.0.2. Must I use VS 2017 in any case?
Sorry, but the answer is yes. The AVX2 functions was in VS2017 and they are used in PyTorch. So VS2017 19.11 is a must.
No problem:) Thank you.
Change:
parameters[2] = make_param() param_list[2] = parameters[2] check() <ins> self.assertEqual(param_list[1:], nn.ParameterList(parameters[1:])) self.assertEqual(param_list[3:], nn.ParameterList(parameters[3:])) self.assertEqual(param_list[:-1], nn.ParameterList(parameters[:-1])) self.assertEqual(param_list[:-3], nn.ParameterList(parameters[:-3])) self.assertEqual(param_list[::-1], nn.ParameterList(parameters[::-1])) </ins> with self.assertRaises(TypeError): param_list += make_param()

Discussion:
it'd be great to pick out elements of of a sequential with array slicing:
I agree this idea!
However,
this has come up again (last in ). Shall we reconsider?
I attempted getting a to work. Is this fine? If this is fine, I can issue a PR too. Let me know what you think. I believe this modification needs to be done in the class.
cc: This issue has been open for quite sometime.
Change:
.. autofunction:: linear <ins> :hidden:`bilinear` ~~~~~~~~~~~~~~~~ .. autofunction:: bilinear </ins> Dropout functions -----------------

Discussion:
is not available in the docs page, but the doc string has been in . This was mentioned earlier in

Oh, bilinear, I missed that last time I looked at it. I'll add a docstring.

returns nothing. Might be useful to have some linter in the build process to find Python functions missing

Pytorch 0.4.0 gives a segmentation fault with dilated 2D convolutions over a specific size. import torch import as nn model = nn.Conv2d(inchannels=64, outchannels=64, kernelsize=5, dilation=2) model(([1, 64, 1166, 1166])) # This runs model(([1, 64, 1167, 1167])) # This crashes PyTorch version: 0.4.0 Is debug build: No CUDA used to build PyTorch: 9.1.85 OS: CentOS Linux 7 (Core) GCC version: (GCC) 4.8.5 (Red Hat 4.8.5-28) CMake version: version 2.8.12.2 Python version: 3.6 Is CUDA available: Yes CUDA runtime version: 8.0.61 GPU models and configuration: GPU 0: GeForce GTX 1080 Ti GPU 1: TITAN X (Pascal) GPU 2: Quadro K2200 Nvidia driver version: 390.59 cuDNN version: Probably one of the following: /usr/local/cuda- /usr/local/cuda- /usr/local/cuda-7.5/lib64/libcudnnstatic.a /usr/local/cuda- /usr/local/cuda-8.0/lib64/libcudnn_static.a
do you think this is the same as the segfault you showed me a week ago on convolution?
ah yes. int overflow. I only fixed conv3d because I didn’t want to hurt perf. I’ll fix all of them.

This should be an easy one.. The function is implemented, just no sign of it in the docs.
Change:
for (int i = 0; i < required_args; ++i) { if (!inputs[i].defined()) { std::stringstream ss; <del> ss << name << ": expected Variable at argument " << i << " (got None)"; </del> <ins> ss << name << ": expected Tensor at argument " << i << " (got None)"; </ins> throw std::runtime_error(ss.str()); } }

Discussion:
Considering the merge of and , and the "removal" of from the API, there are still some messages that reference . This might prove to be confusing for new users starting off from , as they probably might not have heard of . I did a code search to find out multiple instances of this in . PyTorch or Caffe2: PyTorch How you installed PyTorch (conda, pip, source): source (@ ) Build command you used (if compiling from source): OS: Ubuntu 16.04.2 PyTorch version: Python version: 3.5 CUDA/cuDNN version: 8.0.44 GPU models and configuration: GeForce 940M GCC version (if compiling from source): 5.4.0 CMake version: 3.5.1
Change:
if (cur_target == ignore_index) { continue; } <ins> assert(cur_target  >= 0 && cur_target  < n_classes); </ins> Dtype weight = weights ? weights[cur_target] : ScalarConvert<int, Dtype>::to(1); gradInput[index][cur_target] = -weight * gradOutput[index];

Discussion:
Claim: When I use an invalid label with F.crossentropy with reduce=False, I don't receive an error message. I do receive an error message when reduce=True. Code: Result: No error message received. Expected error message: Context: I was using , accidentally used an invalid label, and got extremely large losses and extremely small losses, and inconsistently so. Environment: OS: Ubuntu 16.04 on Docker and host PyTorch version: 0.3.0.post4 How you installed PyTorch: conda Python version: 3.6.3
Change:
- THTensor* self - long k - CONSTANT __last_dim <del> - CONSTANT true - CONSTANT true </del> <ins> - arg: bool largest default: "true" kwarg_only: True - arg: bool sorted default: "true" kwarg_only: True </ins> - arguments: - arg: THTensor* values output: True

Discussion:
Hi.. I think this is a bug: So, it doesn't matter if descending is True or False, just specifying it explicitly requires dimension to be specified.. It would be better (I think) if descending could be specified alone..
Same issue for if is used:
sort was fixed in topk is still a problem.
Change:
if arg['name'] in ['self', 'state', 'dataType', 'handle']: arg['ignore_check'] = True declaration['options'] = self.filter_unique_options(declaration['options']) <del> return declarations </del> <ins> return [d for d in declarations if not d.get('only_register', False)] </ins> def filter_unique_options(self, options): def signature(option):

Discussion:
We should check that the version from cudnn.h matches the version from the loaded shared library. Really weird things happen if they don't match.
Change:
self.ceil_mode = ceil_mode def extra_repr(self): <del> return 'norm_type={norm_type}, kernel_size{kernel_size}, stride={stride}, '  </del> <ins> return 'norm_type={norm_type}, kernel_size={kernel_size}, stride={stride}, '  </ins> 'ceil_mode={ceil_mode}'.format(**self.__dict__)

Discussion:
is not available in the docs page, but the doc string has been in . This was mentioned earlier in

Oh, bilinear, I missed that last time I looked at it. I'll add a docstring.

returns nothing. Might be useful to have some linter in the build process to find Python functions missing

Pytorch 0.4.0 gives a segmentation fault with dilated 2D convolutions over a specific size. import torch import as nn model = nn.Conv2d(inchannels=64, outchannels=64, kernelsize=5, dilation=2) model(([1, 64, 1166, 1166])) # This runs model(([1, 64, 1167, 1167])) # This crashes PyTorch version: 0.4.0 Is debug build: No CUDA used to build PyTorch: 9.1.85 OS: CentOS Linux 7 (Core) GCC version: (GCC) 4.8.5 (Red Hat 4.8.5-28) CMake version: version 2.8.12.2 Python version: 3.6 Is CUDA available: Yes CUDA runtime version: 8.0.61 GPU models and configuration: GPU 0: GeForce GTX 1080 Ti GPU 1: TITAN X (Pascal) GPU 2: Quadro K2200 Nvidia driver version: 390.59 cuDNN version: Probably one of the following: /usr/local/cuda- /usr/local/cuda- /usr/local/cuda-7.5/lib64/libcudnnstatic.a /usr/local/cuda- /usr/local/cuda-8.0/lib64/libcudnn_static.a
do you think this is the same as the segfault you showed me a week ago on convolution?
ah yes. int overflow. I only fixed conv3d because I didn’t want to hurt perf. I’ll fix all of them.

This should be an easy one.. The function is implemented, just no sign of it in the docs.
Change:
self.assertEqual(output[0][0].sum().data[0], 0) self.assertEqual(output[1][2].sum().data[0], 0) <ins> embedding = nn.Embedding(10, 20, padding_idx=0, sparse=True) input = Variable(torch.LongTensor([[0, 2, 4, 5], [4, 3, 0, 9]])) output = embedding(input) self.assertEqual(output[0][0].sum().data[0], 0) self.assertEqual(output[1][2].sum().data[0], 0) </ins> def test_embedding_max_norm(self): embedding = nn.Embedding(22, 5, max_norm=1.0) input = Variable(torch.LongTensor([2, 8, 8, 6]))

Discussion:
As reported here with repro. When sparse=True, paddingidx should probably error out (as the easiest solution). Alternatively, the gradients for paddingidx indices should be zeroed out explicitly in the sparse=True backward codepath.
Change:
torch.randn(4, 4).cuda()) @unittest.skipIf(not TEST_MULTIGPU, "multi-GPU not supported") <ins> def test_broadcast_not_requiring_grad(self): variables = [ Variable(torch.randn(1, 2).cuda(), requires_grad=True), Variable(torch.randn(1, 2).cuda(), requires_grad=False), Variable(torch.randn(1, 2).cuda(), requires_grad=False), Variable(torch.randn(1, 2).cuda(), requires_grad=True), Variable(torch.randn(1, 2).cuda(), requires_grad=True), ] broadcasted_variables = Broadcast.apply((0, 1), *variables) for output_idx, broadcasted_var in enumerate(broadcasted_variables): input_var = variables[output_idx % len(variables)] self.assertEqual(input_var.requires_grad, broadcasted_var.requires_grad) @unittest.skipIf(not TEST_MULTIGPU, "multi-GPU not supported") </ins> def test_replicate(self): module = nn.Linear(10, 5).float().cuda() input = Variable(torch.randn(2, 10).float().cuda())

Discussion:
PyTorch version: 0.3.0 Python version: 3.5 GPU models and configuration: 8x NVIDIA Titan I found that requiresgrad setting in module is ignored when module is wrapped with nn.DataParallel. For example, This code should be executed because very small part of network activation (only fc layer) is stored for backward computation, but it results to run-time error: Internally, I found a crack in function which is in . In replicate function, it copies all parameter in module (# of replica times) with . In broadcasting code, it just defines new with default constructor parameter, which is always set to True. I think there can be some choices to fix this issue. is intended behavior for DataParallel, so we should use volatile (or ) to implement transfer learning in DataParallel model should be fixed to handle requires_grad when copying parameters. should be synchronizing in I think any of solution doesn't need lots of effort to fix it.
Change:
"torch/csrc/autograd/functions/convolution.cpp", "torch/csrc/autograd/functions/init.cpp", "torch/csrc/nn/THNN_generic.cpp", <ins> *split_types("torch/csrc/Tensor.cpp") </ins> ] try:

Discussion:
Trying to run on github actions via AWS instance gives me: I will of course set up now, but it shouldn't crash like that. May be it should use as a fallback? pt-2.0 cc
Is this a multitenant setup? Also curious which distro are you running, I was under the impression that most linux distros set home by default
this is AWS image. one default user. I think it's user, but it wants you to run as root, which is weird. I have manually now set to overcome this.
We should update triton to use pathlib or something