jiebi/CodeConvo huggingface dataset:
Change:
<ins> #include <stdint.h> #include <stddef.h> enum THPByteOrder { THP_LITTLE_ENDIAN = 0, THP_BIG_ENDIAN = 1 }; THPByteOrder THP_nativeByteOrder(); void THP_decodeInt16Buffer(int16_t* dst, const uint8_t* src, THPByteOrder order, size_t len); void THP_decodeInt32Buffer(int32_t* dst, const uint8_t* src, THPByteOrder order, size_t len); void THP_decodeInt64Buffer(int64_t* dst, const uint8_t* src, THPByteOrder order, size_t len); void THP_decodeFloatBuffer(float* dst, const uint8_t* src, THPByteOrder order, size_t len); void THP_decodeDoubleBuffer(double* dst, const uint8_t* src, THPByteOrder order, size_t len); </ins>

Discussion:
<!-- A clear and concise description of what the bug is. --The target name is used by a test executable in mkldnn. Steps to reproduce the behavior: BUILDTEST=1 build <!-- If you have a code sample, error messages, stack traces, please provide it here as well --<!-- A clear and concise description of what you expected to happen. --CMake configure passes. Please copy and paste the output from our (or fill out the checklist below manually). You can get the script and run it with: PyTorch Version (e.g., 1.0): v1.5.0-rc2 OS (e.g., Linux): Windows How you installed PyTorch (, , source): source Build command you used (if compiling from source): python build Python version: 3.x CUDA/cuDNN version: 9.2 / 10.1 / 10.2 / NA GPU models and configuration: Tesla M60 Any other relevant information: <!-- Add any other context about the problem here. --We should not build tests for mkldnn.
I have encountered the same error, how do I resolve it?
It is actually mkldnn that is out of sync. So running the following script should help.
Thanks a lot for this I posted this on the PyTorch Discussions but to no avail, and now I've finally understood the reason of this error. Thanks a lot. Surprisingly, for me the build works out if I build from the master repo, but changing to a different version always fails.
Change:
input=(torch.randn(2, 3, 5, 5, 5) * 1000)), OldModuleTest(nn.VolumetricMaxPooling, (2, 2, 2, 2, 2, 2), <del> input_size=(2, 3, 5, 5, 5), </del> <ins> input=(torch.randn(2, 3, 5, 5, 5) * 1000), </ins> desc='stride'), OldModuleTest(nn.VolumetricMaxPooling, (2, 2, 2, 2, 2, 2, 1, 1, 1), <del> input_size=(2, 3, 5, 5, 5), </del> <ins> input=(torch.randn(2, 3, 5, 5, 5) * 1000), </ins> desc='stride_padding'), OldModuleTest(nn.VolumetricReplicationPadding, (1, 2, 3, 4, 5, 6),

Discussion:
Repro: stack: says he's already investigating a similar problem in internal models. cc
we think may be root cause
I think this might be correct? I am seeing CPU devices explicitly created in the model, and in dynamo. I do not believe this is a bug, but we need much better tooling around detecting devices.
In that case we need to add float16 support to the cpu backend (ugh)
The node being copied CPU=CUDA is: so the only CPU op involved is -- which we are actually generating ourselves. This patch might help, by doing the conversion on GPU: Though after applying it we get a new error:
After applying some partial fixes in pytorch/torchdynamo, I tried the minifier -- but it didn't seem to work. cc I was running I get a lot of printouts of: But then it finishes I get:
If I change where the cpu device is set to "cuda" in the model source - and remove an assert on our end, I can get the same exact error. That might make for an easier minification and repro story. Let me see if that minimizes on my end.
Please rebase to this PR,
pytorch/torchdynamo gets the same error without needing to do dtype conversion, potentially easier for debugging?
Minified version (requires pytorch/torchdynamo)
ref pytorch/torchdynamo
Change:
THCAssertSameGPU(THCTensor_(checkGPU)(state, 2, self_, src_)); THArgCheck(src_->nDimension == 2, 1, "expected a matrix"); <del> THCTensor *src = src_; if (self_ == src_) src = THCTensor_(newContiguous)(state, src_); </del> <ins> if (self_ != src_) THCTensor_(resizeAs)(state, self_, src_); </ins> <del> int64_t stride0 = src->stride[0]; int64_t stride1 = src->stride[1]; real *start = THCTensor_(data)(state, src); </del> <ins> int64_t stride0 = self_->stride[0]; int64_t stride1 = self_->stride[1]; real *start = THCTensor_(data)(state, self_); </ins> TensorTriOp<real, 0> op(start, stride0, stride1, k); if (self_ == src_) { <del> if (!THC_pointwiseApply1(state, src, op)) { </del> <ins> if (!THC_pointwiseApply1(state, src_, op)) { </ins> THArgCheck(false, 2, CUTORCH_DIM_WARNING); } } else { <del> THCTensor_(resizeAs)(state, self_, src); </del> <ins> THCTensor_(resizeAs)(state, self_, src_); </ins> <del> if (!THC_pointwiseApply2(state, self_, src, op)) { </del> <ins> if (!THC_pointwiseApply2(state, self_, src_, op)) { </ins> THArgCheck(false, 2, CUTORCH_DIM_WARNING); } } <del> if (self_ == src_) THCTensor_(freeCopyTo)(state, src, src_); </del> THCudaCheck(cudaGetLastError()); }

Discussion:
On recent PyTorch master (built from source; 0.4.0a0+) on a P100, CUDA 9:
I think the results on CPU meets the defination of triu (upper triangular part of the matrix). It is really hard to handle with a tensor from expand operation with multi-thread not only on GPU but also on CPU. The unexpected result will also happen on CPU if we apply openmp to it. I think it is necessary to eleminate the difference between the implementation and defination.
It doesn’t matter if it’s cpu or not. triu doesn’t have an underscore which means it doesn’t operate in-place and should always give the same result (in new memory)
Change:
inputs, module_kwargs = scatter_kwargs(inputs, module_kwargs, device_ids, dim) if len(device_ids) == 1: return module(*inputs[0], **module_kwargs[0]) <del> replicas = replicate(module, device_ids[:len(inputs)]) outputs = parallel_apply(replicas, inputs, module_kwargs, device_ids) </del> <ins> used_device_ids = device_ids[:len(inputs)] replicas = replicate(module, used_device_ids) outputs = parallel_apply(replicas, inputs, module_kwargs, used_device_ids) </ins> return gather(outputs, output_device, dim)

Discussion:
has broken DataParallel tests. Now that deviceids are explicitly sent to parallelapply, this assert gets triggered if inputs are not big enough to be on all devices (e.g. batch size of 20 on 8 GPUs gets chunked into 6*3+2, so that 8-th GPU is idle, and assert gets triggered).
your latest fix in doesn't fix this completely. Here's a failing stack-trace:
How is it possible that the tests didn't catch that in the PR?
it's because our build box which only has 2 GPUs doesn't hit this case. It's failing on 4-GPU setting (maybe batch size becomes too small then)
Change:
grad = grad.index(c); } <del> int64_t num_features = grad.size(-1); </del> <ins> int64_t num_features = grad_.size(-1); </ins> auto weight_size = std::array<int64_t, 2>{{ num_weights, num_features }}; <ins> auto& dense_type = grad.type(); auto& sparse_type = dense_type.toBackend(grad.is_cuda() ? kSparseCUDA : kSparseCPU); // check if all our grad come from padding_idx if (grad.numel() == 0) { return sparse_type.sparse_coo_tensor(indices_.type().tensor(), dense_type.tensor(), weight_size); } </ins> auto index = indices.view({1, -1}); auto values = grad.contiguous().view({-1, num_features}); <del> auto& sparse_type = grad.type().toBackend(grad.is_cuda() ? kSparseCUDA : kSparseCPU); </del> return sparse_type.sparse_coo_tensor(index, values, weight_size); }

Discussion:
I have a straightforward use case. Given a token (or word), lookup the embedding of the word, pass that through a linear layer and perform L2 regression. If the token corresponds to the of the embedding layer, throws an error saying OS: macOS High Sierra PyTorch version: 0.3.1 How you installed PyTorch (conda, pip, source): Tried both conda and pip Python version: 3.6.4 CUDA/cuDNN version: CPU GPU models and configuration: NA GCC version (if compiling from source): NA A script to reproduce the bug:
I got the same trouble, and it throw this error after some epochs of training. Does anyone have some solutions?
Change:
THCTensor *newValues = THCTensor_(newNarrow)(state, sorted, dimension, k, 1); THCudaLongTensor *newIndices = THCudaLongTensor_newNarrow(state, sorted_indices, dimension, k, 1); <ins> THCTensor_(free)(state, sorted); THCudaLongTensor_free(state, sorted_indices); </ins> if (!keepdim) { THCTensor_(squeeze1d)(state, newValues, newValues, dimension); THCudaLongTensor_squeeze1d(state, newIndices, newIndices, dimension);

Discussion:
has a memory leak on the GPU when using PyTorch version 0.3.1. The memory leak goes away if I downgrade to PyTorch 0.3.0. I was able to reproduce the memory leak on 3 separate machines using P100s, 1080TIs and Titan Xs.How you installed PyTorch (conda, pip, source): conda OS: Ubuntu 16.04 PyTorch version: 0.3.1.post3 Python version: 3.6.3 CUDA version: 8.0.61 GPU models and configuration: 4x P100s, 2x 1080 TIs, 4x Titan X (3 separate machines)
thanks for the report. I can reproduce the bug on master. We'll get it fixed.
Change:
os.mkdir(os.path.join(tmp_dir, 'barrier')) os.mkdir(os.path.join(tmp_dir, 'test_dir')) if backend == 'mpi': <del> mpiexec = 'mpiexec -n 3 --noprefix {}'.format(python) </del> <ins> # test mpiexec for --noprefix option devnull = open(os.devnull, 'w') noprefix_opt = '--noprefix' if subprocess.call( 'mpiexec -n 1 --noprefix bash -c ""', shell=True, stdout=devnull, stderr=subprocess.STDOUT) == 0 else '' mpiexec = 'mpiexec -n 3 {} {}'.format(noprefix_opt, python) </ins> return_code = run_test(mpiexec, test_module, test_directory, options) else:

Discussion:
I installed to get available, but the test run failed this way: Log: So it sounds like we need a specific MPI backend for the tests, any old one won't do. CC and
Hmm can you try removing ?
Thank you, It's work!
I am sorry and , how to remove --noprefix,,? should I reinstall the mpi using --prefix,,? thank you
It needs to be removed from the script
Thank you so much adam, I have solved it Sent from Mail for Windows 10
Change:
lr=1e-1) ) <ins> def test_adagrad_sparse(self): self._test_rosenbrock_sparse( lambda params: optim.Adagrad(params, lr=1e-1) ) </ins> def test_adamax(self): self._test_rosenbrock( lambda params: optim.Adamax(params, lr=1e-1),

Discussion:
and I talked about the intended semantics of sparsemask when given a sparse tensor as an argument, and we came to the following conclusions: with an uncoalesced sparse tensor does not make sense, full stop. Unlike values, where your update might be linear and work correctly on an uncoalesced sparse tensor, there isn't any good reason why you want to pluck out values from the dense tensor multiple times, it is basically always a bug. the (eventual) documentation for sparsemask, we will recommend calling coalesce before doing any operations involving sparsemask (or nonlinear updates to indices/values). should add a test to sparsemask which checks if the flag of the input tensor is true, and raise an error if it is not. This is a cheap safety check which will prevent people from forgetting to call coalesce (as in (2)). There is a performance consequence to calling coalesce every iteration of training. If the gradient is producing coalesced vectors, we can alleviate this problem by adding a preprocessing step to coalesce() which tests if the input is already coalesced, and bugs out in that case. However, I think this is out of scope for this particular ticket for now.
Change:
set /a NO_NNPACK=1 ) <ins> IF "%DEBUG%"=="1" ( set BUILD_TYPE=Debug ) ELSE ( set BUILD_TYPE=Release ) </ins> IF "%CMAKE_GENERATOR%"=="" ( set CMAKE_GENERATOR_COMMAND= set MAKE_COMMAND=msbuild INSTALL.vcxproj /p:Configuration=Release

Discussion:
The current master triggers the following error messages from Visual Studio 2015 (Debug build): 2------ Build started: Project: ATen, Configuration: Debug x64 ------ 2cl : Command line error D8016: '/Ox' and '/RTC1' command-line options are incompatible ... 14LINK : fatal error LNK1104: cannot open file '....srcATenDebug' It appears that they are triggered by *.c files in the ATen project, like "srcTHvectorAVX.c"
Why do you need Debug build?
Isn't ATen meant to support both Debug and Release builds? FYI I'm involved with the project at which does support both build configurations, and we would like to be able to link our project with ATen.
Well, you can remove the flags in as a workground.
We should patch this file to ignore this flag when building the library in debug mode. It shouldn't be too hard
I catched similar errors , when I Tryed to build Debug Version I try to build Release and catched next errors e:pytorchatensrcatencpuvec256vec256int.h(184): note: see reference to function template instantiation 'at::vec256::anonymous-namespace'::Vec256<int16t::blend<1(at::vec256::anonymous-namespace'::Vec256<int16t)' being compiled 28e:pytorchatensrcatencpuvec256vec256int.h(147): error C3861: 'mm256extract_epi16': identifier not found What I should do?
There're two issues in your description. The first question is easy. You'll also have to specify the flag to 1. The second question is about your compiler. I guess you are using VS2015 or earlier versions of VS2017. Make sure the version in the output is 19.11 after your enter in CMD.
Yes I am using VS2015. My Compiler is 19.0.2. Must I use VS 2017 in any case?
Sorry, but the answer is yes. The AVX2 functions was in VS2017 and they are used in PyTorch. So VS2017 19.11 is a must.
No problem:) Thank you.
Change:
cudnnDestroyConvolutionDescriptor(desc); } void set(cudnnDataType_t dataType, int dim, int* pad, int* stride, int * upscale) { <ins> cudnnDataType_t mathType = dataType; if (dataType == CUDNN_DATA_HALF) mathType = CUDNN_DATA_FLOAT; </ins> CHECK(cudnnSetConvolutionNdDescriptor(desc, dim, pad, stride, upscale, <del> CUDNN_CROSS_CORRELATION, dataType)); </del> <ins> CUDNN_CROSS_CORRELATION, mathType)); </ins> } };

Discussion:
to reproduce Will result in:
What's your GPU and CUDA version?
TitanX Maxwell, cuda 8.0
In cudnn bindings convolution math type is inferred from input data type (and inferred to be equal to input data type): Math type for convolutions with half inputs and weights should be CUDNNDATAFLOAT because 1) computations with half math are not supported anywhere except Pascal and TX1 2) Even on Pascal, 6.1 cards have very low performance fp16 math 3) fp16 math comes with a set of accuracy problems
Thanks. Seems like batchnorm fp16 support is still missing. e.g: will result in
Ah, yes. fp16 support was removed from cudnn bn backend (because it still requires fp32 for parameters), and probably it is not properly routed to or not supported in nn backend. BTW, be careful with truncating bn params to 16 bits, in our experiments on wide resnets/cifar it cost ~ half a percent of accuracy, though there may be ways to overcome this.
Is there an update regarding this? Could you describe how to truncate the bn params?
did you update PyTorch to a recent version? The snippet no longer raises an error for me
Thanks for the fast response. I installed the newest version via conda. Just retried the above code snippet, it works indeed. Shouldn't it be possible to "half" a entire model like this: For this i get: CUDNNSTATUSBAD_PARAM
Your input size is wrong, models in torchvision expect specific input sizes. You can play around with the batch size, but number of channels and image size are fixed. When I change input size to (64,3,224,224) the snippet above does not return an error.
Change:
fn.w_desc = rnn.init_weight_descriptor(fn, fn.weight_buf) # Slice off views into weight_buf <del> params = rnn.get_parameters(fn, handle, fn.weight_buf) </del> all_weights = [[p.data for p in l] for l in self.all_weights] <ins> params = rnn.get_parameters(fn, handle, fn.weight_buf) </ins> # Copy weights and update their storage rnn._copyParams(all_weights, params)

Discussion:
While the fast path for cuDNN RNNs introduced in includes a check intended to fall back to copying weights if weights are moved or aliased, this only works if the aliasing happens after the weights are first compacted. But the most natural place to tie weights is inside a module's method, which runs before , the typical time when is first invoked. Here's an MWE. The three runs (CPU, weight tying before , and weight tying after ) should all give the same results, and they do before , but they have not since:
Change:
return !was_null && requires_grad == var.requires_grad() && is_volatile == var.is_volatile(); } <del> Node* recordTraceHelper(std::string op, at::ArrayRef<Variable> inputs, at::ArrayRef<Variable> outputs); // These overloads are intended to simplify code generation inline Node* recordTrace(std::string op, std::initializer_list<Variable> inputs, const Variable& output) { return recordTraceHelper(op, inputs, {output}); } inline Node* recordTrace(std::string op, std::initializer_list<Variable> inputs, const std::tuple<Variable, Variable>& outputs) { return recordTraceHelper(op, inputs, {std::get<0>(outputs), std::get<1>(outputs)}); } inline Node* recordTrace(std::string op, std::initializer_list<Variable> inputs, const std::tuple<Variable, Variable, Variable>& outputs) { return recordTraceHelper(op, inputs, {std::get<0>(outputs), std::get<1>(outputs), std::get<2>(outputs)}); } inline Node* recordTrace(std::string op, at::TensorList inputs, const Variable& output) { // TODO: Eliminate the intermediate vector allocation return recordTraceHelper(op, variable_list(inputs.begin(), inputs.end()), {output}); } </del> <ins> Node* recordTrace(std::string op, at::ArrayRef<Variable> inputs, at::ArrayRef<Variable> outputs); </ins> }}} // namespace torch::jit::tracer

Discussion:
when executing:
I'm seeing this as well as of last night. Its preceded by many warnings about : But the first error is the one about . I went back, and I get a good install as late as commit
Change:
('topk', small_3d_unique, lambda t: [2, 1, True, True], 'dim_desc_sort'), ('trace', medium_2d, lambda t: [],), ('tril', medium_2d, lambda t: [],), <ins> ('tril', medium_2d_expanded, lambda t: [], 'zero_stride', types, True), </ins> ('tril', medium_2d, lambda t: [2], 'positive'), ('tril', medium_2d, lambda t: [-2], 'negative'), ('triu', medium_2d, lambda t: [],), <ins> ('triu', medium_2d_expanded, lambda t: [], 'zero_stride', types, True), </ins> ('triu', medium_2d, lambda t: [2], 'positive'), ('triu', medium_2d, lambda t: [-2], 'negative'), ('unsqueeze', new_t(2, 3, 4), lambda t: [2],),

Discussion:
On recent PyTorch master (built from source; 0.4.0a0+) on a P100, CUDA 9:
I think the results on CPU meets the defination of triu (upper triangular part of the matrix). It is really hard to handle with a tensor from expand operation with multi-thread not only on GPU but also on CPU. The unexpected result will also happen on CPU if we apply openmp to it. I think it is necessary to eleminate the difference between the implementation and defination.
It doesn’t matter if it’s cpu or not. triu doesn’t have an underscore which means it doesn’t operate in-place and should always give the same result (in new memory)
Change:
strt = _tensor_str(self) size_str = 'x'.join(str(size) for size in self.size()) <del> device_str = '' if not self.is_cuda else ' (GPU {})'.format(self.get_device()) </del> <ins> device_str = '' if not self.is_cuda else  ' (GPU {})'.format(self.get_device()) </ins> strt += '[{} of size {}{}]n'.format(torch.typename(self), <del> size_str, device_str) </del> <ins> size_str, device_str) </ins> return 'n' + strt

Discussion:
It seems that the following link that can be reached from here (check the screenshot) is broken: Page: Broken link: Screenshot: ! cc
this should be transferred to , but I don't have permissions to do that. Fix is available at pytorch/hub.
Change:
optimizer (Optimizer): Wrapped optimizer. milestones (list): List of epoch indices. Must be increasing. gamma (float): Multiplicative factor of learning rate decay. <del> Default: -0.1. </del> <ins> Default: 0.1. </ins> last_epoch (int): The index of last epoch. Default: -1. Example:

Discussion:
as shown in source code, this parameters is not used. Any reason that we keep it?
this can be cleaned up.
Change:
"""Fills the input Tensor or Variable with values according to the method described in "Understanding the difficulty of training deep feedforward neural networks" - Glorot, X. & Bengio, Y. (2010), using a uniform distribution. The resulting tensor will have values sampled from :math:`U(-a, a)` where <del> :math:`a = gain times sqrt{2 / (fan_in + fan_out)} times sqrt{3}`. Also known as Glorot initialisation. </del> <ins> :math:`a = gain times sqrt{2 / (fan_in + fan_out)} times sqrt{3}`. Also known as Glorot initialisation. </ins> Args: tensor: an n-dimensional torch.Tensor or autograd.Variable

Discussion:
why 1.21.6? because that's the oldest supported numpy on windows/python3.10: all cc
Here is the test for pinning numpy to 1.21.6 The reason its was not pinned to this version is that conda can't properly resolve all the dependencies at this step here:
It looks like you are correct, I will put up a PR to pin lower then already pinned:
Here is the PR:
Change:
AT_PRIVATE_CASE_TYPE(at::ScalarType::Long, int64_t, __VA_ARGS__)         AT_PRIVATE_CASE_TYPE(at::ScalarType::Short, int16_t, __VA_ARGS__)        default:                                                                 <del> AT_ERROR("%s not implemented for '%s'", (NAME), the_type.toString());  </del> <ins> AT_ERROR(#NAME, " not implemented for '", the_type.toString(), "'");  </ins> }                                                                          }()

Discussion:
is not available in the docs page, but the doc string has been in . This was mentioned earlier in

Oh, bilinear, I missed that last time I looked at it. I'll add a docstring.

returns nothing. Might be useful to have some linter in the build process to find Python functions missing

Pytorch 0.4.0 gives a segmentation fault with dilated 2D convolutions over a specific size. import torch import as nn model = nn.Conv2d(inchannels=64, outchannels=64, kernelsize=5, dilation=2) model(([1, 64, 1166, 1166])) # This runs model(([1, 64, 1167, 1167])) # This crashes PyTorch version: 0.4.0 Is debug build: No CUDA used to build PyTorch: 9.1.85 OS: CentOS Linux 7 (Core) GCC version: (GCC) 4.8.5 (Red Hat 4.8.5-28) CMake version: version 2.8.12.2 Python version: 3.6 Is CUDA available: Yes CUDA runtime version: 8.0.61 GPU models and configuration: GPU 0: GeForce GTX 1080 Ti GPU 1: TITAN X (Pascal) GPU 2: Quadro K2200 Nvidia driver version: 390.59 cuDNN version: Probably one of the following: /usr/local/cuda- /usr/local/cuda- /usr/local/cuda-7.5/lib64/libcudnnstatic.a /usr/local/cuda- /usr/local/cuda-8.0/lib64/libcudnn_static.a
do you think this is the same as the segfault you showed me a week ago on convolution?
ah yes. int overflow. I only fixed conv3d because I didn’t want to hurt perf. I’ll fix all of them.

This should be an easy one.. The function is implemented, just no sign of it in the docs.
Change:
char *bytes = (char *) data; int64_t remaining = sizeof(real) * size; while (remaining > 0) { <del> ssize_t result = write(fd, bytes, remaining); </del> <ins> // we write and read in 1GB blocks to avoid bugs on some OSes ssize_t result = write(fd, bytes, THMin(remaining, 1073741824)); </ins> if (result < 0) throw std::system_error(result, std::system_category()); bytes += result;

Discussion:
Running this on OSX: More details of failure in Ubuntu 14.04.1 in
Maybe it's caused by a ? I just test this code and it doesn't work (on Mac OS X 10.12.3 with Python 3.6):
That thread links to this one , which says it’s an OS bug and proposes a workaround patch that hasn’t been merged. There’s also a numpy issue thread, which links to this Torch7 commit that implements a workaround
We're doing a loop in case it writes less, but we always pass the full amount of remaining bytes. Putting a cap (std::min with INT_MAX) on the size of a single write might help.
I can't reproduce it on CentOS. I think it might have been a report for a PyTorch version prior to fixing .