jiebi/CodeConvo huggingface dataset:
Change:
#   method name, #   input size/constructing fn, #   args (tuple represents shape of a tensor arg), <del> #   test variant name (will be used at test name suffix),  // optional #   indices for possible dim arg,                          // optional #   output indices that should be gradcheck'ed,            // optional </del> <ins> #   test variant name (will be used at test name suffix),    // optional #   indices for possible dim arg,                            // optional #   fn mapping output to part that should be gradcheck'ed,   // optional </ins> # ) method_tests = [ ('add', (S, S, S), ((S, S, S),)),

Discussion:
I am having problem in running the backward propagation with svd. In particular, it seems working when the first dimension of the matrix x (below) is greater than the second. But I get the following error if the first is smaller than the second. PyTorch version: 2018.04.20 Is debug build: No CUDA used to build PyTorch: 8.0.61 OS: Ubuntu 16.04.4 LTS GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.9) 5.4.0 CMake version: Could not collect Python version: 3.5 Is CUDA available: No CUDA runtime version: No CUDA GPU models and configuration: No CUDA Nvidia driver version: No CUDA cuDNN version: No CUDA Versions of relevant libraries: [pip3] numpy (1.13.3) [pip3] torch (2018.4.20) [pip3] torchvision (0.2.0) [conda] pytorch 0.3.1 py35cuda8.0.61cudnn7.0.52 pytorch [conda] pytorch-nightly 2018.04.20 py35cuda8.0.61cudnn7.1.21 pytorch [conda] torchvision 0.2.0 py35heaa392f_1 pytorch
Repro'ed on 0.4 branch. works btw.
What does "Repro'ed" means? Should I install from that branch? I need to compute the kernel of a matrix, therefore I need the last columns of the matrix V (which is why I have to pass ).
It's a shorthand for "reproduced", which means that the bug still hasn't been fixed. Sorry, we'll try to get to it soon.
Change:
TensorTriOp(T *start_, int64_t stride0_, int64_t stride1_, int64_t k_) : start(start_), stride0(stride0_), stride1(stride1_), k(k_) {} <del> __device__ __forceinline__ int mask(T *in) { ptrdiff_t n = in - start; </del> <ins> __device__ __forceinline__ int mask(T *out) { ptrdiff_t n = out - start; </ins> int64_t row, col; if (stride0 > stride1) {

Discussion:
On recent PyTorch master (built from source; 0.4.0a0+) on a P100, CUDA 9:
I think the results on CPU meets the defination of triu (upper triangular part of the matrix). It is really hard to handle with a tensor from expand operation with multi-thread not only on GPU but also on CPU. The unexpected result will also happen on CPU if we apply openmp to it. I think it is necessary to eleminate the difference between the implementation and defination.
It doesn’t matter if it’s cpu or not. triu doesn’t have an underscore which means it doesn’t operate in-place and should always give the same result (in new memory)
Change:
LDFLAGS="-L$INSTALL_DIR/lib " LD_POSTFIX=".so.1" LD_POSTFIX_UNVERSIONED=".so" <del> if [[ $(uname) == 'Darwin' ]]; then </del> <ins> if [[ $(uname) == 'Darwin' ]]; then </ins> LDFLAGS="$LDFLAGS -Qunused-arguments -Wl,-rpath,@loader_path" LD_POSTFIX=".1.dylib" LD_POSTFIX_UNVERSIONED=".dylib"

Discussion:
there is a bug. the code is from The code for installing cuda10.2 should be corrected to The code for installing cuda10.2 should be corrected to cc
Thank you for reporting the typo

This is how I implement the decoder of a sequence to sequence model I'm not sure about the new autograd mechanics but this worked in the previous version. If I didn't make the unrolling codes a function it will work. It will also work with CPU. I compiled from source with Python 2.7, CUDA 8.0 and Cudnn 6
Thanks for the minimal reproduction! It broke OpenNMT and a big internal model but we weren't looking forward to isolating what parts of those had the issue.
That's great, I'll take a look today. Thanks!
I've pushed a fix to .
How does the inference work in this case? I mean, once you train it how do you test it in the test phase?
Change:
return t(*sizes).copy_(torch.randn(*sizes)) return tmp <ins> # Content of each tuple: # - function name # - constructor for the tensor,    signature: fn(tensor_type) -> tensor # - constructor for the arguments, signature: fn(tensor_type) -> list # - postfix name for the test (must be unique for a given function) (default='') # - tensor types to use (default=types) # - disable inplace test, if set to True, no inplace test will be done (default=False) </ins> tests = [ ('add', small_3d, lambda t: [number(3.14, 3, t)]), ('add', small_3d, lambda t: [small_3d_positive(t)], 'tensor'),

Discussion:
On recent PyTorch master (built from source; 0.4.0a0+) on a P100, CUDA 9:
I think the results on CPU meets the defination of triu (upper triangular part of the matrix). It is really hard to handle with a tensor from expand operation with multi-thread not only on GPU but also on CPU. The unexpected result will also happen on CPU if we apply openmp to it. I think it is necessary to eleminate the difference between the implementation and defination.
It doesn’t matter if it’s cpu or not. triu doesn’t have an underscore which means it doesn’t operate in-place and should always give the same result (in new memory)
Change:
} void THCTensor_(sparseMask)(THCState *state, THCSTensor *r_, THCTensor *t, THCSTensor *mask) { <ins> THArgCheck(mask->coalesced, 2, "mask is uncoalesced"); </ins> THCAssertSameGPU(THCSTensor_(checkGPU)(state, 2, 3, r_, mask, t)); if(!THCSTensor_(isSameSizeAsDense)(state, mask, t)) { THError("sparseMask operands have incompatible sizes");

Discussion:
and I talked about the intended semantics of sparsemask when given a sparse tensor as an argument, and we came to the following conclusions: with an uncoalesced sparse tensor does not make sense, full stop. Unlike values, where your update might be linear and work correctly on an uncoalesced sparse tensor, there isn't any good reason why you want to pluck out values from the dense tensor multiple times, it is basically always a bug. the (eventual) documentation for sparsemask, we will recommend calling coalesce before doing any operations involving sparsemask (or nonlinear updates to indices/values). should add a test to sparsemask which checks if the flag of the input tensor is true, and raise an error if it is not. This is a cheap safety check which will prevent people from forgetting to call coalesce (as in (2)). There is a performance consequence to calling coalesce every iteration of training. If the gradient is producing coalesced vectors, we can alleviate this problem by adding a preprocessing step to coalesce() which tests if the input is already coalesced, and bugs out in that case. However, I think this is out of scope for this particular ticket for now.
Change:
(Addcdiv,       (),                 ((S, S), (S, S), torch.rand(S, S) + 1e-2)   ), (Addcdiv,       (0.6,),             ((S, S), (S, S), torch.rand(S, S) + 1e-2), 'scale'), (IndexAdd,      (0,),               ((S, S), index_variable(2, S), (2, S))      ), <del> (IndexCopy,     (0,),               ((S, S), index_variable(2, S), (2, S))      ), </del> <ins> # (IndexCopy,     (0,),               ((S, S), index_variable(2, S), (2, S))      ), </ins> (IndexFill,     (0, 2),             ((S, S), index_variable(2, S))              ), (IndexSelect,   (0,),               ((S, S), index_variable(2, S))              ), (Gather,        (0,),               ((M, S), gather_variable((S, S), 1, M))     ),

Discussion:
This happens when the index tensor contains duplicate elements. If this is not allowed indexcopy should raise an exception when this happens and we should fix the test. Otherwise we need to fix the autograd op. Here's an example: cc
Why would you with repeated indices? Sounds for me like UB is a reasonable. We should check that in C
I ran into this in and fixed the test for now. I'm not sure how to interpret last comment: did you mean UB behavior is "unreasonable" rather than "a reasonable"? The last sentence reads that way.
I'd be ok with saying that it's UB in such case. There's no natural choice, and it can decrease performance.
Is this issue still worth working on? If adding the additional check can decrease performance, can we add another argument specifying that indexes contain duplicate elements?
Change:
#include <Python.h> #include "THP.h" <ins> #include "torch/csrc/utils/auto_gil.h" </ins> // Adapted from fblualib void* ObjectPtrAllocator::malloc(ptrdiff_t size) {

Discussion:
When both PyTorch and Pandas are imported, calling results in a segfault. The issue was originally reported by It looks like calling returns a null pointer here: Examining in GDB: I suspect it's both PyTorch and Pandas doing threading magic and somehow getting in each other's way. I haven't yet been able to provide a minimal example that triggers this behaviour and does not use my package; it's likely that I'm doing something odd that causes this. More documentation here: Possibly related issues: and Python version: 3.6 PyTorch version: v0.2.0 Pandas version: 0.20.1
Change:
#include <vector> #include <string> <ins> #include "torch/csrc/utils/object_ptr.h" </ins> #define THPUtils_(NAME) TH_CONCAT_4(THP,Real,Utils_,NAME) #define THPUtils_typename(obj) (Py_TYPE(obj)->tp_name)

Discussion:
Go to Look at the html. It contains the following: ! This problem manifests in google search. Searching for 1.8.1 docs brings up the "1.8.0 docs": ! We can probably fix this by doing a quick sed on the website but we should develop a solution for future releases. I think this has happened before in the past and am not sure what causes it. cc
Change:
name: binary_linux_manywheel_3_7m_cu102_devtoolset7_test requires: - binary_linux_manywheel_3_7m_cu102_devtoolset7_build <del> resource_class: gpu.medium </del> <ins> resource_class: gpu.nvidia.small </ins> use_cuda_docker_runtime: "1" - binary_linux_test: build_environment: libtorch 3.7m cpu devtoolset7

Discussion:
As it still using deprecated resource class, see See example of the failure in
Followup: we need to plan better --we knew it was gonna happen and none of us fixed it early enough. Action item: go through tasks about deprecation in sprint planning
Change:
optimizer (Optimizer): Wrapped optimizer. step_size (int): Period of learning rate decay. gamma (float): Multiplicative factor of learning rate decay. <del> Default: -0.1. </del> <ins> Default: 0.1. </ins> last_epoch (int): The index of last epoch. Default: -1. Example:

Discussion:
as shown in source code, this parameters is not used. Any reason that we keep it?
this can be cleaned up.
Change:
"torch/csrc/Generator.cpp", "torch/csrc/Tensor.cpp", "torch/csrc/Storage.cpp", <ins> "torch/csrc/byte_order.cpp", </ins> "torch/csrc/utils.cpp", "torch/csrc/allocators.cpp", "torch/csrc/serialization.cpp",

Discussion:
<!-- A clear and concise description of what the bug is. --The target name is used by a test executable in mkldnn. Steps to reproduce the behavior: BUILDTEST=1 build <!-- If you have a code sample, error messages, stack traces, please provide it here as well --<!-- A clear and concise description of what you expected to happen. --CMake configure passes. Please copy and paste the output from our (or fill out the checklist below manually). You can get the script and run it with: PyTorch Version (e.g., 1.0): v1.5.0-rc2 OS (e.g., Linux): Windows How you installed PyTorch (, , source): source Build command you used (if compiling from source): python build Python version: 3.x CUDA/cuDNN version: 9.2 / 10.1 / 10.2 / NA GPU models and configuration: Tesla M60 Any other relevant information: <!-- Add any other context about the problem here. --We should not build tests for mkldnn.
I have encountered the same error, how do I resolve it?
It is actually mkldnn that is out of sync. So running the following script should help.
Thanks a lot for this I posted this on the PyTorch Discussions but to no avail, and now I've finally understood the reason of this error. Thanks a lot. Surprisingly, for me the build works out if I build from the master repo, but changing to a different version always fails.
Change:
grouped_args += [group(t) for t in args] res.append(impl[fn_name](self, self._bufs[g], *grouped_args)) if fn_name == 'grad_params': <del> return [torch.cat(t, 0) for t in zip(*res)] </del> <ins> return [torch.cat(t, 0) if t[0] is not None else None for t in zip(*res)] </ins> else: return torch.cat(res, 1)

Discussion:
x = Variable(((16, 64, 8, 8))) res = nn.Conv2d(16, 16, kernelsize=3, groups=2, bias=False)(x) results in: return ('updateoutput', input, weight, bias) File "", line 158, in thnn groupedargs += [group(t) for t in args] File "", line 153, in group dim = 0 if () == 1 else 1 AttributeError: 'NoneType' object has no attribute 'dim' The in line 158 is because there is no bias. I am running on CPU so this is in the code.
Change:
if mode not in {'min', 'max'}: raise ValueError('mode ' + mode + ' is unknown!') if threshold_mode not in {'rel', 'abs'}: <del> raise ValueError('threshold mode ' + mode + ' is unknown!') </del> <ins> raise ValueError('threshold mode ' + threshold_mode + ' is unknown!') </ins> if mode == 'min' and threshold_mode == 'rel': rel_epsilon = 1. - threshold self.is_better = lambda a, best: a < best * rel_epsilon

Discussion:
Trivial report, unfortunately I can't make a PR at the moment, but I can submit a report. line 329 in should replace with
Change:
.. automethod:: log2 .. automethod:: log2_ .. automethod:: log_normal_ <ins> .. automethod:: logsumexp </ins> .. automethod:: long .. automethod:: lt .. automethod:: lt_

Discussion:
Leaving this issue here for higher visibility. Originally reported by here: Also mentioned by Piggybacking off of this issue, has strange behavior (reported here: )
I have fixed this locally. Could I send a PR?
Yes, go ahead!
Change:
] extra_compile_args += ['-DWITH_DISTRIBUTED_MW'] include_dirs += [tmp_install_path + "/include/THD"] <del> main_link_args += [THD_LIB] </del> <ins> main_link_args += [THD_LIB, THPP_LIB] </ins> if platform.system() == 'Linux': main_link_args += [GLOO_LIB]

Discussion:
It seems that the shared object file is not being built correctly and this is leading to build errors on Travis.
this is weird, i saw this intermittently on travis as well. But couldn't reproduce on subsequent builds on Travis or locally.
I ran a clean build on the latest commit locally just now and I can't reproduce it either. Is it that ccache is preserving some build files and those are interfering with subsequent builds?
Actually, I apologize, I am able to reproduce. Importing torch in an ipython shell gives the same issue.
I meet the same problem. I am using the newest version "commit ". And I checked all dependent libraries of , all of them are available. Any advices are welcome. Much thanks.
I have reproduced something similar. It seems to only happen when you do . EDIT: No, my problem was actually that I was running Python inside my PyTorch checkout.
It also happens when you do
Varun, i cant reproduce this. What's your OS version and python version?
I have met the same problem as is mentioned above. I'm using Ubuntu 16.04.3 and Anaconda python 2.7.13.
Ubuntu 16.04.2 Python 3.6.2 Cuda 8.0
one last thing, what's your gcc version, and did you do python clean, or clone a fresh new folder
I did a , but am running it again to make sure.
On definitively running , I am not getting the error.
thanks everyone, should be fixed on master now.
Change:
num_devices = 0; } #endif <del> ready_queues = std::vector<std::shared_ptr<ReadyQueue>>(num_devices + 1); for (auto& queue : ready_queues) { </del> <ins> int num_threads = num_devices + 1; ready_queues = std::vector<std::shared_ptr<ReadyQueue>>(num_threads); for (int i = 0; i < num_threads; ++i) { auto& queue = ready_queues[i]; </ins> queue.reset(new ReadyQueue()); <del> std::thread t(&Engine::thread_main, this, queue); </del> <ins> std::thread t(&Engine::thread_main, this, queue, i - 1); </ins> t.detach(); } }

Discussion:
Consider following code, and run it with multi gpus (e.g. 4): It will output: That is, the default device would always be zero even in DataParallel. My pytorch version is 0.1.122. I think this is not the desired behavior. It would cause some troubles. I tried to insert my own cuda kernel into backward to calculate the gradients, it become very slow, and I fixed it by (grad.getdevice()). Anyway, I think currentdevice in forward and backward should be the same, or could anyone explain to me why they are different?
this seems like something we should fix: ?
<!-- drci-comment-start --:pagefacingup: Preview :pagefacingup: Preview :question: Need help or want to give feedback on the CI? Visit our Note: Links to docs will display an error until the docs builds have been completed. This comment was automatically generated by Dr. CI and updates every 15 minutes. <!-- drci-comment-end --
Change:
return make_tensor(t, M, M) <ins> def medium_2d_expanded(t): return t(1).expand(M, M) </ins> def medium_2d_scaled(t, scale=10): return make_tensor(t, M, M).mul(scale)

Discussion:
On recent PyTorch master (built from source; 0.4.0a0+) on a P100, CUDA 9:
I think the results on CPU meets the defination of triu (upper triangular part of the matrix). It is really hard to handle with a tensor from expand operation with multi-thread not only on GPU but also on CPU. The unexpected result will also happen on CPU if we apply openmp to it. I think it is necessary to eleminate the difference between the implementation and defination.
It doesn’t matter if it’s cpu or not. triu doesn’t have an underscore which means it doesn’t operate in-place and should always give the same result (in new memory)
Change:
else: sizes = torch.LongStorage(args) sizes = _infer_sizes(sizes, self.nelement()) <del> if reduce(lambda a,b: a * b, sizes) != self.nelement(): raise RuntimeError('Invalid size for view. Input size: ' + 'x'.join(map(lambda v: str(v), self.size())) + ', output size: ' + 'x'.join(map(lambda v: str(v), sizes)) + '.') assert self.is_contiguous(), "expecting a contiguous tensor" dst.set_(self.storage(), self.storage_offset(), sizes) </del> <ins> numel = reduce(lambda a,b: a * b, sizes) if len(sizes) > 0 else 0 if numel != self.nelement(): def format_size(size): return 'x'.join(str(v) for v in size) if len(size) > 0 else '0' raise ValueError( "view of size '{0}' is invalid for input of size '{1}'" .format(format_size(sizes), format_size(self.size()))) if not self.is_contiguous(): raise ValueError("input should be contiguous") if self.storage() is not None: dst.set_(self.storage(), self.storage_offset(), sizes) </ins> return dst def view_as(self, tensor):

Discussion:
Please check and fix all the PyTorch web links format from to , like in the official site.
closed via
Change:
-DNO_NNPACK=%NO_NNPACK% ^ -DCUDNN_INCLUDE_DIR="%CUDNN_INCLUDE_DIR%" ^ -DCUDNN_LIB_DIR="%CUDNN_LIB_DIR%" ^ <del> -DCMAKE_BUILD_TYPE=Release </del> <ins> -DCMAKE_BUILD_TYPE=%BUILD_TYPE% </ins> %MAKE_COMMAND% cd ../..

Discussion:
The current master triggers the following error messages from Visual Studio 2015 (Debug build): 2------ Build started: Project: ATen, Configuration: Debug x64 ------ 2cl : Command line error D8016: '/Ox' and '/RTC1' command-line options are incompatible ... 14LINK : fatal error LNK1104: cannot open file '....srcATenDebug' It appears that they are triggered by *.c files in the ATen project, like "srcTHvectorAVX.c"
Why do you need Debug build?
Isn't ATen meant to support both Debug and Release builds? FYI I'm involved with the project at which does support both build configurations, and we would like to be able to link our project with ATen.
Well, you can remove the flags in as a workground.
We should patch this file to ignore this flag when building the library in debug mode. It shouldn't be too hard
I catched similar errors , when I Tryed to build Debug Version I try to build Release and catched next errors e:pytorchatensrcatencpuvec256vec256int.h(184): note: see reference to function template instantiation 'at::vec256::anonymous-namespace'::Vec256<int16t::blend<1(at::vec256::anonymous-namespace'::Vec256<int16t)' being compiled 28e:pytorchatensrcatencpuvec256vec256int.h(147): error C3861: 'mm256extract_epi16': identifier not found What I should do?
There're two issues in your description. The first question is easy. You'll also have to specify the flag to 1. The second question is about your compiler. I guess you are using VS2015 or earlier versions of VS2017. Make sure the version in the output is 19.11 after your enter in CMD.
Yes I am using VS2015. My Compiler is 19.0.2. Must I use VS 2017 in any case?
Sorry, but the answer is yes. The AVX2 functions was in VS2017 and they are used in PyTorch. So VS2017 19.11 is a must.
No problem:) Thank you.
Change:
for i, (sample, target) in enumerate(loader): idx = i * batch_size self.assertEqual(sample, self.data[idx:idx + batch_size]) <del> self.assertEqual(target, self.labels[idx:idx + batch_size].view(-1, 1)) </del> <ins> self.assertEqual(target, self.labels[idx:idx + batch_size]) </ins> self.assertEqual(i, math.floor((len(self.dataset) - 1) / batch_size)) def _test_shuffle(self, loader):

Discussion:
TensorDataset adds casts 1D target tensor to 2D which is not acceptable by NLL loss layer.
Fixed in .
Change:
(hx + cx).sum().backward() <del> @unittest.skipIf(not TEST_CUDNN, "needs cudnn") @default_tensor_type(torch.FloatTensor)  # FIXME: just until torch.cuda.DoubleTensor.sum() implemented def test_RNN_cpu_vs_cudnn(self): </del> <ins> def _test_RNN_cpu_vs_cudnn(self, dropout): </ins> def forward_backward(cuda, rnn, input_val, hx_val, weights_val): is_lstm = type(rnn) == nn.LSTM

Discussion:
cuDNN backend for RNN stabilizes at large mem usage. The following code stabilizes for me at 14+GiB reported by nvidia-smi. The same RNN without cuDNN uses <2.5GiB.
Oops... any idea what's going on with the garbage collector in this case?
are you running this on a P100 GPU?
Yes, I observe similar behavior on Titan X Pascal
hmmm, do cudnn RNNs have weird memory-oversubscribing properties? If it wasn't for CuDNN RNN, then it's not unreasonable. we use a custom CUDA allocator that will oversubscribe if needed, but will free the memory blocks it's holding onto if it hits OOM / failed allocations.
Tried the same RNN in torch and am getting ~2.6GiB on nvidia-smi.
With cuDNN I'm getting a constant usage of 2241MB on my machine using PyTorch (M40).
are you using cudnnv5, or v6?
v5
no, cudnn does not do memory oversubscribing, and already tried the same network in lua torch with cudnn, where it gets expected memory usage. Also, he is getting OOMs on slightly bigger networks, where apparently caching allocator should have tried to free previous blocks. Interesting that can not repro this.
Just to make sure did you set when you ran? I left it as False in the repro.
I can repro in the docker image built from the current version on P100, and I go OOM after 8 batches with the same image on 1080, which generally should not happen - if there is enough memory to go through 1 batch, memory usage should not increase afterwards.
No, I just copied your script and ran it. Weird thing is that it goes OOM after 18 batches once I enable . Without it memory remains stable.
A lot of cudaMalloc allocations go not through THCudaMalloc, and never get freed:
So not going through THCudaMalloc is actually expected for storages allocated with THCStorage_(newWithSize), but what's not expected is to every time cudaMalloc bytes of rnn workspace, instead of reusing earlier allocations. (~784 MB allocations are for workspace, ~156MB allocations happen somewhere in criterion).
My guess is that the bug is at the Python level.

Either: Something is holding on to a Tensor longer than it should Something is creating a circular reference which prevents quick clean-up
The issue is that in creates cyclic references. It's made worse by the fact that it holds on to more memory than it should (like the workspace). You can see this by adding a as the first line in your for loop. With the explicit call to the GC every iteration, the memory usage doesn't explode.
how can these cyclic references be fixed?
Not sure yet, but I will fix it tomorrow with