jiebi/CodeConvo huggingface dataset:
Change:
return None return match.group(1) <ins> def run_and_return_first_line(run_lambda, command): """Runs command using run_lambda and returns first line if output is not empty""" rc, out, _ = run_lambda(command) if rc != 0: return None return out.split('n')[0] </ins> def get_conda_packages(run_lambda): if get_platform() == 'win32':

Discussion:
TensorPipe uses some advanced features of the Linux kernel. It does so on a "best effort" basis (i.e., use them if available, fall back on something else if not). This logic is unfortunately hard to test because it would require specific kernel versions on our CI machines. So it occasionally happens that we find out about issues when users report them. To help investigate these reports we've found it useful to know the kernel version and the glibc version used by the user (both when building and when running, if they're different). It would be great if these could be to the report produced by collect_env.
CC
Change:
export PATH="$PWD:$PATH" popd <del> time test/run_test.sh </del> <ins> time test/run_test.sh -- -v </ins> rm -rf ninja

Discussion:
If a test segfaults, it's hard to determine which test it is because the output looks like: ! I can work on this if we think this is a good idea.
Last time I asked about this, he didn't want to because it adds a lot of scrolling. I also did some work to move the CI tests to so that we can output XML status files, but (1) this is unlikely to help in case of segfault, and (2) you can't do this straightforwardly, because we run some test files multiple times with different environment variables, and the XML exporter isn't really set up to handle this case. For the record, I am personally OK with just slapping on the tests and calling it a day.
in the conbuild sounds good to me too
closed via
Change:
std::unique_ptr<npy_intp[]> sizes; std::unique_ptr<npy_intp[]> strides; <ins> THPUtils_assert(PyTuple_Size(args) == 0, "numpy() takes no positional arguments"); </ins> // Numpy and Torch disagree on empty tensors. In Torch, an empty tensor // is a tensor with zero dimensions. In Numpy, a tensor with zero dimensions // is a scalar (with one element). So we'll convert an empty Torch tensor

Discussion:
Currently due to the way cwrap generates wrapping code for the c source, passing an argument to .numpy() would not result in any error. I think probably this should not be the desired behaviour for the API?
fixed via
Change:
// See Note [Numpy memory management] void* NumpyArrayAllocator::realloc(void* ptr, ptrdiff_t size) { <del> PyArrayObject *array_ptr = (PyArrayObject*)object.get(); if (array_ptr && ptr == PyArray_DATA(array_ptr)) { void* newPtr = this->malloc(size); memcpy(newPtr, ptr, std::min((size_t) size, (size_t) PyArray_NBYTES(array_ptr))); // Whee! We're done! object = nullptr; return newPtr; } return allocator->realloc(allocatorContext, ptr, size); </del> <ins> throw std::logic_error("NumpyArrayAllocator::realloc() not supported"); </ins> } // See Note [Numpy memory management]

Discussion:
When both PyTorch and Pandas are imported, calling results in a segfault. The issue was originally reported by It looks like calling returns a null pointer here: Examining in GDB: I suspect it's both PyTorch and Pandas doing threading magic and somehow getting in each other's way. I haven't yet been able to provide a minimal example that triggers this behaviour and does not use my package; it's likely that I'm doing something odd that causes this. More documentation here: Possibly related issues: and Python version: 3.6 PyTorch version: v0.2.0 Pandas version: 0.20.1
Change:
"""Fills the input Tensor or Variable with values according to the method described in "Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification" - He, K. et al. (2015), using a uniform distribution. The resulting tensor will have values sampled from :math:`U(-bound, bound)` where <del> :math:`bound = sqrt{2 / ((1 + a^2) times fan_in)} times sqrt{3}`. Also known as He initialisation. </del> <ins> :math:`bound = sqrt{2 / ((1 + a^2) times fan_in)} times sqrt{3}`. Also known as He initialisation. </ins> Args: tensor: an n-dimensional torch.Tensor or autograd.Variable

Discussion:
why 1.21.6? because that's the oldest supported numpy on windows/python3.10: all cc
Here is the test for pinning numpy to 1.21.6 The reason its was not pinned to this version is that conda can't properly resolve all the dependencies at this step here:
It looks like you are correct, I will put up a PR to pin lower then already pinned:
Here is the PR:
Change:
sequence.add_(start - 1) return sequence.resize_(*size) <del> def test_index(self): reference = self._consecutive((3, 3, 3)) </del> <ins> @staticmethod def _test_index(self, conv_fn): def consec(size, start=1): sequence = torch.ones(int(torch.Tensor(size).prod(0)[0])).cumsum(0) sequence.add_(start - 1) return sequence.view(*size) reference = conv_fn(consec((3, 3, 3))) </ins> # empty tensor indexing <del> self.assertEqual(reference[torch.LongTensor()], reference.new()) </del> <ins> self.assertEqual(reference[conv_fn(torch.LongTensor())], reference.new()) </ins> <del> self.assertEqual(reference[0], self._consecutive((3, 3)), 0) self.assertEqual(reference[1], self._consecutive((3, 3), 10), 0) self.assertEqual(reference[2], self._consecutive((3, 3), 19), 0) self.assertEqual(reference[0, 1], self._consecutive((3,), 4), 0) self.assertEqual(reference[0:2], self._consecutive((2, 3, 3)), 0) </del> <ins> self.assertEqual(reference[0], consec((3, 3)), 0) self.assertEqual(reference[1], consec((3, 3), 10), 0) self.assertEqual(reference[2], consec((3, 3), 19), 0) self.assertEqual(reference[0, 1], consec((3,), 4), 0) self.assertEqual(reference[0:2], consec((2, 3, 3)), 0) </ins> self.assertEqual(reference[2, 2, 2], 27, 0) <del> self.assertEqual(reference[:], self._consecutive((3, 3, 3)), 0) </del> <ins> self.assertEqual(reference[:], consec((3, 3, 3)), 0) </ins> # indexing with Ellipsis self.assertEqual(reference[..., 2], torch.Tensor([[3, 6, 9],

Discussion:
Arguably, indexing a Tensor by a LongTensor with no dimension should return a LongTensor with no dimension. Consider the following case: This breaks indexing by indices when the list of indices is empty, which can be hard to detect. In my use-case, I use such a masking to select rows of a tensor corresponding to "valid" entries in a vector; I had not considered that it would not work when there are no valid entries.
Thanks! I'll take a look.
Change:
<del> import torch from torch.autograd.function import Function from torch._thnn import type2backend import torch.backends.cudnn as cudnn class BatchNorm(Function): def __init__(self, running_mean, running_var, training, momentum, eps): super(BatchNorm, self).__init__() self.running_mean = running_mean self.running_var = running_var self.training = training self.momentum = momentum self.eps = eps def forward(self, input, weight=None, bias=None): self.save_for_backward(input, weight, bias) # don't use cuDNN for half inputs because cuDNN requires the weight and # bias tensors to be floats, unlike THCUNN which requires half tensors. self.use_cudnn = (cudnn.is_acceptable(input) and cudnn.version() > 5110 and weight is not None and bias is not None and not isinstance(input, torch.cuda.HalfTensor)) # temporary buffers used in forward and backward num_features = input.size(1) _save_mean = input.new(num_features) _save_std = input.new(num_features) output = input.new(input.size()) if self.use_cudnn: torch._C._cudnn_batch_norm_forward( input, output, weight, bias, self.running_mean, self.running_var, _save_mean, _save_std, self.training, self.momentum, self.eps) else: backend = type2backend[type(input)] backend.BatchNormalization_updateOutput( backend.library_state, input, output, weight, bias, self.running_mean, self.running_var, _save_mean, _save_std, self.training, self.momentum, self.eps) if self.requires_grad: self._save_mean = _save_mean self._save_std = _save_std return output def backward(self, grad_output): input, weight, bias = self.saved_tensors grad_input, grad_weight, grad_bias = None, None, None if self.needs_input_grad[0] or self.use_cudnn: grad_input = input.new(input.size()) if (len(self.needs_input_grad) > 1 and self.needs_input_grad[1]) or self.use_cudnn: grad_weight = weight.new(weight.size()).zero_() if (len(self.needs_input_grad) > 1 and self.needs_input_grad[2]) or self.use_cudnn: grad_bias = bias.new(bias.size()).zero_() if self.use_cudnn and self.training: # cudnn does not support backward in evaluate mode torch._C._cudnn_batch_norm_backward( input, grad_output, grad_input, grad_weight, grad_bias, weight, self.running_mean, self.running_var, self._save_mean, self._save_std, self.training, self.eps) else: grad_output = grad_output.contiguous() backend = type2backend[type(input)] backend.BatchNormalization_backward( backend.library_state, input, grad_output, grad_input, grad_weight, grad_bias, weight, self.running_mean, self.running_var, self._save_mean, self._save_std, self.training, 1.0, self.eps) return grad_input, grad_weight, grad_bias </del>

Discussion:
As Arthur Szlam reports, fb-internal cudnn is still lagging behind, and giving batch-size 1024 with batchnorm is raising an error from cudnn. Need to check for compile-time version and disable this codepath
The 1024 limitation was removed in 5.1.10
Change:
for i in range(10): Variable(torch.randn(10, 10), _grad_fn=CollectOnDelete()) <del> @unittest.skipIf(not torch.cuda.is_available() or torch.cuda.device_count() < 2, "CUDA not available or <2 GPUs detected") </del> <ins> @unittest.skipIf(torch.cuda.device_count() < 2, "no multi-GPU") </ins> def test_unused_output_gpu(self): from torch.nn.parallel._functions import Broadcast x = Variable(torch.randn(5, 5).float().cuda(), requires_grad=True)

Discussion:
Consider following code, and run it with multi gpus (e.g. 4): It will output: That is, the default device would always be zero even in DataParallel. My pytorch version is 0.1.122. I think this is not the desired behavior. It would cause some troubles. I tried to insert my own cuda kernel into backward to calculate the gradients, it become very slow, and I fixed it by (grad.getdevice()). Anyway, I think currentdevice in forward and backward should be the same, or could anyone explain to me why they are different?
this seems like something we should fix: ?
<!-- drci-comment-start --:pagefacingup: Preview :pagefacingup: Preview :question: Need help or want to give feedback on the CI? Visit our Note: Links to docs will display an error until the docs builds have been completed. This comment was automatically generated by Dr. CI and updates every 15 minutes. <!-- drci-comment-end --
Change:
res = [] for g in range(self.groups): def group(tensor, dim=None): <ins> if tensor is None: return None </ins> if dim is None: dim = 0 if tensor.dim() == 1 else 1 n = tensor.size(dim) // self.groups

Discussion:
x = Variable(((16, 64, 8, 8))) res = nn.Conv2d(16, 16, kernelsize=3, groups=2, bias=False)(x) results in: return ('updateoutput', input, weight, bias) File "", line 158, in thnn groupedargs += [group(t) for t in args] File "", line 153, in group dim = 0 if () == 1 else 1 AttributeError: 'NoneType' object has no attribute 'dim' The in line 158 is because there is no bias. I am running on CPU so this is in the code.
Change:
self.assertEqual(n[1], l2) self.assertEqual(n[2], l3) self.assertEqual(n[3], l4) <ins> self.assertEqual(n[1:], nn.Sequential(l2, l3, l4)) self.assertEqual(n[3:], nn.Sequential(l4)) self.assertEqual(n[:-1], nn.Sequential(l1, l2, l3)) self.assertEqual(n[:-3], nn.Sequential(l1)) self.assertEqual(n[::-1], nn.Sequential(l4, l3, l2, l1)) </ins> def test_ModuleList(self): modules = [nn.ReLU(), nn.Linear(5, 5)]

Discussion:
it'd be great to pick out elements of of a sequential with array slicing:
I agree this idea!
However,
this has come up again (last in ). Shall we reconsider?
I attempted getting a to work. Is this fine? If this is fine, I can issue a PR too. Let me know what you think. I believe this modification needs to be done in the class.
cc: This issue has been open for quite sometime.
Change:
id_map = {old_id: p for old_id, p in zip(chain(*(g['params'] for g in saved_groups)), chain(*(g['params'] for g in groups)))} <del> state = {id_map.get(k, k): v for k, v in state_dict['state'].items()} </del> <ins> state = defaultdict( dict, {id_map.get(k, k): v for k, v in state_dict['state'].items()}) </ins> # Update parameter groups, setting their 'params' value def update_group(group, new_group):

Discussion:
The state attribute in the opt. class is a defaultdict which helps to add new parameter states in the state dict as they are being used while training. Once the optimizer's statedict is saved and loaded, the state attribute is a dict and not a defaultdict which assumes that before saving, all the parameters in the network are present in the state dict. When loading the opt. using a saved statedict the state attribute should be a defaultdict. I currently do the following to have this behavior Corresponding
Hi all, it is my first time trying to contribute to an open-source project. I took a look on the bug. It seems that it is related to the function "statedict()" of class "". When the state is transformed to packedstate, it uses dict instead of defaultdict, which is inconsistent with the type of state. The solution I am proposing is to let the packedstate be defaultdict to maintain consistency with Here is my proposed change: packedstate = defaultdict(dict, {(id(k) if isinstance(k, Variable) else k): v for k, v in ()}) Is there anything I am missing? Is "" in the pytorch/test enough to validate my change? Where could I add a test point to ensure the type of state remain the same after saving and loading? Thanks a lot for your patience.
closed via thanks to !! Thanks a lot !
Change:
std::tie(self, indices) = transposeToFront(self, indices); } auto linearIndex = computeLinearIndex(self, indices); <del> return std::make_tuple<>(self, linearIndex); </del> <ins> return std::make_tuple(self, linearIndex); </ins> } Tensor index(const Tensor & self, TensorList indices) {

Discussion:
In PyTorch master 0.4 , variables cannot be indexed. This used to work in PyTorch 0.2 :
This is breaking some of Pyro's inference algorithms on PyTorch master:
Change:
if not self.shutdown: self.shutdown = True self.done_event.set() <del> # if worker_manager_thread is waiting to put while not self.data_queue.empty(): self.data_queue.get() </del> <ins> # if worker_manager_thread is waiting to put, make place for it try: while not self.data_queue.empty(): self.data_queue.get() except FileNotFoundError: # FileNotFoundError can happen when we rebuild the fd # fetched from the queue but the socket is already closed # from the worker side (e.g. due to Python shutting down). pass </ins> for _ in self.workers: self.index_queue.put(None) # done_event should be sufficient to exit worker_manager_thread,

Discussion:
I'm creating a in my script ([]), and I get an error when this script is close to ending. I am using MPI through mpi4py, not the MPI PyTorch distributed backend. This error only happens when under , not under (which completes successfully). Nothing in the traceback reveals this is an issue with my code, and I haven't touched the DataLoader at all. I've been running this code for a couple months in the same environment, and I haven't seen this bug pop before. That is, this command does not raise an error: However, this command does raise an error: Traceback: PyTorch version: 0.3.1.post2 How you installed PyTorch (conda, pip, source): conda Python version: 3.6.1 OS/CUDA/cuDNN version: A modification of the Deep Learning AMI GPU models and configuration: Amazon instance []
I do not see this bug with Pytorch 0.3.0.post4`
Change:
@property def saved_tensors(self): flat_tensors = super(NestedIOFunction, self).saved_tensors <del> return _map_tensor_fromiter(iter(flat_tensors))(self._to_save_nested) </del> <ins> return _unflatten(flat_tensors, self._to_save_nested) </ins> def mark_dirty(self, *args, **kwargs): self.dirty_tensors = tuple(_iter_tensors((args, kwargs)))

Discussion:
cuDNN backend for RNN stabilizes at large mem usage. The following code stabilizes for me at 14+GiB reported by nvidia-smi. The same RNN without cuDNN uses <2.5GiB.
Oops... any idea what's going on with the garbage collector in this case?
are you running this on a P100 GPU?
Yes, I observe similar behavior on Titan X Pascal
hmmm, do cudnn RNNs have weird memory-oversubscribing properties? If it wasn't for CuDNN RNN, then it's not unreasonable. we use a custom CUDA allocator that will oversubscribe if needed, but will free the memory blocks it's holding onto if it hits OOM / failed allocations.
Tried the same RNN in torch and am getting ~2.6GiB on nvidia-smi.
With cuDNN I'm getting a constant usage of 2241MB on my machine using PyTorch (M40).
are you using cudnnv5, or v6?
v5
no, cudnn does not do memory oversubscribing, and already tried the same network in lua torch with cudnn, where it gets expected memory usage. Also, he is getting OOMs on slightly bigger networks, where apparently caching allocator should have tried to free previous blocks. Interesting that can not repro this.
Just to make sure did you set when you ran? I left it as False in the repro.
I can repro in the docker image built from the current version on P100, and I go OOM after 8 batches with the same image on 1080, which generally should not happen - if there is enough memory to go through 1 batch, memory usage should not increase afterwards.
No, I just copied your script and ran it. Weird thing is that it goes OOM after 18 batches once I enable . Without it memory remains stable.
A lot of cudaMalloc allocations go not through THCudaMalloc, and never get freed:
So not going through THCudaMalloc is actually expected for storages allocated with THCStorage_(newWithSize), but what's not expected is to every time cudaMalloc bytes of rnn workspace, instead of reusing earlier allocations. (~784 MB allocations are for workspace, ~156MB allocations happen somewhere in criterion).
My guess is that the bug is at the Python level.

Either: Something is holding on to a Tensor longer than it should Something is creating a circular reference which prevents quick clean-up
The issue is that in creates cyclic references. It's made worse by the fact that it holds on to more memory than it should (like the workspace). You can see this by adding a as the first line in your for loop. With the explicit call to the GC every iteration, the memory usage doesn't explode.
how can these cyclic references be fixed?
Not sure yet, but I will fix it tomorrow with
Change:
using thpp::Tensor; <ins> #ifndef CUDNN_BN_MIN_EPSILON #define CUDNN_BN_MIN_EPSILON 0 #endif </ins> auto BatchNormForward::apply(const variable_list& inputs) -> variable_list { check_input_variables("BatchNorm", inputs, 3, 1);

Discussion:
Hi, I have two linux servers. My code works in one machine but not the other. In the one that it doesn't work, the error message is: occured during batch normalization( convolution is fine). Also, the same code works in CPU but not in GPU. I guess this problem is mainly related to GPU libraries like cuda or cudnn. The one that doesn't work has the following configurations: Nvidia driver 375.26 cuda 8.0.61 cudnn 5.1.10 While in the one it does work, I have the following configurations: Nvidia driver 367.48 cuda 8.0.44 cudnn 5.1.5 TensorFlow can run in the above environment flawless though. So I am wondering if this is caused by the incompatibility between PyTorch 0.1.12 and cuda 8.0.61/cudnn 5.1.10 ?
you use the pytorch binaries, we ship cudnn6 in them, use it it's great. 5.1.10 maybe has a bug. cc: . do you have a small test case to reproduce this? does not use CuDNN batchnorm by default, so maybe that's why?
Small test case would help, thanks.
After some experimentation, I have pin pointed where exactly the error comes from. It seems batch norm in cudnn 5.1.10 doesn't support eps as small as 1e-6. A small test case would be:
Yes, cudnn.h for 5.1.10 defines CUDNNBNEPSILON to be 1e-5 (minimum allowable value). I can't check header for 5.1.5. Even with 1e-5 epsilon for many input data distributions the results become pretty unstable. Is there a case for this restriction to be lifted, does using very small epsilon provide benefits?
I see. Not sure if small epsilon is necessary. I am using 1e-6 just to replicate others' configurations. Should PyTorch provide more informative error message about this restriction?
Ideally, CUDNN should provide more informative error messages (pytorch can not know why cudnn is returning an error code in each particular case) , but that's a long standing discussion.
It just hits on me.

As mentioned by , TensorFlow does not use CuDNN batchnorm by default. So is this possible in Pytorch, to not use default CuDNN batchnorm?
Never mind. I found the switch in .
Change:
self.assertEqual(list(tensor), [4, 4, 4, 4]) p.join() <ins> def _test_empty_tensor_sharing(self, dtype): q = mp.Queue() empty = torch.tensor([], dtype=dtype) q.put(empty) out = q.get(timeout=1) self.assertEqual(out, empty) def test_empty_tensor_sharing(self): self._test_empty_tensor_sharing(torch.float32) self._test_empty_tensor_sharing(torch.int64) @unittest.skipIf(not torch.cuda.is_available(), 'CUDA not available') def test_empty_tensor_sharing_cuda(self): self._test_empty_tensor_sharing(torch.cuda.float32) self._test_empty_tensor_sharing(torch.cuda.int64) </ins> def _test_autograd_sharing(self, var): ready = mp.Event() master_modified = mp.Event()

Discussion:
PyTorch version: 0.4.0a0+ Repro: Result:
Emm. I fail to reproduce this bug in Python 3.5.0 and Mac OS 10.13.3. What's your system environment?
The bug is probably specific to the sharing mode. Mac OS uses by default.
Change:
.. automethod:: repeat .. automethod:: resize_ .. automethod:: resize_as_ <del> .. automethod:: retain </del> .. automethod:: round .. automethod:: round_ .. automethod:: rsqrt

Discussion:
Dataloader call getitem as many times as indices in the current batch. In case datasets support a list of indices in one call, or a native python slice object, add a getbatch (optional) to pytorch dataset class. If dataloader sees the dataset has such a method implemented, make it fetch the batch corresping to the list of indices with one getbatch call rather than many getitem calls. This saves the trouble of calling getitem many times in cases where for example a disk file has to be opened then closed for every call and the underlying data structure supports a slicing or lists of indices for example hdf5 files.
will support something similar
is the latest version of this.
Change:
// This Engine's ReadyQueues and their corresponding threads are leaked here Engine::~Engine() = default; <del> auto Engine::thread_main(std::shared_ptr<ReadyQueue> queue) -> void { </del> <ins> auto Engine::thread_main(std::shared_ptr<ReadyQueue> queue, int device) -> void { </ins> THInferNumThreads(); <ins> AutoGPU guard(device); </ins> while (1) { FunctionTask task = queue->pop_back(); if (!task.base->has_error.load()) {

Discussion:
Consider following code, and run it with multi gpus (e.g. 4): It will output: That is, the default device would always be zero even in DataParallel. My pytorch version is 0.1.122. I think this is not the desired behavior. It would cause some troubles. I tried to insert my own cuda kernel into backward to calculate the gradients, it become very slow, and I fixed it by (grad.getdevice()). Anyway, I think currentdevice in forward and backward should be the same, or could anyone explain to me why they are different?
this seems like something we should fix: ?
<!-- drci-comment-start --:pagefacingup: Preview :pagefacingup: Preview :question: Need help or want to give feedback on the CI? Visit our Note: Links to docs will display an error until the docs builds have been completed. This comment was automatically generated by Dr. CI and updates every 15 minutes. <!-- drci-comment-end --