jiebi/CodeConvo huggingface dataset:
Change:
import math from bisect import bisect_right <ins> from functools import partial </ins> from .optimizer import Optimizer

Discussion:
I'm working on checkpointing in our nn trainer. When saving the state objects, I get an error: PicklingError: Can't pickle <type 'function': attribute lookup builtin.function failed I tracked it down to trying to save the scheduler ReduceLROnPlateau. I think the issue is due to the scheduler referencing the optimizer, which has to be saved with statedict(). I presume the same error will occur with other schedulers too. It seems like the problem could be solved by providing a statedict() and loadstatedict() in the scheduler classes. pytorch master rev (Jan 22) installed from source x86_64 on centos 7.4 cuda 9.0 cudnn 7 k80 gcc 5.4 python 2.7.14
The following script reproduces the crash: import torch import from import ReduceLROnPlateau if name == 'main': model = (10,10) optim = (model.parameters()) sched = ReduceLROnPlateau(optim, mode='max', factor=0.5) with open('','wb') as f: # Works # ((model.statedict(), optim.statedict()), f) # Crashes ((model.statedict(), optim.state_dict(), sched), f)
Change:
ctx.num_inputs = len(inputs) ctx.input_device = inputs[0].get_device() outputs = comm.broadcast_coalesced(inputs, ctx.target_gpus) <ins> non_differentiables = [] for idx, input_requires_grad in enumerate(ctx.needs_input_grad[1:]): if not input_requires_grad: for output in outputs: non_differentiables.append(output[idx]) ctx.mark_non_differentiable(*non_differentiables) </ins> return tuple([t for tensors in outputs for t in tensors]) @staticmethod

Discussion:
PyTorch version: 0.3.0 Python version: 3.5 GPU models and configuration: 8x NVIDIA Titan I found that requiresgrad setting in module is ignored when module is wrapped with nn.DataParallel. For example, This code should be executed because very small part of network activation (only fc layer) is stored for backward computation, but it results to run-time error: Internally, I found a crack in function which is in . In replicate function, it copies all parameter in module (# of replica times) with . In broadcasting code, it just defines new with default constructor parameter, which is always set to True. I think there can be some choices to fix this issue. is intended behavior for DataParallel, so we should use volatile (or ) to implement transfer learning in DataParallel model should be fixed to handle requires_grad when copying parameters. should be synchronizing in I think any of solution doesn't need lots of effort to fix it.
Change:
THCDeviceTensor<THCIndex_t, 1> target, THCDeviceTensor<Dtype, 1> output, Dtype *weights, <ins> int n_classes, </ins> int ignore_index) { CUDA_KERNEL_LOOP(index, batch_size) {

Discussion:
Claim: When I use an invalid label with F.crossentropy with reduce=False, I don't receive an error message. I do receive an error message when reduce=True. Code: Result: No error message received. Expected error message: Context: I was using , accidentally used an invalid label, and got extremely large losses and extremely small losses, and inconsistently so. Environment: OS: Ubuntu 16.04 on Docker and host PyTorch version: 0.3.0.post4 How you installed PyTorch: conda Python version: 3.6.3
Change:
skipTestIf = test[5] if len(test) >= 6 else [] <del> test_output_indices = test[6] if len(test) >= 7 else None </del> <ins> output_process_fn = test[6] if len(test) >= 7 else lambda x: x </ins> for dim_perm in product([-1, 1], repeat=len(dim_args_idx)): test_name = basic_test_name

Discussion:
I am having problem in running the backward propagation with svd. In particular, it seems working when the first dimension of the matrix x (below) is greater than the second. But I get the following error if the first is smaller than the second. PyTorch version: 2018.04.20 Is debug build: No CUDA used to build PyTorch: 8.0.61 OS: Ubuntu 16.04.4 LTS GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.9) 5.4.0 CMake version: Could not collect Python version: 3.5 Is CUDA available: No CUDA runtime version: No CUDA GPU models and configuration: No CUDA Nvidia driver version: No CUDA cuDNN version: No CUDA Versions of relevant libraries: [pip3] numpy (1.13.3) [pip3] torch (2018.4.20) [pip3] torchvision (0.2.0) [conda] pytorch 0.3.1 py35cuda8.0.61cudnn7.0.52 pytorch [conda] pytorch-nightly 2018.04.20 py35cuda8.0.61cudnn7.1.21 pytorch [conda] torchvision 0.2.0 py35heaa392f_1 pytorch
Repro'ed on 0.4 branch. works btw.
What does "Repro'ed" means? Should I install from that branch? I need to compute the kernel of a matrix, therefore I need the last columns of the matrix V (which is why I have to pass ).
It's a shorthand for "reproduced", which means that the bug still hasn't been fixed. Sorry, we'll try to get to it soon.
Change:
y.sum().backward() self.assertEqual(x.grad.data, torch.ones(5, 5) * 2) <ins> @unittest.skipIf(torch.cuda.device_count() < 2, "no multi-GPU") def test_backward_device(self): # check that current device matches the variable's device device = [None] class Identity(torch.autograd.Function): @staticmethod def forward(ctx, x): return x.clone() @staticmethod def backward(ctx, grad_output): device[0] = torch.cuda.current_device() return grad_output.clone() v = Variable(torch.randn(1).cuda(1), requires_grad=True) Identity.apply(v).backward() self.assertEqual(device[0], 1) </ins> def test_detach(self): x = Variable(torch.randn(10, 10), requires_grad=True) y = x + 2

Discussion:
Consider following code, and run it with multi gpus (e.g. 4): It will output: That is, the default device would always be zero even in DataParallel. My pytorch version is 0.1.122. I think this is not the desired behavior. It would cause some troubles. I tried to insert my own cuda kernel into backward to calculate the gradients, it become very slow, and I fixed it by (grad.getdevice()). Anyway, I think currentdevice in forward and backward should be the same, or could anyone explain to me why they are different?
this seems like something we should fix: ?
<!-- drci-comment-start --:pagefacingup: Preview :pagefacingup: Preview :question: Need help or want to give feedback on the CI? Visit our Note: Links to docs will display an error until the docs builds have been completed. This comment was automatically generated by Dr. CI and updates every 15 minutes. <!-- drci-comment-end --
Change:
with values from the interval ``[start, end)`` taken with common difference :attr:`step` beginning from `start`. <ins> Note that non-integer `step` is subject to floating point rounding errors when comparing against `end`; to avoid inconsistency, we advise adding a small epsilon to `end` in such cases. </ins> .. math:: out_{i+1} = out_{i} + mathrm{step}

Discussion:
! OS: Ubuntu 17.10 PyTorch version: 0.3.1 How you installed PyTorch (conda, pip, source): conda Python version: 3.6.4 CUDA/cuDNN version: - GPU models and configuration: -
This is a floating point problem that has been mitigated in master with this commit: But even with that fix, arange doesn't always give the expected result. You can avoid this issue by doing
It’s impossible to have arange work in all kinda of possible conditions due to rounding errors in floating point math. What we did in master was to make it consistent with numpy. An alternate solution would be to avoid having the length of the interval be exactly divisible by its length (you can make it a tiny bit larger).
The numpy docs have a note saying "When using a non-integer step, such as 0.1, the results will often not be consistent." Maybe this could be to the doc as well?
Change:
torch/csrc/nn/THNN_generic.cwrap torch/csrc/nn/THNN_generic.cpp torch/csrc/nn/THNN_generic.h <ins> torch/csrc/generated </ins> docs/src/**/* test/data/legacy_modules.t7 test/data/gpu_tensors.pt

Discussion:
Trying to run on github actions via AWS instance gives me: I will of course set up now, but it shouldn't crash like that. May be it should use as a fallback? pt-2.0 cc
Is this a multitenant setup? Also curious which distro are you running, I was under the impression that most linux distros set home by default
this is AWS image. one default user. I think it's user, but it wants you to run as root, which is weird. I have manually now set to overcome this.
We should update triton to use pathlib or something
Change:
:math:`p_n > 1` increases the recall, :math:`p_n < 1` increases the precision. For example, if a dataset contains 100 positive and 300 negative examples of a single class, <del> then `pos_weight` for the class should be equal to math:`frac{300}{100}=3`. </del> <ins> then `pos_weight` for the class should be equal to :math:`frac{300}{100}=3`. </ins> The loss would act as if the dataset contains math:`3times 100=300` positive examples. Args:

Discussion:
is not available in the docs page, but the doc string has been in . This was mentioned earlier in

Oh, bilinear, I missed that last time I looked at it. I'll add a docstring.

returns nothing. Might be useful to have some linter in the build process to find Python functions missing

Pytorch 0.4.0 gives a segmentation fault with dilated 2D convolutions over a specific size. import torch import as nn model = nn.Conv2d(inchannels=64, outchannels=64, kernelsize=5, dilation=2) model(([1, 64, 1166, 1166])) # This runs model(([1, 64, 1167, 1167])) # This crashes PyTorch version: 0.4.0 Is debug build: No CUDA used to build PyTorch: 9.1.85 OS: CentOS Linux 7 (Core) GCC version: (GCC) 4.8.5 (Red Hat 4.8.5-28) CMake version: version 2.8.12.2 Python version: 3.6 Is CUDA available: Yes CUDA runtime version: 8.0.61 GPU models and configuration: GPU 0: GeForce GTX 1080 Ti GPU 1: TITAN X (Pascal) GPU 2: Quadro K2200 Nvidia driver version: 390.59 cuDNN version: Probably one of the following: /usr/local/cuda- /usr/local/cuda- /usr/local/cuda-7.5/lib64/libcudnnstatic.a /usr/local/cuda- /usr/local/cuda-8.0/lib64/libcudnn_static.a
do you think this is the same as the segfault you showed me a week ago on convolution?
ah yes. int overflow. I only fixed conv3d because I didn’t want to hurt perf. I’ll fix all of them.

This should be an easy one.. The function is implemented, just no sign of it in the docs.
Change:
{ transpose_m1 = (transpose_r == 'n' ? 't' : 'n'); m1_ = THTensor_(newContiguous)(m1); <ins> free_m1 = 1; </ins> } /* m2 */

Discussion:
Here is an example of what seems to be a memory leak in pytorch, although it could be numpy? The leak was only occurring when using the or array unsqueezing methods in numpy. Also worth noting that the issue is resolved by removing the from and casting the torch tensor to type. An easy workaround is to reshape the array after turning it into a torch Tensor.
Change:
l.param_attr = None self.assertEqual(num_params(), 3) <ins> def test_assignment(self): l = nn.Module() a = nn.Parameter(torch.randn(2)) b = nn.Parameter(torch.randn(3)) c = nn.Parameter(torch.randn(4)) q = nn.Linear(4, 4) r = nn.Linear(5, 5) w = nn.Linear(6, 6) def test_assignments(get_list, a, b, c): # Check that None can be shadowed l.a = None self.assertIsNone(l.a) self.assertIn('a', l.__dict__) l.a = a self.assertIs(l.a, a) self.assertEqual(get_list(), [a]) self.assertNotIn('a', l.__dict__) # Assign second object l.b = None self.assertIsNone(l.b) self.assertIn('b', l.__dict__) l.b = b self.assertIs(l.b, b) self.assertEqual(get_list(), [a, b]) self.assertNotIn('b', l.__dict__) # Remove and add the object back. Order should be unchanged. l.a = None self.assertIsNone(l.a) self.assertEqual(get_list(), [b]) l.a = a self.assertIs(l.a, a) self.assertEqual(get_list(), [a, b]) # Replace object with another one. Order should be unchanged. l.a = c self.assertIs(l.a, c) self.assertEqual(get_list(), [c, b]) # Remove and reassign an attribute. It should appear at the end of the list now. del l.a self.assertFalse(hasattr(l, 'a')) l.a = a self.assertIs(l.a, a) self.assertEqual(get_list(), [b, a]) test_assignments(lambda: list(l.parameters()), a, b, c) del l.a, l.b self.assertEqual(list(l.parameters()), []) test_assignments(lambda: list(l.children()), q, r, w) del l.a, l.b self.assertEqual(list(l.children()), []) buf = torch.randn(10) l.register_buffer('buf', buf) self.assertIs(l.buf, buf) l.buf = None self.assertIs(l.buf, None) self.assertNotIn('buf', l.__dict__)  # should be stored in l._buffers l.buf = buf self.assertIn('buf', l.state_dict()) self.assertIs(l.state_dict()['buf'], buf) </ins> @unittest.skipIf(not TEST_CUDA, 'CUDA not available') def test_Conv2d_large_workspace(self): # These sizes require huge cuDNN workspaces. Make sure we choose a

Discussion:
They do one pair of these calls for each slicing operation.
Does this actually have a performance impact? These calls tend to be very quick
Yes, it's a major reason why small unrolled LSTMs (which involve in addition to two GEMMs: a chunk into four gates, for 4 pairs of calls, five unary pointwise ops, for 5 more, and five binary pointwise ops, for another 5) are much slower than they are in Theano and TensorFlow.
Also, there are excessive cudaGetDevice calls for each pointwise operation (9 cudaGetDevice) even without chunking, as in the following repro code: As for small unrolled LSTMs it does show up.
This should no longer be an issue, now that is implemented as a native ATen function. I think we're still using the Python version for tensors, but this will be resolved soon when we stop using tensors altogether.
Change:
<ins> import os this_file = os.path.dirname(os.path.abspath(__file__)) generated_dir = os.path.abspath(os.path.join(this_file, '..', '..', 'torch', 'csrc', 'generated')) line_start = '//generic_include ' types = [ 'Double', 'Float', 'Half', 'Long', 'Int', 'Short', 'Char', 'Byte' ] generic_include = '#define {lib}_GENERIC_FILE "{path}"' generate_include = '#include "{lib}/{lib}Generate{type}Type.h"' def split_types(file_name): assert file_name.startswith('torch/csrc/') if not os.path.exists(generated_dir): os.makedirs(generated_dir) with open(file_name, 'r') as f: lines = f.read().split('n') # Find //generic_include for i, l in enumerate(lines): if l.startswith(line_start): args = l[len(line_start):] lib_prefix, generic_file = filter(bool, args.split()) break else: raise RuntimeError("generic include not found") gen_name_prefix = file_name[len('torch/csrc/'):].replace('/', '_').replace('.cpp', '') gen_path_prefix = os.path.join(generated_dir, gen_name_prefix) prefix = 'n'.join(lines[:i]) suffix = 'n'.join(lines[i + 1:]) to_build = [] g_include = generic_include.format(lib=lib_prefix, path=generic_file) for t in types: t_include = generate_include.format(lib=lib_prefix, type=t) gen_path = gen_path_prefix + t + '.cpp' to_build.append(gen_path) with open(gen_path, 'w') as f: f.write(prefix + 'n' + g_include + 'n' + t_include + 'n' + suffix) return to_build </ins>

Discussion:
Trying to run on github actions via AWS instance gives me: I will of course set up now, but it shouldn't crash like that. May be it should use as a fallback? pt-2.0 cc
Is this a multitenant setup? Also curious which distro are you running, I was under the impression that most linux distros set home by default
this is AWS image. one default user. I think it's user, but it wants you to run as root, which is weird. I have manually now set to overcome this.
We should update triton to use pathlib or something
Change:
self.add_module(str(idx), module) def __getitem__(self, idx): <del> if not (-len(self) <= idx < len(self)): raise IndexError('index {} is out of range'.format(idx)) if idx < 0: idx += len(self) it = iter(self._modules.values()) for i in range(idx): next(it) return next(it) </del> <ins> if isinstance(idx, slice): return Sequential(OrderedDict(list(self._modules.items())[idx])) else: if not (-len(self) <= idx < len(self)): raise IndexError('index {} is out of range'.format(idx)) if idx < 0: idx += len(self) it = iter(self._modules.values()) for i in range(idx): next(it) return next(it) </ins> def __len__(self): return len(self._modules)

Discussion:
it'd be great to pick out elements of of a sequential with array slicing:
I agree this idea!
However,
this has come up again (last in ). Shall we reconsider?
I attempted getting a to work. Is this fine? If this is fine, I can issue a PR too. Let me know what you think. I believe this modification needs to be done in the class.
cc: This issue has been open for quite sometime.
Change:
cudnn=True, ), dict( <ins> fullname='Conv2d_groups_thnn', constructor=lambda: nn.Conv2d(4, 6, (3, 2), groups=2), input_size=(2, 4, 6, 5), ), dict( </ins> module_name='ConvTranspose2d', constructor_args=(3, 4, 3, (3, 2), 1, (1, 1)), cudnn=True,

Discussion:
x = Variable(((16, 64, 8, 8))) res = nn.Conv2d(16, 16, kernelsize=3, groups=2, bias=False)(x) results in: return ('updateoutput', input, weight, bias) File "", line 158, in thnn groupedargs += [group(t) for t in args] File "", line 153, in group dim = 0 if () == 1 else 1 AttributeError: 'NoneType' object has no attribute 'dim' The in line 158 is because there is no bias. I am running on CPU so this is in the code.
Change:
:members: :hidden:`InstanceNorm1d` <del> ~~~~~~~~~~~~~~~~~~~~~ </del> <ins> ~~~~~~~~~~~~~~~~~~~~~~~~ </ins> .. autoclass:: InstanceNorm1d :members: :hidden:`InstanceNorm2d` <del> ~~~~~~~~~~~~~~~~~~~~~ </del> <ins> ~~~~~~~~~~~~~~~~~~~~~~~~ </ins> .. autoclass:: InstanceNorm2d :members: :hidden:`InstanceNorm3d` <del> ~~~~~~~~~~~~~~~~~~~~~ </del> <ins> ~~~~~~~~~~~~~~~~~~~~~~~~ </ins> .. autoclass:: InstanceNorm3d :members:

Discussion:
why 1.21.6? because that's the oldest supported numpy on windows/python3.10: all cc
Here is the test for pinning numpy to 1.21.6 The reason its was not pinned to this version is that conda can't properly resolve all the dependencies at this step here:
It looks like you are correct, I will put up a PR to pin lower then already pinned:
Here is the PR:
Change:
for t in types: tensor = t() gpu_tensor = get_gpu_type(t)() <ins> # Default values desc = '' type_subset = types no_inplace = False </ins> if len(decl) == 3: name, constr, arg_constr = decl <del> desc = '' </del> elif len(decl) == 4: name, constr, arg_constr, desc = decl elif len(decl) == 5: name, constr, arg_constr, desc, type_subset = decl <del> if t not in type_subset: continue </del> <ins> elif len(decl) == 6: name, constr, arg_constr, desc, type_subset, no_inplace = decl if t not in type_subset: continue </ins> precision = custom_precision.get(name, TestCuda.precision) for inplace in (True, False): <ins> if inplace and no_inplace: continue </ins> if inplace: name_inner = name + '_' else:

Discussion:
On recent PyTorch master (built from source; 0.4.0a0+) on a P100, CUDA 9:
I think the results on CPU meets the defination of triu (upper triangular part of the matrix). It is really hard to handle with a tensor from expand operation with multi-thread not only on GPU but also on CPU. The unexpected result will also happen on CPU if we apply openmp to it. I think it is necessary to eleminate the difference between the implementation and defination.
It doesn’t matter if it’s cpu or not. triu doesn’t have an underscore which means it doesn’t operate in-place and should always give the same result (in new memory)
Change:
RECORD_TRACE = CodeTemplate(""" if (jit::tracer::isTracing({ ${tensor_args} })) { <del> jit::Node *n = jit::tracer::recordTrace( "${api_name}", { ${tensor_args} }, ${return_name} ); </del> <ins> jit::Node *n = jit::tracer::recordTrace( "${api_name}", ${trace_inputs}, ${trace_outputs} ); </ins> ${record_attributes} } """)

Discussion:
when executing:
I'm seeing this as well as of last night. Its preceded by many warnings about : But the first error is the one about . I went back, and I get a good install as late as commit
Change:
#include "tuple_parser.h" <ins> #include <string> </ins> #include "torch/csrc/DynamicTypes.h" #include "torch/csrc/autograd/python_variable.h" #include "python_numbers.h"

Discussion:
Version include <stringin some cpp files: define order in The pre-defined struct is not passed by clang in macOS, so we should reorder them.
Simple fix
Change:
#ifndef TH_GENERIC_FILE #define TH_GENERIC_FILE "generic/Tensor.cpp" #else

Discussion:
When I don't have a working numpy dependency detected (due to unrelated shenanigans), the build fails with this: And then a big pile of related errors. CC
This is most likely the problem:
CC
Ok, should probably read
Should I go ahead and make the change? I'll also remove the at line 193. The rest looks ok to me.
go for it.
Actually, the change I proposed is incorrect (I misread the second defined), but I'm on it.
Change:
if (free_b) THTensor_(free)(b);), "gels", info,""); <del> /* rb__ is currently ldb by nrhs; resize it to n by nrhs */ rb__->size[0] = n; if (rb__ != rb_) </del> <ins> /* * In the m < n case, if the input b is used as the result (so b == _rb), * then rb_ was originally m by nrhs but now should be n by nrhs. * This is larger than before, so we need to expose the new rows by resizing. */ if (m < n && b == rb_) { </ins> THTensor_(resize2d)(rb_, n, nrhs); <ins> } </ins> THTensor_(freeCopyTo)(ra__, ra_); THTensor_(freeCopyTo)(rb__, rb_);

Discussion:
b = (600,1) A = (600,2) X, = (b,A) X.size() #torch.Size([2, 1]) Y, = (b.cuda(),A.cuda()) Y.size() #torch.Size([600, 1])
that is interesting. says that the first rows returned ( here) include the solution of the least-squares problem. I've verified that the first 2 rows of X and Y are the same so you can probably use that for now while we figure out what the rest of the output is.
Change:
gtest_reports_dir="${TEST_DIR}/cpp" pytest_reports_dir="${TEST_DIR}/python" <ins> # This is needed to work around ROCm using old docker images until # the transition to new images is complete. # TODO: Remove once ROCm CI is using new images. if [[ $BUILD_ENVIRONMENT == py3.6-devtoolset7-rocmrpm-centos* ]]; then # This file is sourced multiple times, only install conda the first time. # We must install conda where we have write access. CONDA_DIR="$ROOT_DIR/conda" if [[ ! -d $CONDA_DIR ]]; then ANACONDA_PYTHON_VERSION=3.6 BASE_URL="https://repo.anaconda.com/miniconda" CONDA_FILE="Miniconda3-latest-Linux-x86_64.sh" mkdir $CONDA_DIR pushd /tmp wget -q "${BASE_URL}/${CONDA_FILE}" chmod +x "${CONDA_FILE}" ./"${CONDA_FILE}" -b -f -p "$CONDA_DIR" popd export PATH="$CONDA_DIR/bin:$PATH" # Ensure we run conda in a directory that jenkins has write access to pushd $CONDA_DIR # Track latest conda update conda update -n base conda # Install correct Python version conda install python="$ANACONDA_PYTHON_VERSION" conda_install() { # Ensure that the install command don't upgrade/downgrade Python # This should be called as #   conda_install pkg1 pkg2 ... [-c channel] conda install -q -y python="$ANACONDA_PYTHON_VERSION" $* } # Install PyTorch conda deps, as per https://github.com/pytorch/pytorch README conda_install numpy pyyaml mkl mkl-include setuptools cffi typing future six # TODO: This isn't working atm conda_install nnpack -c killeent # Install some other packages # Need networkx 2.0 because bellmand_ford was moved in 2.1 . Scikit-image by # defaults installs the most recent networkx version, so we install this lower # version explicitly before scikit-image pulls it in as a dependency pip install networkx==2.0 # TODO: Why is scipy pinned # numba & llvmlite is pinned because of https://github.com/numba/numba/issues/4368 # scikit-learn is pinned because of # https://github.com/scikit-learn/scikit-learn/issues/14485 (affects gcc 5.5 # only) pip install --progress-bar off pytest scipy==1.1.0 scikit-learn==0.20.3 scikit-image librosa>=0.6.2 psutil numba==0.46.0 llvmlite==0.30.0 # click - onnx # hypothesis - tests # jupyter - for tutorials pip install --progress-bar off click hypothesis jupyter protobuf tabulate virtualenv mock typing-extensions popd else export PATH="$CONDA_DIR/bin:$PATH" fi fi </ins> # Figure out which Python to use PYTHON="$(which python)" if [[ "${BUILD_ENVIRONMENT}" =~ py((2|3).?[0-9]?.?[0-9]?) ]]; then

Discussion:
See e.g.: cc
Can we close it?
The build is passing now, although the tests might be failing (although it might have been an environment failure).