internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: chainer/chainer

Issue:
`Optimizer` runs unnecessary state array conversion
#6171 has been introduced for the specific `Optimizer` which replaces its state arrays such as `SMORMS3`. But it is also executed for every `Optimizer` which have `states` attribute (e.g. `Adam`) and generates additional overhead.


Discussion:
The comparison with `arr` should be replaced with `st` at line 295.
Oops..., not `st` but `self.state[state_name]` at line 297, before calling `update_core()`.

Resolving patch:
diff --git a/chainer/optimizer.py b/chainer/optimizer.py
--- a/chainer/optimizer.py
+++ b/chainer/optimizer.py
@@ -294,8 +294,9 @@ def update_core_chainerx(self, param):
         for state_name, st in self.state.items():
             st = self.state[state_name]
             if isinstance(st, chainerx.ndarray):
-                self.state[state_name] = backend.from_chainerx(st)
-                chainerx_state_arrays[state_name] = st
+                fallback_arr = backend.from_chainerx(st)
+                self.state[state_name] = fallback_arr
+                chainerx_state_arrays[state_name] = (st, fallback_arr)
 
         # Create a temporary parameter with memory-shared NumPy/CuPy array
         # If the ChainerX parameter has a cached NumPy/CuPy copy, use the
@@ -316,9 +317,12 @@ def update_core_chainerx(self, param):
         update_core(temp_param)
 
         # Restore state arrays
-        for state_name, arr in chainerx_state_arrays.items():
+        for state_name, (arr, fallback_arr) in chainerx_state_arrays.items():
             cur_arr = self.state[state_name]
-            if cur_arr is not arr:
+            if cur_arr is not fallback_arr:
+                # The optimizer altered the reference of the state, instead of
+                # updating it in-place. We need to convert the new state back
+                # to ChainerX.
                 arr = backend.to_chainerx(cur_arr)
             self.state[state_name] = arr
 


Test patch:
Repository: chainer/chainer

Issue:
test_backprop.py does not pass in debug mode
Version: current master 03157f852002dcdcb04e03e6da03315976ce7434

```
$ CHAINER_DEBUG=1 pytest tests/chainer_tests/test_backprop.py
```

#### Result

```
E   AttributeError: 'NoneType' object has no attribute 'dtype'
 TestFunctionOutputNone_use_chainerx_false__chainerx_device_None__use_cuda_true__cuda_device_0__use_cudnn_never__cudnn_deterministic_false__autotune_false__cudnn_fast_batch_normalization_false__use_ideep_never_param_1_{function_node=False}.
test_backprop
tests/chainer_tests/test_backprop.py:483: in test_backprop
    y1, y2, y3 = self.call_func(backend_config, x1)
tests/chainer_tests/test_backprop.py:472: in call_func
    return self.call_func_function(backend_config, x1)
tests/chainer_tests/test_backprop.py:422: in call_func_function
    return Func()(x1)
chainer/function.py:269: in __call__
    ret = node.apply(inputs)
chainer/function_node.py:343: in apply
    for out in outputs):
chainer/function_node.py:343: in <genexpr>
    for out in outputs):
chainer/backend.py:31: in _contains_nan
    if x.dtype.kind in ('f', 'c'):
E   AttributeError: Parameterized test failed.
E
E   Base test method: TestFunctionOutputNone_use_chainerx_false__chainerx_device_None__use_cuda_true__cuda_device_0__use_cudnn_never__cudnn_deterministic_false__autotune_false__cudnn_fast_batch_normalization_false__use_ideep_never.test_back
prop
E   Test parameters:
E     function_node: False
E
E
E   (caused by)
E   AttributeError: 'NoneType' object has no attribute 'dtype'
```


Discussion:

Resolving patch:
diff --git a/chainer/function_node.py b/chainer/function_node.py
--- a/chainer/function_node.py
+++ b/chainer/function_node.py
@@ -339,11 +339,11 @@ def apply(self, inputs):
 
         # NaN check of output values
         if is_debug:
-            if any(chainer.backend._contains_nan(out)
-                   for out in outputs):
-                msg = ('NaN is detected on forward computation of '
-                       '{}'.format(self.label))
-                raise RuntimeError(msg)
+            for out in outputs:
+                if out is not None and chainer.backend._contains_nan(out):
+                    msg = ('NaN is detected on forward computation of '
+                           '{}'.format(self.label))
+                    raise RuntimeError(msg)
 
         self._output_count = len(outputs)
 


Test patch:
Repository: chainer/chainer

Issue:
Validation loss and perplexity not printed in seq2seq example
With the fix #6091, `validation/main/bleu` is now printed, but `validation/main/loss` and `validation/main/perp` are not.

```
epoch       iteration   main/loss   validation/main/loss  main/perp   validation/main/perp  validation/main/bleu  elapsed_time
0           10          34.986                            1114.84                                                 3.61583       
1           20          3.3503                            1.19844                           0.178536              16.5124       
# source : <UNK> <UNK> <UNK> <UNK> <UNK>
# result : <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UN
K> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK>
 <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK>
# expect : <UNK> <UNK> <UNK> <UNK> <UNK> <UNK> <UNK>
1           30          2.78501                           1.16146                                                 19.6621       
2           40          2.65923                           1.15034                           0.178536              32.4583       
```


Discussion:
These columns can simply be removed.

Resolving patch:
diff --git a/examples/seq2seq/seq2seq.py b/examples/seq2seq/seq2seq.py
--- a/examples/seq2seq/seq2seq.py
+++ b/examples/seq2seq/seq2seq.py
@@ -371,9 +371,8 @@ def _filter_func(s, t):
     trainer.extend(extensions.LogReport(
         trigger=(args.log_interval, 'iteration')))
     trainer.extend(extensions.PrintReport(
-        ['epoch', 'iteration', 'main/loss', 'validation/main/loss',
-         'main/perp', 'validation/main/perp', 'validation/main/bleu',
-         'elapsed_time']),
+        ['epoch', 'iteration', 'main/loss', 'main/perp',
+         'validation/main/bleu', 'elapsed_time']),
         trigger=(args.log_interval, 'iteration'))
 
     if args.validation_source and args.validation_target:


Test patch:
Repository: chainer/chainer

Issue:
[document] The input argument name in `F.rrelu` is inconsistent
https://docs.chainer.org/en/stable/reference/generated/chainer.functions.rrelu.html#chainer.functions.rrelu

The slope for x < 0 is named `a` in the body text, but `r` is used for the input argument.

> Randomized Leaky Rectified Liner Unit function.
> This function is expressed as
> f(x)=max(x,ax),
> where a is a random number sampled from a uniform distribution U(l,u).
> ...
> ・r (N-dimensional array or None) – The r to be used for rrelu. The shape and dtype must be the same as x[0] and should be on the same device. If r is not specified or set to None, a r will be generated randomly according to the given l and u. If r is specified, l and u will be ignored.



Discussion:
According to the original paper (https://arxiv.org/pdf/1505.00853.pdf), `a` is the correct one.

However, if the name of the input argument is modified, the users may have to amend their code.
I don't have an idea which one should be fixed...
We discussed this issue and concluded that:
* Keep name of the input argument as is.
* Replace `a` with `r` in the body text.
* Add notes like ``This value corresponds to `a` in the original paper.`` to the description of `r`.

@fiarabbit Do you have interest in making a PR?

Resolving patch:
diff --git a/chainer/functions/activation/rrelu.py b/chainer/functions/activation/rrelu.py
--- a/chainer/functions/activation/rrelu.py
+++ b/chainer/functions/activation/rrelu.py
@@ -96,12 +96,15 @@ def rrelu(x, l=1. / 8, u=1. / 3, **kwargs):
 
     This function is expressed as
 
-    .. math:: f(x)=\\max(x, ax),
+    .. math:: f(x)=\\max(x, rx),
 
-    where :math:`a` is a random number sampled from a uniform distribution
+    where :math:`r` is a random number sampled from a uniform distribution
     :math:`U(l, u)`.
 
-    See: https://arxiv.org/pdf/1505.00853.pdf
+    .. note::
+
+        The :math:`r` corresponds to :math:`a` in the original
+        paper (https://arxiv.org/pdf/1505.00853.pdf).
 
     Args:
         x (:class:`~chainer.Variable` or :ref:`ndarray`):
@@ -112,7 +115,7 @@ def rrelu(x, l=1. / 8, u=1. / 3, **kwargs):
             The r to be used for rrelu.
             The shape and dtype must be the same as ``x[0]`` and should be on
             the same device.
-            If ``r``  is not specified or set to ``None``, a ``r`` will be
+            If ``r``  is not specified or set to ``None``, an ``r`` will be
             generated randomly according to the given ``l`` and ``u``.
             If ``r`` is specified, ``l`` and ``u`` will be ignored.
         return_r (bool):


Test patch:
Repository: chainer/chainer

Issue:
Support ChainerX in F.GetItem backward
`GetItemGrad` does not suport it yet.

Related: #5944


Discussion:

Resolving patch:
diff --git a/chainer/functions/array/get_item.py b/chainer/functions/array/get_item.py
--- a/chainer/functions/array/get_item.py
+++ b/chainer/functions/array/get_item.py
@@ -56,19 +56,23 @@ def __init__(self, slices, in_shape):
         self._in_shape = in_shape
 
     def forward(self, inputs):
+        slices = tuple([
+            backend.from_chx(s) if isinstance(s, chainerx.ndarray) else s
+            for s in self.slices])
+
         gy, = inputs
         xp = backend.get_array_module(*inputs)
         gx = xp.zeros(self._in_shape, gy.dtype)
         if xp is numpy:
             try:
-                numpy.add.at(gx, self.slices, gy)
+                numpy.add.at(gx, slices, gy)
             except IndexError:
                 done = False
                 # In numpy<1.13, 0-dim boolean index is not supported in
                 # numpy.add.at and it's supported for 0-dim arr in
                 # arr.__getitem__.
-                if not _numpy_supports_0d_bool_index and len(self.slices) == 1:
-                    idx = numpy.asanyarray(self.slices[0])
+                if not _numpy_supports_0d_bool_index and len(slices) == 1:
+                    idx = numpy.asanyarray(slices[0])
                     if idx.dtype == numpy.dtype(bool):
                         # Convert the array and the mask to 1-dim.
                         # numpy.add.at with them is supported in older numpy.
@@ -86,7 +90,7 @@ def forward(self, inputs):
 '''
                     raise IndexError(msg)
         else:
-            gx.scatter_add(self.slices, inputs[0])
+            gx.scatter_add(slices, inputs[0])
         return gx,
 
     def backward(self, indexes, ggx):


Test patch:
diff --git a/tests/chainer_tests/functions_tests/array_tests/test_get_item.py b/tests/chainer_tests/functions_tests/array_tests/test_get_item.py
--- a/tests/chainer_tests/functions_tests/array_tests/test_get_item.py
+++ b/tests/chainer_tests/functions_tests/array_tests/test_get_item.py
@@ -20,10 +20,9 @@
     + [{'use_cuda': True}]
     # ChainerX tests
     + [
-        # TODO(niboshi): Add the following configurations
-        # {'use_chainerx': True, 'chainerx_device': 'native:0'},
-        # {'use_chainerx': True, 'chainerx_device': 'cuda:0'},
-        # {'use_chainerx': True, 'chainerx_device': 'cuda:1'},
+        {'use_chainerx': True, 'chainerx_device': 'native:0'},
+        {'use_chainerx': True, 'chainerx_device': 'cuda:0'},
+        {'use_chainerx': True, 'chainerx_device': 'cuda:1'},
     ])
Repository: chainer/chainer

Issue:
Emit warning if compiled in Debug mode
In debug mode ChainerX runs significantly slower.
However sometimes it's difficult notice that.


Discussion:

Resolving patch:
diff --git a/chainerx/__init__.py b/chainerx/__init__.py
--- a/chainerx/__init__.py
+++ b/chainerx/__init__.py
@@ -1,5 +1,6 @@
 import os
 import sys
+import warnings
 
 
 if sys.version_info[0] < 3:
@@ -70,3 +71,9 @@ def __init__(self, *args, **kwargs):
 
 def is_available():
     return _available
+
+
+if _available and _core._is_debug():
+    # Warn if the ChainerX core binary is built in debug mode
+    warnings.warn(
+        'ChainerX core binary is built in debug mode.', stacklevel=2)


Test patch:
Repository: chainer/chainer

Issue:
Why `F.dropout` usually does not use CuDNN?
Currently, `F.dropout` uses cudnn only when `using_config('use_cudnn', 'always')`
https://github.com/chainer/chainer/blob/d7adbfb45d46fafa6bd0bbb9f0cebe5894489437/chainer/functions/noise/dropout.py#L46

But IMO, cudnn should be enabled when `>=auto`.


Discussion:
It seems it's ok to enable it by default, but I'd like to see the benchmark result first.

@bonprosoft Let me know if you have any special reason to use `==always` in #3369. 
I don't have any special reason for the code.
I think it is OK to use cuDNN dropout if it doesn't decrease performance.
I confirmed that enabling cuDNN is better.

```
Forward
=========== =============
  backend                
----------- -------------
    gpu      7.29±0.02ms 
 gpu-cudnn   2.64±0.04ms 
    cpu       1.49±0.1s  
 cpu-ideep     80.6±4ms  
=========== =============

Backward
=========== =============
  backend                
----------- -------------
    gpu      3.16±0.03ms 
 gpu-cudnn   2.68±0.02ms 
    cpu       109±0.8ms  
 cpu-ideep     47.1±3ms  
=========== =============
```
(https://github.com/chainer/chainer-benchmark/pull/31)

Resolving patch:
diff --git a/chainer/functions/noise/dropout.py b/chainer/functions/noise/dropout.py
--- a/chainer/functions/noise/dropout.py
+++ b/chainer/functions/noise/dropout.py
@@ -44,7 +44,7 @@ def forward_cpu(self, x):
         return y,
 
     def forward_gpu(self, x):
-        if (chainer.should_use_cudnn('==always', 5000)
+        if (chainer.should_use_cudnn('>=auto', 5000)
                 and x[0].flags.c_contiguous
                 and self.mask is None
                 and not self.return_mask):
@@ -83,7 +83,7 @@ def _forward_ideep(self, x):
         return y,
 
     def backward(self, x, gy):
-        if chainer.should_use_cudnn('==always', 5000) and self._use_cudnn:
+        if chainer.should_use_cudnn('>=auto', 5000) and self._use_cudnn:
             return DropoutGradCuDNN(self.states, self.dropout_ratio).apply(gy)
         else:
             return DropoutGrad(self.mask).apply(gy)


Test patch:
diff --git a/tests/chainer_tests/functions_tests/noise_tests/test_dropout.py b/tests/chainer_tests/functions_tests/noise_tests/test_dropout.py
--- a/tests/chainer_tests/functions_tests/noise_tests/test_dropout.py
+++ b/tests/chainer_tests/functions_tests/noise_tests/test_dropout.py
@@ -8,7 +8,6 @@
 from chainer import gradient_check
 from chainer import testing
 from chainer.testing import attr
-from chainer.testing import backend
 
 
 @testing.parameterize(
@@ -17,7 +16,7 @@
     {'dtype': numpy.float64, 'ratio': 0.5},
     {'dtype': numpy.float64, 'ratio': 0.0},
 )
-@backend.inject_backend_tests(
+@testing.inject_backend_tests(
     ['test_forward', 'test_backward', 'test_double_backward',
      'test_immutable'],
     # CPU tests
Repository: chainer/chainer

Issue:
ChainerX example support: image_captioning: train.py



Discussion:

Resolving patch:
diff --git a/examples/image_captioning/train.py b/examples/image_captioning/train.py
--- a/examples/image_captioning/train.py
+++ b/examples/image_captioning/train.py
@@ -37,12 +37,24 @@ def main():
     parser.add_argument('--rnn', type=str, default='nsteplstm',
                         choices=['nsteplstm', 'lstm'],
                         help='Language model layer type')
-    parser.add_argument('--gpu', type=int, default=0,
-                        help='GPU ID (negative value indicates CPU)')
+    parser.add_argument('--device', '-d', type=str, default='-1',
+                        help='Device specifier. Either ChainerX device '
+                        'specifier or an integer. If non-negative integer, '
+                        'CuPy arrays with specified device id are used. If '
+                        'negative integer, NumPy arrays are used')
     parser.add_argument('--max-caption-length', type=int, default=30,
                         help='Maxium caption length when using LSTM layer')
+    group = parser.add_argument_group('deprecated arguments')
+    group.add_argument('--gpu', '-g', dest='device',
+                       type=int, nargs='?', const=0,
+                       help='GPU ID (negative value indicates CPU)')
     args = parser.parse_args()
 
+    device = chainer.get_device(args.device)
+
+    print('Device: {}'.format(device))
+    print()
+
     # Load the MSCOCO dataset. Assumes that the dataset has been downloaded
     # already using e.g. the `download.py` script
     train, val = datasets.get_mscoco(args.mscoco_root)
@@ -60,10 +72,7 @@ def main():
     # NStepLSTM layers
     model = ImageCaptionModel(
         vocab_size, dropout_ratio=args.dropout_ratio, rnn=args.rnn)
-
-    if args.gpu >= 0:
-        chainer.backends.cuda.get_device_from_id(args.gpu).use()
-        model.to_gpu()
+    model.to_device(device)
 
     def transform(in_data):
         # Called for each sample and applies necessary preprocessing to the
@@ -85,6 +94,7 @@ def transform(in_data):
     optimizer = optimizers.Adam()
     optimizer.setup(model)
 
+    @chainer.dataset.converter()
     def converter(batch, device):
         # The converted receives a batch of input samples any may modify it if
         # necessary. In our case, we need to align the captions depending on if
@@ -99,7 +109,7 @@ def converter(batch, device):
             batch, device, max_caption_length=max_caption_length)
 
     updater = training.updater.StandardUpdater(
-        train_iter, optimizer=optimizer, device=args.gpu, converter=converter)
+        train_iter, optimizer=optimizer, device=device, converter=converter)
 
     trainer = training.Trainer(
         updater, out=args.out, stop_trigger=(args.max_iters, 'iteration'))
@@ -108,7 +118,7 @@ def converter(batch, device):
             val_iter,
             target=model,
             converter=converter,
-            device=args.gpu
+            device=device,
         ),
         trigger=(args.val_iter, 'iteration')
     )


Test patch:
Repository: chainer/chainer

Issue:
VAE example does not work with ChainerX
Version: latest master(ac4e62d3d5c53069d1f4a224989396a28b19c3d8)
Running VAE example with a ChainerX device results in an error.

```
$ ./train_vae.py --device native:0
...
  File "/home/niboshi/w/repos/chainer/chainer/distributions/bernoulli.py", line 51, in backward
    nan = xp.array(xp.nan).astype(dlogit.dtype)
AttributeError: module 'chainerx' has no attribute 'nan'
```


Discussion:

Resolving patch:
diff --git a/chainerx/__init__.py b/chainerx/__init__.py
--- a/chainerx/__init__.py
+++ b/chainerx/__init__.py
@@ -15,6 +15,12 @@
 
 if _available:
     from numpy import dtype  # NOQA
+    from numpy import (  # NOQA
+        Inf, Infinity, NAN, NINF, NZERO, NaN, PINF, PZERO,
+        e, euler_gamma,
+        inf, infty, nan,
+        newaxis,
+        pi)
     from numpy import (
         bool_, int8, int16, int32, int64, uint8, float16, float32, float64)  # NOQA
     all_dtypes = (


Test patch:
diff --git a/tests/chainerx_tests/unit_tests/test_constants.py b/tests/chainerx_tests/unit_tests/test_constants.py
new file mode 100644
--- /dev/null
+++ b/tests/chainerx_tests/unit_tests/test_constants.py
@@ -0,0 +1,21 @@
+import numpy
+
+import chainerx
+
+
+def test_constants():
+    assert chainerx.Inf is numpy.Inf
+    assert chainerx.Infinity is numpy.Infinity
+    assert chainerx.NAN is numpy.NAN
+    assert chainerx.NINF is numpy.NINF
+    assert chainerx.NZERO is numpy.NZERO
+    assert chainerx.NaN is numpy.NaN
+    assert chainerx.PINF is numpy.PINF
+    assert chainerx.PZERO is numpy.PZERO
+    assert chainerx.e is numpy.e
+    assert chainerx.euler_gamma is numpy.euler_gamma
+    assert chainerx.inf is numpy.inf
+    assert chainerx.infty is numpy.infty
+    assert chainerx.nan is numpy.nan
+    assert chainerx.newaxis is numpy.newaxis
+    assert chainerx.pi is numpy.pi
Repository: chainer/chainer

Issue:
Improve error message when ChainerX is unavailable
When a user attempt to use `cuda:0` device, ChainerX says the device specifier is invalid if `chainerx.is_available()` is `False`. It seems a bit difficult to deduce the actual problem from the message.

```
$ python train_mnist.py -d cuda:0
Traceback (most recent call last):
  File "train_mnist.py", line 134, in <module>
    main()
  File "train_mnist.py", line 56, in main
    device = chainer.get_device(args.device)
  File "/path/to/chainer/chainer/backend.py", line 157, in get_device
    raise ValueError('Invalid device specifier: {}'.format(device_spec))
ValueError: Invalid device specifier: cuda:0
```


Discussion:

Resolving patch:
diff --git a/chainer/backend.py b/chainer/backend.py
--- a/chainer/backend.py
+++ b/chainer/backend.py
@@ -154,7 +154,16 @@ def get_device(device_spec):
                 if not colon:
                     return intel64.Intel64Device()
 
-        elif chainerx.is_available():
+        else:
+            # String device specifier without '@' prefix is assumed to be a
+            # ChainerX device.
+            if not chainerx.is_available():
+                raise RuntimeError(
+                    'Tried to parse ChainerX device specifier \'{}\', '
+                    'but ChainerX is not available. '
+                    'Note that device specifiers without \'@\' prefix are '
+                    'assumed to be ChainerX device '
+                    'specifiers.'.format(device_spec))
             return _chainerx.ChainerxDevice(chainerx.get_device(device_spec))
 
     raise ValueError('Invalid device specifier: {}'.format(device_spec))


Test patch:
diff --git a/tests/chainer_tests/test_backend.py b/tests/chainer_tests/test_backend.py
--- a/tests/chainer_tests/test_backend.py
+++ b/tests/chainer_tests/test_backend.py
@@ -279,6 +279,14 @@ def test_tuple_invalid(self):
         # tuple is no longer supported from Chainer
         self.check_invalid(('native', 0))
 
+    @unittest.skipIf(
+        chainerx.is_available(), 'Only tested when ChainerX is not built')
+    def test_chx_device_spec_without_chx_available(self):
+        # If chainerx is not available, get_device() with unprefixed string
+        # should mention ChainerX unavailability in the error message.
+        with pytest.raises(RuntimeError, match=r'.*ChainerX.*'):
+            chainer.get_device('foo')
+
 
 class TestDevice(unittest.TestCase):