Repository: chainer/chainer

Issue:
Behavior of extract() in ResNet and VGG models is different in v1 and v2
In chainer v1, it was assumed that `extract()` methods in ``ResNetLayers`` and ``VGG16Layers`` extract feature vectors in *test* mode.
```python
import chainer.links as L
model = L.ResNet50Layers()
feature = mode.extract([image])  # it is equivalent to extract([image], test=True)
```

However, in v2 we have to explicitly use ``with chainer.use_config()`` clause like that:
```python
with chainer.use_config('train', False):
    feature = mode.extract([image])
```
This specification is a bit confusing; users might expect that ``extract()`` literally extracts feature vectors regardless of the training mode, but it is not correct. Although I do not think that it is right thing to change this specification from now on, it would be better to add the above description to the comment.


Discussion:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 30 days if no further activity occurs. Thank you for your contributions.

The other solution would be to turn off `train` mode  in `extract` method forcibly. Do you think it restricts the usage of the method?
This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 30 days if no further activity occurs. Thank you for your contributions.

bump
If we are to fix this, we should include this in v4 release, as this is obviously no-compat.
I discussed this issue with @rezoo and want to propose the followings:
- Do not change the default value of the training mode from the current implementation. Instead, write explicitly in the document of `extract` that the behavior has been changed from v2.
- Check the value of `chainer.config.train` in `extract` and raise an warning when it is `True` possibly in debug mode. (This might be inappropriate because we do want to set it to `True` in some case (e.g. test-time dropout).

@kmaehashi  @rezoo How do you think of the proposal?


Resolving patch:
diff --git a/chainer/links/model/vision/googlenet.py b/chainer/links/model/vision/googlenet.py
--- a/chainer/links/model/vision/googlenet.py
+++ b/chainer/links/model/vision/googlenet.py
@@ -252,14 +252,37 @@ def extract(self, images, layers=None, size=(224, 224), **kwargs):
         it is also interpreted as a shortcut method that implicitly calls
         ``prepare`` and ``__call__`` functions.
 
+        Unlike ``predict`` method, this method does not override
+        ``chainer.config.train`` and ``chainer.config.enable_backprop``
+        configuration. If you want to extract features without updating
+        model parameters, you need to manually set configuration when
+        calling this method as follows:
+
+         .. code-block:: python
+
+             # model is an instance of `GoogLeNet`
+             with chainer.using_config('train', False):
+                 with chainer.using_config('enable_backprop', False):
+                     feature = model.extract([image])
+
         .. warning::
 
-           ``train`` and ``volatile`` arguments are not supported anymore since
-           v2.
-           Instead, use ``chainer.using_config('train', train)`` and
-           ``chainer.using_config('enable_backprop', not volatile)``
-           respectively.
-           See :func:`chainer.using_config`.
+           ``train`` and ``volatile`` arguments are not supported
+           anymore since v2. Instead, users should configure
+           training and volatile modes with ``train`` and
+           ``enable_backprop``, respectively.
+
+           Note that default behavior of this method is different
+           between v1 and later versions. Specifically,
+           the default values of ``train`` arguments in v1 were
+           ``False`` and ``OFF``, while that of
+           ``chainer.config.train`` are ``True``.
+           Therefore, users need to explicitly switch ``train``
+           to ``False`` to run the code in test mode to turn off
+           coputational graph construction.
+
+           See the `upgrade guide <https://docs.chainer.org/en/stable\
+           /upgrade_v2.html#training-mode-is-configured-by-a-thread-local-flag>`_.
 
         Args:
             images (iterable of PIL.Image or numpy.ndarray): Input images.
diff --git a/chainer/links/model/vision/resnet.py b/chainer/links/model/vision/resnet.py
--- a/chainer/links/model/vision/resnet.py
+++ b/chainer/links/model/vision/resnet.py
@@ -221,14 +221,37 @@ def extract(self, images, layers=None, size=(224, 224), **kwargs):
         it is also interpreted as a shortcut method that implicitly calls
         ``prepare`` and ``__call__`` functions.
 
+        Unlike ``predict`` method, this method does not override
+        ``chainer.config.train`` and ``chainer.config.enable_backprop``
+        configuration. If you want to extract features without updating
+        model parameters, you need to manually set configuration when
+        calling this method as follows:
+
+         .. code-block:: python
+
+             # model is an instance of ResNetLayers (50 or 101 or 152 layers)
+             with chainer.using_config('train', False):
+                 with chainer.using_config('enable_backprop', False):
+                     feature = model.extract([image])
+
         .. warning::
 
-           ``test`` and ``volatile`` arguments are not supported anymore since
-           v2.
-           Instead, use ``chainer.using_config('train', train)`` and
-           ``chainer.using_config('enable_backprop', not volatile)``
-           respectively.
-           See :func:`chainer.using_config`.
+           ``test`` and ``volatile`` arguments are not supported
+           anymore since v2. Instead, users should configure
+           training and volatile modes with ``train`` and
+           ``enable_backprop``, respectively.
+
+           Note that default behavior of this method is different
+           between v1 and later versions. Specifically,
+           the default values of ``test`` in v1 were ``True`` (test mode).
+           But that of ``chainer.config.train`` is also ``True``
+           (train mode). Therefore, users need to explicitly switch
+           ``train`` to ``False`` to run the code in test mode and
+           ``enable_backprop`` to ``False`` to turn off
+           coputational graph construction.
+
+           See the `upgrade guide <https://docs.chainer.org/en/stable\
+           /upgrade_v2.html#training-mode-is-configured-by-a-thread-local-flag>`_.
 
         Args:
             images (iterable of PIL.Image or numpy.ndarray): Input images.
diff --git a/chainer/links/model/vision/vgg.py b/chainer/links/model/vision/vgg.py
--- a/chainer/links/model/vision/vgg.py
+++ b/chainer/links/model/vision/vgg.py
@@ -215,14 +215,37 @@ def extract(self, images, layers=None, size=(224, 224), **kwargs):
         it is also interpreted as a shortcut method that implicitly calls
         ``prepare`` and ``__call__`` functions.
 
+        Unlike ``predict`` method, this method does not override
+        ``chainer.config.train`` and ``chainer.config.enable_backprop``
+        configuration. If you want to extract features without updating
+        model parameters, you need to manually set configuration when
+        calling this method as follows:
+
+         .. code-block:: python
+
+             # model is an instance of VGG16Layers
+             with chainer.using_config('train', False):
+                 with chainer.using_config('enable_backprop', False):
+                     feature = model.extract([image])
+
         .. warning::
 
-           ``test`` and ``volatile`` arguments are not supported anymore since
-           v2.
-           Instead, use ``chainer.using_config('train', train)`` and
-           ``chainer.using_config('enable_backprop', not volatile)``
-           respectively.
-           See :func:`chainer.using_config`.
+           ``test`` and ``volatile`` arguments are not supported
+           anymore since v2. Instead, users should configure
+           training and volatile modes with ``train`` and
+           ``enable_backprop``, respectively.
+
+           Note that default behavior of this method is different
+           between v1 and later versions. Specifically,
+           the default values of ``test`` in v1 were ``True`` (test mode).
+           But that of ``chainer.config.train`` is also ``True``
+           (train mode). Therefore, users need to explicitly switch
+           ``train`` to ``False`` to run the code in test mode and
+           ``enable_backprop`` to ``False`` to turn off
+           coputational graph construction.
+
+           See the `upgrade guide <https://docs.chainer.org/en/stable\
+           /upgrade_v2.html#training-mode-is-configured-by-a-thread-local-flag>`_.
 
         Args:
             images (iterable of PIL.Image or numpy.ndarray): Input images.


Test patch:
diff --git a/tests/chainer_tests/links_tests/model_tests/test_vision.py b/tests/chainer_tests/links_tests/model_tests/test_vision.py
--- a/tests/chainer_tests/links_tests/model_tests/test_vision.py
+++ b/tests/chainer_tests/links_tests/model_tests/test_vision.py
@@ -1,7 +1,9 @@
 import unittest
+import warnings
 
 import numpy
 
+import chainer
 from chainer.backends import cuda
 from chainer.links.model.vision import googlenet
 from chainer.links.model.vision import resnet
@@ -110,6 +112,47 @@ def test_extract_gpu(self):
         self.link.to_gpu()
         self.check_extract()
 
+    def check_warning_in_extract(self, debug, train):
+        # Compute running statistics of BatchNormalization
+        # layers to avoid NaN.
+        if debug and not train:
+            with chainer.using_config('debug', False):
+                with chainer.using_config('train', True):
+                    for _ in range(10):
+                        x1 = numpy.random.uniform(
+                            0, 255, (320, 240, 3)).astype(numpy.uint8)
+                        x2 = numpy.random.uniform(
+                            0, 255, (320, 240, 3)).astype(numpy.uint8)
+                        self.link.extract([x1, x2])
+
+        with numpy.errstate(all='ignore'):
+            with chainer.using_config('debug', debug):
+                with chainer.using_config('train', train):
+                    with warnings.catch_warnings(record=True) as w:
+                        x1 = numpy.random.uniform(
+                            0, 255, (320, 240, 3)).astype(numpy.uint8)
+                        x2 = numpy.random.uniform(
+                            0, 255, (320, 240, 3)).astype(numpy.uint8)
+                        self.link.extract([x1, x2])
+
+        if debug and train:
+            assert len(w) == 1
+            assert w[0].category is RuntimeWarning
+        else:
+            assert len(w) == 0
+
+    def test_warning_in_extract_0(self):
+        self.check_warning_in_extract(True, True)
+
+    def test_warning_in_extract_1(self):
+        self.check_warning_in_extract(True, False)
+
+    def test_warning_in_extract_2(self):
+        self.check_warning_in_extract(False, True)
+
+    def test_warning_in_extract_3(self):
+        self.check_warning_in_extract(False, False)
+
     def check_predict(self):
         x1 = numpy.random.uniform(0, 255, (320, 240, 3)).astype(numpy.uint8)
         x2 = numpy.random.uniform(0, 255, (320, 240)).astype(numpy.uint8)
@@ -225,6 +268,35 @@ def test_extract_gpu(self):
         self.link.to_gpu()
         self.check_extract()
 
+    def check_warning_in_extract(self, debug, train):
+        with numpy.errstate(all='ignore'):
+            with chainer.using_config('debug', debug):
+                with chainer.using_config('train', train):
+                    with warnings.catch_warnings(record=True) as w:
+                        x1 = numpy.random.uniform(
+                            0, 255, (320, 240, 3)).astype(numpy.uint8)
+                        x2 = numpy.random.uniform(
+                            0, 255, (320, 240, 3)).astype(numpy.uint8)
+                        self.link.extract([x1, x2])
+
+        if debug and train:
+            assert len(w) == 1
+            assert w[0].category is RuntimeWarning
+        else:
+            assert len(w) == 0
+
+    def test_warning_in_extract_0(self):
+        self.check_warning_in_extract(True, True)
+
+    def test_warning_in_extract_1(self):
+        self.check_warning_in_extract(True, False)
+
+    def test_warning_in_extract_2(self):
+        self.check_warning_in_extract(False, True)
+
+    def test_warning_in_extract_3(self):
+        self.check_warning_in_extract(False, False)
+
     def check_predict(self):
         x1 = numpy.random.uniform(0, 255, (320, 240, 3)).astype(numpy.uint8)
         x2 = numpy.random.uniform(0, 255, (320, 240)).astype(numpy.uint8)
@@ -358,6 +430,35 @@ def test_extract_gpu(self):
         self.link.to_gpu()
         self.check_extract()
 
+    def check_warning_in_extract(self, debug, train):
+        with numpy.errstate(all='ignore'):
+            with chainer.using_config('debug', debug):
+                with chainer.using_config('train', train):
+                    with warnings.catch_warnings(record=True) as w:
+                        x1 = numpy.random.uniform(
+                            0, 255, (320, 240, 3)).astype(numpy.uint8)
+                        x2 = numpy.random.uniform(
+                            0, 255, (320, 240, 3)).astype(numpy.uint8)
+                        self.link.extract([x1, x2])
+
+        if debug and train:
+            assert len(w) == 1
+            assert w[0].category is RuntimeWarning
+        else:
+            assert len(w) == 0
+
+    def test_warning_in_extract_0(self):
+        self.check_warning_in_extract(True, True)
+
+    def test_warning_in_extract_1(self):
+        self.check_warning_in_extract(True, False)
+
+    def test_warning_in_extract_2(self):
+        self.check_warning_in_extract(False, True)
+
+    def test_warning_in_extract_3(self):
+        self.check_warning_in_extract(False, False)
+
     def check_predict(self):
         x1 = numpy.random.uniform(0, 255, (320, 240, 3)).astype(numpy.uint8)
         x2 = numpy.random.uniform(0, 255, (320, 240)).astype(numpy.uint8)