Repository: huggingface/transformers (Python)
Date: 2021-10-04 15:07:51+00:00
Task type: Bug Fix

Issue:
Default arguments of clm example are confusing
I was having a look at the `run_clm.py` script and which new arguments are available to push to the hub.

```sh
python transformers\examples\pytorch\language-modeling\run_clm.py -h
```

I see the following options (note the True defaults for all):

```
  --no_keep_linebreaks  Whether to keep line breaks when using TXT files or not. (default: True)
  --keep_linebreaks [KEEP_LINEBREAKS]
                        Whether to keep line breaks when using TXT files or not. (default: True)

  --no_dataloader_pin_memory
                        Whether or not to pin memory for DataLoader. (default: True)
  --dataloader_pin_memory [DATALOADER_PIN_MEMORY]
                        Whether or not to pin memory for DataLoader. (default: True)

  --no_skip_memory_metrics
                        Whether or not to skip adding of memory profiler reports to metrics. (default: True)
  --skip_memory_metrics [SKIP_MEMORY_METRICS]
                        Whether or not to skip adding of memory profiler reports to metrics. (default: True)
```

From this, I cannot figure out what the default behaviour is or what I should change to become the expected behavior. I do not know what the use case is for this but it seems much better to only keep one of each option. If one the two for each option is deprecated, then that could be added in the description too.

I'm on current master (4.12 dev).

### Who can help

@sgugger, @patil-suraj




Issue discussion:
Unfortunately, since the two arguments are accepted, there is no way for us to automate a better documentation of them from the `HfArgumentParser` (if you have ideas, by all means!) so you should rely on the documentation of [`TrainingArguments`](https://huggingface.co/transformers/main_classes/trainer.html#trainingarguments).
I went looking for the `no_*` arguments. It seems that they are dynamically generated:

https://github.com/huggingface/transformers/blob/3a8de58c5192b620228128430ea52e6eda81c40a/src/transformers/hf_argparser.py#L112-L113

But I do not quite understand the use case for this. If the documentation only shows the version without `no_`, then why do they exist? Having two arguments for a boolean argument seems overkill.

That being said, I am sure there are reasons for that. My suggestion to make this more usable would be to negate the default value for the `no_` field. This doe snot change the default behaviour as far as I tested and makes it clear to the user what the default behavior is.

```
import argparse

cparser = argparse.ArgumentParser(formatter_class=argparse.ArgumentDefaultsHelpFormatter)

cparser.add_argument("--dataloader_pin_memory", default=True, action="store_true", help="Enable memory pinning for DataLoader")
cparser.add_argument("--no_dataloader_pin_memory", default=False, action="store_false", dest="dataloader_pin_memory", help="Disable memory pinning for DataLoader")

cargs = cparser.parse_args()

print(vars(cargs))
```

Help will look like this (with `False` on the no_ option):

```
optional arguments:
  -h, --help            show this help message and exit
  --dataloader_pin_memory
                        Enable memory pinning for DataLoader (default: True)
  --no_dataloader_pin_memory
                        Disable memory pinning for DataLoader (default: False)
```

Behaviour as before:
- default: {'dataloader_pin_memory': True}
- `--dataloader_pin_memory`: {'dataloader_pin_memory': True}
- `--no_dataloader_pin_memory`: {'dataloader_pin_memory': False}

The "whether or not" in the original help description may also be confusing. Because you generate the second field dynamically, you could go so far as to be consistent with your description and simply do `field_help.replace("Enable", "Disable)`.
Like I said, the `no-` are automagically generated by the `HfArgumentParser`. We can't remove them without creating a breakign change. At the same time there is no point in adding the `no-` argument to the `TrainingArguments` class (or other dataclasses) which can also be used as is in a notebook.
I think you misunderstood my reply. I am suggesting to change this default True:

https://github.com/huggingface/transformers/blob/3a8de58c5192b620228128430ea52e6eda81c40a/src/transformers/hf_argparser.py#L112-L113

into False

```
if field.default is True:
    parser.add_argument(f"--no_{field.name}", default=False, action="store_false", dest=field.name, **kwargs)
```

which as far I tested does not break anything as the result should be identical. But it has the changed bonus that the argparser --help is less ambiguous as it would have defaults dataloader_pin_memory: True, no_dataloader_pin_memory: False.
Let me double check, but that seems like a good change indeed. Thanks for explaining it to me!
Mmm, actually it looks like changing this `default` to `False` changes the default value in the argparser: tried to laundh the script with and without `--no_dataloader_pin_memory` and printed the value of `training_args.dataloader_pin_memory`. Currently we get False and True respectively (as it should).
With the changed of default you are suggesting, I always get False.
The reason that it is False is because of the order of the arguments. The `no_` variant is added to the argparser first (before the actual argument), therefore its defaults will get precedence down the line. I can make a suggestion in a PR to move things around? 

That would involve moving this line 

https://github.com/huggingface/transformers/blob/3a8de58c5192b620228128430ea52e6eda81c40a/src/transformers/hf_argparser.py#L112-L113

to after this line

https://github.com/huggingface/transformers/blob/3a8de58c5192b620228128430ea52e6eda81c40a/src/transformers/hf_argparser.py#L146

It is not visually as pleasing to repeat the if-clause but I'd argue that it could be worth it when documented well enough.
Oh the code of HfArgumentParser is not visually pleasing so that's not a problem ;-)
If you can suggest a PR, I'll test on the branch that everything is good with it.

Solution patch:
diff --git a/src/transformers/hf_argparser.py b/src/transformers/hf_argparser.py
--- a/src/transformers/hf_argparser.py
+++ b/src/transformers/hf_argparser.py
@@ -17,6 +17,7 @@
 import re
 import sys
 from argparse import ArgumentDefaultsHelpFormatter, ArgumentParser, ArgumentTypeError
+from copy import copy
 from enum import Enum
 from pathlib import Path
 from typing import Any, Iterable, List, NewType, Optional, Tuple, Union
@@ -101,6 +102,9 @@ def _add_dataclass_arguments(self, dtype: DataClassType):
                 ):
                     field.type = prim_type
 
+            # A variable to store kwargs for a boolean field, if needed
+            # so that we can init a `no_*` complement argument (see below)
+            bool_kwargs = {}
             if isinstance(field.type, type) and issubclass(field.type, Enum):
                 kwargs["choices"] = [x.value for x in field.type]
                 kwargs["type"] = type(kwargs["choices"][0])
@@ -109,8 +113,9 @@ def _add_dataclass_arguments(self, dtype: DataClassType):
                 else:
                     kwargs["required"] = True
             elif field.type is bool or field.type == Optional[bool]:
-                if field.default is True:
-                    parser.add_argument(f"--no_{field.name}", action="store_false", dest=field.name, **kwargs)
+                # Copy the currect kwargs to use to instantiate a `no_*` complement argument below.
+                # We do not init it here because the `no_*` alternative must be instantiated after the real argument
+                bool_kwargs = copy(kwargs)
 
                 # Hack because type=bool in argparse does not behave as we want.
                 kwargs["type"] = string_to_bool
@@ -145,6 +150,14 @@ def _add_dataclass_arguments(self, dtype: DataClassType):
                     kwargs["required"] = True
             parser.add_argument(field_name, **kwargs)
 
+            # Add a complement `no_*` argument for a boolean field AFTER the initial field has already been added.
+            # Order is important for arguments with the same destination!
+            # We use a copy of earlier kwargs because the original kwargs have changed a lot before reaching down
+            # here and we do not need those changes/additional keys.
+            if field.default is True and (field.type is bool or field.type == Optional[bool]):
+                bool_kwargs["default"] = False
+                parser.add_argument(f"--no_{field.name}", action="store_false", dest=field.name, **bool_kwargs)
+
     def parse_args_into_dataclasses(
         self, args=None, return_remaining_strings=False, look_for_args_file=True, args_filename=None
     ) -> Tuple[DataClass, ...]:


Test patch:
diff --git a/tests/test_hf_argparser.py b/tests/test_hf_argparser.py
--- a/tests/test_hf_argparser.py
+++ b/tests/test_hf_argparser.py
@@ -126,8 +126,10 @@ def test_with_default_bool(self):
 
         expected = argparse.ArgumentParser()
         expected.add_argument("--foo", type=string_to_bool, default=False, const=True, nargs="?")
-        expected.add_argument("--no_baz", action="store_false", dest="baz")
         expected.add_argument("--baz", type=string_to_bool, default=True, const=True, nargs="?")
+        # A boolean no_* argument always has to come after its "default: True" regular counter-part
+        # and its default must be set to False
+        expected.add_argument("--no_baz", action="store_false", default=False, dest="baz")
         expected.add_argument("--opt", type=string_to_bool, default=None)
         self.argparsersEqual(parser, expected)