Repository: codespell-project/codespell

Issue:
Multiple dictionaries (add default one) for `dictionary` config setting
we have 

```shell
❯ grep dictionary .codespellrc
dictionary = .codespell_dict
```
in https://github.com/bids-standard/bids-specification/blob/HEAD/.codespellrc but that disables default dictionaries, and I need to specify `-D-` to get them used:

```shell
❯ codespell
❯ codespell -D-
./DECISION-MAKING.md:17: Curent ==> Current
./src/common-principles.md:188: entites ==> entities
```

Unfortunately, `-D` is the different style than e.g. `--skip` which expects them comma separated and `-D` doesn't work that way:

```shell
❯ git diff
diff --git a/.codespellrc b/.codespellrc
index 28aeafee..a513a1a7 100644
--- a/.codespellrc
+++ b/.codespellrc
@@ -2,5 +2,5 @@
 skip = *.js,*.svg,*.eps,.git
 ignore-words-list = fo,te,als
 builtin = clear,rare
-dictionary = .codespell_dict
+dictionary = -,.codespell_dict
 exclude-file = src/CHANGES.md
❯ codespell
...
codespell: error: argument -D/--dictionary: expected one argument
```

so the question is how could I keep using default dictionaries + custom one while relying on specification in the config file?


Discussion:
Let's start with command line options, we'll look into configuration files later. The output of `--help` reads:
```
  -D DICTIONARY, --dictionary DICTIONARY
                        custom dictionary file that contains spelling corrections. If this flag is not specified or equals "-"
                        then the default dictionary is used. This option can be specified multiple times.
```
I would expect `-D - -D .codespell_dict` to work. Can you check it does?

If it does work, chances are this works in the configuration file:
```
dictionary = -
dictionary = .codespell_dict
```
Thank you @DimitriPapadopoulos for the rapid follow up -- sorry I missed it .

<details>
<summary>-D- -D .codespell_dict -- works!</summary> 

```shell
❯ codespell -D- -D .codespell_dict
./README.md:63: folders ==> directories
./README.md:64: sub-directory ==> subdirectory
./README.md:65: zumba ==> zebra
./DECISION-MAKING.md:17: Curent ==> Current
./src/common-principles.md:188: entites ==> entities
❯ codespell -D-
./DECISION-MAKING.md:17: Curent ==> Current
./src/common-principles.md:188: entites ==> entities
❯ cat .codespell_dict
sub-directory->subdirectory
sub-directories->subdirectories
file name->filename
file names->filenames
folder->directory
folders->directories
zumba->zebra

```
</details>
<details>

<summary>Two lines in config file -- does not!</summary> 

```shell
❯ grep dictionary .codespellrc
dictionary = -
dictionary = .codespell_dict
changes on filesystem:                                                                                                                                       
 .codespellrc | 1 +
❯ codespell --version
2.2.2
changes on filesystem:                                                                                                                                       
 .codespellrc | 1 +
❯ codespell
Traceback (most recent call last):
  File "/usr/bin/codespell", line 33, in <module>
    sys.exit(load_entry_point('codespell==0.0.0', 'console_scripts', 'codespell')())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/codespell_lib/_codespell.py", line 767, in _script_main
    return main(*sys.argv[1:])
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/codespell_lib/_codespell.py", line 772, in main
    options, parser = parse_options(args)
                      ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/codespell_lib/_codespell.py", line 427, in parse_options
    config.read(cfg_files)
  File "/usr/lib/python3.11/configparser.py", line 712, in read
    self._read(fp, filename)
  File "/usr/lib/python3.11/configparser.py", line 1111, in _read
    raise DuplicateOptionError(sectname, optname,
configparser.DuplicateOptionError: While reading from '.codespellrc' [line  6]: option 'dictionary' in section 'codespell' already exists

```
</details>
While `configparser` raises the `DuplicationError`, it also provides a _stict_ option to [customize parser behaviour](https://docs.python.org/3/library/configparser.html#customizing-parser-behaviour):
* _strict_, default value: `True`
  When set to `True`, the parser will not allow for any section or option duplicates while reading from a single source (using `read_file()`, `read_string()` or `read_dict()`). It is recommended to use strict parsers in new applications.

Since this is not a new application, I shall propose to set _strict_ to `False`, for consistency between commend line option `-D` and config file option `dictionary`.

We can think about adding support for multiple dictionaries separated by `,` in a later merge request.
no good. With

<details>
<summary>the patch adding strict=False</summary> 

```patch
diff --git a/codespell_lib/_codespell.py b/codespell_lib/_codespell.py
index a082e6d1..8f975179 100644
--- a/codespell_lib/_codespell.py
+++ b/codespell_lib/_codespell.py
@@ -541,7 +541,7 @@ def parse_options(
     cfg_files = ["setup.cfg", ".codespellrc"]
     if options.config:
         cfg_files.append(options.config)
-    config = configparser.ConfigParser(interpolation=None)
+    config = configparser.ConfigParser(interpolation=None, strict=False)
 
     # Read toml before other config files.
     toml_files = []
@@ -571,7 +571,7 @@ def parse_options(
     # Collect which config files are going to be used
     used_cfg_files = []
     for cfg_file in cfg_files:
-        _cfg = configparser.ConfigParser()
+        _cfg = configparser.ConfigParser(strict=False)
         _cfg.read(cfg_file)
         if _cfg.has_section("codespell"):
             used_cfg_files.append(cfg_file)

```
</details>

it no longer crashes but it just takes the 2nd value as the value, not a list of values (ie dictionaries).
Right, [configparser.ConfigParser.read_file](https://docs.python.org/3/library/configparser.html#configparser.ConfigParser.read_file) does not raise a `DuplicateOptionError` any more, but appears to be silently overwriting old values instead of creating a list of values. I cannot find a relevant option in [Customizing Parser Behaviour](https://docs.python.org/3/library/configparser.html#customizing-parser-behaviour), actually the documentation does not specify the behaviour at all.

I need to dig in `configparser`, but if the behaviour is not specified in the documentation, I suspect whatever solution I try will be fragile. See for example:
* [ConfigParser Duplicate keys in python](https://stackoverflow.com/questions/30095471/configparser-duplicate-keys-in-python)
* [Python issue 12662: Add support for duplicate options in configparser](https://bugs.python.org/issue12662)
@larsoner I see multiple issues in _codespell_ options:
1. Some command line options “_can be specified multiple times_” (`-D`/`--dictionary`), others expect “_comma-separated list of files_” (`-S`/`--skip`), others don't even bother to document how to specify multiple files (`-I`/`--ignore-words` and `-x`/`--exclude-file`), if possible at all.
2. In the case of `-D`/`--dictionary`, what works for command line options will not work in INI files: you cannot have multiple occurrences of the same key in INI files by default – or at least [configparser](https://docs.python.org/3/library/configparser.html) will at best take the last value into account and discard previous values.

Solutions:
1. We could modify the behaviour of [configparser](https://docs.python.org/3/library/configparser.html) to accept multiple occurrences of keys, using a custom `dict_type` for [configparser.ConfigParser](https://docs.python.org/3/library/configparser.html#configparser.ConfigParser). This would result in non-standard INI files, but command line options that “_can be specified multiple times_” could be used the same way in a config file.
2. We could modify all the above options that accept a file to accept a “_comma-separated list of files_”. Then command line options and config file options could be equivalent.

We could do both, of course.

Thoughts?
I prefer solution (2) to (1)
Solution 2 is being implemented in https://github.com/codespell-project/codespell/pull/2767.

Resolving patch:
diff --git a/codespell_lib/_codespell.py b/codespell_lib/_codespell.py
--- a/codespell_lib/_codespell.py
+++ b/codespell_lib/_codespell.py
@@ -390,10 +390,9 @@ def parse_options(
         "-D",
         "--dictionary",
         action="append",
-        help="custom dictionary file that contains spelling "
-        "corrections. If this flag is not specified or "
-        'equals "-" then the default dictionary is used. '
-        "This option can be specified multiple times.",
+        help="comma-separated list of custom dictionary files that "
+        "contain spelling corrections. If this flag is not specified "
+        'or equals "-" then the default dictionary is used.',
     )
     builtin_opts = "\n- ".join(
         [""] + [f"{d[0]!r} {d[1]}" for d in _builtin_dictionaries]
@@ -423,26 +422,26 @@ def parse_options(
         "-I",
         "--ignore-words",
         action="append",
-        metavar="FILE",
-        help="file that contains words that will be ignored "
-        "by codespell. File must contain 1 word per line."
-        " Words are case sensitive based on how they are "
-        "written in the dictionary file",
+        metavar="FILES",
+        help="comma-separated list of files that contain "
+        "words to be ignored by codespell. Files must contain "
+        "1 word per line. Words are case sensitive based on "
+        "how they are written in the dictionary file.",
     )
     parser.add_argument(
         "-L",
         "--ignore-words-list",
         action="append",
         metavar="WORDS",
-        help="comma separated list of words to be ignored "
+        help="comma-separated list of words to be ignored "
         "by codespell. Words are case sensitive based on "
-        "how they are written in the dictionary file",
+        "how they are written in the dictionary file.",
     )
     parser.add_argument(
         "--uri-ignore-words-list",
         action="append",
         metavar="WORDS",
-        help="comma separated list of words to be ignored "
+        help="comma-separated list of words to be ignored "
         "by codespell in URIs and emails only. Words are "
         "case sensitive based on how they are written in "
         'the dictionary file. If set to "*", all '
@@ -494,11 +493,13 @@ def parse_options(
     parser.add_argument(
         "-x",
         "--exclude-file",
+        action="append",
         type=str,
-        metavar="FILE",
-        help="ignore whole lines that match those "
-        "in the file FILE. The lines in FILE "
-        "should match the to-be-excluded lines exactly",
+        metavar="FILES",
+        help="ignore whole lines that match those in "
+        "the comma-separated list of files EXCLUDE. "
+        "The lines in these files should match the "
+        "to-be-excluded lines exactly",
     )
 
     parser.add_argument(
@@ -1166,20 +1167,22 @@ def main(*args: str) -> int:
     else:
         ignore_word_regex = None
 
-    ignore_words_files = options.ignore_words or []
     ignore_words, ignore_words_cased = parse_ignore_words_option(
         options.ignore_words_list
     )
-
-    for ignore_words_file in ignore_words_files:
-        if not os.path.isfile(ignore_words_file):
-            print(
-                f"ERROR: cannot find ignore-words file: {ignore_words_file}",
-                file=sys.stderr,
-            )
-            parser.print_help()
-            return EX_USAGE
-        build_ignore_words(ignore_words_file, ignore_words, ignore_words_cased)
+    if options.ignore_words:
+        ignore_words_files = flatten_clean_comma_separated_arguments(
+            options.ignore_words
+        )
+        for ignore_words_file in ignore_words_files:
+            if not os.path.isfile(ignore_words_file):
+                print(
+                    f"ERROR: cannot find ignore-words file: {ignore_words_file}",
+                    file=sys.stderr,
+                )
+                parser.print_help()
+                return EX_USAGE
+            build_ignore_words(ignore_words_file, ignore_words, ignore_words_cased)
 
     uri_regex = options.uri_regex or uri_regex_def
     try:
@@ -1196,7 +1199,7 @@ def main(*args: str) -> int:
         itertools.chain(*parse_ignore_words_option(options.uri_ignore_words_list))
     )
 
-    dictionaries = options.dictionary or ["-"]
+    dictionaries = flatten_clean_comma_separated_arguments(options.dictionary or ["-"])
 
     use_dictionaries = []
     for dictionary in dictionaries:
@@ -1258,7 +1261,9 @@ def main(*args: str) -> int:
 
     exclude_lines: Set[str] = set()
     if options.exclude_file:
-        build_exclude_hashes(options.exclude_file, exclude_lines)
+        exclude_files = flatten_clean_comma_separated_arguments(options.exclude_file)
+        for exclude_file in exclude_files:
+            build_exclude_hashes(exclude_file, exclude_lines)
 
     file_opener = FileOpener(options.hard_encoding_detection, options.quiet_level)
 


Test patch:
diff --git a/codespell_lib/tests/test_basic.py b/codespell_lib/tests/test_basic.py
--- a/codespell_lib/tests/test_basic.py
+++ b/codespell_lib/tests/test_basic.py
@@ -342,6 +342,20 @@ def test_ignore_dictionary(
     fname = tmp_path / "ignore.txt"
     fname.write_text("abandonned\nabilty\r\nackward")
     assert cs.main("-I", fname, bad_name) == 1
+    # missing file in ignore list
+    fname_missing = tmp_path / "missing.txt"
+    result = cs.main("-I", fname_missing, bad_name, std=True)
+    assert isinstance(result, tuple)
+    code, _, stderr = result
+    assert code == EX_USAGE
+    assert "ERROR:" in stderr
+    # comma-separated list of files
+    fname_dummy1 = tmp_path / "dummy1.txt"
+    fname_dummy1.touch()
+    fname_dummy2 = tmp_path / "dummy2.txt"
+    fname_dummy2.touch()
+    assert cs.main("-I", fname_dummy1, "-I", fname, "-I", fname_dummy2, bad_name) == 1
+    assert cs.main("-I", f"{fname_dummy1},{fname},{fname_dummy2}", bad_name) == 1
 
 
 def test_ignore_words_with_cases(
@@ -495,6 +509,13 @@ def test_exclude_file(
     )
     assert cs.main(bad_name) == 18
     assert cs.main("-x", fname, bad_name) == 1
+    # comma-separated list of files
+    fname_dummy1 = tmp_path / "dummy1.txt"
+    fname_dummy1.touch()
+    fname_dummy2 = tmp_path / "dummy2.txt"
+    fname_dummy2.touch()
+    assert cs.main("-x", fname_dummy1, "-x", fname, "-x", fname_dummy2, bad_name) == 1
+    assert cs.main("-x", f"{fname_dummy1},{fname},{fname_dummy2}", bad_name) == 1
 
 
 def test_encoding(