Repository: codespell-project/codespell

Issue:
[2.2.2 regression] Codespell tries and fails to decode and fix `.git\objects` files
Using Codespell 2.2.2 on Python 3.10 in a Git repository, Codespell errors with an permission error trying to modify Git objects:
```
PermissionError: [Errno 13] Permission denied: '.\\.git\\objects\\28\\719cdd2c5741cadfa2eede24b624249600f2aa'
```
This behaviour was not present in Codespell 2.2.1 (I tested that multiple times on multiple branches) so I think it's a 2.2.2 regression.

Full log:
```
(Py310) C:\Users\Ewout\Documents\GitHub\EMAworkbench>codespell . -w
.\CONTRIBUTING.md:24: wil ==> will, well
WARNING: Decoding file using encoding=utf-8 failed: .\.git\objects\00\a64bbac844bee64e9747d950e769870ac01fc8
WARNING: Trying next encoding iso-8859-1
WARNING: Decoding file using encoding=utf-8 failed: .\.git\objects\0c\d404eaacd02cd51a30438e14ecbadba2bc25d7
WARNING: Trying next encoding iso-8859-1
WARNING: Decoding file using encoding=utf-8 failed: .\.git\objects\0e\221584b2ca3efb1d0c227644f863571a195ce5
WARNING: Trying next encoding iso-8859-1
WARNING: Decoding file using encoding=utf-8 failed: .\.git\objects\0e\b5063aece7ad0f777839db61502047e92b982a
WARNING: Trying next encoding iso-8859-1
WARNING: Decoding file using encoding=utf-8 failed: .\.git\objects\0e\b88e5cd03acfd5ace58bf042313c7f22fbffda
WARNING: Trying next encoding iso-8859-1
WARNING: Decoding file using encoding=utf-8 failed: .\.git\objects\0e\dd7ff99fb009e57273889f9ec19ef612b4d271
WARNING: Trying next encoding iso-8859-1
WARNING: Decoding file using encoding=utf-8 failed: .\.git\objects\0f\eb1bea2475bb2b103d2499f28a4411154d336c
WARNING: Trying next encoding iso-8859-1
WARNING: Decoding file using encoding=utf-8 failed: .\.git\objects\17\ec18a5d54ae157814d94b5adb70a2545af6e37
WARNING: Trying next encoding iso-8859-1
WARNING: Decoding file using encoding=utf-8 failed: .\.git\objects\18\9acca87135099b8ec590204b8141b131b69796
WARNING: Trying next encoding iso-8859-1
WARNING: Decoding file using encoding=utf-8 failed: .\.git\objects\19\cba58fe9824793572e1e06a45ae98bc0fa8b06
WARNING: Trying next encoding iso-8859-1
WARNING: Decoding file using encoding=utf-8 failed: .\.git\objects\1f\8d3993763dd7f55d06edaaf7bef3750a8750c6
WARNING: Trying next encoding iso-8859-1
WARNING: Decoding file using encoding=utf-8 failed: .\.git\objects\25\5e3d6d9639dfe6fd4e797e1c63d59ba0522c2d
WARNING: Trying next encoding iso-8859-1
WARNING: Decoding file using encoding=utf-8 failed: .\.git\objects\27\471e6e6b17405d4bcac965e226e65bda20f70a
WARNING: Trying next encoding iso-8859-1
WARNING: Decoding file using encoding=utf-8 failed: .\.git\objects\27\f2cf2ad8af4f240709fe46980adf2718ef85d6
WARNING: Trying next encoding iso-8859-1
WARNING: Decoding file using encoding=utf-8 failed: .\.git\objects\28\719cdd2c5741cadfa2eede24b624249600f2aa
WARNING: Trying next encoding iso-8859-1
FIXED: .\.git\objects\28\719cdd2c5741cadfa2eede24b624249600f2aa
Traceback (most recent call last):
  File "C:\Users\Ewout\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\Ewout\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Users\Ewout\.virtualenvs\Py310\Scripts\codespell.exe\__main__.py", line 7, in <module>
  File "C:\Users\Ewout\.virtualenvs\Py310\lib\site-packages\codespell_lib\_codespell.py", line 767, in _script_main
    return main(*sys.argv[1:])
  File "C:\Users\Ewout\.virtualenvs\Py310\lib\site-packages\codespell_lib\_codespell.py", line 910, in main
    bad_count += parse_file(
  File "C:\Users\Ewout\.virtualenvs\Py310\lib\site-packages\codespell_lib\_codespell.py", line 760, in parse_file
    with codecs.open(filename, 'w', encoding=encoding) as f:
  File "C:\Users\Ewout\AppData\Local\Programs\Python\Python310\lib\codecs.py", line 905, in open
    file = builtins.open(filename, mode, buffering)
PermissionError: [Errno 13] Permission denied: '.\\.git\\objects\\28\\719cdd2c5741cadfa2eede24b624249600f2aa'
```



Discussion:
While I can reproduce this issue, I wonder whether it is really a regression. Indeed, I had similar errors with former versions of codespell, see #2189.

When just reading files, without`-w`:
```
$ mkdir -p foo/bar
$ touch foo/bar/file
$ chmod a= foo/bar/file
$ cd foo
$ codespell
Traceback (most recent call last):
  File "/home/username/.local/bin/codespell", line 8, in <module>
    sys.exit(_script_main())
  File "/my/path/codespell/codespell_lib/_codespell.py", line 767, in _script_main
    return main(*sys.argv[1:])
  File "/my/path/codespell/codespell_lib/_codespell.py", line 910, in main
    bad_count += parse_file(
  File "/my/path/codespell/codespell_lib/_codespell.py", line 651, in parse_file
    text = is_text_file(filename)
  File "/my/path/codespell/codespell_lib/_codespell.py", line 509, in is_text_file
    with open(filename, mode='rb') as f:
PermissionError: [Errno 13] Permission denied: './bar/file'
$ 
```
When attempting to fix files, with `-w`:
```
$ codespell -w
Traceback (most recent call last):
  File "/home/username/.local/bin/codespell", line 8, in <module>
    sys.exit(_script_main())
  File "/my/path/codespell/codespell_lib/_codespell.py", line 767, in _script_main
    return main(*sys.argv[1:])
  File "/my/path/codespell/codespell_lib/_codespell.py", line 910, in main
    bad_count += parse_file(
  File "/my/path/codespell/codespell_lib/_codespell.py", line 651, in parse_file
    text = is_text_file(filename)
  File "/my/path/codespell/codespell_lib/_codespell.py", line 509, in is_text_file
    with open(filename, mode='rb') as f:
PermissionError: [Errno 13] Permission denied: './bar/file'
$ 
```

Could it be that codespell used to skip `.git`, and now does not skip `.git`, for some reason?
As for the following warnings, they are indeed new and the result of 900f18654b570df28365d07a8908e7cf01917ac9, yet they are expected. It's just that the encoding detection code used to have a bug, and this bug has been fixed by 900f18654b570df28365d07a8908e7cf01917ac9.
```
WARNING: Decoding file using encoding=utf-8 failed: .\.git\objects\00\a64bbac844bee64e9747d950e769870ac01fc8
WARNING: Trying next encoding iso-8859-1
```
So I should probably let codespell ignore my `.git` folder from this point onwards? Or is the error still undesired behaviour?
Does just `codespell` work @EwoutH when in the right directory?

We should ignore hidden files already as per e.g.:
https://github.com/codespell-project/codespell/issues/2539#issuecomment-1282769046

Do the tests pass on Windows for you?

I suspect it's a bug somewhere, but I don't know if it's our hidden directory checking, or our Windows behaviour or something else...
~~You should certainly add `skip = ./.git` to your configuration file. As far as I know, codespell has never been skipping `.git` by itself.~~
**Edit:** My wrong, codespell does seem to be skipping hidden files, I wonder why I had or thought I had to add `skip = ./.git` in some projects:
```
$ mkdir .git
$ touch .git/file
$ chmod a= .git/file
$ codespell -w
$ 
```

I still wonder why you cannot reproduce this issue with prior versions of codespell. I seem to be able to trigger this error with codespell 2.2.0:
```
$ mkdir -p foo/bar
$ touch foo/bar/file
$ chmod a= foo/bar/file
$ cd foo
$ codespell --version
2.2.0
$ codespell
Traceback (most recent call last):
  File "/home/username/.local/bin/codespell", line 8, in <module>
    sys.exit(_script_main())
  File "/my/path/codespell/codespell_lib/_codespell.py", line 767, in _script_main
    return main(*sys.argv[1:])
  File "/my/path/codespell/codespell_lib/_codespell.py", line 910, in main
    bad_count += parse_file(
  File "/my/path/codespell/codespell_lib/_codespell.py", line 651, in parse_file
    text = is_text_file(filename)
  File "/my/path/codespell/codespell_lib/_codespell.py", line 509, in is_text_file
    with open(filename, mode='rb') as f:
PermissionError: [Errno 13] Permission denied: './bar/file'
$ 
```
OK, it appears codespell skips **hidden files** but does not skip **hidden directories** reliably. That has always been the case, hence the need to explicitly skip `.git` in the configuration file.
```
$ echo 'errror' > .hidden_file
$ 
$ mkdir .hidden_directory
$ echo 'errror' > .hidden_directory/file
$ echo 'errror' > .hidden_directory/a/file
$ echo 'errror' > .hidden_directory/a/b/file
$ echo 'errror' > .hidden_directory/a/b/c/file
$ 
$ codespell 
./.hidden_directory/a/file:1: errror ==> error
./.hidden_directory/a/b/file:1: errror ==> error
./.hidden_directory/a/b/c/file:1: errror ==> error
$ 
```
As you can see, it's all a question of depth. Hidden files and files directly under a hidden directory are skipped. Files in subdirectories of hidden directories are not skipped. This is not a 2.2.2 regression, it is just not a known bug. I guess you were lucky not to hit the bug before.

Will have to modify the call to [os.walk](https://docs.python.org/3/library/os.html#os.walk) and make sure we use the default `topdown=True` to be able to prune hidden directories in place.
> When _topdown_ is `True`, the caller can modify the _dirnames_ list in-place (perhaps using [del](https://docs.python.org/3/reference/simple_stmts.html#del) or slice assignment), and [walk()](https://docs.python.org/3/library/os.html#os.walk) will only recurse into the subdirectories whose names remain in _dirnames_; this can be used to prune the search, [...]

Resolving patch:
diff --git a/codespell_lib/_codespell.py b/codespell_lib/_codespell.py
--- a/codespell_lib/_codespell.py
+++ b/codespell_lib/_codespell.py
@@ -1171,7 +1171,12 @@ def main(*args: str) -> int:
                     )
 
                 # skip (relative) directories
-                dirs[:] = [dir_ for dir_ in dirs if not glob_match.match(dir_)]
+                dirs[:] = [
+                    dir_
+                    for dir_ in dirs
+                    if not glob_match.match(dir_)
+                    and not is_hidden(dir_, options.check_hidden)
+                ]
 
         elif not glob_match.match(filename):  # skip files
             bad_count += parse_file(


Test patch:
diff --git a/codespell_lib/tests/test_basic.py b/codespell_lib/tests/test_basic.py
--- a/codespell_lib/tests/test_basic.py
+++ b/codespell_lib/tests/test_basic.py
@@ -455,19 +455,33 @@ def test_check_hidden(
     capsys: pytest.CaptureFixture[str],
 ) -> None:
     """Test ignoring of hidden files."""
-    fname = tmp_path / "test.txt"
     # visible file
-    fname.write_text("abandonned\n")
+    #
+    #         tmp_path
+    #         └── test.txt
+    #
+    fname = tmp_path / "test.txt"
+    fname.write_text("erorr\n")
     assert cs.main(fname) == 1
     assert cs.main(tmp_path) == 1
+
     # hidden file
+    #
+    #         tmp_path
+    #         └── .test.txt
+    #
     hidden_file = tmp_path / ".test.txt"
     fname.rename(hidden_file)
     assert cs.main(hidden_file) == 0
     assert cs.main(tmp_path) == 0
     assert cs.main("--check-hidden", hidden_file) == 1
     assert cs.main("--check-hidden", tmp_path) == 1
+
     # hidden file with typo in name
+    #
+    #         tmp_path
+    #         └── .abandonned.txt
+    #
     typo_file = tmp_path / ".abandonned.txt"
     hidden_file.rename(typo_file)
     assert cs.main(typo_file) == 0
@@ -476,16 +490,28 @@ def test_check_hidden(
     assert cs.main("--check-hidden", tmp_path) == 1
     assert cs.main("--check-hidden", "--check-filenames", typo_file) == 2
     assert cs.main("--check-hidden", "--check-filenames", tmp_path) == 2
+
     # hidden directory
+    #
+    #         tmp_path
+    #         ├── .abandonned
+    #         │   ├── .abandonned.txt
+    #         │   └── subdir
+    #         │       └── .abandonned.txt
+    #         └── .abandonned.txt
+    #
     assert cs.main(tmp_path) == 0
     assert cs.main("--check-hidden", tmp_path) == 1
     assert cs.main("--check-hidden", "--check-filenames", tmp_path) == 2
-    hidden_dir = tmp_path / ".abandonned"
-    hidden_dir.mkdir()
-    copyfile(typo_file, hidden_dir / typo_file.name)
+    hidden = tmp_path / ".abandonned"
+    hidden.mkdir()
+    copyfile(typo_file, hidden / typo_file.name)
+    subdir = hidden / "subdir"
+    subdir.mkdir()
+    copyfile(typo_file, subdir / typo_file.name)
     assert cs.main(tmp_path) == 0
-    assert cs.main("--check-hidden", tmp_path) == 2
-    assert cs.main("--check-hidden", "--check-filenames", tmp_path) == 5
+    assert cs.main("--check-hidden", tmp_path) == 3
+    assert cs.main("--check-hidden", "--check-filenames", tmp_path) == 8
     # check again with a relative path
     try:
         rel = op.relpath(tmp_path)
@@ -494,20 +520,28 @@ def test_check_hidden(
         pass
     else:
         assert cs.main(rel) == 0
-        assert cs.main("--check-hidden", rel) == 2
-        assert cs.main("--check-hidden", "--check-filenames", rel) == 5
+        assert cs.main("--check-hidden", rel) == 3
+        assert cs.main("--check-hidden", "--check-filenames", rel) == 8
+
     # hidden subdirectory
-    assert cs.main(tmp_path) == 0
-    assert cs.main("--check-hidden", tmp_path) == 2
-    assert cs.main("--check-hidden", "--check-filenames", tmp_path) == 5
+    #
+    #         tmp_path
+    #         ├── .abandonned
+    #         │   ├── .abandonned.txt
+    #         │   └── subdir
+    #         │       └── .abandonned.txt
+    #         ├── .abandonned.txt
+    #         └── subdir
+    #             └── .abandonned
+    #                 └── .abandonned.txt
     subdir = tmp_path / "subdir"
     subdir.mkdir()
-    hidden_subdir = subdir / ".abandonned"
-    hidden_subdir.mkdir()
-    copyfile(typo_file, hidden_subdir / typo_file.name)
+    hidden = subdir / ".abandonned"
+    hidden.mkdir()
+    copyfile(typo_file, hidden / typo_file.name)
     assert cs.main(tmp_path) == 0
-    assert cs.main("--check-hidden", tmp_path) == 3
-    assert cs.main("--check-hidden", "--check-filenames", tmp_path) == 8
+    assert cs.main("--check-hidden", tmp_path) == 4
+    assert cs.main("--check-hidden", "--check-filenames", tmp_path) == 11
 
 
 def test_case_handling(