Repository: huggingface/transformers

Issue:
Doc styling utils adds parasites new lines
## Environment info
     
- `transformers` version: 4.2.0dev0
- Platform: Windows-10-10.0.18362-SP0
- Python version: 3.7.9
- PyTorch version (GPU?): 1.7.1 (False)
- Tensorflow version (GPU?): 2.3.1 (False)
- Using GPU in script?: Nope
- Using distributed or parallel set-up in script?: Nope

### Who can help

@sgugger 

## Information

Running the python util to style docs adds parasite new lines in every single docstring. See:

```bash
$ python utils/style_doc.py src/transformers docs/source --max_len 119 --check_only
Traceback (most recent call last):
  File "utils/style_doc.py", line 491, in <module>
    main(*args.files, max_len=args.max_len, check_only=args.check_only)
  File "utils/style_doc.py", line 479, in main
    raise ValueError(f"{len(changed)} files should be restyled!")
ValueError: 345 files should be restyled!
```

See this commit for an example of what it does: https://github.com/huggingface/transformers/pull/9150/commits/b4dedd5ca25f043c66d12c774fa00a34c74dffb2

## To reproduce

Steps to reproduce the behavior:

1. Checkout and update master branch
2. run `python utils/style_doc.py src/transformers docs/source --max_len 119 --check-only` from transformers root

Output:
```python
Traceback (most recent call last):
  File "utils/style_doc.py", line 491, in <module>
    main(*args.files, max_len=args.max_len, check_only=args.check_only)
  File "utils/style_doc.py", line 479, in main
    raise ValueError(f"{len(changed)} files should be restyled!")
ValueError: 345 files should be restyled!
```

It might have something to do with Windows or a particular setup of my machine because behavior cannot be reproduced by @patrickvonplaten.

## Expected behavior

On master branch, documentation should not need to be restyled



Discussion:
@sgugger do you maybe have an idea here? Now that I see that it's Window I think this could be the reason
Mmm, @jplu didn't have any issue with this I believe.
Sorry, reading again, you're not running make style but launching the script directly. There is nothing there to properly support Windows special line endings so it would probably require rewriting it from scratch to fix this. Can you use WSL for the styling?
I can run properly `python utils/style_doc.py src/transformers docs/source --max_len 119` on Windows (not on WSL) without any error. In order to properly run all the make targets on Windows I use the steps given in the [CONTRIBUTING readme](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#develop-on-windows).
Ah, there was one special magic option from GitHub for the line endings that could help, @LysandreJik do you remember which one? We had this problem with Niels on TAPAS.
It is the `core.autocrlf` config, you can see how to set it in the doc https://git-scm.com/book/fr/v2/Personnalisation-de-Git-Configuration-de-Git
Asked Lysandre and running `git config core.autocrlf false` solved the issue last time a contributor ran into it. @SBrandeis could you test if it does solve the issue for you? If that's the case, we'll add it to the `CONTRIBUTING` guide.
Setting `git config core.autocrlf` to `false` did not solve my issue, neither did running the python util from WSL.
 
Just tried on my Windows laptop and I'm unable to reproduce, the line runs just fine on my side :-/, so it must be something else. The `newline="\n"` that jplu added everywhere we open a file should make it so that there is no different line endings problem in the first place, but there is still something here...

From the diff, it comes from [this regex](https://github.com/huggingface/transformers/blob/1c19b423bf274a465f95725a79819bf82f71329e/utils/style_doc.py#L417) but there should be no weird `\r` from Windows at this stage.

Anyhow, will rework the regex as a for loop and it should work everywhere hopefully.
(Not sure the PR above will actually fix the issue since I can't reproduce, please confirm if it does or no @SBrandeis )
Hi @sgugger, thanks a lot for the PR.
Unfortunately, it does not solve the issue on my side (the style_doc util still updates 196 files).
Since none of @jplu, @patrickvonplaten  and you can reproduce the issue, it must be related to my particular setup.
Not sure what is the cause though, but I'll let you know if I figure this out !
@SBrandeis as we are both on Windows, do you want we check that together offline?
I actually forgot to push the changes in #9488 because it was Friday evening and my brain was dead :-/
Will open a new PR.

Resolving patch:
diff --git a/utils/style_doc.py b/utils/style_doc.py
--- a/utils/style_doc.py
+++ b/utils/style_doc.py
@@ -61,8 +61,8 @@
 _re_arg_def = re.compile(r"^\s*(Args?|Parameters?|Params|Arguments?|Environment|Attributes?)\s*:\s*$")
 # Matches the return introduction in docstrings.
 _re_return = re.compile(r"^\s*(Returns?|Raises?|Note)\s*:\s*$")
-# Matches any doc special word without an empty line before.
-_re_any_doc_special_word = re.compile(r"[^\n]\n([ \t]*)(" + "|".join(DOC_SPECIAL_WORD) + r")(::?\s*)\n")
+# Matches any doc special word.
+_re_any_doc_special_word = re.compile(r"^\s*(" + "|".join(DOC_SPECIAL_WORD) + r")::?\s*$")
 
 
 class SpecialBlock(Enum):
@@ -375,6 +375,19 @@ def _add_new_lines_before_list(text):
     return "\n".join(new_lines)
 
 
+def _add_new_lines_before_doc_special_words(text):
+    lines = text.split("\n")
+    new_lines = []
+    for idx, line in enumerate(lines):
+        # Detect if the line is the start of a new list.
+        if _re_any_doc_special_word.search(line) is not None:
+            # If the line before is non empty, add an extra new line.
+            if idx > 0 and len(lines[idx - 1]) != 0:
+                new_lines.append("")
+        new_lines.append(line)
+    return "\n".join(new_lines)
+
+
 def style_rst_file(doc_file, max_len=119, check_only=False):
     """ Style one rst file `doc_file` to `max_len`."""
     with open(doc_file, "r", encoding="utf-8", newline="\n") as f:
@@ -414,7 +427,7 @@ def style_docstring(docstring, max_len=119):
         indent = indent_search.groups()[0] if indent_search is not None else ""
 
     # Add missing new lines before Args/Returns etc.
-    docstring = _re_any_doc_special_word.sub(r"\n\n\1\2\3\n", docstring)
+    docstring = _add_new_lines_before_doc_special_words(docstring)
     # Add missing new lines before lists
     docstring = _add_new_lines_before_list(docstring)
     # Style


Test patch: