Repository: psf/black

Issue:
All-whitespace files do not get their newlines removed
From @akaihola in #779:

In akaihola/darker#164 ([comment](https://github.com/akaihola/darker/issues/164#issuecomment-882014824)) we noticed that `format_str()` gives different results for some files than running `black` from the command line or `black.main(..., standalone_mode=False)`. This should probably be fixed before making `format_str()` an officially public Python API.

Consider this test script `compare_black.py`:
```python
import io
import sys
from contextlib import redirect_stdout
from subprocess import PIPE, check_output

from black import FileMode, diff, format_str, main

path = sys.argv[1]

print("============================= format_str =============================")
with open(path) as f:
    content = f.read()
format_str_result = format_str(content, mode=FileMode())
print(diff(content, format_str_result, path, path))

print("================================ main ================================")
with redirect_stdout(io.TextIOWrapper(io.BytesIO())) as main_stdout:
    main(["--quiet", "--diff", path], standalone_mode=False)
main_stdout.seek(0)
print(main_stdout.read())

print("============================= subprocess =============================")
subprocess_result = check_output(["black", "--quiet", "--diff", path], encoding="UTF-8")
print(subprocess_result)
```

If you create a valid Python source code file with just a few newlines:
```bash
echo "



" >newlines.py
```
and reformat it using the above script:
```bash
python compare_black.py newlines.py                   
============================= format_str =============================
--- /tmp/myfile.py
+++ /tmp/myfile.py
@@ -1,5 +0,0 @@
-
-
-
-
-

================================ main ================================

============================= subprocess =============================

```
You see that `format_str()` removes those blank lines while `main()` and `black` leave the file untouched.

I haven't tried to found other cases where these outputs don't match. I wonder if it only happens to all-whitespace files. **Edit:** Yes, only with all-whitespace files.

_Originally posted by @akaihola in https://github.com/psf/black/issues/779#issuecomment-882233332_


Discussion:
>All-whitespace files do not get their newlines removed

So @JelleZijlstra the expected behavior is for all-whitespace files to be truncated instead of being left unmodified? Makes sense.

So it's actually `format_str()` which does the right thing, while the `black` command line, `black.main(..., standalone_mode=False)` and `black.format_file_contents()` are wrong since they don't reformat all-whitespace files.
I'm open to persuasion otherwise, but it makes sense to me to truncate such files to either empty or a single newline. I suppose it doesn't make a huge difference in practice.
> I'm open to persuasion otherwise, but it makes sense to me to truncate such files to either empty or a single newline. I suppose it doesn't make a huge difference in practice.

I think a single newline is the right choice here given that under unix a trailing newline indicates text files.

Resolving patch:
diff --git a/src/black/__init__.py b/src/black/__init__.py
--- a/src/black/__init__.py
+++ b/src/black/__init__.py
@@ -917,7 +917,7 @@ def format_file_contents(src_contents: str, *, fast: bool, mode: Mode) -> FileCo
     valid by calling :func:`assert_equivalent` and :func:`assert_stable` on it.
     `mode` is passed to :func:`format_str`.
     """
-    if not src_contents.strip():
+    if not mode.preview and not src_contents.strip():
         raise NothingChanged
 
     if mode.is_ipynb:
@@ -1014,6 +1014,9 @@ def format_ipynb_string(src_contents: str, *, fast: bool, mode: Mode) -> FileCon
     Operate cell-by-cell, only on code cells, only for Python notebooks.
     If the ``.ipynb`` originally had a trailing newline, it'll be preserved.
     """
+    if mode.preview and not src_contents:
+        raise NothingChanged
+
     trailing_newline = src_contents[-1] == "\n"
     modified = False
     nb = json.loads(src_contents)
@@ -1106,6 +1109,13 @@ def _format_str_once(src_contents: str, *, mode: Mode) -> str:
     dst_contents = []
     for block in dst_blocks:
         dst_contents.extend(block.all_lines())
+    if mode.preview and not dst_contents:
+        # Use decode_bytes to retrieve the correct source newline (CRLF or LF),
+        # and check if normalized_content has more than one line
+        normalized_content, _, newline = decode_bytes(src_contents.encode("utf-8"))
+        if "\n" in normalized_content:
+            return newline
+        return ""
     return "".join(dst_contents)
 
 


Test patch:
diff --git a/tests/data/preview/whitespace.py b/tests/data/preview/whitespace.py
new file mode 100644
--- /dev/null
+++ b/tests/data/preview/whitespace.py
@@ -0,0 +1,6 @@
+
+		
+
+           
+
+# output
diff --git a/tests/test_black.py b/tests/test_black.py
--- a/tests/test_black.py
+++ b/tests/test_black.py
@@ -25,6 +25,7 @@
     List,
     Optional,
     Sequence,
+    Type,
     TypeVar,
     Union,
 )
@@ -153,6 +154,34 @@ def test_empty_ff(self) -> None:
             os.unlink(tmp_file)
         self.assertFormatEqual(expected, actual)
 
+    @patch("black.dump_to_file", dump_to_stderr)
+    def test_one_empty_line(self) -> None:
+        mode = black.Mode(preview=True)
+        for nl in ["\n", "\r\n"]:
+            source = expected = nl
+            assert_format(source, expected, mode=mode)
+
+    def test_one_empty_line_ff(self) -> None:
+        mode = black.Mode(preview=True)
+        for nl in ["\n", "\r\n"]:
+            expected = nl
+            tmp_file = Path(black.dump_to_file(nl))
+            if system() == "Windows":
+                # Writing files in text mode automatically uses the system newline,
+                # but in this case we don't want this for testing reasons. See:
+                # https://github.com/psf/black/pull/3348
+                with open(tmp_file, "wb") as f:
+                    f.write(nl.encode("utf-8"))
+            try:
+                self.assertFalse(
+                    ff(tmp_file, mode=mode, write_back=black.WriteBack.YES)
+                )
+                with open(tmp_file, "rb") as f:
+                    actual = f.read().decode("utf8")
+            finally:
+                os.unlink(tmp_file)
+            self.assertFormatEqual(expected, actual)
+
     def test_experimental_string_processing_warns(self) -> None:
         self.assertWarns(
             black.mode.Deprecated, black.Mode, experimental_string_processing=True
@@ -971,8 +1000,8 @@ def err(msg: str, **kwargs: Any) -> None:
         )
 
     def test_format_file_contents(self) -> None:
-        empty = ""
         mode = DEFAULT_MODE
+        empty = ""
         with self.assertRaises(black.NothingChanged):
             black.format_file_contents(empty, mode=mode, fast=False)
         just_nl = "\n"
@@ -990,6 +1019,17 @@ def test_format_file_contents(self) -> None:
             black.format_file_contents(invalid, mode=mode, fast=False)
         self.assertEqual(str(e.exception), "Cannot parse: 1:7: return if you can")
 
+        mode = black.Mode(preview=True)
+        just_crlf = "\r\n"
+        with self.assertRaises(black.NothingChanged):
+            black.format_file_contents(just_crlf, mode=mode, fast=False)
+        just_whitespace_nl = "\n\t\n \n\t \n \t\n\n"
+        actual = black.format_file_contents(just_whitespace_nl, mode=mode, fast=False)
+        self.assertEqual("\n", actual)
+        just_whitespace_crlf = "\r\n\t\r\n \r\n\t \r\n \t\r\n\r\n"
+        actual = black.format_file_contents(just_whitespace_crlf, mode=mode, fast=False)
+        self.assertEqual("\r\n", actual)
+
     def test_endmarker(self) -> None:
         n = black.lib2to3_parse("\n")
         self.assertEqual(n.type, black.syms.file_input)
@@ -1281,8 +1321,51 @@ def test_reformat_one_with_stdin_and_existing_path(self) -> None:
             report.done.assert_called_with(expected, black.Changed.YES)
 
     def test_reformat_one_with_stdin_empty(self) -> None:
+        cases = [
+            ("", ""),
+            ("\n", "\n"),
+            ("\r\n", "\r\n"),
+            (" \t", ""),
+            (" \t\n\t ", "\n"),
+            (" \t\r\n\t ", "\r\n"),
+        ]
+
+        def _new_wrapper(
+            output: io.StringIO, io_TextIOWrapper: Type[io.TextIOWrapper]
+        ) -> Callable[[Any, Any], io.TextIOWrapper]:
+            def get_output(*args: Any, **kwargs: Any) -> io.TextIOWrapper:
+                if args == (sys.stdout.buffer,):
+                    # It's `format_stdin_to_stdout()` calling `io.TextIOWrapper()`,
+                    # return our mock object.
+                    return output
+                # It's something else (i.e. `decode_bytes()`) calling
+                # `io.TextIOWrapper()`, pass through to the original implementation.
+                # See discussion in https://github.com/psf/black/pull/2489
+                return io_TextIOWrapper(*args, **kwargs)
+
+            return get_output
+
+        mode = black.Mode(preview=True)
+        for content, expected in cases:
+            output = io.StringIO()
+            io_TextIOWrapper = io.TextIOWrapper
+
+            with patch("io.TextIOWrapper", _new_wrapper(output, io_TextIOWrapper)):
+                try:
+                    black.format_stdin_to_stdout(
+                        fast=True,
+                        content=content,
+                        write_back=black.WriteBack.YES,
+                        mode=mode,
+                    )
+                except io.UnsupportedOperation:
+                    pass  # StringIO does not support detach
+                assert output.getvalue() == expected
+
+        # An empty string is the only test case for `preview=False`
         output = io.StringIO()
-        with patch("io.TextIOWrapper", lambda *args, **kwargs: output):
+        io_TextIOWrapper = io.TextIOWrapper
+        with patch("io.TextIOWrapper", _new_wrapper(output, io_TextIOWrapper)):
             try:
                 black.format_stdin_to_stdout(
                     fast=True,