Repository: xonsh/xonsh
Pull request: Fix `cat` can't read pseudo files with zero size
<!--- Thanks for opening a PR on xonsh! Please include a news entry with your PR
to help keep our changelog up to date! There are instructions available here:
https://xon.sh/devguide.html#changelog -->

<!--- If there is specific issue / feature request that this PR is addressing,
please link to the corresponding issue by using the `#issuenumber` syntax.
Thanks again! -->

I hope this patch will fix #3182 and #3199.
I tried `cat /proc/cpuinfo` in my machine.


Problem statement:
coreutils cat can't read /proc/*
Hey ! cat from coreutils doesn't return anything when reading files from `/proc` :
```
$ cat /proc/cpuinfo                                                                                                                                                                            
$ /usr/bin/cat /proc/cpuinfo                                                                                                                                                                   
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 60
model name      : Intel(R) Core(TM) i5-4590 CPU @ 3.30GHz
```

Here is my config :
```
+------------------+---------------------+
| xonsh            | 0.9.6               |
| Git SHA          | dabd4bd3            |
| Commit Date      | Jun 6 09:44:17 2019 |
| Python           | 3.7.3               |
| PLY              | 3.11                |
| have readline    | True                |
| prompt toolkit   | 2.0.9               |
| shell type       | prompt_toolkit2     |
| pygments         | 2.4.2               |
| on posix         | True                |
| on linux         | True                |
| distro           | unknown             |
| on darwin        | False               |
| on windows       | False               |
| on cygwin        | False               |
| on msys2         | False               |
| is superuser     | False               |
| default encoding | utf-8               |
| xonsh encoding   | utf-8               |
| encoding errors  | surrogateescape     |
+------------------+---------------------+
```

Hints from issue discussion:

Commit message: add to restore the deleted new line

Patch to review:
diff --git a/news/3182-fix-cat-proc-file.rst b/news/3182-fix-cat-proc-file.rst
new file mode 100644
index 0000000000..c26c588255
--- /dev/null
+++ b/news/3182-fix-cat-proc-file.rst
@@ -0,0 +1,23 @@
+**Added:**
+
+* <news item>
+
+**Changed:**
+
+* <news item>
+
+**Deprecated:**
+
+* <news item>
+
+**Removed:**
+
+* <news item>
+
+**Fixed:**
+
+* Fix ``cat`` can't read pseudo files with zero size such as /proc/\* or /sys/\* (#3182, #3199)
+
+**Security:**
+
+* <news item>
diff --git a/tests/test_xoreutils.py b/tests/test_xoreutils.py
index 34642e2046..2253b77b76 100644
--- a/tests/test_xoreutils.py
+++ b/tests/test_xoreutils.py
@@ -191,6 +191,24 @@ def test_cat_single_file_work_exist_content(self, cat_env_fixture):
         assert stdout_buf.getvalue() == bytes(expected_content, "utf-8")
         assert stderr_buf.getvalue() == b''
 
+    def test_cat_single_file_with_end_newline(self, cat_env_fixture):
+        content = "this is a content withe \\n\nfor testing xoreutil's cat\n"
+        with open(self.tempfile, "w") as f:
+            f.write(content)
+        expected_content = content.replace("\n", os.linesep)
+
+        stdin = io.StringIO()
+        stdout_buf = io.BytesIO()
+        stderr_buf = io.BytesIO()
+        stdout = io.TextIOWrapper(stdout_buf)
+        stderr = io.TextIOWrapper(stderr_buf)
+        opts = cat._cat_parse_args([])
+        cat._cat_single_file(opts, self.tempfile, stdin, stdout, stderr)
+        stdout.flush()
+        stderr.flush()
+        assert stdout_buf.getvalue() == bytes(expected_content, "utf-8")
+        assert stderr_buf.getvalue() == b''
+
     def test_cat_empty_file(self, cat_env_fixture):
         with open(self.tempfile, "w") as f:
             f.write("")
diff --git a/xonsh/xoreutils/cat.py b/xonsh/xoreutils/cat.py
index 2a89f14193..4f1a5c5984 100644
--- a/xonsh/xoreutils/cat.py
+++ b/xonsh/xoreutils/cat.py
@@ -1,6 +1,5 @@
 """Implements a cat command for xonsh."""
 import os
-import stat
 import time
 import builtins
 
@@ -12,15 +11,17 @@ def _cat_line(
     f, sep, last_was_blank, line_count, opts, out, enc, enc_errors, read_size
 ):
     _r = r = f.readline(size=80)
+    restore_newline = False
     if isinstance(_r, str):
         _r = r = _r.encode(enc, enc_errors)
     if r == b"":
-        last_was_blank, line_count, read_size, True
+        return last_was_blank, line_count, read_size, True
     if r.endswith(sep):
         _r = _r[: -len(sep)]
+        restore_newline = True
     this_one_blank = _r == b""
     if last_was_blank and this_one_blank and opts["squeeze_blank"]:
-        last_was_blank, line_count, read_size, False
+        return last_was_blank, line_count, read_size, False
     last_was_blank = this_one_blank
     if opts["number_all"] or (opts["number_nonblank"] and not this_one_blank):
         start = ("%6d " % line_count).encode(enc, enc_errors)
@@ -28,6 +29,8 @@ def _cat_line(
         line_count += 1
     if opts["show_ends"]:
         _r = _r + b"$"
+    if restore_newline:
+        _r = _r + sep
     out.buffer.write(_r)
     out.flush()
     read_size += len(r)
@@ -39,7 +42,7 @@ def _cat_single_file(opts, fname, stdin, out, err, line_count=1):
     enc = env.get("XONSH_ENCODING")
     enc_errors = env.get("XONSH_ENCODING_ERRORS")
     read_size = 0
-    file_size = fobj = None
+    fobj = None
     if fname == "-":
         f = stdin
     elif os.path.isdir(fname):
@@ -49,15 +52,11 @@ def _cat_single_file(opts, fname, stdin, out, err, line_count=1):
         print("cat: No such file or directory: {}".format(fname), file=err)
         return True, line_count
     else:
-        fstat = os.stat(fname)
-        file_size = fstat.st_size
-        if file_size == 0 and not stat.S_ISREG(fstat.st_mode):
-            file_size = None
         fobj = open(fname, "rb")
         f = xproc.NonBlockingFDReader(fobj.fileno(), timeout=0.1)
     sep = os.linesep.encode(enc, enc_errors)
     last_was_blank = False
-    while file_size is None or read_size < file_size:
+    while True:
         try:
             last_was_blank, line_count, read_size, endnow = _cat_line(
                 f,


Code review comments:
diff_hunk: @@ -49,15 +52,11 @@ def _cat_single_file(opts, fname, stdin, out, err, line_count=1):
         print("cat: No such file or directory: {}".format(fname), file=err)
         return True, line_count
     else:
-        fstat = os.stat(fname)
-        file_size = fstat.st_size
-        if file_size == 0 and not stat.S_ISREG(fstat.st_mode):
-            file_size = None
         fobj = open(fname, "rb")
         f = xproc.NonBlockingFDReader(fobj.fileno(), timeout=0.1)
     sep = os.linesep.encode(enc, enc_errors)
     last_was_blank = False
-    while file_size is None or read_size < file_size:
+    while True:
path: xonsh/xoreutils/cat.py
text: @user1:
I think some of these conditions might need to be implemented in the body for safety

@user1:
pinginf this again @author

@author:
Thanks for reviewing and sorry for the delayed response.
I think you are right but I don't know what is the good condition for safety...
Does checking `endnow` meet safety?

@user1:
Is there a reason the original check doesn't apply anymore is what I was asking. I think it could be restored.

@author:
The original check ( 344671ba ) caused #3182 because `stat.S_ISREG(os.stat("/proc/cpuinfo").st_mode)` returns 0 and the file size of `/proc/cpuinfo` is 0.
But the previous of original check( 7b7a9a30 ) works fine, so I think it can be restored. I will try it.