Repository: cython/cython
Pull request: cython, cythonize commands return error when file does not exist
cython and cythonize commands return No such file or directory error when passed file does not exist.

Fixes #4628


Problem statement:
[BUG] cython and cythonize commands does not error out when file does not exist
**Describe the bug**
All standard linux commands print out error when file is not present:

```
$ ls missing_file
ls: missing_file: No such file or directory
$ cat missing_file
cat: missing_file: No such file or directory
$ python3.9 missing_file
/usr/local/bin/python3.9: can't open file '/Users/matus/missing_file': [Errno 2] No such file or directory
```

cython and cythonize commands do not follow this semantics:
```
$ cython missing_file
/Users/matus/dev/cython/missing_file

$ cythonize missing_file
$
```

**To Reproduce**

Run `cython` or `cythonize` commands with non-existing files

**Expected behavior**

Error is printed out as in other standard tools

**Environment (please complete the following information):**
 - OS: MacOS
 - Python version 3.9.9
 - Cython version: master

Hints from issue discussion:

Commit message: Try to fix failing tests

Patch to review:
diff --git a/Cython/Build/Cythonize.py b/Cython/Build/Cythonize.py
index 1f79589f875..49df02abe13 100644
--- a/Cython/Build/Cythonize.py
+++ b/Cython/Build/Cythonize.py
@@ -1,6 +1,6 @@
 #!/usr/bin/env python
 
-from __future__ import absolute_import
+from __future__ import absolute_import, print_function
 
 import os
 import shutil
@@ -45,10 +45,12 @@ def find_package_base(path):
         package_path = '%s/%s' % (parent, package_path)
     return base_dir, package_path
 
-
 def cython_compile(path_pattern, options):
-    pool = None
     all_paths = map(os.path.abspath, extended_iglob(path_pattern))
+    _cython_compile_files(all_paths, options)
+
+def _cython_compile_files(all_paths, options):
+    pool = None
     try:
         for path in all_paths:
             if options.build_inplace:
@@ -222,8 +224,15 @@ def parse_args(args):
 def main(args=None):
     options, paths = parse_args(args)
 
+    all_paths = []
     for path in paths:
-        cython_compile(path, options)
+        expanded_path = [os.path.abspath(p) for p in extended_iglob(path)]
+        if not expanded_path:
+            import sys
+            print("{}: No such file or directory: '{}'".format(sys.argv[0], path), file=sys.stderr)
+            sys.exit(1)
+        all_paths.extend(expanded_path)
+    _cython_compile_files(all_paths, options)
 
 
 if __name__ == '__main__':
diff --git a/Cython/Compiler/CmdLine.py b/Cython/Compiler/CmdLine.py
index ffff6a61c96..025c7627c72 100644
--- a/Cython/Compiler/CmdLine.py
+++ b/Cython/Compiler/CmdLine.py
@@ -4,11 +4,17 @@
 
 from __future__ import absolute_import
 
+import sys
 import os
 from argparse import ArgumentParser, Action, SUPPRESS
 from . import Options
 
 
+if sys.version_info[:2] < (3, 3):
+    # TODO: This workaround can be removed in Cython 3.1
+    FileNotFoundError = IOError
+
+
 class ParseDirectivesAction(Action):
     def __call__(self, parser, namespace, values, option_string=None):
         old_directives = dict(getattr(namespace, self.dest,
@@ -203,6 +209,10 @@ def filter_out_embed_options(args):
 def parse_command_line(args):
     parser = create_cython_argparser()
     arguments, sources = parse_command_line_raw(parser, args)
+    for source in sources:
+        if not os.path.exists(source):
+            import errno
+            raise FileNotFoundError(errno.ENOENT, os.strerror(errno.ENOENT), source)
 
     options = Options.CompilationOptions(Options.default_options)
     for name, value in vars(arguments).items():
diff --git a/Cython/Compiler/Main.py b/Cython/Compiler/Main.py
index 764d9af2175..6dd527ec13b 100644
--- a/Cython/Compiler/Main.py
+++ b/Cython/Compiler/Main.py
@@ -2,7 +2,7 @@
 #   Cython Top Level
 #
 
-from __future__ import absolute_import
+from __future__ import absolute_import, print_function
 
 import os
 import re
@@ -716,7 +716,12 @@ def main(command_line = 0):
     args = sys.argv[1:]
     any_failures = 0
     if command_line:
-        options, sources = parse_command_line(args)
+        try:
+            options, sources = parse_command_line(args)
+        except IOError as e:
+            # TODO: IOError can be replaced with FileNotFoundError in Cython 3.1
+            print("{}: No such file or directory: '{}'".format(sys.argv[0], e.filename), file=sys.stderr)
+            sys.exit(1)
     else:
         options = CompilationOptions(default_options)
         sources = args
diff --git a/Cython/Compiler/Tests/TestCmdLine.py b/Cython/Compiler/Tests/TestCmdLine.py
index 5953112dc1c..4f1b15303f8 100644
--- a/Cython/Compiler/Tests/TestCmdLine.py
+++ b/Cython/Compiler/Tests/TestCmdLine.py
@@ -1,6 +1,10 @@
 import os
 import sys
 from unittest import TestCase
+if sys.version_info < (3, 3):
+    from mock import patch, Mock
+else:
+    from unittest.mock import patch, Mock
 try:
     from StringIO import StringIO
 except ImportError:
@@ -11,7 +15,14 @@
 
 from .Utils import backup_Options, restore_Options, check_global_options
 
+unpatched_exists = os.path.exists
 
+def patched_exists(path):
+    if path in ('source.pyx', 'file.pyx', 'file1.pyx', 'file2.pyx', 'file3.pyx'):
+        return True
+    return unpatched_exists(path)
+
+@patch('os.path.exists', new=Mock(side_effect=patched_exists))
 class CmdLineParserTest(TestCase):
     def setUp(self):
         self._options_backup = backup_Options()
diff --git a/Cython/Utils.py b/Cython/Utils.py
index d6a49f97359..598df33bf64 100644
--- a/Cython/Utils.py
+++ b/Cython/Utils.py
@@ -543,6 +543,7 @@ def print_captured(captured, output, header_line=None):
     if captured:
         if header_line:
             output.write(header_line)
+        captured = captured.encode(errors='replace').decode()
         output.write(captured)
 
 
diff --git a/test-requirements-27.txt b/test-requirements-27.txt
index a3ad0439e31..0e2e2188c84 100644
--- a/test-requirements-27.txt
+++ b/test-requirements-27.txt
@@ -60,3 +60,4 @@ wcwidth==0.2.5
 webencodings==0.5.1
 widgetsnbextension==3.5.1
 zipp==1.2.0
+mock==3.0.5
diff --git a/tests/run/cython_no_files.srctree b/tests/run/cython_no_files.srctree
new file mode 100644
index 00000000000..455258c038c
--- /dev/null
+++ b/tests/run/cython_no_files.srctree
@@ -0,0 +1,34 @@
+PYTHON test_cythonize_no_files.py
+PYTHON test_cython_no_files.py
+
+######## a.py ###########
+a = 1
+
+######## b.py ###########
+b = 2
+
+######## c.pyx ###########
+c = 3
+
+######## d.pyx ###########
+d = 4
+
+######## test_cythonize_no_files.py ###########
+import subprocess
+import sys
+
+cmd = [sys.executable, '-c', 'from Cython.Build.Cythonize import main; main()', 'a.py', 'b.py', 'c.py', '*.pyx']
+proc = subprocess.Popen(cmd, stderr=subprocess.PIPE)
+_, err = proc.communicate()
+assert proc.returncode == 1, proc.returncode
+assert b"No such file or directory: 'c.py'" in err, err
+
+######## test_cython_no_files.py ###########
+import subprocess
+import sys
+
+cmd = [sys.executable, '-c', 'from Cython.Compiler.Main import main; main(command_line = 1)', 'a.py', 'b.py', 'c.py', '*.pyx']
+proc = subprocess.Popen(cmd, stderr=subprocess.PIPE)
+_, err = proc.communicate()
+assert proc.returncode == 1, proc.returncode
+assert b"No such file or directory: 'c.py'" in err, err


Code review comments:
diff_hunk: @@ -543,6 +543,7 @@ def print_captured(captured, output, header_line=None):
     if captured:
         if header_line:
             output.write(header_line)
+        captured = captured.encode(errors='replace').decode()
path: Cython/Utils.py
text: @author:
Funny, this line helps. Not sure why it is working because:
1. `output.write()` accepts only unicode, not sure why since `output` is `sys.stderr`/`sys.stdout`.
2. If I understand that correctly `capture` is  prepared by `prepare_captured()` to ensure that string is serialisable. Not sure why we are getting this error.

@user1:
> `UnicodeEncodeError: 'ascii' codec can't encode characters in position 105-107: ordinal not in range(128)`

I think the code line that you added is the wrong way to deal with this issue. In Python 3, it probably doesn't do anything because the default encoding tends to be UTF-8, for which encoding+decoding is a no-op. In Python 2, this is only needed if the output (or default?) encoding cannot handle (non-ascii) Unicode text. I don't actually know why it encodes (to 'ascii') here. The logs are gone, so I cannot look up the IPython/Jupyter version that was in use. You were using an older 5.x version locally, so maybe that's a difference? I cannot reproduce it with an 8.x version either though (on Linux).

@user2:
If I remember correctly, this was a CI issue and nobody could reproduce it locally

@author:
OK. Now I tried to remove this line. Let's see what will say CI...

@author:
Hmm I tried to remove the line and now the CI works fine. Not sure what was the issue.