Repository: python/mypy

Issue:
Use a different error code on early exit
When running mypy from a script, it's sometimes nice to know when mypy successfully completed a full typechecking run vs exited early due to blocking errors.  Perhaps mypy could exit with exit code `2` when exiting due to blocking errors?


Discussion:
Good idea. Then it should also exit 2 if it can't open a file and for other "early stage" errors. (It already uses exit 2 for command line parsing errors.)
Maybe there should be 3 error exit codes?

1) Typechecking ran to completion and there are some type errors.
2) The code could not be analyzed at all because e.g. the command line flags don't parse or a file cannot be opened.  This should indicate a major configuration error.
3) There was some code analysis, but it could not be completed because of a parse error or early semantic analysis error.
Maybe, but the problem is that there's no standard for codes other than 0,
1, 2. So perhaps this not such a good idea after all? What's the use case
for distinguishing blockers?

Whoops, I searched for exit status but didn't find this issue. Sorry! Maybe it's time to increase priority?
If you think there's a current need for this, sure!
Mostly that I wish to remove certain code from an internal mypy integration script (you know the one).
Let's implement the following:
- exit code 2 for command line syntax errors, missing files, and blocking errors
- exit code 1 if there are only non-blocking errors
- exit code 0 if there are no errors (though perhaps warnings or notes)
Also, note that parse errors are implicitly treated as blocking errors.

Resolving patch:
diff --git a/mypy/main.py b/mypy/main.py
--- a/mypy/main.py
+++ b/mypy/main.py
@@ -72,12 +72,14 @@ def flush_errors(new_messages: List[str], serious: bool) -> None:
                 f.write(msg + '\n')
             f.flush()
         except BrokenPipeError:
-            sys.exit(1)
+            sys.exit(2)
 
     serious = False
+    blockers = False
     try:
         type_check_only(sources, bin_dir, options, flush_errors)
     except CompileError as e:
+        blockers = True
         if not e.use_stdout:
             serious = True
     if options.warn_unused_configs and options.unused_configs:
@@ -89,7 +91,8 @@ def flush_errors(new_messages: List[str], serious: bool) -> None:
         t1 = time.time()
         util.write_junit_xml(t1 - t0, serious, messages, options.junit_xml)
     if messages:
-        sys.exit(1)
+        code = 2 if blockers else 1
+        sys.exit(code)
 
 
 def find_bin_directory(script_path: str) -> str:


Test patch:
diff --git a/mypy/test/testcmdline.py b/mypy/test/testcmdline.py
--- a/mypy/test/testcmdline.py
+++ b/mypy/test/testcmdline.py
@@ -58,6 +58,7 @@ def test_python_cmdline(testcase: DataDrivenTestCase) -> None:
     outb = process.stdout.read()
     # Split output into lines.
     out = [s.rstrip('\n\r') for s in str(outb, 'utf8').splitlines()]
+    result = process.wait()
     # Remove temp file.
     os.remove(program_path)
     # Compare actual output to expected.
@@ -78,6 +79,9 @@ def test_python_cmdline(testcase: DataDrivenTestCase) -> None:
                                            path))
     else:
         out = normalize_error_messages(out)
+        obvious_result = 1 if out else 0
+        if obvious_result != result:
+            out.append('== Return code: {}'.format(result))
         assert_string_arrays_equal(testcase.output, out,
                                    'Invalid output ({}, line {})'.format(
                                        testcase.file, testcase.line))
diff --git a/test-data/unit/cmdline.test b/test-data/unit/cmdline.test
--- a/test-data/unit/cmdline.test
+++ b/test-data/unit/cmdline.test
@@ -4,7 +4,10 @@
 -- The initial line specifies the command line, in the format
 --
 --   # cmd: mypy <options>
-
+--
+-- '== Return code: <value>' is added to the output when the process return code
+-- is "nonobvious" -- that is, when it is something other than 0 if there are no
+-- messages and 1 if there are.
 
 -- Directories/packages on the command line
 -- ----------------------------------------
@@ -89,6 +92,7 @@ sub.pkg is not a valid Python package name
 # coding: uft-8
 [out]
 mypy: can't decode file 'a.py': unknown encoding: uft-8
+== Return code: 2
 
 [case testCannotIgnoreDuplicateModule]
 # cmd: mypy one/mod/__init__.py two/mod/__init__.py
@@ -98,6 +102,7 @@ mypy: can't decode file 'a.py': unknown encoding: uft-8
 # type: ignore
 [out]
 two/mod/__init__.py: error: Duplicate module named 'mod'
+== Return code: 2
 
 [case testFlagsFile]
 # cmd: mypy @flagsfile
@@ -225,6 +230,7 @@ x.py:1: error: Function is missing a type annotation
 [file mypy.ini]
 [out]
 mypy.ini: No [mypy] section in config file
+== Return code: 0
 
 [case testConfigErrorUnknownFlag]
 # cmd: mypy -c pass
@@ -233,6 +239,7 @@ mypy.ini: No [mypy] section in config file
 bad = 0
 [out]
 mypy.ini: [mypy]: Unrecognized option: bad = 0
+== Return code: 0
 
 [case testConfigErrorBadBoolean]
 # cmd: mypy -c pass
@@ -241,6 +248,7 @@ mypy.ini: [mypy]: Unrecognized option: bad = 0
 ignore_missing_imports = nah
 [out]
 mypy.ini: [mypy]: ignore_missing_imports: Not a boolean: nah
+== Return code: 0
 
 [case testConfigErrorNotPerFile]
 # cmd: mypy -c pass
@@ -250,6 +258,7 @@ mypy.ini: [mypy]: ignore_missing_imports: Not a boolean: nah
 python_version = 3.4
 [out]
 mypy.ini: [mypy-*]: Per-module sections should only specify per-module flags (python_version)
+== Return code: 0
 
 [case testConfigMypyPath]
 # cmd: mypy file.py
@@ -467,6 +476,7 @@ main.py:1: error: Cannot find module named 'a.b'
 python_version = 1.0
 [out]
 mypy.ini: [mypy]: python_version: Python major version '1' out of range (must be 2 or 3)
+== Return code: 0
 
 [case testPythonVersionTooOld26]
 # cmd: mypy -c pass
@@ -475,6 +485,7 @@ mypy.ini: [mypy]: python_version: Python major version '1' out of range (must be
 python_version = 2.6
 [out]
 mypy.ini: [mypy]: python_version: Python 2.6 is not supported (must be 2.7)
+== Return code: 0
 
 [case testPythonVersionTooOld32]
 # cmd: mypy -c pass
@@ -483,6 +494,7 @@ mypy.ini: [mypy]: python_version: Python 2.6 is not supported (must be 2.7)
 python_version = 3.2
 [out]
 mypy.ini: [mypy]: python_version: Python 3.2 is not supported (must be 3.3 or higher)
+== Return code: 0
 
 [case testPythonVersionTooNew28]
 # cmd: mypy -c pass
@@ -491,6 +503,7 @@ mypy.ini: [mypy]: python_version: Python 3.2 is not supported (must be 3.3 or hi
 python_version = 2.8
 [out]
 mypy.ini: [mypy]: python_version: Python 2.8 is not supported (must be 2.7)
+== Return code: 0
 
 [case testPythonVersionTooNew40]
 # cmd: mypy -c pass
@@ -499,6 +512,7 @@ mypy.ini: [mypy]: python_version: Python 2.8 is not supported (must be 2.7)
 python_version = 4.0
 [out]
 mypy.ini: [mypy]: python_version: Python major version '4' out of range (must be 2 or 3)
+== Return code: 0
 
 [case testPythonVersionAccepted27]
 # cmd: mypy -c pass
@@ -1008,3 +1022,27 @@ def get_tasks(self):
     return 'whatever'
 [out]
 a.py:1: error: Function is missing a type annotation
+
+[case testMissingFile]
+# cmd: mypy nope.py
+[out]
+mypy: can't read file 'nope.py': No such file or directory
+== Return code: 2
+
+[case testParseError]
+# cmd: mypy a.py
+[file a.py]
+def foo(
+[out]
+a.py:1: error: unexpected EOF while parsing
+== Return code: 2
+
+[case testParseErrorAnnots]
+# cmd: mypy a.py
+[file a.py]
+def foo(x):
+    # type: (str, int) -> None
+    return
+[out]
+a.py:1: error: Type signature has too many arguments
+== Return code: 2
diff --git a/test-data/unit/reports.test b/test-data/unit/reports.test
--- a/test-data/unit/reports.test
+++ b/test-data/unit/reports.test
@@ -12,6 +12,7 @@
 bad_report = .
 [out]
 mypy.ini: [mypy]: Unrecognized report type: bad_report
+== Return code: 0
 
 [case testCoberturaParser]
 # cmd: mypy --cobertura-xml-report build pkg