Repository: python/mypy

Issue:
Default "--soft-error-limit" behavior causes tons of headache for mypy newbies with large code bases
**Bug Report**

While not strictly a bug, I would argue that there is a "bug" in the logic behind the decision to add a default `--soft-error-limit` of `200` (added in https://github.com/python/mypy/pull/10579). The idea was to make it easier to identify missing stubs as the root cause for a flood of errors, especially useful when settings are strict. However, I would assume that strict settings are typically used by `mypy` experts while newbies would typically start with the default settings and gradually work their way up to strict settings (if at all). So while the default values of the strictness settings favor newbies, I think the default behavior of `--soft-error-limits` inconsistently (and unhelpfully) favors experts at the expense of newbies.

Furthermore, this error limit applies to all errors, regardless of whether they are caused by missing stubs or not. This leads to very confusing error outputs for large code bases that are in the process of being typed by `mypy` newbies. Since Python typing already is a complicated heap of **(insert word here)**, even for devs with  tons of experience in non-typed Python, I would think it is of importance for the `mypy` project to try to lower the threshold for adoption by newbies as much as possible. However, the default setting of `--soft-error-limit` does the opposite: It is counterintuitive that `mypy` by default has an error limit that silences errors.

The default behavior is especially confusing when `mypy` is used in the context of an IDE such as PyCharm (also mentioned by @ilevkivskyi in this comment https://github.com/python/mypy/pull/10579#discussion_r645623803). In this case, the error messages are filtered by module name so that the note `(Skipping most remaining errors due to unresolved imports or missing stubs; fix these first)` will appear in a random module and most probably not be read by the user. In my personal experience, trying to debug the discrepancy between the inline `mypy` errors and the lack of the same errors in the global view made me in the end want to pull out my last few remaining strands of hair rather than continue the hunt. This particular discrepancy was one of the worst issues I have ever tried to debug, as there were no patterns to which errors were silenced. Debugging, on the other hand, is typically done by trying to find the pattern that cause the bug to appear:

> **Example debugging session**: In my particular debugging approach, I tried figuring out why an inline error in a particular module was silenced in a global mypy run by carrying out a binary search of including/excluding modules to locate the exact reason for the silencing, while `grep`-ing the output on the specific module I selected. `grep`-ing of course silenced the real error message (however it did appear in my module of interest once in a while, but with no apparent pattern). In the end I had a set of unrelated modules where, for some unexplained reason, removing any one of them from the `mypy` run would "un-silence" the error for the module of interest. In the end, I concluded that there was no pattern, there had to be a limit somewhere. Googling the note text mentioned above also did not directly lead me to the answer, that also took a while. I am sure other approaches to debugging this issue can be equally or more confusing, as exemplified in issues https://github.com/python/mypy/issues/10956 and https://github.com/python/mypy/issues/14389.


**To Reproduce**

As a `mypy` newbie, run `mypy` on a large code base that have been partially or fully typed, but where detailed fixing of `mypy` errors have been postponed due to the priority of developing actual features. To increase the pain level to the maximum, do this in PyCharm or other IDE where errors are filtered by module. Then:

1. Be motivated that there are not more than around 200 errors in the whole code base!
2. Start fixing errors in a module of choice
3. Discover that the inline errors from the module do not appear in the global view
4. Try to debug the discrepancy: choose your favorite debugging method and whatever pattern seems most likely to you as the cause
5. Discover that the more you debug, the less sense you are able to make of the issue
6. Start questioning your own sanity!
7. Alternatively, start questioning the sanity of the `mypy` library
8. (If you endure wasting hours of your precious time): Find the actual cause of the bug. Set `--soft-error-limit=-1` and witness the total number of errors multiply
9. Despair at the real number of errors you need to fix!

**Expected Behavior**

1. All errors are reported by default
2. (Alternatively) All errors not caused by missing stubs are reported by default
3. If `--soft-error-limit` is set, a note should be added at the __very end__ of the error output that __also__ mentions the actual config/command-line setting to tweak
4. A config setting `soft_error_limit` should be added that corresponds to the command-line flag
5. Log output at `verbosity=1` should, for each module, log whether any errors were silenced due to the setting

**Actual Behavior**

No more than 200 errors are reported (in my case only 139 errors and 41 notes were reported, for some reason). Errors are randomly silenced in global runs for no apparent reason and with no easily noticed error messages. Reading the log at `verbosity=1` gives no clue to the reason why errors are silenced.

**Your Environment**

<!-- Include as many relevant details about the environment you experienced the bug in -->

- Mypy version used: 1.1.1
- Mypy command-line flags: `--show-column-numbers --follow-imports silent` (default for PyCharm `mypy` plugin)
- Mypy configuration options from `mypy.ini` (and other config files): 
mypy_path = "src:tests"
- Python version used: 3.10

<!-- You can freely edit this text, please remove all the lines you believe are unnecessary. -->



Discussion:
You convinced me that the soft error limit can be confusing and should not be enabled by default. Let's just disable it by default as a first step, and we can improve it later. 

Resolving patch:
diff --git a/mypy/defaults.py b/mypy/defaults.py
--- a/mypy/defaults.py
+++ b/mypy/defaults.py
@@ -44,5 +44,5 @@
 ]
 
 # Threshold after which we sometimes filter out most errors to avoid very
-# verbose output
-MANY_ERRORS_THRESHOLD: Final = 200
+# verbose output. The default is to show all errors.
+MANY_ERRORS_THRESHOLD: Final = -1


Test patch: