Repository: zulip/zulip

Issue:
Add test suite for regex linters
Since we now have ~100 custom lint rules, it'd be good to have a test suite for them.

I'm thinking a simple test suite where you can declare a list like this:
```
good_lines = [
     ("foo.py", "foo = [] # type", "Error: Missing space before '#'"
```

where each tuple is a filename, line, and then some text in the expected error message.

I imagine we'd start with some simple small as part of `tools/test-tools` and then add things as we went.

(Note I made up the error message in the example above; it's probably off from the real one)

The cool is in `tools/linter_lib`; tagging this as bite size since I don't think one needs any special Zulip expertise to do this :).



Discussion:
Hello @zulip/server-tooling members, this issue was labeled with the **area: tooling** label, so you may want to check it out!

@zulipbot claim
Congratulations, @shailysangwan, you've successfully claimed your first issue! Contributing to Zulip will (hopefully) be a fun and rewarding experience, and I'm here to help get you off to a good start.

First things first:
* Join me on the [Zulip developers' server](https://chat.zulip.org), to get help, chat about this issue, and meet the other developers.
* Sign the [Dropbox Contributor License Agreement](https://opensource.dropbox.com/cla/), so that Zulip can use your code.
* [Unwatch this repository](https://help.github.com/articles/unwatching-repositories/), so that you don't get 100 emails a day.

As you work on this issue, you'll also want to refer to the [Zulip code contribution guide](http://zulip.readthedocs.io/en/latest/index.html#code-docs), as well as the rest of the developer documentation on that site.


See you on the other side (that is, the pull request side)!

Seeing this just made me realize the issue's similarity to #5406 . I submitted #6472 for adding a test suite just like described here, with the difference that it doesn't test the rules themselves yet, only the validity of the filenames. Positing here to avoid any potential duplicate work. @shailysangwan @timabbott how do we want to proceed here? 
#6472 provides a test file, which is a great start.  I think there's probably 2 things that need to happen here.  First, we need to write a test system for the rule tests, which has some nice helper function supporting tests like what I described above.  Then, we can add a bunch of tests, and ideally some sort of meta-test that can be used eventually to check if we have "test coverage" of all the rules, if we can figure out a clean way to do that.

Thinking about this a bit more, I think it's possible we want want the system to be that you add a line in the actual rule definition like this for the "two spaces before comment" test (with a list of lines that pass/fail):

`test_bad: ['x = 1 # comment']`
`test_good: ['x = 1  # comment']`

and the test system would confirm that the test fired that way on that particular rule (I think we can just assume the file matches the filter rules if it's being checked, and I don't think we want to test the descriptions, since that's basically just duplicate work).  Or something along those lines.  Should be able to cover 95% of the lint rules this way in a pretty compact fashion.

@derAnfaenger since you have some experience with writing test frameworks at this point, maybe you should work on adding the framework, and then @shailysangwan can review and help with adding individual tests?


Resolving patch:
diff --git a/tools/linter_lib/custom_check.py b/tools/linter_lib/custom_check.py
--- a/tools/linter_lib/custom_check.py
+++ b/tools/linter_lib/custom_check.py
@@ -131,7 +131,9 @@ def build_custom_checkers(by_lang):
          'description': 'Fix trailing whitespace'},
         {'pattern': '^#+[A-Za-z0-9]',
          'strip': '\n',
-         'description': 'Missing space after # in heading'},
+         'description': 'Missing space after # in heading',
+         'good_lines': ['### some heading', '# another heading'],
+         'bad_lines': ['###some heading', '#another heading']},
     ]  # type: RuleList
     js_rules = cast(RuleList, [
         {'pattern': '[^_]function\(',


Test patch:
diff --git a/tools/tests/test_linter_custom_check.py b/tools/tests/test_linter_custom_check.py
--- a/tools/tests/test_linter_custom_check.py
+++ b/tools/tests/test_linter_custom_check.py
@@ -1,17 +1,18 @@
 import os
 
 from itertools import chain
-from mock import patch, MagicMock
+from mock import patch
 from unittest import TestCase
 
 from typing import Any, Dict, List
 
 from tools.linter_lib.custom_check import build_custom_checkers
+from tools.linter_lib.custom_check import custom_check_file
 
 ROOT_DIR = os.path.abspath(os.path.join(__file__, '..', '..', '..'))
 CHECK_MESSAGE = "Fix the corresponding rule in `tools/linter_lib/custom_check.py`."
 
-class TestCustomRulesFormat(TestCase):
+class TestCustomRules(TestCase):
 
     def setUp(self):
         # type: () -> None
@@ -45,3 +46,22 @@ def test_paths_in_rules(self):
                 if not os.path.splitext(path)[1]:
                     self.assertTrue(path.endswith('/'),
                                     "The path '{}' should end with '/'. {}".format(path, CHECK_MESSAGE))
+
+    def test_rule_patterns(self):
+        # type: () -> None
+        """Verifies that the search regex specified in a custom rule actually matches
+           the expectation and doesn't throw false positives."""
+        for rule in self.all_rules:
+            pattern = rule['pattern']
+            for line in rule.get('good_lines', []):
+                # create=True is superfluous when mocking built-ins in Python >= 3.5
+                with patch('builtins.open', return_value=iter((line+'\n\n').splitlines()), create=True, autospec=True):
+                    self.assertFalse(custom_check_file('foo.bar', 'baz', [rule], ''),
+                                     "The pattern '{}' matched the line '{}' while it shouldn't.".format(pattern, line))
+
+            for line in rule.get('bad_lines', []):
+                # create=True is superfluous when mocking built-ins in Python >= 3.5
+                with patch('builtins.open',
+                           return_value=iter((line+'\n\n').splitlines()), create=True, autospec=True), patch('builtins.print'):
+                    self.assertTrue(custom_check_file('foo.bar', 'baz', [rule], ''),
+                                    "The pattern '{}' didn't match the line '{}' while it should.".format(pattern, line))