Repository: facelessuser/wcmatch

Issue:
Reverse match convention of translate exclusion patterns
One thing I regret was outputting exclusion patterns in translate as matches that are `True` if the folder is not excluded. This utilizes lookaheads to first check for anything that would exclude the file, and then grabs the filepath. This is slower than if we had just had a match (`True`) exclude the file.

Yes it is a clever pattern, but performance wise, it is not ideal. Internally, we are using an internal flag to force them in the better, faster orientation. I'm thinking we should just tear off the bandage in 4.0 and just output them in the better, faster way. Then we can get rid of the internal flag.

3.0 and 4.0 has really been a focus of getting things the way we ultimately want things. Fixing overlooked issues, and making course corrections before more people start using the library. I would have preferred to do this all in 3.0, but I didn't actually expect to have the motivation to move through 3.0 and 4.0 so fast, or even realize the full scope of what I wanted to do.

I imagine most people are using glob or fnmatch directly, and are not using the translate API. I also imagine the number of people using this library is quite small.

This will not affect Rummage or pyspelling, our file search and replace tool and document spellchecker, which is the biggest impact to me personally. Translation is really only used by me for internal stuff, and as I mentioned, we are forcing it the way we want it with a special internal flag.

In short, I'm willing to break API to work the better way, and possibly anger a very small few, if any, to get things to where we want them moving forward.  I'm hoping with the course corrections that took place in 3.0 and 4.0, that we won't have to make any drastic changes moving forward.


Discussion:

Patch:
diff --git a/docs/src/markdown/changelog.md b/docs/src/markdown/changelog.md
index f2c08e1..64916a3 100644
--- a/docs/src/markdown/changelog.md
+++ b/docs/src/markdown/changelog.md
@@ -9,6 +9,7 @@
 - **NEW**: Exclusion patterns (enabled with `NEGATE`) now always enable `DOTALL` in the exclusion patterns. They also will match symlinks in `**` patterns. Only non `NEGATE` patterns that are paired with a `NEGATE` pattern are subject to symlinks and dot rules. Exclusion patterns themselves allow dots and symlinks to make filtering easier.
 - **NEW**: Exclusion patterns no longer provide a default regular pattern if one is not applied. Exclusion patterns are
 meant to filter the results of normal patterns. You can either use the `SPLIT` flag and provide a pattern with your default ('default_pattern|!exclusion'), or feed in a list of multiple patterns instead of a single string (`['default_pattern', '!exclusion']`). If you really need the old behavior, you can use the `NEGDEFAULT` flag which will provide a default of `**` which is subject to the `GLOBSTAR` flag. `NEGDEFAULT` will raise a deprecation warning and will be removed in the future.
+- **NEW**: Translate now outputs exclusion patterns so that if they match, the file is excluded. This is opposite logic to how it used to be, but is more efficient.
 - **FIX**: An empty pattern in `glob` should not match slashes.
 
 ## 3.0.2
diff --git a/docs/src/markdown/fnmatch.md b/docs/src/markdown/fnmatch.md
index 6d49a18..23afc98 100644
--- a/docs/src/markdown/fnmatch.md
+++ b/docs/src/markdown/fnmatch.md
@@ -115,16 +115,19 @@ def fnsplit(pattern, *, flags=0):
 def translate(patterns, *, flags=0):
 ```
 
-`translate` takes a file pattern (or list of patterns) and returns two lists: one for normal patterns and one for exclusion patterns. The lists contain the regular expressions used for matching the given patterns. All patterns are constructed in such a way that a matched pattern is what is desired, regardless of whether the pattern is an exclusion pattern or regular pattern. It should be noted that a file is considered matched if it matches at least one regular pattern and all of the exclusion patterns.
+`translate` takes a file pattern (or list of patterns) and returns two lists: one for normal patterns and one for exclusion patterns. The lists contain the regular expressions used for matching the given patterns. It should be noted that a file is considered matched if it matches at least one regular pattern and matches **none** of the exclusion patterns.
 
 ```pycon3
 >>> from wcmatch import translate
 >>> fnmatch.translate(r'*.{a,{b,c}}', flags=fnmatch.BRACE)
 (['^(?s:(?=.).*?\\.a)$', '^(?s:(?=.).*?\\.b)$', '^(?s:(?=.).*?\\.c)$'], [])
 >>> fnmatch.translate(r'**|!*.{a,{b,c}}', flags=fnmatch.BRACE | fnmatch.NEGATE | fnmatch.SPLIT)
-(['^(?s:(?=.)(?![.]).*?)$'], ['^(?!(?s:(?=.).*?\\.a)$).*?$', '^(?!(?s:(?=.).*?\\.b)$).*?$', '^(?!(?s:(?=.).*?\\.c)$).*?$'])
+(['^(?s:(?=.)(?![.]).*?)$'], ['^(?s:(?=.).*?\\.a)$', '^(?s:(?=.).*?\\.b)$', '^(?s:(?=.).*?\\.c)$'])
 ```
 
+!!! warning "Changed 4.0"
+    Translate now outputs exclusion patterns so that if they match, the file is excluded. This is opposite logic to how it used to be, but is more efficient.
+
 ## Flags
 
 #### `fnmatch.FORCECASE, fnmatch.F` {: #fnmatchforcecase}
diff --git a/docs/src/markdown/glob.md b/docs/src/markdown/glob.md
index f74a879..462c3cb 100644
--- a/docs/src/markdown/glob.md
+++ b/docs/src/markdown/glob.md
@@ -134,19 +134,20 @@ Exclusion patterns are allowed as well.
 
 ```pycon3
 >>> from wcmatch import glob
->>> glob.globmatch('some/path/test.txt', r'!**/*/*.txt', flags=glob.NEGATE)
-False
->>> glob.globmatch('some/path/test.py', r'!**/*/*.txt', flags=glob.NEGATE)
+>>> glob.globmatch('some/path/test.py', r'**|!**/*.txt', flags=glob.NEGATE | glob.GLOBSTAR | glob.SPLIT)
 True
+>>> glob.globmatch('some/path/test.txt', r'**|!**/*.txt', flags=glob.NEGATE | glob.GLOBSTAR | glob.SPLIT)
+False
+>>>
 ```
 
 When exclusion patterns are used in conjunction with other patterns, a path will be considered matched if one of the positive patterns match **and** none of the exclusion patterns match. If an exclusion pattern is given without any regular patterns, the pattern will match nothing. Exclusion patterns are meant to filter other patterns, not match anything by themselves.
 
 ```pycon3
 >>> from wcmatch import glob
->>> glob.globmatch('some/path/test.txt', [r'**/*/*.txt', r'!**/*/avoid.txt'], flags=glob.NEGATE)
+>>> glob.globmatch('some/path/test.txt', [r'*/*/*.txt', r'!*/*/avoid.txt'], flags=glob.NEGATE)
 True
->>> glob.globmatch('some/path/avoid.txt', [r'**/*/*.txt', r'!**/*/avoid.txt'], flags=glob.NEGATE)
+>>> glob.globmatch('some/path/avoid.txt', [r'*/*/*.txt', r'!*/*/avoid.txt'], flags=glob.NEGATE)
 False
 ```
 
@@ -217,16 +218,19 @@ def globsplit(pattern, *, flags=0):
 def translate(patterns, *, flags=0):
 ```
 
-`translate` takes a file pattern (or list of patterns) and returns two lists: one for normal patterns and one for exclusion patterns. The lists contain the regular expressions used for matching the given patterns. All patterns are constructed in such a way that a matched pattern is what is desired, regardless of whether the pattern is an exclusion pattern or regular pattern. It should be noted that a file is considered matched if it matches at least one regular pattern and all of the exclusion patterns.
+`translate` takes a file pattern (or list of patterns) and returns two lists: one for normal patterns and one for exclusion patterns. The lists contain the regular expressions used for matching the given patterns. It should be noted that a file is considered matched if it matches at least one regular pattern and matches **none** of the exclusion patterns.
 
 ```pycon3
 >>> from wcmatch import glob
 >>> glob.translate(r'**/*.{py,txt}')
 (['^(?s:(?:(?!(?:\\/|^)\\.).)*?(?:^|$|\\/)+(?=.)(?!(?:\\.{1,2})(?:$|\\/))(?:(?!\\.)[^\\/]*?)?\\.py[\\/]*?)$', '^(?s:(?:(?!(?:\\/|^)\\.).)*?(?:^|$|\\/)+(?=.)(?!(?:\\.{1,2})(?:$|\\/))(?:(?!\\.)[^\\/]*?)?\\.txt[\\/]*?)$'], [])
 >>> glob.translate(r'**|!**/*.{py,txt}', flags=glob.NEGATE | glob.SPLIT)
-(['^(?s:(?=.)(?!(?:\\.{1,2})(?:$|\\/))(?:(?!\\.)[^\\/]*?)?[\\/]*?)$'], ['^(?!(?s:(?=.)(?!(?:\\.{1,2})(?:$|\\/))[^\\/]*?\\/+(?=.)(?!(?:\\.{1,2})(?:$|\\/))[^\\/]*?\\.\\{py\\,txt\\}[\\/]*?)$).*?$'])
+(['^(?s:(?=.)(?!(?:\\.{1,2})(?:$|\\/))(?:(?!\\.)[^\\/]*?)?[\\/]*?)$'], ['^(?s:(?=.)(?!(?:\\.{1,2})(?:$|\\/))[^\\/]*?\\/+(?=.)(?!(?:\\.{1,2})(?:$|\\/))[^\\/]*?\\.\\{py\\,txt\\}[\\/]*?)$'])
 ```
 
+!!! warning "Changed 4.0"
+    Translate now outputs exclusion patterns so that if they match, the file is excluded. This is opposite logic to how it used to be, but is more efficient.
+
 #### `glob.escape`
 
 ```py3
diff --git a/wcmatch/_wcparse.py b/wcmatch/_wcparse.py
index 54c3f49..850b665 100644
--- a/wcmatch/_wcparse.py
+++ b/wcmatch/_wcparse.py
@@ -93,8 +93,7 @@ NODIR = 0x4000
 _FORCEWIN = 0x100000000  # Forces Windows behavior (used to not assume Unix/Linux because of `FORCECASE` on Windows).
 _TRANSLATE = 0x200000000  # Lets us know we are performing a translation, and we just want the regex.
 _ANCHOR = 0x400000000  # The pattern, if it starts with a slash, is anchored to the working directory; strip the slash.
-_NO_TRANSLATE = 0x800000000  # Don't return translation pattern in positive form, but in the faster negative form.
-NEGDEFAULT = 0x1000000000  # Provide a default for exclude patterns
+NEGDEFAULT = 0x800000000  # Provide a default for exclude patterns
 
 FLAG_MASK = (
     FORCECASE |
@@ -114,8 +113,7 @@ FLAG_MASK = (
     NEGDEFAULT |
     _FORCEWIN |
     _TRANSLATE |
-    _ANCHOR |
-    _NO_TRANSLATE
+    _ANCHOR
 )
 CASE_FLAGS = FORCECASE | IGNORECASE
 
@@ -700,8 +698,6 @@ class WcParse(object):
         self.realpath = bool(flags & REALPATH) and self.pathname
         self.translate = bool(flags & _TRANSLATE)
         self.globstar_capture = self.realpath and not self.translate
-        if flags & _NO_TRANSLATE:
-            self.translate = False
         self.dot = bool(flags & DOTMATCH)
         self.extend = bool(flags & EXTMATCH)
         self.matchbase = bool(flags & MATCHBASE)
@@ -1337,13 +1333,9 @@ class WcParse(object):
 
         case_flag = 'i' if not self.case_sensitive else ''
         if util.PY36:
-            pattern = (
-                r'^(?!(?s%s:%s)$).*?$' if self.negative and self.translate else r'^(?s%s:%s)$'
-            ) % (case_flag, ''.join(result))
+            pattern = r'^(?s{}:{})$'.format(case_flag, ''.join(result))
         else:
-            pattern = (
-                r'(?s%s)^(?!(?:%s)$).*?$' if self.negative and self.translate else r'(?s%s)^(?:%s)$'
-            ) % (case_flag, ''.join(result))
+            pattern = r'(?s{})^(?:{})$'.format(case_flag, ''.join(result))
 
         if self.is_bytes:
             pattern = pattern.encode('latin-1')
diff --git a/wcmatch/fnmatch.py b/wcmatch/fnmatch.py
index f070df4..a52fc20 100644
--- a/wcmatch/fnmatch.py
+++ b/wcmatch/fnmatch.py
@@ -27,7 +27,7 @@ __all__ = (
     "EXTMATCH", "FORCECASE", "IGNORECASE", "RAWCHARS",
     "NEGATE", "MINUSNEGATE", "DOTMATCH", "BRACE", "SPLIT",
     "NEGDEFAULT",
-    "F", "I", "R", "N", "M", "D", "E", "S",
+    "F", "I", "R", "N", "M", "D", "E", "S", "B",
     "translate", "fnmatch", "filter", "fnsplit"
 )
 
diff --git a/wcmatch/glob.py b/wcmatch/glob.py
index 20342ec..6df0144 100644
--- a/wcmatch/glob.py
+++ b/wcmatch/glob.py
@@ -30,8 +30,8 @@ from . import util
 __all__ = (
     "FORCECASE", "IGNORECASE", "RAWCHARS", "DOTGLOB", "DOTMATCH",
     "EXTGLOB", "EXTMATCH", "GLOBSTAR", "NEGATE", "MINUSNEGATE", "BRACE",
-    "REALPATH", "FOLLOW", "MATCHBASE", "MARK", "NEGDEFAULT",
-    "F", "I", "R", "D", "E", "G", "N", "M", "P", "L", "X", 'K',
+    "REALPATH", "FOLLOW", "MATCHBASE", "MARK", "NEGDEFAULT", "NODIR",
+    "F", "I", "R", "D", "E", "G", "N", "M", "B", "P", "L", "S", "X", 'K', "O",
     "iglob", "glob", "globsplit", "globmatch", "globfilter", "escape"
 )
 
@@ -126,7 +126,7 @@ class Glob(object):
 
         self.pattern = []
         self.npatterns = []
-        nflags = self.flags | _wcparse.REALPATH | _wcparse._NO_TRANSLATE
+        nflags = self.flags | _wcparse.REALPATH
         for p in pattern:
             if _wcparse.is_negative(p, self.flags):
                 # Treat the inverse pattern as a normal pattern if it matches, we will exclude.
@@ -232,7 +232,6 @@ class Glob(object):
                             is_dir = f.is_dir()
                             if is_dir:
                                 is_link = f.is_symlink()
-                                self.symlinks[path] = is_link
                             else:
                                 # We don't care if a file is a link
                                 is_link = False
@@ -253,7 +252,6 @@ class Glob(object):
                     is_dir = os.path.isdir(path)
                     if is_dir:
                         is_link = os.path.islink(path)
-                        self.symlinks[path] = is_link
                     else:
                         is_link = False
                     if deep and not self.follow_links and is_link:
@@ -377,9 +375,6 @@ class Glob(object):
     def glob(self):
         """Starts off the glob iterator."""
 
-        # Cached symlinks
-        self.symlinks = {}
-
         if self.is_bytes:
             curdir = os.fsencode(os.curdir)
         else:
diff --git a/wcmatch/wcmatch.py b/wcmatch/wcmatch.py
index eb58e43..4c32e61 100644
--- a/wcmatch/wcmatch.py
+++ b/wcmatch/wcmatch.py
@@ -29,7 +29,7 @@ __all__ = (
     "FORCECASE", "IGNORECASE", "RAWCHARS", "FILEPATHNAME", "DIRPATHNAME",
     "EXTMATCH", "GLOBSTAR", "BRACE", "MINUSNEGATE", "SYMLINKS", "HIDDEN", "RECURSIVE",
     "MATCHBASE", "NEGDEFAULT",
-    "F", "I", "R", "P", "E", "G", "M", "DP", "FP", "SL", "HD", "RV", "X",
+    "F", "I", "R", "P", "E", "G", "M", "DP", "FP", "SL", "HD", "RV", "X", "B",
     "WcMatch"
 )
 


Test patch:
diff --git a/tests/test_fnmatch.py b/tests/test_fnmatch.py
index 0019436..f96229c 100644
--- a/tests/test_fnmatch.py
+++ b/tests/test_fnmatch.py
@@ -307,10 +307,10 @@ class TestFnMatchTranslate(unittest.TestCase):
         p1, p2 = self.split_translate('*test[a-z]?|*test2[a-z]?|!test[!a-z]|!test[!-|a-z]', flags | fnmatch.N)
         if util.PY36:
             self.assertEqual(p1, [r'^(?s:(?=.).*?test[a-z].)$', r'^(?s:(?=.).*?test2[a-z].)$'])
-            self.assertEqual(p2, [r'^(?!(?s:test[^a-z])$).*?$', r'^(?!(?s:test[^\-\|a-z])$).*?$'])
+            self.assertEqual(p2, [r'^(?s:test[^a-z])$', r'^(?s:test[^\-\|a-z])$'])
         else:
             self.assertEqual(p1, [r'(?s)^(?:(?=.).*?test[a-z].)$', r'(?s)^(?:(?=.).*?test2[a-z].)$'])
-            self.assertEqual(p2, [r'(?s)^(?!(?:test[^a-z])$).*?$', r'(?s)^(?!(?:test[^\-\|a-z])$).*?$'])
+            self.assertEqual(p2, [r'(?s)^(?:test[^a-z])$', r'(?s)^(?:test[^\-\|a-z])$'])
 
         p1, p2 = self.split_translate('test[]][!][][]', flags | fnmatch.F)
         if util.PY36:
@@ -342,10 +342,10 @@ class TestFnMatchTranslate(unittest.TestCase):
         p1, p2 = self.split_translate('-|-test|-', flags=flags | fnmatch.N | fnmatch.M)
         if util.PY36:
             self.assertEqual(p1, [])
-            self.assertEqual(p2, [r'^(?!(?s:)$).*?$', r'^(?!(?s:test)$).*?$', r'^(?!(?s:)$).*?$'])
+            self.assertEqual(p2, [r'^(?s:)$', r'^(?s:test)$', r'^(?s:)$'])
         else:
             self.assertEqual(p1, [])
-            self.assertEqual(p2, [r'(?s)^(?!(?:)$).*?$', r'(?s)^(?!(?:test)$).*?$', r'(?s)^(?!(?:)$).*?$'])
+            self.assertEqual(p2, [r'(?s)^(?:)$', r'(?s)^(?:test)$', r'(?s)^(?:)$'])
 
         p1, p2 = self.split_translate('test[^chars]', flags)
         if util.PY36: