Repository: celery/celery

Issue:
glob behavior in task_routes interprets * as one or more characters, not zero or more
<!--
Please fill this template entirely and do not erase parts of it.
We reserve the right to close without a response
bug reports which are incomplete.
-->
# Checklist
<!--
To check an item on the list replace [ ] with [x].
-->

- [x] I have read the relevant section in the
  [contribution guide](http://docs.celeryproject.org/en/latest/contributing.html#other-bugs)
  on reporting bugs.
- [x] I have checked the [issues list](https://github.com/celery/celery/issues?q=is%3Aissue+label%3A%22Issue+Type%3A+Bug+Report%22+-label%3A%22Category%3A+Documentation%22)
  for similar or identical bug reports.
- [x] I have checked the [pull requests list](https://github.com/celery/celery/pulls?q=is%3Apr+label%3A%22PR+Type%3A+Bugfix%22+-label%3A%22Category%3A+Documentation%22)
  for existing proposed fixes.
- [x] I have checked the [commit log](https://github.com/celery/celery/commits/master)
  to find out if the bug was already fixed in the master branch.
- [x] I have included all related issues and possible duplicate issues
  in this issue (If there are none, check this box anyway).

## Mandatory Debugging Information

- [x] I have included the output of ``celery -A proj report`` in the issue.
    (if you are not able to do this, then at least specify the Celery
     version affected).
- [x] I have verified that the issue exists against the `master` branch of Celery.
- [x] I have included the contents of ``pip freeze`` in the issue.
- [x] I have included all the versions of all the external dependencies required
  to reproduce this bug.

## Optional Debugging Information
<!--
Try some of the below if you think they are relevant.
It will help us figure out the scope of the bug and how many users it affects.
-->
- [ ] I have tried reproducing the issue on more than one Python version
  and/or implementation.
- [ ] I have tried reproducing the issue on more than one message broker and/or
  result backend.
- [ ] I have tried reproducing the issue on more than one version of the message
  broker and/or result backend.
- [ ] I have tried reproducing the issue on more than one operating system.
- [ ] I have tried reproducing the issue on more than one workers pool.
- [ ] I have tried reproducing the issue with autoscaling, retries,
  ETA/Countdown & rate limits disabled.
- [ ] I have tried reproducing the issue after downgrading
  and/or upgrading Celery and its dependencies.

## Related Issues and Possible Duplicates
<!--
Please make sure to search and mention any related issues
or possible duplicates to this issue as requested by the checklist above.

This may or may not include issues in other repositories that the Celery project
maintains or other repositories that are dependencies of Celery.

If you don't know how to mention issues, please refer to Github's documentation
on the subject: https://help.github.com/en/articles/autolinked-references-and-urls#issues-and-pull-requests
-->

#### Related Issues

- None

#### Possible Duplicates

- None

## Environment & Settings
<!-- Include the contents of celery --version below -->
**Celery version**:
<!-- Include the output of celery -A proj report below -->
<details>
<summary><b><code>celery report</code> Output:</b></summary>
<p>

```
Commit 02f95470a781369d2b9c4fa7105d879fb0dae3b1 (March 4 2016) and later
```

</p>
</details>

# Steps to Reproduce

The [`task_routes`](http://docs.celeryproject.org/en/latest/userguide/configuration.html#task-routes) configuration parameter allows defining a task route using a glob syntax, for example:

```
task_routes = {
    'feed.tasks.*': 'feeds',
}
```

Under the hood, [celery's code calls `celery.app.routes.glob_to_re`](https://github.com/celery/celery/blob/2c029c4f33d48c1951cc97628655a04024d50a0d/celery/app/routes.py#L36-L38) to convert these globs to a regular expression.  The replacement interprets `*` as the regular expression `.+?`, or, "one or more characters (non-greedy)".

This behavior differs from globbing in other systems, like [bash](https://github.com/celery/celery/blob/2c029c4f33d48c1951cc97628655a04024d50a0d/celery/app/routes.py#L36-L38), [python's glob module](https://docs.python.org/3.5/library/glob.html#glob.glob), [node glob](https://www.npmjs.com/package/glob), and more, which all treat `*` as "zero or more characters".

Given the "zero or more characters" standard, developers will probably expect this pattern:
```
task_routes = {
  'feed.tasks.foo*': 'feeds',
}
```
to match the task `feed.tasks.foo`.  In celery right now, it does not.

## Required Dependencies
<!-- Please fill the required dependencies to reproduce this issue -->
* **Minimal Python Version**: N/A
* **Minimal Celery Version**: N/A
* **Minimal Kombu Version**: N/A
* **Minimal Broker Version**: N/A
* **Minimal Result Backend Version**: N/A
* **Minimal OS and/or Kernel Version**: N/A
* **Minimal Broker Client Version**: N/A
* **Minimal Result Backend Client Version**: N/A

### Python Packages
<!-- Please fill the contents of pip freeze below -->
<details>
<summary><b><code>pip freeze</code> Output:</b></summary>
<p>

```
Not relevant for this ticket.
```

</p>
</details>

### Other Dependencies
<!--
Please provide system dependencies, configuration files
and other dependency information if applicable
-->
<details>
<p>
N/A
</p>
</details>

## Minimally Reproducible Test Case
<!--
Please provide a reproducible test case.
Refer to the Reporting Bugs section in our contribution guide.

We prefer submitting test cases in the form of a PR to our integration test suite.
If you can provide one, please mention the PR number below.
If not, please attach the most minimal code example required to reproduce the issue below.
If the test case is too large, please include a link to a gist or a repository below.
-->

<details>
<p>

```python
from celery.app.routes import glob_to_re

regex = re.compile(glob_to_re('feed.tasks.foo*'))
assert regex.match('feed.tasks.foo')
```

</p>
</details>

# Expected Behavior
<!-- Describe in detail what you expect to happen -->

This task route:
```
task_routes = {
  'feed.tasks.foo*': 'feeds',
}
```
should match a task named `feed.tasks.foo`, in addition to `feed.tasks.foox`.


# Actual Behavior
<!--
Describe in detail what actually happened.
Please include a backtrace and surround it with triple backticks (```).
In addition, include the Celery daemon logs, the broker logs,
the result backend logs and system logs below if they will help us debug
the issue.
-->

THe task `feed.tasks.foo` does not match, as celery expects an additional character to satisfy its interpretation of a glob `*` as "one or more characters (non-greedy)".



Discussion:
Weird, the code seems to be: https://github.com/celery/celery/blob/2c029c4f33d48c1951cc97628655a04024d50a0d/celery/app/routes.py#L36-L38

It ends up as `.+?`, which seems like it should have the same behavior as `.*`, wonder why that wasn't used in the first place...

Looks like that function has no tests either. Was added in #1137, and a comment there suggests using `.*`...
Oops, looks like there are tests, but not specifically for that function, just for the `MapRoute` class.
> Weird, the code seems to be:
> 
> https://github.com/celery/celery/blob/2c029c4f33d48c1951cc97628655a04024d50a0d/celery/app/routes.py#L36-L38
> 
> It ends up as `.+?`, which seems like it should have the same behavior as `.*`, wonder why that wasn't used in the first place...
> 
> Looks like that function has no tests either. Was added in #1137, and a comment there suggests using `.*`...

what would be a possible right solution? did you figure anything?
distutils has an implementation of [glob_to_re](https://www.kite.com/python/docs/distutils.filelist.glob_to_re).
Since the distutils module is being deprecated I propose we copy the code (and attribute it correctly) and see what happens.

```python
def glob_to_re(pattern):
    """Translate a shell-like glob pattern to a regular expression; return
    a string containing the regex.  Differs from 'fnmatch.translate()' in
    that '*' does not match "special characters" (which are
    platform-specific).
    """
    pattern_re = fnmatch.translate(pattern)

    # '?' and '*' in the glob pattern become '.' and '.*' in the RE, which
    # IMHO is wrong -- '?' and '*' aren't supposed to match slash in Unix,
    # and by extension they shouldn't match such "special characters" under
    # any OS.  So change all non-escaped dots in the RE to match any
    # character except the special characters (currently: just os.sep).
    sep = os.sep
    if os.sep == '\\':
        # we're using a regex to manipulate a regex, so we need
        # to escape the backslash twice
        sep = r'\\\\'
    escaped = r'\1[^%s]' % sep
    pattern_re = re.sub(r'((?<!\\)(\\\\)*)\.', escaped, pattern_re)
    return pattern_re
```
It turns out switching to `fnmatch.translate` alone is enough.

Resolving patch:
diff --git a/celery/app/routes.py b/celery/app/routes.py
--- a/celery/app/routes.py
+++ b/celery/app/routes.py
@@ -2,8 +2,8 @@
 
 Contains utilities for working with task routers, (:setting:`task_routes`).
 """
+import fnmatch
 import re
-import string
 from collections import OrderedDict
 from collections.abc import Mapping
 
@@ -23,11 +23,6 @@
 __all__ = ('MapRoute', 'Router', 'prepare')
 
 
-def glob_to_re(glob, quote=string.punctuation.replace('*', '')):
-    glob = ''.join('\\' + c if c in quote else c for c in glob)
-    return glob.replace('*', '.+?')
-
-
 class MapRoute:
     """Creates a router out of a :class:`dict`."""
 
@@ -39,7 +34,7 @@ def __init__(self, map):
             if isinstance(k, Pattern):
                 self.patterns[k] = v
             elif '*' in k:
-                self.patterns[re.compile(glob_to_re(k))] = v
+                self.patterns[re.compile(fnmatch.translate(k))] = v
             else:
                 self.map[k] = v
 
@@ -126,6 +121,7 @@ def expand_router_string(router):
 
 def prepare(routes):
     """Expand the :setting:`task_routes` setting."""
+
     def expand_route(route):
         if isinstance(route, (Mapping, list, tuple)):
             return MapRoute(route)


Test patch:
diff --git a/t/unit/app/test_routes.py b/t/unit/app/test_routes.py
--- a/t/unit/app/test_routes.py
+++ b/t/unit/app/test_routes.py
@@ -16,6 +16,7 @@ def Router(app, *args, **kwargs):
 def E(app, queues):
     def expand(answer):
         return Router(app, [], queues).expand_destination(answer)
+
     return expand
 
 
@@ -46,6 +47,7 @@ def setup(self):
         @self.app.task(shared=False)
         def mytask(*args, **kwargs):
             pass
+
         self.mytask = mytask
 
     def assert_routes_to_queue(self, queue, router, name,
@@ -56,7 +58,8 @@ def assert_routes_to_queue(self, queue, router, name,
             kwargs = {}
         if args is None:
             args = []
-        assert router.route(options, name, args, kwargs)['queue'].name == queue
+        assert router.route(options, name, args, kwargs)[
+                   'queue'].name == queue
 
     def assert_routes_to_default_queue(self, router, name, *args, **kwargs):
         self.assert_routes_to_queue(
@@ -85,10 +88,13 @@ def test_route_for_task__glob(self):
         from re import compile
 
         route = routes.MapRoute([
+            ('proj.tasks.bar*', {'queue': 'routeC'}),
             ('proj.tasks.*', 'routeA'),
             ('demoapp.tasks.bar.*', {'exchange': 'routeB'}),
             (compile(r'(video|image)\.tasks\..*'), {'queue': 'media'}),
         ])
+        assert route('proj.tasks.bar') == {'queue': 'routeC'}
+        assert route('proj.tasks.bar.baz') == {'queue': 'routeC'}
         assert route('proj.tasks.foo') == {'queue': 'routeA'}
         assert route('demoapp.tasks.bar.moo') == {'exchange': 'routeB'}
         assert route('video.tasks.foo') == {'queue': 'media'}
@@ -97,7 +103,7 @@ def test_route_for_task__glob(self):
 
     def test_expand_route_not_found(self):
         expand = E(self.app, self.app.amqp.Queues(
-                   self.app.conf.task_queues, False))
+            self.app.conf.task_queues, False))
         route = routes.MapRoute({'a': {'queue': 'x'}})
         with pytest.raises(QueueNotFound):
             expand(route('a'))