Repository: apache/airflow

Issue:
filesensor wildcard matching does not recognize directories
**Apache Airflow version**: 2.1.0


**Environment**:

- **Cloud provider or hardware configuration**:
- **OS** (e.g. from /etc/os-release):
- **Kernel** (e.g. `uname -a`):
- **Install tools**:
- **Others**:

**What happened**:  FileSensor does not recognize directories with wildcard glob matching.

**What you expected to happen**: FileSensor would sense a directory that contains files if it matches with the wild card option.

**How to reproduce it**: Create a directory with a pattern that matches a wild card using glob



**Anything else we need to know**: Code from FileSensor source that I believe to cause the issue:

```
for path in glob(full_path):
            if os.path.isfile(path):
                mod_time = os.path.getmtime(path)
                mod_time = datetime.datetime.fromtimestamp(mod_time).strftime('%Y%m%d%H%M%S')
                self.log.info('Found File %s last modified: %s', str(path), str(mod_time))
                return True

            for _, _, files in os.walk(full_path):
                if len(files) > 0:
                    return True
        return False

```
I believe to resolve the issue `full_path` in os.walk should be `path` instead.




Discussion:
Thanks for opening your first issue here! Be sure to follow the issue template!

Would you be willing to open PR for that ? 
We should change the logic to use `pathlib.Path.glob()` instead, which automatically handles recursive globbing. (Or `glob.rglob()` but pathlib is easier to use IMO.)
I can open a PR for the quick fix to substitute path for full_path in the walk, but I'm not very familiar with `pathlib.Path.glob()` to be able to make that change in the logic. Also never submitted a PR to a big project like Airflow before (pretty new to coding in general) so gotta go figure out that process.
Hmm, after reading the relevant code, `pathlib.Path.glob()` might not be the best choice. It’s easier to do `glob(full_path, recursive=True)` instead. So you’ll need to do that. After that, please follow the instructions in https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst to submit a pull request for your change.

Resolving patch:
diff --git a/airflow/sensors/filesystem.py b/airflow/sensors/filesystem.py
--- a/airflow/sensors/filesystem.py
+++ b/airflow/sensors/filesystem.py
@@ -38,15 +38,19 @@ class FileSensor(BaseSensorOperator):
     :param filepath: File or folder name (relative to
         the base path set within the connection), can be a glob.
     :type filepath: str
+    :param recursive: when set to ``True``, enables recursive directory matching behavior of
+        ``**`` in glob filepath parameter. Defaults to ``False``.
+    :type recursive: bool
     """
 
     template_fields = ('filepath',)
     ui_color = '#91818a'
 
-    def __init__(self, *, filepath, fs_conn_id='fs_default', **kwargs):
+    def __init__(self, *, filepath, fs_conn_id='fs_default', recursive=False, **kwargs):
         super().__init__(**kwargs)
         self.filepath = filepath
         self.fs_conn_id = fs_conn_id
+        self.recursive = recursive
 
     def poke(self, context):
         hook = FSHook(self.fs_conn_id)
@@ -54,7 +58,7 @@ def poke(self, context):
         full_path = os.path.join(basepath, self.filepath)
         self.log.info('Poking for file %s', full_path)
 
-        for path in glob(full_path):
+        for path in glob(full_path, recursive=self.recursive):
             if os.path.isfile(path):
                 mod_time = os.path.getmtime(path)
                 mod_time = datetime.datetime.fromtimestamp(mod_time).strftime('%Y%m%d%H%M%S')


Test patch:
diff --git a/tests/sensors/test_filesystem.py b/tests/sensors/test_filesystem.py
--- a/tests/sensors/test_filesystem.py
+++ b/tests/sensors/test_filesystem.py
@@ -131,6 +131,27 @@ def test_wildcard_file(self):
             task._hook = self.hook
             task.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE, ignore_ti_state=True)
 
+    def test_wildcared_directory(self):
+        temp_dir = tempfile.mkdtemp()
+        subdir = tempfile.mkdtemp(dir=temp_dir)
+        task = FileSensor(
+            task_id='test',
+            filepath=temp_dir + "/**",
+            fs_conn_id='fs_default',
+            dag=self.dag,
+            timeout=0,
+            poke_interval=1,
+            recursive=True,
+        )
+        task._hook = self.hook
+
+        try:
+            # `touch` the dir
+            open(subdir + "/file", "a").close()
+            task.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE, ignore_ti_state=True)
+        finally:
+            shutil.rmtree(temp_dir)
+
     def test_subdirectory_not_empty(self):
         suffix = '.txt'
         temp_dir = tempfile.mkdtemp()