Repository: apache/airflow

Issue:
S3KeySensor wildcard_match only matching key prefixes instead of full patterns
### Apache Airflow Provider(s)

amazon

### Versions of Apache Airflow Providers

3.4.0

### Apache Airflow version

2.3.2 (latest released)

### Operating System

Debian GNU/Linux 10

### Deployment

Docker-Compose

### Deployment details

_No response_

### What happened

For patterns like "*.zip" the S3KeySensor succeeds for all files, does not take full pattern into account i.e. the ".zip" part). 
Bug introduced in https://github.com/apache/airflow/pull/22737

### What you think should happen instead

Full pattern match as in version 3.3.0 (in S3KeySensor poke()):

```
...
if self.wildcard_match:
            return self.get_hook().check_for_wildcard_key(self.bucket_key, self.bucket_name)
...
```

alternatively the files obtained by `files = self.get_hook().get_file_metadata(prefix, bucket_name)` which only match the prefix should be further filtered.

### How to reproduce

create a DAG with a key sensor task containing wildcard and suffix, e.g. the following task should succeed only if any ZIP-files are available in "my-bucket", but succeeds for all instead:

`S3KeySensor(task_id="wait_for_file", bucket_name="my-bucket", bucket_key="*.zip", wildcard_match=True)`

### Anything else

Not directly part of this issue but at the same time I would suggest to include additional file attributes in method _check_key, e.g. the actual key of the files. This way more filters, e.g. exclude specific keys, could be implemented by using the check_fn.

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)



Discussion:
Thanks for opening your first issue here! Be sure to follow the issue template!

Would it be possible you submit a fix for it @wegmatho ?  I think that's the fastest way to fix it. And I am about to release provider's release so quick fix to that might be released in a couple of days if you manage to fix it.
cc @vincbeck 
I'll look into it

Resolving patch:
diff --git a/airflow/providers/amazon/aws/sensors/s3.py b/airflow/providers/amazon/aws/sensors/s3.py
--- a/airflow/providers/amazon/aws/sensors/s3.py
+++ b/airflow/providers/amazon/aws/sensors/s3.py
@@ -16,7 +16,7 @@
 # specific language governing permissions and limitations
 # under the License.
 #
-
+import fnmatch
 import os
 import re
 import sys
@@ -112,12 +112,13 @@ def _check_key(self, key):
         """
         if self.wildcard_match:
             prefix = re.split(r'[\[\*\?]', key, 1)[0]
-            files = self.get_hook().get_file_metadata(prefix, bucket_name)
-            if len(files) == 0:
+            keys = self.get_hook().get_file_metadata(prefix, bucket_name)
+            key_matches = [k for k in keys if fnmatch.fnmatch(k['Key'], key)]
+            if len(key_matches) == 0:
                 return False
 
             # Reduce the set of metadata to size only
-            files = list(map(lambda f: {'Size': f['Size']}, files))
+            files = list(map(lambda f: {'Size': f['Size']}, key_matches))
         else:
             obj = self.get_hook().head_object(key, bucket_name)
             if obj is None:
@@ -341,8 +342,7 @@ def __init__(
             stacklevel=2,
         )
 
-        self.prefix = prefix
-        prefixes = [self.prefix] if isinstance(self.prefix, str) else self.prefix
+        prefixes = [prefix] if isinstance(prefix, str) else prefix
         keys = [pref if pref.endswith(delimiter) else pref + delimiter for pref in prefixes]
 
         super().__init__(bucket_key=keys, **kwargs)


Test patch:
diff --git a/tests/providers/amazon/aws/sensors/test_s3_key.py b/tests/providers/amazon/aws/sensors/test_s3_key.py
--- a/tests/providers/amazon/aws/sensors/test_s3_key.py
+++ b/tests/providers/amazon/aws/sensors/test_s3_key.py
@@ -160,25 +160,31 @@ def test_poke_wildcard(self, mock_get_file_metadata):
         assert op.poke(None) is False
         mock_get_file_metadata.assert_called_once_with("file", "test_bucket")
 
-        mock_get_file_metadata.return_value = [{'Size': 0}]
+        mock_get_file_metadata.return_value = [{'Key': 'dummyFile', 'Size': 0}]
+        assert op.poke(None) is False
+
+        mock_get_file_metadata.return_value = [{'Key': 'file1', 'Size': 0}]
         assert op.poke(None) is True
 
     @mock.patch('airflow.providers.amazon.aws.sensors.s3.S3Hook.get_file_metadata')
     def test_poke_wildcard_multiple_files(self, mock_get_file_metadata):
         op = S3KeySensor(
             task_id='s3_key_sensor',
-            bucket_key=['s3://test_bucket/file1*', 's3://test_bucket/file2*'],
+            bucket_key=['s3://test_bucket/file*', 's3://test_bucket/*.zip'],
             wildcard_match=True,
         )
 
-        mock_get_file_metadata.side_effect = [[{'Size': 0}], []]
+        mock_get_file_metadata.side_effect = [[{'Key': 'file1', 'Size': 0}], []]
+        assert op.poke(None) is False
+
+        mock_get_file_metadata.side_effect = [[{'Key': 'file1', 'Size': 0}], [{'Key': 'file2', 'Size': 0}]]
         assert op.poke(None) is False
 
-        mock_get_file_metadata.side_effect = [[{'Size': 0}], [{'Size': 0}]]
+        mock_get_file_metadata.side_effect = [[{'Key': 'file1', 'Size': 0}], [{'Key': 'test.zip', 'Size': 0}]]
         assert op.poke(None) is True
 
-        mock_get_file_metadata.assert_any_call("file1", "test_bucket")
-        mock_get_file_metadata.assert_any_call("file2", "test_bucket")
+        mock_get_file_metadata.assert_any_call("file", "test_bucket")
+        mock_get_file_metadata.assert_any_call("", "test_bucket")
 
     @mock.patch('airflow.providers.amazon.aws.sensors.s3.S3Hook.head_object')
     def test_poke_with_check_function(self, mock_head_object):