Repository: apache/airflow

Issue:
The S3ToGCSOperator fails on templated `dest_gcs` URL
<!--

Welcome to Apache Airflow!  For a smooth issue process, try to answer the following questions.
Don't worry if they're not all applicable; just try to include what you can :-)

If you need to include code snippets or logs, please put them in fenced code
blocks.  If they're super-long, please use the details tag like
<details><summary>super-long log</summary> lots of stuff </details>

Please delete these comment blocks before submitting the issue.

-->

<!--

IMPORTANT!!!

PLEASE CHECK "SIMILAR TO X EXISTING ISSUES" OPTION IF VISIBLE
NEXT TO "SUBMIT NEW ISSUE" BUTTON!!!

PLEASE CHECK IF THIS ISSUE HAS BEEN REPORTED PREVIOUSLY USING SEARCH!!!

Please complete the next sections or the issue will be closed.
These questions are the first thing we need to know to understand the context.

-->

**Apache Airflow version**:


**Kubernetes version (if you are using kubernetes)** (use `kubectl version`):

**Environment**:

- **Cloud provider or hardware configuration**:
- **OS** (e.g. from /etc/os-release):
- **Kernel** (e.g. `uname -a`):
- **Install tools**: Docker

**What happened**:

When passing a templatized `dest_gcs` argument to the `S3ToGCSOperator` operator, the DAG fails to import because the constructor attempts to test the validity of the URL before the template has been populated in `execute`.

The error is:

```
Broken DAG: [/opt/airflow/dags/bad_gs_dag.py] Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/google/cloud/hooks/gcs.py", line 1051, in gcs_object_is_directory
    _, blob = _parse_gcs_url(bucket)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/google/cloud/hooks/gcs.py", line 1063, in _parse_gcs_url
    raise AirflowException('Please provide a bucket name')
airflow.exceptions.AirflowException: Please provide a bucket name
```

**What you expected to happen**:

The DAG should successfully parse when using a templatized `dest_gcs` value.

**How to reproduce it**:

Instantiating a `S3ToGCSOperator` task with `dest_gcs="{{ var.gcs_url }}"` fails. 

<details>

```python
from airflow.decorators import dag
from airflow.utils.dates import days_ago
from airflow.providers.google.cloud.transfers.s3_to_gcs import S3ToGCSOperator


@dag(
    schedule_interval=None,
    description="Demo S3-to-GS Bug",
    catchup=False,
    start_date=days_ago(1),
)
def demo_bug():

    S3ToGCSOperator(
        task_id="transfer_task",
        bucket="example_bucket",
        prefix="fake/prefix",
        dest_gcs="{{ var.gcs_url }}",
    )


demo_dag = demo_bug()
```
</details>


**Anything else we need to know**:

Should be fixable by moving the code that evaluates whether the URL is a folder to `execute()`.



Discussion:
Thanks for opening your first issue here! Be sure to follow the issue template!

It may be the same case for `AzureFileShareToGCSOperator`
I'd like to work on this issue. Seems a similar one might affect other operators as well - @TobKed mentioned [AzureFileShareToGCSOperator](https://github.com/apache/airflow/blob/866a601b76e219b3c043e1dbbc8fb22300866351/airflow/providers/google/cloud/transfers/azure_fileshare_to_gcs.py#L106-L114). [`LocalFilesystemToS3Operator`](https://github.com/apache/airflow/blob/1632c9f519510ff218656bbc1554c80cb158e85a/airflow/providers/amazon/aws/transfers/local_to_s3.py#L97-L98) might work incorrect as well.
Basically any validation of input parameter, which can be templated, shouldn't be inside operator's `__init__` function, as its real value will be available at earliest only during task preparation stage.

How it'd be better to handle? Move validation to the top of `execute()` function?

I'll search then for other potential operators which might be affected

Resolving patch:
diff --git a/airflow/providers/amazon/aws/transfers/local_to_s3.py b/airflow/providers/amazon/aws/transfers/local_to_s3.py
--- a/airflow/providers/amazon/aws/transfers/local_to_s3.py
+++ b/airflow/providers/amazon/aws/transfers/local_to_s3.py
@@ -94,10 +94,12 @@ def __init__(
         self.gzip = gzip
         self.acl_policy = acl_policy
 
+    def _check_inputs(self):
         if 's3://' in self.dest_key and self.dest_bucket is not None:
             raise TypeError('dest_bucket should be None when dest_key is provided as a full s3:// file path.')
 
     def execute(self, context):
+        self._check_inputs()
         s3_hook = S3Hook(aws_conn_id=self.aws_conn_id, verify=self.verify)
         s3_hook.load_file(
             self.filename,
diff --git a/airflow/providers/google/cloud/transfers/azure_fileshare_to_gcs.py b/airflow/providers/google/cloud/transfers/azure_fileshare_to_gcs.py
--- a/airflow/providers/google/cloud/transfers/azure_fileshare_to_gcs.py
+++ b/airflow/providers/google/cloud/transfers/azure_fileshare_to_gcs.py
@@ -103,7 +103,8 @@ def __init__(
         self.gzip = gzip
         self.google_impersonation_chain = google_impersonation_chain
 
-        if dest_gcs and not gcs_object_is_directory(self.dest_gcs):
+    def _check_inputs(self) -> None:
+        if self.dest_gcs and not gcs_object_is_directory(self.dest_gcs):
             self.log.info(
                 'Destination Google Cloud Storage path is not a valid '
                 '"directory", define a path that ends with a slash "/" or '
@@ -114,6 +115,7 @@ def __init__(
             )
 
     def execute(self, context):
+        self._check_inputs()
         azure_fileshare_hook = AzureFileShareHook(self.azure_fileshare_conn_id)
         files = azure_fileshare_hook.list_files(
             share_name=self.share_name, directory_name=self.directory_name
diff --git a/airflow/providers/google/cloud/transfers/s3_to_gcs.py b/airflow/providers/google/cloud/transfers/s3_to_gcs.py
--- a/airflow/providers/google/cloud/transfers/s3_to_gcs.py
+++ b/airflow/providers/google/cloud/transfers/s3_to_gcs.py
@@ -147,7 +147,8 @@ def __init__(
         self.gzip = gzip
         self.google_impersonation_chain = google_impersonation_chain
 
-        if dest_gcs and not gcs_object_is_directory(self.dest_gcs):
+    def _check_inputs(self) -> None:
+        if self.dest_gcs and not gcs_object_is_directory(self.dest_gcs):
             self.log.info(
                 'Destination Google Cloud Storage path is not a valid '
                 '"directory", define a path that ends with a slash "/" or '
@@ -158,6 +159,7 @@ def __init__(
             )
 
     def execute(self, context):
+        self._check_inputs()
         # use the super method to list all the files in an S3 bucket/key
         files = super().execute(context)
 


Test patch:
diff --git a/tests/providers/amazon/aws/transfers/test_local_to_s3.py b/tests/providers/amazon/aws/transfers/test_local_to_s3.py
--- a/tests/providers/amazon/aws/transfers/test_local_to_s3.py
+++ b/tests/providers/amazon/aws/transfers/test_local_to_s3.py
@@ -61,16 +61,17 @@ def test_init(self):
         assert operator.encrypt == self._config['encrypt']
         assert operator.gzip == self._config['gzip']
 
-    def test_init_exception(self):
+    def test_execute_exception(self):
+        operator = LocalFilesystemToS3Operator(
+            task_id='file_to_s3_operatro_exception',
+            dag=self.dag,
+            filename=self.testfile1,
+            dest_key=f's3://dummy/{self.dest_key}',
+            dest_bucket=self.dest_bucket,
+            **self._config,
+        )
         with self.assertRaises(TypeError):
-            LocalFilesystemToS3Operator(
-                task_id='file_to_s3_operatro_exception',
-                dag=self.dag,
-                filename=self.testfile1,
-                dest_key=f's3://dummy/{self.dest_key}',
-                dest_bucket=self.dest_bucket,
-                **self._config,
-            )
+            operator.execute(None)
 
     @mock_s3
     def test_execute(self):