Repository: apache/airflow

Issue:
Bytes cast to String in `airflow.providers.google.cloud.transfers.gcs_to_local.GCSToLocalFilesystemOperator` ~142
### Apache Airflow Provider(s)

google

### Versions of Apache Airflow Providers

```
$ pip freeze | grep apache-airflow-providers
apache-airflow-providers-ftp==2.0.1
apache-airflow-providers-google==6.3.0
apache-airflow-providers-http==2.0.2
apache-airflow-providers-imap==2.1.0
apache-airflow-providers-pagerduty==2.1.0
apache-airflow-providers-sftp==2.4.0
apache-airflow-providers-sqlite==2.0.1
apache-airflow-providers-ssh==2.3.0
```

### Apache Airflow version

2.2.3 (latest released)

### Operating System

Ubuntu 20.04.3 LTS

### Deployment

Composer

### Deployment details

_No response_

### What happened

Using `airflow.providers.google.cloud.transfers.gcs_to_local.GCSToLocalFilesystemOperator` to load the contents of a file into `xcom` unexpectedly casts the file bytes to string.

### What you expected to happen

`GCSToLocalFilesystemOperator` should not cast to string

### How to reproduce

Store a file on gcs;
```
Hello World!
```

Read file to xcom
```
my_task = GCSToLocalFilesystemOperator(
    task_id='my_task',
    bucket=bucket,
    object_name=object_path,
    store_to_xcom_key='my_xcom_key',
)
```

Access via jinja;
```
{{ ti.xcom_pull(task_ids="my_task", key="my_xcom_key") }}
```

XCom result is;
```
b'Hello World!'
```

### Anything else

_No response_

### Are you willing to submit PR?

- [X] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Bytes cast to String in `airflow.providers.google.cloud.transfers.gcs_to_local.GCSToLocalFilesystemOperator` ~142
### Apache Airflow Provider(s)

google

### Versions of Apache Airflow Providers

```
$ pip freeze | grep apache-airflow-providers
apache-airflow-providers-ftp==2.0.1
apache-airflow-providers-google==6.3.0
apache-airflow-providers-http==2.0.2
apache-airflow-providers-imap==2.1.0
apache-airflow-providers-pagerduty==2.1.0
apache-airflow-providers-sftp==2.4.0
apache-airflow-providers-sqlite==2.0.1
apache-airflow-providers-ssh==2.3.0
```

### Apache Airflow version

2.2.3 (latest released)

### Operating System

Ubuntu 20.04.3 LTS

### Deployment

Composer

### Deployment details

_No response_

### What happened

Using `airflow.providers.google.cloud.transfers.gcs_to_local.GCSToLocalFilesystemOperator` to load the contents of a file into `xcom` unexpectedly casts the file bytes to string.

### What you expected to happen

`GCSToLocalFilesystemOperator` should not cast to string

### How to reproduce

Store a file on gcs;
```
Hello World!
```

Read file to xcom
```
my_task = GCSToLocalFilesystemOperator(
    task_id='my_task',
    bucket=bucket,
    object_name=object_path,
    store_to_xcom_key='my_xcom_key',
)
```

Access via jinja;
```
{{ ti.xcom_pull(task_ids="my_task", key="my_xcom_key") }}
```

XCom result is;
```
b'Hello World!'
```

### Anything else

_No response_

### Are you willing to submit PR?

- [X] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Bytes cast to String in `airflow.providers.google.cloud.transfers.gcs_to_local.GCSToLocalFilesystemOperator` ~142
### Apache Airflow Provider(s)

google

### Versions of Apache Airflow Providers

```
$ pip freeze | grep apache-airflow-providers
apache-airflow-providers-ftp==2.0.1
apache-airflow-providers-google==6.3.0
apache-airflow-providers-http==2.0.2
apache-airflow-providers-imap==2.1.0
apache-airflow-providers-pagerduty==2.1.0
apache-airflow-providers-sftp==2.4.0
apache-airflow-providers-sqlite==2.0.1
apache-airflow-providers-ssh==2.3.0
```

### Apache Airflow version

2.2.3 (latest released)

### Operating System

Ubuntu 20.04.3 LTS

### Deployment

Composer

### Deployment details

_No response_

### What happened

Using `airflow.providers.google.cloud.transfers.gcs_to_local.GCSToLocalFilesystemOperator` to load the contents of a file into `xcom` unexpectedly casts the file bytes to string.

### What you expected to happen

`GCSToLocalFilesystemOperator` should not cast to string

### How to reproduce

Store a file on gcs;
```
Hello World!
```

Read file to xcom
```
my_task = GCSToLocalFilesystemOperator(
    task_id='my_task',
    bucket=bucket,
    object_name=object_path,
    store_to_xcom_key='my_xcom_key',
)
```

Access via jinja;
```
{{ ti.xcom_pull(task_ids="my_task", key="my_xcom_key") }}
```

XCom result is;
```
b'Hello World!'
```

### Anything else

_No response_

### Are you willing to submit PR?

- [X] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)



Discussion:
Thanks for opening your first issue here! Be sure to follow the issue template!

This is expected. We cannot treat object stored in GCS as "string". It's always retrieved as Bytes. If you know what encoding is used you can easily decode it `ti.xcom_pull(task_ids="my_task", key="my_xcom_key").decode('utf-8') for example.
@potiuk please look at the code here; https://github.com/apache/airflow/blob/main/airflow/providers/google/cloud/transfers/gcs_to_local.py#L142

This is a bug.

You cannot cast `byte` to `str` in this way and successfully decode it after. The `str()` method must be supplied with an `encoding` param like this;

```
str(file_bytes, encoding='utf-8')
```
Ah. I see - this actually converts the value of the bytes to "nice string" representation: https://docs.python.org/3/library/stdtypes.html#str

> Passing a bytes object to str() without the encoding or errors arguments falls under the first case of returning the informal string representation (see also the -b command-line option to Python). For example:

```
str(b'Zoot!')
"b'Zoot!'"
```

So those are not "bytes" but string containing `b'` :scream: 

Yeah. Agree it's a bug then. Would you like to fix it maybe? What do you think should be the fix @danneaves-ee  ?
Thanks for reopening @potiuk :smile: 

GCS provides a "Content-Encoding" meta key, however, it's optional so I guess not many people would use it. 

Since XComs must be serializable, we have replaced the lines as follows in our work-around;
```
# airflow.providers.google.cloud.transfers.gcs_to_local.GCSToLocalFilesystemOperator ~142
- context['ti'].xcom_push(key=self.store_to_xcom_key, value=str(file_bytes))
+ context['ti'].xcom_push(key=self.store_to_xcom_key, value=str(file_bytes, encoding='utf-8'))
```

But it could be safer to use `sys.getdefaultencoding()` instead of `utf-8`. Or even expose the encoding as a parameter to the class?

What do you think @potiuk?

I think the problem is that we really want to store binary data as string. What I thing should be really good solution is to convert the bytes into string assuming that the bytes follow the encoding specified. And if the data is encoded using different encoding, it will fail.

I think adding optional "encoding" parameter which falls-back to `utf-8` is a better idea. getdefaultencoding is something I do not really trust in Python. In many places (including str() built-in function, encoding=utf-8 is default rather than getdefaultencoding.
I assigned you to it :)
Thanks for opening your first issue here! Be sure to follow the issue template!

This is expected. We cannot treat object stored in GCS as "string". It's always retrieved as Bytes. If you know what encoding is used you can easily decode it `ti.xcom_pull(task_ids="my_task", key="my_xcom_key").decode('utf-8') for example.
@potiuk please look at the code here; https://github.com/apache/airflow/blob/main/airflow/providers/google/cloud/transfers/gcs_to_local.py#L142

This is a bug.

You cannot cast `byte` to `str` in this way and successfully decode it after. The `str()` method must be supplied with an `encoding` param like this;

```
str(file_bytes, encoding='utf-8')
```
Ah. I see - this actually converts the value of the bytes to "nice string" representation: https://docs.python.org/3/library/stdtypes.html#str

> Passing a bytes object to str() without the encoding or errors arguments falls under the first case of returning the informal string representation (see also the -b command-line option to Python). For example:

```
str(b'Zoot!')
"b'Zoot!'"
```

So those are not "bytes" but string containing `b'` :scream: 

Yeah. Agree it's a bug then. Would you like to fix it maybe? What do you think should be the fix @danneaves-ee  ?
Thanks for reopening @potiuk :smile: 

GCS provides a "Content-Encoding" meta key, however, it's optional so I guess not many people would use it. 

Since XComs must be serializable, we have replaced the lines as follows in our work-around;
```
# airflow.providers.google.cloud.transfers.gcs_to_local.GCSToLocalFilesystemOperator ~142
- context['ti'].xcom_push(key=self.store_to_xcom_key, value=str(file_bytes))
+ context['ti'].xcom_push(key=self.store_to_xcom_key, value=str(file_bytes, encoding='utf-8'))
```

But it could be safer to use `sys.getdefaultencoding()` instead of `utf-8`. Or even expose the encoding as a parameter to the class?

What do you think @potiuk?

I think the problem is that we really want to store binary data as string. What I thing should be really good solution is to convert the bytes into string assuming that the bytes follow the encoding specified. And if the data is encoded using different encoding, it will fail.

I think adding optional "encoding" parameter which falls-back to `utf-8` is a better idea. getdefaultencoding is something I do not really trust in Python. In many places (including str() built-in function, encoding=utf-8 is default rather than getdefaultencoding.
I assigned you to it :)
Thanks for opening your first issue here! Be sure to follow the issue template!

This is expected. We cannot treat object stored in GCS as "string". It's always retrieved as Bytes. If you know what encoding is used you can easily decode it `ti.xcom_pull(task_ids="my_task", key="my_xcom_key").decode('utf-8') for example.
@potiuk please look at the code here; https://github.com/apache/airflow/blob/main/airflow/providers/google/cloud/transfers/gcs_to_local.py#L142

This is a bug.

You cannot cast `byte` to `str` in this way and successfully decode it after. The `str()` method must be supplied with an `encoding` param like this;

```
str(file_bytes, encoding='utf-8')
```
Ah. I see - this actually converts the value of the bytes to "nice string" representation: https://docs.python.org/3/library/stdtypes.html#str

> Passing a bytes object to str() without the encoding or errors arguments falls under the first case of returning the informal string representation (see also the -b command-line option to Python). For example:

```
str(b'Zoot!')
"b'Zoot!'"
```

So those are not "bytes" but string containing `b'` :scream: 

Yeah. Agree it's a bug then. Would you like to fix it maybe? What do you think should be the fix @danneaves-ee  ?
Thanks for reopening @potiuk :smile: 

GCS provides a "Content-Encoding" meta key, however, it's optional so I guess not many people would use it. 

Since XComs must be serializable, we have replaced the lines as follows in our work-around;
```
# airflow.providers.google.cloud.transfers.gcs_to_local.GCSToLocalFilesystemOperator ~142
- context['ti'].xcom_push(key=self.store_to_xcom_key, value=str(file_bytes))
+ context['ti'].xcom_push(key=self.store_to_xcom_key, value=str(file_bytes, encoding='utf-8'))
```

But it could be safer to use `sys.getdefaultencoding()` instead of `utf-8`. Or even expose the encoding as a parameter to the class?

What do you think @potiuk?

I think the problem is that we really want to store binary data as string. What I thing should be really good solution is to convert the bytes into string assuming that the bytes follow the encoding specified. And if the data is encoded using different encoding, it will fail.

I think adding optional "encoding" parameter which falls-back to `utf-8` is a better idea. getdefaultencoding is something I do not really trust in Python. In many places (including str() built-in function, encoding=utf-8 is default rather than getdefaultencoding.
I assigned you to it :)

Resolving patch:
diff --git a/airflow/providers/google/cloud/transfers/gcs_to_local.py b/airflow/providers/google/cloud/transfers/gcs_to_local.py
--- a/airflow/providers/google/cloud/transfers/gcs_to_local.py
+++ b/airflow/providers/google/cloud/transfers/gcs_to_local.py
@@ -72,6 +72,9 @@ class GCSToLocalFilesystemOperator(BaseOperator):
         Service Account Token Creator IAM role to the directly preceding identity, with first
         account from the list granting this role to the originating account (templated).
     :type impersonation_chain: Union[str, Sequence[str]]
+    :param file_encoding: Optional encoding used to decode file_bytes into a serializable
+        string that is suitable for storing to XCom. (templated).
+    :type file_encoding: str
     """
 
     template_fields: Sequence[str] = (
@@ -80,6 +83,7 @@ class GCSToLocalFilesystemOperator(BaseOperator):
         'filename',
         'store_to_xcom_key',
         'impersonation_chain',
+        'file_encoding',
     )
     ui_color = '#f0eee4'
 
@@ -94,6 +98,7 @@ def __init__(
         google_cloud_storage_conn_id: Optional[str] = None,
         delegate_to: Optional[str] = None,
         impersonation_chain: Optional[Union[str, Sequence[str]]] = None,
+        file_encoding: Optional[str] = 'utf-8',
         **kwargs,
     ) -> None:
         # To preserve backward compatibility
@@ -126,6 +131,7 @@ def __init__(
         self.gcp_conn_id = gcp_conn_id
         self.delegate_to = delegate_to
         self.impersonation_chain = impersonation_chain
+        self.file_encoding = file_encoding
 
     def execute(self, context: 'Context'):
         self.log.info('Executing download: %s, %s, %s', self.bucket, self.object_name, self.filename)
@@ -139,7 +145,7 @@ def execute(self, context: 'Context'):
             file_size = hook.get_size(bucket_name=self.bucket, object_name=self.object_name)
             if file_size < MAX_XCOM_SIZE:
                 file_bytes = hook.download(bucket_name=self.bucket, object_name=self.object_name)
-                context['ti'].xcom_push(key=self.store_to_xcom_key, value=str(file_bytes))
+                context['ti'].xcom_push(key=self.store_to_xcom_key, value=str(file_bytes, self.file_encoding))
             else:
                 raise AirflowException('The size of the downloaded file is too large to push to XCom!')
         else:


Test patch:
diff --git a/tests/providers/google/cloud/transfers/test_gcs_to_local.py b/tests/providers/google/cloud/transfers/test_gcs_to_local.py
--- a/tests/providers/google/cloud/transfers/test_gcs_to_local.py
+++ b/tests/providers/google/cloud/transfers/test_gcs_to_local.py
@@ -34,7 +34,11 @@
 TEST_OBJECT = "dir1/test-object"
 LOCAL_FILE_PATH = "/home/airflow/gcp/test-object"
 XCOM_KEY = "some_xkom_key"
-FILE_CONTENT = "some file content"
+FILE_CONTENT_STR = "some file content"
+FILE_CONTENT_BYTES_UTF8 = b"some file content"
+FILE_CONTENT_BYTES_UTF16 = (
+    b'\xff\xfes\x00o\x00m\x00e\x00 \x00f\x00i\x00l\x00e\x00 \x00c\x00o\x00n\x00t\x00e\x00n\x00t\x00'
+)
 
 
 class TestGoogleCloudStorageDownloadOperator(unittest.TestCase):
@@ -61,7 +65,7 @@ def test_size_lt_max_xcom_size(self, mock_hook):
             store_to_xcom_key=XCOM_KEY,
         )
         context = {"ti": MagicMock()}
-        mock_hook.return_value.download.return_value = FILE_CONTENT
+        mock_hook.return_value.download.return_value = FILE_CONTENT_BYTES_UTF8
         mock_hook.return_value.get_size.return_value = MAX_XCOM_SIZE - 1
 
         operator.execute(context=context)
@@ -71,7 +75,7 @@ def test_size_lt_max_xcom_size(self, mock_hook):
         mock_hook.return_value.download.assert_called_once_with(
             bucket_name=TEST_BUCKET, object_name=TEST_OBJECT
         )
-        context["ti"].xcom_push.assert_called_once_with(key=XCOM_KEY, value=FILE_CONTENT)
+        context["ti"].xcom_push.assert_called_once_with(key=XCOM_KEY, value=FILE_CONTENT_STR)
 
     @mock.patch("airflow.providers.google.cloud.transfers.gcs_to_local.GCSHook")
     def test_size_gt_max_xcom_size(self, mock_hook):
@@ -82,8 +86,30 @@ def test_size_gt_max_xcom_size(self, mock_hook):
             store_to_xcom_key=XCOM_KEY,
         )
         context = {"ti": MagicMock()}
-        mock_hook.return_value.download.return_value = FILE_CONTENT
+        mock_hook.return_value.download.return_value = FILE_CONTENT_BYTES_UTF8
         mock_hook.return_value.get_size.return_value = MAX_XCOM_SIZE + 1
 
         with pytest.raises(AirflowException, match="file is too large"):
             operator.execute(context=context)
+
+    @mock.patch("airflow.providers.google.cloud.transfers.gcs_to_local.GCSHook")
+    def test_xcom_encoding(self, mock_hook):
+        operator = GCSToLocalFilesystemOperator(
+            task_id=TASK_ID,
+            bucket=TEST_BUCKET,
+            object_name=TEST_OBJECT,
+            store_to_xcom_key=XCOM_KEY,
+            file_encoding='utf-16',
+        )
+        context = {"ti": MagicMock()}
+        mock_hook.return_value.download.return_value = FILE_CONTENT_BYTES_UTF16
+        mock_hook.return_value.get_size.return_value = MAX_XCOM_SIZE - 1
+
+        operator.execute(context=context)
+        mock_hook.return_value.get_size.assert_called_once_with(
+            bucket_name=TEST_BUCKET, object_name=TEST_OBJECT
+        )
+        mock_hook.return_value.download.assert_called_once_with(
+            bucket_name=TEST_BUCKET, object_name=TEST_OBJECT
+        )
+        context["ti"].xcom_push.assert_called_once_with(key=XCOM_KEY, value=FILE_CONTENT_STR)