Repository: apache/airflow

Issue:
[AIRFLOW-4438] Add Gzip compression to S3_hook
 - Added bool parameter gzip to load_file to s3_hook
 - Tested the load_file with load_file_gzip unittest
 - Updated the load_file docstring to reflect the extra parameter

---
Issue link: [AIRFLOW-4438](https://issues.apache.org/jira/browse/AIRFLOW-4438)

Make sure to mark the boxes below before creating PR: [x]

- [x] Description above provides context of the change
- [x] Commit message/PR title starts with `[AIRFLOW-NNNN]`. AIRFLOW-NNNN = JIRA ID<sup>*</sup>
- [x] Unit tests coverage for changes (not needed for documentation changes)
- [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)"
- [x] Relevant documentation is updated including usage instructions.
- [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).

<sup>*</sup> For document-only changes commit message can start with `[AIRFLOW-XXXX]`.

---
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information.



Discussion:
# [Codecov](https://codecov.io/gh/apache/airflow/pull/7680?src=pr&el=h1) Report
> Merging [#7680](https://codecov.io/gh/apache/airflow/pull/7680?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/e6af02fdf4fa431739f49e444c93c54ca9f5a8e0?src=pr&el=desc) will **decrease** coverage by `22.52%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/apache/airflow/pull/7680/graphs/tree.svg?width=650&token=WdLKlKHOAU&height=150&src=pr)](https://codecov.io/gh/apache/airflow/pull/7680?src=pr&el=tree)

```diff
@@             Coverage Diff             @@
##           master    #7680       +/-   ##
===========================================
- Coverage   86.99%   64.47%   -22.53%     
===========================================
  Files         904      903        -1     
  Lines       43728    43723        -5     
===========================================
- Hits        38043    28189     -9854     
- Misses       5685    15534     +9849
```


| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7680?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/providers/amazon/aws/hooks/s3.py](https://codecov.io/gh/apache/airflow/pull/7680/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYW1hem9uL2F3cy9ob29rcy9zMy5weQ==) | `96.46% <100%> (+0.12%)` | :arrow_up: |
| [...low/contrib/operators/wasb\_delete\_blob\_operator.py](https://codecov.io/gh/apache/airflow/pull/7680/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy93YXNiX2RlbGV0ZV9ibG9iX29wZXJhdG9yLnB5) | `0% <0%> (-100%)` | :arrow_down: |
| [airflow/contrib/hooks/vertica\_hook.py](https://codecov.io/gh/apache/airflow/pull/7680/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL3ZlcnRpY2FfaG9vay5weQ==) | `0% <0%> (-100%)` | :arrow_down: |
| [airflow/contrib/sensors/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/7680/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvX19pbml0X18ucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [airflow/hooks/mssql\_hook.py](https://codecov.io/gh/apache/airflow/pull/7680/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9tc3NxbF9ob29rLnB5) | `0% <0%> (-100%)` | :arrow_down: |
| [...viders/docker/example\_dags/example\_docker\_swarm.py](https://codecov.io/gh/apache/airflow/pull/7680/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZG9ja2VyL2V4YW1wbGVfZGFncy9leGFtcGxlX2RvY2tlcl9zd2FybS5weQ==) | `0% <0%> (-100%)` | :arrow_down: |
| [airflow/hooks/webhdfs\_hook.py](https://codecov.io/gh/apache/airflow/pull/7680/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy93ZWJoZGZzX2hvb2sucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [airflow/contrib/sensors/emr\_base\_sensor.py](https://codecov.io/gh/apache/airflow/pull/7680/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvZW1yX2Jhc2Vfc2Vuc29yLnB5) | `0% <0%> (-100%)` | :arrow_down: |
| [...irflow/contrib/operators/slack\_webhook\_operator.py](https://codecov.io/gh/apache/airflow/pull/7680/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9zbGFja193ZWJob29rX29wZXJhdG9yLnB5) | `0% <0%> (-100%)` | :arrow_down: |
| [...providers/google/cloud/example\_dags/example\_dlp.py](https://codecov.io/gh/apache/airflow/pull/7680/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL2V4YW1wbGVfZGFncy9leGFtcGxlX2RscC5weQ==) | `0% <0%> (-100%)` | :arrow_down: |
| ... and [487 more](https://codecov.io/gh/apache/airflow/pull/7680/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7680?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7680?src=pr&el=footer). Last update [e6af02f...89b6207](https://codecov.io/gh/apache/airflow/pull/7680?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

Awesome work, congrats on your first merged pull request!

Thanks @OmairK !

Your welcome @potiuk  :smile: . Thanks a lot for guiding me through my first PR. 
Hello, @OmairK 

I'm afraid this isn't working like it should. When you pass a string filename, how can filename.name be derived? https://github.com/apache/airflow/pull/7680/files#diff-411e5140735ea97206a32c7b390e2800R459

Or I'm misunderstanding how this should work.

thx for update. 

Resolving patch:
diff --git a/airflow/providers/amazon/aws/hooks/s3.py b/airflow/providers/amazon/aws/hooks/s3.py
--- a/airflow/providers/amazon/aws/hooks/s3.py
+++ b/airflow/providers/amazon/aws/hooks/s3.py
@@ -465,8 +465,9 @@ def load_file(self,
         if encrypt:
             extra_args['ServerSideEncryption'] = "AES256"
         if gzip:
-            filename_gz = filename.name + '.gz'
-            with open(filename.name, 'rb') as f_in:
+            filename_gz = ''
+            with open(filename, 'rb') as f_in:
+                filename_gz = f_in.name + '.gz'
                 with gz.open(filename_gz, 'wb') as f_out:
                     shutil.copyfileobj(f_in, f_out)
                     filename = filename_gz


Test patch:
diff --git a/tests/providers/amazon/aws/hooks/test_s3.py b/tests/providers/amazon/aws/hooks/test_s3.py
--- a/tests/providers/amazon/aws/hooks/test_s3.py
+++ b/tests/providers/amazon/aws/hooks/test_s3.py
@@ -18,6 +18,7 @@
 #
 
 import gzip as gz
+import os
 import tempfile
 from unittest.mock import Mock
 
@@ -277,25 +278,27 @@ def test_load_fileobj_acl(self, s3_bucket):
 
     def test_load_file_gzip(self, s3_bucket):
         hook = S3Hook()
-        with tempfile.NamedTemporaryFile() as temp_file:
+        with tempfile.NamedTemporaryFile(delete=False) as temp_file:
             temp_file.write(b"Content")
             temp_file.seek(0)
-            hook.load_file(temp_file, "my_key", s3_bucket, gzip=True)
+            hook.load_file(temp_file.name, "my_key", s3_bucket, gzip=True)
             resource = boto3.resource('s3').Object(s3_bucket, 'my_key')  # pylint: disable=no-member
             assert gz.decompress(resource.get()['Body'].read()) == b'Content'
+            os.unlink(temp_file.name)
 
     def test_load_file_acl(self, s3_bucket):
         hook = S3Hook()
-        with tempfile.NamedTemporaryFile() as temp_file:
+        with tempfile.NamedTemporaryFile(delete=False) as temp_file:
             temp_file.write(b"Content")
             temp_file.seek(0)
-            hook.load_file(temp_file, "my_key", s3_bucket, gzip=True,
+            hook.load_file(temp_file.name, "my_key", s3_bucket, gzip=True,
                            acl_policy='public-read')
             response = boto3.client('s3').get_object_acl(Bucket=s3_bucket,
                                                          Key="my_key",
                                                          RequestPayer='requester')  # pylint: disable=no-member # noqa: E501 # pylint: disable=C0301
             assert ((response['Grants'][1]['Permission'] == 'READ') and
                     (response['Grants'][0]['Permission'] == 'FULL_CONTROL'))
+            os.unlink(temp_file.name)
 
     def test_copy_object_acl(self, s3_bucket):
         hook = S3Hook()