Repository: ejulio/spider-feeder

Issue:
Fallback to scrapy AWS settings if not provided
Currently, when loading files from _S3_, we need to set `SPIDERFEEDER_AWS_ACCESS_KEY_ID`.
However, it should be common to have the same user for data delivery and loading the files from _S3_.
So, to avoid configuring `spider-feeder` and `scrapy`, we can fallback in `spider-feeder` to `scrapy` settings if the specific ones for `spider-feeder` are not set.

[scrapy aws settings](https://docs.scrapy.org/en/latest/topics/feed-exports.html#s3)

Basically, read from `SPIDERFEEDER_AWS_ACCESS_KEY_ID` or the URI.
If it is empty, get from `AWS_ACCESS_KEY_ID`

Discussion:

Patch:
diff --git a/CHANGES.md b/CHANGES.md
index 56a5d28..31b5399 100644
--- a/CHANGES.md
+++ b/CHANGES.md
@@ -16,6 +16,7 @@ Whenever possible, link the given PR with the feature/fix.
 ### Added
 
 * New setting `SPIDERFEEDER_INPUT_FORMAT` to set the file format or fall back to the file extension. [PR#18](https://github.com/ejulio/spider-feeder/pull/18)
+* Fall back to `scrapy` AWS settings if not provided. [PR#20](https://github.com/ejulio/spider-feeder/pull/20)
 * Fixed AWS settings set in Dash (Scrapy Cloud) UI. [PR#21](https://github.com/ejulio/spider-feeder/pull/21)
 
 ## 0.2.0 (2019-08-27)
diff --git a/README.md b/README.md
index e57a373..747b426 100644
--- a/README.md
+++ b/README.md
@@ -93,6 +93,8 @@ There are two extensions to load input data to your spiders.
 * Supported schemes are: `''` or `file` for local files and `s3` for AWS S3 (requires `botocore`)
 * When using `s3`, the URI must be formatted as `s3://key_id:secret_key@bucket/blob.txt`
 * If `key_id` and `secret_key` are not provided in the URI, they can be provided by the following settings: `SPIDERFEEDER_AWS_ACCESS_KEY_ID` and `SPIDERFEEDER_AWS_SECRET_ACCESS_KEY`.
+    * If they are not provided by these settings, they will fall back to `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`.
+    * If not set, they can be set as environment variables from `botocore`, but a warning will be logged by `spider-feeder`.
 * When using `collections`, it'll load URLs from [Scrapinghub Collections](https://doc.scrapinghub.com/api/collections.html)
 
 `SPIDERFEEDER_INPUT_FILE_ENCODING` sets the file encoding. DEFAULT = `'utf-8'`.
diff --git a/spider_feeder/store/file_handler/s3.py b/spider_feeder/store/file_handler/s3.py
index edd0389..dc46e4b 100644
--- a/spider_feeder/store/file_handler/s3.py
+++ b/spider_feeder/store/file_handler/s3.py
@@ -3,23 +3,52 @@ This module handles `open()` for files stored in AWS S3.
 '''
 from io import StringIO
 from urllib.parse import urlparse
+import logging
 
 from botocore.session import get_session
 
 
+logger = logging.getLogger(__name__)
+
+
 def open(blob_uri, encoding, settings):
     parsed = urlparse(blob_uri)
 
+    (aws_access_key_id, aws_secret_access_key) = _get_aws_keys(parsed, settings)
     session = get_session()
     client = session.create_client(
         's3',
-        aws_access_key_id=parsed.username or settings['SPIDERFEEDER_AWS_ACCESS_KEY_ID'],
-        aws_secret_access_key=parsed.password or settings['SPIDERFEEDER_AWS_SECRET_ACCESS_KEY'],
+        aws_access_key_id=aws_access_key_id,
+        aws_secret_access_key=aws_secret_access_key,
     )
 
     bucket_name = parsed.hostname
     key_name = parsed.path[1:]
-
-    response = client.get_object(Bucket=bucket_name, Key=key_name, ResponseContentEncoding=encoding)
+    response = client.get_object(
+        Bucket=bucket_name,
+        Key=key_name,
+        ResponseContentEncoding=encoding
+    )
     content = response['Body'].read()
     return StringIO(content.decode(encoding))
+
+
+def _get_aws_keys(parsed_uri, settings):
+    aws_access_key_id = parsed_uri.username
+    aws_secret_access_key = parsed_uri.password
+
+    if not aws_access_key_id and not aws_secret_access_key:
+        aws_access_key_id = settings.get('SPIDERFEEDER_AWS_ACCESS_KEY_ID')
+        aws_secret_access_key = settings.get('SPIDERFEEDER_AWS_SECRET_ACCESS_KEY')
+
+    if not aws_access_key_id and not aws_secret_access_key:
+        aws_access_key_id = settings.get('AWS_ACCESS_KEY_ID')
+        aws_secret_access_key = settings.get('AWS_SECRET_ACCESS_KEY')
+
+    if not aws_access_key_id and not aws_secret_access_key:
+        logger.warning(
+            'No AWS keys were set in the input URI or project settings. '
+            'If that was intentional, make sure to have them set as environment variables.'
+        )
+
+    return (aws_access_key_id, aws_secret_access_key)


Test patch:
diff --git a/tests/store/file_handler/test_s3.py b/tests/store/file_handler/test_s3.py
index 8abbebd..5895be6 100644
--- a/tests/store/file_handler/test_s3.py
+++ b/tests/store/file_handler/test_s3.py
@@ -134,3 +134,29 @@ def test_open_s3_blob_using_project_credentials(botocore_client, mocker):
             aws_access_key_id='key_id',
             aws_secret_access_key='secret',
         )
+
+
+def test_open_s3_blob_using_scrapy_credentials(botocore_client, mocker):
+    (stubber, session_mock) = botocore_client(mocker)
+    with stubber:
+        file_content = 'http://url1.com\nhttps://url1.com'
+        response = get_object_response(file_content)
+        expected_params = {
+            'Bucket': 'bucket', 'Key': 'blob.txt', 'ResponseContentEncoding': 'utf-8'
+        }
+        stubber.add_response('get_object', response, expected_params)
+
+        s3.open(
+            's3://bucket/blob.txt',
+            encoding='utf-8',
+            settings=Settings({
+                'AWS_ACCESS_KEY_ID': 'some_key_id',
+                'AWS_SECRET_ACCESS_KEY': 'some_secret'
+            })
+        )
+
+        session_mock.create_client.assert_called_once_with(
+            's3',
+            aws_access_key_id='some_key_id',
+            aws_secret_access_key='some_secret',
+        )