Repository: iterative/dvc

Issue:
s3: Support AWS credential settings from environment variables
dvc should also load credential information from system environment variables as awscli does.

Reference: https://docs.aws.amazon.com/cli/latest/userguide/cli-environment.html

For example: we should be able to run `dvc pull` despite absence of AWS profiles in _~/.aws_ if we had set the following environment variables:

```
$ export AWS_ACCESS_KEY_ID=somekey
$ export AWS_SECRET_ACCESS_KEY=somekey
$ export AWS_DEFAULT_REGION=someregion
```

The following command works:
``` 
aws s3 ls s3://bucketname
```

But this one does not:
```
dvc pull
```

A usecase for this is doing `dvc pull` in docker image building process, which would be much easier if if we can override configuration via environment variables. 


Discussion:
Hi @jeepkd !

Good point! I was sure that we support this already since this is how we pass creds in our travis tests, but turns out we write it to `.aws/` first. Will send a patch soon.

Thanks,
Ruslan
Hi @jeepkd !

FYI: 0.19.5 with a patch for this is out.

Thanks,
Ruslan
This was quick! Thank you so much, Ruslan!

Jeep
The issue still exists. Not sure if I can re-open this issue? 
PS. I will have a pull request for this one
Re-opened. Are there any specific env vars that are not supported?
I still get an on the 3 envs discussed above

```
$ export AWS_ACCESS_KEY_ID=somekey
$ export AWS_SECRET_ACCESS_KEY=somekey
$ export AWS_DEFAULT_REGION=someregion
$ dvc pull
Preparing to pull data from s3://[myrepo]
[##################            ] 60% Collecting informationError: Failed to pull data from the cloud: Unable to locate credentials
```

A fix is WIP 
https://github.com/jeepkd/dvc/tree/fix/aws-env
But I have to go outside for now. Will be back soon ;)

Resolving patch:
diff --git a/dvc/remote/s3.py b/dvc/remote/s3.py
--- a/dvc/remote/s3.py
+++ b/dvc/remote/s3.py
@@ -57,19 +57,13 @@ def __init__(self, project, config):
             creds_conf = configobj.ConfigObj(credentialpath)
             creds = creds_conf.get(self.profile, {})
         else:
-            creds = {}
+            creds = self._get_aws_config_from_env()
 
         self.creds = creds
         self.region = creds.get('region', self.region)
         self.aws_access_key_id = creds.get('aws_access_key_id', None)
         self.aws_secret_access_key = creds.get('aws_secret_access_key', None)
 
-        self.region = os.getenv('AWS_DEFAULT_REGION', self.region)
-        self.aws_access_key_id = os.getenv('AWS_ACCESS_KEY_ID',
-                                           self.aws_access_key_id)
-        self.aws_secret_access_key = os.getenv('AWS_SECRET_ACCESS_KEY',
-                                               self.aws_secret_access_key)
-
     @property
     def bucket(self):
         return urlparse(self.url).netloc
@@ -89,6 +83,14 @@ def s3(self):
                              region_name=self.region)
         return session.client('s3', endpoint_url=self.endpoint_url)
 
+    def _get_aws_config_from_env(self):
+        creds = {
+            'region': os.getenv('AWS_DEFAULT_REGION'),
+            'aws_access_key_id': os.getenv('AWS_ACCESS_KEY_ID'),
+            'aws_secret_access_key': os.getenv('AWS_SECRET_ACCESS_KEY'),
+        }
+        return {k: v for k, v in creds.items() if v is not None}
+
     def get_etag(self, bucket, key):
         try:
             obj = self.s3.head_object(Bucket=bucket, Key=key)


Test patch: