Repository: serverless/serverless (JavaScript)
Date: 2017-05-29 11:54:36+00:00
Task type: Feature

Issue:
Support for --profile argument when deploying to AWS
##### Feature Request:

it should be possible to run `serverless deploy --profile production` to use the production AWS profile. At the moment this has to be set through environment variables
##### Benefits:
- being able to easily switch profiles to deploy with



Issue discussion:
Just to clarify, could the argument after the --profile option be any named profile that is configured on the AWS cli?

@kevinaud exactly. 

okay, I'd like to work on this

Awesome, thanks. I set the labels for it. Its probably best to add it somewhere here: https://github.com/serverless/serverless/blob/990d7c5a089c2047cdd342fca88f6c006c53c4c3/lib/plugins/aws/index.js#L83

Let me know if you have any questions.

Should setting the profile with environment variables still be supported once this is in place?

I would hope not, generally we should be using the `~/.aws/credentials` file. Also leaves less room for WTF moments 😄 

EDIT: Apparently the AWS-SDK supports both, so I guess Serverless should as well 😕 

So the three possible scenarios as I understand it are:
1. `--profile` option **isn't** used and the `AWS_PROFILE` environment variable **isn't** set, in which case deploy with the 'default' profile
2. `--profile` option **isn't** used and the `AWS_PROFILE` environment variable **is** set, in which case use the value of that environment variable as name of the profile to deploy with 
3. `--profile` option **is** used and, in which case use the value of the `--profile` argument as name of the profile to deploy with regardless of whether or not the `AWS_PROFILE` environment variable is set

EDIT: basing this off of the getCredentials() function that @flomotlik pointed me towards above

I believe I saw a precedence order [here](https://github.com/serverless/serverless/issues/1801#issuecomment-238600591): 

CLI Option > ENV > serverless.yml

Given that, it seems (3) is the likely scenario 👍 

Is setting the profile in serverless.yml currently supported? I don't see it in the docs https://github.com/serverless/serverless/blob/master/docs/understanding-serverless/serverless-yml.md

EDIT: oh wiat I didn't see that you had a link on 'here', I'm checking it out now

I don't think it is supported **yet**, possibly in the next beta. Also, I don't think the docs reflect the state of v1 too well, GitHub issues are a better bet at the moment.

@hassankhan @kevinaud setting the profile will most likely not be supported in the serverless.yml directly.

as @hassankhan already mentioned CLI > ENV

But as CLI always has precedence basically what we can do is if there is a cli option we're setting the profile directly, if no CLI option is given we don't set anything (not even default) and let the AWS SDK deal with it.

@kevinaud just wanted to ping if you had time to look into it or need any help?

@flomotlik Sorry I've been in the process of moving back to college (which includes travelling across most of the US for me) but I'm about to submit a pull request for you to check out.

To anyone else that's not sure how to set the environment variables for the AWS profile during deployment:

You configure the additional profiles in AWS credentials with (using serverless_admin as an example):
`$ aws configure --profile serverless_admin`

And you deploy with that profile:
`$ AWS_PROFILE=serverless_admin serverless deploy`

And you invoke the same way:
`$ AWS_PROFILE=serverless_admin serverless invoke --function <function name>`

Or of course you can also just export the AWS_PROFILE variable for that terminal session or put the environment variable in your bash (or whatever) profile for terminal sessions.

I just tested Serverless 1.0.0-beta2 and was very confused how to choose the AWS profile. Previously Serveless 0.x has asked for it (with a menu) and stored it in admin.env, which we have stored in Git, so that the project is always automatically deployed to the right profile. I hope this functionality can be restored in 1.0.

(Slightly annoyed, because I deployed my test to the wrong profile and now have to clean it up.)

@kevinaud submitted the PR now btw: #1905 great job and will be reviewed and hopefully merged soon.

@kennu so you want to hardcode the profile name in the serverless.yml? Our current approach would be to follow AWS defaults and use `--profile` if its available, AWS_PROFILE if that is available or the default profile in the .aws/credentials file.

Are you the only person deploying those services so you want to hardcode them, or is the hardcoded profile used across a few people in the team?

In my case (workflow with Serverless 0.5 so far)
- I deploy to 17 different AWS accounts, so each project needs some way to remember the profile or things get difficult
- AWS profile names are standardized between team members so that admin.env can be shared in Git

Until now, we could trust that "sls xxx deploy" does the right thing automatically, so it's why I'm missing that feature. Having to prefix commands with AWS_PROFILE=xxx manually is fairly inconvenient and error-prone.

Ok thats good to know. I can see something like

``` yaml
provider:
  profile: production
```

for example. If you provide it via `--profile` you can override that setting, but by default it should work and deploy properly.

@flomotlik I think that would work well. In fact I tried put `profile: xxx` in serverless.yml, because GitHub search found some test file indicating it might work. And then was disappointed. :-)

It is also a larger problem, because it would be also desirable for AWS CLI to use a specific profile when working under a project. But that's another story and with Serverless, there's generally not that much need to use AWS CLI every day.

This also reminds me of another discussion about overriding AWS profiles for each deployment target. We have many cases where production stage is in a separate AWS account. It would be nice to have `sls deploy -s prod` deploy to the right profile automatically.

I'm not sure @flomotlik's example would work - wouldn't that set your profile for your entire provider (i.e. AWS)? I think profile needs to be set per stage for the reasons @kennu mentioned (i.e. deploying different stages to different AWS accounts - it's just best practice).

While the `--profile` flag is a step in the right direction, I would really like to set a profile to use for deployment in configuration (probably `serverless.env.yml`?) I'm thinking something like this:

``` yaml
stages:
  dev:
    vars:
      profile: my-dev-profile
    regions:
      ap-southeast-2:
        vars:
  prod:
    vars:
      profile: my-prod-profile
    regions:
      ap-southeast-2:
        vars:
```

Where `my-dev-profile` and `my-prod-profile` just happen to be different AWS accounts.

Does that make sense, or am I missing something? I saw [this test](https://github.com/serverless/serverless/blob/master/lib/plugins/aws/tests/index.js#L87) and thought it would "just work" but it doesn't seem to be the case - any `sls` commands just bail if I don't have my `AWS_PROFILE` env var set...

After a bit more investigation, I think the issue I described above is not with Serverless - it seems to be getting the profile from config correctly. It seems like the AWS SDK is not aware of the setting... Sorry for the issue hijack :neutral_face:

@rowanu you can do per stage setting with nesting of environment variables as described in #1834 

@flomotlik the problem is the variables weren't honoured by the SDK. Please see my (recently updated) bug #1940 and PR #1945.

@rowanu we're introducing the new variable system in the next few days which will make things like this easier going forward

Being a bit needy here, but can we have a beta.3 drop with the profiles per stage ready? 

@mwawrusch this issue is not for profiles per stage, you want #1940 and #2015.

Btw, Serverless 1.0-RC2 still _silently ignores_ the --profile option. No error message, just deploys to the wrong profile (default). I got bitten by this just now once again. I think it should at least show an error message and abort when you specify an unsupported option.

I'm seeing an issue where serverless doesn't honor a role based setup of credentials:

```
[default]
aws_access_key_id=AAAAAAAAAAAA
aws_secret_access_key=XXXXXXXXXXXXXXXXXXXXXXX

[sandbox-user]
role_arn = arn:aws:iam::AAAAAA:role/FullAdmin
source_profile = default
mfa_serial = arn:aws:iam::ZZZZZZZ:mfa/kold
```

I've added the sandbox-user profile to serverless.yaml and get `Error: ServerlessError: ServerlessError: AWS provider credentials not found. You can find more info on how to set up provider credentials in our docs here: https://git.io/viZAC` when I run it.  I've also tried the env vars but nothing works.

@kennu @kevinaud guys, what you've both described are valid issues, but neither belongs on this issue. Please open a new issue or raise it in [the forum](http://forum.serverless.com/) if you want it addressed.

I agree that **if** support is added to specify the profile in the `serverless.yml` file like it was in `admin.env` pre 1.0, it must support the configuration of different profiles per stage, and a default profile. That supports the (common) usecase where you deploy developers' builds to one account, integration builds to another, and production to yet another.

Is this not supported with the variables work and loading a variable from eg. `${opt:profile}`?

Yup as @andymac4182 said this should be already done and was expanded on recently in https://github.com/serverless/serverless/pull/2229

Documented here: https://serverless.com/framework/docs/providers/aws/setup/

While this doesn't have `--profile` directly built into as an option I think its a cleaner implementation because its more specific on the users end. We might add this more directly through options in the future though, should be easy with that new implementation.

As I think this can be closed now I'll close.

Bad doc link?  I think this is the new one: https://serverless.com/framework/docs/providers/aws/guide/credentials/ 

This flag would be great for numerous reasons.

Can we revisit this? 

I can implement

@nikgraf @DavidWells that's what you need for the dashboard, right?

Similar to @kevinold, I have a role-based AWS setup and I'm not sure how I can deploy to that account. 
Setting the env variable `AWS_PROFILE=my-role-based-profile` doesn't help as I get `ServerlessError: ServerlessError: AWS provider credentials not found.`

It used to work in the 0.5.x because every stage was tied to a specific profile.

@skilledDeveloper I ended up creating another user in my role-based account and used that user as my profile and all worked well.
From AWS documentation, the environment variable is `AWS_DEFAULT_PROFILE`

https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html#cli-environment

Is the `AWS_PROFILE` variable discussed here intentionally different?
Currently I use a setup where I have 24 AWS profiles configured. There is one AWS account, the rest are ROLE based profiles. The only way I can make serverless work is by creating STS sessions and populating ENV variables. This means the values will expire every 1 hour.

serverless should support role based aws cli profiles.
@nappa32 that might be better served with an `--assume-role` CLI parameter, maybe?
In the meantime...I wrote this to solve the issue for me: [aws-profile-switcher](https://www.npmjs.com/package/aws-profile-switcher)
It switches the profile in the actual credentials file so that I don't have to deal with environment variables. Worked really well for me when I was testing out aws permissions for various profiles.

Solution patch:
diff --git a/docs/providers/aws/guide/credentials.md b/docs/providers/aws/guide/credentials.md
index 2530c7a07c8..9df7ca11f06 100644
--- a/docs/providers/aws/guide/credentials.md
+++ b/docs/providers/aws/guide/credentials.md
@@ -128,6 +128,14 @@ Now you can switch per project (/ API) by executing once when you start your pro
 in the Terminal. Now everything is set to execute all the `serverless` CLI options like `sls deploy`.
 The AWS region setting is to prevent issues with specific services, so adapt if you need another default region.
 
+##### Using the `aws-profile` option
+
+You can always speficy the profile which should be used via the `aws-profile` option like this:
+
+```bash
+serverless deploy --aws-profile devProfile
+```
+
 #### Per Stage Profiles
 
 As an advanced use-case, you can deploy different stages to different accounts by using different profiles per stage. In order to use different profiles per stage, you must leverage [variables](https://serverless.com/framework/docs/providers/aws/guide/variables) and the provider profile setting.
diff --git a/lib/plugins/aws/provider/awsProvider.js b/lib/plugins/aws/provider/awsProvider.js
index a9fc5ac448c..3d0f48cef34 100644
--- a/lib/plugins/aws/provider/awsProvider.js
+++ b/lib/plugins/aws/provider/awsProvider.js
@@ -178,9 +178,16 @@ class AwsProvider {
     const result = {};
     const stageUpper = this.getStage() ? this.getStage().toUpperCase() : null;
 
+    let profile;
+    if (this.options['aws-profile']) {
+      profile = this.options['aws-profile'];
+    } else if (this.serverless.service.provider.profile) {
+      profile = this.serverless.service.provider.profile;
+    }
+
     // add specified credentials, overriding with more specific declarations
     impl.addCredentials(result, this.serverless.service.provider.credentials); // config creds
-    impl.addProfileCredentials(result, this.serverless.service.provider.profile);
+    impl.addProfileCredentials(result, profile);
     impl.addEnvironmentCredentials(result, 'AWS'); // creds for all stages
     impl.addEnvironmentProfile(result, 'AWS');
     impl.addEnvironmentCredentials(result, `AWS_${stageUpper}`); // stage specific creds


Test patch:
diff --git a/lib/plugins/aws/provider/awsProvider.test.js b/lib/plugins/aws/provider/awsProvider.test.js
index d183db68ce3..53d38858df4 100644
--- a/lib/plugins/aws/provider/awsProvider.test.js
+++ b/lib/plugins/aws/provider/awsProvider.test.js
@@ -251,6 +251,7 @@ describe('AwsProvider', () => {
     let originalProviderCredentials;
     let originalProviderProfile;
     let originalEnvironmentVariables;
+
     beforeEach(() => {
       originalProviderCredentials = serverless.service.provider.credentials;
       originalProviderProfile = serverless.service.provider.profile;
@@ -273,6 +274,7 @@ describe('AwsProvider', () => {
       );
       newAwsProvider = new AwsProviderProxyquired(serverless, newOptions);
     });
+
     afterEach(() => {
       replaceEnv(originalEnvironmentVariables);
       serverless.service.provider.profile = originalProviderProfile;
@@ -409,6 +411,13 @@ describe('AwsProvider', () => {
       const credentials = newAwsProvider.getCredentials();
       expect(credentials.credentials.profile).to.equal('notDefault');
     });
+
+    it('should get credentials when profile is provied via --aws-profile option', () => {
+      newAwsProvider.options['aws-profile'] = 'notDefault';
+
+      const credentials = newAwsProvider.getCredentials();
+      expect(credentials.credentials.profile).to.equal('notDefault');
+    });
   });
 
   describe('#getRegion()', () => {