Repository: Miserlou/Zappa

Issue:
Environment Variable keys are byte array in Python 3.6
## Context
When running in python 3 (not an issue in python 2) the environment variables get written to zappa_settings.py as byte arrays rather than strings. Meaning you would have to get your environment variable like this: environ[b'MY_VAR'] rather than like this environ['MY_VAR'].

This problem was unmasked by the recent changes in #962 but has likely existed before that. In #962 anything that was an environment_variable in zappa settings (rather than an aws_environment_variable) is no longer written to the aws environment and relies on zappa saving the environment variable to zappa_settings.py and loading that at runtime. 


## Expected Behavior
When zappa_settings has: 'environment_variables': { 'MY_VAR': 'someVal' }
My code can retrieve the environment variable like so: environ['MY_VAR']

## Actual Behavior
When zappa_settings has: 'environment_variables': { 'MY_VAR': 'someVal' }
My code can must retrieve the  the environment variable like so: environ[b'MY_VAR']

## Possible Fix
When creating the zappa_settings.py file, decode the byte array back to a string (PR coming soon)

## Steps to Reproduce
zappa_settings.json:
```
{
    "dev": {
        "apigateway_enabled": true,
        "apigateway_description": "hello world", 
        "app_function": "main.app",
        "aws_region": "us-west-2",
        "lambda_description": "hello world", 
        "project_name": "hello-world",
        "s3_bucket": "<YOUR BUCKET>",
        "timeout_seconds": 30,
        "environment_variables": {
            "TEST_VAR": "TestValue"
         }
    }
}

```

main.py:
```
import json
from flask import Flask, request, Response
from os import environ

app = Flask(__name__)

@app.route('/test')
def stuff():
    for y in environ:
        print (y,':',environ[y])
    return '[{ "hello": "world"}]', 200

if __name__ == '__main__':
        app.run(host='0.0.0.0', port=5001)
```

zappa deploy dev
Hit the /test endpoint
Examine the output in cloud watch, the key for TEST_VAR is b'TEST_VAR' rather than 'TEST_VAR'

## Your Environment
Zappa master branch @ b7c16d3ac62aa496dfce8ba533dc64cd20376269
Ubuntu 16.04
Python 3.6
argcomplete==1.8.2
base58==0.2.4
boto3==1.4.4
botocore==1.5.78
certifi==2017.4.17
chardet==3.0.4
click==6.7
docutils==0.13.1
durationpy==0.4
Flask==0.12.2
future==0.16.0
futures==3.1.1
hjson==2.0.7
idna==2.5
itsdangerous==0.24
Jinja2==2.9.6
jmespath==0.9.3
kappa==0.6.0
lambda-packages==0.16.1
MarkupSafe==1.0
placebo==0.8.1
python-dateutil==2.6.0
python-slugify==1.2.4
PyYAML==3.12
requests==2.18.1
s3transfer==0.1.10
six==1.10.0
toml==0.9.2
tqdm==4.14.0
troposphere==1.9.4
Unidecode==0.4.21
urllib3==1.21.1
Werkzeug==0.12.2
wsgi-request-logger==0.4.6


Environment Variable keys are byte array in Python 3.6
## Context
When running in python 3 (not an issue in python 2) the environment variables get written to zappa_settings.py as byte arrays rather than strings. Meaning you would have to get your environment variable like this: environ[b'MY_VAR'] rather than like this environ['MY_VAR'].

This problem was unmasked by the recent changes in #962 but has likely existed before that. In #962 anything that was an environment_variable in zappa settings (rather than an aws_environment_variable) is no longer written to the aws environment and relies on zappa saving the environment variable to zappa_settings.py and loading that at runtime. 


## Expected Behavior
When zappa_settings has: 'environment_variables': { 'MY_VAR': 'someVal' }
My code can retrieve the environment variable like so: environ['MY_VAR']

## Actual Behavior
When zappa_settings has: 'environment_variables': { 'MY_VAR': 'someVal' }
My code can must retrieve the  the environment variable like so: environ[b'MY_VAR']

## Possible Fix
When creating the zappa_settings.py file, decode the byte array back to a string (PR coming soon)

## Steps to Reproduce
zappa_settings.json:
```
{
    "dev": {
        "apigateway_enabled": true,
        "apigateway_description": "hello world", 
        "app_function": "main.app",
        "aws_region": "us-west-2",
        "lambda_description": "hello world", 
        "project_name": "hello-world",
        "s3_bucket": "<YOUR BUCKET>",
        "timeout_seconds": 30,
        "environment_variables": {
            "TEST_VAR": "TestValue"
         }
    }
}

```

main.py:
```
import json
from flask import Flask, request, Response
from os import environ

app = Flask(__name__)

@app.route('/test')
def stuff():
    for y in environ:
        print (y,':',environ[y])
    return '[{ "hello": "world"}]', 200

if __name__ == '__main__':
        app.run(host='0.0.0.0', port=5001)
```

zappa deploy dev
Hit the /test endpoint
Examine the output in cloud watch, the key for TEST_VAR is b'TEST_VAR' rather than 'TEST_VAR'

## Your Environment
Zappa master branch @ b7c16d3ac62aa496dfce8ba533dc64cd20376269
Ubuntu 16.04
Python 3.6
argcomplete==1.8.2
base58==0.2.4
boto3==1.4.4
botocore==1.5.78
certifi==2017.4.17
chardet==3.0.4
click==6.7
docutils==0.13.1
durationpy==0.4
Flask==0.12.2
future==0.16.0
futures==3.1.1
hjson==2.0.7
idna==2.5
itsdangerous==0.24
Jinja2==2.9.6
jmespath==0.9.3
kappa==0.6.0
lambda-packages==0.16.1
MarkupSafe==1.0
placebo==0.8.1
python-dateutil==2.6.0
python-slugify==1.2.4
PyYAML==3.12
requests==2.18.1
s3transfer==0.1.10
six==1.10.0
toml==0.9.2
tqdm==4.14.0
troposphere==1.9.4
Unidecode==0.4.21
urllib3==1.21.1
Werkzeug==0.12.2
wsgi-request-logger==0.4.6




Discussion:

Resolving patch:
diff --git a/zappa/cli.py b/zappa/cli.py
--- a/zappa/cli.py
+++ b/zappa/cli.py
@@ -2137,12 +2137,12 @@ def create_package(self, output=None):
                 env_dict['AWS_REGION'] = self.aws_region
             env_dict.update(dict(self.environment_variables))
 
-            # Environment variable keys can't be Unicode
+            # Environment variable keys must be ascii
             # https://github.com/Miserlou/Zappa/issues/604
             try:
-                env_dict = dict((k.encode('ascii'), v) for (k, v) in env_dict.items())
-            except Exception: # pragma: no cover
-                    raise ValueError("Environment variable keys must not be unicode.")
+                env_dict = dict((k.encode('ascii').decode('ascii'), v) for (k, v) in env_dict.items())
+            except Exception:
+                raise ValueError("Environment variable keys must be ascii.")
 
             settings_s = settings_s + "ENVIRONMENT_VARIABLES={0}\n".format(
                     env_dict


Test patch:
diff --git a/tests/test_non_ascii_environment_var_key.json b/tests/test_non_ascii_environment_var_key.json
new file mode 100644
--- /dev/null
+++ b/tests/test_non_ascii_environment_var_key.json
@@ -0,0 +1,18 @@
+{
+    "ttt888": {
+       "touch": false,
+       "s3_bucket": "lmbda",
+       "app_function": "tests.test_app.hello_world",
+       "callbacks": {
+           "settings": "test_settings.callback",
+           "post": "test_settings.callback",
+           "zip": "test_settings.callback"
+       },
+       "delete_local_zip": true,
+       "debug": true,
+       "parameter_depth": 2,
+        "environment_variables": {
+            "REMOTE_HOST☣": "example.net"
+        },
+    }
+}
diff --git a/tests/tests.py b/tests/tests.py
--- a/tests/tests.py
+++ b/tests/tests.py
@@ -1653,7 +1653,7 @@ def test_settings_py_generation(self):
 
             # validate environment variables
             self.assertIn('ENVIRONMENT_VARIABLES', settings)
-            self.assertEqual(settings['ENVIRONMENT_VARIABLES'][b'TEST_ENV_VAR'], "test_value")
+            self.assertEqual(settings['ENVIRONMENT_VARIABLES']['TEST_ENV_VAR'], "test_value")
 
             # validate Context header mappings
             self.assertIn('CONTEXT_HEADER_MAPPINGS', settings)
@@ -1662,6 +1662,14 @@ def test_settings_py_generation(self):
 
         zappa_cli.remove_local_zip()
 
+    def test_only_ascii_env_var_allowed(self):
+        zappa_cli = ZappaCLI()
+        zappa_cli.api_stage = 'ttt888'
+        zappa_cli.load_settings('tests/test_non_ascii_environment_var_key.json')
+        with self.assertRaises(ValueError) as context:
+            zappa_cli.create_package()
+        self.assertEqual('Environment variable keys must be ascii.', str(context.exception))
+
 
 
 if __name__ == '__main__':