Solving the frequent Fernet Key cryptography error for containerized Apache-Airflow 

![AirflowLogo.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/538885/a8895a6d-3950-c59f-420f-68f2b4e51a69.png)

### The Issue

For security purposes, sensitive connection and administrative information is encrypted with a _Fernet key_ before being stored in Airflow's backend database. This includes any passwords for your connection objects as well as service account keys for e.g. Google Cloud. 

However, if you have built Airflow webserver as a containerized service, then every time you modify and rebuild your container you run the risk of invalidating your Fernet key and losing access to your connections. 

Airflow finds the Fernet key you would like to use from the config file, which by default gets generated and added to `airflow/airflow.cfg` when you first run the `airflow initdb` command. There is some insecurity built into this approach, since the key gets hard-coded into the file. 

For this diagnosis and If you're using the [puckel/docker-airflow](https://github.com/puckel/docker-airflow) repository's Dockerfile or docker-compose.yaml as a base for building your Airflow service, then the point at which your Fernet key gets generated is here, in the `scripts/entrypoint.sh` file:

```bash
: "${AIRFLOW__CORE__FERNET_KEY:=${FERNET_KEY:=$(python -c "from cryptography.fernet \
import Fernet; FERNET_KEY = Fernet.generate_key().decode(); print(FERNET_KEY)")}}"
```

That's a pretty sexy, maybe brilliant, one-liner as far as bash-scripted one-liner's go. If you're not familiar with bash script, the breakdown is as follows: 

- the `:` at the start of the line allows you to define a variable in a script with a default value. Here, `entrypoint.sh` creates the AIRFLOW_\_CORE_\_FERNET_KEY variable for the script, if the variable does not already exist in the environment. So, you could override it by specifying this variable in your Dockerfile or docker-compose.yaml file with something like 
`ENV AIRFLOW__CORE__FERNET_KEY='some string you generated or made up'`.
- `=${FERNET_KEY:= ` This portion assigns the value of FERNET_KEY to AIRFLOW_\_CORE_\_FERNET_KEY if it already exists in the environment (maybe you decided to pass it from somewhere else); if it does not exist, then the `:=` part here tells bash to make a default value with the one-line call to Python's `cryptography` library.
- the `python -c ...` you can probably understand if you know at least some Python--the string gets passed as a -c (command) to python, and the print(FERNET_KEY) call prints out the random Fernet Key to STDOUT.
- Placing the whole Python section into `$( )` tells bash to evaluate the entire expression in a sub-process and to return the STDOUT output. So, the output of this portion is the Fernet key itself.

At first I wondered why Puckel defined two different variables for the key in `entrypoint.sh`, but I realized that it is necessary to have two places where the user can manually define it, depending on their use case. 

1. `AIRFLOW__CORE__FERNET_KEY` is the environment variable the `airflow initdb` command will look for when creating the back-end database, and so if the user wants to change it and uses docker-compose, she should set it in the docker-compose file. 
2. If she just wants to build the webserver by itself, she can set `FERNET_KEY` in the Dockerfile, because that is accessible to `entrypoint.sh`, which gets executed at the end of the Dockerfile. 

In either case, the final value of `$FERNET_KEY` gets assigned to the `airflow.cfg` file in line 122:

```python
# Secret key to save connection passwords in the db
fernet_key = $FERNET_KEY
```

### The Solution
So, now that we understand exactly what is going on, we can troubleshoot the `cryptography.fernet` error that might appear in your DAG task execution logs when a task fails, as it tries to access the back-end database for connection and other runtime data. This will likely happen every time you rebuild your airflow_webserver container, unless you rebuild the database and it's data dump each time (if you're like me, you prefer to keep it as a persistent volume so you have some permanency to your Airflow execution and scheduling data). 

The easiest thing to do is just re-enter your connections and other entries that use the Fernet key for cryptographic encoding in the Airflow UI, though if you have many connections, that will become very tedious.

The second easiest thing is to create a task that recreates connections and other database entries you need, scheduled to run `@once` so that you can just trigger it after rebuilding your webserver container.

The Python task would look something like this:

```python
from airflow.models import Connection, Variable, Session

import airflow import DAG
from airflow.operators.python_operator import PythonOperator
from airflow import models, settings

dag = DAG( .... )

def set_connection(**config):
    for k,v in config.items():
        conn = Connection()
        setattr(conn, k, v)
    session = settings.Session()
    session.add(conn)
    session.commit()
    session.close()

task = PythonOperator(
    dag = dag,
    task_id = 'set-connections',
    python_callable = set_connection,
    ...........
)
```

Of course, try to avoid hard-coding the config for your connections directly into your file. You can store it in a more secure place, such as a dedicated database with encryption, in a gcloud bucket, etc., and pull that connection configuration data into your script. Also, for added security, Airflow `connection` objects have a `rotate_fernet_key` attribute you can explore to change the encryption in the backend database regularly!


#### Sources
https://airflow.apache.org/docs/stable/_modules/airflow/models/connection.html#Connection.rotate_fernet_key