Repository: apache/airflow

Issue:
Masking passwords with empty connection passwords make some logs unreadable in 2.1.0
Discovered in this [Slack conversation](https://apache-airflow.slack.com/archives/CCQ7EGB1P/p1621752408213700).

When you have connections with empty passwords masking logs masks all the character breaks:
```
[2021-05-23 04:00:23,309] {{logging_mixin.py:104}} WARNING - ***-***-***-*** ***L***o***g***g***i***n***g*** ***e***r***r***o***r*** ***-***-***-***
[2021-05-23 04:00:23,309] {{logging_mixin.py:104}} WARNING - ***T***r***a***c***e***b***a***c***k*** ***(***m***o***s***t*** ***r***e***c***e***n***t*** ***c***a***l***l*** ***l***a***s***t***)***:***
[2021-05-23 04:00:23,309] {{logging_mixin.py:104}} WARNING - *** *** ***F***i***l***e*** ***"***/***u***s***r***/***l***o***c***a***l***/***l***i***b***/***p***y***t***h***o***n***3***.***8***/***l***o***g***g***i***n***g***/***_***_***i***n***i***t***_***_***.***p***y***"***,*** ***l***i***n***e*** ***1***0***8***1***,*** ***i***n*** ***e***m***i***t***
*** *** *** *** ***m***s***g*** ***=*** ***s***e***l***f***.***f***o***r***m***a***t***(***r***e***c***o***r***d***)***
```

Until this is fixed, an easy workaround is to disable masking via disabling sensitive connection masking in configuration:

```
[core]
hide_sensitive_var_conn_fields = False
```

or vial env variable:

```
AIRFLOW__CORE__HIDE_SENSITIVE_VAR_CONN_FIELDS="False"
```

This is only happening if the task accesses the connection that has empty password. However there are a number of cases where such an empty password might be "legitimate" - for example in `google` provider you might authenticate using env variable or workload identity and connection will contain an empty password then.


Discussion:
Possibly we should only mask password longer than X characters (for example 3 or 5) - masking shorter passwords does not improve security much but it can cause some really weird logs.

@ashb  WDYT?
I thought I had an `if conn.login` guard. I guess not.

We could have a minimum length, but I'm generally adverse to adding extra config options without a strong need as each one we add makes understanding airflow harder
I discovered that if fernet key is not set, you cannot view connections on the UI when load_example dags is True. Maybe it’s related to this issue?

Steps to reproduce:

1. create virtualenv & activate it
2. Run python setup.py develop & airflow standalone. 
2. Visit connections on the UI and you’ll get an error 
> I discovered that if fernet key is not set, you cannot view connections on the UI when load_example dags is True. Maybe it’s related to this issue?
> 
> Steps to reproduce:
> 
> 1. create virtualenv & activate it
> 2. Run python setup.py develop & airflow standalone.
> 3. Visit connections on the UI and you’ll get an error

I think it is not that - the user in Slack discussion mentioned that it was only in some "Task" logs, which suggests that this was when particular connection was used. I am not 100% sure about the path to reproduce it indeed however, but empty "password" looks like a very probable (and quite valid) case.

> I thought I had an `if conn.login` guard. I guess not.

I had a quick look at the PR and have not found any guard - that's why I raised it :). But think it's `if conn.password` case more than `conn.login`. I can imagine case where we have login and no password.

> We could have a minimum length, but I'm generally adverse to adding extra config options without a strong need as each one we add makes understanding airflow harder

I thought more about hard-coding minimum length of password actually. And I think '3' might be a good one especially that it's replaced by '***' :D and we do not want to make logs any longer. But on a more serious note - I can imagine cases where people use some simple connections with very short passwords to test them ( `test`, `admin`, `123` or even `a`, `p` are often used for that purpose). In this case masking logs replacing all `a`s with `***` might lead to a very confusing output. That's why I thought having `len<3` or `len<5` hard-coded might be a good idea. 

Especially that those "Very short" passwords actually ARE causing a potential security threat mentioned in the PR already. 

Imagine that someone, by mistake sets 'a' password on a connection. Suddenly all "a's" get replaced with `***` and it's going to be super-easy to figure out what the password is. That's an interesting vector of attack (very difficult to exploit, more by accident than deliberate action, but still possible). This is less of a problem for longer passwords - figuring out longer passwords replaced in logs might be less probable and less obvious. But maybe we should also not mask the password if it is a common "word" in a dictionary? That would decrease the probability of it appearing in the logs and being discovered by accident.

Those are quite low-probability scenarios, so I am not very strong on having them fixed, but if it is an easy thing to do (seems it is) I think we should do it - ideally both: len < 3 + dictionary check.

@potiuk The guarding I am thinking of is in Connection, not in the masker (The top if here is in the `__init__` function):

https://github.com/apache/airflow/blob/9c98a60cdd29f0b005bf3abdbfc42aba419fded8/airflow/models/connection.py#L145-L151

The attack scenario you mentioned was also bought up by @uranusjr here https://github.com/apache/airflow/pull/15599#issuecomment-830104043 and my reply

> Possibly, but I don't think we need to defend against that level of attack -- it would only show up in the UI like that if it matches a connection the task has accessed.
>
> So yes, there's a theoretical attack surface here, but with the planned work of per-Connection ACLs etc, I think that is mitigated. Plus it only lets you validate the password if you already know it, but if you can change the DAG code, there are easier ways of exfil-ing the credentials.

How about this:

Currently we somewhat "blindly" replace in the rendered output, but I think it is also possible that we could operate on the format strings. Consider the following case

```
conn = Connection.from_uri("foo://a:b@host")
logging.info("Password: %s", conn.password)
logging.info("Password: " + conn.password)
```

The first log line we could, I think _always_ replace the password since it's a whole string, but replacing in a longer message only could be done only when the password length is >=4 for example.

(I think this is achievable on a technical level).

That said I'm not sure _how_ they ended up with every character being masked.
Oh, I think I know -- I think they had something like this in their extra field:

```
{"password": ""}
```

That would do it.

Resolving patch:
diff --git a/airflow/utils/log/secrets_masker.py b/airflow/utils/log/secrets_masker.py
--- a/airflow/utils/log/secrets_masker.py
+++ b/airflow/utils/log/secrets_masker.py
@@ -201,6 +201,8 @@ def add_mask(self, secret: Union[str, dict, Iterable], name: str = None):
             for k, v in secret.items():
                 self.add_mask(v, k)
         elif isinstance(secret, str):
+            if not secret:
+                return
             pattern = re.escape(secret)
             if pattern not in self.patterns and (not name or should_hide_value_for_key(name)):
                 self.patterns.add(pattern)


Test patch:
diff --git a/tests/utils/log/test_secrets_masker.py b/tests/utils/log/test_secrets_masker.py
--- a/tests/utils/log/test_secrets_masker.py
+++ b/tests/utils/log/test_secrets_masker.py
@@ -156,6 +156,8 @@ def test_exc_tb(self, logger, caplog):
             # When the "sensitive value" is a dict, don't mask anything
             # (Or should this be mask _everything_ under it ?
             ("api_key", {"other": "innoent"}, set()),
+            (None, {"password": ""}, set()),
+            (None, "", set()),
         ],
     )
     def test_mask_secret(self, name, value, expected_mask):