Repository: apache/airflow

Issue:
Error on logging empty line to Cloudwatch
**Apache Airflow version**: 2.0.1

**Environment**:

- **Cloud provider or hardware configuration**: AWS

**What happened**:
I have Airflow with Cloudwatch-based remote logging running. I also have `BashOperator` that does, for example, `rsync` with invalid parameters, for example `rsync -av test test`. The output of the `rsync` error is formatted and contains empty line. Once that empty line is logged to the Cloudwatch, i receive an error:
```
2021-04-06 19:29:22,318] /home/airflow/.local/lib/python3.6/site-packages/watchtower/__init__.py:154 WatchtowerWarning: Failed to deliver logs: Parameter validation failed:
Invalid length for parameter logEvents[5].message, value: 0, valid range: 1-inf
[2021-04-06 19:29:22,320] /home/airflow/.local/lib/python3.6/site-packages/watchtower/__init__.py:158 WatchtowerWarning: Failed to deliver logs: None
```

So basically empty lines can't be submitted to the Cloudwatch and as result the whole output of the process doesn't appear in logs.

**What you expected to happen**:

I expect to have an output of the bash command in logs. Empty lines can be skipped or replaced with something.

**How to reproduce it**:

For example: run `BashOperator` with `rsync` command that fails on Airflow with Cloudwatch-based remote logging. It could be any other command that produces empty line in the output.


Discussion:
Are you on the latest version of watchtower? Sounds like they should be ignored now: kislyuk/watchtower#43

Either way, I'm not sure Airflow should be responsible for skipping/replacing empty lines.
Airflow currently pins watchtower to `~=0.7.3`, so the fix (only available in 1.x, I believe) cannot be used in this context.
Ah, good catch. @codenamestif, would it be possible for you to test 1.x in your environment? I suspect there wasn't an issue, just no one has tried it.
@jedcunningham I run airflow from the docker image based on `apache/airflow:2.0.1`. I installed `watchtower==1.0.6` under `airflow` user and `pip freeze` confirms it. However I still see the same error. Dunno exactly if this is a correct way to update the lib for the docker image.
To clarify, you did it in your image custom, not in an already running container, right? If so, then yeah that is the right way to test it. Bummer. Might be worth an issue over in the watchtower repo.
@jedcunningham no, it wasn't a running container. I updated my image and started a new application. I also see in build logs:
```
#14 6.959 Installing collected packages: watchtower
#14 6.959   Attempting uninstall: watchtower
#14 6.961     Found existing installation: watchtower 0.7.3
#14 6.965     Uninstalling watchtower-0.7.3:
#14 6.974       Successfully uninstalled watchtower-0.7.3
#14 7.095 ERROR: apache-airflow-providers-amazon 1.1.0 requires watchtower~=0.7.3, but you'll have watchtower 1.0.6 which is incompatible.
#14 7.096 Successfully installed watchtower-1.0.6
```
So i assume i updated the lib properly and still see the error.
Hey y'all, i'm having the same issue now, but i found a workaround, you can use this awk command to delete the newlines in the Bash command:

`my_cool_command | awk NF`

In case you have the same problem as i did (In my case it happens with running dbt, which puts empty newlines for styling between it's messages)

Have a great day!
Another approach is to wrap the LogHandler into something like this:

```(python)
import watchtower


class NoEmptyMessageHandler(watchtower.CloudWatchLogHandler):
    def emit(self, message):
        if isinstance(message.msg, str) and not message.msg:
            pass
        else:
            super().emit(message)
```

I just saw that a new version of apache-airflow-providers-amazon was released (2.0.0) with an updated watchtower, i'll test it later :smile: 
> I just saw that a new version of apache-airflow-providers-amazon was released (2.0.0) with an updated watchtower, i'll test it later 😄

I just tested it, the error is still there
I had a lot of jobs freezing with cloudwatch, migrated to using an EFS filesystem to keep logs myself (using ECS Fargate to host Airflow)
i am also seeing this error even with apparently the latest versions:

```
apache-airflow==2.1.3
apache-airflow-providers-amazon==2.1.0
...
watchtower==1.0.6
```

```
[2021-08-31 18:33:50,305] /<home dir>/lib/python3.9/site-packages/watchtower/__init__.py:201 WatchtowerWarning: Failed to deliver logs: Parameter validation failed:
Invalid length for parameter logEvents[19].message, value: 0, valid min length: 1
Invalid length for parameter logEvents[21].message, value: 0, valid min length: 1
[2021-08-31 18:33:50,308] /<home dir>/lib/python3.9/site-packages/watchtower/__init__.py:205 WatchtowerWarning: Failed to deliver logs: None
```

it is difficult to debug as the log gets truncated. not sure where the blanks are coming from. the underlying `BashOperator` `TaskInstance` should not be generating any blank log lines.

is it easy to setup airflow to log to two locations simultaneously?
> I had a lot of jobs freezing with cloudwatch, migrated to using an EFS filesystem to keep logs myself (using ECS Fargate to host Airflow)

I have noticed some jobs randomly freezing with nothing in the logs. how were you able to pinpoint the issue to cloudwatch logging?
We are also seeing this issue, airflow version 2.1.4, using the kubernetesPodOperator. Is there a workaround?
i've gotten around it by creating a `log_config.py` with the following:

```
from copy import deepcopy
import logging

from airflow.config_templates.airflow_local_settings import DEFAULT_LOGGING_CONFIG

LOGGING_CONFIG = deepcopy(DEFAULT_LOGGING_CONFIG)
class EmptyLogFilter(logging.Filter):
    def filter(self, record: logging.LogRecord) -> bool:
        if not record.getMessage():
            return False
        else:
            return True


LOGGING_CONFIG['filters']['remove_blanks'] = {'()': 'log_config.EmptyLogFilter'}

CLOUDWATCH_HANDLER = 'airflow.providers.amazon.aws.log.cloudwatch_task_handler.CloudwatchTaskHandler'

TASK_HANDLER = LOGGING_CONFIG['handlers']['task']
if TASK_HANDLER['class'] == CLOUDWATCH_HANDLER:
    # This thing blows up on empty log messages so apply filter
    filters = TASK_HANDLER.get('filters', [])
    filters.append('remove_blanks')
    TASK_HANDLER['filters'] = filters
```

don't forget to enable this logger in `airflow.cfg` with `logging_config_class = log_config.LOGGING_CONFIG`
> i've gotten around it by creating a `log_config.py` with the following:
> 
> ```
> from copy import deepcopy
> import logging
> 
> from airflow.config_templates.airflow_local_settings import DEFAULT_LOGGING_CONFIG
> 
> LOGGING_CONFIG = deepcopy(DEFAULT_LOGGING_CONFIG)
> class EmptyLogFilter(logging.Filter):
>     def filter(self, record: logging.LogRecord) -> bool:
>         if not record.getMessage():
>             return False
>         else:
>             return True
> 
> 
> LOGGING_CONFIG['filters']['remove_blanks'] = {'()': 'log_config.EmptyLogFilter'}
> 
> CLOUDWATCH_HANDLER = 'airflow.providers.amazon.aws.log.cloudwatch_task_handler.CloudwatchTaskHandler'
> 
> TASK_HANDLER = LOGGING_CONFIG['handlers']['task']
> if TASK_HANDLER['class'] == CLOUDWATCH_HANDLER:
>     # This thing blows up on empty log messages so apply filter
>     filters = TASK_HANDLER.get('filters', [])
>     filters.append('remove_blanks')
>     TASK_HANDLER['filters'] = filters
> ```
> 
> don't forget to enable this logger in `airflow.cfg` with `logging_config_class = log_config.LOGGING_CONFIG`

Where did you add the log_config.py file? In the airflow docker image or with the dags?


@kevlubkcm ☝️ 
I cut [an issue](https://github.com/kislyuk/watchtower/issues/154) to the watchtower repo (since the other was closed) with some possible fixes I had tested. But it seems like there is already a fix that *just* got released into Watchtower 2.0.

I have yet to do some testing with the new release, but once I get around to (and assuming it works as expected) it should be an easy fix to bump the watchtower version in Airflow to 2.0.

I'm happy to be assigned this issue if no one else is working on it.
Update: My testing revealed another edge case where empty messages were failing watchtower/cloudwatch, I've submitted a PR to the watchtower repo to fix it: https://github.com/kislyuk/watchtower/pull/162

Resolving patch:
diff --git a/airflow/providers/amazon/aws/log/cloudwatch_task_handler.py b/airflow/providers/amazon/aws/log/cloudwatch_task_handler.py
--- a/airflow/providers/amazon/aws/log/cloudwatch_task_handler.py
+++ b/airflow/providers/amazon/aws/log/cloudwatch_task_handler.py
@@ -81,7 +81,7 @@ def set_context(self, ti):
         self.handler = watchtower.CloudWatchLogHandler(
             log_group=self.log_group,
             stream_name=self._render_filename(ti, ti.try_number),
-            boto3_session=self.hook.get_session(self.region_name),
+            boto3_client=self.hook.get_conn(),
         )
 
     def close(self):
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -189,7 +189,7 @@ def write_version(filename: str = os.path.join(*[my_dir, "airflow", "git_version
 ]
 amazon = [
     'boto3>=1.15.0,<1.19.0',
-    'watchtower~=1.0.6',
+    'watchtower~=2.0.1',
     'jsonpath_ng>=1.5.3',
     'redshift_connector~=2.0.888',
     'sqlalchemy_redshift~=0.8.6',


Test patch: