Repository: apache/airflow

Issue:
UnicodeDecodeError in bash_operator.py
Hi,

I see a lot of these errors when running `airflow backfill` : 

```
Traceback (most recent call last):
  File "/usr/lib/python2.7/logging/__init__.py", line 851, in emit
    msg = self.format(record)
  File "/usr/lib/python2.7/logging/__init__.py", line 724, in format
    return fmt.format(record)
  File "/usr/lib/python2.7/logging/__init__.py", line 467, in format
    s = self._fmt % record.__dict__
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 13: ordinal not in range(128)
Logged from file bash_operator.py, line 72
```



Discussion:
I encountered the same problem. I have hacked the [line#73 in airflow/operators/bash_operator.py](https://github.com/airbnb/airflow/blob/master/airflow/operators/bash_operator.py#L73) to 
`line = line.decode('utf-8').strip()` to fix my problem.

error log

```
[2015-12-22 18:17:51,354] {bash_operator.py:70} INFO - Output:
[2015-12-22 18:17:51,359] {models.py:1041} ERROR - 'ascii' codec can't decode byte 0xe5 in position 4: ordinal not in range(128)
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 1000, in run
    result = task_copy.execute(context=context)
  File "/usr/local/lib/python2.7/dist-packages/airflow/operators/bash_operator.py", line 73, in execute
    line = line.decode().strip()
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe5 in position 4: ordinal not in range(128)
[2015-12-22 18:17:51,388] {models.py:1077} ERROR - 'ascii' codec can't decode byte 0xe5 in position 4: ordinal not in range(128)
```

There is already a `help-wanted` tag posted on #721 

@GeiliCode How about submit a pull request for this? I think it's a common problem. thx

@haitaoyao I am not familiar with encoding, so I don't know how to deal with encoding which is not  `ascii` nor `utf-8`.


Resolving patch:
diff --git a/airflow/operators/bash_operator.py b/airflow/operators/bash_operator.py
--- a/airflow/operators/bash_operator.py
+++ b/airflow/operators/bash_operator.py
@@ -1,7 +1,6 @@
 
 from builtins import bytes
 import logging
-import sys
 from subprocess import Popen, STDOUT, PIPE
 from tempfile import gettempdir, NamedTemporaryFile
 
@@ -22,6 +21,7 @@ class BashOperator(BaseOperator):
         of inheriting the current process environment, which is the default
         behavior.
     :type env: dict
+    :type output_encoding: output encoding of bash command
     """
     template_fields = ('bash_command', 'env')
     template_ext = ('.sh', '.bash',)
@@ -33,6 +33,7 @@ def __init__(
             bash_command,
             xcom_push=False,
             env=None,
+            output_encoding='utf-8',
             *args, **kwargs):
         """
         If xcom_push is True, the last line written to stdout will also
@@ -42,6 +43,7 @@ def __init__(
         self.bash_command = bash_command
         self.env = env
         self.xcom_push_flag = xcom_push
+        self.output_encoding = output_encoding
 
     def execute(self, context):
         """
@@ -70,7 +72,7 @@ def execute(self, context):
                 logging.info("Output:")
                 line = ''
                 for line in iter(sp.stdout.readline, b''):
-                    line = line.decode().strip()
+                    line = line.decode(self.output_encoding).strip()
                     logging.info(line)
                 sp.wait()
                 logging.info("Command exited with "


Test patch:
diff --git a/tests/core.py b/tests/core.py
--- a/tests/core.py
+++ b/tests/core.py
@@ -331,6 +331,14 @@ def test_bash_operator(self):
             dag=self.dag)
         t.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE, force=True)
 
+    def test_bash_operator_multi_byte_output(self):
+        t = operators.BashOperator(
+                task_id='test_multi_byte_bash_operator',
+                bash_command=u"echo \u2600",
+                dag=self.dag,
+                output_encoding='utf-8')
+        t.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE, force=True)
+
     def test_trigger_dagrun(self):
         def trigga(context, obj):
             if True: