Issue 1:
Apache Airflow version

main (development)

What happened

Variables with newlines in them are not always reproduced exactly as they are pasted into the web UI. Depending on how they are rendered, Unix newlines (\n) are silently converted to Windows newlines (\r\n) when rendered.

Additionally, when users try to specify \r CR/"carriage return" in Python code, that gets rendered as a \n LF/"line feed" character.

Tested on both astro and Airflow with Breeze.

What you think should happen instead

Unix newlines (\n) should always be rendered as Unix newlines (\n); Windows newlines (\r\n) should always be rendered as Windows newlines. Not 100% sure anymore.

How to reproduce

Steps to reproduce:

Spin up Airflow with Breeze (breeze start-airflow) - I tested with the PostgreSQL database backend

Install dos2unix within the Airflow container: apt update; apt install --yes dos2unix

Create a Variable in the Airflow UI named newlines, and ensure that its content contains newlines. Note that the non-newline content doesn't really matter. I used:

Line 1
Line 2
Line 3

Run this dag:

from datetime import datetime

from airflow import DAG
from airflow.operators.bash import BashOperator
from airflow.operators.smooth import SmoothOperator

with DAG("test_newlines", schedule=None) as dag:
test_bash_operator_multiline_env_var = BashOperator(
dag=dag,
task_id="test_bash_operator_multiline_env_var",
start_date=datetime.now(),
env={
"MULTILINE_ENV_VAR": """Line 1
Line 2
Line 3"""
},
append_env=True,
bash_command='''
if [[ "$(echo $MULTILINE_ENV_VAR)" != "$(echo $MULTILINE_ENV_VAR | dos2unix)" ]]; then
echo >&2 "Multiline environment variable contains newlines incorrectly converted to Windows CRLF"
exit 1
fi''',
)

test_bash_operator_heredoc_contains_newlines = BashOperator(
dag=dag,
task_id="test_bash_operator_heredoc_contains_newlines",
start_date=datetime.now(),
bash_command="""
diff <(
cat <<EOF
Line 1
Line 2
Line 3
EOF
) <(
cat <<EOF | dos2unix # Hat-tip: @potiuk
Line 1
Line 2
Line 3
EOF
) || {
echo >&2 "Bash heredoc contains newlines incorrectly converted to Windows CRLF"
exit 1
}
""",
)

test_bash_operator_env_var_from_variable_jinja_interpolation = BashOperator(
dag=dag,
task_id="test_bash_operator_env_var_from_variable_jinja_interpolation",
start_date=datetime.now(),
env={
"ENV_VAR_AIRFLOW_VARIABLE_WITH_NEWLINES": "{{ var.value['newlines'] }}",
},
append_env=True,
bash_command='''
diff <(echo "$ENV_VAR_AIRFLOW_VARIABLE_WITH_NEWLINES") <(echo "$ENV_VAR_AIRFLOW_VARIABLE_WITH_NEWLINES" | dos2unix) || {
echo >&2 "Environment variable contains newlines incorrectly converted to Windows CRLF"
exit 1
}
''',
)

test_bash_operator_from_variable_jinja_interpolation = BashOperator(
dag=dag,
task_id="test_bash_operator_from_variable_jinja_interpolation",
start_date=datetime.now(),
bash_command='''
diff <(echo "{{ var.value['newlines'] }}") <(echo "{{ var.value['newlines'] }}" | dos2unix) || {
echo >&2 "Jinja interpolated string contains newlines incorrectly converted to Windows CRLF"
exit 1
}
''',
)

test_bash_operator_backslash_n_not_equals_backslash_r = BashOperator(
dag=dag,
task_id="test_bash_operator_backslash_n_not_equals_backslash_r",
start_date=datetime.now(),
bash_command='''
if [[ "\r" == "\n" ]]; then
echo >&2 "Backslash-r has been incorrectly converted into backslash-n"
exit 1
fi''',
)

passing = SmoothOperator(dag=dag, task_id="passing", start_date=datetime.now())
failing = SmoothOperator(dag=dag, task_id="failing", start_date=datetime.now())

[
test_bash_operator_multiline_env_var,
test_bash_operator_heredoc_contains_newlines,
] >> passing
[
test_bash_operator_env_var_from_variable_jinja_interpolation,
test_bash_operator_from_variable_jinja_interpolation,
test_bash_operator_backslash_n_not_equals_backslash_r,
] >> failing

Warning: The test_bash_operator_heredoc_contains_newlines may not actually ever complete in Breeze. It does complete successfully when using astro. That might be another bug. @potiuk's fix may have fixed this.

Operating System

macOS 11.6.8 "Big Sur"

Versions of Apache Airflow Providers

No response

Deployment

Astronomer

Deployment details

Local breeze environment

Local astro environment

Anything else

Are you willing to submit PR?

Yes I am willing to submit a PR!

Code of Conduct

I agree to follow this project's Code of Conduct

Issue 2:
Description

The old documentation for Contributor's guide should be reviewed and incorporated in CONTRIBUTING.rst

Use case / motivation

To remove documentation duplication and clean it up.

Related Issues

#10178