Issue 1:
Apache Airflow version

2.2.0 (latest released)

Operating System

Debian GNU/Linux 11 (bullseye)

Versions of Apache Airflow Providers

apache-airflow-providers-ftp==2.0.1

apache-airflow-providers-http==2.0.1

apache-airflow-providers-imap==2.0.1

apache-airflow-providers-postgres==2.3.0

apache-airflow-providers-sqlite==2.0.1

Deployment

Other Docker-based deployment

Deployment details

The host machine runs Windows 10 Enterprise build 19043.1237. The version of Docker Desktop is 4.1.1 (69879). Docker image is based on python:3.8-slim, and it's a single-machine deployment.

What happened

If runs a python script with BashOperator and supplies any "env" argument to BashOperator, the sys.executable variable of the python session becomes None

What you expected to happen

sys.executable should populates the path of python executable when using BashOperator with "env" argument

How to reproduce

I was running the following DAG

import datetime

from airflow import DAG
from airflow.operators.bash import BashOperator

with DAG(
dag_id='testdag',
start_date=datetime.datetime(2021,10,1),
schedule_interval='0 11 * * *', # daily 11 AM UTC
catchup=False
) as dag:

test_task = BashOperator(
task_id = 'test_task'
,bash_command= 'echo $(python3 -c "import sys;print(sys.executable)")'
,env={'test':'test'}
)

And the log showed:

[2021-10-22, 12:02:09 EDT] {taskinstance.py:1035} INFO - Dependencies all met for <TaskInstance: testdag.test_task manual__2021-10-22T16:02:07.768715+00:00 [queued]>
[2021-10-22, 12:02:09 EDT] {taskinstance.py:1035} INFO - Dependencies all met for <TaskInstance: testdag.test_task manual__2021-10-22T16:02:07.768715+00:00 [queued]>
[2021-10-22, 12:02:09 EDT] {taskinstance.py:1241} INFO -
--------------------------------------------------------------------------------
[2021-10-22, 12:02:09 EDT] {taskinstance.py:1242} INFO - Starting attempt 1 of 1
[2021-10-22, 12:02:09 EDT] {taskinstance.py:1243} INFO -
--------------------------------------------------------------------------------
[2021-10-22, 12:02:09 EDT] {taskinstance.py:1262} INFO - Executing <Task(BashOperator): test_task> on 2021-10-22 16:02:07.768715+00:00
[2021-10-22, 12:02:09 EDT] {standard_task_runner.py:52} INFO - Started process 15435 to run task
[2021-10-22, 12:02:09 EDT] {standard_task_runner.py:76} INFO - Running: ['airflow', 'tasks', 'run', 'testdag', 'test_task', 'manual__2021-10-22T16:02:07.768715+00:00', '--job-id', '31', '--raw', '--subdir', 'DAGS_FOLDER/test.py', '--cfg-path', '/tmp/tmpwzxrybgg', '--error-file', '/tmp/tmpqqsq91p4']
[2021-10-22, 12:02:09 EDT] {standard_task_runner.py:77} INFO - Job 31: Subtask test_task
[2021-10-22, 12:02:09 EDT] {logging_mixin.py:109} INFO - Running <TaskInstance: testdag.test_task manual__2021-10-22T16:02:07.768715+00:00 [running]> on host 8f4f12aea4ed
[2021-10-22, 12:02:10 EDT] {taskinstance.py:1412} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=airflow
AIRFLOW_CTX_DAG_ID=testdag
AIRFLOW_CTX_TASK_ID=test_task
AIRFLOW_CTX_EXECUTION_DATE=2021-10-22T16:02:07.768715+00:00
AIRFLOW_CTX_DAG_RUN_ID=manual__2021-10-22T16:02:07.768715+00:00
[2021-10-22, 12:02:10 EDT] {subprocess.py:62} INFO - Tmp dir root location:
/tmp
[2021-10-22, 12:02:10 EDT] {subprocess.py:74} INFO - Running command: ['bash', '-c', 'echo $(python3 -c "import sys;print(sys.executable)")']
[2021-10-22, 12:02:10 EDT] {subprocess.py:85} INFO - Output:
[2021-10-22, 12:02:10 EDT] {subprocess.py:89} INFO -
[2021-10-22, 12:02:10 EDT] {subprocess.py:93} INFO - Command exited with return code 0
[2021-10-22, 12:02:10 EDT] {taskinstance.py:1270} INFO - Marking task as SUCCESS. dag_id=testdag, task_id=test_task, execution_date=20211022T160207, start_date=20211022T160209, end_date=20211022T160210
[2021-10-22, 12:02:10 EDT] {local_task_job.py:154} INFO - Task exited with return code 0
[2021-10-22, 12:02:10 EDT] {local_task_job.py:264} INFO - 0 downstream tasks scheduled from follow-on schedule check

In my tests, as long as something is passed through "env" argument to the BashOperator which runs a python scripts, the spawned python session will have the sys.executable variable as None.

Anything else

The actual python script I tried to run is using Snowflake.connector library. The library tries to identify the underlying OS/platform using platform.libc_ver(sys.executable) function call. Very weirdly, in this case, the value of sys.executable becomes the temporary execution directory (such as /tmp/airflowtmpyes4ub8l in the following log). Not sure if this info helps.

DAG

import datetime

from airflow import DAG
from airflow.operators.bash import BashOperator

with DAG(
dag_id='testdag',
start_date=datetime.datetime(2021,10,1),
schedule_interval='0 11 * * *', # daily 11 AM UTC
catchup=False
) as dag:

test_task = BashOperator(
task_id = 'test_task'
,bash_command = 'python3 -c "import snowflake.connector"'
,env={'test':'test'}

The log

[2021-10-22, 12:34:52 EDT] {taskinstance.py:1035} INFO - Dependencies all met for <TaskInstance: testdag.test_task manual__2021-10-22T16:34:48.077920+00:00 [queued]>
[2021-10-22, 12:34:52 EDT] {taskinstance.py:1035} INFO - Dependencies all met for <TaskInstance: testdag.test_task manual__2021-10-22T16:34:48.077920+00:00 [queued]>
[2021-10-22, 12:34:52 EDT] {taskinstance.py:1241} INFO -
--------------------------------------------------------------------------------
[2021-10-22, 12:34:52 EDT] {taskinstance.py:1242} INFO - Starting attempt 1 of 1
[2021-10-22, 12:34:52 EDT] {taskinstance.py:1243} INFO -
--------------------------------------------------------------------------------
[2021-10-22, 12:34:52 EDT] {taskinstance.py:1262} INFO - Executing <Task(BashOperator): test_task> on 2021-10-22 16:34:48.077920+00:00
[2021-10-22, 12:34:52 EDT] {standard_task_runner.py:52} INFO - Started process 17378 to run task
[2021-10-22, 12:34:52 EDT] {standard_task_runner.py:76} INFO - Running: ['airflow', 'tasks', 'run', 'testdag', 'test_task', 'manual__2021-10-22T16:34:48.077920+00:00', '--job-id', '37', '--raw', '--subdir', 'DAGS_FOLDER/test.py', '--cfg-path', '/tmp/tmp0smgg6b6', '--error-file', '/tmp/tmpa9trmhpd']
[2021-10-22, 12:34:52 EDT] {standard_task_runner.py:77} INFO - Job 37: Subtask test_task
[2021-10-22, 12:34:52 EDT] {logging_mixin.py:109} INFO - Running <TaskInstance: testdag.test_task manual__2021-10-22T16:34:48.077920+00:00 [running]> on host 8f4f12aea4ed
[2021-10-22, 12:34:52 EDT] {taskinstance.py:1412} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=airflow
AIRFLOW_CTX_DAG_ID=testdag
AIRFLOW_CTX_TASK_ID=test_task
AIRFLOW_CTX_EXECUTION_DATE=2021-10-22T16:34:48.077920+00:00
AIRFLOW_CTX_DAG_RUN_ID=manual__2021-10-22T16:34:48.077920+00:00
[2021-10-22, 12:34:52 EDT] {subprocess.py:62} INFO - Tmp dir root location:
/tmp
[2021-10-22, 12:34:52 EDT] {subprocess.py:74} INFO - Running command: ['bash', '-c', 'python3 -c "import snowflake.connector"']
[2021-10-22, 12:34:52 EDT] {subprocess.py:85} INFO - Output:
[2021-10-22, 12:34:52 EDT] {subprocess.py:89} INFO - Traceback (most recent call last):
[2021-10-22, 12:34:52 EDT] {subprocess.py:89} INFO - File "<string>", line 1, in <module>
[2021-10-22, 12:34:52 EDT] {subprocess.py:89} INFO - File "/usr/local/lib/python3.8/site-packages/snowflake/connector/__init__.py", line 15, in <module>
[2021-10-22, 12:34:52 EDT] {subprocess.py:89} INFO - from .connection import SnowflakeConnection
[2021-10-22, 12:34:52 EDT] {subprocess.py:89} INFO - File "/usr/local/lib/python3.8/site-packages/snowflake/connector/connection.py", line 31, in <module>
[2021-10-22, 12:34:52 EDT] {subprocess.py:89} INFO - from . import errors, proxy
[2021-10-22, 12:34:52 EDT] {subprocess.py:89} INFO - File "/usr/local/lib/python3.8/site-packages/snowflake/connector/errors.py", line 14, in <module>
[2021-10-22, 12:34:52 EDT] {subprocess.py:89} INFO - from .description import CLIENT_NAME, SNOWFLAKE_CONNECTOR_VERSION
[2021-10-22, 12:34:52 EDT] {subprocess.py:89} INFO - File "/usr/local/lib/python3.8/site-packages/snowflake/connector/description.py", line 17, in <module>
[2021-10-22, 12:34:52 EDT] {subprocess.py:89} INFO - PLATFORM = platform.platform()
[2021-10-22, 12:34:52 EDT] {subprocess.py:89} INFO - File "/usr/local/lib/python3.8/platform.py", line 1206, in platform
[2021-10-22, 12:34:52 EDT] {subprocess.py:89} INFO - libcname, libcversion = libc_ver(sys.executable)
[2021-10-22, 12:34:52 EDT] {subprocess.py:89} INFO - File "/usr/local/lib/python3.8/platform.py", line 193, in libc_ver
[2021-10-22, 12:34:52 EDT] {subprocess.py:89} INFO - with open(executable, 'rb') as f:
[2021-10-22, 12:34:52 EDT] {subprocess.py:89} INFO - IsADirectoryError: [Errno 21] Is a directory: '/tmp/airflowtmpyes4ub8l'
[2021-10-22, 12:34:52 EDT] {subprocess.py:93} INFO - Command exited with return code 1
[2021-10-22, 12:34:52 EDT] {taskinstance.py:1686} ERROR - Task failed with exception
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1324, in _run_raw_task
self._execute_task_with_callbacks(context)
File "/usr/local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1443, in _execute_task_with_callbacks
result = self._execute_task(context, self.task)
File "/usr/local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1499, in _execute_task
result = execute_callable(context=context)
File "/usr/local/lib/python3.8/site-packages/airflow/operators/bash.py", line 187, in execute
raise AirflowException(
airflow.exceptions.AirflowException: Bash command failed. The command returned a non-zero exit code 1.
[2021-10-22, 12:34:52 EDT] {taskinstance.py:1270} INFO - Marking task as FAILED. dag_id=testdag, task_id=test_task, execution_date=20211022T163448, start_date=20211022T163452, end_date=20211022T163452
[2021-10-22, 12:34:52 EDT] {standard_task_runner.py:88} ERROR - Failed to execute job 37 for task test_task
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/airflow/task/task_runner/standard_task_runner.py", line 85, in _start_by_fork
args.func(args, dag=self.dag)
File "/usr/local/lib/python3.8/site-packages/airflow/cli/cli_parser.py", line 48, in command
return func(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/airflow/utils/cli.py", line 92, in wrapper
return f(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/airflow/cli/commands/task_command.py", line 292, in task_run
_run_task_by_selected_method(args, dag, ti)
File "/usr/local/lib/python3.8/site-packages/airflow/cli/commands/task_command.py", line 107, in _run_task_by_selected_method
_run_raw_task(args, ti)
File "/usr/local/lib/python3.8/site-packages/airflow/cli/commands/task_command.py", line 180, in _run_raw_task
ti._run_raw_task(
File "/usr/local/lib/python3.8/site-packages/airflow/utils/session.py", line 70, in wrapper
return func(*args, session=session, **kwargs)
File "/usr/local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1324, in _run_raw_task
self._execute_task_with_callbacks(context)
File "/usr/local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1443, in _execute_task_with_callbacks
result = self._execute_task(context, self.task)
File "/usr/local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1499, in _execute_task
result = execute_callable(context=context)
File "/usr/local/lib/python3.8/site-packages/airflow/operators/bash.py", line 187, in execute
raise AirflowException(
airflow.exceptions.AirflowException: Bash command failed. The command returned a non-zero exit code 1.
[2021-10-22, 12:34:52 EDT] {local_task_job.py:154} INFO - Task exited with return code 1
[2021-10-22, 12:34:52 EDT] {local_task_job.py:264} INFO - 0 downstream tasks scheduled from follow-on schedule check

Thank you so much for the help!

Are you willing to submit PR?

Yes I am willing to submit a PR!

Code of Conduct

I agree to follow this project's Code of Conduct

Issue 2:
Body

I have a kind request for all the contributors to the latest provider packages release.

Could you help us to test the RC versions of the providers and let us know in the comment,

if the issue is addressed there.

Providers that need testing

Those are providers that require testing as there were some substantial changes introduced:

Provider amazon: 2.3.0rc1

Add IAM Role Credentials to S3ToRedshiftTransfer and RedshiftToS3Transfer (#18156): @john-jac

Adding missing replace param directive in docstring of SalesforceToS3Operator (#18241): @josh-fell

Added upsert method on S3ToRedshift operator (#18027): @mariotaddeucci

Add Spark to the EMR cluster for the job flow examples (#17563): @snolan-amount

Update s3_list.py (#18561): @Xilorole

ECSOperator realtime logging (#17626): @codenamestif

Deprecate default pod name in EKSPodOperator (#18036): @mik-laj

Aws secrets manager backend (#17448): @JavierLopezT

sftp_to_s3 stream file option (#17609): @john-jac

AwsBaseHook make client_type & resource_type optional params for get_client_type & get_resource_type (#17987): @eladkal

Delete unnecessary parameters in EKSPodOperator (#17960): @mik-laj

ECSOperator returns last logs when ECS task fails (#17209): @pmalafosse

Refresh credentials for long-running pods on EKS (#17951): @mik-laj

Provider apache.cassandra: 2.1.0rc1

Adding a default conn ID value for Apache Cassandra sensors (#18620): @josh-fell

Provider apache.hdfs: 2.1.1rc1

hdfs provider: fix deprecation warning in WebHDFSHook (#18331): @Aakcht

Provider cncf.kubernetes: 2.0.3rc1

Fix KubernetesPodOperator reattach when not deleting pods (#18070): @jedcunningham

Make Kubernetes job description fit on one log line (#18377): @collinmcnulty

Do not fail KubernetesPodOperator tasks if log following fails (#17649): @baryluk

Provider databricks: 2.0.2rc1

Move DB call out of DatabricksHook.__init__ (#18339): @kaxil

Provider docker: 2.2.0rc1

Add a Docker Taskflow decorator (#15330): @dimberman

Provider google: 6.0.0rc1

Migrate Google Cloud Build from Discovery API to Python SDK (#18184): @ryanyuan

Add index to the dataset name to have separate dataset for each DAG (#18459): @deedmitrij

Add missing init.py files for some test packages (#18142): @potiuk

Add possibility to run DAGs from system tests and see DAGs logs (#17868): @deedmitrij

Rename AzureDataLakeStorage to ADLS (#18493): @eladkal

Make next_dagrun_info take a data interval (#18088): @uranusjr

Use parameters instead of params (#18143): @msumit

New google operator: SQLToGoogleSheetsOperator (#17887): @mariotaddeucci

Fix part of Google system tests (#18494): @mnojek

Fix kubernetes engine system test (#18548): @deedmitrij

Fix BigQuery system test (#18373): @deedmitrij

Fix error when create external table using table resource (#17998): @tnyz

Fix BigQuery data extraction in BigQueryToMySqlOperator (#18073): @wolfier

Fix providers tests in main branch with eager upgrades (#18040): @potiuk

fix(CloudSqlProxyRunner): don't query connections from Airflow DB (#18006): @m1racoli

Remove check for at least one schema in GCSToBigquery (#18150): @c-nuro

deduplicate running jobs on BigQueryInsertJobOperator (#17496): @blcksrx

Provider hashicorp: 2.1.1rc1

Fixing Vault AppRole authentication with CONN_URI (#18064): @nathadfield

Provider microsoft.azure: 3.2.0rc1

Rename AzureDataLakeStorage to ADLS (#18493): @eladkal

Creating ADF pipeline run operator, sensor + ADF custom conn fields (#17885): @josh-fell

Rename LocalToAzureDataLakeStorageOperator to LocalFilesystemToADLSOperator (#18168): @eladkal

Rename FileToWasbOperator to LocalFilesystemToWasbOperator (#18109): @eladkal

Fixed wasb hook attempting to create container when getting a blob client (#18287): @ignaski

Removing redundant relabeling of ADX conn field (#18386): @josh-fell

Proper handling of Account URL custom conn field in AzureBatchHook (#18456): @josh-fell

Provider microsoft.psrp: 1.0.1rc1

Fix unexpected bug in exiting PSRPHook.client context manager (#18014): @malthe

Provider neo4j: 2.0.2rc1

fix Neo4jHook to get the query response (#18007): @minu7

Provider papermill: 2.1.0rc1

Add support for templated fields in PapermillOperator (#18357): @eladkal

Fix usage of range(len()) to enumerate (#18174): @kaxil

Provider postgres: 2.3.0rc1

Added upsert method on S3ToRedshift operator (#18027): @mariotaddeucci

Fix example dag of PostgresOperator (#18236): @eladkal

Provider slack: 4.1.0rc1

Restore 'filename' to template_fields of SlackAPIFileOperator (#18466): @SayTen

Provider snowflake: 2.2.0rc1

Add Snowflake DQ Operators (#17741): @denimalpaca

Provider ssh: 2.2.0rc1

[Airflow 16364] Add conn_timeout and cmd_timeout params to SSHOperator; add conn_timeout param to SSHHook (#17236): @fatmumuhomer

New Providers:

Provider influxdb: 1.0.0rc1: @subkanthi

Providers that do not need testing

We have a number of providers that had doc-only changes so they are not released.

Committer

I acknowledge that I am a maintainer/committer of the Apache Airflow project.