Issue 1:
Apache Airflow version: 1.10.12

Environment: VMware VM

Cloud provider or hardware configuration: 8 G RAM, 50G HDD, 4Cores

OS:

NAME="CentOS Linux"

VERSION="7 (Core)"

ID="centos"

ID_LIKE="rhel fedora"

VERSION_ID="7"

PRETTY_NAME="CentOS Linux 7 (Core)"

ANSI_COLOR="0;31"

CPE_NAME="cpe:/o:centos:centos:7"

HOME_URL="https://www.centos.org/"

BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"

CENTOS_MANTISBT_PROJECT_VERSION="7"

REDHAT_SUPPORT_PRODUCT="centos"

REDHAT_SUPPORT_PRODUCT_VERSION="7"

Kernel:

Linux mycentosvm 3.10.0-1160.6.1.el7.x86_64 #1 SMP Tue Nov 17 13:59:11 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Install tools:

yum groupinstall "Development tools"

yum install zlib-devel bzip2-devel openssl-devel ncurses-devel sqlite-devel python3-devel wget gcc gcc-c++ libffi-devel cyrus-sasl-devel

yum install mysql-server [version8]

[set global_explicit, create airflow database and user then give permissions]

alternative python 3.7 [pip 20.2.4 setuptools 47.1.0 ]

What happened:

All kinds of errors:

I copied https://raw.githubusercontent.com/apache/airflow/constraints-1.10.12/constraints-3.7.txt to a file called req.txt to be able to change the dependencies

Error1:

ERROR: Could not satisfy constraints for 'pyhive': installation from path or url cannot be constrained to a version

My solution1:

Install specified pyhive manually

pip install PyHive==0.6.3

Error2:

ERROR: Cannot install apache-airflow, apache-airflow and apache-airflow because these package versions have conflicting dependencies.

The conflict is caused by:
email-validator 1.1.1 depends on idna>=2.0.0
requests 2.24.0 depends on idna<3 and >=2.5
moto 1.3.14 depends on idna<2.9 and >=2.5

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies

My solution2:

pip install idna==2.9

Error3:

ERROR: Cannot install apache-airflow[async,celery,devel,devel-hadoop,druid,hdfs,hive,jdbc,mysql,rabbitmq,ssh] 1.10.12, apache-airflow and apache-airflow because these package versions have conflicting dependencies.

The conflict is caused by:
apache-airflow[async,celery,devel,devel-hadoop,druid,hdfs,hive,jdbc,mysql,rabbitmq,ssh] 1.10.12 depends on pytest<6.0.0; extra == "devel_hadoop"
pytest-rerunfailures 9.0 depends on pytest>=5.0
pytest-xdist 2.0.0 depends on pytest>=6.0.0

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies

My solution3:

pip install pytest==6.0.0

Error4:

Downloading eventlet-0.14.0.tar.gz (279 kB)
|████████████████████████████████| 279 kB 404 kB/s
ERROR: Command errored out with exit status 1:
command: /usr/local/bin/python3.7 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-ozxuwjov/eventlet_711991891ddc448ba8dd97630910301c/setup.py'"'"'; __file__='"'"'/tmp/pip-install-ozxuwjov/eventlet_711991891ddc448ba8dd97630910301c/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-aw4qpa8h
cwd: /tmp/pip-install-ozxuwjov/eventlet_711991891ddc448ba8dd97630910301c/
Complete output (8 lines):
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-install-ozxuwjov/eventlet_711991891ddc448ba8dd97630910301c/setup.py", line 3, in <module>
from eventlet import __version__
File "/tmp/pip-install-ozxuwjov/eventlet_711991891ddc448ba8dd97630910301c/eventlet/__init__.py", line 42
except ImportError, e:
^
SyntaxError: invalid syntax
----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

My solution4:

pip install eventlet==0.29.1

Error5:

bitmq,ssh]==1.10.12) (0.4.17)
ERROR: Could not find a version that satisfies the requirement pytest<6.0.0; extra == "devel_hadoop" (from apache-airflow[async,celery,devel,devel-hadoop,druid,hdfs,hive,jdbc,mysql,rabbitmq,ssh])
ERROR: No matching distribution found for pytest<6.0.0; extra == "devel_hadoop"

then install older version and i get this error:

Downloading paramiko-2.7.1-py2.py3-none-any.whl (206 kB)
|████████████████████████████████| 206 kB 377 kB/s
Collecting pytest-xdist
Using cached pytest_xdist-2.0.0-py2.py3-none-any.whl (36 kB)
ERROR: Cannot install apache-airflow[async,celery,devel,devel-hadoop,druid,hdfs,hive,jdbc,mysql,rabbitmq,ssh] 1.10.12 and apache-airflow because these package versions have conflicting dependencies.

The conflict is caused by:
apache-airflow[async,celery,devel,devel-hadoop,druid,hdfs,hive,jdbc,mysql,rabbitmq,ssh] 1.10.12 depends on pytest<6.0.0; extra == "devel_hadoop"
pytest-xdist 2.0.0 depends on pytest>=6.0.0

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies

I even don't know any of my solutions even work!

How to reproduce it:

Run this command

pip install \
apache-airflow[async,celery,devel,devel_hadoop,druid,hdfs,hive,jdbc,mysql,rabbitmq,ssh]==1.10.12 \
--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-1.10.12/constraints-3.7.txt" \
--use-feature=2020-resolver

Anything else we need to know:

According to the announcement linked below, pip will introduce a new dependency resolver in October 2020, which will be more robust but might break some existing setups. Therefore they are suggesting users to try running their pip install scripts at least once (in dev mode) with this option: --use-feature=2020-resolver to anticipate any potential issue before the new resolver becomes the default in October 2020 with pip version 20.3.

How often does this problem occur? Once? Every time etc?

Everytime, the dependency problem is a nightmare, especially with adding hadoop extras

Issue 2:
Description

We should consider adding pip check command in our CI system to verify that dependencies that we use (together with constrains) are fully compatible. This should be run for biggest possible user-installation (apache-airflow[all]).

Use case / motivation

Successful pip check is often a business requirement for releasing new version of Airflow. It assure us that all dependencies we use to build and run Airflow are compatible and we won't face any unexpected errors.

Related Issues

N/A