Repository: apache/airflow

Issue:
Snowflake python connector monkeypatches urllib and makes many services unusable.
Curreently wnen you run snowflke provider, it monkeypatches urlllb in a way that is not compatible with other libraries (for example presto SSL with kerberos, google,  amazon, qubole and many others).

This is not critical (as in 2.0 we have provider separation and snowflake code will not even be there until you choose [snowflake] extra or install provider manually,

For now we decided to release but immediately yank the snowflake provider!

Additional links: 

* Issue: https://github.com/snowflakedb/snowflake-connector-python/issues/324

Offending code:

* https://github.com/snowflakedb/snowflake-connector-python/blob/133d6215f7920d304c5f2d466bae38127c1b836d/src/snowflake/connector/network.py#L89-L92




Discussion:
cc: @mik-laj 
Raised the issue with my contact at Snowflake
Seems like a fix is finally on the way https://github.com/snowflakedb/snowflake-connector-python/pull/557
I'm still not sure _why_ they are even doing such monkeypatching. Does anyone know why they do this?
I read all issues related to that there and never found why they did that...
The main issue was open since June. But with community backlash seems like they started working on it. :pray: 
@ashb OCSP support. 
Yeah. OCSP support was the root cause, but the solution to monkeypatch a library that others use was, well.. offensive. They could have vendoered it in as we did with some other packages in the past 
Or submitted the patch upstream :) 
fyi, we have been trying to get snowflake to fix this for months.

As an alternative, I made a [lite version of snowflake-connector package](https://github.com/manugarri/snowflake-connector-python-lite) that i use in my airflow environments in case someone wants to check it out
Thanks. I think we are just trying to push snowflake (and we have some heavyweights helping us) to fix it, so let's wait and see :).
**Update:** just heard back from Snowflake. The current estimation is that they will fix it by end of this year.
I'll keep track
great @eladkal . thanks!
Thanks @eladkal !
Yeah, I think NOT switching to a slimmed down version of the lib is a better incentive for Snowflake to fix it :)
agree @potiuk , but i dont work in a heavyweight company so i had to make our data pipelines and reports work.  

I would definitely recommend a full rewrite of the python package, how it handles dependencies is an absolute mess.
Let's see. I hope things will get better with Snowflke's integrations and ecosystem in general. They have a very, very cool product.
I would assume this https://github.com/apache/airflow/issues/13255 issue I raised is related (?)
Absolutely @davido912 
Snowflake released new version of the connector

https://pypi.org/project/snowflake-connector-python/
```
v2.3.8
Arrow result conversion speed up.
Send all Python Connector exceptions to in-band or out-of-band telemetry.
Vendoring requests and urllib3 to contain OCSP monkey patching to our library only.
Declare dependency on setuptools.
```
Woooho! Switching to it!
what a day. let us know if this solved anything! :D 

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -429,14 +429,10 @@ def get_sphinx_theme_version() -> str:
     # once it is merged, we can move those two back to `azure` extra.
     'azure-storage-blob',
     'azure-storage-common',
-    # snowflake is not compatible with latest version.
-    # This library monkey patches the requests library, so SSL is broken globally.
-    # See: https://github.com/snowflakedb/snowflake-connector-python/issues/324
-    'requests<2.24.0',
-    # Newest version drop support for old version of azure-storage-blob
-    # Until #12188 is solved at least we need to limit maximum version.
-    # https://github.com/apache/airflow/pull/12188
-    'snowflake-connector-python>=1.5.2,<=2.3.6',
+    # Snowflake conector > 2.3.8 is needed because it has vendored urrllib3 and requests libraries which
+    # are monkey-patched. In earlier versions of the library, monkeypatching the libraries by snowflake
+    # caused other providers to fail (Google, Amazon etc.)
+    'snowflake-connector-python>=2.3.8',
     'snowflake-sqlalchemy>=1.1.0',
 ]
 spark = [
@@ -473,11 +469,6 @@ def get_sphinx_theme_version() -> str:
 ]
 # End dependencies group
 
-############################################################################################################
-# IMPORTANT NOTE!!!!!!!!!!!!!!!
-# IF you are removing dependencies from this list, please make sure that you also increase
-# PIP_DEPENDENCIES_EPOCH_NUMBER in the Dockerfile.ci and Dockerfile
-############################################################################################################
 devel = [
     'beautifulsoup4~=4.7.1',
     'black',
@@ -522,26 +513,9 @@ def get_sphinx_theme_version() -> str:
     'yamllint',
 ]
 
-############################################################################################################
-# IMPORTANT NOTE!!!!!!!!!!!!!!!
-# If you are removing dependencies from the above list, please make sure that you also increase
-# PIP_DEPENDENCIES_EPOCH_NUMBER in the Dockerfile.ci and Dockerfile
-############################################################################################################
-
 devel_minreq = cgroups + devel + doc + kubernetes + mysql + password
 devel_hadoop = devel_minreq + hdfs + hive + kerberos + presto + webhdfs
 
-############################################################################################################
-# IMPORTANT NOTE!!!!!!!!!!!!!!!
-# If you have a 'pip check' problem with dependencies, it might be because some dependency has been
-# installed via 'install_requires' in setup.cfg in higher version than required in one of the options below.
-# For example pip check was failing with requests=2.25.1 installed even if in some dependencies below
-# < 2.24.0 was specified for it. Solution in such case is to add such limiting requirement to
-# install_requires in setup.cfg (we've added requests<2.24.0 there to limit requests library).
-# This should be done with appropriate comment explaining why the requirement was added.
-############################################################################################################
-
-
 # Dict of all providers which are part of the Apache Airflow repository together with their requirements
 PROVIDERS_REQUIREMENTS: Dict[str, List[str]] = {
     'amazon': amazon,
@@ -641,7 +615,6 @@ def add_extras_for_all_providers() -> None:
 
 add_extras_for_all_providers()
 
-#############################################################################################################
 #############################################################################################################
 #  The whole section can be removed in Airflow 3.0 as those old aliases are deprecated in 2.* series
 #############################################################################################################
@@ -750,10 +723,9 @@ def add_extras_for_all_deprecated_aliases() -> None:
 
 # Those packages are excluded because they break tests (downgrading mock and few other requirements)
 # and they are not needed to run our test suite. This can be removed as soon as we get non-conflicting
-# requirements for the apache-beam as well. This waits for azure + snowflake fixes:
+# requirements for the apache-beam as well. This waits for azure fixes:
 #
 # * Azure: https://github.com/apache/airflow/issues/11968
-# * Snowflake: https://github.com/apache/airflow/issues/12881
 #
 PACKAGES_EXCLUDED_FOR_CI = [
     'apache-beam',


Test patch:
diff --git a/scripts/in_container/run_ci_tests.sh b/scripts/in_container/run_ci_tests.sh
--- a/scripts/in_container/run_ci_tests.sh
+++ b/scripts/in_container/run_ci_tests.sh
@@ -18,6 +18,8 @@
 # shellcheck source=scripts/in_container/_in_container_script_init.sh
 . "$( dirname "${BASH_SOURCE[0]}" )/_in_container_script_init.sh"
 
+reinstall_azure_storage_blob
+
 echo
 echo "Starting the tests with those pytest arguments:" "${@}"
 echo
diff --git a/scripts/in_container/run_install_and_test_provider_packages.sh b/scripts/in_container/run_install_and_test_provider_packages.sh
--- a/scripts/in_container/run_install_and_test_provider_packages.sh
+++ b/scripts/in_container/run_install_and_test_provider_packages.sh
@@ -75,12 +75,6 @@ function install_airflow_as_specified() {
     group_end
 }
 
-function install_deps() {
-    group_start "Installs all remaining dependencies that are not installed by '${AIRFLOW_EXTRAS}' "
-    install_remaining_dependencies
-    group_end
-}
-
 function install_provider_packages() {
     group_start "Install provider packages"
     if [[ ${PACKAGE_FORMAT} == "wheel" ]]; then
@@ -202,7 +196,8 @@ function discover_all_field_behaviours() {
 setup_provider_packages
 verify_parameters
 install_airflow_as_specified
-install_deps
+install_remaining_dependencies
+reinstall_azure_storage_blob
 install_provider_packages
 import_all_provider_classes
 
diff --git a/tests/providers/presto/hooks/test_presto.py b/tests/providers/presto/hooks/test_presto.py
--- a/tests/providers/presto/hooks/test_presto.py
+++ b/tests/providers/presto/hooks/test_presto.py
@@ -217,23 +217,6 @@ def test_should_record_records(self):
         records = hook.get_records(sql)
         self.assertEqual([['Customer#000000001'], ['Customer#000000002'], ['Customer#000000003']], records)
 
-    @pytest.mark.xfail(
-        condition=True,
-        reason="""
-This test will fail when full suite of tests are run, because of Snowflake monkeypatching urllib3
-library as described in https://github.com/snowflakedb/snowflake-connector-python/issues/324
-the offending code is here:
-https://github.com/snowflakedb/snowflake-connector-python/blob/133d6215f7920d304c5f2d466bae38127c1b836d/src/snowflake/connector/network.py#L89-L92
-
-This test however runs fine when run in total isolation. We could move it to Heisentests, but then we would
-have to enable integrations there and make sure no snowflake gets imported.
-
-In the future Snowflake plans to get rid of the MonkeyPatching.
-
-Issue to keep track of it: https://github.com/apache/airflow/issues/12881
-
-""",
-    )
     @pytest.mark.integration("presto")
     @pytest.mark.integration("kerberos")
     def test_should_record_records_with_kerberos_auth(self):