Repository: apache/airflow

Issue:
Custom XCom backends circular import when using cli command like airflow connections list
<!--

Welcome to Apache Airflow!  For a smooth issue process, try to answer the following questions.
Don't worry if they're not all applicable; just try to include what you can :-)

If you need to include code snippets or logs, please put them in fenced code
blocks.  If they're super-long, please use the details tag like
<details><summary>super-long log</summary> lots of stuff </details>

Please delete these comment blocks before submitting the issue.

-->

<!--

IMPORTANT!!!

PLEASE CHECK "SIMILAR TO X EXISTING ISSUES" OPTION IF VISIBLE
NEXT TO "SUBMIT NEW ISSUE" BUTTON!!!

PLEASE CHECK IF THIS ISSUE HAS BEEN REPORTED PREVIOUSLY USING SEARCH!!!

Please complete the next sections or the issue will be closed.
These questions are the first thing we need to know to understand the context.

-->

**Apache Airflow version**: 2.0.1


**Kubernetes version (if you are using kubernetes)** (use `kubectl version`):

**Environment**:

- **Cloud provider or hardware configuration**:
- **OS** (e.g. from /etc/os-release):
- **Kernel** (e.g. `uname -a`):
- **Install tools**:
- **Others**:

**What happened**:

While running `airflow connections list` with a `xcom_backend = xcom.MyXComBackend` in `airflow.cfg` I will get a 
 
    ImportError: cannot import name 'BaseHook' from partially initialized module 'airflow.hooks.base' (most likely due to a circular import) (/Users/rubelagu/tmp/airflow-xcom-backend/venv/lib/python3.8/site-packages/airflow/hooks/base.py)

(see below for complete traceback).


The issue seems to be that `airflow/cli/commands/connection_command.py` imports `BaseHook` directly and `BaseHook -> Connection -> BaseOperator -> TaskInstance -> XCom -> MyXComBackend -> S3Hook -> BaseHook ` (again see complete traceback below)
    
```
airflow connections list
[2021-04-12 10:51:34,020] {configuration.py:459} ERROR - cannot import name 'BaseHook' from partially initialized module 'airflow.hooks.base' (most likely due to a circular import) (/Users/rubelagu/tmp/airflow-xcom-backend/venv/lib/python3.8/site-packages/airflow/hooks/base.py)
Traceback (most recent call last):
  File "/Users/rubelagu/tmp/airflow-xcom-backend/venv/lib/python3.8/site-packages/airflow/configuration.py", line 457, in getimport
    return import_string(full_qualified_path)
  File "/Users/rubelagu/tmp/airflow-xcom-backend/venv/lib/python3.8/site-packages/airflow/utils/module_loading.py", line 32, in import_string
    module = import_module(module_path)
  File "/Users/rubelagu/.pyenv/versions/3.8.8/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/Users/rubelagu/tmp/airflow-xcom-backend/plugins/xcom.py", line 4, in <module>
    from airflow.providers.amazon.aws.hooks.s3 import S3Hook
  File "/Users/rubelagu/tmp/airflow-xcom-backend/venv/lib/python3.8/site-packages/airflow/providers/amazon/aws/hooks/s3.py", line 37, in <module>
    from airflow.providers.amazon.aws.hooks.base_aws import AwsBaseHook
  File "/Users/rubelagu/tmp/airflow-xcom-backend/venv/lib/python3.8/site-packages/airflow/providers/amazon/aws/hooks/base_aws.py", line 41, in <module>
    from airflow.hooks.base import BaseHook
ImportError: cannot import name 'BaseHook' from partially initialized module 'airflow.hooks.base' (most likely due to a circular import) (/Users/rubelagu/tmp/airflow-xcom-backend/venv/lib/python3.8/site-packages/airflow/hooks/base.py)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/rubelagu/tmp/airflow-xcom-backend/venv/bin/airflow", line 8, in <module>
    sys.exit(main())
  File "/Users/rubelagu/tmp/airflow-xcom-backend/venv/lib/python3.8/site-packages/airflow/__main__.py", line 40, in main
    args.func(args)
  File "/Users/rubelagu/tmp/airflow-xcom-backend/venv/lib/python3.8/site-packages/airflow/cli/cli_parser.py", line 47, in command
    func = import_string(import_path)
  File "/Users/rubelagu/tmp/airflow-xcom-backend/venv/lib/python3.8/site-packages/airflow/utils/module_loading.py", line 32, in import_string
    module = import_module(module_path)
  File "/Users/rubelagu/.pyenv/versions/3.8.8/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/Users/rubelagu/tmp/airflow-xcom-backend/venv/lib/python3.8/site-packages/airflow/cli/commands/connection_command.py", line 30, in <module>
    from airflow.hooks.base import BaseHook
  File "/Users/rubelagu/tmp/airflow-xcom-backend/venv/lib/python3.8/site-packages/airflow/hooks/base.py", line 23, in <module>
    from airflow.models.connection import Connection
  File "/Users/rubelagu/tmp/airflow-xcom-backend/venv/lib/python3.8/site-packages/airflow/models/__init__.py", line 20, in <module>
    from airflow.models.baseoperator import BaseOperator, BaseOperatorLink
  File "/Users/rubelagu/tmp/airflow-xcom-backend/venv/lib/python3.8/site-packages/airflow/models/baseoperator.py", line 55, in <module>
    from airflow.models.taskinstance import Context, TaskInstance, clear_task_instances
  File "/Users/rubelagu/tmp/airflow-xcom-backend/venv/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 58, in <module>
    from airflow.models.xcom import XCOM_RETURN_KEY, XCom
  File "/Users/rubelagu/tmp/airflow-xcom-backend/venv/lib/python3.8/site-packages/airflow/models/xcom.py", line 289, in <module>
    XCom = resolve_xcom_backend()
  File "/Users/rubelagu/tmp/airflow-xcom-backend/venv/lib/python3.8/site-packages/airflow/models/xcom.py", line 279, in resolve_xcom_backend
    clazz = conf.getimport("core", "xcom_backend", fallback=f"airflow.models.xcom.{BaseXCom.__name__}")
  File "/Users/rubelagu/tmp/airflow-xcom-backend/venv/lib/python3.8/site-packages/airflow/configuration.py", line 460, in getimport
    raise AirflowConfigException(
airflow.exceptions.AirflowConfigException: The object could not be loaded. Please check "xcom_backend" key in "core" section. Current value: "xcom.MyXComBackend".
```

**What you expected to happen**:

I expected to be able to use S3Hook or GCSHook from a cusom XCom backedn following the example in what I though was the canonical example at https://medium.com/apache-airflow/airflow-2-0-dag-authoring-redesigned-651edc397178 



**How to reproduce it**:

* In `airflow.cfg` set `xcom_backend = xcom.MyXComBackend`
* Create file `plugins/xcom.py` with the following contents
```
from airflow.models.xcom import BaseXCom
from airflow.providers.amazon.aws.hooks.s3 import S3Hook

class MyXComBackend(BaseXCom):
    pass
```


**Anything else we need to know**:

This can be workaround by not importing the hooks on the module level but at the method level, but that is
* really ugly
* not evident for anybody trying to create a custom xcom backend for the first time.


```
from airflow.models.xcom import BaseXCom
class MyXComBackend(BaseXCom):


    @staticmethod
    def serialize_value(value):
        from airflow.providers.amazon.aws.hooks.s3 import S3Hook
        hook = S3Hook()
        pass
    @staticmethod
    def deserialize_value(value):
        from airflow.providers.amazon.aws.hooks.s3 import S3Hook
        hook = S3Hook()
        pass
```




Discussion:
The “ultimate” fix IMO would be to make the XCom class a proxy that’s lazily loaded instead (similar to [how Django loads the user class](https://docs.djangoproject.com/en/3.2/topics/auth/customizing/#django.contrib.auth.get_user_model)). But for a simple workaround, `airflow.hooks.base` only reference `Connection` once in `BaseHook.get_connection()`, so that’s a good place to break the cycle.
If airflow was Python 3.7+ instead of Python 3.6+ we could change `airflow/models/xcom.py` so that instead of resolving the XCom backend right away it would be delayed until first use. 

So instead of: 
```
XCom = resolve_xcom_backend()
``` 

```
def __getattr__(name):
  if name == "XCom":
    xcom = resolve_xcom_backend()
    sys.module[__name__].XCom = xcom
    return xcom
  raise AttributeError(f"module {__name__} has no attribute {name}")
```

I'm going to try to use  
```
XCom = lazy_object_proxy.Proxy(resolve_xcom_backend)
```
if it works I'll create a PR

Well, it turns out it does not work `airflow.models.xcom.XCom` is used in `datamodel = AirflowModelView.CustomSQLAInterface(XCom)` for example that breaks if XCom is a Proxy object. 
I was thinking more about proxying the class itself, something like

```python
class _XComMeta(type):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self._impl = resolve_xcom_backend()

    def __getattr__(self, name):
        return getattr(self._impl, name)


class XCom(BaseXCom, metaclass=_XComMeta):
    def __new__(cls, *args, **kwargs):
        return cls._impl(*args, **kwargs)
```

There are other things that need consideration such as `__issubclass__` and `__isinstance__`, but it should be possible with the right tweaks to the code base. Whether that’s too much work, however…
Uh, so a straight-up proxy class doesn’t work since `XCom` is also a SQLAlchemy model, and a more significant is needed. The strategy can still be simple (Django’s example still applies; they are also dealing with an ORM class there), it’s the refactoring (and potential backward compatibility issues) needed behind it that’s significant. That’s *probably* still worthwhile in the long run, but will need more discussion.

In the meantime, I’ll submit the “easy” fix to untangle the circular import.

Resolving patch:
diff --git a/airflow/hooks/base.py b/airflow/hooks/base.py
--- a/airflow/hooks/base.py
+++ b/airflow/hooks/base.py
@@ -18,12 +18,14 @@
 """Base class for all hooks"""
 import logging
 import warnings
-from typing import Any, Dict, List
+from typing import TYPE_CHECKING, Any, Dict, List
 
-from airflow.models.connection import Connection
 from airflow.typing_compat import Protocol
 from airflow.utils.log.logging_mixin import LoggingMixin
 
+if TYPE_CHECKING:
+    from airflow.models.connection import Connection  # Avoid circular imports.
+
 log = logging.getLogger(__name__)
 
 
@@ -37,7 +39,7 @@ class BaseHook(LoggingMixin):
     """
 
     @classmethod
-    def get_connections(cls, conn_id: str) -> List[Connection]:
+    def get_connections(cls, conn_id: str) -> List["Connection"]:
         """
         Get all connections as an iterable, given the connection id.
 
@@ -53,13 +55,15 @@ def get_connections(cls, conn_id: str) -> List[Connection]:
         return [cls.get_connection(conn_id)]
 
     @classmethod
-    def get_connection(cls, conn_id: str) -> Connection:
+    def get_connection(cls, conn_id: str) -> "Connection":
         """
         Get connection, given connection id.
 
         :param conn_id: connection id
         :return: connection
         """
+        from airflow.models.connection import Connection
+
         conn = Connection.get_connection_from_secrets(conn_id)
         if conn.host:
             log.info(


Test patch: