Repository: apache/airflow

Issue:
Version 2.3.3 breaks "Plugins as Python packages" feature
### Apache Airflow version

2.3.3 (latest released)

### What happened

In 2.3.3

If I use https://airflow.apache.org/docs/apache-airflow/stable/plugins.html#plugins-as-python-packages feature, then I see these Error:

short:
`ValueError: The name 'airs' is already registered for this blueprint. Use 'name=' to provide a unique name.`

long:
> i'm trying to reproduce it...

If I don't use it(workarounding by AIRFLOW__CORE__PLUGINS_FOLDER), errors doesn't occur.

It didn't happend in 2.3.2 and earlier 

### What you think should happen instead

Looks like plugins are import multiple times if it is plugins-as-python-packages.

Perhaps flask's major version change  is the main cause.
Presumably, in flask 1.0, duplicate registration of blueprint was quietly filtered out, but in 2.0 it seems to have been changed to generate an error. (I am trying to find out if this hypothesis is correct)

Anyway, use the latest version of FAB is important. we will have to adapt to this change, so plugins will have to be imported once regardless how it defined.

### How to reproduce

> It was reproduced in the environment used at work, but it is difficult to disclose or explain it.
> I'm working to reproduce it with the breeze command, and I open the issue first with the belief that it's not just me.

### Operating System

CentOS Linux release 7.9.2009 (Core)

### Versions of Apache Airflow Providers

```sh
$ SHIV_INTERPRETER=1 airsflow -m pip freeze | grep apache-
apache-airflow==2.3.3
apache-airflow-providers-apache-hive==3.1.0
apache-airflow-providers-apache-spark==2.1.0
apache-airflow-providers-celery==3.0.0
apache-airflow-providers-common-sql==1.0.0
apache-airflow-providers-ftp==3.1.0
apache-airflow-providers-http==3.0.0
apache-airflow-providers-imap==3.0.0
apache-airflow-providers-postgres==5.1.0
apache-airflow-providers-redis==3.0.0
apache-airflow-providers-sqlite==3.1.0
```

but I think these are irrelevant.

### Deployment

Other 3rd-party Helm chart

### Deployment details

docker image based on centos7, python 3.9.10 interpreter, self-written helm2 chart ....

... but I think these are irrelevant.

### Anything else

_No response_

### Are you willing to submit PR?

- [X] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)



Discussion:
Thanks for opening your first issue here! Be sure to follow the issue template!

Yeah. It would be great to get to the bottom of this. Happy to help with investigation if you have some more findings.
In FAB 2.X, registering the same blueprint more than once is prohibited. 

https://github.com/pallets/flask/blob/2.1.2/src/flask/blueprints.py#L293-L296

Problem is `entry_points_with_dist('airflow.plugins')` is returning the plugin twice.
https://github.com/apache/airflow/blob/2.3.3/airflow/plugins_manager.py#L224-L225

We need a way to prevents double registration for plugins..
The most likely cause is the plugin appears twice in `sys.path` and thus loaded multiple times by `importlib.metadata`. We can add some sort of registry that keeps track what distributions have been loaded (by distribution name) to avoid this.

Resolving patch:
diff --git a/airflow/plugins_manager.py b/airflow/plugins_manager.py
--- a/airflow/plugins_manager.py
+++ b/airflow/plugins_manager.py
@@ -113,7 +113,7 @@ class EntryPointSource(AirflowPluginSource):
     """Class used to define Plugins loaded from entrypoint."""
 
     def __init__(self, entrypoint: importlib_metadata.EntryPoint, dist: importlib_metadata.Distribution):
-        self.dist = dist.metadata['name']
+        self.dist = dist.metadata['Name']
         self.version = dist.version
         self.entrypoint = str(entrypoint)
 
diff --git a/airflow/utils/entry_points.py b/airflow/utils/entry_points.py
--- a/airflow/utils/entry_points.py
+++ b/airflow/utils/entry_points.py
@@ -15,15 +15,20 @@
 # specific language governing permissions and limitations
 # under the License.
 
+from __future__ import annotations
+
+from typing import Iterator
+
+from packaging.utils import canonicalize_name
+
 try:
-    import importlib_metadata
+    import importlib_metadata as metadata
 except ImportError:
-    from importlib import metadata as importlib_metadata  # type: ignore
+    from importlib import metadata  # type: ignore[no-redef]
 
 
-def entry_points_with_dist(group: str):
-    """
-    Return EntryPoint objects of the given group, along with the distribution information.
+def entry_points_with_dist(group: str) -> Iterator[tuple[metadata.EntryPoint, metadata.Distribution]]:
+    """Retrieve entry points of the given group.
 
     This is like the ``entry_points()`` function from importlib.metadata,
     except it also returns the distribution the entry_point was loaded from.
@@ -31,7 +36,12 @@ def entry_points_with_dist(group: str):
     :param group: Filter results to only this entrypoint group
     :return: Generator of (EntryPoint, Distribution) objects for the specified groups
     """
-    for dist in importlib_metadata.distributions():
+    loaded: set[str] = set()
+    for dist in metadata.distributions():
+        key = canonicalize_name(dist.metadata["Name"])
+        if key in loaded:
+            continue
+        loaded.add(key)
         for e in dist.entry_points:
             if e.group != group:
                 continue


Test patch:
diff --git a/tests/plugins/test_plugins_manager.py b/tests/plugins/test_plugins_manager.py
--- a/tests/plugins/test_plugins_manager.py
+++ b/tests/plugins/test_plugins_manager.py
@@ -298,6 +298,7 @@ def test_entrypoint_plugin_errors_dont_raise_exceptions(self, caplog):
         from airflow.plugins_manager import import_errors, load_entrypoint_plugins
 
         mock_dist = mock.Mock()
+        mock_dist.metadata = {"Name": "test-dist"}
 
         mock_entrypoint = mock.Mock()
         mock_entrypoint.name = 'test-entrypoint'
@@ -387,7 +388,7 @@ def test_should_return_correct_source_details(self):
         mock_entrypoint.module = 'module_name_plugin'
 
         mock_dist = mock.Mock()
-        mock_dist.metadata = {'name': 'test-entrypoint-plugin'}
+        mock_dist.metadata = {'Name': 'test-entrypoint-plugin'}
         mock_dist.version = '1.0.0'
         mock_dist.entry_points = [mock_entrypoint]
 
diff --git a/tests/utils/test_entry_points.py b/tests/utils/test_entry_points.py
new file mode 100644
--- /dev/null
+++ b/tests/utils/test_entry_points.py
@@ -0,0 +1,49 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+from typing import Iterable
+from unittest import mock
+
+from airflow.utils.entry_points import entry_points_with_dist, metadata
+
+
+class MockDistribution:
+    def __init__(self, name: str, entry_points: Iterable[metadata.EntryPoint]) -> None:
+        self.metadata = {"Name": name}
+        self.entry_points = entry_points
+
+
+class MockMetadata:
+    def distributions(self):
+        return [
+            MockDistribution(
+                "dist1",
+                [metadata.EntryPoint("a", "b", "group_x"), metadata.EntryPoint("c", "d", "group_y")],
+            ),
+            MockDistribution("Dist2", [metadata.EntryPoint("e", "f", "group_x")]),
+            MockDistribution("dist2", [metadata.EntryPoint("g", "h", "group_x")]),  # Duplicated name.
+        ]
+
+
+@mock.patch("airflow.utils.entry_points.metadata", MockMetadata())
+def test_entry_points_with_dist():
+    entries = list(entry_points_with_dist("group_x"))
+
+    # The second "dist2" is ignored. Only "group_x" entries are loaded.
+    assert [dist.metadata["Name"] for _, dist in entries] == ["dist1", "Dist2"]
+    assert [ep.name for ep, _ in entries] == ["a", "e"]
diff --git a/tests/www/views/test_views.py b/tests/www/views/test_views.py
--- a/tests/www/views/test_views.py
+++ b/tests/www/views/test_views.py
@@ -88,7 +88,7 @@ def test_plugin_should_list_entrypoint_on_page_with_details(admin_client):
     mock_plugin = AirflowPlugin()
     mock_plugin.name = "test_plugin"
     mock_plugin.source = EntryPointSource(
-        mock.Mock(), mock.Mock(version='1.0.0', metadata={'name': 'test-entrypoint-testpluginview'})
+        mock.Mock(), mock.Mock(version='1.0.0', metadata={'Name': 'test-entrypoint-testpluginview'})
     )
     with mock_plugin_manager(plugins=[mock_plugin]):
         resp = admin_client.get('/plugin')