Repository: Parsl/parsl

Issue:
parsl[monitoring] pins sqlalchemy to <1.4
**Describe the bug**
Parsl[monitoring] currently requires sqlalchemy <1.4, making it difficult to integrate with other packages that depend on newer versions.

This issue is related to #1994, solved in fd8072f7, where it was found that sqlalchemy-utils 0.36.8 was incompatible with squalchemy 1.4.0.

However the newest sqlalchemy-utils 0.38.2 does not seem to break with sqlalchemy 1.4.37. Indeed,

`pytest parsl/tests/test_monitoring/ --config local`

succeeds with 4/4 passed.

**To Reproduce**
n/a

**Expected behavior**
Parsl[monitoring] should depend on the newest version that does not explicitly break functionality

**Environment**
- OS: `Linux 5.17.9-arch1-1 #1 SMP PREEMPT Wed, 18 May 2022 17:30:11 +0000 x86_64 GNU/Linux`
- Python 3.10.4
 - Parsl 1.2.0

**Distributed Environment**
- Where are you running the Parsl script from?
 Currently testing on a local machine. Will deploy on a cluster as part of a larger service whose web interface could benefit from newer sqlalchemy

- Where do you need the workers to run ?
 Workers will run mostly on a slurm cluster, although the issue is relative to the master only, the only one to have the monitoring



Discussion:

Resolving patch:
diff --git a/parsl/monitoring/db_manager.py b/parsl/monitoring/db_manager.py
--- a/parsl/monitoring/db_manager.py
+++ b/parsl/monitoring/db_manager.py
@@ -21,6 +21,8 @@
 try:
     import sqlalchemy as sa
     from sqlalchemy import Column, Text, Float, Boolean, Integer, DateTime, PrimaryKeyConstraint, Table
+    from sqlalchemy.orm import Mapper
+    from sqlalchemy.orm import mapperlib
     from sqlalchemy.orm import sessionmaker
     from sqlalchemy.ext.declarative import declarative_base
 except ImportError:
@@ -28,12 +30,6 @@
 else:
     _sqlalchemy_enabled = True
 
-try:
-    from sqlalchemy_utils import get_mapper
-except ImportError:
-    _sqlalchemy_utils_enabled = False
-else:
-    _sqlalchemy_utils_enabled = True
 
 WORKFLOW = 'workflow'    # Workflow table includes workflow metadata
 TASK = 'task'            # Task table includes task metadata
@@ -50,11 +46,6 @@ class Database:
         raise OptionalModuleMissing(['sqlalchemy'],
                                     ("Default database logging requires the sqlalchemy library."
                                      " Enable monitoring support with: pip install 'parsl[monitoring]'"))
-    if not _sqlalchemy_utils_enabled:
-        raise OptionalModuleMissing(['sqlalchemy_utils'],
-                                    ("Default database logging requires the sqlalchemy_utils library."
-                                     " Enable monitoring support with: pip install 'parsl[monitoring]'"))
-
     Base = declarative_base()
 
     def __init__(self,
@@ -75,18 +66,39 @@ def __init__(self,
         Session = sessionmaker(bind=self.eng)
         self.session = Session()
 
+    def _get_mapper(self, table_obj: Table) -> Mapper:
+        if hasattr(mapperlib, '_all_registries'):
+            all_mappers = set()
+            for mapper_registry in mapperlib._all_registries():  # type: ignore
+                all_mappers.update(mapper_registry.mappers)
+        else:  # SQLAlchemy <1.4
+            all_mappers = mapperlib._mapper_registry  # type: ignore
+        mapper_gen = (
+            mapper for mapper in all_mappers
+            if table_obj in mapper.tables
+        )
+        try:
+            mapper = next(mapper_gen)
+            second_mapper = next(mapper_gen, False)
+        except StopIteration:
+            raise ValueError(f"Could not get mapper for table {table_obj}")
+
+        if second_mapper:
+            raise ValueError(f"Multiple mappers for table {table_obj}")
+        return mapper
+
     def update(self, *, table: str, columns: List[str], messages: List[Dict[str, Any]]) -> None:
         table_obj = self.meta.tables[table]
         mappings = self._generate_mappings(table_obj, columns=columns,
                                            messages=messages)
-        mapper = get_mapper(table_obj)
+        mapper = self._get_mapper(table_obj)
         self.session.bulk_update_mappings(mapper, mappings)
         self.session.commit()
 
     def insert(self, *, table: str, messages: List[Dict[str, Any]]) -> None:
         table_obj = self.meta.tables[table]
         mappings = self._generate_mappings(table_obj, messages=messages)
-        mapper = get_mapper(table_obj)
+        mapper = self._get_mapper(table_obj)
         self.session.bulk_insert_mappings(mapper, mappings)
         self.session.commit()
 
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -8,8 +8,7 @@
 
 extras_require = {
     'monitoring' : [
-        'sqlalchemy>=1.3.0,!=1.3.4,<1.4',
-        'sqlalchemy_utils',
+        'sqlalchemy>=1.3.0',
         'pydot',
         'networkx>=2.5,<2.6',
         'Flask>=1.0.2',


Test patch: