Repository: BSC-ES/autosubmit
Pull request #2107: Add DbManager factory function
Author: login: LuiggiTenorioK
Created: 2025-02-05T16:25:19Z | Merged: 2025-03-25T15:03:42Z

Related to #1285 and PR #1914 

The idea of this PR is to include the code organization changes related to #1914. In this case:

* Adding the DbManager factory function
* Update JobListPersistenceDb and JobPackagePersistence `__init__` arguments
* Fix tests to mock the classes correctly changed above

These changes will make it easier to implement and review the changes needed to enable the support to Postgres.


Diff:
diff --git a/autosubmit/autosubmit.py b/autosubmit/autosubmit.py
index 58684ad4f..a259aa590 100644
--- a/autosubmit/autosubmit.py
+++ b/autosubmit/autosubmit.py
@@ -1538,8 +1538,7 @@ def inspect(expid, lst, filter_chunks, filter_status, filter_section, notransiti
             safetysleeptime = as_conf.get_safetysleeptime()
             Log.debug("The Experiment name is: {0}", expid)
             Log.debug("Sleep: {0}", safetysleeptime)
-            packages_persistence = JobPackagePersistence(os.path.join(BasicConfig.LOCAL_ROOT_DIR, expid, "pkl"),
-                                                         "job_packages_" + expid)
+            packages_persistence = JobPackagePersistence(expid)
             os.chmod(os.path.join(BasicConfig.LOCAL_ROOT_DIR, expid,
                                   "pkl", "job_packages_" + expid + ".db"), 0o644)
 
@@ -2033,8 +2032,7 @@ def prepare_run(expid, notransitive=False, start_time=None, start_after=None,
         Log.debug("Loading job packages")
         # Packages == wrappers and jobs inside wrappers. Name is also misleading.
         try:
-            packages_persistence = JobPackagePersistence(os.path.join(
-                BasicConfig.LOCAL_ROOT_DIR, expid, "pkl"), "job_packages_" + expid)
+            packages_persistence = JobPackagePersistence(expid)
         except IOError as e:
             raise AutosubmitError(
                 "job_packages not found", 6016, str(e))
@@ -2708,8 +2706,7 @@ def monitor(expid, file_format, lst, filter_chunks, filter_status, filter_sectio
         try:
             if len(as_conf.experiment_data.get("WRAPPERS", {})) > 0 and check_wrapper:
                 # Class constructor creates table if it does not exist
-                packages_persistence = JobPackagePersistence(os.path.join(BasicConfig.LOCAL_ROOT_DIR, expid, "pkl"),
-                                                             "job_packages_" + expid)
+                packages_persistence = JobPackagePersistence(expid)
                 # Permissions
                 os.chmod(os.path.join(BasicConfig.LOCAL_ROOT_DIR, expid, "pkl", "job_packages_" + expid + ".db"), 0o644)
                 # Database modification
@@ -2721,11 +2718,9 @@ def monitor(expid, file_format, lst, filter_chunks, filter_status, filter_sectio
                                                            packages_persistence, True)
 
                 packages = packages_persistence.load(True)
-                packages += JobPackagePersistence(os.path.join(BasicConfig.LOCAL_ROOT_DIR, expid, "pkl"),
-                                                  "job_packages_" + expid).load()
+                packages += JobPackagePersistence(expid).load()
             else:
-                packages = JobPackagePersistence(os.path.join(BasicConfig.LOCAL_ROOT_DIR, expid, "pkl"),
-                                                 "job_packages_" + expid).load()
+                packages = JobPackagePersistence(expid).load()
         except BaseException as e:
             if profile:
                 profiler.stop()
@@ -3039,8 +3034,7 @@ def recovery(expid, noplot, save, all_jobs, hide, group_by=None, expand=list(),
             raise AutosubmitCritical("Couldn't restore the experiment workflow", 7040, str(e))
 
         try:
-            packages = JobPackagePersistence(os.path.join(BasicConfig.LOCAL_ROOT_DIR, expid, "pkl"),
-                                             "job_packages_" + expid).load()
+            packages = JobPackagePersistence(expid).load()
 
             groups_dict = dict()
             if group_by:
@@ -4560,8 +4554,7 @@ def create(expid, noplot, hide, output='pdf', group_by=None, expand=list(), expa
                     job_list.save()
                     as_conf.save()
                     try:
-                        packages_persistence = JobPackagePersistence(
-                            os.path.join(BasicConfig.LOCAL_ROOT_DIR, expid, "pkl"), "job_packages_" + expid)
+                        packages_persistence = JobPackagePersistence(expid)
                         packages_persistence.reset_table()
                         packages_persistence.reset_table(True)
                     except Exception:
@@ -5377,9 +5370,7 @@ def set_status(expid, noplot, save, final, filter_list, filter_chunks, filter_st
                 if not noplot:
                     from .monitor.monitor import Monitor
                     if as_conf.get_wrapper_type() != 'none' and check_wrapper:
-                        packages_persistence = JobPackagePersistence(
-                            os.path.join(BasicConfig.LOCAL_ROOT_DIR, expid, "pkl"),
-                            "job_packages_" + expid)
+                        packages_persistence = JobPackagePersistence(expid)
                         os.chmod(os.path.join(BasicConfig.LOCAL_ROOT_DIR,
                                               expid, "pkl", "job_packages_" + expid + ".db"), 0o775)
                         packages_persistence.reset_table(True)
@@ -5391,8 +5382,7 @@ def set_status(expid, noplot, save, final, filter_list, filter_chunks, filter_st
 
                         packages = packages_persistence.load(True)
                     else:
-                        packages = JobPackagePersistence(os.path.join(BasicConfig.LOCAL_ROOT_DIR, expid, "pkl"),
-                                                         "job_packages_" + expid).load()
+                        packages = JobPackagePersistence(expid).load()
                     groups_dict = dict()
                     if group_by:
                         status = list()
@@ -5550,8 +5540,7 @@ def _get_job_list_persistence(expid, as_conf):
         if storage_type == 'pkl':
             return JobListPersistencePkl()
         elif storage_type == 'db':
-            return JobListPersistenceDb(os.path.join(BasicConfig.LOCAL_ROOT_DIR, expid, "pkl"),
-                                        "job_list_" + expid)
+            return JobListPersistenceDb(expid)
         raise AutosubmitCritical('Storage type not known', 7014)
 
     @staticmethod
diff --git a/autosubmit/database/db_manager.py b/autosubmit/database/db_manager.py
index 89e62fb65..c55ca3918 100644
--- a/autosubmit/database/db_manager.py
+++ b/autosubmit/database/db_manager.py
@@ -19,14 +19,14 @@
 
 import sqlite3
 import os
-
+from typing import Any, Dict, Iterable, List, Protocol, Union, cast
 
 class DbManager(object):
     """
     Class to manage an SQLite database.
     """
 
-    def __init__(self, root_path, db_name, db_version):
+    def __init__(self, root_path: str, db_name: str, db_version: int):
         self.root_path = root_path
         self.db_name = db_name
         self.db_version = db_version
@@ -94,6 +94,17 @@ def insertMany(self, table_name, data):
         cursor.executemany(insert_many_command, data)
         self.connection.commit()
 
+    def delete_where(self, table_name: str, where: list[str]):
+        """
+        Deletes the rows of the given table that matches the given where conditions
+        :param table_name: str
+        :param where: [str]
+        """
+        cursor = self.connection.cursor()
+        delete_command = self.generate_delete_command(table_name, where[:])
+        cursor.execute(delete_command)
+        self.connection.commit()
+
     def select_first(self, table_name):
         """
         Returns the first row of the given table
@@ -142,7 +153,7 @@ def drop(self):
         if os.path.exists(self._get_db_filepath()):
             os.remove(self._get_db_filepath())
 
-    def _get_db_filepath(self):
+    def _get_db_filepath(self) -> str:
         """
         Returns the path of the .db file
         :return path: int
@@ -162,7 +173,7 @@ def _initialize_database(self):
         self.insert(options_table_name, columns, ['name', self.db_name])
         self.insert(options_table_name, columns, ['version', self.db_version])
 
-    def _select_with_all_fields(self, table_name, where=[]):
+    def _select_with_all_fields(self, table_name: str, where: List[str] = []) -> sqlite3.Cursor:
         """
         Returns the cursor of the select command with the given parameters
         :param table_name: str
@@ -179,7 +190,7 @@ def _select_with_all_fields(self, table_name, where=[]):
     """
 
     @staticmethod
-    def generate_create_table_command(table_name, fields):
+    def generate_create_table_command(table_name: str, fields: List[str]) -> str:
         create_command = 'CREATE TABLE IF NOT EXISTS ' + table_name + ' (' + fields.pop(0)
         for field in fields:
             create_command += (', ' + field)
@@ -187,12 +198,14 @@ def generate_create_table_command(table_name, fields):
         return create_command
 
     @staticmethod
-    def generate_drop_table_command(table_name):
+    def generate_drop_table_command(table_name: str) -> str:
         drop_command = 'DROP TABLE IF EXISTS ' + table_name
         return drop_command
 
     @staticmethod
-    def generate_insert_command(table_name, columns, values):
+    def generate_insert_command(
+        table_name: str, columns: List[str], values: List[str]
+    ) -> str:
         insert_command = 'INSERT INTO ' + table_name + '(' + columns.pop(0)
         for column in columns:
             insert_command += (', ' + column)
@@ -203,7 +216,7 @@ def generate_insert_command(table_name, columns, values):
         return insert_command
 
     @staticmethod
-    def generate_insert_many_command(table_name, num_of_values):
+    def generate_insert_many_command(table_name: str, num_of_values: int) -> str:
         insert_command = 'INSERT INTO ' + table_name + ' VALUES (?'
         num_of_values -= 1
         while num_of_values > 0:
@@ -213,14 +226,78 @@ def generate_insert_many_command(table_name, num_of_values):
         return insert_command
 
     @staticmethod
-    def generate_count_command(table_name):
+    def generate_count_command(table_name: str) -> str:
         count_command = 'SELECT count(*) FROM ' + table_name
         return count_command
 
     @staticmethod
-    def generate_select_command(table_name, where=[]):
+    def generate_select_command(table_name: str, where: List[str] = []) -> str:
         basic_select = 'SELECT * FROM ' + table_name
         select_command = basic_select if len(where) == 0 else basic_select + ' WHERE ' + where.pop(0)
         for condition in where:
             select_command += ' AND ' + condition
         return select_command
+    
+    @staticmethod
+    def generate_delete_command(table_name: str, where: List[str] = []) -> str:
+        delete_command = "DELETE FROM " + table_name + " WHERE " + where.pop(0)
+        for condition in where:
+            delete_command += " AND " + condition
+        return delete_command
+
+class DatabaseManager(Protocol):
+    """Common interface for database managers.
+    We used a protocol here to avoid having to modify the existing
+    SQLite code (as we would if we used an abstract/ABC class).
+    And the new database manager will "quack" like the other one does.
+    """
+
+    connection: Union[sqlite3.Connection]
+
+    def backup(self): ...
+    def restore(self): ...
+    def disconnect(self): ...
+    def create_table(self, table_name: str, fields: List[str]): ...
+    def drop_table(self, table_name: str): ...
+    def insert(self, table_name: str, columns: List[str], values: List[str]): ...
+    def insertMany(self, table_name: str, data: List[Union[Iterable, Dict]]): ...
+    def delete_where(self, table_name: str, where: List[str]): ...
+    def select_first(self, table_name: str) -> List[Any]: ...
+    def select_first_where(self, table_name: str, where: List[str]) -> List[Any]: ...
+    def select_all(self, table_name: str) -> List[List[Any]]: ...
+    def select_all_where(self, table_name: str, where: List[str]) -> List[List[Any]]: ...
+    def count(self, table_name: str) -> int: ...
+    def drop(self): ...
+
+
+def create_db_manager(db_engine: str, **options) -> DatabaseManager:
+    """
+    Creates a Postgres or SQLite database manager based on the Autosubmit configuration.
+    Note that you must provide the options even if they are optional, in which case
+    you must provide ``options=None``, or you will get a ``KeyError``.
+    Later we might be able to drop the SQLite database manager. So, for the moment,
+    please call the function providing the database engine type, and the arguments
+    for both SQLite and for SQLAlchemy.
+    This means you do not have to do an ``if/else`` in your code, just give
+    this function the engine type, and all the valid options, and it should
+    handle choosing and building the database manager for you. e.g.
+    ```python
+    from autosubmit.database.db_manager import create_db_manager
+    options = {
+        # these are for sqlite
+        'root_path': '/tmp/',
+        'db_name': 'name.db',
+        'db_version': 1,
+        # and these for sqlalchemy -- not very elegant, but this is
+        # to work-effectively-with-legacy-code (as in that famous book).
+        'schema': 'a001'
+    }
+    db_manager = create_db_manager(db_engine='postgres', **options)
+    ```
+    :param db_engine: The database engine type.
+    :return: A ``DatabaseManager``.
+    :raises ValueError: If the database engine type is not valid.
+    :raises KeyError: If the ``options`` dictionary is missing a required parameter for an engine.
+    """
+    # TODO Create SqlAlchemyDbManager and add it to the if/else
+    return cast(DatabaseManager, DbManager(options['root_path'], options['db_name'], options['db_version']))
\ No newline at end of file
diff --git a/autosubmit/job/job_list.py b/autosubmit/job/job_list.py
index 5f24a1ae7..42e9b52a3 100644
--- a/autosubmit/job/job_list.py
+++ b/autosubmit/job/job_list.py
@@ -3205,14 +3205,12 @@ def retrieve_packages(BasicConfig, expid, current_jobs=None):
         # monitor = Monitor()
         packages = None
         try:
-            packages = JobPackagePersistence(os.path.join(BasicConfig.LOCAL_ROOT_DIR, expid, "pkl"),
-                                             "job_packages_" + expid).load(wrapper=False)
+            packages = JobPackagePersistence(expid).load(wrapper=False)
         except Exception as ex:
             print("Wrapper table not found, trying packages.")
             packages = None
             try:
-                packages = JobPackagePersistence(os.path.join(BasicConfig.LOCAL_ROOT_DIR, expid,
-                                                "pkl"), "job_packages_" + expid).load(wrapper=True)
+                packages = JobPackagePersistence(expid).load(wrapper=True)
             except Exception as exp2:
                 packages = None
                 pass
diff --git a/autosubmit/job/job_list_persistence.py b/autosubmit/job/job_list_persistence.py
index 91f9bcd4a..737c3f0e5 100644
--- a/autosubmit/job/job_list_persistence.py
+++ b/autosubmit/job/job_list_persistence.py
@@ -20,9 +20,11 @@
 import pickle
 from sys import setrecursionlimit, getrecursionlimit
 import shutil
-from autosubmit.database.db_manager import DbManager
+from autosubmit.database.db_manager import create_db_manager
 from log.log import AutosubmitCritical, Log
 from contextlib import suppress
+from autosubmitconfigparser.config.basicconfig import BasicConfig
+from pathlib import Path
 
 
 class JobListPersistence(object):
@@ -140,8 +142,15 @@ class JobListPersistenceDb(JobListPersistence):
         "wrapper_type",
     ]
 
-    def __init__(self, persistence_path, persistence_file):
-        self.db_manager = DbManager(persistence_path, persistence_file, self.VERSION)
+    def __init__(self, expid: str):
+        options = {
+            "root_path": str(Path(BasicConfig.LOCAL_ROOT_DIR, expid, "pkl")),
+            "db_name": f"job_list_{expid}",
+            "db_version": self.VERSION,
+            "schema": expid
+        }
+        self.expid = expid
+        self.db_manager = create_db_manager(BasicConfig.DATABASE_BACKEND, **options)
 
     def load(self, persistence_path, persistence_file):
         """
diff --git a/autosubmit/job/job_package_persistence.py b/autosubmit/job/job_package_persistence.py
index 766147f77..7089048b0 100644
--- a/autosubmit/job/job_package_persistence.py
+++ b/autosubmit/job/job_package_persistence.py
@@ -17,7 +17,9 @@
 # You should have received a copy of the GNU General Public License
 # along with Autosubmit.  If not, see <http://www.gnu.org/licenses/>.
 
-from autosubmit.database.db_manager import DbManager
+from autosubmit.database.db_manager import create_db_manager
+from autosubmitconfigparser.config.basicconfig import BasicConfig
+from pathlib import Path
 from log.log import AutosubmitCritical
 from typing import Any, List
 
@@ -39,8 +41,14 @@ class JobPackagePersistence(object):
     WRAPPER_JOB_PACKAGES_TABLE = 'wrapper_job_package'
     TABLE_FIELDS = ['exp_id', 'package_name', 'job_name', 'wallclock' ]  # new field, needs a new autosubmit create
 
-    def __init__(self, persistence_path, persistence_file):
-        self.db_manager = DbManager(persistence_path, persistence_file, self.VERSION)
+    def __init__(self, expid: str):
+        options = {
+            'root_path': str(Path(BasicConfig.LOCAL_ROOT_DIR, expid, "pkl")),
+            'db_name': f"job_packages_{expid}",
+            'db_version': self.VERSION,
+            'schema': expid
+        }
+        self.db_manager = create_db_manager(BasicConfig.DATABASE_BACKEND, **options)
         self.db_manager.create_table(self.JOB_PACKAGES_TABLE, self.TABLE_FIELDS)
         self.db_manager.create_table(self.WRAPPER_JOB_PACKAGES_TABLE, self.TABLE_FIELDS)
 
diff --git a/autosubmit/job/job_utils.py b/autosubmit/job/job_utils.py
index 39fd47563..7ade0147a 100644
--- a/autosubmit/job/job_utils.py
+++ b/autosubmit/job/job_utils.py
@@ -264,8 +264,8 @@ def get_job_package_code(expid: str, job_name: str) -> int:
     try:
         basic_conf = BasicConfig()
         basic_conf.read()
-        packages_wrapper = JobPackagePersistence(os.path.join(basic_conf.LOCAL_ROOT_DIR, expid, "pkl"),"job_packages_" + expid).load(wrapper=True)
-        packages_wrapper_plus = JobPackagePersistence(os.path.join(basic_conf.LOCAL_ROOT_DIR, expid, "pkl"),"job_packages_" + expid).load(wrapper=False)
+        packages_wrapper = JobPackagePersistence(expid).load(wrapper=True)
+        packages_wrapper_plus = JobPackagePersistence(expid).load(wrapper=False)
         if packages_wrapper or packages_wrapper_plus:
             packages = packages_wrapper if len(packages_wrapper) > len(packages_wrapper_plus) else packages_wrapper_plus
             for exp, package_name, _job_name in packages:
diff --git a/test/unit/test_checkpoints.py b/test/unit/test_checkpoints.py
index 3b16b9dcf..b8c4f7ddc 100644
--- a/test/unit/test_checkpoints.py
+++ b/test/unit/test_checkpoints.py
@@ -17,6 +17,8 @@
 
 import shutil
 from random import randrange
+from pathlib import Path
+from unittest.mock import patch
 
 import pytest
 
@@ -34,8 +36,21 @@ def setup_job_list(autosubmit_config, tmpdir, mocker, prepare_basic_config):
     as_conf.experiment_data = dict()
     as_conf.experiment_data["JOBS"] = dict()
     as_conf.experiment_data["PLATFORMS"] = dict()
-    job_list = JobList(experiment_id, as_conf, YAMLParserFactory(),
-                       JobListPersistenceDb(tmpdir, 'db'))
+
+    prepare_basic_config.DB_DIR = tmpdir
+    prepare_basic_config.DB_FILE = Path(tmpdir, 'test-db.db')
+    prepare_basic_config.DB_PATH = prepare_basic_config.DB_FILE
+    prepare_basic_config.LOCAL_PROJ_DIR = tmpdir
+    prepare_basic_config.LOCAL_ROOT_DIR = tmpdir
+    prepare_basic_config.LOCAL_TMP_DIR = tmpdir
+
+    Path(prepare_basic_config.DB_FILE).touch()
+    Path(prepare_basic_config.LOCAL_ROOT_DIR, experiment_id, 'pkl').mkdir(parents=True, exist_ok=True)
+
+    with patch('autosubmit.job.job_list_persistence.BasicConfig', prepare_basic_config):
+        job_list = JobList(experiment_id, prepare_basic_config, YAMLParserFactory(),
+                            JobListPersistenceDb(experiment_id))
+
     dummy_serial_platform = mocker.MagicMock()
     dummy_serial_platform.name = 'serial'
     dummy_platform = mocker.MagicMock()
diff --git a/test/unit/test_dependencies.py b/test/unit/test_dependencies.py
index b2b4f057d..45743a8d1 100644
--- a/test/unit/test_dependencies.py
+++ b/test/unit/test_dependencies.py
@@ -1,7 +1,6 @@
-from datetime import datetime
-
 import mock
 import pytest
+from datetime import datetime
 from mock.mock import MagicMock
 from networkx import DiGraph  # type: ignore
 
@@ -20,6 +19,13 @@
 _DATE_LIST = ["20020201", "20020202", "20020203", "20020204", "20020205", "20020206", "20020207",
               "20020208", "20020209", "20020210"]
 
+_EXPID = 'a000'
+
+
+@pytest.fixture(autouse=True)
+def as_conf(autosubmit_config):
+    return autosubmit_config(_EXPID, experiment_data={})
+
 
 @pytest.fixture
 def relationships_dates():
@@ -124,7 +130,7 @@ def joblist(tmp_path):
     as_conf.experiment_data["JOBS"] = dict()
     as_conf.jobs_data = as_conf.experiment_data["JOBS"]
     as_conf.experiment_data["PLATFORMS"] = dict()
-    job_list_persistence = JobListPersistenceDb(str(tmp_path), 'db')
+    job_list_persistence = JobListPersistenceDb(_EXPID)
     joblist = JobList(experiment_id, as_conf, YAMLParserFactory(), job_list_persistence)
     joblist._date_list = _DATE_LIST
     joblist._member_list = _MEMBER_LIST
@@ -684,7 +690,7 @@ def test_add_special_conditions_chunks_to_once(mocker, joblist):
 
 def test_job_dict_get_jobs_filtered(mocker, joblist):
     # Test the get_jobs_filtered function
-    as_conf = mock.Mock()
+    as_conf = mocker.Mock()
     as_conf.experiment_data = dict()
     as_conf.experiment_data = {
         'CONFIG': {'AUTOSUBMIT_VERSION': '4.1.2', 'MAXWAITINGJOBS': 20, 'TOTALJOBS': 20, 'SAFETYSLEEPTIME': 10,
@@ -756,10 +762,7 @@ def test_job_dict_get_jobs_filtered(mocker, joblist):
     assert expected_output == result
 
 
-def test_normalize_auto_keyword(autosubmit_config, mocker):
-    as_conf = autosubmit_config('a000', experiment_data={
-
-    })
+def test_normalize_auto_keyword(as_conf, mocker):
     job_list = JobList(
         as_conf.expid,
         as_conf,
@@ -768,21 +771,21 @@ def test_normalize_auto_keyword(autosubmit_config, mocker):
     )
     dependency = Dependency("test")
 
-    job = Job("a000_20001010_fc1_2_1_test", 1, Status.READY, 1)
+    job = Job(f"{_EXPID}_20001010_fc1_2_1_test", 1, Status.READY, 1)
     job.running = "chunk"
     job.section = "test"
     job.date = "20001010"
     job.member = "fc1"
     job.splits = 5
 
-    job_minus = Job("a000_20001010_fc1_1_1_minus", 1, Status.READY, 1)
+    job_minus = Job(f"{_EXPID}_20001010_fc1_1_1_minus", 1, Status.READY, 1)
     job_minus.running = "chunk"
     job_minus.section = "minus"
     job_minus.date = "20001010"
     job_minus.member = "fc1"
     job_minus.splits = 40
 
-    job_plus = Job("a000_20001010_fc1_3_1_plus", 1, Status.READY, 1)
+    job_plus = Job(f"{_EXPID}_20001010_fc1_3_1_plus", 1, Status.READY, 1)
     job_plus.running = "chunk"
     job_plus.section = "plus"
     job_plus.date = "20001010"
diff --git a/test/unit/test_dic_jobs.py b/test/unit/test_dic_jobs.py
index e83625be1..cd3348358 100644
--- a/test/unit/test_dic_jobs.py
+++ b/test/unit/test_dic_jobs.py
@@ -16,9 +16,8 @@
 # along with Autosubmit.  If not, see <http://www.gnu.org/licenses/>.
 
 import math
-from datetime import datetime
-
 import pytest
+from datetime import datetime
 from mock import Mock
 
 from autosubmit.job.job import Job
diff --git a/test/unit/test_job_graph.py b/test/unit/test_job_graph.py
index 5b1aaf8b1..430edb251 100644
--- a/test/unit/test_job_graph.py
+++ b/test/unit/test_job_graph.py
@@ -47,7 +47,7 @@ def job_list(autosubmit_config, tmp_path):
         'JOBS': {},
         'PLATFORMS': {},
     })
-    job_list_persistence = JobListPersistenceDb(str(tmp_path), 'db')
+    job_list_persistence = JobListPersistenceDb(_EXPID)
     job_list = JobList(_EXPID, as_conf, YAMLParserFactory(), job_list_persistence)
     # Basic workflow with SETUP, INI, SIM, POST, CLEAN
     setup_job = _create_dummy_job('expid_SETUP', Status.READY)
diff --git a/test/unit/test_job_grouping.py b/test/unit/test_job_grouping.py
index 7a4b53971..e43da04f1 100644
--- a/test/unit/test_job_grouping.py
+++ b/test/unit/test_job_grouping.py
@@ -15,10 +15,9 @@
 # You should have received a copy of the GNU General Public License
 # along with Autosubmit.  If not, see <http://www.gnu.org/licenses/>.
 
-from random import randrange
-
 import pytest
 from bscearth.utils.date import parse_date, date2str
+from random import randrange
 
 from autosubmit.job.job import Job
 from autosubmit.job.job_common import Status
@@ -47,7 +46,7 @@ def job_list(autosubmit_config, tmp_path):
         'JOBS': {},
         'PLATFORMS': {}
     })
-    job_list_persistence = JobListPersistenceDb(str(tmp_path), 'db')
+    job_list_persistence = JobListPersistenceDb('a000')
     job_list = JobList(as_conf.expid, as_conf, YAMLParserFactory(), job_list_persistence)
 
     # Basic workflow with SETUP, INI, SIM, POST, CLEAN
diff --git a/test/unit/test_job_package.py b/test/unit/test_job_package.py
index 06ade9402..896fb0e03 100644
--- a/test/unit/test_job_package.py
+++ b/test/unit/test_job_package.py
@@ -106,7 +106,7 @@ def fn(options: dict) -> JobPackageVertical:
 @pytest.fixture
 def joblist(tmp_path, as_conf):
     job_list = JobList('a000', as_conf, YAMLParserFactory(),
-                       JobListPersistenceDb(str(tmp_path), 'db'))
+                       JobListPersistenceDb(str(tmp_path)))
     job_list._ordered_jobs_by_date_member["WRAPPERS"] = dict()
     return job_list
 
diff --git a/test/unit/test_packages_persistence.py b/test/unit/test_packages_persistence.py
index 70d98d827..5e3bf8294 100644
--- a/test/unit/test_packages_persistence.py
+++ b/test/unit/test_packages_persistence.py
@@ -30,7 +30,7 @@ def test_load(mocker):
     mocker.patch('autosubmit.database.db_manager.DbManager.select_all').return_value = [
         ['random-id"', 'vertical-wrapper', 'dummy-job', '02:00']]
     mocker.patch('sqlite3.connect').return_value = mocker.MagicMock()
-    job_package_persistence = JobPackagePersistence('dummy/path', 'dummy/file')
+    job_package_persistence = JobPackagePersistence('dummy/expid')
     assert job_package_persistence.load(wrapper=True) == [['random-id"', 'vertical-wrapper', 'dummy-job', '02:00']]
     mocker.patch('autosubmit.database.db_manager.DbManager.select_all').return_value = [
         ['random-id"', 'vertical-wrapper', 'dummy-job']]
diff --git a/test/unit/test_wrappers.py b/test/unit/test_wrappers.py
index e77ae7a03..85842524d 100644
--- a/test/unit/test_wrappers.py
+++ b/test/unit/test_wrappers.py
@@ -15,18 +15,18 @@
 # You should have received a copy of the GNU General Public License
 # along with Autosubmit.  If not, see <http://www.gnu.org/licenses/>.
 
+from collections import OrderedDict
+
 import copy
 import inspect
+import mock
+import pytest
 import shutil
 import tempfile
-from collections import OrderedDict
+from mock import MagicMock
 from pathlib import Path
 from random import randrange
 
-import mock
-import pytest
-from mock import MagicMock
-
 import log.log
 from autosubmit.job.job import Job
 from autosubmit.job.job_common import Status
@@ -195,8 +195,13 @@ def setup_method(self):
         self.as_conf.experiment_data["PLATFORMS"] = dict()
         self.as_conf.experiment_data["WRAPPERS"] = dict()
         self.temp_directory = tempfile.mkdtemp()
-        self.job_list = JobList(self.experiment_id, self.as_conf, YAMLParserFactory(),
-                                JobListPersistenceDb(self.temp_directory, 'db'))
+        # TODO: The ``MagicMock`` argument is replacing this old call:
+        #       ``JobListPersistenceDb(self.experiment_id)``. The reason is that we need the pytest
+        #       fixtures to mock the DB_PATH/DB_DIR/etc., but the ones we have are function-scoped.
+        #       Once this code gets ported to function-based, instead of class-based, we can use
+        #       those fixtures, removing that mock (we have plenty of other places testing already
+        #       ``JobList``, but less mocking is always better.
+        self.job_list = JobList(self.experiment_id, self.as_conf, YAMLParserFactory(), MagicMock())
 
         self.parser_mock = MagicMock(spec='SafeConfigParser')
 


Reviews:
user: login: isimo00
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-07T09:16:41Z
user: login: LuiggiTenorioK
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-07T09:34:19Z
user: login: LuiggiTenorioK
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-07T09:38:01Z
user: login: isimo00
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-12T11:10:41Z
user: login: dbeltrankyl
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-03-25T14:18:05Z
user: login: LuiggiTenorioK
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-25T14:56:19Z

Review comments:
diff_hunk: @@ -94,6 +94,17 @@ def insertMany(self, table_name, data):
         cursor.executemany(insert_many_command, data)
         self.connection.commit()
 
+    def delete_where(self, table_name: str, where: list[str]):
path: autosubmit/database/db_manager.py
user: login: isimo00
body: To be consistent with type hints, should we add the `None` type to the output of the function?

`def delete_where(self, table_name: str, where: list[str]) -> None:`

created_at: 2025-03-07T09:12:04Z
author_association: CONTRIBUTOR
diff_hunk: @@ -234,3 +245,67 @@ def generate_select_command(table_name, where=[]):
         for condition in where:
             select_command += ' AND ' + condition
         return select_command
+    
+    @staticmethod
+    def generate_delete_command(table_name: str, where: list[str] = []):
path: autosubmit/database/db_manager.py
user: login: isimo00
body: Type hint return string - see other comment -?
created_at: 2025-03-07T09:14:35Z
author_association: CONTRIBUTOR
diff_hunk: @@ -94,6 +94,17 @@ def insertMany(self, table_name, data):
         cursor.executemany(insert_many_command, data)
         self.connection.commit()
 
+    def delete_where(self, table_name: str, where: list[str]):
path: autosubmit/database/db_manager.py
user: login: LuiggiTenorioK
body: I think adding the type when it only returns `None` is not mandatory.

If it may return any other type or `None` (`Union[<OTHER_TYPE>, None]`), it will be useful to have that. So, when someone else calls that function should also handle the `None` case.
created_at: 2025-03-07T09:34:19Z
author_association: MEMBER
diff_hunk: @@ -234,3 +245,67 @@ def generate_select_command(table_name, where=[]):
         for condition in where:
             select_command += ' AND ' + condition
         return select_command
+    
+    @staticmethod
+    def generate_delete_command(table_name: str, where: list[str] = []):
path: autosubmit/database/db_manager.py
user: login: LuiggiTenorioK
body: I'm sure that @kinow didn't put the hints there to avoid making more changes and keep it consistent which is valid when the PR is big.

I'll add the return types for the generate commands now because this PR is not that big.
created_at: 2025-03-07T09:38:01Z
author_association: MEMBER
diff_hunk: @@ -140,8 +142,14 @@ class JobListPersistenceDb(JobListPersistence):
         "wrapper_type",
     ]
 
-    def __init__(self, persistence_path, persistence_file):
-        self.db_manager = DbManager(persistence_path, persistence_file, self.VERSION)
+    def __init__(self, expid: str):
+        options = {
+            "root_path": str(Path(BasicConfig.LOCAL_ROOT_DIR, expid, "pkl")),
+            "db_name": f"job_list_{expid}",
+            "db_version": self.VERSION
path: autosubmit/job/job_list_persistence.py
user: login: LuiggiTenorioK
body: ```suggestion
            "db_version": self.VERSION,
            "schema": expid
```

I think we forgot to add the schema here :point_up: 

About the changes in the tests, they look way simpler than before. Nice change! :raised_hands: 
created_at: 2025-03-25T14:56:19Z
author_association: MEMBER

PR comments:
user: login: codecov-commenter
created_at: 2025-02-05T16:31:22Z
body: ## [Codecov](https://app.codecov.io/gh/BSC-ES/autosubmit/pull/2107?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=BSC-ES) Report
:x: Patch coverage is `67.30769%` with `17 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 53.90%. Comparing base ([`c452dc6`](https://app.codecov.io/gh/BSC-ES/autosubmit/commit/c452dc68aa2e59aa20987209ed021a238b97c132?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=BSC-ES)) to head ([`c79c992`](https://app.codecov.io/gh/BSC-ES/autosubmit/commit/c79c992bd8c112993f95d199a604f9588d197ca7?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=BSC-ES)).
:warning: Report is 298 commits behind head on master.

| [Files with missing lines](https://app.codecov.io/gh/BSC-ES/autosubmit/pull/2107?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=BSC-ES) | Patch % | Lines |
|---|---|---|
| [autosubmit/database/db\_manager.py](https://app.codecov.io/gh/BSC-ES/autosubmit/pull/2107?src=pr&el=tree&filepath=autosubmit%2Fdatabase%2Fdb_manager.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=BSC-ES#diff-YXV0b3N1Ym1pdC9kYXRhYmFzZS9kYl9tYW5hZ2VyLnB5) | 68.00% | [8 Missing :warning: ](https://app.codecov.io/gh/BSC-ES/autosubmit/pull/2107?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=BSC-ES) |
| [autosubmit/autosubmit.py](https://app.codecov.io/gh/BSC-ES/autosubmit/pull/2107?src=pr&el=tree&filepath=autosubmit%2Fautosubmit.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=BSC-ES#diff-YXV0b3N1Ym1pdC9hdXRvc3VibWl0LnB5) | 30.00% | [7 Missing :warning: ](https://app.codecov.io/gh/BSC-ES/autosubmit/pull/2107?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=BSC-ES) |
| [autosubmit/job/job\_list.py](https://app.codecov.io/gh/BSC-ES/autosubmit/pull/2107?src=pr&el=tree&filepath=autosubmit%2Fjob%2Fjob_list.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=BSC-ES#diff-YXV0b3N1Ym1pdC9qb2Ivam9iX2xpc3QucHk=) | 0.00% | [2 Missing :warning: ](https://app.codecov.io/gh/BSC-ES/autosubmit/pull/2107?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=BSC-ES) |

<details><summary>Additional details and impacted files</summary>



```diff
@@            Coverage Diff             @@
##           master    #2107      +/-   ##
==========================================
+ Coverage   53.88%   53.90%   +0.01%     
==========================================
  Files          72       72              
  Lines       17219    17242      +23     
  Branches     3351     3352       +1     
==========================================
+ Hits         9279     9294      +15     
- Misses       7095     7103       +8     
  Partials      845      845              
```

| [Flag](https://app.codecov.io/gh/BSC-ES/autosubmit/pull/2107/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=BSC-ES) | Coverage Δ | |
|---|---|---|
| [fast-tests](https://app.codecov.io/gh/BSC-ES/autosubmit/pull/2107/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=BSC-ES) | `53.90% <67.30%> (+0.01%)` | :arrow_up: |

Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=BSC-ES#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/BSC-ES/autosubmit/pull/2107?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=BSC-ES).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=BSC-ES).
<details><summary> :rocket: New features to boost your workflow: </summary>

- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
author_association: NONE
user: login: LuiggiTenorioK
created_at: 2025-02-06T09:21:48Z
body: @kinow FYI as some of this code is yours
author_association: MEMBER
user: login: LuiggiTenorioK
created_at: 2025-03-07T09:58:27Z
body: Did a few changes in the latest commit:

- Added type hints for the methods that aren't covered by the Protocol class
- Added return type hints inside the Protocol class
author_association: MEMBER
user: login: kinow
created_at: 2025-03-25T09:23:05Z
body: Rebased, git conflicts fixed. Now waiting for GH Actions :+1: 
author_association: MEMBER
user: login: kinow
created_at: 2025-03-25T09:35:12Z
body: Fixing tests...
author_association: MEMBER
user: login: kinow
created_at: 2025-03-25T09:48:19Z
body: Done, and GH Actions passing now, cc @LuiggiTenorioK .
author_association: MEMBER
user: login: kinow
created_at: 2025-03-25T12:05:34Z
body: Luiggi and I cannot review this one (we are both authors of parts of the code), can you do a final review whenever you have time, please, @dbeltrankyl ? Thanks!
author_association: MEMBER
user: login: kinow
created_at: 2025-03-25T14:14:07Z
body: Fixing conflicts...
author_association: MEMBER
user: login: dbeltrankyl
created_at: 2025-03-25T14:17:47Z
body: Looks good! 

I don't see anything that could make conflicts with the changes of the memory

Sorry for not looking at it earlier; I didn't notice I was assigned. 

It has now conflicts likely due to the https://github.com/BSC-ES/autosubmit/pull/2238 😓 
author_association: COLLABORATOR
user: login: kinow
created_at: 2025-03-25T14:46:09Z
body: Thanks @dbeltrankyl ! Conflicts fixed, had to adjust some of the tests after the conflicts were resolved as the `JobListPersistence` class will create a sqlite file that will fail unless the `autosubmit_config` fixture is loaded. But I believe the tests will pass. @LuiggiTenorioK can you have a look once GH Actions pass if everything looks in order, please? (After fixing conflicts a couple of times I stop being able to see the wood for the trees :sweat_smile: )
author_association: MEMBER
user: login: kinow
created_at: 2025-03-25T14:59:09Z
body: Thanks for the final review, @LuiggiTenorioK . I accepted your suggestion. I'm about to log off for today. Feel free to take over and merge if GH Actions pass :+1: Otherwise, I can merge it tomorrow morning. Cheers
author_association: MEMBER
user: login: LuiggiTenorioK
created_at: 2025-03-25T15:03:24Z
body: Thanks @kinow! Everything seems sorted now. Merging :rocket: 
author_association: MEMBER