Repository: flux-framework/flux-accounting
Pull request #491: plugin: add enforcement of max running jobs limit for a queue per-association
Author: login: cmoussa1
Labels: name: new feature
description: new feature
name: merge-when-passing
name: plugin
description: related to the multi-factor priority plugin
Created: 2024-09-27T18:24:32Z | Merged: 2025-03-21T03:39:09Z

#### Problem

The priority plugin does not support enforcement of a max running jobs limit in a queue for an association, i.e the only max running jobs limit currently enforced for an association is one across _all_ of their running jobs, regardless of queue.

---

This PR looks to add enforcement of a new limit to the priority plugin: max running jobs in a queue per-association. To achieve this, new members are added to the `Queue` class and the `Association` class:

##### Queue

* `max_running_jobs`: the max number of running jobs an association can have in a queue

##### Association

* `queue_usage`: a hash-map storing the name of the queue and the number of running jobs the association has in that queue
* `queue_held_jobs`: a hash-map storing the name of the queue and a list of any held jobs the association has in that queue

If a queue is specified on submission, the priority plugin will keep track of these running jobs per-association by incrementing the association's running jobs counter for that queue when a job enters `job.state.run` and decrementing when it enters `job.state.inactive`.

If an association has the max number of running jobs in a queue and submits another job to that queue, a *queue* running jobs dependency is added in `job.state.depend`. The job ID will be stored in the `Association` object and held until a currently running job in that queue transitions to `job.state.inactive`.

In `job.state.inactive`, if a queue is specified for a job, a check is performed to see if the association has any other jobs waiting to be run in that queue. If one is specified (and the association is now under the max running jobs limit), the dependency is removed from the first held job in that queue and it can proceed to run.

I've added some basic tests that simulate an association submitting the max number of jobs to a queue and having a dependency added to another submitted job. But, while at this limit, the association can successfully submit jobs to other queues and have them run. Once a currently running job in the queue where a job is held completes, the held job can transition to run.


Diff:
diff --git a/src/bindings/python/fluxacct/accounting/__init__.py.in b/src/bindings/python/fluxacct/accounting/__init__.py.in
index dc38e81f8..a656ed479 100644
--- a/src/bindings/python/fluxacct/accounting/__init__.py.in
+++ b/src/bindings/python/fluxacct/accounting/__init__.py.in
@@ -1,6 +1,6 @@
 DB_DIR = "@X_LOCALSTATEDIR@/lib/flux/"
 DB_PATH = "@X_LOCALSTATEDIR@/lib/flux/FluxAccounting.db"
-DB_SCHEMA_VERSION = 26
+DB_SCHEMA_VERSION = 27
 
 # flux-accounting DB table column names
 ASSOCIATION_TABLE = [
@@ -29,6 +29,7 @@ QUEUE_TABLE = [
     "max_nodes_per_job",
     "max_time_per_job",
     "priority",
+    "max_running_jobs",
 ]
 PROJECT_TABLE = ["project_id", "project", "usage"]
 JOBS_TABLE = [
diff --git a/src/bindings/python/fluxacct/accounting/create_db.py b/src/bindings/python/fluxacct/accounting/create_db.py
index 83c98cc45..64918fd43 100755
--- a/src/bindings/python/fluxacct/accounting/create_db.py
+++ b/src/bindings/python/fluxacct/accounting/create_db.py
@@ -182,6 +182,7 @@ def create_db(
                 max_nodes_per_job   int(11)    DEFAULT 1   NOT NULL    ON CONFLICT REPLACE DEFAULT 1,
                 max_time_per_job    int(11)    DEFAULT 60  NOT NULL    ON CONFLICT REPLACE DEFAULT 60,
                 priority            int(11)    DEFAULT 0   NOT NULL    ON CONFLICT REPLACE DEFAULT 0,
+                max_running_jobs    int(11)    DEFAULT 100 NOT NULL    ON CONFLICT REPLACE DEFAULT 100,
                 PRIMARY KEY (queue)
             );"""
     )
diff --git a/src/bindings/python/fluxacct/accounting/queue_subcommands.py b/src/bindings/python/fluxacct/accounting/queue_subcommands.py
index dc9d09c31..60ea52267 100644
--- a/src/bindings/python/fluxacct/accounting/queue_subcommands.py
+++ b/src/bindings/python/fluxacct/accounting/queue_subcommands.py
@@ -31,7 +31,9 @@ def view_queue(conn, queue, parsable=False):
         raise sqlite3.OperationalError(f"an sqlite3.OperationalError occurred: {exc}")
 
 
-def add_queue(conn, queue, min_nodes=1, max_nodes=1, max_time=60, priority=0):
+def add_queue(
+    conn, queue, min_nodes=1, max_nodes=1, max_time=60, priority=0, max_running_jobs=100
+):
     try:
         insert_stmt = """
                       INSERT INTO queue_table (
@@ -39,8 +41,9 @@ def add_queue(conn, queue, min_nodes=1, max_nodes=1, max_time=60, priority=0):
                         min_nodes_per_job,
                         max_nodes_per_job,
                         max_time_per_job,
-                        priority
-                      ) VALUES (?, ?, ?, ?, ?)
+                        priority,
+                        max_running_jobs
+                      ) VALUES (?, ?, ?, ?, ?, ?)
                       """
         conn.execute(
             insert_stmt,
@@ -50,6 +53,7 @@ def add_queue(conn, queue, min_nodes=1, max_nodes=1, max_time=60, priority=0):
                 max_nodes,
                 max_time,
                 priority,
+                max_running_jobs,
             ),
         )
 
@@ -78,6 +82,7 @@ def edit_queue(
     max_nodes_per_job=None,
     max_time_per_job=None,
     priority=None,
+    max_running_jobs=None,
 ):
     params = locals()
     editable_fields = [
@@ -85,6 +90,7 @@ def edit_queue(
         "max_nodes_per_job",
         "max_time_per_job",
         "priority",
+        "max_running_jobs",
     ]
 
     for field in editable_fields:
diff --git a/src/cmd/flux-account-priority-update.py b/src/cmd/flux-account-priority-update.py
index a39cffe7f..a08c09856 100755
--- a/src/cmd/flux-account-priority-update.py
+++ b/src/cmd/flux-account-priority-update.py
@@ -112,6 +112,7 @@ def bulk_update(path):
             "max_nodes_per_job": int(row["max_nodes_per_job"]),
             "max_time_per_job": int(row["max_time_per_job"]),
             "priority": int(row["priority"]),
+            "max_running_jobs": int(row["max_running_jobs"]),
         }
         bulk_q_data.append(single_q_data)
 
diff --git a/src/cmd/flux-account-service.py b/src/cmd/flux-account-service.py
index 42c5f5976..5eb70afa3 100755
--- a/src/cmd/flux-account-service.py
+++ b/src/cmd/flux-account-service.py
@@ -443,6 +443,7 @@ def add_queue(self, handle, watcher, msg, arg):
                 msg.payload["max_nodes_per_job"],
                 msg.payload["max_time_per_job"],
                 msg.payload["priority"],
+                msg.payload["max_running_jobs"],
             )
 
             payload = {"add_queue": val}
@@ -500,6 +501,7 @@ def edit_queue(self, handle, watcher, msg, arg):
                 msg.payload["max_nodes_per_job"],
                 msg.payload["max_time_per_job"],
                 msg.payload["priority"],
+                msg.payload["max_running_jobs"],
             )
 
             payload = {"edit_queue": val}
diff --git a/src/cmd/flux-account.py b/src/cmd/flux-account.py
index ad056a3ba..65f96591b 100755
--- a/src/cmd/flux-account.py
+++ b/src/cmd/flux-account.py
@@ -568,6 +568,12 @@ def add_add_queue_arg(subparsers):
         default=0,
         metavar="PRIORITY",
     )
+    subparser_add_queue.add_argument(
+        "--max-running-jobs",
+        help="max number of running jobs an association can have in the queue",
+        default=100,
+        metavar="MAX_RUNNING_JOBS",
+    )
 
 
 def add_view_queue_arg(subparsers):
@@ -625,6 +631,13 @@ def add_edit_queue_arg(subparsers):
         default=None,
         metavar="PRIORITY",
     )
+    subparser_edit_queue.add_argument(
+        "--max-running-jobs",
+        type=int,
+        help="max number of running jobs an association can have in the queue",
+        default=None,
+        metavar="MAX_RUNNING_JOBS",
+    )
 
 
 def add_delete_queue_arg(subparsers):
diff --git a/src/plugins/accounting.cpp b/src/plugins/accounting.cpp
index 864a34139..55eb9a974 100644
--- a/src/plugins/accounting.cpp
+++ b/src/plugins/accounting.cpp
@@ -84,10 +84,29 @@ json_t* Association::to_json () const
         }
     }
 
+    json_t *queue_usage_json = json_object ();
+    if (!queue_usage_json) {
+        json_decref (held_job_ids);
+        json_decref (user_queues);
+        json_decref (user_projects);
+        return nullptr;
+    }
+    for (const auto &entry : queue_usage) {
+        if (json_object_set_new (queue_usage_json,
+                                 entry.first.c_str (),
+                                 json_integer (entry.second)) < 0) {
+            json_decref (held_job_ids);
+            json_decref (user_queues);
+            json_decref (user_projects);
+            json_decref (queue_usage_json);
+            return nullptr;
+        }
+    }
+
     // 'o' steals the reference for both held_job_ids and user_queues
     json_t *u = json_pack ("{s:s, s:f, s:i, s:i, s:i, s:i, s:o,"
                            " s:o, s:i, s:o, s:s, s:i, s:i, s:i,"
-                           " s:i, s:i}",
+                           " s:i, s:i, s:o}",
                            "bank_name", bank_name.c_str (),
                            "fairshare", fairshare,
                            "max_run_jobs", max_run_jobs,
@@ -103,7 +122,8 @@ json_t* Association::to_json () const
                            "max_cores", max_cores,
                            "cur_nodes", cur_nodes,
                            "cur_cores", cur_cores,
-                           "active", active);
+                           "active", active,
+                           "queue_usage", queue_usage_json);
 
     if (!u)
         return nullptr;
@@ -232,3 +252,14 @@ int get_project_info (const char *project,
 
     return 0;
 }
+
+
+int max_run_jobs_per_queue (const std::map<std::string, Queue> &queues,
+                            const std::string &queue)
+{
+    auto it = queues.find (queue);
+    if (it == queues.end ())
+        return -1;
+
+    return it->second.max_running_jobs;
+}
diff --git a/src/plugins/accounting.hpp b/src/plugins/accounting.hpp
index 8e9b9c47a..952b48fbb 100644
--- a/src/plugins/accounting.hpp
+++ b/src/plugins/accounting.hpp
@@ -27,6 +27,7 @@ extern "C" {
 #include <iterator>
 #include <sstream>
 #include <algorithm>
+#include <unordered_map>
 
 // all attributes are per-user/bank
 class Association {
@@ -48,6 +49,11 @@ class Association {
     int max_cores;                     // max num cores across all running jobs
     int cur_nodes;                     // current number of used nodes
     int cur_cores;                     // current number of used cores
+    std::unordered_map<std::string, int>
+      queue_usage;                     // track num of running jobs per queue
+    std::unordered_map<std::string,
+                       std::vector<long int>>
+      queue_held_jobs;                // keep track of held job ID's per queue
 
     // methods
     json_t* to_json () const;    // convert object to JSON string
@@ -77,6 +83,7 @@ class Queue {
     int max_nodes_per_job;
     int max_time_per_job;
     int priority;
+    int max_running_jobs;
 };
 
 // get an Association object that points to user/bank in the users map;
@@ -113,4 +120,8 @@ int get_project_info (const char *project,
                       std::vector<std::string> &permissible_projects,
                       std::vector<std::string> projects);
 
+// fetch the max number of running jobs a queue can have per-association
+int max_run_jobs_per_queue (const std::map<std::string, Queue> &queues,
+                            const std::string &queue);
+
 #endif // ACCOUNTING_H
diff --git a/src/plugins/mf_priority.cpp b/src/plugins/mf_priority.cpp
index fc22b2266..d6bddfaca 100644
--- a/src/plugins/mf_priority.cpp
+++ b/src/plugins/mf_priority.cpp
@@ -395,6 +395,7 @@ static void rec_q_cb (flux_t *h,
 {
     char *queue = NULL;
     int min_nodes_per_job, max_nodes_per_job, max_time_per_job, priority = 0;
+    int max_running_jobs = 0;
     json_t *data, *jtemp = NULL;
     json_error_t error;
     int num_data = 0;
@@ -417,12 +418,13 @@ static void rec_q_cb (flux_t *h,
         json_t *el = json_array_get(data, i);
 
         if (json_unpack_ex (el, &error, 0,
-                            "{s:s, s:i, s:i, s:i, s:i}",
+                            "{s:s, s:i, s:i, s:i, s:i, s:i}",
                             "queue", &queue,
                             "min_nodes_per_job", &min_nodes_per_job,
                             "max_nodes_per_job", &max_nodes_per_job,
                             "max_time_per_job", &max_time_per_job,
-                            "priority", &priority) < 0)
+                            "priority", &priority,
+                            "max_running_jobs", &max_running_jobs) < 0)
             flux_log (h, LOG_ERR, "mf_priority unpack: %s", error.text);
 
         Queue *q;
@@ -432,6 +434,7 @@ static void rec_q_cb (flux_t *h,
         q->max_nodes_per_job = max_nodes_per_job;
         q->max_time_per_job = max_time_per_job;
         q->priority = priority;
+        q->max_running_jobs = max_running_jobs;
     }
 
     if (flux_respond (h, msg, NULL) < 0)
@@ -866,12 +869,15 @@ static int depend_cb (flux_plugin_t *p,
     int userid;
     long int id;
     Association *b;
+    char *queue = NULL;
 
     flux_t *h = flux_jobtap_get_flux (p);
     if (flux_plugin_arg_unpack (args,
                                 FLUX_PLUGIN_ARG_IN,
-                                "{s:i, s:I}",
-                                "userid", &userid, "id", &id) < 0) {
+                                "{s:i, s:I, s{s{s{s?s}}}}",
+                                "userid", &userid, "id", &id,
+                                "jobspec", "attributes", "system",
+                                "queue", &queue) < 0) {
         flux_log (h,
                   LOG_ERR,
                   "flux_plugin_arg_unpack: %s",
@@ -892,6 +898,38 @@ static int depend_cb (flux_plugin_t *p,
         return -1;
     }
 
+    if (queue != NULL) {
+        // safely assign "queue" to an std::string
+        std::string value (queue);
+        // fetch max number of running jobs in this queue
+        int queue_max_run_jobs = max_run_jobs_per_queue (queues, queue);
+        if (queue_max_run_jobs < 0) {
+            // can't find a max_run_jobs limit for this queue because it
+            // might not be defined in the flux-accounting DB, so just set
+            // it to a large number
+            queue_max_run_jobs = std::numeric_limits<int>::max ();
+        }
+
+        // look up the association's current number of running jobs;
+        // if queue cannot be found, an entry in the Association object will be
+        // initialized with a current running jobs count of 0
+        int assoc_cur_run_jobs = b->queue_usage[queue];
+        if (assoc_cur_run_jobs >= queue_max_run_jobs) {
+            // association is already at their max number of running jobs
+            // in this queue; add a dependency
+            if (flux_jobtap_dependency_add (p, id, "max-run-jobs-queue") < 0) {
+                flux_jobtap_raise_exception (p, FLUX_JOBTAP_CURRENT_JOB,
+                                            "mf_priority", 0, "failed to "
+                                            "add dependency for max run jobs "
+                                            "per-queue limit");
+                return -1;
+            }
+            b->queue_held_jobs[queue].push_back (id);
+
+            return 0;
+        }
+    }
+
     // if user has already hit their max running jobs count, add a job
     // dependency to hold job until an already running job has finished
     if ((b->max_run_jobs > 0) && (b->cur_run_jobs == b->max_run_jobs)) {
@@ -919,12 +957,15 @@ static int run_cb (flux_plugin_t *p,
     int userid;
     Association *b;
     json_t *jobspec = NULL;
+    char *queue = NULL;
 
     flux_t *h = flux_jobtap_get_flux (p);
     if (flux_plugin_arg_unpack (args,
                                 FLUX_PLUGIN_ARG_IN,
-                                "{s:o}",
-                                "jobspec", &jobspec) < 0) {
+                                "{s:o, s{s{s{s?s}}}}",
+                                "jobspec", &jobspec,
+                                "jobspec", "attributes", "system",
+                                "queue", &queue) < 0) {
         flux_log (h,
                   LOG_ERR,
                   "flux_plugin_arg_unpack: %s",
@@ -945,7 +986,12 @@ static int run_cb (flux_plugin_t *p,
         return -1;
     }
 
-    // increment the user's current running jobs and resources counts
+    if (queue != NULL)
+        // a queue was passed-in; increment counter of the number of
+        // queue-specific running jobs for this association
+        b->queue_usage[std::string (queue)]++;
+
+    // increment the user's current running jobs count
     b->cur_run_jobs++;
     if (jobspec == NULL) {
         flux_jobtap_raise_exception (p,
@@ -1181,13 +1227,16 @@ static int inactive_cb (flux_plugin_t *p,
     int userid;
     Association *b;
     json_t *jobspec = NULL;
+    char *queue = NULL;
 
     flux_t *h = flux_jobtap_get_flux (p);
     if (flux_plugin_arg_unpack (args,
                                 FLUX_PLUGIN_ARG_IN,
-                                "{s:i, s:o}",
+                                "{s:i, s:o, s{s{s{s?s}}}}",
                                 "userid", &userid,
-                                "jobspec", &jobspec) < 0) {
+                                "jobspec", &jobspec,
+                                "jobspec", "attributes", "system",
+                                "queue", &queue) < 0) {
         flux_log (h,
                   LOG_ERR,
                   "flux_plugin_arg_unpack: %s",
@@ -1236,6 +1285,43 @@ static int inactive_cb (flux_plugin_t *p,
         }
     }
 
+    if (queue != NULL) {
+        // safely assign "queue" to an std::string
+        std::string value (queue);
+        if (b->queue_usage[queue] > 0) {
+            // decrement the counter of running jobs the association in queue
+            b->queue_usage[queue]--;
+
+            // fetch max number of running jobs in queue
+            int queue_max_run_jobs = max_run_jobs_per_queue (queues, queue);
+            if (queue_max_run_jobs < 0) {
+                // can't find a max_run_jobs limit for this queue because it
+                // might not be defined in the flux-accounting DB, so just set
+                // it to a large number
+                queue_max_run_jobs = std::numeric_limits<int>::max ();
+            }
+
+            if ((b->queue_held_jobs[queue].size () > 0) &&
+                (b->queue_usage[queue] < queue_max_run_jobs)) {
+                // association has at least one held job in queue;
+                // remove the dependency from the first held job
+                long int id = b->queue_held_jobs[queue].front ();
+                if (flux_jobtap_dependency_remove (p,
+                                                   id,
+                                                   "max-run-jobs-queue") < 0) {
+                    flux_jobtap_raise_exception (p, id, "mf_priority",
+                                                 0, "failed to remove job "
+                                                 " dependency for max run jobs "
+                                                 "per-queue limit");
+                    return -1;
+                }
+                b->queue_held_jobs[queue].erase (
+                    b->queue_held_jobs[queue].begin ()
+                );
+            }
+        }
+    }
+
     // if the user/bank combo has any currently held jobs and the user is now
     // under their max jobs limit, remove the dependency from first held job
     if ((b->held_jobs.size () > 0) && (b->cur_run_jobs < b->max_run_jobs)) {
diff --git a/src/plugins/test/accounting_test01.cpp b/src/plugins/test/accounting_test01.cpp
index 7bb73414f..a98ed5b17 100644
--- a/src/plugins/test/accounting_test01.cpp
+++ b/src/plugins/test/accounting_test01.cpp
@@ -58,7 +58,9 @@ void add_user_to_map (
         a.max_nodes,
         a.max_cores,
         a.cur_nodes,
-        a.cur_cores
+        a.cur_cores,
+        a.queue_usage,
+        a.queue_held_jobs
     };
 }
 
@@ -70,9 +72,11 @@ void initialize_map (
     std::map<int, std::map<std::string, Association>> &users)
 {
     Association user1 = {"bank_A", 0.5, 5, 0, 7, 0, {},
-                         {}, 0, 1, {"*"}, "*", 2147483647, 2147483647, 0, 0};
+                         {}, 0, 1, {"*"}, "*", 2147483647, 2147483647, 0, 0,
+                         {}, {}};
     Association user2 = {"bank_A", 0.5, 5, 0, 7, 0, {},
-                         {}, 0, 1, {"*"}, "*", 2147483647, 2147483647, 0, 0};
+                         {}, 0, 1, {"*"}, "*", 2147483647, 2147483647, 0, 0,
+                         {}, {}};
 
     add_user_to_map (users, 1001, "bank_A", user1);
     users_def_bank[1001] = "bank_A";
@@ -86,9 +90,9 @@ void initialize_map (
  * helper function to add test queues to the queues map
  */
 void initialize_queues () {
-    queues["bronze"] = {0, 5, 60, 100};
-    queues["silver"] = {0, 5, 60, 200};
-    queues["gold"] = {0, 5, 60, 300};
+    queues["bronze"] = {0, 5, 60, 100, 100};
+    queues["silver"] = {0, 5, 60, 200, 100};
+    queues["gold"] = {0, 5, 60, 300, 100};
 }
 
 
@@ -275,7 +279,7 @@ static void test_check_map_dne_true ()
 
     Association tmp_user = {"DNE", 0.5, 5, 0, 7, 0, {},
                             {}, 0, 1, {"*"}, "*", 2147483647, 2147483647,
-                            0, 0};
+                            0, 0, {}, {}};
     add_user_to_map (users, 9999, "DNE", tmp_user);
     users_def_bank[9999] = "DNE";
 
diff --git a/t/Makefile.am b/t/Makefile.am
index a8b719226..0337d562e 100644
--- a/t/Makefile.am
+++ b/t/Makefile.am
@@ -51,6 +51,7 @@ TESTSCRIPTS = \
 	t1049-issue580.t \
 	t1050-mf-priority-update-on-reload.t \
 	t1051-list-users.t \
+	t1052-mf-priority-queue-limits.t \
 	t5000-valgrind.t \
 	python/t1000-example.py \
 	python/t1001_db.py \
diff --git a/t/t1052-mf-priority-queue-limits.t b/t/t1052-mf-priority-queue-limits.t
new file mode 100755
index 000000000..d7f4185a5
--- /dev/null
+++ b/t/t1052-mf-priority-queue-limits.t
@@ -0,0 +1,207 @@
+#!/bin/bash
+
+test_description='test priority plugin max-running-jobs per-queue limits'
+
+. `dirname $0`/sharness.sh
+
+mkdir -p conf.d
+
+MULTI_FACTOR_PRIORITY=${FLUX_BUILD_DIR}/src/plugins/.libs/mf_priority.so
+SUBMIT_AS=${SHARNESS_TEST_SRCDIR}/scripts/submit_as.py
+DB_PATH=$(pwd)/FluxAccountingTest.db
+
+export TEST_UNDER_FLUX_SCHED_SIMPLE_MODE="limited=1"
+test_under_flux 16 job -o,--config-path=$(pwd)/conf.d
+
+flux setattr log-stderr-level 1
+
+test_expect_success 'allow guest access to testexec' '
+	flux config load <<-EOF
+	[exec.testexec]
+	allow-guests = true
+	EOF
+'
+
+test_expect_success 'create flux-accounting DB' '
+	flux account -p ${DB_PATH} create-db
+'
+
+test_expect_success 'start flux-accounting service' '
+	flux account-service -p ${DB_PATH} -t
+'
+
+test_expect_success 'load multi-factor priority plugin' '
+	flux jobtap load -r .priority-default ${MULTI_FACTOR_PRIORITY}
+'
+
+test_expect_success 'check that mf_priority plugin is loaded' '
+	flux jobtap list | grep mf_priority
+'
+
+test_expect_success 'add some banks' '
+	flux account add-bank root 1 &&
+	flux account add-bank --parent-bank=root bankA 1
+'
+
+test_expect_success 'add queues with different running jobs limits' '
+	flux account add-queue bronze --max-running-jobs=3 &&
+	flux account add-queue silver --max-running-jobs=2 &&
+	flux account add-queue gold --max-running-jobs=1
+'
+
+test_expect_success 'add a user' '
+	flux account add-user \
+		--username=user1 \
+		--userid=5001 \
+		--bank=bankA \
+		--queues="bronze,silver,gold" \
+		--max-running-jobs=100 \
+		--max-active-jobs=100
+'
+
+test_expect_success 'send the user and queue information to the plugin' '
+	flux account-priority-update -p ${DB_PATH}
+'
+
+test_expect_success 'configure flux with those queues' '
+	cat >conf.d/queues.toml <<-EOT &&
+	[queues.bronze]
+	[queues.silver]
+	[queues.gold]
+	EOT
+	flux config reload &&
+	flux queue start --all
+'
+
+# In this set of tests, an association belongs to all three available queues,
+# and each queue has a different limit on the number of running jobs available
+# per-association. The association will submit the max number of running jobs
+# to the silver queue (2 jobs). A dependency specific to the number of running
+# jobs per-queue is added to the third submitted job in the silver queue, but
+# jobs submitted to other queues will still receive an alloc event.
+#
+# Once one of the currently running jobs in the silver queue completes and is
+# cleaned up, the job with a dependency added to it will have its dependency
+# removed and will receive its alloc event.
+test_expect_success 'submit max number of jobs to silver queue' '
+	job1=$(flux python ${SUBMIT_AS} 5001 --queue=silver sleep 60) &&
+	job2=$(flux python ${SUBMIT_AS} 5001 --queue=silver sleep 60) &&
+	flux job wait-event -vt 5 ${job1} alloc &&
+	flux job wait-event -vt 5 ${job2} alloc
+'
+
+test_expect_success 'running jobs count for the queues are incremented once jobs start' '
+	flux jobtap query mf_priority.so > silver.json &&
+	jq -e ".mf_priority_map[] | \
+		select(.userid == 5001) | \
+		.banks[0].queue_usage.silver == 2" <silver.json
+'
+
+test_expect_success 'a third job to the silver queue results in a dependency-add' '
+	job3=$(flux python ${SUBMIT_AS} 5001 --queue=silver sleep 60) &&
+	flux job wait-event -vt 5 \
+		--match-context=description="max-run-jobs-queue" \
+		${job3} dependency-add
+'
+
+test_expect_success 'association can submit other jobs to other queues in the meantime' '
+	job4=$(flux python ${SUBMIT_AS} 5001 --queue=bronze sleep 60) &&
+	flux job wait-event -vt 5 ${job4} alloc
+'
+
+test_expect_success 'check overall jobs counts for user' '
+	flux jobtap query mf_priority.so > user1.json &&
+	jq -e ".mf_priority_map[] | \
+		select(.userid == 5001) | \
+		.banks[0].cur_run_jobs == 3" <user1.json &&
+	jq -e ".mf_priority_map[] | \
+		select(.userid == 5001) | \
+		.banks[0].cur_active_jobs == 4" <user1.json
+'
+
+test_expect_success 'cancel currently running job in silver queue' '
+	flux cancel ${job1} &&
+	flux job wait-event -vt 5 ${job1} clean
+'
+
+test_expect_success 'wait for alloc on held job and then cancel second and third jobs' '
+	flux job wait-event -vt 5 ${job3} alloc &&
+	flux cancel ${job2} &&
+	flux cancel ${job3} &&
+	flux job wait-event -vt 5 ${job2} clean &&
+	flux job wait-event -vt 5 ${job3} clean
+'
+
+test_expect_success 'cancel job in bronze queue' '
+	flux cancel ${job4} &&
+	flux job wait-event -vt 5 ${job4} clean
+'
+
+test_expect_success 'running jobs count for the queues are decremented once jobs exit' '
+	flux jobtap query mf_priority.so > query.json &&
+	jq -e ".mf_priority_map[] | \
+		select(.userid == 5001) | \
+		.banks[0].queue_usage.silver == 0" <query.json &&
+	jq -e ".mf_priority_map[] | \
+		select(.userid == 5001) | \
+		.banks[0].cur_run_jobs == 0" <query.json &&
+	jq -e ".mf_priority_map[] | \
+		select(.userid == 5001) | \
+		.banks[0].cur_active_jobs == 0" <query.json
+'
+
+# In this set of tests, the association will have a max running jobs limit
+# that is less than the number of jobs they can run in a given queue. In this
+# case, the association will have a more general running jobs limit dependency
+# added to their job instead of the queue-specific dependency.
+test_expect_success 'edit the max-running-jobs limit of the association' '
+	flux account edit-user user1 --max-running-jobs=2 &&
+	flux account-priority-update -p ${DB_PATH}
+'
+
+test_expect_success 'submit max running jobs to bronze queue' '
+	job1=$(flux python ${SUBMIT_AS} 5001 --queue=bronze sleep 60) &&
+	job2=$(flux python ${SUBMIT_AS} 5001 --queue=bronze sleep 60) &&
+	flux job wait-event -vt 5 ${job1} alloc &&
+	flux job wait-event -vt 5 ${job2} alloc
+'
+
+test_expect_success 'a third submitted job (regardless of queue) results in dependency-add' '
+	job3=$(flux python ${SUBMIT_AS} 5001 --queue=silver sleep 60) &&
+	flux job wait-event -vt 5 \
+		--match-context=description="max-running-jobs-user-limit" \
+		${job3} dependency-add
+'
+
+test_expect_success 'check active/running jobs counts' '
+	flux jobtap query mf_priority.so > user1.json &&
+	jq -e ".mf_priority_map[] | \
+		select(.userid == 5001) | \
+		.banks[0].held_jobs | length == 1" <user1.json &&
+	jq -e ".mf_priority_map[] | \
+		select(.userid == 5001) | \
+		.banks[0].cur_run_jobs == 2" <user1.json &&
+	jq -e ".mf_priority_map[] | \
+		select(.userid == 5001) | \
+		.banks[0].cur_active_jobs == 3" <user1.json
+'
+
+test_expect_success 'cancel currently running job; held job gets alloc event' '
+	flux cancel ${job1} &&
+	flux job wait-event -vt 5 ${job1} clean &&
+	flux job wait-event -vt 5 ${job3} alloc &&
+	flux job wait-event -vt 5 \
+		--match-context=description="max-running-jobs-user-limit" \
+		${job3} dependency-remove
+'
+
+test_expect_success 'cancel running jobs' '
+	flux cancel ${job2} &&
+	flux cancel ${job3}
+'
+
+test_expect_success 'shut down flux-accounting service' '
+	flux python -c "import flux; flux.Flux().rpc(\"accounting.shutdown_service\").get()"
+'
+
+test_done


Reviews:
user: login: jameshcorbett
body: NICE
state: APPROVED
author_association: MEMBER
submitted_at: 2025-03-20T22:52:15Z
user: login: cmoussa1
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-21T03:22:56Z

Review comments:
diff_hunk: @@ -895,6 +898,38 @@ static int depend_cb (flux_plugin_t *p,
         return -1;
     }
 
+    if (queue != NULL) {
+        // safely assign "queue" to an std::string
+        std::string value (queue);
+        // fetch max number of running jobs in this queue
+        int queue_max_run_jobs = max_run_jobs_per_queue (queues, queue);
+        if (queue_max_run_jobs < 0) {
+            // can't find a max_run_jobs limit for this queue because it
+            // might not be defined in the flux-accounting DB, so just set
+            // it to a large number
+            queue_max_run_jobs = std::numeric_limits<int>::max ();
+        }
+
+        // look up the association's current number of running jobs;
+        // if queue cannot be found, an entry in the Association object will be
+        // initialized with a current running jobs count of 0
+        int assoc_cur_run_jobs = b->queue_usage[queue];
+        if (assoc_cur_run_jobs == queue_max_run_jobs) {
path: src/plugins/mf_priority.cpp
user: login: jameshcorbett
body: Could be `>=` here right? Shouldn't hurt? or help I guess
created_at: 2025-03-20T22:52:07Z
author_association: MEMBER
diff_hunk: @@ -895,6 +898,38 @@ static int depend_cb (flux_plugin_t *p,
         return -1;
     }
 
+    if (queue != NULL) {
+        // safely assign "queue" to an std::string
+        std::string value (queue);
+        // fetch max number of running jobs in this queue
+        int queue_max_run_jobs = max_run_jobs_per_queue (queues, queue);
+        if (queue_max_run_jobs < 0) {
+            // can't find a max_run_jobs limit for this queue because it
+            // might not be defined in the flux-accounting DB, so just set
+            // it to a large number
+            queue_max_run_jobs = std::numeric_limits<int>::max ();
+        }
+
+        // look up the association's current number of running jobs;
+        // if queue cannot be found, an entry in the Association object will be
+        // initialized with a current running jobs count of 0
+        int assoc_cur_run_jobs = b->queue_usage[queue];
+        if (assoc_cur_run_jobs == queue_max_run_jobs) {
path: src/plugins/mf_priority.cpp
user: login: cmoussa1
body: Oh yeah, good suggestion, you're right. Shouldn't hurt!
created_at: 2025-03-21T03:22:56Z
author_association: MEMBER

PR comments:
user: login: cmoussa1
created_at: 2024-09-30T17:04:42Z
body: OK, I've added a couple more tests here and I think this might be ready for some initial review. I don't expect this to make this month's release, so no rush here. Let me know if the second and third commit could perhaps be squashed? I tried to keep them separate so they'd be easier to review, but can squash if we feel like it'd make more sense.
author_association: MEMBER
user: login: cmoussa1
created_at: 2024-10-03T17:29:55Z
body: @ryanday36 would you like to give this PR just a high-level look and confirm that this solves the case you brought up in #402? Let me know if I understood the requirements correctly. 
author_association: MEMBER
user: login: cmoussa1
created_at: 2025-03-19T16:45:36Z
body: @ryanday36 this has now been rebased if you want to give this PR a look :-)
author_association: MEMBER
user: login: ryanday36
created_at: 2025-03-19T18:45:11Z
body: Thanks @cmoussa1! Sorry I never got back to you on this in October. It sounds like it's doing what we want.
author_association: CONTRIBUTOR