Repository: dmwm/WMCore
Pull request #12302: Improve monitoring information of WMAgent components
Author: login: vkuznet
Labels: name: New Feature
name: WMAgent
name: WMStats
name: Monitoring
Created: 2025-03-11T17:45:51Z | Merged: 2025-05-07T15:14:08Z

Fixes #12145 

#### Status
ready

#### Description
To improve WMagent component's monitoring we introduced the following changes:

- Add new procfs module with `processStatus` function based on `/proc` info for given PID. This function returns information about process threads which later can be used by WMStats. For example, here is addition to [`agentInfo`](https://cmsweb-testbed.cern.ch/couchdb/wmstats/_design/WMStats/_view/agentInfo) dictionary we supply to CouchDB:
```
  "components": {
...
    "DBS3Upload": [
      {"process": "python", "pid": "1921054", "status": "S (sleeping)",  "type": "process"},
      {"process": "python", "pid": "1921059", "status": "S (sleeping)",  "type": "thread" }
    ],
...
}
```
It shows the main process and its threads, in this case main process pid is 1921054, and pid of its threads (in this case only one additional thread) is 1921059 which is in sleeping state.

- add `processThreadsInfo` function in `Utils/ProcessStats.py` module which provides process and thread monitoring information:
```
{"process_name": .., "pid": .., "status": .., "ppid": .., "cmdline":.., 
 "cpu_usage_percent": .., "memory_usage_percent": .., "memory_rss": .., 
 "num_open_files": .., "num_connections": .., 
 "threads": [..]}
```
and thread info:
```
{"thread_id": .., "user_time":.., "system_time":.., "cpu_usage_percent": ..,
  "memory_usage_bytes":.., "num_open_files":.., "num_connections":..,
  "state":.., "name":..}
```
- update `WMComponent/AnalyticsDataCollector/DataCollectAPI.py` parts with information about dead threads via `threadsDetails` function within `agentInfo['down_component_detail']` used by WMStats web UI
- update `bin/wmcoreD` where we provide status of component's process along with its threads, e.g.
```
manage status
...
Component:WorkQueueManager 3545240 running with threads [3545241, 3545242, 3545243, 3545244]
Component:DBS3Upload 3545249 running with threads [3545254]
...
```
Each component now reports all its threads and if thread is missing/died/stuck it will provide proper message about it, e.g. component can be in `partially-running` state where its main process and some of the threads are ok but other threads misbehave

- introduced `trheads.json` file in component area along with `Daemon.xml` to capture initial conditions of component state. This information is used by aforementioned new functionality to determine if original threads are running or died

- propagate thread information into `WMComponent/AnalyticsDataCollector/DataCollectAPI.py` who use it and analyze state of specific component and properly report its state (with new threads information) to CouchDB.

**Note:** The information reported in WMStats web UI **agentInfo** tab can be accessed via the following URL: `https://xxxx.cern.ch/couchdb/wmstats/_design/WMStats/_view/agentInfo` which return nested dictionary which is used by WMStats JavaScript engine to report this information in **agentInfo** tab.

**Pylint note:** I noticed degradation in modified `bin/wmcoreD` codebase which is not related to proposed changes. For instance, there are underlined variables used in this codebase. And, I explicitly did not touch those errors/warnings which most likely come from stricter rules imposed by pylint over time. If those should be addressed I suggest to make an explicit request for them and I'll be happy to fix those issues.

**Note II:** The proposed changed to `wmcoreD` will affect output of `manage status` for list of components. In particular the new output for each individual component will have the following outputs:
- if everything is fine
```
Component:ABC 123 running with threads X
```
- if main process is dead
```
Component:ABC 123 not-running
```

- if main process is alive but one of its threads is dead
```
Component:ABC 123 partially-running with threads X, lost Y threads
```


#### Is it backward compatible (if not, which system it affects?)
YES

#### Related PRs

#### External dependencies / deployment changes


Linked issue #12145: Improve and simplify status check of WMAgent components
**Impact of the new feature**
Reduce impact on shifter to walk through every WMAgent to check status of all components.

**Is your feature request related to a problem? Please describe.**
During investigation of DBS3Upload backlog I found that on vocms0282 it was stuck. The ComponentLog shown no issues. We checked with CMS DBA and found that no ORACLE connection was performed either. I logged into docker image and found that this component was not running via `manage status` command. It means that inspecting ComponentLog is not sufficient to know if component is running or not, and therefore to properly know the status of component the shifter must login to each individual WMAgent docker image, issue `manage status` command and pay attention about status of the components.

**Describe the solution you'd like**
Review how WMStats UI agent info is populated and revisit how we report status of components

**Describe alternatives you've considered**
Keep things as is and rely on manual checks from shifters.

**Additional context**

Linked issue comments: user: login: vkuznet
created_at: 2025-03-10T17:42:50Z
body: The actual workflow which reports WMAgent component status is divided into two parts.

### WMAgent Component
The WMAgent runs AgentStatusPoller component to collect data about specific agent, using the following chain of steps:
- the workflow is initiated by [AgentStatusPoller.py](https://github.com/dmwm/WMCore/blob/master/src/python/WMComponent/AgentStatusWatcher/DrainStatusPoller.py)
- it calls [DataCollectAPI.py](https://github.com/dmwm/WMCore/blob/master/src/python/WMComponent/AnalyticsDataCollector/DataCollectAPI.py) module and invokes [`getComponentStatus`](https://github.com/dmwm/WMCore/blob/master/src/python/WMComponent/AnalyticsDataCollector/DataCollectAPI.py#L237) API
- this API calls [Details.py](https://github.com/dmwm/WMCore/blob/master/src/python/WMCore/Agent/Daemon/Details.py) module
- which provides [`isAlive`](https://github.com/dmwm/WMCore/blob/master/src/python/WMCore/Agent/Daemon/Details.py#L76) method
- which by itself calls `ps -p PID`

Now, the PID of the WMAgent Component can be obtained from its `Daemon.xml` file, e.g. on vocms264  `TaskArchiver/Daemon.xml` has the following content
```
<Daemon>
        <ProcessID Value="1090"/>
        <ParentProcessID Value="1"/>
        <ProcessGroupID Value="1089"/>
        <UserID Value="31961"/>
        <EffectiveUserID Value="31961"/>
        <GroupID Value="1399"/>
        <EffectiveGroupID Value="1399"/>
</Daemon>
```
The PID of the component is extracted from ProcessID. So, and then `ps` command is executed for that PID, e.g.
```
ps -p 1090
    PID TTY          TIME CMD
   1090 ?        15:40:21 python
```
But it only shows parent process, the component, see
```
ps auxww | grep 1090
cmst1       1090  2.4  0.6 1285116 775628 ?      Sl   Feb11 940:21 python /usr/local/bin/wmcoreD --start --config=/data/srv/wmagent/2.3.9/config/config.py
```
Therefore, the existing codebase never looks at status of component threads. To do that we have two different solutions:
1. use `ps -eLf | grep <pid>` command to show all running threads and check if any of them has `Z` (zombie) or `T (stopped)` flag in it, e.g.
```
ps -eLf | grep 1090
cmst1       1090       1    1090  0    4 Feb11 ?        00:00:00 python /usr/local/bin/wmcoreD --start --config=/data/srv/wmagent/2.3.9/config/config.py
cmst1       1090       1    1091  0    4 Feb11 ?        00:00:49 python /usr/local/bin/wmcoreD --start --config=/data/srv/wmagent/2.3.9/config/config.py
cmst1       1090       1    1092  0    4 Feb11 ?        02:55:54 python /usr/local/bin/wmcoreD --start --config=/data/srv/wmagent/2.3.9/config/config.py
cmst1       1090       1    1093  1    4 Feb11 ?        12:41:14 python /usr/local/bin/wmcoreD --start --config=/data/srv/wmagent/2.3.9/config/config.py
```
Here is list of states according to Linux OS: 
- `R` → Running
- `S` → Sleeping (Idle but can be scheduled)
- `D` → Uninterruptible sleep (e.g., I/O wait)
- `Z` → **Zombie (Dead thread, needs cleanup)**
- `T` → **Stopped (Suspended, e.g., by `SIGSTOP`)**
 
2. use `/proc` file system to obtain status of the process and its threads. It involves two steps:- find all PIDs of the component, e.g.
```
ls /proc/1090/task
1090  1091  1092  1093
```
- inspect individual PID and check out its state, e.g.
```
cat /proc/1090/task/1091/status | grep State
State:  S (sleeping)
```

Therefore, to obtain correct state of the component including its threads either of the above approaches should be used. But that raises another issue:
- how to properly report it to upstream server, e.g. should we provide individual dict entry for parent process ID, and its thread?
- which data structure to use for that, and how to update upstream server (WMStats) to properly parse and report it on its web UI

### WMStats web UI
The WMStats service (web UI) reports under **agent info** tab the status of all WMAgents. To display particular state of the component it relies on the following codebase
- [WMStats.Agents.js](https://github.com/dmwm/WMCore/blob/master/src/couchapps/WMStats/_attachments/js/DataStruct/WMStats.Agents.js) module to obtain agent data from wmstats CouchDB
- this information is then rendered by [WMStats.AgentModel.js](https://github.com/dmwm/WMCore/blob/master/src/couchapps/WMStats/_attachments/js/Models/WMStats.AgentModel.js) and [WMStats.View.IndexHTML.js](https://github.com/dmwm/WMCore/blob/master/src/couchapps/WMStats/_attachments/js/Views/WMStats.View.IndexHTML.js)

### Solution
The data which obtained from WMStats CouchDB is pushed by the component (see **WMAgent Component** section). Therefore, to implement more comprehensive solution we need
- change data structure in WMAgent component
- parse this data structure in WMStats server (JavaScripts)
author_association: CONTRIBUTOR
user: login: vkuznet
created_at: 2025-03-10T19:17:06Z
body: Additional info (thanks to @amaltaro ):
- we have `wm_workers` table within the agent
- to access MariaDB we can use `docker exec -it wmagent bash` command and within MariaDB prompt we can
- check entries in this table
```
MariaDB [wmagent]> select * from wm_workers;
+--------------+--------------------------------+--------------+---------+-------+---------------+------------+------------+---------+------------>
| component_id | name                           | last_updated | state   | pid   | poll_interval | last_error | cycle_time | outcome | error_messa>
+--------------+--------------------------------+--------------+---------+-------+---------------+------------+------------+---------+------------>
|           91 | AgentStatusPoller              |   1741633497 | Running |  1262 |           300 |       NULL |    27.4943 | NULL    | NULL       >
|           89 | AnalyticsPoller                |   1741633591 | Running |  1246 |           600 |       NULL |    219.318 | NULL    | NULL       >
|           90 | ArchiveDataPoller              |   1741633619 | Running |  1254 |           300 |       NULL |     0.0046 | NULL    | NULL       >
|           88 | CleanCouchPoller               |   1741631799 | Running |  1229 |          1200 |       NULL |     5426.6 | NULL    | NULL       >
|           78 | DBSUploadPoller                |   1741633613 | Running |  1111 |           100 |       NULL |      1.494 | NULL    | NULL       >
|           91 | DrainStatusPoller              |   1741633627 | Running |  1262 |          3600 |       NULL |    19.0053 | NULL    | NULL       >
|           85 | ErrorHandlerPoller             |   1741633618 | Running |  1185 |           240 |       NULL |     0.3769 | NULL    | NULL       >
|           79 | JobAccountantPoller            |   1741633614 | Running |  1125 |           300 |       NULL |      0.001 | NULL    | NULL       >
|           87 | JobArchiverPoller              |   1741633614 | Running |  1217 |           120 |       NULL |    129.399 | NULL    | NULL       >
|           80 | JobCreatorPoller               |   1741633590 | Running |  1134 |           120 |       NULL |     0.0009 | NULL    | NULL       >
|           81 | JobSubmitterPoller             |   1741633627 | Running |  1144 |           120 |       NULL |    283.856 | NULL    | NULL       >
|           82 | JobTrackerPoller               |   1741633604 | Running |  1156 |            60 |       NULL |     0.0015 | NULL    | NULL       >
|           84 | JobUpdaterPoller               |   1741631642 | Running |  1174 |           120 |       NULL |    4644.76 | NULL    | NULL       >
|           91 | ResourceControlUpdater         |   1741633603 | Running |  1262 |           900 |       NULL |    1091.36 | NULL    | NULL       >
|           86 | RetryManagerPoller             |   1741633513 | Running |  1198 |           240 |       NULL |     0.0038 | NULL    | NULL       >
|           92 | RucioInjectorPoller            |   1741633633 | Running |  1277 |           300 |       NULL |     8.2923 | NULL    | NULL       >
|           95 | StatusPoller                   |   1741633624 | Running | 35756 |            60 |       NULL |     8.7041 | NULL    | NULL       >
|           88 | TaskArchiverPoller             |   1741633588 | Running |  1229 |           240 |       NULL |     68.248 | NULL    | NULL       >
|           94 | WorkflowUpdaterPoller          |   1741633624 | Running |  2684 |         28800 |       NULL |     0.0078 | NULL    | NULL       >
|           77 | WorkQueueManagerCleaner        |   1741633599 | Running |  1102 |           180 |       NULL |    14.7266 | NULL    | NULL       >
|           77 | WorkQueueManagerLocationPoller |   1741633627 | Running |  1102 |          3600 |       NULL |      0.041 | NULL    | NULL       >
|           77 | WorkQueueManagerWMBSFileFeeder |   1741633595 | Running |  1102 |           180 |       NULL |     12.703 | NULL    | NULL       >
|           77 | WorkQueueManagerWorkPoller     |   1741633591 | Running |  1102 |           180 |       NULL |     1.0831 | NULL    | NULL       >
+--------------+--------------------------------+--------------+---------+-------+---------------+------------+------------+---------+------------>
```

But the entries report status of individual components rather their threads, e.g. 
```
# component info
ls /proc/1262/task
1262  1265  1266  1269

# component thread status
cat /proc/1262/task/1265/status | grep State
State:  S (sleeping)
```
author_association: CONTRIBUTOR
user: login: anpicci
created_at: 2025-04-07T14:20:26Z
body: @vkuznet @amaltaro , @mapellidario pointed out that by solving this issue, #12091 will also benefit from it.
What do you think? 
author_association: CONTRIBUTOR
user: login: vkuznet
created_at: 2025-04-13T17:18:31Z
body: > [@vkuznet](https://github.com/vkuznet) [@amaltaro](https://github.com/amaltaro) , [@mapellidario](https://github.com/mapellidario) pointed out that by solving this issue, [#12091](https://github.com/dmwm/WMCore/issues/12091) will also benefit from it. What do you think?

@anpicci , I think it is another way around, please read https://github.com/dmwm/WMCore/issues/12091#issuecomment-2800038963 where I pointed out that using function which I provided here can be used to kill all components threads along with its main process. Therefore, if you agree to merge this PR, then solving https://github.com/dmwm/WMCore/issues/12091 will be done by using `processThreadsInfo(pid)` function from this PR.
author_association: CONTRIBUTOR
user: login: amaltaro
created_at: 2025-05-09T17:42:43Z
body: After building a new release with these changes, either in 2.4.0rc1 or 2.4.0rc2, the results are not very positive. 
Please refer to the screenshots attached to this issue. I am reopening this issue.

![Image](https://github.com/user-attachments/assets/88b74ea7-f097-48be-ac8d-0ff85a291031)

<img width="481" alt="Image" src="https://github.com/user-attachments/assets/345b921b-f16e-4547-b7a8-1b671863aee9" />
author_association: CONTRIBUTOR
user: login: vkuznet
created_at: 2025-05-12T13:28:51Z
body: @amaltaro I introduced typo when switched to `extractFromXML` function upon your last review. The corresponding PR is here https://github.com/dmwm/WMCore/pull/12362
author_association: CONTRIBUTOR
user: login: vkuznet
created_at: 2025-05-13T13:32:45Z
body: I patched vocms266 with changes proposed in https://github.com/dmwm/WMCore/pull/12362, in particular it was two issues:
- one typo which I reported, and another
- mismatch if pid data-type used in codebase (in a code it is str while `psutils.Process` used it as `int`). The details can be found in https://github.com/dmwm/WMCore/pull/12362#issuecomment-2876534183

With these changes the vocms266 is up and running.
author_association: CONTRIBUTOR
user: login: amaltaro
created_at: 2025-05-20T15:20:32Z
body: As suspected, the WMStats monitoring issue (see the screenshot above with all components marked as red) has not been solved with the 2nd PR provided in: https://github.com/dmwm/WMCore/pull/12302

I upgraded vocms0192 to the latest version - released yesterday - and one can see the problem from there. Reopening this once again. 
author_association: CONTRIBUTOR
user: login: vkuznet
created_at: 2025-05-20T16:20:49Z
body: If I login to vocms0192 I can see that everything is running fine:
```
(WMAgent-2.4.0rc3) [cmst1@vocms0192:current]$ manage status
...
Component:WorkQueueManager 1095 running with threads [1096, 1097, 1098, 1099, 1100]
Component:DBS3Upload 1105 running with threads [1106, 1114]
Component:JobAccountant 1127 running with threads [1128, 1129]
Component:JobCreator 1136 running with threads [1137, 1138]
Component:JobSubmitter 1145 running with threads [1146, 1148]
Component:JobTracker 1160 running with threads [1161, 1163]
Component:JobStatusLite 1170 running with threads [1171, 1173]
Component:JobUpdater 1180 running with threads [1181, 1183]
Component:ErrorHandler 1207 running with threads [1226, 1234]
Component:RetryManager 1251 running with threads [1252, 1253]
Component:JobArchiver 1260 running with threads [1261, 1262]
Component:TaskArchiver 1270 running with threads [1271, 1272, 1273]
Component:AnalyticsDataCollector 1299 running with threads [1307, 1308]
Component:ArchiveDataReporter 1318 running with threads [1320, 1321]
Component:AgentStatusWatcher 1328 running with threads [1329, 1332, 1333, 1334]
Component:RucioInjector 1393 running with threads [1417, 1418]
Component:WorkflowUpdater 1443 running with threads [1448, 1449, 1450]
```
And, I can confirm that if I look at individual components, e.g. DBS3Upload:
```
ls /proc/1105/task/
1105  1106  1114

# and then look into individual pid or thread id
(WMAgent-2.4.0rc3) [cmst1@vocms0192:current]$ grep -i state /proc/1105/task/1105/status
State:  S (sleeping)
(WMAgent-2.4.0rc3) [cmst1@vocms0192:current]$ grep -i state /proc/1106/task/1106/status
State:  S (sleeping)
```
But according to WMStats it lists:
```
DBS3Upload
worker thread: null
status: Lost threads
last updated: 2025/5/20 (Tue) 16:13:35 UTC
pid: 1105
error message:
Lost threads: [{'process': 'python', 'pid': '1105', 'status': 'sleeping', 'type': 'process'}, {'process': 'python-thread', 'pid': '1106', 'status': 'running', 'type': 'thread'}, {'process': 'python-thread', 'pid': '1114', 'status': 'running', 'type': 'thread'}]
```
which interprets threads as lost. So, we need to dig down to wmstats update which report this as down process threads. I'll have a look further.
author_association: CONTRIBUTOR
user: login: vkuznet
created_at: 2025-05-20T16:44:07Z
body: @amaltaro , I provided new fix in https://github.com/dmwm/WMCore/pull/12372. The issue with WMStats report that we switched along the review process from `procfs` I used in testing to use `psutils` which use slightly different status values, `S (sleeping)` vs `sleeping` and `R (running)` vs `running` which we check in `DataCollectAPI.py`. I identified the difference by running directly code used in `DataCollectAPI.py` and printout the state of the processes. You may try to patch vocms0192 with proposed changes to see if issue will be resolved in WMStats reporting.
author_association: CONTRIBUTOR
user: login: amaltaro
created_at: 2025-05-20T20:03:34Z
body: Just so everything is reported in the issue ticket instead of getting lost in the PR comments, there are 2 more issues (or 1) that were overlooked, as reported in:
https://github.com/dmwm/WMCore/pull/12372#issuecomment-2895672013
and
https://github.com/dmwm/WMCore/pull/12372#issuecomment-2895689537
author_association: CONTRIBUTOR
user: login: amaltaro
created_at: 2025-07-16T12:38:22Z
body: The submit10 node has been showing false-alarms for AgentStatusWatcher for the last couple of days.

Looking into the message provided by WMStats, we can observe:
* worker thread is `null`. Shouldn't it contain the name of the thread that went down?
* It reports "Lost threads", but the dictionary contains `status=running`, which look inconsistent as well.

I am also attaching a screenshot for what is reported in WMStats.

Additionally, agent `manage status` does not report any problems with that component:
```
Component:AgentStatusWatcher 26095 running with threads [26098, 26099, 26106]
```

Finally, I do not see any worker threads terminated in the AgentStatusWatcher log file. I have already restarted this component once, but the problem persists.

Interestingly, the other agents running on the same version `2.4.1` are not having such problems.

@vkuznet can you please look into this once you are back? FYI @anpicci 

<img width="723" height="256" alt="Image" src="https://github.com/user-attachments/assets/00e2e834-df3c-4a60-a106-f02c50f7b5e8" />
author_association: CONTRIBUTOR
user: login: amaltaro
created_at: 2025-07-17T13:19:59Z
body: After stopping, restarting and waiting for the containers and components on submit10, we have yet another false alarm.

WMStats reports (I waited for multiple refreshes to make sure it was not transient):
```
agent: cmsgwms-submit10.fnal.gov (2.4.1)
agent last updated: 2025/7/17 (Thu) 13:09:04 UTC : 0 h 4 m
data last updated: 0 h 6 m
status: Draining Agent; Components or Thread down;
team: production
component errors for: WorkQueueManager
undefined

    worker thread: undefined
    status: undefined
    last updated: NaN/NaN/NaN (undefined) NaN:NaN:NaN UTC
    pid: undefined
    error message:    undefined
```

Then checking the agent, the manage script reports some lost threads for `WorkQueueManager`:
```
$ manage status
...
Component:WorkQueueManager 1161 partially-running with threads [], lost [1177, 1178, 1179, 1180] threads
```

while the threads.json content is:
```
$ cat install/WorkQueueManager/threads.json 
[{"process": "python", "pid": "1161", "status": "sleeping", "type": "process"}]
```

then the Daemon.xml file reports:
```
$ cat install/WorkQueueManager/Daemon.xml
<Daemon>
	<ProcessID Value="1161"/>
	<ParentProcessID Value="1160"/>
	<ProcessGroupID Value="1160"/>
	<UserID Value="42935"/>
	<EffectiveUserID Value="42935"/>
	<GroupID Value="5063"/>
	<EffectiveGroupID Value="5063"/>
</Daemon>
```

then checking this process ID and the parent ID, we have:
```
$ ps -Tfww -p 1160
UID          PID    SPID    PPID  C STIME TTY          TIME CMD
```
and
```
$ ps -Tfww -p 1161
UID          PID    SPID    PPID  C STIME TTY          TIME CMD
cmsdata+    1161    1161       1  0 10:45 ?        00:00:07 python /usr/local/bin/wmcoreD --start --config=/data/srv/wmagent/2.4.1/config/config.py
cmsdata+    1161    1177       1  0 10:45 ?        00:00:01 python /usr/local/bin/wmcoreD --start --config=/data/srv/wmagent/2.4.1/config/config.py
cmsdata+    1161    1178       1  0 10:45 ?        00:00:01 python /usr/local/bin/wmcoreD --start --config=/data/srv/wmagent/2.4.1/config/config.py
cmsdata+    1161    1179       1  0 10:45 ?        00:00:00 python /usr/local/bin/wmcoreD --start --config=/data/srv/wmagent/2.4.1/config/config.py
cmsdata+    1161    1180       1  0 10:45 ?        00:00:03 python /usr/local/bin/wmcoreD --start --config=/data/srv/wmagent/2.4.1/config/config.py
```

lastly, tailing/grepping the component, I do see that worker threads are running and have actually recently (few minutes) completed a cycle:
```
$ uptime
 13:13:26 up 20:27,  0 user,  load average: 0.23, 0.32, 0.24

$ tail -n10000 install/WorkQueueManager/ComponentLog | grep 'secs to execute' | tail -n 5
2025-07-17 13:08:19,173:140475363227328:INFO:BaseWorkerThread:WorkQueueManagerWorkPoller took 1.019 secs to execute
2025-07-17 13:09:02,384:140475338049216:INFO:BaseWorkerThread:WorkQueueManagerCleaner took 2.207 secs to execute
2025-07-17 13:11:02,702:140475354834624:INFO:BaseWorkerThread:WorkQueueManagerWMBSFileFeeder took 0.641 secs to execute
2025-07-17 13:11:20,885:140475363227328:INFO:BaseWorkerThread:WorkQueueManagerWorkPoller took 1.038 secs to execute
2025-07-17 13:12:05,282:140475338049216:INFO:BaseWorkerThread:WorkQueueManagerCleaner took 2.216 secs to execute
```

I am afraid this will require deeper investigation and proper fixes to be applied to avoid the false alarms that we wanted to resolve with this development. Reopening this ticket.
author_association: CONTRIBUTOR
user: login: amaltaro
created_at: 2025-07-24T08:48:59Z
body: In addition to the two posts above, I see vocms0192 is reporting a (false) problem right now:
```
Component:AgentStatusWatcher 1133 partially-running with threads [1134, 1137, 1138], lost [1144] threads
```

which might be useful for debugging as well.
author_association: CONTRIBUTOR
user: login: vkuznet
created_at: 2025-07-24T13:18:22Z
body: > In addition to the two posts above, I see vocms0192 is reporting a (false) problem right now:
> 
> ```
> Component:AgentStatusWatcher 1133 partially-running with threads [1134, 1137, 1138], lost [1144] threads
> ```
> 
> which might be useful for debugging as well.

Alan, thanks for reporting this. Since this agent component is still running I logged to the node and inspected its stack. To my surprise the report is correct since according to Linux filesystem the process with PID=1133 has the following info in `/proc` file system:
```
ls /proc/1133/task
1133  1134  1137  1138  1144
```
As you can see the pid `1144` is listed among other threads. I also composed stripped version of `wmcoreD` and confirmed that code correctly reports it:
```
import psutil
import os
import json

pid=1133


mypid=os.getpid()
compDir="/data/srv/wmagent/current/install/AgentStatusWatcher/"

jsonFile = os.path.join(compDir, "threads.json")
with open(jsonFile, "r", encoding="utf-8") as istream:
  data = json.load(istream)

# Extract process and its threads
threadPids = []

for entry in data:
  if str(entry["pid"]) == str(pid) and entry["type"] == "process":
      continue
  elif entry["type"] == "thread":
      threadPids.append(entry["pid"])

print("thread pids", threadPids)
print("this python pid", mypid)

process = psutil.Process(int(pid))
for thread in process.threads():
    print("process thread", thread)
    if str(thread.id) == str(pid):
        continue  # skip pid thread
    if str(thread.id) in threadPids:
        print("running thread", pid, thread.id)
    else:
        print("orphan thread", pid, thread.id)
```
And this code shows me the following output:
```
this python pid 3191388
process thread pthread(id=1133, user_time=0.33, system_time=0.1)
process thread pthread(id=1134, user_time=21.08, system_time=23.67)
running thread 1133 1134
process thread pthread(id=1137, user_time=858.45, system_time=143.39)
running thread 1133 1137
process thread pthread(id=1138, user_time=520.63, system_time=68.84)
running thread 1133 1138
process thread pthread(id=1144, user_time=404.51, system_time=40.3)
orphan thread 1133 1144
```
Therefore, I conclude that both psutils and `/proc` file system reports consistently that main process `1133` (the AgentStatusWatcher component) currently running on a system indeed has thread `1144` which appears both in `/proc` file system and reported by `psutils`. Moreover, this pid has the following content in `/proc` file system:
```
ls /proc/1133/task/1144/
arch_status  clear_refs          cwd      gid_map            loginuid   net        oom_score_adj  root       smaps         status
attr         cmdline             environ  io                 maps       ns         pagemap        sched      smaps_rollup  syscall
auxv         comm                exe      ksm_merging_pages  mem        numa_maps  patch_state    schedstat  stack         uid_map
cgroup       cpu_resctrl_groups  fd       ksm_stat           mountinfo  oom_adj    personality    sessionid  stat          wchan
children     cpuset              fdinfo   limits             mounts     oom_score  projid_map     setgroups  statm
```
and this thread has proper parent (process 1133, the AgentStatusWatcher)
```
head /proc/1133/task/1144/status
Name:   python
Umask:  0022
State:  S (sleeping)
Tgid:   1133
Ngid:   0
Pid:    1144
PPid:   1
TracerPid:      0
```

Therefore, by inspecting psutils output and `/proc` file system I conclude that code correctly reports it. It is also confirmed with `ps` output:
```
ps -Tfww -p 1133
UID          PID    SPID    PPID  C STIME TTY          TIME CMD
cmst1       1133    1133       1  0 Jun18 ?        00:35:34 python /usr/local/bin/wmcoreD --start --config=/data/srv/wmagent/2.4.1/config/config.py
cmst1       1133    1134       1  0 Jun18 ?        00:00:44 python /usr/local/bin/wmcoreD --start --config=/data/srv/wmagent/2.4.1/config/config.py
cmst1       1133    1137       1  0 Jun18 ?        00:16:42 python /usr/local/bin/wmcoreD --start --config=/data/srv/wmagent/2.4.1/config/config.py
cmst1       1133    1138       1  0 Jun18 ?        00:09:49 python /usr/local/bin/wmcoreD --start --config=/data/srv/wmagent/2.4.1/config/config.py
cmst1       1133    1144       1  0 Jun18 ?        00:07:24 python /usr/local/bin/wmcoreD --start --config=/data/srv/wmagent/2.4.1/config/config.py
```
And, it is not shown in `threads.json` file:
```
cat /data/srv/wmagent/current/install/AgentStatusWatcher/threads.json  | jq
[
  {
    "process": "python",
    "pid": "1133",
    "status": "sleeping",
    "type": "process"
  },
  {
    "process": "python-thread",
    "pid": "1134",
    "status": "running",
    "type": "thread"
  },
  {
    "process": "python-thread",
    "pid": "1137",
    "status": "running",
    "type": "thread"
  },
  {
    "process": "python-thread",
    "pid": "1138",
    "status": "running",
    "type": "thread"
  }
]
```

The only explanation I have for such behavior is that this thread was not started initially and was spanned after component has started. Is it possible within current code? In other words do we start new threads after some delay or restart a thread internally?

In other words, the code behavior is correct, i.e. at the time we started the component the thread with pid `1144` was not present. At the time we evaluate the component this thread appears in component threads stack. And, the question is how we get new thread after component startup.
author_association: CONTRIBUTOR
user: login: amaltaro
created_at: 2025-07-24T13:56:09Z
body: `AgentStatusWatcher` is the main thread that spawns the other threads, as can be seen in this module:
https://github.com/dmwm/WMCore/blob/master/src/python/WMComponent/AgentStatusWatcher/AgentStatusWatcher.py

I don't know if it makes any sense to have it "down", as that would imply that the other threads would be down as well.

And yes, all the worker threads are spawned during the component initialization, which corresponds to the following section of the log:
```
2025-06-18 12:30:56,876:140147361467456:INFO:Harness:>>>Starting: AgentStatusWatcher<<<
2025-06-18 12:30:56,877:140147361467456:INFO:Harness:>>>Initializing default database
2025-06-18 12:30:56,877:140147361467456:INFO:Harness:>>>Check if connection is through socket
2025-06-18 12:30:56,877:140147361467456:INFO:Harness:>>>Setting config for thread: 
2025-06-18 12:30:56,877:140147361467456:INFO:Harness:>>>Building database connection string
2025-06-18 12:30:56,880:140147361467456:INFO:DBCore:Instantiating base WM DBInterface
2025-06-18 12:30:56,880:140147361467456:INFO:WorkerThreadManager:Started
2025-06-18 12:30:56,880:140147361467456:INFO:WorkerThreadManager:All worker threads paused
2025-06-18 12:30:56,881:140147361467456:INFO:Harness:>>>Initialize transaction dictionary
2025-06-18 12:30:56,881:140147361467456:INFO:Harness:Harness part constructor finished
2025-06-18 12:30:56,881:140147361467456:INFO:Harness:>>>Registering Component - AgentStatusWatcher
2025-06-18 12:30:57,235:140147361467456:INFO:Harness:>>>Starting initialization
2025-06-18 12:30:57,235:140147361467456:INFO:Harness:>>>Setting default transaction
2025-06-18 12:30:57,235:140147361467456:INFO:AgentStatusWatcher:AgentStatusWatcher.preInitialization
2025-06-18 12:30:57,235:140147361467456:INFO:AgentStatusWatcher:Setting AgentStatusPoller poll interval to 300 seconds
2025-06-18 12:30:57,240:140147361467456:INFO:WorkerThreadManager:Created worker thread <WMComponent.AgentStatusWatcher.AgentStatusPoller.AgentStatusPoller object at 0x7f7695f0ac00>
2025-06-18 12:30:57,241:140146934544064:INFO:BaseWorkerThread:Initialising worker thread <WMComponent.AgentStatusWatcher.AgentStatusPoller.AgentStatusPoller object at 0x7f7695f0ac00>
2025-06-18 12:30:57,241:140147361467456:INFO:AgentStatusWatcher:Setting ResourceControlUpdater poll interval to 900 seconds
2025-06-18 12:30:57,241:140146934544064:INFO:BaseWorkerThread:Initialising default database
2025-06-18 12:30:57,242:140147361467456:INFO:Grafana:MonIT service initialized with parameters: {'endpoint': 'https://monit-grafana.cern.ch', 'cacheduration': 0, 'headers': {'Accept': 'application/json', 'Content-Type': 'application/json'}}
2025-06-18 12:30:57,242:140146934544064:INFO:DBCore:Instantiating base WM DBInterface
2025-06-18 12:30:57,242:140146934544064:INFO:BaseWorkerThread:Initialising default transaction
2025-06-18 12:30:57,243:140147361467456:INFO:WorkerThreadManager:Created worker thread <WMComponent.AgentStatusWatcher.ResourceControlUpdater.ResourceControlUpdater object at 0x7f7695e6a3f0>
2025-06-18 12:30:57,244:140146926151360:INFO:BaseWorkerThread:Initialising worker thread <WMComponent.AgentStatusWatcher.ResourceControlUpdater.ResourceControlUpdater object at 0x7f7695e6a3f0>
2025-06-18 12:30:57,244:140147361467456:INFO:AgentStatusWatcher:Setting DrainStatusPoller poll interval to 3600 seconds
2025-06-18 12:30:57,244:140146926151360:INFO:BaseWorkerThread:Initialising default database
2025-06-18 12:30:57,891:140146926151360:INFO:DBCore:Instantiating base WM DBInterface
2025-06-18 12:30:57,891:140146926151360:INFO:BaseWorkerThread:Initialising default transaction
2025-06-18 12:30:57,892:140147361467456:INFO:WorkerThreadManager:Created worker thread <WMComponent.AgentStatusWatcher.DrainStatusPoller.DrainStatusPoller object at 0x7f7695e7f950>
2025-06-18 12:30:57,893:140146934544064:INFO:LogDB:<LogDB(url=https://cmsweb-testbed.cern.ch/couchdb/wmstats_logdb, identifier=vocms0192.cern.ch, agent=1)>
2025-06-18 12:30:57,893:140146917758656:INFO:BaseWorkerThread:Initialising worker thread <WMComponent.AgentStatusWatcher.DrainStatusPoller.DrainStatusPoller object at 0x7f7695e7f950>
```
author_association: CONTRIBUTOR
user: login: vkuznet
created_at: 2025-07-24T14:31:24Z
body: The `threads.json` file is created by the following chain of code:
- the `wmcoreD` creates new `threads.json` file and put content in it from `processStatus(pid)` function, see https://github.com/dmwm/WMCore/blob/master/bin/wmcoreD#L238-L244
- the `processStatus(pid)` function is defined in `Utils/ProcFS.py` module, see https://github.com/dmwm/WMCore/blob/master/src/python/Utils/ProcFS.py
- and we use psutils threads to extract all process thread pids, see https://github.com/dmwm/WMCore/blob/master/src/python/Utils/ProcFS.py#L31

Therefore, if `threads.json` does not contain one thread `1144` the question remains how it is created? If we believe to psutils it did not exist at startup of the component, otherwise it will be visible as others threads. Since the log output does not show thread PIDs we can't conclude if it was there or not during startup of the component. Also, since psutils captured all other threads I tend to believe that this particular thread was not there at startup, and somehow was created later after startup of the component. What we can do to improve the situation is only dump thread pid within component constructor and then observe it in a log and compare to reported ones by manage script.

Any other suggestions?
author_association: CONTRIBUTOR
user: login: amaltaro
created_at: 2025-07-24T15:25:24Z
body: Valentin, I have no plans to dive into the implementation of this feature. However, I wanted to emphasize that the goal was to:
* detect when a component is down
* detect when worker threads are down

such that people can properly take actions on parts of the system that are either down or stuck. To me, it looks like we are not properly tracking the component process/thread correctly, as I see no information for it in the Daemon.xml file 
```
Component:AgentStatusWatcher 1133 partially-running with threads [1134, 1137, 1138], lost [1144] threads
Component:RucioInjector 1143 running with threads [1193, 1199]
Component:WorkflowUpdater 1233 running with threads [1254, 1256, 1257]
----------------------------------------------------------------------
$ cat install/AgentStatusWatcher/Daemon.xml 
<Daemon>
	<ProcessID Value="1133"/>
	<ParentProcessID Value="1"/>
	<ProcessGroupID Value="1132"/>
	<UserID Value="31961"/>
	<EffectiveUserID Value="31961"/>
	<GroupID Value="1399"/>
	<EffectiveGroupID Value="1399"/>
</Daemon>
```
author_association: CONTRIBUTOR
user: login: vkuznet
created_at: 2025-07-24T15:41:51Z
body: The issue here is not about detection which is working as expected but rather about reporting of all threads which is  what is leading to false alarm. In other words our mechanism of reporting all threads associated with particular component is not working properly but the mechanism of detecting down threads seems to be working just fine. This is why I'm asking questions how threads are created. To me there is something in a code which literally creates a thread after we capture initial state of the component. And, this is what I'm trying to understand.

My conclusion stands that there is nothing wrong with initial issue of reporting and detecting misbehavior in threads of the process. Instead, we have a different problem of not reporting all threads initially (or if threads are created post startup phase of the component). I would appreciate if @amaltaro and @anpicci evaluate this observation and decide should we create new issue for this or continue working within this thread (which again was about reporting which is working as expected).
author_association: CONTRIBUTOR

Diff:
diff --git a/bin/wmcoreD b/bin/wmcoreD
index 40c773adb5..4ebdcd60d2 100755
--- a/bin/wmcoreD
+++ b/bin/wmcoreD
@@ -35,12 +35,15 @@ __version__ = "$Revision: 1.16 $"
 __author__ = "fvlingen@caltech.edu"
 
 import getopt
-import logging
 import os
 import subprocess
 import sys
 import time
+import json
+import psutil
 
+from Utils.Utilities import extractFromXML
+from Utils.ProcFS import processStatus
 from WMCore.Agent.Daemon.Details import Details
 from WMCore.Configuration import loadConfigurationFile
 from WMCore.WMFactory import WMFactory
@@ -177,7 +180,7 @@ def connectionTest(configFile):
         print("skipped.")
         return
 
-    (dialect, junk) = config.CoreDatabase.connectUrl.split(":", 1)
+    dialect, _ = config.CoreDatabase.connectUrl.split(":", 1)
     socket = getattr(config.CoreDatabase, "socket", None)
 
     try:
@@ -235,6 +238,15 @@ def startup(configFile):
         else:
             print('Path for daemon file does not exist!')
             sys.exit(1)
+        # write into component area process status information
+        cpath = os.path.join(compDir, "threads.json")
+        if os.path.exists(cpath):
+            os.remove(cpath)
+        cpid = extractFromXML(daemonXML, "ProcessID")
+        with open(cpath, 'w', encoding="utf-8") as istream:
+            procStatus = processStatus(cpid)
+            istream.write(json.dumps(procStatus))
+            print("Component %s started with %s threads, see %s\n" % (component, len(procStatus), cpath))
 
     return
     
@@ -274,6 +286,10 @@ def shutdown(configFile):
                 daemon.removeAndBackupDaemonFile()
             else:
                 daemon.killWithPrejudice()
+        # remove component threads.json file
+        cpath = os.path.join(compDir, "threads.json")
+        if os.path.exists(cpath):
+            os.remove(cpath)
         if doLogCleanup:
             #  //
             # // Log Cleanup
@@ -295,7 +311,7 @@ def shutdown(configFile):
             exitCode = subprocess.call(["rm", "-rf", "%s" % compDir])
             if exitCode:
                 msg = "Failed to clean up dir: %s\n" % compDir
-                msg += output
+                msg += f"with exit code {exitCode}"
                 print(msg)
 
     return
@@ -320,18 +336,62 @@ def status(configFile):
             return
         compDir = config.section_(component).componentDir
         compDir = os.path.expandvars(compDir)
-        daemonXml = os.path.join(compDir, "Daemon.xml")
-        if not os.path.exists(daemonXml):
-            print("Component:%s Not Running" % component)
-            continue
-        daemon = Details(daemonXml)
-        if not daemon.isAlive():
-            print("Component:%s Not Running" % component)
-        else:
-            print("Component:%s Running:%s" % (component, daemon['ProcessID']))
+        checkProcessThreads(component, compDir)
 
     sys.exit(0)
 
+def checkProcessThreads(component, compDir):
+    """
+    Helper function to check process and its threads for their statuses
+    :param component: component name
+    :return: prints status of the component process and its threads
+    """
+    # check if component daemon exists
+    daemonXml = os.path.join(compDir, "Daemon.xml")
+    if not os.path.exists(daemonXml):
+        print("Component:%s Not Running" % component)
+        return
+    pid = extractFromXML(daemonXML, "ProcessID")
+
+    jsonFile = os.path.join(compDir, "threads.json")
+    with open(jsonFile, "r", encoding="utf-8") as istream:
+        data = json.load(istream)
+
+    # Extract process and its threads
+    threadPids = []
+
+    for entry in data:
+        if str(entry["pid"]) == str(pid) and entry["type"] == "process":
+            continue
+        elif entry["type"] == "thread":
+            threadPids.append(entry["pid"])
+
+    # Check if process is running
+    processRunning = psutil.pid_exists(int(pid))
+
+    # Check if threads are running
+    runningThreads = []
+    orphanThreads = []
+    process = psutil.Process(int(pid))
+    for thread in process.threads():
+        if str(thread.id) == str(pid):
+            continue  # skip pid thread
+        if str(thread.id) in threadPids:
+            runningThreads.append(thread.id)
+        else:
+            orphanThreads.append(thread.id)
+
+    # Output result
+    status = "running" if processRunning else "not-running"
+    msg = f"Component:{component} {pid} {status} with threads {runningThreads}"
+    if status == "running":
+        if len(orphanThreads) > 0:
+            status = "partially-running" if processRunning else "not-running"
+            msg = f"Component:{component} {pid} {status} with threads {runningThreads}, lost {orphanThreads} threads"
+    else:
+        msg = f"Component:{component} {pid} {status}"
+    print(msg)
+
 def restart(config):
     """
     _restart_
diff --git a/src/python/Utils/ProcFS.py b/src/python/Utils/ProcFS.py
new file mode 100644
index 0000000000..c5fc798981
--- /dev/null
+++ b/src/python/Utils/ProcFS.py
@@ -0,0 +1,112 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+"""
+File       : ProcFS.py
+Author     : Valentin Kuznetsov <vkuznet AT gmail dot com>
+Description: This module provides function which rely on Linux proc fs
+"""
+# system modules
+import os
+
+# 3rd party modules
+try:
+    import psutil
+    usePsutil = True
+except ImportError:
+    usePsutil = False
+
+
+def processStatus(pid, method=None):
+    """
+    Return status of the given process PID along with its threads using psutil.
+    :param pid: Process ID to inspect
+    :param method: define which method to use procfs or psutil
+    :return: list of dictionaries of `{process, pid, status}` data-structure
+    """
+    if method == 'procfs' or usePsutil == False:
+        return processStatusViaProcFS(pid)
+    return processStatusViaPsutil(pid)
+
+
+def processStatusViaPsutil(pid):
+    """
+    Return status of the given process PID along with its threads using psutil.
+    :param pid: Process ID to inspect
+    :return: list of dictionaries with {process, pid, status, type}
+    """
+    statusList = []
+
+    try:
+        proc = psutil.Process(pid)
+        processName = proc.name()
+        processStatus = proc.status()
+        statusList.append({
+            "process": processName,
+            "pid": str(pid),
+            "status": processStatus,
+            "type": "process"
+        })
+
+        # Iterate over threads
+        threads = proc.threads()
+        for thread in threads:
+            if str(pid) == str(thread.id):
+                continue
+            threadId = thread.id
+            # psutil doesn't provide thread name or detailed state, so we approximate
+            statusList.append({
+                "process": f"{processName}-thread",
+                "pid": str(threadId),
+                "status": "running",  # threads are assumed running if present
+                "type": "thread"
+            })
+
+    except psutil.NoSuchProcess:
+        return [{"error": f"Process {pid} not found"}]
+    except Exception as e:
+        return [{"error": str(e)}]
+
+    return statusList
+
+
+def processStatusViaProcFS(pid):
+    """
+    Return status of given process PID along with its threads via /proc FS look-up (available on all Linux OSes)
+    :return: list of dictionaries of `{process, pid, status}` data-structure
+    """
+    statusList = []
+    procPath = f"/proc/{pid}"
+
+    if not os.path.exists(procPath):
+        return [{"error": f"Process {pid} not found"}]
+
+    try:
+        with open(f"{procPath}/status", encoding='utf-8') as f:
+            processName = ""
+            processState = ""
+            for line in f:
+                if line.startswith("Name:"):
+                    processName = line.split(":")[1].strip()
+                elif line.startswith("State:"):
+                    processState = line.split(":")[1].strip()
+            statusList.append({"process": processName, "pid": str(pid), "status": processState, "type": "process"})
+
+        taskPath = f"{procPath}/task"
+        if os.path.exists(taskPath):
+            for threadId in os.listdir(taskPath):
+                if str(threadId) == str(pid):
+                    continue
+                with open(f"{taskPath}/{threadId}/status", encoding='utf-8') as f:
+                    threadName = ""
+                    threadState = ""
+                    for line in f:
+                        if line.startswith("Name:"):
+                            threadName = line.split(":")[1].strip()
+                        elif line.startswith("State:"):
+                            threadState = line.split(":")[1].strip()
+                    statusList.append({"process": threadName, "pid": threadId, "status": threadState, "type": "thread"})
+
+    except Exception as e:
+        return [{"error": str(e)}]
+
+    return statusList
diff --git a/src/python/Utils/ProcessStats.py b/src/python/Utils/ProcessStats.py
index 80a7233111..ec6e35b733 100755
--- a/src/python/Utils/ProcessStats.py
+++ b/src/python/Utils/ProcessStats.py
@@ -89,6 +89,90 @@ def threadStack():
         threads.append(tdict)
     return dict(threads=threads)
 
+def processThreadsInfo(pid):
+    """
+    Provides information about process and its threads using psutils
+    :param pid: process PID, string
+    :return: dictionary statistics about process and threads
+    """
+    try:
+        process = psutil.Process(pid)
+
+        # Get overall process details
+        process_info = {
+            "process_name": process.name(),
+            "pid": process.pid,
+            "status": process.status(),
+            "ppid": process.ppid(),  # Parent Process ID
+            "cmdline": process.cmdline(),  # Full command-line arguments
+            "cpu_usage_percent": process.cpu_percent(interval=1.0),  # CPU usage in percentage
+            "memory_usage_percent": process.memory_percent(),  # RAM usage in percentage
+            "memory_rss": process.memory_info().rss,  # Resident Set Size (bytes)
+            "num_open_files": len(process.open_files()),  # Number of open file descriptors
+            "num_connections": len(process.connections()),  # Number of active connections
+            "threads": []
+        }
+
+        # Iterate over threads to get per-thread details
+        for thread in process.threads():
+            if str(pid) == str(thread.id):
+                continue
+            thread_id = thread.id
+
+            thread_info = {
+                "thread_id": thread_id,
+                "user_time": thread.user_time,  # CPU time spent in user mode
+                "system_time": thread.system_time,  # CPU time spent in system mode
+                "cpu_usage_percent": None,
+                "memory_usage_bytes": None,
+                "num_open_files": None,
+                "num_connections": None,
+                "state": "unknown",
+                "name": "thread"
+            }
+            if str(pid) == str(thread_id):
+                thread_info['name'] = "process"
+
+            # Try getting thread status
+            try:
+                thread_status = psutil.Process(thread_id).status()
+                thread_info["state"] = thread_status
+            except psutil.NoSuchProcess:
+                thread_info["state"] = "zombie"
+
+            # Try getting per-thread CPU usage
+            try:
+                thread_info["cpu_usage_percent"] = psutil.Process(thread_id).cpu_percent(interval=1.0)
+            except Exception:
+                thread_info["cpu_usage_percent"] = "N/A"
+
+            # Try getting per-thread memory usage
+            try:
+                thread_info["memory_usage_bytes"] = psutil.Process(thread_id).memory_info().rss
+            except Exception:
+                thread_info["memory_usage_bytes"] = "N/A"
+
+            # Try getting per-thread open file descriptors
+            try:
+                thread_info["num_open_files"] = len(psutil.Process(thread_id).open_files())
+            except Exception:
+                thread_info["num_open_files"] = "N/A"
+
+            # Try getting per-thread open connections
+            try:
+                thread_info["num_connections"] = len(psutil.Process(thread_id).connections())
+            except Exception:
+                thread_info["num_connections"] = "N/A"
+
+            # Append to process_info
+            process_info["threads"].append(thread_info)
+
+        return process_info
+    except psutil.NoSuchProcess:
+        return {"error": f"No process found with PID {pid}"}
+    except Exception as e:
+        return {"error": str(e)}
+
 def main():
     "Main function to use this module as a stand-along script."
     parser = argparse.ArgumentParser(prog='PROG')
@@ -97,7 +181,12 @@ def main():
 
     pdict = processStatus(int(opts.pid))
     pdict.update(threadStack())
-    print(json.dumps(pdict))
+    print(f"Process status for {opts.pid}")
+    print(json.dumps(pdict, indent=4))
+
+    processInfo = processThreadsInfo(int(opts.pid))
+    print(f"Process/threads status for {opts.pid}")
+    print(json.dumps(processInfo, indent=4))
 
 if __name__ == '__main__':
     main()
diff --git a/src/python/Utils/Utilities.py b/src/python/Utils/Utilities.py
index ef8b704d9e..fd2212c066 100644
--- a/src/python/Utils/Utilities.py
+++ b/src/python/Utils/Utilities.py
@@ -11,6 +11,16 @@
 from types import ModuleType, FunctionType
 from gc import get_referents
 
+import xml.etree.ElementTree as ET
+
+def extractFromXML(xmlFile, xmlElement):
+    tree = ET.parse(xmlFile)
+    root = tree.getroot()
+    element = root.find(f".//{xmlElement}")
+    if element is not None:
+        return element.get("Value")
+    return None
+
 def lowerCmsHeaders(headers):
     """
     Lower CMS headers in provided header's dict. The WMCore Authentication
diff --git a/src/python/WMComponent/AnalyticsDataCollector/DataCollectAPI.py b/src/python/WMComponent/AnalyticsDataCollector/DataCollectAPI.py
index 27e92016d1..b6a594cbcd 100644
--- a/src/python/WMComponent/AnalyticsDataCollector/DataCollectAPI.py
+++ b/src/python/WMComponent/AnalyticsDataCollector/DataCollectAPI.py
@@ -7,6 +7,7 @@
 
 import os
 import time
+import json
 import logging
 
 import WMCore
@@ -19,6 +20,34 @@
 from WMComponent.AnalyticsDataCollector.DataCollectorEmulatorSwitch import emulatorHook
 
 
+def threadsDetails(component, pid, downProcessThreads):
+    """
+    Helper function to provide information about down component in dictionary
+    data-format used by agentInfo
+    :param component: name of the component
+    :param pid: pid of the component process
+    :param downProcessThreads: is a list of process thread dictionaries
+    :return: dictionary used by down_component_detail part of agentInfo
+
+    NOTE: we preserve data-structure used in down_component_detail which comes from
+    WMCore/Agent/Database/MySQL/GetAllHeartbeatInfo.py SQL query and only fill
+    out necessary information about component threads and nothing else.
+    """
+    data = {
+            "name": component,
+            "pid": pid,
+            "worker_name": None,
+            "state": "Lost threads",
+            "last_updated": int(time.time()),
+            "update_threshold": None,
+            "poll_interval": None,
+            "cycle_time": None,
+            "outcome": None,
+            "last_error": None,
+            "error_message": f"Lost threads: {downProcessThreads}"
+          }
+    return data
+
 @emulatorHook
 class LocalCouchDBData(object):
     def __init__(self, couchURL, statSummaryDB, summaryLevel):
@@ -248,6 +277,7 @@ def getComponentStatus(self, config):
 
         components = config.listComponents_() + config.listWebapps_()
         # check the component status
+        agentComponents = {}
         for component in components:
             compDir = config.section_(component).componentDir
             compDir = os.path.expandvars(compDir)
@@ -259,11 +289,38 @@ def getComponentStatus(self, config):
                 daemon = Details(daemonXml)
                 if not daemon.isAlive():
                     downFlag = True
+                # add individual component thread status
+                compName = compDir.split('/')[-1]
+                compProcessStatus = daemon.processStatus()
+                agentComponents.update({compName: compProcessStatus})
+                # check if number of component threads is equal to initial set
+                cpath = os.path.join(compDir, "threads.json")
+                downProcessThreads = []
+                if os.path.exists(cpath):
+                    origThreads = []
+                    with open(cpath, 'r', encoding='utf-8') as istream:
+                        origThreads = json.load(istream)
+                    if len(origThreads) != len(compProcessStatus):
+                        downFlag = True
+                        for proc in origThreads:
+                            if proc not in compProcessStatus:
+                                downProcessThreads.append(proc)
+                # check if all component process' threads are alive, otherwise set down flag
+                for proc in compProcessStatus:
+                    # the alive process should be either in sleeping or running states
+                    if proc['status'] not in ["S (sleeping)", "R (running)"]:
+                        downFlag = True
+                        downProcessThreads.append(proc)
             if downFlag and component not in agentInfo['down_components']:
                 agentInfo['status'] = 'down'
                 agentInfo['down_components'].append(component)
-                agentInfo['down_component_detail'].append(component)
+                if len(downProcessThreads) > 0:
+                    agentInfo['down_component_detail'].append(
+                            threadsDetails(component, daemon['ProcessID'], downProcessThreads))
+                else:
+                    agentInfo['down_component_detail'].append(component)
 
+        agentInfo['components'] = agentComponents
         agentInfo['workers'] = self.listWorkers.execute()
         return agentInfo
 
diff --git a/src/python/WMCore/Agent/Daemon/Details.py b/src/python/WMCore/Agent/Daemon/Details.py
index 8153e70a38..5f8d409862 100644
--- a/src/python/WMCore/Agent/Daemon/Details.py
+++ b/src/python/WMCore/Agent/Daemon/Details.py
@@ -21,6 +21,7 @@
 
 from Utils.PythonVersion import PY3
 from Utils.Utilities import encodeUnicodeToBytesConditional
+from Utils.ProcFS import processStatus
 
 
 def run(command):
@@ -86,6 +87,13 @@ def isAlive(self):
             return False
         return True
 
+    def processStatus(self):
+        """
+        Return process status along with status of all running threads
+        :return: process status, list of dictionaries of `{process, pid, status}` data-structure
+        """
+        return processStatus(self['ProcessID'])
+
     def kill(self, signal=15):
         """
         _kill_
diff --git a/test/python/Utils_t/ProcFS_t.py b/test/python/Utils_t/ProcFS_t.py
new file mode 100644
index 0000000000..2a18a51cb9
--- /dev/null
+++ b/test/python/Utils_t/ProcFS_t.py
@@ -0,0 +1,82 @@
+#!/usr/bin/env python
+"""
+Unittests for ProcFS functions
+"""
+
+# system modules
+import unittest
+import os
+import time
+import threading
+
+# WMCore modules
+from Utils.ProcFS import processStatus
+
+
+def daemon(nloops=2):
+    """
+    Dummy daemon with sleep loops
+    :param: number of loops to perform, default 2
+    """
+    for _ in range(nloops):
+        time.sleep(1)
+
+
+class TestProcessStatus(unittest.TestCase):
+    """
+    TestProcessStauts unit test class
+    """
+    def testProcessStatus(self):
+        """Test processStatus function with the current process PID."""
+        pid = os.getpid()  # Get the current process ID
+        thread = threading.Thread(target=daemon, args={})  # Create a dummy thread
+        thread.start()
+        try:
+            statusList = processStatus(pid)
+
+            # Check that at least one entry exists (main process)
+            self.assertGreater(len(statusList), 0, "Status list should not be empty")
+
+            # Ensure the first entry is the main process
+            mainProcess = statusList[0]
+            self.assertEqual(mainProcess["pid"], str(pid), "Main process PID should match")
+            self.assertIn(mainProcess["status"][0], "running", "Unexpected process state")
+
+            # Ensure that we have two pids: one for main process and another for daemon thread
+            pids = [entry["pid"] for entry in statusList]
+            self.assertTrue(len(pids), 2)
+            self.assertTrue(pids[0] != pids[1], True)
+
+        finally:
+            thread.join()  # Ensure thread finishes execution
+
+    def worker(self):
+        """A simple worker thread that runs for a short time."""
+        time.sleep(2)  # Keep the thread alive for 2 seconds
+
+    def testProcessStatusWithStoppedThread(self):
+        """Test processStatus when a thread is running and then stopped."""
+        pid = os.getpid()  # Get current process ID
+        thread = threading.Thread(target=self.worker)
+        thread.start()  # Start the thread
+
+        time.sleep(0.5)  # Wait for the thread to be fully initialized
+
+        # Check status while the thread is running
+        statusList = processStatus(pid)
+        pids = [entry["pid"] for entry in statusList]
+        self.assertTrue(len(pids), 2)
+
+        # Let the thread finish (simulate stopping)
+        thread.join()
+
+        time.sleep(1)  # Allow time for the system to reflect thread termination
+
+        # Check status after thread stops
+        statusList = processStatus(pid)
+        pids = [entry["pid"] for entry in statusList]
+        self.assertTrue(len(pids), 1)
+
+
+if __name__ == "__main__":
+    unittest.main()
diff --git a/test/python/Utils_t/ProcessStats_t.py b/test/python/Utils_t/ProcessStats_t.py
new file mode 100644
index 0000000000..9ad6902dda
--- /dev/null
+++ b/test/python/Utils_t/ProcessStats_t.py
@@ -0,0 +1,27 @@
+#!/usr/bin/env python
+"""
+Unittests for ProcessStats module
+"""
+
+# system modules
+import unittest
+import os
+
+# WMCore modules
+from Utils.ProcessStats import processThreadsInfo
+
+
+class TestProcessTHreadsInfo(unittest.TestCase):
+    """
+    TestProcessThreadsInfo unit test class
+    """
+    def testProcessThreadsInfo(self):
+        pid = os.getpid()  # Get current process ID
+        info = processThreadsInfo(pid)
+        self.assertTrue(info['pid'] == pid)
+        self.assertTrue(info['status'] == 'running')
+        self.assertTrue(len(info['threads']) == 0)
+
+
+if __name__ == "__main__":
+    unittest.main()
diff --git a/test/python/Utils_t/Utilities_t.py b/test/python/Utils_t/Utilities_t.py
index 9be1009dd3..0da9ebdcb2 100644
--- a/test/python/Utils_t/Utilities_t.py
+++ b/test/python/Utils_t/Utilities_t.py
@@ -4,11 +4,14 @@
 """
 
 from builtins import object
+import os
+import tempfile
 import unittest
 
 from Utils.Utilities import makeList, makeNonEmptyList, strToBool, \
     safeStr, rootUrlJoin, zipEncodeStr, lowerCmsHeaders, getSize, \
-    encodeUnicodeToBytes, diskUse, numberCouchProcess, normalize_spaces
+    encodeUnicodeToBytes, diskUse, numberCouchProcess, normalize_spaces, \
+    extractFromXML
 
 
 class UtilitiesTests(unittest.TestCase):
@@ -235,6 +238,28 @@ def testNumberCouchProcess(self):
         # there should be at least one process, but who knows...
         self.assertTrue(data >= 0)
 
+    def testExtractFromXML(self):
+        """Test extracting an existing element from XML."""
+
+        # create new xmlContent file
+        xmlContent = """<?xml version="1.0"?>
+        <Daemon>
+            <ProcessID Value="1953175"/>
+            <ParentProcessID Value="1"/>
+            <ProcessGroupID Value="1953174"/>
+        </Daemon>"""
+
+        tempXML = tempfile.NamedTemporaryFile(delete=False, suffix=".xml")
+        tempXML.write(xmlContent.encode("utf-8"))
+        tempXML.close()
+
+        value = extractFromXML(tempXML.name, "ProcessID")
+        self.assertEqual(value, "1953175")
+        value = extractFromXML(tempXML.name, "NonExistingElement")
+        self.assertIsNone(value)
+
+        # remove tempXML
+        os.unlink(tempXML.name)
 
 if __name__ == '__main__':
     unittest.main()


Reviews:
user: login: amaltaro
body: Without having a deep look into the details of the worker threads implementation, but seeing each component worker thread inherits a `BaseWorkerThread`:
https://github.com/dmwm/WMCore/blob/master/src/python/WMCore/WorkerThreads/BaseWorkerThread.py

and runs as a daemon, spawning a Daemon.xml file, e.g.:
```
(WMAgent-2.3.9.2) [xxx@vocms0xxx:current]$ cat install/WorkQueueManager/Daemon.xml
```

Can't we consolidate most of this development into this module:
https://github.com/dmwm/WMCore/blob/master/src/python/WMCore/Agent/Daemon/Details.py
?

Or perhaps we could have the relevant worker thread monitoring logic in the BaseWorkerThread itself:
https://github.com/dmwm/WMCore/blob/master/src/python/WMCore/WorkerThreads/BaseWorkerThread.py

In addition, I see the Harness module is used for starting up a component:
https://github.com/dmwm/WMCore/blob/master/src/python/WMCore/Agent/Harness.py
but I think it is a higher layer and would not fit well for the worker threads monitoring.
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-25T22:47:51Z
user: login: anpicci
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-31T12:11:38Z
user: login: amaltaro
body: Valentin, please find a few more comments along the code.
In addition, given the amount of information in the PR description, I feel like we could move/enhance all of that to our documentation, for instance: https://gitlab.cern.ch/dmwm/wmcore-docs/-/blob/master/docs/wmcore/WMAgent-monitoring.md?ref_type=heads

and keep only a clear and concise summary of the changes provided with this PR (pointing to the documentation for further details). For instance, mentioning in the PR description about the new threads.json file and the new process_info would be useful.
state: CHANGES_REQUESTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-17T03:29:20Z
user: login: amaltaro
body: @vkuznet from the discussion we had this morning, I thought you had addressed all comments/reviews, but I still see several comments along the code. 

Can you please update them accordingly? Or follow up with any discussion still needed? FYI @anpicci 
state: CHANGES_REQUESTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-06T15:25:37Z
user: login: vkuznet
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-06T17:23:52Z
user: login: vkuznet
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-06T17:24:12Z
user: login: vkuznet
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-06T17:32:02Z
user: login: vkuznet
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-06T17:35:51Z
user: login: vkuznet
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-06T17:37:33Z
user: login: amaltaro
body: Other than one import that I would strongly suggest to update, it looks good to me Valentin. However, please see inline for further feedback.
state: CHANGES_REQUESTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-07T02:34:08Z
user: login: vkuznet
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-07T11:15:39Z
user: login: vkuznet
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-07T11:15:57Z
user: login: vkuznet
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-07T11:17:33Z
user: login: vkuznet
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-07T11:29:19Z
user: login: mapellidario
body: thanks valentin for the great effort, i approve this PR
state: APPROVED
author_association: MEMBER
submitted_at: 2025-05-07T12:43:03Z
user: login: anpicci
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-07T13:17:53Z
user: login: anpicci
body: Thank you for taking care of these changes. I approve them since to me no further changes are needed, we can keep `ProcFS.py` and `ProcessStats.py` (as well as their implementation) as they are, and come back to this whether needed
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-07T13:20:44Z
user: login: anpicci
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-07T13:25:26Z

Review comments:
diff_hunk: @@ -4,18 +4,38 @@
 """
 
 from builtins import object
+import os
+import tempfile
 import unittest
 
 from Utils.Utilities import makeList, makeNonEmptyList, strToBool, \
     safeStr, rootUrlJoin, zipEncodeStr, lowerCmsHeaders, getSize, \
-    encodeUnicodeToBytes, diskUse, numberCouchProcess, normalize_spaces
+    encodeUnicodeToBytes, diskUse, numberCouchProcess, normalize_spaces, \
+    extractFromXML
 
 
 class UtilitiesTests(unittest.TestCase):
     """
     unittest for Utilities functions
     """
 
+    def setUp(self):
path: test/python/Utils_t/Utilities_t.py
user: login: amaltaro
body: Given that setUp/tearDown is executed for every unit test, and only 1 unit test will consume this XML file, I would suggest to create/remove this file only within the `testExtractFromXML` context.
created_at: 2025-04-17T02:56:14Z
author_association: CONTRIBUTOR
diff_hunk: @@ -89,6 +89,88 @@ def threadStack():
         threads.append(tdict)
     return dict(threads=threads)
 
+def processThreadsInfo(pid):
path: src/python/Utils/ProcessStats.py
user: login: amaltaro
body: Given all the code in this function, can you please try creating a unit test for it?
created_at: 2025-04-17T03:03:37Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,52 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+"""
+File       : ProcFS.py
path: src/python/Utils/ProcFS.py
user: login: amaltaro
body: I had understood we would perform this monitoring with the psutils library, instead of directly accessing /proc. Leaving this comment again, in case the previous one got lost.
created_at: 2025-04-17T03:05:31Z
author_association: CONTRIBUTOR
diff_hunk: @@ -320,18 +338,61 @@ def status(configFile):
             return
         compDir = config.section_(component).componentDir
         compDir = os.path.expandvars(compDir)
-        daemonXml = os.path.join(compDir, "Daemon.xml")
-        if not os.path.exists(daemonXml):
-            print("Component:%s Not Running" % component)
-            continue
-        daemon = Details(daemonXml)
-        if not daemon.isAlive():
-            print("Component:%s Not Running" % component)
-        else:
-            print("Component:%s Running:%s" % (component, daemon['ProcessID']))
+        checkProcessThreads(component, compDir)
 
     sys.exit(0)
 
+def checkProcessThreads(component, compDir):
+    """
+    Helper function to check process and its threads for their statuses
+    :param component: component name
+    :return: prints status of the component process and its threads
+    """
+    # check if component daemon exists
+    daemonXml = os.path.join(compDir, "Daemon.xml")
+    if not os.path.exists(daemonXml):
+        print("Component:%s Not Running" % component)
+        return
+    daemon = Details(daemonXml)
+    pid = daemon['ProcessID']
+
+    jsonFile = os.path.join(compDir, "threads.json")
+    with open(jsonFile, "r", encoding="utf-8") as istream:
+        data = json.load(istream)
+
+    # Extract process and its threads
+    processEntry = None
+    threadPids = []
+
+    for entry in data:
+        if str(entry["pid"]) == str(pid) and entry["type"] == "process":
+            processEntry = entry
+        elif entry["type"] == "thread":
+            threadPids.append(entry["pid"])
+
+    # Check if process is running
+    processRunning = psutil.pid_exists(int(pid))
+
+    # Check if threads are running
+    runningThreads = []
+    orphanThreads = []
+    process = psutil.Process(int(pid))
+    for thread in process.threads():
+        if str(thread.id) == str(pid):
+            continue  # skip pid thread
+        if str(thread.id) in threadPids:
+            runningThreads.append(thread.id)
+        else:
+            orphanThreads.append(thread.id)
+
+    # Output result
+    status = "running" if processRunning else "not-running"
path: bin/wmcoreD
user: login: amaltaro
body: Are these the words that we are currently using for component status result? If not, I would suggest to keep them consistent.
created_at: 2025-04-17T03:13:56Z
author_association: CONTRIBUTOR
diff_hunk: @@ -320,18 +338,61 @@ def status(configFile):
             return
         compDir = config.section_(component).componentDir
         compDir = os.path.expandvars(compDir)
-        daemonXml = os.path.join(compDir, "Daemon.xml")
-        if not os.path.exists(daemonXml):
-            print("Component:%s Not Running" % component)
-            continue
-        daemon = Details(daemonXml)
-        if not daemon.isAlive():
-            print("Component:%s Not Running" % component)
-        else:
-            print("Component:%s Running:%s" % (component, daemon['ProcessID']))
+        checkProcessThreads(component, compDir)
 
     sys.exit(0)
 
+def checkProcessThreads(component, compDir):
+    """
+    Helper function to check process and its threads for their statuses
+    :param component: component name
+    :return: prints status of the component process and its threads
+    """
+    # check if component daemon exists
+    daemonXml = os.path.join(compDir, "Daemon.xml")
+    if not os.path.exists(daemonXml):
+        print("Component:%s Not Running" % component)
+        return
+    daemon = Details(daemonXml)
+    pid = daemon['ProcessID']
+
+    jsonFile = os.path.join(compDir, "threads.json")
+    with open(jsonFile, "r", encoding="utf-8") as istream:
+        data = json.load(istream)
+
+    # Extract process and its threads
+    processEntry = None
+    threadPids = []
+
+    for entry in data:
+        if str(entry["pid"]) == str(pid) and entry["type"] == "process":
+            processEntry = entry
+        elif entry["type"] == "thread":
+            threadPids.append(entry["pid"])
+
+    # Check if process is running
+    processRunning = psutil.pid_exists(int(pid))
+
+    # Check if threads are running
+    runningThreads = []
+    orphanThreads = []
+    process = psutil.Process(int(pid))
+    for thread in process.threads():
+        if str(thread.id) == str(pid):
+            continue  # skip pid thread
+        if str(thread.id) in threadPids:
+            runningThreads.append(thread.id)
+        else:
+            orphanThreads.append(thread.id)
+
+    # Output result
+    status = "running" if processRunning else "not-running"
+    msg = f"Component:{component} {pid} {status} with threads {runningThreads}"
+    if len(orphanThreads) > 0:
+        status = "partially-running" if processRunning else "not-running"
path: bin/wmcoreD
user: login: amaltaro
body: And this new status means the parent process is gone but not the thread, is it correct? Perhaps we should add a simple legend at the top of this module and/or in the PR description?
created_at: 2025-04-17T03:14:40Z
author_association: CONTRIBUTOR
diff_hunk: @@ -4,18 +4,38 @@
 """
 
 from builtins import object
+import os
+import tempfile
 import unittest
 
 from Utils.Utilities import makeList, makeNonEmptyList, strToBool, \
     safeStr, rootUrlJoin, zipEncodeStr, lowerCmsHeaders, getSize, \
-    encodeUnicodeToBytes, diskUse, numberCouchProcess, normalize_spaces
+    encodeUnicodeToBytes, diskUse, numberCouchProcess, normalize_spaces, \
+    extractFromXML
 
 
 class UtilitiesTests(unittest.TestCase):
     """
     unittest for Utilities functions
     """
 
+    def setUp(self):
path: test/python/Utils_t/Utilities_t.py
user: login: vkuznet
body: resolved.
created_at: 2025-05-06T17:23:52Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,52 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+"""
+File       : ProcFS.py
path: src/python/Utils/ProcFS.py
user: login: vkuznet
body: I adjusted the module to provide both implementation and use psutil as default one.
created_at: 2025-05-06T17:24:12Z
author_association: CONTRIBUTOR
diff_hunk: @@ -89,6 +89,88 @@ def threadStack():
         threads.append(tdict)
     return dict(threads=threads)
 
+def processThreadsInfo(pid):
path: src/python/Utils/ProcessStats.py
user: login: vkuznet
body: done
created_at: 2025-05-06T17:32:01Z
author_association: CONTRIBUTOR
diff_hunk: @@ -320,18 +338,61 @@ def status(configFile):
             return
         compDir = config.section_(component).componentDir
         compDir = os.path.expandvars(compDir)
-        daemonXml = os.path.join(compDir, "Daemon.xml")
-        if not os.path.exists(daemonXml):
-            print("Component:%s Not Running" % component)
-            continue
-        daemon = Details(daemonXml)
-        if not daemon.isAlive():
-            print("Component:%s Not Running" % component)
-        else:
-            print("Component:%s Running:%s" % (component, daemon['ProcessID']))
+        checkProcessThreads(component, compDir)
 
     sys.exit(0)
 
+def checkProcessThreads(component, compDir):
+    """
+    Helper function to check process and its threads for their statuses
+    :param component: component name
+    :return: prints status of the component process and its threads
+    """
+    # check if component daemon exists
+    daemonXml = os.path.join(compDir, "Daemon.xml")
+    if not os.path.exists(daemonXml):
+        print("Component:%s Not Running" % component)
+        return
+    daemon = Details(daemonXml)
+    pid = daemon['ProcessID']
+
+    jsonFile = os.path.join(compDir, "threads.json")
+    with open(jsonFile, "r", encoding="utf-8") as istream:
+        data = json.load(istream)
+
+    # Extract process and its threads
+    processEntry = None
+    threadPids = []
+
+    for entry in data:
+        if str(entry["pid"]) == str(pid) and entry["type"] == "process":
+            processEntry = entry
+        elif entry["type"] == "thread":
+            threadPids.append(entry["pid"])
+
+    # Check if process is running
+    processRunning = psutil.pid_exists(int(pid))
+
+    # Check if threads are running
+    runningThreads = []
+    orphanThreads = []
+    process = psutil.Process(int(pid))
+    for thread in process.threads():
+        if str(thread.id) == str(pid):
+            continue  # skip pid thread
+        if str(thread.id) in threadPids:
+            runningThreads.append(thread.id)
+        else:
+            orphanThreads.append(thread.id)
+
+    # Output result
+    status = "running" if processRunning else "not-running"
path: bin/wmcoreD
user: login: vkuznet
body: these words `running` or `not-running` will appear once someone runs `manage` script which calls `wmcoreD`. The actual message is defined a few lines below like:

```
    status = "running" if processRunning else "not-running"
    msg = f"Component:{component} {pid} {status} with threads {runningThreads}"
    if len(orphanThreads) > 0:
        status = "partially-running" if processRunning else "not-running"
        msg = f"Component:{component} {pid} {status} with threads {runningThreads}, lost {orphanThreads} threads"
    print(msg)
```
created_at: 2025-05-06T17:35:51Z
author_association: CONTRIBUTOR
diff_hunk: @@ -320,18 +338,61 @@ def status(configFile):
             return
         compDir = config.section_(component).componentDir
         compDir = os.path.expandvars(compDir)
-        daemonXml = os.path.join(compDir, "Daemon.xml")
-        if not os.path.exists(daemonXml):
-            print("Component:%s Not Running" % component)
-            continue
-        daemon = Details(daemonXml)
-        if not daemon.isAlive():
-            print("Component:%s Not Running" % component)
-        else:
-            print("Component:%s Running:%s" % (component, daemon['ProcessID']))
+        checkProcessThreads(component, compDir)
 
     sys.exit(0)
 
+def checkProcessThreads(component, compDir):
+    """
+    Helper function to check process and its threads for their statuses
+    :param component: component name
+    :return: prints status of the component process and its threads
+    """
+    # check if component daemon exists
+    daemonXml = os.path.join(compDir, "Daemon.xml")
+    if not os.path.exists(daemonXml):
+        print("Component:%s Not Running" % component)
+        return
+    daemon = Details(daemonXml)
+    pid = daemon['ProcessID']
+
+    jsonFile = os.path.join(compDir, "threads.json")
+    with open(jsonFile, "r", encoding="utf-8") as istream:
+        data = json.load(istream)
+
+    # Extract process and its threads
+    processEntry = None
+    threadPids = []
+
+    for entry in data:
+        if str(entry["pid"]) == str(pid) and entry["type"] == "process":
+            processEntry = entry
+        elif entry["type"] == "thread":
+            threadPids.append(entry["pid"])
+
+    # Check if process is running
+    processRunning = psutil.pid_exists(int(pid))
+
+    # Check if threads are running
+    runningThreads = []
+    orphanThreads = []
+    process = psutil.Process(int(pid))
+    for thread in process.threads():
+        if str(thread.id) == str(pid):
+            continue  # skip pid thread
+        if str(thread.id) in threadPids:
+            runningThreads.append(thread.id)
+        else:
+            orphanThreads.append(thread.id)
+
+    # Output result
+    status = "running" if processRunning else "not-running"
+    msg = f"Component:{component} {pid} {status} with threads {runningThreads}"
+    if len(orphanThreads) > 0:
+        status = "partially-running" if processRunning else "not-running"
path: bin/wmcoreD
user: login: vkuznet
body: yes, it means process is running (not gone) but one of its thread is dead such that message we get from `manage status` would be
```
Component:ABC 123 partially-running with threads X, lost Y threads
```
created_at: 2025-05-06T17:37:32Z
author_association: CONTRIBUTOR
diff_hunk: @@ -235,6 +238,16 @@ def startup(configFile):
         else:
             print('Path for daemon file does not exist!')
             sys.exit(1)
+        # write into component area process status information
+        cpath = os.path.join(compDir, "threads.json")
+        if os.path.exists(cpath):
+            os.remove(cpath)
+        cxml = os.path.join(compDir, "Daemon.xml")
path: bin/wmcoreD
user: login: amaltaro
body: This line is dup of L231 (other than the variable name).
created_at: 2025-05-07T02:18:59Z
author_association: CONTRIBUTOR
diff_hunk: @@ -320,18 +337,63 @@ def status(configFile):
             return
         compDir = config.section_(component).componentDir
         compDir = os.path.expandvars(compDir)
-        daemonXml = os.path.join(compDir, "Daemon.xml")
-        if not os.path.exists(daemonXml):
-            print("Component:%s Not Running" % component)
-            continue
-        daemon = Details(daemonXml)
-        if not daemon.isAlive():
-            print("Component:%s Not Running" % component)
-        else:
-            print("Component:%s Running:%s" % (component, daemon['ProcessID']))
+        checkProcessThreads(component, compDir)
 
     sys.exit(0)
 
+def checkProcessThreads(component, compDir):
+    """
+    Helper function to check process and its threads for their statuses
+    :param component: component name
+    :return: prints status of the component process and its threads
+    """
+    # check if component daemon exists
+    daemonXml = os.path.join(compDir, "Daemon.xml")
+    if not os.path.exists(daemonXml):
+        print("Component:%s Not Running" % component)
+        return
+    daemon = Details(daemonXml)
path: bin/wmcoreD
user: login: amaltaro
body: I would suggest to be consistent on the parsing of the XML file, at least within these changes. Here you use the long implemented class to parse the XML, but at the top of this module you use `extractFromXML()`.
created_at: 2025-05-07T02:24:03Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,52 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+"""
+File       : ProcFS.py
path: src/python/Utils/ProcFS.py
user: login: amaltaro
body: To me, it would make sense to not fork the process status check into yet another module, and instead I'd integrate these changes together with src/python/Utils/ProcessStats.py. 
That's just my opinion though, perhaps others will share their view. Otherwise, please do not modify anything and let us move on with this.
created_at: 2025-05-07T02:30:36Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,108 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+"""
+File       : ProcFS.py
+Author     : Valentin Kuznetsov <vkuznet AT gmail dot com>
+Description: This module provides function which rely on Linux proc fs
+"""
+# system modules
+import os
+
+# 3rd party modules
+import psutil
path: src/python/Utils/ProcFS.py
user: login: amaltaro
body: I fear that importing this module in the worker node might hit an import error exception (as we do not have psutils in the grid nodes). So you might want to add a try/except here.
created_at: 2025-05-07T02:31:27Z
author_association: CONTRIBUTOR
diff_hunk: @@ -235,6 +238,16 @@ def startup(configFile):
         else:
             print('Path for daemon file does not exist!')
             sys.exit(1)
+        # write into component area process status information
+        cpath = os.path.join(compDir, "threads.json")
+        if os.path.exists(cpath):
+            os.remove(cpath)
+        cxml = os.path.join(compDir, "Daemon.xml")
path: bin/wmcoreD
user: login: vkuznet
body: ok, done
created_at: 2025-05-07T11:15:39Z
author_association: CONTRIBUTOR
diff_hunk: @@ -320,18 +337,63 @@ def status(configFile):
             return
         compDir = config.section_(component).componentDir
         compDir = os.path.expandvars(compDir)
-        daemonXml = os.path.join(compDir, "Daemon.xml")
-        if not os.path.exists(daemonXml):
-            print("Component:%s Not Running" % component)
-            continue
-        daemon = Details(daemonXml)
-        if not daemon.isAlive():
-            print("Component:%s Not Running" % component)
-        else:
-            print("Component:%s Running:%s" % (component, daemon['ProcessID']))
+        checkProcessThreads(component, compDir)
 
     sys.exit(0)
 
+def checkProcessThreads(component, compDir):
+    """
+    Helper function to check process and its threads for their statuses
+    :param component: component name
+    :return: prints status of the component process and its threads
+    """
+    # check if component daemon exists
+    daemonXml = os.path.join(compDir, "Daemon.xml")
+    if not os.path.exists(daemonXml):
+        print("Component:%s Not Running" % component)
+        return
+    daemon = Details(daemonXml)
path: bin/wmcoreD
user: login: vkuznet
body: switched to use extractFromXML
created_at: 2025-05-07T11:15:57Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,108 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+"""
+File       : ProcFS.py
+Author     : Valentin Kuznetsov <vkuznet AT gmail dot com>
+Description: This module provides function which rely on Linux proc fs
+"""
+# system modules
+import os
+
+# 3rd party modules
+import psutil
path: src/python/Utils/ProcFS.py
user: login: vkuznet
body: Added try/except and fallback to use `processStatusViaProcFS`. But obvious question here is why do we need psutil implementation if on worker node we don't have it. The initial implementation via procfs is free from this issue.
created_at: 2025-05-07T11:17:33Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,52 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+"""
+File       : ProcFS.py
path: src/python/Utils/ProcFS.py
user: login: vkuznet
body: Originally, the idea was to have dedicated module for handling `procfs` functionality. That's why I created separate `ProcFS.py` from `ProcessStats.py` module. Now, per request from reviewers we converge on using `psutils` even though in your another comment you mentioned that usage of `psutil` on worker node may fail. Therefore, here I see few approaches:
- rollback to procfs (since psutil is not present everywhere) and then we may keep it in separate module, or
- merge `ProcFS.py` into `ProcessStats.py` as they both use psutil and provide into about process
- merge `ProcFS.py` into `ProcessStats.py` as they contain code about process/thread pids statuses

Either approach is fine with me and I can do whatever you'll consider appropriate.
created_at: 2025-05-07T11:28:32Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,52 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+"""
+File       : ProcFS.py
path: src/python/Utils/ProcFS.py
user: login: anpicci
body: In my view, we can leave as is, we can live with two scripts for the time being
created_at: 2025-05-07T13:17:53Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,108 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+"""
+File       : ProcFS.py
+Author     : Valentin Kuznetsov <vkuznet AT gmail dot com>
+Description: This module provides function which rely on Linux proc fs
+"""
+# system modules
+import os
+
+# 3rd party modules
+import psutil
path: src/python/Utils/ProcFS.py
user: login: anpicci
body: I know that is not ideal, but if the try/except properly handle the events where `psutils` cannot be loaded, we are fine with that. Sorry if this point was missed previously, eventually requiring additional dev effort
created_at: 2025-05-07T13:25:26Z
author_association: CONTRIBUTOR

PR comments:
user: login: dmwm-bot
created_at: 2025-03-11T17:59:02Z
body: Jenkins results:
 * Python3 Unit tests: failed
   * 1 tests added
   * 2 changes in unstable tests
 * Python3 Pylint check: failed
   * 4 warnings and errors that must be fixed
   * 4 warnings
   * 50 comments to review
 * Pycodestyle check: succeeded
   * 3 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/WMCore-PR-Report/463/artifact/artifacts/PullRequestReport.html

author_association: NONE
user: login: dmwm-bot
created_at: 2025-03-11T18:22:05Z
body: Jenkins results:
 * Python3 Unit tests: failed
   * 1 tests added
   * 3 changes in unstable tests
 * Python3 Pylint check: succeeded
   * 4 warnings
   * 34 comments to review
 * Pycodestyle check: succeeded
   * 3 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/WMCore-PR-Report/464/artifact/artifacts/PullRequestReport.html

author_association: NONE
user: login: dmwm-bot
created_at: 2025-03-11T19:50:53Z
body: Jenkins results:
 * Python3 Unit tests: failed
   * 1 tests added
   * 4 changes in unstable tests
 * Python3 Pylint check: succeeded
   * 4 warnings
   * 34 comments to review
 * Pycodestyle check: succeeded
   * 1 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/WMCore-PR-Report/465/artifact/artifacts/PullRequestReport.html

author_association: NONE
user: login: dmwm-bot
created_at: 2025-03-12T13:11:29Z
body: Jenkins results:
 * Python3 Unit tests: failed
   * 1 tests added
   * 3 changes in unstable tests
 * Python3 Pylint check: succeeded
   * 4 warnings
   * 34 comments to review
 * Pycodestyle check: succeeded
   * 1 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/WMCore-PR-Report/471/artifact/artifacts/PullRequestReport.html

author_association: NONE
user: login: dmwm-bot
created_at: 2025-03-12T13:31:25Z
body: Jenkins results:
 * Python3 Unit tests: succeeded
   * 2 tests added
 * Python3 Pylint check: succeeded
   * 4 warnings
   * 34 comments to review
 * Pycodestyle check: succeeded
   * 1 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/WMCore-PR-Report/472/artifact/artifacts/PullRequestReport.html

author_association: NONE
user: login: dmwm-bot
created_at: 2025-03-12T17:48:50Z
body: Jenkins results:
 * Python3 Unit tests: succeeded
   * 2 tests added
   * 2 changes in unstable tests
 * Python3 Pylint check: succeeded
   * 4 warnings
   * 34 comments to review
 * Pycodestyle check: succeeded
   * 1 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/WMCore-PR-Report/476/artifact/artifacts/PullRequestReport.html

author_association: NONE
user: login: dmwm-bot
created_at: 2025-03-13T17:35:18Z
body: Jenkins results:
 * Python3 Unit tests: succeeded
   * 3 tests added
   * 3 changes in unstable tests
 * Python3 Pylint check: failed
   * 14 warnings and errors that must be fixed
   * 5 warnings
   * 130 comments to review
 * Pycodestyle check: succeeded
   * 89 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/WMCore-PR-Report/483/artifact/artifacts/PullRequestReport.html

author_association: NONE
user: login: dmwm-bot
created_at: 2025-03-25T13:34:51Z
body: Jenkins results:
 * Python3 Unit tests: succeeded
   * 3 tests added
   * 5 changes in unstable tests
 * Python3 Pylint check: failed
   * 20 warnings and errors that must be fixed
   * 10 warnings
   * 136 comments to review
 * Pycodestyle check: succeeded
   * 103 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/WMCore-PR-Report/519/artifact/artifacts/PullRequestReport.html

author_association: NONE
user: login: vkuznet
created_at: 2025-03-26T13:41:28Z
body: > src/python/WMCore/WorkerThreads/BaseWorkerThread.py

Alan (@amaltaro), please go through the review process to avoid triggering unnecessary work. In my view, what you suggest does not fit in existing architectural design because we don't have ability to communicate with threads from external process. In other words, our components and pollers are python processes rather (HTTP) services where you can request some monitoring metrics. In external process with thread model what can be done is to push metrics somewhere, e.g. we can push such information to internal database, or Monit infrastructure and then use them afterwards. At the moment we rely on process ID (extracted from Daemon.xml) to inspect its status about run-time environment of components/daemons.

I provided a stand-along functions for that and they are used in `Details.py` as you mentioned in your comment. Therefore, I recommend that you review the current changes, and if you have specific idea how to implement thread monitoring via integration with `[WMCore/WorkerThreads/BaseWorkerThread.py](https://github.com/dmwm/WMCore/blob/master/src/python/WMCore/WorkerThreads/BaseWorkerThread.py)` I suggest that you outline your idea as proposal for such monitoring. For instance
- add process metrics to BaseWorkerThread
- how to push/collect these metrics elsewhere (please specify where)
- how to extract these metrics from elsewhere and use it for monitoring of running components/daemons. For instance:
  - how to use these metrics in `manage status`
  - how to use these metrics in `WMComponent/AnalyticsDataCollector/DataCollectAPI.py` which report them to CouchDB, i.e. how it can communicate with run-time environment (`BaseWorkerThread.py`)
  - maybe we can push metrics directly from `BaseWorkerThread.py` to CouchDB, is it desired?

My point is that without global picture with all details of monitoring metrics workflows it is very hard to understand what you have in mind, and simply mentioning different parts of WMCore does not help to understand if it is feasible to accomplish with current architectural design.
author_association: CONTRIBUTOR
user: login: amaltaro
created_at: 2025-03-27T11:24:39Z
body: @vkuznet I am puzzled with the fragmentation of process-related code. Despite the extensive PR description (thanks!), please help me understand the following:
a) what is `wmcoreD` bin used for? I guess it is used by the `manage` script, hence the fragmentation from the components codebase
b) can't we merge the functionality in the new module `ProcFS` into `ProcessStats`? Is there any reason to keep those separated?
c) instead of extending `Utilities` module, can't we use the `Daemon/Details` module - which already loads the Daemon.xml and gives you the ProcessID element?
d) in `DataCollectAPI`, instead of relying in a broken information from the database - and the new data struct created - don't you think it is better to persist the actual worker thread ID in the database (wm_workers table?)

Checking the information already avaialble in WMStats `agentInfo` view, I see:
```
        "workers": [
          {
            "name": "AgentStatusPoller",
            "last_updated": 1743074171,
            "state": "Running",
            "poll_interval": 300,
            "cycle_time": 1.5446
          },
          {
            "name": "AnalyticsPoller",
            "last_updated": 1743074124,
            "state": "Running",
            "poll_interval": 600,
            "cycle_time": 0.8201
          },
...
```

and I would be in favor of always reporting the state of all worker threads (+ main component), instead of just publishing when one of the threads is down.

What do you think?
author_association: CONTRIBUTOR
user: login: mapellidario
created_at: 2025-03-28T10:17:36Z
body: Before I leave a review, I have a curiosity, mainly for my education, and if it has already been discussed i am sorry if i missed it. Is it there any reason why you use read the content of the `/proc/$pid` files in some places and use the python module `psutils` somewhere else? do they provide different information? if `psutil` can provide all the information we need, i would be in favor of never reading directly from `/prod/$pid` files
author_association: MEMBER
user: login: vkuznet
created_at: 2025-03-30T17:47:48Z
body: @amaltaro , here are my responses to your questions:
a) the `wmcoreD` is used within `manage script`, e.g.
```
start_agent(){
    if  _init_valid $wmaInitUsing ; then
        echo "Starting WMAgent..."
        wmcoreD --start --config=$WMA_CONFIG_DIR/config.py
    else
        echo "ERROR: This agent is not fully initialized. Cannot use it."
        return $(false)
    fi
}
```
b) we can merge `ProcFS` into `ProcessStats` but they have code not related to each other. The former keeps code to work with Linux file system, while later the code to about process monitoring. Because of this reason I kept them separately but I don't mind to merge them.
c) the only reason I added `extractFromXML` due to its generic nature, i.e. this function can be used with any XML parsing task. Moving this from Utilities to `Daemon/Details` is acceptable but based on my previous reviews you always wanted to keep pieces in right places, and to me keeping generic function in Utilities is more appropriate then in `Daemon/Details`. And, I do use `processStatus` function I provided in `Daemon/Details`. If my answers are not satisfactory to you please elaborate more how do you prefer relocate the code. In my view everything is in right places and used accordingly.
d) I don't really have a clue why we keep process information in a database as it is outdated due to different time it is used and collected. My understanding that we want to get up-to-date info about component to report it to the couch. The database write seems useless to me since knowing process ID we can get up-to-date process information. Regarding structure used `agentInfo` I intentionally kept it backward compatible to avoid even more changes on wmstats JavaScript (client) side. And, so far we only improving what we have. Instead the entire process of collecting info is outdated and I'm in favor of collecting metrics and pushing them to local Prometheus instances (which then can push it to CMS Monitoring one). Here again, I tried to minimize the effort (which seems already extensive).

@mapellidario , answering your question. Even though `psutil` is well known python package the Linux `/proc` file system is foundation of everything in Linux OS. I trust it more :) That said, I provided both ways, either use `ProcFS` module based on `/proc` FS, or use psutil approach in `ProcessStats` module.
author_association: CONTRIBUTOR
user: login: amaltaro
created_at: 2025-04-03T15:56:13Z
body: Thank you for these clarifications, Valentin.

> b) we can merge ProcFS into ProcessStats but they have code not related to each other. The former keeps code to work with Linux file system, while later the code to about process monitoring. Because of this reason I kept them separately but I don't mind to merge them.

Given that we are still checking information for processes, I would merge them all together (probably keeping ProcessStats module). 

> c) the only reason I added extractFromXML due to its generic nature, i.e. this function can be used with any XML parsing task. Moving this from Utilities to Daemon/Details is acceptable but based on my previous reviews you always wanted to keep pieces in right places, and to me keeping generic function in Utilities is more appropriate then in Daemon/Details. And, I do use processStatus function I provided in Daemon/Details. If my answers are not satisfactory to you please elaborate more how do you prefer relocate the code. In my view everything is in right places and used accordingly.

Yes, generic functions are better placed under the `Utilities` package. However, in this case you are dealing with an agent component/daemon, which creates an XML. From what I can see, the Details.py module:
https://github.com/dmwm/WMCore/blob/master/src/python/WMCore/Agent/Daemon/Details.py

exist exactly to deal with that Daemon.xml file. So it is somehow natural to me that we would use that module to retrieve the node that you are looking for. If not yet available, then we can extend that class with the required method.

Generic module does the job and can be used by anything else that needs to parse an xml. The inconsistency accessing Daemon.xml is what bugs me. Just my 2 cents, stick to whatever you/people prefer.

> d) I don't really have a clue why we keep process information in a database as it is outdated due to different time it is used and collected. ...

Data in the database will only be stale if a process dies, which is actually our clue to see that that component/process is no longer running.
The way the agent works is that, whenever we start a new component, it self-registers in the database (plus some information). If the row was already populated with an outdated PID, it is updated as well. So that table is always giving us a summary of components / worker threads / process id (well, without the thread id, which is being resolved with this PR). I am adding a couple of database screenshots for your consideration as well.

> @mapellidario , answering your question. Even though psutil is well known python package the Linux /proc file system is foundation of everything in Linux OS. I trust it more :) That said, I provided both ways, either use ProcFS module based on /proc FS, or use psutil approach in ProcessStats module.

It looks like `psutil` relies on `/proc` and `/sys` for linux process management, so we are probably fine if we adopt only one solution. I'd stick with one well tested solution, than having 2 and not being sure which one to take.

<img width="435" alt="wm_components" src="https://github.com/user-attachments/assets/aa9967ae-1727-429f-a849-4f606ed341ad" />
<img width="1076" alt="wm_workers" src="https://github.com/user-attachments/assets/7343648e-f0ea-4976-80cd-c341a560862c" />


author_association: CONTRIBUTOR
user: login: vkuznet
created_at: 2025-04-14T12:47:54Z
body: > Data in the database will only be stale if a process dies, which is actually our clue to see that that component/process is no longer running.

The data in the database will not be correct if **ANY** thread will die with main process still alive. For instance, if main process is fine and all of its threads too then your statement is correct. But if main process is fine but any of its threads are not (it is possible that thread dies but main process is alive) then content of database is not representing actual state of the component. For that reason I think we cannot rely on database content for any component (in this regard I don't know why do we need the database table for them either).

Before making any changes I want you to review this statement and decide how to proceed. In my view the database entries are not useful to represent state of the component and we **must** use either proc fs or pstuils to determine state of all component threads each time of polling cycle.

I'm tagging everyone to make awareness of this situation: @amaltaro , @anpicci , @todor-ivanov , @mapellidario , @khurtado , @d-ylee 
author_association: CONTRIBUTOR
user: login: amaltaro
created_at: 2025-04-15T19:52:34Z
body: > The data in the database will not be correct if ANY thread will die with main process still alive. For instance, if main process is fine and all of its threads too then your statement is correct. But if main process is fine but any of its threads are not (it is possible that thread dies but main process is alive) then content of database is not representing actual state of the component. For that reason I think we cannot rely on database content for any component (in this regard I don't know why do we need the database table for them either).

And this is exactly the expected behavior. Let me expand on the algorithm that was probably considered when these tables were designed (which makes sense to me):
1. component starts and it registers its PID in the database (including worker threads)
2. whatever is monitoring these components/worker threads, reads from the database to find out what is the **expected** PID for those threads
3. it checks their status. If anything is not alive, it then reports that it is down. If every PID is alive, then it knows all worker threads are behaving properly.
4. if someone stops a component, all its worker threads are stopped as well - which goes back to point 3 above, meaning monitoring will report those as down.
5. whenever a component is restarted, all its worker threads are respawned and the table gets updated with new PIDs

> In my view the database entries are not useful to represent state of the component

I guess here is the miscommunication. The database is **not** meant to track the state of components and worker threads, it is only meant to track the expected (registered) PID for them. Monitoring of their state needs to happen "outside" of the table - which implies, the table is not updated, other than the heartbeat, if we want to.
author_association: CONTRIBUTOR
user: login: vkuznet
created_at: 2025-04-16T12:38:57Z
body: @amaltaro , thanks for explaining the algorithm but this is not what is implemented in a code. According to
- [isAlive](https://github.com/dmwm/WMCore/blob/master/src/python/WMCore/Agent/Daemon/Details.py#L84) function of Daemon/Details.py module it checks 
```
        # Reference: ps -T -p 1946167 -o euser,pid,ppid,lwp,nlwp,stat,start
        # it prints the user, process and its threads, number of threads, etc
        dummyse, dummyso, rc = run('ps -p %s' % self['ProcessID'])
```
Please note that comment about ps command is correct (it should extract info about main process along with threads) the actual action does not do that, i.e. it only looks-up main process
```
# here is lookup of PID of Component:WorkQueueManager in my agent
ps -p 3545240
    PID TTY          TIME CMD
3545240 ?        04:16:45 python

# while if we'll supply -T flag it will print out all threads
ps -T -p 3545240
    PID    SPID TTY          TIME CMD
3545240 3545240 ?        00:00:00 python
3545240 3545241 ?        00:14:15 python
3545240 3545242 ?        00:10:18 python
3545240 3545243 ?        00:02:28 python
3545240 3545244 ?        03:49:19 python
```
- moreover, the ps command only show running process but does not show its actual state, the process may be alive but stuck on somthing (I/O), i.e. we need to know the state of the process which is what I suggest in this PR
- the [kill](https://github.com/dmwm/WMCore/blob/master/src/python/WMCore/Agent/Daemon/Details.py#L96) function does not kill threads either it kills main process and even though it is rare in UNIX/Linux but it is not impossible for the main process to be killed and its thread be alive in a Zombie state. This happens when a thread is stuck in a kernel-level uninterruptible sleep (D state) like waiting for I/O (disk/NFS/network). In this case even SIGKILL (signal 9) may not immediately remove them until the blocking operation completes. And, even more the default parameter to `kill` function is signal 15 which is graceful shutdown and not even signal 9. Therefore, even though current code may work in majority of the case it is not 100%.

Therefore, according to my findins keeping main PID in DB table and according to current code implementation we do not cover by 100% all the case to kill components along with its threads but we do cover majority of them as it is usually sufficient to use main PID to kill process with its threads. In my view, the current code implementation is not sufficient to cover all corner cases and we should re-evaluate it further.
author_association: CONTRIBUTOR
user: login: amaltaro
created_at: 2025-04-17T03:38:52Z
body: Now replying on top of your last comment:

> moreover, the ps command only show running process but does not show its actual state, the process may be alive but stuck on somthing (I/O), i.e. we need to know the state of the process which is what I suggest in this PR

At this stage, I don't think we need/want to implement actions based on the state of the process. So having the state in our monitoring is probably what we should focus on this contribution.

> the [kill](https://github.com/dmwm/WMCore/blob/master/src/python/WMCore/Agent/Daemon/Details.py#L96) function does not kill threads either it kills ...

I might have missed it, but we do not have any changes in this PR on how components/threads are started or terminated, do we? I would suggest not to make any changes on that aspect, as I don't recall having such issues while operating agents. If needed, this is something that we can address in the future.

> Therefore, according to my findins keeping main PID in DB table and according to current code implementation we do not cover by 100% all the case to kill components along with its threads but we do cover majority of them as it is usually sufficient to use main PID to kill process with its threads. In my view, the current code implementation is not sufficient to cover all corner cases and we should re-evaluate it further.

As said above, I feel like the focus of this contribution is to have a better monitoring of the components and worker threads. If we have to modify how they are started up and/or killed/terminated, I would suggest to deal with this in a different ticket.

About the database table, I believe it could be a replacement to the threads.json file, assuming that the database would only give us the PID and the worker thread name that we want to look at. However, I think the `manage` script would not be able to easily use that - making the effort to make the DB table consistent probably not worth it. If I am missing anything, please let me know.
author_association: CONTRIBUTOR
user: login: amaltaro
created_at: 2025-04-18T01:25:14Z
body: The following thought just occurred. Instead of all this extra work to have a proper monitoring of the worker threads, why don't we separate each worker thread in its own component/process? I see many benefits in doing so, in addition to an easy way to scale out component instances in the future.
author_association: CONTRIBUTOR
user: login: vkuznet
created_at: 2025-04-18T13:22:35Z
body: > The following thought just occurred. Instead of all this extra work to have a proper monitoring of the worker threads, why don't we separate each worker thread in its own component/process? I see many benefits in doing so, in addition to an easy way to scale out component instances in the future.

Alan, I'm glad that you brought this up, but if we are serious about refactoring component threads into separate entities I rather suggest to go extra mile and convert each thread into individual service. The benefits of this are the following:
- The services can be accessible via HTTP and therefore provide extra flexibility for management, e.g. you can change at run-time the polling cycle, or you can pass new set of parameters, etc.
- The services would easy to monitor via Prometeus if service provide its own set of /metrics and therefore we can have much better monitoring in place
- The services provides GET/POST/PUT/DELETE operations meaning that we can communicate with they externally. At the moment the threads manage their data via persistent storage, e.g. store JSON to CouchDB, or MariaDB, while with services we can let client to perform operation with service, e.g. send new document to it. This brings us closer to data-driven service oriented architecture.

I suggest that we discuss this further within members of the team. FYI: @anpicci 
author_association: CONTRIBUTOR
user: login: amaltaro
created_at: 2025-04-21T10:52:18Z
body: Right. There are several benefits in separating these worker threads, and the disadvantages is only that some objects won't be shared between a components worker thread - but honestly, I don't know how much we are using this at the moment.

Nonetheless, I would not consider web-based components at the moment, because there are also security concerns, depending on where these services would be running. Other than separating worker threads in their own component, I think there are no further requirements to be changed in order to deliver a better monitoring of the components/worker threads - so it is a good compromise within the scope of this ticket.
author_association: CONTRIBUTOR
user: login: vkuznet
created_at: 2025-04-21T11:46:28Z
body: @amaltaro , so what do you propose to do with this PR? Should we leave and merge it, should we change PR to separate all threads into individual processes, if so how? Should we open new issue for such activity. It would be nice if you'll provide concrete steps how to move forward with issue in question.
author_association: CONTRIBUTOR
user: login: amaltaro
created_at: 2025-04-21T15:29:16Z
body: Before deciding how to proceed here, I think we need to answer the following questions:
a) do we resolve the ill worker thread monitoring if we isolate each in their own component? I think so, but are we missing anything else important?
b) which parts of this current implementation in this PR we **must** keep (resolves an immediate problem)?
c) which parts of this current implementation in this PR we **can** keep (might be useful at some point, but does not resolve any immediate problem)?

What I am trying to see is how much extra work is the component separation and whether that separation would suffice or not.
author_association: CONTRIBUTOR
user: login: vkuznet
created_at: 2025-04-21T16:11:32Z
body: Alan, let me answer your question one by one:
a) separation of component threads into individual processes will allow to use their PIDs to monitor their state. Therefore, the answer is yes.
b) and c):I think this PR just generalize **any** process monitoring since it provides additional functionality to obtain process information along with its threads. Will the process have threads or not does not concern of this PR. That said, if we'll have any daemon we want to get info about its process and threads this PR is beneficial. If we will never have threaded daemons or apps then we may not need it. For instance, any HTTP service we have is threaded (as we use CherryPy threaded framework), and therefore having proposed changes is beneficial in my view.

Finally, the component separation into individual processes has nothing to do with this PR, and it should go into separate issue. Moreover, it will require additional thoughts on several items like:
- agreement on naming convention. For instance, when we'll separate WorkQueueManager into separate processes, how we are going to name them?
- we may need to think how to adjust start-up procedure since right now it has one set of processes and we will expand it further
- we may need to adjust `manage` script for WMAgent to handle new processes too
- we should decide if we need to adjust any changes in internal database tables since we'll have more processes
- etc.
author_association: CONTRIBUTOR
user: login: anpicci
created_at: 2025-04-22T09:52:32Z
body: Hi @vkuznet @amaltaro, thank you for the fruitful discussion. I am in favor of concluding this PR as is, without further major development, and opening an issue to collect the idea outlined in the last messages.
Closing the issue related to this PR will impact https://github.com/dmwm/WMCore/pull/12308, and this is another motivation to moving along with merge this PR sooner rather than later
author_association: CONTRIBUTOR
user: login: dmwm-bot
created_at: 2025-04-28T12:29:34Z
body: Jenkins results:
 * Python3 Unit tests: succeeded
   * 3 tests added
   * 1 changes in unstable tests
 * Python3 Pylint check: failed
   * 14 warnings and errors that must be fixed
   * 10 warnings
   * 137 comments to review
 * Pycodestyle check: succeeded
   * 103 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/WMCore-PR-Report/628/artifact/artifacts/PullRequestReport.html

author_association: NONE
user: login: vkuznet
created_at: 2025-04-28T12:34:12Z
body: @amaltaro , @anpicci this PR is ready, all commits are squashed and I don't know if you still want to have another review or not. Just in case I request Alan's review again.

Also, I submitted this wf `valya_TC_Nano_vocms266_forceComplete_250428_123656_8774` to cmsweb-test10 with pointing to be executed on vocms266 agent where these changes are deployed
```
python3 inject-test-wfs.py -u "https://cmsweb-test10.cern.ch" -m Integration -f TC_Nano.json -c vocms266_Validation -r vocms266_forceComplete -t test10-vocms266 -a Integ_Test -p tc_nano
```
author_association: CONTRIBUTOR
user: login: amaltaro
created_at: 2025-05-06T15:44:37Z
body: Valentin, in addition to the previous review, please update this documentation:
https://gitlab.cern.ch/dmwm/wmcore-docs/-/blob/master/docs/wmcore/ReqMgr2-monitoring.md

with the new content that will start getting pushed to WMStats (for the agent monitoring). I believe none of this affects the documents posted to AMQ, but please clarify that as well if needed.
author_association: CONTRIBUTOR
user: login: dmwm-bot
created_at: 2025-05-06T17:57:06Z
body: Jenkins results:
 * Python3 Unit tests: failed
   * 3 tests no longer failing
   * 4 tests added
   * 1 changes in unstable tests
 * Python3 Pylint check: failed
   * 15 warnings and errors that must be fixed
   * 11 warnings
   * 139 comments to review
 * Pycodestyle check: succeeded
   * 107 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/WMCore-PR-Report/659/artifact/artifacts/PullRequestReport.html

author_association: NONE
user: login: dmwm-bot
created_at: 2025-05-06T19:07:31Z
body: Jenkins results:
 * Python3 Unit tests: succeeded
   * 3 tests no longer failing
   * 4 tests added
   * 2 changes in unstable tests
 * Python3 Pylint check: failed
   * 15 warnings and errors that must be fixed
   * 11 warnings
   * 139 comments to review
 * Pycodestyle check: succeeded
   * 106 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/WMCore-PR-Report/661/artifact/artifacts/PullRequestReport.html

author_association: NONE
user: login: vkuznet
created_at: 2025-05-06T19:16:56Z
body: > @vkuznet from the discussion we had this morning, I thought you had addressed all comments/reviews, but I still see several comments along the code.
> 
> Can you please update them accordingly? Or follow up with any discussion still needed? FYI @anpicci

@amaltaro , some of the changes were not pushed and I apologize for that. But I also forgot to update the comments you provided. Now it is done, and changes are pushed upstream. I verified that Jenkins are also fine. Per your request I updated the description with new statuses, see **Note II**. Once I'll update documentation I'll post relevant PR. So far this PR is ready.

@mapellidario , @anpicci I would appreciate if you'll make another round of review as well.
author_association: CONTRIBUTOR
user: login: dmwm-bot
created_at: 2025-05-07T11:26:07Z
body: Jenkins results:
 * Python3 Unit tests: succeeded
   * 4 tests added
   * 1 changes in unstable tests
 * Python3 Pylint check: failed
   * 16 warnings and errors that must be fixed
   * 11 warnings
   * 142 comments to review
 * Pycodestyle check: succeeded
   * 107 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/WMCore-PR-Report/666/artifact/artifacts/PullRequestReport.html

author_association: NONE
user: login: vkuznet
created_at: 2025-05-07T11:31:33Z
body:   @amaltaro , I applied changes based on your review and squashed commits. But I raised a question about `psutil` (see in my comments). Feel free to review again and let me know how to proceed.
author_association: CONTRIBUTOR