Repository: frappe/agent
Pull request #178: feat: Snapshots to facilitate stuck process investigation
Author: login: Aradhya-Tripathi
Created: 2025-03-21T15:19:05Z | Merged: 2025-04-02T13:55:33Z

Addresses [this](https://github.com/frappe/press/issues/2450)

REQUIRES
- frappe ALL = (root) NOPASSWD: /home/frappe/agent/env/bin/py-spy
in /etc/sudoers.d/frappe on all agent servers.





Diff:
diff --git a/agent/bench.py b/agent/bench.py
index c46d914d..ecc5b22e 100644
--- a/agent/bench.py
+++ b/agent/bench.py
@@ -316,6 +316,35 @@ def fetch_monitor_data(self):
             traceback.print_exc()
         return lines
 
+    def _parse_pids(self, lines):
+        pids = []
+        lines = lines.strip().split("\n")
+
+        for line in lines:
+            parts = line.strip().split()
+            name, pid = parts[0], parts[1]
+            pids.append((name, pid))
+
+        return pids
+
+    def get_worker_pids(self):
+        """Get all the processes running gunicorn for now"""
+        return self._parse_pids(self.execute(f"docker top {self.name} | grep gunicorn")["output"])
+
+    def take_snapshot(self, pid_info: list[tuple[str, str]]):
+        snapshots = {}
+        pyspy_bin = os.path.join(self.server.directory, "env/bin/py-spy")
+
+        for name, pid in pid_info:
+            try:
+                snapshots[f"{name}:{pid}"] = json.loads(
+                    self.execute(f"sudo {pyspy_bin} dump --pid {pid} --json")["output"]
+                )
+            except AgentException as e:
+                snapshots[f"{name}:{pid}"] = str(e)
+
+        return snapshots
+
     def status(self):
         status = {
             "sites": {site: {"scheduler": True, "web": True} for site in self.sites},
diff --git a/agent/utils.py b/agent/utils.py
index ad57bc6e..7455b2c6 100644
--- a/agent/utils.py
+++ b/agent/utils.py
@@ -169,3 +169,7 @@ def get_mariadb_table_name_from_path(path: str) -> str:
     filename = os.path.splitext(filename)[0]
     # Decode the filename
     return decode_mariadb_filename(filename)
+
+
+def check_installed_pyspy(server_dir: str) -> bool:
+    return os.path.exists(os.path.join(server_dir, "env/bin/py-spy"))
diff --git a/agent/web.py b/agent/web.py
index d42fe363..150c9d79 100644
--- a/agent/web.py
+++ b/agent/web.py
@@ -33,6 +33,7 @@
 from agent.security import Security
 from agent.server import Server
 from agent.ssh import SSHProxy
+from agent.utils import check_installed_pyspy
 
 if TYPE_CHECKING:
     from datetime import datetime, timedelta
@@ -161,6 +162,21 @@ def ping():
     return {"message": "pong"}
 
 
+@application.route("/snapshot/<string:bench_name>")
+def get_snapshots(bench_name: str):
+    server = Server()
+    bench = server.benches.get(bench_name)
+
+    if not bench:
+        return {"message": f"No such bench {bench_name}"}, 400
+
+    if not check_installed_pyspy(server.directory):
+        return {"message": "PySpy is not installed on this server"}, 400
+
+    pids = bench.get_worker_pids()
+    return bench.take_snapshot(pids)
+
+
 @application.route("/ping_job", methods=["POST"])
 def ping_job():
     return {
diff --git a/requirements.txt b/requirements.txt
index 65fcf1b5..144616cb 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -27,3 +27,4 @@ docker==6.1.2
 filelock==3.13.1
 sentry-sdk[flask, rq]==2.1.1
 sql_metadata==2.15.0
+py-spy==0.4.0
\ No newline at end of file


Reviews:
user: login: tanmoysrt
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-01T07:14:53Z
user: login: Aradhya-Tripathi
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-01T12:19:00Z
user: login: tanmoysrt
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-01T14:29:26Z

Review comments:
diff_hunk: @@ -316,6 +316,35 @@ def fetch_monitor_data(self):
             traceback.print_exc()
         return lines
 
+    def _parse_pids(self, lines):
+        pids = []
+        lines = lines.strip().split("\n")
+
+        for line in lines:
+            parts = line.strip().split()
+            name, pid = parts[0], parts[1]
+            pids.append((name, pid))
+
+        return pids
+
+    def get_worker_pids(self):
+        """Get all the processes running gunicorn for now"""
+        return self._parse_pids(self.execute("ps aux | grep gunicorn")["output"])
path: agent/bench.py
user: login: tanmoysrt
body: ig You are trying to find `gunicorn` processes inside bench's container .
If that's the case, Use `docker_execute` function to run the `ps aux ....`  inside container
created_at: 2025-04-01T07:14:53Z
author_association: MEMBER
diff_hunk: @@ -316,6 +316,35 @@ def fetch_monitor_data(self):
             traceback.print_exc()
         return lines
 
+    def _parse_pids(self, lines):
+        pids = []
+        lines = lines.strip().split("\n")
+
+        for line in lines:
+            parts = line.strip().split()
+            name, pid = parts[0], parts[1]
+            pids.append((name, pid))
+
+        return pids
+
+    def get_worker_pids(self):
+        """Get all the processes running gunicorn for now"""
+        return self._parse_pids(self.execute("ps aux | grep gunicorn")["output"])
path: agent/bench.py
user: login: Aradhya-Tripathi
body: As of now I was just testing py-spy outside the container and getting a dump, will look into running py-spy in a [docker container](https://github.com/benfred/py-spy/blob/master/README.md#how-do-i-run-py-spy-in-docker) as well, also I guess enabling SYS_PTRACE is unsafe.


created_at: 2025-04-01T12:19:00Z
author_association: CONTRIBUTOR
diff_hunk: @@ -316,6 +316,35 @@ def fetch_monitor_data(self):
             traceback.print_exc()
         return lines
 
+    def _parse_pids(self, lines):
+        pids = []
+        lines = lines.strip().split("\n")
+
+        for line in lines:
+            parts = line.strip().split()
+            name, pid = parts[0], parts[1]
+            pids.append((name, pid))
+
+        return pids
+
+    def get_worker_pids(self):
+        """Get all the processes running gunicorn for now"""
+        return self._parse_pids(self.execute("ps aux | grep gunicorn")["output"])
path: agent/bench.py
user: login: tanmoysrt
body: It's fine to run `py-spy` outside.

I was just talking about getting the pids of the gunicorn process for a particular bench.
The current code will grep gunicorn process of other benches as well. 

So, either you need to run `ps aux ...` inside container or use [`docker top`](https://docs.docker.com/reference/cli/docker/container/top/) to get the pids of process inside container
created_at: 2025-04-01T14:29:26Z
author_association: MEMBER

PR comments:
user: login: ankush
created_at: 2025-04-14T08:46:39Z
body: @Aradhya-Tripathi any way to view this in UI right now? Even dumb report on desk is fine 
author_association: MEMBER
user: login: Aradhya-Tripathi
created_at: 2025-04-14T08:50:35Z
body: > @Aradhya-Tripathi any way to view this in UI right now? Even dumb report on desk is fine

We have [this](https://github.com/frappe/press/pull/2576) for now.

author_association: CONTRIBUTOR