Repository: hpcc-systems/HPCC-Platform
Pull request #19588: HPCC-27051 create a sasha service to clean up post mortem files
Author: login: streeterd
Created: 2025-03-05T16:19:27Z | Merged: 2025-04-10T15:39:58Z

<!-- Thank you for submitting a pull request to the HPCC project

 PLEASE READ the following before proceeding.

 This project only accepts pull requests related to open JIRA issues.
 If suggesting a new feature or change, please discuss it in a JIRA issue first.
 If fixing a bug, there should be an issue describing it with steps to reproduce.
 The title line of the pull request (and of each commit within it) should refer to the
 associated issue using the format:

 HPCC-nnnnn Short description of issue

 This will allow the Jira ticket to be automatically updated to refer to this pull request,
 and will ensure that the automatically-generated changelog is properly formatted.
 Where a pull request contains a single commit the pull request title will be set automatically,
 assuming that the commit has followed the proper guidelines.

 Please go over all the following points, and put an `x` in all the boxes that apply. You may find
 it easier to press the 'Create' button first then click on the checkboxes to edit the comment.
-->

## Type of change:
- [ ] This change is a bug fix (non-breaking change which fixes an issue).
- [x] This change is a new feature (non-breaking change which adds functionality).
- [ ] This change improves the code (refactor or other change that does not change the functionality)
- [ ] This change fixes warnings (the fix does not alter the functionality or the generated code)
- [ ] This change is a breaking change (fix or feature that will cause existing behavior to change).
- [ ] This change alters the query API (existing queries will have to be recompiled)

## Checklist:
- [x] My code follows the code style of this project.
  - [ ] My code does not create any new warnings from compiler, build system, or lint.
- [x] The commit message is properly formatted and free of typos.
  - [ ] The commit message title makes sense in a changelog, by itself.
  - [ ] The commit is signed.
- [x] My change requires a change to the documentation.
  - [ ] I have updated the documentation accordingly, or...
  - [ ] I have created a JIRA ticket to update the documentation.
  - [ ] Any new interfaces or exported functions are appropriately commented.
- [x] I have read the CONTRIBUTORS document.
- [x] The change has been fully tested:
  - [ ] I have added tests to cover my changes.
  - [ ] All new and existing tests passed.
  - [ ] I have checked that this change does not introduce memory leaks.
  - [ ] I have used Valgrind or similar tools to check for potential issues.
- [ ] I have given due consideration to all of the following potential concerns:
  - [ ] Scalability
  - [ ] Performance
  - [ ] Security
  - [ ] Thread-safety
  - [ ] Cloud-compatibility
  - [ ] Premature optimization
  - [ ] Existing deployed queries will not be broken
  - [ ] This change fixes the problem, not just the symptom
  - [ ] The target branch of this pull request is appropriate for such a change.
- [ ] There are no similar instances of the same problem that should be addressed
  - [ ] I have addressed them here
  - [ ] I have raised JIRA issues to address them separately
- [ ] This is a user interface / front-end modification
  - [ ] I have tested my changes in multiple modern browsers
  - [ ] The component(s) render as expected

## Smoketest:
- [x] Send notifications about my Pull Request position in Smoketest queue.
- [x] Test my draft Pull Request.

## Testing:
<!-- Please describe how this change has been tested.-->

<!-- Thank you for taking the time to submit this pull request and to answer all of the above-->



Diff:
diff --git a/dali/sasha/CMakeLists.txt b/dali/sasha/CMakeLists.txt
index 1ba66a6e957..1c1f969fe73 100644
--- a/dali/sasha/CMakeLists.txt
+++ b/dali/sasha/CMakeLists.txt
@@ -42,6 +42,7 @@ set (    SRCS
          sautil.cpp 
          saverify.cpp 
          saxref.cpp 
+         sadbghk.cpp
     )
 
 include_directories ( 
diff --git a/dali/sasha/sadbghk.cpp b/dali/sasha/sadbghk.cpp
new file mode 100644
index 00000000000..f0fe7f277bb
--- /dev/null
+++ b/dali/sasha/sadbghk.cpp
@@ -0,0 +1,181 @@
+#include "platform.h"
+
+#include "jfile.hpp"
+#include "jlib.hpp"
+#include "jlog.hpp"
+#include "jiface.hpp"
+#include "jptree.hpp"
+#include "jstring.hpp"
+
+#include "saserver.hpp"
+#include "sautil.hpp"
+
+static constexpr unsigned defaultExpiryDays = 14;
+static constexpr unsigned defaultHouseKeepingIntervalHours = 24;
+
+#define LOGDBGHK "DEBUGPLANEHOUSEKEEPING: "
+
+// Debug Housekeeping monitor
+
+class CSashaDebugPlaneHousekeepingServer : public ISashaServer, public Thread
+{
+    std::atomic<bool> stopped{true};
+    Semaphore stopSem;
+    Mutex runMutex;
+    StringBuffer debugDir;
+    unsigned expiryDays{0};
+    unsigned interval{0};
+
+public:
+    IMPLEMENT_IINTERFACE_USING(Thread);
+
+    CSashaDebugPlaneHousekeepingServer()
+        : Thread("CSashaDebugPlaneHousekeepingServer")
+    {
+    }
+
+    virtual void start() override
+    {
+        Owned<IPropertyTree> compConfig = getComponentConfig();
+        expiryDays = compConfig->getPropInt("@expiryDays", defaultExpiryDays);
+        if (expiryDays == 0)
+            return;
+        interval = compConfig->getPropInt("@interval", defaultHouseKeepingIntervalHours);
+        if (interval == 0)
+            return;
+
+        // get debug plane dir
+        StringBuffer planeName;
+        if (!getDefaultPlane(planeName, "@debugPlane", "debug"))
+        {
+            WARNLOG("Failed to get default debug plane");
+            return;
+        }
+        Owned<IPropertyTree> plane = getStoragePlane(planeName);
+        verifyex(plane && plane->getProp("@prefix", debugDir));
+
+        stopped = false;
+        Thread::start(false);
+    }
+
+    virtual void ready() override
+    {
+    }
+
+    virtual void stop() override
+    {
+        if (stopped)
+            return;
+
+        stopped = true;
+        stopSem.signal();
+        synchronized block(runMutex); // hopefully stopped should stop
+        if (!join(1000 * 60 * 3))
+            OERRLOG("CSashaDebugPlaneHousekeepingServer aborted");
+    }
+
+    void runDebugHousekeeping()
+    {
+        synchronized block(runMutex);
+        if (stopped)
+            return;
+
+        unsigned dirsDeleted{0};
+        unsigned filesDeleted{0};
+        unsigned filesFailedToBeDeleted{0};
+
+        // iterate debug plane selecting files and directories for housekeeping
+        Owned<IDirectoryIterator> pDirIter = createDirectoryIterator(debugDir.str(), "*", false, true);
+        ForEach(*pDirIter)
+        {
+            if (stopped)
+                break;
+
+            IFile &iFile = pDirIter->query();
+            const char *filePath = iFile.queryFilename();
+
+            if (iFile.isDirectory() == fileBool::foundYes)
+            {
+                // Process directories
+
+                // Exclude the ".", ".."
+                if (streq(filePath, ".") || streq(filePath, ".."))
+                    continue;
+
+                // Process expired directories for housekeeping
+                if (hasExpired(iFile))
+                {
+                    recursiveRemoveDirectory(&iFile);
+                    ++dirsDeleted; // should be really be conditional on success of recursiveRemoveDirectory
+                }
+            }
+            else
+            {
+                // Process file exclude non expired files from housekeeping
+                if (hasExpired(iFile))
+                {
+                    if (iFile.remove())
+                        ++filesDeleted;
+                    else
+                        ++filesFailedToBeDeleted;
+                }
+            }
+        }
+
+        if (dirsDeleted || filesDeleted || filesFailedToBeDeleted)
+            PROGLOG(LOGDBGHK "Deleted dirs: %d, files: %d, files failed to be deleted: %d", dirsDeleted, filesDeleted, filesFailedToBeDeleted);
+    }
+
+    virtual int run() override
+    {
+        CSashaSchedule schedule;
+        schedule.init(getComponentConfigSP(), interval);
+        while (!stopped)
+        {
+            stopSem.wait(1000 * 60);
+            if (stopped)
+                break;
+            if (!schedule.ready())
+                continue;
+            try
+            {
+                runDebugHousekeeping();
+            }
+            catch (IException *e)
+            {
+                EXCLOG(e, LOGDBGHK);
+                e->Release();
+            }
+        }
+        return 0;
+    }
+
+private:
+    bool hasExpired(IFile &iFile)
+    {
+        CDateTime expiredModifiedTime;
+        if (!iFile.getTime(nullptr, &expiredModifiedTime, nullptr))
+        {
+            WARNLOG("Failed to get modified time for file %s", iFile.queryFilename());
+            return false;
+        }
+        expiredModifiedTime.adjustTime(60 * 24 * expiryDays);
+
+        CDateTime now;
+        now.setNow();
+
+        if (now.compare(expiredModifiedTime, false) > 0)
+            return true;
+        else
+            return false;
+    }
+
+} *sashaDebugPlaneHousekeepingServer = nullptr;
+
+ISashaServer *createSashaDebugPlaneHousekeepingServer()
+{
+    assertex(isContainerized());
+    assertex(!sashaDebugPlaneHousekeepingServer); // initialization problem
+    sashaDebugPlaneHousekeepingServer = new CSashaDebugPlaneHousekeepingServer();
+    return sashaDebugPlaneHousekeepingServer;
+}
diff --git a/dali/sasha/sadbghk.hpp b/dali/sasha/sadbghk.hpp
new file mode 100644
index 00000000000..57dad4df9d5
--- /dev/null
+++ b/dali/sasha/sadbghk.hpp
@@ -0,0 +1,4 @@
+#pragma once
+
+interface ISashaServer;
+extern ISashaServer *createSashaDebugPlaneHousekeepingServer();
diff --git a/dali/sasha/saserver.cpp b/dali/sasha/saserver.cpp
index acdf1775772..eb1c2d8c693 100644
--- a/dali/sasha/saserver.cpp
+++ b/dali/sasha/saserver.cpp
@@ -40,6 +40,7 @@
 #include "saarch.hpp"
 #include "saverify.hpp"
 #include "saxref.hpp"
+#include "sadbghk.hpp"
 #include "saqmon.hpp"
 #include "sacoalescer.hpp"
 #include "sacmd.hpp"
@@ -53,8 +54,9 @@ extern void LDStest();
 Owned<IPropertyTree> serverConfig;
 static IArrayOf<ISashaServer> servers;
 static std::atomic<unsigned> StopSuspendCount{0};
-static bool stopped = false;
+static bool stopped{false};
 static Semaphore stopSem;
+static bool isDaliClient{false};
 
 const char *sashaProgramName;
 
@@ -238,6 +240,13 @@ class CSashaCmdThread : public CInterface, implements IPooledThread
 
 void SashaMain()
 {
+    if (!isDaliClient) // If a Dali client, SashaMain will be listening to MP and cancelled by CStopThread
+    {
+        // NB: requestStop signals stopSem
+        stopSem.wait();
+        return;
+    }
+
     IInterCommunicator & comm=queryWorldCommunicator();
     unsigned start = msTick();
     unsigned timeout = serverConfig->getPropInt("@autoRestartInterval")*1000*60*60;
@@ -329,6 +338,7 @@ int main(int argc, const char* argv[])
     try
     {
         serverConfig.setown(loadConfiguration(defaultYaml, argv, "sasha", "SASHA", "sashaconf.xml", nullptr));
+        isDaliClient = isContainerized() ? serverConfig->hasProp("[access='dali']") : true;
 
         Owned<IFile> sentinelFile;
 
@@ -378,16 +388,20 @@ int main(int argc, const char* argv[])
                 writeSentinelFile(sentinelFile);
         }
 #endif
-        StringBuffer daliServer;
-        if (!serverConfig->getProp("@daliServers", daliServer))
-            serverConfig->getProp("@DALISERVERS", daliServer); // @DALISERVERS legacy/bare-metal
-        if (0 == daliServer.length())
+        if (isDaliClient)
         {
-            PROGLOG("DALISERVERS not specified in sashaconf.xml");
-            return 1;
+            PROGLOG("Connecting to DALISERVERS.");
+            StringBuffer daliServer;
+            if (!serverConfig->getProp("@daliServers", daliServer))
+                serverConfig->getProp("@DALISERVERS", daliServer); // @DALISERVERS legacy/bare-metal
+            if (0 == daliServer.length())
+            {
+                PROGLOG("DALISERVERS not specified in sashaconf.xml");
+                return 1;
+            }
+            Owned<IGroup> serverGroup = createIGroupRetry(daliServer.str(), DALI_SERVER_PORT);
+            initClientProcess(serverGroup, DCR_SashaServer, port, nullptr, nullptr, MP_WAIT_FOREVER, true);
         }
-        Owned<IGroup> serverGroup = createIGroupRetry(daliServer.str(), DALI_SERVER_PORT);
-        initClientProcess(serverGroup, DCR_SashaServer, port, nullptr, nullptr, MP_WAIT_FOREVER, true);
 
         if (stop)
             stopSashaServer((argc>2)?argv[2]:"", DEFAULT_SASHA_PORT);
@@ -435,6 +449,8 @@ int main(int argc, const char* argv[])
                     servers.append(*createSashaFileExpiryServer());
                 else if (strieq(service, "thor-qmon"))
                     servers.append(*createSashaQMonitorServer());
+                else if (strieq(service, "debugplane-housekeeping"))
+                    servers.append(*createSashaDebugPlaneHousekeepingServer());
                 //else if (strieq(service, "xref")) // TODO
                 //    servers.append(*createSashaXrefServer());
                 else
@@ -444,9 +460,6 @@ int main(int argc, const char* argv[])
                 AddServers();
 #endif
 
-                StringBuffer eps;
-                PROGLOG("SASERVER starting on %s",queryMyNode()->endpoint().getEndpointHostText(eps).str());
-
                 ForEachItemIn(i1,servers)
                 {
                     ISashaServer &server=servers.item(i1);
@@ -469,11 +482,14 @@ int main(int argc, const char* argv[])
                         if (!stopped)
                         {
                             stopped = true;
-                            IInterCommunicator &comm=queryWorldCommunicator();
+                            IInterCommunicator &comm = queryWorldCommunicator();
                             comm.cancel(NULL, MPTAG_SASHA_REQUEST);
                         }
                     }
-                } *stopThread = new CStopThread;
+                } *stopThread{};
+                if (isDaliClient)
+                    stopThread = new CStopThread;
+                // else, SashaMain waits directly on stopSem
                 addThreadExceptionHandler(&exceptionStopHandler);
 #ifdef _CONTAINERIZED
                 if (!serverConfig->getPropBool("@inDaliPod"))
@@ -484,8 +500,11 @@ int main(int argc, const char* argv[])
                 SashaMain();
                 removeThreadExceptionHandler(&exceptionStopHandler);
 
-                stopSem.signal();
-                delete stopThread;
+                if (isDaliClient)
+                {
+                    stopSem.signal();
+                    delete stopThread;
+                }
 
                 PROGLOG("SASERVER exiting");
 #ifndef _CONTAINERIZED
@@ -514,9 +533,12 @@ int main(int argc, const char* argv[])
     serverConfig.clear();
     try
     {
-        closeDllServer();
         closeEnvironment();
-        closedownClientProcess();
+        if (isDaliClient)
+        {
+            closeDllServer();
+            closedownClientProcess();
+        }
     }
     catch (IException *) {  // dali may be down
     }
diff --git a/dali/sasha/sasha.md b/dali/sasha/sasha.md
new file mode 100644
index 00000000000..ce21b5efb33
--- /dev/null
+++ b/dali/sasha/sasha.md
@@ -0,0 +1,63 @@
+# Sasha
+
+## Services
+
+### Debug Plane Housekeeping Server
+
+#### Overview
+
+- This is a maintenance service that automatically manages files and directories on the debug plane using a configured retention period.
+- This is required to control the disc space used by post-mortem and other files on the debug plane.
+- This service is designed to execute on a schedule; a single log entry for a summary of any deleted (or failed to be deleted) files or deleted directories will be generated.
+
+#### Installation and Configuration
+
+- The service is installed and active by default.
+- It is configuration-driven with fallback to defaults in the configuration yaml files.
+  - For example `values.yaml` :
+
+    |attribute|description|default|required|
+    |:--------|:----------|:------|:-------|
+    |disabled|Disable or enable this service.|false|optional|
+    |interval|Minimum interval between debug plane housekeeping expiry checking (in hours, 0 disables).|24|optional|
+    |at|Schedule to check for debug plane expired files and directories (cron format).|* 3 * * *|optional|
+    |expiryDays|Default number of days passed that debug plane files or directories are deleted.|14|optional|
+
+#### User Guide
+
+##### How it works in steps
+
+1. The `disabled` value is read from the component configuration
+2. If `disabled` is `true` then the service is not started.
+3. The schedule is read from the component configuration.
+4. If the `interval` is 0 then the service is stopped.
+5. The debug plane location is read from the component configuration.
+6. The service `Thread` is started.
+7. A loop of the following is started until the service is stopped:
+    1. A one minute wait is imposed.
+    2. If the schedule using `at` and `interval` indicates that the housekeeping is required then:
+        1. Any exceptions from the following are caught and logged.
+            1. The debug plane directory list is read.
+            2. The list of files and directories in the debug plane are processed:
+                1. If the file or directory's modified date is over `expiryDays` old then:
+                    1. The file or directory is deleted.
+                    2. A counter for each file or directory is maintained.
+            3. A log entry is made giving the total number of files and directories deleted.
+
+##### Inputs and Outputs
+
+- The component configuration as per [Installation and Configuration](#installation-and-configuration) is the input for this service.
+- The outputs are any logs generated for errors and deletion statistics.
+
+##### Error Handling
+
+- Known exceptions are handled to prevent service interruption.
+
+#### Limitations and Considerations
+
+- The service ensures it only runs in containerized environments.
+
+#### Future Possible Enhancements
+
+- A bare metal implementation.
+- Logging freed up disc space statistics.
diff --git a/helm/examples/vault-pki-remote/values-hpcc1.yaml b/helm/examples/vault-pki-remote/values-hpcc1.yaml
index a21e9ccc765..2cac210c2d5 100644
--- a/helm/examples/vault-pki-remote/values-hpcc1.yaml
+++ b/helm/examples/vault-pki-remote/values-hpcc1.yaml
@@ -52,6 +52,8 @@ sasha:
     disabled: true
   file-expiry:
     disabled: true
+  debugplane-housekeeping:
+    disabled: true
 
 esp:
 - name: eclwatch
diff --git a/helm/examples/vault-pki-remote/values-hpcc2.yaml b/helm/examples/vault-pki-remote/values-hpcc2.yaml
index 799f529572a..c8d1b2bf172 100644
--- a/helm/examples/vault-pki-remote/values-hpcc2.yaml
+++ b/helm/examples/vault-pki-remote/values-hpcc2.yaml
@@ -52,6 +52,8 @@ sasha:
     disabled: true
   file-expiry:
     disabled: true
+  debugplane-housekeeping:
+    disabled: true
 
 esp:
 - name: eclwatch
diff --git a/helm/hpcc/templates/_helpers.tpl b/helm/hpcc/templates/_helpers.tpl
index 26d6436a7df..c44c4e996dd 100644
--- a/helm/hpcc/templates/_helpers.tpl
+++ b/helm/hpcc/templates/_helpers.tpl
@@ -1715,6 +1715,7 @@ dali
 dali data
 {{- else if (eq "thor-qmon" .name) -}}
 dali queues
+{{- else if (eq "debugplane-housekeeping" .name) -}}
 {{- else -}}
 {{- $_ := fail (printf "Unknown sasha service:" .name ) -}}
 {{- end -}}
diff --git a/helm/hpcc/values.schema.json b/helm/hpcc/values.schema.json
index 436d7247b80..22f7e7fd23b 100644
--- a/helm/hpcc/values.schema.json
+++ b/helm/hpcc/values.schema.json
@@ -3058,6 +3058,27 @@
       },
       "additionalProperties": false
     },
+    "sasha-debugplane-housekeeping": {
+      "type": "object",
+      "allOf": [{ "$ref": "#/definitions/sashacommon" }],
+      "properties": {
+        "expiryDays": {
+          "type": "integer",
+          "description": "Default number of days passed that debug plane files or directories are deleted.",
+          "default": "14"
+        },
+        "disabled": {},
+        "interval": {},
+        "at" : {},
+        "image": {},
+        "resources": {},
+        "annotations": {},
+        "labels": {},
+        "expert": {},
+        "terminationGracePeriodSeconds": {}
+      },
+      "additionalProperties": false
+    },
     "sashaservice": {
       "oneOf": [
         {
@@ -3082,6 +3103,9 @@
             "thor-qmon": {
               "$ref": "#/definitions/sasha-thor-qmon"
             },
+            "debugplane-housekeeping": {
+              "$ref": "#/definitions/sasha-debugplane-housekeeping"
+            },
             "disabled": {
               "type": "boolean"
             }
diff --git a/helm/hpcc/values.yaml b/helm/hpcc/values.yaml
index dada3ca5c03..a37f5ef9456 100644
--- a/helm/hpcc/values.yaml
+++ b/helm/hpcc/values.yaml
@@ -513,6 +513,12 @@ sasha:
     #switchMinTime: 1
     #queues: "*"
 
+  debugplane-housekeeping: {} # NB: no properties defined, use defaults
+    #disabled: true
+    #interval: 24
+    #at: "* 3 * * *"
+    #expiryDays: 14
+
 
 dfuserver:
 - name: dfuserver
diff --git a/testing/helm/tests/multicertdomains.yaml b/testing/helm/tests/multicertdomains.yaml
index d406fc54119..3f82b5c3b95 100644
--- a/testing/helm/tests/multicertdomains.yaml
+++ b/testing/helm/tests/multicertdomains.yaml
@@ -49,6 +49,8 @@ sasha:
     disabled: true
   file-expiry:
     disabled: true
+  debugplane-housekeeping:
+    disabled: true
 
 esp:
 - name: eclwatch
diff --git a/testing/helm/tests/resourced.yaml b/testing/helm/tests/resourced.yaml
index a8d3a0053e6..893aca28a0c 100644
--- a/testing/helm/tests/resourced.yaml
+++ b/testing/helm/tests/resourced.yaml
@@ -511,6 +511,15 @@ sasha:
       cpu: "1"
       memory: "4Gi"
 
+  debugplane-housekeeping:
+    #disabled: true
+    #interval: 1
+    #at: "* 3 * * *"
+    #expiryDays: 14
+    resources:
+      cpu: "1"
+      memory: "1Gi"
+
 dfuserver:
 - name: dfuserver
   maxJobs: 1
diff --git a/testing/helm/tests/resourced2.yaml b/testing/helm/tests/resourced2.yaml
index 6b4f95d615f..a0159d61fda 100644
--- a/testing/helm/tests/resourced2.yaml
+++ b/testing/helm/tests/resourced2.yaml
@@ -513,6 +513,15 @@ sasha:
       cpu: "1"
       memory: "4Gi"
 
+  debugplane-housekeeping:
+    #disabled: true
+    #interval: 1
+    #at: "* 3 * * *"
+    #expiryDays: 14
+    resources:
+      cpu: "1"
+      memory: "1Gi"
+
 dfuserver:
 - name: dfuserver
   maxJobs: 1
diff --git a/testing/helm/tests/values-hpcc1.yaml b/testing/helm/tests/values-hpcc1.yaml
index a21e9ccc765..2cac210c2d5 100644
--- a/testing/helm/tests/values-hpcc1.yaml
+++ b/testing/helm/tests/values-hpcc1.yaml
@@ -52,6 +52,8 @@ sasha:
     disabled: true
   file-expiry:
     disabled: true
+  debugplane-housekeeping:
+    disabled: true
 
 esp:
 - name: eclwatch
diff --git a/testing/helm/tests/values-hpcc2.yaml b/testing/helm/tests/values-hpcc2.yaml
index 799f529572a..c8d1b2bf172 100644
--- a/testing/helm/tests/values-hpcc2.yaml
+++ b/testing/helm/tests/values-hpcc2.yaml
@@ -52,6 +52,8 @@ sasha:
     disabled: true
   file-expiry:
     disabled: true
+  debugplane-housekeeping:
+    disabled: true
 
 esp:
 - name: eclwatch


Reviews:
user: login: copilot-pull-request-reviewer[bot]
body: ## PR Overview

This PR creates a new "debug-housekeeping" configuration for the sasha service, intended to support cleanup of post mortem files by providing an optional debug mode for housekeeping tasks.
- Introduces a "debug-housekeeping" block in the primary values file with default (empty object) settings.
- Adds corresponding "debug-housekeeping" configurations in multiple test YAML files, enabling or disabling the debug housekeeping feature.
- Augments resourced test files with a "resources" block under "debug-housekeeping" for additional resource allocation.

### Reviewed Changes

| File                                      | Description                                               |
| ----------------------------------------- | --------------------------------------------------------- |
| helm/hpcc/values.yaml                     | Added "debug-housekeeping" with default settings          |
| testing/helm/tests/multicertdomains.yaml   | Added "debug-housekeeping" with disabled flag             |
| testing/helm/tests/values-hpcc2.yaml         | Added "debug-housekeeping" with disabled flag             |
| testing/helm/tests/values-hpcc1.yaml         | Added "debug-housekeeping" with disabled flag             |
| testing/helm/tests/resourced.yaml          | Added "debug-housekeeping" with commented config and resource sub-block |
| testing/helm/tests/resourced2.yaml         | Added "debug-housekeeping" with commented config and resource sub-block |

Copilot reviewed 21 out of 21 changed files in this pull request and generated no comments.

<details>
<summary>Comments suppressed due to low confidence (1)</summary>

**helm/hpcc/values.yaml:516**
* [nitpick] Consider removing or clarifying the commented configuration options below 'debug-housekeeping'. Either define a full default structure or remove the commented lines to provide clearer guidance to users.
```
debug-housekeeping: {} # NB: no properties defined, use defaults
```
</details>


state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-05T16:20:05Z
user: login: copilot-pull-request-reviewer[bot]
body: ## PR Overview

This pull request creates a new Sasha service to clean up post mortem files by adding a new debugging configuration for housekeeping tasks. The changes introduce a "debug-housekeeping" block in multiple Helm values files to control the behavior of the service, with variations between production and test configurations.
- Added a debug-housekeeping block with an empty/default configuration in helm/hpcc/values.yaml.
- Enabled/disabled debug-housekeeping in various test files (multicertdomains.yaml, values-hpcc1.yaml, values-hpcc2.yaml, resourced.yaml, resourced2.yaml).

### Reviewed Changes

| File                                   | Description                                                  |
| -------------------------------------- | ------------------------------------------------------------ |
| helm/hpcc/values.yaml                  | Added a debug-housekeeping block with default (empty) values.  |
| testing/helm/tests/multicertdomains.yaml | Added debug-housekeeping block; set to disabled for testing.   |
| testing/helm/tests/values-hpcc2.yaml     | Added debug-housekeeping block; set to disabled for testing.   |
| testing/helm/tests/values-hpcc1.yaml     | Added debug-housekeeping block; set to disabled for testing.   |
| testing/helm/tests/resourced.yaml        | Added debug-housekeeping block with additional resource limits.|
| testing/helm/tests/resourced2.yaml       | Added debug-housekeeping block with additional resource limits.|

Copilot reviewed 20 out of 20 changed files in this pull request and generated no comments.



state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-07T11:51:18Z
user: login: streeterd
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-10T10:36:30Z
user: login: streeterd
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-10T10:37:38Z
user: login: streeterd
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-10T10:39:15Z
user: login: copilot-pull-request-reviewer[bot]
body: ## PR Overview

This PR introduces a new sasha service for cleaning up post mortem files by adding a debug-housekeeping configuration option. The key changes include:
- Adding a debug-housekeeping section with default and commented optional properties in helm/hpcc/values.yaml.
- Updating multiple test configuration files to include a debug-housekeeping configuration with the "disabled" flag.
- Adding resource configuration for debug-housekeeping in resourced test files.

### Reviewed Changes

| File                                    | Description                                                  |
| --------------------------------------- | ------------------------------------------------------------ |
| helm/hpcc/values.yaml                   | Adds a debug-housekeeping configuration with default values. |
| testing/helm/tests/values-hpcc1.yaml      | Introduces debug-housekeeping with "disabled: true".         |
| testing/helm/tests/values-hpcc2.yaml      | Introduces debug-housekeeping with "disabled: true".         |
| testing/helm/tests/multicertdomains.yaml  | Introduces debug-housekeeping with "disabled: true".         |
| testing/helm/tests/resourced2.yaml        | Adds debug-housekeeping with resource limits.                |
| testing/helm/tests/resourced.yaml         | Adds debug-housekeeping with resource limits.                |

Copilot reviewed 17 out of 17 changed files in this pull request and generated 1 comment.

<details>
<summary>Comments suppressed due to low confidence (1)</summary>

**testing/helm/tests/resourced.yaml:514**
* The debug-housekeeping block contains several commented out properties alongside an active resources block, which may cause confusion; consider cleaning up the commented properties or providing clear documentation to standardize its use.
```
debug-housekeeping:
```
</details>


state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-11T09:58:27Z
user: login: streeterd
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-11T10:01:30Z
user: login: jakesmith
body: @streeterd - please see comments
state: CHANGES_REQUESTED
author_association: MEMBER
submitted_at: 2025-03-12T10:14:14Z
user: login: jakesmith
body: @streeterd - looking close - please see comments.
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-13T19:16:47Z
user: login: streeterd
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-14T08:54:20Z
user: login: jakesmith
body: @streeterd - please see comments.
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-17T15:54:54Z
user: login: streeterd
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-18T09:17:23Z
user: login: jakesmith
body: @streeterd - looks good. Please squash.

After that, happy for it to be merged and then add a separate linked documentation JIRA, or could add the documentation md in this PR if you prefer.


state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-19T09:26:57Z
user: login: jakesmith
body: @streeterd - looks good, but you're previous commit with a table detailing the config options looked clearer I think.
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-19T13:04:03Z
user: login: jakesmith
body: @streeterd - spotted a couple of minors worth addressing before this is merged.
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-20T09:15:38Z
user: login: jakesmith
body: @streeterd - looks good, 1 trivial comment.
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-20T11:46:54Z
user: login: jakesmith
body: @streeterd - looks good. Please squash.
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-20T12:14:20Z
user: login: jakesmith
body: @streeterd - looks good to go.
state: DISMISSED
author_association: MEMBER
submitted_at: 2025-03-20T12:28:29Z
user: login: jakesmith
body: Pending on investigations into why k8s GH test failing.
Was failing because was [trying] to connect to Dali, but had no network policy to do so, and isn't needed. Making it conditional caused a seg. fault - to investigate.
state: CHANGES_REQUESTED
author_association: MEMBER
submitted_at: 2025-03-23T11:21:08Z
user: login: jakesmith
body: @streeterd - passing this back to you for now, as BM regression suites are failing
state: CHANGES_REQUESTED
author_association: MEMBER
submitted_at: 2025-03-31T09:11:55Z
user: login: jakesmith
body: @streeterd - please see comments.
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-03T16:16:37Z
user: login: jakesmith
body: @streeterd - Final comments, nothing significant, could be merged now.

Please address and go ahead and squash.
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-08T11:39:48Z
user: login: streeterd
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-08T12:41:20Z
user: login: streeterd
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-08T12:41:41Z
user: login: streeterd
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-08T12:47:08Z
user: login: streeterd
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-08T12:47:14Z
user: login: streeterd
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-08T12:47:19Z
user: login: streeterd
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-08T12:47:25Z
user: login: streeterd
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-08T12:47:30Z
user: login: jakesmith
body: @streeterd - looks good. Please squash.
state: APPROVED
author_association: MEMBER
submitted_at: 2025-04-08T14:33:25Z
user: login: jakesmith
body: @streeterd - looks good.
state: APPROVED
author_association: MEMBER
submitted_at: 2025-04-09T09:41:41Z

Review comments:
diff_hunk: @@ -136,6 +136,15 @@
                                   hpcc:presetValue="0" hpcc:tooltip="minimum idle time before job will switch queues. (in minutes, 0 disables)"/>
                 </xs:attributeGroup>
 
+                <xs:attributeGroup name="dbgHk" hpcc:groupByName="DbgHk" hpcc:docid="sh.t5">
+                    <xs:attribute name="ExpiryInterval" type="xs:nonNegativeInteger" hpcc:displayName="Expiry Interval (hours)"
+                                  hpcc:presetValue="24" hpcc:tooltip="minimum interval between checking for distributed file expiry (in hours, 0 disables)"/>
+                    <xs:attribute name="ExpiryAt" type="cronSchedule" hpcc:displayName="Expiry Schedule (cron format)"
+                                  hpcc:presetValue="* 3 * * *" hpcc:tooltip="schedule to check to remove post-mortem files (cron format)"/>
+                    <xs:attribute name="ExpiryDefault" type="xs:nonNegativeInteger" hpcc:displayName="Expiry Default (days)"
+                                  hpcc:presetValue="14" hpcc:tooltip="Default number of days passed that post-mortem files are deleted after their associated workunit was completed"/>
+                </xs:attributeGroup>
path: initfiles/componentfiles/configschema/xsd/sasha.xsd
user: login: jakesmith
body: componentfiles/* are BM config files. This service will likely never exist in the BM version, and we are trying to keep BM config changes down to a bare minimum (with an aspiration if it ever happened, for BM to use containerized style config). So best not to make these changes at this juncture.
created_at: 2025-03-10T09:27:56Z
author_association: MEMBER
diff_hunk: @@ -513,6 +513,13 @@ sasha:
     #switchMinTime: 1
     #queues: "*"
 
+  debug-housekeeping: {} # NB: no properties defined, use defaults
+    #disabled: true
+    #interval: 1
+    #at: "* 3 * * *"
+    #expiryDefault: 7
+    #user: sasha
path: helm/hpcc/values.yaml
user: login: jakesmith
body: doesn't need a 'user', which is used by file-expiry service to lookup a logical file, and hence needs a user id to authorize.
created_at: 2025-03-10T09:29:52Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,228 @@
+#include "platform.h"
+
+#include "jlib.hpp"
+#include "jiface.hpp"
+#include "jptree.hpp"
+#include "jregexp.hpp"
+
+#include "dadfs.hpp"
+#include "dalienv.hpp"
+
+#include "saserver.hpp"
+#include "sautil.hpp"
+
+#define DEFAULT_EXPIRY_INTERVAL 24 // hours
+
+#define DEFAULT_EXPIRYDAYS 7
+
+#define LOGDBGHK "DEBUGHOUSEKEEPING: "
+
+// Debug Housekeeping monitor
+
+class CSashaDebugHousekeepingServer : public ISashaServer, public Thread
+{
+    bool stopped;
+    Semaphore stopsem;
+    Mutex runmutex;
+    Owned<IUserDescriptor> udesc;
+    Linked<IPropertyTree> props;
+
+public:
+    IMPLEMENT_IINTERFACE_USING(Thread);
+
+    CSashaDebugHousekeepingServer(IPropertyTree *_config)
+        : Thread("CSashaDebugHousekeepingServer"), props(_config)
+    {
+        stopped = false;
+
+        StringBuffer userName;
+        props->getProp("@user", userName);
+        udesc.setown(createUserDescriptor());
+        udesc->set(userName.str(), nullptr);
+    }
+
+    ~CSashaDebugHousekeepingServer()
+    {
+    }
+
+    void start()
+    {
+        Thread::start(false);
+    }
+
+    void ready()
+    {
+    }
+
+    void stop()
+    {
+        if (!stopped)
+        {
+            stopped = true;
+            stopsem.signal();
+        }
+        synchronized block(runmutex); // hopefully stopped should stop
+        if (!join(1000 * 60 * 3))
+            OERRLOG("CSashaDebugHousekeepingServer aborted");
+    }
+
+    void runDebugHousekeeping()
+    {
+        synchronized block(runmutex);
+        if (stopped)
+            return;
+        PROGLOG(LOGDBGHK "Started");
+        unsigned defaultExpireDays = props->getPropInt("@expiryDefault", DEFAULT_EXPIRYDAYS);
+
+        // get debug plane dir
+        StringBuffer debugDir;
+        StringBuffer planeName;
+        if (!getDefaultPlane(planeName, "@debugPlane", "debug"))
+        {
+            WARNLOG("Exception handlers configured, but debug plane is missing");
+            return;
+        }
+        Owned<IPropertyTree> plane = getStoragePlane(planeName);
+        assertex(plane);
+        verifyex(plane->getProp("@prefix", debugDir));
+
+        // iterate debug plane selecting post-mortem directories for housekeeping
+        Owned<IDirectoryIterator> pDirIter = createDirectoryIterator(debugDir.str(), "*", false, true);
+        addPathSepChar(debugDir);
+        ForEach(*pDirIter)
+        {
+            if (stopped)
+                break;
+
+            IFile &iDirFile = pDirIter->query();
+            const char *dirPath = iDirFile.queryFilename();
+
+            if (!dirPath || !*dirPath)
+                continue;
+
+            // Process directories, exclude the ".", "..", non post-mortem and not expired post-mortem directories
+            if (iDirFile.isDirectory() == fileBool::foundYes && *dirPath != '.')
+            {
+                // Ensure directory name only
+                StringBuffer dirNameOnly;
+                String dirNameStr(dirPath);
+                int fwdSlashIndex = dirNameStr.lastIndexOf('/');
+                if (fwdSlashIndex > -1)
+                {
+                    String *tmpStr = dirNameStr.substring(fwdSlashIndex + 1);
+                    dirNameOnly.append(*tmpStr);
+                    delete tmpStr;
+                }
+                else
+                {
+                    dirNameOnly.append(dirNameStr);
+                }
+
+                if (isPostMortemDirPath(dirNameOnly) && isExpiredDirPath(dirNameOnly, defaultExpireDays))
+                {
+                    recursiveRemoveDirectory(dirPath);
+                    PROGLOG(LOGDBGHK "Deleted %s", dirPath);
+                }
+            }
+        }
+        pDirIter.clear();
+
+        PROGLOG(LOGDBGHK "%s", stopped ? "Stopped" : "Done");
+    }
+
+    int run()
+    {
+        unsigned interval = props->getPropInt("@interval", DEFAULT_EXPIRY_INTERVAL);
+        if (!interval)
+            stopped = true;
+        PROGLOG(LOGDBGHK "min interval = %d hr", interval);
+        unsigned initinterval = (interval - 1) / 2; // wait a bit til dali has started
+        CSashaSchedule schedule;
+        if (interval)
+            schedule.init(props, interval, initinterval);
+        initinterval *= 60 * 60 * 1000; // ms
+        unsigned started = msTick();
+        while (!stopped)
+        {
+            stopsem.wait(1000 * 60);
+            if (stopped)
+                break;
+            if (!interval || ((started != (unsigned)-1) && (msTick() - started < initinterval)))
+                continue;
+            started = (unsigned)-1;
+            if (!schedule.ready())
+                continue;
+            try
+            {
+                runDebugHousekeeping();
+            }
+            catch (IException *e)
+            {
+                StringBuffer s;
+                EXCLOG(e, LOGDBGHK);
+                e->Release();
+            }
+        }
+        PROGLOG(LOGDBGHK "Exit");
+        return 0;
+    }
+
+private:
+    bool isPostMortemDirPath(const StringBuffer &dirName)
+    {
+        // Expecting a directory name like "W20250225-101112"
+        RegExpr postMortemDirRegEx("^W[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]-[0-9][0-9][0-9][0-9][0-9][0-9]$");
+        if (postMortemDirRegEx.find(dirName.str()))
+        {
+            PROGLOG(LOGDBGHK "Post-mortem dir: %s", dirName.str());
+
+            return true;
+        }
+        else
+        {
+            PROGLOG(LOGDBGHK "Non post-mortem dir: %s", dirName.str());
path: dali/sasha/sadbghk.cpp
user: login: streeterd
body: @jakesmith do we need this log?
I added it in case there were folders that we not expected.
created_at: 2025-03-10T10:36:30Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,228 @@
+#include "platform.h"
+
+#include "jlib.hpp"
+#include "jiface.hpp"
+#include "jptree.hpp"
+#include "jregexp.hpp"
+
+#include "dadfs.hpp"
+#include "dalienv.hpp"
+
+#include "saserver.hpp"
+#include "sautil.hpp"
+
+#define DEFAULT_EXPIRY_INTERVAL 24 // hours
+
+#define DEFAULT_EXPIRYDAYS 7
+
+#define LOGDBGHK "DEBUGHOUSEKEEPING: "
+
+// Debug Housekeeping monitor
+
+class CSashaDebugHousekeepingServer : public ISashaServer, public Thread
+{
+    bool stopped;
+    Semaphore stopsem;
+    Mutex runmutex;
+    Owned<IUserDescriptor> udesc;
+    Linked<IPropertyTree> props;
+
+public:
+    IMPLEMENT_IINTERFACE_USING(Thread);
+
+    CSashaDebugHousekeepingServer(IPropertyTree *_config)
+        : Thread("CSashaDebugHousekeepingServer"), props(_config)
+    {
+        stopped = false;
+
+        StringBuffer userName;
+        props->getProp("@user", userName);
+        udesc.setown(createUserDescriptor());
+        udesc->set(userName.str(), nullptr);
+    }
+
+    ~CSashaDebugHousekeepingServer()
+    {
+    }
+
+    void start()
+    {
+        Thread::start(false);
+    }
+
+    void ready()
+    {
+    }
+
+    void stop()
+    {
+        if (!stopped)
+        {
+            stopped = true;
+            stopsem.signal();
+        }
+        synchronized block(runmutex); // hopefully stopped should stop
+        if (!join(1000 * 60 * 3))
+            OERRLOG("CSashaDebugHousekeepingServer aborted");
+    }
+
+    void runDebugHousekeeping()
+    {
+        synchronized block(runmutex);
+        if (stopped)
+            return;
+        PROGLOG(LOGDBGHK "Started");
+        unsigned defaultExpireDays = props->getPropInt("@expiryDefault", DEFAULT_EXPIRYDAYS);
+
+        // get debug plane dir
+        StringBuffer debugDir;
+        StringBuffer planeName;
+        if (!getDefaultPlane(planeName, "@debugPlane", "debug"))
+        {
+            WARNLOG("Exception handlers configured, but debug plane is missing");
+            return;
+        }
+        Owned<IPropertyTree> plane = getStoragePlane(planeName);
+        assertex(plane);
+        verifyex(plane->getProp("@prefix", debugDir));
+
+        // iterate debug plane selecting post-mortem directories for housekeeping
+        Owned<IDirectoryIterator> pDirIter = createDirectoryIterator(debugDir.str(), "*", false, true);
+        addPathSepChar(debugDir);
+        ForEach(*pDirIter)
+        {
+            if (stopped)
+                break;
+
+            IFile &iDirFile = pDirIter->query();
+            const char *dirPath = iDirFile.queryFilename();
+
+            if (!dirPath || !*dirPath)
+                continue;
+
+            // Process directories, exclude the ".", "..", non post-mortem and not expired post-mortem directories
+            if (iDirFile.isDirectory() == fileBool::foundYes && *dirPath != '.')
+            {
+                // Ensure directory name only
+                StringBuffer dirNameOnly;
+                String dirNameStr(dirPath);
+                int fwdSlashIndex = dirNameStr.lastIndexOf('/');
+                if (fwdSlashIndex > -1)
+                {
+                    String *tmpStr = dirNameStr.substring(fwdSlashIndex + 1);
+                    dirNameOnly.append(*tmpStr);
+                    delete tmpStr;
+                }
+                else
+                {
+                    dirNameOnly.append(dirNameStr);
+                }
+
+                if (isPostMortemDirPath(dirNameOnly) && isExpiredDirPath(dirNameOnly, defaultExpireDays))
+                {
+                    recursiveRemoveDirectory(dirPath);
+                    PROGLOG(LOGDBGHK "Deleted %s", dirPath);
+                }
+            }
+        }
+        pDirIter.clear();
+
+        PROGLOG(LOGDBGHK "%s", stopped ? "Stopped" : "Done");
+    }
+
+    int run()
+    {
+        unsigned interval = props->getPropInt("@interval", DEFAULT_EXPIRY_INTERVAL);
+        if (!interval)
+            stopped = true;
+        PROGLOG(LOGDBGHK "min interval = %d hr", interval);
+        unsigned initinterval = (interval - 1) / 2; // wait a bit til dali has started
+        CSashaSchedule schedule;
+        if (interval)
+            schedule.init(props, interval, initinterval);
+        initinterval *= 60 * 60 * 1000; // ms
+        unsigned started = msTick();
+        while (!stopped)
+        {
+            stopsem.wait(1000 * 60);
+            if (stopped)
+                break;
+            if (!interval || ((started != (unsigned)-1) && (msTick() - started < initinterval)))
+                continue;
+            started = (unsigned)-1;
+            if (!schedule.ready())
+                continue;
+            try
+            {
+                runDebugHousekeeping();
+            }
+            catch (IException *e)
+            {
+                StringBuffer s;
+                EXCLOG(e, LOGDBGHK);
+                e->Release();
+            }
+        }
+        PROGLOG(LOGDBGHK "Exit");
+        return 0;
+    }
+
+private:
+    bool isPostMortemDirPath(const StringBuffer &dirName)
+    {
+        // Expecting a directory name like "W20250225-101112"
+        RegExpr postMortemDirRegEx("^W[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]-[0-9][0-9][0-9][0-9][0-9][0-9]$");
+        if (postMortemDirRegEx.find(dirName.str()))
+        {
+            PROGLOG(LOGDBGHK "Post-mortem dir: %s", dirName.str());
path: dali/sasha/sadbghk.cpp
user: login: streeterd
body: @jakesmith do we need this log?
Could it be useful as a DBGLOG ?
created_at: 2025-03-10T10:37:38Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,228 @@
+#include "platform.h"
+
+#include "jlib.hpp"
+#include "jiface.hpp"
+#include "jptree.hpp"
+#include "jregexp.hpp"
+
+#include "dadfs.hpp"
+#include "dalienv.hpp"
+
+#include "saserver.hpp"
+#include "sautil.hpp"
+
+#define DEFAULT_EXPIRY_INTERVAL 24 // hours
+
+#define DEFAULT_EXPIRYDAYS 7
+
+#define LOGDBGHK "DEBUGHOUSEKEEPING: "
+
+// Debug Housekeeping monitor
+
+class CSashaDebugHousekeepingServer : public ISashaServer, public Thread
+{
+    bool stopped;
+    Semaphore stopsem;
+    Mutex runmutex;
+    Owned<IUserDescriptor> udesc;
+    Linked<IPropertyTree> props;
+
+public:
+    IMPLEMENT_IINTERFACE_USING(Thread);
+
+    CSashaDebugHousekeepingServer(IPropertyTree *_config)
+        : Thread("CSashaDebugHousekeepingServer"), props(_config)
+    {
+        stopped = false;
+
+        StringBuffer userName;
+        props->getProp("@user", userName);
+        udesc.setown(createUserDescriptor());
+        udesc->set(userName.str(), nullptr);
+    }
+
+    ~CSashaDebugHousekeepingServer()
+    {
+    }
+
+    void start()
+    {
+        Thread::start(false);
+    }
+
+    void ready()
+    {
+    }
+
+    void stop()
+    {
+        if (!stopped)
+        {
+            stopped = true;
+            stopsem.signal();
+        }
+        synchronized block(runmutex); // hopefully stopped should stop
+        if (!join(1000 * 60 * 3))
+            OERRLOG("CSashaDebugHousekeepingServer aborted");
+    }
+
+    void runDebugHousekeeping()
+    {
+        synchronized block(runmutex);
+        if (stopped)
+            return;
+        PROGLOG(LOGDBGHK "Started");
+        unsigned defaultExpireDays = props->getPropInt("@expiryDefault", DEFAULT_EXPIRYDAYS);
+
+        // get debug plane dir
+        StringBuffer debugDir;
+        StringBuffer planeName;
+        if (!getDefaultPlane(planeName, "@debugPlane", "debug"))
+        {
+            WARNLOG("Exception handlers configured, but debug plane is missing");
+            return;
+        }
+        Owned<IPropertyTree> plane = getStoragePlane(planeName);
+        assertex(plane);
+        verifyex(plane->getProp("@prefix", debugDir));
+
+        // iterate debug plane selecting post-mortem directories for housekeeping
+        Owned<IDirectoryIterator> pDirIter = createDirectoryIterator(debugDir.str(), "*", false, true);
+        addPathSepChar(debugDir);
+        ForEach(*pDirIter)
+        {
+            if (stopped)
+                break;
+
+            IFile &iDirFile = pDirIter->query();
+            const char *dirPath = iDirFile.queryFilename();
+
+            if (!dirPath || !*dirPath)
+                continue;
+
+            // Process directories, exclude the ".", "..", non post-mortem and not expired post-mortem directories
+            if (iDirFile.isDirectory() == fileBool::foundYes && *dirPath != '.')
+            {
+                // Ensure directory name only
+                StringBuffer dirNameOnly;
+                String dirNameStr(dirPath);
+                int fwdSlashIndex = dirNameStr.lastIndexOf('/');
+                if (fwdSlashIndex > -1)
+                {
+                    String *tmpStr = dirNameStr.substring(fwdSlashIndex + 1);
+                    dirNameOnly.append(*tmpStr);
+                    delete tmpStr;
+                }
+                else
+                {
+                    dirNameOnly.append(dirNameStr);
+                }
+
+                if (isPostMortemDirPath(dirNameOnly) && isExpiredDirPath(dirNameOnly, defaultExpireDays))
+                {
+                    recursiveRemoveDirectory(dirPath);
+                    PROGLOG(LOGDBGHK "Deleted %s", dirPath);
+                }
+            }
+        }
+        pDirIter.clear();
+
+        PROGLOG(LOGDBGHK "%s", stopped ? "Stopped" : "Done");
+    }
+
+    int run()
+    {
+        unsigned interval = props->getPropInt("@interval", DEFAULT_EXPIRY_INTERVAL);
+        if (!interval)
+            stopped = true;
+        PROGLOG(LOGDBGHK "min interval = %d hr", interval);
+        unsigned initinterval = (interval - 1) / 2; // wait a bit til dali has started
+        CSashaSchedule schedule;
+        if (interval)
+            schedule.init(props, interval, initinterval);
+        initinterval *= 60 * 60 * 1000; // ms
+        unsigned started = msTick();
+        while (!stopped)
+        {
+            stopsem.wait(1000 * 60);
+            if (stopped)
+                break;
+            if (!interval || ((started != (unsigned)-1) && (msTick() - started < initinterval)))
+                continue;
+            started = (unsigned)-1;
+            if (!schedule.ready())
+                continue;
+            try
+            {
+                runDebugHousekeeping();
+            }
+            catch (IException *e)
+            {
+                StringBuffer s;
+                EXCLOG(e, LOGDBGHK);
+                e->Release();
+            }
+        }
+        PROGLOG(LOGDBGHK "Exit");
+        return 0;
+    }
+
+private:
+    bool isPostMortemDirPath(const StringBuffer &dirName)
+    {
+        // Expecting a directory name like "W20250225-101112"
+        RegExpr postMortemDirRegEx("^W[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]-[0-9][0-9][0-9][0-9][0-9][0-9]$");
+        if (postMortemDirRegEx.find(dirName.str()))
+        {
+            PROGLOG(LOGDBGHK "Post-mortem dir: %s", dirName.str());
+
+            return true;
+        }
+        else
+        {
+            PROGLOG(LOGDBGHK "Non post-mortem dir: %s", dirName.str());
+
+            return false;
+        }
+    }
+
+    bool isExpiredDirPath(StringBuffer &dirName, const unsigned &defaultExpireDays)
+    {
+        // Directory name is like "W20250225-101112"
+        StringBuffer cDateTimeoprmattedString;
+        cDateTimeoprmattedString.appendf("%c%c%c%c-%c%c-%c%cT%c%c:%c%c:%c%c",
+                                         dirName.charAt(1), dirName.charAt(2), dirName.charAt(3), dirName.charAt(4),
+                                         dirName.charAt(5), dirName.charAt(6),
+                                         dirName.charAt(7), dirName.charAt(8),
+                                         dirName.charAt(10), dirName.charAt(11),
+                                         dirName.charAt(12), dirName.charAt(13),
+                                         dirName.charAt(14), dirName.charAt(15));
+
+        CDateTime now;
+        now.setNow();
+
+        CDateTime expires;
+        expires.setString(cDateTimeoprmattedString.str());
+        expires.adjustTime(60 * 24 * defaultExpireDays);
+
+        if (now.compare(expires, false) > 0)
+        {
+            PROGLOG(LOGDBGHK "Post-mortem dir: %s has expired", dirName.str());
path: dali/sasha/sadbghk.cpp
user: login: streeterd
body: @jakesmith is this log needed?
The dir will be deleted and logged anyway.
created_at: 2025-03-10T10:39:15Z
author_association: CONTRIBUTOR
diff_hunk: @@ -513,6 +513,13 @@ sasha:
     #switchMinTime: 1
     #queues: "*"
 
+  debug-housekeeping: {} # NB: no properties defined, use defaults
+    #disabled: true
path: helm/hpcc/values.yaml
user: login: jakesmith
body: more/later: if enabled, we could validate statically (at helm install time) that a debug plane exists (it's optional so it doesn't necessarily exist)
created_at: 2025-03-11T09:50:14Z
author_association: MEMBER
diff_hunk: @@ -513,6 +513,13 @@ sasha:
     #switchMinTime: 1
     #queues: "*"
 
+  debug-housekeeping: {} # NB: no properties defined, use defaults
path: helm/hpcc/values.yaml
user: login: Copilot
body: The inline declaration using '{}' differs in structure from the block style used in test files; consider aligning the structure (e.g. using a block style with a 'disabled' property) to ensure consistent configuration behavior.
created_at: 2025-03-11T09:58:26Z
author_association: CONTRIBUTOR
diff_hunk: @@ -513,6 +513,13 @@ sasha:
     #switchMinTime: 1
     #queues: "*"
 
+  debug-housekeeping: {} # NB: no properties defined, use defaults
path: helm/hpcc/values.yaml
user: login: streeterd
body: The new code matches the existing code.
created_at: 2025-03-11T10:01:30Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,224 @@
+#include "platform.h"
+
+#include "jlib.hpp"
+#include "jiface.hpp"
+#include "jptree.hpp"
+#include "jregexp.hpp"
+
+#include "dadfs.hpp"
+#include "dalienv.hpp"
+
+#include "saserver.hpp"
+#include "sautil.hpp"
+
+#define DEFAULT_EXPIRY_INTERVAL 24 // hours
+
+#define DEFAULT_EXPIRYDAYS 7
+
+#define LOGDBGHK "DEBUGHOUSEKEEPING: "
+
+// Debug Housekeeping monitor
+
+class CSashaDebugHousekeepingServer : public ISashaServer, public Thread
path: dali/sasha/sadbghk.cpp
user: login: jakesmith
body: trivial: for clarity, I prefer it to reference '..DebugPlane..' vs '..Debug..' here/elsewhere.
created_at: 2025-03-12T09:46:01Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,224 @@
+#include "platform.h"
+
+#include "jlib.hpp"
+#include "jiface.hpp"
+#include "jptree.hpp"
+#include "jregexp.hpp"
+
+#include "dadfs.hpp"
+#include "dalienv.hpp"
+
+#include "saserver.hpp"
+#include "sautil.hpp"
+
+#define DEFAULT_EXPIRY_INTERVAL 24 // hours
+
+#define DEFAULT_EXPIRYDAYS 7
+
+#define LOGDBGHK "DEBUGHOUSEKEEPING: "
+
+// Debug Housekeeping monitor
+
+class CSashaDebugHousekeepingServer : public ISashaServer, public Thread
+{
+    bool stopped;
path: dali/sasha/sadbghk.cpp
user: login: jakesmith
body: style: there's plenty of code like this, but it is preferable to initialize member variables at this point, rather than in the ctor.
created_at: 2025-03-12T09:46:47Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,224 @@
+#include "platform.h"
+
+#include "jlib.hpp"
+#include "jiface.hpp"
+#include "jptree.hpp"
+#include "jregexp.hpp"
+
+#include "dadfs.hpp"
+#include "dalienv.hpp"
+
+#include "saserver.hpp"
+#include "sautil.hpp"
+
+#define DEFAULT_EXPIRY_INTERVAL 24 // hours
+
+#define DEFAULT_EXPIRYDAYS 7
+
+#define LOGDBGHK "DEBUGHOUSEKEEPING: "
+
+// Debug Housekeeping monitor
+
+class CSashaDebugHousekeepingServer : public ISashaServer, public Thread
+{
+    bool stopped;
+    Semaphore stopsem;
+    Mutex runmutex;
+    Linked<IPropertyTree> props;
path: dali/sasha/sadbghk.cpp
user: login: jakesmith
body: See other comments - as this is a containerized service, not really needed - should use getComponentConfig*() directly.
created_at: 2025-03-12T09:47:50Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,224 @@
+#include "platform.h"
+
+#include "jlib.hpp"
+#include "jiface.hpp"
+#include "jptree.hpp"
+#include "jregexp.hpp"
+
+#include "dadfs.hpp"
+#include "dalienv.hpp"
+
+#include "saserver.hpp"
+#include "sautil.hpp"
+
+#define DEFAULT_EXPIRY_INTERVAL 24 // hours
+
+#define DEFAULT_EXPIRYDAYS 7
+
+#define LOGDBGHK "DEBUGHOUSEKEEPING: "
+
+// Debug Housekeeping monitor
+
+class CSashaDebugHousekeepingServer : public ISashaServer, public Thread
+{
+    bool stopped;
+    Semaphore stopsem;
+    Mutex runmutex;
+    Linked<IPropertyTree> props;
+
+public:
+    IMPLEMENT_IINTERFACE_USING(Thread);
+
+    CSashaDebugHousekeepingServer(IPropertyTree *_config)
path: dali/sasha/sadbghk.cpp
user: login: jakesmith
body: re. containerized only, no need for passed in config.
created_at: 2025-03-12T09:48:06Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,224 @@
+#include "platform.h"
+
+#include "jlib.hpp"
+#include "jiface.hpp"
+#include "jptree.hpp"
+#include "jregexp.hpp"
+
+#include "dadfs.hpp"
+#include "dalienv.hpp"
+
+#include "saserver.hpp"
+#include "sautil.hpp"
+
+#define DEFAULT_EXPIRY_INTERVAL 24 // hours
+
+#define DEFAULT_EXPIRYDAYS 7
+
+#define LOGDBGHK "DEBUGHOUSEKEEPING: "
+
+// Debug Housekeeping monitor
+
+class CSashaDebugHousekeepingServer : public ISashaServer, public Thread
+{
+    bool stopped;
+    Semaphore stopsem;
+    Mutex runmutex;
+    Linked<IPropertyTree> props;
+
+public:
+    IMPLEMENT_IINTERFACE_USING(Thread);
+
+    CSashaDebugHousekeepingServer(IPropertyTree *_config)
+        : Thread("CSashaDebugHousekeepingServer"), props(_config)
+    {
+        stopped = false;
path: dali/sasha/sadbghk.cpp
user: login: jakesmith
body: trivial: initialize in ctor.
created_at: 2025-03-12T09:48:18Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,224 @@
+#include "platform.h"
+
+#include "jlib.hpp"
+#include "jiface.hpp"
+#include "jptree.hpp"
+#include "jregexp.hpp"
+
+#include "dadfs.hpp"
+#include "dalienv.hpp"
+
+#include "saserver.hpp"
+#include "sautil.hpp"
+
+#define DEFAULT_EXPIRY_INTERVAL 24 // hours
+
+#define DEFAULT_EXPIRYDAYS 7
+
+#define LOGDBGHK "DEBUGHOUSEKEEPING: "
+
+// Debug Housekeeping monitor
+
+class CSashaDebugHousekeepingServer : public ISashaServer, public Thread
+{
+    bool stopped;
+    Semaphore stopsem;
+    Mutex runmutex;
+    Linked<IPropertyTree> props;
+
+public:
+    IMPLEMENT_IINTERFACE_USING(Thread);
+
+    CSashaDebugHousekeepingServer(IPropertyTree *_config)
+        : Thread("CSashaDebugHousekeepingServer"), props(_config)
+    {
+        stopped = false;
+
+        StringBuffer userName;
+        props->getProp("@user", userName);
path: dali/sasha/sadbghk.cpp
user: login: jakesmith
body: no user needed - the service is deleting physical files from the debug plane only.
created_at: 2025-03-12T09:48:46Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,224 @@
+#include "platform.h"
+
+#include "jlib.hpp"
+#include "jiface.hpp"
+#include "jptree.hpp"
+#include "jregexp.hpp"
+
+#include "dadfs.hpp"
+#include "dalienv.hpp"
+
+#include "saserver.hpp"
+#include "sautil.hpp"
+
+#define DEFAULT_EXPIRY_INTERVAL 24 // hours
+
+#define DEFAULT_EXPIRYDAYS 7
+
+#define LOGDBGHK "DEBUGHOUSEKEEPING: "
+
+// Debug Housekeeping monitor
+
+class CSashaDebugHousekeepingServer : public ISashaServer, public Thread
+{
+    bool stopped;
+    Semaphore stopsem;
+    Mutex runmutex;
+    Linked<IPropertyTree> props;
+
+public:
+    IMPLEMENT_IINTERFACE_USING(Thread);
+
+    CSashaDebugHousekeepingServer(IPropertyTree *_config)
+        : Thread("CSashaDebugHousekeepingServer"), props(_config)
+    {
+        stopped = false;
+
+        StringBuffer userName;
+        props->getProp("@user", userName);
+    }
+
+    ~CSashaDebugHousekeepingServer()
+    {
+    }
path: dali/sasha/sadbghk.cpp
user: login: jakesmith
body: trivial/style: no need for empty dtor.
created_at: 2025-03-12T09:49:00Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,224 @@
+#include "platform.h"
+
+#include "jlib.hpp"
+#include "jiface.hpp"
+#include "jptree.hpp"
+#include "jregexp.hpp"
+
+#include "dadfs.hpp"
+#include "dalienv.hpp"
+
+#include "saserver.hpp"
+#include "sautil.hpp"
+
+#define DEFAULT_EXPIRY_INTERVAL 24 // hours
+
+#define DEFAULT_EXPIRYDAYS 7
+
+#define LOGDBGHK "DEBUGHOUSEKEEPING: "
+
+// Debug Housekeeping monitor
+
+class CSashaDebugHousekeepingServer : public ISashaServer, public Thread
+{
+    bool stopped;
+    Semaphore stopsem;
+    Mutex runmutex;
+    Linked<IPropertyTree> props;
+
+public:
+    IMPLEMENT_IINTERFACE_USING(Thread);
+
+    CSashaDebugHousekeepingServer(IPropertyTree *_config)
+        : Thread("CSashaDebugHousekeepingServer"), props(_config)
+    {
+        stopped = false;
+
+        StringBuffer userName;
+        props->getProp("@user", userName);
+    }
+
+    ~CSashaDebugHousekeepingServer()
+    {
+    }
+
+    void start()
path: dali/sasha/sadbghk.cpp
user: login: jakesmith
body: style: we should always add 'virtual' for clarity and 'override' to prevent accidental overloads on implementation of virtual abstract methods.
created_at: 2025-03-12T09:50:17Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,224 @@
+#include "platform.h"
+
+#include "jlib.hpp"
+#include "jiface.hpp"
+#include "jptree.hpp"
+#include "jregexp.hpp"
+
+#include "dadfs.hpp"
+#include "dalienv.hpp"
+
+#include "saserver.hpp"
+#include "sautil.hpp"
+
+#define DEFAULT_EXPIRY_INTERVAL 24 // hours
+
+#define DEFAULT_EXPIRYDAYS 7
+
+#define LOGDBGHK "DEBUGHOUSEKEEPING: "
+
+// Debug Housekeeping monitor
+
+class CSashaDebugHousekeepingServer : public ISashaServer, public Thread
+{
+    bool stopped;
+    Semaphore stopsem;
+    Mutex runmutex;
+    Linked<IPropertyTree> props;
+
+public:
+    IMPLEMENT_IINTERFACE_USING(Thread);
+
+    CSashaDebugHousekeepingServer(IPropertyTree *_config)
+        : Thread("CSashaDebugHousekeepingServer"), props(_config)
+    {
+        stopped = false;
+
+        StringBuffer userName;
+        props->getProp("@user", userName);
+    }
+
+    ~CSashaDebugHousekeepingServer()
+    {
+    }
+
+    void start()
+    {
+        Thread::start(false);
+    }
+
+    void ready()
+    {
+    }
+
+    void stop()
+    {
+        if (!stopped)
+        {
+            stopped = true;
+            stopsem.signal();
+        }
+        synchronized block(runmutex); // hopefully stopped should stop
+        if (!join(1000 * 60 * 3))
+            OERRLOG("CSashaDebugHousekeepingServer aborted");
+    }
+
+    void runDebugHousekeeping()
+    {
+        synchronized block(runmutex);
+        if (stopped)
+            return;
+        PROGLOG(LOGDBGHK "Started");
+        unsigned defaultExpireDays = props->getPropInt("@expiryDefault", DEFAULT_EXPIRYDAYS);
+
+        // get debug plane dir
+        StringBuffer debugDir;
+        StringBuffer planeName;
+        if (!getDefaultPlane(planeName, "@debugPlane", "debug"))
+        {
+            WARNLOG("Exception handlers configured, but debug plane is missing");
+            return;
+        }
+        Owned<IPropertyTree> plane = getStoragePlane(planeName);
+        assertex(plane);
+        verifyex(plane->getProp("@prefix", debugDir));
path: dali/sasha/sadbghk.cpp
user: login: jakesmith
body: should get expiryDays and debug plane details once and store as members, e.g. in ctor.
created_at: 2025-03-12T09:51:35Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,224 @@
+#include "platform.h"
+
+#include "jlib.hpp"
+#include "jiface.hpp"
+#include "jptree.hpp"
+#include "jregexp.hpp"
+
+#include "dadfs.hpp"
+#include "dalienv.hpp"
+
+#include "saserver.hpp"
+#include "sautil.hpp"
+
+#define DEFAULT_EXPIRY_INTERVAL 24 // hours
+
+#define DEFAULT_EXPIRYDAYS 7
+
+#define LOGDBGHK "DEBUGHOUSEKEEPING: "
+
+// Debug Housekeeping monitor
+
+class CSashaDebugHousekeepingServer : public ISashaServer, public Thread
+{
+    bool stopped;
+    Semaphore stopsem;
+    Mutex runmutex;
+    Linked<IPropertyTree> props;
+
+public:
+    IMPLEMENT_IINTERFACE_USING(Thread);
+
+    CSashaDebugHousekeepingServer(IPropertyTree *_config)
+        : Thread("CSashaDebugHousekeepingServer"), props(_config)
+    {
+        stopped = false;
+
+        StringBuffer userName;
+        props->getProp("@user", userName);
+    }
+
+    ~CSashaDebugHousekeepingServer()
+    {
+    }
+
+    void start()
+    {
+        Thread::start(false);
+    }
+
+    void ready()
+    {
+    }
+
+    void stop()
+    {
+        if (!stopped)
+        {
+            stopped = true;
+            stopsem.signal();
+        }
+        synchronized block(runmutex); // hopefully stopped should stop
+        if (!join(1000 * 60 * 3))
+            OERRLOG("CSashaDebugHousekeepingServer aborted");
+    }
+
+    void runDebugHousekeeping()
+    {
+        synchronized block(runmutex);
+        if (stopped)
+            return;
+        PROGLOG(LOGDBGHK "Started");
+        unsigned defaultExpireDays = props->getPropInt("@expiryDefault", DEFAULT_EXPIRYDAYS);
path: dali/sasha/sadbghk.cpp
user: login: jakesmith
body: DEFAULT_EXPIRYDAYS is a default - but the configuration option isn't really a "default", since no way of overriding the expiryDefault (unlike the file expiry service, where the config default can be overridden by individual files settings).

I suggest renaming to "expiryCutoff".
created_at: 2025-03-12T09:53:17Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,224 @@
+#include "platform.h"
+
+#include "jlib.hpp"
+#include "jiface.hpp"
+#include "jptree.hpp"
+#include "jregexp.hpp"
+
+#include "dadfs.hpp"
+#include "dalienv.hpp"
+
+#include "saserver.hpp"
+#include "sautil.hpp"
+
+#define DEFAULT_EXPIRY_INTERVAL 24 // hours
+
+#define DEFAULT_EXPIRYDAYS 7
+
+#define LOGDBGHK "DEBUGHOUSEKEEPING: "
+
+// Debug Housekeeping monitor
+
+class CSashaDebugHousekeepingServer : public ISashaServer, public Thread
+{
+    bool stopped;
+    Semaphore stopsem;
+    Mutex runmutex;
+    Linked<IPropertyTree> props;
+
+public:
+    IMPLEMENT_IINTERFACE_USING(Thread);
+
+    CSashaDebugHousekeepingServer(IPropertyTree *_config)
+        : Thread("CSashaDebugHousekeepingServer"), props(_config)
+    {
+        stopped = false;
+
+        StringBuffer userName;
+        props->getProp("@user", userName);
+    }
+
+    ~CSashaDebugHousekeepingServer()
+    {
+    }
+
+    void start()
+    {
+        Thread::start(false);
+    }
+
+    void ready()
+    {
+    }
+
+    void stop()
+    {
+        if (!stopped)
+        {
+            stopped = true;
+            stopsem.signal();
+        }
+        synchronized block(runmutex); // hopefully stopped should stop
+        if (!join(1000 * 60 * 3))
+            OERRLOG("CSashaDebugHousekeepingServer aborted");
+    }
+
+    void runDebugHousekeeping()
+    {
+        synchronized block(runmutex);
+        if (stopped)
+            return;
+        PROGLOG(LOGDBGHK "Started");
+        unsigned defaultExpireDays = props->getPropInt("@expiryDefault", DEFAULT_EXPIRYDAYS);
+
+        // get debug plane dir
+        StringBuffer debugDir;
+        StringBuffer planeName;
+        if (!getDefaultPlane(planeName, "@debugPlane", "debug"))
+        {
+            WARNLOG("Exception handlers configured, but debug plane is missing");
+            return;
+        }
+        Owned<IPropertyTree> plane = getStoragePlane(planeName);
+        assertex(plane);
+        verifyex(plane->getProp("@prefix", debugDir));
+
+        // iterate debug plane selecting post-mortem directories for housekeeping
+        Owned<IDirectoryIterator> pDirIter = createDirectoryIterator(debugDir.str(), "*", false, true);
+        ForEach(*pDirIter)
+        {
+            if (stopped)
+                break;
+
+            IFile &iDirFile = pDirIter->query();
+            const char *dirPath = iDirFile.queryFilename();
+
+            if (!dirPath || !*dirPath)
path: dali/sasha/sadbghk.cpp
user: login: jakesmith
body: I don't think it can ever be blank - (isEmptyString() preferrable if testing this kind of thing)
created_at: 2025-03-12T09:54:37Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,224 @@
+#include "platform.h"
+
+#include "jlib.hpp"
+#include "jiface.hpp"
+#include "jptree.hpp"
+#include "jregexp.hpp"
+
+#include "dadfs.hpp"
+#include "dalienv.hpp"
+
+#include "saserver.hpp"
+#include "sautil.hpp"
+
+#define DEFAULT_EXPIRY_INTERVAL 24 // hours
+
+#define DEFAULT_EXPIRYDAYS 7
+
+#define LOGDBGHK "DEBUGHOUSEKEEPING: "
+
+// Debug Housekeeping monitor
+
+class CSashaDebugHousekeepingServer : public ISashaServer, public Thread
+{
+    bool stopped;
+    Semaphore stopsem;
+    Mutex runmutex;
+    Linked<IPropertyTree> props;
+
+public:
+    IMPLEMENT_IINTERFACE_USING(Thread);
+
+    CSashaDebugHousekeepingServer(IPropertyTree *_config)
+        : Thread("CSashaDebugHousekeepingServer"), props(_config)
+    {
+        stopped = false;
+
+        StringBuffer userName;
+        props->getProp("@user", userName);
+    }
+
+    ~CSashaDebugHousekeepingServer()
+    {
+    }
+
+    void start()
+    {
+        Thread::start(false);
+    }
+
+    void ready()
+    {
+    }
+
+    void stop()
+    {
+        if (!stopped)
+        {
+            stopped = true;
+            stopsem.signal();
+        }
+        synchronized block(runmutex); // hopefully stopped should stop
+        if (!join(1000 * 60 * 3))
+            OERRLOG("CSashaDebugHousekeepingServer aborted");
+    }
+
+    void runDebugHousekeeping()
+    {
+        synchronized block(runmutex);
+        if (stopped)
+            return;
+        PROGLOG(LOGDBGHK "Started");
+        unsigned defaultExpireDays = props->getPropInt("@expiryDefault", DEFAULT_EXPIRYDAYS);
+
+        // get debug plane dir
+        StringBuffer debugDir;
+        StringBuffer planeName;
+        if (!getDefaultPlane(planeName, "@debugPlane", "debug"))
+        {
+            WARNLOG("Exception handlers configured, but debug plane is missing");
+            return;
+        }
+        Owned<IPropertyTree> plane = getStoragePlane(planeName);
+        assertex(plane);
+        verifyex(plane->getProp("@prefix", debugDir));
+
+        // iterate debug plane selecting post-mortem directories for housekeeping
+        Owned<IDirectoryIterator> pDirIter = createDirectoryIterator(debugDir.str(), "*", false, true);
+        ForEach(*pDirIter)
+        {
+            if (stopped)
+                break;
+
+            IFile &iDirFile = pDirIter->query();
+            const char *dirPath = iDirFile.queryFilename();
+
+            if (!dirPath || !*dirPath)
+                continue;
+
+            // Process directories, exclude the ".", "..", non post-mortem and not expired post-mortem directories
+            if (iDirFile.isDirectory() == fileBool::foundYes && *dirPath != '.')
path: dali/sasha/sadbghk.cpp
user: login: jakesmith
body: we want to delete all files that and directories (excluding special ones like . and ..) from the debug plane if expired..

created_at: 2025-03-12T09:55:13Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,224 @@
+#include "platform.h"
+
+#include "jlib.hpp"
+#include "jiface.hpp"
+#include "jptree.hpp"
+#include "jregexp.hpp"
+
+#include "dadfs.hpp"
+#include "dalienv.hpp"
+
+#include "saserver.hpp"
+#include "sautil.hpp"
+
+#define DEFAULT_EXPIRY_INTERVAL 24 // hours
+
+#define DEFAULT_EXPIRYDAYS 7
+
+#define LOGDBGHK "DEBUGHOUSEKEEPING: "
+
+// Debug Housekeeping monitor
+
+class CSashaDebugHousekeepingServer : public ISashaServer, public Thread
+{
+    bool stopped;
+    Semaphore stopsem;
+    Mutex runmutex;
+    Linked<IPropertyTree> props;
+
+public:
+    IMPLEMENT_IINTERFACE_USING(Thread);
+
+    CSashaDebugHousekeepingServer(IPropertyTree *_config)
+        : Thread("CSashaDebugHousekeepingServer"), props(_config)
+    {
+        stopped = false;
+
+        StringBuffer userName;
+        props->getProp("@user", userName);
+    }
+
+    ~CSashaDebugHousekeepingServer()
+    {
+    }
+
+    void start()
+    {
+        Thread::start(false);
+    }
+
+    void ready()
+    {
+    }
+
+    void stop()
+    {
+        if (!stopped)
+        {
+            stopped = true;
+            stopsem.signal();
+        }
+        synchronized block(runmutex); // hopefully stopped should stop
+        if (!join(1000 * 60 * 3))
+            OERRLOG("CSashaDebugHousekeepingServer aborted");
+    }
+
+    void runDebugHousekeeping()
+    {
+        synchronized block(runmutex);
+        if (stopped)
+            return;
+        PROGLOG(LOGDBGHK "Started");
+        unsigned defaultExpireDays = props->getPropInt("@expiryDefault", DEFAULT_EXPIRYDAYS);
+
+        // get debug plane dir
+        StringBuffer debugDir;
+        StringBuffer planeName;
+        if (!getDefaultPlane(planeName, "@debugPlane", "debug"))
+        {
+            WARNLOG("Exception handlers configured, but debug plane is missing");
+            return;
+        }
+        Owned<IPropertyTree> plane = getStoragePlane(planeName);
+        assertex(plane);
+        verifyex(plane->getProp("@prefix", debugDir));
+
+        // iterate debug plane selecting post-mortem directories for housekeeping
+        Owned<IDirectoryIterator> pDirIter = createDirectoryIterator(debugDir.str(), "*", false, true);
+        ForEach(*pDirIter)
+        {
+            if (stopped)
+                break;
+
+            IFile &iDirFile = pDirIter->query();
+            const char *dirPath = iDirFile.queryFilename();
+
+            if (!dirPath || !*dirPath)
+                continue;
+
+            // Process directories, exclude the ".", "..", non post-mortem and not expired post-mortem directories
+            if (iDirFile.isDirectory() == fileBool::foundYes && *dirPath != '.')
+            {
+                // Ensure directory name only
+                StringBuffer dirNameOnly;
+                String dirNameStr(dirPath);
+                int fwdSlashIndex = dirNameStr.lastIndexOf('/');
+                if (fwdSlashIndex > -1)
+                {
+                    String *tmpStr = dirNameStr.substring(fwdSlashIndex + 1);
+                    dirNameOnly.append(*tmpStr);
+                    delete tmpStr;
+                }
+                else
+                {
+                    dirNameOnly.append(dirNameStr);
+                }
+
+                if (isPostMortemDirPath(dirNameOnly) && isExpiredDirPath(dirNameOnly, defaultExpireDays))
path: dali/sasha/sadbghk.cpp
user: login: jakesmith
body: all files should be deleted if expired, not just ones that meet this pattern, so I think isPostMortemDirPath can be deleted.

created_at: 2025-03-12T09:56:09Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,224 @@
+#include "platform.h"
+
+#include "jlib.hpp"
+#include "jiface.hpp"
+#include "jptree.hpp"
+#include "jregexp.hpp"
+
+#include "dadfs.hpp"
+#include "dalienv.hpp"
+
+#include "saserver.hpp"
+#include "sautil.hpp"
+
+#define DEFAULT_EXPIRY_INTERVAL 24 // hours
+
+#define DEFAULT_EXPIRYDAYS 7
+
+#define LOGDBGHK "DEBUGHOUSEKEEPING: "
+
+// Debug Housekeeping monitor
+
+class CSashaDebugHousekeepingServer : public ISashaServer, public Thread
+{
+    bool stopped;
+    Semaphore stopsem;
+    Mutex runmutex;
+    Linked<IPropertyTree> props;
+
+public:
+    IMPLEMENT_IINTERFACE_USING(Thread);
+
+    CSashaDebugHousekeepingServer(IPropertyTree *_config)
+        : Thread("CSashaDebugHousekeepingServer"), props(_config)
+    {
+        stopped = false;
+
+        StringBuffer userName;
+        props->getProp("@user", userName);
+    }
+
+    ~CSashaDebugHousekeepingServer()
+    {
+    }
+
+    void start()
+    {
+        Thread::start(false);
+    }
+
+    void ready()
+    {
+    }
+
+    void stop()
+    {
+        if (!stopped)
+        {
+            stopped = true;
+            stopsem.signal();
+        }
+        synchronized block(runmutex); // hopefully stopped should stop
+        if (!join(1000 * 60 * 3))
+            OERRLOG("CSashaDebugHousekeepingServer aborted");
+    }
+
+    void runDebugHousekeeping()
+    {
+        synchronized block(runmutex);
+        if (stopped)
+            return;
+        PROGLOG(LOGDBGHK "Started");
+        unsigned defaultExpireDays = props->getPropInt("@expiryDefault", DEFAULT_EXPIRYDAYS);
+
+        // get debug plane dir
+        StringBuffer debugDir;
+        StringBuffer planeName;
+        if (!getDefaultPlane(planeName, "@debugPlane", "debug"))
+        {
+            WARNLOG("Exception handlers configured, but debug plane is missing");
+            return;
+        }
+        Owned<IPropertyTree> plane = getStoragePlane(planeName);
+        assertex(plane);
+        verifyex(plane->getProp("@prefix", debugDir));
+
+        // iterate debug plane selecting post-mortem directories for housekeeping
+        Owned<IDirectoryIterator> pDirIter = createDirectoryIterator(debugDir.str(), "*", false, true);
+        ForEach(*pDirIter)
+        {
+            if (stopped)
+                break;
+
+            IFile &iDirFile = pDirIter->query();
+            const char *dirPath = iDirFile.queryFilename();
+
+            if (!dirPath || !*dirPath)
+                continue;
+
+            // Process directories, exclude the ".", "..", non post-mortem and not expired post-mortem directories
+            if (iDirFile.isDirectory() == fileBool::foundYes && *dirPath != '.')
+            {
+                // Ensure directory name only
+                StringBuffer dirNameOnly;
+                String dirNameStr(dirPath);
+                int fwdSlashIndex = dirNameStr.lastIndexOf('/');
+                if (fwdSlashIndex > -1)
+                {
+                    String *tmpStr = dirNameStr.substring(fwdSlashIndex + 1);
+                    dirNameOnly.append(*tmpStr);
+                    delete tmpStr;
+                }
+                else
+                {
+                    dirNameOnly.append(dirNameStr);
+                }
+
+                if (isPostMortemDirPath(dirNameOnly) && isExpiredDirPath(dirNameOnly, defaultExpireDays))
path: dali/sasha/sadbghk.cpp
user: login: jakesmith
body: isExpired() should be based on the file's (iFile) modified date stamp ..
created_at: 2025-03-12T09:56:53Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,224 @@
+#include "platform.h"
+
+#include "jlib.hpp"
+#include "jiface.hpp"
+#include "jptree.hpp"
+#include "jregexp.hpp"
+
+#include "dadfs.hpp"
+#include "dalienv.hpp"
+
+#include "saserver.hpp"
+#include "sautil.hpp"
+
+#define DEFAULT_EXPIRY_INTERVAL 24 // hours
+
+#define DEFAULT_EXPIRYDAYS 7
+
+#define LOGDBGHK "DEBUGHOUSEKEEPING: "
+
+// Debug Housekeeping monitor
+
+class CSashaDebugHousekeepingServer : public ISashaServer, public Thread
+{
+    bool stopped;
+    Semaphore stopsem;
+    Mutex runmutex;
+    Linked<IPropertyTree> props;
+
+public:
+    IMPLEMENT_IINTERFACE_USING(Thread);
+
+    CSashaDebugHousekeepingServer(IPropertyTree *_config)
+        : Thread("CSashaDebugHousekeepingServer"), props(_config)
+    {
+        stopped = false;
+
+        StringBuffer userName;
+        props->getProp("@user", userName);
+    }
+
+    ~CSashaDebugHousekeepingServer()
+    {
+    }
+
+    void start()
+    {
+        Thread::start(false);
+    }
+
+    void ready()
+    {
+    }
+
+    void stop()
+    {
+        if (!stopped)
+        {
+            stopped = true;
+            stopsem.signal();
+        }
+        synchronized block(runmutex); // hopefully stopped should stop
+        if (!join(1000 * 60 * 3))
+            OERRLOG("CSashaDebugHousekeepingServer aborted");
+    }
+
+    void runDebugHousekeeping()
+    {
+        synchronized block(runmutex);
+        if (stopped)
+            return;
+        PROGLOG(LOGDBGHK "Started");
+        unsigned defaultExpireDays = props->getPropInt("@expiryDefault", DEFAULT_EXPIRYDAYS);
+
+        // get debug plane dir
+        StringBuffer debugDir;
+        StringBuffer planeName;
+        if (!getDefaultPlane(planeName, "@debugPlane", "debug"))
+        {
+            WARNLOG("Exception handlers configured, but debug plane is missing");
+            return;
+        }
+        Owned<IPropertyTree> plane = getStoragePlane(planeName);
+        assertex(plane);
+        verifyex(plane->getProp("@prefix", debugDir));
+
+        // iterate debug plane selecting post-mortem directories for housekeeping
+        Owned<IDirectoryIterator> pDirIter = createDirectoryIterator(debugDir.str(), "*", false, true);
+        ForEach(*pDirIter)
+        {
+            if (stopped)
+                break;
+
+            IFile &iDirFile = pDirIter->query();
+            const char *dirPath = iDirFile.queryFilename();
+
+            if (!dirPath || !*dirPath)
+                continue;
+
+            // Process directories, exclude the ".", "..", non post-mortem and not expired post-mortem directories
+            if (iDirFile.isDirectory() == fileBool::foundYes && *dirPath != '.')
+            {
+                // Ensure directory name only
+                StringBuffer dirNameOnly;
+                String dirNameStr(dirPath);
+                int fwdSlashIndex = dirNameStr.lastIndexOf('/');
+                if (fwdSlashIndex > -1)
+                {
+                    String *tmpStr = dirNameStr.substring(fwdSlashIndex + 1);
+                    dirNameOnly.append(*tmpStr);
+                    delete tmpStr;
+                }
+                else
+                {
+                    dirNameOnly.append(dirNameStr);
+                }
+
+                if (isPostMortemDirPath(dirNameOnly) && isExpiredDirPath(dirNameOnly, defaultExpireDays))
+                {
+                    recursiveRemoveDirectory(dirPath);
+                    PROGLOG(LOGDBGHK "Deleted %s", dirPath);
+                }
+            }
+        }
+        pDirIter.clear();
+
+        PROGLOG(LOGDBGHK "%s", stopped ? "Stopped" : "Done");
+    }
+
+    int run()
+    {
+        unsigned interval = props->getPropInt("@interval", DEFAULT_EXPIRY_INTERVAL);
+        if (!interval)
+            stopped = true;
path: dali/sasha/sadbghk.cpp
user: login: jakesmith
body: I notice this is same in file expiry service, but if !interval, stopped= true, then should exit at this point (return).
As it stands, if interval = 0, it will fall through and to initinterval = (unsigned)-1 / 2 ..
created_at: 2025-03-12T09:58:12Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,224 @@
+#include "platform.h"
+
+#include "jlib.hpp"
+#include "jiface.hpp"
+#include "jptree.hpp"
+#include "jregexp.hpp"
+
+#include "dadfs.hpp"
+#include "dalienv.hpp"
+
+#include "saserver.hpp"
+#include "sautil.hpp"
+
+#define DEFAULT_EXPIRY_INTERVAL 24 // hours
+
+#define DEFAULT_EXPIRYDAYS 7
+
+#define LOGDBGHK "DEBUGHOUSEKEEPING: "
+
+// Debug Housekeeping monitor
+
+class CSashaDebugHousekeepingServer : public ISashaServer, public Thread
+{
+    bool stopped;
+    Semaphore stopsem;
+    Mutex runmutex;
+    Linked<IPropertyTree> props;
+
+public:
+    IMPLEMENT_IINTERFACE_USING(Thread);
+
+    CSashaDebugHousekeepingServer(IPropertyTree *_config)
+        : Thread("CSashaDebugHousekeepingServer"), props(_config)
+    {
+        stopped = false;
+
+        StringBuffer userName;
+        props->getProp("@user", userName);
+    }
+
+    ~CSashaDebugHousekeepingServer()
+    {
+    }
+
+    void start()
+    {
+        Thread::start(false);
+    }
+
+    void ready()
+    {
+    }
+
+    void stop()
+    {
+        if (!stopped)
+        {
+            stopped = true;
+            stopsem.signal();
+        }
+        synchronized block(runmutex); // hopefully stopped should stop
+        if (!join(1000 * 60 * 3))
+            OERRLOG("CSashaDebugHousekeepingServer aborted");
+    }
+
+    void runDebugHousekeeping()
+    {
+        synchronized block(runmutex);
+        if (stopped)
+            return;
+        PROGLOG(LOGDBGHK "Started");
+        unsigned defaultExpireDays = props->getPropInt("@expiryDefault", DEFAULT_EXPIRYDAYS);
+
+        // get debug plane dir
+        StringBuffer debugDir;
+        StringBuffer planeName;
+        if (!getDefaultPlane(planeName, "@debugPlane", "debug"))
+        {
+            WARNLOG("Exception handlers configured, but debug plane is missing");
+            return;
+        }
+        Owned<IPropertyTree> plane = getStoragePlane(planeName);
+        assertex(plane);
+        verifyex(plane->getProp("@prefix", debugDir));
+
+        // iterate debug plane selecting post-mortem directories for housekeeping
+        Owned<IDirectoryIterator> pDirIter = createDirectoryIterator(debugDir.str(), "*", false, true);
+        ForEach(*pDirIter)
+        {
+            if (stopped)
+                break;
+
+            IFile &iDirFile = pDirIter->query();
+            const char *dirPath = iDirFile.queryFilename();
+
+            if (!dirPath || !*dirPath)
+                continue;
+
+            // Process directories, exclude the ".", "..", non post-mortem and not expired post-mortem directories
+            if (iDirFile.isDirectory() == fileBool::foundYes && *dirPath != '.')
+            {
+                // Ensure directory name only
+                StringBuffer dirNameOnly;
+                String dirNameStr(dirPath);
+                int fwdSlashIndex = dirNameStr.lastIndexOf('/');
+                if (fwdSlashIndex > -1)
+                {
+                    String *tmpStr = dirNameStr.substring(fwdSlashIndex + 1);
+                    dirNameOnly.append(*tmpStr);
+                    delete tmpStr;
+                }
+                else
+                {
+                    dirNameOnly.append(dirNameStr);
+                }
+
+                if (isPostMortemDirPath(dirNameOnly) && isExpiredDirPath(dirNameOnly, defaultExpireDays))
+                {
+                    recursiveRemoveDirectory(dirPath);
+                    PROGLOG(LOGDBGHK "Deleted %s", dirPath);
+                }
+            }
+        }
+        pDirIter.clear();
+
+        PROGLOG(LOGDBGHK "%s", stopped ? "Stopped" : "Done");
+    }
+
+    int run()
+    {
+        unsigned interval = props->getPropInt("@interval", DEFAULT_EXPIRY_INTERVAL);
+        if (!interval)
+            stopped = true;
+        PROGLOG(LOGDBGHK "min interval = %d hr", interval);
+        unsigned initinterval = (interval - 1) / 2; // wait a bit til dali has started
path: dali/sasha/sadbghk.cpp
user: login: jakesmith
body: there's no value in the initinterval for this service..
It's there to give Dali a chance to startup in the case of the [logical] file expiry service, but in this case it doesn't serve any purpose, so better to delete this/related code.
created_at: 2025-03-12T09:59:13Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,224 @@
+#include "platform.h"
+
+#include "jlib.hpp"
+#include "jiface.hpp"
+#include "jptree.hpp"
+#include "jregexp.hpp"
+
+#include "dadfs.hpp"
+#include "dalienv.hpp"
+
+#include "saserver.hpp"
+#include "sautil.hpp"
+
+#define DEFAULT_EXPIRY_INTERVAL 24 // hours
+
+#define DEFAULT_EXPIRYDAYS 7
+
+#define LOGDBGHK "DEBUGHOUSEKEEPING: "
+
+// Debug Housekeeping monitor
+
+class CSashaDebugHousekeepingServer : public ISashaServer, public Thread
+{
+    bool stopped;
+    Semaphore stopsem;
+    Mutex runmutex;
+    Linked<IPropertyTree> props;
+
+public:
+    IMPLEMENT_IINTERFACE_USING(Thread);
+
+    CSashaDebugHousekeepingServer(IPropertyTree *_config)
+        : Thread("CSashaDebugHousekeepingServer"), props(_config)
+    {
+        stopped = false;
+
+        StringBuffer userName;
+        props->getProp("@user", userName);
+    }
+
+    ~CSashaDebugHousekeepingServer()
+    {
+    }
+
+    void start()
+    {
+        Thread::start(false);
+    }
+
+    void ready()
+    {
+    }
+
+    void stop()
+    {
+        if (!stopped)
+        {
+            stopped = true;
+            stopsem.signal();
+        }
+        synchronized block(runmutex); // hopefully stopped should stop
+        if (!join(1000 * 60 * 3))
+            OERRLOG("CSashaDebugHousekeepingServer aborted");
+    }
+
+    void runDebugHousekeeping()
+    {
+        synchronized block(runmutex);
+        if (stopped)
+            return;
+        PROGLOG(LOGDBGHK "Started");
+        unsigned defaultExpireDays = props->getPropInt("@expiryDefault", DEFAULT_EXPIRYDAYS);
+
+        // get debug plane dir
+        StringBuffer debugDir;
+        StringBuffer planeName;
+        if (!getDefaultPlane(planeName, "@debugPlane", "debug"))
+        {
+            WARNLOG("Exception handlers configured, but debug plane is missing");
+            return;
+        }
+        Owned<IPropertyTree> plane = getStoragePlane(planeName);
+        assertex(plane);
+        verifyex(plane->getProp("@prefix", debugDir));
+
+        // iterate debug plane selecting post-mortem directories for housekeeping
+        Owned<IDirectoryIterator> pDirIter = createDirectoryIterator(debugDir.str(), "*", false, true);
+        ForEach(*pDirIter)
+        {
+            if (stopped)
+                break;
+
+            IFile &iDirFile = pDirIter->query();
+            const char *dirPath = iDirFile.queryFilename();
+
+            if (!dirPath || !*dirPath)
+                continue;
+
+            // Process directories, exclude the ".", "..", non post-mortem and not expired post-mortem directories
+            if (iDirFile.isDirectory() == fileBool::foundYes && *dirPath != '.')
+            {
+                // Ensure directory name only
+                StringBuffer dirNameOnly;
+                String dirNameStr(dirPath);
+                int fwdSlashIndex = dirNameStr.lastIndexOf('/');
+                if (fwdSlashIndex > -1)
+                {
+                    String *tmpStr = dirNameStr.substring(fwdSlashIndex + 1);
+                    dirNameOnly.append(*tmpStr);
+                    delete tmpStr;
+                }
+                else
+                {
+                    dirNameOnly.append(dirNameStr);
+                }
+
+                if (isPostMortemDirPath(dirNameOnly) && isExpiredDirPath(dirNameOnly, defaultExpireDays))
+                {
+                    recursiveRemoveDirectory(dirPath);
+                    PROGLOG(LOGDBGHK "Deleted %s", dirPath);
+                }
+            }
+        }
+        pDirIter.clear();
+
+        PROGLOG(LOGDBGHK "%s", stopped ? "Stopped" : "Done");
+    }
+
+    int run()
+    {
+        unsigned interval = props->getPropInt("@interval", DEFAULT_EXPIRY_INTERVAL);
+        if (!interval)
+            stopped = true;
+        PROGLOG(LOGDBGHK "min interval = %d hr", interval);
+        unsigned initinterval = (interval - 1) / 2; // wait a bit til dali has started
+        CSashaSchedule schedule;
+        if (interval)
+            schedule.init(props, interval, initinterval);
+        initinterval *= 60 * 60 * 1000; // ms
+        unsigned started = msTick();
+        while (!stopped)
+        {
+            stopsem.wait(1000 * 60);
+            if (stopped)
+                break;
+            if (!interval || ((started != (unsigned)-1) && (msTick() - started < initinterval)))
+                continue;
+            started = (unsigned)-1;
+            if (!schedule.ready())
+                continue;
+            try
+            {
+                runDebugHousekeeping();
+            }
+            catch (IException *e)
+            {
+                StringBuffer s;
+                EXCLOG(e, LOGDBGHK);
+                e->Release();
+            }
+        }
+        PROGLOG(LOGDBGHK "Exit");
+        return 0;
+    }
+
+private:
+    bool isPostMortemDirPath(const StringBuffer &dirName)
path: dali/sasha/sadbghk.cpp
user: login: jakesmith
body: see previous comments - isPostMortemDirPath function not needed.
created_at: 2025-03-12T09:59:49Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,228 @@
+#include "platform.h"
+
+#include "jlib.hpp"
+#include "jiface.hpp"
+#include "jptree.hpp"
+#include "jregexp.hpp"
+
+#include "dadfs.hpp"
+#include "dalienv.hpp"
+
+#include "saserver.hpp"
+#include "sautil.hpp"
+
+#define DEFAULT_EXPIRY_INTERVAL 24 // hours
+
+#define DEFAULT_EXPIRYDAYS 7
+
+#define LOGDBGHK "DEBUGHOUSEKEEPING: "
+
+// Debug Housekeeping monitor
+
+class CSashaDebugHousekeepingServer : public ISashaServer, public Thread
+{
+    bool stopped;
+    Semaphore stopsem;
+    Mutex runmutex;
+    Owned<IUserDescriptor> udesc;
+    Linked<IPropertyTree> props;
+
+public:
+    IMPLEMENT_IINTERFACE_USING(Thread);
+
+    CSashaDebugHousekeepingServer(IPropertyTree *_config)
+        : Thread("CSashaDebugHousekeepingServer"), props(_config)
+    {
+        stopped = false;
+
+        StringBuffer userName;
+        props->getProp("@user", userName);
+        udesc.setown(createUserDescriptor());
+        udesc->set(userName.str(), nullptr);
+    }
+
+    ~CSashaDebugHousekeepingServer()
+    {
+    }
+
+    void start()
+    {
+        Thread::start(false);
+    }
+
+    void ready()
+    {
+    }
+
+    void stop()
+    {
+        if (!stopped)
+        {
+            stopped = true;
+            stopsem.signal();
+        }
+        synchronized block(runmutex); // hopefully stopped should stop
+        if (!join(1000 * 60 * 3))
+            OERRLOG("CSashaDebugHousekeepingServer aborted");
+    }
+
+    void runDebugHousekeeping()
+    {
+        synchronized block(runmutex);
+        if (stopped)
+            return;
+        PROGLOG(LOGDBGHK "Started");
+        unsigned defaultExpireDays = props->getPropInt("@expiryDefault", DEFAULT_EXPIRYDAYS);
+
+        // get debug plane dir
+        StringBuffer debugDir;
+        StringBuffer planeName;
+        if (!getDefaultPlane(planeName, "@debugPlane", "debug"))
+        {
+            WARNLOG("Exception handlers configured, but debug plane is missing");
+            return;
+        }
+        Owned<IPropertyTree> plane = getStoragePlane(planeName);
+        assertex(plane);
+        verifyex(plane->getProp("@prefix", debugDir));
+
+        // iterate debug plane selecting post-mortem directories for housekeeping
+        Owned<IDirectoryIterator> pDirIter = createDirectoryIterator(debugDir.str(), "*", false, true);
+        addPathSepChar(debugDir);
+        ForEach(*pDirIter)
+        {
+            if (stopped)
+                break;
+
+            IFile &iDirFile = pDirIter->query();
+            const char *dirPath = iDirFile.queryFilename();
+
+            if (!dirPath || !*dirPath)
+                continue;
+
+            // Process directories, exclude the ".", "..", non post-mortem and not expired post-mortem directories
+            if (iDirFile.isDirectory() == fileBool::foundYes && *dirPath != '.')
+            {
+                // Ensure directory name only
+                StringBuffer dirNameOnly;
+                String dirNameStr(dirPath);
+                int fwdSlashIndex = dirNameStr.lastIndexOf('/');
+                if (fwdSlashIndex > -1)
+                {
+                    String *tmpStr = dirNameStr.substring(fwdSlashIndex + 1);
+                    dirNameOnly.append(*tmpStr);
+                    delete tmpStr;
+                }
+                else
+                {
+                    dirNameOnly.append(dirNameStr);
+                }
+
+                if (isPostMortemDirPath(dirNameOnly) && isExpiredDirPath(dirNameOnly, defaultExpireDays))
+                {
+                    recursiveRemoveDirectory(dirPath);
+                    PROGLOG(LOGDBGHK "Deleted %s", dirPath);
+                }
+            }
+        }
+        pDirIter.clear();
+
+        PROGLOG(LOGDBGHK "%s", stopped ? "Stopped" : "Done");
+    }
+
+    int run()
+    {
+        unsigned interval = props->getPropInt("@interval", DEFAULT_EXPIRY_INTERVAL);
+        if (!interval)
+            stopped = true;
+        PROGLOG(LOGDBGHK "min interval = %d hr", interval);
+        unsigned initinterval = (interval - 1) / 2; // wait a bit til dali has started
+        CSashaSchedule schedule;
+        if (interval)
+            schedule.init(props, interval, initinterval);
+        initinterval *= 60 * 60 * 1000; // ms
+        unsigned started = msTick();
+        while (!stopped)
+        {
+            stopsem.wait(1000 * 60);
+            if (stopped)
+                break;
+            if (!interval || ((started != (unsigned)-1) && (msTick() - started < initinterval)))
+                continue;
+            started = (unsigned)-1;
+            if (!schedule.ready())
+                continue;
+            try
+            {
+                runDebugHousekeeping();
+            }
+            catch (IException *e)
+            {
+                StringBuffer s;
+                EXCLOG(e, LOGDBGHK);
+                e->Release();
+            }
+        }
+        PROGLOG(LOGDBGHK "Exit");
+        return 0;
+    }
+
+private:
+    bool isPostMortemDirPath(const StringBuffer &dirName)
+    {
+        // Expecting a directory name like "W20250225-101112"
+        RegExpr postMortemDirRegEx("^W[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]-[0-9][0-9][0-9][0-9][0-9][0-9]$");
+        if (postMortemDirRegEx.find(dirName.str()))
+        {
+            PROGLOG(LOGDBGHK "Post-mortem dir: %s", dirName.str());
path: dali/sasha/sadbghk.cpp
user: login: jakesmith
body: we need to be very careful not to be too verbose - logging is expensive.
DBGLOG would be 'ok' in theory, but in fact there our default logging level in various places is still too high and would include DBGLOG's (DBGLOG confusingly is emitted in release too).

You do truly want it to be a DEBUG only log, during development only. For now, we should either wrap them in #ifdef _DEBUG if you really want to leave them in code, or remove them entirely when done.
created_at: 2025-03-12T10:02:47Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,228 @@
+#include "platform.h"
+
+#include "jlib.hpp"
+#include "jiface.hpp"
+#include "jptree.hpp"
+#include "jregexp.hpp"
+
+#include "dadfs.hpp"
+#include "dalienv.hpp"
+
+#include "saserver.hpp"
+#include "sautil.hpp"
+
+#define DEFAULT_EXPIRY_INTERVAL 24 // hours
+
+#define DEFAULT_EXPIRYDAYS 7
+
+#define LOGDBGHK "DEBUGHOUSEKEEPING: "
+
+// Debug Housekeeping monitor
+
+class CSashaDebugHousekeepingServer : public ISashaServer, public Thread
+{
+    bool stopped;
+    Semaphore stopsem;
+    Mutex runmutex;
+    Owned<IUserDescriptor> udesc;
+    Linked<IPropertyTree> props;
+
+public:
+    IMPLEMENT_IINTERFACE_USING(Thread);
+
+    CSashaDebugHousekeepingServer(IPropertyTree *_config)
+        : Thread("CSashaDebugHousekeepingServer"), props(_config)
+    {
+        stopped = false;
+
+        StringBuffer userName;
+        props->getProp("@user", userName);
+        udesc.setown(createUserDescriptor());
+        udesc->set(userName.str(), nullptr);
+    }
+
+    ~CSashaDebugHousekeepingServer()
+    {
+    }
+
+    void start()
+    {
+        Thread::start(false);
+    }
+
+    void ready()
+    {
+    }
+
+    void stop()
+    {
+        if (!stopped)
+        {
+            stopped = true;
+            stopsem.signal();
+        }
+        synchronized block(runmutex); // hopefully stopped should stop
+        if (!join(1000 * 60 * 3))
+            OERRLOG("CSashaDebugHousekeepingServer aborted");
+    }
+
+    void runDebugHousekeeping()
+    {
+        synchronized block(runmutex);
+        if (stopped)
+            return;
+        PROGLOG(LOGDBGHK "Started");
+        unsigned defaultExpireDays = props->getPropInt("@expiryDefault", DEFAULT_EXPIRYDAYS);
+
+        // get debug plane dir
+        StringBuffer debugDir;
+        StringBuffer planeName;
+        if (!getDefaultPlane(planeName, "@debugPlane", "debug"))
+        {
+            WARNLOG("Exception handlers configured, but debug plane is missing");
+            return;
+        }
+        Owned<IPropertyTree> plane = getStoragePlane(planeName);
+        assertex(plane);
+        verifyex(plane->getProp("@prefix", debugDir));
+
+        // iterate debug plane selecting post-mortem directories for housekeeping
+        Owned<IDirectoryIterator> pDirIter = createDirectoryIterator(debugDir.str(), "*", false, true);
+        addPathSepChar(debugDir);
+        ForEach(*pDirIter)
+        {
+            if (stopped)
+                break;
+
+            IFile &iDirFile = pDirIter->query();
+            const char *dirPath = iDirFile.queryFilename();
+
+            if (!dirPath || !*dirPath)
+                continue;
+
+            // Process directories, exclude the ".", "..", non post-mortem and not expired post-mortem directories
+            if (iDirFile.isDirectory() == fileBool::foundYes && *dirPath != '.')
+            {
+                // Ensure directory name only
+                StringBuffer dirNameOnly;
+                String dirNameStr(dirPath);
+                int fwdSlashIndex = dirNameStr.lastIndexOf('/');
+                if (fwdSlashIndex > -1)
+                {
+                    String *tmpStr = dirNameStr.substring(fwdSlashIndex + 1);
+                    dirNameOnly.append(*tmpStr);
+                    delete tmpStr;
+                }
+                else
+                {
+                    dirNameOnly.append(dirNameStr);
+                }
+
+                if (isPostMortemDirPath(dirNameOnly) && isExpiredDirPath(dirNameOnly, defaultExpireDays))
+                {
+                    recursiveRemoveDirectory(dirPath);
+                    PROGLOG(LOGDBGHK "Deleted %s", dirPath);
+                }
+            }
+        }
+        pDirIter.clear();
+
+        PROGLOG(LOGDBGHK "%s", stopped ? "Stopped" : "Done");
+    }
+
+    int run()
+    {
+        unsigned interval = props->getPropInt("@interval", DEFAULT_EXPIRY_INTERVAL);
+        if (!interval)
+            stopped = true;
+        PROGLOG(LOGDBGHK "min interval = %d hr", interval);
+        unsigned initinterval = (interval - 1) / 2; // wait a bit til dali has started
+        CSashaSchedule schedule;
+        if (interval)
+            schedule.init(props, interval, initinterval);
+        initinterval *= 60 * 60 * 1000; // ms
+        unsigned started = msTick();
+        while (!stopped)
+        {
+            stopsem.wait(1000 * 60);
+            if (stopped)
+                break;
+            if (!interval || ((started != (unsigned)-1) && (msTick() - started < initinterval)))
+                continue;
+            started = (unsigned)-1;
+            if (!schedule.ready())
+                continue;
+            try
+            {
+                runDebugHousekeeping();
+            }
+            catch (IException *e)
+            {
+                StringBuffer s;
+                EXCLOG(e, LOGDBGHK);
+                e->Release();
+            }
+        }
+        PROGLOG(LOGDBGHK "Exit");
+        return 0;
+    }
+
+private:
+    bool isPostMortemDirPath(const StringBuffer &dirName)
+    {
+        // Expecting a directory name like "W20250225-101112"
+        RegExpr postMortemDirRegEx("^W[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]-[0-9][0-9][0-9][0-9][0-9][0-9]$");
+        if (postMortemDirRegEx.find(dirName.str()))
+        {
+            PROGLOG(LOGDBGHK "Post-mortem dir: %s", dirName.str());
+
+            return true;
+        }
+        else
+        {
+            PROGLOG(LOGDBGHK "Non post-mortem dir: %s", dirName.str());
path: dali/sasha/sadbghk.cpp
user: login: jakesmith
body: well isPostMortemDirPath is needed so moot point.
created_at: 2025-03-12T10:03:25Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,224 @@
+#include "platform.h"
+
+#include "jlib.hpp"
+#include "jiface.hpp"
+#include "jptree.hpp"
+#include "jregexp.hpp"
+
+#include "dadfs.hpp"
+#include "dalienv.hpp"
+
+#include "saserver.hpp"
+#include "sautil.hpp"
+
+#define DEFAULT_EXPIRY_INTERVAL 24 // hours
+
+#define DEFAULT_EXPIRYDAYS 7
+
+#define LOGDBGHK "DEBUGHOUSEKEEPING: "
+
+// Debug Housekeeping monitor
+
+class CSashaDebugHousekeepingServer : public ISashaServer, public Thread
+{
+    bool stopped;
+    Semaphore stopsem;
+    Mutex runmutex;
+    Linked<IPropertyTree> props;
+
+public:
+    IMPLEMENT_IINTERFACE_USING(Thread);
+
+    CSashaDebugHousekeepingServer(IPropertyTree *_config)
+        : Thread("CSashaDebugHousekeepingServer"), props(_config)
+    {
+        stopped = false;
+
+        StringBuffer userName;
+        props->getProp("@user", userName);
+    }
+
+    ~CSashaDebugHousekeepingServer()
+    {
+    }
+
+    void start()
+    {
+        Thread::start(false);
+    }
+
+    void ready()
+    {
+    }
+
+    void stop()
+    {
+        if (!stopped)
+        {
+            stopped = true;
+            stopsem.signal();
+        }
+        synchronized block(runmutex); // hopefully stopped should stop
+        if (!join(1000 * 60 * 3))
+            OERRLOG("CSashaDebugHousekeepingServer aborted");
+    }
+
+    void runDebugHousekeeping()
+    {
+        synchronized block(runmutex);
+        if (stopped)
+            return;
+        PROGLOG(LOGDBGHK "Started");
+        unsigned defaultExpireDays = props->getPropInt("@expiryDefault", DEFAULT_EXPIRYDAYS);
+
+        // get debug plane dir
+        StringBuffer debugDir;
+        StringBuffer planeName;
+        if (!getDefaultPlane(planeName, "@debugPlane", "debug"))
+        {
+            WARNLOG("Exception handlers configured, but debug plane is missing");
+            return;
+        }
+        Owned<IPropertyTree> plane = getStoragePlane(planeName);
+        assertex(plane);
+        verifyex(plane->getProp("@prefix", debugDir));
+
+        // iterate debug plane selecting post-mortem directories for housekeeping
+        Owned<IDirectoryIterator> pDirIter = createDirectoryIterator(debugDir.str(), "*", false, true);
+        ForEach(*pDirIter)
+        {
+            if (stopped)
+                break;
+
+            IFile &iDirFile = pDirIter->query();
+            const char *dirPath = iDirFile.queryFilename();
+
+            if (!dirPath || !*dirPath)
+                continue;
+
+            // Process directories, exclude the ".", "..", non post-mortem and not expired post-mortem directories
+            if (iDirFile.isDirectory() == fileBool::foundYes && *dirPath != '.')
+            {
+                // Ensure directory name only
+                StringBuffer dirNameOnly;
+                String dirNameStr(dirPath);
+                int fwdSlashIndex = dirNameStr.lastIndexOf('/');
+                if (fwdSlashIndex > -1)
+                {
+                    String *tmpStr = dirNameStr.substring(fwdSlashIndex + 1);
+                    dirNameOnly.append(*tmpStr);
+                    delete tmpStr;
+                }
+                else
+                {
+                    dirNameOnly.append(dirNameStr);
+                }
+
+                if (isPostMortemDirPath(dirNameOnly) && isExpiredDirPath(dirNameOnly, defaultExpireDays))
+                {
+                    recursiveRemoveDirectory(dirPath);
+                    PROGLOG(LOGDBGHK "Deleted %s", dirPath);
+                }
+            }
+        }
+        pDirIter.clear();
+
+        PROGLOG(LOGDBGHK "%s", stopped ? "Stopped" : "Done");
+    }
+
+    int run()
+    {
+        unsigned interval = props->getPropInt("@interval", DEFAULT_EXPIRY_INTERVAL);
+        if (!interval)
+            stopped = true;
+        PROGLOG(LOGDBGHK "min interval = %d hr", interval);
+        unsigned initinterval = (interval - 1) / 2; // wait a bit til dali has started
+        CSashaSchedule schedule;
+        if (interval)
+            schedule.init(props, interval, initinterval);
+        initinterval *= 60 * 60 * 1000; // ms
+        unsigned started = msTick();
+        while (!stopped)
+        {
+            stopsem.wait(1000 * 60);
+            if (stopped)
+                break;
+            if (!interval || ((started != (unsigned)-1) && (msTick() - started < initinterval)))
+                continue;
+            started = (unsigned)-1;
+            if (!schedule.ready())
+                continue;
+            try
+            {
+                runDebugHousekeeping();
+            }
+            catch (IException *e)
+            {
+                StringBuffer s;
+                EXCLOG(e, LOGDBGHK);
+                e->Release();
+            }
+        }
+        PROGLOG(LOGDBGHK "Exit");
+        return 0;
+    }
+
+private:
+    bool isPostMortemDirPath(const StringBuffer &dirName)
+    {
+        // Expecting a directory name like "W20250225-101112"
+        RegExpr postMortemDirRegEx("^W[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]-[0-9][0-9][0-9][0-9][0-9][0-9]$");
+        if (postMortemDirRegEx.find(dirName.str()))
+        {
+            PROGLOG(LOGDBGHK "Post-mortem dir: %s", dirName.str());
+
+            return true;
+        }
+        else
+        {
+            PROGLOG(LOGDBGHK "Non post-mortem dir: %s", dirName.str());
+
+            return false;
+        }
+    }
+
+    bool isExpiredDirPath(StringBuffer &dirName, const unsigned &defaultExpireDays)
path: dali/sasha/sadbghk.cpp
user: login: jakesmith
body: see other comments, should use iFile and iFile.getTime to look at modified-time.
created_at: 2025-03-12T10:03:49Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,228 @@
+#include "platform.h"
+
+#include "jlib.hpp"
+#include "jiface.hpp"
+#include "jptree.hpp"
+#include "jregexp.hpp"
+
+#include "dadfs.hpp"
+#include "dalienv.hpp"
+
+#include "saserver.hpp"
+#include "sautil.hpp"
+
+#define DEFAULT_EXPIRY_INTERVAL 24 // hours
+
+#define DEFAULT_EXPIRYDAYS 7
+
+#define LOGDBGHK "DEBUGHOUSEKEEPING: "
+
+// Debug Housekeeping monitor
+
+class CSashaDebugHousekeepingServer : public ISashaServer, public Thread
+{
+    bool stopped;
+    Semaphore stopsem;
+    Mutex runmutex;
+    Owned<IUserDescriptor> udesc;
+    Linked<IPropertyTree> props;
+
+public:
+    IMPLEMENT_IINTERFACE_USING(Thread);
+
+    CSashaDebugHousekeepingServer(IPropertyTree *_config)
+        : Thread("CSashaDebugHousekeepingServer"), props(_config)
+    {
+        stopped = false;
+
+        StringBuffer userName;
+        props->getProp("@user", userName);
+        udesc.setown(createUserDescriptor());
+        udesc->set(userName.str(), nullptr);
+    }
+
+    ~CSashaDebugHousekeepingServer()
+    {
+    }
+
+    void start()
+    {
+        Thread::start(false);
+    }
+
+    void ready()
+    {
+    }
+
+    void stop()
+    {
+        if (!stopped)
+        {
+            stopped = true;
+            stopsem.signal();
+        }
+        synchronized block(runmutex); // hopefully stopped should stop
+        if (!join(1000 * 60 * 3))
+            OERRLOG("CSashaDebugHousekeepingServer aborted");
+    }
+
+    void runDebugHousekeeping()
+    {
+        synchronized block(runmutex);
+        if (stopped)
+            return;
+        PROGLOG(LOGDBGHK "Started");
+        unsigned defaultExpireDays = props->getPropInt("@expiryDefault", DEFAULT_EXPIRYDAYS);
+
+        // get debug plane dir
+        StringBuffer debugDir;
+        StringBuffer planeName;
+        if (!getDefaultPlane(planeName, "@debugPlane", "debug"))
+        {
+            WARNLOG("Exception handlers configured, but debug plane is missing");
+            return;
+        }
+        Owned<IPropertyTree> plane = getStoragePlane(planeName);
+        assertex(plane);
+        verifyex(plane->getProp("@prefix", debugDir));
+
+        // iterate debug plane selecting post-mortem directories for housekeeping
+        Owned<IDirectoryIterator> pDirIter = createDirectoryIterator(debugDir.str(), "*", false, true);
+        addPathSepChar(debugDir);
+        ForEach(*pDirIter)
+        {
+            if (stopped)
+                break;
+
+            IFile &iDirFile = pDirIter->query();
+            const char *dirPath = iDirFile.queryFilename();
+
+            if (!dirPath || !*dirPath)
+                continue;
+
+            // Process directories, exclude the ".", "..", non post-mortem and not expired post-mortem directories
+            if (iDirFile.isDirectory() == fileBool::foundYes && *dirPath != '.')
+            {
+                // Ensure directory name only
+                StringBuffer dirNameOnly;
+                String dirNameStr(dirPath);
+                int fwdSlashIndex = dirNameStr.lastIndexOf('/');
+                if (fwdSlashIndex > -1)
+                {
+                    String *tmpStr = dirNameStr.substring(fwdSlashIndex + 1);
+                    dirNameOnly.append(*tmpStr);
+                    delete tmpStr;
+                }
+                else
+                {
+                    dirNameOnly.append(dirNameStr);
+                }
+
+                if (isPostMortemDirPath(dirNameOnly) && isExpiredDirPath(dirNameOnly, defaultExpireDays))
+                {
+                    recursiveRemoveDirectory(dirPath);
+                    PROGLOG(LOGDBGHK "Deleted %s", dirPath);
+                }
+            }
+        }
+        pDirIter.clear();
+
+        PROGLOG(LOGDBGHK "%s", stopped ? "Stopped" : "Done");
+    }
+
+    int run()
+    {
+        unsigned interval = props->getPropInt("@interval", DEFAULT_EXPIRY_INTERVAL);
+        if (!interval)
+            stopped = true;
+        PROGLOG(LOGDBGHK "min interval = %d hr", interval);
+        unsigned initinterval = (interval - 1) / 2; // wait a bit til dali has started
+        CSashaSchedule schedule;
+        if (interval)
+            schedule.init(props, interval, initinterval);
+        initinterval *= 60 * 60 * 1000; // ms
+        unsigned started = msTick();
+        while (!stopped)
+        {
+            stopsem.wait(1000 * 60);
+            if (stopped)
+                break;
+            if (!interval || ((started != (unsigned)-1) && (msTick() - started < initinterval)))
+                continue;
+            started = (unsigned)-1;
+            if (!schedule.ready())
+                continue;
+            try
+            {
+                runDebugHousekeeping();
+            }
+            catch (IException *e)
+            {
+                StringBuffer s;
+                EXCLOG(e, LOGDBGHK);
+                e->Release();
+            }
+        }
+        PROGLOG(LOGDBGHK "Exit");
+        return 0;
+    }
+
+private:
+    bool isPostMortemDirPath(const StringBuffer &dirName)
+    {
+        // Expecting a directory name like "W20250225-101112"
+        RegExpr postMortemDirRegEx("^W[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]-[0-9][0-9][0-9][0-9][0-9][0-9]$");
+        if (postMortemDirRegEx.find(dirName.str()))
+        {
+            PROGLOG(LOGDBGHK "Post-mortem dir: %s", dirName.str());
+
+            return true;
+        }
+        else
+        {
+            PROGLOG(LOGDBGHK "Non post-mortem dir: %s", dirName.str());
+
+            return false;
+        }
+    }
+
+    bool isExpiredDirPath(StringBuffer &dirName, const unsigned &defaultExpireDays)
+    {
+        // Directory name is like "W20250225-101112"
+        StringBuffer cDateTimeoprmattedString;
+        cDateTimeoprmattedString.appendf("%c%c%c%c-%c%c-%c%cT%c%c:%c%c:%c%c",
+                                         dirName.charAt(1), dirName.charAt(2), dirName.charAt(3), dirName.charAt(4),
+                                         dirName.charAt(5), dirName.charAt(6),
+                                         dirName.charAt(7), dirName.charAt(8),
+                                         dirName.charAt(10), dirName.charAt(11),
+                                         dirName.charAt(12), dirName.charAt(13),
+                                         dirName.charAt(14), dirName.charAt(15));
+
+        CDateTime now;
+        now.setNow();
+
+        CDateTime expires;
+        expires.setString(cDateTimeoprmattedString.str());
+        expires.adjustTime(60 * 24 * defaultExpireDays);
+
+        if (now.compare(expires, false) > 0)
+        {
+            PROGLOG(LOGDBGHK "Post-mortem dir: %s has expired", dirName.str());
path: dali/sasha/sadbghk.cpp
user: login: jakesmith
body: right, ensure logging by default is at a minimum, logging when deleted is more than enough. So this func can just return true/false only.
created_at: 2025-03-12T10:04:35Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,224 @@
+#include "platform.h"
+
+#include "jlib.hpp"
+#include "jiface.hpp"
+#include "jptree.hpp"
+#include "jregexp.hpp"
+
+#include "dadfs.hpp"
+#include "dalienv.hpp"
+
+#include "saserver.hpp"
+#include "sautil.hpp"
+
+#define DEFAULT_EXPIRY_INTERVAL 24 // hours
+
+#define DEFAULT_EXPIRYDAYS 7
+
+#define LOGDBGHK "DEBUGHOUSEKEEPING: "
+
+// Debug Housekeeping monitor
+
+class CSashaDebugHousekeepingServer : public ISashaServer, public Thread
+{
+    bool stopped;
+    Semaphore stopsem;
+    Mutex runmutex;
+    Linked<IPropertyTree> props;
+
+public:
+    IMPLEMENT_IINTERFACE_USING(Thread);
+
+    CSashaDebugHousekeepingServer(IPropertyTree *_config)
+        : Thread("CSashaDebugHousekeepingServer"), props(_config)
+    {
+        stopped = false;
+
+        StringBuffer userName;
+        props->getProp("@user", userName);
+    }
+
+    ~CSashaDebugHousekeepingServer()
+    {
+    }
+
+    void start()
+    {
+        Thread::start(false);
+    }
+
+    void ready()
+    {
+    }
+
+    void stop()
+    {
+        if (!stopped)
+        {
+            stopped = true;
+            stopsem.signal();
+        }
+        synchronized block(runmutex); // hopefully stopped should stop
+        if (!join(1000 * 60 * 3))
+            OERRLOG("CSashaDebugHousekeepingServer aborted");
+    }
+
+    void runDebugHousekeeping()
+    {
+        synchronized block(runmutex);
+        if (stopped)
+            return;
+        PROGLOG(LOGDBGHK "Started");
+        unsigned defaultExpireDays = props->getPropInt("@expiryDefault", DEFAULT_EXPIRYDAYS);
+
+        // get debug plane dir
+        StringBuffer debugDir;
+        StringBuffer planeName;
+        if (!getDefaultPlane(planeName, "@debugPlane", "debug"))
+        {
+            WARNLOG("Exception handlers configured, but debug plane is missing");
+            return;
+        }
+        Owned<IPropertyTree> plane = getStoragePlane(planeName);
+        assertex(plane);
+        verifyex(plane->getProp("@prefix", debugDir));
+
+        // iterate debug plane selecting post-mortem directories for housekeeping
+        Owned<IDirectoryIterator> pDirIter = createDirectoryIterator(debugDir.str(), "*", false, true);
+        ForEach(*pDirIter)
+        {
+            if (stopped)
+                break;
+
+            IFile &iDirFile = pDirIter->query();
+            const char *dirPath = iDirFile.queryFilename();
+
+            if (!dirPath || !*dirPath)
+                continue;
+
+            // Process directories, exclude the ".", "..", non post-mortem and not expired post-mortem directories
+            if (iDirFile.isDirectory() == fileBool::foundYes && *dirPath != '.')
+            {
+                // Ensure directory name only
+                StringBuffer dirNameOnly;
+                String dirNameStr(dirPath);
+                int fwdSlashIndex = dirNameStr.lastIndexOf('/');
+                if (fwdSlashIndex > -1)
+                {
+                    String *tmpStr = dirNameStr.substring(fwdSlashIndex + 1);
+                    dirNameOnly.append(*tmpStr);
+                    delete tmpStr;
+                }
+                else
+                {
+                    dirNameOnly.append(dirNameStr);
+                }
+
+                if (isPostMortemDirPath(dirNameOnly) && isExpiredDirPath(dirNameOnly, defaultExpireDays))
+                {
+                    recursiveRemoveDirectory(dirPath);
+                    PROGLOG(LOGDBGHK "Deleted %s", dirPath);
+                }
+            }
+        }
+        pDirIter.clear();
+
+        PROGLOG(LOGDBGHK "%s", stopped ? "Stopped" : "Done");
+    }
+
+    int run()
+    {
+        unsigned interval = props->getPropInt("@interval", DEFAULT_EXPIRY_INTERVAL);
+        if (!interval)
+            stopped = true;
+        PROGLOG(LOGDBGHK "min interval = %d hr", interval);
+        unsigned initinterval = (interval - 1) / 2; // wait a bit til dali has started
+        CSashaSchedule schedule;
+        if (interval)
+            schedule.init(props, interval, initinterval);
+        initinterval *= 60 * 60 * 1000; // ms
+        unsigned started = msTick();
+        while (!stopped)
+        {
+            stopsem.wait(1000 * 60);
+            if (stopped)
+                break;
+            if (!interval || ((started != (unsigned)-1) && (msTick() - started < initinterval)))
+                continue;
+            started = (unsigned)-1;
+            if (!schedule.ready())
+                continue;
+            try
+            {
+                runDebugHousekeeping();
+            }
+            catch (IException *e)
+            {
+                StringBuffer s;
+                EXCLOG(e, LOGDBGHK);
+                e->Release();
+            }
+        }
+        PROGLOG(LOGDBGHK "Exit");
+        return 0;
+    }
+
+private:
+    bool isPostMortemDirPath(const StringBuffer &dirName)
+    {
+        // Expecting a directory name like "W20250225-101112"
+        RegExpr postMortemDirRegEx("^W[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]-[0-9][0-9][0-9][0-9][0-9][0-9]$");
+        if (postMortemDirRegEx.find(dirName.str()))
+        {
+            PROGLOG(LOGDBGHK "Post-mortem dir: %s", dirName.str());
+
+            return true;
+        }
+        else
+        {
+            PROGLOG(LOGDBGHK "Non post-mortem dir: %s", dirName.str());
+
+            return false;
+        }
+    }
+
+    bool isExpiredDirPath(StringBuffer &dirName, const unsigned &defaultExpireDays)
+    {
+        // Directory name is like "W20250225-101112"
+        StringBuffer cDateTimeoprmattedString;
+        cDateTimeoprmattedString.appendf("%c%c%c%c-%c%c-%c%cT%c%c:%c%c:%c%c",
+                                         dirName.charAt(1), dirName.charAt(2), dirName.charAt(3), dirName.charAt(4),
+                                         dirName.charAt(5), dirName.charAt(6),
+                                         dirName.charAt(7), dirName.charAt(8),
+                                         dirName.charAt(10), dirName.charAt(11),
+                                         dirName.charAt(12), dirName.charAt(13),
+                                         dirName.charAt(14), dirName.charAt(15));
+
+        CDateTime now;
+        now.setNow();
+
+        CDateTime expires;
+        expires.setString(cDateTimeoprmattedString.str());
+        expires.adjustTime(60 * 24 * defaultExpireDays);
+
+        if (now.compare(expires, false) > 0)
+        {
+            PROGLOG(LOGDBGHK "Post-mortem dir: %s has expired", dirName.str());
+
+            return true;
+        }
+        else
+        {
+            return false;
+        }
+    }
+
+} *sashaDebugHousekeepingServer = NULL;
+
+ISashaServer *createSashaDebugHousekeepingServer()
+{
+    assertex(!sashaDebugHousekeepingServer); // initialization problem
+    Linked<IPropertyTree> config = serverConfig;
+    sashaDebugHousekeepingServer = new CSashaDebugHousekeepingServer(config);
path: dali/sasha/sadbghk.cpp
user: login: jakesmith
body: see other comments re. containerized, and using getComponentConfig*(). SO no need to pass in config here.
Worth adding a assertex(isContainerized() in this func for clarity.
created_at: 2025-03-12T10:05:23Z
author_association: MEMBER
diff_hunk: @@ -435,6 +437,8 @@ int main(int argc, const char* argv[])
                     servers.append(*createSashaFileExpiryServer());
                 else if (strieq(service, "thor-qmon"))
                     servers.append(*createSashaQMonitorServer());
+                else if (strieq(service, "debug-housekeeping"))
path: dali/sasha/saserver.cpp
user: login: jakesmith
body: let's rename to "debugplane-housekeeping"
created_at: 2025-03-12T10:05:54Z
author_association: MEMBER
diff_hunk: @@ -75,6 +76,7 @@ static void AddServers()
     servers.append(*createSashaDaFSMonitorServer());
     servers.append(*createSashaQMonitorServer());
     servers.append(*createSashaFileExpiryServer()); 
+    servers.append(*createSashaDebugHousekeepingServer());
path: dali/sasha/saserver.cpp
user: login: jakesmith
body: this is BM only, and this service will not be useable there, this line should be removed.
created_at: 2025-03-12T10:06:21Z
author_association: MEMBER
diff_hunk: @@ -99,6 +99,7 @@ dali.sasha.coalescer
 sasha.dfurecovery-archiver
 sasha.dfuwu-archiver
 sasha.file-expiry
+sasha.debug-housekeeping
path: docs/EN_US/ContainerizedHPCC/ContainerizedMods/LocalDeployment.xml
user: login: jakesmith
body: debugplane-housekeeping .. and in other yaml's
created_at: 2025-03-12T10:06:52Z
author_association: MEMBER
diff_hunk: @@ -2417,6 +2417,29 @@ sudo -u hpcc cp /etc/HPCCSystems/source/NewEnvironment.xml /etc/HPCCSystems/envi
 
         <?hard-pagebreak ?>
 
+        <sect3>
path: docs/EN_US/ConfiguringHPCC/ConfiguringHPCC.xml
user: login: jakesmith
body: docs/*/ConfiguringHPCC/ are BM only docs, should be left unchanged as this service will not be part of BM.
created_at: 2025-03-12T10:11:10Z
author_association: MEMBER
diff_hunk: @@ -1715,6 +1715,8 @@ dali
 dali data
 {{- else if (eq "thor-qmon" .name) -}}
 dali queues
+{{- else if (eq "debug-housekeeping" .name) -}}
+dali data
path: helm/hpcc/templates/_helpers.tpl
user: login: jakesmith
body: does not need 'dali' access.
created_at: 2025-03-12T10:11:32Z
author_association: MEMBER
diff_hunk: @@ -3045,6 +3045,35 @@
       },
       "additionalProperties": false
     },
+    "sasha-debug-housekeeping": {
+      "type": "object",
+      "allOf": [{ "$ref": "#/definitions/sashacommon" }],
+      "properties": {
+        "expiryDefault": {
+          "type": "integer",
+          "description": "Default number of days passed that post-mortem files are deleted after their associated workunit was completed",
+          "default": "14"
+        },
+        "user": {
+          "type": "string",
+          "description": "A username authorized to access and remove post-mortem files"
path: helm/hpcc/values.schema.json
user: login: jakesmith
body: does not need a 'user'.
created_at: 2025-03-12T10:11:54Z
author_association: MEMBER
diff_hunk: @@ -3045,6 +3045,35 @@
       },
       "additionalProperties": false
     },
+    "sasha-debug-housekeeping": {
+      "type": "object",
+      "allOf": [{ "$ref": "#/definitions/sashacommon" }],
+      "properties": {
+        "expiryDefault": {
path: helm/hpcc/values.schema.json
user: login: jakesmith
body: see other comment, let's rename to "expiryDays'
created_at: 2025-03-12T10:12:08Z
author_association: MEMBER
diff_hunk: @@ -510,6 +510,16 @@ sasha:
     resources:
       cpu: "1"
       memory: "4Gi"
+      
+  debug-housekeeping:
+    #disabled: true
+    #interval: 1
+    #at: "* 3 * * *"
+    #expiryDefault: 7
+    #user: sasha
+    resources:
+      cpu: "1"
+      memory: "4Gi"
path: testing/helm/tests/resourced.yaml
user: login: jakesmith
body: this service in particular should require a trivial amount of memory, to emphasis that, let's reduce this to 1Gi
created_at: 2025-03-12T10:12:51Z
author_association: MEMBER
diff_hunk: @@ -3,55 +3,60 @@
 #include "jlib.hpp"
 #include "jiface.hpp"
 #include "jptree.hpp"
-#include "jregexp.hpp"
+#include "jstring.hpp"
 
 #include "dadfs.hpp"
 #include "dalienv.hpp"
 
 #include "saserver.hpp"
 #include "sautil.hpp"
 
-#define DEFAULT_EXPIRY_INTERVAL 24 // hours
+#define DEFAULT_HOUSEKEEPING_INTERVAL_HOURS 24
path: dali/sasha/sadbghk.cpp
user: login: jakesmith
body: trivial:
```
static constexpr unsigned defaultHouseKeepingIntervalHours = 24;
```
more modern/nicer to the compiler.
created_at: 2025-03-13T18:42:52Z
author_association: MEMBER
diff_hunk: @@ -3,55 +3,60 @@
 #include "jlib.hpp"
 #include "jiface.hpp"
 #include "jptree.hpp"
-#include "jregexp.hpp"
+#include "jstring.hpp"
 
 #include "dadfs.hpp"
 #include "dalienv.hpp"
 
 #include "saserver.hpp"
 #include "sautil.hpp"
 
-#define DEFAULT_EXPIRY_INTERVAL 24 // hours
+#define DEFAULT_HOUSEKEEPING_INTERVAL_HOURS 24
 
-#define DEFAULT_EXPIRYDAYS 7
-
-#define LOGDBGHK "DEBUGHOUSEKEEPING: "
+#define LOGDBGHK "DEBUGPLANEHOUSEKEEPING: "
 
 // Debug Housekeeping monitor
 
-class CSashaDebugHousekeepingServer : public ISashaServer, public Thread
+class CSashaDebugPlaneHousekeepingServer : public ISashaServer, public Thread
 {
-    bool stopped;
+    bool stopped{false};
     Semaphore stopsem;
     Mutex runmutex;
-    Linked<IPropertyTree> props;
+    StringBuffer debugDir;
+    unsigned expiryDays{0};
 
 public:
     IMPLEMENT_IINTERFACE_USING(Thread);
 
-    CSashaDebugHousekeepingServer(IPropertyTree *_config)
-        : Thread("CSashaDebugHousekeepingServer"), props(_config)
+    CSashaDebugPlaneHousekeepingServer()
+        : Thread("CSashaDebugPlaneHousekeepingServer")
     {
-        stopped = false;
-
-        StringBuffer userName;
-        props->getProp("@user", userName);
     }
 
-    ~CSashaDebugHousekeepingServer()
-    {
-    }
 
-    void start()
+    virtual void start() override
     {
         Thread::start(false);
+
+        expiryDays = getComponentConfig()->getPropInt("@expiryDays");
path: dali/sasha/sadbghk.cpp
user: login: jakesmith
body: leaks. getComponentConfig returns a linked IPropertyTree - and nothing releases it here.
Either need to do a : Owned<IPropertyTree> config = getComponentConfig();  - so owned, and Owned dtor releases it, or, most suitable in this case, use getComponentConfigSP() - which will invoke an inlined Owned which will be destroyed after use.
created_at: 2025-03-13T18:48:36Z
author_association: MEMBER
diff_hunk: @@ -3,55 +3,60 @@
 #include "jlib.hpp"
 #include "jiface.hpp"
 #include "jptree.hpp"
-#include "jregexp.hpp"
+#include "jstring.hpp"
 
 #include "dadfs.hpp"
 #include "dalienv.hpp"
 
 #include "saserver.hpp"
 #include "sautil.hpp"
 
-#define DEFAULT_EXPIRY_INTERVAL 24 // hours
+#define DEFAULT_HOUSEKEEPING_INTERVAL_HOURS 24
 
-#define DEFAULT_EXPIRYDAYS 7
-
-#define LOGDBGHK "DEBUGHOUSEKEEPING: "
+#define LOGDBGHK "DEBUGPLANEHOUSEKEEPING: "
 
 // Debug Housekeeping monitor
 
-class CSashaDebugHousekeepingServer : public ISashaServer, public Thread
+class CSashaDebugPlaneHousekeepingServer : public ISashaServer, public Thread
 {
-    bool stopped;
+    bool stopped{false};
     Semaphore stopsem;
     Mutex runmutex;
-    Linked<IPropertyTree> props;
+    StringBuffer debugDir;
+    unsigned expiryDays{0};
 
 public:
     IMPLEMENT_IINTERFACE_USING(Thread);
 
-    CSashaDebugHousekeepingServer(IPropertyTree *_config)
-        : Thread("CSashaDebugHousekeepingServer"), props(_config)
+    CSashaDebugPlaneHousekeepingServer()
+        : Thread("CSashaDebugPlaneHousekeepingServer")
     {
-        stopped = false;
-
-        StringBuffer userName;
-        props->getProp("@user", userName);
     }
 
-    ~CSashaDebugHousekeepingServer()
-    {
-    }
 
-    void start()
+    virtual void start() override
     {
         Thread::start(false);
+
+        expiryDays = getComponentConfig()->getPropInt("@expiryDays");
+
+        // get debug plane dir
+        StringBuffer planeName;
+        if (!getDefaultPlane(planeName, "@debugPlane", "debug"))
+        {
+            WARNLOG("Exception handlers configured, but debug plane is missing");
path: dali/sasha/sadbghk.cpp
user: login: jakesmith
body: not sure what this message relates to ? copy and pasted?
created_at: 2025-03-13T18:49:01Z
author_association: MEMBER
diff_hunk: @@ -60,28 +65,14 @@ class CSashaDebugHousekeepingServer : public ISashaServer, public Thread
         }
         synchronized block(runmutex); // hopefully stopped should stop
         if (!join(1000 * 60 * 3))
-            OERRLOG("CSashaDebugHousekeepingServer aborted");
+            OERRLOG("CSashaDebugPlaneHousekeepingServer aborted");
     }
 
     void runDebugHousekeeping()
     {
         synchronized block(runmutex);
         if (stopped)
             return;
-        PROGLOG(LOGDBGHK "Started");
-        unsigned defaultExpireDays = props->getPropInt("@expiryDefault", DEFAULT_EXPIRYDAYS);
-
-        // get debug plane dir
-        StringBuffer debugDir;
-        StringBuffer planeName;
-        if (!getDefaultPlane(planeName, "@debugPlane", "debug"))
-        {
-            WARNLOG("Exception handlers configured, but debug plane is missing");
-            return;
-        }
-        Owned<IPropertyTree> plane = getStoragePlane(planeName);
-        assertex(plane);
-        verifyex(plane->getProp("@prefix", debugDir));
 
         // iterate debug plane selecting post-mortem directories for housekeeping
path: dali/sasha/sadbghk.cpp
user: login: jakesmith
body: comment no longer accurate
created_at: 2025-03-13T18:49:48Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,164 @@
+#include "platform.h"
+
+#include "jlib.hpp"
+#include "jiface.hpp"
+#include "jptree.hpp"
+#include "jstring.hpp"
+
+#include "dadfs.hpp"
+#include "dalienv.hpp"
+
+#include "saserver.hpp"
+#include "sautil.hpp"
+
+#define DEFAULT_HOUSEKEEPING_INTERVAL_HOURS 24
+
+#define LOGDBGHK "DEBUGPLANEHOUSEKEEPING: "
+
+// Debug Housekeeping monitor
+
+class CSashaDebugPlaneHousekeepingServer : public ISashaServer, public Thread
+{
+    bool stopped{false};
+    Semaphore stopsem;
+    Mutex runmutex;
+    StringBuffer debugDir;
+    unsigned expiryDays{0};
+
+public:
+    IMPLEMENT_IINTERFACE_USING(Thread);
+
+    CSashaDebugPlaneHousekeepingServer()
+        : Thread("CSashaDebugPlaneHousekeepingServer")
+    {
+    }
+
+    virtual void start() override
+    {
+        Thread::start(false);
+
+        expiryDays = getComponentConfig()->getPropInt("@expiryDays");
+
+        // get debug plane dir
+        StringBuffer planeName;
+        if (!getDefaultPlane(planeName, "@debugPlane", "debug"))
+        {
+            WARNLOG("Exception handlers configured, but debug plane is missing");
+            return;
+        }
+        Owned<IPropertyTree> plane = getStoragePlane(planeName);
+        assertex(plane);
+        verifyex(plane->getProp("@prefix", debugDir));
+    }
+
+    virtual void ready() override
+    {
+    }
+
+    virtual void stop() override
+    {
+        if (!stopped)
+        {
+            stopped = true;
+            stopsem.signal();
+        }
+        synchronized block(runmutex); // hopefully stopped should stop
+        if (!join(1000 * 60 * 3))
+            OERRLOG("CSashaDebugPlaneHousekeepingServer aborted");
+    }
+
+    void runDebugHousekeeping()
+    {
+        synchronized block(runmutex);
+        if (stopped)
+            return;
+
+        // iterate debug plane selecting post-mortem directories for housekeeping
+        Owned<IDirectoryIterator> pDirIter = createDirectoryIterator(debugDir.str(), "*", false, true);
+        ForEach(*pDirIter)
+        {
+            if (stopped)
+                break;
+
+            IFile &iFile = pDirIter->query();
+            const char *filePath = iFile.queryFilename();
+
+            // Process directories, exclude the ".", "..", non post-mortem and not expired post-mortem directories
path: dali/sasha/sadbghk.cpp
user: login: jakesmith
body: comment needs revising
created_at: 2025-03-13T18:52:35Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,164 @@
+#include "platform.h"
+
+#include "jlib.hpp"
+#include "jiface.hpp"
+#include "jptree.hpp"
+#include "jstring.hpp"
+
+#include "dadfs.hpp"
+#include "dalienv.hpp"
+
+#include "saserver.hpp"
+#include "sautil.hpp"
+
+#define DEFAULT_HOUSEKEEPING_INTERVAL_HOURS 24
+
+#define LOGDBGHK "DEBUGPLANEHOUSEKEEPING: "
+
+// Debug Housekeeping monitor
+
+class CSashaDebugPlaneHousekeepingServer : public ISashaServer, public Thread
+{
+    bool stopped{false};
+    Semaphore stopsem;
+    Mutex runmutex;
+    StringBuffer debugDir;
+    unsigned expiryDays{0};
+
+public:
+    IMPLEMENT_IINTERFACE_USING(Thread);
+
+    CSashaDebugPlaneHousekeepingServer()
+        : Thread("CSashaDebugPlaneHousekeepingServer")
+    {
+    }
+
+    virtual void start() override
+    {
+        Thread::start(false);
+
+        expiryDays = getComponentConfig()->getPropInt("@expiryDays");
+
+        // get debug plane dir
+        StringBuffer planeName;
+        if (!getDefaultPlane(planeName, "@debugPlane", "debug"))
+        {
+            WARNLOG("Exception handlers configured, but debug plane is missing");
+            return;
+        }
+        Owned<IPropertyTree> plane = getStoragePlane(planeName);
+        assertex(plane);
+        verifyex(plane->getProp("@prefix", debugDir));
+    }
+
+    virtual void ready() override
+    {
+    }
+
+    virtual void stop() override
+    {
+        if (!stopped)
+        {
+            stopped = true;
+            stopsem.signal();
+        }
+        synchronized block(runmutex); // hopefully stopped should stop
+        if (!join(1000 * 60 * 3))
+            OERRLOG("CSashaDebugPlaneHousekeepingServer aborted");
+    }
+
+    void runDebugHousekeeping()
+    {
+        synchronized block(runmutex);
+        if (stopped)
+            return;
+
+        // iterate debug plane selecting post-mortem directories for housekeeping
+        Owned<IDirectoryIterator> pDirIter = createDirectoryIterator(debugDir.str(), "*", false, true);
+        ForEach(*pDirIter)
+        {
+            if (stopped)
+                break;
+
+            IFile &iFile = pDirIter->query();
+            const char *filePath = iFile.queryFilename();
+
+            // Process directories, exclude the ".", "..", non post-mortem and not expired post-mortem directories
+            if (iFile.isDirectory() == fileBool::foundYes)
+            {
+                if (streq(filePath, ".") || streq(filePath, ".."))
+                    continue;
+
+                if (isExpiredModifiedDateTime(filePath))
path: dali/sasha/sadbghk.cpp
user: login: jakesmith
body: already have iFile, so better to pass that in, rather than create one inside isExpiredModifiedDateTime
created_at: 2025-03-13T18:53:59Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,164 @@
+#include "platform.h"
+
+#include "jlib.hpp"
+#include "jiface.hpp"
+#include "jptree.hpp"
+#include "jstring.hpp"
+
+#include "dadfs.hpp"
+#include "dalienv.hpp"
+
+#include "saserver.hpp"
+#include "sautil.hpp"
+
+#define DEFAULT_HOUSEKEEPING_INTERVAL_HOURS 24
+
+#define LOGDBGHK "DEBUGPLANEHOUSEKEEPING: "
+
+// Debug Housekeeping monitor
+
+class CSashaDebugPlaneHousekeepingServer : public ISashaServer, public Thread
+{
+    bool stopped{false};
+    Semaphore stopsem;
+    Mutex runmutex;
+    StringBuffer debugDir;
+    unsigned expiryDays{0};
+
+public:
+    IMPLEMENT_IINTERFACE_USING(Thread);
+
+    CSashaDebugPlaneHousekeepingServer()
+        : Thread("CSashaDebugPlaneHousekeepingServer")
+    {
+    }
+
+    virtual void start() override
+    {
+        Thread::start(false);
+
+        expiryDays = getComponentConfig()->getPropInt("@expiryDays");
+
+        // get debug plane dir
+        StringBuffer planeName;
+        if (!getDefaultPlane(planeName, "@debugPlane", "debug"))
+        {
+            WARNLOG("Exception handlers configured, but debug plane is missing");
+            return;
+        }
+        Owned<IPropertyTree> plane = getStoragePlane(planeName);
+        assertex(plane);
+        verifyex(plane->getProp("@prefix", debugDir));
+    }
+
+    virtual void ready() override
+    {
+    }
+
+    virtual void stop() override
+    {
+        if (!stopped)
+        {
+            stopped = true;
+            stopsem.signal();
+        }
+        synchronized block(runmutex); // hopefully stopped should stop
+        if (!join(1000 * 60 * 3))
+            OERRLOG("CSashaDebugPlaneHousekeepingServer aborted");
+    }
+
+    void runDebugHousekeeping()
+    {
+        synchronized block(runmutex);
+        if (stopped)
+            return;
+
+        // iterate debug plane selecting post-mortem directories for housekeeping
+        Owned<IDirectoryIterator> pDirIter = createDirectoryIterator(debugDir.str(), "*", false, true);
+        ForEach(*pDirIter)
+        {
+            if (stopped)
+                break;
+
+            IFile &iFile = pDirIter->query();
+            const char *filePath = iFile.queryFilename();
+
+            // Process directories, exclude the ".", "..", non post-mortem and not expired post-mortem directories
+            if (iFile.isDirectory() == fileBool::foundYes)
+            {
+                if (streq(filePath, ".") || streq(filePath, ".."))
+                    continue;
+
+                if (isExpiredModifiedDateTime(filePath))
path: dali/sasha/sadbghk.cpp
user: login: jakesmith
body: personal: "hasExpired" would suffice.
created_at: 2025-03-13T18:54:25Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,164 @@
+#include "platform.h"
+
+#include "jlib.hpp"
+#include "jiface.hpp"
+#include "jptree.hpp"
+#include "jstring.hpp"
+
+#include "dadfs.hpp"
+#include "dalienv.hpp"
+
+#include "saserver.hpp"
+#include "sautil.hpp"
+
+#define DEFAULT_HOUSEKEEPING_INTERVAL_HOURS 24
+
+#define LOGDBGHK "DEBUGPLANEHOUSEKEEPING: "
+
+// Debug Housekeeping monitor
+
+class CSashaDebugPlaneHousekeepingServer : public ISashaServer, public Thread
+{
+    bool stopped{false};
+    Semaphore stopsem;
+    Mutex runmutex;
+    StringBuffer debugDir;
+    unsigned expiryDays{0};
+
+public:
+    IMPLEMENT_IINTERFACE_USING(Thread);
+
+    CSashaDebugPlaneHousekeepingServer()
+        : Thread("CSashaDebugPlaneHousekeepingServer")
+    {
+    }
+
+    virtual void start() override
+    {
+        Thread::start(false);
+
+        expiryDays = getComponentConfig()->getPropInt("@expiryDays");
+
+        // get debug plane dir
+        StringBuffer planeName;
+        if (!getDefaultPlane(planeName, "@debugPlane", "debug"))
+        {
+            WARNLOG("Exception handlers configured, but debug plane is missing");
+            return;
+        }
+        Owned<IPropertyTree> plane = getStoragePlane(planeName);
+        assertex(plane);
+        verifyex(plane->getProp("@prefix", debugDir));
+    }
+
+    virtual void ready() override
+    {
+    }
+
+    virtual void stop() override
+    {
+        if (!stopped)
+        {
+            stopped = true;
+            stopsem.signal();
+        }
+        synchronized block(runmutex); // hopefully stopped should stop
+        if (!join(1000 * 60 * 3))
+            OERRLOG("CSashaDebugPlaneHousekeepingServer aborted");
+    }
+
+    void runDebugHousekeeping()
+    {
+        synchronized block(runmutex);
+        if (stopped)
+            return;
+
+        // iterate debug plane selecting post-mortem directories for housekeeping
+        Owned<IDirectoryIterator> pDirIter = createDirectoryIterator(debugDir.str(), "*", false, true);
+        ForEach(*pDirIter)
+        {
+            if (stopped)
+                break;
+
+            IFile &iFile = pDirIter->query();
+            const char *filePath = iFile.queryFilename();
+
+            // Process directories, exclude the ".", "..", non post-mortem and not expired post-mortem directories
+            if (iFile.isDirectory() == fileBool::foundYes)
+            {
+                if (streq(filePath, ".") || streq(filePath, ".."))
+                    continue;
+
+                if (isExpiredModifiedDateTime(filePath))
+                {
+                    recursiveRemoveDirectory(filePath);
path: dali/sasha/sadbghk.cpp
user: login: jakesmith
body: similarly, use iFile (there's a recursiveRemoveDirectory that takes an IFile).
created_at: 2025-03-13T18:55:46Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,164 @@
+#include "platform.h"
+
+#include "jlib.hpp"
+#include "jiface.hpp"
+#include "jptree.hpp"
+#include "jstring.hpp"
+
+#include "dadfs.hpp"
+#include "dalienv.hpp"
+
+#include "saserver.hpp"
+#include "sautil.hpp"
+
+#define DEFAULT_HOUSEKEEPING_INTERVAL_HOURS 24
+
+#define LOGDBGHK "DEBUGPLANEHOUSEKEEPING: "
+
+// Debug Housekeeping monitor
+
+class CSashaDebugPlaneHousekeepingServer : public ISashaServer, public Thread
+{
+    bool stopped{false};
+    Semaphore stopsem;
+    Mutex runmutex;
+    StringBuffer debugDir;
+    unsigned expiryDays{0};
+
+public:
+    IMPLEMENT_IINTERFACE_USING(Thread);
+
+    CSashaDebugPlaneHousekeepingServer()
+        : Thread("CSashaDebugPlaneHousekeepingServer")
+    {
+    }
+
+    virtual void start() override
+    {
+        Thread::start(false);
+
+        expiryDays = getComponentConfig()->getPropInt("@expiryDays");
+
+        // get debug plane dir
+        StringBuffer planeName;
+        if (!getDefaultPlane(planeName, "@debugPlane", "debug"))
+        {
+            WARNLOG("Exception handlers configured, but debug plane is missing");
+            return;
+        }
+        Owned<IPropertyTree> plane = getStoragePlane(planeName);
+        assertex(plane);
+        verifyex(plane->getProp("@prefix", debugDir));
+    }
+
+    virtual void ready() override
+    {
+    }
+
+    virtual void stop() override
+    {
+        if (!stopped)
+        {
+            stopped = true;
+            stopsem.signal();
+        }
+        synchronized block(runmutex); // hopefully stopped should stop
+        if (!join(1000 * 60 * 3))
+            OERRLOG("CSashaDebugPlaneHousekeepingServer aborted");
+    }
+
+    void runDebugHousekeeping()
+    {
+        synchronized block(runmutex);
+        if (stopped)
+            return;
+
+        // iterate debug plane selecting post-mortem directories for housekeeping
+        Owned<IDirectoryIterator> pDirIter = createDirectoryIterator(debugDir.str(), "*", false, true);
+        ForEach(*pDirIter)
+        {
+            if (stopped)
+                break;
+
+            IFile &iFile = pDirIter->query();
+            const char *filePath = iFile.queryFilename();
+
+            // Process directories, exclude the ".", "..", non post-mortem and not expired post-mortem directories
+            if (iFile.isDirectory() == fileBool::foundYes)
+            {
+                if (streq(filePath, ".") || streq(filePath, ".."))
+                    continue;
+
+                if (isExpiredModifiedDateTime(filePath))
+                {
+                    recursiveRemoveDirectory(filePath);
+                }
+            }
+            else
+            {
+                if (isExpiredModifiedDateTime(filePath))
+                {
+                    iFile.remove();
+                }
+            }
+        }
+        pDirIter.clear();
path: dali/sasha/sadbghk.cpp
user: login: jakesmith
body: unnecessary
created_at: 2025-03-13T18:55:59Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,164 @@
+#include "platform.h"
+
+#include "jlib.hpp"
+#include "jiface.hpp"
+#include "jptree.hpp"
+#include "jstring.hpp"
+
+#include "dadfs.hpp"
+#include "dalienv.hpp"
+
+#include "saserver.hpp"
+#include "sautil.hpp"
+
+#define DEFAULT_HOUSEKEEPING_INTERVAL_HOURS 24
+
+#define LOGDBGHK "DEBUGPLANEHOUSEKEEPING: "
+
+// Debug Housekeeping monitor
+
+class CSashaDebugPlaneHousekeepingServer : public ISashaServer, public Thread
+{
+    bool stopped{false};
+    Semaphore stopsem;
+    Mutex runmutex;
+    StringBuffer debugDir;
+    unsigned expiryDays{0};
+
+public:
+    IMPLEMENT_IINTERFACE_USING(Thread);
+
+    CSashaDebugPlaneHousekeepingServer()
+        : Thread("CSashaDebugPlaneHousekeepingServer")
+    {
+    }
+
+    virtual void start() override
+    {
+        Thread::start(false);
+
+        expiryDays = getComponentConfig()->getPropInt("@expiryDays");
+
+        // get debug plane dir
+        StringBuffer planeName;
+        if (!getDefaultPlane(planeName, "@debugPlane", "debug"))
+        {
+            WARNLOG("Exception handlers configured, but debug plane is missing");
+            return;
+        }
+        Owned<IPropertyTree> plane = getStoragePlane(planeName);
+        assertex(plane);
+        verifyex(plane->getProp("@prefix", debugDir));
+    }
+
+    virtual void ready() override
+    {
+    }
+
+    virtual void stop() override
+    {
+        if (!stopped)
+        {
+            stopped = true;
+            stopsem.signal();
+        }
+        synchronized block(runmutex); // hopefully stopped should stop
+        if (!join(1000 * 60 * 3))
+            OERRLOG("CSashaDebugPlaneHousekeepingServer aborted");
+    }
+
+    void runDebugHousekeeping()
+    {
+        synchronized block(runmutex);
+        if (stopped)
+            return;
+
+        // iterate debug plane selecting post-mortem directories for housekeeping
+        Owned<IDirectoryIterator> pDirIter = createDirectoryIterator(debugDir.str(), "*", false, true);
+        ForEach(*pDirIter)
+        {
+            if (stopped)
+                break;
+
+            IFile &iFile = pDirIter->query();
+            const char *filePath = iFile.queryFilename();
+
+            // Process directories, exclude the ".", "..", non post-mortem and not expired post-mortem directories
+            if (iFile.isDirectory() == fileBool::foundYes)
+            {
+                if (streq(filePath, ".") || streq(filePath, ".."))
+                    continue;
+
+                if (isExpiredModifiedDateTime(filePath))
+                {
+                    recursiveRemoveDirectory(filePath);
+                }
+            }
+            else
+            {
+                if (isExpiredModifiedDateTime(filePath))
+                {
+                    iFile.remove();
+                }
+            }
+        }
+        pDirIter.clear();
+    }
+
+    virtual int run() override
+    {
+        CSashaSchedule schedule;
+        unsigned interval = getComponentConfig()->getPropInt("@interval", DEFAULT_HOUSEKEEPING_INTERVAL_HOURS);
path: dali/sasha/sadbghk.cpp
user: login: jakesmith
body: leaks. See prev. comment.
created_at: 2025-03-13T18:56:39Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,164 @@
+#include "platform.h"
+
+#include "jlib.hpp"
+#include "jiface.hpp"
+#include "jptree.hpp"
+#include "jstring.hpp"
+
+#include "dadfs.hpp"
+#include "dalienv.hpp"
+
+#include "saserver.hpp"
+#include "sautil.hpp"
+
+#define DEFAULT_HOUSEKEEPING_INTERVAL_HOURS 24
+
+#define LOGDBGHK "DEBUGPLANEHOUSEKEEPING: "
+
+// Debug Housekeeping monitor
+
+class CSashaDebugPlaneHousekeepingServer : public ISashaServer, public Thread
+{
+    bool stopped{false};
+    Semaphore stopsem;
+    Mutex runmutex;
+    StringBuffer debugDir;
+    unsigned expiryDays{0};
+
+public:
+    IMPLEMENT_IINTERFACE_USING(Thread);
+
+    CSashaDebugPlaneHousekeepingServer()
+        : Thread("CSashaDebugPlaneHousekeepingServer")
+    {
+    }
+
+    virtual void start() override
+    {
+        Thread::start(false);
+
+        expiryDays = getComponentConfig()->getPropInt("@expiryDays");
+
+        // get debug plane dir
+        StringBuffer planeName;
+        if (!getDefaultPlane(planeName, "@debugPlane", "debug"))
+        {
+            WARNLOG("Exception handlers configured, but debug plane is missing");
+            return;
+        }
+        Owned<IPropertyTree> plane = getStoragePlane(planeName);
+        assertex(plane);
+        verifyex(plane->getProp("@prefix", debugDir));
+    }
+
+    virtual void ready() override
+    {
+    }
+
+    virtual void stop() override
+    {
+        if (!stopped)
+        {
+            stopped = true;
+            stopsem.signal();
+        }
+        synchronized block(runmutex); // hopefully stopped should stop
+        if (!join(1000 * 60 * 3))
+            OERRLOG("CSashaDebugPlaneHousekeepingServer aborted");
+    }
+
+    void runDebugHousekeeping()
+    {
+        synchronized block(runmutex);
+        if (stopped)
+            return;
+
+        // iterate debug plane selecting post-mortem directories for housekeeping
+        Owned<IDirectoryIterator> pDirIter = createDirectoryIterator(debugDir.str(), "*", false, true);
+        ForEach(*pDirIter)
+        {
+            if (stopped)
+                break;
+
+            IFile &iFile = pDirIter->query();
+            const char *filePath = iFile.queryFilename();
+
+            // Process directories, exclude the ".", "..", non post-mortem and not expired post-mortem directories
+            if (iFile.isDirectory() == fileBool::foundYes)
+            {
+                if (streq(filePath, ".") || streq(filePath, ".."))
+                    continue;
+
+                if (isExpiredModifiedDateTime(filePath))
+                {
+                    recursiveRemoveDirectory(filePath);
+                }
+            }
+            else
+            {
+                if (isExpiredModifiedDateTime(filePath))
+                {
+                    iFile.remove();
+                }
+            }
+        }
+        pDirIter.clear();
+    }
+
+    virtual int run() override
+    {
+        CSashaSchedule schedule;
+        unsigned interval = getComponentConfig()->getPropInt("@interval", DEFAULT_HOUSEKEEPING_INTERVAL_HOURS);
+        if (interval == 0)
+        {
+            stopped = true;
+            return 0;
+        }
+        schedule.init(getComponentConfig(), interval);
path: dali/sasha/sadbghk.cpp
user: login: jakesmith
body: leaks.
created_at: 2025-03-13T18:56:45Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,164 @@
+#include "platform.h"
+
+#include "jlib.hpp"
+#include "jiface.hpp"
+#include "jptree.hpp"
+#include "jstring.hpp"
+
+#include "dadfs.hpp"
+#include "dalienv.hpp"
+
+#include "saserver.hpp"
+#include "sautil.hpp"
+
+#define DEFAULT_HOUSEKEEPING_INTERVAL_HOURS 24
+
+#define LOGDBGHK "DEBUGPLANEHOUSEKEEPING: "
+
+// Debug Housekeeping monitor
+
+class CSashaDebugPlaneHousekeepingServer : public ISashaServer, public Thread
+{
+    bool stopped{false};
+    Semaphore stopsem;
+    Mutex runmutex;
+    StringBuffer debugDir;
+    unsigned expiryDays{0};
+
+public:
+    IMPLEMENT_IINTERFACE_USING(Thread);
+
+    CSashaDebugPlaneHousekeepingServer()
+        : Thread("CSashaDebugPlaneHousekeepingServer")
+    {
+    }
+
+    virtual void start() override
+    {
+        Thread::start(false);
+
+        expiryDays = getComponentConfig()->getPropInt("@expiryDays");
+
+        // get debug plane dir
+        StringBuffer planeName;
+        if (!getDefaultPlane(planeName, "@debugPlane", "debug"))
+        {
+            WARNLOG("Exception handlers configured, but debug plane is missing");
+            return;
+        }
+        Owned<IPropertyTree> plane = getStoragePlane(planeName);
+        assertex(plane);
+        verifyex(plane->getProp("@prefix", debugDir));
+    }
+
+    virtual void ready() override
+    {
+    }
+
+    virtual void stop() override
+    {
+        if (!stopped)
+        {
+            stopped = true;
+            stopsem.signal();
+        }
+        synchronized block(runmutex); // hopefully stopped should stop
+        if (!join(1000 * 60 * 3))
+            OERRLOG("CSashaDebugPlaneHousekeepingServer aborted");
+    }
+
+    void runDebugHousekeeping()
+    {
+        synchronized block(runmutex);
+        if (stopped)
+            return;
+
+        // iterate debug plane selecting post-mortem directories for housekeeping
+        Owned<IDirectoryIterator> pDirIter = createDirectoryIterator(debugDir.str(), "*", false, true);
+        ForEach(*pDirIter)
+        {
+            if (stopped)
+                break;
+
+            IFile &iFile = pDirIter->query();
+            const char *filePath = iFile.queryFilename();
+
+            // Process directories, exclude the ".", "..", non post-mortem and not expired post-mortem directories
+            if (iFile.isDirectory() == fileBool::foundYes)
+            {
+                if (streq(filePath, ".") || streq(filePath, ".."))
+                    continue;
+
+                if (isExpiredModifiedDateTime(filePath))
+                {
+                    recursiveRemoveDirectory(filePath);
+                }
+            }
+            else
+            {
+                if (isExpiredModifiedDateTime(filePath))
+                {
+                    iFile.remove();
+                }
+            }
+        }
+        pDirIter.clear();
+    }
+
+    virtual int run() override
+    {
+        CSashaSchedule schedule;
+        unsigned interval = getComponentConfig()->getPropInt("@interval", DEFAULT_HOUSEKEEPING_INTERVAL_HOURS);
+        if (interval == 0)
+        {
+            stopped = true;
+            return 0;
+        }
+        schedule.init(getComponentConfig(), interval);
+        while (!stopped)
+        {
+            stopsem.wait(1000 * 60);
+            if (stopped)
+                break;
+            if (!schedule.ready())
+                continue;
+            try
+            {
+                runDebugHousekeeping();
+            }
+            catch (IException *e)
+            {
+                StringBuffer s;
path: dali/sasha/sadbghk.cpp
user: login: jakesmith
body: unused
created_at: 2025-03-13T18:57:09Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,164 @@
+#include "platform.h"
+
+#include "jlib.hpp"
+#include "jiface.hpp"
+#include "jptree.hpp"
+#include "jstring.hpp"
+
+#include "dadfs.hpp"
+#include "dalienv.hpp"
+
+#include "saserver.hpp"
+#include "sautil.hpp"
+
+#define DEFAULT_HOUSEKEEPING_INTERVAL_HOURS 24
+
+#define LOGDBGHK "DEBUGPLANEHOUSEKEEPING: "
+
+// Debug Housekeeping monitor
+
+class CSashaDebugPlaneHousekeepingServer : public ISashaServer, public Thread
+{
+    bool stopped{false};
+    Semaphore stopsem;
+    Mutex runmutex;
+    StringBuffer debugDir;
+    unsigned expiryDays{0};
+
+public:
+    IMPLEMENT_IINTERFACE_USING(Thread);
+
+    CSashaDebugPlaneHousekeepingServer()
+        : Thread("CSashaDebugPlaneHousekeepingServer")
+    {
+    }
+
+    virtual void start() override
+    {
+        Thread::start(false);
+
+        expiryDays = getComponentConfig()->getPropInt("@expiryDays");
+
+        // get debug plane dir
+        StringBuffer planeName;
+        if (!getDefaultPlane(planeName, "@debugPlane", "debug"))
+        {
+            WARNLOG("Exception handlers configured, but debug plane is missing");
+            return;
+        }
+        Owned<IPropertyTree> plane = getStoragePlane(planeName);
+        assertex(plane);
+        verifyex(plane->getProp("@prefix", debugDir));
+    }
+
+    virtual void ready() override
+    {
+    }
+
+    virtual void stop() override
+    {
+        if (!stopped)
+        {
+            stopped = true;
+            stopsem.signal();
+        }
+        synchronized block(runmutex); // hopefully stopped should stop
+        if (!join(1000 * 60 * 3))
+            OERRLOG("CSashaDebugPlaneHousekeepingServer aborted");
+    }
+
+    void runDebugHousekeeping()
+    {
+        synchronized block(runmutex);
+        if (stopped)
+            return;
+
+        // iterate debug plane selecting post-mortem directories for housekeeping
+        Owned<IDirectoryIterator> pDirIter = createDirectoryIterator(debugDir.str(), "*", false, true);
+        ForEach(*pDirIter)
+        {
+            if (stopped)
+                break;
+
+            IFile &iFile = pDirIter->query();
+            const char *filePath = iFile.queryFilename();
+
+            // Process directories, exclude the ".", "..", non post-mortem and not expired post-mortem directories
+            if (iFile.isDirectory() == fileBool::foundYes)
+            {
+                if (streq(filePath, ".") || streq(filePath, ".."))
+                    continue;
+
+                if (isExpiredModifiedDateTime(filePath))
+                {
+                    recursiveRemoveDirectory(filePath);
+                }
+            }
+            else
+            {
+                if (isExpiredModifiedDateTime(filePath))
+                {
+                    iFile.remove();
+                }
+            }
+        }
+        pDirIter.clear();
+    }
+
+    virtual int run() override
+    {
+        CSashaSchedule schedule;
+        unsigned interval = getComponentConfig()->getPropInt("@interval", DEFAULT_HOUSEKEEPING_INTERVAL_HOURS);
+        if (interval == 0)
+        {
+            stopped = true;
+            return 0;
+        }
+        schedule.init(getComponentConfig(), interval);
+        while (!stopped)
+        {
+            stopsem.wait(1000 * 60);
+            if (stopped)
+                break;
+            if (!schedule.ready())
+                continue;
+            try
+            {
+                runDebugHousekeeping();
+            }
+            catch (IException *e)
+            {
+                StringBuffer s;
+                EXCLOG(e, LOGDBGHK);
+                e->Release();
+            }
+        }
+        return 0;
+    }
+
+private:
+    bool isExpiredModifiedDateTime(const char *filePath)
+    {
+        Owned<IFile> file = createIFile(filePath);
path: dali/sasha/sadbghk.cpp
user: login: jakesmith
body: minor: see other comments, can pass in the IFile that the iterator is returning instead of reconstructing one.
created_at: 2025-03-13T18:57:51Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,164 @@
+#include "platform.h"
+
+#include "jlib.hpp"
+#include "jiface.hpp"
+#include "jptree.hpp"
+#include "jstring.hpp"
+
+#include "dadfs.hpp"
+#include "dalienv.hpp"
+
+#include "saserver.hpp"
+#include "sautil.hpp"
+
+#define DEFAULT_HOUSEKEEPING_INTERVAL_HOURS 24
+
+#define LOGDBGHK "DEBUGPLANEHOUSEKEEPING: "
+
+// Debug Housekeeping monitor
+
+class CSashaDebugPlaneHousekeepingServer : public ISashaServer, public Thread
+{
+    bool stopped{false};
+    Semaphore stopsem;
+    Mutex runmutex;
+    StringBuffer debugDir;
+    unsigned expiryDays{0};
+
+public:
+    IMPLEMENT_IINTERFACE_USING(Thread);
+
+    CSashaDebugPlaneHousekeepingServer()
+        : Thread("CSashaDebugPlaneHousekeepingServer")
+    {
+    }
+
+    virtual void start() override
+    {
+        Thread::start(false);
+
+        expiryDays = getComponentConfig()->getPropInt("@expiryDays");
+
+        // get debug plane dir
+        StringBuffer planeName;
+        if (!getDefaultPlane(planeName, "@debugPlane", "debug"))
+        {
+            WARNLOG("Exception handlers configured, but debug plane is missing");
+            return;
+        }
+        Owned<IPropertyTree> plane = getStoragePlane(planeName);
+        assertex(plane);
+        verifyex(plane->getProp("@prefix", debugDir));
+    }
+
+    virtual void ready() override
+    {
+    }
+
+    virtual void stop() override
+    {
+        if (!stopped)
+        {
+            stopped = true;
+            stopsem.signal();
+        }
+        synchronized block(runmutex); // hopefully stopped should stop
+        if (!join(1000 * 60 * 3))
+            OERRLOG("CSashaDebugPlaneHousekeepingServer aborted");
+    }
+
+    void runDebugHousekeeping()
+    {
+        synchronized block(runmutex);
+        if (stopped)
+            return;
+
+        // iterate debug plane selecting post-mortem directories for housekeeping
+        Owned<IDirectoryIterator> pDirIter = createDirectoryIterator(debugDir.str(), "*", false, true);
+        ForEach(*pDirIter)
+        {
+            if (stopped)
+                break;
+
+            IFile &iFile = pDirIter->query();
+            const char *filePath = iFile.queryFilename();
+
+            // Process directories, exclude the ".", "..", non post-mortem and not expired post-mortem directories
+            if (iFile.isDirectory() == fileBool::foundYes)
+            {
+                if (streq(filePath, ".") || streq(filePath, ".."))
+                    continue;
+
+                if (isExpiredModifiedDateTime(filePath))
+                {
+                    recursiveRemoveDirectory(filePath);
+                }
+            }
+            else
+            {
+                if (isExpiredModifiedDateTime(filePath))
+                {
+                    iFile.remove();
+                }
+            }
+        }
+        pDirIter.clear();
+    }
+
+    virtual int run() override
+    {
+        CSashaSchedule schedule;
+        unsigned interval = getComponentConfig()->getPropInt("@interval", DEFAULT_HOUSEKEEPING_INTERVAL_HOURS);
+        if (interval == 0)
+        {
+            stopped = true;
+            return 0;
+        }
+        schedule.init(getComponentConfig(), interval);
+        while (!stopped)
+        {
+            stopsem.wait(1000 * 60);
+            if (stopped)
+                break;
+            if (!schedule.ready())
+                continue;
+            try
+            {
+                runDebugHousekeeping();
+            }
+            catch (IException *e)
+            {
+                StringBuffer s;
+                EXCLOG(e, LOGDBGHK);
+                e->Release();
+            }
+        }
+        return 0;
+    }
+
+private:
+    bool isExpiredModifiedDateTime(const char *filePath)
+    {
+        Owned<IFile> file = createIFile(filePath);
+        CDateTime expiredModifiedTime;
+        file->getTime(nullptr, &expiredModifiedTime, nullptr);
path: dali/sasha/sadbghk.cpp
user: login: jakesmith
body: probably should error check getTime and return false if errors (and issue a warning)
created_at: 2025-03-13T18:58:54Z
author_association: MEMBER
diff_hunk: @@ -3045,6 +3045,31 @@
       },
       "additionalProperties": false
     },
+    "sasha-debugplane-housekeeping": {
+      "type": "object",
+      "allOf": [{ "$ref": "#/definitions/sashacommon" }],
+      "properties": {
+        "expiryDays": {
+          "type": "integer",
+          "description": "Default number of days passed that post-mortem files are deleted after their associated workunit was completed",
path: helm/hpcc/values.schema.json
user: login: jakesmith
body: 'after their associated workunit was completed' doesn't really make sense.
created_at: 2025-03-13T19:01:28Z
author_association: MEMBER
diff_hunk: @@ -3045,6 +3045,31 @@
       },
       "additionalProperties": false
     },
+    "sasha-debugplane-housekeeping": {
+      "type": "object",
+      "allOf": [{ "$ref": "#/definitions/sashacommon" }],
+      "properties": {
+        "expiryDays": {
+          "type": "integer",
+          "description": "Default number of days passed that post-mortem files are deleted after their associated workunit was completed",
+          "default": "14"
+        },
+        "disabled": {},
+        "interval": {},
+        "service": {},
path: helm/hpcc/values.schema.json
user: login: jakesmith
body: I can't remember how these work exactly, I think they're necessary if you want to inherit them in the schema,
but "service" we don't want as part of this sasha component.
Others that are not relevant:
- throttle
- plane
- egress
created_at: 2025-03-13T19:09:27Z
author_association: MEMBER
diff_hunk: @@ -513,6 +513,12 @@ sasha:
     #switchMinTime: 1
     #queues: "*"
 
+  debugplane-housekeeping: {} # NB: no properties defined, use defaults
+    #disabled: true
+    #interval: 1
path: helm/hpcc/values.yaml
user: login: jakesmith
body: let's make this 24 - in keeping with the default in the code (and in other example/test yamls)
created_at: 2025-03-13T19:10:14Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,164 @@
+#include "platform.h"
+
+#include "jlib.hpp"
+#include "jiface.hpp"
+#include "jptree.hpp"
+#include "jstring.hpp"
+
+#include "dadfs.hpp"
+#include "dalienv.hpp"
path: dali/sasha/sadbghk.cpp
user: login: jakesmith
body: these ^^ two aren't needed I think.
created_at: 2025-03-13T19:11:17Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,164 @@
+#include "platform.h"
+
+#include "jlib.hpp"
+#include "jiface.hpp"
+#include "jptree.hpp"
+#include "jstring.hpp"
+
+#include "dadfs.hpp"
+#include "dalienv.hpp"
+
+#include "saserver.hpp"
+#include "sautil.hpp"
+
+#define DEFAULT_HOUSEKEEPING_INTERVAL_HOURS 24
+
+#define LOGDBGHK "DEBUGPLANEHOUSEKEEPING: "
+
+// Debug Housekeeping monitor
+
+class CSashaDebugPlaneHousekeepingServer : public ISashaServer, public Thread
+{
+    bool stopped{false};
+    Semaphore stopsem;
+    Mutex runmutex;
path: dali/sasha/sadbghk.cpp
user: login: jakesmith
body: super picky: camelCase prefered (for stopSem and runMutex)
created_at: 2025-03-13T19:12:00Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,164 @@
+#include "platform.h"
+
+#include "jlib.hpp"
+#include "jiface.hpp"
+#include "jptree.hpp"
+#include "jstring.hpp"
+
+#include "dadfs.hpp"
+#include "dalienv.hpp"
+
+#include "saserver.hpp"
+#include "sautil.hpp"
+
+#define DEFAULT_HOUSEKEEPING_INTERVAL_HOURS 24
+
+#define LOGDBGHK "DEBUGPLANEHOUSEKEEPING: "
+
+// Debug Housekeeping monitor
+
+class CSashaDebugPlaneHousekeepingServer : public ISashaServer, public Thread
+{
+    bool stopped{false};
path: dali/sasha/sadbghk.cpp
user: login: jakesmith
body: should really be an atomic
created_at: 2025-03-13T19:13:13Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,164 @@
+#include "platform.h"
+
+#include "jlib.hpp"
+#include "jiface.hpp"
+#include "jptree.hpp"
+#include "jstring.hpp"
+
+#include "dadfs.hpp"
+#include "dalienv.hpp"
+
+#include "saserver.hpp"
+#include "sautil.hpp"
+
+#define DEFAULT_HOUSEKEEPING_INTERVAL_HOURS 24
+
+#define LOGDBGHK "DEBUGPLANEHOUSEKEEPING: "
+
+// Debug Housekeeping monitor
+
+class CSashaDebugPlaneHousekeepingServer : public ISashaServer, public Thread
+{
+    bool stopped{false};
+    Semaphore stopsem;
+    Mutex runmutex;
+    StringBuffer debugDir;
+    unsigned expiryDays{0};
+
+public:
+    IMPLEMENT_IINTERFACE_USING(Thread);
+
+    CSashaDebugPlaneHousekeepingServer()
+        : Thread("CSashaDebugPlaneHousekeepingServer")
+    {
+    }
+
+    virtual void start() override
+    {
+        Thread::start(false);
+
+        expiryDays = getComponentConfig()->getPropInt("@expiryDays");
+
+        // get debug plane dir
+        StringBuffer planeName;
+        if (!getDefaultPlane(planeName, "@debugPlane", "debug"))
+        {
+            WARNLOG("Exception handlers configured, but debug plane is missing");
+            return;
+        }
+        Owned<IPropertyTree> plane = getStoragePlane(planeName);
+        assertex(plane);
+        verifyex(plane->getProp("@prefix", debugDir));
+    }
+
+    virtual void ready() override
+    {
+    }
+
+    virtual void stop() override
+    {
+        if (!stopped)
+        {
+            stopped = true;
+            stopsem.signal();
+        }
+        synchronized block(runmutex); // hopefully stopped should stop
+        if (!join(1000 * 60 * 3))
+            OERRLOG("CSashaDebugPlaneHousekeepingServer aborted");
+    }
+
+    void runDebugHousekeeping()
+    {
+        synchronized block(runmutex);
+        if (stopped)
+            return;
+
+        // iterate debug plane selecting post-mortem directories for housekeeping
+        Owned<IDirectoryIterator> pDirIter = createDirectoryIterator(debugDir.str(), "*", false, true);
+        ForEach(*pDirIter)
+        {
+            if (stopped)
+                break;
+
+            IFile &iFile = pDirIter->query();
+            const char *filePath = iFile.queryFilename();
+
+            // Process directories, exclude the ".", "..", non post-mortem and not expired post-mortem directories
+            if (iFile.isDirectory() == fileBool::foundYes)
+            {
+                if (streq(filePath, ".") || streq(filePath, ".."))
+                    continue;
+
+                if (isExpiredModifiedDateTime(filePath))
+                {
+                    recursiveRemoveDirectory(filePath);
+                }
+            }
+            else
+            {
+                if (isExpiredModifiedDateTime(filePath))
+                {
+                    iFile.remove();
+                }
+            }
+        }
+        pDirIter.clear();
+    }
+
+    virtual int run() override
+    {
+        CSashaSchedule schedule;
+        unsigned interval = getComponentConfig()->getPropInt("@interval", DEFAULT_HOUSEKEEPING_INTERVAL_HOURS);
+        if (interval == 0)
+        {
+            stopped = true;
+            return 0;
+        }
+        schedule.init(getComponentConfig(), interval);
+        while (!stopped)
+        {
+            stopsem.wait(1000 * 60);
+            if (stopped)
+                break;
+            if (!schedule.ready())
+                continue;
+            try
+            {
+                runDebugHousekeeping();
+            }
+            catch (IException *e)
+            {
+                StringBuffer s;
+                EXCLOG(e, LOGDBGHK);
+                e->Release();
+            }
+        }
+        return 0;
+    }
+
+private:
+    bool isExpiredModifiedDateTime(const char *filePath)
+    {
+        Owned<IFile> file = createIFile(filePath);
+        CDateTime expiredModifiedTime;
+        file->getTime(nullptr, &expiredModifiedTime, nullptr);
+        expiredModifiedTime.adjustTime(60 * 24 * expiryDays);
+
+        CDateTime now;
+        now.setNow();
+
+        if (now.compare(expiredModifiedTime, false) > 0)
+            return true;
+        else
+            return false;
+    }
+
+} *sashaDebugPlaneHousekeepingServer = NULL;
path: dali/sasha/sadbghk.cpp
user: login: jakesmith
body: picky!: nullptr preferred
created_at: 2025-03-13T19:14:40Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,7 @@
+#ifndef SADBGHKIF_HPP
+#define SADBGHKIF_HPP
path: dali/sasha/sadbghk.hpp
user: login: jakesmith
body: trivial: I know lots of code uses this approach, but we should use this instead:
```
#pragma once
```
created_at: 2025-03-13T19:15:54Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,164 @@
+#include "platform.h"
+
+#include "jlib.hpp"
+#include "jiface.hpp"
+#include "jptree.hpp"
+#include "jstring.hpp"
+
+#include "dadfs.hpp"
+#include "dalienv.hpp"
path: dali/sasha/sadbghk.cpp
user: login: streeterd
body: dadfs.hpp is needed for IFile and IDirectoryIterator
dalienv.hpp is needed for the LOGs
created_at: 2025-03-14T08:54:20Z
author_association: CONTRIBUTOR
diff_hunk: @@ -137,11 +137,15 @@ class CSashaDebugPlaneHousekeepingServer : public ISashaServer, public Thread
     }
 
 private:
-    bool isExpiredModifiedDateTime(const char *filePath)
+    bool hasExpired(IFile &iFile)
     {
-        Owned<IFile> file = createIFile(filePath);
         CDateTime expiredModifiedTime;
-        file->getTime(nullptr, &expiredModifiedTime, nullptr);
+        iFile.getTime(nullptr, &expiredModifiedTime, nullptr);
+        if (expiredModifiedTime.isNull())
path: dali/sasha/sadbghk.cpp
user: login: jakesmith
body: was there a reason not to use boolean result of getTime here instead?
created_at: 2025-03-17T14:52:00Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,164 @@
+#include "platform.h"
+
+#include "jlib.hpp"
+#include "jiface.hpp"
+#include "jptree.hpp"
+#include "jstring.hpp"
+
+#include "dadfs.hpp"
+#include "dalienv.hpp"
path: dali/sasha/sadbghk.cpp
user: login: jakesmith
body: that's not where those things are defined.
created_at: 2025-03-17T15:00:42Z
author_association: MEMBER
diff_hunk: @@ -83,41 +83,42 @@ class CSashaDebugPlaneHousekeepingServer : public ISashaServer, public Thread
             IFile &iFile = pDirIter->query();
             const char *filePath = iFile.queryFilename();
 
-            // Process directories, exclude the ".", "..", non post-mortem and not expired post-mortem directories
             if (iFile.isDirectory() == fileBool::foundYes)
             {
+                // Process directories, exclude the ".", "..", and non expired directories from housekeeping
                 if (streq(filePath, ".") || streq(filePath, ".."))
-                    continue;
+                        continue;
path: dali/sasha/sadbghk.cpp
user: login: jakesmith
body: extra indentation.
created_at: 2025-03-17T15:45:42Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,168 @@
+#include "platform.h"
+
+#include "jlib.hpp"
+#include "jiface.hpp"
+#include "jptree.hpp"
+#include "jstring.hpp"
+
+#include "dadfs.hpp"
+#include "dalienv.hpp"
+
+#include "saserver.hpp"
+#include "sautil.hpp"
+
+static constexpr unsigned defaultHouseKeepingIntervalHours = 24;
+
+#define LOGDBGHK "DEBUGPLANEHOUSEKEEPING: "
+
+// Debug Housekeeping monitor
+
+class CSashaDebugPlaneHousekeepingServer : public ISashaServer, public Thread
+{
+    std::atomic<bool> stopped{false};
+    Semaphore stopSem;
+    Mutex runMutex;
+    StringBuffer debugDir;
+    unsigned expiryDays{0};
path: dali/sasha/sadbghk.cpp
user: login: jakesmith
body: let's add a static constexpr unsigned defaultExpiryDays = 14; and use here and on line 40 too.
And in start (before Thread::start) check that if set to 0, then bails out (doesn't start).

In fact the "Failed to get default debug plane" bail out should before the Thread::start(false) too.
Move the Thread::start(false) to the end of the function.
created_at: 2025-03-17T15:51:08Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,168 @@
+#include "platform.h"
+
+#include "jlib.hpp"
+#include "jiface.hpp"
+#include "jptree.hpp"
+#include "jstring.hpp"
+
+#include "dadfs.hpp"
+#include "dalienv.hpp"
+
+#include "saserver.hpp"
+#include "sautil.hpp"
+
+static constexpr unsigned defaultHouseKeepingIntervalHours = 24;
+
+#define LOGDBGHK "DEBUGPLANEHOUSEKEEPING: "
+
+// Debug Housekeeping monitor
+
+class CSashaDebugPlaneHousekeepingServer : public ISashaServer, public Thread
+{
+    std::atomic<bool> stopped{false};
+    Semaphore stopSem;
+    Mutex runMutex;
+    StringBuffer debugDir;
+    unsigned expiryDays{0};
+
+public:
+    IMPLEMENT_IINTERFACE_USING(Thread);
+
+    CSashaDebugPlaneHousekeepingServer()
+        : Thread("CSashaDebugPlaneHousekeepingServer")
+    {
+    }
+
+    virtual void start() override
+    {
+        Thread::start(false);
+
+        expiryDays = getComponentConfigSP()->getPropInt("@expiryDays");
+
+        // get debug plane dir
+        StringBuffer planeName;
+        if (!getDefaultPlane(planeName, "@debugPlane", "debug"))
+        {
+            WARNLOG("Failed to get default debug plane");
+            return;
+        }
+        Owned<IPropertyTree> plane = getStoragePlane(planeName);
+        assertex(plane);
+        verifyex(plane->getProp("@prefix", debugDir));
+    }
+
+    virtual void ready() override
+    {
+    }
+
+    virtual void stop() override
+    {
+        if (!stopped)
+        {
+            stopped = true;
+            stopSem.signal();
+        }
+        synchronized block(runMutex); // hopefully stopped should stop
+        if (!join(1000 * 60 * 3))
+            OERRLOG("CSashaDebugPlaneHousekeepingServer aborted");
+    }
+
+    void runDebugHousekeeping()
+    {
+        synchronized block(runMutex);
+        if (stopped)
+            return;
+
+        // iterate debug plane selecting files and directories for housekeeping
+        Owned<IDirectoryIterator> pDirIter = createDirectoryIterator(debugDir.str(), "*", false, true);
+        ForEach(*pDirIter)
+        {
+            if (stopped)
+                break;
+
+            IFile &iFile = pDirIter->query();
+            const char *filePath = iFile.queryFilename();
+
+            if (iFile.isDirectory() == fileBool::foundYes)
+            {
+                // Process directories, exclude the ".", "..", and non expired directories from housekeeping
+                if (streq(filePath, ".") || streq(filePath, ".."))
+                        continue;
+
+                if (hasExpired(iFile))
+                {
path: dali/sasha/sadbghk.cpp
user: login: jakesmith
body: style/personal: don't normally add block curlies for single line blocks.
created_at: 2025-03-17T15:52:40Z
author_association: MEMBER
diff_hunk: @@ -513,6 +513,12 @@ sasha:
     #switchMinTime: 1
     #queues: "*"
 
+  debugplane-housekeeping: {} # NB: no properties defined, use defaults
+    #disabled: true
+    #interval: 24
+    #at: "* 3 * * *"
+    #expiryDays: 7
path: helm/hpcc/values.yaml
user: login: jakesmith
body: for consistency, make this 14, i.e. so same as stated default in schema.
created_at: 2025-03-17T15:54:12Z
author_association: MEMBER
diff_hunk: @@ -510,6 +510,15 @@ sasha:
     resources:
       cpu: "1"
       memory: "4Gi"
+      
+  debugplane-housekeeping:
+    #disabled: true
+    #interval: 1
+    #at: "* 3 * * *"
+    #expiryDays: 7
path: testing/helm/tests/resourced.yaml
user: login: jakesmith
body: for consistency, make this 14, i.e. so same as stated default in schema.
created_at: 2025-03-17T15:54:24Z
author_association: MEMBER
diff_hunk: @@ -512,6 +512,15 @@ sasha:
     resources:
       cpu: "1"
       memory: "4Gi"
+      
+  debugplane-housekeeping:
+    #disabled: true
+    #interval: 1
+    #at: "* 3 * * *"
+    #expiryDays: 7
path: testing/helm/tests/resourced2.yaml
user: login: jakesmith
body: for consistency, make this 14, i.e. so same as stated default in schema.
created_at: 2025-03-17T15:54:29Z
author_association: MEMBER
diff_hunk: @@ -137,11 +137,15 @@ class CSashaDebugPlaneHousekeepingServer : public ISashaServer, public Thread
     }
 
 private:
-    bool isExpiredModifiedDateTime(const char *filePath)
+    bool hasExpired(IFile &iFile)
     {
-        Owned<IFile> file = createIFile(filePath);
         CDateTime expiredModifiedTime;
-        file->getTime(nullptr, &expiredModifiedTime, nullptr);
+        iFile.getTime(nullptr, &expiredModifiedTime, nullptr);
+        if (expiredModifiedTime.isNull())
path: dali/sasha/sadbghk.cpp
user: login: streeterd
body: Not that I can remember, I will use the return of `getTime`
created_at: 2025-03-18T09:17:23Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,66 @@
+# Sasha
+
+## Services
+
+### Debug Plane Housekeeping Server
+
+#### Overview
+
+- This is a maintenance service that automatically manages files and directories on the debug plane using a configured retention period.
+- This is required to control the disc space used by post-mortem and other files on the debug plane.
+- This service is designed to execute on a schedule with no interaction other; a single log entry for a summary of any deleted files or directories will be generated.
+
+#### Installation and Configuration
+
+- The service is installed and active by default.
+- It is configuration-driven with fallback to defaults in the configuration yaml files.
+  - For example `values.yaml` :
+
+    |attribute|description|values|default|required|
+    |:--------|:----------|:-----|:------|:-------|
+    |disabled|Disable or enable this service.|false|false|optional|
+    |interval|Minimum interval between debug plane housekeeping expiry checking (in hours, 0 disables).|24||optional|
+    |at|Schedule to check for debug plane expired files and directories (cron format).|* 3 * * *||optional|
+    |expiryDays|Default number of days passed that debug plane files or directories are deleted.|14|14|optional|
+
+#### User Guide
+
+##### How it works in steps
+
+1. The `disabled` value is read from the component configuration
+2. If `disabled` is `true` then the service is not started.
+3. If the service is being run in the bare metal environment then the service is stopped.
path: dali/sasha/sasha.md
user: login: jakesmith
body: Should remove this line. The service is never created in BM
created_at: 2025-03-20T09:03:41Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,66 @@
+# Sasha
+
+## Services
+
+### Debug Plane Housekeeping Server
+
+#### Overview
+
+- This is a maintenance service that automatically manages files and directories on the debug plane using a configured retention period.
+- This is required to control the disc space used by post-mortem and other files on the debug plane.
+- This service is designed to execute on a schedule with no interaction other; a single log entry for a summary of any deleted files or directories will be generated.
+
+#### Installation and Configuration
+
+- The service is installed and active by default.
+- It is configuration-driven with fallback to defaults in the configuration yaml files.
+  - For example `values.yaml` :
+
+    |attribute|description|values|default|required|
+    |:--------|:----------|:-----|:------|:-------|
+    |disabled|Disable or enable this service.|false|false|optional|
+    |interval|Minimum interval between debug plane housekeeping expiry checking (in hours, 0 disables).|24||optional|
+    |at|Schedule to check for debug plane expired files and directories (cron format).|* 3 * * *||optional|
+    |expiryDays|Default number of days passed that debug plane files or directories are deleted.|14|14|optional|
+
+#### User Guide
+
+##### How it works in steps
+
+1. The `disabled` value is read from the component configuration
+2. If `disabled` is `true` then the service is not started.
+3. If the service is being run in the bare metal environment then the service is stopped.
+4. The `expiryDays` value is read from the component configuration.
+5. The debug plane location is read from the component configuration.
+6. The service `Thread` is started.
+7. The schedule `at` is read from the component configuration.
+8. The `interval` is read from the component configuration.
path: dali/sasha/sasha.md
user: login: jakesmith
body: Step 4, 7 and 8 are a bit OTT to say config option is read from config file..
created_at: 2025-03-20T09:05:00Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,66 @@
+# Sasha
+
+## Services
+
+### Debug Plane Housekeeping Server
+
+#### Overview
+
+- This is a maintenance service that automatically manages files and directories on the debug plane using a configured retention period.
+- This is required to control the disc space used by post-mortem and other files on the debug plane.
+- This service is designed to execute on a schedule with no interaction other; a single log entry for a summary of any deleted files or directories will be generated.
+
+#### Installation and Configuration
+
+- The service is installed and active by default.
+- It is configuration-driven with fallback to defaults in the configuration yaml files.
+  - For example `values.yaml` :
+
+    |attribute|description|values|default|required|
+    |:--------|:----------|:-----|:------|:-------|
+    |disabled|Disable or enable this service.|false|false|optional|
+    |interval|Minimum interval between debug plane housekeeping expiry checking (in hours, 0 disables).|24||optional|
+    |at|Schedule to check for debug plane expired files and directories (cron format).|* 3 * * *||optional|
+    |expiryDays|Default number of days passed that debug plane files or directories are deleted.|14|14|optional|
+
+#### User Guide
+
+##### How it works in steps
+
+1. The `disabled` value is read from the component configuration
+2. If `disabled` is `true` then the service is not started.
+3. If the service is being run in the bare metal environment then the service is stopped.
+4. The `expiryDays` value is read from the component configuration.
+5. The debug plane location is read from the component configuration.
+6. The service `Thread` is started.
+7. The schedule `at` is read from the component configuration.
+8. The `interval` is read from the component configuration.
+9. If the `interval` is 0 then the service is stopped.
path: dali/sasha/sasha.md
user: login: jakesmith
body: really, should check in start() and not start in the 1st place.
created_at: 2025-03-20T09:06:43Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,180 @@
+#include "platform.h"
+
+#include "jfile.hpp"
+#include "jlib.hpp"
+#include "jlog.hpp"
+#include "jiface.hpp"
+#include "jptree.hpp"
+#include "jstring.hpp"
+
+#include "saserver.hpp"
+#include "sautil.hpp"
+
+static constexpr unsigned defaultExpiryDays = 14;
+static constexpr unsigned defaultHouseKeepingIntervalHours = 24;
+
+#define LOGDBGHK "DEBUGPLANEHOUSEKEEPING: "
+
+// Debug Housekeeping monitor
+
+class CSashaDebugPlaneHousekeepingServer : public ISashaServer, public Thread
+{
+    std::atomic<bool> stopped{false};
+    Semaphore stopSem;
+    Mutex runMutex;
+    StringBuffer debugDir;
+    unsigned expiryDays{0};
+
+public:
+    IMPLEMENT_IINTERFACE_USING(Thread);
+
+    CSashaDebugPlaneHousekeepingServer()
+        : Thread("CSashaDebugPlaneHousekeepingServer")
+    {
+    }
+
+    virtual void start() override
+    {
+        expiryDays = getComponentConfigSP()->getPropInt("@expiryDays", defaultExpiryDays);
+        if (expiryDays == 0)
+        {
+            WARNLOG("Failed to get expiry days for debug plane");
path: dali/sasha/sadbghk.cpp
user: login: jakesmith
body: Not "failed to get" - it had to be deliberately configured as 0 for this to be true.
I would treat as disabled and exit without starting.

In fact a better pattern is to set 'stopped' to true in member initialization,
and set to false just before the Thread::start() call.

It would also be good to check 'interval' here for 0, and not start, rather than check it once started and then stop.

created_at: 2025-03-20T09:11:05Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,180 @@
+#include "platform.h"
+
+#include "jfile.hpp"
+#include "jlib.hpp"
+#include "jlog.hpp"
+#include "jiface.hpp"
+#include "jptree.hpp"
+#include "jstring.hpp"
+
+#include "saserver.hpp"
+#include "sautil.hpp"
+
+static constexpr unsigned defaultExpiryDays = 14;
+static constexpr unsigned defaultHouseKeepingIntervalHours = 24;
+
+#define LOGDBGHK "DEBUGPLANEHOUSEKEEPING: "
+
+// Debug Housekeeping monitor
+
+class CSashaDebugPlaneHousekeepingServer : public ISashaServer, public Thread
+{
+    std::atomic<bool> stopped{false};
+    Semaphore stopSem;
+    Mutex runMutex;
+    StringBuffer debugDir;
+    unsigned expiryDays{0};
+
+public:
+    IMPLEMENT_IINTERFACE_USING(Thread);
+
+    CSashaDebugPlaneHousekeepingServer()
+        : Thread("CSashaDebugPlaneHousekeepingServer")
+    {
+    }
+
+    virtual void start() override
+    {
+        expiryDays = getComponentConfigSP()->getPropInt("@expiryDays", defaultExpiryDays);
+        if (expiryDays == 0)
+        {
+            WARNLOG("Failed to get expiry days for debug plane");
+            return;
+        }
+
+        // get debug plane dir
+        StringBuffer planeName;
+        if (!getDefaultPlane(planeName, "@debugPlane", "debug"))
+        {
+            WARNLOG("Failed to get default debug plane");
+            return;
+        }
+        Owned<IPropertyTree> plane = getStoragePlane(planeName);
+        assertex(plane);
+        verifyex(plane->getProp("@prefix", debugDir));
+
+        Thread::start(false);
+    }
+
+    virtual void ready() override
+    {
+    }
+
+    virtual void stop() override
+    {
+        if (!stopped)
path: dali/sasha/sadbghk.cpp
user: login: jakesmith
body: I know this is c&p from other code, but if already stopped, why continue..
I think this should be:
```
if (stopped)
    return;
stopped = true;
stopSem.signal();
....
```
created_at: 2025-03-20T09:13:04Z
author_association: MEMBER
diff_hunk: @@ -35,12 +36,9 @@ class CSashaDebugPlaneHousekeepingServer : public ISashaServer, public Thread
 
     virtual void start() override
     {
-        expiryDays = getComponentConfigSP()->getPropInt("@expiryDays", defaultExpiryDays);
-        if (expiryDays == 0)
-        {
-            WARNLOG("Failed to get expiry days for debug plane");
-            return;
-        }
+        Owned<IPropertyTree> compConfig = getComponentConfig();
+        expiryDays = compConfig->getPropInt("@expiryDays", defaultExpiryDays);
+        interval = compConfig->getPropInt("@interval", defaultHouseKeepingIntervalHours);
path: dali/sasha/sadbghk.cpp
user: login: jakesmith
body: minor: cleaner to check these here and exit (return), and then unconditionally set stopped to false, just before thread starts.
created_at: 2025-03-20T11:46:11Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,176 @@
+#include "platform.h"
+
+#include "jfile.hpp"
+#include "jlib.hpp"
+#include "jlog.hpp"
+#include "jiface.hpp"
+#include "jptree.hpp"
+#include "jstring.hpp"
+
+#include "saserver.hpp"
+#include "sautil.hpp"
+
+static constexpr unsigned defaultExpiryDays = 14;
+static constexpr unsigned defaultHouseKeepingIntervalHours = 24;
+
+#define LOGDBGHK "DEBUGPLANEHOUSEKEEPING: "
+
+// Debug Housekeeping monitor
+
+class CSashaDebugPlaneHousekeepingServer : public ISashaServer, public Thread
+{
+    std::atomic<bool> stopped{true};
+    Semaphore stopSem;
+    Mutex runMutex;
+    StringBuffer debugDir;
+    unsigned expiryDays{0};
+    unsigned interval{0};
+
+public:
+    IMPLEMENT_IINTERFACE_USING(Thread);
+
+    CSashaDebugPlaneHousekeepingServer()
+        : Thread("CSashaDebugPlaneHousekeepingServer")
+    {
+    }
+
+    virtual void start() override
+    {
+        Owned<IPropertyTree> compConfig = getComponentConfig();
+        expiryDays = compConfig->getPropInt("@expiryDays", defaultExpiryDays);
+        if (expiryDays == 0)
+            return;
+        interval = compConfig->getPropInt("@interval", defaultHouseKeepingIntervalHours);
+        if (interval == 0)
+            return;
+
+        // get debug plane dir
+        StringBuffer planeName;
+        if (!getDefaultPlane(planeName, "@debugPlane", "debug"))
+        {
+            WARNLOG("Failed to get default debug plane");
+            return;
+        }
+        Owned<IPropertyTree> plane = getStoragePlane(planeName);
+        assertex(plane);
+        verifyex(plane->getProp("@prefix", debugDir));
+
+        stopped = false;
+        Thread::start(false);
+    }
+
+    virtual void ready() override
+    {
+    }
+
+    virtual void stop() override
+    {
+        if (stopped)
+            return;
+
+        stopped = true;
+        stopSem.signal();
+        synchronized block(runMutex); // hopefully stopped should stop
+        if (!join(1000 * 60 * 3))
+            OERRLOG("CSashaDebugPlaneHousekeepingServer aborted");
+    }
+
+    void runDebugHousekeeping()
+    {
+        synchronized block(runMutex);
+        if (stopped)
+            return;
+
+        unsigned dirsDeleted{0};
+        unsigned filesDeleted{0};
+
+        // iterate debug plane selecting files and directories for housekeeping
+        Owned<IDirectoryIterator> pDirIter = createDirectoryIterator(debugDir.str(), "*", false, true);
+        ForEach(*pDirIter)
+        {
+            if (stopped)
+                break;
+
+            IFile &iFile = pDirIter->query();
+            const char *filePath = iFile.queryFilename();
+
+            if (iFile.isDirectory() == fileBool::foundYes)
+            {
+                // Process directories, exclude the ".", "..", and non expired directories from housekeeping
path: dali/sasha/sadbghk.cpp
user: login: jakesmith
body: trivial: comment out of date? I think these lines used to check expired to, now done via hasExpired which is self-explanatory
created_at: 2025-04-03T15:29:36Z
author_association: MEMBER
diff_hunk: @@ -238,6 +240,29 @@ class CSashaCmdThread : public CInterface, implements IPooledThread
 
 void SashaMain()
 {
+    if (!isDaliClient)
+    {
+        bool stopped = false;
path: dali/sasha/saserver.cpp
user: login: jakesmith
body: should really be an atomic.
created_at: 2025-04-03T16:06:00Z
author_association: MEMBER
diff_hunk: @@ -238,6 +240,29 @@ class CSashaCmdThread : public CInterface, implements IPooledThread
 
 void SashaMain()
 {
+    if (!isDaliClient)
+    {
+        bool stopped = false;
+        class AbortHandler : implements CSimpleInterfaceOf<IAbortHandler>
+        {
+            bool stopped = false;
+        public:
+            AbortHandler(bool &_stopped) : stopped(_stopped) {}
+            virtual bool onAbort() override
+            {
+                PROGLOG("Aborted");
+                stopped = true;
path: dali/sasha/saserver.cpp
user: login: jakesmith
body: This isn't going to affect the 'stopped' outside of the class, so it won't cause it to stop at the moment.
this.stopped needs to be a reference to the stopped passed in.
created_at: 2025-04-03T16:08:27Z
author_association: MEMBER
diff_hunk: @@ -469,8 +498,11 @@ int main(int argc, const char* argv[])
                         if (!stopped)
                         {
                             stopped = true;
-                            IInterCommunicator &comm=queryWorldCommunicator();
-                            comm.cancel(NULL, MPTAG_SASHA_REQUEST);
+                            if (isDaliClient)
+                            {
+                                IInterCommunicator &comm = queryWorldCommunicator();
+                                comm.cancel(NULL, MPTAG_SASHA_REQUEST);
+                            }
                         }
                     }
                 } *stopThread = new CStopThread;
path: dali/sasha/saserver.cpp
user: login: jakesmith
body: unconditionally creating CStopThread here, but conditionally deleting it on line 521.
If not a Dali client, then doesn't need this thread.
It doesn't need to signal the stopSem either (doesn't need the stopSem at all, but harmless).
created_at: 2025-04-03T16:15:23Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,179 @@
+#include "platform.h"
+
+#include "jfile.hpp"
+#include "jlib.hpp"
+#include "jlog.hpp"
+#include "jiface.hpp"
+#include "jptree.hpp"
+#include "jstring.hpp"
+
+#include "saserver.hpp"
+#include "sautil.hpp"
+
+static constexpr unsigned defaultExpiryDays = 14;
+static constexpr unsigned defaultHouseKeepingIntervalHours = 24;
+
+#define LOGDBGHK "DEBUGPLANEHOUSEKEEPING: "
+
+// Debug Housekeeping monitor
+
+class CSashaDebugPlaneHousekeepingServer : public ISashaServer, public Thread
+{
+    std::atomic<bool> stopped{true};
+    Semaphore stopSem;
+    Mutex runMutex;
+    StringBuffer debugDir;
+    unsigned expiryDays{0};
+    unsigned interval{0};
+
+public:
+    IMPLEMENT_IINTERFACE_USING(Thread);
+
+    CSashaDebugPlaneHousekeepingServer()
+        : Thread("CSashaDebugPlaneHousekeepingServer")
+    {
+    }
+
+    virtual void start() override
+    {
+        Owned<IPropertyTree> compConfig = getComponentConfig();
+        expiryDays = compConfig->getPropInt("@expiryDays", defaultExpiryDays);
+        if (expiryDays == 0)
+            return;
+        interval = compConfig->getPropInt("@interval", defaultHouseKeepingIntervalHours);
+        if (interval == 0)
+            return;
+
+        // get debug plane dir
+        StringBuffer planeName;
+        if (!getDefaultPlane(planeName, "@debugPlane", "debug"))
+        {
+            WARNLOG("Failed to get default debug plane");
+            return;
+        }
+        Owned<IPropertyTree> plane = getStoragePlane(planeName);
+        assertex(plane);
+        verifyex(plane->getProp("@prefix", debugDir));
path: dali/sasha/sadbghk.cpp
user: login: jakesmith
body: trivial: reads odd that line 55 is an assertex and this is a verifyex. I'd change it to a single verifyex:
```
verifyex(plane && plane->getProp("@prefix", debugDir));
```

created_at: 2025-04-08T11:25:06Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,179 @@
+#include "platform.h"
+
+#include "jfile.hpp"
+#include "jlib.hpp"
+#include "jlog.hpp"
+#include "jiface.hpp"
+#include "jptree.hpp"
+#include "jstring.hpp"
+
+#include "saserver.hpp"
+#include "sautil.hpp"
+
+static constexpr unsigned defaultExpiryDays = 14;
+static constexpr unsigned defaultHouseKeepingIntervalHours = 24;
+
+#define LOGDBGHK "DEBUGPLANEHOUSEKEEPING: "
+
+// Debug Housekeeping monitor
+
+class CSashaDebugPlaneHousekeepingServer : public ISashaServer, public Thread
+{
+    std::atomic<bool> stopped{true};
+    Semaphore stopSem;
+    Mutex runMutex;
+    StringBuffer debugDir;
+    unsigned expiryDays{0};
+    unsigned interval{0};
+
+public:
+    IMPLEMENT_IINTERFACE_USING(Thread);
+
+    CSashaDebugPlaneHousekeepingServer()
+        : Thread("CSashaDebugPlaneHousekeepingServer")
+    {
+    }
+
+    virtual void start() override
+    {
+        Owned<IPropertyTree> compConfig = getComponentConfig();
+        expiryDays = compConfig->getPropInt("@expiryDays", defaultExpiryDays);
+        if (expiryDays == 0)
+            return;
+        interval = compConfig->getPropInt("@interval", defaultHouseKeepingIntervalHours);
+        if (interval == 0)
+            return;
+
+        // get debug plane dir
+        StringBuffer planeName;
+        if (!getDefaultPlane(planeName, "@debugPlane", "debug"))
+        {
+            WARNLOG("Failed to get default debug plane");
+            return;
+        }
+        Owned<IPropertyTree> plane = getStoragePlane(planeName);
+        assertex(plane);
+        verifyex(plane->getProp("@prefix", debugDir));
+
+        stopped = false;
+        Thread::start(false);
+    }
+
+    virtual void ready() override
+    {
+    }
+
+    virtual void stop() override
+    {
+        if (stopped)
+            return;
+
+        stopped = true;
+        stopSem.signal();
+        synchronized block(runMutex); // hopefully stopped should stop
+        if (!join(1000 * 60 * 3))
+            OERRLOG("CSashaDebugPlaneHousekeepingServer aborted");
+    }
+
+    void runDebugHousekeeping()
+    {
+        synchronized block(runMutex);
+        if (stopped)
+            return;
+
+        unsigned dirsDeleted{0};
+        unsigned filesDeleted{0};
+
+        // iterate debug plane selecting files and directories for housekeeping
+        Owned<IDirectoryIterator> pDirIter = createDirectoryIterator(debugDir.str(), "*", false, true);
+        ForEach(*pDirIter)
+        {
+            if (stopped)
+                break;
+
+            IFile &iFile = pDirIter->query();
+            const char *filePath = iFile.queryFilename();
+
+            if (iFile.isDirectory() == fileBool::foundYes)
+            {
+                // Process directories
+                
+                // Exclude the ".", ".."
+                if (streq(filePath, ".") || streq(filePath, ".."))
+                    continue;
+
+                // Process expired directories for housekeeping
+                if (hasExpired(iFile))
+                {
+                    recursiveRemoveDirectory(&iFile);
path: dali/sasha/sadbghk.cpp
user: login: jakesmith
body: recursiveRemoveDirectory should really return false if it fails.. but beyond scope of this PR.
But maybe add a comment on end of ++dirsDeleted; e.g. // should be really be conditional on success recursiveRemoveDirectory
created_at: 2025-04-08T11:29:55Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,179 @@
+#include "platform.h"
+
+#include "jfile.hpp"
+#include "jlib.hpp"
+#include "jlog.hpp"
+#include "jiface.hpp"
+#include "jptree.hpp"
+#include "jstring.hpp"
+
+#include "saserver.hpp"
+#include "sautil.hpp"
+
+static constexpr unsigned defaultExpiryDays = 14;
+static constexpr unsigned defaultHouseKeepingIntervalHours = 24;
+
+#define LOGDBGHK "DEBUGPLANEHOUSEKEEPING: "
+
+// Debug Housekeeping monitor
+
+class CSashaDebugPlaneHousekeepingServer : public ISashaServer, public Thread
+{
+    std::atomic<bool> stopped{true};
+    Semaphore stopSem;
+    Mutex runMutex;
+    StringBuffer debugDir;
+    unsigned expiryDays{0};
+    unsigned interval{0};
+
+public:
+    IMPLEMENT_IINTERFACE_USING(Thread);
+
+    CSashaDebugPlaneHousekeepingServer()
+        : Thread("CSashaDebugPlaneHousekeepingServer")
+    {
+    }
+
+    virtual void start() override
+    {
+        Owned<IPropertyTree> compConfig = getComponentConfig();
+        expiryDays = compConfig->getPropInt("@expiryDays", defaultExpiryDays);
+        if (expiryDays == 0)
+            return;
+        interval = compConfig->getPropInt("@interval", defaultHouseKeepingIntervalHours);
+        if (interval == 0)
+            return;
+
+        // get debug plane dir
+        StringBuffer planeName;
+        if (!getDefaultPlane(planeName, "@debugPlane", "debug"))
+        {
+            WARNLOG("Failed to get default debug plane");
+            return;
+        }
+        Owned<IPropertyTree> plane = getStoragePlane(planeName);
+        assertex(plane);
+        verifyex(plane->getProp("@prefix", debugDir));
+
+        stopped = false;
+        Thread::start(false);
+    }
+
+    virtual void ready() override
+    {
+    }
+
+    virtual void stop() override
+    {
+        if (stopped)
+            return;
+
+        stopped = true;
+        stopSem.signal();
+        synchronized block(runMutex); // hopefully stopped should stop
+        if (!join(1000 * 60 * 3))
+            OERRLOG("CSashaDebugPlaneHousekeepingServer aborted");
+    }
+
+    void runDebugHousekeeping()
+    {
+        synchronized block(runMutex);
+        if (stopped)
+            return;
+
+        unsigned dirsDeleted{0};
+        unsigned filesDeleted{0};
+
+        // iterate debug plane selecting files and directories for housekeeping
+        Owned<IDirectoryIterator> pDirIter = createDirectoryIterator(debugDir.str(), "*", false, true);
+        ForEach(*pDirIter)
+        {
+            if (stopped)
+                break;
+
+            IFile &iFile = pDirIter->query();
+            const char *filePath = iFile.queryFilename();
+
+            if (iFile.isDirectory() == fileBool::foundYes)
+            {
+                // Process directories
+                
+                // Exclude the ".", ".."
+                if (streq(filePath, ".") || streq(filePath, ".."))
+                    continue;
+
+                // Process expired directories for housekeeping
+                if (hasExpired(iFile))
+                {
+                    recursiveRemoveDirectory(&iFile);
+                    ++dirsDeleted;
+                }
+            }
+            else
+            {
+                // Process file exclude non expired files from housekeeping
+                if (hasExpired(iFile))
+                {
+                    iFile.remove();
+                    ++filesDeleted;
path: dali/sasha/sadbghk.cpp
user: login: jakesmith
body: should be conditional on result of iFile.remove().
Could track failed delete count too.
created_at: 2025-04-08T11:30:40Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,63 @@
+# Sasha
+
+## Services
+
+### Debug Plane Housekeeping Server
+
+#### Overview
+
+- This is a maintenance service that automatically manages files and directories on the debug plane using a configured retention period.
+- This is required to control the disc space used by post-mortem and other files on the debug plane.
+- This service is designed to execute on a schedule with no interaction other; a single log entry for a summary of any deleted files or directories will be generated.
path: dali/sasha/sasha.md
user: login: jakesmith
body: >with no interaction other

with no other interaction?
created_at: 2025-04-08T11:34:04Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,63 @@
+# Sasha
+
+## Services
+
+### Debug Plane Housekeeping Server
+
+#### Overview
+
+- This is a maintenance service that automatically manages files and directories on the debug plane using a configured retention period.
+- This is required to control the disc space used by post-mortem and other files on the debug plane.
+- This service is designed to execute on a schedule with no interaction other; a single log entry for a summary of any deleted files or directories will be generated.
+
+#### Installation and Configuration
+
+- The service is installed and active by default.
+- It is configuration-driven with fallback to defaults in the configuration yaml files.
+  - For example `values.yaml` :
+
+    |attribute|description|values|default|required|
+    |:--------|:----------|:-----|:------|:-------|
+    |disabled|Disable or enable this service.|false|false|optional|
path: dali/sasha/sasha.md
user: login: jakesmith
body: what is 'values' column vs default?
created_at: 2025-04-08T11:35:27Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,63 @@
+# Sasha
+
+## Services
+
+### Debug Plane Housekeeping Server
+
+#### Overview
+
+- This is a maintenance service that automatically manages files and directories on the debug plane using a configured retention period.
+- This is required to control the disc space used by post-mortem and other files on the debug plane.
+- This service is designed to execute on a schedule with no interaction other; a single log entry for a summary of any deleted files or directories will be generated.
+
+#### Installation and Configuration
+
+- The service is installed and active by default.
+- It is configuration-driven with fallback to defaults in the configuration yaml files.
+  - For example `values.yaml` :
+
+    |attribute|description|values|default|required|
+    |:--------|:----------|:-----|:------|:-------|
+    |disabled|Disable or enable this service.|false|false|optional|
path: dali/sasha/sasha.md
user: login: streeterd
body: 'values' will be removed.
created_at: 2025-04-08T12:41:20Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,63 @@
+# Sasha
+
+## Services
+
+### Debug Plane Housekeeping Server
+
+#### Overview
+
+- This is a maintenance service that automatically manages files and directories on the debug plane using a configured retention period.
+- This is required to control the disc space used by post-mortem and other files on the debug plane.
+- This service is designed to execute on a schedule with no interaction other; a single log entry for a summary of any deleted files or directories will be generated.
path: dali/sasha/sasha.md
user: login: streeterd
body: I will remove 'with no other interaction'
created_at: 2025-04-08T12:41:41Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,179 @@
+#include "platform.h"
+
+#include "jfile.hpp"
+#include "jlib.hpp"
+#include "jlog.hpp"
+#include "jiface.hpp"
+#include "jptree.hpp"
+#include "jstring.hpp"
+
+#include "saserver.hpp"
+#include "sautil.hpp"
+
+static constexpr unsigned defaultExpiryDays = 14;
+static constexpr unsigned defaultHouseKeepingIntervalHours = 24;
+
+#define LOGDBGHK "DEBUGPLANEHOUSEKEEPING: "
+
+// Debug Housekeeping monitor
+
+class CSashaDebugPlaneHousekeepingServer : public ISashaServer, public Thread
+{
+    std::atomic<bool> stopped{true};
+    Semaphore stopSem;
+    Mutex runMutex;
+    StringBuffer debugDir;
+    unsigned expiryDays{0};
+    unsigned interval{0};
+
+public:
+    IMPLEMENT_IINTERFACE_USING(Thread);
+
+    CSashaDebugPlaneHousekeepingServer()
+        : Thread("CSashaDebugPlaneHousekeepingServer")
+    {
+    }
+
+    virtual void start() override
+    {
+        Owned<IPropertyTree> compConfig = getComponentConfig();
+        expiryDays = compConfig->getPropInt("@expiryDays", defaultExpiryDays);
+        if (expiryDays == 0)
+            return;
+        interval = compConfig->getPropInt("@interval", defaultHouseKeepingIntervalHours);
+        if (interval == 0)
+            return;
+
+        // get debug plane dir
+        StringBuffer planeName;
+        if (!getDefaultPlane(planeName, "@debugPlane", "debug"))
+        {
+            WARNLOG("Failed to get default debug plane");
+            return;
+        }
+        Owned<IPropertyTree> plane = getStoragePlane(planeName);
+        assertex(plane);
+        verifyex(plane->getProp("@prefix", debugDir));
path: dali/sasha/sadbghk.cpp
user: login: streeterd
body: Fixed in 341117ff45e2e48c46e8281ce781a1ad90ead3ba
created_at: 2025-04-08T12:47:08Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,179 @@
+#include "platform.h"
+
+#include "jfile.hpp"
+#include "jlib.hpp"
+#include "jlog.hpp"
+#include "jiface.hpp"
+#include "jptree.hpp"
+#include "jstring.hpp"
+
+#include "saserver.hpp"
+#include "sautil.hpp"
+
+static constexpr unsigned defaultExpiryDays = 14;
+static constexpr unsigned defaultHouseKeepingIntervalHours = 24;
+
+#define LOGDBGHK "DEBUGPLANEHOUSEKEEPING: "
+
+// Debug Housekeeping monitor
+
+class CSashaDebugPlaneHousekeepingServer : public ISashaServer, public Thread
+{
+    std::atomic<bool> stopped{true};
+    Semaphore stopSem;
+    Mutex runMutex;
+    StringBuffer debugDir;
+    unsigned expiryDays{0};
+    unsigned interval{0};
+
+public:
+    IMPLEMENT_IINTERFACE_USING(Thread);
+
+    CSashaDebugPlaneHousekeepingServer()
+        : Thread("CSashaDebugPlaneHousekeepingServer")
+    {
+    }
+
+    virtual void start() override
+    {
+        Owned<IPropertyTree> compConfig = getComponentConfig();
+        expiryDays = compConfig->getPropInt("@expiryDays", defaultExpiryDays);
+        if (expiryDays == 0)
+            return;
+        interval = compConfig->getPropInt("@interval", defaultHouseKeepingIntervalHours);
+        if (interval == 0)
+            return;
+
+        // get debug plane dir
+        StringBuffer planeName;
+        if (!getDefaultPlane(planeName, "@debugPlane", "debug"))
+        {
+            WARNLOG("Failed to get default debug plane");
+            return;
+        }
+        Owned<IPropertyTree> plane = getStoragePlane(planeName);
+        assertex(plane);
+        verifyex(plane->getProp("@prefix", debugDir));
+
+        stopped = false;
+        Thread::start(false);
+    }
+
+    virtual void ready() override
+    {
+    }
+
+    virtual void stop() override
+    {
+        if (stopped)
+            return;
+
+        stopped = true;
+        stopSem.signal();
+        synchronized block(runMutex); // hopefully stopped should stop
+        if (!join(1000 * 60 * 3))
+            OERRLOG("CSashaDebugPlaneHousekeepingServer aborted");
+    }
+
+    void runDebugHousekeeping()
+    {
+        synchronized block(runMutex);
+        if (stopped)
+            return;
+
+        unsigned dirsDeleted{0};
+        unsigned filesDeleted{0};
+
+        // iterate debug plane selecting files and directories for housekeeping
+        Owned<IDirectoryIterator> pDirIter = createDirectoryIterator(debugDir.str(), "*", false, true);
+        ForEach(*pDirIter)
+        {
+            if (stopped)
+                break;
+
+            IFile &iFile = pDirIter->query();
+            const char *filePath = iFile.queryFilename();
+
+            if (iFile.isDirectory() == fileBool::foundYes)
+            {
+                // Process directories
+                
+                // Exclude the ".", ".."
+                if (streq(filePath, ".") || streq(filePath, ".."))
+                    continue;
+
+                // Process expired directories for housekeeping
+                if (hasExpired(iFile))
+                {
+                    recursiveRemoveDirectory(&iFile);
path: dali/sasha/sadbghk.cpp
user: login: streeterd
body: Fixed in 341117ff45e2e48c46e8281ce781a1ad90ead3ba
created_at: 2025-04-08T12:47:14Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,179 @@
+#include "platform.h"
+
+#include "jfile.hpp"
+#include "jlib.hpp"
+#include "jlog.hpp"
+#include "jiface.hpp"
+#include "jptree.hpp"
+#include "jstring.hpp"
+
+#include "saserver.hpp"
+#include "sautil.hpp"
+
+static constexpr unsigned defaultExpiryDays = 14;
+static constexpr unsigned defaultHouseKeepingIntervalHours = 24;
+
+#define LOGDBGHK "DEBUGPLANEHOUSEKEEPING: "
+
+// Debug Housekeeping monitor
+
+class CSashaDebugPlaneHousekeepingServer : public ISashaServer, public Thread
+{
+    std::atomic<bool> stopped{true};
+    Semaphore stopSem;
+    Mutex runMutex;
+    StringBuffer debugDir;
+    unsigned expiryDays{0};
+    unsigned interval{0};
+
+public:
+    IMPLEMENT_IINTERFACE_USING(Thread);
+
+    CSashaDebugPlaneHousekeepingServer()
+        : Thread("CSashaDebugPlaneHousekeepingServer")
+    {
+    }
+
+    virtual void start() override
+    {
+        Owned<IPropertyTree> compConfig = getComponentConfig();
+        expiryDays = compConfig->getPropInt("@expiryDays", defaultExpiryDays);
+        if (expiryDays == 0)
+            return;
+        interval = compConfig->getPropInt("@interval", defaultHouseKeepingIntervalHours);
+        if (interval == 0)
+            return;
+
+        // get debug plane dir
+        StringBuffer planeName;
+        if (!getDefaultPlane(planeName, "@debugPlane", "debug"))
+        {
+            WARNLOG("Failed to get default debug plane");
+            return;
+        }
+        Owned<IPropertyTree> plane = getStoragePlane(planeName);
+        assertex(plane);
+        verifyex(plane->getProp("@prefix", debugDir));
+
+        stopped = false;
+        Thread::start(false);
+    }
+
+    virtual void ready() override
+    {
+    }
+
+    virtual void stop() override
+    {
+        if (stopped)
+            return;
+
+        stopped = true;
+        stopSem.signal();
+        synchronized block(runMutex); // hopefully stopped should stop
+        if (!join(1000 * 60 * 3))
+            OERRLOG("CSashaDebugPlaneHousekeepingServer aborted");
+    }
+
+    void runDebugHousekeeping()
+    {
+        synchronized block(runMutex);
+        if (stopped)
+            return;
+
+        unsigned dirsDeleted{0};
+        unsigned filesDeleted{0};
+
+        // iterate debug plane selecting files and directories for housekeeping
+        Owned<IDirectoryIterator> pDirIter = createDirectoryIterator(debugDir.str(), "*", false, true);
+        ForEach(*pDirIter)
+        {
+            if (stopped)
+                break;
+
+            IFile &iFile = pDirIter->query();
+            const char *filePath = iFile.queryFilename();
+
+            if (iFile.isDirectory() == fileBool::foundYes)
+            {
+                // Process directories
+                
+                // Exclude the ".", ".."
+                if (streq(filePath, ".") || streq(filePath, ".."))
+                    continue;
+
+                // Process expired directories for housekeeping
+                if (hasExpired(iFile))
+                {
+                    recursiveRemoveDirectory(&iFile);
+                    ++dirsDeleted;
+                }
+            }
+            else
+            {
+                // Process file exclude non expired files from housekeeping
+                if (hasExpired(iFile))
+                {
+                    iFile.remove();
+                    ++filesDeleted;
path: dali/sasha/sadbghk.cpp
user: login: streeterd
body: Fixed in 341117ff45e2e48c46e8281ce781a1ad90ead3ba
created_at: 2025-04-08T12:47:19Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,63 @@
+# Sasha
+
+## Services
+
+### Debug Plane Housekeeping Server
+
+#### Overview
+
+- This is a maintenance service that automatically manages files and directories on the debug plane using a configured retention period.
+- This is required to control the disc space used by post-mortem and other files on the debug plane.
+- This service is designed to execute on a schedule with no interaction other; a single log entry for a summary of any deleted files or directories will be generated.
path: dali/sasha/sasha.md
user: login: streeterd
body: Fixed in 341117ff45e2e48c46e8281ce781a1ad90ead3ba
created_at: 2025-04-08T12:47:25Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,63 @@
+# Sasha
+
+## Services
+
+### Debug Plane Housekeeping Server
+
+#### Overview
+
+- This is a maintenance service that automatically manages files and directories on the debug plane using a configured retention period.
+- This is required to control the disc space used by post-mortem and other files on the debug plane.
+- This service is designed to execute on a schedule with no interaction other; a single log entry for a summary of any deleted files or directories will be generated.
+
+#### Installation and Configuration
+
+- The service is installed and active by default.
+- It is configuration-driven with fallback to defaults in the configuration yaml files.
+  - For example `values.yaml` :
+
+    |attribute|description|values|default|required|
+    |:--------|:----------|:-----|:------|:-------|
+    |disabled|Disable or enable this service.|false|false|optional|
path: dali/sasha/sasha.md
user: login: streeterd
body: Fixed in 341117ff45e2e48c46e8281ce781a1ad90ead3ba
created_at: 2025-04-08T12:47:30Z
author_association: CONTRIBUTOR

PR comments:
user: login: github-actions[bot]
created_at: 2025-03-05T16:19:42Z
body: Jira Issue: https://hpccsystems.atlassian.net//browse/HPCC-27051

Jirabot Action Result:
Workflow Transition To: Merge Pending
Additional PR: https://github.com/hpcc-systems/HPCC-Platform/pull/19588

author_association: NONE
user: login: streeterd
created_at: 2025-03-18T10:01:37Z
body: > @streeterd - please see comments.

@jakesmith Fixed in a4adb7505207c5a52be8cc1c6cace2ff1158bff7
author_association: CONTRIBUTOR
user: login: streeterd
created_at: 2025-03-20T12:25:26Z
body: @jakesmith  squashed.
author_association: CONTRIBUTOR
user: login: jakesmith
created_at: 2025-03-20T16:20:56Z
body: @streeterd - can you double check if any last minute changes broke k8s startup?
author_association: MEMBER
user: login: streeterd
created_at: 2025-03-20T17:06:33Z
body: @jakesmith I ran up the containerised build after having to exclude mongodb due to a compilation failure:

```
c/include -isystem /hpcc-dev/build/vcpkg_installed/x64-linux-dynamic/include/bsoncxx/v_noabi -isystem /hpcc-dev/build/vcpkg_installed/x64-linux-dynamic/include/mongocxx/v_noabi -frtti -fPIC -fmessage-length=0 -Werror=format -Wformat-security -Wformat-nonliteral -pthread -Wuninitialized -Werror=return-type -Werror=format-nonliteral -Wno-psabi -Wparentheses -g -D_DEBUG -DDEBUG -fPIC   -fsanitize=undefined -fno-sanitize=alignment -fsanitize-undefined-trap-on-error -MD -MT plugins/mongodb/CMakeFiles/mongodbembed.dir/mongodbembed.cpp.o -MF plugins/mongodb/CMakeFiles/mongodbembed.dir/mongodbembed.cpp.o.d -o plugins/mongodb/CMakeFiles/mongodbembed.dir/mongodbembed.cpp.o -c /hpcc-dev/HPCC-Platform/plugins/mongodb/mongodbembed.cpp
In file included from /hpcc-dev/build/vcpkg_installed/x64-linux-dynamic/include/bsoncxx/v_noabi/bsoncxx/stdx/optional.hpp:50,
                 from /hpcc-dev/build/vcpkg_installed/x64-linux-dynamic/include/bsoncxx/v_noabi/bsoncxx/document/element.hpp:27,
                 from /hpcc-dev/build/vcpkg_installed/x64-linux-dynamic/include/bsoncxx/v_noabi/bsoncxx/document/view.hpp:23,
                 from /hpcc-dev/build/vcpkg_installed/x64-linux-dynamic/include/mongocxx/v_noabi/mongocxx/client_session.hpp:28,
                 from /hpcc-dev/build/vcpkg_installed/x64-linux-dynamic/include/mongocxx/v_noabi/mongocxx/client.hpp:28,
                 from /hpcc-dev/HPCC-Platform/plugins/mongodb/mongodbembed.hpp:28,
                 from /hpcc-dev/HPCC-Platform/plugins/mongodb/mongodbembed.cpp:18:
/hpcc-dev/build/vcpkg_installed/x64-linux-dynamic/include/bsoncxx/v_noabi/bsoncxx/stdx/operators.hpp:88:34: error: 'bsoncxx::detail::strong_ordering::less' declared weak after being used
   88 |     static const strong_ordering less;
      |                                  ^~~~
/hpcc-dev/build/vcpkg_installed/x64-linux-dynamic/include/bsoncxx/v_noabi/bsoncxx/stdx/operators.hpp:89:34: error: 'bsoncxx::detail::strong_ordering::greater' declared weak after being used
   89 |     static const strong_ordering greater;
      |                                  ^~~~~~~

```
The pods run up ok
![image](https://github.com/user-attachments/assets/b10552c5-1422-4dac-b42c-4349a95e0940)

The log shows the debug housekeeping pod is ok:

```
dave@RISKAZ-06905526:~/HPCC-Platform/dockerfiles$ kl sasha-debugplane-housekeeping-6f7f868f64-wqr9b
running saserver --logging.postMortem=1000 --config=/etc/config/sasha-debugplane-housekeeping.yaml --daliServers=mydali:7070 --service=debugplane-housekeeping
Waiting for child process 7
00000000 USR PRO 2025-03-20 16:49:32.707     7     7 UNK     "JLog: queuing enabled: queueLen=512"
00000001 PRG INF 2025-03-20 16:49:32.707     7     7 UNK     "Build community_9.11.0-trunk0Debug[heads/HPCC-27051_Create_a_Sasha_service_to_clean_up_post-mortem_files-0-g7de389-dirty]"
00000002 USR PRO 2025-03-20 16:49:32.708     7     7 UNK     "MP TLS: off acceptThreadPoolSize: 100"
00000003 USR PRO 2025-03-20 16:49:34.915     7     7 UNK     "initializeStoragePlanes update"
00000004 PRG INF 2025-03-20 16:49:34.916     7     7 UNK     "Modify functions should be registered before the configuration is loaded"
00000005 USR PRO 2025-03-20 16:49:34.916     7     7 UNK     "SASERVER starting on sasha-debugplane-housekeeping-6f7f868f64-wqr9b:8877"
00000006 PRG INF 2025-03-20 16:49:34.916     7     7 UNK     "Creating sentinel file /tmp/sasha-debugplane-housekeeping.sentinel"
dave@RISKAZ-06905526:~/HPCC-Platform/dockerfiles$

```

I then completed a delete test of expired file and dir:

```
dave@RISKAZ-06905526:~$ kl sasha-debugplane-housekeeping-6f7f868f64-wqr9b
running saserver --logging.postMortem=1000 --config=/etc/config/sasha-debugplane-housekeeping.yaml --daliServers=mydali:7070 --service=debugplane-housekeeping
Waiting for child process 7
00000000 USR PRO 2025-03-20 16:49:32.707     7     7 UNK     "JLog: queuing enabled: queueLen=512"
00000001 PRG INF 2025-03-20 16:49:32.707     7     7 UNK     "Build community_9.11.0-trunk0Debug[heads/HPCC-27051_Create_a_Sasha_service_to_clean_up_post-mortem_files-0-g7de389-dirty]"
00000002 USR PRO 2025-03-20 16:49:32.708     7     7 UNK     "MP TLS: off acceptThreadPoolSize: 100"
00000003 USR PRO 2025-03-20 16:49:34.915     7     7 UNK     "initializeStoragePlanes update"
00000004 PRG INF 2025-03-20 16:49:34.916     7     7 UNK     "Modify functions should be registered before the configuration is loaded"
00000005 USR PRO 2025-03-20 16:49:34.916     7     7 UNK     "SASERVER starting on sasha-debugplane-housekeeping-6f7f868f64-wqr9b:8877"
00000006 PRG INF 2025-03-20 16:49:34.916     7     7 UNK     "Creating sentinel file /tmp/sasha-debugplane-housekeeping.sentinel"
00000007 USR PRO 2025-03-20 16:59:44.024     7    18 UNK     "DEBUGPLANEHOUSEKEEPING: Deleted dirs: 1, files: 1"
dave@RISKAZ-06905526:~$

```

Still running ok after that:
![image](https://github.com/user-attachments/assets/3f617551-5de1-41fe-bd87-4602a750b84e)

All looks ok.
author_association: CONTRIBUTOR
user: login: jakesmith
created_at: 2025-03-20T17:45:59Z
body: Not related to the k8s stall, but noticed some spurious white space in a couple of the yaml files in this PR:

![image](https://github.com/user-attachments/assets/443e5467-bcae-490d-8e2c-599a1be58634)

author_association: MEMBER
user: login: streeterd
created_at: 2025-04-04T11:06:52Z
body: @jakesmith I tested the latest commit ba5a4ffcfc1a4f0635c8a18a25ce6e75f7f53aa4

In containerized:

```
running saserver --logging.postMortem=1000 --config=/etc/config/sasha-debugplane-housekeeping.yaml --daliServers=mydali:7070 --service=debugplane-housekeeping
Waiting for child process 7
00000000 USR PRO 2025-04-04 10:32:09.446     7     7 UNK     "JLog: queuing enabled: queueLen=512"
00000001 PRG INF 2025-04-04 10:32:09.446     7     7 UNK     "Build community_9.11.0-trunk0Debug[heads/HPCC-27051_Create_a_Sasha_service_to_clean_up_post-mortem_files-0-gba5a4f-dirty]"
00000002 USR PRO 2025-04-04 10:32:09.447     7     7 UNK     "initializeStoragePlanes update"
00000003 PRG INF 2025-04-04 10:32:09.447     7     7 UNK     "Modify functions should be registered before the configuration is loaded"
00000004 PRG INF 2025-04-04 10:32:09.447     7     7 UNK     "Creating sentinel file /tmp/sasha-debugplane-housekeeping.sentinel"
00000005 USR PRO 2025-04-04 10:39:50.037     7    13 UNK     "DEBUGPLANEHOUSEKEEPING: Deleted dirs: 1, files: 1"
EXIT via signal for 7
Sending SIGTERM to process (PID 7)
00000006 USR PRO 2025-04-04 10:40:23.751     7     7 UNK     "Stop signalled"
00000007 USR PRO 2025-04-04 10:40:23.751     7     7 UNK     "Stop requested"
00000008 USR PRO 2025-04-04 10:40:23.751     7     7 UNK     "SASERVER exiting"
00000009 USR PRO 2025-04-04 10:40:23.751     7     7 UNK     "Stopping 0"
Child process 7 exited with exit code 0
```

and

Bare metal:

```
dave@RISKAZ-06905526:~/HPCC-Platform$ bms
Starting mydafilesrv ...       [   OK    ]
Starting mydali ...            [   OK    ]
Starting mydfuserver ...       [   OK    ]
Starting myeclagent ...        [   OK    ]
Starting myeclccserver ...     [   OK    ]
Starting myeclscheduler ...    [   OK    ]
Starting myesp ...             [   OK    ]
Starting myroxie ...           [   OK    ]
Starting mysasha ...           [   OK    ]
Starting mythor ...            [   OK    ]
Starting mytoposerver ...      [   OK    ]
dave@RISKAZ-06905526:~/HPCC-Platform$ bme
Stopping mytoposerver...       [   OK    ]
Stopping mythor...             [   OK    ]
Stopping mysasha...            [   OK    ]
Stopping myroxie...            [   OK    ]
Stopping myesp...              [   OK    ]
Stopping myeclscheduler...     [   OK    ]
Stopping myeclccserver...      [   OK    ]
Stopping myeclagent...         [   OK    ]
Stopping mydfuserver...        [   OK    ]
Stopping mydali...             [   OK    ]
Service dafilesrv, mydafilesrv is still running.
To stop it, run "/etc/init.d/dafilesrv stop".
Stopping mydafilesrv...        [   OK    ]
dave@RISKAZ-06905526:~/hpccDebug/etc/init.d$

```

author_association: CONTRIBUTOR
user: login: streeterd
created_at: 2025-04-09T08:02:30Z
body: @jakesmith  squashed
author_association: CONTRIBUTOR
user: login: github-actions[bot]
created_at: 2025-04-10T15:47:39Z
body: Jirabot Action Result:
Added fix version: 9.12.0
Workflow Transition: 'Resolve issue'

author_association: NONE