Repository: spacetelescope/jwql

Issue:
Create script to delete old log files
_Issue [JWQL-165](https://jira.stsci.edu/browse/JWQL-165) was created on JIRA by [Bryan Hilbert](https://jira.stsci.edu/secure/ViewProfile.jspa?name=hilbert):_

With instrument monitors and other scripts running via cron job, JWQL builds up a lot of log files very quickly. We should create a script that will periodically clean out old log files, in order to save disk space and to limit the number of files present in the log file directories.

It probably makes sense to have the script only examine and delete files that were created by the servers and ignore those created by other machines, since the latter will be coming from development work and older logs may be useful.


Discussion:

Resolving patch:
diff --git a/jwql/website/apps/jwql/clean_old_log_files.py b/jwql/website/apps/jwql/clean_old_log_files.py
new file mode 100644
--- /dev/null
+++ b/jwql/website/apps/jwql/clean_old_log_files.py
@@ -0,0 +1,84 @@
+#! /usr/bin/env python
+
+"""Clean old log files from the collection of log files
+
+Authors
+-------
+
+    - Bryan Hilbert
+
+Use
+---
+
+    To delete log files that are older than some threshold age:
+    ::
+
+    > python clean_old_log_files.py --time_limit 7  # days
+
+"""
+import argparse
+from datetime import datetime, timedelta
+import os
+import socket
+
+from jwql.utils.utils import get_config
+
+HOSTNAME = socket.gethostname()
+configs = get_config()
+LOG_BASE_DIR = configs['log_dir']
+
+
+def define_options():
+    """Create parser to take the time limit.
+
+    Returns
+    -------
+    parser : argparse.ArgumentParser
+        Parser containing time limit
+    """
+    usage = 'clean_old_log_files.py -t 14'
+    parser = argparse.ArgumentParser(usage=usage)
+    parser.add_argument('-t', '--time_limit', type=int, default=14,
+                        help='Time limit in days. Log files older than this will be deleted.')
+    return parser
+
+
+def run(time_limit=timedelta(days=14)):
+    """Look through log directories and delete log files that are older than ``time_limit``.
+    Have time_limit default to be 14 days.
+
+    Inputs
+    ------
+    time_limit : datetime.timdelta
+        Files older than this time limit will be deleted
+    """
+    now = datetime.now()
+
+    if 'pljwql' in HOSTNAME:
+        subdir = 'ops'
+    elif 'tljwql' in HOSTNAME:
+        subdir = 'test'
+    else:
+        # This should cover the dev server as well as local machines
+        subdir = 'dev'
+
+    log_dir = os.path.join(LOG_BASE_DIR, subdir)
+    for logtype in os.scandir(log_dir):
+        if logtype.is_dir():
+            for item in os.scandir(logtype):
+                # We only try to delete log files produced by the machine on which
+                # this script is running. e.g. log files produced by the test server
+                # can only be deleted by running this script on the test server.
+                if HOSTNAME in item.name and item.name[-4:] == '.log':
+                    stat_result = item.stat()
+                    last_modified_time = datetime.fromtimestamp(stat_result.st_mtime)
+                    age = now - last_modified_time
+                    if age > time_limit:
+                        full_path = os.path.join(log_dir, logtype, item)
+                        os.remove(full_path)
+
+
+if __name__ == '__main__':
+    parser = define_options()
+    args = parser.parse_args()
+    run(timedelta(days=args.time_limit))


Test patch: