Repository: CSCfi/rems
Pull request #2728 (2021-09-16 11:51:46)

Expired application eraser
implements #2665

* add new configuration options (default values here):
  * :application-expiration nil
  * takes in configuration map with form `{:application.state/{state} "P90D"}` where duration is expressed as [ISO_8601 duration](https://en.wikipedia.org/wiki/ISO_8601#Durations)
  * multiple application-state/duration combinations can be defined in the same configuration object using application state as key: `{:application.state/draft "P90D" :application.state/closed "P7D"}`
* add new ns `rems.application.eraser`:
  * if enabled, function `remove-expired-applications!` is scheduled to run once every day (determined by server start time)
  * it searches all applications that have a defined state in config and were last updated over `[:application-expiration :application.state/{state}]` duration ago, and removes them, together with attachments
* updated .gitignore (clj-kondo)

to do: email notifications to applicant before application deletion (to be done in another ticket)

# Checklist for author

Remove items that aren't applicable, check items that are done.

## Reviewability
- [x] Link to issue

## Backwards compatibility
- [x] Config is backwards compatible

## Documentation
- [x] Update changelog if necessary
- [x] New config options in config-defaults.edn

## Testing
- [x] Complex logic is unit tested
- [x] Valuable features are integration / browser / acceptance tested automatically

## Follow-up
- [x] New tasks are created for pending or remaining tasks https://github.com/CSCfi/rems/issues/2729



Patch:
diff --git a/.gitignore b/.gitignore
index cf98e8428..a720b08cf 100644
--- a/.gitignore
+++ b/.gitignore
@@ -23,4 +23,5 @@
 /browsertest-downloads/
 /browsertest-accessibility-report/
 /.calva/
-/.lsp/
\ No newline at end of file
+/.lsp/
+/.clj-kondo/
\ No newline at end of file
diff --git a/CHANGELOG.md b/CHANGELOG.md
index adb6d3da2..dbae45cea 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -18,6 +18,7 @@ Changes since v2.20
 - There is a new v2 push API to sync entitlements outside of REMS. So far the only supported type is `:ega`. See [configuration.md](docs/configuration.md) for more details. (#2466)
 - Handlers can be invited to a workflow by email using the API (#2650)
 - Entitlement expiration field in application actions will now accept only future dates (#2674)
+- Applications can now be removed automatically after an expiration threshold is exceeded, using configuration option `:application-expiration`. This feature is disabled by default. (#2665)
 
 ### Changes
 - Reviewers are no longer able to see private field questions in generated application pdf. Forms are also no longer rendered in application page if they contain only private fields (#2161)
diff --git a/dev-config.edn b/dev-config.edn
index f7b587890..5926be1ea 100644
--- a/dev-config.edn
+++ b/dev-config.edn
@@ -31,6 +31,7 @@
  :application-id-column :generated-and-assigned-external-id
  :enable-assign-external-id-ui true
  :attachment-max-size 10000000 ;; 10MB
+ :application-expiration nil
  :enable-pdf-api true
  :enable-permissions-api true
  ;; example keys from https://tools.ietf.org/html/rfc7517#appendix-A
diff --git a/resources/config-defaults.edn b/resources/config-defaults.edn
index 6ae6afe8e..37e344b75 100644
--- a/resources/config-defaults.edn
+++ b/resources/config-defaults.edn
@@ -292,10 +292,17 @@
  ;; Show "Assign external id" action button in UI for handlers
  :enable-assign-external-id-ui false
 
- ;; maximum size of attachment files in bytes (10000000 = 10MB)
+ ;; Maximum size of attachment files in bytes (10000000 = 10MB).
  ;; nil = no limit
  :attachment-max-size 10000000
 
+ ;; Should applications be deleted after a set period of time has passed?
+ ;; Use map with key as required application state, and value as ISO-8601 duration:
+ ;; {:application.state/draft "P90D" :application.state/closed "P7D"}
+ ;; nil = feature not enabled.
+ ;; https://en.wikipedia.org/wiki/ISO_8601#Durations for duration formatting reference.
+ :application-expiration nil
+
  ;; Experimental features
 
  ;; enable /api/applications/:id/experimental/pdf
diff --git a/src/clj/rems/api/services/command.clj b/src/clj/rems/api/services/command.clj
index 5cb1fc45d..16ade1a9b 100644
--- a/src/clj/rems/api/services/command.clj
+++ b/src/clj/rems/api/services/command.clj
@@ -32,7 +32,7 @@
 (defn- delete-applications [new-events]
   (doseq [event new-events]
     (when (= :application.event/deleted (:event/type event))
-      (applications/delete-application! (:application/id event))))
+      (applications/delete-application-and-reload-cache! (:application/id event))))
   [])
 
 ;; Process managers react to events with side effects & new commands.
diff --git a/src/clj/rems/application/eraser.clj b/src/clj/rems/application/eraser.clj
new file mode 100644
index 000000000..8fec5aee9
--- /dev/null
+++ b/src/clj/rems/application/eraser.clj
@@ -0,0 +1,64 @@
+(ns rems.application.eraser
+  (:require [mount.core :as mount]
+            [clj-time.core :as time]
+            [clojure.test :refer :all]
+            [clojure.tools.logging :as log]
+            [rems.scheduler :as scheduler]
+            [rems.db.applications :as applications]
+            [rems.config :refer [env]])
+  (:import [org.joda.time Period]))
+
+(defn- is-expired?
+  [application]
+  (let [last-activity (:application/last-activity application)
+        state (:application/state application)
+        expiration-threshold (some-> env :application-expiration state Period/parse)]
+    (if expiration-threshold
+      (time/before? last-activity (time/minus (time/now) expiration-threshold))
+      false)))
+
+(deftest test-is-expired
+  (with-redefs [env {:application-expiration {:application.state/draft "P90D"
+                                              :application.state/closed "P7D"}}]
+    (testing "should identify expired application by state"
+      (let [over-90d-ago (time/minus (time/now) (time/days 90) (time/seconds 1))
+            over-7d-ago (time/minus (time/now) (time/days 7) (time/seconds 1))
+            over-1d-ago (time/minus (time/now) (time/days 1) (time/seconds 1))]
+        (is (is-expired? {:application/state :application.state/draft
+                          :application/last-activity over-90d-ago}))
+        (is (not (is-expired? {:application/state :application.state/draft
+                               :application/last-activity over-7d-ago})))
+        (is (is-expired? {:application/state :application.state/closed
+                          :application/last-activity over-7d-ago}))
+        (is (not (is-expired? {:application/state :application.state/closed
+                               :application/last-activity over-1d-ago})))
+        (is (not (is-expired? {:application/state :application.state/rejected
+                               :application/last-activity over-90d-ago})))))))
+
+(defn remove-expired-applications!
+  []
+  (log/info :start #'remove-expired-applications!)
+  (doseq [app-id (->> (applications/get-all-unrestricted-applications)
+                      (filter is-expired?)
+                      (map :application/id))]
+    (try
+      (applications/delete-application! app-id)
+      (log/info "Succesfully removed application" app-id)
+      (catch Throwable t
+        (log/error "Failed to remove application" app-id ":" t))))
+  (applications/reload-cache!)
+  (log/info :finish #'remove-expired-applications!))
+
+(mount/defstate expired-application-poller
+  :start (when (:application-expiration env)
+           (scheduler/start! remove-expired-applications! (.toStandardDuration (time/days 1))))
+  :stop (scheduler/stop! expired-application-poller))
+
+(comment
+  (mount/defstate expired-application-poller-test
+    :start (scheduler/start! (fn [] (with-redefs [env {:application-expiration {:application.state/draft "P90D"}}]
+                                      (remove-expired-applications!))) (.toStandardDuration (time/seconds 10)))
+    :stop (scheduler/stop! expired-application-poller-test))
+  (mount/start #{#'expired-application-poller-test})
+  (mount/stop #{#'expired-application-poller-test}))
+
diff --git a/src/clj/rems/db/applications.clj b/src/clj/rems/db/applications.clj
index 0263d96f7..5f29e8b37 100644
--- a/src/clj/rems/db/applications.clj
+++ b/src/clj/rems/db/applications.clj
@@ -8,6 +8,7 @@
             [conman.core :as conman]
             [medley.core :refer [map-vals]]
             [mount.core :as mount]
+            [clj-time.core :as time]
             [rems.application.events-cache :as events-cache]
             [rems.application.model :as model]
             [rems.auth.util :refer [throw-forbidden]]
@@ -282,11 +283,15 @@
   :start (scheduler/start! reload-cache! (Duration/standardHours 1))
   :stop (scheduler/stop! all-applications-cache-reloader))
 
-(defn delete-application! [app-id]
+(defn delete-application!
+  [app-id]
   (let [application (get-application app-id)]
     (assert (= :application.state/draft (:application/state application))
             (str "Tried to delete application " app-id " which is not a draft!")))
   (db/delete-application-attachments! {:application app-id})
   (db/delete-application-events! {:application app-id})
-  (db/delete-application! {:application app-id})
-  (reload-cache!))
+  (db/delete-application! {:application app-id}))
+
+(defn delete-application-and-reload-cache! [app-id]
+  (delete-application! app-id)
+  (reload-cache!))
\ No newline at end of file
diff --git a/src/clj/rems/handler.clj b/src/clj/rems/handler.clj
index 1ea3dbb0d..4d4d7f44e 100644
--- a/src/clj/rems/handler.clj
+++ b/src/clj/rems/handler.clj
@@ -15,6 +15,7 @@
             [rems.css.styles :as styles]
             [rems.db.catalogue :as catalogue]
             [rems.email.core] ;; to enable email polling
+            [rems.application.eraser] ;; to enable expired application clean-up job
             [rems.entitlements :as entitlements]
             [rems.layout :as layout]
             [rems.middleware :refer [wrap-cacheable wrap-base]]


Test patch:
diff --git a/test-config.edn b/test-config.edn
index 404fbfd1b..d97bf3a42 100644
--- a/test-config.edn
+++ b/test-config.edn
@@ -12,6 +12,7 @@
                          {:attribute "picture"}]
  :languages [:en :fi :sv]
  :attachment-max-size 10000
+ :application-expiration {:application.state/draft "P90D"}
  :enable-pdf-api true
  :enable-permissions-api true
  ;; example keys from https://tools.ietf.org/html/rfc7517#appendix-A
diff --git a/test/clj/rems/application/test_eraser.clj b/test/clj/rems/application/test_eraser.clj
new file mode 100644
index 000000000..b11cc4bf9
--- /dev/null
+++ b/test/clj/rems/application/test_eraser.clj
@@ -0,0 +1,58 @@
+(ns ^:integration rems.application.test-eraser
+  (:require [clj-time.core :as time]
+            [clojure.test :refer :all]
+            [rems.db.core :as db]
+            [rems.db.events :as events]
+            [rems.db.applications :as applications]
+            [rems.application.eraser :as eraser]
+            [rems.db.testing :refer [test-db-fixture rollback-db-fixture]]))
+
+(use-fixtures :once test-db-fixture)
+(use-fixtures :each rollback-db-fixture)
+
+(defn- add-dummy-application!
+  [{:keys [date-time actor draft?]}]
+  (let [app-id (:id (db/create-application!))
+        created-event {:event/type :application.event/created
+                       :event/time date-time
+                       :event/actor actor
+                       :application/id app-id
+                       :application/external-id ""
+                       :application/resources []
+                       :application/licenses []
+                       :application/forms []
+                       :workflow/id 0
+                       :workflow/type :workflow/default}
+        draft-saved-event {:event/type :application.event/draft-saved
+                           :event/time date-time
+                           :event/actor actor
+                           :application/id app-id
+                           :application/field-values []}
+        submitted-event {:event/type :application.event/submitted
+                         :event/time date-time
+                         :event/actor actor
+                         :application/id app-id}]
+    (events/add-event! created-event)
+    (events/add-event! draft-saved-event)
+    (when (not draft?)
+      (events/add-event! submitted-event))
+    app-id))
+
+(deftest test-eraser
+  (testing "removes expired draft applications"
+    (let [app-id-1 (add-dummy-application! {:draft? true :date-time (time/now) :actor "alice"})
+          app-id-2 (add-dummy-application! {:date-time (time/now) :actor "alice"})
+          app-id-3 (add-dummy-application! {:date-time (time/minus (time/now) (time/days 120)) :actor "alice"})
+          expired-app-id (add-dummy-application! {:draft? true
+                                                  :date-time (time/minus (time/now) (time/days 90) (time/seconds 1))
+                                                  :actor "alice"})
+          expired-app-id-2 (add-dummy-application! {:draft? true
+                                                    :date-time (time/minus (time/now) (time/days 120))
+                                                    :actor "alice"})]
+      (let [before-apps (applications/get-all-applications "alice")]
+        (is (= #{app-id-1 app-id-2 app-id-3 expired-app-id expired-app-id-2}
+               (set (map :application/id before-apps)))))
+      (eraser/remove-expired-applications!)
+      (let [after-apps (applications/get-all-applications "alice")]
+        (is (= #{app-id-1 app-id-2 app-id-3}
+               (set (map :application/id after-apps))))))))
\ No newline at end of file
diff --git a/test/clj/rems/db/test_applications.clj b/test/clj/rems/db/test_applications.clj
index fd947435a..8521a71f0 100644
--- a/test/clj/rems/db/test_applications.clj
+++ b/test/clj/rems/db/test_applications.clj
@@ -81,10 +81,10 @@
     (is (= "1980/2" (application-external-id! (DateTime. #inst "1980-12-12"))))
     (is (= "1981/4" (application-external-id! (DateTime. #inst "1981-04-01"))))))
 
-(deftest test-delete-application!
+(deftest test-delete-application-and-reload-cache!!
   (let [app-id (test-helpers/create-application! {:actor "applicant"})]
     (is (applications/get-application app-id))
-    (applications/delete-application! app-id)
+    (applications/delete-application-and-reload-cache! app-id)
     (testing "deleted draft is gone"
       (is (not (applications/get-application app-id))))
     (testing "events are gone"
@@ -96,9 +96,9 @@
                             :type :application.command/submit
                             :actor "applicant"})
     (testing "can't delete submitted application"
-      (is (thrown? AssertionError (applications/delete-application! app-id))))
+      (is (thrown? AssertionError (applications/delete-application-and-reload-cache! app-id))))
     (test-helpers/command! {:application-id app-id
                             :type :application.command/return
                             :actor "developer"})
     (testing "can't delete returned application"
-      (is (thrown? AssertionError (applications/delete-application! app-id))))))
+      (is (thrown? AssertionError (applications/delete-application-and-reload-cache! app-id))))))