Repository: CSCfi/rems
Pull request #3154 (2023-05-10 05:46:19)

Plugins
For #3133, #3097 

Introduces a new configurable plugin architecture based on executing scripts with [SCI](https://github.com/babashka/sci).

## Must solve
- [x] loading plugins (in config mount)
- [x] finish documentation and changelog

# Checklist for author

Remove items that aren't applicable, check items that are done.

## Reviewability
- [x] Link to issue

## Backwards compatibility
- [x] Config is backwards compatible

## Documentation
- [x] Update changelog if necessary
- [x] Update docs/ (if applicable)
- [ ] ADR for major architectural decisions or experiments (split to #3164)
- [x] New config options in config-defaults.edn

## Testing
- [x] Complex logic is unit tested
- [ ] Valuable features are integration / browser / acceptance tested automatically

## Follow-up
- [x] New tasks are created for pending or remaining tasks (split to #3164)



Patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 70cebe751..aedee584f 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -11,6 +11,7 @@ Changes since v2.32
 ### Additions
 - PDF output can be customized a little. This enables changing the font, which should fix missing diacritics. (#3158)
 - Workflows can now disable commands dynamically by application state, user role, or both. Rules can be set in workflow administration page under disable commands field. (#3131)
+- Plugins can be written to extend REMS functionality in select extension points. So far there is support for plugins written in Clojure embedded in Markdown files. See `docs/plugins.md` for details. (#3133)
 
 ### Fixes
 - Label and header fields have URLs made into links. (#3155)
diff --git a/dev-config.edn b/dev-config.edn
index de8e0f1ea..644c924dd 100644
--- a/dev-config.edn
+++ b/dev-config.edn
@@ -17,7 +17,8 @@
  ;; some attributes that google oauth returns:
  :oidc-extra-attributes [{:attribute "nickname" :name {:en "Nickname" :fi "Lempinimi"}}
                          {:attribute "picture"}
-                         {:attribute "organizations"}]
+                         {:attribute "organizations"}
+                         {:attribute "groups" :name {:en "Groups" :fi "Ryhmät" :sv "Grupper"}}]
  :languages [:en :fi :sv]
  :public-url "http://localhost:3000/"
  :extra-pages [{:id "about"
@@ -83,4 +84,26 @@
  :enable-duo true
  :enable-catalogue-tree true
  :enable-save-compaction true
- :enable-autosave true}
+ :enable-autosave true
+
+ ;; let's use plugins in dev
+ :oidc-require-name false
+ :oidc-require-email false
+ :plugins [{:id :plugin/AARC-G069-group-split
+            :filename "resources/plugins/AARC-G069-group-split.md"
+            :attribute-name "eduperson_entitlement"
+            :trusted-authorities ["perun.aai.lifescience-ri.eu"]}
+
+           {:id :plugin/validate-attributes
+            :filename "resources/plugins/validate-attributes.md"
+            :required-attributes [{:attribute-name "name" :error-key :t.login.errors/name}
+                                  {:attribute-name "email" :error-key :t.login.errors/email}]}
+
+           {:id :plugin/validate-group-membership ; NB: not in use at the moment
+            :filename "resources/plugins/validate-group-membership.md"
+            :attribute-name "groups"
+            :valid-groups ["VO1"]
+            :error-key :t.login.errors/group}]
+
+ :extension-points {:extension-point/transform-user-data [:plugin/AARC-G069-group-split]
+                    :extension-point/validate-user-data [:plugin/validate-attributes]}}
diff --git a/docs/hooks.md b/docs/hooks.md
index 98d3875d5..d6a5406fb 100644
--- a/docs/hooks.md
+++ b/docs/hooks.md
@@ -1,5 +1,7 @@
 # Hooks
 
+See also [plugins](./plugins.md).
+
 ## Extra scripts
 
 You can configure REMS to include extra script files by defining configuration such as:
diff --git a/docs/plugins.md b/docs/plugins.md
new file mode 100644
index 000000000..3828e3768
--- /dev/null
+++ b/docs/plugins.md
@@ -0,0 +1,87 @@
+# Plugins
+
+REMS can be extended with plugins in certain extension points.
+
+The plugins are loaded dynamically from external code found in the specified file.
+
+There should be a function defined in the plugin, that is called at the right time. The name of the function
+depends on the type of the extension point.
+
+All the functions receive `config` where the plugin's configuration is (from the config file). Also context
+specific `data` will be passed.
+
+Certain types of libraries have been exposed to plugins. Please post an issue if you want more added.
+
+Examples of plugins can be found in the `resources/plugins` directory.
+
+## Types of plugins
+
+### Transform
+
+Take `data` and do any kind of transformations to it and return the new `data`.
+
+Return the original data if nothing should be done.
+
+```clj
+(defn transform [config data]
+  ...)
+```
+
+### Process
+
+Processes the passed `data` for side-effects, such as integration to another server using HTTP requests.
+
+Returns the errors so an empty sequence (`nil` or `[]` for example) should be returned if everything was good.
+Any errors will prevent the possible next process from running.
+
+In the case of a failure in processing of the same `data`, the process will be retried again, so the implementation should be idempotent. A retry can also happen for a successful process, if a processing plugin configured after this plugin fails.
+
+```clj
+(defn process [config data]
+  ...)
+```
+
+### Validate
+
+Validates the passed `data`.
+
+Returns the errors so an empty sequence (`nil` or `[]` for example) should be returned if everything was good.
+Any errors will prevent the possible next validation from running, and generally the action from happening (e.g. logging in).
+
+```clj
+(defn validate [config data]
+  ...)
+```
+
+## Extension points
+
+Next are all the current extension points and the function they expect to find in the plugin.
+
+### `:extension-point/transform-user-data`
+
+After logging in, after opening the OIDC token and potentially fetching the user info, allow transforming that data further. For example, a complex field can be parsed and the result stored in new fields.
+
+Expects `transform` function.
+
+See [AARC-G069-group-split.md](../resources/plugins/AARC-G069-group-split.md)
+
+### `:extension-point/validate-user-data`
+
+After logging in, after the user data is finalized, allow validating it to prevent invalid users from logging in.
+
+Expects `validate` function.
+
+The first returned error will be shown to the user on an error page. It can have the keys:
+- `:key` The translation key of the error message such as `:t.login.errors/invalid-user` (possibly from extra translations provided).
+- `:args` Additional arguments for the message translation (`%1`,`%2`, ...) if any. These must be translation keys too.
+
+See [validate-attributes.md](../resources/plugins/validate-attributes.md)
+
+### `:extension-point/process-entitlements`
+
+After entitlements have been updated, the new entitlements can be processed, and for example updated to
+another system.
+
+Expects `process` function.
+
+See [LS-AAI-GA4GH-push.md](../resources/plugins/LS-AAI-GA4GH-push.md)
diff --git a/example-theme/extra-translations/en.edn b/example-theme/extra-translations/en.edn
index 398f57d64..4c21b4e3b 100644
--- a/example-theme/extra-translations/en.edn
+++ b/example-theme/extra-translations/en.edn
@@ -2,4 +2,5 @@
  {:footer "CSC – Overridden in Extra Translations"
   ;; the extra %1 and the :unused-key are for rems.test-locales
   :create-license {:license-text "Text %1"}
-  :unused-key "Unused"}}
+  :unused-key "Unused"
+  :login {:errors {:group "group"}}}}
diff --git a/project.clj b/project.clj
index d192c9777..4ac68611f 100644
--- a/project.clj
+++ b/project.clj
@@ -61,7 +61,9 @@
                  [ring/ring-defaults "0.3.4"]
                  [ring/ring-devel "1.9.6"]
                  [ring/ring-servlet "1.9.6"]
-                 [nano-id "1.0.0"]]
+                 [nano-id "1.0.0"]
+                 [org.babashka/sci "0.7.39"]
+                 [com.nextjournal/beholder "1.0.2"]]
 
   :min-lein-version "2.9.8"
 
@@ -83,6 +85,7 @@
   :antq {}
 
   :cljfmt {:paths ["project.clj" "src/clj" "src/cljc" "src/cljs" "test/clj" "test/cljc" "test/cljs"] ; need explicit paths to include cljs
+           :indents {delay [[:inner 0]]}
            :remove-consecutive-blank-lines? false} ; too many changes for now, probably not desirable
 
   :clean-targets ["target"]
diff --git a/resources/config-defaults.edn b/resources/config-defaults.edn
index 5f47b2422..4890d7128 100644
--- a/resources/config-defaults.edn
+++ b/resources/config-defaults.edn
@@ -446,4 +446,23 @@
  ;; :pdf-metadata {:font {:encoding :unicode
  ;;                       :ttf-name "custom-font-file.ttf"}}
  ;;
- :pdf-metadata {}}
+ :pdf-metadata {}
+
+ ;; REMS can be extended using plugins:
+ ;;
+ ;; Define the plugins with `:plugins` and then
+ ;; attach them to the extension points using `:extension-points`.
+ ;;
+ ;; Example:
+ ;; :plugins [{:id :plugin/group-split
+ ;;            :filename "plugins/group-split.md"
+ ;;            :attribute-name "user_groups"
+ ;;            :split-with ";"}
+ ;;
+ ;; :extension-points {:extension-point/transform-user-data [:plugin/group-split] ; should match the previous id
+ ;;                    :extension-point/validate-user-data [...
+ ;;                                                         ]}
+ ;;
+ ;; See docs/plugins.md for more information.
+ :plugins nil ; by default no plugins
+ :extension-points nil} ; by default no plugins
diff --git a/resources/plugins/AARC-G069-group-split.md b/resources/plugins/AARC-G069-group-split.md
new file mode 100644
index 000000000..9ade07a6f
--- /dev/null
+++ b/resources/plugins/AARC-G069-group-split.md
@@ -0,0 +1,72 @@
+# AARC-G069 group parse
+
+A plugin to parse group attributes according to AARC-G069.
+
+The format comes from AARC-G069 as so:
+
+    <NAMESPACE>:group:<GROUP>[:<SUBGROUP>*][:role=<ROLE>][#<AUTHORITY>]
+
+The OIDC attribute name that is parsed is configured in the `:attribute-name` config. With these example attibutes:
+
+    urn:geant:lifescience-ri.eu:group:example-vo.lifescience-ri.eu#aai.lifescience-ri.eu
+    urn:geant:elixir-europe.org:group:elixir:ELIXIR%20AAI:staff#perun.elixir-czech.cz
+
+The <NAMESPACE> is:
+
+    urn:geant:lifescience-ri.eu
+    urn:geant:elixir-europe.org
+
+The <AUTHORITY> comes after `#`:
+
+    aai.lifescience-ri.eu
+    perun.elixir-czech.cz
+
+The part `:group:` is just a splitter like the `#` is.
+
+The middle part are the <GROUP> (and <SUBGROUP>s) of the person:
+
+    example-vo.lifescience-ri.eu (only group)
+    elixir:ELIXIR%20AAI:staff (group "elixir", subgroup "ELIXIR AAI", subgroup "staff")
+
+The <ROLE> is not present in the example but it is currently handled as just another subgroup. So
+a `role=R` would literally be just an additional `role=R` group.
+
+For a group to be valid the authority must declared in config `:trusted-authorities`.
+
+```clj
+(require '[clojure.string :as str])
+(require '[clojure.tools.logging :as log])
+(require '[rems.config :refer [env]])
+(require '[rems.common.util :refer [getx]])
+
+(defn transform [config data]
+  (when (:log-authentication-details env)
+    (log/info "Data" data))
+
+  (let [attribute-name (getx config :attribute-name)]
+    (if-some [attribute-value (get data (keyword attribute-name))]
+      (let [group-index (str/index-of attribute-value ":group:")
+            authority-index (str/index-of attribute-value "#")]
+
+        (when (:log-authentication-details env)
+          (log/info "Indices" group-index authority-index))
+
+        (if (and group-index
+                 authority-index
+                 (< -1 group-index authority-index))
+          (let [namespace (subs attribute-value 0 group-index)
+                groups-and-role (subs attribute-value (+ group-index (count ":group:")) authority-index)
+                authority (subs attribute-value (+ authority-index (count "#")))
+
+                groups (str/split groups-and-role #":")]
+
+            (when (:log-authentication-details env)
+              (log/info "Parts" namespace groups authority))
+
+            ;; TODO: check trusted authority
+
+            (assoc data :groups groups))
+          data))
+      data)))
+
+```
diff --git a/resources/plugins/LS-AAI-GA4GH-push.md b/resources/plugins/LS-AAI-GA4GH-push.md
new file mode 100644
index 000000000..652ff0ed9
--- /dev/null
+++ b/resources/plugins/LS-AAI-GA4GH-push.md
@@ -0,0 +1,11 @@
+# LifeScience AAI GA4GH push
+
+A plugin to push REMS entitlements to LifeScience AAI using GA4GH visas.
+
+See [GA4GH](../../docs/ga4gh-visas.md).
+
+```clj
+(defn process [config data]
+  ;; TODO implement
+  )
+```
diff --git a/resources/plugins/validate-attributes.md b/resources/plugins/validate-attributes.md
new file mode 100644
index 000000000..76b55411a
--- /dev/null
+++ b/resources/plugins/validate-attributes.md
@@ -0,0 +1,22 @@
+# Validate attributes
+
+A plugin to validate that specified attributes are present and not empty strings.
+
+```clj
+(require '[clojure.string :as str])
+(require '[clojure.tools.logging :as log])
+
+(defn empty-attributes [user attributes]
+  (for [{:keys [attribute-name error-key]} attributes
+        :let [attribute-value (get user (keyword attribute-name))
+              error? (str/blank? attribute-value)]
+        :when error?]
+    error-key))
+
+(defn validate [config data]
+  (when-some [invalid-attributes (seq (empty-attributes data (get config :required-attributes)))]
+    (log/info invalid-attributes)
+    [{:key :t.login.errors/invalid-user
+      :args invalid-attributes}]))
+
+```
diff --git a/resources/plugins/validate-group-membership.md b/resources/plugins/validate-group-membership.md
new file mode 100644
index 000000000..0c5d3e358
--- /dev/null
+++ b/resources/plugins/validate-group-membership.md
@@ -0,0 +1,22 @@
+# Validate group membership
+
+Check that the specified group attribute contains at least
+one of the valid groups.
+
+```clj
+(require '[rems.config :refer [env]])
+(require '[clojure.string :as str])
+(require '[clojure.tools.logging :as log])
+
+(defn validate [config data]
+  (let [{:keys [attribute-name valid-groups error-key]} config
+        groups (get data (keyword attribute-name))]
+
+    (when (:log-authentication-details env)
+      (log/info "Groups" groups))
+
+    (when (or (empty? groups)
+              (empty? (clojure.set/intersection (set groups) valid-groups)))
+      [{:key error-key}])))
+
+```
diff --git a/src/clj/rems/auth/fake_login.clj b/src/clj/rems/auth/fake_login.clj
index 9538277e1..e0ccd14ee 100644
--- a/src/clj/rems/auth/fake_login.clj
+++ b/src/clj/rems/auth/fake_login.clj
@@ -6,6 +6,7 @@
             [rems.common.util :refer [escape-element-id]]
             [rems.config :refer [env]]
             [rems.db.test-data-users :as test-data-users]
+            [rems.plugins :as plugins]
             [ring.util.response :refer [redirect]]))
 
 (defn get-fake-user-descriptions []
@@ -44,6 +45,7 @@
   (let [id-data (get-fake-id-data username)
         user-info (get-fake-user-info username)
         user-data (merge id-data user-info)
+        user-data (plugins/transform :extension-point/transform-user-data user-data)
         user (oidc/find-or-create-user! user-data)]
     (-> (redirect "/redirect")
         (assoc :session session)
@@ -95,3 +97,4 @@
 (defroutes routes
   (GET (login-url) req (fake-login-screen req))
   (GET (logout-url) req (fake-logout req)))
+
diff --git a/src/clj/rems/auth/oidc.clj b/src/clj/rems/auth/oidc.clj
index 5c6a25868..5dd427087 100644
--- a/src/clj/rems/auth/oidc.clj
+++ b/src/clj/rems/auth/oidc.clj
@@ -11,6 +11,7 @@
             [rems.ga4gh :as ga4gh]
             [rems.json :as json]
             [rems.jwt :as jwt]
+            [rems.plugins :as plugins]
             [rems.util :refer [getx]]
             [ring.util.response :refer [redirect]])
   (:import [java.time Instant]))
@@ -91,6 +92,7 @@
     user))
 
 (defn- get-user-attributes [user-data]
+  ;; XXX: consider using a plugin for the renaming
   ;; TODO all attributes could support :rename
   (let [userid (or (find-user user-data) (get-new-userid user-data))
         _ (assert userid (when (:log-authentication-details env) {:user-data user-data}))
@@ -104,12 +106,19 @@
 ;; XXX: consider joining with rems.db.users/invalid-user?
 (defn- validate-user! [user]
   ;; userid already checked
+  ;; XXX: we can migrate this old way to the new plugin system
   (when-let [errors (seq (remove nil?
                                  [(when (and (:oidc-require-name env) (str/blank? (:name user))) :t.login.errors/name)
                                   (when (and (:oidc-require-email env) (str/blank? (:email user))) :t.login.errors/email)]))]
     (throw (ex-info "Invalid user"
                     {:key :t.login.errors/invalid-user
                      :args errors
+                     :user user})))
+
+  (when-let [errors (seq (remove nil? (plugins/validate :extension-point/validate-user-data user)))]
+    (throw (ex-info "Invalid user"
+                    {:key (or (:key (first errors)) :t.login.errors/invalid-user)
+                     :args (:args (first errors))
                      :user user}))))
 
 (defn find-or-create-user! [user-data]
@@ -165,9 +174,12 @@
                                 json/parse-string))
                 researcher-status (ga4gh/passport->researcher-status-by user-info)
                 user-data (merge id-data user-info researcher-status)
+                user-data (plugins/transform :extension-point/transform-user-data user-data)
                 user (find-or-create-user! user-data)]
+
             (when (:log-authentication-details env)
               (log/info "logged in" user-data user))
+
             (-> (redirect "/redirect")
                 (assoc :session (:session request))
                 (assoc-in [:session :access-token] access-token)
diff --git a/src/clj/rems/db/entitlements.clj b/src/clj/rems/db/entitlements.clj
index 8021cdfe2..db1d65e64 100644
--- a/src/clj/rems/db/entitlements.clj
+++ b/src/clj/rems/db/entitlements.clj
@@ -12,6 +12,7 @@
             [rems.db.outbox :as outbox]
             [rems.ga4gh :as ga4gh]
             [rems.json :as json]
+            [rems.plugins :as plugins]
             [rems.scheduler :as scheduler]
             [rems.service.ega :as ega]))
 
@@ -27,6 +28,10 @@
 
 (defn- post-entitlements! [{:keys [action type entitlements config]}]
   (case type
+    :plugin
+    (plugins/process :extension-point/process-entitlements entitlements)
+
+    ;; TODO consider removing in favour of plugins
     :ega
     (when config
       (try (doseq [entitlement entitlements] ; technically these could be grouped per user & api-key
@@ -34,8 +39,9 @@
            (catch Exception e
              (log/error "POST failed" e)
              (or (ex-data e)
-                 {:status "exception"}))))
+                 [{:status "exception"}]))))
 
+    ;; TODO consider removing in favour of plugins
     :basic ; TODO: let's move this :entitlements-target (v1) at some point to :entitlement-post (v2)
     (when-let [target (get-in env [:entitlements-target action])]
       (let [payload (get-entitlements-payload entitlements action)
@@ -57,7 +63,7 @@
             (log/infof "Posted entitlements to %s: %s -> %s" target payload status)
             (do
               (log/warnf "Entitlement post failed: %s", response)
-              (str "failed: " status))))))))
+              [(str "failed: " status)])))))))
 
 ;; TODO argh adding these everywhere sucks
 ;; TODO consider using schema coercions
@@ -71,8 +77,8 @@
   (doseq [entry (mapv fix-entry-from-db
                       (outbox/get-due-entries :entitlement-post))]
     ;; TODO could send multiple entitlements at once instead of one outbox entry at a time
-    (if-let [error (post-entitlements! (:outbox/entitlement-post entry))]
-      (let [entry (outbox/attempt-failed! entry error)]
+    (if-let [errors (post-entitlements! (:outbox/entitlement-post entry))]
+      (let [entry (outbox/attempt-failed! entry errors)]
         (when (not (:outbox/next-attempt entry))
           (log/warn "all attempts to send entitlement post id " (:outbox/id entry) "failed")))
       (outbox/attempt-succeeded! (:outbox/id entry)))))
@@ -101,6 +107,7 @@
     ;; TODO could generate only one outbox entry per application. Currently one per user-resource pair.
     (let [entitlements (db/get-entitlements {:application application-id :user user-id :resource resource-id})]
       (add-to-outbox! :add :basic entitlements nil)
+      (add-to-outbox! :add :plugin entitlements nil)
       (doseq [config (:entitlement-push env)]
         (add-to-outbox! :add (:type config) entitlements config)))))
 
diff --git a/src/clj/rems/markdown.clj b/src/clj/rems/markdown.clj
new file mode 100644
index 000000000..e90de0dcf
--- /dev/null
+++ b/src/clj/rems/markdown.clj
@@ -0,0 +1,91 @@
+(ns rems.markdown
+  (:require [clojure.string :as str]
+            [clojure.test :refer [deftest is]]))
+
+(defn- starts-clj-block? [line]
+  (or (str/starts-with? (str/triml line) "```clj")
+      (str/starts-with? (str/triml line) "```clojure")))
+
+(defn- ends-clj-block? [line]
+  (str/starts-with? (str/triml line) "```"))
+
+(defn strip-to-clj-content
+  "Strips every non-Clojure code line from the content in `lines`.
+
+  Preserves the lines so that empty blank lines will be injected
+  instead of removing them completely. This is to preserve line
+  (line-seq )numbers in exceptions."
+  [lines]
+  (let [blank-line ""
+        newline "\n"]
+    (loop [lines lines
+           clojure-code-block? false
+           result-lines []]
+      (let [line (first lines)]
+        (cond (not line) ; eof
+              (mapv #(str % newline) result-lines)
+
+              (and (not clojure-code-block?)
+                   (starts-clj-block? line))
+              (recur (rest lines)
+                     true
+                     (conj result-lines blank-line))
+
+              (and clojure-code-block?
+                   (ends-clj-block? line))
+              (recur (rest lines)
+                     false
+                     (conj result-lines blank-line))
+
+              clojure-code-block?
+              (recur (rest lines)
+                     true
+                     (conj result-lines line))
+
+              :else
+              (recur (rest lines)
+                     false
+                     (conj result-lines blank-line)))))))
+
+(deftest test-strip-to-clj-content
+  (is (= ["\n"] (strip-to-clj-content [""])))
+
+  (is (= ["\n"
+          "\n"
+          "\n"]
+         (strip-to-clj-content ["# title"
+                                ""
+                                "content"]))
+      "produces empty lines for each stripped line")
+
+  (is (= ["\n"
+          "\n"
+          "(+ 1\n"
+          "   2\n"
+          "   3)\n"
+          "\n"
+          "\n"]
+         (strip-to-clj-content ["```clj"
+                                ""
+                                "(+ 1"
+                                "   2"
+                                "   3)"
+                                ""
+                                "```"]))
+      "produces empty lines for each stripped line")
+
+  (is (= ["\n"
+          "\n"
+          "\n"
+          "(fun 1\n"
+          "     2 (fan 3))\n"
+          "\n"
+          "\n"]
+         (strip-to-clj-content ["# code"
+                                ""
+                                "```clj"
+                                "(fun 1"
+                                "     2 (fan 3))"
+                                "```"
+                                "the end"]))
+      "keeps Clojure code intact"))
diff --git a/src/clj/rems/middleware.clj b/src/clj/rems/middleware.clj
index 0e2cbe3a1..3e1f5d5fe 100644
--- a/src/clj/rems/middleware.clj
+++ b/src/clj/rems/middleware.clj
@@ -9,7 +9,6 @@
             [mount.core :as mount]
             [nano-id.core :as nano-id]
             [rems.auth.auth :as auth]
-            [rems.auth.util :refer [throw-forbidden throw-unauthorized]]
             [rems.common.util :refer [assoc-some-in]]
             [rems.config :refer [env]]
             [rems.context :as context]
diff --git a/src/clj/rems/plugins.clj b/src/clj/rems/plugins.clj
new file mode 100644
index 000000000..e4845030d
--- /dev/null
+++ b/src/clj/rems/plugins.clj
@@ -0,0 +1,207 @@
+(ns rems.plugins
+  "REMS can be extended with plugins in certain extension points.
+
+  The plugins are loaded dynamically from external
+  code found in the specified file.
+
+  See also docs/plugins.md"
+  (:require [clj-http.client]
+            [clojure.java.io :as io]
+            [clojure.tools.logging :as log]
+            [clojure.string :as str]
+            [mount.core :as mount]
+            [nextjournal.beholder :as beholder]
+            [rems.common.util :refer [getx index-by]]
+            [rems.config :refer [env]]
+            [rems.markdown :as md]
+            [sci.core :as sci]))
+
+;; logging is difficult to expose so we wrap it
+
+(def ^:dynamic *logging-ctx*)
+
+(defn log-wrapper
+  "Wraps the `clojure.tools.logging` macros and presents helper functions for plugins.
+
+  The keys `extension-point-id` and `plugin-id` are automatically included in the logs from the
+  `*logging-ctx*`."
+  [level]
+  (fn [x & args]
+    (let [{:keys [extension-point-id plugin-id]} *logging-ctx*]
+      (if (instance? Throwable x)
+        (log/log level x (apply print-str extension-point-id plugin-id "-" args))
+        (log/log level (apply print-str extension-point-id plugin-id "-" x args))))))
+
+
+
+
+
+;;; SCI
+
+(def sci-ctx
+  "Default SCI context for plugins."
+  (delay
+    (sci/init {:namespaces {'rems.config {'env env}
+                            'clojure.string (sci/copy-ns clojure.string (sci/create-ns 'clojure.string))
+                            'clj-http.client (sci/copy-ns clj-http.client (sci/create-ns 'clj-http.client))
+                            'clojure.tools.logging {'trace (log-wrapper :trace)
+                                                    'debug (log-wrapper :debug)
+                                                    'info (log-wrapper :info)
+                                                    'warn (log-wrapper :warn)
+                                                    'error (log-wrapper :error)
+                                                    'fatal (log-wrapper :fatal)}
+
+                            'rems.common.util {'getx getx}}})))
+
+;; make sure SCI printing goes to stdout
+(sci/alter-var-root sci/out (constantly *out*))
+(sci/alter-var-root sci/err (constantly *err*))
+
+
+
+
+;;; Plugins
+
+;; TODO: use common cache architecture?
+(def plugin-configs (atom {}))
+(def plugin-ctx-cache (atom {}))
+
+(defn- load-plugin-configs! []
+  (reset! plugin-configs
+          (->> env
+               :plugins
+               (index-by [:id]))))
+
+(defn- get-plugin-config [plugin-id]
+  (getx @plugin-configs plugin-id))
+
+(defn- get-plugins-at [extension-point-id]
+  (->> (get-in env [:extension-points extension-point-id])
+       (mapv get-plugin-config)))
+
+(defn- load-plugin [plugin-config]
+  (let [plugin-filename (getx plugin-config :filename)
+        _ (log/info "Loading plugin" (:id plugin-config) "from" (pr-str plugin-filename))
+        ctx (-> @sci-ctx sci/fork)]
+    (-> plugin-filename
+        io/reader
+        line-seq
+        md/strip-to-clj-content
+        str/join
+        (->> (sci/eval-string* ctx))) ; evaluate plugin (defs) into context
+    ctx))
+
+(defn- load-plugins! []
+  (reset! plugin-ctx-cache {})
+  (doseq [plugin-config (vals @plugin-configs)
+          :let [plugin-id (getx plugin-config :id)]]
+    (swap! plugin-ctx-cache assoc plugin-id (load-plugin plugin-config))))
+
+
+(defn- plugin-files-changed [{:keys [path] :as _event}]
+  (let [filename (.toString (.toAbsolutePath path))
+        _ (log/debug "Changed plugin file" filename)
+        changed-plugins (for [plugin-config (:plugins env)
+                              :when (.endsWith path (:filename plugin-config))]
+                          plugin-config)]
+    (doseq [plugin-config changed-plugins
+            :let [plugin-id (:id plugin-config)]]
+      (swap! plugin-ctx-cache assoc plugin-id (load-plugin plugin-config)))))
+
+(defn- remove-filename [filename]
+  (.getParent (io/file filename)))
+
+
+
+
+;;; Plugins (re-)loaded as files change
+
+(mount/defstate plugin-loader
+  :start (let [plugin-paths (->> env
+                                 :plugins
+                                 (mapv :filename)
+                                 (mapv remove-filename)
+                                 (into #{}))]
+           (log/info "Loading plugins from:" (pr-str plugin-paths))
+           ;; NB: while plugins can be reloaded
+           ;; the plugin configs are read only at mount
+           ;; start as they are part of the config
+           (load-plugin-configs!)
+           (load-plugins!)
+           (apply beholder/watch plugin-files-changed plugin-paths))
+  :stop (beholder/stop plugin-loader))
+
+
+
+
+
+;;; Internal entrypoint
+
+(defn- run-plugin
+  "Run the plugin from `plugin-config` passing in `data` at `extension-point-id` (context used for logging).
+
+  The plugin runs through SCI with some context injected (data, config, env, log etc.)"
+  [symbol-name extension-point-id data plugin-config]
+  (let [plugin-id (getx plugin-config :id)]
+    (log/debug "Running plugin" plugin-id)
+    (let [plugin (getx @plugin-ctx-cache plugin-id)
+          ctx (-> plugin
+                  sci/fork
+                  (sci/merge-opts {:namespaces {'user {'config plugin-config
+                                                       'data data}}}))]
+      (binding [*logging-ctx* {:extension-point-id extension-point-id
+                               :plugin-id plugin-id}]
+        (sci/eval-string* ctx (str "(" symbol-name " config data)"))))))
+
+
+
+
+
+;;; Public API
+
+(defn process
+  "Executes the configured plugins of `extension-point-id` to process the given `data`.
+
+  Each plugin gets a turn with the same original `data`.
+
+  The plugins will be executed in turn, in the same order as they have been configured.
+  The plugins are expected to do something side-effectful, like a HTTP request.
+
+  Each plugin should return a sequence of errors. The first non-empty result will
+  be returned (the first errors encountered) or `nil` (no errors). This would
+  will cause the processing to be re-tried later so you should prefer idempotent
+  implementations and external services."
+  [extension-point-id data]
+  (let [plugin-configs (get-plugins-at extension-point-id)]
+    (some (fn [plugin-config]
+            (seq (run-plugin "process" extension-point-id data plugin-config))) ; can return n problems
+          plugin-configs)))
+
+(defn transform
+  "Reduces the configured plugins of `extension-point-id` to transform the given `data`.
+
+  Each plugin should return the same or modified `data`.
+
+  The plugins will be executed in turn, in the same order as they have been configured. So
+  the return value of the one plugin will be the `data` of the next.
+
+  Returns whatever the last plugin returns."
+  [extension-point-id data]
+  (let [plugin-configs (get-plugins-at extension-point-id)]
+    (reduce (partial run-plugin "transform" extension-point-id)
+            data
+            plugin-configs)))
+
+(defn validate
+  "Validates `data` with the plugins of `extension-point-id`.
+
+  The plugins will be executed in turn, in the same order as they have been configured.
+
+  Each plugin should return a sequence of errors. The first non-empty result will
+  be returned (the first errors encountered) or `nil` (no errors)."
+  [extension-point-id data]
+  (let [plugin-configs (get-plugins-at extension-point-id)]
+    (some (fn [plugin-config]
+            (seq (run-plugin "validate" extension-point-id data plugin-config))) ; can return n problems
+          plugin-configs)))
+


Test patch:
diff --git a/src/clj/rems/db/test_data_users.clj b/src/clj/rems/db/test_data_users.clj
index 054455101..8f8a9f92e 100644
--- a/src/clj/rems/db/test_data_users.clj
+++ b/src/clj/rems/db/test_data_users.clj
@@ -68,7 +68,7 @@
    "new-user" {:sub "new-user" :email "new-user@example.com" :name "New User"}})
 
 (def +fake-user-info+
-  {"alice" {:researcher-status-by "so"}
+  {"alice" {:researcher-status-by "so" :eduperson_entitlement "urn:lifescience-ri.eu:group:VO1:B:1:role=R#perun.aai.lifescience-ri.eu"}
    "elixir-alice" {:researcher-status-by "so"}})
 
 (def +user-descriptions+
diff --git a/test-data/example-request-plugin.md b/test-data/example-request-plugin.md
new file mode 100644
index 000000000..16468a564
--- /dev/null
+++ b/test-data/example-request-plugin.md
@@ -0,0 +1,17 @@
+# HTTP request plugin
+
+This plugin does a HTTP request. Useful for tests.
+
+```clj
+(require '[clj-http.client :as client])
+
+(defn process [config data]
+  (assert (:url config))
+  (assert (:value data))
+  (let [response (client/post (:url config)
+                              {:accept :json
+                               :form-params (select-keys data [:value])
+                               :headers {"x-a-header" (:value data)}})]
+    response))
+
+```
diff --git a/test-data/exception-plugin.md b/test-data/exception-plugin.md
new file mode 100644
index 000000000..c8e71dfb5
--- /dev/null
+++ b/test-data/exception-plugin.md
@@ -0,0 +1,9 @@
+# Exception plugin
+
+This plugin throws an exception when used. Useful for tests.
+
+```clj
+(defn transform [config data]
+  (throw (Exception. "hello from plugin")))
+
+```
diff --git a/test-data/fail-neg-plugin.md b/test-data/fail-neg-plugin.md
new file mode 100644
index 000000000..51742d121
--- /dev/null
+++ b/test-data/fail-neg-plugin.md
@@ -0,0 +1,10 @@
+# Fail neg plugin
+
+This plugin validates that the ``:value` is positive`. Useful for tests.
+
+```clj
+(defn validate [config data]
+  (when-not (pos? (:value data))
+    [{:result :fail :reason :neg}])) ; could return multiple errors
+
+```
diff --git a/test-data/fail-odd-plugin.md b/test-data/fail-odd-plugin.md
new file mode 100644
index 000000000..53a84d95a
--- /dev/null
+++ b/test-data/fail-odd-plugin.md
@@ -0,0 +1,10 @@
+# Fail odd plugin
+
+This plugin validates that the ``:value` is even`. Useful for tests.
+
+```clj
+(defn validate [config data]
+  (when (odd? (:value data))
+    [{:result :fail :reason :odd}])) ; could return multiple errors
+
+```
diff --git a/test-data/fail-process-plugin.md b/test-data/fail-process-plugin.md
new file mode 100644
index 000000000..ab8384f10
--- /dev/null
+++ b/test-data/fail-process-plugin.md
@@ -0,0 +1,9 @@
+# Fail process plugin
+
+This plugin fails processing. Useful for tests.
+
+```clj
+(defn process [config data]
+  [{:result :fail}]) ; could return multiple errors
+
+```
diff --git a/test-data/failing-plugin.md b/test-data/failing-plugin.md
new file mode 100644
index 000000000..2c6320db7
--- /dev/null
+++ b/test-data/failing-plugin.md
@@ -0,0 +1,11 @@
+# Failing plugin
+
+This plugin fails when used. Useful for tests.
+
+```clj
+(+ 1 2 3)
+
+(+ 1 2
+   (invalid code here))
+
+```
diff --git a/test-data/http-request-plugin.md b/test-data/http-request-plugin.md
new file mode 100644
index 000000000..67d6500b6
--- /dev/null
+++ b/test-data/http-request-plugin.md
@@ -0,0 +1,14 @@
+# HTTP request plugin
+
+This plugin does a HTTP request. Useful for tests.
+
+```clj
+(require ['clj-http.client :as client])
+
+(defn process [config data]
+  (let [response (client/get "http://example.com/process/123"
+                             {:accept :json
+                              :headers "X-a-header" "value"})]
+    response))
+
+```
diff --git a/test-data/identity-plugin.md b/test-data/identity-plugin.md
new file mode 100644
index 000000000..bef6f1459
--- /dev/null
+++ b/test-data/identity-plugin.md
@@ -0,0 +1,9 @@
+# Identity plugin
+
+This plugin does nothing interesting. Useful for tests.
+
+```clj
+(defn transform [config data]
+  data)
+
+```
diff --git a/test-data/increment-plugin.md b/test-data/increment-plugin.md
new file mode 100644
index 000000000..0ce6f34c9
--- /dev/null
+++ b/test-data/increment-plugin.md
@@ -0,0 +1,9 @@
+# Increment plugin
+
+This plugin increments the `:value`. Useful for tests.
+
+```clj
+(defn transform [config data]
+  (update data :value inc))
+
+```
diff --git a/test/clj/rems/db/test_entitlements.clj b/test/clj/rems/db/test_entitlements.clj
index 42e1aed46..671ac04c9 100644
--- a/test/clj/rems/db/test_entitlements.clj
+++ b/test/clj/rems/db/test_entitlements.clj
@@ -65,16 +65,16 @@
     (run-with-server
      {:status 404}
      (fn [_]
-       (is (= "failed: 404" (#'entitlements/post-entitlements! {:action :add :type :basic :entitlements +entitlements+}))))))
+       (is (= ["failed: 404"] (#'entitlements/post-entitlements! {:action :add :type :basic :entitlements +entitlements+}))))))
   (testing "timeout"
     (run-with-server
      {:status 200 :delay 5000} ;; timeout of 2500 in code
      (fn [_]
-       (is (= "failed: exception" (#'entitlements/post-entitlements! {:action :add :type :basic :entitlements +entitlements+}))))))
+       (is (= ["failed: exception"] (#'entitlements/post-entitlements! {:action :add :type :basic :entitlements +entitlements+}))))))
   (testing "invalid url"
     (with-redefs [rems.config/env (assoc rems.config/env
                                          :entitlements-target {:add "http://invalid/entitlements"})]
-      (is (= "failed: exception" (#'entitlements/post-entitlements! {:action :add :type :basic :entitlements +entitlements+})))))
+      (is (= ["failed: exception"] (#'entitlements/post-entitlements! {:action :add :type :basic :entitlements +entitlements+})))))
   (testing "no server configured"
     (is (nil? (#'entitlements/post-entitlements! {:action :add :type :basic :entitlements +entitlements+})))))
 
@@ -329,10 +329,10 @@
       (run-with-ega-server
        {"/p/permissions" {:status 200 :delay 5000}} ;; timeout of 2500 in code
        (fn [server]
-         (is (= {:status "exception"} (#'entitlements/post-entitlements! (post-data-for server)))))))
+         (is (= [{:status "exception"}] (#'entitlements/post-entitlements! (post-data-for server)))))))
 
     (testing "invalid url"
-      (is (= {:status "exception"} (#'entitlements/post-entitlements! (post-data-for {:uri "http://invalid/address/altogether"})))))
+      (is (= [{:status "exception"}] (#'entitlements/post-entitlements! (post-data-for {:uri "http://invalid/address/altogether"})))))
 
     (testing "no server configured means no problem"
       (is (= nil (#'entitlements/post-entitlements! (dissoc (post-data-for {:uri "http://invalid/address/no/problem"}) :config)))))))
diff --git a/test/clj/rems/test_plugins.clj b/test/clj/rems/test_plugins.clj
new file mode 100644
index 000000000..e16f0eead
--- /dev/null
+++ b/test/clj/rems/test_plugins.clj
@@ -0,0 +1,148 @@
+(ns rems.test-plugins
+  (:require [clojure.test :refer [deftest is testing]]
+            [rems.config :refer [env]]
+            [rems.json :as json]
+            [rems.plugins :as plugins]
+            [stub-http.core :as stub]))
+
+(deftest test-transform
+  (testing "without plugins configured"
+    (with-redefs [env {}]
+      (#'plugins/load-plugin-configs!)
+      (#'plugins/load-plugins!)
+      (is (= {} (plugins/transform :test {})))
+      (is (= {:anything 52} (plugins/transform :test {:anything 52})))))
+
+  (testing "with identity plugin"
+    (with-redefs [env {:plugins [{:id :plugin/test
+                                  :filename "test-data/identity-plugin.md"}]
+                       :extension-points {:test [:plugin/test]}}]
+      (#'plugins/load-plugin-configs!)
+      (#'plugins/load-plugins!)
+      (is (= {} (plugins/transform :test {})))
+      (is (= {:anything 52} (plugins/transform :test {:anything 52})))))
+
+  (testing "with increment plugin"
+    (with-redefs [env {:plugins [{:id :plugin/test
+                                  :filename "test-data/increment-plugin.md"}]
+                       :extension-points {:test [:plugin/test]}}]
+      (#'plugins/load-plugin-configs!)
+      (#'plugins/load-plugins!)
+      (is (= {:value 2} (plugins/transform :test {:value 1})))))
+
+  (testing "with compile failing plugin"
+    (with-redefs [env {:plugins [{:id :plugin/test
+                                  :filename "test-data/failing-plugin.md"}]
+                       :extension-points {:test [:plugin/test]}}]
+      (try
+        (#'plugins/load-plugin-configs!)
+        (#'plugins/load-plugins!)
+        ;; NB: error is thrown at load-time before plugin is used
+
+        (catch clojure.lang.ExceptionInfo e
+          (is (= {:type :sci/error
+                  :line 9
+                  :column 4
+                  :message "Could not resolve symbol: invalid"
+                  :file nil
+                  :phase "analysis"}
+                 (dissoc (ex-data e) :sci.impl/callstack))
+              "code analysis will fail and the location is reported")))))
+
+  (testing "with exception throwing plugin"
+    (with-redefs [env {:plugins [{:id :plugin/test
+                                  :filename "test-data/exception-plugin.md"}]
+                       :extension-points {:test [:plugin/test]}}]
+      (#'plugins/load-plugin-configs!)
+      (#'plugins/load-plugins!)
+
+      (try
+        (plugins/transform :test {})
+
+        (catch clojure.lang.ExceptionInfo e
+          (is (= {:type :sci/error
+                  :line 7
+                  :column 3
+                  :message "hello from plugin"
+                  :file nil}
+                 (dissoc (ex-data e) :sci.impl/callstack))
+              "the exception is wrapped fail and returned"))))))
+
+(deftest test-process
+  (with-open [server (stub/start! {"/" (fn [request]
+                                         {:status 200
+                                          :content-type "application/json"
+                                          :body (json/generate-string {:success true
+                                                                       :value (:value (:query-params request))
+                                                                       :header (get-in request [:headers :x-a-header])})})})]
+
+    (testing "with failing plugin"
+      (with-redefs [env {:plugins [{:id :plugin/fail-process
+                                    :filename "test-data/fail-process-plugin.md"}
+                                   {:id :plugin/http-example-request
+                                    :filename "test-data/example-request-plugin.md"
+                                    :url (:uri server)}]
+                         :extension-points {:test [:plugin/fail-process :plugin/http-example-request]}}]
+        (#'plugins/load-plugin-configs!)
+        (#'plugins/load-plugins!)
+
+        (is (empty? (stub/recorded-responses server))
+            (is (= [{:result :fail}]
+                   (plugins/process :test {:value 42}))
+                "the first process to return errors is where the execution stops (short-circuit evaluation)")))
+
+      (testing "with process plugin"
+        (with-redefs [env {:plugins [{:id :plugin/fail-process
+                                      :filename "test-data/fail-process-plugin.md"} ; NB: declared but not used
+                                     {:id :plugin/http-example-request
+                                      :filename "test-data/example-request-plugin.md"
+                                      :url (:uri server)}]
+                           :extension-points {:test [:plugin/http-example-request]}}]
+
+          (#'plugins/load-plugin-configs!)
+          (#'plugins/load-plugins!)
+
+          (plugins/process :test {:value 42})
+
+          (is (= {:success true
+                  :header "42"
+                  :value "42"}
+                 (json/parse-string (:body (first (stub/recorded-responses server)))))
+              "HTTP request succeeds"))))))
+
+(deftest test-validate
+  (testing "with validate plugin"
+    (with-redefs [env {:plugins [{:id :plugin/fail-odd
+                                  :filename "test-data/fail-odd-plugin.md"}
+                                 {:id :plugin/fail-neg
+                                  :filename "test-data/fail-neg-plugin.md"}]
+                       :extension-points {:test [:plugin/fail-odd :plugin/fail-neg]}}]
+      (#'plugins/load-plugin-configs!)
+      (#'plugins/load-plugins!)
+
+      (testing "failing at runtime"
+        (try
+          (plugins/validate :test {})
+
+          (catch clojure.lang.ExceptionInfo e
+            (is (= {:type :sci/error
+                    :line 7
+                    :column 9
+                    :message "Argument must be an integer: "
+                    :file nil}
+                   (dissoc (ex-data e) :sci.impl/callstack))
+                "code will fail at runtime and the location is reported"))))
+
+      (is (= nil (plugins/validate :test {:value 2}))
+          "pass both validate")
+
+      (is (= [{:result :fail :reason :odd}] (plugins/validate :test {:value 1}))
+          "first validate")
+
+      (is (= [{:result :fail :reason :neg}] (plugins/validate :test {:value -2}))
+          "first validate is a pass but the second fails")
+
+      (is (= [{:result :fail :reason :odd}]
+             (plugins/validate :test {:value -1}))
+          "the first validate to return errors is where the execution stops (short-circuit evaluation)"))))
+