Repository: privacyidea/privacyidea

Issue:
mangle event handler
# Background pain

In certain scenarios the input data is not given in an optimal way.

Especially in the case of usernames in realm this can be very complicated.

E.g. the UPN of a user may be ``givenname.surname@company.com`` and a user would authenticate with this UPN at a given application. But if this username hits privacyidea

1. in case of splitAtSign=False, the user ``givenname.surname@company.com`` can not be found in the default realm, since this user does not exist in the default realm, but in the realm ``company.com``.

2. In case of splitAtSign=True, the user might be searched in the realm ``company.com`` but the user ``givenname.surname`` can not be found in this realm, because the attribte ``givenname.surname`` does not exist. The sAMAccountname might be ``username`` and the UPN is ``givenname.surname@company.com``.

Thus we need a more flexible way to tweek or mangle attributes. In this case a request with 
* user = ``givenname.surname@company.com``
* realm = ""

must result in 

* user = ``givenname.surname@company.com``
* realm = ``company.com``
This way the UPN could be found in the non-default realm ``company.com``.

# Approach of the solution

We can create a mangle event handler, that acts as a pre-Event handler, that can 
* modify existing attributes
* delete existing attributes
* create new attributes based on existing attributes.

The "allowed_positions" would probably on be ``[pre]``.
https://github.com/privacyidea/privacyidea/blob/master/privacyidea/lib/eventhandler/base.py#L102

Possible actions could be:
* modify
  * options:
    * name of the parameter to match
       * regular expression to match the parameter  
    * name of the parameter to modify 
       * result how the changed value should look like
  (This way we could also either overwrite the existing parameter or create a new parameter)
* add
  * options:
     * name of the parameter to add
       *value to add
* delete
  * options:
    * name of the parameter to match
      * regular expression, when to delete
    * name of the parameter to delete
   (This way we could also either delete the matching parameter or check the matching of Paramter A and delete Parameter B)
 



Discussion:
I think "add" could also modify an existing parameter a.k.a. "modify" could add the new parameter, if it does not exist yet. So I would go with:

* add
  * options:
    * name of the parameter to match
       (Although this might look like a condition, we actually need this matching, since the result of the matching is used to add or modify the new parameter)
       * regular expression to match the parameter  
    * name of the parameter to add / modify 
       * result how the changed value should look like
  (This way we could also either overwrite the existing parameter or create a new parameter)
* delete
  A matching parameter in this case would be a condition to delete the parameter. So I would remove the matching here.
  * options:
    * name of the parameter to delete
 

Resolving patch:
diff --git a/privacyidea/api/event.py b/privacyidea/api/event.py
--- a/privacyidea/api/event.py
+++ b/privacyidea/api/event.py
@@ -67,6 +67,7 @@ def get_eventhandling(eventid=None):
     elif eventid == "handlermodules":
         # TODO: We need to provide a dynamic list of event handlers
         res = ["UserNotification", "Token", "Federation", "Script", "Counter"]
+        res.append("RequestMangler")
         res.append("ResponseMangler")
     else:
         res = g.event_config.get_event(eventid)
diff --git a/privacyidea/lib/event.py b/privacyidea/lib/event.py
--- a/privacyidea/lib/event.py
+++ b/privacyidea/lib/event.py
@@ -159,6 +159,7 @@ def get_handler_object(handlername):
     from privacyidea.lib.eventhandler.federationhandler import \
         FederationEventHandler
     from privacyidea.lib.eventhandler.counterhandler import CounterEventHandler
+    from privacyidea.lib.eventhandler.requestmangler import RequestManglerEventHandler
     from privacyidea.lib.eventhandler.responsemangler import ResponseManglerEventHandler
     h_obj = None
     if handlername == "UserNotification":
@@ -171,6 +172,8 @@ def get_handler_object(handlername):
         h_obj = FederationEventHandler()
     elif handlername == "Counter":
         h_obj = CounterEventHandler()
+    elif handlername == "RequestMangler":
+        h_obj = RequestManglerEventHandler()
     elif handlername == "ResponseMangler":
         h_obj = ResponseManglerEventHandler()
     return h_obj
diff --git a/privacyidea/lib/eventhandler/requestmangler.py b/privacyidea/lib/eventhandler/requestmangler.py
new file mode 100644
--- /dev/null
+++ b/privacyidea/lib/eventhandler/requestmangler.py
@@ -0,0 +1,174 @@
+# -*- coding: utf-8 -*-
+#
+#  2019-09-06 Cornelius Kölbel <cornelius.koelbel@netknights.it>
+#             Initial writup
+#
+# License:  AGPLv3
+# (c) 2019. Cornelius Kölbel
+#
+# This code is free software; you can redistribute it and/or
+# modify it under the terms of the GNU AFFERO GENERAL PUBLIC LICENSE
+# License as published by the Free Software Foundation; either
+# version 3 of the License, or any later version.
+#
+# This code is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU AFFERO GENERAL PUBLIC LICENSE for more details.
+#
+# You should have received a copy of the GNU Affero General Public
+# License along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+#
+__doc__ = """This is the event handler module modifying request parameters.
+"""
+from privacyidea.lib.eventhandler.base import BaseEventHandler
+from privacyidea.lib.token import (get_token_types, set_validity_period_end,
+                                   set_validity_period_start)
+from privacyidea.lib.realm import get_realms
+from privacyidea.lib.token import (set_realms, remove_token, enable_token,
+                                   unassign_token, init_token, set_description,
+                                   set_count_window, add_tokeninfo,
+                                   set_failcounter, delete_tokeninfo)
+from privacyidea.lib.utils import (parse_date, is_true,
+                                   parse_time_offset_from_now)
+from privacyidea.lib.tokenclass import DATE_FORMAT, AUTH_DATE_FORMAT
+from privacyidea.lib import _
+import json
+import logging
+import re
+import datetime
+import yaml
+from dateutil.parser import parse as parse_date_string
+from dateutil.tz import tzlocal
+
+log = logging.getLogger(__name__)
+
+
+class ACTION_TYPE(object):
+    """
+    Allowed actions
+    """
+    SET = "set"
+    DELETE = "delete"
+
+
+class RequestManglerEventHandler(BaseEventHandler):
+    """
+    An Eventhandler needs to return a list of actions, which it can handle.
+
+    It also returns a list of allowed action and conditions
+
+    It returns an identifier, which can be used in the eventhandlig definitions
+    """
+
+    identifier = "RequestMangler"
+    description = "This event handler can modify the parameters in the request."
+
+    @property
+    def allowed_positions(cls):
+        """
+        This returns the allowed positions of the event handler definition.
+        :return: list of allowed positions
+        """
+        """
+        Usually we would only modify the parameters in the PRE location, so that
+        the request is handled with the modified parameters. 
+        
+        But we could also modify the parameters in the POST location, so that the request
+        is handled with the original parameters, but *after* the request is handled,
+        some parameters can be changed to that an event handler, that is called *After*
+        the RequestMangler gets other input parameters.
+        
+        At the time of writing I can not make up a scenario, but technically it could 
+        make sense.
+        """
+        return ["post", "pre"]
+
+    @property
+    def actions(cls):
+        """
+        This method returns a dictionary of allowed actions and possible
+        options in this handler module.
+
+        :return: dict with actions
+        """
+        actions = {ACTION_TYPE.DELETE:
+                       {"parameter":
+                            {"type": "str",
+                             "required": True,
+                             "description": _("The parameter that should be deleted.")}
+                        },
+                   ACTION_TYPE.SET:
+                       {"parameter":
+                            {"type": "str",
+                             "required": True,
+                             "description": _("The parameter that should be added or modified.")
+                             },
+                        "value":
+                            {"type": "str",
+                             "required": True,
+                             "description": _("The new value of the parameter. Can contain tags like {0}, {1} for  "
+                                              "the matched sub strings.")
+                            },
+                        "match_parameter":
+                            {"type": "str",
+                             "description": _("The parameter, that should match some values.")
+                            },
+                        "match_pattern":
+                            {"type": "str",
+                             "description": _("The value of the match_parameter. It can contain a regular "
+                                              "expression and '()' to transfer values to the new parameter.")
+                            }
+                        }
+                   }
+        return actions
+
+    def do(self, action, options=None):
+        """
+        This method executes the defined action in the given event.
+
+        :param action:
+        :param options: Contains the flask parameters g, request, response
+            and the handler_def configuration
+        :type options: dict
+        :return:
+        """
+        ret = True
+        request = options.get("request")
+        handler_def = options.get("handler_def")
+        handler_options = handler_def.get("options", {})
+        parameter = handler_options.get("parameter")
+
+        if parameter:
+            if action.lower() == ACTION_TYPE.DELETE:
+                if parameter in request.all_data:
+                    del(request.all_data[parameter])
+            elif action.lower() == ACTION_TYPE.SET:
+                value = handler_options.get("value")
+                match_parameter = handler_options.get("match_parameter")
+                match_pattern = handler_options.get("match_pattern")
+                if value is not None:
+                    # We only take action, if we have a value, even an empty string "".
+                    if not match_parameter:
+                        # simple setting a parameter
+                        request.all_data[parameter] = value
+                    elif match_pattern and match_parameter in request.all_data:
+                        # setting a parameter depending on another value,
+                        # but only set it, if match_parameter exists
+                        """
+                        Note: Beware user supplied  format-string like "match_pattern", it can
+                        be dangerous: http://lucumr.pocoo.org/2016/12/29/careful-with-str-format/
+                        but in our case it is fine because no objects are involved, as m.groups() 
+                        always returns a tuple of strings
+                        """
+                        m = re.match("^" + match_pattern + "$", request.all_data.get(match_parameter))
+                        if m:
+                            # Now we set the new value with the matching tuple
+                            try:
+                                request.all_data[parameter] = value.format(*m.groups())
+                            except IndexError:
+                                log.warning(u"The number of found tags ({0!r}) "
+                                            u"do not match the required number ({1!r}).".format(m.groups(), value))
+
+        return ret


Test patch:
diff --git a/tests/test_lib_events.py b/tests/test_lib_events.py
--- a/tests/test_lib_events.py
+++ b/tests/test_lib_events.py
@@ -20,6 +20,7 @@
 from privacyidea.lib.eventhandler.responsemangler import ResponseManglerEventHandler
 from privacyidea.models import EventCounter, TokenOwner
 from privacyidea.lib.eventhandler.federationhandler import FederationEventHandler
+from privacyidea.lib.eventhandler.requestmangler import RequestManglerEventHandler
 from privacyidea.lib.eventhandler.base import BaseEventHandler, CONDITION
 from privacyidea.lib.smtpserver import add_smtpserver
 from privacyidea.lib.smsprovider.SMSProvider import set_smsgateway
@@ -911,6 +912,195 @@ def test_02_forward_admin_request(self):
                          "https://remote/token/init")
 
 
+class RequestManglerTestCase(MyTestCase):
+
+    def test_01_delete_request_parameter(self):
+        actions = RequestManglerEventHandler().actions
+        self.assertTrue("delete" in actions, actions)
+        self.assertTrue("set" in actions, actions)
+
+        pos = RequestManglerEventHandler().allowed_positions
+        self.assertEqual(set(pos), {"post", "pre"})
+
+        g = FakeFlaskG()
+        audit_object = FakeAudit()
+        builder = EnvironBuilder(method='POST',
+                                 data={'serial': "SPASS01"},
+                                 headers={})
+        env = builder.get_environ()
+        # Set the remote address so that we can filter for it
+        env["REMOTE_ADDR"] = "10.0.0.1"
+        g.client_ip = env["REMOTE_ADDR"]
+        req = Request(env)
+        req.all_data = {"serial": "SPASS01", "type": "spass", "deleteme": "topsecret"}
+        resp = Response()
+
+        # Request
+        options = {"g": g,
+                   "request": req,
+                   "response": resp,
+                   "handler_def": {"options":
+                                       {"parameter": "deleteme"}
+                                   }
+                   }
+        r_handler = RequestManglerEventHandler()
+        res = r_handler.do("delete", options=options)
+        self.assertNotIn("deleteme", req.all_data)
+        self.assertTrue(res)
+
+        # Delete a non-existing value
+        options = {"g": g,
+                   "request": req,
+                   "response": resp,
+                   "handler_def": {"options":
+                                       {"parameter": "doesnotexist"}
+                                   }
+                   }
+        r_handler = RequestManglerEventHandler()
+        res = r_handler.do("delete", options=options)
+        self.assertNotIn("doesnotexist", req.all_data)
+        self.assertTrue(res)
+
+    def test_02_set_parameter(self):
+        g = FakeFlaskG()
+        audit_object = FakeAudit()
+        builder = EnvironBuilder(method='POST',
+                                 data={'serial': "SPASS01"},
+                                 headers={})
+        env = builder.get_environ()
+        # Set the remote address so that we can filter for it
+        env["REMOTE_ADDR"] = "10.0.0.1"
+        g.client_ip = env["REMOTE_ADDR"]
+        req = Request(env)
+        req.all_data = {"serial": "SPASS01", "type": "spass"}
+        resp = Response()
+
+        # simple add a parameter with a fixed value
+        options = {"g": g,
+                   "request": req,
+                   "response": resp,
+                   "handler_def": {"options":
+                                       {"parameter": "newone",
+                                        "value": "simpleadd"}
+                                   }
+                   }
+        r_handler = RequestManglerEventHandler()
+        res = r_handler.do("set", options=options)
+        self.assertTrue(res)
+        self.assertEqual("simpleadd", req.all_data.get("newone"))
+
+        # overwrite an existing parameter with a fixed value
+        options = {"g": g,
+                   "request": req,
+                   "response": resp,
+                   "handler_def": {"options":
+                                       {"parameter": "serial",
+                                        "value": "FUN007"}
+                                   }
+                   }
+        r_handler = RequestManglerEventHandler()
+        res = r_handler.do("set", options=options)
+        self.assertTrue(res)
+        self.assertEqual("FUN007", req.all_data.get("serial"))
+
+        # Change a parameter with the part of another parameter
+        req.all_data = {"user": "givenname.surname@company.com",
+                        "realm": ""}
+
+        options = {"g": g,
+                   "request": req,
+                   "response": resp,
+                   "handler_def": {"options":
+                                       {"parameter": "realm",
+                                        "value": "{0}",
+                                        "match_parameter": "user",
+                                        "match_pattern": ".*@(.*)"}
+                                   }
+                   }
+        r_handler = RequestManglerEventHandler()
+        res = r_handler.do("set", options=options)
+        self.assertTrue(res)
+        self.assertEqual("company.com", req.all_data.get("realm"))
+        self.assertEqual("givenname.surname@company.com", req.all_data.get("user"))
+
+        # Only match the complete value, not a subvalue
+        req.all_data = {"user": "givenname.surname@company.company",
+                        "realm": ""}
+        options = {"g": g,
+                   "request": req,
+                   "response": resp,
+                   "handler_def": {"options":
+                                       {"parameter": "realm",
+                                        "value": "newrealm",
+                                        "match_parameter": "user",
+                                        "match_pattern": ".*@company.com"}
+                                   }
+                   }
+        r_handler = RequestManglerEventHandler()
+        res = r_handler.do("set", options=options)
+        self.assertTrue(res)
+        # The realm is not changed!
+        self.assertEqual("", req.all_data.get("realm"))
+
+        # Now we change the parameter itself.
+        # Change company.com of a user to newcompany.com
+        req.all_data = {"user": "givenname.surname@company.com"}
+
+        options = {"g": g,
+                   "request": req,
+                   "response": resp,
+                   "handler_def": {"options":
+                                       {"parameter": "user",
+                                        "value": "{0}@newcompany.com",
+                                        "match_parameter": "user",
+                                        "match_pattern": "(.*)@company.com"}
+                                   }
+                   }
+        r_handler = RequestManglerEventHandler()
+        res = r_handler.do("set", options=options)
+        self.assertTrue(res)
+        self.assertEqual("givenname.surname@newcompany.com", req.all_data.get("user"))
+
+        # The request does not contain the match_parameter, thus the
+        # parameter in question will not be modified
+        req.all_data = {"user": "givenname.surname@company.com" }
+        options = {"g": g,
+                   "request": req,
+                   "response": resp,
+                   "handler_def": {"options":
+                                       {"parameter": "user",
+                                        "value": "{0}@newcompany.com",
+                                        "match_parameter": "username",
+                                        "match_pattern": "(.*)@company.com"}
+                                   }
+                   }
+        r_handler = RequestManglerEventHandler()
+        res = r_handler.do("set", options=options)
+        self.assertTrue(res)
+        # The name is still the old one - since there was nothing to match
+        self.assertEqual("givenname.surname@company.com", req.all_data.get("user"))
+
+        # Do some nasty replacing, that will not work out
+        # We require two tags, but only have one!
+        req.all_data = {"user": "givenname.surname@company.com"}
+
+        options = {"g": g,
+                   "request": req,
+                   "response": resp,
+                   "handler_def": {"options":
+                                       {"parameter": "user",
+                                        "value": "{1} <{0}@newcompany.com>",
+                                        "match_parameter": "user",
+                                        "match_pattern": "(.*)@company.com"}
+                                   }
+                   }
+        r_handler = RequestManglerEventHandler()
+        res = r_handler.do("set", options=options)
+        self.assertTrue(res)
+        # The user was not modified, since the number of tags did not match
+        self.assertEqual("givenname.surname@company.com", req.all_data.get("user"))
+
+
 class ResponseManglerTestCase(MyTestCase):
 
     def test_01_delete_response(self):