Repository: privacyidea/privacyidea

Issue:
Add an endpoint for polling the transaction status which automatically deletes answered challenges
Assume a user has enrolled a push token and tries to log in via the owncloud app. The app triggers a challenge in the privacyIDEA server, the user smartphone receives a push notification, the user accepts the challenge. The smartphone app sends a challenge response to the ``/ttype/push`` endpoint, which sets the ``otp_valid`` flag of the challenge in the database. According to the [push token protocol](https://github.com/privacyidea/privacyidea/wiki/concept%3A-PushToken#polling-endpoint-for-application), the owncloud app polls the ``/token/challenges`` endpoint and authenticates the user if it finds a challenge with ``otp_valid`` set to true.

After authentication, the accepted challenge *stays* in the database (https://github.com/privacyidea/privacyidea-owncloud-app/issues/73). Eventually, it will be removed by the challenge janitor, but the cleanup timestamp is not really defined: The challenge janitor is, for example, invoked when a new challenge is triggered. But the challenge could potentially stay in the database for a long time.

This means that theoretically, one could still authenticate with the old transaction ID until the challenge is cleaned up. An app should not allow the user to specify a transaction ID, but privacyIDEA should not allow this behavior nonetheless.

An initial idea was to add a boolean flag to the response of ``/token/challenges`` that indicates the validity of the challenge (#1837). However, a better idea might be the following:

We add a new endpoint, like ``POST /token/finalize_transaction`` (or so), which takes a parameter ``transaction_id`` and

1) checks for a valid challenge with the given transaction ID and ``otp_valid=true``
2) if such a challenge exists, the endpoint returns ``true`` and removes *all* challenges with the given transaction ID

This endpoint could be polled by apps to check the challenge response status of out-of-band tokens (currently push + tiqr).


Discussion:
The documentation of the TiQR token actually proposes that applications poll the ``/validate/check`` endpoint with the transaction ID and an empty password: https://privacyidea.readthedocs.io/en/latest/modules/lib/tokentypes/tiqr.html#authentication

The ``/validate/check`` endpoint then calls ``TiqrTokenClass.check_challenge_response``, which checks for a valid challenge with ``otp_valid=true``. If it finds such a challenge, it returns true and then *actually deletes* the challenge (but other challenges with the same transaction ID stay in the database).

https://github.com/privacyidea/privacyidea/blob/f1543ced95df9dd7214dbd2499e3f57bd3990f98/privacyidea/lib/tokens/tiqrtoken.py#L465

Thus, the behavior of ``/validate/check`` is really close to the behavior of the ``POST /token/finalize_transaction`` endpoint proposed above. ~~Polling ``/validate/check`` for push tokens has a different behavior, as it *does not* delete a matching valid challenge:~~ (this is not correct, see below)

https://github.com/privacyidea/privacyidea/blob/f1543ced95df9dd7214dbd2499e3f57bd3990f98/privacyidea/lib/tokens/pushtoken.py#L660

... but I'm wondering why? @cornelinux wrote in #1494:

> The thing is, that we keep the challenge for a while, so that we can fetch the information several times.
> The old expired challenge gets deleted by the challenge janitor, when a new challenge is created for this very token.
> https://github.com/privacyidea/privacyidea/blob/1342-pushtoken/privacyidea/lib/tokens/pushtoken.py#L530

The question is: Should we

1) modify ``PushTokenClass.check_challenge_response`` such that it also removes a correctly answered challenge, and require apps to poll ``/validate/check`` instead of ``GET /token/challenges``?
2) or add a new endpoint as described above, which is used for out-of-band tokens
We should at least use the __same__ behavior for all "out-of-band" tokens.
TiQR calls ``/validate/check?transaction_id=12345&pass=`` with an empty password to check, whether a challenge was answered correctly *in time*. The reason to *not* delete this transaction in this case when it was answered correctly is very simple:
An attacke could "guess" the transaction_id, call the unauthenticated /validate/check endpoint and thus clean the challenge. The real user would not be able to login, since the information, that he answered the challenge successfully has been deleted.

Polling the /validate/check endpoint is simpler for the application. We could do this for the push token app, too.

A side question would be, which events you would want to see in the audit log as an administrator.

So I would also like to have an additional endpoint with a nicde name than ``/token/finalize_transaction`` like ``POST /validate/<namingishard>``. I think putting it in the validate endpoint makes more sence, since this would be an endpoint dedicated for authentication while `GET /token/challenges`` is dedicated for managing tokens! 

Thanks for your thoughts!

> TiQR calls `/validate/check?transaction_id=12345&pass=` with an empty password to check, whether a challenge was answered correctly _in time_. The reason to _not_ delete this transaction in this case when it was answered correctly is very simple:
> An attacke could "guess" the transaction_id, call the unauthenticated /validate/check endpoint and thus clean the challenge. The real user would not be able to login, since the information, that he answered the challenge successfully has been deleted.

I think this line of thought makes sense -- but as it turns out, the ``/validate/check?transaction_id=12345&pass=`` request *does* delete the challenge in case of the TiQR token:

https://github.com/privacyidea/privacyidea/blob/a128999d61f7d6fed114ed371528ab7d766be4d8/privacyidea/lib/tokens/tiqrtoken.py#L465

As @plettich said above, we should at least have the same behavior for all out-of-band tokens.

So I'll add an unauthenticated POST endpoint to ``/validate/`` that
* takes a mandatory transaction ID
* checks if any challenge with that transaction ID has ``otp_valid=1`` and ``otp_received=1``
* if yes, remove *all* challenges with this transaction and return true
* otherwise, return false
We should keep in mind that SAML plugins expect several SAML-specific attributes from the privacyIDEA server. Currently, they are provided by ``/validate/samlcheck``. If we really create a new endpoint for out-of-band tokens, we will need to provide the attributes somehow (see https://github.com/privacyidea/simplesamlphp-module-privacyidea/issues/68)
OK, disregard everything I said above. :) I had a closer look at the behavior of ``/validate/check`` when it is provided with a correct challenge-response response for a transaction, and as it turns out, it *does* actually remove *all* challenges with that transaction ID (this was implemented in #654). This is because of this line in ``check_token_list``:
https://github.com/privacyidea/privacyidea/blob/d140e767438706a6da47a3e432f9b4c59ab1a69b/privacyidea/lib/token.py#L2240

So apparently ``/validate/check`` already implements the behavior that we want from our ominous endpoint above :)

In this case, I would propose to simply change the push token documentation, such that clients should poll the ``/validate/check`` endpoint with a transaction ID and an empty password? 
We decided on the following proposal:

Out-of-band tokens (like push or tiqr) are different to other challenge-response tokens in the sense that they require the authentication plugin to poll the privacyIDEA server and check whether the out-of-band challenge has been answered correctly.
We define the authentication flow as follows:

1. the plugin triggers a challenge (via ``/validate/check`` or ``/validate/triggerchallenge``) and gets a transaction ID
2. the plugin polls a new endpoint like ``/validate/pollchallenges`` which takes only a transaction ID and only returns ``true`` or ``false``. ``true`` means that the plugin should attempt an authentication, ``false`` means that no challenge to this transaction has been answered yet. The endpoint does not require authentication, but this could be added by a ``api_key`` policy. We should probably also return ``false`` if there is no challenge with the respective transaction ID, to avoid information leakage.
3. the user answers the challenge (e.g. using a smartphone app). By communication with a tokentype-specific endpoint, the challenge is marked as valid.
4. Hence, ``/validate/pollchallenges`` returns ``true`` to the application, which then sends a ``/validate/check`` request to finalize the authentication. This request includes the transaction ID and an empty password. Upon successful authentication, this endpoint removes *all* challenges associated with the given transaction ID.

This behavior should be the same for all out-of-band tokens.

So we should do the following:
* [x] pick a cool name for the endpoint
* [x] implement that endpoint
* [ ] document this flow somewhere and change the PUSH specification
* [ ] change the ownCloud plugin, which currently polls ``/token/challenges``

@cornelinux  @NuvandaPV what do you think? Let me know if I forgot something.
I would prefer the normal documentation.
We have some infos in the tokens descriptions themselves.
https://privacyidea.readthedocs.io/en/latest/configuration/tokenconfig/email.html?highlight=transaction_id#first-step

https://privacyidea.readthedocs.io/en/latest/configuration/tokenconfig/sms.html?highlight=transaction_id#first-step

But maybe we should add a more generic description, that describes:

1. normal "single shot" authentication
2. challenge response authentication
3. out of band authentication

Maybe we should add it before the tokentyes (https://privacyidea.readthedocs.io/en/latest/configuration/token_config.html#supported-tokentypes) to tell, what generic types of authentication exist.
As #1859 is merged now, the next step is to document the authentication flow for out-of-band tokens. I'll add this to the token docs as @cornelinux suggested.

Resolving patch:
diff --git a/privacyidea/lib/tokens/tiqrtoken.py b/privacyidea/lib/tokens/tiqrtoken.py
--- a/privacyidea/lib/tokens/tiqrtoken.py
+++ b/privacyidea/lib/tokens/tiqrtoken.py
@@ -68,8 +68,12 @@
    authentication is stored in the Challenge DB table.
    (No need for the application to take any action)
 5. Now, the application needs to poll
-   ``/validate/check?user=<user>&transaction_id=*&pass=`` to verifiy the
-   successful authentication. The ``pass`` can be empty.
+   ``/validate/polltransaction?transaction_id=<transaction_id>`` to check
+   the transaction status. If the endpoint returns ``false``, the challenge
+   has not been answered yet.
+6. Once ``/validate/polltransaction`` returns true, the application needs to finalize
+   the authentication with a request ``/validate/check?user=<user>&transaction_id=<transaction_id>&pass=``.
+   The ``pass`` can be empty.
    If ``value=true`` is returned, the user authenticated successfully
    with the TiQR token.
 


Test patch: