Pylons/pyramid — Issue #436: Authorisation incorrectly implements 403 Forbidden against RFC 2616
Author: simonyarde
Created: 2012-02-14 12:36:02+00:00
State: open

I have provided an explanation of the issue as-I-see-it and an outline for
potential improvements to the cookbook re issue #433 since I believe
they are related, hence the length of the article as I've attempted to capture
both concerns here.

### Background

There are two common approaches to authentication: HTTP Authentication re
[RFC2617](http://www.ietf.org/rfc/rfc2617.txt); or so called "custom authentication", also known as "forms
authentication" or "cookie authentication".

With custom authentication, a web application masks 401 Unauthorised responses
with redirection to a custom login-form and uses cookies to persist
authentication information in so-called "auth tickets". Following successful
authentication the user is typically redirected to the protected resource
initially requested, or to a default page if the login request did not originate
from a masked 401. In such designs, 401 responses are never sent because they
are always replaced by 302 Found (since HTTP/1.0), or more appropriate 303 See
Other (since HTTP/1.1). Custom authentication is common-place because it has
been seen to allow developers greater control over the login process,
particularly in relation to presentation, log-out, feedback regarding forgotten
passwords, and the ability to implement the kinds of increased-security features
employed by financial institutions.

HTTP Authorisation is appropriate where a user-agent needs to access a protected
resource without user-interaction such as a private RSS-feed. In this
circumstance custom authentication is not possible because there is no defined
standard. HTTP Authentication is also applicable in many cases where basic
log/pass functionality is required. As browsers improve the user-experience of
login-challenges triggered by 401 responses they may become more accepted. Users
are increasingly accustomed to mobile interfaces that create much greater
consistency between browser and operating-system UI, and indeed less distinction
between web-pages and apps connected to web-services.

Hybrid login systems providing both HTTP Authorisation and custom authorisation
are a likely interim situation for the foreseeable future as described in this
article by Atif Aziz and Scott Mitchell -
http://msdn.microsoft.com/en-us/library/aa479391.aspx - and are especially
relevant as the web becomes more RESTful or web-service-like.

### Issue

Pyramid's authorisation raises a 403 Forbidden exception. I believe this is the
wrong exception and response because 403 explicitly states that authorisation
will not help, i.e. **the problem has nothing to do with authorisation
whatsoever**:

10.4.4 403 Forbidden

The server understood the request, but is refusing to fulfill it. Authorization
will not help and the request SHOULD NOT be repeated. If the request method was
not HEAD and the server wishes to make public why the request has not been
fulfilled, it SHOULD describe the reason for the refusal in the entity. If the
server does not wish to make this information available to the client, the
status code 404 (Not Found) can be used instead.

http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.4

401 is the meaningful exception and correct HTTP response for *all* matters
relating to authentication and authorisation:

10.4.2 401 Unauthorized

The request requires user authentication. The response MUST include a
WWW-Authenticate header field (section 14.47) containing a challenge applicable
to the requested resource. The client MAY repeat the request with a suitable
Authorization header field (section 14.8). If the request already included
Authorization credentials, then the 401 response indicates that authorization
has been refused for those credentials. If the 401 response contains the same
challenge as the prior response, and the user agent has already attempted
authentication at least once, then the user SHOULD be presented the entity that
was given in the response, since that entity might include relevant diagnostic
information. HTTP access authentication is explained in "HTTP Authentication:
Basic and Digest Access Authentication" [43].

http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.2

The introduction of HTTP standards into Pyramid's exception-language creates an
expectation for correct usage that is disrupted by the misuse of 403 in relation
to authorisation. This is having negative repercussions in the tutorial content,
which might otherwise have gently introduced new developers to web-standards and
to the logic of a custom auth mechanism that is complimentary to HTTP standards
and HTTP Authorisation.

Furthermore, because the 403 Forbidden view is in use by authorisation it is
difficult to devise code that responds to genuine 403 exceptions (see Twitter
example discussed below).

Applications wishing to use HTTP Authorisation currently have to intercept a 403
Forbidden exception and mask it with a 401 response in order to trigger
user-agent login mechanisms; again, it is hard to devise good code and
counterintuitive to communicate in documentation and tutorials.

Regarding permissions, 401 is the correct exception/response where a user does
not have a particular permission because the user should be given a chance to
login with different credentials, and additionally be informed of the reason why
their login failed:

If the request already included Authorization credentials, then the 401
response indicates that authorization has been refused for those
credentials.

RFC2616

and as mentioned by @mmdonc in relation to Issue #433:

.. in reality what the
login-form-invoked-as-the-result-of-an-HTTPForbidden-when-the-user-already-
is-not-just-Everyone could say is "your current credentials do not allow you
to perform the specified operation, but you can try to log in with another
set of credentials if you like, and the operation will then be performed if
those credentials are permitted to perform the operation". This is also the
answer to the second question above.

For contrast I highlight the example of Twitter which responds with 403
Forbidden where the number of requests for a given user has been exceeded; the
issue is temporary and is *not* related to an authorisation problem (auth will not
help), but *has* been caused by the user, and so is the correct use of 403. The
corresponding server-error 503 Service Unavailable would have been appropriate
had it been the server that was at fault.

### Related Note re Cookbook and Docs

https://github.com/Pylons/pyramid/issues/433

I believe the problem needs to be solved root-and-branch, starting with the
correct use of 401 as discussed above. Otherwise developers need to understand
that 403 Forbidden is a meaningless exception that they will have to
intercept and mask. As mentioned previously it becomes hard to write good code
that clearly separates authentication logic from informational 403 exceptions.

It would be beneficial to highlight the two common approaches to authorisation,
i.e. HTTP Authorisation and custom authorisation, and explain their relationship.

It would probably be best to focus on custom authorisation since the
implementation is easier and a more common-place requirement.

The cookbook could explain briefly that HTTP Authorisation can be integrated
at a later stage to create a hybrid authorisation mechanism to support certain
user-agents, where the 401 response would be returned rather than masked
with a redirect.

Developers need to understand that if they implement an ACL but no means for a
user or user-agent to authenticate, then the user will be presented with the
browser's login challenge or be redirected to a login page (if configured); the
content will remain inaccessible in any case.