Package: django-icetea
# django-icetea``django-icetea`` is a package built on top of [Django](https://www.djangoproject.com/) and provides abstractions for creating REST APIs.It has been influenced by the architecture of [django-piston](https://bitbucket.org/jespern/django-piston/wiki/Home) and [piston-perfect](https://github.com/smartpr/piston-perfect).I decided to build ``django-icetea``, in order to have an API framework with tight foundations, consistent and intuitive behaviour, *readable code*, and of course, easy to use.## Installation``django-icetea`` is registered in [PyPI](http://pypi.python.org/pypi/django-icetea/), soinstalling it is as easy as running:pip install django-iceteaHowever, I would suggest using the latest version from github. The master branch is always stable.### Settings parametersIn your application's ``settings.py`` file, you can specify the followingparameters related to ``django-icetea``:* ``ICETEA_ERRORS``: With ``True``, enables the sending of emails to theapplications's admins, in the case of Server Errors. Default is ``True``.* ``ICETEA_DISPLAY_ERRORS``: With ``True``, returns well-formed error messages in the case ofServer Errors. It requires that ``DEBUG=True``. Default is ``True``.## DocumentationThe code is thoroughly documented. Use [epydoc](http://epydoc.sourceforge.net/) to parse it and generate adocument out of it. For example, in order to create an *html* page with thedocumentation, ``cd`` into the *django-icetea* folder, and issue:epydoc --html icetea -o docsThis will create folder *docs*. Open the file *index.html* for the whole codedocumentation.## Philosophy``django-icetea`` aims to provide the abstractions for providing out-of-the-boxfunctionality for creating APIs. It strives to keep things clear and explicit,without any unnecessary magic behind the scenes.It is very extensible, and the default behaviour can be overridden, extendedand modified at will.As in any project though, some assumptions have to be made, and someconventions need to be predefined.For example, although *HTTP* is anapplication protocol, it is mostly used for interaction with Webbrowsers. When applied in more generic request/response schemes, there arescenarios that the protocol itself does not indicate the correct behavior.For this reason, I mostly view *HTTP* as the means of transmissionfor requests and responses. It is unaware of business logic, and therefore itlacks the means of mapping application specific semantics or errors to *HTTPResponses*.A specific case in which the *HTTP* protocol doesn't really specify thebehaviour, is the following:> Say we need to create a model instance; We issue a *POST* request to the> API, and we expect a response which will indicate *if* the> resource has been created, and if yes, return the resource.> The server first needs to validate the data it has received. If the data> are invalid, the API should return a ``400 Bad Request`` response. If the data> are valid, but upon creation the database fails, what do we do? Do we> return ``400 Bad Request``? No way. This will confuse the user and indicate> that the data provided were invalid. The request was validated successfully,> so this is not the case, Do we return a Successful response ``200 OK``,> and empty data? I choose for the latter. This indicates that the data were> indeed valid, but the resource failed to be created.In anycase developing an API is all about consistent and unambiguouscommunication between the client and the server. This has been one of my maingoals with this project. If different applications require different semantics,*django-icetea*'s code can easily be modified to support them.Moreover, following the [Principle of the least astonishment](http://en.wikipedia.org/wiki/Principle_of_least_astonishment) which is what *Python* in general, and *Django* in particular encourage, Ihave tried to follow the general behavior that *Django* users are familiarwith. An example of this is the ``validation`` method of the ``ModelHandler`` class (in the ``handlers.py`` module). Itcleans the data, creates model instances (without committing them to thedatabase) and validates them using the model's ``full-clean`` method. Once thisis done, we are certain that we are dealing with perfectly valid modelinstances, which we can safely write to the database. This means that we don'thave the need to do all these steps manually, since they are offered by*django-icetea* out of the box. It is exactly howvalidation for Django *Modelform* works, and how most *Django* developers areused to doing things..The means have changed (REST API instead of Forms), but the procedure is still thesame.## Short Introduction*django-icetea* offers 2 types of handlers:* ``ModelHandler``: Used to expose *Django* models to the API. Offers CRUDfunctionality out of the box.* ``BaseHandler``: Used to expose data that don't map on a model. Most of thefunctionality will need to be written manually.### Glossary``Singular Request``: A request that refers to a single resource. The resourceis usually identified by the url. For example a *GET/PUT/DELETE* request on ``/resource/<id>/`` is asingular request.``Plural Request``: A request that affects(retrieves or modifies) a group of resources (usually a subset or all the resources that the client has the right to view). It could be plural GET, plural PUT,or plural DELETE.``Bulk Request``: Request with an array of data in its request body. It only makessense for *POST* requests, and aims to create multiple instances in one request. For Bulk POST requests thereis no recommended behavior or semantics, so we defined our own semantics, in order to make surethat the functionality is predictable and makes the most sense. More detailscan be seen in section [Bulk POST requests](https://github.com/stargazer/django-icetea#bulk-post-requests).### AssumptionsThe only assumption that *django-icetea* makes is that singular requests aredenoted by the keyword argument ``id``. So for example a GET request of thefollowing form ``/resource/<id>/`` requests the resource with ``id=<id>``.This is essential mostly for security related checks, which mainly controlwhether the request is plural, and if such a request has been explicitlyallowed.### Incoming RequestsThe ``Content-type`` header for incoming requests should be``application/json``. This is currently the only request body format that*django-icetea* recognizes.### Outgoing responsesThe outgoing responses of *django-icetea* can be of one of the followingformats:*   ``application/json``*   ``text/xml``*   ``text/html``*   ``application/vnd.ms-excel``The default is ``application/json``.Please not that in the case of outputting ``json``, or ``xml``, it is easy to serializedata structures in the response. However in the case of ``html`` andespecially ``xls`` format, there should (probably) be application specific semantics applied to the outputemitters.### Status codesThe Status codes have the following meanings:*   ``200 OK``: Request was served successfully*   ``403 Forbidden``: The client is not authenticated*   ``405 Method Not Allowed``: The request method was performed on a resource that does not support that method*   ``410 Gone``: The resource is not available*   ``422 UnprocessableEntity``: The request was valid but could not be processed due to the semantics of the resource (for example, a *DELETE* request on a resource that belongs to the authenticated client. We might choose not to allow deletion of the specific resource if its field *x* has a specific value. In that case we respond with a *422 UnprocessableEntity* response).For a very detailed description of the expected responses for anykind of request, please check section [Request and response protocol](https://github.com/stargazer/django-icetea#request-and-response-protocol).### Tests``django-icetea`` comes with its own test suite, found in the ``tests.py`` module. This module defines Base Test Classes, which are used to test ``django-icetea`` itself, and can also be used to test any API implementation.### CSRF tokensDjango uses *CSRF tokens*, in order to deal with web browsers'[CSRF vulnerability](http://www.squarefree.com/securitytips/web-developers.html#CSRF).Django's *CsrfViewMiddleware* inserts a *CSRF token* as a hidden field in every form using the*POST method*, before sending the form to the web browser. For every subsequent*POST request* from the web browser, the same middleware checks the token, to ensure thatit contains the expected value. If not a *403 Forbidden* response is returned.However, since *django-icetea* is an API and does not make use of forms, theCSRF token doesn't make a lot of sense. So by default *django-icetea* views are*CSRF exempted*, meaning they don't require the CSRF token.## UsageSay we have an API built on top of ``django-icetea``. Let's assume we havea Django app called ``foo``, with a model ``FooModel``.We want to define 2 API handlers; One that exposes the model ``FooModel``,and another one that exposes some other non-model data.Other than defining the business logic, handlers also act as means ofrepresentation. For example, ``ModelHandler`` classes, define how thecorresponding model will be represented within that handler(which fields shouldbe exposed), but also in cases that it is nested in the responses of other handlers.### foo/handlers.pyTODO: Example with a BaseHandlerHere we define our API handler, which is the implementer of the businesslogic``` pythonfrom foo.models import foomodelfrom icetea.handlers import ModelHandlerclass FooHandler(ModelHandler):authentication = Truemodel = FooModelread = Truecreate = Trueallowed_out_fields = ('id','field1','field2',)allowed_in_fields = ('field1','field2',)```### foo/urls.pyWe need to create resources(equivalent to Django views), which will initiatethe serving of API requests``` pythonfrom djanco.conf.urls.defaults import *from foo.handlers import FooHandlerfrom icetea.resource import Resourcefoo_handler = Resource(FooHandler)urlpatterns = patterns('',url(r'^foo/$ ', foo_handler),url(r'^foo/(?P<id>\d+)/$ ', foo_handler),)```## Handler level attributes### Relevant for all handlers#### read, create, update, deleteIf any of these parameters is ``True``, then the handler allows ``GET``,``POST``, ``PUT`` and ``DELETE`` requests respectively.If instead they are defined as methods, eg:``` pythondef read(self, request, *args, **kwargs):pass```Then the corresponding action is enabled, and the default functionality isoverridden by the method we defined.#### bulk_createIf ``True`` enables bulk-POST requests. Default is ``False``. See section [Notes](https://github.com/stargazer/django-icetea#notes) for more information.Requires that ``create = True``.When enabled, you should anticipate on ``400 Bad Request`` responses, with a list in their body.#### plural_updateIf ``True``, enables plural PUT requests, which means updating multipleresources in one request. It is a potentially catastrophic operation, and forthis reason is should be explicitly allowed. Default is ``False``.Requires that ``update = True``.When enabled, you should anticipate on ``400 Bad Request`` responses, with a list in their body.#### plural_deleteIf ``True`` enables plural DELETE requests, which means deleting multipleresources in one request. It is a potentially catastrophic operation, and forthis reason it should be explicitly allowed. Default is ``False``.Requires that ``delete=True``.When enabled, you should anticipate on ``422 Unprocessable Entity`` responses, with a list in their body.#### request_fieldsIndicates which querystring parameter will act as a a request-level fieldselector. If ``True``, then the selector is ``field``. If ``False``, there will be no field selection. Default is ``True``.#### orderIndicates which querystring parameter will act as the order-type selectoron the result set of the requested operation.If ``True``, then the parameter is ``order``. If ``False``, no order-typeselection can be performed. Default is ``False``.The order logic, should be implemented in the handler's ``order_data``method.#### sliceIndicates which querystring parameter will be used to request slicing ofthe result set of the requested operation.If ``True``, then the parameter is ``slice``. If ``False``, no slicing willbe possible. Default is ``False``.The slicing notation follows Python's *slice notation*, of ``start:stop:step``.#### authenticationIf ``True``, only authenticated users can access the handler. The *Djangoauthentication* is used. Default value is ``False``.#### allowed_out_fieldsTuple of fields, which indicates the fields that the handler is allowed tooutput.In the case of a ``ModelHandler``, it indicates model fields.In the case of a ``BaseHandler``, it only has sense if the handler returns dictionaries, or lists of dictionaries, and it indicates the dictionary keys that the handler is allowed to output.The actual fields that a request will eventually output, is a function ofthis parameter, as well as the request-level field selection, indicated bythe ``field``.#### allowed_in_fieldsTuple of fields, which indicates the fields that the handler is allowed totake from the incoming request body. In the case of ``ModelHandler``classes, no primary keys or related keys are allowed.### Relevant only for handlers extending ModelHandler#### modelThe database model which the Handler exposes.#### filtersA dictionary of ``filter name``: ``filter_operation`` couples. ``filtername`` defines the querystring parameter used to apply the filtering on thecurrent request. ``filter_operation`` corresponds to a Django lookupfilter, which will be applied on the request's resuls data.#### exclude_nestedFields which should be excluded when the model is nested in anotherhandler's response.#### excel_filenameThe filename to be given to the attachment, if the the request needs to outputto ``excel`` format. It can either be a string or a handler method that returns astring. Default value is ``file.xls``## Notes### Adding extra (fake) fields on a ModelHandlerIt's possible that we want to add fake fields on the output of a ``ModelHandler``.By *fake* I mean fields that are not actual physical model fields, but simply extrainformation that we wish to include on the API handler's output. Doing so is very easy.In the model class,  you simply need to define the ``fake_fields`` tuple, with thenames of the fake fields. Then we define the class method``compute_fake_fields(self, field)``, which should return the values of the fake fields.For example:``` pythonfake_fields = ('num_tweets', 'num_retweets',)def _compute_fake_fields(self, field):if field == 'num_tweets':return self.tweets.count()elif field == 'num_retweets':return Retweet.objects.filter(tweet__in=self.tweets.all()).count()```The method ``_compute_fake_fields`` is invoked by the ``Emitter`` class, whichconstructs the output of the handler. The ``field`` parameter is the field namethat is evaluated. So the ``_compute_fake_fields`` method should be able to computeall the field names in the ``fake_fields`` tuple.From this point on, the API handlers can treat these fields as normal model fields.Meaning, they can be included in the tuples ``allowed_out_fields``,``exclude_nested``, etc, depending on how you want to treat them.### Bulk POST requests*Bulk POST request* refers to a single ``POST`` request which attempts to createmultiple resources. The specifications of ``REST`` or ``HTTP`` don't specifyany standard behaviour for such requests, and instead discourage its use, dueto poor semantics.For example, how would the API signal an error on one of the data objects inthe request body? Or, how would it signal a database error, when all the dataobjects in the request body were valid?I chose the following behavior:* Any error in the request body, will return a ``Bad Request`` response.For example when the data in the request body refer to Django models, ifeven one of the models fails to validate, the response will be ``400 BadRequest``. The response body will include a list, with all the objects thatcould not be validated. Every object should have an ``index`` parameter, thatspecifies a zero-bazed index of the request body parameter that could not bevalidated.(Similarly a ``POST`` request for a single instance, returns ``Bad request``if the request body does not contain valid data)* If the request body is valid, the response is ``OK``, and its bodycontains a list of all the successfully added model instances. If one modelinstance failed to be created (due for example to a database error),although it contained valid data, it will not be part of the responsedata.(Similarly a POST request for a single instance, returns an ``OK``response, and the model instance in the request body. If the modelinstance failed to be created, although it was valid, we return an ``OK``response, with ``null`` in the response body)This is in my opinion the most intuitive behavior. However I think that it alldepends on the requirements of each application, and the clients using the API.So feel free to modify the existing behavior.By default *bulk POST requests* are disabled. They can be enabled by setting``bulk_create = True`` in the handler class.### Building inheritable handlers... Metaclass magicIn this subsection, the term ``operation`` means one of ``read``, ``create``,``update``, ``delete``.When a handler class sets ``read = True``, basically it says to the system:> I want to inherit the standard ``read`` functionality. Please provide me with it.This works with some metaclass magic. Clearly some magic needs to be inplease in order to convert the boolean attribute ``read``, to a method.The way metaclasses work, is that when a class is initialized, the Pythoninterpreter scans its own member attributes, and *then* runs the code of themetaclass. In this case, what the metaclass does, is remove all thoseoperations that have been defined with ``True``, like `read = True``, in orderto make space for them to be inherited. The metaclass runs on classinitialization, whereas the [PythonMRO](http://www.python.org/getit/releases/2.3/mro/) runs on runtime.For this reason, when a handler class is defined, in order to provideinheritable behaviour for other handlers, unless it defines the operations asmethods, it needs to provide them as ``read = True``. This way its metaclasswill remove these attributes and make "space", for classes that inheritfrom it, to inherit the behaviour that these operations define. Of course thehandlers that inherit from a base handler, will need to first explicitly allowan operation, in case they want to inherit its functionality.So the way to see it when building handlers:>    Setting ``read = True``, means that the handler itself and handlers that>    inherit from it, will inherit the ``read`` functionality, given that they>    allow so.>>    Setting ``read = False``, or not setting the ``read`` attribute at all, will block>    the ``read`` functionality for the handler and handlers that inherit from it.### Request and response protocolHere we describe the responses (status code and response body) for all types ofHTTP requests, successful or not.*   GET*   Singular*   Successful:*   Status code: 200*   Response body: Dictionary* Errors:*   Status code: 403*   Response body: Empty*   Status code: 405*   Response body: Empty*   Status code: 410*   Response body: Empty*   Plural*   Successful:*   Status code: 200*   Response body: List*   Errors:*   Status code: 403*   Response body: Empty*   Status code: 405*   Response body: Empty*   POST*   Refers to single resource (Dictionary in request body)*   Successful:*   Status code: 200*   Response body: Dictionary*   Response body: ``null`` (Happens in the case when adatabase failure prevents the data object from beingwritten to the database)*   Errors:*   Status code: 400*   Response body: Dictionary*   Status code: 403*   Response body: Empty*   Status code: 405*   Response body: Empty*   Status code: 422*   Response body: Dictionary*   Bulk (List in request body)*   Successful*   Status code: 200*   Response body: List*   Errors*   Status code: 400*   Response body: List (list items are dictionaries. Everydictionary should have an``index`` parameter which defines azero-based index of the request body instance that wasinvalid)*   Status code: 403*   Response body: Empty*   Status code: 405*   Response body: Empty*   Status code: 422*   Response body: List (list items are dictionaries. Everydictionary should have an ``index`` parameter which defines azero-based index of the request body instance that causedthe error)*   PUT*   Singular*   Successful*   Status code: 200*   Response body: Dictionary*   Errors*   Status code: 400*   Response body: Dictionary*   Status code: 403*   Response body: Empty*   Status code: 405*   Response body: Empty*   Status code: 410*   Response body: Empty*   Status code: 422*   Response body: Dictionary*   Plural*   Successful*   Status code: 200*   Response body: List*   Errors*   Status code: 400*   Response body: List (list items are dictionaries. Everydictionary should provide an ``id`` parameter which definesthe ``id`` of the (model) instance that was invalid)*   Status code: 403*   Response body: Empty*   Status code: 405*   Response body: Empty*   Status code: 422*   Response body: List (list items are dictionaries. Everydictionary should provide an ``id`` parameter which definesthe ``id` of the (model) instance that caused the error)*   DELETE*   Singular*   Successful*   Status code: 200*   Response body: Dictionary*   Errors*   Status code: 403*   Response body: Empty*   Status code: 405*   Response body: Empty*   Status code: 410*   Response body: Empty*   Status code: 422*   Response body: Dictionary*   Plural*   Successful*   Status code: 200*   Response body: List*   Errors*   Status code: 403*   Response body: Empty*   Status code: 405*   Response body: Empty*   Status code: 422*   Response body: List (list items are dictionaries. Everydictionary should provide an ``id`` parameter which definedthe ``id`` of the (model) instance that caused the error)#### NoteIn the presence of errors, if the response body is a dictionary or a list,every error instance(*one* in the case of dictionary, *multiple* in the case of alist) will contain the following keys:*   ``errors``: It will be a dictionary of {``field``: [``error``]} pairs where possible,or a list of strings describing the errors.*   ``type``  : Error typeIn the case of a list of errors, every item will contain the key ``index``, or``id``, which will specify which request body item, or which data model instance caused the corresponding error.##### Examples``` python# Error response of a POST request for a single resource{"errors": {"text": ["This field cannot be blank"]},"type": "Validation Error"}`````` python# Error response of a Bulk POST request[{"index": 0,"errors": {"gender": ["Value u'bi' is not a valid choice."],"email": ["Invalid Email"]},"type": "Validation Error"},{"index": 3,"errors":   {"postcode": ["Invalid Postcode"]"type": "Validation Error"}}]`````` python# Error response of a plural DELETE request[{"id": 2,"errors": ["Instance cannot be deleted"],"type": "Unprocessable Entity Error"},{"id": 4,"errors": ["Instance cannot be deleted"],"type": "Unprocessable Entity Error"}]```