tomchristie/django-rest-framework — pull request #1994 by akaariai (2014-10-29 11:01:16+00:00)
Title: A couple of speedup ideas

I've implemented a couple of ideas to speed up rest framework. The most important idea is to memoize situations where simple getattr calls can be used instead of the full blown get_attribute. This patch also does slight speedup to serializer.to_representation(), and allows using non-sorted output from serializers.

The test case I used was rendering 100000 simple objects with ten simple fields to a JSON list. I am using ensure_ascii when rendering, as that is fastest option.

Current master renders the data to JSON in 24s, while version-3.0 renders it in around 8.2s. When using the patched version with ordered output, the run speed is around 5.1s, and with unordered output, the speed is around 3.1s. All combined the speedup is 8x compared to master. The patched version is 2.6x faster than version-3.0. Dumping the data directly without a serializer takes around 0.9s.

The biggest drawbacks of the used approach are:
- added complexity
- not guaranteed to work if users do runtime monkey patching
- it is possible that you have an instance where an attribute is sometimes an callable, and sometimes not. Something like: `data = [Obj(some_attr=datetime.now if i > 1 else None) for i in range(0, 10)]` would not work with the current implementation. Notably caching is_simple_callable is impossible for such data, and most of the speedup comes from bypassing is_simple_callable checks.

While the patch passes all tests the work is at proof of concept stage.

After the changes the time is mostly consumed in iterating the fields, calling get_attribute, and then calling field.to_representation(). There isn't any obvious extra work done any more. The top lines of the profile are:

```
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.190 0.190 12.452 12.452 tester.py:1()
1 0.000 0.000 9.875 9.875 serializers.py:126(data)
1 0.150 0.150 9.875 9.875 serializers.py:495(to_representation)
100000 3.258 0.000 9.721 0.000 serializers.py:415(to_representation)
1000000 2.469 0.000 3.574 0.000 fields.py:254(get_attribute)
1003256/1003228 1.109 0.000 1.109 0.000 {getattr}
100000 0.541 0.000 0.924 0.000 tester.py:32(__init__)
800000 0.917 0.000 0.917 0.000 fields.py:517(to_representation)
100000 0.483 0.000 0.898 0.000 fields.py:820(to_representation)
1 0.000 0.000 0.817 0.817 renderers.py:81(render)
1 0.041 0.041 0.790 0.790 __init__.py:185(dumps)
1 0.000 0.000 0.749 0.749 encoder.py:180(encode)
100000 0.622 0.000 0.726 0.000 serializers.py:167(__init__)
```
The test script used was:
```
import os
os.environ['DJANGO_SETTINGS_MODULE'] = 'django_test.settings'
from django import setup
setup()
from datetime import datetime
import json

from rest_framework import serializers
from rest_framework.renderers import JSONRenderer

class CommentSerializer(serializers.Serializer):
email = serializers.EmailField()
content1 = serializers.CharField(max_length=200)
content2 = serializers.CharField(max_length=200)
content3 = serializers.CharField(max_length=200)
content4 = serializers.CharField(max_length=200)
content5 = serializers.CharField(max_length=200)
content6 = serializers.CharField(max_length=200)
content7 = serializers.CharField(max_length=200)
content8 = serializers.CharField(max_length=200)
date = serializers.DateField()

sorted_output = False

class Comment(object):
def __init__(self):
self.content1 = u'foo'
self.content2 = u'foo'
self.content3 = u'foo'
self.content4 = u'foo'
self.content5 = u'foo'
self.content6 = u'foo'
self.content7 = u'foo'
self.content8 = u'foo'
self.email = u'a@b.fi'
self.date = datetime.now().date()

comments = [Comment() for i in range(0, 100000)]
start = datetime.now()
json.dumps([
{
'content1': getattr(c, 'content1'), 'email': getattr(c, 'email'),
'date': getattr(c, 'date').isoformat(),
'content2': getattr(c, 'content2'),
'content3': getattr(c, 'content3'),
'content4': getattr(c, 'content4'),
'content5': getattr(c, 'content5'),
'content6': getattr(c, 'content6'),
'content7': getattr(c, 'content7'),
'content8': getattr(c, 'content8'),
} for c in comments])
print(datetime.now() - start)
start = datetime.now()
serializer = CommentSerializer(comments, many=True)
class FastJSONRenderer(JSONRenderer):
ensure_ascii = True

FastJSONRenderer().render(serializer.data)
print(datetime.now() - start)
```