gsehub/xblock-mentoring — pull request #1 by FiloSottile (2014-03-28 16:40:28+00:00)
Title: Optimize mentoring-dataexport times by using a raw SQL cursor

The bulk reading of the Answer objects to export have been converted to chunked reading of the three needed columns from a raw DB connection cursor, executed with a slightly modified version of the Django query.

A new util function, `chunked_cursor_iterator()`, has been added.

The before and after benchmark and data verification follows. Note that the new code has been executed first to avoid second-to-run cache advantages.

```
$ time curl 'http://preview.localhost:8000/courses/edX/Open_DemoX/edx_demo_course/xblock/i4x:;_;_edX;_Open_DemoX;_mentoring-dataexport;_0e508e490f5c47e08e929c639d7e62cf/handler/download_csv' -H 'Accept-Encoding: gzip,deflate,sdch' -H 'Accept-Language: en-US,en;q=0.8,it-IT;q=0.6,it;q=0.4' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.152 Safari/537.36' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8' -H 'Referer: http://preview.localhost:8000/courses/edX/Open_DemoX/edx_demo_course/courseware/graded_interactions/d0526a768886466daf83bb02ef1ee348/' -H 'Cookie: djdt=hide; edxloggedin=true; sessionid=dce732d78d28fe8434856631c59e6bdf; csrftoken=SrZqmCgvxBgOywdQqwfnF8yHPxci9PUF' -H 'Connection: keep-alive' --compressed > new.csv
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 44.3M 0 44.3M 0 0 3155k 0 --:--:-- 0:00:14 --:--:-- 5357k
curl -H 'Accept-Encoding: gzip,deflate,sdch' -H -H -H -H -H -H > 0,27s user 0,76s system 7% cpu 14,421 total

$ time curl 'http://preview.localhost:8000/courses/edX/Open_DemoX/edx_demo_course/xblock/i4x:;_;_edX;_Open_DemoX;_mentoring-dataexport;_0e508e490f5c47e08e929c639d7e62cf/handler/download_csv' -H 'Accept-Encoding: gzip,deflate,sdch' -H 'Accept-Language: en-US,en;q=0.8,it-IT;q=0.6,it;q=0.4' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.152 Safari/537.36' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8' -H 'Referer: http://preview.localhost:8000/courses/edX/Open_DemoX/edx_demo_course/courseware/graded_interactions/d0526a768886466daf83bb02ef1ee348/' -H 'Cookie: djdt=hide; edxloggedin=true; sessionid=dce732d78d28fe8434856631c59e6bdf; csrftoken=SrZqmCgvxBgOywdQqwfnF8yHPxci9PUF' -H 'Connection: keep-alive' --compressed > old.csv
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 44.3M 0 44.3M 0 0 656k 0 --:--:-- 0:01:09 --:--:-- 1739k
curl -H 'Accept-Encoding: gzip,deflate,sdch' -H -H -H -H -H -H > 0,32s user 1,01s system 1% cpu 1:09,28 total

$ sha1sum new.csv old.csv
a78a1a3fc47619882884a65eb374de1d27ba7ee3 new.csv
a78a1a3fc47619882884a65eb374de1d27ba7ee3 old.csv
```