emoncms/emoncms — pull request #57 by CapnBry (2013-04-30 20:53:40+00:00)
Title: Long rawdata visualization load times

My emoncms is on a machine that doesn't have mysql on it, on account of it being flash media that can't take the update frequency my plug monitors put out (once every 10 seconds per plug, 4 plugs so far). When adding a rawdata visualization, the browser spins for over *13 seconds* to load the graph. Adding just a few graphs or a multigraph was making the page minutes to load.

I turned on some SQL tracing and saw that the Feed model was making 1,000 calls to the database to quantize the data, one for each datapoint. The data it was returning was also incorrect on account of it just pulls 1 random sample from each quantization with may massively misrepresent the value of the dataset.

What I've done is modified the class to push the work off onto the mysql server and have it return a dataset which averages the data for each interval. This reduced the load time from 13 seconds to 0.175s. This may not be viable for massive datasets but I don't have a database with millions of rows to test with.