dustinhiatt-wf/table — pull request #1 by dustinhiatt-wf (2013-10-28 20:16:50+00:00)
Title: Import matrices

This represents the work needed to 'hide' the actual table implementation from a consumer's perspective. There are currently two modes of operation.

1) All table manipulation occurs on GAE. What is returned from that context is a wrapper that wraps a Table which is an ndb.Model. This wrapper exposes Matrix functionality and performs those calculations/updates/etc on the actual model. This is very quick, but because the whole model is in memory, we are limited in numbers here.

2) All table manipulation occurs on GCE. What is returned from that context is a wrapper that wraps REST endpoint calls. All calls into this functionality actually initiates an rpc which tells GCE to update the table it has in memory. Calling put on this wrapper will initiate a call to tell GCE to manually save the table. Because all of these operations cause an RPC, this wrapper is slower but there are basically no memory constraints as the table exists in memory on GCE.

I'm not 100% thrilled with my REST endpoints at this moment.

GET /v1/table//row//column/ will get you a value at a position, but getting a range has to look like...
GET /v1/table//rows?start_row=&stop_row=&start_column=...

I can't think of a better way of doing that. I can probably also clean up some of that code to reduce repetitive code, but I wanted to get this out there so you can see where I am at.

Next steps:

1) Implement a third context which is the gce server context. This has to be different than GAE context simply because all the ndb tools we have on GAE don't exist on GCE. This should make deploying a new server and integrating with GAE easy as it's simply a switch of contexts.

2) Use autobahn to actually create the endpoints. I hope that won't be too bad.

3) We REALLY REALLY need something around Elements so that I can be notified when an Element's value is changed.

4) APIs specific to bigsky. What I have here is pretty generic, but the code that scrapes section WFML for cells will have to be pretty specific and will probably have to live in bigsky. Most of this code is actually complete already and lives in DS-1313.

I think #3 is actually the biggest hurdle, but I think we are actually pretty close to being able to actually use something like this to store values for calculations for current tables (especially if we limit ourselves to GAE contexts initially).

While you review this, you might also review general repo structure. I'm not sure if the license is really needed or not and I just copied some generic text. I also need to generate a readme. I would like to open source this at some point.

This code review looks long, but in reality you have all seen most of this code before. The new pieces of the code center around the GCE context material.