sakaiproject/Hilary — pull request #442 by simong (2013-03-25 19:26:12+00:00)
Title: Collaborative documents

## What
This PR introduces the collaborative documents with concurrent editing. It uses [etherpad-lite](https://github.com/ether/etherpad-lite) under the hood and will become a standard dependency if this gets merged in. You technically don't need it to run Hilary, but you do need it when creating/editing/publishing collaborative documents

This PR adds support for:
* Collaborative documents (collabdocs)
* Preview processing of collabdocs
* Adding the html of collabdocs to the search index
* Storing the html on publications in a revision and on the content object

## How does it work.

### Document creation

The following interaction with the etherpad API occurs:
* An Etherpad group specifically for this content ID is created
* An etherpad grouppad spefically for this content ID is created

### Document editing

When a user wants to edit a document the UI needs to make a call to
`/api/content/:id/join`. This endpoint will return a URL that the UI can use as the src of an iframe and then inject in the DOM.
The endpoint performs the following:
* Create an etherpad author for this user if it doesn't exist yet.
* Create a session that is tied to:
* The author
* The group
* Valid for 24 hours.

The url looks like:
http://0.etherpad.oae.com/p/c_cam_abc123?sessionID=abcef12345&signature=<..>

Each etherpad instance should have the [Sakai OAE Etherpad](https://github.com/sakaiproject/ep_oae) installed which takes care of verifying the signatures and initializing the session.

### Document publishing.

When publishing a document, the latest HTML from the pad will be fetched and stored in Cassandra on:
* the content object (for easy retrieval)
* a revision object.

Thus each publication creates a revision object in Cassandra.

### Document viewing

This is fairly straightforward, the HTML is available on the Content object at `contentObj.collabdoc.html` and can easily be returned.

## Problems
### 1. Etherpad doesn't have any support for clustering/instant failover (yet.)

The following is a naive implementation of spreading load.

Hilary knows each etherpad host and where it can reach them.
When a user creates a collabdoc we create a corresponding pad in etherpad. We use the content id as the padname. Because etherpad doesn't cluster very well, we need to direct all users for the same pad to the same etherpad instance. This happens as follows:
```
var servers = [ .. ]; // All the etherpad servers known to Hilary
var index = hash(contentId) % servers.length
var etherpadServer = servers[index];
```

Now this won't spread the load evenly so it is entirely possible that given 3 etherpad instances all users end up on the first one, but I can't see any other way to do it.

### 2. Adding/removing capacity
Since Hilary needs to know each etherpad instance's location adding extra nodes require changing the config.js and rebooting Hilary app nodes.

If we're assuming that etherpad instances are reachable on a subdomain that is predictable we could make 2 Admin UI config options as follows:
* How many etherpad instances do you have
* Provide the domain where they are reachable in the following format:
* http://${i}.etherpad.oae-performance.sakaiproject.org (i would be a positive integer with an excluse upper bound of the first option)

Note: Failover isn't really affected by this as that can happen at the proxy layer.

### 3. Performance
It's a bit of an unknown. Performance varies grossly on how many writers you have. Apparently lurkers cost next to nothing. There aren't any real good tools out there afaict.

## TODO
1. Restoring revisions

**TL;DR: Concurrent editting!**