csfrancis/semian — pull request #3 by csfrancis (2014-10-03 18:00:58+00:00)
Title: Register refactor

This refactors `Semian.register` to make it safe to be called across multiple processes.

### Problem
When rolling out Semian on Shopify job servers, multiple job server processes are calling `register` with a non-zero ticket count. This caused Semian to reset the ticket count on every call, even if some tickets were already in use. So for example:
```
WORKER1: WORKER2:
register(100)
acquire register(100)
release
```
When `WORKER1` releases the ticcket, the count will now be 101, which is higher than the configured value of 100.

Another problem is that there is no safe way to resize the number of tickets on a resource.

### Solution
The solution updates the underlying semaphore set to contain 3 values:
1. The ticket count
2. The maximum ticket count
3. A resource lock

In almost all cases, when `register` is called, we check if the ticket count passed is the same as the maximum ticket count and if they are equal, we simply return. There are some special cases:

* The ticket count has changed:
* In this case, we acquire the resource lock and add the delta of (tickets - max_ticket_count) to the existing ticket count. This allows us to resize the semaphore without concerning any tickets that may already be acquired.
* Multiple workers trying to create the resource at the same time:
* When we create the underlying semaphore, we pass `IPC_EXCL`. If the resource already exists, the current client will not attempt to initialize the semaphore and will wait until it's been initialized by another worker. If the resource does not exist, this worker will be responsible for configuring the initial ticket counts and resource lock.

I've tried to make these internal operations resilient, in that they should never block forever and will fail fast. That being said, if they DO fail, things will probably be in a very bad state and the underlying semaphores will most likely need to be deleted in order to recover.