KeyBasedLocalStorageLink resend on error

Quoting an email discussion:
My recommendation would be to make "send" put the packet on a FIFO queue.  Have a sendImpl (or better name) function that attempts to send everything on the queue.  This would let us implement packet fragmentation, later, by just pushing the fragments onto the queue. 

There are only two reasons that we couldn't write the packet.  The first is that the packet being larger than the localStorage quota (5mb on non-IE, 10mb on IE).  This we just need to prevent, or better yet implement fragmenting. 

The other reason is that there's enough in-flight packets such that there's no room left for this packet.  The window for an in-flight packet is extremely small, though-- one line of JavaScript:

localStorage.setItem(JSON.stringify(packet),"");
ozpIwc.metrics.counter('links.keyBroadcastLocalStorage.packets.sent').inc();
localStorage.removeItem(JSON.stringify(packet),"");


The only window for a packet to take space against the quota is after the release of the mutex in setItem and before it's acquired in removeItem.  So we can use some rather aggressive retry counts.

Another consideration is that Chrome, Firefox, and possibly other browsers (I haven't checked), a tab that is not in focus will have any timers padded out to 1 second.

My thought is to try to resend 3 times immediately to optimistically try to get past the one-line collision window.  Only after that, set an aggressive timer at max(1,2^(X-1))-1 with X starting at 0 and an implicit Z of 1ms.  This would give retries of at 0,0,1,3,7,15, etc, simply yielding the event loop on the first couple tries.  The max retries for this should be fairly small.  If we're still colliding, it's not likely to resolve.

This is different than how TCP does it because the TCP backoff is focused on congestion, we're only focused on collision.  Congestion, in our case, is going to present itself via memory use due to backlogs of storage events, which will have to be dealt with separately.