State machine for start, stop, and pause

Stop has been tested and works, pause may not work yet.

Signed-off-by: Adam C. Emerson <aemerson@linuxbox.com>
Implement new deferment styles for the thread fridge

 * Queue: What we had before, stick the job on a queue and immediately
          exit.

 * Block: Wait (with timeout) for a thread to become available.

 * Fail: Error on any submission when the fridge is full.

Switch request processing over to use blocking deferment.

A simple test (set the max to two, hammer it with three machines, and
insert prints on blocking and scheduling) shows that blocking
deferment works.  I'm not sure how to induce a timeout in request
processing, so as yet the behaviour of rearming the transport on
timeout isn't tested.

Signed-off-by: Adam C. Emerson <aemerson@linuxbox.com>
Rename fridgethr_get to fridgethr_submit

The name fridgethr_get did not make the function's purpose apparent.

Signed-off-by: Adam C. Emerson <aemerson@linuxbox.com>
Add per-task cleanup to thread fridge

Signed-off-by: Adam C. Emerson <aemerson@linuxbox.com>
Fix up formatting of FSAL_UP

It was a rewrite anyway, so there's nothing we can foreport from 1.5.
It's not that busy an area in the code.
And I'm about to make lots of changes anyway by stuffing it in the fridge.

Signed-off-by: Adam C. Emerson <aemerson@linuxbox.com>
Put FSAL UPcall thread in the fridge

And if we find ourselves with lots of upcall traffic, we can trivially
increase the size of the fridge.

Also take the opportunity to simplify upcall dispatch and get rid of
the gigantic switch statements.

Signed-off-by: Adam C. Emerson <aemerson@linuxbox.com>
Pull out stat thread launches

According to Jim Lieb they don't even compile.

Signed-off-by: Adam C. Emerson <aemerson@linuxbox.com>
Swallow up state async request thread

This is not tested beyond making sure it comes up and shuts down.  I
do not yet have an FSAL that supports asynchronous blocking locks, so
someone with one will need to go over this.

Signed-off-by: Adam C. Emerson <aemerson@linuxbox.com>
Add support for looper fridges

Signed-off-by: Adam C. Emerson <aemerson@linuxbox.com>
Add 'woke' entry to fridgethr_context

Signed-off-by: Adam C. Emerson <aemerson@linuxbox.com>
Support changing the wait time of a running fridge

Signed-off-by: Adam C. Emerson <aemerson@linuxbox.com>
Swallow LRU

Signed-off-by: Adam C. Emerson <aemerson@linuxbox.com>
Swallow the Reaper.

Signed-off-by: Adam C. Emerson <aemerson@linuxbox.com>
Clean up no longer used stuff

Specifically get rid of the pthread_ts for things that are now in a
fridge, and bits of data related to the removed flusher threads (that
were part of the removed content cache.)

Signed-off-by: Adam C. Emerson <aemerson@linuxbox.com>
Clean up some no-longer-used parameters

Signed-off-by: Adam C. Emerson <aemerson@linuxbox.com>
Add per-thread initialize/finalizer and data

Signed-off-by: Adam C. Emerson <aemerson@linuxbox.com>
Let looper processes do their own waiting

This model avoids infecting the loopers with the fridge's state
machine logic.  The fridge parameters provide the fridge with a
function that will wake all sleepers, and the fridge provide a
function, fridgethr_you_should_break that the function should call
before sleeping or doing work (and return if it's true.)

Signed-off-by: Adam C. Emerson <aemerson@linuxbox.com>
Swallow the worker threads

Signed-off-by: Adam C. Emerson <aemerson@linuxbox.com>
Remove TCB

Signed-off-by: Adam C. Emerson <aemerson@linuxbox.com>
Cache Inode Destroyer

Add functionality to break down the inode cache, closing file
descriptors, and relasing state in the FSAL.  This is all done without
taking any locks so that the shutdown procedure doesn't hang in the
event of a thread being cancelled while holding a lock.

Signed-off-by: Adam C. Emerson <aemerson@linuxbox.com>