Implement new deferment styles for the thread fridge

 * Queue: What we had before, stick the job on a queue and immediately
          exit.

 * Block: Wait (with timeout) for a thread to become available.

 * Fail: Error on any submission when the fridge is full.

Switch request processing over to use blocking deferment.

A simple test (set the max to two, hammer it with three machines, and
insert prints on blocking and scheduling) shows that blocking
deferment works.  I'm not sure how to induce a timeout in request
processing, so as yet the behaviour of rearming the transport on
timeout isn't tested.

Signed-off-by: Adam C. Emerson <aemerson@linuxbox.com>
Rename fridgethr_get to fridgethr_submit

The name fridgethr_get did not make the function's purpose apparent.

Signed-off-by: Adam C. Emerson <aemerson@linuxbox.com>
Add per-task cleanup to thread fridge

Signed-off-by: Adam C. Emerson <aemerson@linuxbox.com>
Fix up formatting of FSAL_UP

It was a rewrite anyway, so there's nothing we can foreport from 1.5.
It's not that busy an area in the code.
And I'm about to make lots of changes anyway by stuffing it in the fridge.

Signed-off-by: Adam C. Emerson <aemerson@linuxbox.com>
Put FSAL UPcall thread in the fridge

And if we find ourselves with lots of upcall traffic, we can trivially
increase the size of the fridge.

Also take the opportunity to simplify upcall dispatch and get rid of
the gigantic switch statements.

Signed-off-by: Adam C. Emerson <aemerson@linuxbox.com>
Pull out stat thread launches

According to Jim Lieb they don't even compile.

Signed-off-by: Adam C. Emerson <aemerson@linuxbox.com>
Swallow up state async request thread

This is not tested beyond making sure it comes up and shuts down.  I
do not yet have an FSAL that supports asynchronous blocking locks, so
someone with one will need to go over this.

Signed-off-by: Adam C. Emerson <aemerson@linuxbox.com>
Add support for looper fridges

Signed-off-by: Adam C. Emerson <aemerson@linuxbox.com>
Add 'woke' entry to fridgethr_context

Signed-off-by: Adam C. Emerson <aemerson@linuxbox.com>
Swallow LRU

Signed-off-by: Adam C. Emerson <aemerson@linuxbox.com>
Swallow the Reaper.

Signed-off-by: Adam C. Emerson <aemerson@linuxbox.com>
Clean up no longer used stuff

Specifically get rid of the pthread_ts for things that are now in a
fridge, and bits of data related to the removed flusher threads (that
were part of the removed content cache.)

Signed-off-by: Adam C. Emerson <aemerson@linuxbox.com>
Clean up some no-longer-used parameters

Signed-off-by: Adam C. Emerson <aemerson@linuxbox.com>
Add per-thread initialize/finalizer and data

Signed-off-by: Adam C. Emerson <aemerson@linuxbox.com>
Add functionality to the fridge to let a fridge wait on its own

Signed-off-by: Adam C. Emerson <aemerson@linuxbox.com>
Swallow the worker threads

Signed-off-by: Adam C. Emerson <aemerson@linuxbox.com>