avoid the spin lock call if there is nothing

Signed-off-by: Sébastien Boisvert <boisvert@anl.gov>
add comment about which code call spin_lock

Profiling before 4f84f9c90b:

Each sample counts as 0.01 seconds.
  %   cumulative   self              self     total
 time   seconds   seconds    calls  ms/call  ms/call  name
 66.36     27.40    27.40 126206996     0.00     0.00  bsal_fifo_pop
 28.70     39.25    11.85                             bsal_worker_thread_main
  3.51     40.70     1.45 124603913     0.00     0.00  bsal_fifo_array_pop
  0.90     41.07     0.37                             bsal_fifo_array_unit
  0.53     41.29     0.22      156     1.41     1.41  bsal_fifo_push

After 4f84f9c90b:

Each sample counts as 0.01 seconds.
  %   cumulative   self              self     total
 time   seconds   seconds    calls  us/call  us/call  name
 43.33      0.07     0.07   743509     0.09     0.09  bsal_fifo_empty
 26.67      0.11     0.04                             bsal_worker_thread_main
 10.00      0.12     0.02                             bsal_fifo_bytes
  6.67      0.13     0.01     5070     1.97     1.97  bsal_worker_pool_run
  6.67      0.14     0.01      739    13.53    13.53  bsal_actor_dead

bsal_fifo_empty reads a volatile variable and uses no spin lock when
the FIFO is empty.

Signed-off-by: Sébastien Boisvert <boisvert@anl.gov>
improve documentation in readme

Signed-off-by: Sébastien Boisvert <boisvert@anl.gov>
add runtime parameter -workers-per-node

This sets the number of workers per node.

Signed-off-by: Sébastien Boisvert <boisvert@anl.gov>