alf_queue_bench: start benchmarking queue bulking

This adds a simple benchmark for testing queue bulking.

The killer feature of the alf_queue implementation is the support
for "bulking", that is enqueue or dequeue of several elements in
one go.

The main saving of this bulk feature is to amortize the cost of
the locked cmpxchg operation per element.

(Results from E5-2695)

None-bulking:

 Type:ALF-simple Per elem: 27 cycles(tsc) 10.913 ns (step:0)
  - (measurement period time:0.218277470 sec time_interval:218277470)
  - (invoke count:20000000 tsc_interval:545692625)

 Type:ALF-multi Per elem: 33 cycles(tsc) 13.589 ns (step:128)
  - (measurement period time:0.347897160 sec time_interval:347897160)
  - (invoke count:25600000 tsc_interval:869741700)

Bulking results, cost per element operation:

 Type:ALF-bulk2 Per elem: 12 cycles(tsc) 5.193 ns (step:2)
  - (measurement period time:0.207737059 sec time_interval:207737059)
  - (invoke count:40000000 tsc_interval:519341734)

 Type:ALF-bulk4 Per elem: 8 cycles(tsc) 3.437 ns (step:4)
  - (measurement period time:0.274989687 sec time_interval:274989687)
  - (invoke count:80000000 tsc_interval:687473203)

 Type:ALF-bulk6 Per elem: 5 cycles(tsc) 2.269 ns (step:6)
  - (measurement period time:0.272292094 sec time_interval:272292094)
  - (invoke count:120000000 tsc_interval:680729225)

 Type:ALF-bulk8 Per elem: 5 cycles(tsc) 2.002 ns (step:8)
  - (measurement period time:0.320363793 sec time_interval:320363793)
  - (invoke count:160000000 tsc_interval:800908403)

 Type:ALF-bulk16 Per elem: 3 cycles(tsc) 1.365 ns (step:16)
  - (measurement period time:0.436851012 sec time_interval:436851012)
  - (invoke count:320000000 tsc_interval:1092126213)

Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
alf_queue: reorganize __helper_alf_{enqueue,dequeue}

Group the enqueue and dequeue helpers together. In preperation
for testing different variants for these helpers for
optimal performance benefits.

Only a single of these helpers will survive upstream submission.
alf_queue: optimize simple helper with "unlikely" branch

The added "unlikely()" statement to help the branch prediction
does help when doing bulking in:
__helper_alf_enqueue_store_simple()
__helper_alf_dequeue_load_simple

The before and after:

 bulk2 : 6.126 ns -> 6.177 ns
 bulk4 : 3.758 ns -> 3.192 ns
 bulk6 : 2.957 ns -> 2.507 ns
 bulk8 : 2.502 ns -> 2.023 ns
 bulk16: 1.882 ns -> 1.443 ns

This is an improvement, but compared to the "mask" helper, it is
still slightly slower:

Comparing enqueue/dequeue helpers "simple" -> "mask".

 bulk2 : 6.177 ns -> 5.193 ns
 bulk4 : 3.192 ns -> 3.371 ns
 bulk6 : 2.507 ns -> 2.269 ns
 bulk8 : 2.023 ns -> 1.997 ns
 bulk16: 1.443 ns -> 1.493 ns

Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
alf_queue: loop unrolled enqueue/dequeue helpers

Implemented:
 __helper_alf_enqueue_store_unroll()
 __helper_alf_dequeue_load_unroll()

Which implements loop unrolling in case of bulking.

Comparing enqueue/dequeue helpers "mask" -> "unroll":

It is interesting to see that none-bulking benchmarks is not hurt,
it is even slightly improved:

 (none-bulking)
 simple: 10.912 ns -> 10.149 ns
 multi : 13.589 ns -> 13.579 ns

The loop unrolling does improve performance, but not much, and
mostly for larger and 4 aligned cases.

 (bulking)
 bulk2 :  5.193 ns -> 5.489 ns
 bulk4 :  3.371 ns -> 3.225 ns
 bulk6 :  2.269 ns -> 2.211 ns
 bulk8 :  1.997 ns -> 1.907 ns
 bulk16:  1.493 ns -> 1.197 ns

I was hoping to see a larger effect of this loop unrolling, not
because of the unroll, but because I was hoping to see a memory
pipeline effect.

Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
alf_queue: reduce "AND" operations in enqueue/dequeue helper "mask_less"

It is possible to avoid the masking of the index with q->mask,
if we know that wrapping will not occur.

This is implemented in:
 __helper_alf_enqueue_store_mask_less()
 __helper_alf_dequeue_load_mask_less()

Comparing enqueue/dequeue helpers "mask" -> "mask_less":

It is somewhat surprising to see the improvement for the
none-bulking case.

 (none-bulking)
 simple: 10.912 ns ->  9.438 ns
 multi : 13.589 ns -> 13.411 ns

There is an regression in case of bulking only 2 elements, but
else its a slight improvement.

 (bulking)
 bulk2 :  5.193 ns -> 6.785 ns
 bulk4 :  3.371 ns -> 3.214 ns
 bulk6 :  2.269 ns -> 2.388 ns
 bulk8 :  1.997 ns -> 1.998 ns
 bulk16:  1.493 ns -> 1.399 ns

Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
alf_queue: extending "unroll" with masked-AND avoidance

In the "unroll" helper, use the trick to avoid the masking of the
index with q->mask, if we know that wrapping will not occur (like
previous commit).

Comparing enqueue/dequeue helpers:
 "mask" -> "unroll" -> new "unroll" (without q->mask):

 (none-bulking)
          "mask"   -> "unroll"  -> new "unroll"
 simple: 10.912 ns -> 10.149 ns ->  9.133 ns
 multi : 13.589 ns -> 13.579 ns -> 13.232 ns

The only outliner is bulk2, else we see slight improvements.

 (bulking)
 bulk2 :  5.193 ns -> 5.489 ns -> 6.166 ns
 bulk4 :  3.371 ns -> 3.225 ns -> 3.137 ns
 bulk6 :  2.269 ns -> 2.211 ns -> 2.242 ns
 bulk8 :  1.997 ns -> 1.907 ns -> 1.955 ns
 bulk16:  1.493 ns -> 1.197 ns -> 1.191 ns

Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
alf_queue: optimize "unroll" for remainder handling

The bulk2 case were not very good for "unroll" helper.  Try to
fix that.

The bulk2 case would always hit the remainder handling
in "unroll", which deployes a Duff's device fall-through switch
trick.

Drop the Duff's device trick, and instead optimize the remainder
handling explicitly.

Comparing enqueue/dequeue helpers:
 "mask" -> old "unroll" -> "unroll" (without q->mask) -> his commit

 (none-bulking)
 simple: 10.912 ns -> 10.149 ns ->  9.133 ns ->  9.444 ns
 multi : 13.589 ns -> 13.579 ns -> 13.232 ns -> 13.409 ns

This seems to have restored bulk2 performance:

 (bulking)
 bulk2 :  5.193 ns -> 5.489 ns -> 6.166 ns -> 5.407 ns
 bulk4 :  3.371 ns -> 3.225 ns -> 3.137 ns -> 3.137 ns
 bulk6 :  2.269 ns -> 2.211 ns -> 2.242 ns -> 2.155 ns
 bulk8 :  1.997 ns -> 1.907 ns -> 1.955 ns -> 1.915 ns
 bulk16:  1.493 ns -> 1.197 ns -> 1.191 ns -> 1.191 ns

Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>