qmempool: save one call to in_serving_softirq() on fastpath

Its possible to avoid calling in_serving_softirq() twice on
our fast-path, by propagating this state between the calls
__qmempool_preempt_disable() -> __qmempool_preempt_enable().

Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
qmempool: be consistant with variable name "elem"

The code uses variable name "elem" for an element, except in function
main allocation function __qmempool_alloc_node(), where "element" name
is used.  Choose to be consistant and change this to "elem".

Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
qmempool: only support GFP_ATOMIC alloc use-case

The preemption disable/enable mechanism were getting too complicated
to allow SLAB alloactions to run with preemption enabled.

Instead focus our qmempool use-case for preemption safe contexts,
which is required anyway for the fast-path lock-free queue usage.
Thus, we deem that we don't loose too much by adding this limit
to the use-cases qmempool can be applied.

Change also include removing:  qmempool_alloc_node().

And adding: qmempool_alloc_softirq(), which can avoid any of the
preempt checks, as it must only be used from softirq context.

Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
qmempool: remove BUG_ON(in_irq()) check

When relying on the protection acheived from softirq context, then
qmempool must not be used from interrupt context.  We had a check
with BUG_ON(in_irq()) in __qmempool_preempt_disable() but practical
benchmarks show that this cost too much to enable per default.

Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
qmempool: remove BUG_ON(in_irq()) check in __qmempool_free_to_sharedq()

Remove another BUG_ON() check in __qmempool_free_to_sharedq() which is
only needed if the alf_queue API changes.  If it does change we should
just update this code, instead of keeping this runtime check.

Add a notes about to watchout for alf_queue API changes.

Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
qmempool: move __qmempool_free_to_sharedq() to qmempool.c file

After the preempt disable/enable usage changes, the function
__qmempool_free_to_sharedq() can now be moved to qmempool.c file.

Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
qmempool: move code from __qmempool_free() to __qmempool_free_to_sharedq()

Moved some code from __qmempool_free() to __qmempool_free_to_sharedq()
to help icache, but this might not be good because __qmempool_free_to_sharedq()
will more likely need reloading to icache.

Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
qmempool: remove some optimization code to reduce code size

The remove the optimization of adding the cache hot element to localq,
in __qmempool_free_to_sharedq(), because it seems the cost of the
increased code/icache size matters more (in full forwarding test).

Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>