dm crypt: Make crypto work queue multi-threaded
Revert "dm crypt: Make crypto work queue single-threaded"

This reverts commit 1f6706007f6de860dcd582526831e7e9c9f8d502.

Revert "Revert "dm crypt: add loop aes iv generator""

This reverts commit 0c1917ac99407a024f055008ae55afa334c256a2.

Revert "Revert "dm crypt: wipe keys string immediately after key is set""

This reverts commit ca0b08bd6d222955ea8f4e69aec8e18f18a02402.

dm-crypt: remove per-cpu structure

Dm-crypt used per-cpu structures to hold pointers to ablkcipher_request.
The code assumed that the work item keeps executing on a single CPU, so it
used no synchronization when accessing this structure.

When we disable a CPU by writing zero to
/sys/devices/system/cpu/cpu*/online, the work item could be moved to
another CPU. This causes crashes in dm-crypt because the code starts using
a wrong ablkcipher_request.

This patch fixes this bug by removing the percpu definition. The structure
ablkcipher_request is accessed via a pointer from convert_context.
Consequently, if the work item is rescheduled to a different CPU, the
thread still uses the same ablkcipher_request.

Signed-off-by: Mikulas Patocka <mpatocka@redhat.com>

Signed-off-by: Paul Reioux <reioux@gmail.com>

bio: use kmalloc alignment for bio slab

Various subsystems can ask the bio subsystem to create a bio slab cache
with some free space before the bio. This free space can be used for any
purpose. Device mapper uses this feature to place some target-specific and
device-mapper specific data before the bio, so that the target-specific
data doesn't have to be allocated separatedly.

This mechanism is used in place of kmalloc, so we need that the allocated
slab have the same memory alignment as memory allocated with kmalloc.

This patch changes the function bio_find_or_create_slab so that it uses
ARCH_KMALLOC_MINALIGN alignment when creating the slab cache. This patch
is needed so that dm-crypt can use per-bio data for encryption - the
crypto subsystem assumes that these data have the same alignment as
kmallocated memory.

Signed-off-by: Mikulas Patocka <mpatocka@redhat.com>

Signed-off-by: Paul Reioux <reioux@gmail.com>

dm-crypt: use per-bio data

This patch changes dm-crypt so that it uses auxiliary data allocated with
the bio.

Dm-crypt requires two allocations per request - struct dm_crypt_io and
struct ablkcipher_request (with other data appended to it). It used
mempool for the allocation.

Some requests may require more dm_crypt_ios and ablkcipher_requests,
however most requests need just one of each of these two structures to
complete.

This patch changes it so that the first dm_crypt_io and ablkcipher_request
and allocated with the bio (using target per_bio_data_size option). If the
request needs additional values, they are allocated from the mempool.

Signed-off-by: Mikulas Patocka <mpatocka@redhat.com>

backported for Linux 3.10

Signed-off-by: Paul Reioux <reioux@gmail.com>

dm: stop using WQ_NON_REENTRANT

dbf2576e37 ("workqueue: make all workqueues non-reentrant") made
WQ_NON_REENTRANT no-op and the flag is going away.  Remove its usages.

This patch doesn't introduce any behavior changes.

Signed-off-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Mike Snitzer <snitzer@redhat.com>
Signed-off-by: Alasdair G Kergon <agk@redhat.com>
Acked-by: Joe Thornber <ejt@redhat.com>

dm-crypt: use unbound workqueue for request processing

Use unbound workqueue so that work is automatically ballanced between
available CPUs.

Signed-off-by: Mikulas Patocka <mpatocka@redhat.com>

Signed-off-by: Paul Reioux <reioux@gmail.com>

dm-crypt: don't allocate pages for a partial request.

This patch changes crypt_alloc_buffer so that it always allocates pages for
a full request.

This change enables further simplification and removing of one refcounts
in the next patches.

Note: the next patch is needed to fix a theoretical deadlock

Signed-off-by: Mikulas Patocka <mpatocka@redhat.com>

backported for Linux 3.10

Signed-off-by: Paul Reioux <reioux@gmail.com>

dm-crypt: avoid deadlock in mempools

This patch fixes a theoretical deadlock introduced in the previous patch.

The function crypt_alloc_buffer may be called concurrently. If we allocate
from the mempool concurrently, there is a possibility of deadlock.
For example, if we have mempool of 256 pages, two processes, each wanting 256,
pages allocate from the mempool concurrently, it may deadlock in a situation
where both processes have allocated 128 pages and the mempool is exhausted.

In order to avoid this scenarios, we allocate the pages under a mutex.

In order to not degrade performance with excessive locking, we try
non-blocking allocations without a mutex first and if it fails, we fallback
to a blocking allocation with a mutex.

Signed-off-by: Mikulas Patocka <mpatocka@redhat.com>

backported for Linux 3.10

Signed-off-by: Paul Reioux <reioux@gmail.com>

dm-crypt: remove io_pool

Remove io_pool and _crypt_io_pool because they are unused.

Signed-off-by: Mikulas Patocka <mpatocka@redhat.com>

Signed-off-by: Paul Reioux <reioux@gmail.com>

dm-crypt: offload writes to thread

Submitting write bios directly in the encryption thread caused serious
performance degradation. On multiprocessor machine encryption requests
finish in a different order than they were submitted in. Consequently, write
requests would be submitted in a different order and it could cause severe
performance degradation.

This patch moves submitting write requests to a separate thread so that
the requests can be sorted before submitting.

Sorting is implemented in the next patch.

Note: it is required that a previous patch "dm-crypt: don't allocate pages
for a partial request." is applied before applying this patch. Without
that, this patch could introduce a crash.

Signed-off-by: Mikulas Patocka <mpatocka@redhat.com>

Signed-off-by: Paul Reioux <reioux@gmail.com>

dm-crypt: sort writes

Write requests are sorted in a red-black tree structure and are submitted
in the sorted order.

In theory the sorting should be performed by the underlying disk scheduler,
however, in practice the disk scheduler accepts and sorts only 128 requests.
In order to sort more requests, we need to implement our own sorting.

Signed-off-by: Mikulas Patocka <mpatocka@redhat.com>

Signed-off-by: Paul Reioux <reioux@gmail.com>

dm-crypt.c: fix merge derp

Signed-off-by: Paul Reioux <reioux@gmail.com>

dm-crypt: use 2x number of write threads to saturate each cpus pipeline

Signed-off-by: Paul Reioux <reioux@gmail.com>

Conflicts:
	drivers/md/dm-crypt.c