raid5: create multiple threads to handle stripes

This is a new tempt to make raid5 handle stripes in multiple threads, as
suggested by Neil to have maxium flexibility and better numa binding. It
basically is a combination of my first and second generation patches. By
default, no multiple thread is enabled (all stripes are handled by raid5d).

An example to enable multiple threads:
#echo 3 > /sys/block/md0/md/auxthread_number
This will create 3 auxiliary threads to handle stripes. The threads can run
on any cpus and handle stripes produced by any cpus.

#echo 1-3 > /sys/block/md0/md/auxth0/cpulist
This will bind auxiliary thread 0 to cpu 1-3, and this thread will only handle
stripes produced by cpu 1-3. User tool can further change the thread's
affinity, but the thread can only handle stripes produced by cpu 1-3 till the
sysfs entry is changed again.

If stripes produced by a CPU aren't handled by any auxiliary thread, such
stripes will be handled by raid5d. Otherwise, raid5d doesn't handle any
stripes.

Signed-off-by: Shaohua Li <shli@fusionio.com>
Signed-off-by: NeilBrown <neilb@suse.de>
md: never update metadata when array is read-only.

Normally we don't even try to update the metadata if
the array is read-only.  However future patches
will increase the number of things that can happen on a read-only
array, so it is safest to explicitly disable this.

Every time that mddev->ro is set to 0, either
 - md_update_sb will be called again (at least if MD_CHANGE_DEVS
   is set) or
 - the mddev->thread is scheduled, which will also run
   md_update_sb if needed.

So this is safe: if the array ever become read-write the
metadata will be updated.

Signed-off-by: NeilBrown <neilb@suse.de>
md: use common code for all calls to ->hot_remove_disk()

slot_store and remove_and_add_spares both call ->hot_remove_disk(),
but with slightly different tests and consequences, which is
at least untidy and might be buggy.

So modify remove_and_add_spaces() so that it can be asked
to remove a specific device, and call it from slot_store().

We also clear the Blocked flag to ensure that doesn't prevent
removal.  The purpose of Blocked is to prevent automatic removal
by the kernel before an error is acknowledged.
If the array is read/write then user-space would have not reason
to remove a device unless it was known to be 'spare' or 'faulty' in
which it would have already cleared the Blocked flag.
If the array is read-only, the flag might still be blocked, but
there is no harm in clearing the flag for read-only arrays.

Signed-off-by: NeilBrown <neilb@suse.de>
md: HOT_DISK_REMOVE shouldn't make a read-auto device active.

If a fail device or a spare is removed from an array, there is
not need to make the array 'active'.  If/when the array does become
active for some other reason the metadata will be update to reflect
the removal.
If that never happens and the array is stopped while still read-auto,
then there is no loss in forgetting the that the device had 'failed'.

A read-only array will leave failed devices attached to
the array personality, so we need to explicitly call
remove_and_add_spares() to free it (clearing Blocked just
like we do in store_slot()).

Signed-off-by: NeilBrown <neilb@suse.de>
md: use set_bit_le and clear_bit_le

The value returned by test_and_set_bit_le() drivers/md/bitmap.c is not used.
So just use set_bit_le(). The same goes for test_and_clear_bit_le().

Signed-off-by: Akinobu Mita <akinobu.mita@gmail.com>
Cc: Neil Brown <neilb@suse.de>
Cc: linux-raid@vger.kernel.org
Signed-off-by: NeilBrown <neilb@suse.de>
md/raid5: Change or of some order to improve efficiency.

As the function call is the most expensive of these tests it should be
done later in the chain so that it can be avoided in some cases.

Signed-off-by: Jianpeng Ma <majianpeng@gmail.com>
Signed-off-by: NeilBrown <neilb@suse.de>
md/raid5: avoid an extra write when writing to a known-bad-block.

If we write to a known-bad-block it will be flags as having
a ReadError by analyse_stripe, but the write will proceed anyway
(as it should).  Then the read-error handling will kick in an
write again, then re-read.

We don't need that 'write-again', so set R5_ReWrite so it looks like
it has already been done.  Then we will just get the re-read, which we
want.

Reported-by: majianpeng <majianpeng@gmail.com>
Signed-off-by: NeilBrown <neilb@suse.de>
MD: Fix typos in MD documentation

MD:  Fix some typos/grammer in MD documentation

Reviewed-by: Paul Menzel <paulepanter@users.sourceforge.net>
Signed-off-by: Jonathan Brassow <jbrassow@redhat.com>
Signed-off-by: NeilBrown <neilb@suse.de>
md/raid10: Allow skipping recovery when clean arrays are assembled

When an array is assembled incrementally with mdadm -I -R
and the array switches to "active" mode, md starts a recovery.

If the array was clean, the "fullsync" flag will be 0. Skip
the full recovery in this case, as RAID1 does (the code was
actually copied from the sync_request() method of RAID1).

Signed-off-by: Martin Wilck <mwilck@arcor.de>
Signed-off-by: NeilBrown <neilb@suse.de>