rename 'add_recovery_fn' to 'push_recovery_fn'

Rename to 'push_recovery_fn' and provide a 'pop_recovery_fn'
to remove the last entry on the array.

Signed-off-by: Hannes Reinecke <hare@suse.de>
Testcase 16: Reboot during recovery

Add monitor_testcase_16: reboot during recovery.

Signed-off-by: Hannes Reinecke <hare@suse.de>
monitor_testcase_functions: stop iostat properly

Update stop_iotest() to not print out the annoying message
'Terminated iostat' when shutting down the testcase.

Signed-off-by: Hannes Reinecke <hare@suse.de>
monitor_testcase_functions: Update DASD layout on linux021

Signed-off-by: Hannes Reinecke <hare@suse.de>
monitor_testcase_functions: abort test if programs could not be started

If either md_monitor or mdadm failed to start the whole test is
not likely to succeed, so we should abort directly here.

Signed-off-by: Hannes Reinecke <hare@suse.de>
monitor_testcase_functions: stop iostat before stopping array

Having iostat running when the array is already down is pointless
and will only lead to errors. So terminate iostat before shutting
down the array.

Signed-off-by: Hannes Reinecke <hare@suse.de>
monitor_testcase_1: Fixup system logging

We should register in syslog the start time and the final result
of the test.

Signed-off-by: Hannes Reinecke <hare@suse.de>
monitor_testcase_2: Fixup system logging

Signed-off-by: Hannes Reinecke <hare@suse.de>
md_monitor: print current state when failing a device

When failing a device we should be printing out the current state.
This makes debugging easier.

Signed-off-by: Hannes Reinecke <hare@suse.de>
monitor_testcase_2: Add 'wait_md' after failing devices

Calling 'mdadm --fail <list of devices>' has the unfortunate
effect that mdadm will be calling 'Fail' for each device in turn.
This will confuse md_monitor as it'll start recovery for the first
device, then get a 'Fail' request for the second, and start failing
the entire array _again_.

So we should rather wait for md to finish the 'failing' sequence
before proceeding further.

Signed-off-by: Hannes Reinecke <hare@suse.de>
md_monitor: Fixup typo in discover_md_components

There was a typo in discover_md_components which cause a
re-discovery after recovery to issue error messages and
the device to be not attached correctly.

Signed-off-by; Hannes Reinecke <hare@suse.de>
monitor_testcase_3: use 'MonitorStatus' instead of 'ArrayStatus'

ArrayStatus prints out the full description of the array, within
which the order of the devices is basically arbitrary. In particular
it might change after reassembling, causing random failures.
As we're only interested that md_monitor picked up every device
we can just compare the 'MonitorStatus' CLI command which
doesn't suffer from this issue.

Signed-off-by: Hannes Reinecke <hare@suse.de>
monitor_testcase_3: Fixup system logging

Signed-off-by: Hannes Reinecke <hare@suse.de>
monitor_testcase_4: Use push_recovery_fn

Instead of hardcoding a reset for each and every possible
device upon exit we should be pushing a reset function
for the device in question.

Signed-off-by: Hannes Reinecke <hare@suse.de>
monitor_testcase_5: Fix vary on/off setting

For some reason it's not possible to have a 'configure standby'
on linux021. As the testcase works perfectly without and disconnects
the devices as desired we can ignore this error and continue with
the testcase.

Signed-off-by: Hannes Reinecke <hare@suse.de>
monitor_testcase_6: Fixup system logging

Signed-off-by: Hannes Reinecke <hare@suse.de>