xfstests 297: test xfs direct IO nested transaction deadlock

Regression test case for commit:

437a255 xfs: fix direct IO nested transaction deadlock.

Signed-off-by: Eryu Guan <eguan@redhat.com>
Reviewed-by: Ben Myers <bpm@sgi.com>
Signed-off-by: Ben Myers <bpm@sgi.com>
xfstests: Provide dumpe2fs via standard common.config interface

dumpe2fs can be now accessed via $DUMPE2FS_PROG, tests that require it
can check for its availability by _require_dumpe2fs function.

Signed-off-by: Tomas Racek <tracek@redhat.com>
Reviewed-by: Ben Myers <bpm@sgi.com>
Signed-off-by: Rich Johnston <rjohnston@sgi.com>
xfstests: Add new standard loop handling functions

Add _create_loop_device and _destroy_loop_device to uniformly handle
loopback devices.

V1->V2: Fix possible race

Signed-off-by: Tomas Racek <tracek@redhat.com>
Reviewed-by: Rich Johnston <rjohnston@sgi.com>
Signed-off-by: Rich Johnston <rjohnston@sgi.com>
xfstests: 298: Test that FS sends discard requests only on free blocks

V1->V2: Change way of testing suggested by Dave Chinner
1. Create image file with FS on it
2. Call fstrim to discard free blocks
3. Check that every punched hole in the image file is in the area
   that is marked as free

Signed-off-by: Tomas Racek <tracek@redhat.com>
Reviewed-by: Rich Johnston <rjohnston@sgi.com>
Signed-off-by: Rich Johnston <rjohnston@sgi.com>
xfstests: add fio requirement V2

FIO is very flexible io generator, I would call it IO swiss knife.
Currently we have tons of hardcoded application which reproduces
some predefined scenario. This approach has obvious disadvantages
1) Lack of flexibility: one written it is hard to modify it in future
2) Code base is large, many routines written again and again

At the same time add new fio based test, just add simple INI file.
This greatly simplifies code review. I do believe that some day we will
replace most of hardcoded io binaries with fio.

One who is planning to run $FIO_PROG should first check that system
contains appropriate version which is able to handle jobfile
for example:  _require_fio 286-job.fio

Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
Reviewed-by: Rich Johnston <rjohnston@sgi.com>
Signed-off-by: Rich Johnston <rjohnston@sgi.com>
xfstest: add configurable load factors

Most stress test has probable behavior, the longer test run the
larger corner cases will be cover. It is reasonable to allow
user to provide some sort of system load factor.
This patch introduce two global variables
LOAD_FACTOR: Usually means factor number of running tasks
TIME_FACTOR: Usually means factor of run time, or number of operations
If not specified both variables defined to 1, so original behavior
preserved.

TODO: Change all stress tests to use this variables

Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
Reviewed-by: Rich Johnston <rjohnston@sgi.com>
Signed-off-by: Rich Johnston <rjohnston@sgi.com>
xfstests: hardcode fops for determinable fsstests runs

106,107 and 117 are frozen tests which use known seed, it is
reasonable to explicitly hardcode file operations in order to avoid
implicit changes caused by future changes in fsstress.

NOTE: options generated like follows: fsstress -S c $ORIG_ARGS

Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
Reviewed-by: Rich Johnston <rjohnston@sgi.com>
Signed-off-by: Rich Johnston <rjohnston@sgi.com>
xfstest: allow fsstress to use load factor where appropriate

1) Add _scale_fsstress_args function which transform arguments according
   to load factors
2) Let all non deterministic fsstress tests to use scaled arguments

Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
Reviewed-by: Rich Johnston <rjohnston@sgi.com>
Signed-off-by: Rich Johnston <rjohnston@sgi.com>
xfstest: move run_check to common.rc

Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
Reviewed-by: Rich Johnston <rjohnston@sgi.com>
Signed-off-by: Rich Johnston <rjohnston@sgi.com>

Move run_check to common.rc.
xfstest: add fallocate/truncate vs AIO/DIO stress test

Run DIO, fallocate and truncate threads on a common file in parallel.
If a race exists, the old dio request may rewrite blocks after it was
allocated to another file, we will catch that by verifying blocks content.

this patch known to catch deadlock for ext4
http://lists.openwall.net/linux-ext4/2012/09/06/3

Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
Reviewed-by: Rich Johnston <rjohnston@sgi.com>
Signed-off-by: Rich Johnston <rjohnston@sgi.com>
xfstest: add fallocate/punch_hole vs AIO/DIO stress test

Run random AIO/DIO activity (fio's job:direct_aio_raicer)
random fallocate activity(fio's job:falloc_raicer)
and random punch_hole activity(punch_hole_raicer) on a common
file in parallel. If a race exists, old dio request may rewrite
punched block after it was allocated to another file, we will
catch that by verifier fio's job: "aio-dio-verifier".

Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
Reviewed-by: Rich Johnston <rjohnston@sgi.com>
Signed-off-by: Rich Johnston <rjohnston@sgi.com>
xfstest: add defragmentation stress tests for ext4

Perform various regression tests for ext4defrag subsystem

 301'th Test1: Defragment file while other task does direct AIO
 302'th Test2: Perform defragmentation on file under buffered AIO
 	while third task does direct AIO to donor file
 303'th Test3: Two defrag tasks use common donor file.
 304'th Test4: Stress defragmentation. Several threads perform
 	fragmentation at random position use inplace=1 will
	allocate and free blocks inside defrag event improve
	load pressure.

This tests are considered dangerous because 302'th and 303'th are known
to trigger OOPS on recent kernels see:https://gist.github.com/dmonakhov/4770294

Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
Reviewed-by: Rich Johnston <rjohnston@sgi.com>
Signed-off-by: Rich Johnston <rjohnston@sgi.com>
xfstests: add disk failure simulation test

There are many situations where disk may fail for example
1) brutal usb dongle unplug
2) iscsi (or any other netbdev) failure due to network issues
In this situation filesystem which use this blockdevice is
expected to fail(force RO remount, abort, etc) but whole system
should still be operational. In other words:
1) Kernel should not panic
2) Memory should not leak
3) Data integrity operations (sync,fsync,fdatasync, directio) should fail
   for affected filesystem
4) It should be possible to umount broken filesystem

Later when disk becomes available again we expect(only for journaled filesystems):
5) It will be possible to mount filesystem w/o explicit fsck (in order to catch
   issues like https://patchwork.kernel.org/patch/1983981/)
6) Filesystem should be operational
7) After mount/umount has being done all errors should be fixed so fsck should
   not spot any issues.

This test use fault injection (CONFIG_FAULT_INJECTION=y,
CONFIG_FAIL_MAKE_REQUEST=y and CONFIG_FAULT_INJECTION_DEBUG_FS=y config
options) to force all new IO requests to fail for a given device. Xfs
already has XFS_IOC_GOINGDOWN ioctl which provides similar behavior, but it
is fs specific and it does it in an easy way because it performs freeze_bdev()
before actual shutdown.

Test run fsstress in background and then force disk failure.
Once disk failed it check that (1)-(4) is true.
Then makes disk available again and check that (5)-(7) is also true

BE CAREFUL!! test known to cause memory corruption for XFS
see: https://gist.github.com/dmonakhov/4953045

Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
Reviewed-by: Rich Johnston <rjohnston@sgi.com>
Signed-off-by: Rich Johnston <rjohnston@sgi.com>
xfstests: use right defrag program path

In _require_defrag defragmentation command path is fixed.  That will
cause that this test case is skipped in some distributions.

Signed-off-by: Zheng Liu <wenqing.lz@taobao.com>
Reviewed-by: Eric Sandeen <sandeen@redhat.com>
Signed-off-by: Rich Johnston <rjohnston@sgi.com>
xfstests 273: fix a typo

Make the porter log cp failure into $seq.full by appending, not overwriting,
which can help debug.

Signed-off-by: Wang Sheng-Hui <shhuiw@gmail.com>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Rich Johnston <rjohnston@sgi.com>
xfstests 276: fix error 'FIBMAP: Invalid argument'

Commit 05dadc0 ("Btrfs: add fiemap's flag check") added validity
checks to the fiemap flags and hence invalid flags are now being
rejected. Test 276 passes an invalid fiemap flag to btrfs, and so
that test fails since this commit was added.

Btrfs doesn't support FIEMAP_FLAG_XATTR, which is enabled by -x option
of filefrag, and will fail with
	'FIBMAP: Invalid argument'
for 'filefrag -vx'. 'filefrag -vx' fails on btrfs with
     'FIEMAP failed with unsupported flags 2'
Remove the '-x' option.

Signed-off-by: Wang Sheng-Hui <shhuiw@gmail.com>
Reviewed-by: Eric Sandeen <sandeen@redhat.com>
Signed-off-by: Rich Johnston <rjohnston@sgi.com>
xfstests: don't assume that falloc_punch implies falloc in test 255

As of Linux 3.9-rc1, ext4 will support the punch operation on file
systems using indirect blocks, but it can not support the fallocate
operation (since there is no way to mark a block as uninitialized
using indirect block scheme).  This caused test 255 to fail, since it
only used _require_xfS_io_falloc_punch assuming that all file systems
which supported punch can also support fallocate.  Fix this.

Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
Reviewed-by: Rich Johnston <rjohnston@sgi.com>
Signed-off-by: Rich Johnston <rjohnston@sgi.com>
xfstests: don't remove trailing zeros from integers

_within_tolerance strips trailing zeros from the min and max range
values it outputs.  This leads to damage if the min or max value is
an integer containing trailing zeros rather than a real number with
a fractional part containing trailing zeros. Xfstest 289 can exhibit
this problem when its input is out of range.  Modify the code so it
will only remove trailing zeros found after a decimal point.

V1->V2: Remove decimal points not followed by digits
V2->V3: Per Dave Chinner, simplify by using multiple sed expressions

Signed-off-by: Eric Whitney <enwlinux@gmail.com>
Reviewed-by: Eric Sandeen <sandeen@redhat.com>
Signed-off-by: Rich Johnston <rjohnston@sgi.com>
xfstests: use native definition of O_DIRECT flag

The definition of O_DIRECT in src/trunc.c causes xfstest 125 to fail
when run on a Pandaboard.  On ARM, the value used (0x040000) is
O_DIRECTORY rather than O_DIRECT as it is on x86.  Prefer the platform's
native definition of O_DIRECT supplied by fcntl.h if available.  Also,
fix a couple of error messages to properly reflect their context.

Signed-off-by: Eric Whitney <enwlinux@gmail.com>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Rich Johnston <rjohnston@sgi.com>
xfstests: handle new mkfs.btrfs -f option cleanly

I added an "-f" option to mkfs.btrfs to force overwrite
of an existing filesystem.  Now on an xfstests run, new
mkfs.btrfs requires it, and old mkfs.btrfs cannot accept
it.

So, add a helper which works out whether -f is needed,
and add it to the MKFS_BTRFS_PROG env. var as necessary,
so that it is an always-included option during the tests.

Signed-off-by: Eric Sandeen <sandeen@redhat.com>
Reviewed-by: Rich Johnston <rjohnston@sgi.com> 
Signed-off-by: Rich Johnston <rjohnston@sgi.com>