client: fix compile warning on platforms where pfds->fd is not an int

It's apparently a long on some platforms.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
Rework descriptions for mailing list, options, job files

In general, make the existing document less ambiguous:

o Use consistent formatting/layout when describing options
o always refer to options with "--" prefix.
o reduce use of pronouns: "its" and "this" are often ambigous.
o remove use of "you" to describe user actions/choices
o add reference to examples/ directory

Signed-off-by: Grant Grundler <grundler@chromium.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Remove duplicate LDFLAGS on -o lines

Only need to list LDFLAGS once on each line.

Signed-off-by: Grant Grundler <grundler@chromium.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Fix problem with --showcmd and callbacks that verify

David reports:

I'm using "fio --showcmd file.job" to convert a script to plain
commandline arguments and this fails if the directory specified in job
file does not exist. While this has to be an error if the job is being
executed, it should not be in context of --showcmd.

To reproduce:
$ cat job.fio
[global]
directory=/x

$ fio --showcmd job.fio
fio: /x is not a directory
fio: failed parsing directory=/x
fio: job global dropped

Expected output:
fio --directory=/x

Signed-off-by: Jens Axboe <axboe@kernel.dk>
Catch more places where the callback should just return on parse/check

Signed-off-by: Jens Axboe <axboe@kernel.dk>
Fix broken requeue of io_us

Suresh reports:

-----

Does a requeued io_u work get submitted again through __get_io_u() ? If
so, the io_u item is taken out of requeue list & then the io_u->file is
set to NULL. We don't make a distinction about item being already setup
Vs a new io_u in __get_io_u(). In get_io_u(), when an io_u may come out
of requeue list, we depend on io_u->file to be non NULL to detect the
work as old pending one.

Am I missing something here or is this a bug?

1274         io_u = __get_io_u(td);
1275         if (!io_u) {
1276                 dprint(FD_IO, "__get_io_u failed\n");
1277                 return NULL;
1278         }
1279
1280         if (check_get_verify(td, io_u))
1281                 goto out;
1282         if (check_get_trim(td, io_u))
1283                 goto out;
1284
1285         /*
1286          * from a requeue, io_u already setup
1287          */
1288         if (io_u->file)
1289                 goto out;
1290
"io_u.c" line 1286 of 1667 --77%-- col 3-10

1130 struct io_u *__get_io_u(struct thread_data *td)
1131 {
1132         struct io_u *io_u;
1133
1134         td_io_u_lock(td);
1135
1136 again:
1137         if (!io_u_rempty(&td->io_u_requeues))
1138                 io_u = io_u_rpop(&td->io_u_requeues);
1139         else if (!io_u_qempty(&td->io_u_freelist))
1140                 io_u = io_u_qpop(&td->io_u_freelist);
1141
1142         if (io_u) {
1143                 io_u->buflen = 0;
1144                 io_u->resid = 0;
1145                 io_u->file = NULL;
1146                 io_u->end_io = NULL;
1147         }
"io_u.c" line 1130 of 1667 --67%-- col 1

-----

We want a retry of a requeue to be identical, not some
new chunk (and drop the latter one on the floor). This got broken with
the introduction of the rings instead of lists, where a single if
section was split into two.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
debug: ensure that __dprint() is also logged over the network

We force local logging now, but that's not terribly useful
if you can't see the output from the backend. Make it follow
the usual rules of logging over the network if a backend, to
syslog, or locally.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
Add condition to stop issuing io in do_io().

Problem:
When specifying iodepth > 1 while doing random async io
without a random map (and using the default random generator), the
main while loop in do_io() will issue extra io in the amount of io
allowed to accumulate in the queue. For example, a job with size=24k
and bs=8k will perform 4 io instead of the expected 3.

Reason:
The above behavior occurs because the while loop in do_io()
will continue until the amount of "completed io" >= size, but io
in the queue are not accounted for because they have not been
completed by io_u_queued_complete().

Exceptions:
The above behavior does not manifest itself when:
	using random_generator=lfsr bacause lfsr knows when to stop
	generating, and get_io_u() inside the while loop returns NULL,
	breaking out of the loop.
	using random map because random map ensures that all blocks
	are hit, and after they are all hit, no more are generated.

Proposed Solution:
Stop while loop based on bytes issued instead of bytes completed.

Signed-off-by: Juan Casse <jcasse@chromium.org>
Reviewed-by: Grant Grundler <grundler@chromium.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Fixes bug: stale LAST_POS(f) is not being reset

Problem:
When running fio with ioengine=sync, LAST_POS(f) state is not
reset after the file is closed. This causes workloads with
readwrite=randread and loops > 1 to fail verification if the
state of LAST_POS(f) at the beginning of the next loop is the
same as the io_u->offset. If that is the case, lseek in not
invoked, but the file position is at 0 and not at io_u->offset.

Proposed Solution:
Other ioengines, such as binject and fusion-aw, set f->engine_data
to 0 when closing the file. The sync ioengin uses f->engine_data to
store the LAST_POS(f) state. The proposed solution is to set
f->engine_data = 0 when closing a file in generic_close_file().

Signed-off-by: Juan Casse <jcasse@chromium.org>
Reviewed-by: Grant Grundler <grundler@chromium.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Fio 2.1.3

Signed-off-by: Jens Axboe <axboe@kernel.dk>
Fix misspellings using 'codespell' tool

Signed-off-by: Jens Axboe <axboe@kernel.dk>
Update man page

Fix verifysort typo, broken formatting after roundrobin and tidy up some
grammar.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Update man page

Many of fio's options are not described in the man page. Copy the short
text to the man page for those commands listed by:

./fio --cmdhelp=all | sed 's/ *:.*$//' | sed 's/ \+//' \
| while read line; do grep "\b$line\b" -q fio.1; if [[ $? -eq 1 ]]; then \
echo $line; fi; done

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
log: disable log, if realloc fails

Right now we just segfault, if realloc() returns NULL. Handle
this a bit more gracefully, by just disabling logging if that
happens. Fio will also print an error for that case.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
Merge branch 'manpage' of https://github.com/sitsofe/fio
man page: correct typoe

Signed-off-by: Jens Axboe <axboe@kernel.dk>
man page: Fix typos

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
man page: manpagify things that look like syscalls.

Use the bold style with section number to mark syscalls.

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
man page: fix broken formatting before OUTPUT section

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>
man page: Bump up release date

Signed-off-by: Sitsofe Wheeler <sitsofe@yahoo.com>