cfq-iosched: fix cfq_cic_link() race confition

commit 5eb46851de3904cd1be9192fdacb8d34deadc1fc upstream.

cfq_cic_link() has race condition. When some processes which shared ioc
issue I/O to same block device simultaneously, cfq_cic_link() returns -EEXIST
sometimes. The race condition might stop I/O by following steps:

step  1: Process A: Issue an I/O to /dev/sda
step  2: Process A: Get an ioc (iocA here) in get_io_context() which does not
		    linked with a cic for the device
step  3: Process A: Get a new cic for the device (cicA here) in
		    cfq_alloc_io_context()

step  4: Process B: Issue an I/O to /dev/sda
step  5: Process B: Get iocA in get_io_context() since process A and B share the
		    same ioc
step  6: Process B: Get a new cic for the device (cicB here) in
		    cfq_alloc_io_context() since iocA has not been linked with a
		    cic for the device yet

step  7: Process A: Link cicA to iocA in cfq_cic_link()
step  8: Process A: Dispatch I/O to driver and finish it

step  9: Process B: Try to link cicB to iocA in cfq_cic_link()
		    But it fails with showing "cfq: cic link failed!" kernel
		    message, since iocA has already linked with cicA at step 7.
step 10: Process B: Wait for finishig I/O in get_request_wait()
		    The function does not wake up, when there is no I/O to the
		    device.

When cfq_cic_link() returns -EEXIST, it means ioc has already linked with cic.
So when cfq_cic_link() return -EEXIST, retry cfq_cic_lookup().

Signed-off-by: Yasuaki Ishimatsu <isimatu.yasuaki@jp.fujitsu.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Signed-off-by: Greg Kroah-Hartman <gregkh@suse.de>
block: prepare I/O context code for BFQ

BFQ uses struct cfq_io_context to store its per-process per-device data,
reusing the same code for cic handling of CFQ.  The code is not shared
ATM to minimize the impact of these patches.

This patch introduces a new hlist to each io_context to store all the
cic's allocated by BFQ to allow calling the right destructor on module
unload; the radix tree used for cic lookup needs to be duplicated
because it can contain dead keys inserted by a scheduler and later
retrieved by the other one.

Update the io_context exit and free paths to take care also of
the BFQ cic's.

Change the type of cfqq inside struct cfq_io_context to void *
to use it also for BFQ per-queue data.

A new bfq-specific ioprio_changed field is necessary, too, to avoid
clobbering cfq's one, so switch ioprio_changed to a bitmap, with one
element per scheduler.

Signed-off-by: Fabio Checconi <fabio@gandalf.sssup.it>
Signed-off-by: Paolo Valente <paolo.valente@unimore.it>
Signed-off-by: Francesco Allertsen <fallertsen@gmail.com>
block: add cgroups, kconfig and build bits for BFQ

Add a Kconfig option and do the related Makefile changes to compile
the BFQ I/O scheduler.  Also let the cgroups subsystem know about the
BFQ I/O controller.

Signed-off-by: Fabio Checconi <fabio@gandalf.sssup.it>
Signed-off-by: Paolo Valente <paolo.valente@unimore.it>

Conflicts:

	block/Makefile
block: introduce the BFQ I/O scheduler

Add the BFQ-v1-r1 I/O scheduler to 2.6.35.
The general structure is borrowed from CFQ, as much of the code. A (bfq_)queue
is associated to each task doing I/O on a device, and each time a scheduling
decision has to be taken a queue is selected and it is served until it expires.

The main differences are:
     - slices are given in the service domain: tasks are assigned budgets,
       measured in number of sectors. Once got the disk, a task must
       however consume its assigned budget within a configurable maximum time
       (by default, the maximum possible value of the budgets is automaticall
       computed to comply with this timeout). This allows the desired latency
       vs "throughput boosting" tradeoff to be set.

    - Budgets are scheduled according to a variant of WF2Q+, implemented
      using an augmented rb-tree to take eligibility into account while
      preserving an O(log N) overall complexity.

    - BFQ supports full hierarchical scheduling, exporting a cgroups
      interface.  Each node has a full scheduler, so each group can
      be assigned its own ioprio and an ioprio_class.

    - If the cgroups interface is used, weights can be explictly assigned,
      otherwise ioprio values are mapped to weights using the relation
      weight = IOPRIO_BE_NR - ioprio.

    - ioprio classes are served in strict priority order, i.e., lower
      priority queues are not served as long as there are higher priority
      queues.  Among queues in the same classes the bandwidth is distributed
      in proportion to the weights of each queue.

Regarding what has not changed it is worth noting:
    - the handling of cfq_io_contexts to associate queues to tasks.
      Much
      of the code has been reused just renaming it.  (There is room for
      code sharing with CFQ but we wanted to minimize the impact of this
      patch.)

    - The handling of async queues.

    - The handling of idle windows.

    - The handling of merging.

    - The heuristics to assert that a task is worth an idle window (with
      minor modifications to hw_tag/CIC_SEEKY detection).

Signed-off-by: Fabio Checconi <fabio@gandalf.sssup.it>
Signed-off-by: Paolo Valente <paolo.valente@unimore.it>

Ported the following commits to bfq:
    - cfq-iosched: compact io_context radix_tree (80b15c7389caa)
    - cfq-iosched: remove dead_key from cfq_io_context (bca4b914b5da3d8e)
Fix BFQ after __blk_run_queue was changed

See commit 02d63a7d.
Updated BFQ to v2.
Switch from BFQ-v2 to BFQ-v2-r1.

Improvements:

When the first new request for the active queue arrives during disk idling,
the device is not immediately unplugged if this request is small. This prevents
the disk to be immediately committed to serve a small request. In fact, further
requests are likely to be merged to this one quickly, after which the device is
unplugged by some block layer function and larger requests are dispatched.

The idling timeout and the weight-raising max time have been tuned a little bit
too, as detailed below.

Reduced a little bit the disk idling timeout for weight-raised
queues. This helps reduce latencies for applications made of several
processes that do something and then pass the token to the next
process. By reducing the timeout we reduce the time that may be wasted
to wait for the next request of a process that has just passed the token,
and hence that will not issue any new request.

Reduced the weight-raising max time from 8 to 7.5 seconds: this
reduces a little bit the time interval during which a just-started
disk-bound process may unproperly be weight-raised.