block: ensure that bio_add_page() always accepts a page for an empty bio

With commit 762380ad9322 added support for chunk sizes and no merging
across them, it broke the rule of always allowing adding of a single
page to an empty bio. So relax the restriction a bit to allow for that,
similarly to what we have always done.

This fixes a crash with mkfs.xfs and 512b sector sizes on NVMe.

Reported-by: Keith Busch <keith.busch@intel.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
block: add __init to elv_register

elv_register is only called by elevator init functions:

__init cfq_init
__init deadline_init
__init noop_init

Cc: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Fabian Frederick <fabf@skynet.be>
Signed-off-by: Jens Axboe <axboe@fb.com>
block: add __init to blkcg_policy_register

blkcg_policy_register is only called by
__init functions:

__init cfq_init
__init throtl_init

Cc: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Fabian Frederick <fabf@skynet.be>
Signed-off-by: Jens Axboe <axboe@fb.com>
NVMe: Update copyright headers

Make the copyright dates accurate and remove the final paragraph that
includes the address of the FSF.

Signed-off-by: Matthew Wilcox <matthew.r.wilcox@intel.com>
NVMe: Improve error messages

Help people diagnose what is going wrong at initialisation time by
printing out which command has gone wrong and what the device returned.
Also fix the error message printed while waiting for reset.

Signed-off-by: Matthew Wilcox <matthew.r.wilcox@intel.com>
Reviewed-by: Keith Busch <keith.busch@intel.com>
NVMe: Protect against badly formatted CQEs

If a misbehaving device posts a CQE with a command id < depth but for
one that was never allocated, the command info will have a callback
function set to NULL and we don't want to try invoking that.

Signed-off-by: Keith Busch <keith.busch@intel.com>
Signed-off-by: Matthew Wilcox <matthew.r.wilcox@intel.com>
NVMe: Add tracepoints

Adding tracepoints for bio_complete and block_split into nvme to help
with gathering IO info using blktrace and blkparse.

Signed-off-by: Keith Busch <keith.busch@intel.com>
Signed-off-by: Matthew Wilcox <matthew.r.wilcox@intel.com>
NVMe: Configure support for block flush

This configures an nvme request_queue as flush capable if the device
has a volatile write cache present.

Signed-off-by: Keith Busch <keith.busch@intel.com>
Signed-off-by: Matthew Wilcox <matthew.r.wilcox@intel.com>
NVMe: Flush with data support

It is possible a filesystem may send a flush flagged bio with write
data. There is no such composite NVMe command, so the driver sends flush
and write separately.

The device is allowed to execute these commands in any order, so it was
possible the driver ends the bio after the write completes, but while the
flush is still active. We don't want to let a filesystem believe flush
succeeded before it really has; this could cause data corruption on a
power loss between these events. To fix, this patch splits the flush
and write into chained bios.

Signed-off-by: Keith Busch <keith.busch@intel.com>
Signed-off-by: Matthew Wilcox <matthew.r.wilcox@intel.com>
NVMe: Update namespace and controller identify structures to the 1.1a spec

Controller: add CNTLID, AVSCC, APSTA, NVSCC, ACWU, SGLS fields.

Namespace: add NMIC, RESCAP, EUI64 fields. EUI64 is specifically
interesting, since it can be used to construct an UEFI NVMe device
path for a boot entry.

As per NVM Express 1.1a spec:
http://www.nvmexpress.org/wp-content/uploads/NVM-Express-1_1a.pdf

Signed-off-by: Dimitri John Ledkov <dimitri.ledkov@canonical.com>
Signed-off-by: Matthew Wilcox <matthew.r.wilcox@intel.com>
NVMe: Enable BUILD_BUG_ON checks

Since _nvme_check_size() wasn't being called from anywhere, the compiler
was optimising it away ... along with all the link-time build failures
that would result if any of the structures were the wrong size.  Call it
from nvme_exit() for no particular reason.

Signed-off-by: Matthew Wilcox <matthew.r.wilcox@intel.com>
NVMe: Update data structures for NVMe 1.2

Include changes from the current set of ratified Technical Proposals
for NVMe 1.2.

Signed-off-by: Matthew Wilcox <matthew.r.wilcox@intel.com>
NVMe: Fix the buffer size passed in GetLogPage(CDW10.NUMD)

In GetLogPage the buffer size passed to device is a 0's based value.

Signed-off-by: Indraneel M <indraneel.m@samsung.com>
Reported-by: Shiro Itou <shiro.itou@outlook.com>
Reviewed-by: Vishal Verma <vishal.l.verma@linux.intel.com>
Signed-off-by: Matthew Wilcox <matthew.r.wilcox@intel.com>
NVMe: Prevent possible NULL pointer dereference

kmalloc() used by the nvme_alloc_iod() to allocate memory for 'iod'
can fail. So check the return value.

Signed-off-by: Santosh Y <santosh.sy@samsung.com>
Signed-off-by: Matthew Wilcox <matthew.r.wilcox@intel.com>
NVMe: Make iod bio timeout a parameter

This was originally set to 4 times the IO timeout, but that was when
the IO timeout was 5 seconds instead of 30. 20 seconds for total time
to failure seemed more reasonable than 2 minutes for most, but other
users have requested to make this a module parameter instead.

Signed-off-by: Keith Busch <keith.busch@intel.com>
[renamed the module parameter to retry_time]
[made retry_time static]
Signed-off-by: Matthew Wilcox <matthew.r.wilcox@intel.com>
NVMe: Make admin timeout a module parameter

Signed-off-by: Keith Busch <keith.busch@intel.com>
[made admin_timeout static]
Signed-off-by: Matthew Wilcox <matthew.r.wilcox@intel.com>
NVMe: Delete NVME_GET_FEAT_TEMP_THRESH

This define isn't used, and any code that wanted to use it should use
NVME_FEAT_TEMP_THRESH instead.

Signed-off-by: Matthew Wilcox <matthew.r.wilcox@intel.com>
NVMe: Fix nvme get/put queue semantics

The routines to get and lock nvme queues required the caller to "put"
or "unlock" them even if getting one returned NULL. This patch fixes that.

Signed-off-by: Keith Busch <keith.busch@intel.com>
Signed-off-by: Matthew Wilcox <matthew.r.wilcox@intel.com>
NVMe: Adhere to request queue block accounting enable/disable

Recently, a new sysfs control "iostats" was added to selectively
enable or disable io statistics collection for request queues.  This
patch hooks that control.

IO statistics collection is rather expensive on large, multi-node
machines with drives pushing millions of iops.  Having the ability to
disable collection if not needed can improve throughput significantly.

As a data point, on a quad E5-4640, I see more than 50% throughput
improvement when io statistics accounting is disabled during heavily
multi-threaded small block random read benchmarks where device
performance is in the million iops+ range.

Signed-off-by: Sam Bradshaw <sbradshaw@micron.com>
Signed-off-by: Matthew Wilcox <matthew.r.wilcox@intel.com>
NVMe: Use last bytes of f/w rev SCSI Inquiry

After skipping right-padded spaces, use the last four bytes of the
firmware revision when reporting the Inquiry Product Revision. These
are generally more indicative to what is running.

Signed-off-by: Keith Busch <keith.busch@intel.com>
Acked-by: Vishal Verma <vishal.l.verma@linux.intel.com>
Signed-off-by: Matthew Wilcox <matthew.r.wilcox@intel.com>