Illumos #3552

3552 condensing one space map burns 3 seconds of CPU in spa_sync()
     thread (fix race condition)

References:
  https://www.illumos.org/issues/3552
  illumos/illumos-gate@03f8c366886542ed249a15d755ae78ea4e775d9d

Ported-by: Richard Yao <ryao@gentoo.org>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>

Porting notes:

This fixes an upstream regression that was introduced in commit
zfsonlinux/zfs@e51be06697762215dc3b679f8668987034a5a048, which
ported the Illumos 3552 changes. This fix was added to upstream
rather quickly, but at the time of the port, no one spotted it and
the race was rare enough that it passed our regression tests. I
discovered this when comparing our metaslab.c to the illumos
metaslab.c.

Without this change it is possible for metaslab_group_alloc() to
consume a large amount of cpu time.  Since this occurs under a
mutex in a rcu critical section the kernel will log this to the
console as a self-detected cpu stall as follows:

  INFO: rcu_sched self-detected stall on CPU { 0}
  (t=60000 jiffies g=11431890 c=11431889 q=18271)

Closes #1687
Closes #1720
Closes #1731
Closes #1747
Revert "Add new kstat for monitoring time in dmu_tx_assign"

This reverts commit 92334b14ec378b1693573b52c09816bbade9cf3e.

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Revert "Add txgs-<pool> kstat file"

This reverts commit e95853a331529a6cb96fdf10476c53441e59f4e1.
Add visibility in to arc_read

This change is an attempt to add visibility into the arc_read calls
occurring on a system, in real time. To do this, a list was added to the
in memory SPA data structure for a pool, with each element on the list
corresponding to a call to arc_read. These entries are then exported
through the kstat interface, which can then be interpreted in userspace.

For each arc_read call, the following information is exported:

 * A unique identifier (uint64_t)
 * The time the entry was added to the list (hrtime_t)
   (*not* wall clock time; relative to the other entries on the list)
 * The objset ID (uint64_t)
 * The object number (uint64_t)
 * The indirection level (uint64_t)
 * The block ID (uint64_t)
 * The name of the function originating the arc_read call (char[24])
 * The arc_flags from the arc_read call (uint32_t)
 * The PID of the reading thread (pid_t)
 * The command or name of thread originating read (char[16])

From this exported information one can see, in real time, exactly what
is being read, what function is generating the read, and whether or not
the read was found to be already cached.

There is still some work to be done, but this should serve as a good
starting point.

Specifically, dbuf_read's are not accounted for in the currently
exported information. Thus, a follow up patch should probably be added
to export these calls that never call into arc_read (they only hit the
dbuf hash table). In addition, it might be nice to create a utility
similar to "arcstat.py" to digest the exported information and display
it in a more readable format. Or perhaps, log the information and allow
for it to be "replayed" at a later time.

Signed-off-by: Prakash Surya <surya1@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Add visibility in to txg sync behavior

This change is an attempt to add visibility in to how txgs are being
formed on a system, in real time. To do this, a list was added to the
in memory SPA data structure for a pool, with each element on the list
corresponding to txg. These entries are then exported through the kstat
interface, which can then be interpreted in userspace.

For each txg, the following information is exported:

 * Unique txg number (uint64_t)
 * The time the txd was born (hrtime_t)
   (*not* wall clock time; relative to the other entries on the list)
 * The current txg state ((O)pen/(Q)uiescing/(S)yncing/(C)ommitted)
 * The number of reserved bytes for the txg (uint64_t)
 * The number of bytes read during the txg (uint64_t)
 * The number of bytes written during the txg (uint64_t)
 * The number of read operations during the txg (uint64_t)
 * The number of write operations during the txg (uint64_t)
 * The time the txg was closed (hrtime_t)
 * The time the txg was quiesced (hrtime_t)
 * The time the txg was synced (hrtime_t)

Note that while the raw kstat now stores relative hrtimes for the
open, quiesce, and sync times.  Those relative times are used to
calculate how long each state took and these deltas and printed by
output handlers.

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Add visibility in to dmu_tx_assign times

This change adds a new kstat to gain some visibility into the
amount of time spent in each call to dmu_tx_assign. A histogram
is exported via the new dmu_tx_assign file. The information
contained in this histogram is the frequency dmu_tx_assign
took to complete given an interval range.

Signed-off-by: Prakash Surya <surya1@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Add visibility in to cached dbufs

Currently there is no mechanism to inspect which dbufs are being
cached by the system.  There are some coarse counters in arcstats
by they only give a rough idea of what's being cached.  This patch
aims to improve the current situation by adding a new dbufs kstat.

When read this new kstat will walk all cached dbufs linked in to
the dbuf_hash.  For each dbuf it will dump detailed information
about the buffer.  It will also dump additional information about
the referenced arc buffer and its related dnode.  This provides a
more complete view in to exactly what is being cached.

With this generic infrastructure in place utilities can be written
to post-process the data to understand exactly how the caching is
working.  For example, the data could be processed to show a list
of all cached dnodes and how much space they're consuming.  Or a
similar list could be generated based on dnode type.  Many other
ways to interpret the data exist based on what kinds of questions
you're trying to answer.

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Add dbufstat.py command

Add initial version of 'dbufstat.py' command.  This is not yet
complete the following still needs to be done:

* Make default formatting fit in 80 columns.
* Add the --eXtended option to print everything.
* Add the --Fields option to select specific fields for display.
* Add the --Values option to display all possible fields.
* Add pretty print functionality.
* (optional) Add the -s option to custom field separator.
* (optional) Add the -o option to output to a specified file.
* (optional) Add sort functionality by field.

The idea here is for dbufstat.py to feel as similar to arcstat.py
as possible.  Consistency is good.

Signed-off-by: Prakash Surya <surya1@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Improve dbufstat.py command

This commit is an attempt to make the dbufstat.py command to be more
like the arcstat.py command in regards to its usage and output.

In addition:

    * The default formatting now fits in 80 characters
    * Added the --extended (-x) option
    * Added the --columns (-f) option
    * Added the --seperator (-s) option
    * Added the --outfile (-o) option
    * Added the --infile (-i) option
    * Added pretty printing of values
    * Replace --objects option with --dnodes (-d)

Missing functionality:

    * The --buffers (-b) option is not yet implemented
    * Sort functionality non-existent

Signed-off-by: Prakash Surya <surya1@llnl.gov>
Change "objects_" prefix to "dnodes_"

The "--objects"/"-o" option has be changed to "--dnodes"/"-d". The
function names still needed to be changes to match.

Signed-off-by: Prakash Surya <surya1@llnl.gov>
Retab dbufstat.py to use 4 spaces for indentation

Signed-off-by: Prakash Surya <surya1@llnl.gov>
Implement "--buffers"/"-b" option

Signed-off-by: Prakash Surya <surya1@llnl.gov>
Use dynamic indexing for "-d" option

Signed-off-by: Prakash Surya <surya1@llnl.gov>
Use dynamic indexing for "-t" option

Signed-off-by: Prakash Surya <surya1@llnl.gov>
Create update_dict_helper function

Signed-off-by: Prakash Surya <surya1@llnl.gov>
Consolidate [dnodes|types]_print_dict functions

Signed-off-by: Prakash Surya <surya1@llnl.gov>
Remove update_dict_helper indirection

Signed-off-by: Prakash Surya <surya1@llnl.gov>