Raise the Twitter version to t9-dev.
Bug#67718: InnoDB drastically under-fills pages in certain conditions

The problem arises from InnoDB's B+ tree page split algorithm that
attempts to optimize for sequential inserts but might end up causing
poor space utilization depending on the distribution pattern of index
key values throughout the index.

If an insert that causes a page to be split is inserting a key value
that is an immediate successor or predecessor to the last inserted key
value in the same page, the insertion point is used as the split point
irrespective of the actual distribution of values in the page. For
example, further inserts into the page with a key value that is greater
than any key value already in the page, but lower than the minimum key
value of its sibling page, will lead to a series of drastically
under-filled pages.

This page split behavior is especially a problem for workloads with a
random distribution of inserts across the pages. Since any given insert
into a page might be an immediate successor or predecessor to the last
insert, a pattern of sequential inserts might be incorrectly assumed
and lead to an unbalanced distribution of data through page splits.

The solution is to use the standard B+ tree split algorithm while still
preserving some form of optimization for sequential inserts. When a
page needs to be split, the median key value in a page is used as the
split point so that the data is distributed in a symmetric fashion. In
order to improve space utilization for sequential inserts, if a pattern
of sequential inserts is detected, the insert point might be used as
the split point if the page is the right-most or left-most page in the
tree and the key value being inserted is above the largest or below the
smallest value in the index.

A new variable named "innodb_index_page_split_mode" is introduced to
provide a way to control the page split behavior. The variable accepts
a set of permitted values: "symmetric", "lower" and "upper". Currently,
"symmetric" is always required and will be added if not stated. If its
value is empty, the original aforementioned behavior is used instead.

Using "symmetric" alone, pages are always split roughly in the middle.
When "symmetric,lower" or "symmetric,upper" are set, sequential inserts
into the left-most or right-most page in the tree will cause the
insertion point to be used as the split point. Using "symmetric,upper,
lower" will cause sequential-insert behavior to split at the insertion
point (as above) for both the left-most and right-most pages in the
tree (or more precisely, at their level).
Log checkpoint triggered flushes might be synchronous and asynchronous.

To better reflect the synchronization status of such flushes, add a
separate counter for asynchronous flushes.
Avoid recalculating the number of reserved fragment pages.

The segment is locked exclusively while its number of reserved pages
is being calculated, so it suffices (and is safe and more efficient
too) to calculate it only once.
Fix spurious failures of the rpl_start_stop_slave test case.

Wait for the slave thread to reach the desired state (connected to
master and waiting for events to arrive) before attempting to retrieve
its connection ID.
InnoDB wastes 62 of every 16,384 pages in XDES/IBUF_BITMAP extent

The problem is that once the segments of a tablespace are bigger than
32 pages, fragment pages are no longer allocated for use, yet they are
still reserved whenever a new fragment extent is allocated. This is a
direct consequence of mainly two facts: whenever a new descriptor page
is needed (every 16384 pages), the extent that contains the descriptor
page cannot be assigned to a segment and is instead used as a fragment
extent; and a segment can only allocate up to 32 fragment pages since
the array used to track fragment pages belonging to a segment is limited
to 32 entries per segment.

The solution is to allow for fragment extents to be leased to segments
whenever there are free fragment extents available. A fragment extent
is considered available if the only used pages in the extent are the
extent descriptor and ibuf bitmap pages. A new extent state is used to
tag leased extents and to ensure that they are returned to the space
free fragment list once no longer being used by a segment.

Additionally, a new system variable named innodb_lease_fragment_extents
is introduced to control whether free fragment extents are allocated to
segments.

This is an incompatible change. Once a fragment extent is allocated to
a segment, the table that contains the segment is no longer compatible
with earlier MySQL versions.

Also, the two reserved pages per leased extent are counted towards the
size of the segment.  This means that index and table size information
provided by statements such as SHOW TABLE STATUS will include the size
of the two additional pages per leased extent.
Introduce new status variables for insert buffering operations.

- Innodb_ibuf_pages
  The current size (in pages) of the ibuf tree.

- Innodb_ibuf_merged_pages
  Number of index pages that absorbed buffered changes.

- Innodb_ibuf_merged_inserts
  Number of insert operations merged.

- Innodb_ibuf_merged_delete_marks
  Number of delete mark operations merged.

- Innodb_ibuf_merged_deletes
  Number of delete operations merged.

- Innodb_ibuf_discarded_inserts
  Number of discarded buffered delete operations.

- Innodb_ibuf_discarded_delete_marks
  Number of discarded buffered delete mark operations.

- Innodb_ibuf_discarded_deletes
  Number of discarded buffered delete operations.
Expose purge age information

This change introduces new status variables that expose the purge age
in transactions and undo log. Innodb_purge_trx_no indicates the maximum
transaction ID for which records have been purged. Innodb_purge_undo_no
represents the maximum undo log record number for which records have
been purged. Finally, Innodb_trx_max_id represents the smallest number
not yet assigned as a transaction ID.
Add counters for successful page merges and page discards

Currently Innodb_page_merges counts only merge attempts but there is
no metric for successful merges. This change introduces a new status
variable named Innodb_page_merges_succeeded which indicates the number
of successful page merge operations (that is, the number of pages
successfully merged into another page).

Additionally, this change also introduces a new status variable named
Innodb_page_discards which represents the number of pages that have
become empty and are thus discarded.
Support for floating-point system variables using the plugin interface.

Augment the server plugin interface to allow plugins to define and
expose floating-point system variables of type double. The convenience
macros MYSQL_SYSVAR_DOUBLE and MYSQL_THDVAR_DOUBLE are introduced and
can be used by plugins to declare system variables of type double.
The fractional part of the def, min and max values of system variables
is ignored.

Allow the default, minimum and maximum values of system variables of
type double to have a meaningful fractional part. Since the command-line
option parsing interface (my_getopt) uses fields of type unsigned long
long (ull) to store these values, the double values were being stored in
a lossy way that discards the fractional part.

This change introduces a couple of functions that can be used to store
the raw representation of a double value in the raw bits of unsigned
long long field in a way that the binary representation remains the
same. Using these function the real value can be converted back and
forth between the types.
Change the type of the system variables innodb_segment_fill_factor and
innodb_index_fill_factor to double.

These variables are used to represent percentages but could not be set
to percentages with fractional parts. Furthermore, their initial default
values were fractional percentages, but had to be changed because the
interface did not support them. Now these system variables are made of
type double and their default values are restored to their original
form.
InnoDB reserves an excessive amount of space in large tables for write
operations.

When performing operations that are expected to expand a table (for
example, allocate new pages due to a page split), InnoDB currently
preallocates and reserves up to 1% of the total size of the tablespace
as a measure to ensure that enough free extents (that is, disk space)
are available for the operation and to ensure that if running out of
disk space, these operations are preemptively failed as to reserve any
remaining free space to operations that end up freeing space (that is,
delete data).

The percentage is reasonable for tables smaller than a few gigabytes,
but not for tables sized at tens of gigabytes or more, at which point
the percentage won't correctly estimate the free space needed to perform
operations and may cause an excessive amount of free extents to be
preallocated. Also, this reservation approach is of dubious need
considering that disk space usage is normally monitored and that
transactions normally won't cause large expansions. In the worst case,
it is even possible to expand the underlying filesystem if the database
goes into a deadlock where it lacks free space for cleaning operations.

The solution is to provide a way to either completely disable free
extents reservation or to control the amount of free extents that are
reserved for such operations. This changes introduces two new system
variables to accomplish both. The variable innodb_reserve_free_extents
can be used to enable or disable free extents reservation and
innodb_free_extents_reservation_factor can be used to control what
percentage of a space size is reserved for operations that may cause
more space to be used.
Increase timeout for test case innodb_mysql.

The test case might take a long time to execute when running under
valgrind and debugging is enabled.
Suppress unsafe statement warnings generated by a replication test.

The rpl_filter_tables_not_exist test case generates unsafe statement
warnings both on the master and slave, so add a missing suppression
to the master.
Twitter MySQL 5.5.28.t9