Fix minimum setting for a variable.

Summary:
max_dirty_pages_pct_lwm was incorrectly set to 0.001.

It's minimum should be 0 (0.000), to allow it to be disabled.

Test Plan:
The setting of zero has been used in prod, works fine.
Jenkins will run the test suite also.

Reviewers: jtolmer

Reviewed By: jtolmer
Fix double-allocate and double-free of table stats lock on exit.

Summary:
LOCK_global_table_stats was created twice, deleted twice.

...fixed that.

Test Plan: Re-enabled safe mutexes by fixing the cmake debug builds.

Reviewers: jtolmer, rudradevbasak

Reviewed By: rudradevbasak
Fix master hang during shutdown

Summary:
We are seeing cases of masters hanging during shutdown because bin log
dump threads are sitting waiting for more bin log events to get generated,
which will never come. This problem is one that I've seen in the past.
Fix by merging the fix I made in the Google 5.1 patch with the 5.6 version
of kill_zombie_dump_threads to 5.6ify it.

Test Plan: Jenkins

Reviewers: santoshb

Reviewed By: santoshb
only buffer aio requests on os_aio_read_array

Summary:
The merged aio request feature was originally designed for aio read only
but it could be invoked on other arrays. Put a check so it won't accidentally
buffer other io operations without submitting.

Test Plan: jenkins

Reviewers: steaphan, rudradevbasak

Reviewed By: steaphan
Enhance semi-sync

Summary:
Enhance semi-sync by placing the semisync wait
before the storage engine commit. Group commit is changed
to include a new stage called semisync_stage which does
the semisync wait. The phases in the group commit
are now
  flush phase    // holds LOCK_log
  sync phase     // holds LOCK_sync
  semisync phase // holds LOCK_semisync
  commit phase   // holds LOCK_commit

The wait after storage engine commit is completely removed
to avoid unnecessary plugin calls. The change in the
test cases is due to the bug http://bugs.mysql.com/bug.php?id=70410
which is fixed here.

Removed rpl_semi_sync_deadlock.test which will consistently
fail with enhance semi-sync. This is due to the following
deadlock caused by the test.

  thread 1 with LOCK_semisync waits for before_rotate_binlog debug sync point
  rotate thread needs to signal before_rotate_binlog waits on LOCK_log
  thread 2 with LOCK_log waits on LOCK_sync
  thread 3 with LOCK_sync waits on LOCK_semisync

The deadlock scenario tested by rpl_semi_sync_deadlock.test is
not possible here because commit thread after acquiring
LOCK_commit doesn't wait for dump thread.

Test Plan: arc unit --everything

Reviewers: jtolmer

Reviewed By: jtolmer
Add support for millisecond timeouts for reads, writes, and connects

Summary:
This diff exposes three new client options:

MYSQL_OPT_CONNECT_TIMEOUT_MS
MYSQL_OPT_READ_TIMEOUT_MS
MYSQL_OPT_WRITE_TIMEOUT_MS

Which are similar to the non-_MS options, except the value is, of
course, in milliseconds.

This diff also changes a number of timeout-related codepaths to use a
structure rather than a naked integer.  This helps prevent many, many
classes of errors that come from accidentally multiplying or dividing by
1000 to convert (or forgetting to), and creates a form of type safety
for timeouts.

Test Plan: jenkins, mtr

Reviewers: steaphan, jtolmer

Reviewed By: steaphan
handle sigterm in mysqlbinlog

Summary:
kill the connection of mysqlbinlog
on server when mysqlbinlog process receives
SIGTERM signal.

Test Plan:
kill -SIGTERM <pid_of_mysqlbinlog>
and verified the connection is closed in mysqlbinlog.

Reviewers: rudradevbasak, steaphan

Reviewed By: steaphan
Make defragment command support single index defragmentation

Summary:
Modify th defragmentation command to support single index defragmentation:
alter table XXX defragment [index YYY] [async_commit];

This will enable a script to defragment a single index rather than a full table.

Test Plan: modified main.alter_table_defragment and innodb.innodb_alter_table_defragment to test this new feature.

Reviewers: steaphan

Reviewed By: steaphan
Move the counter read to inside mutex for os_aio_linux_dispatch

Summary:
The counter read in os_aio_linux_dispatch was out of mutex protection before.
So there's a possibility that multiple threads will read the counter value as
slots_per_segment and call submit, which is not necessary. Move the counter read
to inside mutex protection.

Also use total_submitted rather than total_count to update the srv_stats counter.

Test Plan: jenkins

Reviewers: rudradevbasak, steaphan

Reviewed By: steaphan
signal dump threads about binlog update after binlog fsync

Summary:
update_binlog_pos() gives signal to all the binlog dump
threads about new data in the binlog. This should be called after
binlog fsync to avoid inconsistencies where slaves end up with some
transactions which may not exist anymore in master's binlog due
to os crash just before the binlog fsync.

The initial version of this triggered a test failure due to binlog dump
thread which couldn't receive latest position information while the
commit thread waits indefinitely causing a timeout. LOCK_sync is used
lock_commits() while INNODB SNAPSHOT but it will be a no-op now since no
other thread takes LOCK_sync.

This diff fixes these problems by using 51 logic of doing fsync before
signal_update() while holding lock_log.  Removed the sync stage from
binlog group commit.

Test Plan:
mysqltest.sh --big-tests
mysqltest.sh --big-tests --valgrind

Reviewers: jtolmer, yoshinori, steaphan

Reviewed By: steaphan
remove assert in update_binlog_end_pos

Summary:
The callers of update_binlog_end_pos don't hold
LOCK_log when the log is not in use. So this assert is
unnecessary and breaks lots of test cases.

Test Plan: mysqltest.sh

Reviewers: steaphan

Reviewed By: steaphan
hold data_lock while calling mi->get_mi_description_event()

Summary:
hold data_lock while calling mi->get_mi_description_event()
to avoid assertion.

Test Plan: mysqltest.sh

Reviewers: steaphan

Reviewed By: steaphan
tracking stats for defragmentation

Summary:
Add stats for num_pages, num_pages_freed, num_btr_compress, num_btr_compress_failure, num_page_split
to index stats. These statistics are intended to be used for defragmentation, it's not necessarily accurate.
num_pages and num_pages_freed are only updated when running defragmentation, so it will give a summary of how
last defragmentation on the index executed. num_btr_compress, num_btr_comrpess_failure and num_page_split
are cleared after a index defragmentation finished. The number can be used to estimate whether another
defragmentation is necessary.

Test Plan: jenkins

Reviewers: steaphan, jtolmer, nizamordulu

Reviewed By: steaphan
make gtid enabled slave with less durable settings crash safe

Summary:
With less durable settins (sync_binlog !=1 and
innodb_flush_log_at_trx_commit !=1), a slave with GTIDs/MTS
is not crash safe. Two possible scenarios causing slave
inconsistency

1) slave's binary log is behind innodb transaction log.
2) slave's binary log is ahead of innodb transaction log.

The slave_gtid_info transaction table consistently stores
gtid information and handles scenario 1. But in case
of scenario 2, even though the slave_gtid_info is consistent
with innodb, slave will skip executing some transactions if it's
GTID is logged in the binlog even though it is not committed in
innodb.

This diff fixes scenario 2 by changing gtid_executed when a
slave is initialized based on the binlog file and binlog position
which are logged inside innodb trx log. When gtid_executed is set
to an old value which is consistent with innodb, slave doesn't
miss any transactions.

Changing gtid_executed after crash recovery should be done only on slaves
with gtid enabled. Crash recovery on master should not do this because
it will create duplicate GTID log events for different transactions
causing data inconsistency on slaves.

Note that slaves may have duplicate GTID log events, but those
GTIDs corresponds to same transaction. Leaving the extra binlog
events on the slave doesn't create any problems because any slave
receiving those extra events skip executing any duplicate GTIDs.

Note that binlog file and binlog position in innodb header may be behind
binlog since there may be some transaction that are rolled
forward during crash recovery, but slave_gtid_info table handles
those cases by skipping the transactions but printing them to the binlog.

Test Plan: repeatedly crashed a slave with innodb_flush_log_at_trx_commit=0

Reviewers: jtolmer, yoshinori

Reviewed By: jtolmer
Remove -DWITH_DEBUG from cmake setup.

Summary:
This causes confusion, even apparently from Oracle.

The cmakeage adding it actually made Debug and WITH_DEBUG into two
different debug builds.

Now, we print an error and exit if someone passes -DWITH_DEBUG to cmake.

Test Plan: Jenkins

Reviewers: jtolmer

Reviewed By: jtolmer
Improve apply-log performance

Summary:
This patch has three fixes to improve performance of apply-log, especially
for disk-based machines:

1) Hash the pair (space, page_no) in such a way so that pages within
   the same table are ordered by page_no, while still maintaining a
   good enough hash quality.

   A couple of other places had hardcoded the hash function being used.
   Updated those to call recv_fold() instead.

2) Buffer aio read requests during recovery.

3) In the second run of xtrabackup to create txlogs,
   skip extending .ibd files

Test Plan:
Ran normal crash recovery, and also xtrabackup

Testing on disk based machines show considerable improvement.
Changes (1) and (2) cause the time taken to apply hashed log
records by 50%

For instances with a lot of ibd files, extending files may also take a
lot of time. For example, on disk based machines, an instance with
400k files, it takes 45 minutes to extend them. Change (3) stops
xtrabackup from doing a second redundant run of extending files.

Overall, these 3 changes reduce the time taken during apply-log by
between 30-50% on disk based machines, depending on the size
of xtrabackup_logfile and the number of ibd files.

Reviewers: yoshinori, rongrong, steaphan, santoshb

Reviewed By: steaphan
Lazy Flushing

Summary:
Adds innodb_idle_flush_pct to enable tuning of the page flushing rate
when the system is relatively idle.  We care about this, since doing
extra unnecessary flash writes shortens the lifespan of the flash.

Yoshinori actually wrote the code for this change, I just cleaned it
up, tested it, and am taking credit for it.

Test Plan:
The default behavior is unchanged (since the default is 100).

Yoshinori wrote the basics of this, and he tested it.
Setting it to 0 does result in stable lazy flushing, as desired.

All basic mtr tests passed ('arc unit').

Reviewers: rudradevbasak, mcallaghan

Reviewed By: rudradevbasak
Port count of lock waits and lock wait timeouts to table_statistics

Summary:
Port the following commit from 5.1 to 5.6 with minimal changes:
"Add count of lock waits and lock wait timeouts to table_statistics"

Test Plan:
1. Check that patch builds.
2. Run innodb_lock_wait* unit tests and verify that they pass.
3. Run full test suite and ensure that the diff doesn't introduce
regression bugs.
4. Manually cause a lock wait e.g. using
'select * from t1 where i=1 for update'. Verify that the innodb_row_lock_waits
and +innodb_row_lock_wait_timeouts counter in the info scheme increased.

Reviewers: rongrong

Reviewed By: rongrong