Verify secondary index count after all stress test runs

Summary:
Just make sure the number of rows in the primary and secondary
indexes are the same at the end of this test.

Test Plan: Jenkins will tell of if this is wrong.

Reviewers: jtolmer

Reviewed By: jtolmer
Use aligned memory for reduced innodb_doublewrite mode

Summary:
Align the "header" used for reduced innodb_doublewrite mode to the
size of that buffer (4096), so aio writes don't fail with O_DIRECT.

Test Plan:
We saw O_DIRECT writes fail in some cases in our initial prod testing.
We will repeat that testing with this diff.

Added a new test, which tests this in O_DIRECT.
It fails without this change... but passes with it.
The new test is exactly the same as old test, but:
"--innodb-flush-method=O_DIRECT" is added to the -master.opt file.

mysqltest.sh --debug --repeat=16 --parallel=4 \
	innodb.innodb_doublewrite \
	innodb.innodb_doublewrite_odirect \
	innodb.innodb_doublewrite \
	innodb.innodb_doublewrite_odirect

mysqltest.sh --release --repeat=16 --parallel=4 \
	innodb.innodb_doublewrite \
	innodb.innodb_doublewrite_odirect \
	innodb.innodb_doublewrite \
	innodb.innodb_doublewrite_odirect

Jenkins - 'arc unit' - all tests pass.

Reviewers: rudradevbasak

Reviewed By: rudradevbasak
Make threads_running stat properly atomic

Summary:
This got out of sync, and since it's unsigned, a drift into -1 meant
it shot up to 4B, making some monitoring charts useless.

Test Plan:
main.threads_running still passes.
Jenkins - all still pass.

Reviewers: rudradevbasak

Reviewed By: rudradevbasak
Fix the errno in os_aio_linux_dispatch_read_array_submit()

Summary:
os_aio_linux_dispatch_read_array_submit would set the wrong errno if
io_submit fails in any iteration other than the first one.

This diff fixes this bug.

Test Plan: jenkins

Reviewers: steaphan

Reviewed By: steaphan
Fix typo in some comments.

Summary:
This was a typo in an earlier diff, which was then copied a bunch of
times in a later diff.

Test Plan: Changes to comments only.  No real code changes.

Reviewers: jtolmer

Reviewed By: jtolmer
Disable mysql-5.6 client's detection of the OS' charset

Summary:
The default charset of "auto" now just means "default".

This disables auto-detection.  We don't want to use utf8, even if the
OS supports it, because we already use utf8... but called "latin1",
and we don't want to corrupt that by trying to convert it.

I also ripped out the test which tests this feature.

Test Plan:
Connected with client and looked at the charset settings negotiated
by the connection, both before this change (utf8) and after (latin1).

Built a dev rpm and tested this, confirmed it fixes the issue.

Reviewers: jtolmer

Reviewed By: jtolmer
Redo-log insert buffer merge operations

Summary:
This is Oracle's patch for http://bugs.mysql.com/bug.php?id=69122. It applied
cleanly to our branch.

Test Plan:
We will push a test for this failure, either in the rpm test system,
or as a follow-up diff here, or both.

Reviewers: rongrong, jtolmer, santoshb, nizamordulu

Reviewed By: nizamordulu
Remove last_gtid column in slave_worker_info

Summary:
last_gtid column in slave_worker_info is added for
better debugging purposes with MTS. mysql_upgrade can't
upgrade schema if there is just one column change.
Any problem with MTS requires looking at binlog so last_gtid
in here is not required, so remove it to avoid any problems
during upgrade.

Test Plan: arc unit

Reviewers: steaphan

Reviewed By: steaphan
PSI for gtid_info table

Summary:
PSI for gtid_info table is added here.
This fixes seg faults caused on a slave with PSI.

Test Plan:
mysqlrpm.sh with PSI_INTERFACE and run rpm on
both slave and master

Reviewers: steaphan

Reviewed By: steaphan
fix missing big transaction with GTIDs

Summary:
This diff ports Oracles's patch for missing big transaction
with GTIDs. This diff also includes fix for missing big transaction
with GTIDs when parallel replication is enabled.

Io_thread may receive only a partial transaction before
it was stopped using stop slave. This causes a partial transaction
with GTID to get logged in the relay log. When the slave is restarted
again, it misses the transaction because GTID protocol assumes that the logged
GTID in relay log is complete. This is fixed by removing the last GTID
in the relay log from the gtid_retrieved_set causing the master to resend that
whole transaction.

Possible cases:
1) If there is a partial transaction, the whole transaction is retrieved
again into the next relay log which will be executed by SQL thread. SQL thread
rollbacks the partial transaction after seeing the FDE in the next relay log
and starts executing the same transaction which was retrieved again. In MTS
mode SQL thread appends a ROLLBACK query to the slave worker queue which got
the partial transaction.

2) I/O thread would have retrieved full transaction already and SQL thread
would have already executed it. In that case, We are not going to remove last
retrieved GTID from "Retrieved_gtid_set" otherwise we will see gaps in
"Retrieved set".

3) I/O thread would have retrieved full transaction already in the first time
itself and SQL thread has not applied it yet while requesting dump but applied
it after I/O thread started receiving events from master. In this case
retrieving the same transaction again will not cause problem because GTID
number is same, Hence SQL thread will not commit it again.

Please note there will be partial transactions written in relay log but they
will not cause any problem in case of transactional tables.  But in case
of non-transaction tables, partial transaction will create inconsistency
between master and slave.  In that case, users need to check manually.
This is not a problem for us since we are using transactional tables.

Test Plan:
Added a test to verify all the scenarios with and without MTS. Also
ran mysqltest.sh --parallel=32 with and without valgrind.

Reviewers: rudradevbasak, jtolmer, yoshinori

Reviewed By: jtolmer
Counting gtid incompatible statements in user_stats

Summary:
Add counters in user_stats to track gtid incompatible statements.

skip gtid unsafe errors if gtids are not actually getting
generated. This is required if gtid_mode is enabled on slave
regions but not on master. mysqld should not fire error in those
scenarios. should_write_gtids() identifies the above mentioned
scenario. Gtid unsafe queries are blocked by enforce_gtid_consistency
The type of queries are described in
http://dev.mysql.com/doc/refman/5.6/en/replication-options-gtids.html#sysvar_enforce_gtid_consistency

Test Plan:
binlog.binlog_enforce_gtid_consistency is modified to show stats collected.

mtr test to generate three types of unsafe queries which are
CREATE TABLE ... SELECT
CREATE TEMPORARY TABLE inside a transaction
statement that updates both transactional and non transactional updates

The slave user stats are checked for valid counters even though queries
get sucessfully executed on slave. The test case is intentionally made
to run only under statement based replication because row based events
are applied directly by passing all of these unsafe checks causing
different counter stats values.

Reviewers: santoshb, jtolmer, steaphan

Reviewed By: santoshb, jtolmer
status counter for sql_slave_skip_counter usage

Summary:
print warning about the sql_slave_skip_counter functionality getting
deprecated and update a status counter whenever someone tries to
use sql_slave_skip_counter. We want to track the usage of sql_slave_skip_counter.
For more info
http://dev.mysql.com/doc/refman/5.0/en/set-global-sql-slave-skip-counter.html.
This functionality will not work when gtid_mode is enabled

Test Plan: mtr_test

Reviewers: steaphan, rudradevbasak

Reviewed By: steaphan
reduce lock_plugin contention

Summary:
reduce lock_plugin contention by making sure
plugin hooks are not called for non semi-sync slaves

is_semi_sync_slave() function in plugin/semisync/semisync_master.cc
is copied over to sql/rpl_master.cc. is_semi_sync_slave() is
checked in every plugin function which is called
from dump thread. In each of these plugin calls, nothing is
actually done if the slave is not semi-sync enabled. semisync
plugin interface for dump thread is
  repl_semi_binlog_dump_start
  repl_semi_binlog_dump_end
  repl_semi_reserve_header
  repl_semi_before_send_event
  repl_semi_after_send_event

Avoid plugin calls results in performance improvement with many
binlog readers. We are seeing some bottleneck in is_semi_sync_slave()
function which is why this diff avoids unnecessary calls to
is_semi_sync_slave() by re-using it's return value.

Note that this diff is based on the assumption that the plugin hooks
are for semi-sync only. This is fine because no other plugin
uses any of these calls.

Test Plan: arc unit --everything

Reviewers: jtolmer, rudradevbasak

Reviewed By: rudradevbasak
completely remove lock_log usage in dump thread

Summary:
Remove lock_log usage in dump thread. This is
based on mysql-5.7.2 logic for making dump thread
read binlogs without acquiring LOCK_log mutex.

Currently dump threads needs to take LOCK_log before
waiting for a signal update. Modified the code such
that dump thread takes a new mutex LOCK_binlog_end_pos
which protects binlog_end_pos variable.

binlgo_end_pos stores the last valid offset in the active binlog
and dump thread is not allowed to read past that position
if it is reading the active binlog. If dump thread gets
LOG_READ_BINLOG_LAST_VALID_POS while reading an active
binlog it takes LOCK_binlog_end_pos and waits for
signal update if it had read all the events
upto binlog_end_pos.

Test Plan: arc unit and arc unit --everything

Reviewers: jtolmer, steaphan

Reviewed By: jtolmer
More logging about xids during crash recovery

Summary: More logging about xids during crash recovery

Test Plan: arc unit

Reviewers: jtolmer, steaphan

Reviewed By: jtolmer
Increase max user name length to 32

Summary:
This mitigates upstream bug: http://bugs.mysql.com/3083

User name length limit of 16 was too short, so made it 32.

This can be used as a template to make this even bigger, in the future.

Test Plan:
Added long user names to basic grant and replication tests.

Did full build and test run without perfschema, no failures.

Did full build and test run with perfschema, no new failures.

Jenkins both 'arc unit', and 'arc unit --everything', pass

Built a test rpm, ran it through rpm tests, passed.

Installed the test rpm on a test server, then downgraded it:
"use mysql; show create table user;", and got:
* Before upgrade: "...`User` char(16)..."
* After upgrade: "...`User` char(16)..."
* After restart: "...`User` char(16)..."
* After running mysql_upgrade: "...`User` char(32)..."
* After downgrade: "...`User` char(32)..."
* After restart: "...`User` char(32)..."
* After running mysql_upgrade: "...`User` char(16)..."

Reviewers: rudradevbasak, jtolmer, chip, santoshb

Reviewed By: santoshb
Implement online defragmentation

Summary:
This diff implements the "alter table xxx defragment" command.

* Adds a sysvar, innodb_defragment_n_pages, which decides how many pages
  are considered when attempting defragmentation.

* A background thread btr_defragment_thread takes items from a work queue
  which includes a persistent cursor to some index page. It will defragment
  innodb_defragment_n_pages number of pages, then add a new item to the
  work queue to defragment the next pages until defragmentation reaches the
  end of index.

* btr_defragment_n_pages does the actual defragmentation work. It try to
  defragment N consecutive pages starting from the page cursor points to
  to as few pages as possible.

* support for async alter table, use
  "alter table xxx defragment async_commit"

* removed a "continue" in load_generator.py which will skip validation
  for secondary only index lookup

Test Plan:
* sysvar tests

* functional test:
    innodb_alter_table_defragment

* stress tests:
    added defragmentation thread to all stress tests

Reviewers: nizamordulu, jtolmer, steaphan, rudradevbasak, santoshb

Reviewed By: nizamordulu