Replication IO thread retries more frequently

Summary:
Change get_master_version_and_clock so that it returns a retryable error
code when queries to the master fail, regardless of the failure. It will
still return a non-retryable error if the master query succeeds but
returns data the slave views as bad.

Test Plan: mtr

Reviewers: steaphan

Reviewed By: steaphan
Port v5.1 Compression Stats

Summary:
Stats added to table_stats:
  COMPRESSED_PAGE_SIZE
  COMPRESS_OPS
  COMPRESS_OPS_OK
  COMPRESS_PRIMARY_OPS
  COMPRESS_PRIMARY_OPS_OK
  COMPRESS_USECS
  COMPRESS_OK_USECS
  COMPRESS_PRIMARY_USECS
  COMPRESS_PRIMARY_OK_USECS
  UNCOMPRESS_OPS
  UNCOMPRESS_USECS

Stats added to show global status:
  buffer_pool_pages_unzip_lru,
  zip_{1024/2048/4096/8192/16384}_compressed,
  zip_{1024/2048/4096/8192/16384}_compressed_ok,
  zip_{1024/2048/4096/8192/16384}_compressed_seconds,
  zip_{1024/2048/4096/8192/16384}_compressed_ok_seconds,
  zip_{1024/2048/4096/8192/16384}_compressed_primary,
  zip_{1024/2048/4096/8192/16384}_compressed_primary_ok,
  zip_{1024/2048/4096/8192/16384}_compressed_primary_seconds,
  zip_{1024/2048/4096/8192/16384}_compressed_primary_ok_seconds,
  zip_{1024/2048/4096/8192/16384}_compressed_secondary,
  zip_{1024/2048/4096/8192/16384}_compressed_secondary_ok,
  zip_{1024/2048/4096/8192/16384}_compressed_secondary_seconds,
  zip_{1024/2048/4096/8192/16384}_compressed_secondary_ok_seconds,
  zip_{1024/2048/4096/8192/16384}_decompressed,
  zip_{1024/2048/4096/8192/16384}_decompressed_seconds,
  zip_{1024/2048/4096/8192/16384}_decompressed_primary,
  zip_{1024/2048/4096/8192/16384}_decompressed_primary_seconds,
  zip_{1024/2048/4096/8192/16384}_decompressed_secondary,
  zip_{1024/2048/4096/8192/16384}_decompressed_secondary_seconds

Test Plan: mysqltest.sh

Reviewers: steaphan

Reviewed By: steaphan
Make replication and client connection compression configurable.

Summary:
Adds the sysvar net_compression_level.

Test Plan: mysqltest.sh

Reviewers: rongrong, mcallaghan

Reviewed By: rongrong
Port disable_slave_update_table_stats change to 5.6

Summary:
Adds config disable_slave_update_table_stats to disable table stats in
slave thread

Test Plan: mysqltest.sh

Reviewers: steaphan

Reviewed By: steaphan
Port unix datagram socket from 5.1 to 5.6

Summary:
This implements the same functionality of unix datagram slocket of 5.0/5.1.

Test Plan:
All test cases ported, and all pass.

Reviewers: steaphan, jtolmer, zshao

Reviewed By: jtolmer
Ported Working Set Size to 5.6

Summary:
Added a new DB level statistics tables.

Added statistics 'working_set_size' to above table.

Working set size is computed using a modified version of HyperLogLog
algorithm.

Sysbench results:

Without this change:
Transactions : 57.71 per sec
Read/Write requests : 1038.86 per sec

With this change:
Transactions : 57.05 per sec
Read/Write requests : 1027.24 per sec

Test Plan:
MTR tests included for correctness
Performance tested using Sysbench

Reviewers: steaphan, nponnekanti

Reviewed By: steaphan
Check how max_binlog_cache_size affects LOAD DATA INFILE

Summary:

Need to check whether load data infile fails if it tries to create a
binlog cache fails with size greater than max_binlog_cache_size. Its
failing with ER_TRANS_CACHE_FULL error.

Added a test case so that any future modifications in the code notifies
the error.

Test Plan:
Created a dat file greater than 4k and set max_binlog_cache_size to 4k and ran the load data infile on that file.
Added a test in mtr

Reviewers: jtolmer, steaphan, kkwiatkowski

Reviewed By: jtolmer
Fix an inifinite loop while reading compressed pages

Summary:
After loading a compressed page in buf_page_get_gen() we
allocate a new block for decompression. The problem is that the compressed
page is neither buffer-fixed nor I/O-fixed by the time we call
buf_LRU_get_free_block(), so it may end up being evicted and returned
back as a new block.

We then call buf_page_hash_get() to see if the page hash has been
modified, and indeed it returns NULL because the original page in not in
the buffer pool anymore. In which case we go back to read the page
again causing an infinite loop.

Details of the bug is in here http://bugs.mysql.com/bug.php?id=61132.
The bug is fixed as same as described in
http://bugs.mysql.com/bug.php?id=61132. First the block is buffer-fixed so
that it cannot be evicted or relocated.

Test Plan: arc unit, mtr.

Reviewers: jtolmer, nponnekanti, steaphan

Reviewed By: jtolmer
Port additional session stats in SHOW PROCESSLIST

Summary:
stats added:
- examined rows
- sent rows

Test Plan: show_processlist.test

Reviewers: steaphan

Reviewed By: steaphan
Port v5.1 fix for bug #60682 to 5.6

Summary:
Fix http://bugs.mysql.com/60682, which is a deadlock when InnoDB calls
thd_security_context while holding kernel_mutex. The change is to use
pthread_mutex_trylock in thd_security_context and display "::BUSY::" for
the query text when the lock is not acquired.

Test Plan: Added test with reproduction case from bug 60682

Reviewers: jtolmer, steaphan

Reviewed By: steaphan
Make DROP DATABASE replicate safely with FK constraints

Summary:
Currently if you issue a DROP DATABASE dbname, MySQL will
actually log to the binary log a multi-table DROP TABLE statement.

If a database consists of enough tables, this DROP TABLE statement can
be split into 2 or more multi-table statements.

If a table in the first multi-table DROP TABLE statement has a FK
constraint that declares it a parent of a table not included in the DROP
TABLE statement, then when a replication slave or binary log playback of
the event occurs, MySQL will reject the DROP TABLE statement saying that
it would violate the FK constraint.

This bug can be avoided if the DROP TABLE statements are in a correct order
(child tables are dropped before parent tables and all tables in a
circular reference should be in one statement) or setting
OPTION_NO_FOREIGN_KEY_CHECKS flag in Query_log_event in which case
FK references do not matter when deleting parent tables. I used the later
since the order do not really matter in slave as they were already deleted
in master.

Test Plan: Added a test which reproduces the replicaiton bug.

Reviewers: jtolmer, lachlan, chip

Reviewed By: jtolmer
Check for a killed connection while building a previous version.

Summary:
There are O(n^2) undo page reads on secondary index lookups
without any checks for whether the query has been killed. Right now,
when this hits in prod mysqld has to be killed. Add a killed check
so that the query will exit without having to restart the server.

Test Plan: Adds a new test.

Reviewers: steaphan

Reviewed By: steaphan
Replication Monitoring to 5.6

Summary:
Add more SHOW STATUS counters for monitoring replication:

Relay_log_io_events, Relay_log_io_bytes:
Number of events and bytes written by the IO thread

Relay_log_sql_events, Relay_log_sql_bytes:
Number of events and bytes written by the SQL thread

Relay_log_sql_wait_seconds:
Number of seconds that the SQL thread waits for the IO thread to provide
events. This occurs when the master is idle or the network between
master/slave is slow.

Test Plan: mysqltest.sh

Reviewers: rongrong

Reviewed By: rongrong
Port rpl_read_size to 5.6

Summary:
This adds the my.cnf variable "rpl_read_size" with a default
of 8kb. It must be a multiple of 4kb. This sets the size for reads
done from the relay log and binlog. Making it larger might help with
IO stalls while reading these files when they are not in the OS buffer
cache.

Test Plan: Ported test case and added sys_var test.

Reviewers: jtolmer, steaphan

Reviewed By: jtolmer
Add reset_seconds_behind_master and Relay_log_io_connected

Summary:
Adds my.cnf var reset_seconds_behind_master:

When TRUE (default value) reset_seconds_behind_master preserves original
behavior which is to reset Seconds_Behind_Master to zero when the SQL
thread catches up to the IO thread.

When FALSE the reset is not done to avoid SBM flip-flops between zero and
the real lag.

Adds SHOW STATUS value Relay_log_io_connected:

The SHOW STATUS value Relay_log_io_connected counts the number of times the IO
thread connects to the master. This might detect network problems from frequent
disconnects.

Test Plan: mtr

Reviewers: steaphan

Reviewed By: steaphan
Reduce LOCK_log mutex contention during lock during binlog reads.

Summary:
This removes acquisition of LOCK_log mutex during binlog reads.

binlog_last_valid_pos is set whenever binlog event is written to
log_file.  When binlog dump threads read from the binlog file, EOF is
changed to the previous stable position by using
get_binlog_last_valid_pos() which causes it to read complete or valid
events. If there are no further complete events to read, it waits for
signal_update() before starting reading from the binlog again. Also if
reading from inactive binlogs(binlogs which already got rotated and no
events will be written) mutex is never held.

Test Plan:
mtr. Need to verify the mutex contention when feasible.
innodb_stress_tests are passing.

Reviewers: nizamordulu, jtolmer, dmituzas, mcallaghan, chip, steaphan

Reviewed By: jtolmer
Port Stats: Peak Lag to 5.6

Summary:
Adds peak slave lag measurement, 'peak_lag', in 'show slave status'.

Test Plan:
mysqltest.sh

Reviewers: steaphan

Reviewed By: steaphan
Fix failure of set relay_log_info_repository = 'file'

Summary:
set relay_log_info_repository = 'file' fails when rli file has old
format. This is due to the fact that old relay log info format has no
number in the first line, internal_id(used to differentiate multiple rli
instances) is initialized to 0 instead of 1(default value). This causing
error "Multiple replication metadata repository instances found with
data in them" when trying to set rli repo mode to file.

Test Plan:
mtr,
built a new rpm and upgraded a test slave to 5.6 and verified:
	set @@global.relay_log_info_repository='file'
works properly without any error.

Reviewers: jtolmer, steaphan

Reviewed By: jtolmer
Fix checksum errors in 5.6 slave when 5.6 master is downgraded to 5.1

Summary:
When master and slave are running 5.6 with replicaiton on, downgrading
master to 5.1 and starting replication, 5.1 master needs to send 5.6 FDE
and other events (since slave may ask to dump events in the same binlog
file as before downgrading). When master sends FDE and RE it needs to
recompute checksum, since contents of the events are changed (for e.g.
log_pos is set to 0 in fde). 5.1 master don't recomputes the checksum
causing replication to break with error "events are corrupt due to
network error".

Test Plan:
allocated 2 test instances and setup cascading replicaiton
and ran the inplace_upgrade_downgrade rpmtest to see replication
working. Ran mysqlrpmtest.sh fully just to ensure no other problems.

Reviewers: jtolmer, steaphan

Reviewed By: jtolmer
Test autoinc intvar not persisted across relay-log events on slave

Summary:
Adds a test case to check autoinc intvar are not persisted across
relay-log events on slave.

When master and slave have different schemas, in particular different
AUTO_INCREMENT columns, INSERT_ID events logged for a given table on
master may be applied to a different table on slave on SBR. For example
in the following sequence:

1) Insert into table t3 created only in slave with auto-inc column.
2) Insert into table t1 with auto-inc column in master but not in
   slave.
3) Insert or update into table t2 with no auto-inc column and has a
   trigger only in slave but not in master. The trigger inserts a row into
   table t3.
4) The insert into table t3 gets INSERT_ID from the previous insert into table
   t1 causing duplicate key errors.

The update on t2 causes inconsistency since the INSERT_ID values from
previous event (insert into t1) was used for inserting into table t3.
This is fixed in 5.6.10 branch. Added a test case to confirm.

Test Plan: mtr

Reviewers: jtolmer

Reviewed By: jtolmer