Fix rebase error in per-page-type table stats.

Summary:
During a rebase to a new upstream mysql version, this was fixed wrong.

Stat gen was linked to the wrong variable.  The "slot" variable was the
index of an earlier loop, not current slot (which is "aio_slot").

Test Plan:
This was found with the new valgrind tests in Jenkins.

This was also found by the included mtr test, but this was fixed wrong.
The results file was updated with the incorrect results, so it passed.

Reviewers: rongrong

Reviewed By: rongrong
Fix io perf stats in table_stats for partitioned tables

Summary:
table_stats only report the io perf stats from the last partition processed when
reporting stats for partitioned tables. Modify it to aggregate data from all partitions
of a table.

Test Plan: show_table_stats_partition_io_perf should report aggregated results now.

Reviewers: steaphan, rudradevbasak

Reviewed By: rudradevbasak
Added count of bytes of comments to table_statistics

Summary:
Added a new field "COMMENT_BYTES" to the information_schema.table_statistics table which counts the bytes of 4 types of comments in queries , per table.The 4 types are :
	1. C style comments, most commonly used : /*   */
	2. Comments beginning with "#" to the end of line
	3. Comments beginning wiht "-- " to the end of line
    4. Versionedn comments : /*!xxxxx   */
                                 In this type of comment , xxxxx is supposed to represent the version of MySQL server currently being used.If the current version is greater than or equal to the
                                 xxxxx value , then this is not treated as a comment and the syntax within it is executed . However if the current running version of MySQL
                                 is less than this xxxxx value , then this whole thing is treated as a comment.

	Reference : http://dev.mysql.com/doc/refman/5.6/en/comments.html

For a join query , the comment bytes are added to the statistics of all the tables involved in the join.
This also supports multiple hyrbid comments in a single query.

Test Plan:
count_bytes_comments.test
	   jenkins

Reviewers: rongrong, steaphan

Reviewed By: rongrong
Added query_comment_bytes to user_statistics

Summary:
I added a new field "query_comment_bytes" to
information_schema.user_statistics which counts the bytes of comments
for all queries per user.

Test Plan:
user_stats_query_comment_bytes
information_schema
is_columns_is
user_stats_commands
jenkins

Reviewers: rongrong, steaphan

Reviewed By: rongrong
Ported v5.1: Index Stats to MySQL 5.6

Summary:
This diff adds information_schema.index_statistics .
It has columns similar to table_statistics, but the data is counted per index.

desc information_schema.index_statistics;
Field Type Null Key Default Extra
TABLE_SCHEMA varchar(192) NO
TABLE_NAME varchar(192) NO
INDEX_NAME varchar(192) NO
TABLE_ENGINE varchar(192) NO
ROWS_INSERTED bigint(21) NO 0
ROWS_UPDATED bigint(21) NO 0
ROWS_DELETED bigint(21) NO 0
ROWS_READ bigint(21) NO 0
ROWS_REQUESTED bigint(21) NO 0
ROWS_INDEX_FIRST bigint(21) NO 0
ROWS_INDEX_NEXT bigint(21) NO 0
IO_READ_BYTES bigint(21) NO 0
IO_READ_REQUESTS bigint(21) NO 0
IO_READ_SVC_USECS bigint(21) NO 0
IO_READ_SVC_USECS_MAX bigint(21) NO 0
IO_READ_WAIT_USECS bigint(21) NO 0
IO_READ_WAIT_USECS_MAX bigint(21) NO 0
IO_READ_OLD_IOS bigint(21) NO 0

Test Plan:
Jenkins
Specifically, used 5 test cases to test the porting
	   	information_schema.test
		information_schema_db.test
		mysqlshow.test
		index_stats_primary.test
	   	index_stats_secondary.test

Reviewers: rongrong, steaphan
Make InnoDB 5.6 checksums fully backward compatible

Summary:
There are 4 different kinds of checksums in use.
1) crc32c = ut_crc32(..) ^ ut_crc32(..)
2) crc32cfb = ut_crc32(..) + ut_crc32(..)
3) new_csum = ut_fold_binary(..) + ut_fold_binary(..)
4) old_csum = ut_fold_binary(..)

For checking corruptness, we try to match the value of
checksum_field1 (aka "new style" = mach_read_from_4(buf +
FIL_PAGE_SPACE_OR_CHKSUM) ) to all of the following:

In InnoDB5.1 / Innochecksum5.1 : crc32cfb, new_csum
In InnoDB5.6 : crc32c, crc32cfb
In Innochecksum5.6 : crc32c, crc32cfb, new_csum

This patch adds new_csum to the check within InnoDB5.6
So that any page which is not corrupted according to 5.1 is also not
corrupted according to 5.6

Test Plan:
Ran XtraBackup against an instance which uses new_csum, and against
which XtraBackup used to crash. With this patch, it works correctly.

Reviewers: steaphan

Reviewed By: steaphan
Oracle patch for Bug #68501

Summary:
Direct port of the patch provided by Oracle for Bug #68501:
  InnoDB fails to merge under-filled pages depending on deletion order

InnoDB only attempts to merge an under-filled (less than half full) page with its left sibling,
failing to check if the sibling page to the right has enough space.

Test Plan: mysqltest.sh

Reviewers: steaphan

Reviewed By: steaphan
semi sync replication is very slow with many clients

Summary:
This is a direct port of oracle's fix for the bug
http://bugs.mysql.com/bug.php?id=69341.

If rpl_semi_sync_master_timeout is set to a huge value, semi-synchronous replication
becomes very slow, especially when many sessions are working parallel. semisync code
uses a loop method to calculate woken time, before it starts to wait. Bigger
rpl_semi_sync_master_timeout value causes more interations.

In the patch, an effective method is used instead of the original method to calculate
woken time. The code related to woken time(abstime) and wait_file_name_ is optimized
to execute only once in the first interation of wait loop.

Test Plan:
yoshi tested the patch and results are fine. mtr tests
are also fine.

Reviewers: steaphan

Reviewed By: steaphan
Add fix for deleted tablespaces in xtrabackup apply log

Summary:
Without this fix xtrabackup apply log may hang at the end, if it failed
to open some .ibd file in the past.

This patch was part of the old innodb51.patch supplied by Percona, but
for some weird reason it is not part of their current innodb56.patch on
which we have based our patches.

Test Plan:
Added test where table is modified and then dropped during backup.
Succeeds with this patch, fails otherwise.

Reviewers: steaphan

Reviewed By: steaphan
XtraBackup: Prevent indefinite sleep

Summary:
In the apply log phase, do not sleep forever if for some reason
recv_sys->n_addrs did not reach zero, and crash out instead.

Test Plan:
Manually modify recv_sys->n_addrs in gdb, and confirm that it crashes.

Reviewers: steaphan

Reviewed By: steaphan
Make xtrabackup handle O_DIRECT_NO_FSYNC option

Summary:
Add this option, so that xtrabackup doesnt error out when it sees this
in my.cnf

Test Plan: Ran xtrabackup with innodb_flush_method=O_DIRECT_NO_FSYNC in my.cnf

Reviewers: steaphan

Reviewed By: steaphan
Added more tests for xtrabackup

Summary:
Added two more tests:
Perform backup on a compressed table.
Perform backup on a partitioned table.

Test Plan: mtr

Reviewers: steaphan

Reviewed By: steaphan
Increase timeout for replication sync in xtrabackup tests

Summary:
Under valgrind, in some of the xtrabackup tests, the slave takes longer
than the default timeout of 300 seconds to sync up. Hence increase the
timeout by using the wrapper wait_for_slave_to_sync_with_master.inc

Test Plan: Tests pass under valgrind now.

Reviewers: steaphan, jtolmer

Reviewed By: steaphan
Port FAST DROP TABLE from 5.5/5.1 to 5.6.

Summary:
This diff is a port of the FAST DROP TABLE changes that were not ported by
Oracle in 5.6.

Original Summary:

revno: 3768 committer: Inaam Rana <inaam.rana@oracle.com> branch nick:
mysql-5.5 timestamp: Mon 2012-03-26 09:45:01 -0400 message:
Bug#13704145: ELIMINATE LRU SCAN WHEN DROPPING A TABLE
rb://942
  approved by: Marko Makela

  We don't need to scan LRU for dropping AHI entries when DROPing a table.
  AHI entries are already removed when we free up extents for the btree.
------------------------------------------------------------
revno: 3644
committer: Inaam Rana <inaam.rana@oracle.com>
branch nick: mysql-5.5
timestamp: Wed 2011-12-07 09:12:53 -0500
message:
  Bug#11759044 - 51325: DROPPING AN EMPTY INNODB TABLE TAKES A LONG TIME
  WITH LARGE BUFFER POOL

  (Note: this a backport of revno:3472 from mysql-trunk)

  rb://845
  approved by: Marko

    When dropping a table (with an .ibd file i.e.: with
    innodb_file_per_table set) we scan entire LRU to invalidate pages from
    that table. This can be painful in case of large buffer pools as we hold
    the buf_pool->mutex for the scan. Note that gravity of the problem does
    not depend on the size of the table. Even with an empty table but a
    large and filled up buffer pool we'll end up scanning a very long LRU list.

    The fix is to scan flush_list and just remove the blocks belonging to
    the table from the flush_list, marking them as non-dirty. The blocks
    are left in the LRU list for eventual eviction due to aging. The
    flush_list is typically much smaller than the LRU list but for cases
    where it is very long we have the solution of releasing the
    buf_pool->mutex after scanning 1K pages.

    buf_page_[set|unset]_sticky(): Use new IO-state BUF_IO_PIN to ensure
    that a block stays in the flush_list and LRU list when we release
    buf_pool->mutex. Previously we have been abusing BUF_IO_READ to achieve
    this.

This is a partial port as most of the changes were already there in 5.6.

Test Plan:
* Jenkins (arc unit)

* Deploy 3 rpms, one each for 5.6 with patch, 5.6 without patch, and
* 5.1, allocate a large buffer pool (~60GB), and fill it about 95%
* (CREATE + LOAD DATA). Once the buffer pool is filled, create and drop
* a table (empty), and compare the performance:

Query: 'create table a (id int) engine=innodb; drop table a;'

*5.6 with patch:

Query OK, 0 rows affected (0.00 sec)
Query OK, 0 rows affected (0.00 sec)

*5.6 without patch:

Query OK, 0 rows affected (0.00 sec)
Query OK, 0 rows affected (0.15 sec)

*5.1

Query OK, 0 rows affected (0.00 sec)
Query OK, 0 rows affected (0.00 sec)

Reviewers: steaphan

Reviewed By: steaphan
New conf option for TCP input buffer size in replicaiton readers

Summary:
A new conf option rpl_receive_buffer_size is added
to control the input buffer size of the socket used in slave io_thread.

A new cmd line option --receive_buffer_size is added to mysqlbinlog utility
to control the input buffer size of socket used for connection to the server.

Test Plan:
All mtr tests are passing. Added sys_var unit tests.
Yoshi verified the patch and it worked.

Reviewers: steaphan

Reviewed By: steaphan
Port v5.1: Compress until the end of the last record

Summary:
Ports v5.1's "Compress until the end of the last record in
page_zip_compress_sec()" to 5.6.

page_zip_compress_sec() compressed only until the starting
position of the data part of the last record on a secondary index page.
It relied on page_zip_compress() to add the last record's data to the
compression stream while page_zip_compress() was adding the trailing
garbage.

The removal of compression of trailing garbage will be done in
separate diffs, and this diff forms a base for those. There is no
backward or forward data incompatibility because the amount of trailing
garbage compressed doesn't matter. Even if it did, this diff still
compresses all of the garbage it just passes until the end of the last
record before the trailing garbage.

Test Plan: Jenkins (arc unit)

Reviewers: rongrong

Reviewed By: rongrong
Port v5.1: Pass page_zip object to page_zip_compress_*

Summary:
Port v5.1 New Compression "pass page_zip object to page_zip_compress_*"
to 5.6

page_zip_compress_* functions take a lot of parameters because
they need to write to a temporary buffer and they need pointers to the
trailer of the buffer. Instead of passing these pointers we can pass a
temporary page_zip object that points to the temporary compression
buffer and these functions can recompute the trailer pointers
themselves. This adds some overhead but is well worth the cost because
we are going to use the page_zip object for prefix encoding of trx id
and rollback pointers in future diffs.

Test Plan: Jenkins (arc unit)

Reviewers: rongrong

Reviewed By: rongrong
Port v5.1: Refactor page_zip_compress_clust()

Summary:
Port v5.1 New Compression "Refactor page_zip_compress_clust() to handle
compression and storage of uncompressed fields separately" to 5.6

Title says it all. By encapsulating the storing of uncompressed
fields in a different function we can eliminate the dependancy of prefix
encoding of transaction id and rollback pointers on the compression of
the page. This revision also refactors page_zip_compress_clust() so that
it calls page_zip_compress_clust_rec() to add a new record with no blob
pointers to the compression stream so page_zip_compress_clust() becomes
a wrapper around two smaller functions that handle the compression of
one record.

Test Plan: Jenkins (arc unit)

Reviewers: rongrong, jtolmer, nizamordulu

Reviewed By: rongrong
mysqlbinlog should be more user friendly with gtids

Summary:
New options --start-gtid --find-gtid-position are
added to the mysqlbinlog utility. Both require --index-file option
which should specify path to the binlog index file. Index file is
required to build previous_gtid_set_map which avoids opening each
binlog file to read previous_gtid_log_event. --start-gtid dumps
every event starting from the Gtid_log_event corresponding to the
given GTID. --find-gtid-position prints the binlog file name and
starting binlog position of Gtid_log_event corresponding to the
given GTID.

Test Plan: mtr test

Reviewers: jtolmer

Reviewed By: jtolmer
Do not use timestamp adjustment

Summary: mi->clock_diff_with_master stores the time difference of master and
slave clocks. seconds_behind_master is adjusted by this difference every time
it is queried. However the current method of detecting clock_diff_with_master
is broken because it does not compensate for network delays etc. And since we
know that all of our clocks are already synced, we can always set
clock_diff_with_master to zero.

Test Plan: MTR

Reviewers: jtolmer

Reviewed By: jtolmer