Switch to normal mysql public phabricator config

Summary:
This will let us handle reviews with external groups directly.

Test Plan:
This diff was reviewed via public phabricator.  :)

Reviewers: jtolmer

Reviewed By: jtolmer
Stop supressing all warnings in stress tests

Summary:
Stress tests should not generate most warnings unless something is wrong.

This diff might have made some of our recent bugs easier to find.

Test Plan:
Jenkins

Reviewers: jtolmer, santoshb

Reviewed By: santoshb
Merge aio page read requests

Summary:
Tries to submit multiple aio page read requests together to improve read
performance.

This code adds an array to buffer aio requests in os_aio_array_t - though
only for os_aio_read_array, for now. A new parameter (should_buffer) is
added to indicate whether an aio request should be buffered or submitted.
If should_submit is true, it will submit all bufferred aio requests on
the os_aio_array.

Only buf_read_ahead_linear is modified to utilize this functionality so
far. All other call sites are setting should_submit to true. Other
os_aio_array_t arrays will also ignore this.

If one thread calling buf_read_ahead_linear is buffering io requests but
another thread issues a normal os_aio_request, that other request will
submit all the buffered requests from buf_read_ahead_linear. This is
still better than nothing I suppose.

Test Plan:
Perf tests were run manually and approved by Yoshinori.
Jenkins.

Reviewers: steaphan, jtolmer, yoshinori, mcallaghan, rudradevbasak

Reviewed By: steaphan, nizamordulu
Add logical read-ahead to InnoDB

Summary:
When the session variable innodb_lra_size is set to N, we issue async
read requests for the next M logical pages where the total size of the M
pages on disk is N megabytes. The max allowed value of innodb_lra_size
is is 16384 which corresponds to prefetching 16GB of data. We may choose
to use smaller values in production.

When the flashcache is available, the logical-read-ahead tells
flashcache to not cache the pages it reads.

We always sort the page numbers before issuing read requests for them
because sorting is cheap and this way we don't have to rely on the block
layer of linux kernel to sort our read requests. Another advantage is
that if the read array is small then block layer doesn't get a chance to
coalesce reads.

I added status variables for the number of pages prefetched by
logical-read-ahead, the number of pages that are missed (a page is
missed if we notice that it was not prefetched while doing the scan.
this can happen if the b-tree was modified since the last time we
prefetched pages.), and the number of pages that were already in buffer
pool. These are

  innodb_logical_read_ahead_prefetched
  innodb_logical_read_ahead_missed
  innodb_logical_read_ahead_in_buf_pool.

There are two more session variables that control the behaviour of
logical read ahead:

  innodb_lra_n_node_recs_before_sleep: this variable determines how many
node pointer records should be traversed by logical read ahead before
going to sleep.

  innodb_lra_sleep: this is the amount of time (in milliseconds) that
logical read ahead sleeps in order to give other transactions a chance
to x-latch the index lock.

I had to make the following modifications:

* Persistent cursor can not be restored on a level other than the leaf
level. I provided a way to do this but it only works for PAGE_CUR_LE mode.

* Make btr_pcur_restore_position_func() always re-traverse the B-tree in
debug build. This is to test the above functionality in debug build.

Test Plan:
* innodb_logical_read_ahead.test tests that logical read ahead fetches
all pages necessary when the read ahead size is large enough. It also
tests whether the asynchronous io requests made for the table was the
same as the number of prefetches done by the logical read ahead.

* Add a unit test where one thread creates a lot of splits and merges on
the B-tree while the other scans the table. This is to stress test the
correctness of the changes.

* innodb_logical_read_ahead_correctness.test: This test tests for the
base cases (empty table, table with one row, table with many rows) and a
case where pages are merged while the scan is being performed. If the
code was not careful in restoring the cursor in row_read_ahead_logical()
or if row_search_for_mysql() did not check the return value of this
function, then we could skip a record. If the return value of
row_read_ahead_logical() in row_search_for_mysql() is ignored, this test
fails.

* Add stress test for logical read ahead: checksum of the table is
computed in a separate thread while the table is modified by bunch of
threads. This makes sure that innodb_lra_size works under concurrency.

* Run on a production table without traffic.

* Run on a production shadow with read/write traffic.

Also, this passes:
mysqltest.sh --valgrind --jenkins innodb.innodb_logical_read_ahead

Reviewers: rongrong, mcallaghan, steaphan, yoshinori, jtolmer

Reviewed By: steaphan, jtolmer
Disable logical read-ahead when transaction is switching between tables

Summary:
Every time a transaction start to scan a new table, LRA will clean all previous buffers
and start prefetching on the new index from beginning. This doesn't work well when a transaction
is switching frequently among many tables, which happens for partitioned tables in many cases.
This change disables LRA when too many tables are involved in a transaction. A new session variable,
innodb_lra_n_spaces, is introduced to set the maximum number of table access switches allowed before
LRA is disabled. The default value of this parameter is 3 (LRA will be disabled if LRA is accessing
different tables 3 times in a row).

Test Plan: jenkins, tested manually for partitioned table, perf regression disappears

Reviewers: steaphan, rudradevbasak

Reviewed By: rudradevbasak
Enhance aio page read merging, to handle logical read ahead

Summary:
* Added a sysvar innodb_aio_outstanding_requests, which configs the max
number of outstanding requests.  This parameter is not strictly enforced.
The idea is to only schedule a new batch of requests when the number of
outstanding requests gets below the configured value.

* Added a global status innodb_oustanding_aio_requests to show the
current outstanding aio requests.

* Added a debug-only global status innodb_max_outstanding_aio_requests
for test purpose.

Note: Technically, this change does not require logical read ahead.
However, the test for it does.

Test Plan:
Added a new test: innodb.innodb_logical_read_ahead_throttling

Reviewers: jtolmer, nizamordulu, steaphan

Reviewed By: steaphan
Add mysqldump support for logical read ahead

Summary:
Adds options to mysqldump:
	--lra-size=X
	--lra-sleep=X
	--lra-n-node-recs-before-sleep=X

These just inject SET statements to set these session variables.

Test Plan:
Using a previous version of this which triggered an error if LRA was
not present in the server, not just a warning (as this version does):

Ran the mysqldump test on mysqld WITHOUT logical_read_ahead:

--lra-size=5
	Failed on "SET innodb_lra_size=5" with unknown variable:

	mysqldump: Couldn't execute 'SET innodb_lra_size=5':
	Unknown system variable 'innodb_lra_size' (1193)

--lra-size=5 --lra-sleep=10
	Failed on "SET innodb_lra_size=5" with unknown variable:

	mysqldump: Couldn't execute 'SET innodb_lra_size=5':
	Unknown system variable 'innodb_lra_size' (1193)

--lra-sleep=10
	Worked (passed nothing) - forced the code to send anyway, and got:
	mysqldump: Couldn't execute 'SET innodb_lra_sleep=10':
	Unknown system variable 'innodb_lra_sleep' (1193)

	...and likewise for --lra-n-node-recs-before-sleep=X

Ran all of this on mysqld WITH logical_read_ahead:
	Tests all passed without error.

Using this version, which only generates a warning:

Ran mysqldump --lra-size=8 on a table on 5.1 and 5.6 servers.
On 5.1, printed the warning, and dumped normally.
On 5.6, no warning, and dumped normally.

Reviewers: nizamordulu

Reviewed By: nizamordulu
Refactor some client codepaths, switching to state machines

Summary:

This is preparatory work for making these codepaths usable in an async
context.  The goal is to represent the same state machine (async or
sync) via the context structure and calling function pointers that make
use of the context's data to perform io, handle results, etc.

This splits the connect codepaths into smaller units (generally
separating at the IO boundaries).  It changes the mechanism for
executing the connect sequence to mimic what the async version will do.

It also makes a state machine for the authentication flow, similar to
the connect flow -- breaking it apart at the IO points.

This doesn't change the plugin portion yet, but does make the rest of it
state machine'd.

This also changes the invocation of mysqladmin to always have
a --defaults-file to avoid picking up local my.cnf files.

Test Plan: mtr

Reviewers: santoshb, jtolmer

Reviewed By: santoshb
Port the Async MySQL client to 5.6

Summary:
This diff builds loosely upon the 5.1 client work and brings async to
5.6.  It is not fully feature complete (no SSL. no compression, only
supports the default native auth) but it is sufficient for our needs and
we can extend it.

This breaks the API slightly by cleaning it up and making it more
succinct.  Code using the old async API will need to change in minor
ways (though a compatibility layer could be introduced).

This also adds a #define to indicate the nonblocking client is available.

Test Plan: mtr, testing against internal tools

Reviewers: santoshb, steaphan, jtolmer

Reviewed By: santoshb, steaphan, jtolmer
Add more buffer pool stats

Summary:
Feature: Buffer Pool Stats

Display more buffer pool status in SHOW ENGINE INNODB STATUS:

SHOW STATUS - These status vars are added:
  Innodb_buffer_pool_pages_flushed_lru:
    Number of writes done for pages from the LRU list.

  Innodb_buffer_pool_pages_flushed_list:
    Number of writes done for pages from the flush list.

  Innodb_buffer_pool_pages_flushed_page:
    Number of writes done for other reasons.

  Innodb_buffer_pool_pages_lru_old:
    Number of old pages on the LRU list.

SHOW ENGINE INNODB STATUS - These output lines are added:
  Read ahead: X
  Percent pages dirty: X
  Pending writes: X
  Total writes: X
  Evicted after read ahead without access: X

Test Plan:
mtr

A new test, innodb.innodb_buffer_pool_flushed_stats is added.

This new test ensures that the sum of the by-type stats roughly equals
the total stat.

The total stat is a fuzzy counter which has no thread protection and
isn't guaranteed to be 100% accurate. So, it does only an in-bounds
compare, not an exact one.

Reviewers: chip, rongrong

Reviewed By: chip, rongrong
Add innodb_page_cleaner_interval_millis

Summary:
Feature: Buffer Pool Control Enhancements

This adds a my.cnf option to set the interval at which the
page cleaner thread runs. The default is 1000 millis and the range
is 100 to 1000. The thread runs at most once per this many millis.
Setting it to a value less than 1000 allows smaller values to be
used for innodb_io_capacity and innodb_lru_scan_depth and hopefully
reduce mutex contention. Note that the page cleaner thread is
responsible for flushing dirty pages from the flush and LRU lists
and innodb_io_capacity determines how many pages can be flushed
from the flush list, innodb_lru_scan_depth does the same for the
LRU list. Those limits are per buffer pool instance, not global.

This has results for sysbench where a query is update 1 row by PK.
The database is ~32gb and the buffer pool is 4gb. The host has flash
and the test is IO bound. The results are for MySQL 5.6 with a few
patches from Oracle to fix stalls.

io=X means that innodb_io_capacity=X and innodb_lru_scan_depth=X

-- results without this change
Updates/second
    8    16    32    64   128   256 concurrent clients
15192 20771 24702 24643 24519 24516 fil_flush patch,O_DIRECT, io=1024
17231 23661 22663 21892 21230 21401 fil_flush patch,O_DIRECT, io=8192

--- results with this change and page_cleaner_interval_millis=125
Updates/second
    8    16    32    64   128   256    concurrent clients
17697 24218 26822 27034 27014 27020 fil_flush patch,O_DIRECT, io=1024
17486 24316 26651 26721 26493 26719 fil_flush patch,O_DIRECT, io=2048
17575 24035 25749 25505 25439 25231 fil_flush patch,O_DIRECT, io=4096
17211 23500 25157 24080 23812 24408 fil_flush patch,O_DIRECT, io=8192

Test Plan: mtr

Reviewers: steaphan

Reviewed By: steaphan
Increase thread stack max by 64k

Summary:
This increases the default max by 64k.

In 5.6.14, the calculation of the thread stack size was changed.
Apparently this made the default limit pretty close for two tests:
	sys_vars.max_sp_recursion_depth_func
	main.signal_demo3

Then, once some other changes hit, the stack size was upped slightly,
and the max was exceeeded in these two tests.

This adds enough space in the default max to avoid this.

Test Plan:
Ran this before and after applying this change:
mysqltest.sh sys_vars.max_sp_recursion_depth_func main.signal_demo3

Before:
CURRENT_TEST: sys_vars.max_sp_recursion_depth_func
mysqltest: At line 64: query 'CALL sp_addRecords(0,8)' failed: 1436: Thread stack overrun:  134320 bytes used of a 262144 byte stack, and 128000 bytes needed.  Use 'mysqld --thread_stack=#' to specify a bigger stack.
CURRENT_TEST: main.signal_demo3
mysqltest: At line 107: query 'call proc_1()' failed with wrong errno 1436: 'Thread stack overrun:  135568 bytes used of a 262144 byte stack, and 128000 bytes needed.  Use 'mysqld --thread_stack=#' to specify a bigger stack.', instead of 1644...

After:
Both pass.

Reviewers: jtolmer, rudradevbasak, rongrong

Reviewed By: rongrong
Improve optimizer row estimates for partitioned tables.

Summary:
This fixes:  bugs.mysql.com/70265

This reverts the change in this diff:
	http://bazaar.launchpad.net/~mysql/mysql-server/5.6/revision/2661.835.1#sql/ha_partition.cc

It makes the behavior more like in 5.1, but instead of a fixed maximum of
10 partitions, the new variable "part_scan_max" controls this, though
it still defaults to 10.

This version also scans from the last partition to the first, as was
recommended by db-perf.

Test Plan:
I verified all the test changes do not indicate any failures of what the
original tests were intended to check.  All the changes still do scare be
a bit, but I can find no concrete reason to be worried.

Jenkins - all tests pass, including --big tests.

New mtr test added - passes.

Reviewers: santoshb, rudradevbasak, jtolmer

Reviewed By: rudradevbasak, jtolmer
Block non temporary myisam table creation outside mysql db

Summary:
Block non temporary myisam table creation outside of mysql
and mtr schemas. This is controlled by the cnf option block_create_myisam
which by default is true.

Test Plan: mtr test

Reviewers: jtolmer

Reviewed By: jtolmer
Allow blocking of locking forms of flush tables

Summary:
This diff adds a mechanism (block_ftwrl sysvar) to block running
"FLUSH TABLES WITH READ LOCK" and "FLUSH TABLES .. FOR EXPORT".

The defalt value of the config option block_ftwrl is FALSE.

Test Plan: mtr tests

Reviewers: steaphan

Reviewed By: steaphan
Adding unmodified xtrabackup-2.0.5 files

Summary:
This adds the unmodified xtrabackup distribution from Percona.

Some files not useful to us (.bzr and patch files) are excluded.

Test Plan:
This is just the base, it doesn't do or change anything.

Reviewers: rudradevbasak

Reviewed By: rudradevbasak
Add Facebooks's integrated version of xtrabackup

Summary:
This began with just the "innodb56.patch" from Percona, with some
fixes from them as well, including:

* https://bugs.launchpad.net/percona-xtrabackup/+bug/950334
* A fix for deleted tablespaces in xtrabackup apply log (from 5.1 version)

But, this is all done differently than that patch.  Instead of a
separate patch, this is integrated into the main code.

All changes are now protected via the XTRABACKUP define.

This allows the same code to compile for both mysqld and xtrabackup.

This should have a net ZERO effect, until the xtrabackup build is added.

We build this in our tree, beside our normal out-of-dir cmake build.

Basically, this can be built just by running this on a clean source tree:

	cd xtrabackup; ./utils/build.sh innodb56; cd ..

Note that this only works right in a clean source tree.
But, since we build out-of-dir, our source tree is always clean.

Also note that some corner-cases (Debug builds of xtrabackup, for example)
will probably still require more build magic to work.  The real solution
here would be to integrate this build into the main cmake build of
mysql-5.6 itself, but we didn't go that far with this (yet).

Sections not required for mysql-5.6 were removed, for simplicity.

Also, some compile/build warnings/errors were cleaned up.

And, innobackupex was modified.  For apply log, if xtrabackup binary is
explicitly specified, dont try to autodetect.

Also, we initialize error messages for xtrabackup, and we do not use
thread-specific locales for error messages.

Also, we create the xtrabackup_logfile in the supplied target_dir.

Also, we removed the FLUSH TABLES WITH READ LOCK calls from innobackupex.

Also, we added a fix to prevent an indefinite sleep of xtrabackup.

Also, we added support for the O_DIRECT_NO_FSYNC option.

And, we don't use O_DIRECT for xtrabackup_logfile.

Test Plan:
We created a small suite of xtrabackup tests that run within mtr.
These are included here, and all pass.

Jenkins (arc unit).  All pass.

xtrabackup build works (in CLEAN source dir):
	cd xtrabackup; ./utils/build.sh innodb56; cd ..

Artificially threw some warnings in xtrabackup.
Successfully carried out a full backup with warning (segfaulted before).

Reviewers: steaphan, jtolmer, nizamordulu

Reviewed By: steaphan, jtolmer, nizamordulu
Parse, but ignore, MEMCACHE_DIRTY keyword.

Summary:
Port only the parsing of the MEMCACHE_DIRTY keyword.

Test Plan: MTR tests included.

Reviewers: chip

Reviewed By: chip
Use both upstream and our crc algo, migrating to upstream.

Summary:
Facebook's 5.1 mysql wrote checksums not compatible with mysql 5.6.
This change allows us to read either fb or upstream checksums, but
always writes the new upstream (standard mysql-5.6) checksums.

This calculates both the real crc32c algo, and our addition-based one,
and compares to each on read, accepting either as valid.

On write, it only writes the real crc32c (as used upstream).

This dual implementation should not be much more computationally complex
than only calculating one or the other.

Also updated extra/innochecksum program.

This change handles compressed and uncompressed checksums, which (of
course) differ slightly in their codepath.

Test Plan: mtr as well as running as a slave

Reviewers: chip

Reviewed By: chip
Make "FACEBOOK" the default crc setting.

Summary:
This sets the default of this var so that upgrading to 5.6 will work,
without requiring a new my.cnf var to be set.

Also added description of this option to the var's info text.

Test Plan: Deployed in an rpm, got past this blocker to the next one.

Reviewers: chip

Reviewed By: chip