Switch to normal mysql public phabricator config

Summary:
This will let us handle reviews with external groups directly.

Test Plan:
This diff was reviewed via public phabricator.  :)

Reviewers: jtolmer

Reviewed By: jtolmer
Stop supressing all warnings in stress tests

Summary:
Stress tests should not generate most warnings unless something is wrong.

This diff might have made some of our recent bugs easier to find.

Test Plan:
Jenkins

Reviewers: jtolmer, santoshb

Reviewed By: santoshb
Refactor some client codepaths, switching to state machines

Summary:

This is preparatory work for making these codepaths usable in an async
context.  The goal is to represent the same state machine (async or
sync) via the context structure and calling function pointers that make
use of the context's data to perform io, handle results, etc.

This splits the connect codepaths into smaller units (generally
separating at the IO boundaries).  It changes the mechanism for
executing the connect sequence to mimic what the async version will do.

It also makes a state machine for the authentication flow, similar to
the connect flow -- breaking it apart at the IO points.

This doesn't change the plugin portion yet, but does make the rest of it
state machine'd.

This also changes the invocation of mysqladmin to always have
a --defaults-file to avoid picking up local my.cnf files.

Test Plan: mtr

Reviewers: santoshb, jtolmer

Reviewed By: santoshb
Port the Async MySQL client to 5.6

Summary:
This diff builds loosely upon the 5.1 client work and brings async to
5.6.  It is not fully feature complete (no SSL. no compression, only
supports the default native auth) but it is sufficient for our needs and
we can extend it.

This breaks the API slightly by cleaning it up and making it more
succinct.  Code using the old async API will need to change in minor
ways (though a compatibility layer could be introduced).

This also adds a #define to indicate the nonblocking client is available.

Test Plan: mtr, testing against internal tools

Reviewers: santoshb, steaphan

Reviewed By: santoshb, steaphan
Add more buffer pool stats

Summary:
Feature: Buffer Pool Stats

Display more buffer pool status in SHOW ENGINE INNODB STATUS:

SHOW STATUS - These status vars are added:
  Innodb_buffer_pool_pages_flushed_lru:
    Number of writes done for pages from the LRU list.

  Innodb_buffer_pool_pages_flushed_list:
    Number of writes done for pages from the flush list.

  Innodb_buffer_pool_pages_flushed_page:
    Number of writes done for other reasons.

  Innodb_buffer_pool_pages_lru_old:
    Number of old pages on the LRU list.

SHOW ENGINE INNODB STATUS - These output lines are added:
  Read ahead: X
  Percent pages dirty: X
  Pending writes: X
  Total writes: X
  Evicted after read ahead without access: X

Test Plan:
mtr

A new test, innodb.innodb_buffer_pool_flushed_stats is added.

This new test ensures that the sum of the by-type stats roughly equals
the total stat.

The total stat is a fuzzy counter which has no thread protection and
isn't guaranteed to be 100% accurate. So, it does only an in-bounds
compare, not an exact one.

Reviewers: chip, rongrong

Reviewed By: chip, rongrong
Add innodb_page_cleaner_interval_millis

Summary:
Feature: Buffer Pool Control Enhancements

This adds a my.cnf option to set the interval at which the
page cleaner thread runs. The default is 1000 millis and the range
is 100 to 1000. The thread runs at most once per this many millis.
Setting it to a value less than 1000 allows smaller values to be
used for innodb_io_capacity and innodb_lru_scan_depth and hopefully
reduce mutex contention. Note that the page cleaner thread is
responsible for flushing dirty pages from the flush and LRU lists
and innodb_io_capacity determines how many pages can be flushed
from the flush list, innodb_lru_scan_depth does the same for the
LRU list. Those limits are per buffer pool instance, not global.

This has results for sysbench where a query is update 1 row by PK.
The database is ~32gb and the buffer pool is 4gb. The host has flash
and the test is IO bound. The results are for MySQL 5.6 with a few
patches from Oracle to fix stalls.

io=X means that innodb_io_capacity=X and innodb_lru_scan_depth=X

-- results without this change
Updates/second
    8    16    32    64   128   256 concurrent clients
15192 20771 24702 24643 24519 24516 fil_flush patch,O_DIRECT, io=1024
17231 23661 22663 21892 21230 21401 fil_flush patch,O_DIRECT, io=8192

--- results with this change and page_cleaner_interval_millis=125
Updates/second
    8    16    32    64   128   256    concurrent clients
17697 24218 26822 27034 27014 27020 fil_flush patch,O_DIRECT, io=1024
17486 24316 26651 26721 26493 26719 fil_flush patch,O_DIRECT, io=2048
17575 24035 25749 25505 25439 25231 fil_flush patch,O_DIRECT, io=4096
17211 23500 25157 24080 23812 24408 fil_flush patch,O_DIRECT, io=8192

Test Plan: mtr

Reviewers: steaphan

Reviewed By: steaphan
Increase thread stack max by 64k

Summary:
This increases the default max by 64k.

In 5.6.14, the calculation of the thread stack size was changed.
Apparently this made the default limit pretty close for two tests:
	sys_vars.max_sp_recursion_depth_func
	main.signal_demo3

Then, once some other changes hit, the stack size was upped slightly,
and the max was exceeeded in these two tests.

This adds enough space in the default max to avoid this.

Test Plan:
Ran this before and after applying this change:
mysqltest.sh sys_vars.max_sp_recursion_depth_func main.signal_demo3

Before:
CURRENT_TEST: sys_vars.max_sp_recursion_depth_func
mysqltest: At line 64: query 'CALL sp_addRecords(0,8)' failed: 1436: Thread stack overrun:  134320 bytes used of a 262144 byte stack, and 128000 bytes needed.  Use 'mysqld --thread_stack=#' to specify a bigger stack.
CURRENT_TEST: main.signal_demo3
mysqltest: At line 107: query 'call proc_1()' failed with wrong errno 1436: 'Thread stack overrun:  135568 bytes used of a 262144 byte stack, and 128000 bytes needed.  Use 'mysqld --thread_stack=#' to specify a bigger stack.', instead of 1644...

After:
Both pass.

Reviewers: jtolmer, rudradevbasak, rongrong

Reviewed By: rongrong
Improve optimizer row estimates for partitioned tables.

Summary:
This fixes:  bugs.mysql.com/70265

This reverts the change in this diff:
	http://bazaar.launchpad.net/~mysql/mysql-server/5.6/revision/2661.835.1#sql/ha_partition.cc

It makes the behavior more like in 5.1, but instead of a fixed maximum of
10 partitions, the new variable "part_scan_max" controls this, though
it still defaults to 10.

This version also scans from the last partition to the first, as was
recommended by db-perf.

Test Plan:
I verified all the test changes do not indicate any failures of what the
original tests were intended to check.  All the changes still do scare be
a bit, but I can find no concrete reason to be worried.

Jenkins - all tests pass, including --big tests.

New mtr test added - passes.

Reviewers: santoshb, rudradevbasak, jtolmer

Reviewed By: rudradevbasak, jtolmer
Block non temporary myisam table creation outside mysql db

Summary:
Block non temporary myisam table creation outside of mysql
and mtr schemas. This is controlled by the cnf option block_create_myisam
which by default is true.

Test Plan: mtr test

Reviewers: jtolmer

Reviewed By: jtolmer
Allow blocking of locking forms of flush tables

Summary:
This diff adds a mechanism (block_ftwrl sysvar) to block running
"FLUSH TABLES WITH READ LOCK" and "FLUSH TABLES .. FOR EXPORT".

The defalt value of the config option block_ftwrl is FALSE.

Test Plan: mtr tests

Reviewers: steaphan

Reviewed By: steaphan
Adding unmodified xtrabackup-2.0.5 files

Summary:
This adds the unmodified xtrabackup distribution from Percona.

Some files not useful to us (.bzr and patch files) are excluded.

Test Plan:
This is just the base, it doesn't do or change anything.

Reviewers: rudradevbasak

Reviewed By: rudradevbasak
Add Facebooks's integrated version of xtrabackup

Summary:
This began with just the "innodb56.patch" from Percona, with some
fixes from them as well, including:

* https://bugs.launchpad.net/percona-xtrabackup/+bug/950334
* A fix for deleted tablespaces in xtrabackup apply log (from 5.1 version)

But, this is all done differently than that patch.  Instead of a
separate patch, this is integrated into the main code.

All changes are now protected via the XTRABACKUP define.

This allows the same code to compile for both mysqld and xtrabackup.

This should have a net ZERO effect, until the xtrabackup build is added.

We build this in our tree, beside our normal out-of-dir cmake build.

Basically, this can be built just by running this on a clean source tree:

	cd xtrabackup; ./utils/build.sh innodb56; cd ..

Note that this only works right in a clean source tree.
But, since we build out-of-dir, our source tree is always clean.

Also note that some corner-cases (Debug builds of xtrabackup, for example)
will probably still require more build magic to work.  The real solution
here would be to integrate this build into the main cmake build of
mysql-5.6 itself, but we didn't go that far with this (yet).

Sections not required for mysql-5.6 were removed, for simplicity.

Also, some compile/build warnings/errors were cleaned up.

And, innobackupex was modified.  For apply log, if xtrabackup binary is
explicitly specified, dont try to autodetect.

Also, we initialize error messages for xtrabackup, and we do not use
thread-specific locales for error messages.

Also, we create the xtrabackup_logfile in the supplied target_dir.

Also, we removed the FLUSH TABLES WITH READ LOCK calls from innobackupex.

Also, we added a fix to prevent an indefinite sleep of xtrabackup.

Also, we added support for the O_DIRECT_NO_FSYNC option.

And, we don't use O_DIRECT for xtrabackup_logfile.

Test Plan:
We created a small suite of xtrabackup tests that run within mtr.
These are included here, and all pass.

Jenkins (arc unit).  All pass.

xtrabackup build works (in CLEAN source dir):
	cd xtrabackup; ./utils/build.sh innodb56; cd ..

Artificially threw some warnings in xtrabackup.
Successfully carried out a full backup with warning (segfaulted before).

Reviewers: steaphan, jtolmer, nizamordulu

Reviewed By: steaphan, jtolmer, nizamordulu
Parse, but ignore, MEMCACHE_DIRTY keyword.

Summary:
Port only the parsing of the MEMCACHE_DIRTY keyword.

Test Plan: MTR tests included.

Reviewers: chip

Reviewed By: chip
Use both upstream and our crc algo, migrating to upstream.

Summary:
Facebook's 5.1 mysql wrote checksums not compatible with mysql 5.6.
This change allows us to read either fb or upstream checksums, but
always writes the new upstream (standard mysql-5.6) checksums.

This calculates both the real crc32c algo, and our addition-based one,
and compares to each on read, accepting either as valid.

On write, it only writes the real crc32c (as used upstream).

This dual implementation should not be much more computationally complex
than only calculating one or the other.

Also updated extra/innochecksum program.

This change handles compressed and uncompressed checksums, which (of
course) differ slightly in their codepath.

Test Plan: mtr as well as running as a slave

Reviewers: chip

Reviewed By: chip
Make "FACEBOOK" the default crc setting.

Summary:
This sets the default of this var so that upgrading to 5.6 will work,
without requiring a new my.cnf var to be set.

Also added description of this option to the var's info text.

Test Plan: Deployed in an rpm, got past this blocker to the next one.

Reviewers: chip

Reviewed By: chip
Fix our acl damage in mysql_upgrade

Summary:
For Admission Control, we added two columns to the acl table.
The code to read this in mysql 5.1 and 5.6 is completely terrible.
So, acl table data created by our 5.1 causes 5.6 to explode.

Skip reading the last two acl columns, if there aren't at least 3 there.
This allows us to dodge those we used for admission control in 5.1, since,
if the extra columns from 5.6 are present, there will be at least 3 more.

This works around reading these tables, long enough to run mysql_upgrade.

Also mysql_upgrade now drops these two obsolete Admission Control columns,
if present, before addding the three new 5.6 acl columns.

For this to work, mysql_upgrade also had to ignore:
	ERROR 1091: Can't DROP '<column_name>'; ...
So, this was also added to its expected errors list.

This, in turn, required updating of some perfschema test results,
since they include line numbers of errors.

Note that if mysql_upgrade is not run, things (like new GRANTS) will cause
bad failures in 5.6 if our old 5.1 format tables are used.

Test Plan:
Deployed 5.1 and 5.6 RPMs on a spare, and upgraded cleanly!
Jenkins 'arc unit' likes this too.

Reviewers: jtolmer

Reviewed By: jtolmer
Port v5.1 SQL_NO_FCACHE to 5.6

Summary:
When running full table scans, provide a mechanism to disable
caching in flash so that the working set is not evicted.  This is
particularly useful when doing database dumps and not blowing out the
entire working set for data that most likely will not be read again.
This should also reduce the overall writes performed to the flash,
improving endurance.

Adds Innodb_dblwr_page_number to SHOW GLOBAL STATUS.  This is the page
number at which the doublewrite buffer starts in the system tablespace.
We need this for flashcache.

We turn off caching by default in flashcache and only enable it when a
process asks for it.  As long as the process group has flashcache
enabled, then all members will have it enabled.

Test Plan:
Setup a slave and did the following:
1. SELECT * FROM TABLE INTO OUTFILE 'foo' saw writes to flash.
2. SELECT SQL_NO_FCACHE * FROM TABLE2 INTO OUTFILE 'foo2' and saw no
   writes to flash.
3. SELECT /*!40001 SQL_NO_FCACHE */ * FROM TABLE3 INTO OUTFILE 'foo2'
   and saw no writes to flash.
4. SELECT /*!40001 SQL_NO_CACHE */ /*!40002 SQL_NO_FCACHE */ * FROM
   TABLE4 INTO OUTFILE 'foo2' and saw no writes to flash.
5. Ran mysqldump on a host which supports SQL_NO_FCACHE and it auto
   detected properly
6. Ran mysqldump on a host which does not support SQL_NO_FCACHE and it
   does not add the comment hint to the SQL statement.

Reviewers: chip, mcallaghan, jtolmer, santoshb

Reviewed By: chip, jtolmer, santoshb
Port Control Cross-Table Access to 5.6

Summary:
Ported our v5.1 sysvar allow_noncurrent_db_rw, which detects when a query
accesses data not in the current database.  Depending on its setting, when
a query tries to read or write to another database, it will:

	ON:
	Allow it.

	LOG:
	Allow it, but log that it occurred.

	LOG_WARN:
	Allow it, log it occurred, and send an error to client.

	OFF:
	Block it, and send an error to the client.

Test Plan:
mtr test: main.allow_noncurrent_db_rw

Reviewers: heyongqiang, mcallaghan, hfisk, steaphan, nponnekanti

Reviewed By: nponnekanti
Added new fast timer support

Summary:
This replaces the fast_timers we added in 5.1, using the timers which
are now included in upstream v5.6 for perfschema.

Initializes the timers, selects which one to use, points the global
"my_timer_now" function pointer to the correct timer, puts the stats
for that timer in the global struct "my_timer", and sets the new
SHOW STATUS variable "Timer_in_use" to indicate which was chosen.

Added my_timer_ok mtr test to confirm the timer chosen is not "None".

Added my_timer_good mtr test to confirm the timer chosen is millisecond
precision or better.

All timers are type ulonglong, and are stored in native timer units.

Provides my_timer_now() function pointer to get current time.

Provides my_timer_since(t) function to get time passed since time t,
automatically adjusted for the estimated timer overhead.

Provides my_timer_since_and_update(&t) function to get time passed
since time t, adjusted for the estimated timer overhead.  Also updates
t to the current time, skipping that overhead.  To count the time not
counted here, add my_timer.overhead for each call to this function.

Provides my_timer_to_seconds(t) function to convert native timer units
to seconds (including fractions, in a double) for the current timer.
Also adds _milliseconds and _microseconds versions of this.

Provides microseconds_to_my_timer(t) function to convert microseconds
to native timer units.

Adds a new show variable type "SHOW_TIMER" which stores its value as
a 64-bit unsigned integer, in native timer units, but displays as a
double, in seconds, automatically converting it based on the current
timer's frequency.  Since this conversion is only done for display,
no precision is ever lost in the actual variable, but the variable is
still shown as seconds with a fractional component, just like doubles.

This is the base for all our various stats which require timers.

Test Plan:
Jenkins, including follow-up diff which use these.  All pass.

Reviewers: chip, mcallaghan

Reviewed By: chip
Port v5.1 Extra Stats: Basic Stats

Summary:
Improve InnoDB disk IO performance counters:
This renames variables used for InnoDB IO performance counters, cleans
up some of the code that updates the counters, displays aggregate
performance for all async requests in SHOW INNODB STATUS.

Report InnoDB fsync time in SHOW INNODB STATUS:
Report microseconds waiting for fsync or fdatasync in SHOW INNODB STATUS

Report time for synchronous IO requests in SHOW INNODB STATUS:
Time and report via SHOW INNODB STATUS number of synchronous IO requests
and the average time per call.

Report old IO requests in SHOW INNODB STATUS:
An old request is one that is in the IO request array for more than 2
seconds.

Refactor innodb IO performance counters to use a common struct

Report per-tablespace IO statistics in SHOW INNODB STATUS:
When innodb_file_per_table is used, each table uses a separate tablespace
and this feature then provides per-table IO stats. The results look like:
--------------
TABLESPACE I/O
--------------
write: ./ibdata1 564 requests, 0 old, 71316.88 bytes/r, svc: 0.91 secs, 1.62 msecs/r, 60.60 max msecs, wait: 33.05 secs 58.61 msecs/r, 60.60 max msecs
write: ./ib_logfile0 2098 requests, 0 old, 4458.65 bytes/r, svc: 3.13 secs, 1.49 msecs/r, 296.15 max msecs, wait: 3.58 secs 1.71 msecs/r, 296.15 max msecs

Added counters for number of pending and completed aio operations.

Brought over SHOW INNODB FILE STATUS from 5.0.84 and put the data into
the information_schema.innodb_file_status table.

Sample of the output follows:

+-------------------------------------+-----------+----------+-----+----------+------------------+----------+---------------------+---------------+-----------+---------------------+----------------+
| FILE | OPERATION | REQUESTS | OLD | BYTES | BYTES/R | SVC:SECS | SVC:MSECS/R | SVC:MAX_MSECS | WAIT:SECS | WAIT:MSECS/R | WAIT:MAX_MSECS |
+-------------------------------------+-----------+----------+-----+----------+------------------+----------+---------------------+---------------+-----------+---------------------+----------------+
| /data/mysql/ibdata1 | read | 320 | 0 | 7307264 | 22835.2 | 1.527591 | 4.773721875 | 17 | 1.527591 | 4.773721875 | 17 |
| /data/mysql/ibdata1 | write | 118 | 0 | 17367040 | 147178.305084746 | 0.052502 | 0.444932203389831 | 2 | 0.555864 | 4.71071186440678 | 4 |
| /data/mysql/ib_logfile0 | read | 6 | 0 | 69632 | 11605.3333333333 | 8.6e-05 | 0.0143333333333333 | 0 | 8.6e-05 | 0.0143333333333333 | 0 |
| /data/mysql/ib_logfile0 | write | 7466 | 0 | 9551360 | 1279.31422448433 | 0.046362 | 0.00620975087061345 | 0 | 0.047235 | 0.00632668095365658 | 0 |
| ./db7733/alerts_log.ibd | read | 217 | 0 | 3588096 | 16535.0046082949 | 1.34283 | 6.18815668202765 | 20 | 1.688229 | 7.77985714285714 | 76 |
| ./db7733/api_data.ibd | read | 93 | 0 | 1523712 | 16384 | 0.517229 | 5.56160215053763 | 14 | 0.517229 | 5.56160215053763 | 14 |
| ./db7733/api_data.ibd | write | 4 | 0 | 65536 | 16384 | 0.000575 | 0.14375 | 0 | 0.004231 | 1.05775 | 2 |
+-------------------------------------+-----------+----------+-----+----------+------------------+----------+---------------------+---------------+-----------+---------------------+----------------+

Changes:

Adapts those stats which used doubles for timing to use new my_timer.

Some of this is actually redundant with the new perfschema stuff.
However, I included it all anyway, so they can be compared.

Test Plan: Jenkins, 'arc unit' all tests passed.

Reviewers: rongrong, chip

Reviewed By: rongrong