Bug #1217426: Empty directory is not backed when stream is used

The problem was that innobackupex ignored empty directories when
creating streaming backups. Normally a database created by the server
contains a ‘db.opt’ file, and so the corresponding directory is not
empty. However, in some circumstances a database may be created with
‘mkdir’. This is what mysql_install_db does to create the default ‘test’
database. Since there are no files to send into the
stream, the resulting stream would contain no references to the empty
directory, so it would not be created when the stream is unpacked and
thus, missing in the backup. Which resulted in inconsistency between the
original data directory and the one after restoring from the backup.

One solution to this problem would be to stream empty directories. This
is rather complicated from the implementation point of view. Since both
xbcrypt and xbstream only deal with files, it would require implementing
support for directory units, i.e. introducing new ‘directory’ chunk
types, bumping the format version, modifications to the datasink API and
utility command line interface, etc.

A much easier solution is to stream a fake empty ‘db.opt’ file for empty
directories. This way the corresponding database will be present in the
backup, and the empty db.opt file will be ignored by the server after
restore.
Automerge from 2.1.
Fix for https://bugs.launchpad.net/percona-xtrabackup/+bug/1185343 - Fixed IV used in Xtrabackup encryption

In order to fix this in the safest possible way, followed general advice from various sources but summed up nicely here:
"An initialization vector has different security requirements than a key, so the IV usually does not need to be secret. However, in most cases, it is important that an initialization vector is never reused under the same key. For CBC and CFB, reusing an IV leaks some information about the first block of plaintext, and about any common prefix shared by the two messages. For OFB and CTR, reusing an IV completely destroys security. This can be seen because both modes effectively create a bitstream that is XORed with the plaintext, and this bitstream is dependent on the password and IV only. Reusing a bitstream destroys security. In CBC mode, the IV must, in addition, be unpredictable at encryption time; in particular, the (previously) common practice of re-using the last ciphertext block of a message as the IV for the next message is insecure (for example, this method was used by SSL 2.0). If an attacker knows the IV (or the previous block of ciphertext) before he specifies the next plaintext, he can check his guess about plaintext of some block that was encrypted with the same key before (this is known as the TLS CBC IV attack)." - http://en.wikipedia.org/wiki/Block_cipher_modes_of_operation

This fix advances the XBCRYPT chunk header version and adds an iv field to store the iv that was used to encrypt that block.

On decryption, if the XBCRYPT version is 1 then the original fixed iv is used to decrypt, otherwise the iv in the block header is used.

On encryption, for each block being encrypted, an iv of the correct length is generated by using values returned from random() until the iv has been fully populated. This iv will be written into the XBCRYPT chunk header along with its size.

One new test case added, xbcrypt.sh. Added two files in inc: decrypt_v1_test_file.txt is a simple text file that was then encrypted @ v1 levels and put in decrypt_v1_test_file.xbcrypt. The new test case tests two simple scenarios:
  1 - Tests that a file can go through an encryption/decryption cycle and be 100% identical to the orginal file.
  2 - Tests that a file previously encrypted with and earlier version can be decrypted witht he current version.
Fix for https://bugs.launchpad.net/percona-xtrabackup/+bug/1185343 - Fixed IV used in Xtrabackup encryption

In order to fix this in the safest possible way, followed general advice from various sources but summed up nicely here:
"An initialization vector has different security requirements than a key, so the IV usually does not need to be secret. However, in most cases, it is important that an initialization vector is never reused under the same key. For CBC and CFB, reusing an IV leaks some information about the first block of plaintext, and about any common prefix shared by the two messages. For OFB and CTR, reusing an IV completely destroys security. This can be seen because both modes effectively create a bitstream that is XORed with the plaintext, and this bitstream is dependent on the password and IV only. Reusing a bitstream destroys security. In CBC mode, the IV must, in addition, be unpredictable at encryption time; in particular, the (previously) common practice of re-using the last ciphertext block of a message as the IV for the next message is insecure (for example, this method was used by SSL 2.0). If an attacker knows the IV (or the previous block of ciphertext) before he specifies the next plaintext, he can check his guess about plaintext of some block that was encrypted with the same key before (this is known as the TLS CBC IV attack)." - http://en.wikipedia.org/wiki/Block_cipher_modes_of_operation

This fix advances the XBCRYPT chunk header version and adds an iv field to store the iv that was used to encrypt that block.

On decryption, if the XBCRYPT version is 1 then the original fixed iv is used to decrypt, otherwise the iv in the block header is used.

On encryption, for each block being encrypted, an iv of the correct length is generated by using values returned from random() until the iv has been fully populated. This iv will be written into the XBCRYPT chunk header along with its size.

One new test case added, xbcrypt.sh. Added two files in inc: decrypt_v1_test_file.txt is a simple text file that was then encrypted @ v1 levels and put in decrypt_v1_test_file.xbcrypt. The new test case tests two simple scenarios:
  1 - Tests that a file can go through an encryption/decryption cycle and be 100% identical to the orginal file.
  2 - Tests that a file previously encrypted with and earlier version can be decrypted witht he current version.
Increase timeouts to fix bug 1214274.
Merge fix for bug 1214274 from 2.1
Fix for bug 1239728: Race condition in start_query_killer
child code can cause parent MySQL connection to close.
Synchronize child and parent by sending a signal from parent when child fully
initialized.
Merge from 2.1
Bug #1239670: xtrabackup_slave_info does not contain GTID purge lists

Modified innobackupex to detect when the slave server is in the GTID
mode, and produce the correct (pseudo-)SQL to setup a new slave in this
case. I.e. xtrabackup_slave_info instead of:

CHANGE MASTER TO MASTER_LOG_FILE=..., MASTER_LOG_POS=...

will contain:

SET GLOBAL gtid_purged=...;
CHANGE MASTER TO MASTER_AUTO_POSITION=1
Automerge from 2.1.
Bug #1247057: innodb_fast_checksum is not covered by the test suite

Added an innodb_fast_checksum test for Percona Server 5.5- servers.
Automerge from 2.1.
Merged lp:~gl-az/percona-xtrabackup/BT23557-bug1185343-2.1.
Merged lp:~gl-az/percona-xtrabackup/BT23557-bug1185343-2.2.
Merged lp:~sergei.glushchenko/percona-xtrabackup/2.1-xb-bug1214274.
Merged lp:~sergei.glushchenko/percona-xtrabackup/2.2-xb-bug1214274.
Merged lp:~sergei.glushchenko/percona-xtrabackup/2.1-xb-bug1239728.
Merged lp:~sergei.glushchenko/percona-xtrabackup/2.2-xb-bug1239728.
Merged lp:~akopytov/percona-xtrabackup/bug1247057-2.1.
Merged lp:~akopytov/percona-xtrabackup/bug1247057-2.2.