Port v5.1 Compression: Reduce Log Records

Summary:

InnoDB logs the entire compressed page every time the page is
compressed/re-compressed in the redo log. These log records contain the
compressed data for the entire page image. Most of the time this is
unnecessary. This revision fixes this partially by eliminating the
compression log records caused by INSERTs. Note that there will still be
some log records that contain the compressed page image because an
INSERT can cause the page to split, when that happens the data on the
page is split into two and placed on two different pages. After this,
the compressed images of both of the pages are logged instead of the
INSERT log records to both pages.

Removes the unnecessary compressed page image logging that was performed
because of UPDATEs. The DELETEs do not incur any compressed page image
log records, because they are performed by just marking records as
deleted. (Later a purge operation compactifies the pages that have holes
in them, but that too doesn't cause unnecessary page image log records)
UPDATEs are performed by first marking the existing record as deleted,
then inserting a new record therefore most of the unnecessary page-image
logging was already removed.

Move page_zip_compress_write_log_no_data() and
page_zip_parse_compress_no_data() to page0zip.ic. UNIV_INLINE functions
must be put in .ic files. It doesn't cause the build to break in debug
mode, but it does so on opt mode.

When innodb_log_compressed_pages is OFF, we may run into problems during
crash recovery because of innodb_compression_level inconsistencies. Two
ways this could happen are:

1- innodb_compression_level may be changed by a DBA during runtime.
After a crash, we don't really know which pages were compressed using
what level of compression.

2- xtrabackup has no way of knowing what level was used to compress
pages in PAGE_COMPRESS_NO_DATA entries.

Page compressions happen during page reorganization too, so we also need
to log the compression level in PAGE_REORGANIZE log records. To overcome
this, we introduce a new log record called MLOG_ZIP_PAGE_REORGANIZE
which contains the compression level used during runtime.

When an INSERT triggers page-recompression (when modification log
becomes full) InnoDB logs INSERT and ZIP_PAGE_COMPRESS_NO_DATA. During
recovery while processing the INSERT record, we should not re-compress
the page even if the INSERT triggers a re-compression because the
compression level we are using might be wrong. We just wait until the
next log record to process the page.

This exposes more zlib config parameters as system vars. The most
important of these variables is innodb_zlib_wrap which makes zlib not
write adler32 checksums between compressed blocks. The default value for
wrapping is OFF now. This should save some calls to adler32 and some
space. Changes are backwards compatible. In other words it's ok to copy
an old compressed server using xtrabackup to a server with the rpm that
has this change. This is because I updated the transaction log records
to include the value for wrap and the compression strategy in addition
to compression level. This was fairly straightforward because the same
was done earlier with compression level. Prior to this change, we stored
the compression level in a byte. Now, the spare bits in that byte are
used to store values for the new options.

innodb_zlib_wrap and innodb_zlib_strategy also effect blob compression.

It's ok to change these values dynamically. this won't break xtrabackup
or crash recovery.

Needed to backport inflateReset2() from a future version of zlib in
order to reset a decompression stream without allocating memory for its
internal state and with specifying windowBits. This was necessary to
automatically determine whether a compressed page was compressed using
adler32 checksums or not. There should not be any issues in terms of
backwards compatibility with this. After this, a clean build is required
for stress tests to pass.

For some reason, zlib's inflate() reads more than just the header if
adler32 checksums were not computed. This causes a problem when a blob
that spans multiple overflow pages is decompressed. The problem exists
for the blobs that are contained within a page too, but in that case
innodb ignores the error code from zlib and assumes that the blob ends
after the first page (which is true).

Added error messages and debug assertions when a blob truncation is
detected.

Change innodb_zlib_wrap randomly in stress tests

Test Plan:
Tested on Jenkins (arc unit)

Reviewers: jtolmer, nizamordulu steaphan

Reviewed By: nizamordulu, steaphan, jtolmer
Port v5.1 Cache buf_page_t Memory Allocations

Summary:
We cache the malloc'd buf_page_t structs in a linked list and reuse
those mallocations next time a buf_page_t needs to be malloc'd. Caching
is only enabled for the codepaths that use buf_page_alloc_descriptor and
buf_page_free_descriptor.

The length of the cache is determined by innodb_malloc_cache_len.

Test Plan: mysqltest.sh

Reviewers: nizamordulu

Reviewed By: nizamordulu
Port v5.1 Cache mem_block_t Memory Allocations

Summary:
We modify mem_block_struct to support caching of the mem_block_t
objects. Note that mem_heap_t is a linked list of allocated memory
blocks.

A call to mem_heap_create_cached() produces a cached heap only if the
requested initial memory size doesn't exceed the cache block size. If
that is not the case than none of the blocks in the returned heap is
cached. This makes it important to have good cache block size estimates.

If the initial block in a heap is cached (i.e. mem_heap_create_cached
returns a heap whose heap->cache is not NULL) then consecutive blocks
are cached if the consecutive requested memory sizes do not exceed the
cache block size. (see the change in the implementation of
mem_heap_add_block() for details) It's possible for a heap to have some
of its blocks cached and some of them not cached. Non-cached blocks will
have block->cache=NULL.

The rule of thumb is that if block->cache is not NULL then the block
must be put back to cache when the heap is freed unless the cache is
full. If the cache is full then the block gets free()'d.

This diff also fixes the memory usage estimations used in
page_zip_compress() page_zip_decompress(). Earlier estimations did not
take into account the size of deflate_state and the size of
inflate_state respectively. I went through the ZALLOC() calls in
deflate.c and inflate.c to obtain the new estimations.

Also fix the tests innodb_log_compressed_pages* and
innodb_change_compression_level* because their result file depended on
the default value of innodb_log_compressed_pages.

We use the same malloc_cache object to cache both page compressions and
blob compressions. blob compressions allocate 250K whereas page
compressions require about 350K.
We also use the same malloc_cache for blob decompressions and page
decompressions. blob decompressions allocate 40K and page decompressions
allocate around 50K.

Change memLevel for blob compressions from 7 to 8. I made this change
because:
1- it's backwards compatible. we don't need to recompress the data in
the existing tables.
2- Since we are using malloc_cache we already have excess memory in
cached malloc blocks. By using a higher memLevel we can get better
compression for blobs at no cost.

Test Plan: mysqltest.sh, arc unit (Jenkins)

Reviewers: nizamordulu

Reviewed By: nizamordulu
Stabalize main.partition_innodb test

Summary:
Replace the old (restart) test stabalization attempt with a new one.

This inserts sleeps to ensure we lose the races with the purge thread.

Test Plan: mysqltest.sh --repeat=8 --parallel=8 main.partition_innodb(x8)

Reviewers: jtolmer, rongrong

Reviewed By: rongrong