Mksquashfs: introduce additional per CPU fragment process threads

The following commit

	commit 287f3ef6b1cd670f3ff347dcaaa7b0bade03096b
	Date:   Sun May 19 04:27:38 2013 +0100
	mksquashfs: queue fragment and empty file buffers directly
		to main thread

moved fragment and empty file buffer processing away from the
deflate thread(s), and queued them directly to the main thread.
This was done to ensure the "queue dump" facility added to
Mksquashfs when it showed the "reader -> deflate queue size"
was showing the number of *file buffers* waiting to be
compressed by the deflate thread(s).  Previously when the
queue also held fragments (because sparse checking of fragments
was piggy-backed onto the deflate threads), the queue size shown
included both fragments and file buffers waiting to be compressed.
This limited the usefulness of the queue dump in determining what
was happening within Mksquashfs.

A consequence of this, however, was that sparse checking of
fragments was moved to the main thread.  In hindsight,
this could cause a performance regression in certain cases because
there is only one main thread (it is not per CPU) and it is tasked
with overall coordination of Mksquashfs, adding the additional
burden of sparse checking could bottleneck the main thread,
leading to bottlenecking of the other threads and a loss of
parallelism.

In general this change broke the general rule that the main thread
should only deal with coordination and processing should be
done on other per CPU threads.

So fix this by introducing an additional per CPU fragment processing
thread which deals with sparse checking of fragments.

Note: it is expected that moving sparse checking to the main thread may
only have caused a problem with duplicate checking where the main thread
becomes otherwise engaged, and even then only in the extremely rare
case many fragments are sparse.  In general, fragments are mostly
trivially verifiable as non-sparse (first few bytes non-zero), and the
sparse check is therefore extremely fast.

Moving sparse checking to fragment processing threads is a first stage
iteration, the aim is to move the other more CPU intensive fragment
processing which is currently done on the main thread to the new
fragment processing threads too, as this should improve performance.

Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
Mksquashfs: change cache_rehash() to cache_hash()

Originally caches in Mksquashfs on calling cache_get() were always
hashed to the index supplied on calling cache_get().

For writer blocks obtained in the deflator() threads this
created a problem because at the time of getting the cache
block the ultimate index was not known (the index is the disk
location where the block is finally stored, which is not known
until the writer block is processed by the main thread).

The solution adopted then was to cache_get() the writer block
with a dummy index, and later when the real index was known
to rehash the writer block.

This was good and worked well.

However, since then a new cache_get_nohash() call has been
implemented.  This was introduced for reader buffers which
are not looked up and therefore never need to be hashed.

We can use this new call here by changing cache_rehash() to
cache_hash() - get an initial unhashed buffer via cache_get_nohash()
and later create a hash mapping via cache_hash().

The semantic changes from cache_rehash() to cache_hash() are because
we are non-longer rehashing (removing an existing hash and adding a
new hash) but adding a hash to a cache entry that has never been hashed.

Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
Makefile: fix header file dependencies for mksquashfs.c

Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
Makefile: fix header file dependencies for read_fs.c

Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
Makefile: fix header file dependencies for sort.c

Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
Makefile: fix header file dependencies for pseudo.c

Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
Makefile: fix header file dependencies for read_xattrs.c

Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
Makefile: fix header file dependencies for action.c

Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
Makefile: fix header file dependencies for progressbar.c

Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
Makefile: fix header file dependencies for read_file.c

Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
Makefile: fix header file dependencies for info.c

Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
Makefile: fix header file dependencies for restore.c

Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
Makefile: fix header file dependencies for caches-queues-lists.c

Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
Makefile: fix header file dependencies for gzip_wrapper.c

Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
Makefile: fix header file dependencies for lzo_wrapper.c

Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
Makefile: fix header file dependencies for lz4_wrapper.c

Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
Makefile: fix header file dependencies for xz_wrapper.c

Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
xattr: fix include of non system header file

Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
caches-queues-lists: collapse struct file_buffer unions into one with structs

Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
actions: fix parsing of "G/M/K" size specifiers in number parsing

Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>