caches-queues-lists: move definitions of {insert|remove}_fragment_list

from mksquashfs.c to here.

Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
queues-caches-lists: move cache freelist allocate logic

into here, and make it per a per cache option.

On appending we want to change the allocate logic for the
fragment and writer caches to be grow first, rather than allocate
from the free list first.  This is because on appending with the use
free list first logic the caches never grow very large leading
to reduced cache effectiveness.

By making it a per cache option we fix the issue that on appending
we also made the reader cache grow first.  This was *not* a major
problem except because we never do lookup on the reader cache we
don't need to do it!

This commit is mainly because it always should have been a
cache private variable rather than a global.

Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
sort: move extern definitions to sort.h from mksquashfs.c

Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
Fix email address

Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
squashfs_swap: Use explicit sizeof(struct xxx) rather than sizeof(*s)

Many years ago when Squashfs used bitfields the copying swap
macros relied on s(ource) (the first parameter) to be a pointer
to the appropriate structure type.  Beginning with the removal
of the bitfields the swap macros were rewritten, and over
a number of iterations the need for s to be a pointer to the
appropriate type has been removed - except for one vestigal (*)
requirement, when the host is a little endian system but the
destination/source is unaligned, no swapping is
required and memcpy is used to copy from the source
buffer to the destination buffer.  This code still requires
s to be pointer of the appropriate structure type to compute
the size of the memcpy, but, this is unnecessary, so replace
it with an explicit sizeof(struct xxx).  This finally removes
any requirement for either s(ource) or d(estination) to be of
any type.

(*) Note: the new SQUASHFS_INSWAP_XXX macros that were added
at the time of the macro rewrite to more efficiently swap
inplace rather than copying (where possible) still require the
passed pointer to be of the correct type, but, this is entirely
correct as the code by definition relies on a correctly aligned
structure of the appropriate type.

The whole point of removing the need for pointer typing from the
copying macros is it should prevent the compiler from making
assumptions about alignment based on type.  Problems with this
on ARM is what prompted the initial changes!

Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
Swap source and destination in macros in disk -> host reading code

Many years ago when Squashfs used bitfields the copying swap
macros relied on s(source) (the first parameter) to be a pointer
to the appropriate structure type.

When swapping from the host order to disk order, source was a
pointer to the appropriate type, and destination was an unaligned
pointer.  However, when swapping from disk order to host order in
code that read from disk, it was the destination that was a pointer
to the appropriate type, and source was an unaligned pointer.  To
accomodate this, the sense of the parameters was swapped, so the
destination pointer was given first, and then the source pointer.

This was a fairly horrible hack.  Now that the rewritten swap
macros no longer require either the source or destination pointer
to be of the appropriate type, we can swap the parameters around
in the disk -> host reading code, so that all code uses
source then destination order.

Note: the Unsquashfs code that reads legacy 1.x, 2.x and 3.x
filesystems still uses the original bitfield swapping macros,
and the requirement that the first parameter is of the appropriate
type remains, this could be changed too (as there's no longer
any code which uses the macros to swap from host -> disk order)
but the swapping code is fossilised and in deep maintenance mode, and
only essential changes are made.  This largely cosmetic change
(especially as the code isn't developed anymore and so any
additional swapping macros is unlikely) isn't considered sufficiently
essential to risk adding inadvertant bugs.

Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
Move SQUASHFS_LEXX and SQUASHFS_SWAP_{SHORTS:INTS:LONGS} into squashfs_swap.h

Plus move definition of SQUASHFS_MEMCPY as well into squashfs_swap.h

Now that the separate implementations in mksquashfs.h and read_fs.h
are identical we can have one implementation, and we can
put it with the other swap macros.

Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
read_fs: move extern definitions into read_fs.h

Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
restore: move extern definition into restore.h

Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
info: move filename printing into separate function

Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
info: handle asprintf failure properly

Cleanup asprintf error handling.  Print an ERROR and
return without doing anything else.

Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
mksquashfs: add a comment regarding the progressbar and SQUASHFS_TRACE

Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
progressbar: move progress bar state to progressbar.c and set via function

At the moment the progress_enabled flag is set in progressbar.c if
the progressbar should be enabled via enable_progress_bar() and disabled
via disable_progress_bar().  However, this does not allow for
temporary disabling of the progress bar.  A
{disable/enable}_progress_bar() sequence will enable the progress bar
even if it was never enabled in the first place.

Solve this by moving the overall progress flag (which determines if
the progress bar should be initially enabled) into progressbar.c
and set via the accessor function set_progressbar_state().

Now enable_progress_bar() only sets progress_enabled if the overall
display_progress_bar flag is set.  In effect progress_enabled
becomes a temporary status flag controlled by the overall
display_progress_bar flag.

Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
info: add initial code to dump queue state when sent SIGHUP

Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
info: add some newlines to the queue printfs

Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
caches-queues-lists: add include for printf

Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
info: add code to dump cache state

Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
info: add a used parameter to the cache dump information

Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
caches-queues-lists: get rid of "keep" blocks and make behaviour more explicit

Originally the caches in mksquashfs were "buffer pools" that
expanded as necessary when readahead/number of buffers in flight
increased, and shrank as the number of buffers in flight decreased.

It was realised that the "buffer pools" could also be used to lookup
blocks in duplicate checking.  Firstly, when the fragment/blocks of the
file being duplicate checked were "still in flight" it was faster to
look them up in the "buffer pool" rather than have to wait for the
blocks to be written to disk (as was originally necessary because
blocks in flight were inaccessible).  Secondly, when blocks were
given back to the "buffer pool", if rather than shrinking the
cache, the blocks were put onto a freelist, duplicate checking might
find the blocks there which is faster than having to read them back
from the output filesystem.

From this evolved the concept of "keep" blocks, which when a
block was obtained from the cache via cache_get() if the keep flag
was set this signified that the block was to be kept when released
via cache_put().

This was perfectly adequate until now when code is being added to
monitor/record cache behaviour.  As the keep flag is a per block
flag there is no cache specific information that indicates whether
a cache is being used as a non-shrinking lookup cache or as shrinking
non-lookup based "cache" (in this case it is really acting as a buffer
pool).  This is important because it turns out the statistics to be
gathered differs depending on the cache usage.  A non-shrinking lookup
cache has count which represents the maximum size the cache has
grown to, and used which represents the size of the cache currently in
use.  A shrinking non-lookup cache on the other hand shrinks as blocks
become unused, and so count and used are one and the same.  Given this
there is no way of determining what the maximum size the shrinking
cache has historically grown to using just count and used; used is
completely redundant, and the maximum historical size is not recorded.

So as first step, rearchitect the "keep" code moving the flag from
being a per block flag to being a per cache flag, making it explicit
whether the cache is being used as a non-shrinking lookup cache or as
a shrinking non-lookup based cache.

Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
caches-queues-lists: enable cache_get() to use negative indexes

The caches previously could not handle negative indexes well
because the hash table implementation used the hash macro

CALCULATE_HASH(start) (start & 0xffff)

With negative indexes this will always use the one hash table
entry of 65535.

Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>