restore: simplify SIGINT/SIGTERM/SIGUSR1 and interrupted handling

Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
restore: moving pthread cancelling into restore.c from mksquashfs.c

Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
caches-queues-lists: implement queue_flush()

Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
caches-queues-lists: abstract hash table size and don't hard code it everywhere

Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
caches-queues-lists: implement seq_queue_flush()

Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
gzip_wrapper: fix -Xwindow-size option return value

Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
restore: redo thread termination when restoring filesystem on appending

I recently reworked filesystem restoration in Mksquashfs where ^C
is hit twice when appending.  When this happens Mksquashfs has
to terminate all the threads and restore the filesystem.

Originally, when Mksquashfs was enhanced to do multi-threading
(parallelised), this was done by sending all the threads a signal
and then having them go to sleep in their signal handlers.
This was an acceptable solution, but, it suffered from the fact
threads could go to sleep holding pthread mutexes, and so workarounds
had to be added for that.  But an advantage was thread cancellation
was instantaneous.

The rework redid things to be cleaner, restoring is now done on a
special restore thread, rather than the setjmp/longjmp inherited from
the original single threaded Mksquashfs, and thread cancellation is
now done via the pthread_cancel mechanism, ensuring threads do
not go to sleep holding mutexes, allowing the workarounds to be
removed.

However, on slow computers, I have noticed an annoying tendency for
the new code to "hang" for an apppreciable time when ^C is hit
twice, without any activity apparently taking place.

The new cache/queue dump code (which is run when ^\ is hit twice, and
which can be run while Mksquashfs is restoring), neatly pin-pointed
the cause.  During thread cancellation, the decompress and
fragment decompress threads continue to run taking input from their
input queues, and delivering it to their output queues, and only
terminate when either there is no more input or there is no more
space in the output queue.

This behaviour was tracked down to the fact that the decompress and
fragment decompress threads do not ordinarily execute any code which
is a pthread cancellation point - the only call executed which is
a pthread cancellation point is pthread_cond_wait which the threads
only execute if either a queue is empty or full.

Now, this problem could be fixed by the insertion of an explicit
pthread cancellation check call (pthread_testcancel) but this
potentially slows down the most compute intensive threads in
Mksquashfs, for an eventuality (thread termination on appending
if and only if ^C is hit twice) which is very rarely going to
happen.

So the solution that has been adopted is to rework the thread
termination code to forcibly cause each thread to idle.  What this
means is thread termination is now carefully done, thread
termination is now done starting from the ultimate source thread
(the reader thread), the queues from that are flushed, causing the
"downstream" threads to idle, which are then terminated, their
output queues are flushed, causing the threads downstream from them
to idle, which are then cancelled, and so on until the ultimate
destination thread (the writer thread) is cancelled.

This results again in instantaneous thread cancellation when
^C is hit twice, irrespective of slow CPU speed or slow output
devices which may have resulted in large queues of unprocessed
buffers.

Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
Mksquashfs: use pthread_kill() to signal restore thread

Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
Mksquashfs: add new -exit-on-error option

Mksquashfs recognises two kinds of error:

	- Fatal errors which cause it to abort, and
	- Non fatal errors which it considers can be safely ignored

These "non-fatal" errors are generally failure to read files or
directories, when this happens Mksquashfs skips or stores an empty
file and continues (flagging up what it has done to stderr).

From early feedback from users, this is generally considered a better
option than aborting.  Often-times users are aware that there are files
which cannot be read by Mksquashfs, and want Mksquashfs to automatically
ignore them without having to explicitly exclude them on the command line
(for instance it may be known that there are files owned by other users).
Or for instance archiving a filesystem which is known to be partially
corrupted, and it is expected it will generate I/O errors on one or more
files (which are obviously unknown beforehand and cannot be explicitly
excluded).  Additionally, often-times users do not know that there are
files which cannot be read, but would rather Mksquashfs flag these
errors and continue, rather than aborting what may be many hours of
compression.

But I have got feedback which shows in some use-cases this
behaviour is undesireable, and would rather Mksquashfs abort at
the failure to read any files.

So add a new option -exit-on-error which makes Mksquashfs abort on
these errors.

Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
Mksquashfs: reorganise struct file_info

Saves 8 bytes on 64-bit architecture.

Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
Mksquashfs: split checksum_flag

Split checksum_flag in file_info structure into two, one
each for fragments and blocks.

This allows the pre frag duplicate check routine to skip the
block checksum computation, and in future it will allow the
fragment checksum for all files in a fragment (when the fragment
is read off disk in appending) to be computed, skipping the
block checksum computation.

Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
Mksquashfs: move a couple of things into mksquashfs.h

Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
xattr: fix ERROR_EXIT()

Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
Mksquashfs: optimise duplicate checking when appending

When Mksquashfs is appending, the checksums for the fragments
in the original filesystem are lazily computed on demand - when
we get a possible duplicate where the possible duplicate fragment
is stored in a fragment block in the original filesystem, it is
read off disk, decompressed, and the fragment checksum computed.

This allows fast startup as the fragment checksums are not computed
upfront, and of course if we never get a possible duplicate stored
in a particular original fragment block, it is never read off disk.

But, when we read the original fragment block off disk, we compute
the checksum for the fragment of interest only, and then store
the fragment block in the fragment cache.

If we get a reference to another fragment stored in that
fragment block, and the fragment block is still in the cache, then
we re-use it.  If on the other hand, the fragment block has been
aged out of the cache, we have to re-read and re-decompress
the same fragment block.  This adds uncessary overhead, and in
certain filesystems, where there's lots of duplicate matches
but distributed in time, we may end up re-reading the same
fragment blocks many times.

This commit adds an optimisation, when we read a fragment block off
disk, we compute the fragment checksums for *all* the fragments
stored in the fragment block.  This avoids the situation where we
have to re-read the fragment block off disk if the fragment block
disappears from the fragment cache before we get another
duplicate check reference to that fragment block.

Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>