mksquashfs: fix phys mem calculation for 32-bit processes on PAE/64-bit kernels

When adding the code to base default memory usage on physical memory
(by default use 25% of physical memory), I made an oversight.  I assumed
the process would be able to address 25% of physical memory.

However, for 32-bit processes running on a PAE kernel or 64-bit kernel,
25% of physical memory can easily exceed the addressible memory for a
32-bit process, e.g. if a machine has 24 GB of physical memory, the
code would asume the process could easily use 6 GB.

A 32-bit process by definition can only address 4 GB (32-bit pointers).
But, due to the typical kernel/user-space split (1GB/3GB, or 2GB/2GB)
on PAE kernels, a 32-bit process may only be able to address 2 GB.

So, if Mksquashfs is a 32-bit application running on a PAE/64-bit kernel,
the code assumes it can address much more memory than it really can, which
means it runs out of memory.

The fix is to impose a maximum default limit on 32-bit kernels, or
otherwise to never use a value more than 25% of the address space.  If
we assume the maximum address space is 2 GB, then the maximum becomes
512 MB.  But, given most kernels used the 1GB/3GB split, that may be
unduely conservative, and 25% of 3 GB (756 MB) may be better.  This
patch compromises on 640 MB, which is mid-way between the 512 MB and 756 MB
values.  It is also the fixed default value previously used by Mksquashfs.

This patch also alters the code which imposes a maximum size.  Previously
it was believed limiting to the physical memory size was adequate.  But
obviously this needs to be updated to take into account a 32-bit process
may only be able to address 2 GB.  In the process I've also taken the
opportunity to limit all requests to no more than 75% of physical memory.

Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
mksquashfs: ensure value does not overflow a signed int in -mem option

Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
mksquashfs: fix abort on failure to get physical memory due to /proc missing

sysconf(_SC_PHYS_PAGES) which Mksquashfs uses to get the amount of
physical pages fails if /proc isn't mounted, returning -1.  This causes
Mksquashfs to abort with a physical memory less than SQUASHFS_LOWMEM
error.

Fix this by defaulting to the minimal value of 16 Mbytes, and
allow the user to use -mem to set the value to a higher value.  Also
issue a warning to this effect.

Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
actions: add new exists() test operator for symbolic links

Add new exists() operator that tests whether the file pointed
to by the symlink exists (in the source filesystem).

This can be used to exclude symlinks in the filesystem which are
dangling.

Note, this operator does not attempt to determine if the referenced
file is inside or outside of the filesystem being squashed.

Other additional test operators will be added to do that.

Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
actions: add new absolute() test operator for symbolic links

Add new absolute() test operator that tests if a symlink has an
absolute path, which by definition means the symbolic link may
be broken (even if the absolute path does point into the filesystem
being squashed, because the resultant filesystem can be
mounted/unsquashed anywhere, it is unlikely the absolute path will still
still point to the right place).  If you know that an absolute symlink will
point to the right place then you don't need to use this function, and/or
these symlinks can be excluded by use of other test operators.

This is the easiest and most basic test if a symlink becomes dangling
test operator.

Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
actions: add new contained() test operator for symbolic links

Test if a symlink appears to be within the source filesystem,
that is, it has a relative path, and the relative path does not
appear to backtrack outside the source filesystem using "..".

This test function does not evaluate the path for symlinks - this
is deliberate, as this makes the test fast, but in the presence
of symlinks in the path it can fail to identify paths which do go
outside the sourcefilesystem.  If this circumstance may occur, then
you should use contained_realpath(), which does follow symlinks,
but which is hence a much more expensive test.

Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
actions: deal with the unlikely case readlink returns 0 bytes

This should never happen, but, it's easy to adjust the error check
to deal with it, so why not ...

Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
actions: readlink doesn't 0 terminate the returned path

Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
actions: fix empty action

The empty action leaves orphaned inode entries in the inode_info[] table.
These entries because they are orphaned do not (correctly) get allocated
an inode number.  However, write_inode_lookup_table() does not
deal with these orphaned inode entries with an inode number of 0, and
thus writes outside of the bounds of inode_lookup_table[] causing
memory corruption.

Fix this by adding code to skip any orphaned inode entries - that is
an inode number of 0.

Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
actions: fix file type test

S_IFSOCK and S_IFLNK are not disjoint from the other file types, i.e.
the S_IFLNK value of 0120000 shares the bits for S_IFREG (0100000) and
S_IFCHR (0020000).

This means a test such as (st_mode & S_IFREG) will also match on
S_IFSOCK and S_IFLNK as they set bit 0100000.

Fix the checks in actions by introducting a helper file_type_match()
which does more nuanced checks.

Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
actions: add dir_ent to action_data structure

dir_ent is needed for the contained() test operator and for the
new contained_followlink() test operator.

Note: action_data contains some fields which are copied from dir_ent,
and are thus now duplicated.  This will be cleaned up in the near
future.

Historical note: the action_data structure when it was originally
created back in 2012 didn't use the dir_ent structure because at
that time the dir_ent didn't exist when the actions code was called
to evaluate the exclude action().  Iff the exclude action returned
FALSE the dir_ent structure was created then.  Since then dir_scan1()
has been restructured to create the dir_ent structure before any
actions code is called.

Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
actions: add new contained_followlink() test operator for symbolic links

Test if a symlink appears to be within the source filesystem,
that is, it has a relative path, and the relative path does not
appear to backtrack outside the source filesystem using "..".

This test function evaluates the path for symlinks - that is it
follows any symlinks (and any symlinks that it may contain etc.), to
discover the canonicalised relative path.

This is expensive, if possible consider using the faster
contained() alternative which does not try to follow symlinks

Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
actions: contained() should check for nonstandard_pathname

If nonstandard_pathname is set, then we have a symlink entered on the
command line, this by definition is a dangling symlink.

Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
actions: optimise string handling in follow_link()

Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
actions: use access() as an initial validity check in contained_followlink()

if access() returns error then the symlink cannot be dereferenced
for some reason, and it is pointless walking the symlink because it
will fail.  This also deals with cases where a symbolic link is
circular, generating -ELOOP, and so we don't have to worry about
dealing with loops ourselves.

Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>
actions: implement stat(expr) - eval expr on the file pointed to by a symlink

evaluate the expression using the context of the inode
pointed to by the symlink.  This allows the inode attributes
of the file pointed to by the symlink to be evaluated, rather
than the symlink itself.

Note, stat() deliberately does not evaluate the pathname, name or
depth of the symlink, these are left with the symlink values.
This allows stat() to be used on any symlink, rather than
just symlinks which are contained (if the symlink is *not*
contained then pathname, name and depth are meaningless as they
are relative to the filesystem being squashed)

Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>