mksquashfs: ensure value does not overflow a signed int in -mem option

Signed-off-by: Phillip Lougher <phillip@squashfs.org.uk>