src: always check value passed to set_memsize

Move the minimum memory check from the environment parsing to
set_memsize, so the limit is actually enforced also when using the API.

Adapt the rhbz557655.sh test to the invalid memsize values being
rejected now, and add a new test for checking invalid parameters
explicitly.
rescue: fix sscanf placeholders for --smp and --memsize

Use %d to parse them as int (since the variables for them as int)
instead of %u, even if they both need to be at least > 0; the library
will check for the validity of them anyway.