Retry on EINVAL from pthread_attr_setstacksize()

EINVAL means that the requested stack size is either not a multiple
of the system page size or that it's smaller than PTHREAD_STACK_MIN.
Figure out what the case is, fix it up and retry.  If it still fails,
give up, like before.

Suggestions for future improvements:

  * don't fail!() but instead signal a condition, or
  * silently ignore the error and use a default sized stack.

Fixes #11694.