Properly handle shm_open validation. Fixes bug 16274.