test_shm_mq: Fix checks for negative queue size.

Noted while addressing compiler warnings pointed out on pgsql-hackers.