Improve listener_test to avoid possible false alarm

Summary:
Improve listener_test to avoid possible false alarm

Test Plan:
./listener_test