Correct issue of handlers running on all hosts incorrectly

Also adds an integration test to catch this bug in the future.

Fixes #7559