Fixed pyinotify performance regression in 15f82c7011

Fixes again #9722.
Maybe we could make it less coupled if there were some attribute on the handler class that could be checked here.