Fixed pyinotify performance regression in 15f82c7011

Fixes again #9722.
You can check if the approach in the second commit looks better.