INotify backend: honor ignored files

Actually this needs a careful redesign,
but this is good enough to fix the issue.

Fixes: #763