We not being interrested in an event does not mean the parent windows are.

This should fix the out-of-nowhere performance drop that happens when
an event is filtered on Xorg level's.