Speedup event filtering

This code improve two thing:
* filter using class name and set instead of loop and `isinstance` call
* drop event as soon as possible

I got a 30% boost on my computer for a simple benchmark.
What do you think abou ttaking the allowed_events list and just adding the both events we always need?