Fix out of order events when remove handlers

This is a work in progress for fix #1277