Re-implement removeEventListener.

But still, the code seems too complex for what is being done. There must
be better a way to do it.