Fix removing listener, explicit remove from map

Thanks @Tungstwenty