animationComplete: Remove only the handler attached by us

Fixes gh-7265