Move the mouseleave event listener to the element and add a remove event to it