Put the element to indexMap when it is removed