Fix remove button bug when current layer is removed