Remove this.$element on destroy but don't nullify

Bug: T84897
Change-Id: I794fa007358515022598dd3d66683beba94275a4