Remove this.$element on destroy but don't nullify

Bug: T84897
Change-Id: I794fa007358515022598dd3d66683beba94275a4
(cherry picked from commit 7bba15fba90590e32c763c30c2f08114937b8e8f)