Revert "kmselement: Avoid deadlock not setting pad to inactive before removing it"

This reverts commit f4d370fc6a76e843d8b2925df3a79829f0f5770a.

Change-Id: Ib262fc0dc7f932a465dd9453623d4ada5871b8a1