Refactor sidebar code to properly cleanup after itself