Panel: Fix link listener on removed panel

After a panel is removed, its link listener will still fire. If a new
panel with the same id is added, the previously removed panel will be
chosen for any panel link click events.