CoreWM: Fix panel destruction