pointer/keyboard: Make sure to clean up the destroy listeners when releasing