Don't use what we're hovering over, but what we selected when changing
difficulties.