Mariah Carey and Nicki Minaj left the panel after one season