Both judges Mariah Carey and Nicki Minaj also decided to leave after one season to focus on their music careers