Across multiple sources, Mandarin Chinese is the number one language listed as the most challenging to learn. The Defense Language Institute Foreign Language Center puts Mandarin in Category IV, which is the list of the most difficult languages to learn for English speakers.