It's a large component of it, yes. There are some grammar rules that make Mandarin kind of difficult for English speakers, but the tones can be a huge problem if you aren't really used to controlling your pitch while you speak in your native language.