They think they're doing you a favor by speaking in your native language, but they're actually handicapping your ability to learn.
It's your English that actually gets worse as a result of these interactions, as you're constantly having to dumb it down so it makes sense to them.