It is the hardest thing I teach, even though it is only one idea, but once you have mastered it, it will never leave you. Think about the English stops, for example. If you're like most American speakers, not much of a puff. But what if an English speaker DIDN'T vary the pronunciation of stops depending on where they occurred within a word. The answer is yes, though such speech would certainly sound odd.
Linguists call such insignificant varieties allophones of phonemes.
It is important to remember that every language has a different collection of phonemes and allophones. In many of the languages of India, for example, aspirated stops and unaspirated stops are separate phonemes. The Sapir-Whorf Hypothesis Edward Sapir and Benjamin Lee Whorf were very different men, each of whom, in his own way, was a brilliant linguist. Among the many issues they considered, the one they are best known for is the relationship between the language an individual or group speaks and the patterns of thought of that individual or group. The problem is that what Sapir and Whorf said about language and thought varied from lecture to lecture, and from article to article.