While removing most of the coffee grounds is recommended for a clean drinking experience, removing every dirt is unnecessary. Some residue may remain, but it should be fine as long as it doesn't affect your enjoyment of the beverage or pose any health concerns.