Some people remove these and wash them, using a special flexible brush to scrub the inside of the tube. I then fill the keg with hot tap water and again hook it up to the gas, then use the gas to drive the entire contents through the keg and dispensing system. In this way, it is thoroughly rinsed and at the same time filled with C02 to prevent growth until the next time I'm ready to use the keg. It does waste some gas, but I've never had a problem with dirty kegs with this method. The latex seals are a major source of leaks and contamination.
Rubber is porous, therefore it retains small amounts of residue.
Rubber also wears out over time and will leak, causing loss of your dispensing gas. When you first obtain a used keg, you should replace all of the latex rings and seals. Only the gasket needs to be removed for cleaning between each batch of beer, the rest should be adequately cleaned using the method I outlined above for the keg itself. I recommend replacing all of the rubber in your kegs once a year - mainly because of wear, rather than contamination concerns. The two poppet valves inside the nipples are usually stainless and plastic and don't require replacement unless you are having leaks.