The point is not the remote possibility that we are all brains in vats but that we may be just as deluded in ways we cannot even imagine. The whole of experience is a stream of nerve impulses. The roughness of a stone, the sound of a coin dropping on the floor, the odor of a fart. The conventional picture of a real, external world is not the only possible explanation for the neural experience. We are forced to admit that an evil genius or a brains in vats experiment could explain the neural experience just as well.
Experience is forever equivocal.
Science places great faith in the evidence of the senses. Most people are skeptical about ghosts, the Loch Ness monster, and flying saucers, not because they are inherently stupid notions, but only because no one has produced unquestionable sensory evidence for them. Brains in vats turns this apparently reasonable skepticism inside out. The belief that you are NOT a brain in a vat can never be disproven empirically. At issue is not some bit of trivia such as the color of a tyrannosaurus.