Taufer was born Venčeslav Taufer in Ljubljana, Slovenia, then part of the Kingdom of Yugoslavia. His father was a left liberal activist, and prominent member of the Sokol athletic movement. Due to his political activities, he was transferred by the Yugoslav conservative regime to the heavily industrialized Central Sava Valley in central Slovenia, where Veno spent his childhood. In 1943, during the German occupation of Yugoslavia in World War II, his father was killed by the Nazis as one of the leaders of the local partisan resistance.