Pascoe was born in Richmond, Victoria in 1947. He grew up in a poor working-class family; his father, Alf, was a carpenter, and his mother, Gloria Pascoe, went on to win a gold medal in lawn bowls at the 1980 Arnhem Paralympics. Pascoe spent his early years on King Island where his father worked at the tungsten mine. His family moved to Mornington, Victoria, when he was 10 years old, and then two years later moved to the Melbourne suburb of Fawkner. He attended the local state school before completing his secondary education at University High School, where his sister had won an academic scholarship. Pascoe went on to attend the University of Melbourne, initially studying commerce but then transferring to Melbourne State College. After graduating with a Bachelor of Education, he was posted to a small township near Shepparton. He later taught at Bairnsdale for nine years.