Fraser was raised on his father's sheep stations, and after studying at Magdalen College, Oxford, returned to Australia to take over the family property in the Western District of Victoria. After an initial defeat in 1954, he was elected to the House of Representatives at the 1955 federal election, standing in the Division of Wannon. He was 25 at the time, making him one of the youngest people ever elected to parliament. When Harold Holt became prime minister in 1966, Fraser was appointed Minister for the Army. After Holt's disappearance and replacement by John Gorton, Fraser became Minister for Education and Science (1968–1969) and then Minister for Defence (1969–1971). In 1971, Fraser resigned from cabinet and denounced Gorton as "unfit to hold the great office of prime minister"; this precipitated the replacement of Gorton with William McMahon. He subsequently returned to his old education and science portfolio.