Following the Conservative's defeat in the February 1974 general election he became Shadow Secretary of State for Scotland under Edward Heath. When Margaret Thatcher succeeded Heath as Conservative leader "The Glasgow Herald" reported speculation that Buchanan-Smith was one of a group of "top Tories" who might refuse to serve under her. Ultimately he remained in post under Thatcher, but resigned in 1976, along with his junior shadow minister Malcolm Rifkind, when she changed the Conservative Party's policy to oppose Scottish devolution. Subsequently, he led the Conservative contribution to the Yes campaign in the 1979 Scottish devolution referendum.