Willoughby was born in Springburn, Glasgow, and educated at Petershill Primary School and Colston Junior Secondary School, both located in the Springburn area. He played youth football with his local Boys' Brigade team, and then with Drumchapel Amateurs before commencing his career with Rangers as a schoolboy signing at 15 years of age in 1959. He subsequently signed professional terms with Rangers in 1961. He scored in the 1963 Scottish League Cup Final win over Morton, but was omitted from the 1967 European Cup Winners' Cup Final team to the surprise of many, with Roger Hynd preferred, despite Willoughby having scored 17 goals in 14 games in the three months prior.