Lee was a full-back who began at Hednesford Town and joined Aston Villa in October 1955, who were then struggling to survive in the First Division under the stewardship of Eric Houghton. They avoided relegation in 1955–56 on virtue of their superior goal average to Huddersfield Town. Villa finished tenth in the 1956–57 and won the FA Cup, though Lee did not feature in the final. After a 14th-place finish in the 1957–58 campaign, the club were bottom of the league by December of the 1958–59 season, and new boss Joe Mercer could not prevent the drop.