Spencer played local amateur football for a number of clubs in his local area before joining Football League First Division side Aston Villa from Birchfield Trinity in April 1892 as an amateur, turning professional two years later in June 1894. He made his debut for the club four months later on 13 October 1894 during a 3–1 victory over their West Midlands rivals West Bromwich Albion. At the end of his first season with the club, Spencer played in the 1895 FA Cup Final victory over West Brom at the age of 18 and, the following season, helped Villa to win the First Division title. This was followed by the league and FA cup Double winning season of 1896-97. Spencer was a key member of the Villa team that won back-to-back titles in 1898-99 and 1899–1900.