At Chelsea, Spencer enjoyed a consistent period of playing in his career. Between August 1992 and October 1996 he made 137 appearances and scored 43 goals in all competitions. He featured in the teams which lost 4–0 to Manchester United in the 1994 FA Cup Final and reached the semi-finals of the 1994–95 European Cup Winners' Cup and the 1995–96 FA Cup.