A year after Tottenham Hotspur manager Terry Neill took over at Arsenal, he signed Pat Jennings from his old club and Rimmer was sold to Aston Villa, having played 146 games for Arsenal. At Villa, Rimmer was No. 1 for the next six seasons, winning a First Division winners' medal in 1981. The following year Villa reached the European Cup final, but Rimmer was injured after only nine minutes and had to be replaced by Nigel Spink. Villa beat Bayern Munich 1–0, meaning Rimmer became the second player, after Saul Malatrasi, to win a European Cup winners' medal at two clubs.