Rimmer joined Manchester United as a schoolboy in 1963, turning professional two years later. He spent eleven years at Old Trafford, chiefly as Alex Stepney's understudy, including being on the bench in the 1968 European Cup Final. Rimmer only played 46 times for United and in October 1973 he was loaned to Swansea City. Arsenal signed him from United in February 1974, as a long-term replacement for Bob Wilson.