On joining the Royal Engineers in 1873, Stafford commenced playing association football, becoming a "resolute forward, working to the very last", although he was "a little too fond of a charge". At this time, the Royal Engineers were among the top football teams in England, having reached the final of the first FA Cup tournament in 1872 and again two years later, finishing on both occasions as runners-up. In the run to the 1875 final, Stafford scored twice in the early rounds in a 5–0 victory over Cambridge University and in the next match, a 3–2 victory over Clapham Rovers. This set up a hard-fought semi-final against Oxford University in which the Engineers emerged victorious with a 1–1 draw followed by a 1–0 victory in the replay, thus reaching the FA Cup Final for the third time in four years.