Following the war, Brown played for Charlton in the FA Cup Final on 27 April 1946, which was lost 4–1 to Derby County at Wembley Stadium. He joined Nottingham Forest for a fee of £6,750 in May 1946. He made 45 appearances and scored 17 goals for Forest before moving to Aston Villa in October 1947 for a fee of £10,000, then the club's record highest fee paid. After making 30 appearances and scoring nine goals, he was appointed as Gorleston player-manager in August 1948. He left the club in May 1956 to retire from football and worked as a bookmaker and a timber merchant, while also scouting for Villa, although he briefly returned as Gorleston manager in 1958 after Joe Jobling left the club. He lived in Forres, Scotland in his later life and died on 27 December 2008, at the age of 93.