Nelson made 45 appearances in all competitions in his two seasons with Brighton. In the second, 1982–83, they were relegated from the First Division but reached the 1983 FA Cup Final, losing to Manchester United 4–0 in a replay after the original match finished 2–2; Nelson was not selected for either match. He retired as a player that summer.