Norman was born in Mulbarton, Norfolk. He began his career at Norwich City, and played 35 league matches for the Canaries between 1952 and 1955. Despite such a short career with the club, he was elected into the Norwich City Hall of Fame in 2002. He signed for Tottenham Hotspur in November 1955 for a £28,000 transfer fee, that also included the return transfer of Ireland international striker Johnny Gavin to Norwich. Norman played his first game for his new club against Cardiff City, and stayed at White Hart Lane until 1965, making 411 first-team appearances and scoring 19 goals for Spurs. He was an integral part of Bill Nicholson's Double-winning Tottenham team of 1960–61 that went on to retain the FA Cup in 1962, and win the 1963 Cup Winners' Cup.