Nelson was drafted by the Baltimore Colts in the eleventh round of the 1957 NFL Draft. Having played both quarterback and defensive back in college, the Colts made him a defensive player. Nelson started 12 games his rookie season, with 5 interceptions for 29 yards. The following season, Nelson would record 8 interceptions for 199 yards and 1 touchdown. The Colts would go on to win the NFL Championship in what was the NFL's first sudden-death overtime game and has since become widely known as "The Greatest Game Ever Played". The Colts would repeat in 1959 and over his seven-year career with the Colts, Nelson would rack up 33 total interceptions for 378 yards and 3 touchdowns. The Colts traded Nelson to the New York Giants for the 1964 season, his last in the NFL.