He was drafted by the Baltimore Colts in the 10th round of the 1960 NFL Draft. In his first season, he played in 11 games, while having 7 interceptions for 132 yards. The following season, he played in 14 games while having 2 interceptions. He also returned 18 punts for 173 yards (with 9.6 yards per return), finishing 4th in the former category and 3rd in the latter category in the league. In his next season, he played in every game once again while having 7 interceptions for 163 yards. He also recovered four fumbles along with returning 3 punts for 23 yards and having 2 carries for 13 yards. He played in 12 games in the 1963 season, but he had 3 interceptions for 17 yards along with recovering 2 fumbles for 34 yards and one touchdown. In the 1964 season, he had 9 interceptions for 185 yards, both career highs while also having one fumble recovery. He also one carry for 25 yards. He was named First Team All-Pro and the Pro Bowl in 1964. He played in his first playoff game that year, playing in the 1964 NFL Championship Game, which the Colts lost 27-0.