Pitts was selected by the Packers in the 13th round of the 1961 NFL draft, 180th overall. He turned down a higher offer from the Boston Patriots of the AFL to play for a better team in the more established league. A reserve for much of his early career behind hall of famer Paul Hornung, he saw his most action for the Packers in 1966, and scored two touchdowns in the first Super Bowl.