Stafford played at Shattuck-Saint Mary's in Faribault, Minnesota, for two years and played on the US national team at the 2003 IIHF World U18 Championships. He then played for three seasons at the University of North Dakota and on the US national team at the 2004 and 2005 World Junior Ice Hockey Championships. Following his freshman season, Stafford was drafted 13th overall in the 2004 NHL Entry Draft by the Buffalo Sabres. After two more seasons, he signed a professional contract with the Sabres in 2006, foregoing his senior season at the University of North Dakota. Stafford finished his three-year collegiate career with 118 points (48 goals and 70 assists).