Allen was drafted by the Boston Celtics in the first round (25th overall) of the 2004 NBA draft on June 24, 2004, and signed a multi-year contract with the Celtics on July 3, 2004. As a rookie in the 2004–05 season, he averaged 6.4 points, 2.9 rebounds and 0.99 steals in 16.4 minutes over 77 games (34 starts). He scored 20 points in his first career start on January 22, 2005, against the Atlanta Hawks. He later participated in the Rookie Challenge at the 2005 NBA All-Star Weekend, scoring a team-high 17 points.