Most people visit for three to four days, and I think that's a perfect amount of time. As someone who's lived there, I can say that, since Boston is so small, you won't waste a lot of time “in transit,” so you can pack a lot into your days.