For business casual environments, stick with tailored sweaters, not pullovers or fleeces. Be mindful of your outerwear as well. If you are dressed in a nice suit, don't ruin it by putting a fleece or sweatshirt over top.