On your torso, the appropriate layers allow you to moderate heat retention. Perhaps most important is a layer of polypropylene, which will wick moisture away from your skin. Unlike cotton, polypro dries quickly. By wearing a second layer that isn't too tight, you'll create a pocket of heat-retaining dead air space. From there, it's a matter of personal preference.
Some riders are comfortable on most days in a short-sleeve cycling jersey and vest.
Some don't even require the vest. Others might need a long-sleeve jersey or vest. And some riders are quite comfortable riding with two or three sub-layers under a cycling jacket. If you do choose to ride in a jacket, you might want to look for one that won't puff to the size of a parasail. They also sound like a luffed sail going down the road.