Unless you have a backpack or panniers this is a hassle. Of course, we aren't even mentioning the problems involved in undressing and redressing in public or in a howling wind. Cyclists generate a lot of sweat. By the time you realize you are too hot, you are already wet. If you don't remove the wet garment, you will chill yourself more than you think as upper layers are removed.
Cyclists can regulate temperature by level of effort.
Winter commuters learn to dress for the temperature. They will start out slightly underdressed , and therefore feeling a little cold. In a couple miles they are warmed up and cranking along. Once underway, most winter commuters will regulate temperature by work level rather than stop and change. Rain gear is the exception.