The major drawback is no hot coffee. At the observatory there is electricity and shelter. I also have a small coffee maker inside. When we are open to the public there is an activity center with heat and bathrooms that we can use. If it's just a couple members we pile into whichever vehicle has the best heat when we see clouds or aurora moving in.
Wind is the cruelest.
If a tear drops onto the eyepiece it's all over for that one. Having to expose any flesh to wind chill can be downright hazardous. A lot of people consider things like eyepiece and dew shield heaters a luxury. For me to stay out for more than two hours they are a necessity. Just keeping my pen warm enough for the ink not to freeze means sticking it behind my ear under my hat.