The problem stems from a lack of control over where the light is being cast and a lack of appreciation about how to allow lighting to work with the eye's natural ability to adjust to lower light levels at night. You would justifiably think they were incompetent. Yet this is exactly the situation we often find ourselves in with regard to common outdoor lighting. What works best in lighting is to light from above and to not try to light a distance beyond four times the mounting height of the fixture. However, when we do, not only does light strike the ground at such a distance from the fixture that it is no longer useful but it also causes substantial glare.
Glare reduces our ability to see at night.
Many older fixtures not only allow light in this glare zone but many also allow too much light to escape unused and wasted directly into the night sky. The glare zone is within 15 degrees below the horizontal plane. With full cut off shielding and reflectors to use the wasted light, fixtures can be downsized using less power. This would mean a large increase in the number of stars that could be seen. The vista of the Milky Way would return to view near our cities again.