Obviously the latter should be the answer, but if your helmet does not fit properly, it may not be affording as much protection as it could.
Heads come in all shapes and sizes and so do riding helmets.
The challenge is to match one to the other as perfectly as possible. The best way to ensure the fit is to try the helmet on and wear it around for several minutes in the shop. Before even trying on the helmet, take time to read the label. It should be ASTM approved or have the SEI label. Both these indicate that the helmet meets a rigorous standard for safety under a variety of conditions.