Horses are flight animals and can always get frightened or fall. Even the most experienced rider on the most well-behaved horse cannot prevent this. You only have one head, protect it as best you can!
There is a wide range of light and airy riding helmets that are comfortable to wear even in warm temperatures. Even the price (usable helmets are available from 40 euros) should not be a reason to forego wearing a riding helmet. Just as you protect your legs or hands with breeches and riding gloves, you should of course do the same with your head.