Your question is an interesting one. I have nothing but anecdotal evidence to confirm your opinion that people with hazel color eyes report eye color change throughout their lifetime.
All eye color is determined by the amount of a single pigment, melanin, in the iris.
The determination of eye color is now known to be controlled by at least three gene pairs with other genes suspected of contributing as modifiers of those genes. People with hazel eyes may have melanin deposited nonuniformly throughout the iris. This alone can make the eye color appear different in different light. It is also possible that there are genes that modify melanin deposition throughout our lifetime. It describes in more detail how eye color is determined and has some interesting links to other genetics sites.