A variety of factors causes caffeine sensitivity, such as genetics and your liver's ability to metabolize caffeine. A caffeine allergy occurs if your immune system mistakes caffeine as a harmful invader and attempts to fight it off with antibodies.