Caffeine is a powerful central nervous system stimulant whose effects diminish with increased usage. Larger and larger doses can become necessary to avoid withdrawal symptoms and sustain stimulation until you need a cup just to function normally. To avoid this habituation, drink caffeine only during or before races and training. Most road cyclists limit their caffeine consumption to a bottle of coke diluted with water during training rides and the last half of longer races or a cup of drip coffee a half an hour before shorter races. The exact mechanisms of caffeine's diminishing effects are not well known but are probably are due to reduced levels of circulating free fatty acids and CNS neurotransmitters.
Caffeine affects different people differently.
For some it only causes uncomfortable nervousness, high blood pressure and stomach acid. A strong cup of coffee, to someone who does not drink caffeine regularly, can last all day and well into the night, leaving a groggy feeling the next day that can be remedied only with another cup. If you do drink more than a cup of coffee every day some cutting back would probably go far towards improving your riding quality. When a cup of coffee is no longer an option but a prerequisite for alertness and energy it is probably doing more harm than good. This is the negative side of drinking the stuff, moderation can be difficult.