For some it only causes uncomfortable 'nervousness', high blood pressure and stomach acid. A strong cup of coffee, to someone who does not drink caffeine regularly, can last all day and well into the night, leaving a groggy feeling the next day which can be remedied only with another cup. If you do drink more than a cup of coffee every day some cutting back would probably go far towards improving your riding quality. When a cup of coffee is no longer an option but a prerequisite for alertness and energy it is probably doing more harm than good. This is the negative side of drinking the stuff, moderation can be difficult.
Caffeine is also a powerful diuretic.
This can mean serious health and performance dangers. Many riders do not drink enough fluids even without caffeine. For this reason it is best to avoid any form of caffeine during hot weather and drink more water than normal with any coffee, cola or tea. Remember, the fatigue caused by dehydration is indistinguishable from the normal fatigue of hard training. The exact mechanisms of caffeine's effects are not well known but are probably are due to altered levels of CNS neurotransmitters and the metabolism of circulating free fatty acids.