The 2500uF capacitor helps to maintain a constant input into the regulator. The 0.1uF capacitor eliminates any high frequency pulses that could otherwise interfere with the operation of the regulator. Voltage regulators are very robust. In both cases the regulator will shut down before damage occurs. The only way to destroy a regulator is to apply reverse voltage to its input.
Reverse polarity destroys the regulator almost instantly.
To avoid this possibility you should always use diode protection of the power supply. Even this short 'test' would destroy the regulator if a protection diode were not used. All of the interfaces described on this site have protection diodes for this reason. A Protection Diode on the power supply input line. As a general rule the input voltage should be limited to 2 to 3 volts above the output voltage.