The most likely reason your car stereo turns off at high volume is because it's protection mechanism is kicking in. Most car stereos have some sort of built-in protection that kicks in when the system is pushed too hard. This protection usually takes the form of a circuit breaker that trips when too much current flows through it. When this happens, the stereo shuts off to prevent damage to the system.
There are a few things that can cause this protection to trip more easily than others. One is if your speakers are low quality or mismatched. If you have a subwoofer, make sure it's properly matched to your other speakers. Another possibility is that your amplifier isn't powerful enough for your speakers. If you're driving high-end speakers, they may require more power than a factory amplifier can provide. Finally, make sure all your connections are tight and free of corrosion. Loose or corroded connections can create resistance, which causes the amplifier to overheat and trip the protection circuit.
If you're not sure what's causing your stereo to turn off at high volume, take it to a qualified technician for diagnosis and repair. In most cases, this is a relatively easy and inexpensive fix.