When dealing with bullies, don't get in the position of being a victim.
Bullies enjoy bullying.
Pushing someone around makes them feel stronger because they feel weak or bad inside much of the time. But bullies are not always bullies. Apologizing is almost always safe. If you have had bad past experience with this kid and he scares you, it is hard to keep your cool, so it is sensible to get away as quickly as possible. But don't let him scare you into being rude yourself.