It's just human nature to accept the program. Even if the risk of accepting one program is low, the risk adds up when you repeatedly accept programs. And when you do get the one bad program, there is no limit on how much damage it can do. The only way to avoid this scenario is to refuse all programs, no matter how fun or interesting they sound, except programs that come from a few people you know well. The main danger in Java comes from the complexity of the software that implements the sandbox.
Common sense says that complicated technology is more likely to break down than simple technology.
Java is pretty complicated, and several breakdowns have happened in the past. If you're the average person, you don't have the time or the desire to examine Java and look for implementation errors. So you have to hope the implementers did everything right. They're smart and experienced and motivated, but that doesn't make them infallible. For example, Java programs are not allowed to access files, so there's no way to write a text editor.