Nuclear reactors produce plutonium, and plutonium is terrible because it can be used to make bombs. Perhaps I have caricatured this argument, but I haven't seen it in a coherent form. Safeguards are indeed needed. There is no policy including a ban on nuclear energy that can prevent a determined country from launching a nuclear war. So far deterrents have sufficed, and a total ban would give an enormous advantage to the first country to violate it.
Plutonium is the most toxic substance in existence.
It isn't the most toxic substance. There are plenty of toxic substances, and they occasionally poison people. Plutonium is one of the easiest to keep under control. Plutonium symbolizes nuclear war. Purely symbolic arguments have no force.