Make sure we never have commands run more than once

Previously could happen if the package checker was adding commands to be
executed while also deleting commands.