Add ability to skip plugins at runtime

(opened on behalf of @spacetrack)