[#70887666] Have runner handle agent-less configuration

Signed-off-by: John Tuley <jtuley@pivotallabs.com>