envoy: fail if agent can't be fetched

Regression since the new IPC stuff landed