Move metrics manager setup from worker to API

Part of #914, required by #822.