Revert "Don't use per puzzle host"

This reverts commit 84ef8745ce38f02b7e18bc6d7edf61faec1b651c.