core: disallow brackets in VM names [GH-4319]