Should be able to prealloc this as `num_perms` is known