Matrix scanned in, will create perm vector now