transform vertices by matrix