Accept heterogeneous eigen vectors in eigen backend