Fix issues with A_mul_Bc and Ac_mul_B for CholmodSparse (#9160).

As described in the title. Is there any test for Cholmod related functions?
bit more accurate to have this return `Csize_t`