Computation number theory homework