[polygamma] Fix gcc-11 test failures.