check malloc return for code in rabin2.c
Fix CID 1134670