TComRom: fix fraction part variation in x265_lambda2_tab for higher QPs