RD merge: refine merge costs with estimated merge mode bits.