cmake: re-split TLibCommon headers to fix VC9 builds
vc9: define UINT64_MAX in our hacked stdint.h
rdcost: overflow check by integer