Fix integer overflow in MallocFrameBuffer()

Promote integers to uint64_t to avoid integer overflow issue during
frame buffer allocation for very large screen sizes