add dealing with maxpayloadsize

reviewed at: https://rbcommons.com/s/OpenH264/r/525/

aslo astyle gmp-openh264.cpp