nghttp2_frame: Return NGHTTP2_ERR_FRAME_SIZE_ERROR instead of .._BUFFER_ERROR

Also updates docs
nghttp2_frame: Add assertion to available buffer size in first buffer
Update doc
Update doc
Code cleanup
nghttp2.h: Use hex style flag definition