avcodec/libx264: Implement reference frame count limiting based on level

This makes libavcodec/libx264.c behave more similar to the x264 command line
util

Fixes Ticket3307

Implementation based on x264

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>