[configure] use pkg-config for rtmp detection. fixes linking with newer librtmp versions
[configure] fix static linking when ffmpeg found nettle/hogweed
Merge pull request #4654 from wsnipex/linking-fix

Fix building on linux with newer librtmp nr2