[configure] use pkg-config for rtmp detection. fixes linking with newer librtmp versions
[configure] fix static linking when ffmpeg found nettle/hogweed
Merge pull request #4642 from wsnipex/linking-fix

Linking fix [fixes #15139]