add missing va_end()s

(Spotted by Coverity: CID 1109741)
fix potential segfault

When switching rtp/udp or changing MTU (spotted by Coverity: CID 1109734).
fix dead code

(spotted by Coverity: CID 1109724)
check setsockopt return value

(spotted by Coverity: CID 1109720 & 1109719)
fix confusing indentation

(spotted by Coverity: CID 1100854)