cppcheck: call by reference, efficient emptyness check

remove unused prototype while here
allocate buffer for PESPacket with pes_alloc (coverity 1047220)
Explicit casts for C++11 standards compliance

Fixes #11845