mpegts: Return proper error code on invalid input data

Signed-off-by: Diego Biurrun <diego@biurrun.de>