avformat/thp: fix variable types to avoid overflows

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
avformat/thp: force moving forward

Fixes infinite loop
Fixes Ticket3098

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>