paltalk: fix undefined behavior (shift out of range)

The shift `(gint32)0xFF << 24` invokes undefined behavior as it may not
fit in a signed integer. Fix this by explicitly casting 0xFF as
unsigned. Caught by `clang -fsanitize=undefined`.

While at it, convert to tvb_captured_length and add modelines.

Change-Id: I241ff8ed91815369ec0c19719750cee4b6b12343
Reviewed-on: https://code.wireshark.org/review/3311
Reviewed-by: Evan Huus <eapache@gmail.com>