DVBSub/PGSSub: Add more error checking when parsing composition object.

Fixes CID #971508.
Remove some dead code.

Fixes CID #965946.
Improve the Get(Cur)DispMode and SetDispMode functions.

Fixes CID #966222.