Fix OTR test case (bug #3)

One of the participants in the OTR test case uses fragmentation. With
OTRv3 the messages need more space, so increase the minimum mss used
by the test.

Fixes bug #3. Thank you, Takashi Kato.
assembler x86: Fix bogus error call

When an instruction had an invalid prefix an error reporting procedure
was called incorrectly.

Also do some minor code cleanup.

Fixes bug #2. Thank you, Takashi Kato.