Fix a number of bugs in signature checking

When given badly formatted  signature or pubkeys like in block
0000000000000001e4241fd0b3469a713f41c5682605451c05d3033288fb2244, transaction
fd9b541d23f6e9bddb34ede15c7684eeec36231118796b691ae525f95578acf1 we could fail
on strange scripts because we returned an error instead of failing the
opcode and putting a FALSE on the stack.

Fixes chainfork issue on the aforementioned block.