pescan: code style fixes

Fix switch indent.

Addressing:
https://github.com/inliniac/suricata/pull/174#discussion_r3332386
pescan: code style fixup: variable naming maxSize

Rename maxSize -> max_size
pescan: test malloc result in unittest