Ticket #5425: Avoid infinite recursion in checkMemsetType

Hi,

This patch avoid infinite recursion upon invalid input and therefore fixes the ticket. Thanks to consider merging.

Cheers,
  Simon