Ticket #5425: Avoid infinite recursion in checkMemsetType for invalid input