[sanitizer] improve the calloc overflow check (spotted by samsonov@)