Only use -Wno-pointer-sign in libyaml configure.