Support both --with-* and --without-* options for libsodium and pgm