Remove unneeded PKG_CONFIG_PATH tweaks

Looks like everything works now even without manipulating the
PKG_CONFIG_PATH