Fix build errors on latest Ubuntu

We need to be more careful about the order of compiler arguments from
pkg-config.