Revert "Add system CFLAGS after vendor library paths"

This reverts commit cea15456af6d329f79c67bf608df68a762c22d84.

For some weird reason this seems to break on Linux.