Remove Darwin specific setup

Relies on pkg-config being available on all platforms.