Remove Darwin specific setup

Relies on pkg-config being available on all platforms.

@pselva @immersive/immersive-collaborators