Do part of cmake's job manually

Who needs to link to dylibs anyways