Build fixes

* improve X11 detection on OS X
* add support for cmake 3.x
* install rdtk and shadow libraries with version (not only .so/.dylib)