Revert cmake libevdev changes

It doesn't do what it's supposed to.

Doesn't point to prefix, only relevant for include path.
Previous setup isn't that bad so revert for now.