Improved logic for setting the lib dir

Use logic similar to that in Avogadro, reduces special cases for Mac,
also brought in the debug logic to help when there are problems with
locating plugins.

Change-Id: I2feaa03ee83fc9232c991af2db432400a98adc41
Merge topic 'mac-lib-dir' into master

a1d34b3b Improved logic for setting the lib dir