Safe-guard against empty directories

CMake will raise an error if you try to remove something from an
unset variable (whereas removing something from an empty list "" is
fine). If the user doesn't have the appropriate library, the search
will turn out empty, and the configuration will fail with a syntax
error instead of a message that the library is not found.
Cleanup unused variable