Improve removing the library path

Fixes #148.