Don't remove the libraries.

The build dependencies and libraries where both defined in $BUILD_DEPS
which caused them to be removed as well during the cleanup. I have now
split them up in two separate variables to fix this.