Build only lib from package

Fixes #724 

Yeah that'd also work.