Remove unused lib dirs from config