Fix mixups between libdir and librarydir

We were using different names, meaning we would not handle the libdir
option properly.