Actually build the directories