Create dir for riemann logs and config