Create cache dir for first run

Probably this should be a task rather than a setting but I couldn't make
that work.