[#1911] Only create the tmp dir if it doesn't already exist