Creates a not_ready directory for non phpunit tests