Maybe instead of doing a "full stack" test, could you do a regular test invoking the task directly? This is because full stack tests are take a lot of time. It will work something like this:

    in_tmp "wrong_name", fn ->
      assert_raise Mix.Error, %r(the name is already used by a dynamo module), fn ->
        Mix.Tasks.Dynamo.run ["dynamo"]
      end
    end

I haven't tested it, so I am not a 100% sure! :D