Repository: beeware/briefcase

Issue:
CI is not Reporting Error States for App Verification Tests
**Describe the bug**
The `briefcase create windows visualstudio` command was erroring for me in my local environment. Then I realized it was doing it on `main`. Then I realized that CI was [logging these errors](https://github.com/beeware/briefcase/actions/runs/3189260683/jobs/5203020654#step:10:73) but not reporting them.

**To Reproduce**
Steps to reproduce the behavior:
Review a recent CI run for `main`; for example: https://github.com/beeware/briefcase/actions/runs/3176163100/jobs/5175282275#step:10:73

**Expected behavior**
We need CI on our team telling us when things don't work :)

If any step in CI fails, then the entire CI run should fail.

**Screenshots**
Error being logged for posterity.
```
[verify-toga] Installing application code...

Log saved to D:\a\briefcase\briefcase\tests\apps\verify-toga\briefcase.2022_10_03-17_48_01.create.log

Traceback (most recent call last):
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\Scripts\briefcase.exe\__main__.py", line 7, in <module>
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\briefcase\__main__.py", line 21, in main
    command(**options)
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\briefcase\commands\create.py", line 784, in __call__
    state = self.create_app(app, **full_options(state, options))
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\briefcase\commands\create.py", line 748, in create_app
    self.install_app_code(app=app)
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\briefcase\commands\create.py", line [52](https://github.com/beeware/briefcase/actions/runs/3176163100/jobs/5175282275#step:10:53)4, in install_app_code
    self.tools.shutil.rmtree(app_path)
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\shutil.py", line 7[59](https://github.com/beeware/briefcase/actions/runs/3176163100/jobs/5175282275#step:10:60), in rmtree
    return _rmtree_unsafe(path, onerror)
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\shutil.py", line 610, in _rmtree_unsafe
    onerror(os.scandir, path, sys.exc_info())
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\shutil.py", line [60](https://github.com/beeware/briefcase/actions/runs/3176163100/jobs/5175282275#step:10:61)7, in _rmtree_unsafe
    with os.scandir(path) as scandir_it:
FileNotFoundError: [WinError 3] The system cannot find the path specified: 'D:\\a\\briefcase\\briefcase\\tests\\apps\\verify-toga\\windows\\VisualStudio\\Hello Toga\\x[64](https://github.com/beeware/briefcase/actions/runs/3176163100/jobs/5175282275#step:10:65)\\Release\\app'
```

[logs_1591.zip](https://github.com/beeware/briefcase/files/9736397/logs_1591.zip)

**Environment:**
 - Operating System: GH CI Microsoft Windows Server 2022
 - Python version: 3.9.10
 - Software versions:
   - Briefcase: 0eed8e1c45c7c1c761dc5e2da08c7a37b0ea3ea7

Discussion:

Patch:
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index c4a1aa76..8b9d4a1f 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -193,6 +193,7 @@ jobs:
         cd tests/apps
         cat verify-${{ matrix.framework }}.config | briefcase new
     - name: Build App
+      shell: bash
       run: |
         cd tests/apps/verify-${{ matrix.framework }}
         briefcase create
@@ -200,6 +201,7 @@ jobs:
         briefcase package --adhoc-sign
     - name: Build Xcode project
       if: matrix.os_name == 'macOS'
+      shell: bash
       run: |
         cd tests/apps/verify-${{ matrix.framework }}
         briefcase create ${{ matrix.os_name }} Xcode
@@ -207,6 +209,7 @@ jobs:
         briefcase package ${{ matrix.os_name }} Xcode --adhoc-sign
     - name: Build Visual Studio project
       if: matrix.os_name == 'windows'
+      shell: bash
       run: |
         cd tests/apps/verify-${{ matrix.framework }}
         briefcase create ${{ matrix.os_name }} VisualStudio
@@ -214,6 +217,7 @@ jobs:
         briefcase package ${{ matrix.os_name }} VisualStudio --adhoc-sign
     - name: Build Flatpak project
       if: matrix.os_name == 'linux' && matrix.framework == 'toga'
+      shell: bash
       run: |
         cd tests/apps/verify-${{ matrix.framework }}
         briefcase create ${{ matrix.os_name }} flatpak
@@ -221,6 +225,7 @@ jobs:
         briefcase package ${{ matrix.os_name }} flatpak --adhoc-sign
     - name: Build Android App
       if: matrix.framework == 'toga'
+      shell: bash
       run: |
         cd tests/apps/verify-${{ matrix.framework }}
         briefcase create android
@@ -228,6 +233,7 @@ jobs:
         briefcase package android --adhoc-sign
     - name: Build iOS App
       if: matrix.platform == 'macos-12' && matrix.framework == 'toga'
+      shell: bash
       run: |
         cd tests/apps/verify-${{ matrix.framework }}
         briefcase create iOS
@@ -235,6 +241,7 @@ jobs:
         briefcase package iOS --adhoc-sign
     - name: Build Web App
       if: matrix.framework == 'toga'
+      shell: bash
       run: |
         cd tests/apps/verify-${{ matrix.framework }}
         briefcase create web
diff --git a/changes/908.misc.rst b/changes/908.misc.rst
new file mode 100644
index 00000000..9f8b1328
--- /dev/null
+++ b/changes/908.misc.rst
@@ -0,0 +1,1 @@
+Failures on intermediate commands in CI are correctly surfaced as CI errors.
diff --git a/changes/909.misc.rst b/changes/909.misc.rst
new file mode 100644
index 00000000..b5bae0ee
--- /dev/null
+++ b/changes/909.misc.rst
@@ -0,0 +1,1 @@
+``briefcase create`` no longer fails if a template doesn't provide an empty ``app`` folder.
diff --git a/src/briefcase/commands/create.py b/src/briefcase/commands/create.py
index d1ef8e00..dfff0ff8 100644
--- a/src/briefcase/commands/create.py
+++ b/src/briefcase/commands/create.py
@@ -530,9 +530,10 @@ class CreateCommand(BaseCommand):
 
         :param app: The config object for the app
         """
-        # Remove existing app folder
+        # Remove existing app folder if it exists
         app_path = self.app_path(app)
-        self.tools.shutil.rmtree(app_path)
+        if app_path.exists():
+            self.tools.shutil.rmtree(app_path)
         self.tools.os.mkdir(app_path)
 
         # Install app code.


Test patch:
diff --git a/tests/commands/create/test_install_app_code.py b/tests/commands/create/test_install_app_code.py
index 0a34cec6..bd395cf1 100644
--- a/tests/commands/create/test_install_app_code.py
+++ b/tests/commands/create/test_install_app_code.py
@@ -7,6 +7,8 @@ import pytest
 import briefcase
 from briefcase.commands.create import MissingAppSources
 
+from ...utils import create_file
+
 
 def assert_dist_info(app_path):
     dist_info_path = app_path / "my_app-1.2.3.dist-info"
@@ -123,20 +125,18 @@ def test_source_dir(
     #     shallow.py
     #     submodule /
     #       deeper.py
-    first_src = tmp_path / "project" / "src" / "first" / "demo.py"
-    first_src.parent.mkdir(parents=True)
-    with first_src.open("w") as f:
-        f.write("print('hello first')\n")
-
-    second_shallow_src = tmp_path / "project" / "src" / "second" / "shallow.py"
-    second_deep_src = (
-        tmp_path / "project" / "src" / "second" / "submodule" / "deeper.py"
+    create_file(
+        tmp_path / "project" / "src" / "first" / "demo.py",
+        "print('hello first')\n",
+    )
+    create_file(
+        tmp_path / "project" / "src" / "second" / "shallow.py",
+        "print('hello shallow second')\n",
+    )
+    create_file(
+        tmp_path / "project" / "src" / "second" / "submodule" / "deeper.py",
+        "print('hello deep second')\n",
     )
-    second_deep_src.parent.mkdir(parents=True)
-    with second_shallow_src.open("w") as f:
-        f.write("print('hello shallow second')\n")
-    with second_deep_src.open("w") as f:
-        f.write("print('hello deep second')\n")
 
     # Set the app definition, and install sources
     myapp.sources = ["src/first", "src/second"]
@@ -169,14 +169,14 @@ def test_source_file(
     # src /
     #   demo.py
     # other.py
-    first_src = tmp_path / "project" / "src" / "demo.py"
-    first_src.parent.mkdir(parents=True)
-    with first_src.open("w") as f:
-        f.write("print('hello first')\n")
-
-    second_src = tmp_path / "project" / "other.py"
-    with second_src.open("w") as f:
-        f.write("print('hello second')\n")
+    create_file(
+        tmp_path / "project" / "src" / "demo.py",
+        "print('hello first')\n",
+    )
+    create_file(
+        tmp_path / "project" / "other.py",
+        "print('hello second')\n",
+    )
 
     # Set the app definition, and install sources
     myapp.sources = ["src/demo.py", "other.py"]
@@ -191,6 +191,68 @@ def test_source_file(
     assert_dist_info(app_path)
 
 
+def test_no_existing_app_folder(
+    create_command,
+    myapp,
+    tmp_path,
+    app_path,
+    app_requirements_path_index,
+):
+    """If there's no pre-existing app folder, one is created."""
+    # Create the mock sources
+    # src /
+    #   first /
+    #     demo.py
+    #   second /
+    #     shallow.py
+    #     submodule /
+    #       deeper.py
+    create_file(
+        tmp_path / "project" / "src" / "first" / "demo.py",
+        "print('hello first')\n",
+    )
+    create_file(
+        tmp_path / "project" / "src" / "second" / "shallow.py",
+        "print('hello shallow second')\n",
+    )
+    create_file(
+        tmp_path / "project" / "src" / "second" / "submodule" / "deeper.py",
+        "print('hello deep second')\n",
+    )
+
+    # Remove the app folder created by the test fixture.
+    shutil.rmtree(app_path)
+
+    # Set the app definition, and install sources
+    myapp.sources = ["src/first/demo.py", "src/second"]
+
+    create_command.install_app_code(myapp)
+
+    # All the new sources exist, and contain the new content.
+    assert (app_path / "demo.py").exists()
+    with (app_path / "demo.py").open() as f:
+        assert f.read() == "print('hello first')\n"
+
+    assert (app_path / "second").exists()
+    assert (app_path / "second" / "shallow.py").exists()
+    with (app_path / "second" / "shallow.py").open() as f:
+        assert f.read() == "print('hello shallow second')\n"
+
+    assert (app_path / "second" / "submodule").exists()
+    assert (app_path / "second" / "submodule" / "deeper.py").exists()
+    with (app_path / "second" / "submodule" / "deeper.py").open() as f:
+        assert f.read() == "print('hello deep second')\n"
+
+    # The stale/broken modules have been removed.
+    assert not (app_path / "stale.py").exists()
+    assert not (app_path / "second" / "stale.py").exists()
+    assert not (app_path / "second" / "broken").exists()
+
+    # Metadata has been updated.
+    assert not (app_path / "my_app-1.2.2.dist-info").exists()
+    assert_dist_info(app_path)
+
+
 def test_replace_sources(
     create_command,
     myapp,
@@ -207,20 +269,18 @@ def test_replace_sources(
     #     shallow.py
     #     submodule /
     #       deeper.py
-    first_src = tmp_path / "project" / "src" / "first" / "demo.py"
-    first_src.parent.mkdir(parents=True)
-    with first_src.open("w") as f:
-        f.write("print('hello first')\n")
-
-    second_shallow_src = tmp_path / "project" / "src" / "second" / "shallow.py"
-    second_deep_src = (
-        tmp_path / "project" / "src" / "second" / "submodule" / "deeper.py"
+    create_file(
+        tmp_path / "project" / "src" / "first" / "demo.py",
+        "print('hello first')\n",
+    )
+    create_file(
+        tmp_path / "project" / "src" / "second" / "shallow.py",
+        "print('hello shallow second')\n",
+    )
+    create_file(
+        tmp_path / "project" / "src" / "second" / "submodule" / "deeper.py",
+        "print('hello deep second')\n",
     )
-    second_deep_src.parent.mkdir(parents=True)
-    with second_shallow_src.open("w") as f:
-        f.write("print('hello shallow second')\n")
-    with second_deep_src.open("w") as f:
-        f.write("print('hello deep second')\n")
 
     # Also create some existing sources:
     # path / to / app /
@@ -234,31 +294,33 @@ def test_replace_sources(
     #     broken /
     #       other.py
     #   my_app-1.2.2.dist-info /
-
-    old_first_src = app_path / "demo.py"
-    with old_first_src.open("w") as f:
-        f.write("print('old hello first')\n")
-
-    old_stale_src = app_path / "stale.py"
-    with old_stale_src.open("w") as f:
-        f.write("print('stale hello first')\n")
+    create_file(
+        app_path / "src" / "demo.py",
+        "print('old hello first')\n",
+    )
+    create_file(
+        app_path / "src" / "stale.py",
+        "print('stale hello first')\n",
+    )
+    create_file(
+        app_path / "src" / "second" / "shallow.py",
+        "print('old hello shallow second')\n",
+    )
+    create_file(
+        app_path / "src" / "second" / "stale.py",
+        "print('hello second stale')\n",
+    )
+    create_file(
+        app_path / "src" / "second" / "submodule" / "deeper.py",
+        "print('hello deep second')\n",
+    )
+    create_file(
+        app_path / "src" / "second" / "broken" / "other.py",
+        "print('hello second deep broken')\n",
+    )
 
     old_dist_info_dir = app_path / "my_app-1.2.2.dist-info"
-    old_second_shallow_src = app_path / "src" / "second" / "shallow.py"
-    old_second_stale_src = app_path / "src" / "second" / "stale.py"
-    old_second_deep_src = app_path / "src" / "second" / "submodule" / "deeper.py"
-    old_second_broken_src = app_path / "src" / "second" / "broken" / "other.py"
     old_dist_info_dir.mkdir()
-    old_second_deep_src.parent.mkdir(parents=True)
-    old_second_broken_src.parent.mkdir(parents=True)
-    with old_second_shallow_src.open("w") as f:
-        f.write("print('old hello shallow second')\n")
-    with old_second_stale_src.open("w") as f:
-        f.write("print('hello second stale')\n")
-    with old_second_deep_src.open("w") as f:
-        f.write("print('old hello deep second')\n")
-    with old_second_broken_src.open("w") as f:
-        f.write("print('hello second deep broken')\n")
 
     # Set the app definition, and install sources
     myapp.sources = ["src/first/demo.py", "src/second"]