Repository: beeware/briefcase

Issue:
Fails if dependencies include .pth files
# Context

As a [Mu Editor](https://github.com/mu-editor/mu) contributor, and having worked a bit on improving its packaging, I joined the sprint today in order to understand how easy/difficult it will be for Mu to be packaged with the newer `briefcase` 0.3 release.

Facts about Mu packaging:
* macOS Application Bundle produced with `briefcase` 0.2.x.
* Windows installer produced with `pynsist`.
* PyPI wheel produced with `wheel` + `setuptools`.
* Also included in Debian, Ubuntu, and Raspbian package repositories. Maybe Fedora, too. Not 100% sure because Linux distribution packaging has been, AFAICT, handled outside of the development tree (or maybe it hasn't had much love, in the last 12 months or so).

Challenges about Mu packaging:
* Having a single source of truth WRT to dependency specification and meta data.
* The current solution is `setuptools` based and all of the information sourced from `setup.py` + `setup.cfg` (from there, we use a non-trivial script to produce the necessary `pynsist` input such that it can do its thing, on Windows).

# The Issue

Packaging Mu on Windows leads to a partially ok Mu installation for various motives. In this issue, I'm focusing on a failure that results when trying to bring up its Python REPL -- it  leads to a crash (Mu's fault) because a particular module fails to import, resulting in an unhandled exception.

Specifics:
* Mu's REPL is based on `qtconsole` that, on Windows, ends up requiring `pywin32`.
* `pywin32` uses `.pth` files to *guide* `site.py` in populating `sys.path` in a specific way.

Bottom line:
* Briefcase packaged Windows applications that depend on `pywin32` fail to import `win32api`, on of its modules.

# Investigation

After a few hints from @freakboy3742 and @dgelessus at Gitter, here's what's going on:

* The Python support package being used is the default one: one of the "embeddable zip file" packages from https://www.python.org/downloads/windows/. (FTR, all if this was explored with Python 3.6.8 64 bit).
* In particular it includes a `pythonXX._pth` that:
  * Makes Python run in [isolated mode](https://docs.python.org/3/using/cmdline.html#interface-options)...
  * ...and prevents [site.py](https://docs.python.org/3/library/site.html) from being loaded...
  * ...per [these docs here](https://docs.python.org/3/using/windows.html#finding-modules).
* Such `pythonXX._pth` file is actually overwritten by `briefcase` in order to:
  * Add the `src\app` and `src\app_packages` to `sys.path` such that both the application and its dependencies can be successfully imported.
* However, the presence of the `._pth` file:
  * Prevents the `site` module from being loaded at startup...
  * ...which would be responsible for populating `sys.path` from any `.pth` files that are found on the `site-packages` directories.

# A Successful Hack

With this information, I invested some time fiddling with these things to see if, at least, I could make it work. Here's what I did that resulted in a working Mu REPL (thus, its underlying `import win32api` working!):

* Hacked the cookiecutter template's `briefcase.toml` file (took me quite figuring out where this file was coming from!!!):
  * Set `app_packages_path` to `src\python\lib\site-packages`, instead.
* Then, hacked `briefcase`'s overwriting of `pythonXX._pth` to produce:

  ```
  pythonXX.zip
  .
  lib\site-packages
  import site
  ..\app
  ```
  * This lets Python find the Standard Library with the first 2 lines...
  * ...find application dependencies with the 3rd line...
  * ...has `site` imported such that `.pth` files in the application dependencies are handled...
  * ...lastly adds the application package path such that it can be imported and run.
* Lastly, I observed that having `site` imported lead to an over-populated, non-safe `sys.path`. For some reason, my local Python installation's `site-packages` was being added, and then maybe some more.
* With that, the last step of the hack, was creating a `sitecustomize.py`, which is automatically picked up when `site` is imported per the revamped `pythonXX._pth`. Here's my take:
  ```
  import re
  import sys

  _IS_LOCAL = re.compile(r'(src\\python|src\\app)', re.IGNORECASE)

  sys.path = [path for path in sys.path if _IS_LOCAL.search(path)]
  ```

With these three changes, the `import win32api` in Mu succeeds and, ultimately Mu works as intended WRT to providing a Python REPL.


# Thoughts

* Handling `.pth` files in dependencies is a must, I'd venture saying.
* My approach is the result of fiddling and exploring and I don't like it very much (but it works!). It feels unnecessarily complicated and, thus, brittle.
* Somewhat orthogonal, but related, maybe having a `venv` to which dependencies are `pip install`ed instead of `pip install --target`ed will be more robust, at least for the platforms where that is feasible. No worries about playing with import PATHs in three distinct places (well, maybe some PATH cleaning up could be in order, to guarantee isolation -- see my `sitecustomize.py`, above).

All in all, I leave this issue here in the hope that some useful discussion comes out of it, while serving as a hack-guide to someone facing a similar failures. I'm not sure yet about what could be a solid, sustainable, simple, future-proof solution.

Shall we discuss? :)

PS: I suppose similar failures will happen on any platform, as long as `.pth` files are brought in by dependencies.

Discussion:

Patch:
diff --git a/changes/2103.feature.rst b/changes/2103.feature.rst
new file mode 100644
index 00000000..561ea064
--- /dev/null
+++ b/changes/2103.feature.rst
@@ -0,0 +1,1 @@
+Cookiecutter filters for escaping XML content and attributes have been added.
diff --git a/changes/2204.removal.rst b/changes/2204.removal.rst
new file mode 100644
index 00000000..d8f14517
--- /dev/null
+++ b/changes/2204.removal.rst
@@ -0,0 +1,1 @@
+The ``app_packages`` folder now occurs *after* the ``app`` folder in the package resolution path on Windows, Linux, macOS and iOS. This will result in subtle changes in behavior if you have packages defined in your ``sources``  that have the same name as packages installed from your ``requires``.
diff --git a/changes/381.bugfix.rst b/changes/381.bugfix.rst
new file mode 100644
index 00000000..59042d48
--- /dev/null
+++ b/changes/381.bugfix.rst
@@ -0,0 +1,1 @@
+``.pth`` files created by packages installed as dependencies are now correctly processed during application startup on macOS, Windows, Linux and iOS.
diff --git a/src/briefcase/integrations/cookiecutter.py b/src/briefcase/integrations/cookiecutter.py
index d691b289..b9ac4540 100644
--- a/src/briefcase/integrations/cookiecutter.py
+++ b/src/briefcase/integrations/cookiecutter.py
@@ -1,6 +1,7 @@
 """Jinja2 extensions."""
 
 import uuid
+from xml.sax.saxutils import escape, quoteattr
 
 from jinja2.ext import Extension
 
@@ -139,7 +140,17 @@ class XMLExtension(Extension):
             """Render value in XML format appropriate for an attribute."""
             return "true" if obj else "false"
 
+        def xml_escape(obj):
+            """Filter to escape characters <, >, &, " and '"""
+            return escape(obj)
+
+        def xml_attr(obj):
+            """ "Filter to quote an XML value appropriately."""
+            return quoteattr(obj)
+
         environment.filters["bool_attr"] = bool_attr
+        environment.filters["xml_escape"] = xml_escape
+        environment.filters["xml_attr"] = xml_attr
 
 
 class UUIDExtension(Extension):


Test patch:
diff --git a/tests/integrations/cookiecutter/test_XMLExtension.py b/tests/integrations/cookiecutter/test_XMLExtension.py
index 54b256a4..7b3199b1 100644
--- a/tests/integrations/cookiecutter/test_XMLExtension.py
+++ b/tests/integrations/cookiecutter/test_XMLExtension.py
@@ -24,3 +24,41 @@ def test_bool_attr(value, expected):
     env.filters = {}
     XMLExtension(env)
     assert env.filters["bool_attr"](value) == expected
+
+
+@pytest.mark.parametrize(
+    "value, expected",
+    [
+        # No special characters
+        ("Hello World", "Hello World"),
+        # Ampersands escaped
+        ("Hello & World", "Hello &amp; World"),
+        # Less than
+        ("Hello < World", "Hello &lt; World"),
+        # Greater than
+        ("Hello > World", "Hello &gt; World"),
+    ],
+)
+def test_xml_escape(value, expected):
+    env = MagicMock()
+    env.filters = {}
+    XMLExtension(env)
+    assert env.filters["xml_escape"](value) == expected
+
+
+@pytest.mark.parametrize(
+    "value, expected",
+    [
+        # A single quote wrapped in double quotes
+        ("Hello ' World", '"Hello \' World"'),
+        # A double quote wrapped in single quotes
+        ('Hello " World', "'Hello \" World'"),
+        # A double quote and a single quote in the same string value
+        ("Hello \" And ' World", '"Hello &quot; And \' World"'),
+    ],
+)
+def test_xml_attr(value, expected):
+    env = MagicMock()
+    env.filters = {}
+    XMLExtension(env)
+    assert env.filters["xml_attr"](value) == expected