Repository: twisted/twisted

Issue:
twisted.web.Resource.putChild works unexpectedly with Unicode path
|[<img alt="derwolfe's avatar" src="https://avatars.githubusercontent.com/u/641357?s=50" width="50" height="50">](https://github.com/derwolfe)| @derwolfe reported|
|-|-|
|Trac ID|trac#8985|
|Type|defect|
|Created|2017-01-10 19:17:12Z|

IResource [https://github.com/twisted/twisted/blob/twisted-16.5.0/src/twisted/web/resource.py#L61-L71] states that the `path` param should be `bytes`. However, if a user passes in a unicode string, no warning is displayed, nor is any exception thrown. On python3, this is especially problematic, as new users might be following the documentation how-to for twisted web [https://twistedmatrix.com/documents/current/web/howto/using-twistedweb.html#resource-trees] and not understand why their unicode path isn't returning the resource that they would expect.

Currently, Resource's implementation [https://github.com/twisted/twisted/blob/twisted-16.5.0/src/twisted/web/resource.py#L209-L220] of `putChild` happily accepts unicode or bytes. 

For example of how this fails, the following twisted site will return `hi` when curl'ed (`curl http://localhost:8888`) and running on python2, but will return a 404 when run with python3.

```
from twisted.internet import reactor
from twisted.web.server import Site
from twisted.web.resource import Resource


class Page(Resource):

    def render_GET(self, request):
        return b"hi"


if __name__ == '__main__':

    root = Resource()
    root.putChild("", Page()) # on python3, "" should be b""

    factory = Site(root)

    reactor.listenTCP(8888, factory)
    reactor.run()
```



<details><summary>Searchable metadata</summary>

```
trac-id__8985 8985
type__defect defect
reporter__derwolfe derwolfe
priority__normal normal
milestone__None None
branch__ 
branch_author__ 
status__new new
resolution__None None
component__web web
keywords__None None
time__1484075832128825 1484075832128825
changetime__1511669305479914 1511669305479914
version__None None
owner__rodrigc rodrigc
cc__derwolfe
```
</details>



Discussion:
|[<img alt="glyph's avatar" src="https://avatars.githubusercontent.com/u/716529?s=50" width="50" height="50">](https://github.com/glyph)<a name="note_15"></a>|@glyph set owner to @rodrigc|
|-|-|

I am a little confused about the purpose of this PR.

It seems to be relaxing a number of bytes/unicode confusions (good) to allow for more idiomatic passing of `str` or `bytes` through to various methods (also good).  But it also appears to put a lot of effort into going _backwards_ on using functions like `networkString` and `intToBytes` and instead hard-coding `isinstance` and `.encode("ascii")` calls (bad).

Granted, `networkString` may be insufficiently expressive, or the wrong thing, in many of these cases, but there is lots and lots of repetition of the pattern

```python
def foo(bar):
    if isinstance(bar, unicode):
        bar = bar.encode("ascii")
```

This is a pattern which propagates type confusion.  If the caller really _should_ be passing `unicode` in the future, then we want a chokepoint that warns about `bytes` eventually; if the caller really should be continuing to pass bytes but we need a convenient way to catch errors during py3 migrations without blowing up, then we need a chokepoint that warns about `unicode`.  Neither is possible if the only way to find this pattern is an AST-grep across the whole codebase.

Also, there appear to be lots of places that change `intToBytes` into `str(x).encode("ascii")` which drops some implicit typechecking that was previously present and bloats the size of the branch considerably.

It does look like this has a few desirable properties and I wish I could approve it as-is, but I'm just too vague on the exact nature of the changes at the moment.  So, my review feedback, mainly to make future reviews more tractable, is:

1. If you can, please minimize this branch to only the necessary changes, to make it easier to review (getting rid of any unnecessary stylistic changes).  It would be a lot easier to review the stylistic stuff in isolation.
2. Please write up exactly what semantic changes this makes in some clear news files; this is way too many type-related changes for just a `.misc`.
3. Please write compat functions which express the semantic desire of both the `if isinstance(..., unicode): ....encode("ascii")` and `if not isinstance(name, bytes): raise TypeError` patterns, and give each a docstring, so it's easier to review at the individual call sites if it's appropriate, and provide some levers for longer-term shepherding of application code towards the correct types.

Thanks!
|[<img alt="derwolfe's avatar" src="https://avatars.githubusercontent.com/u/641357?s=50" width="50" height="50">](https://github.com/derwolfe)<a name="note_3"></a>|@derwolfe commented|
|-|-|

PR available at [https://github.com/twisted/twisted/pull/671]
|[<img alt="wsanchez's avatar" src="https://avatars.githubusercontent.com/u/50002?s=50" width="50" height="50">](https://github.com/wsanchez)<a name="note_10"></a>|@wsanchez commented|
|-|-|

Right, sorry I wasn't more clear, but yes, I think the internal representation for `path` would ideally be unicode, and if we are given bytes as input, we should assume either ASCII or UTF-8; probably the latter.

Thanks.
|[<img alt="wsanchez's avatar" src="https://avatars.githubusercontent.com/u/50002?s=50" width="50" height="50">](https://github.com/wsanchez)<a name="note_5"></a>|@wsanchez commented|
|-|-|

Wouldn't it be better if we actually made unicode work as expected?  Is that not feasible?

It seems weird that we are requiring in API that a URL path component, which is clearly *text* must be encoded as *data* (bytes).  Especially with our own `URL` class is adamant that one must use unicode.

|[<img alt="wsanchez's avatar" src="https://avatars.githubusercontent.com/u/50002?s=50" width="50" height="50">](https://github.com/wsanchez)<a name="note_6"></a>|@wsanchez commented|
|-|-|

I don't know how others feel about the compat argument here, but I'd prefer if we converted bytes to unicode if given bytes, assuming utf-8.

The compat argument in favor of this is that anything other than ASCII bytes is already forces us to assume an encoding at some point.
|[<img alt="derwolfe's avatar" src="https://avatars.githubusercontent.com/u/641357?s=50" width="50" height="50">](https://github.com/derwolfe)<a name="note_7"></a>|@derwolfe commented|
|-|-|

> Wouldn't it be better if we actually made unicode work as expected? Is that not feasible?

I hadn't thought about that initially, but I think it would be better to make unicode work as expected. I started with the bytes-only version mainly because it was the simplest patch I could think of that would provide an obvious error message to someone new to the interface. 

> It seems weird that we are requiring in API that a URL path component, which is clearly *text* must be encoded as *data* (bytes). Especially with our own URL class is adamant that one must use unicode.

I completely agree. If we're treating URL paths as text elsewhere, this seems like a reasonable place to have the same expectation.

Would it be reasonable to

1. expand `t.w.IResource.putChild` to accept both text and bytes for `path`
2. if the path is bytes, just use it.
3. if the path is unicode, encode it as utf-8, save the path as bytes.

A further benefit to making text paths acceptable is that the current how-to docs wouldn't need to change. If the bytes-only patch as currently proposed were merged, we might want to modify the how-to documentation to make it clear that bytes _must_ be used.
|[<img alt="derwolfe's avatar" src="https://avatars.githubusercontent.com/u/641357?s=50" width="50" height="50">](https://github.com/derwolfe)<a name="note_9"></a>|@derwolfe commented|
|-|-|

wsanchez, on rereading your comment, I see that you were suggesting we store the {{ path }} as {{ text }} and not {{ bytes }}. I'll work on a patch that does this. 

Thanks!
|[<img alt="mithrandi's avatar" src="https://avatars.githubusercontent.com/u/856430?s=50" width="50" height="50">](https://github.com/mithrandi)<a name="note_12"></a>|@mithrandi commented|
|-|-|

[#9135](https://github.com/twisted/twisted/issues/9135) was sort of a duplicate of this, but since this ticket has expanded to a wider scope, I'm keeping both.
|[<img alt="markrwilliams's avatar" src="https://avatars.githubusercontent.com/u/704247?s=50" width="50" height="50">](https://github.com/markrwilliams)<a name="note_13"></a>|@markrwilliams commented|
|-|-|

https://github.com/twisted/twisted/pull/911#pullrequestreview-70044476

Resolving patch:
diff --git a/src/twisted/web/distrib.py b/src/twisted/web/distrib.py
--- a/src/twisted/web/distrib.py
+++ b/src/twisted/web/distrib.py
@@ -10,9 +10,8 @@
 """
 
 import copy
-
-# System Imports
 import os
+import sys
 
 try:
     import pwd
@@ -24,8 +23,6 @@
 from twisted.internet import address, reactor
 from twisted.logger import Logger
 from twisted.persisted import styles
-
-# Twisted Imports
 from twisted.spread import pb
 from twisted.spread.banana import SIZE_LIMIT
 from twisted.web import http, resource, server, static, util
@@ -355,18 +352,20 @@ def render_GET(self, request):
         return htmlDoc.encode("utf-8")
 
     def getChild(self, name, request):
-        if name == "":
+        if name == b"":
             return self
 
-        td = ".twistd"
+        td = b".twistd"
 
-        if name[-len(td) :] == td:
+        if name.endswith(td):
             username = name[: -len(td)]
             sub = 1
         else:
             username = name
             sub = 0
         try:
+            # Decode using the filesystem encoding to reverse a transformation
+            # done in the pwd module.
             (
                 pw_name,
                 pw_passwd,
@@ -375,7 +374,7 @@ def getChild(self, name, request):
                 pw_gecos,
                 pw_dir,
                 pw_shell,
-            ) = self._pwd.getpwnam(username)
+            ) = self._pwd.getpwnam(username.decode(sys.getfilesystemencoding()))
         except KeyError:
             return resource.NoResource()
         if sub:
diff --git a/src/twisted/web/resource.py b/src/twisted/web/resource.py
--- a/src/twisted/web/resource.py
+++ b/src/twisted/web/resource.py
@@ -59,16 +59,18 @@ def getChildWithDefault(name, request):
         @type request: L{twisted.web.server.Request}
         """
 
-    def putChild(path, child):
+    def putChild(path: bytes, child: "IResource") -> None:
         """
-        Put a child IResource implementor at the given path.
+        Put a child L{IResource} implementor at the given path.
 
         @param path: A single path component, to be interpreted relative to the
             path this resource is found at, at which to put the given child.
             For example, if resource A can be found at I{http://example.com/foo}
             then a call like C{A.putChild(b"bar", B)} will make resource B
             available at I{http://example.com/foo/bar}.
-        @type path: C{bytes}
+
+            The path component is I{not} URL-encoded -- pass C{b'foo bar'}
+            rather than C{b'foo%20bar'}.
         """
 
     def render(request):
@@ -103,7 +105,7 @@ class Resource:
     """
     Define a web-accessible resource.
 
-    This serves 2 main purposes; one is to provide a standard representation
+    This serves two main purposes: one is to provide a standard representation
     for what HTTP specification calls an 'entity', and the other is to provide
     an abstract directory structure for URL retrieval.
     """
@@ -199,12 +201,17 @@ def getChildWithDefault(self, path, request):
         return self.getChild(path, request)
 
     def getChildForRequest(self, request):
+        """
+        Deprecated in favor of L{getChildForRequest}.
+
+        @see: L{twisted.web.resource.getChildForRequest}.
+        """
         warnings.warn(
             "Please use module level getChildForRequest.", DeprecationWarning, 2
         )
         return getChildForRequest(self, request)
 
-    def putChild(self, path, child):
+    def putChild(self, path: bytes, child: IResource) -> None:
         """
         Register a static child.
 
@@ -213,24 +220,18 @@ def putChild(self, path, child):
         path to be ''.
 
         @param path: A single path component.
-        @type path: L{bytes}
 
         @param child: The child resource to register.
-        @type child: L{IResource}
 
         @see: L{IResource.putChild}
         """
         if not isinstance(path, bytes):
-            warnings.warn(
-                "Path segment must be bytes; "
-                "passing {} has never worked, and "
-                "will raise an exception in the future.".format(type(path)),
-                category=DeprecationWarning,
-                stacklevel=2,
-            )
+            raise TypeError(f"Path segment must be bytes, but {path!r} is {type(path)}")
 
         self.children[path] = child
-        child.server = self.server
+        # IResource is incomplete and doesn't mention this server attribute, see
+        # https://github.com/twisted/twisted/issues/11717
+        child.server = self.server  # type: ignore[attr-defined]
 
     def render(self, request):
         """


Test patch:
diff --git a/src/twisted/web/test/requesthelper.py b/src/twisted/web/test/requesthelper.py
--- a/src/twisted/web/test/requesthelper.py
+++ b/src/twisted/web/test/requesthelper.py
@@ -5,11 +5,12 @@
 Helpers related to HTTP requests, used by tests.
 """
 
+from __future__ import annotations
 
 __all__ = ["DummyChannel", "DummyRequest"]
 
 from io import BytesIO
-from typing import Optional
+from typing import Dict, List, Optional
 
 from zope.interface import implementer, verify
 
@@ -206,6 +207,11 @@ class DummyRequest:
     uri = b"http://dummy/"
     method = b"GET"
     client: Optional[IAddress] = None
+    sitepath: List[bytes]
+    written: List[bytes]
+    prepath: List[bytes]
+    args: Dict[bytes, List[bytes]]
+    _finishedDeferreds: List[Deferred[None]]
 
     def registerProducer(self, prod, s):
         """
@@ -225,7 +231,12 @@ def registerProducer(self, prod, s):
     def unregisterProducer(self):
         self.go = 0
 
-    def __init__(self, postpath, session=None, client=None):
+    def __init__(
+        self,
+        postpath: list[bytes],
+        session: Optional[Session] = None,
+        client: Optional[IAddress] = None,
+    ) -> None:
         self.sitepath = []
         self.written = []
         self.finished = 0
diff --git a/src/twisted/web/test/test_distrib.py b/src/twisted/web/test/test_distrib.py
--- a/src/twisted/web/test/test_distrib.py
+++ b/src/twisted/web/test/test_distrib.py
@@ -29,7 +29,7 @@
 from twisted.web import client, distrib, resource, server, static
 from twisted.web.http_headers import Headers
 from twisted.web.test._util import _render
-from twisted.web.test.test_web import DummyChannel, DummyRequest
+from twisted.web.test.requesthelper import DummyChannel, DummyRequest
 
 
 class MySite(server.Site):
@@ -395,7 +395,7 @@ def test_interface(self):
         """
         self.assertTrue(verifyObject(resource.IResource, self.directory))
 
-    def _404Test(self, name):
+    async def _404Test(self, name: bytes) -> None:
         """
         Verify that requesting the C{name} child of C{self.directory} results
         in a 404 response.
@@ -403,28 +403,24 @@ def _404Test(self, name):
         request = DummyRequest([name])
         result = self.directory.getChild(name, request)
         d = _render(result, request)
+        await d
+        self.assertEqual(request.responseCode, 404)
 
-        def cbRendered(ignored):
-            self.assertEqual(request.responseCode, 404)
-
-        d.addCallback(cbRendered)
-        return d
-
-    def test_getInvalidUser(self):
+    async def test_getInvalidUser(self):
         """
         L{UserDirectory.getChild} returns a resource which renders a 404
         response when passed a string which does not correspond to any known
         user.
         """
-        return self._404Test("carol")
+        await self._404Test(b"carol")
 
-    def test_getUserWithoutResource(self):
+    async def test_getUserWithoutResource(self):
         """
         L{UserDirectory.getChild} returns a resource which renders a 404
         response when passed a string which corresponds to a known user who has
         neither a user directory nor a user distrib socket.
         """
-        return self._404Test("alice")
+        await self._404Test(b"alice")
 
     def test_getPublicHTMLChild(self):
         """
@@ -436,7 +432,7 @@ def test_getPublicHTMLChild(self):
         public_html = home.child("public_html")
         public_html.makedirs()
         request = DummyRequest(["bob"])
-        result = self.directory.getChild("bob", request)
+        result = self.directory.getChild(b"bob", request)
         self.assertIsInstance(result, static.File)
         self.assertEqual(result.path, public_html.path)
 
@@ -450,7 +446,7 @@ def test_getDistribChild(self):
         home.makedirs()
         web = home.child(".twistd-web-pb")
         request = DummyRequest(["bob"])
-        result = self.directory.getChild("bob.twistd", request)
+        result = self.directory.getChild(b"bob.twistd", request)
         self.assertIsInstance(result, distrib.ResourceSubscription)
         self.assertEqual(result.host, "unix")
         self.assertEqual(abspath(result.port), web.path)
diff --git a/src/twisted/web/test/test_resource.py b/src/twisted/web/test/test_resource.py
--- a/src/twisted/web/test/test_resource.py
+++ b/src/twisted/web/test/test_resource.py
@@ -182,19 +182,8 @@ def test_staticChildPathType(self):
         resource = Resource()
         child = Resource()
         sibling = Resource()
-        resource.putChild("foo", child)
-        warnings = self.flushWarnings([self.test_staticChildPathType])
-        self.assertEqual(len(warnings), 1)
-        self.assertIn("Path segment must be bytes", warnings[0]["message"])
-        # We expect an error here because "foo" != b"foo" on Python 3+
-        self.assertIsInstance(
-            resource.getChildWithDefault(b"foo", DummyRequest([])), ErrorPage
-        )
-
-        resource.putChild(None, sibling)
-        warnings = self.flushWarnings([self.test_staticChildPathType])
-        self.assertEqual(len(warnings), 1)
-        self.assertIn("Path segment must be bytes", warnings[0]["message"])
+        self.assertRaises(TypeError, resource.putChild, "foo", child)
+        self.assertRaises(TypeError, resource.putChild, None, sibling)
 
     def test_defaultHEAD(self):
         """
diff --git a/src/twisted/web/test/test_template.py b/src/twisted/web/test/test_template.py
--- a/src/twisted/web/test/test_template.py
+++ b/src/twisted/web/test/test_template.py
@@ -761,7 +761,7 @@ class DummyRenderRequest(DummyRequest):  # type: ignore[misc]
     """
 
     def __init__(self) -> None:
-        super().__init__([""])
+        super().__init__([b""])
         self.site = FakeSite()