Repository: encode/httpx

Issue:
Feature Request: support iterating stream lines
I think it is better to implement a api for iterating stream line by line. This scene is very common (e.g., kubernetes watch api). Now we need to create a wrapper for this.

```Python
import json
import httpx


class StreamWrapper(object):
    def __init__(self, stream):
        self._stream = stream

    def __iter__(self):
        pending = ""

        for chunk in self._stream:
            chunk = pending + chunk
            lines = chunk.splitlines()

            if chunk and lines and lines[-1] and lines[-1][-1] == chunk[-1]:
                pending = lines.pop()
            else:
                pending = ""

            for line in lines:
                yield line

        if pending:
            yield pending


timeout = httpx.TimeoutConfig(
    connect_timeout=5, read_timeout=None, write_timeout=5
)


resp = httpx.get(
    "http://127.0.0.1:18081/api/v1/watch/namespaces/default/pods",
    stream=True,
    timeout=timeout,
)

for chunk in StreamWrapper(resp.stream_text()):
    print(json.loads(chunk))
```


Discussion:
(For your immediate need, is it an option to use [the official Kubernetes Python client](https://github.com/kubernetes-client/python)?)

Note that your wrapping algorithm is accidentally quadratic. Say your stream gives you N bytes, one byte at a time, with only a newline at the end: you'll be calling splitlines() N times, which is undesired because splitlines is O(n) too.

It so happens that transforming a stream into lines is tricky to get right! While the kubernetes watch API is well behaved, your wrapper can't be included as is in httpx.

It also happens that this kind of line splitting is a generic problem that needs to be solved in different places. We're still trying to figure out as a community how to extract this kind of functionality in reusable and composable parts, see https://github.com/python-trio/trio/issues/796 and https://bugs.python.org/issue38242 for detail. In the meantime, even if I don't speak for httpx maintainers, I don't think adding this functionality in httpx itself would be a good move.
Thanks for sharing this snippet @Hanaasagi. I think you’ll understand though that this particular piece of functionality is out of scope for HTTPX, as supported by the reasons @pquentin pointed out.

Edit: see below for nuancing.
Woops, clicked the wrong button. Responding to issues on a phone can be tricky!

I was saying: parsing the stream iterator into a stream of lines might be something our users want to do, though. Your use case may not be isolated.

As noted in https://github.com/python-trio/trio/issues/796 though, most async libraries expose a way to do this. For example, StreamReader.readline() in asyncio.

So we *could* in theory add a new .readline() method to our concurrency backends, accept “lines” (the string) along with True/False on the stream parameter, and then use that method instead of .read() to yield lines instead of raw chunks in the .stream_text() iterator.

I’m really not sure whether this would work well with protocol layer though, and also not sure this is something we want to support inside HTTPX at all.

One immediate sensible point of action out of this would be to document our handling of this use case. 
Related to #24, #183. I'm fairly sure myself and @sethmlarson had some more conversation about this too, but I can't dig it out right now?

Not sure what we want the API to look like, but I'd agree in general with the composable implementation (take a stream iterator, return a linewise iterator)

Resolving patch:
diff --git a/httpx/decoders.py b/httpx/decoders.py
--- a/httpx/decoders.py
+++ b/httpx/decoders.py
@@ -211,6 +211,67 @@ def _detector_result(self) -> str:
         return result
 
 
+class LineDecoder:
+    """
+    Handles incrementally reading lines from text.
+
+    Uses universal line decoding, supporting any of `\n`, `\r`, or `\r\n`
+    as line endings, normalizing to `\n`.
+    """
+
+    def __init__(self) -> None:
+        self.buffer = ""
+
+    def decode(self, text: str) -> typing.List[str]:
+        lines = []
+
+        if text.startswith("\n") and self.buffer and self.buffer[-1] == "\r":
+            # Handle the case where we have an "\r\n" split across
+            # our previous input, and our new chunk.
+            lines.append(self.buffer[:-1] + "\n")
+            self.buffer = ""
+            text = text[1:]
+
+        while text:
+            num_chars = len(text)
+            for idx in range(num_chars):
+                char = text[idx]
+                next_char = None if idx + 1 == num_chars else text[idx + 1]
+                if char == "\n":
+                    lines.append(self.buffer + text[: idx + 1])
+                    self.buffer = ""
+                    text = text[idx + 1 :]
+                    break
+                elif char == "\r" and next_char == "\n":
+                    lines.append(self.buffer + text[:idx] + "\n")
+                    self.buffer = ""
+                    text = text[idx + 2 :]
+                    break
+                elif char == "\r" and next_char is not None:
+                    lines.append(self.buffer + text[:idx] + "\n")
+                    self.buffer = ""
+                    text = text[idx + 1 :]
+                    break
+                elif next_char is None:
+                    self.buffer = text
+                    text = ""
+                    break
+
+        return lines
+
+    def flush(self) -> typing.List[str]:
+        if self.buffer.endswith("\r"):
+            # Handle the case where we had a trailing '\r', which could have
+            # been a '\r\n' pair.
+            lines = [self.buffer[:-1] + "\n"]
+        elif self.buffer:
+            lines = [self.buffer]
+        else:
+            lines = []
+        self.buffer = ""
+        return lines
+
+
 SUPPORTED_DECODERS = {
     "identity": IdentityDecoder,
     "gzip": GZipDecoder,
diff --git a/httpx/models.py b/httpx/models.py
--- a/httpx/models.py
+++ b/httpx/models.py
@@ -17,6 +17,7 @@
     SUPPORTED_DECODERS,
     Decoder,
     IdentityDecoder,
+    LineDecoder,
     MultiDecoder,
     TextDecoder,
 )
@@ -934,6 +935,14 @@ async def stream_text(self) -> typing.AsyncIterator[str]:
             yield decoder.decode(chunk)
         yield decoder.flush()
 
+    async def stream_lines(self) -> typing.AsyncIterator[str]:
+        decoder = LineDecoder()
+        async for text in self.stream_text():
+            for line in decoder.decode(text):
+                yield line
+        for line in decoder.flush():
+            yield line
+
     async def raw(self) -> typing.AsyncIterator[bytes]:
         """
         A byte-iterator over the raw response content.


Test patch:
diff --git a/tests/models/test_responses.py b/tests/models/test_responses.py
--- a/tests/models/test_responses.py
+++ b/tests/models/test_responses.py
@@ -163,6 +163,18 @@ async def test_stream_text():
     assert content == "Hello, world!"
 
 
+@pytest.mark.asyncio
+async def test_stream_lines():
+    response = httpx.Response(200, content=b"Hello,\nworld!")
+
+    await response.read()
+
+    content = []
+    async for line in response.stream_lines():
+        content.append(line)
+    assert content == ["Hello,\n", "world!"]
+
+
 @pytest.mark.asyncio
 async def test_stream_interface_after_read():
     response = httpx.Response(200, content=b"Hello, world!")
diff --git a/tests/test_decoders.py b/tests/test_decoders.py
--- a/tests/test_decoders.py
+++ b/tests/test_decoders.py
@@ -9,6 +9,7 @@
     DeflateDecoder,
     GZipDecoder,
     IdentityDecoder,
+    LineDecoder,
     TextDecoder,
 )
 
@@ -167,6 +168,48 @@ def test_text_decoder_empty_cases():
     assert decoder.flush() == ""
 
 
+def test_line_decoder_nl():
+    decoder = LineDecoder()
+    assert decoder.decode("") == []
+    assert decoder.decode("a\n\nb\nc") == ["a\n", "\n", "b\n"]
+    assert decoder.flush() == ["c"]
+
+    decoder = LineDecoder()
+    assert decoder.decode("") == []
+    assert decoder.decode("a\n\nb\nc\n") == ["a\n", "\n", "b\n", "c\n"]
+    assert decoder.flush() == []
+
+
+def test_line_decoder_cr():
+    decoder = LineDecoder()
+    assert decoder.decode("") == []
+    assert decoder.decode("a\r\rb\rc") == ["a\n", "\n", "b\n"]
+    assert decoder.flush() == ["c"]
+
+    decoder = LineDecoder()
+    assert decoder.decode("") == []
+    assert decoder.decode("a\r\rb\rc\r") == ["a\n", "\n", "b\n"]
+    assert decoder.flush() == ["c\n"]
+
+
+def test_line_decoder_crnl():
+    decoder = LineDecoder()
+    assert decoder.decode("") == []
+    assert decoder.decode("a\r\n\r\nb\r\nc") == ["a\n", "\n", "b\n"]
+    assert decoder.flush() == ["c"]
+
+    decoder = LineDecoder()
+    assert decoder.decode("") == []
+    assert decoder.decode("a\r\n\r\nb\r\nc\r\n") == ["a\n", "\n", "b\n", "c\n"]
+    assert decoder.flush() == []
+
+    decoder = LineDecoder()
+    assert decoder.decode("") == []
+    assert decoder.decode("a\r") == []
+    assert decoder.decode("\n\r\nb\r\nc") == ["a\n", "\n", "b\n"]
+    assert decoder.flush() == ["c"]
+
+
 def test_invalid_content_encoding_header():
     headers = [(b"Content-Encoding", b"invalid-header")]
     body = b"test 123"