Repository: encode/httpx

Issue:
detect length of file-like object?
### Checklist

<!-- Please make sure you check all these items before submitting your feature request. -->

- [x] There are no similar issues or pull requests for this yet.
- [x] I discussed this idea on the [community chat](https://gitter.im/encode/community) and feedback is positive.

### Is your feature related to a problem? Please describe.

I'm using `httpx` to upload files to a WebDAV server, which uses `PUT`. The problem here is that not a lot of server has chunked uploading support for PUT. 

When uploading a file-like object, `request` tries to detect file size to send it in `Content-Length` header using [`super_len()`](https://github.com/psf/requests/blob/8c211a96cdbe9fe320d63d9e1ae15c5c07e179f8/requests/utils.py#L108). But `httpx` uses `Transfer-Encoding: chunked`. So, the files end up empty or the server throws a `411 Length Required` response.

The difference in behaviour with respect to `requests` can be demonstrated as follows:

```python
>>> import requests, httpx
>>> from io import BytesIO

>>> requests.put("https://httpbin.org/anything", data=BytesIO(b"foobar")).request.headers
 {'User-Agent': 'python-requests/2.25.1', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-Length': '6'}

>>> httpx.put("https://httpbin.org/anything", data=BytesIO(b"foobar")).request.headers
Headers({'host': 'httpbin.org', 'accept': '*/*', 'accept-encoding': 'gzip, deflate', 'connection': 'keep-alive', 'user-agent': 'python-httpx/0.16.1', 'transfer-encoding': 'chunked'})
```

## Describe the solution you would like.
It'd be great to have the same behaviour as `requests` here, i.e. trying to detect the size and then falling back to chunked encoding. 

## Describe alternatives you considered

As a workaround, I can use [`peek_filelike_length`](https://github.com/encode/httpx/blob/0f280af8b170ed5cc48c12a894f71a8b5762f748/httpx/_utils.py#L347) and a custom header `Content-Length` which is respected, that works well for me.

## Additional context

We had a long discussion in https://github.com/iterative/dvc/issues/4796 regarding it. `nextcloud-snap` that I use locally for testing, does not have support this either due to a bug in Apache (or, some modules in it): https://github.com/nextcloud/nextcloud-snap/issues/365. 

And, Sabredav (`owncloud`/`nextcloud` uses them internally) [wiki](https://sabre.io/dav/clients/finder/) has a good insight on this (though the article is >6 years old):

> Finder uses Transfer-Encoding: Chunked in PUT request bodies. This is a little used HTTP feature, and therefore not implemented in a bunch of web servers. The only server I've seen so far that handles this reasonably well is Apache + mod_php. Nginx and Lighttpd respond with 411 Length Required, which is completely ignored by Finder. This was seen on nginx 0.7.63. It was recently reported that a development release (1.3.8) no longer had this issue.
>
>When using this with Apache + FastCGI PHP completely drops the request body, so it will seem as if the PUT request was successful, but the file will end up empty.

Note that the recent servers do support chunked encoding, but we still find a lot of old servers or are using certain modules that are buggy wrt chunked encoding.

---

I understand that it's the server that is not http/1.1 compliant. And, if `httpx` does not want to support, I'm okay with the workaround as well.  But I'd happy to make a PR if we decide to do so. Thanks. 🙂 




Discussion:
Okay, so it looks to me like there's a more sensible behaviour that we could be using for binary data, where we attempt to peek the file(like) length, in the same way that we currently do for multipart data.

Currently we're always treating binary data as having an unknown length, and using chunked encoding for it...

https://github.com/encode/httpx/blob/24a55d73c31475f3d17ee46580810bc30efc369b/httpx/_content.py#L77-L103

We probably ought to instead be attempting to use our `peek_filelike_length` function, and only using chunked encoding when we're unable to obtain the expected length... https://github.com/encode/httpx/blob/24a55d73c31475f3d17ee46580810bc30efc369b/httpx/_utils.py#L347

@tomchristie, thanks. I'll try to come up with a PR. I just wanted to hear if that was a good idea that `httpx` want to have. 🙂 
Okay, I think there's two distinctly separate PRs for us to tackle here...

Firstly, a bit of refactoring so that `peek_filelike_length` returns `None` when it can't obtain a length instead of raising an exception. Referenced in the multipart implementation, here... https://github.com/encode/httpx/blob/24a55d73c31475f3d17ee46580810bc30efc369b/httpx/_multipart.py#L94-L100

Then *once* that pull request is done, we can follow up with a pull request using `peek_filelike_length` inside `encode_content`.

One other thing we'll want to be careful about here is what behaviour we end up with if files opened in text mode are used. There's a note in requests about disallowing these in the future. (Which makes sense for us)


Resolving patch:
diff --git a/httpx/_content.py b/httpx/_content.py
--- a/httpx/_content.py
+++ b/httpx/_content.py
@@ -17,7 +17,7 @@
 from ._multipart import MultipartStream
 from ._transports.base import AsyncByteStream, SyncByteStream
 from ._types import RequestContent, RequestData, RequestFiles, ResponseContent
-from ._utils import primitive_value_to_str
+from ._utils import peek_filelike_length, primitive_value_to_str
 
 
 class ByteStream(AsyncByteStream, SyncByteStream):
@@ -82,12 +82,17 @@ def encode_content(
 
     if isinstance(content, (bytes, str)):
         body = content.encode("utf-8") if isinstance(content, str) else content
-        content_length = str(len(body))
-        headers = {"Content-Length": content_length} if body else {}
+        content_length = len(body)
+        headers = {"Content-Length": str(content_length)} if body else {}
         return headers, ByteStream(body)
 
     elif isinstance(content, Iterable):
-        headers = {"Transfer-Encoding": "chunked"}
+        content_length_or_none = peek_filelike_length(content)
+
+        if content_length_or_none is None:
+            headers = {"Transfer-Encoding": "chunked"}
+        else:
+            headers = {"Content-Length": str(content_length_or_none)}
         return headers, IteratorByteStream(content)  # type: ignore
 
     elif isinstance(content, AsyncIterable):
diff --git a/httpx/_multipart.py b/httpx/_multipart.py
--- a/httpx/_multipart.py
+++ b/httpx/_multipart.py
@@ -93,9 +93,8 @@ def get_length(self) -> int:
             return len(headers) + len(to_bytes(self.file))
 
         # Let's do our best not to read `file` into memory.
-        try:
-            file_length = peek_filelike_length(self.file)
-        except OSError:
+        file_length = peek_filelike_length(self.file)
+        if file_length is None:
             # As a last resort, read file and cache contents for later.
             assert not hasattr(self, "_data")
             self._data = to_bytes(self.file.read())
diff --git a/httpx/_utils.py b/httpx/_utils.py
--- a/httpx/_utils.py
+++ b/httpx/_utils.py
@@ -342,7 +342,7 @@ def guess_content_type(filename: typing.Optional[str]) -> typing.Optional[str]:
     return None
 
 
-def peek_filelike_length(stream: typing.IO) -> int:
+def peek_filelike_length(stream: typing.Any) -> typing.Optional[int]:
     """
     Given a file-like stream object, return its length in number of bytes
     without reading it into memory.
@@ -350,7 +350,9 @@ def peek_filelike_length(stream: typing.IO) -> int:
     try:
         # Is it an actual file?
         fd = stream.fileno()
-    except OSError:
+        # Yup, seems to be an actual file.
+        length = os.fstat(fd).st_size
+    except (AttributeError, OSError):
         # No... Maybe it's something that supports random access, like `io.BytesIO`?
         try:
             # Assuming so, go to end of stream to figure out its length,
@@ -358,14 +360,11 @@ def peek_filelike_length(stream: typing.IO) -> int:
             offset = stream.tell()
             length = stream.seek(0, os.SEEK_END)
             stream.seek(offset)
-        except OSError:
+        except (AttributeError, OSError):
             # Not even that? Sorry, we're doomed...
-            raise
-        else:
-            return length
-    else:
-        # Yup, seems to be an actual file.
-        return os.fstat(fd).st_size
+            return None
+
+    return length
 
 
 class Timer:


Test patch:
diff --git a/tests/test_content.py b/tests/test_content.py
--- a/tests/test_content.py
+++ b/tests/test_content.py
@@ -48,6 +48,18 @@ async def test_bytes_content():
     assert async_content == b"Hello, world!"
 
 
+@pytest.mark.asyncio
+async def test_bytesio_content():
+    headers, stream = encode_request(content=io.BytesIO(b"Hello, world!"))
+    assert isinstance(stream, typing.Iterable)
+    assert not isinstance(stream, typing.AsyncIterable)
+
+    content = b"".join([part for part in stream])
+
+    assert headers == {"Content-Length": "13"}
+    assert content == b"Hello, world!"
+
+
 @pytest.mark.asyncio
 async def test_iterator_content():
     def hello_world():