Repository: encode/httpx
Pull request: Add support for multiple files per POST field
fixes #777 
inspired by great work done on #891 
separates files and data as per @florimondmanca suggesiton in gitter
I commented below a mypy error I ignored because I'm clueless on how to deal with it to be honest

Problem statement:
Support for multiple files per POST field
```
async def http_upload():
    async with httpx.AsyncClient(verify=False) as client:
        data_path = os.path.join(PARENT_PATH, 'data_file/blob/data')
        index_path = os.path.join(PARENT_PATH, 'data_file/blob/index')
        files = {
            "upload_file": [
                open(data_path, 'rb'),
                open(index_path, 'rb'),
            ]
        }
        # files = [
        #     ("upload_file", open(data_path, 'rb')),
        #     ("upload_file", open(index_path, 'rb'))
        # ]
        params = {
            "requestId": "10001",
            "sdkVersion": "V1.2.3",
            "area": "CH"
        }

        response = await session.post(url, data=params, files=files)
```

```console
self = <httpx.content_streams.MultipartStream.FileField object at 0x7f503c22ea60>

    def render_data(self) -> bytes:
        if isinstance(self.file, str):
            content = self.file
        else:
>           content = self.file.read()
E           AttributeError: 'list' object has no attribute 'read'

or
    def iter_fields(
        self, data: dict, files: dict
    ) -> typing.Iterator[typing.Union["FileField", "DataField"]]:
        for name, value in data.items():
            if isinstance(value, (list, dict)):
                for item in value:
                    yield self.DataField(name=name, value=item)
            else:
                yield self.DataField(name=name, value=value)

>       for name, value in files.items():
E       AttributeError: 'list' object has no attribute 'items'
```

Excuse me, how should I use。

Hints from issue discussion:
You should read the manual https://www.python-httpx.org/advanced/#multipart-file-encoding
Hi, not sure was the problem was exactly, but going to close this as resolved for now. If there's anything unclear/missing in the docs relative to your problem, feel free to ping back here! Thanks.
It seems to me the OP's problem was the same I'm facing right now in that I'm trying to replicate what is described in the `requests` docs here: https://2.python-requests.org/en/master/user/advanced/#post-multiple-multipart-encoded-files
The `httpx` docs @j178 pointed to only seem to describe posting a single file (which works exactly as for `requests`). However, once multiple files need to be uploaded, `requests` stops taking a `dict` and uses a `list` instead, as a `dict` can't use the same key multiple times.
The question now would be how, if at all, this behavior of the `requests` library could be replicated in `httpx`.
> You should read the manual https://www.python-httpx.org/advanced/#multipart-file-encoding

Upload multiple files at once, not one.Can't HTTPX solve it?
> Hi, not sure was the problem was exactly, but going to close this as resolved for now. If there's anything unclear/missing in the docs relative to your problem, feel free to ping back here! Thanks.

Upload multiple files at once, not one.Can't HTTPX solve it?
The problem is not solved, why is it closed?
Just like requests: https://2.python-requests.org/en/master/user/advanced/#post-multiple-multipart-encoded-files
@ysq1222 Reopened, I think it’s clearer now what this is about: uploading multiple files for a given POST field using HTTPX (we only support 1 file per file right now).

That last link is particularly helpful, especially because it shows what the API should look like for such a feature, i.e. a list of file dicts (which is a reasonable API I guess, even though the typing for `files` is already quite involved).

Since it’s supported by Requests I don’t see why we shouldn’t support it. :)

At this point, I imagine any PRs towards resolving this are much welcome? 😄
Edited the issue description to better reflect the need here. 👍
So, I think we also want to support the "either a dict, or a two-tuple of items" for the `data=...` case too.

Ie, this works in requests, but not in httpx...

```python
import requests

url = 'https://httpbin.org/post'
r = requests.post(url, data=[("a", "1"), ("a", "2")])
print(r.text)
```

In both cases we want to change how we iterate over the items in `data`/`files`...

https://github.com/encode/httpx/blob/b3db9ff0b6554ed3a48ff912682f4a7d10c3f3f8/httpx/_content_streams.py#L293

Eg... 

```python
data_items = data.items() if isinstance(data, dict) else data
files_items = files.items() if isinstance(files, dict) else files
for name, value in data_items:
    ...
for name, value in files_items:
    ...
```

> Reopened

Thanks! Great to see quick progress here now :D

> That last link is particularly helpful

You're welcome! Oh. Wait. 😉
> @ysq1222 Reopened, I think it’s clearer now what this is about: uploading multiple files for a given POST field using HTTPX (we only support 1 file per file right now).
> 
> That last link is particularly helpful, especially because it shows what the API should look like for such a feature, i.e. a list of file dicts (which is a reasonable API I guess, even though the typing for `files` is already quite involved).
> 
> Since it’s supported by Requests I don’t see why we shouldn’t support it. :)
> 
> At this point, I imagine any PRs towards resolving this are much welcome?

Will it be supported in the next version?

Commit message: Fixed some docs typos

Patch to review:
diff --git a/docs/advanced.md b/docs/advanced.md
index 76321dd1b2..40055611bd 100644
--- a/docs/advanced.md
+++ b/docs/advanced.md
@@ -464,6 +464,16 @@ MIME header field.
     It is safe to upload large files this way. File uploads are streaming by default, meaning that only one chunk will be loaded into memory at a time.
  
  Non-file data fields can be included in the multipart form using by passing them to `data=...`.
+ 
+You can also send multiple files in one go with a multiple file field form.
+To do that you'll need to pass a list of tuples that looks like `(form_field, (2_or_3_elements_tuple))` where the `2_or_3_elements_tuple` is described above and the `form_field` is the name of your form field.
+For instance this request sends 2 files, `foo.png` and `bar.png` in one request on the `images` form field: 
+
+```python
+>>> multiple_files = [('images', ('foo.png', open('foo.png', 'rb'), 'image/png')),
+                      ('images', ('bar.png', open('bar.png', 'rb'), 'image/png'))]
+>>> r = httpx.post("https://httpbin.org/post", files=multiple_files)
+```
 
 ## Customizing authentication
 
diff --git a/httpx/_content_streams.py b/httpx/_content_streams.py
index 6436be563a..5c58effd2d 100644
--- a/httpx/_content_streams.py
+++ b/httpx/_content_streams.py
@@ -328,7 +328,8 @@ def _iter_fields(
             else:
                 yield self.DataField(name=name, value=value)
 
-        for name, value in files.items():
+        file_items = files.items() if isinstance(files, dict) else files
+        for name, value in file_items:  # type: ignore
             yield self.FileField(name=name, value=value)
 
     def iter_chunks(self) -> typing.Iterator[bytes]:
diff --git a/httpx/_types.py b/httpx/_types.py
index a8925bbe76..a74020a4ae 100644
--- a/httpx/_types.py
+++ b/httpx/_types.py
@@ -72,4 +72,4 @@
     # (filename, file (or text), content_type)
     Tuple[Optional[str], FileContent, Optional[str]],
 ]
-RequestFiles = Mapping[str, FileTypes]
+RequestFiles = Union[Mapping[str, FileTypes], List[Tuple[str, FileTypes]]]
diff --git a/tests/test_content_streams.py b/tests/test_content_streams.py
index c5eb961ddc..2b2adc92ae 100644
--- a/tests/test_content_streams.py
+++ b/tests/test_content_streams.py
@@ -204,3 +204,50 @@ async def test_empty_request():
 def test_invalid_argument():
     with pytest.raises(TypeError):
         encode(123)
+
+
+@pytest.mark.asyncio
+async def test_multipart_multiple_files_single_input_content():
+    files = [
+        ("file", io.BytesIO(b"<file content 1>")),
+        ("file", io.BytesIO(b"<file content 2>")),
+    ]
+    stream = encode(files=files, boundary=b"+++")
+    sync_content = b"".join([part for part in stream])
+    async_content = b"".join([part async for part in stream])
+
+    assert stream.can_replay()
+    assert stream.get_headers() == {
+        "Content-Length": "271",
+        "Content-Type": "multipart/form-data; boundary=+++",
+    }
+    assert sync_content == b"".join(
+        [
+            b"--+++\r\n",
+            b'Content-Disposition: form-data; name="file"; filename="upload"\r\n',
+            b"Content-Type: application/octet-stream\r\n",
+            b"\r\n",
+            b"<file content 1>\r\n",
+            b"--+++\r\n",
+            b'Content-Disposition: form-data; name="file"; filename="upload"\r\n',
+            b"Content-Type: application/octet-stream\r\n",
+            b"\r\n",
+            b"<file content 2>\r\n",
+            b"--+++--\r\n",
+        ]
+    )
+    assert async_content == b"".join(
+        [
+            b"--+++\r\n",
+            b'Content-Disposition: form-data; name="file"; filename="upload"\r\n',
+            b"Content-Type: application/octet-stream\r\n",
+            b"\r\n",
+            b"<file content 1>\r\n",
+            b"--+++\r\n",
+            b'Content-Disposition: form-data; name="file"; filename="upload"\r\n',
+            b"Content-Type: application/octet-stream\r\n",
+            b"\r\n",
+            b"<file content 2>\r\n",
+            b"--+++--\r\n",
+        ]
+    )


Code review comments:
diff_hunk: @@ -464,6 +464,16 @@ MIME header field.
     It is safe to upload large files this way. File uploads are streaming by default, meaning that only one chunk will be loaded into memory at a time.
  
  Non-file data fields can be included in the multipart form using by passing them to `data=...`.
+ 
+You can also send multiple files in one go with a multiple file field form.
+To do that you'll need to pass a list of tuples that looks like `(form_field, (2_or_3_elements_tuple))` where the `2_or_3_elements_tuple` is described above and the `form_field` is the name of your form field.
+For instance this request sends 2 files, `foo.png` and `bar.png` in one request on the `images` form field: 
+
+```python
+>>> multiple_files = [('images', ('foo.png', open('foo.png', 'rb'), 'image/png')),
path: docs/advanced.md
text: @user1:
Should we stick to naming this variable `files` for consistency with the other examples?

@author:
yep, done

@user1:
Sorry, I meant renaming `multiple_files` to `files` (the `images` field name is very nice considering the example deals with sending images).

@author:
too many files ! :) I changed back and updated the one you had in mind !

diff_hunk: @@ -328,7 +328,8 @@ def _iter_fields(
             else:
                 yield self.DataField(name=name, value=value)
 
-        for name, value in files.items():
+        file_items = files.items() if isinstance(files, dict) else files
+        for name, value in file_items:  # type: ignore
path: httpx/_content_streams.py
text: @author:
No idea how to fix the mypy error on this line `httpx/_content_streams.py:332: error: Unpacking a string is disallowed
`

@user1:
I think it’s because the type hints is using Mapping but we’re only matching against dict (a specific type of mapping), so mypy assumes that the other branch could also receive other types of mappings, and iterating over a mapping yields its keys (strings).

It’s actually possible to `isinstance(obj, typing.Mapping)` (since it’s the same Mapping class than in collections.abc), so I think we can go for that?

Alternatively, switch the `if` to match a tuple (and treat the default case as mapping) - probably a bit less surprising.

@author:
fixed !