Repository: tornadoweb/tornado

Issue:
how to upload files with AsyncHTTPClient
if using requests, codes like:

data = {...}
addr = '...'
with file( ... ) as f:
     response = requests.post( addr, data = data, files = { 'file': f } )

please show me the async code using AsyncHTTPClient.



Discussion:
The purpose of the issue tracker is to report bugs or feature requests for Tornado. Questions like this are better suited for the mailing list or stack overflow.

there are same questions already in stackoverflow. but no answer.
if this feature is not exist. regard this as a feature request.

I have the same problem, all I want is as following but I am not sure how to do that :  
self._http_client = AsyncHTTPClient()
self._http_client.fetch(self._server_url, \
                    method = "POST", \
                    request_timeout = 10, \
                    files = {"f1": "./data/1.txt", "f2":"./data/2.bin"}, \   ### upload files to HttpServer
                    body = json.dumps(params, ensure_ascii = False), \
                    callback = self.post_process) 
            #ioloop will run continually 
            tornado.ioloop.IOLoop.instance().start()

This is a plausible example of uploading a file with AsyncHTTPClient:

https://github.com/bryndin/tornado-flickr-api/blob/master/tornado_flickrapi/multipart.py

I agree it could be a useful feature for AsyncHTTPClient to support file upload more easily. Multipart form encoding is not in the Python standard library so each HTTP implementation has to implement its own eventually.

_Or_, since file upload with AsyncHTTPClient is easily implemented without any changes to Tornado itself, this would make a useful little standalone package.

Omm, Thank you for your kindly reply.  Indeed I use the way as your link to send a file instead of AsyncHTTPClient.

Here's an example. Run this file-receiver server in one terminal window:

```
import tornado.ioloop
import tornado.web


class MainHandler(tornado.web.RequestHandler):
    def post(self):
        for k, (info,) in self.request.files.items():
            name, content_type = info['filename'], info['content_type']
            body = info['body']
            print('%s: %s %s %d bytes' % (k, name, content_type, len(body)))

        self.write('OK')


def make_app():
    return tornado.web.Application([
        (r"/", MainHandler),
    ])


if __name__ == "__main__":
    app = make_app()
    app.listen(8888)
    tornado.ioloop.IOLoop.current().start()

```

Create two files in the local directory named "foo" and "bar" with any contents, and run this file in another terminal window (I've tried to make it compatible with Python 2.7 or Python 3):

```
import mimetypes
from functools import partial
from uuid import uuid4

from tornado import gen, httpclient, ioloop


@gen.coroutine
def body_producer(boundary, filenames, write):
    boundary_bytes = boundary.encode()

    for filename in filenames:
        filename_bytes = filename.encode()
        write(b'--%s\r\n' % (boundary_bytes,))
        write(b'Content-Disposition: form-data; name="%s"; filename="%s"\r\n' %
              (filename_bytes, filename_bytes))

        mtype = mimetypes.guess_type(filename)[0] or 'application/octet-stream'
        write(b'Content-Type: %s\r\n' % (mtype.encode(),))
        write(b'\r\n')
        with open(filename, 'rb') as f:
            while True:
                # 16k at a time.
                chunk = f.read(16 * 1024)
                if not chunk:
                    break
                write(chunk)

                # Let the IOLoop process its event queue.
                yield gen.moment

        write(b'\r\n')
        yield gen.moment

    write(b'--%s--\r\n' % (boundary_bytes,))


@gen.coroutine
def post(client, url, filenames):
    boundary = uuid4().hex
    headers = {'Content-Type': 'multipart/form-data; boundary=%s' % boundary}
    producer = partial(body_producer, boundary, filenames)
    response = yield client.fetch(url,
                                  method='POST',
                                  headers=headers,
                                  body_producer=producer)

    print(response)


ioloop.IOLoop.current().run_sync(lambda: post(httpclient.AsyncHTTPClient(),
                                              'http://localhost:8888/',
                                              ('foo.txt', 'bar')))
```

@bdarnell what's a good next step on this ticket? Shall I add this to `examples/`?

@saintthor I just updated my example to use a streaming upload, and to set each file's mime-type correctly.

that is perfect.

Yeah, I think checking this in as an example is a good idea. We might even want two versions of the file-upload example: one like this with the `multipart/form-data` wrapper, and one that uploads the raw file with a PUT (and therefore can use streaming requests on the server side). We probably want to recommend the PUT version to users who are not constrained by backwards compatibility with HTML forms. 


Resolving patch:
diff --git a/demos/file_upload/file_receiver.py b/demos/file_upload/file_receiver.py
new file mode 100644
--- /dev/null
+++ b/demos/file_upload/file_receiver.py
@@ -0,0 +1,63 @@
+#!/usr/bin/env python
+
+"""Usage: python file_receiver.py
+
+Demonstrates a server that receives a multipart-form-encoded set of files in an
+HTTP POST, or streams in the raw data of a single file in an HTTP PUT.
+
+See file_uploader.py in this directory for code that uploads files in this format.
+"""
+
+import logging
+
+try:
+    from urllib.parse import unquote
+except ImportError:
+    # Python 2.
+    from urllib import unquote
+
+import tornado.ioloop
+import tornado.web
+from tornado import options
+
+
+class POSTHandler(tornado.web.RequestHandler):
+    def post(self):
+        for field_name, files in self.request.files.items():
+            for info in files:
+                filename, content_type = info['filename'], info['content_type']
+                body = info['body']
+                logging.info('POST "%s" "%s" %d bytes',
+                             filename, content_type, len(body))
+
+        self.write('OK')
+
+
+@tornado.web.stream_request_body
+class PUTHandler(tornado.web.RequestHandler):
+    def initialize(self):
+        self.bytes_read = 0
+
+    def data_received(self, chunk):
+        self.bytes_read += len(chunk)
+
+    def put(self, filename):
+        filename = unquote(filename)
+        mtype = self.request.headers.get('Content-Type')
+        logging.info('PUT "%s" "%s" %d bytes', filename, mtype, self.bytes_read)
+        self.write('OK')
+
+
+def make_app():
+    return tornado.web.Application([
+        (r"/post", POSTHandler),
+        (r"/(.*)", PUTHandler),
+    ])
+
+
+if __name__ == "__main__":
+    # Tornado configures logging.
+    options.parse_command_line()
+    app = make_app()
+    app.listen(8888)
+    tornado.ioloop.IOLoop.current().start()
diff --git a/demos/file_upload/file_uploader.py b/demos/file_upload/file_uploader.py
new file mode 100644
--- /dev/null
+++ b/demos/file_upload/file_uploader.py
@@ -0,0 +1,114 @@
+#!/usr/bin/env python
+
+"""Usage: python file_uploader.py [--put] file1.txt file2.png ...
+
+Demonstrates uploading files to a server, without concurrency. It can either
+POST a multipart-form-encoded request containing one or more files, or PUT a
+single file without encoding.
+
+See also file_receiver.py in this directory, a server that receives uploads.
+"""
+
+import mimetypes
+import os
+import sys
+from functools import partial
+from uuid import uuid4
+
+try:
+    from urllib.parse import quote
+except ImportError:
+    # Python 2.
+    from urllib import quote
+
+from tornado import gen, httpclient, ioloop
+from tornado.options import define, options
+
+
+# Using HTTP POST, upload one or more files in a single multipart-form-encoded
+# request.
+@gen.coroutine
+def multipart_producer(boundary, filenames, write):
+    boundary_bytes = boundary.encode()
+
+    for filename in filenames:
+        filename_bytes = filename.encode()
+        write(b'--%s\r\n' % (boundary_bytes,))
+        write(b'Content-Disposition: form-data; name="%s"; filename="%s"\r\n' %
+              (filename_bytes, filename_bytes))
+
+        mtype = mimetypes.guess_type(filename)[0] or 'application/octet-stream'
+        write(b'Content-Type: %s\r\n' % (mtype.encode(),))
+        write(b'\r\n')
+        with open(filename, 'rb') as f:
+            while True:
+                # 16k at a time.
+                chunk = f.read(16 * 1024)
+                if not chunk:
+                    break
+                write(chunk)
+
+                # Let the IOLoop process its event queue.
+                yield gen.moment
+
+        write(b'\r\n')
+        yield gen.moment
+
+    write(b'--%s--\r\n' % (boundary_bytes,))
+
+
+# Using HTTP PUT, upload one raw file. This is preferred for large files since
+# the server can stream the data instead of buffering it entirely in memory.
+@gen.coroutine
+def post(filenames):
+    client = httpclient.AsyncHTTPClient()
+    boundary = uuid4().hex
+    headers = {'Content-Type': 'multipart/form-data; boundary=%s' % boundary}
+    producer = partial(multipart_producer, boundary, filenames)
+    response = yield client.fetch('http://localhost:8888/post',
+                                  method='POST',
+                                  headers=headers,
+                                  body_producer=producer)
+
+    print(response)
+
+
+@gen.coroutine
+def raw_producer(filename, write):
+    with open(filename, 'rb') as f:
+        while True:
+            # 16K at a time.
+            chunk = f.read(16 * 1024)
+            if not chunk:
+                # Complete.
+                break
+
+            write(chunk)
+
+
+@gen.coroutine
+def put(filenames):
+    client = httpclient.AsyncHTTPClient()
+    for filename in filenames:
+        mtype = mimetypes.guess_type(filename)[0] or 'application/octet-stream'
+        headers = {'Content-Type': mtype}
+        producer = partial(raw_producer, filename)
+        url_path = quote(os.path.basename(filename))
+        response = yield client.fetch('http://localhost:8888/%s' % url_path,
+                                      method='PUT',
+                                      headers=headers,
+                                      body_producer=producer)
+    
+        print(response)
+
+
+define("put", type=bool, help="Use PUT instead of POST", group="file uploader")
+
+# Tornado configures logging from command line opts and returns remaining args.
+filenames = options.parse_command_line()
+if not filenames:
+    print("Provide a list of filenames to upload.", file=sys.stderr)
+    sys.exit(1)
+
+method = put if options.put else post
+ioloop.IOLoop.current().run_sync(lambda: method(filenames))
diff --git a/tornado/web.py b/tornado/web.py
--- a/tornado/web.py
+++ b/tornado/web.py
@@ -1670,6 +1670,9 @@ def stream_request_body(cls):
     There is a subtle interaction between ``data_received`` and asynchronous
     ``prepare``: The first call to ``data_received`` may occur at any point
     after the call to ``prepare`` has returned *or yielded*.
+
+    See the `file receiver demo <https://github.com/tornadoweb/tornado/tree/master/demos/file_upload/>`_
+    for example usage.
     """
     if not issubclass(cls, RequestHandler):
         raise TypeError("expected subclass of RequestHandler, got %r", cls)


Test patch: