Repository: aio-libs/aiohttp

Issue:
Uploading files shouldn't always use chunked, especally when HTTP1.0 is used.
aiohttp (0.9.0 when this bug is opened) always uses 'transfer-encoding': 'chunked' for file uploading, even if `chunked` is set to `False` or `version` is set to `HttpVersion10`.

This can be problematic in some cases. When the server doesn't support chunked multipart/form-data -- which is why I opened this issue -- the client doesn't respond for a very long time and then says `aiohttp.errors.ClientConnectionError: Can not read status line`.

IMO, the `request` method should respect the `chunked` switch, or even prefer the non-chunked way if the source is an actual file (content-length can be determined). I checked Firefox and the requests library, they don't send chunked packets by default.



Discussion:
Setting `Content-Length` in the request headers should help.

`aiohttp` uses this field to determine the actual `Transfer-Encoding`:
1. `chunked` set: Use `chunked`;
2. `chunked` not set and `Content-Length` set: Use `Content-Length`;
3. Neither `chunked` nor `Content-Length` is set: Depends on actual content type; File-like objects have no interface for determining real file size, so `aiohttp` uses `chunked` by default

It's all in the code: `ClientRequest.update_transfer_encoding`

Setting `Content-Length` won't work because before `ClientRequest.update_transfer_encoding` is called, `ClientRequest.update_body_from_data(self, data)` changed the value of `chunked`

``` python3
    if data.is_form_data():
        self.chunked = self.chunked or 8196
```


Resolving patch:
diff --git a/aiohttp/client.py b/aiohttp/client.py
--- a/aiohttp/client.py
+++ b/aiohttp/client.py
@@ -9,6 +9,7 @@
 import weakref
 import warnings
 import chardet
+from os import fstat
 
 import aiohttp
 from . import hdrs, helpers, streams
@@ -383,7 +384,13 @@ def update_body_from_data(self, data):
             assert not isinstance(data, io.StringIO), \
                 'attempt to send text data instead of binary'
             self.body = data
-            self.chunked = True
+            if not self.chunked and isinstance(data, io.BufferedReader):
+                # Not chunking if content-length can be determined
+                size = fstat(data.fileno()).st_size - data.tell()
+                self.headers[hdrs.CONTENT_LENGTH] = str(size)
+                self.chunked = False
+            else:
+                self.chunked = True
             if hasattr(data, 'mode'):
                 if data.mode == 'r':
                     raise ValueError('file {!r} should be open in binary mode'


Test patch:
diff --git a/tests/test_client.py b/tests/test_client.py
--- a/tests/test_client.py
+++ b/tests/test_client.py
@@ -7,6 +7,7 @@
 import unittest
 import unittest.mock
 import urllib.parse
+import os.path
 
 import aiohttp
 from aiohttp.client import ClientRequest, ClientResponse
@@ -542,6 +543,39 @@ def test_chunked_length(self):
         self.assertEqual(req.headers['TRANSFER-ENCODING'], 'chunked')
         self.assertNotIn('CONTENT-LENGTH', req.headers)
 
+    def test_file_upload_not_chunked(self):
+        here = os.path.dirname(__file__)
+        fname = os.path.join(here, 'sample.key')
+        with open(fname, 'rb') as f:
+            req = ClientRequest(
+                'post', 'http://python.org/',
+                data=f)
+            self.assertFalse(req.chunked)
+            self.assertEqual(req.headers['CONTENT-LENGTH'],
+                             str(os.path.getsize(fname)))
+
+    def test_file_upload_not_chunked_seek(self):
+        here = os.path.dirname(__file__)
+        fname = os.path.join(here, 'sample.key')
+        with open(fname, 'rb') as f:
+            f.seek(100)
+            req = ClientRequest(
+                'post', 'http://python.org/',
+                data=f)
+            self.assertEqual(req.headers['CONTENT-LENGTH'],
+                             str(os.path.getsize(fname) - 100))
+
+    def test_file_upload_force_chunked(self):
+        here = os.path.dirname(__file__)
+        fname = os.path.join(here, 'sample.key')
+        with open(fname, 'rb') as f:
+            req = ClientRequest(
+                'post', 'http://python.org/',
+                data=f,
+                chunked=True)
+            self.assertTrue(req.chunked)
+            self.assertNotIn('CONTENT-LENGTH', req.headers)
+
     def test_expect100(self):
         req = ClientRequest('get', 'http://python.org/',
                             expect100=True, loop=self.loop)