Repository: scrapy/scrapy
Pull request: [MRG+1] Add flush() method to StreamLogger
Fixes GH-2125

Local tests:

with a pretty simple spider:

```
$ cat stdoutlogging/spiders/example.py 
# -*- coding: utf-8 -*-
import scrapy


class ExampleSpider(scrapy.Spider):
    name = "example"
    allowed_domains = ["example.com"]
    start_urls = (
        'http://www.example.com/',
    )

    def parse(self, response):
        print("hello!")
        return {"somekey": "somevalue"}
```

and Python 3.5:

```
$ scrapy version -v
Scrapy    : 1.2.0dev2
lxml      : 3.6.0.0
libxml2   : 2.9.3
Twisted   : 16.4.0
Python    : 3.5.1+ (default, Mar 30 2016, 22:46:26) - [GCC 5.3.1 20160330]
pyOpenSSL : 16.1.0 (OpenSSL 1.0.2g-fips  1 Mar 2016)
Platform  : Linux-4.4.0-36-generic-x86_64-with-Ubuntu-16.04-xenial
```

Before this change:

```
$ scrapy crawl example -s LOG_STDOUT=1
2016-09-14 12:07:35 [scrapy] INFO: Scrapy 1.2.0dev2 started (bot: stdoutlogging)
2016-09-14 12:07:35 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'stdoutlogging.spiders', 'BOT_NAME': 'stdoutlogging', 'LOG_STDOUT': '1', 'SPIDER_MODULES': ['stdoutlogging.spiders'], 'ROBOTSTXT_OBEY': True}
(...)
2016-09-14 12:07:36 [scrapy] INFO: Spider opened
2016-09-14 12:07:36 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2016-09-14 12:07:36 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023
2016-09-14 12:07:36 [scrapy] DEBUG: Crawled (404) <GET http://www.example.com/robots.txt> (referer: None)
2016-09-14 12:07:36 [scrapy] DEBUG: Crawled (200) <GET http://www.example.com/> (referer: None)
2016-09-14 12:07:36 [stdout] INFO: hello!
2016-09-14 12:07:36 [scrapy] DEBUG: Scraped from <200 http://www.example.com/>
{'somekey': 'somevalue'}
2016-09-14 12:07:36 [scrapy] INFO: Closing spider (finished)
(...stats...)
2016-09-14 12:07:36 [scrapy] INFO: Spider closed (finished)
Exception ignored in: <scrapy.utils.log.StreamLogger object at 0x7f298eb66358>
AttributeError: 'StreamLogger' object has no attribute 'flush'
$
```

After the change:

```
$ scrapy crawl example -s LOG_STDOUT=1
2016-09-14 12:11:33 [scrapy] INFO: Scrapy 1.2.0dev2 started (bot: stdoutlogging)
2016-09-14 12:11:33 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'stdoutlogging.spiders', 'SPIDER_MODULES': ['stdoutlogging.spiders'], 'BOT_NAME': 'stdoutlogging', 'ROBOTSTXT_OBEY': True, 'LOG_STDOUT': '1'}
(...)
2016-09-14 12:11:33 [scrapy] INFO: Spider opened
2016-09-14 12:11:33 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2016-09-14 12:11:33 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023
2016-09-14 12:11:39 [scrapy] DEBUG: Crawled (404) <GET http://www.example.com/robots.txt> (referer: None)
2016-09-14 12:11:39 [scrapy] DEBUG: Crawled (200) <GET http://www.example.com/> (referer: None)
2016-09-14 12:11:39 [stdout] INFO: hello!
2016-09-14 12:11:39 [scrapy] DEBUG: Scraped from <200 http://www.example.com/>
{'somekey': 'somevalue'}
2016-09-14 12:11:39 [scrapy] INFO: Closing spider (finished)
(...stats...)
2016-09-14 12:11:39 [scrapy] INFO: Spider closed (finished)
$
```

Note: I was not able to write a failing-then-passing test for this change in `test_utils_log.StreamLoggerTest`.
I tried with [`testfixtures.OutputCapture`](https://pythonhosted.org/testfixtures/api.html#testfixtures.OutputCapture) but could not capture the stderr output


Problem statement:
scrapy.utils.log.StreamLogger doesn't seem to have flush method - breaks Python3
Might be necessary to add the flush method to StreamLogger class, to make this work with Python3 and `LOG_STDOUT = True`:

```
  File "<frozen importlib._bootstrap>", line 986, in _gcd_import

  File "<frozen importlib._bootstrap>", line 969, in _find_and_load

  File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked

  File "<frozen importlib._bootstrap>", line 673, in _load_unlocked

  File "<frozen importlib._bootstrap_external>", line 658, in exec_module

  File "<frozen importlib._bootstrap_external>", line 764, in get_code

  File "<frozen importlib._bootstrap_external>", line 724, in source_to_code

  File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed

builtins.SyntaxError: invalid syntax (pipelines.py, line 19)
Exception ignored in: <scrapy.utils.log.StreamLogger object at 0x10fddd978>
AttributeError: 'StreamLogger' object has no attribute 'flush'
```

Hints from issue discussion:
Exception ignored in: <scrapy.utils.log.StreamLogger object at 0x7fcd38612f60>
AttributeError: 'StreamLogger' object has no attribute 'flush'
also have this problem on python 3

Commit message: Add flush() method to StreamLogger

Fixes GH-2125

Patch to review:
diff --git a/scrapy/utils/log.py b/scrapy/utils/log.py
index cc2f0b164fa..a28002c08e5 100644
--- a/scrapy/utils/log.py
+++ b/scrapy/utils/log.py
@@ -145,6 +145,9 @@ def write(self, buf):
         for line in buf.rstrip().splitlines():
             self.logger.log(self.log_level, line.rstrip())
 
+    def flush(self):
+        pass
+
 
 class LogCounterHandler(logging.Handler):
     """Record log levels count into a crawler stats"""


Code review comments:
diff_hunk: @@ -145,6 +145,9 @@ def write(self, buf):
         for line in buf.rstrip().splitlines():
             self.logger.log(self.log_level, line.rstrip())
 
+    def flush(self):
+        pass
path: scrapy/utils/log.py
text: @user1:
Should it flush logger handlers? 

http://stackoverflow.com/a/18012843/114795 suggests it can be done using
`[h.flush() for h in my_logger.handlerList]`. 


@author:
Makes sense. Updated.