Repository: scrapy/scrapy

Issue:
inaccurate doc string in utils.python.isbinarytext
It says:
    """Return True if the given text is considered binary, or False
    otherwise, by looking for binary bytes at their chars
    """
However, instead of returning False, a TypeError is raised if text is not bytes.

context:
    Porting responsetypes to Python 3
    on branch tmp-py3, just pulled a few minutes ago
    running
    <tests.test_responsetypes.ResponseTypesTest testMethod=test_from_body>



Discussion:
Yeah, `text` argument name is very confusing. This function checks if data (a bytes object) looks like text or like a binary data. So `text` must be bytes, and the function should raise TypeError, but argument names, docstring and even function name are confusing.

Any suggestions?

It's totally confusing.

How about changing the docstring to:
    """Looks for binary bytes in chars and raises a TypeError if text is not bytes
    """

I wouldn't focus the docstring on raising this TypeError - it is just an expected type of an argument, not the main function purpose. What about this?

```
def isbinarytext(text):
    """Return True if the given ``text`` argument (a ``bytes`` object) 
    contains bytes that are uncommon in textual data.
    """
```

//cc @dangra @eliasdorneles 

Wouldn't `isbinarydata` be a better name?

isbinarydata sounds good to me, it is better than isbinarytext.

I stumbled about the same, while doing this change 691b7f3df73379c04af1140217721aa131b44f26
and had to go confirm it wouldn't just consider it all binary in py3 now.
It's just checking if there are control characters in it or if it is sane to convert it to a string.

So my vote for more clarity, reversing the expectation:

``` py
def binary_is_text(data):
    """Returns True if the given ``data`` argument (a ``bytes`` object) 
    does not contain unprintable control characters.
    """
```

to write

``` py
if binary_is_text(data):
    text = data.decode(encoding)
```

though it breaks the naming convention of `is...`.

I like @nyov's suggestion.

Could `isinstance(text, six.binary_type)` be used instead?

Could `isbinarytext`be renamed `isbinary` or `isbinarytype`?

I also think @nyov's doc string is OK.

> Could isinstance(text, six.binary_type) be used instead?
> Could isbinarytextbe renamed isbinary or isbinarytype?

No, isbinarytext is different. It is not checking the type of argument, it checks its contents. Scrapy receives bytes from network and tries to detect how to decode them; usually there are some clues like HTTP headers, but if conventional methods fail Scrapy uses a heuristic: if contents looks like text (i.e. it doesn't contain control characters) then Scrapy tries to decode it as text. This is what isbinarytext is checking. If used correctly, it should only receive bytes; TypeError is just a sanity check.

This detection method is not perfect, and this function was intended to be internal, an implementation detail.

yeah, I'm seeing that `six.binary_type`is not the same.
I guess my real issue is that `isbinarytext` raises an exception instead of returning False.

If you feed it the correct input, it doesn't raise any exception as @kmike already noted.
You can't feed a complex datatype to a function which expects an integer argument, either, without getting an exception.

I see this now. I was confused about something.
I still think it's a little unclear, but it's not really a bug.


Resolving patch:
diff --git a/scrapy/responsetypes.py b/scrapy/responsetypes.py
--- a/scrapy/responsetypes.py
+++ b/scrapy/responsetypes.py
@@ -10,7 +10,7 @@
 
 from scrapy.http import Response
 from scrapy.utils.misc import load_object
-from scrapy.utils.python import isbinarytext, to_bytes, to_native_str
+from scrapy.utils.python import binary_is_text, to_bytes, to_native_str
 
 
 class ResponseTypes(object):
@@ -94,7 +94,7 @@ def from_body(self, body):
         cannot be guess using more straightforward methods."""
         chunk = body[:5000]
         chunk = to_bytes(chunk)
-        if isbinarytext(chunk):
+        if not binary_is_text(chunk):
             return self.from_mimetype('application/octet-stream')
         elif b"<html>" in chunk.lower():
             return self.from_mimetype('text/html')
diff --git a/scrapy/utils/python.py b/scrapy/utils/python.py
--- a/scrapy/utils/python.py
+++ b/scrapy/utils/python.py
@@ -174,17 +174,25 @@ def new_method(self, *args, **kwargs):
         return cache[self]
     return new_method
 
+
 _BINARYCHARS = {six.b(chr(i)) for i in range(32)} - {b"\0", b"\t", b"\n", b"\r"}
 _BINARYCHARS |= {ord(ch) for ch in _BINARYCHARS}
 
-
+@deprecated("scrapy.utils.python.binary_is_text")
 def isbinarytext(text):
-    """Return True if the given text is considered binary, or False
-    otherwise, by looking for binary bytes at their chars
+    """ This function is deprecated.
+    Please use scrapy.utils.python.binary_is_text, which was created to be more
+    clear about the functions behavior: it is behaving inverted to this one. """
+    return not binary_is_text(text)
+
+
+def binary_is_text(data):
+    """ Returns `True` if the given ``data`` argument (a ``bytes`` object)
+    does not contain unprintable control characters.
     """
-    if not isinstance(text, bytes):
-        raise TypeError("text must be bytes, got '%s'" % type(text).__name__)
-    return any(c in _BINARYCHARS for c in text)
+    if not isinstance(data, bytes):
+        raise TypeError("data must be bytes, got '%s'" % type(data).__name__)
+    return all(c not in _BINARYCHARS for c in data)
 
 
 def get_func_args(func, stripself=False):


Test patch:
diff --git a/tests/test_utils_python.py b/tests/test_utils_python.py
--- a/tests/test_utils_python.py
+++ b/tests/test_utils_python.py
@@ -5,7 +5,7 @@
 import six
 
 from scrapy.utils.python import (
-    memoizemethod_noargs, isbinarytext, equal_attributes,
+    memoizemethod_noargs, binary_is_text, equal_attributes,
     WeakKeyCache, stringify_dict, get_func_args, to_bytes, to_unicode,
     without_none_values)
 
@@ -71,18 +71,18 @@ def noncached(self):
         assert one is not three
 
 
-class IsBinaryTextTest(unittest.TestCase):
-    def test_isbinarytext(self):
-        assert not isbinarytext(b"hello")
+class BinaryIsTextTest(unittest.TestCase):
+    def test_binaryistext(self):
+        assert binary_is_text(b"hello")
 
     def test_utf_16_strings_contain_null_bytes(self):
-        assert not isbinarytext(u"hello".encode('utf-16'))
+        assert binary_is_text(u"hello".encode('utf-16'))
 
     def test_one_with_encoding(self):
-        assert not isbinarytext(b"<div>Price \xa3</div>")
+        assert binary_is_text(b"<div>Price \xa3</div>")
 
     def test_real_binary_bytes(self):
-        assert isbinarytext(b"\x02\xa3")
+        assert not binary_is_text(b"\x02\xa3")