Repository: scrapy/scrapy

Issue:
get_func_args() from scrapy.utils.python crashes on partial functions
I use Python 2.7 and used a partial function (`functools.partial`) as part of an input_processor for the ItemLoader. For instance,

``` python
price = Field(input_processor=Compose(some_partial_func, some_other_func))
```

During execution, `get_func_args() from scrapy.util.python` goes into an infinite loop until

``` python
/usr/local/lib/python2.7/dist-packages/scrapy/utils/python.pyc in get_func_args(func, stripself)
    161             return []
    162         else:
--> 163             return get_func_args(func.__call__, True)
    164     else:
    165         raise TypeError('%s is not callable' % type(func))

/usr/local/lib/python2.7/dist-packages/scrapy/utils/python.pyc in get_func_args(func, stripself)
    161             return []
    162         else:
--> 163             return get_func_args(func.__call__, True)
    164     else:
    165         raise TypeError('%s is not callable' % type(func))

/usr/local/lib/python2.7/dist-packages/scrapy/utils/python.pyc in get_func_args(func, stripself)
    151     if inspect.isfunction(func):
    152         func_args, _, _, _ = inspect.getargspec(func)
--> 153     elif inspect.isclass(func):
    154         return get_func_args(func.__init__, True)
    155     elif inspect.ismethod(func):

/usr/lib/python2.7/inspect.pyc in isclass(object)
     63         __doc__         documentation string
     64         __module__      name of module in which this class was defined"""
---> 65     return isinstance(object, (type, types.ClassType))
     66 
     67 def ismethod(object):

RuntimeError: maximum recursion depth exceeded while calling a Python object
```

happens. Looks like `get_func_args` needs to handle partial functions as a separate case.

get_func_args() from scrapy.utils.python crashes on partial functions
I use Python 2.7 and used a partial function (`functools.partial`) as part of an input_processor for the ItemLoader. For instance,

``` python
price = Field(input_processor=Compose(some_partial_func, some_other_func))
```

During execution, `get_func_args() from scrapy.util.python` goes into an infinite loop until

``` python
/usr/local/lib/python2.7/dist-packages/scrapy/utils/python.pyc in get_func_args(func, stripself)
    161             return []
    162         else:
--> 163             return get_func_args(func.__call__, True)
    164     else:
    165         raise TypeError('%s is not callable' % type(func))

/usr/local/lib/python2.7/dist-packages/scrapy/utils/python.pyc in get_func_args(func, stripself)
    161             return []
    162         else:
--> 163             return get_func_args(func.__call__, True)
    164     else:
    165         raise TypeError('%s is not callable' % type(func))

/usr/local/lib/python2.7/dist-packages/scrapy/utils/python.pyc in get_func_args(func, stripself)
    151     if inspect.isfunction(func):
    152         func_args, _, _, _ = inspect.getargspec(func)
--> 153     elif inspect.isclass(func):
    154         return get_func_args(func.__init__, True)
    155     elif inspect.ismethod(func):

/usr/lib/python2.7/inspect.pyc in isclass(object)
     63         __doc__         documentation string
     64         __module__      name of module in which this class was defined"""
---> 65     return isinstance(object, (type, types.ClassType))
     66 
     67 def ismethod(object):

RuntimeError: maximum recursion depth exceeded while calling a Python object
```

happens. Looks like `get_func_args` needs to handle partial functions as a separate case.



Discussion:
Python 3.3 has [inspect.signature](http://hg.python.org/cpython/file/3.3/Lib/inspect.py#l1366) function that works for functools.partial . Also, there is a getargspec version that supports functools.partial in [sphinx](https://bitbucket.org/birkenfeld/sphinx/src/1b7f2df1f87d/sphinx/util/inspect.py#cl-22). Both implementations tries not to return parameters that are already bound by functools.partial.

Alternatively, [sage.misc.sageinspect.sage_getargspec](http://trac.sagemath.org/browser/src/sage/misc/sageinspect.py) just returns args of underlying function.

Main use of get_func_args is in scrapy.loader.common.wrap_loader_context. wrap_loader_context returns functools.partial, so it seems that this function is not called twice on the same object anywhere (because the problem described in this ticket will be much prominent otherwise). This means that even simplest sage's solution should work for scrapy needs in practice.

Another way is to port a more feature-rich solution like sphinx's or Python 3.3's. 

Do you want to give it a try? Pull requests are welcome :)

Thanks for the pointers. I'll take a look at different implementations you mentioned to see which fits best

Python 3.3 has [inspect.signature](http://hg.python.org/cpython/file/3.3/Lib/inspect.py#l1366) function that works for functools.partial . Also, there is a getargspec version that supports functools.partial in [sphinx](https://bitbucket.org/birkenfeld/sphinx/src/1b7f2df1f87d/sphinx/util/inspect.py#cl-22). Both implementations tries not to return parameters that are already bound by functools.partial.

Alternatively, [sage.misc.sageinspect.sage_getargspec](http://trac.sagemath.org/browser/src/sage/misc/sageinspect.py) just returns args of underlying function.

Main use of get_func_args is in scrapy.loader.common.wrap_loader_context. wrap_loader_context returns functools.partial, so it seems that this function is not called twice on the same object anywhere (because the problem described in this ticket will be much prominent otherwise). This means that even simplest sage's solution should work for scrapy needs in practice.

Another way is to port a more feature-rich solution like sphinx's or Python 3.3's. 

Do you want to give it a try? Pull requests are welcome :)

Thanks for the pointers. I'll take a look at different implementations you mentioned to see which fits best


Resolving patch:
diff --git a/scrapy/utils/python.py b/scrapy/utils/python.py
--- a/scrapy/utils/python.py
+++ b/scrapy/utils/python.py
@@ -10,7 +10,7 @@
 import inspect
 import weakref
 import errno
-from functools import wraps
+from functools import partial, wraps
 from sgmllib import SGMLParser
 
 
@@ -156,6 +156,9 @@ def get_func_args(func, stripself=False):
         return get_func_args(func.__func__, True)
     elif inspect.ismethoddescriptor(func):
         return []
+    elif isinstance(func, partial):
+        return [x for x in get_func_args(func.func)[len(func.args):]
+                if not (func.keywords and x in func.keywords)]
     elif hasattr(func, '__call__'):
         if inspect.isroutine(func):
             return []


Test patch:
diff --git a/scrapy/tests/test_contrib_loader.py b/scrapy/tests/test_contrib_loader.py
--- a/scrapy/tests/test_contrib_loader.py
+++ b/scrapy/tests/test_contrib_loader.py
@@ -1,4 +1,5 @@
 import unittest
+from functools import partial
 
 from scrapy.contrib.loader import ItemLoader
 from scrapy.contrib.loader.processor import Join, Identity, TakeFirst, \
@@ -332,6 +333,29 @@ class TestItemLoader(NameItemLoader):
         item = il.load_item()
         self.assertEqual(item['name'], u'Mart')
 
+    def test_partial_processor(self):
+        def join(values, sep=None, loader_context=None, ignored=None):
+            if sep is not None:
+                return sep.join(values)
+            elif loader_context and 'sep' in loader_context:
+                return loader_context['sep'].join(values)
+            else:
+                return ''.join(values)
+
+        class TestItemLoader(NameItemLoader):
+            name_out = Compose(partial(join, sep='+'))
+            url_out = Compose(partial(join, loader_context={'sep': '.'}))
+            summary_out = Compose(partial(join, ignored='foo'))
+
+        il = TestItemLoader()
+        il.add_value('name', [u'rabbit', u'hole'])
+        il.add_value('url', [u'rabbit', u'hole'])
+        il.add_value('summary', [u'rabbit', u'hole'])
+        item = il.load_item()
+        self.assertEqual(item['name'], u'rabbit+hole')
+        self.assertEqual(item['url'], u'rabbit.hole')
+        self.assertEqual(item['summary'], u'rabbithole')
+
 
 class ProcessorsTest(unittest.TestCase):
 
diff --git a/scrapy/tests/test_utils_python.py b/scrapy/tests/test_utils_python.py
--- a/scrapy/tests/test_utils_python.py
+++ b/scrapy/tests/test_utils_python.py
@@ -1,3 +1,4 @@
+import functools
 import operator
 import unittest
 from itertools import count
@@ -64,7 +65,7 @@ def test_isbinarytext(self):
         assert not isbinarytext("hello")
 
         # utf-16 strings contain null bytes
-        assert not isbinarytext(u"hello".encode('utf-16')) 
+        assert not isbinarytext(u"hello".encode('utf-16'))
 
         # one with encoding
         assert not isbinarytext("<div>Price \xa3</div>")
@@ -175,11 +176,17 @@ def __call__(self, a, b, c):
 
         a = A(1, 2, 3)
         cal = Callable()
+        partial_f1 = functools.partial(f1, None)
+        partial_f2 = functools.partial(f1, b=None)
+        partial_f3 = functools.partial(partial_f2, None)
 
         self.assertEqual(get_func_args(f1), ['a', 'b', 'c'])
         self.assertEqual(get_func_args(f2), ['a', 'b', 'c'])
         self.assertEqual(get_func_args(A), ['a', 'b', 'c'])
         self.assertEqual(get_func_args(a.method), ['a', 'b', 'c'])
+        self.assertEqual(get_func_args(partial_f1), ['b', 'c'])
+        self.assertEqual(get_func_args(partial_f2), ['a', 'c'])
+        self.assertEqual(get_func_args(partial_f3), ['c'])
         self.assertEqual(get_func_args(cal), ['a', 'b', 'c'])
         self.assertEqual(get_func_args(object), [])