Repository: scrapy/scrapy

Issue:
get_func_args maximum recursion
https://github.com/scrapy/scrapy/blob/master/scrapy/utils/python.py#L149

Today I was working on a project were I have to skip the first item of a list, and then join the rest. Instead of writing the typical slice I tried something much more good looking `Compose(itemgetter(slice(1, None)), Join())` but I found out this maximum recursion. I did some research and ask @dangra about it, but nothing came up.
I think the main problem is that `inspect` isn't able recognize `itemgetter` as `something`.

``` python
>>> inspect.getmembers(itemgetter(2))
[('__call__',
  <method-wrapper '__call__' of operator.itemgetter object at 0x7f79aeffb990>),
 ('__class__', <type 'operator.itemgetter'>),
 ('__delattr__',
  <method-wrapper '__delattr__' of operator.itemgetter object at 0x7f79aeffb990>),
 ('__doc__',
  'itemgetter(item, ...) --> itemgetter object\n\nReturn a callable object that fetches the given item(s) from its operand.\nAfter, f=itemgetter(2), the call f(r) returns r[2].\nAfter, g=itemgetter(2,5,3), the call g(r) returns (r[2], r[5], r[3])'),
 ('__format__',
  <built-in method __format__ of operator.itemgetter object at 0x7f79aeffb990>),
 ('__getattribute__',
  <method-wrapper '__getattribute__' of operator.itemgetter object at 0x7f79aeffb990>),
 ('__hash__',
  <method-wrapper '__hash__' of operator.itemgetter object at 0x7f79aeffb990>),
 ('__init__',
  <method-wrapper '__init__' of operator.itemgetter object at 0x7f79aeffb990>),
 ('__new__', <built-in method __new__ of type object at 0x8c1ec0>),
 ('__reduce__',
  <built-in method __reduce__ of operator.itemgetter object at 0x7f79aeffb990>),
 ('__reduce_ex__',
  <built-in method __reduce_ex__ of operator.itemgetter object at 0x7f79aeffb990>),
 ('__repr__',
  <method-wrapper '__repr__' of operator.itemgetter object at 0x7f79aeffb990>),
 ('__setattr__',
  <method-wrapper '__setattr__' of operator.itemgetter object at 0x7f79aeffb990>),
 ('__sizeof__',
  <built-in method __sizeof__ of operator.itemgetter object at 0x7f79aeffb990>),
 ('__str__',
  <method-wrapper '__str__' of operator.itemgetter object at 0x7f79aeffb990>),
 ('__subclasshook__',
  <built-in method __subclasshook__ of type object at 0x8c1ec0>)]
>>> inspect.getargspec(itemgetter(2).__call__)
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/usr/lib/python2.7/inspect.py", line 815, in getargspec
    raise TypeError('{!r} is not a Python function'.format(func))
TypeError: <method-wrapper '__call__' of operator.itemgetter object at 0xb3ddd0> is not a Python function
>>> inspect.getargspec(itemgetter(slice(None, 2)).__init__)
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/usr/lib/python2.7/inspect.py", line 815, in getargspec
    raise TypeError('{!r} is not a Python function'.format(func))
TypeError: <method-wrapper '__init__' of operator.itemgetter object at 0xb3de10> is not a Python function
```

EDIT: Looks like the reason was C functions weren't covered by inspect module until Python 3.4 (http://bugs.python.org/issue17481)



Discussion:
same results for python2 and python3

``` python
>>> ig = itemgetter(0)
>>> [(n, f(ig), f(ig.__call__)) for n, f in vars(inspect).items() if n.startswith('is')]
[('isgenerator', False, False),
 ('isdatadescriptor', False, False),
 ('isframe', False, False),
 ('ismemberdescriptor', False, False),
 ('isabstract', False, False),
 ('isbuiltin', False, False),
 ('isfunction', False, False),
 ('istraceback', False, False),
 ('isgeneratorfunction', False, False),
 ('ismethod', False, False),
 ('iscode', False, False),
 ('isclass', False, False),
 ('ismethoddescriptor', False, False),
 ('isroutine', False, False),
 ('ismodule', False, False),
 ('isgetsetdescriptor', False, False)]
```

at least we can detect it similar to what we did for partial

``` python
>>> isinstance(ig, itemgetter)
True
```

Do we want to fall more than one time on line https://github.com/scrapy/scrapy/blob/master/scrapy/utils/python.py#L166?
I'm thinking that we can just return an empty list the second time we reach that point and avoid the recursion completely. 

> Do we want to fall more than one time on line https://github.com/scrapy/scrapy/blob/master/scrapy/utils/python.py#L166?
> I'm thinking that we can just return an empty list the second time we reach that point and avoid the recursion completely.

+1 I can't imagine a better solution


Resolving patch:
diff --git a/scrapy/utils/python.py b/scrapy/utils/python.py
--- a/scrapy/utils/python.py
+++ b/scrapy/utils/python.py
@@ -147,6 +147,8 @@ def get_func_args(func, stripself=False):
     elif hasattr(func, '__call__'):
         if inspect.isroutine(func):
             return []
+        elif getattr(func, '__name__', None) == '__call__':
+            return []
         else:
             return get_func_args(func.__call__, True)
     else:


Test patch:
diff --git a/scrapy/tests/test_utils_python.py b/scrapy/tests/test_utils_python.py
--- a/scrapy/tests/test_utils_python.py
+++ b/scrapy/tests/test_utils_python.py
@@ -193,6 +193,7 @@ def __call__(self, a, b, c):
         # TODO: how do we fix this to return the actual argument names?
         self.assertEqual(get_func_args(unicode.split), [])
         self.assertEqual(get_func_args(" ".join), [])
+        self.assertEqual(get_func_args(operator.itemgetter(2)), [])
 
 if __name__ == "__main__":
     unittest.main()