Repository: rootpy/rootpy

Issue:
maximum recursion depth exceeded
Dear rootpyers,
(adding @taroni and @ekfriis who might be interested)

I just updated rootpy to the following commit:
99aaea62f16e7db8d2215c50831d85d798026db6 and I started getting a bunch of errors when retrieving an histogram from a view

Exception RuntimeError: 'maximum recursion depth exceeded in **subclasscheck**' in <type 'exceptions.RuntimeError'> ignored

the problem persist updating to the current head but was not there in 44eedc1265023bb366324f134584c76d999a1631

Can you help me sorting this thing out?

Thank you



Discussion:
We'll gladly help out, but we need more information. Can you provide minimal code to reproduce?

Very minimal:

``` python
import rootpy.io as io
import rootpy.plotting.views as views

infile = io.open('SOME_FILE.root') #I know it raises a deprecation warning, but for now I want backward compatibility
general_view = views.SubdirectoryView(infile,'SOME_PATH')
histo = titled_view.Get('SOME_HISTO')
```

and i get back
Exception RuntimeError: 'maximum recursion depth exceeded while calling a Python object' in <type 'exceptions.RuntimeError'> ignored

but the histogram is there. It's just very very annoying when plotting lots of histograms

The following code works for me, can you verify it works for you, and if not, can you find ways to modify what I wrote to make it break? I must be missing something.

``` python
from rootpy.testdata import get_file
from rootpy.plotting.views import SubdirectoryView

def test_view():
    f = get_file("test_file.root")
    view = SubdirectoryView(f, "means")
    view.Get("hist1")
```

(note, this uses test data shipped with `rootpy`, so you don't have to modify it at all, please run `test_view` verbatim)

I've seen this before with python 2.6. More info here: http://bugs.python.org/issue5508. We once had the same issue with Tree objects. I'll take a closer look.

Ok,
@pwaller the code you provided me works, I tried several other combinations and I think I narrowed the problem quite a bit.

``` python
import rootpy.io as io
import rootpy.plotting.views as views

infile = io.open('SOME_FILE.root')

general_view = views.SubdirectoryView(infile,'SOME_DIR')
general_view.Get('SOME_HIST') #WORKS

nested_view = views.SubdirectoryView(general_view,'SOME_NESTED_DIR')
nested_view.Get('SOME_HIST') #RAISES ERROR!

general_view = views.SubdirectoryView(infile,'SOME_DIR/SOME_NESTED_DIR')
general_view.Get('SOME_HIST') #RAISES ERROR!
```

So the problem seems in nesting multiple directories, no matter how. Unfortunately I dod not find any test file shipped with rootpy that has histograms deeper than one directory.

Can you reproduce the error?

So, I narrowed down the problem again, and it seems it does not come where I was expecting it. Also please note that the error is slightly different from the one that I saw the first time, I just got a bunch of both and I did not noticed they were slightly different.

```
>>> import rootpy.io as io
>>> import rootpy.plotting.views as views
INFO:rootpy.logger] | No default log handler configured. See `logging` module |
INFO:rootpy.logger] \    To suppress: 'rootpy.log.basic_config_colorized()'   /
WARNING:ROOT.TUnixSystem.SetDisplay] DISPLAY not set, setting it to 128.141.151.159:0.0
>>> 
>>> infile = io.open('Zjets_M50.root')
/afs/hep.wisc.edu/cms/mverzett/vhanalysis_summer13/src/UWHiggs/dependencies/FinalStateAnalysis/recipe/external/src/rootpy/rootpy/io/file.py:248: DeprecationWarning: Use root_open instead; open is deprecated.
  DeprecationWarning)
>>> general_view = views.SubdirectoryView(infile,'em/f2')
>>> histo = general_view.Get('mPt')
Exception RuntimeError: 'maximum recursion depth exceeded while calling a Python object' in <type 'exceptions.RuntimeError'> ignored
>>> 
>>> general_view = views.SubdirectoryView(infile,'em')
>>> histo = general_view.Get('f2/mPt')
Exception RuntimeError: 'maximum recursion depth exceeded while calling a Python object' in <type 'exceptions.RuntimeError'> ignored
>>> 
>>> general_view = views.SubdirectoryView(infile,'em')
>>> histo = general_view.Get('f2/mPt')
Exception RuntimeError: 'maximum recursion depth exceeded while calling a Python object' in <type 'exceptions.RuntimeError'> ignored
>>> 
>>> histo = infile.Get('em/f2/mPt')
Exception RuntimeError: 'maximum recursion depth exceeded while calling a Python object' in <type 'exceptions.RuntimeError'> ignored
>>> 
>>> import ROOT
>>> tfile = ROOT.TFile.Open('Zjets_M50.root')
>>> 
>>> histo = tfile.Get('em/f2/mPt')
>>> 
>>> general_view = views.SubdirectoryView(tfile,'em/f2')
>>> histo = general_view.Get('mPt')
>>> 
```

So it seems that the problem is inside either File or root_open rather than the views

Indeed, I can reproduce this with Python 2.6:

```
$ python -c "from rootpy.testdata import get_file; f=get_file('test_file.root'); print f.means"
Exception RuntimeError: 'maximum recursion depth exceeded while calling a Python object' in <type 'exceptions.RuntimeError'> ignored
Directory('/home/pwaller/.local/src/rootpy/rootpy/testdata/test_file.root:/means')
```

It may be obvious to you, but the problem is located in File:

```
>>> import rootpy.io as io
>>> rootpy_file = io.File('Zjets_M50.root', 'read')
>>> histo = rootpy_file.Get('em/f2/mPt')
Exception RuntimeError: 'maximum recursion depth exceeded while calling a Python object' in <type 'exceptions.RuntimeError'> ignored
```

yup, that's what my line to reproduce shows.

It seems that the problem originates from here:
https://github.com/rootpy/rootpy/blob/master/rootpy/io/file.py#L82

by the way, one questions:
1) why here https://github.com/rootpy/rootpy/blob/master/rootpy/io/file.py#L154 don't you call for QROOT.TFile init method or for super(...).**init** ?

Ok. The problem is that `asrootpy` [violates](https://github.com/rootpy/rootpy/blob/master/rootpy/__init__.py#L125) this [assumption](https://github.com/rootpy/rootpy/blob/master/rootpy/io/file.py#L88) - `hasattr` will call `__getattr__` if the requested attribute doesn't exist in the object's `__dict__`.

In addition, then, the `self._path` access in the `DoesNotExist` results in infinite recursion.

I don't know why it is manifest on 2.6 but not 2.7. An immediate fix might be to replace the condition `hasattr(thing, '_post_init'):` with `"_post_init" in thing.__dict__`.

I debugged this with the following lines:

``` python
from rootpy import log
log["/stack"].showstack()
log["/stack"].warning("Here")
```

Stack trace:

```
WARNING:stack] Here
WARNING:stack] File "<string>", line 1, in <module>

WARNING:stack] File "/user1/pwaller/.local/src/rootpy/rootpy/io/file.py", line 98, in __getattr__
WARNING:stack]   return self.Get(attr)
WARNING:stack] File "/user1/pwaller/.local/src/rootpy/rootpy/io/file.py", line 56, in get
WARNING:stack]   thing = f(self, _name, **kwargs)
WARNING:stack] File "/user1/pwaller/.local/src/rootpy/rootpy/io/file.py", line 130, in Get
WARNING:stack]   x =  asrootpy(thing, **kwargs)
WARNING:stack] File "/user1/pwaller/.local/src/rootpy/rootpy/__init__.py", line 125, in asrootpy
WARNING:stack]   if hasattr(thing, '_post_init'):
WARNING:stack] File "/user1/pwaller/.local/src/rootpy/rootpy/io/file.py", line 98, in __getattr__
WARNING:stack]   return self.Get(attr)
WARNING:stack] File "/user1/pwaller/.local/src/rootpy/rootpy/io/file.py", line 69, in get
WARNING:stack]   (name, self._path))
WARNING:stack] File "/user1/pwaller/.local/src/rootpy/rootpy/io/file.py", line 95, in __getattr__
WARNING:stack]   log["/stack"].warning("Here")
```

@ndawe, I think this whole bit of code needs to be re-thought for reliability.

`__getattr__` should do an immediate check to see if its attr is valid or not and then raise AttributeError if not. This should involve the absolute minimum of code possible. Otherwise you get into hairy, hard to debug situations.

I would like to propose this solution:

``` python
    def __getattr__(self, attr):
        """
        Natural naming support.
        Now you can get an object from a File/Directory with
        myfile.somedir.otherdir.histname

        Must be careful here... if __getattr__ ends up being called
        in Get this can end up in an "infinite" recursion and stack overflow
        """
        available_keys  = self.GetListOfKeys()
        available_names = [available_keys.At(it).ReadObj().GetName() for it in range(0,available_keys.GetSize())]
        if attr in available_names:
            return self.Get(attr)
        return super(_DirectoryBase,self).__getattr__(attr)

```

What do you think?

Modified a copy-paste error

@mverzett, this is better. Why not just `any(attr == k.GetName() for i in self.GetListOfKeys())` instead? This two benefits: `ReadObj` is expensive and actually decompresses the object - so your proposal necessarily causes decompression of all `TObject` in the file, which might be expensive. `any` is lazy, so it will stop iterating the keys when it finds the first match.

Even better would be to do a `attr in set(k.GetName() for k in self.GetListOfKeys())` and then memoize the set so that it is only computed once, but that might also not be so great.

Fair point. Definitely better.

Re-asking the question, so I know that in the future
by the way, one questions:
1) why here https://github.com/rootpy/rootpy/blob/master/rootpy/io/file.py#L154 don't you call for QROOT.TFile init method or for super(...).**init** ?

I do not know - if I had written it, I would have used `super`. @ndawe ?


Resolving patch:
diff --git a/rootpy/io/file.py b/rootpy/io/file.py
--- a/rootpy/io/file.py
+++ b/rootpy/io/file.py
@@ -70,9 +70,8 @@ def get(self, name, **kwargs):
 
 class _DirectoryBase(Object):
     """
-    A mixin (can't stand alone). To be improved.
+    A mixin (can't stand alone).
     """
-
     def walk(self, top=None, class_pattern=None):
         """
         Calls :func:`rootpy.io.utils.walk`.
@@ -81,14 +80,20 @@ def walk(self, top=None, class_pattern=None):
 
     def __getattr__(self, attr):
         """
-        Natural naming support.
-        Now you can get an object from a File/Directory with
-        myfile.somedir.otherdir.histname
+        Natural naming support. Now you can get an object from a
+        File/Directory with::
 
-        Must be careful here... if __getattr__ ends up being called
-        in Get this can end up in an "infinite" recursion and stack overflow
+            myfile.somedir.otherdir.histname
         """
-        return self.Get(attr)
+        # Be careful! If ``__getattr__`` ends up being called again here,
+        # this can end up in an "infinite" recursion and stack overflow.
+
+        # Directly call ROOT's Get() here since ``attr`` must anyway be a valid
+        # identifier (not a path including subdirectories).
+        thing = self.ROOT_base.Get(self, attr)
+        if not thing:
+            raise AttributeError
+        return asrootpy(thing)
 
     def __getitem__(self, name):
 
@@ -148,10 +153,9 @@ class Directory(_DirectoryBase, QROOT.TDirectoryFile):
     """
     Inherits from TDirectory
     """
-
     def __init__(self, name, title, *args, **kwargs):
 
-        ROOT.TDirectoryFile.__init__(self, name, title, *args)
+        ROOT.TDirectoryFile.__init__(self, name, title, *args, **kwargs)
         self._path = name
         self._parent = None
 
@@ -173,10 +177,9 @@ class File(_DirectoryBase, QROOT.TFile):
     >>> f = File(filename, 'read')
 
     """
+    def __init__(self, name, *args, **kwargs):
 
-    def __init__(self, *args, **kwargs):
-
-        ROOT.TFile.__init__(self, *args, **kwargs)
+        ROOT.TFile.__init__(self, name, *args, **kwargs)
         self._path = self.GetName()
         self._parent = self
 
@@ -211,7 +214,7 @@ class TemporaryFile(File, QROOT.TFile):
     def __init__(self, *args, **kwargs):
 
         self.__fd, self.__tmp_path = tempfile.mkstemp(*args, **kwargs)
-        super(TemporaryFile, self).__init__(self.__tmp_path, 'recreate')
+        File.__init__(self, self.__tmp_path, 'recreate')
 
     def Close(self):
 


Test patch: