Repository: scikit-hep/uproot5

Issue:
parsing a struct with 2D arrays
This is a continuation of https://stackoverflow.com/questions/62519766/parsing-a-struct-using-2d-arrays-using-uproot/62520558#62520558 

Jim, 

  Thanks, for the fix on StackExchange.  Here is a small root file along with the macro that generated it so that you can test the automated interpretation.

[2DinStruct.root.gz](https://github.com/scikit-hep/uproot/files/4827751/2DinStruct.root.gz)
[Test2DinStruct.C.gz](https://github.com/scikit-hep/uproot/files/4827753/Test2DinStruct.C.gz)

Ron



Discussion:
Thanks for this thread, which gave me the pointers needed to interpret a more complicated structure.

In case it's useful, here is the function I wrote to transform the branch title into a dType for interpretation:

```
def get_Python_dType_from_root_branch_title(title):
    '''
    Given a root branch's title, find the dType needed to interpret it
    Input:
        title: root branch title
    return: 
        dtype_list: to be used with interpretation = uproot.AsDtype(np.dtype(dtype_list))
    Throws an error if the type in Root isn't one of those already included in the translation
    By Michael Punch... MP 20210117
    '''
    import re

    # Dictionary to translate Root types to Python dTypes
    # Note: if no type included with the name, then assume it's a float
    trans_dict ={"": ">f4", "I": ">i4", "i": ">u4", "F": ">f4", "S": ">i2"}

    dtype_list = []

    leaves = title.split(':')
    for leaf in leaves:
        # Split into variable name + possible array lengths, and field type
        # If there is no field type, leave this empty
        extname,field = leaf.split("/") if '/' in leaf else (leaf,'')
        
        # Take the part of variable name before any possible array lengths as the name
        name = extname.split('[')[0]

        arr_lens = [int(_) for _ in re.findall('\[(\d+?)\]',extname)]

        try:
            pfield = trans_dict[field] # will give keyerror if unknown key found
        except KeyError:
            print("Error: Field type '"+field+"' in Root not yet translated to Python type. Add correct type to 'trans_dict'")

        line = (name,pfield,(arr_lens)) if len(arr_lens) else (name,pfield)

        dtype_list.append(line)
        
    return dtype_list
```
Ping @jpivarski (sorry if that's not the correct procedure), if you think this is useful / needs work ?
I didn't realize that I needed to take an action on this: I had to go back to the StackOverflow post to see that I had asked for the algorithm that you just gave. Sorry about that!

I'll move this issue to Uproot 4 with a to-do item to see if your algorithm can be integrated into [uproot/interpretation/identify.py](https://github.com/scikit-hep/uproot4/blob/main/uproot/interpretation/identify.py).

Thanks for offering the algorithm, and sorry that I didn't recognize it as exactly the thing I'd asked for! (I thought you were just showing something you were using in your analysis, and I didn't think I needed to comment on it.)
I guess I wasn't clear.  Thanks for taking it into account.

Two worries I would have on the routine as-is is that:

- I have only taken a handful of the root data types into account, so it might fail with other datatypes.
  - so, I have 4 of the data types (I guess the most common) from the list of 12 here: https://root.cern/manual/root_classes_data_types_and_global_variables/#machine-independent-data-types 
- I am assuming that root is giving everything as big-endian (which works for me on x86_64 machine to produce the root files, and for this uproot reading)
  - But, I suppose this is okay, "It also is stored in machine independent format (ASCII, IEEE floating point, Big Endian byte ordering)" from the root docs https://root.cern.ch/root/htmldoc/guides/users-guide/InputOutput.html

If anyone has to hand the list of the data type descriptors that root uses in the branch titles with their equivalent as a Python (numpy) dtype, that would be sufficient... and especially if this is already somewhere in the uproot codebase, it would be better to leverage that.

But I would be happy to do some reverse engineering on that, if needed.


Information about how to interpret a branch is scattered around, and some of it is only available in titles that have to be parsed as strings, such as the [parameters of Double32/Float16](https://github.com/scikit-hep/uproot4/blob/20d8575e941c32559c7b5e62b0ed5f92bc4927d0/uproot/interpretation/identify.py#L206-L214).

Everything in ROOT is big-endian—that, at least, is easy to state as an absolute rule for now. The new RTNtuple will be little-endian, but everything in the RNTuple class (apart from the [RNTuple "anchor"](https://github.com/scikit-hep/uproot4/blob/main/uproot/models/RNTuple.py), which is an entry point) is distinct from the rest of ROOT I/O.

Manually providing an Interpretation is certainly allowed. That was my initial, quick [answer on StackOverflow](https://stackoverflow.com/a/62520558/1623645). Sometimes, that's what I recommend when I can't figure out why someone's classes have internal headers or are missing internal headers, but it's plain from the serialized file whether those headers exist or not, and I give them an Interpretation they can plug in. [Automatically determining the Interpretation](https://github.com/scikit-hep/uproot4/blob/20d8575e941c32559c7b5e62b0ed5f92bc4927d0/uproot/interpretation/identify.py#L275-L291) is, of course, better, and I'll see if your code can be merged in to improve that.
Okay, thanks!

Resolving patch:
diff --git a/uproot/interpretation/identify.py b/uproot/interpretation/identify.py
--- a/uproot/interpretation/identify.py
+++ b/uproot/interpretation/identify.py
@@ -101,35 +101,53 @@ def _leaf_to_dtype(leaf):
 _pair_second = re.compile(r"pair\<[^<>]*,(.*) \>")
 
 
-def _from_leaves(branch, context):
+def _from_leaves_one(leaf, title):
     dims, is_jagged = (), False
-    if len(branch.member("fLeaves")) == 1:
-        leaf = branch.member("fLeaves")[0]
-        title = leaf.member("fTitle")
 
-        m = _title_has_dims.match(title)
-        if m is not None:
-            dims = tuple(int(x) for x in re.findall(_item_dim_pattern, title))
-            if dims == ():
-                if leaf.member("fLen") > 1:
-                    dims = (leaf.member("fLen"),)
+    m = _title_has_dims.match(title)
+    if m is not None:
+        dims = tuple(int(x) for x in re.findall(_item_dim_pattern, title))
+        if dims == ():
+            if leaf.member("fLen") > 1:
+                dims = (leaf.member("fLen"),)
 
-            if any(
-                _item_dim_pattern.match(x) is None
-                for x in re.findall(_item_any_pattern, title)
-            ):
-                is_jagged = True
+        if any(
+            _item_dim_pattern.match(x) is None
+            for x in re.findall(_item_any_pattern, title)
+        ):
+            is_jagged = True
+
+    return dims, is_jagged
+
+
+def _from_leaves(branch, context):
+    if len(branch.member("fLeaves")) == 0:
+        raise UnknownInterpretation(
+            "leaf-list with zero leaves",
+            branch.file.file_path,
+            branch.object_path,
+        )
+
+    elif len(branch.member("fLeaves")) == 1:
+        leaf = branch.member("fLeaves")[0]
+        title = leaf.member("fTitle")
+        return _from_leaves_one(leaf, title)
 
     else:
+        first = True
         for leaf in branch.member("fLeaves"):
-            if _title_has_dims.match(leaf.member("fTitle")):
-                raise UnknownInterpretation(
-                    "leaf-list with square brackets in the title",
-                    branch.file.file_path,
-                    branch.object_path,
-                )
-
-    return dims, is_jagged
+            title = leaf.member("fTitle")
+            if first:
+                dims, is_jagged = _from_leaves_one(leaf, title)
+            else:
+                trial_dims, trial_is_jagged = _from_leaves_one(leaf, title)
+                if dims != trial_dims or is_jagged != trial_is_jagged:
+                    raise UnknownInterpretation(
+                        "leaf-list with different dimensions among the leaves",
+                        branch.file.file_path,
+                        branch.object_path,
+                    )
+        return dims, is_jagged
 
 
 def _float16_double32_walk_ast(node, branch, source):
diff --git a/uproot/interpretation/library.py b/uproot/interpretation/library.py
--- a/uproot/interpretation/library.py
+++ b/uproot/interpretation/library.py
@@ -276,7 +276,8 @@ def _strided_to_awkward(awkward, path, interpretation, data):
     parameters = {
         "__record__": uproot.model.classname_decode(interpretation.model.__name__)[0]
     }
-    return awkward.layout.RecordArray(contents, names, len(data), parameters=parameters)
+    length = len(data) if len(contents) == 0 else None
+    return awkward.layout.RecordArray(contents, names, length, parameters=parameters)
 
 
 # FIXME: _object_to_awkward_json and _awkward_json_to_array are slow functions
@@ -377,8 +378,9 @@ def _awkward_json_to_array(awkward, form, array):
                         _awkward_json_to_array(awkward, subform, array[name])
                     )
                 names.append(name)
+        length = len(array) if len(contents) == 0 else None
         return awkward.layout.RecordArray(
-            contents, names, len(array), parameters=_awkward_p(form)
+            contents, names, length, parameters=_awkward_p(form)
         )
 
     elif form["class"][:15] == "ListOffsetArray":
@@ -415,10 +417,11 @@ def _awkward_json_to_array(awkward, form, array):
             else:
                 keys = _awkward_json_to_array(awkward, key_form, array.content["0"])
                 values = _awkward_json_to_array(awkward, value_form, array.content["1"])
+                length = len(array.content) if len(keys) == 0 else None
                 content = awkward.layout.RecordArray(
                     (keys, values),
                     None,
-                    len(array.content),
+                    length,
                     parameters=_awkward_p(form["content"]),
                 )
             cls = getattr(awkward.layout, form["class"])
@@ -581,6 +584,8 @@ def finalize(self, array, branch, interpretation, entry_start, entry_stop):
                         numpy.array(array[name]), regulararray=True, highlevel=False
                     )
                 )
+            if len(contents) != 0:
+                length = None
             out = awkward.layout.RecordArray(contents, array.dtype.names, length)
             for size in shape[::-1]:
                 out = awkward.layout.RegularArray(out, size)


Test patch: