Repository: d0c-s4vage/pfp (python)

Issue:
Pfp implicit array handling doesn't match 010's
## Describe the bug

Found this when trying to use the DEX.bt template on a dex file as described in #14 . The error occurs as part of this struct:

```c
typedef struct {
    ubyte val <comment="uleb128 element">;
    if(val > 0x7f) {
        ubyte val <comment="uleb128 element">;
        if (val > 0x7f) {
            ubyte val <comment="uleb128 element">;
            if(val > 0x7f) {
                ubyte val <comment="uleb128 element">;
                if(val > 0x7f) {
                    ubyte val <comment="uleb128 element">;
                }
            }
        }
    }
} uleb128 <read=ULeb128Read, comment="Unsigned little-endian base 128 value">;
```
Notice the multiple declarations of `val` - this creates an implicit array of `val`. Comparing the implicit array `val` to an integer is what is causing the problem.

In 010 editor, referencing `val` only resolves to the most recent implicit array item, not the entire array. More testing is needed to determine what the exact 010 behavior here is.

Ignore the mismatched filetypes (png/mp4/etc). The screenshot below is from 010 editor:

![image](https://user-images.githubusercontent.com/5090146/71797338-72a97f00-3002-11ea-8cf1-9341d6ab2a01.png)

and this is from pfp:

```
LE pfp> n
      1 typedef struct {
      2     Int3();
      3     ubyte val <comment="uleb128 element">;
      4     if(val > 0x7f) {
-->   5         ubyte val <comment="uleb128 element">;
      6         if (val > 0x7f) {
      7             ubyte val <comment="uleb128 element">;
      8             if(val > 0x7f) {
      9                 ubyte val <comment="uleb128 element">;
LE pfp> val
UChar(137 [89])
LE pfp> n
      2     Int3();
      3     ubyte val <comment="uleb128 element">;
      4     if(val > 0x7f) {
      5         ubyte val <comment="uleb128 element">;
-->   6         if (val > 0x7f) {
      7             ubyte val <comment="uleb128 element">;
      8             if(val > 0x7f) {
      9                 ubyte val <comment="uleb128 element">;
     10                 if(val > 0x7f) {
LE pfp> val
UChar[2] ('\x89P')
```
Notice how pfp resolves `val` to the full implicit array, instead of the most recent element.

## To Reproduce

Run the template below in 010 editor on the image below:

![python](https://user-images.githubusercontent.com/5090146/71797419-e481c880-3002-11ea-88b9-83f44e56408d.png)

```c
typedef struct {
    ubyte val <comment="uleb128 element">;
    if(val > 0x7f) {
        ubyte val <comment="uleb128 element">;
        if (val > 0x7f) {
            ubyte val <comment="uleb128 element">;
            if(val > 0x7f) {
                ubyte val <comment="uleb128 element">;
                if(val > 0x7f) {
                    ubyte val <comment="uleb128 element">;
                }
            }
        }
    }
} uleb128 <read=ULeb128Read, comment="Unsigned little-endian base 128 value">;

uleb128 test;
while(!FEof()) {
    ubyte val2;
}
```

In pfp, run the template below on the same png above (this template has an `Int3()` added):

```c
typedef struct {
    Int3();
    ubyte val <comment="uleb128 element">;
    if(val > 0x7f) {
        ubyte val <comment="uleb128 element">;
        if (val > 0x7f) {
            ubyte val <comment="uleb128 element">;
            if(val > 0x7f) {
                ubyte val <comment="uleb128 element">;
                if(val > 0x7f) {
                    ubyte val <comment="uleb128 element">;
                }
            }
        }
    }
} uleb128 <read=ULeb128Read, comment="Unsigned little-endian base 128 value">;

uleb128 test;
```

## Expected Behavior

Pfp should have the same behavior as 010 editor

## Implementation/Fix Notes/Thoughts

How will this affect traversing the dom? Perhaps it only matters during parsing, but after-the-fact referencing `val` will always return the full implicit array? As stated before, need to track a few more things down.


Pull request:
Adds consecutive duplicate (implicit) array support
fixes #110 

Patch:
diff --git a/docs/source/differences.rst b/docs/source/differences.rst
new file mode 100644
index 0000000..6ca5f91
--- /dev/null
+++ b/docs/source/differences.rst
@@ -0,0 +1,79 @@
+
+.. _differences:
+
+Differences Between 010 and pfp
+===============================
+
+This section documents the known differences between pfp and 010 editor.
+
+.. toctree::
+    :maxdepth: 1
+
+Duplicate Arrays
+----------------
+
+*TLDR*: Pfp does not [yet] support non-consecutive duplicate arrays.
+Consecutive duplicate arrays are fully supported.
+
+First, some definitions and back story.
+
+Duplicate arrays are what occurs when multiple variables of the same name
+are declared in the same scope. E.g.:
+
+.. code-block:: c
+
+    int x;
+    int x;
+    if (x[0] == x[1] || x[0] == x) {
+        Printf("Same!");
+    }
+
+The 010 template script above declares ``x`` twice, creating a duplicate, or
+as pfp originally called it, an implicit array. Notice the two comparisons -
+they actually perform the same comparison:
+
+.. code-block:: c
+
+    x[0] != x[1]
+
+and
+
+.. code-block:: c
+
+    x[0] == x
+
+In 010, if the duplicate/implicit array is referenced without indexing, the
+most recently parsed field in the duplicate array is returned. I.e., it's treated
+as a normal field and not an array. However, if indexing is done on the duplicate
+array variable, the variable is treated as an array.
+
+Below is a quote on duplicate arrays from the
+`010 Editor documentation <https://www.sweetscape.com/010editor/manual/ArraysDuplicates.htm>`_:
+
+    When writing a template, regular arrays can be declaring using the same syntax
+    as scripts (see Arrays and Strings). However, 010 Editor has a syntax that
+    allows arrays to be built in a special way. When declaring template variables,
+    multiple copies of the same variable can be declared. For example:
+
+    .. code-block:: c
+
+        int x;
+        int y;
+        int x;
+
+    010 Editor allows you to treat the multiple declarations of the variable as
+    an array (this is called a Duplicate Array). In this example, x[0] could
+    be used to reference the first occurrence of x and x[1] could be used to
+    reference the second occurrence of x. Duplicate arrays can even be defined
+    with for or while loops. For example:
+
+    .. code-block:: c
+
+        local int i;
+        for( i = 0; i < 5; i++ )
+            int x;
+
+This breaks down in pfp when non-consecutive arrays are created, as is done
+in the first code sample from the 010 Editor documentation above.
+`Issue #111 <https://github.com/d0c-s4vage/pfp/issues/111>`_ tracks the effort
+to add support for non-consecutive duplicate arrays.
diff --git a/docs/source/index.rst b/docs/source/index.rst
index 9d0c67f..2c8a1b3 100644
--- a/docs/source/index.rst
+++ b/docs/source/index.rst
@@ -106,6 +106,12 @@ find the ``tEXt`` chunk, and change the comment: ::
             chunk.data.tEXt.comment = "NEW COMMENT"
             print("Comment after: {}".format(chunk.data.tEXt.comment))
 
+Notes
+-----
+
+A few differences do exist between 010 Editor and pfp. See the
+:ref:`differences` section for specific, documented differences.
+
 
 Contents:
 
@@ -120,6 +126,7 @@ Contents:
    interpreter
    functions
    bitstream
+   differences
 
 .. automodule:: pfp
    :members:
diff --git a/pfp/fields.py b/pfp/fields.py
index 5a6add1..acfa035 100644
--- a/pfp/fields.py
+++ b/pfp/fields.py
@@ -662,6 +662,38 @@ class Void(Field):
     pass
 
 
+@inherit_hash
+class ImplicitArrayWrapper(Field):
+    """
+    """
+
+    last_field = None
+    implicit_array = None
+
+    def __init__(self, last_field, implicit_array):
+        """Redirect all attribute accesses to the ``last_field``, except for
+        array indexing. Array indexing is forwarded on to the
+        ``implicit_array``.
+        """
+        super(Field, self).__setattr__("last_field", last_field)
+        super(Field, self).__setattr__("implicit_array", implicit_array)
+
+    def __setattr__(self, name, value):
+        """Custom setattr that forwards all sets to the last_field
+        """
+        return setattr(self.last_field, name, value)
+
+    def __getattr__(self, name):
+        """Custom getattr that forwards all gets to last_field.
+        """
+        return getattr(self.last_field, name)
+    
+    def __getitem__(self, key):
+        """Let this ImplicitArrayWrapper act like an array
+        """
+        return self.implicit_array[key]
+
+
 @inherit_hash
 class Struct(Field):
     """The struct field"""
@@ -671,6 +703,10 @@ class Struct(Field):
     _pfp__children = []
     """All children of the struct, in order added"""
 
+    _pfp__implicit_arrays = {}
+    """Mapping of all implicit arrays in this struct. All implicit arrays will
+    be resolved to a concrete array after parsing is complete"""
+
     _pfp__name_collisions = {}
     """Counters for any naming collisions"""
 
@@ -679,6 +715,8 @@ class Struct(Field):
     def __init__(self, stream=None, metadata_processor=None):
         # ordered list of children
         super(Struct, self).__setattr__("_pfp__children", [])
+        # initialize implicit arrays for this struct instance
+        super(Struct, self).__setattr__("_pfp__implicit_arrays", {})
         # for quick child access
         super(Struct, self).__setattr__("_pfp__children_map", {})
 
@@ -690,6 +728,23 @@ class Struct(Field):
         if stream is not None:
             self._pfp__offset = stream.tell()
 
+    def _pfp__finalize(self):
+        """Finalize the results of parsing the data. Currently this involves:
+
+          * resolving implicit arrays to concrete arrays
+        """
+        to_swap = []
+        for child_name, child in six.iteritems(self._pfp__children_map):
+            if isinstance(child, Struct):
+                child._pfp__finalize()
+                continue
+            if child_name not in self._pfp__implicit_arrays:
+                continue
+            to_swap.append((child_name, child))
+
+        for child_name, child in to_swap:
+            self._pfp__children_map[child_name] = self._pfp__implicit_arrays[child_name]
+            
     def _pfp__snapshot(self, recurse=True):
         """Save off the current value of the field
         """
@@ -747,7 +802,25 @@ class Struct(Field):
         ):
             return self._pfp__handle_non_consecutive_duplicate(name, child)
         elif not overwrite and name in self._pfp__children_map:
-            return self._pfp__handle_implicit_array(name, child)
+            implicit_array = self._pfp__handle_implicit_array(name, child)
+
+            # see #110 (https://github.com/d0c-s4vage/pfp/issues/110)
+            # during parsing, duplicate (implicit) arrays should always
+            # reference the last parsed variable of ``name``. However, if the
+            # variable ``name`` is indexed, then the nth duplicate array item
+            # should be returned.
+            #
+            # E.g.
+            #
+            #    int x;
+            #    int x;
+            #    int x;
+            #    Printf("%d\n", x);    // prints the latest x value
+            #    Printf("%d\n", x[0]); // prints the first x value
+            #
+            self._pfp__implicit_arrays[name] = implicit_array
+            self._pfp__children_map[name] = ImplicitArrayWrapper(child, implicit_array)
+            return child
         else:
             child._pfp__parent = self
             self._pfp__children.append(child)
@@ -812,13 +885,14 @@ class Struct(Field):
         """Handle inserting implicit array elements
         """
         existing_child = self._pfp__children_map[name]
-        if isinstance(existing_child, Array):
+        existing_implicit_array = self._pfp__implicit_arrays.get(name, None)
+        if isinstance(existing_implicit_array, Array):
             # I don't think we should check this
             #
             # if existing_child.field_cls != child.__class__:
             #    raise errors.PfpError("implicit arrays must be sequential!")
-            existing_child.append(child)
-            return existing_child
+            existing_implicit_array.append(child)
+            return existing_implicit_array
         else:
             cls = (
                 child._pfp__class
diff --git a/pfp/interp.py b/pfp/interp.py
index 49e938d..ffe3705 100644
--- a/pfp/interp.py
+++ b/pfp/interp.py
@@ -817,6 +817,7 @@ class PfpInterp(object):
         self._dlog("parsed template into ast")
 
         res = self._run(keep_successful)
+        res._pfp__finalize()
         return res
 
     def step_over(self):


Test patch:
diff --git a/tests/test_arrays.py b/tests/test_arrays.py
index 28b30b5..450995d 100644
--- a/tests/test_arrays.py
+++ b/tests/test_arrays.py
@@ -85,6 +85,20 @@ class TestArrays(utils.PfpTestCase):
         self.assertEqual(dom.chars[2], ord("C"))
         self.assertEqual(dom.chars[3], ord("D"))
 
+    def test_implicit_array_same_behavior_as_010(self):
+        dom = self._test_parse_build(
+            "ABCD",
+            """
+                while(!FEof()) {
+                    char x;
+                    Printf("%c", x);
+                }
+            """,
+            stdout="ABCD",
+        )
+        self.assertIsInstance(dom.x, Array)
+        self.assertEqual(dom.x, b"ABCD")
+
     def test_array_length1(self):
         dom = self._test_parse_build(
             "abcd",