Repository: beeware/toga

Issue:
Modifying Tree Widget nodes before the Tree is Expanded/Drawn in macOS
### Describe the bug

If you attempt to append, insert, remove, or modify a child node used as a data source for the Tree widget, before it has expanded and displayed those nodes, an **AttributeError: 'Node' object has no attribute '_impl'** will be generated. It appears that the child nodes are being generated on demand and don't exist until they have been displayed. A work around is to manually expand the Tree before doing any modifications.

### Steps to reproduce

A minimum script to reproduce the error is shown below. If you comment out `tree.expand()` the error will be thrown.

```
import toga

def build(app):
    tree = toga.Tree(headings=["Type", "Text", "Options", "ID"])
    root_node = tree.data.append(dict(type="Root Node", text="", options="", ID="0"))
    leaf_node = dict(type="Spelling", text="word", options="opt", id="0", root=False)
    tree.expand()
    #root_node.append(leaf_node)
    root_node.insert(0, leaf_node)
    return tree

def main():
    return toga.App("Node Test App", "au.com.reefwing.node_test", startup=build)

if __name__ == "__main__":
    main().main_loop()
```

### Expected behavior

It should be possible to modify the underlying Tree data model regardless of the state of the UI widget.

### Screenshots

_No response_

### Environment

- Operating System: macOS
- Python version: 3.11
- Software versions:
  - Briefcase:
  - Toga: 0.4.2


### Logs

```
File "/Users/dsuch/Documents/BeeWare/tests/node_test.py", line 14, in build
    root_node.insert(0, leaf_node)
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/toga/sources/tree_source.py", line 122, in insert
    self._source.notify("insert", parent=self, index=index, item=node)
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/toga/sources/base.py", line 84, in notify
    method(**kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/toga_cocoa/widgets/tree.py", line 223, in insert
    inParent=parent._impl if parent else None,
             ^^^^^^^^^^^^
AttributeError: 'Node' object has no attribute '_impl'
```


### Additional context

_No response_


Discussion:
Thanks for the report. As best as I can work out, this is a macOS specific issue; analogous problems don't exist with other platforms.

The core of the issue is that macOS Trees (and tables, for that matter) have a single render method, and the objects representing the actual tree nodes are created on first render. This is an optimisation; if you have a tree with a million rows, you don't need to create a million objects to display the first 20. 

However, the "on demand" aspect is what causes issues here. If the widget has never been rendered, it doesn't have an implementation; so requesting the that the implementation be updated, or that the the implementation be used as a parent, causes a crash.

In many cases, the fix here will be to just ignore the problem.

In the case of an update to an existing node, if the node hasn't been displayed, there's nothing visible to update, so the lack of an implementation is no problem. The implementation will be created if/when the widget *is* displayed, and should assume the current value at that time.

In the case of deleting a node, if the node (or it's parent) hasn't been displayed, there's no visual representation to delete, so any calls to remove a native representation can be ignored.

The only mildly complicated case is adding a child. If the parent hasn't been displayed, then the child won't need to be displayed either; however, if the parent is later displayed, there *might* be a need to trigger the creation of child nodes. It's possible this *might* "just happen", but that definitely needs to be confirmed.

As a final aside - there's also something odd going on with the provided example code and the `expand` call - on both macOS and GTK, the expand call doesn't appear to be actually expanding the tree.
I think `tree.expand()` isn't doing anything because at the stage that it is called no children have been added. If you add a second expand call after adding the child node it expands the tree.
Interesting... I do wonder if that's a bug or a feature. A node with no children should still be *expandable* - there's just nothing there, and there's something to be said for *showing* there's nothing there. This sounds suspiciously like an `if children:` vs `if children is None:` when `children = []` bug.
Just talking macOS, but given the tree widget is based on NSOutlineView, then the [docs](https://developer.apple.com/documentation/appkit/nsoutlineview/1535323-expanditem?language=objc) suggest that this is expected behaviour. 

> If item is not expandable or is already expanded, does nothing.

Before you append/insert the child there is no disclosure triangle to expand.
Ah - that's where I was going wrong. I mistakenly remembered that the initial state of the root node's children was an empty list, but it's not - it's `None`, indicating no children, and the child list is created when the first child is added. So - my "is None vs children = []" analysis was correct... but I was wrong about it being incorrect behaviour. 

As you suggested, a `try:except` block sorts out the append/insert issue. It also handles removing a child or updating a childs fields before expansion. The additions are in `insert`, `remove`, and `change` respectively. One small problem(?), if you append and then remove a child immediately, i.e.,

```
root_node.append(leaf_node)
root_node.remove(root_node[0])
```

The disclosure triange remains visible, even though there are no children.

While I was here I thought I would test the other functions which include `_impl`. If you try to expand a specific node (e.g., `tree.expand(root_node)`) you get the same AttributeError. The `try:except` block removes the error but the node is not expanded. My thinking is before expanding it needs something like:

```
        try:
            node_impl = node._impl
        except AttributeError:
            node_impl = TogaData.alloc().init()
            node_impl.attrs = {"node": node}
            node._impl = node_impl
```

But with the actual `attrs` for the node assigned. It isn't clear to me how to do this. 

Collapse has the same problem but the `try:except` solution works.

Regarding adding tests for these cases should I add something to test_tree.py in widgets or do something else?
> As you suggested, a `try:except` block sorts out the append/insert issue. It also handles removing a child or updating a childs fields before expansion. The additions are in `insert`, `remove`, and `change` respectively. One small problem(?), if you append and then remove a child immediately, i.e.,
> 
> ```
> root_node.append(leaf_node)
> root_node.remove(root_node[0])
> ```
> 
> The disclosure triange remains visible, even though there are no children.

That sounds entirely consistent to me. It's the other end of the "bug" that I thought I was seeing. 

When a node is created, it sets `children=None` - this node *cannot* have children. That tells the Tree widget that no disclosure triangle is needed.

You then add a child, which sets `children = [child1]`. This tells the tree a disclosure triangle is required, and if expanded, that a child could be displayed.

You then delete the child, which removes the child from children: `children = []`. This tells the tree that the node *could* have children, but it doesn't - so there's a disclosure triangle, but pressing it does nothing.

It's a bit like the difference between an empty directory and a file with the same name in a file explorer. Adding a child to a node is a bit like converting the file into a directory and putting a child file in that directory; removing the child file doesn't change the directory back to a file.
 
> While I was here I thought I would test the other functions which include `_impl`. If you try to expand a specific node (e.g., `tree.expand(root_node)`) you get the same AttributeError. The `try:except` block removes the error but the node is not expanded. My thinking is before expanding it needs something like:
> 
> ```
>         try:
>             node_impl = node._impl
>         except AttributeError:
>             node_impl = TogaData.alloc().init()
>             node_impl.attrs = {"node": node}
>             node._impl = node_impl
> ```
> 
> But with the actual `attrs` for the node assigned. It isn't clear to me how to do this.

You don't need to. The `attrs` in this case are the attributes on the node - they only exist provide a path back to the data node that contains the properties of the item on the tree. 

The code you've got here looks like all you need - except that it's duplicating the code around L32. I'd suggest that you could factor this out into an idempotent `node_impl()` method that can be used anywhere that the user requests `node._impl`, and ensures that the impl exists. That might even be a way around adding the try-except logic everywhere - if you can guarantee that the _impl exists whenever it's accessed, the need to ignore attribute errors goes away (or, rather, the existence of the AttributeError is used to immediately backfill the missing `_impl`).
 
> Collapse has the same problem but the `try:except` solution works.
> 
> Regarding adding tests for these cases should I add something to test_tree.py in widgets or do something else?

To be clear - there's two tests named `test_tree.py` - one in core, and one in testbed. In this case, it's the testbed version that needs to be updated, because we need to confirm specific behavior of the Cocoa implementation (and confirm that the same problem doesn't exist for other platforms). 

But yes - all we need is a modification to the existing expand/insert/etc tests in the testbed `test_tree.py` (or new tests if it isn't easy to shoehorn the specific test case into the existing test data).

Thanks - that all makes sense. Although I think you made up the word idempotent! 8-)
> Thanks - that all makes sense. Although I think you made up the word idempotent! 8-)

Unsure if that was intended as a joke, but just in case: [Idempotence](https://en.wikipedia.org/wiki/Idempotence) is an important concept in computer science - in practice, it means calling a method N times has no more effect on a system than calling it once. If you know a method is idempotent, it radically changes the way you can use that method in a system. 

For example, one of the big differences between HTTP verbs like GET and POST is that GET is definitionally idempotent, but POST isn't. Multiple calls to GET the same resource have the same effect on the system - which means that GET requests can be cached. POST requests aren't idempotent, so they can't be cached. (And before anyone nitpicks: yes, there's more in the HTTP spec that impacts on this, including but not limited to the definition of "safe" methods... but that's a digression for our purposes here)

In Toga's case, "Get the impl, and if it doesn't exist, create it" would be idempotent because the impl is created the first time, but the existing instance is returned on subsequent calls (or if it existed to start with). A version that returned a new impl every time would *not* be idempotent, because the number of calls would alter the numbers of instances that were created, and the instance that is currently in use.
Sorry poor attempt at humour, although I did have to look it up. To be fair when I did Computer Science we were using a Vax 11.

So the mods are working on my tests and pass the existing testbed tests. When I look at test_tree.py, as you would imagine it is already testing expand/collapse/append/insert/remove. But it is always expanding the tree first. I'm thinking all I need to do is to comment out those intitial tree expansions. Is that sufficient?
> Sorry poor attempt at humour, although I did have to look it up. To be fair when I did Computer Science we were using a Vax 11.

Ah, the VAX... such memories... not good ones, mind... but memories :-)

> So the mods are working on my tests and pass the existing testbed tests. When I look at test_tree.py, as you would imagine it is already testing expand/collapse/append/insert/remove. But it is always expanding the tree first. I'm thinking all I need to do is to comment out those intitial tree expansions. Is that sufficient?

We need to retain the existing set of tests, because they're testing the behavior when the tree *is* expanded - but "the same set of tests but not expanded" could quite possibly be sufficient. However, I suspect you'd also want to check "what if I add when non-expanded then expand", and a few other cases like that, so a literal duplicate probably isn't the best match - but the general pattern of the test should be at least useful as a template.

Resolving patch:
diff --git a/cocoa/src/toga_cocoa/widgets/tree.py b/cocoa/src/toga_cocoa/widgets/tree.py
--- a/cocoa/src/toga_cocoa/widgets/tree.py
+++ b/cocoa/src/toga_cocoa/widgets/tree.py
@@ -16,6 +16,15 @@
 from toga_cocoa.widgets.internal.data import TogaData
 
 
+def node_impl(node):
+    try:
+        return node._impl
+    except AttributeError:
+        node._impl = TogaData.alloc().init()
+        node._impl.attrs = {"node": node}
+        return node._impl
+
+
 class TogaTree(NSOutlineView):
     interface = objc_property(object, weak=True)
     impl = objc_property(object, weak=True)
@@ -32,14 +41,8 @@ def outlineView_child_ofItem_(self, tree, child: int, item):
         # Get the Cocoa implementation for the row. If an _impl
         # doesn't exist, create a data object for it, and
         # populate it with initial values for each column.
-        try:
-            node_impl = node._impl
-        except AttributeError:
-            node_impl = TogaData.alloc().init()
-            node_impl.attrs = {"node": node}
-            node._impl = node_impl
 
-        return node_impl
+        return node_impl(node)
 
     @objc_method
     def outlineView_isItemExpandable_(self, tree, item) -> bool:
@@ -220,22 +223,25 @@ def insert(self, parent, index, item):
         index_set = NSIndexSet.indexSetWithIndex(index)
         self.native_tree.insertItemsAtIndexes(
             index_set,
-            inParent=parent._impl if parent else None,
+            inParent=node_impl(parent) if parent else None,
             withAnimation=NSTableViewAnimation.SlideDown.value,
         )
 
     def change(self, item):
-        self.native_tree.reloadItem(item._impl)
+        self.native_tree.reloadItem(node_impl(item))
 
     def remove(self, parent, index, item):
-        index = self.native_tree.childIndexForItem(item._impl)
-        index_set = NSIndexSet.indexSetWithIndex(index)
-        parent = self.native_tree.parentForItem(item._impl)
-        self.native_tree.removeItemsAtIndexes(
-            index_set,
-            inParent=parent,
-            withAnimation=NSTableViewAnimation.SlideUp.value,
-        )
+        try:
+            index = self.native_tree.childIndexForItem(item._impl)
+            index_set = NSIndexSet.indexSetWithIndex(index)
+            parent = self.native_tree.parentForItem(item._impl)
+            self.native_tree.removeItemsAtIndexes(
+                index_set,
+                inParent=parent,
+                withAnimation=NSTableViewAnimation.SlideUp.value,
+            )
+        except AttributeError:
+            pass
 
     def clear(self):
         self.native_tree.reloadData()
@@ -264,13 +270,13 @@ def get_selection(self):
                 return None
 
     def expand_node(self, node):
-        self.native_tree.expandItem(node._impl, expandChildren=True)
+        self.native_tree.expandItem(node_impl(node), expandChildren=True)
 
     def expand_all(self):
         self.native_tree.expandItem(None, expandChildren=True)
 
     def collapse_node(self, node):
-        self.native_tree.collapseItem(node._impl, collapseChildren=True)
+        self.native_tree.collapseItem(node_impl(node), collapseChildren=True)
 
     def collapse_all(self):
         self.native_tree.collapseItem(None, collapseChildren=True)


Test patch:
diff --git a/testbed/tests/widgets/test_tree.py b/testbed/tests/widgets/test_tree.py
--- a/testbed/tests/widgets/test_tree.py
+++ b/testbed/tests/widgets/test_tree.py
@@ -217,6 +217,87 @@ async def test_expand_collapse(widget, probe, source):
     assert not probe.is_expanded(source[2])
     assert not probe.is_expanded(source[2][2])
 
+    # Expanding/collapsing nodes when tree is not expanded
+    # Expand a single root node
+    widget.expand(source[1])
+    await probe.redraw("Root Node 1 has been expanded")
+    assert not probe.is_expanded(source[0])
+    assert not probe.is_expanded(source[0][2])
+    assert probe.is_expanded(source[1])
+    assert probe.is_expanded(source[1][2])
+    assert not probe.is_expanded(source[2])
+    assert not probe.is_expanded(source[2][2])
+
+    # Expand a single root node again
+    widget.expand(source[1])
+    await probe.redraw("Root Node 1 is still expanded")
+    assert not probe.is_expanded(source[0])
+    assert not probe.is_expanded(source[0][2])
+    assert probe.is_expanded(source[1])
+    assert probe.is_expanded(source[1][2])
+    assert not probe.is_expanded(source[2])
+    assert not probe.is_expanded(source[2][2])
+
+    # Expand a single child node
+    widget.expand(source[1][2])
+    await probe.redraw("Child Node 1:2 has been expanded")
+    assert not probe.is_expanded(source[0])
+    assert not probe.is_expanded(source[0][2])
+    assert probe.is_expanded(source[1])
+    assert probe.is_expanded(source[1][2])
+    assert not probe.is_expanded(source[2])
+    assert not probe.is_expanded(source[2][2])
+
+    # Expand the same child node again
+    widget.expand(source[1][2])
+    await probe.redraw("Child Node 1:2 is still expanded")
+    assert not probe.is_expanded(source[0])
+    assert not probe.is_expanded(source[0][2])
+    assert probe.is_expanded(source[1])
+    assert probe.is_expanded(source[1][2])
+    assert not probe.is_expanded(source[2])
+    assert not probe.is_expanded(source[2][2])
+
+    # Attempt to collapse a leaf node
+    widget.collapse(source[1][2][1])
+    await probe.redraw("Leaf node collapse is a no-op")
+    assert not probe.is_expanded(source[0])
+    assert not probe.is_expanded(source[0][2])
+    assert probe.is_expanded(source[1])
+    assert probe.is_expanded(source[1][2])
+    assert not probe.is_expanded(source[2])
+    assert not probe.is_expanded(source[2][2])
+
+    # Attempt to expand a leaf node
+    widget.expand(source[1][2][1])
+    await probe.redraw("Leaf node expansion is a no-op")
+    assert not probe.is_expanded(source[0])
+    assert not probe.is_expanded(source[0][2])
+    assert probe.is_expanded(source[1])
+    assert probe.is_expanded(source[1][2])
+    assert not probe.is_expanded(source[2])
+    assert not probe.is_expanded(source[2][2])
+
+    # Fully collapse the tree
+    widget.collapse()
+    await probe.redraw("All nodes have been collapsed")
+    assert not probe.is_expanded(source[0])
+    assert not probe.is_expanded(source[0][2])
+    assert not probe.is_expanded(source[1])
+    assert not probe.is_expanded(source[1][2])
+    assert not probe.is_expanded(source[2])
+    assert not probe.is_expanded(source[2][2])
+
+    # Fully collapse when already fully collapsed
+    widget.collapse()
+    await probe.redraw("All nodes are still collapsed")
+    assert not probe.is_expanded(source[0])
+    assert not probe.is_expanded(source[0][2])
+    assert not probe.is_expanded(source[1])
+    assert not probe.is_expanded(source[1][2])
+    assert not probe.is_expanded(source[2])
+    assert not probe.is_expanded(source[2][2])
+
     # Fully expand the tree
     widget.expand()
     await probe.redraw("All nodes have been expanded")
@@ -327,7 +408,7 @@ async def test_expand_collapse(widget, probe, source):
     assert probe.is_expanded(source[2])
     assert probe.is_expanded(source[2][2])
 
-    # Attempt to expand a leaf node
+    # Attempt to collapse a leaf node again
     widget.collapse(source[0][2][1])
     await probe.redraw("Leaf node collapse is a no-op")
     assert probe.is_expanded(source[0])
@@ -442,14 +523,14 @@ async def _row_change_test(widget, probe):
     """Meta test for adding and removing data to the table"""
 
     # Change the data source for something smaller
-    widget.data = [
+    small_data = [
         (
             {"a": "A0", "b": "", "c": ""},
             [({"a": f"A{i}", "b": i, "c": MyData(i)}, None) for i in range(0, 5)],
         )
     ]
-    await probe.expand_tree()
-    await probe.redraw("Data source has been changed")
+
+    widget.data = small_data
 
     assert probe.child_count() == 1
     assert probe.child_count((0,)) == 5
@@ -459,6 +540,93 @@ async def _row_change_test(widget, probe):
     probe.assert_cell_content((0, 4), 1, "4")
     probe.assert_cell_content((0, 4), 2, "<data 4>")
 
+    # Test append/insert/modify/remove prior to tree expansion
+    # Append a child to a non-expanded root
+    widget.data[0].append({"a": "AX", "b": "BX", "c": "CX"})
+    await probe.redraw("Full row has been appended to a non-expanded root")
+
+    assert probe.child_count((0,)) == 6
+    probe.assert_cell_content((0, 4), 0, "A4")
+    probe.assert_cell_content((0, 5), 0, "AX")
+
+    # Append a child to a non-expanded child
+    widget.data[0][5].append({"a": "AXX", "b": "BXX", "c": "CXX"})
+    await probe.redraw("Full row has been appended to a non-expanded child")
+
+    assert probe.child_count((0, 5)) == 1
+    probe.assert_cell_content((0, 5, 0), 0, "AXX")
+    probe.assert_cell_content((0, 5, 0), 1, "BXX")
+    probe.assert_cell_content((0, 5, 0), 2, "CXX")
+
+    # Insert a row into the middle of the table;
+    # Row is missing a B accessor
+    widget.data[0].insert(2, {"a": "AY", "c": "CY"})
+    await probe.redraw("Partial row has been appended to a non-expanded root")
+
+    assert probe.child_count((0,)) == 7
+    probe.assert_cell_content((0, 2), 0, "AY")
+    probe.assert_cell_content((0, 5), 0, "A4")
+    probe.assert_cell_content((0, 6), 0, "AX")
+
+    # Missing value has been populated
+    probe.assert_cell_content((0, 2), 1, "MISSING!")
+
+    # Change content on the partial row
+    widget.data[0][2].a = "ANEW"
+    widget.data[0][2].b = "BNEW"
+    await probe.redraw("Partial non-visible row has been updated")
+
+    assert probe.child_count((0,)) == 7
+    probe.assert_cell_content((0, 2), 0, "ANEW")
+    probe.assert_cell_content((0, 5), 0, "A4")
+    probe.assert_cell_content((0, 6), 0, "AX")
+
+    # Missing value has the default empty string
+    probe.assert_cell_content((0, 2), 1, "BNEW")
+
+    # Delete a row
+    del widget.data[0][3]
+    await probe.redraw("Non-visible row has been removed")
+    assert probe.child_count((0,)) == 6
+    probe.assert_cell_content((0, 2), 0, "ANEW")
+    probe.assert_cell_content((0, 4), 0, "A4")
+    probe.assert_cell_content((0, 5), 0, "AX")
+
+    # Insert a new root
+    widget.data.insert(0, {"a": "A!", "b": "B!", "c": "C!"})
+    await probe.redraw("New root row has been appended")
+
+    assert probe.child_count() == 2
+    assert probe.child_count((0,)) == 0
+    probe.assert_cell_content((0,), 0, "A!")
+
+    # Delete a root
+    del widget.data[1]
+    await probe.redraw("Old root row has been removed")
+
+    assert probe.child_count() == 1
+    assert probe.child_count((0,)) == 0
+    probe.assert_cell_content((0,), 0, "A!")
+
+    # Expand tree and check contents still correct
+    await probe.expand_tree()
+    await probe.redraw("Tree expanded")
+
+    assert probe.child_count() == 1
+    assert probe.child_count((0,)) == 0
+    probe.assert_cell_content((0,), 0, "A!")
+    probe.assert_cell_content((0,), 1, "B!")
+    probe.assert_cell_content((0,), 2, "C!")
+
+    # Clear the table
+    widget.data.clear()
+    await probe.redraw("Data has been cleared")
+    assert probe.child_count() == 0
+
+    # Test append/insert/modify/remove after tree expansion
+    widget.data = small_data
+    await probe.redraw("Data source has been changed")
+
     # Append a row to the table
     widget.data[0].append({"a": "AX", "b": "BX", "c": "CX"})
     await probe.redraw("Full row has been appended")