Change:
As NAME noted , the current truncation algorithm breaks parent hashes. This PR implements the "careful truncation" algorithm he proposes.
I think this should be good to go?
Merging this for now to un-break the spec. I will work with Marta to see if we can make a follow-up that removes redundant nodes, so that we can get back to more aggressive truncation. Hi everyone, while testing changes to our tree implementation, we found what we believe is a bug in the specification. More concretely, the tree truncation mandated by the spec is too aggressive and can destroy parent hashes in an otherwise valid tree. For debugging purposes, we had a check that would validate parent hashes of every node every time a commit is processed, which is how we found the bug. Currently, the spec mandates that upon a remove operation, the leaves are removed until the right-most leaf in non-blank. However, this can lead to a situation, where non-blank parent nodes of a non-blank leaf are removed. This removal potentially interrupts valid "parent-hash-chains". More concretely, if the affected leaf node was the last node to commit a path in its subtree, then overly aggressive truncation can lead to the removal of parent nodes in that path. This in turn leads to an error when the parent hash of the parent of the removed parent node is verified. Note that if the parent hash chain is interrupted in this way, the situation is indistinguishable from one where the parent hash of a node is simply invalid. Consider the following example: 15 | / | 7 | _ /  | 3 11 | /  /  | 1 5 9 13 | /  /  /  /  | A B C D E F G - I 1 1 1 1 1 1 1 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 G performs an update, leading to a valid parent hash chain along its direct path (13, 11, 7, 15). A removes I, leading to the following state, since G is the right-most non-blank leaf. 7 | /  3 11 /  /  1 5 9 | /  /  /  | A B C D E F G 1 1 1 0 1 2 3 4 5 6 7 8 9 0 1 2 Since we have removed 13 in the process of removing (leaf) node 14, verification of the parent hash of node 11 will now fail, since none of its children carries its hash in its parent hash field. This will become apparent if a new member is added, either by one of the nodes A-D or by any node without a path. As solution, we propose that the right-most leaf is only removed in the process of truncation if both it and its parent are blank. Raphael pointed out that this "careful" truncation leads to sparser trees and thus makes the protocol overall less efficient (larger tree structures need to be stored, longer direct paths with more secrets and key pairs to derive, etc.). We can fix this by having two forms of truncation: A "careful" one, as described above and the one that is already in the spec right now with one caveat. The careful one replaces the one that is currently described in the spec and is thus performed with each removal operation. The one that we have right now is performed before a committer (or the receiver of a commit) applies a direct path, except that it can only remove nodes that are in the direct path of the committer. This should allow us to get the smallest tree that we can without compromising valid parent hash chains. Cheers, Konrad

Discussion:
Blank the intermediate nodes along the path from L to the root Truncate the tree by removing leaves from the right side of the <del> tree until the rightmost leaf node is not blank. </del> <ins> tree as long as all of the following conditions hold (since non- blank intermediate nodes hold information that is necessary for verifying parent hashes): The rightmost leaf is blank The parent of the rightmost leaf is either blank or the root of the tree </ins> 11.1.4.