sl2: Store the children of displayables in the cache.

This means we don't have to re-add them all the time, and prevents
clear from being called for no-child displayables.