Work from the top down instead of from feature indices

As suggested by @mourner