Moved indexing to after remove calls so we don't add new indexes to data we are about to remove
Restructured to make to/from info more obvious and stops it's own array