properly adjust the character sprites lists on loaded viewport movement

before we were just skipping over them, and this had its own problems
now they are removed directly, and readded if/when they get back
into the loaded viewport