Single array remove instead of multiple

Each List.Remove copies the underlying array. Let's not do that.