now deals with duplicate ids much better