recovery and duplicate removal