add duplicate and json export pipeline