add ignore duplicate inputs option to avoid training duplicate sample