Package: simtester
SimtesterSimtesteris an open-source toolkit for evaluating user simulator of task-oriented dialogue system(TOD). It is
developed based on Python and PyTorch. You can easily construct agents(system variants) with Simtester using either your
own implemented model or our trained model and construct tester with different combinations of agents. In tester-based
evaluation, the user simulator you implemented interacts with agents in the tester, ranks them, and tester calculates
the Exact Distinct score of the user simulator.[paper]InstallationQuick-StartContributionsCitingTeamLicenseInstallationSimtester works with the following operating systems：LinuxWindows 10macOS XSimtester requires Python version 3.6 or later.You can install Simtester from source:gitclonehttps://github.com/Superbooming/simtester&&cdsimtester
pipinstall-e.We will upload Simtester to pypi soon.Quick-StartWhen you construct a agent using your own implemented model, you should fill your model instanceyour_modeland your model interact functionyour_model_interact_fnwhich input the dialogue context and output the response.When you construct a agent using own trained model, you should download model first(link inModels). You
should fill the model name and the responding model directory which store the model you download and the remaining
configuration will be loaded according to default config file(insimtester/config/multiwoz/). Or you can fill your own
config file(.yaml format) path and the config file should include the model name and the responding model directory.See
the config file example insimtester/config/multiwoz/soloist-base.yaml.# construct a agent using your own implemented modelfromsimtesterimportAgentagent=Agent(your_model,your_model_interact_fn)# construct a gent using our trained modelfromsimtesterimportMultiWOZAgentagent=MultiWOZAgent('soloist-base','model/soloist-base')# fill model name and model directoryagent=MultiWOZAgent(config='simtester/config/multiwoz.soloist-base.yaml')# fill config path# interact with agent and get the responseresopnse=agent.interact('I want to book a resturant in the center of city.')# get dialogue context and start a new dialogueagent.get_context()agent.start_dialogue()When you construct a tester using your own agents, you should fill your agent list which includes your own agent
instance and the ground truth agent rank where a smaller number indicates a higher rank. If rank is not filled, the
number is default to incremented.When you construct a agent with strategy we provided, you can just fill the strategy name.When you interact with tester, you can either input the utterance list, or the rank your simulator predicts. If the
utterance list is input, tester will return the response list which includes response responding to each agent and the
dialogue state list which indicates whether each agent ends the dialog. You can set the end_token of tester and when the
agent receives response which includes the end_token, the agent's dialogue will be ended. If the rank your simulator
predicts is input, tester will compare the ground truth rank with input rank and return the result. Notice that the
input of the rank represents the end of the dialogue for all agents, and a new round of dialogue will start.fromsimtesterimportTester# construct tester with your own agentstester=Tester([agentA,agentB,agentC],[2,1,3])# construct tester with strategy we providedtester=Tester('context-tester')# interact with testertester.interact(['I want to book a resturant.','I want to book a hotel.','I want to find a attraction.'])tester.interact(rank=[2,3,1])# get current tester information and exact distinct scoretester.get_info()tester.get_score()# reset testertester.reset()ContributionsPlease let us know if you encounter a bug or have any suggestions
byfiling an issue.We welcome all contributions from bug fixes to new features and extensions.We expect all contributions discussed in the issue tracker and going through PRs.CitingIf you find Simtester useful for your research or development, please cite
ourPaper:@article{sun2022metaphorical,
  title={Metaphorical User Simulators for Evaluating Task-oriented Dialogue Systems},
  author={Sun, Weiwei and Guo, Shuyu and Zhang, Shuo and Ren, Pengjie and Chen, Zhumin and de Rijke, Maarten and Ren, Zhaochun},
  journal={arXiv preprint arXiv:2204.00763},
  year={2022}
}TeamSimtesterwas developed and maintained by Shandong University IR Lab.LicenseSimtesterusesMIT License.