Package: convogym
convogym is a Python library for simulating conversations between chatbots and for creating new training data based on human-chatbot interactions.Gym TypesThere are currently 3 different gym environments:Gym(base class),ActiveGym(for active learning new turn-level goals), andRLGym(use reinforcement learning to guide turn-level goals toward dialog-level goals).The basicGymenvironment is designed to interact with human users (interactive=True) or to simulate dialog episodes with randomly generated personality facts (interactive=False). Users can manually enter personality facts (reset_persona_func=get_custom_persona) to observe different effects on personalized dialog generation.InActiveGym, human users teaches the decoder model to use new turn-level goals which are represented as contextual inputs to the decoder model. At each turn, the user chooses a turn-level goal for which the decoder tries to incorporate into its responses. If the response does not live up to user expectations, the user is prompted to enter a correct response for which to train the decoder model with. Whentrain_mode=True, the decoder parameters are updated after each correction.InRLGym, human users no longer choose the turn-level goals, nor do they provide corrections to the responses. Instead, a dialog policy is trained to output a distribution over turn-level goals for which turn-level goals can be sampled from and used to guide the conversation. The default dialog-level objective function is a ranking loss used to detect relevant personality traits (from the PersonaChat dataset).Core ModulesThere are 6 core modules in convogym.decodersA module to do controlled text generation using language models. Currently supportsHuggingfacetransformers such as GPT-2, DialogGPT, etc. The default model is currently theaf1tang/personaGPTmodel card, which can handle personalities from the PersonaChat dataset.agentsA module that handles response generation, formats tokens for natural language understanding (NLU) and controlled natural language generation (NLG). The base class is theAgentobject, which handles formatting prefix tokens to do conditional text generation.statesA module to learn a representation of dialog histories. The base class isStateEstimatorwhich takes dialog history (tokens) and map to an embedding space (a feature vector inR^n).rewardsA module to train the state estimation and dialog policy. The base class is theRewardobject, which provides a dialog-level object to guide conversations toward specific end points (dialog-level goals).policiesA module which learns to outputturn-level goals– goals that decoder model can use to generate customized responses (e.g., personalized for the user, preserves sensibility of responses). The base class is thePolicyobject which interfaces with the training environment. A defaultaction_spaceis provided which defines a preliminary set of turn-level goals (actions) for which a policy learns to work with. This can be overwritten for specific uses.gymsA module that ties together the core components (decoder, agents, state estimators, reward functions and policy) into an interactive interface from which dialog episodes can be simulated (interactive=False) or from which users can specify ground-truth responses for active learning (interactive=False).Dependenciesconvogym requires the following dependencies:numpy>=1.20.2scipy>=1.6.2pandas>=1.2.4transformers==4.10.0dotenvtqdmInstallationThere are 2 ways to install the current package.The easiest way is to usepipinstallation:pip install -U convogym(09.12.21) Under constructionTo build from source, use Git to clone the latest repo:git clone git://github.com/af1tang/convogym.git
cd convogymOnce in the repo, pip install in developer mode locally.pip install -e ./As of September 2021, convogym is NOT supported on Apple Silicon M1 hardware.Getting StartedInteractive DialogThe decoder model can be interpreted as a straight forward language model that can generate a variety of stylized text. We can load the default decoder model as follows:>>> from convogym.decoders import model, tokenizerWe can initialize a gym environment to conduct a short conversation (3 rounds) with the decoder model as follows.>>> from convogym.gyms import Gym
>>> from convogym.prefixes import get_custom_persona, get_random_persona
>>> gym = Gym(model=model, tokenizer=tokenizer, interactive=True, reset_persona_func=get_custom_persona, length=3)
>>> gym.sim_convos(num_convos=1)In this case,get_custom_personaprompts us to give our partner, the decoder model, a set of personality facts to go off of. Alternatively, we can usereset_persona_func=get_random_personato sample from a list of personas provided in theconvogym/datafolder.>>> gym = Gym(model=model, tokenizer=tokenizer, interactive=False, reset_persona_func=get_random_persona, length=3)
>>> gym.sim_convos(num_convos=3)When we setinteractive=False, conversations are simulated using self-play between 2 decoder models, parameterized by  different personalities which is displayed at the end of each episode. We can also access the dialog history and personalities directly throughgym.data.Decoding w/ Turn-Level GoalsSuppose we want to teach the decoder model to generate responses related to specific topics (e.g., talk about hobbies) rather than personalities. We can create the followingprefix tokensto describe these turn-level goals.>>> from convogym.utils._device import to_var # use GPU
>>> goal = "ask about hobbies."
>>> inp = tokenizer.encode("
" + goal + "


")
>>> print(inp)
[50262, 2093, 546, 45578, 13, 50257, 50260, 50257, 50259]
>>> tokenizer.decode(model.generate(to_var(inp).long().view(1,-1)).tolist()[0][len(inp):] )
'hello do you have any hobbies?
'We can find a default list of turn-level goals using>>> from convogym._action_space import action_space
>>> print(action_space)So how do we train the decoder to utilizenewturn-level goals? The answer is throughActiveGym(Active Learning Gym).>>> from convogym.gyms import ActiveGym
>>> from convogym._configs import opts
>>> from convogym.load_data import prepare_personachat_dataset
>>> train_data, _ = prepare_personachat_dataset(model, tokenizer)
>>> new_goals = ['talk about pokemon.', 'ask about favorite anime.']
>>> gym = ActiveGym(model=model, tokenizer=tokenizer, action_space=new_goals,
>>>                              training_data=train_decoder_data, train_model=True)
>>> gym.sim_convos(1)In this setting, we are prompted to choose a goal fromnew_goalsat each turn. The decodermodelthen tries output the correct response. Whentrain_model=True, the decoder model is fine-tuned with gradient descent whenever we provide corrections.Dialog PolicyNow suppose we want to train a model to output turn-level goals. We can useRLGym(Reinforcement Learning Gym) to interact with a policy model.>>> from convogym.gyms import RLGym
>>> # import modules for RL
>>> from convogym.environments import Env
>>> # base state estimator aggregates hidden state info of input tokens from decoder
>>> from convogym.states import BaseStateEstimator
>>> # manual reward gets user input to rate episode
>>> from convogym.rewards import ManualReward
>>> # DQN policy uses batch q learning to update policy
>>> from convogym.policies import DQNPolicy
>>> # define an action space for the policy (default is designed for personachat)
>>> from convogym.load_data import default_action_space
>>> state_estimator = BaseStateEstimator(model=model, tokenizer=tokenizer)
>>> gym = RLGym( model=model, tokenizer=tokenizer,
                 policy=DQNPolicy(action_space=default_action_space),
                 env=Env(state_estimator),
                 reward_obj=ManualReward(state_estimator),
          )
>>> # simulate and rate 3 convos
>>> gym.sim_convos(num_convos=3, training=True)InManualReward, the user provides a ground truth reward for each dialog trajectory. This assumes that the user already have a task-specific reward in mind.Alternatively, users can also design dialog-level objective functions to train the policy (training=True). For example, the base classRewarduses aranking lossdesigned for the PersonaChat to identify relevant personalities used to parameterize the decoder model.ExamplesExample scripts of various ways to use convogym can be found atconvogym/examples.How to ContributeContributors at all levels of experience are welcomed.If you see an issue, please report it on the issues page. When possible, please provide reproducible code snippet and details of the functions used and the shape of the data involved.Please also open an issue page when making a feature request.To contribute, please clone the git repo and create a fork.A more detailed documentation page to come.How to CiteComing soon.