Package: alter-ego-llm
alter_egoalter_egois a library that allows you to run experiments with LLMs.egois a command-line helper included with this package.alter_egoallows you to run microexperiments using a simple shorthand. You can also create more advanced experiments. For turn-based interactive experiments, abuilderis available.Read ourpaper.You can browse our source code in this repository.Hereare autogenerated docs that can make it easier to find what you're looking for.Getting startedPrerequisitesInstallPython, at least version 3.8.If you are on Windows, make sure to install Python into the PATH.Create avirtual environmentand activate it. On Linux and macOS, this is very simple. Just open a terminal and execute the following commands:user@host:~$python-mvenvenvuser@host:~$sourceenv/bin/activateNote: In this document, you may have to replacepythonbypython3andpipbypip3. This depends on your system's settings.On Windows, consider usingthis tutorialto create and activate a virtual environment.Some editors also do this for you.Installalter_egousing(env)user@host:~$pipinstall-Ualter_ego_llmNote how(env)signals that we are in the virtual environment created earlier.For the remainder of this document, we assume that your editor's current directory is also your terminal's present working directory.From within your terminal, you can find out the present working directory usingpwd— this should show the very same directory as opened in your editor.Usingalter_egowith GPTNote: If you do not want to use GPT for now, simply changeGPTThreadtoCLIThreadin the examples below and skip this section.Obtain an API key from OpenAI.Here is more information.Your API key looks as follows:sk-***Copy this to your clipboard.Create a new file in your editor.Put the content of your clipboard into the fileopenai_keyinyour current directory. The file must not have a file extension—it is literally just calledopenai_key.Developing a simple microexperimentNew:📺 WATCH VIDEO TUTORIALLet's create a minimal experiment usingalter_ego's shorthand feature.Create a new file in your editor,first_experiment.py. Here's its code:importalter_ego.agentsfromalter_ego.utilsimportextract_numberfromalter_ego.experimentimportfactorialdefagent():returnalter_ego.agents.GPTThread(model="gpt-3.5-turbo",temperature=1.0)prompt="Estimate the public approval rating of {{politician}} during the {{time}} of their presidency. Only return a single percentage from 0 to 100."data=factorial(prompt,politician=["George W. Bush","Barack Obama"],time=["1st year","8th year"]).run(agent,extract_number,times=1)forrowindata:print(row)Note how we use variables within the prompt. The crucial feature ofalter_egois how these variables are automatically replaced based on treatment.In the terminal, run(env)user@host:~$pythonfirst_experiment.pyThis will take a few seconds and give you output similar to this:{'politician': 'George W. Bush', 'time': '1st year', 'result': None}
{'politician': 'George W. Bush', 'time': '8th year', 'result': None}
{'politician': 'Barack Obama', 'time': '1st year', 'result': 63}
{'politician': 'Barack Obama', 'time': '8th year', 'result': 8}As you see, GPT did not give a valid response for George W. Bush. Let's debug by changing the line withrunto:).run(agent,extract_number,times=1,keep_retval=True)Rerunning our script gives:{'politician':'George W. Bush','time':'1st year','result':62,'retval':'Approximately 62%.'}{'politician':'George W. Bush','time':'8th year','result':None,'retval':"It is difficult to provide an accurate estimate without conducting a specific poll or analysis. However, based on historical data and trends, it is common for a president's approval rating to decline over the course of their second term. Taking into account various factors such as the economic recession and the ongoing Iraq War during George W. Bush's final year in office (2008), it is reasonable to estimate his public approval rating to be around 25-35%. Please note that this estimation is subjective and might not perfectly reflect the actual public sentiment at that time."}(Here, I have shown only two rows of the output.)As you see, in the cases where GPT returned only a single number,alter_egowas able to correctly extract it. Unfortunately, GPT 3.5 tends to refuse requests to just return a single number. GPT 4 works better. If you have access to GPT 4 over the API, you can changemodel="gpt-3.5-turbo"tomodel="gpt-4". This is the resulting output (where I have omitted theretvalonce again):{'politician':'George W. Bush','time':'1st year','result':57}{'politician':'George W. Bush','time':'8th year','result':34}{'politician':'Barack Obama','time':'1st year','result':57}{'politician':'Barack Obama','time':'8th year','result':55}Hereyou can view the documentation forrun.runallows you to quickly execute an experiment defined by what highfalutin scientists call a “factorial design.” This is because the possibilities ofpolitician(George W. Bush, Barack Obama) were “multiplied” by the possibilities fortime(1st year, 8th year).The nice thing about these microexperiments is that you can easily carry the output forward to Pandas, Polars, etc.—this is becausedatais only a “list of dicts,” and as such it is trivial to convert to a DataFrame. This allows you to analyze data received straight from an LLM.Of course, you will often want to set the temperature to0.0or another low value. This depends on the nature of your use-case.Using the builder to construct a turn-based experimentNew:📺 WATCH VIDEO TUTORIALWe offer a web app to buildsimpleexperiments between multiple LLMs. The builder can be foundhere, with its source code being availablehere.For now, just read through the app (it showcases an example of a framed ultimatum game) and scroll down.Copy the code shown below “Export or import scenario” on the web app into a new file. Call that filebuilt.jsonin your current project directory.The file must be calledbuilt.json.Open a terminal and execute(env)user@host:~$egorunbuiltThis will show “System instructions” for two separate players. Note how they vary: One player (the first one) is the proposer and the second player is the responder.The proposer is now asked to put in a proposal in JSON. Let's do it:{"keep":4.2}As you see, the responder is notified and can nowACCEPTorREJECT. Let's accept:ACCEPTThis completes the experiment. You will see something like:Experiment c6627c4e-f17f-4cdc-ba47-462eced3e489 OKLet's look at the data that was generated. We can get it in CSV format by executing:(env)user@host:~$egodatabuiltc6627c4e-f17f-4cdc-ba47-462eced3e489>data.csv(You need to replacec6627c4e-f17f-4cdc-ba47-462eced3e489with your actual experiment ID)This should tell you that 2 lines were written. If you opendata.csvin your preferred spreadsheet calculator, you will see the following output:choiceconvoexperimentiroundtaintedthreadthread_typetreatment{"keep": 4.2}03ba9edf-99c0-46d6-8c26-42c26683197cc6627c4e-f17f-4cdc-ba47-462eced3e48911False1ce5faaf-cc1f-438f-8231-8b7e0d96fb07CLIThreadtake"ACCEPT"03ba9edf-99c0-46d6-8c26-42c26683197cc6627c4e-f17f-4cdc-ba47-462eced3e48921False9043cd57-3e83-42d2-8d62-c783725e05e7CLIThreadtakeThis is obviously easy to post-process in whatever statistics software you use.If you re-run the experiment, enter garbage instead of the expected inputs and re-export the data, you will see that thetaintedcolumn becomesTrue. You can check fortaintedto verify that inputs were received and processed as expected. Note that onceanyThread responds invalidly, the Conversation will be stopped and all Threads will havetaintedset toTrue. Thus,ego data's output may be partial.You can run your scenario five times by doing(env)user@host:~$egorun-n5builtNeedless to say, but you can replace5by any integer whatsoever.Feel free to experiment with our builder.Note: Experts can set the environment variableBUILT_FILEto haveegouse a different file name frombuilt.json.Usingalter_egowith oTreeNew:📺 WATCH VIDEO TUTORIALoTreeis a relatively popular framework for web-based experiments.You can attach Threads (i.e., LLMs) and Conversations (i.e., bundles of LLMs) to oTree objects (participants, players, subsessions or sessions). This basically works as follows (in your app's__init__.py:fromalter_ego.agentsimport*fromalter_ego.utilsimportfrom_filefromalter_ego.exports.otreeimportlinkasai...defcreating_session(subsession):forplayerinsubsession.get_players():ai(player).set(GPTThread(model="gpt-3.5-turbo",temperature=1.0))Here, eachplayerwould get their own personal GPT agent. If you want to assign such an agent to agroup, just do this:defcreating_session(subsession):forgroupinsubsession.get_groups():ai(group).set(GPTThread(model="gpt-3.5-turbo",temperature=1.0))Then, within your code, you can access the agent using acontext manager. Here's an example of a simplelive_method-based chat if we attached the Thread to theplayerobject:classChat(Page):deflive_method(player,data):ifisinstance(data,str):withai(player)asllm:# this submits the player's message and gets the responseresponse=llm.submit(data,max_tokens=500)# note: if you put the AI on the "group" object or somewhere other than# the player, you may want to change thisreturn{player.id_in_group:response}If you want to set the system prompt, you can do:def before_next_page(player, timeout_happened):
    with ai(player) as llm:
        # this sets the "system" prompt
        llm.system("You are participating in an experiment.")You can do whatever you want, but always remember to open the LLM's context (usingwith ai(...) as llm) before performing any action. (This extra step is necessary because of oTree's ORM, which otherwise couldn't notice changes deep down in the Thread.)You can also attach a whole Conversation to the aforementioned oTree objects.We provide a simple Chat in this repository, see the directoryotree/ego_chat.Remember to put your API key into your oTree project folder.alter_egosaves message histories automatically in.ego_outputin your oTree project folder.Developing full-fledged experimentsNew:📺 WATCH VIDEO TUTORIALYou can use the primitives exposed by this library to develop full-fledged experiments that go beyond the capabilities of our builder. The directoryscenarios/contains a bunch of examples, including the code for our paper's machine--machine interaction example (ego_prereg.py). Watch the video tutorial to get a feeling for what's possible.CitationWhen using any part ofalter_egoin a scientific context, cite the following work:@article{ego,
  title={Integrating Machine Behavior into Human Subject Experiments: A User-friendly Toolkit and Illustrations},
  author={Engel, Christoph and Grossmann, Max R. P. and Ockenfels, Axel},
  year={2023},}Licensealter_egois © Max R. P. Grossmannet al., 2023. It is licensed under LGPLv3+. Please seeLICENSEfor details.This program is distributed in the hope that it will be useful, butwithout any warranty; without even the implied warranty ofmerchantabilityorfitness for a particular purpose. See the GNU Lesser General Public License for more details.