Package: tree-of-thoughts-llm
Official Repo of Tree of Thoughts (ToT)Note: https://github.com/kyegomez/tree-of-thoughts CANNOT replicate paper results.In fact, people have reported that his codecannotproperlyrun, and isprobably automatically generated by ChatGPT, andkyegomez has done so for other popular ML methods, while intentionally refusing to link to official implementations for his own interests (seehttps://github.com/kyegomez/tree-of-thoughts/issues/54,https://github.com/kyegomez/tree-of-thoughts/issues/55,https://github.com/kyegomez/tree-of-thoughts/issues/56).
Unfortunately, Google/Github searches go to kyegomez's malicious repo by default as it has more stars.Please DE-STAR his repo and STAR this to help other people avoid being misled, thanks!Official implementation for paperTree of Thoughts: Deliberate Problem Solving with Large Language Modelswith code, prompts, model outputs.
Also checkits tweet threadin 1min.SetupSet up OpenAI API key and store in environment variableOPENAI_API_KEY(seehere).Install dependencies andtotpackage (PyPI package coming soon):gitclonehttps://github.com/princeton-nlp/tree-of-thought-llmcdtree-of-thought-llm
pipinstall-rrequirements.txt
pipinstall-e.# install `tot` packageQuick StartThe following minimal script will attempt to solve the game of 24 with4 5 6 10(might be a bit slow as it's using GPT-4):importargparsefromtot.methods.bfsimportsolvefromtot.tasks.game24importGame24Taskargs=argparse.Namespace(backend='gpt-4',temperature=0.7,task='game24',naive_run=False,prompt_sample=None,method_generate='propose',method_evaluate='value',method_select='greedy',n_generate_sample=1,n_evaluate_sample=3,n_select_sample=5)task=Game24Task()ys,infos=solve(args,task,900)print(ys[0])And the output would be something like (note it's not deterministic, and sometimes the output can be wrong):10 - 4 = 6 (left: 5 6 6)
5 * 6 = 30 (left: 6 30)
30 - 6 = 24 (left: 24)
Answer: (5 * (10 - 4)) - 6 = 24Paper ExperimentsRun experiments viash scripts/{game24, text, crosswords}/{standard_sampling, cot_sampling, bfs}.sh, except in crosswords we use a DFS algorithm for ToT, which can be run viascripts/crosswords/search_crosswords-dfs.ipynb.The very simplerun.pyimplements the ToT + BFS algorithm, as well as the naive IO/CoT sampling. Some key arguments:--naive_run: if True, run naive IO/CoT sampling instead of ToT + BFS.--prompt_sample(choices=[standard,cot]): sampling prompt--method_generate(choices=[sample,propose]): thought generator, whether to sample independent thoughts (used in Creative Writing) or propose sequential thoughts (used in Game of 24)--method_evaluate(choices=[value,vote]): state evaluator, whether to use the value states independently (used in Game of 24) or vote on states together (used in Creative Writing)--n_generate_sample: number of times to prompt for thought generation--n_evaluate_sample: number of times to prompt for state evaluation--n_select_sample: number of states to keep from each step (i.e.bin the paper's ToT + BFS algorithm)Paper Trajectorieslogs/contains all the trajectories from the paper's experiments, except forlogs/game24/gpt-4_0.7_propose1_value3_greedy5_start900_end1000.jsonwhich was reproduced after the paper (as the original experiment was done in a notebook) and achieved a 69% score instead of the original 74% score due to randomness in GPT decoding. We hope to aggregate multiple runs in the future to account for sampling randomness and update the paper, but this shouldn't affect the main conclusions of the paper.How to Add A New TaskSetting up a new task is easy, and mainly involves two steps.Set up a new task class intot/tasks/and task files intot/data/. Seetot/tasks/game24.pyfor an example. Add the task totot/tasks/__init__.py.Set up task-specific prompts intot/prompts/. Seetot/prompts/game24.pyfor an example. Depending on the nature of the task, choose--method_generate(choices=[sample,propose]) and--method_evaluate(choices=[value,vote]) and their corresponding prompts.CitationsPlease cite the paper and star this repo if you use ToT and find it interesting/useful, thanks! Feel free to contactshunyuyao.cs@gmail.comor open an issue if you have any questions.@misc{yao2023tree,title={{Tree of Thoughts}: Deliberate Problem Solving with Large Language Models},author={Shunyu Yao and Dian Yu and Jeffrey Zhao and Izhak Shafran and Thomas L. Griffiths and Yuan Cao and Karthik Narasimhan},year={2023},eprint={2305.10601},archivePrefix={arXiv},primaryClass={cs.CL}}