Package: autochain
AutoChainLarge language models (LLMs) have shown huge success in different text generation tasks and
enable developers to build generative agents based on objectives expressed in natural language.However, most generative agents require heavy customization for specific purposes, and
supporting different use cases can sometimes be overwhelming using existing tools
and frameworks. As a result, it is still very challenging to build a custom generative agent.In addition, evaluating such generative agents, which is usually done by manually trying different
scenarios, is a very manual, repetitive, and expensive task.AutoChain takes inspiration from LangChain and AutoGPT and aims to solve
both problems by providing a lightweight and extensible framework
for developers to build their own agents using LLMs with custom tools andautomatically evaluatingdifferent user scenarios with simulated
conversations. Experienced user of LangChain would find AutoChain is easy to navigate since
they share similar but simpler concepts.The goal is to enable rapid iteration on generative agents, both by simplifying agent customization
and evaluation.If you have any questions, please feel free to reach out to Yi Luyi.lu@forethought.aiFeatures🚀 lightweight and extensible generative agent pipeline.🔗 agent that can use different custom tools and
support OpenAIfunction calling💾 simple memory tracking for conversation history and tools' outputs🤖 automated agent multi-turn conversation evaluation with simulated conversationsSetupQuick installpipinstallautochainOr install from source after cloning this repositorycdautochain
pyenvvirtualenv3.10.11venv
pyenvlocalvenv

pipinstall.SetPYTHONPATHandOPENAI_API_KEYexportOPENAI_API_KEY=exportPYTHONPATH=`pwd`Run your first conversation with agent interactivelypythonautochain/workflows_evaluation/conversational_agent_eval/generate_ads_test.py-iHow does AutoChain simplify building agents?AutoChain aims to provide a lightweight framework and simplifies the agent building process in a
few
ways, as compared to existing frameworksEasy prompt updateEngineering and iterating over prompts is a crucial part of building generative
agent. AutoChain makes it very easy to update prompts and visualize prompt
outputs. Run with-vflag to output verbose prompt and outputs in console.Up to 2 layers of abstractionAs part of enabling rapid iteration, AutoChain chooses to remove most of the
abstraction layers from alternative frameworksAutomated multi-turn evaluationEvaluation is the most painful and undefined part of building generative agents. Updating the
agent to better perform in one scenario often causes regression in other use cases. AutoChain
provides a testing framework to automatically evaluate agent's ability under different
user scenarios.Example usageIf you have experience with LangChain, you already know 80% of the AutoChain interfaces.AutoChain aims to make building custom generative agents as straightforward as possible, with as
little abstractions as possible.The most basic example uses the default chain andConversationalAgent:fromautochain.chain.chainimportChainfromautochain.memory.buffer_memoryimportBufferMemoryfromautochain.models.chat_openaiimportChatOpenAIfromautochain.agent.conversational_agent.conversational_agentimportConversationalAgentllm=ChatOpenAI(temperature=0)memory=BufferMemory()agent=ConversationalAgent.from_llm_and_tools(llm=llm)chain=Chain(agent=agent,memory=memory)print(chain.run("Write me a poem about AI")['message'])Just like in LangChain, you can add a list of tools to the agenttools=[Tool(name="Get weather",func=lambda*args,**kwargs:"Today is a sunny day",description="""This function returns the weather information""")]memory=BufferMemory()agent=ConversationalAgent.from_llm_and_tools(llm=llm,tools=tools)chain=Chain(agent=agent,memory=memory)print(chain.run("What is the weather today")['message'])AutoChain also added support
forfunction callingin OpenAI models. Behind the scenes, it turns the function spec into OpenAI format without explicit
instruction, so you can keep following the sameToolinterface you are familiar with.llm=ChatOpenAI(temperature=0)agent=OpenAIFunctionsAgent.from_llm_and_tools(llm=llm,tools=tools)Seemore examplesunderautochain/examplesandworkflow
evaluationtest cases which can also be run interactively.Read more about detailedcomponents overviewWorkflow EvaluationIt is notoriously hard to evaluate generative agents in LangChain or AutoGPT. An agent's behavior
is nondeterministic and susceptible to small changes to the prompt or model. As such, it is
hard to know what effects an update to the agent will have on all relevant use cases.The current path for
evaluation is running the agent through a large number of preset queries and evaluate the
generated responses. However, that is limited to single turn conversation, general and not
specific to tasks and expensive to verify.To facilitate agent evaluation, AutoChain introduces the workflow evaluation framework. This
framework runs conversations between a generative agent and LLM-simulated test users. The test
users incorporate various user contexts and desired conversation outcomes, which enables easy
addition of test cases for new user scenarios and fast evaluation. The framework leverages LLMs to
evaluate whether a given multi-turn conversation has achieved the intended outcome.Read more about ourevaluation strategy.How to run workflow evaluationsYou can either run your tests in interactive mode, or run the full suite of test cases at once.autochain/workflows_evaluation/conversational_agent_eval/generate_ads_test.pycontains a few
example test cases.To run all the cases defined in a test file:pythonautochain/workflows_evaluation/conversational_agent_eval/generate_ads_test.pyTo run your tests interactively-i:pythonautochain/workflows_evaluation/conversational_agent_eval/generate_ads_test.py-iLooking for more details on how AutoChain works? See
ourcomponents overview