aiflows/JarvisFlowModule

For a detailed introduction to Jarvis, including Jarvis structures, an example run, etc, visit: https://huggingface.co/aiflows/JarvisFlowModule/blob/main/Introduction_to_Jarvis.md

Table of Contents

• JarvisFlow
  • JarvisFlow
• Controller_JarvisFlow
  • Controller_JarvisFlow
    • __init__
    • instantiate_from_config
    • run
• UpdatePlanAtomicFlow
  • UpdatePlanAtomicFlow
    • run
• Planner_JarvisFlow
  • Planner_JarvisFlow
    • detect_finish_or_continue
• run_Jarvis
• CtrlExMem_JarvisFlow
  • CtrlExMem_JarvisFlow
    • detect_finish_or_continue
• __init__
• IntermediateAns_Jarvis
  • IntermediateAns_Jarvis
• FinalAns_Jarvis
  • FinalAns_Jarvis
    • run

JarvisFlow

JarvisFlow Objects

class JarvisFlow(AbstractBossFlow)

JarvisFlow is a flow module for the boss Jarvis. It inherits from AbstractBossFlow. ( https://huggingface.co/aiflows/AbstractBossFlowModule/tree/main). Jarvis is a general purpose agent empowered by multiple large language models and tools including a code interpreter, to take task commands in natural language, and make plans, write and run code in an interactive fashion to finish the task.

The highlight of Jarvis is that it integrates 17 large language models, each of them prompted differently to achieve seamless inter-model communication and model-user interaction. The structure of Jarvis ensures that it is much more robust, flexible and memory-efficient than previous agents empowered by one single model. What's more, Jarvis integrates modules to allow for llm's memory management, ensuring persisted mid-long term memory and efficient short-term memory management, making its life duration much longer than single-modeled agents, and more powerful, in that it is able to accumulate important knowledge e.g. code library. Jarvis can also take response from the user and the environment (e.g. code execution result), and spontaneously re-plan and re-execute to make the execution more robust and reliable.

Configuration Parameters:

• memory_files (dict): mem_name-memfile_path pairs. mem_name is the name of the memory (plan, logs, code_library), and memfile_path is the path to the corresponding memory file. Configure this either in the .yaml file, or override the memory_files entry when running the flow.
• subflows_config (dict): configs for subflows.
  • MemoryReading: Module used to read in memory (https://huggingface.co/aiflows/MemoryReadingFlowModule), output interface configured so that it outputs the neeed memory.
  • Planner: Module used to interactively write plans for Jarvis, the planner is implemented in the JarvisFlow.
  • CtrlExMem: Module used to execute the plan in a controller-executor manner, and update the memory. It is implemented in the JarvisFlow.

The code interpreter of Jarvis (https://huggingface.co/aiflows/InterpreterFlowModule) relies on open-interpreter (https://github.com/KillianLucas/open-interpreter) We are extracting the specific code from open-interpreter because the litellm version of open-interpreter is not compatible with that of the current version of aiflows(v.0.1.7).

Controller_JarvisFlow

Controller_JarvisFlow Objects

class Controller_JarvisFlow(ChatAtomicFlow)

This class is a controller for JarvisFlow, it takes the plan generated by the planner, logs of previous executions, depending on the initial goal or the subsequent feedback from the branching executors (and the human), to decide which executor to call next (or to exit by calling finish).

Configuration Parameters:

• commands (dict): a dictionary of commands that the controller can call, each command has a name, a description, and a list of input arguments. The commands will be injected into the system message prompt template.
• system_message_prompt_template (str): the template for the system message prompt, there are several components needs to be injected into the template, including the commands, plan, plan_file_location, logs, and the goal. The injection of commands is done then initalizing the flow, the rest of the components are injected at the beginning of each run.
• previous_messages (int): a sliding window of previous messages that will be passed to the model. This is the central part of short-term memory management.

Input Interface Non Initialized:

• goal (str): the initial goal of the conversation, this is the input to the model.
• memory_files (dict): a dictionary of file locations that contains the plan, logs.
• plan (str): the plan generated by the planner, the plan will change (marked as done, or re-plan) as execution preceeds.
• logs (str): the logs of previous executions, the logs will be appended as execution preceeds.

Input Interface Initialized:

• result (str): the result of the previous execution, this is the input to the model.
• memory_files (dict): a dictionary of file locations that contains the plan, logs.
• plan (str): the plan generated by the planner, the plan will change (marked as done, or re-plan) as execution preceeds.
• logs (str): the logs of previous executions, the logs will be appended as execution preceeds.
• goal (str): the initial goal, this is kept because the goal is also injected into the system prompts so that Jarvis does not forget what the goal is, when the memory sliding window is implemented.

Output Interface:

• command (str): the command to be executed by the executor.
• command_args (dict): the arguments of the command to be executed by the executor.

__init__

def __init__(commands: List[Command], **kwargs)

Initialize the flow, inject the commands into the system message prompt template.

Arguments:

• commands (List[Command]): a list of commands that the controller can call.
• kwargs (Dict[str, Any]): other parameters.

instantiate_from_config

@classmethod
def instantiate_from_config(cls, config)

Setting up the flow from the config file. In particular, setting up the prompts, backend, and commands.

Arguments:

• config (Dict[str, Any]): the config file.

Returns:

Controller_JarvisFlow: the instantiated flow.

run

def run(input_data: Dict[str, Any]) -> Dict[str, Any]

Run the flow, update the system prompts, and run the model.

Arguments:

• input_data (Dict[str, Any]): the input data to the flow.

Returns:

Dict[str, Any]: the output of the flow.

UpdatePlanAtomicFlow

UpdatePlanAtomicFlow Objects

class UpdatePlanAtomicFlow(AtomicFlow)

This class is used to update the plan file with the updated plan, called by the controller, when it realizes one step of the plan is done, and provide the updated plan, it is exactly the same as the old plan, except the step that is done is marked as done.

Input Interface:

• updated_plan: the updated plan, exactly the same as the old plan, except the step that is done is marked as done.

Output Interface:

• result: the result of the operation

Configuration Parameters:

• input_interface: the input interface of the atomic flow
• output_interface: the output interface of the atomic flow

run

def run(input_data: Dict[str, Any])

Run the atomic flow.

Arguments:

• input_data (Dict[str, Any]): the input data

Returns:

Dict[str, Any]: the result of the operation

Planner_JarvisFlow

Planner_JarvisFlow Objects

class Planner_JarvisFlow(PlanWriterFlow)

This flow inherits from PlanWriterFlow (https://huggingface.co/aiflows/PlanWriterFlowModule), and is used to generate a plan for Jarvis.

Input Interface:

• goal (str): the goal of the planner, the goal comes from the user's query when calling Jarvis.
• memory_files (dict): a dictionary of memory files, the keys are the names of the memory files, the values are the locations of the memory files.

Output Interfaces:

• plan (str): the generated plan, the plan string will be written to the plan file and returned to the flow state of the Jarvis flow.
• summary (str): the summary of the planner.
• status (str): the status of the planner, can be "finished" or "unfinished".

Configuration Parameters:

• Also refer to PlanWriterFlow (https://huggingface.co/aiflows/PlanWriterFlowModule/blob/main/PlanWriterFlow.py) for more configuration parameters.
• input_interface: the input interface of the flow.
• output_interface: the output interface of the flow.
• subflows_config: the configuration of the subflows of the flow.
• early_exit_key: the key of the early exit signal in the output payload.
• topology: the topology of the subflows.

detect_finish_or_continue

@CircularFlow.output_msg_payload_processor
def detect_finish_or_continue(output_payload: Dict[str, Any],
                              src_flow) -> Dict[str, Any]

This function is used to detect whether the planner should finish or continue.

Arguments:

• output_payload (Dict[str, Any]): the output payload of the flow.
• src_flow (Flow): the flow that generates the output payload.

Returns:

Dict[str, Any]: the output payload of the flow.

run_Jarvis

CtrlExMem_JarvisFlow

CtrlExMem_JarvisFlow Objects

class CtrlExMem_JarvisFlow(CtrlExMemFlow)

This class inherits from the CtrlExMemFlow class from AbstractBossFlowModule. See: https://huggingface.co/aiflows/AbstractBossFlowModule/blob/main/CtrlExMemFlow.py

Input Interface:

• plan
• memory_files
• logs
• goal

Output Interface:

• result
• summary

Configuration Parameters:

• input_interface: the input interface of the flow
• output_interface: the output interface of the flow
• subflows_config: the subflows configuration of the flow
• topology: the topology of the subflows

Take notice that:

1. In the controller, we only keep the previous 3 messages for memory management, that will be: a. The assistant message (controller's last command) b. Manually updated new system prompt (new logs, new plans, etc.) c. The user message (result, feedback)
2. Each time one executor from the branch is executed, the logs is updated, this means: a. The logs file of Jarvis is updated. b. After MemoryReading at the end of each run of the loop, the logs in the flow_state is updated. c. The next time the controller is called, the updated logs is injected into the system prompts.
3. In the prompts of the controller, when the controller realizes one step of the plan is done, we ask the controller to revise what was done and mark the current step as done. This means: a. The plan file is updated. b. The plan in the flow_state is updated. c. The next time the controller is called, the updated plan is injected into the system prompts.

This is basically how the memory management works, to allow for more space for llm execution, and make sure the llm does not forget important information.

detect_finish_or_continue

@CircularFlow.output_msg_payload_processor
def detect_finish_or_continue(output_payload: Dict[str, Any],
                              src_flow) -> Dict[str, Any]

This function is called when the JarvisFlow receives a message from one of its branches. This function

processes the message and decides whether the JarvisFlow should continue or finish.

Arguments:

• output_payload (Dict[str, Any]): the output payload of the branch
• src_flow (str): the source flow of the message

Returns:

Dict[str, Any]: the updated output payload

__init__

IntermediateAns_Jarvis

IntermediateAns_Jarvis Objects

class IntermediateAns_Jarvis(HumanStandardInputFlow)

This class inherits from the HumanStandardInputFlow class. It is used to give an intermediate answer to the user. The user is then able to provide feedback on the intermediate result. Depending on the user's feedback, the controller will decide to do different things (e.g. continue, re-plan, etc.)

Input Interface:

• answer: The intermediate answer to the question asked by the user.

Output Interface:

• result: User's response to the intermediate answer.
• summary: A summary of the action.

Configuration parameters:

• query_message_prompt_template: The template of the message that is shown to the user.
• request_multi_line_input_flag: A flag that indicates whether the user can give a multi-line input.
• end_of_input_string: The string that indicates the end of the input.

FinalAns_Jarvis

FinalAns_Jarvis Objects

class FinalAns_Jarvis(HumanStandardInputFlow)

This class inherits from the HumanStandardInputFlow class. It is used to give the final answer to the user.

Input Interface:

• answer: The answer to the question asked by the user.

Output Interface:

• result: User's response to the final answer.
• summary: A summary of the action.

Configuration parameters:

• query_message_prompt_template: The template of the message that is shown to the user.
• request_multi_line_input_flag: A flag that indicates whether the user can give a multi-line input.
• end_of_input_string: The string that indicates the end of the input.

run

def run(input_data: Dict[str, Any]) -> Dict[str, Any]

The run method of the class.

Arguments:

• input_data (Dict[str, Any]): The input data of the flow.

Returns:

Dict[str, Any]: The output data of the flow.