Hugging Face's logo

aiflows
/
FunSearchFlowModule

Model card Files Community
Edit model card

Table of Contents

FunSearch

FunSearch Objects 

class FunSearch(CompositeFlow)

This class implements FunSearch. This code is an implementation of Funsearch (https://www.nature.com/articles/s41586-023-06924-6) and is heavily inspired by the original code (https://github.com/google-deepmind/funsearch) . It's a Flow in charge of starting, stopping and managing (passing around messages) the FunSearch process. It passes messages around to the following subflows:

  • ProgramDBFlow: which is in charge of storing and retrieving programs.
  • SamplerFlow: which is in charge of sampling programs.
  • EvaluatorFlow: which is in charge of evaluating programs.

Configuration Parameters:

  • name (str): The name of the flow. Default: "FunSearchFlow".
  • description (str): The description of the flow. Default: "A flow implementing FunSearch"
  • subflows_config (Dict[str,Any]): A dictionary of subflows configurations. Default:
    • ProgramDBFlow: By default, it uses the ProgramDBFlow class and uses its default parameters.
    • SamplerFlow: By default, it uses the SamplerFlow class and uses its default parameters.
    • EvaluatorFlow: By default, it uses the EvaluatorFlow class and uses its default parameters.

Input Interface:

  • from (str): The flow from which the message is coming from. It can be one of the following: "FunSearch", "SamplerFlow", "EvaluatorFlow", "ProgramDBFlow".
  • operation (str): The operation to perform. It can be one of the following: "start", "stop", "get_prompt", "get_best_programs_per_island", "register_program".
  • content (Dict[str,Any]): The content associated to an operation. Here is the expected content for each operation:
    • "start":
      • num_samplers (int): The number of samplers to start up. Note that it's still restricted by the number of workers available. Default: 1.
    • "stop":
      • No content. Pass either an empty dictionary or None. Works also with no content.
    • "get_prompt":
      • No content. Pass either an empty dictionary or None. Works also with no content.
    • "get_best_programs_per_island":
      • No content. Pass either an empty dictionary or None. Works also with no content.

Output Interface:

  • retrieved (Dict[str,Any]): The retrieved data.

Citation:

@Article{FunSearch2023, author = {Romera-Paredes, Bernardino and Barekatain, Mohammadamin and Novikov, Alexander and Balog, Matej and Kumar, M. Pawan and Dupont, Emilien and Ruiz, Francisco J. R. and Ellenberg, Jordan and Wang, Pengming and Fawzi, Omar and Kohli, Pushmeet and Fawzi, Alhussein}, journal = {Nature}, title = {Mathematical discoveries from program search with large language models}, year = {2023}, doi = {10.1038/s41586-023-06924-6} }

make_request_for_prompt 

def make_request_for_prompt()

This method makes a request for a prompt. It sends a message to itself with the operation "get_prompt" which will trigger the flow to call the ProgramDBFlow to get a prompt.

request_samplers 

def request_samplers(input_message: FlowMessage)

This method requests samplers. It sends a message to itself with the operation "get_prompt" which will trigger the flow to call the ProgramDBFlow to get a prompt.

Arguments:

  • input_message (FlowMessage): The input message that triggered the request for samplers.

get_next_state 

def get_next_state(input_message: FlowMessage)

This method determines the next state of the flow based on the input message. It will return the next state based on the current state and the message received.

Arguments:

  • input_message (FlowMessage): The input message that triggered the request for the next state.

Returns:

str: The next state of the flow.

set_up_flow_state 

def set_up_flow_state()

This method sets up the state of the flow. It's called at the beginning of the flow.

save_message_to_state 

def save_message_to_state(msg_id: str, message: FlowMessage)

This method saves a message to the state of the flow. It's used to keep track of state on a per message basis (i.e., state of the flow depending on the message received and id).

Arguments:

  • msg_id (str): The id of the message to save.
  • message (FlowMessage): The message to save.

rename_key_message_in_state 

def rename_key_message_in_state(old_key: str, new_key: str)

This method renames a key in the state of the flow in the "msg_requests" dictonary. It's used to rename a key in the state of the flow (i.e., rename a message id).

Arguments:

  • old_key (str): The old key to rename.
  • new_key (str): The new key to rename to.

message_in_state 

def message_in_state(msg_id: str) -> bool

This method checks if a message is in the state of the flow (in "msg_requests" dictionary). It returns True if the message is in the state, otherwise it returns False.

Arguments:

  • msg_id (str): The id of the message to check if it's in the state.

Returns:

bool: True if the message is in the state, otherwise False.

get_message_from_state 

def get_message_from_state(msg_id: str) -> Dict[str, Any]

This method returns the state associated with a message id in the state of the flow (in "msg_requests" dictionary).

Arguments:

  • msg_id (str): The id of the message to get the state from.

Returns:

Dict[str,Any]: The state associated with the message id.

pop_message_from_state 

def pop_message_from_state(msg_id: str) -> Dict[str, Any]

This method pops a message from the state of the flow (in "msg_requests" dictionary). It the state associate to a message and removes it from the state.

Arguments:

  • msg_id (str): The id of the message to pop from the state.

Returns:

Dict[str,Any]: The state associated with the message id.

merge_message_request_state 

def merge_message_request_state(id: str, new_states: Dict[str, Any])

This method merges new states to a message in the state of the flow (in "msg_requests" dictionary). It merges new states to a message in the state.

Arguments:

  • id (str): The id of the message to merge new states to.
  • new_states (Dict[str,Any]): The new states to merge to the message.

register_data_to_state 

def register_data_to_state(input_message: FlowMessage)

This method registers the input message data to the flow state. It's called everytime a new input message is received.

Arguments:

  • input_message (FlowMessage): The input message

call_program_db 

def call_program_db(input_message)

This method calls the ProgramDBFlow. It sends a message to the ProgramDBFlow with the data of the input message.

Arguments:

  • input_message (FlowMessage): The input message to send to the ProgramDBFlow.

call_evaluator 

def call_evaluator(input_message)

This method calls the EvaluatorFlow. It sends a message to the EvaluatorFlow with the data of the input message.

Arguments:

  • input_message (FlowMessage): The input message to send to the EvaluatorFlow.

call_sampler 

def call_sampler(input_message)

This method calls the SamplerFlow. It sends a message to the SamplerFlow with the data of the input message.

Arguments:

  • input_message (FlowMessage): The input message to send to the SamplerFlow.

generate_reply 

def generate_reply(input_message: FlowMessage)

This method generates a reply to a message sent to user. It packages the output message and sends it.

Arguments:

  • input_message (FlowMessage): The input message to generate a reply to.

run 

def run(input_message: FlowMessage)

This method runs the flow. It's the main method of the flow. It's called when the flow is executed.

ProgramDBFlowModule

ProgramDBFlow Objects 

class ProgramDBFlow(AtomicFlow)

This class implements a ProgramDBFlow. It's a flow that stores programs and their scores in a database. It can also query the database for the best programs or generate a prompt containing stored programs in order to evolve them with a SamplerFlow. This code is an implementation of Funsearch (https://www.nature.com/articles/s41586-023-06924-6) and is heavily inspired by the original code (https://github.com/google-deepmind/funsearch)

Configuration Parameters:

  • name (str): The name of the flow. Default: "ProgramDBFlow"
  • description (str): A description of the flow. This description is used to generate the help message of the flow. Default: " A flow that saves programs in a database of islands"
  • artifact_to_evolve_name (str): The name of the artifact/program to evolve. Default: "solve_function"
  • evaluate_function (str): The function used to evaluate the program. No Default value. This MUST be passed as a parameter.
  • evaluate_file_full_content (str): The full content of the file containing the evaluation function. No Default value. This MUST be passed as a parameter.
  • num_islands: The number of islands to use. Default: 3
  • reset_period: The period in seconds to reset the islands. Default: 3600
  • artifacts_per_prompt: The number of previous artifacts/programs to include in a prompt. Default: 2
  • temperature: The temperature of the island. Default: 0.1
  • temperature_period: The period in seconds to change the temperature. Default: 30000
  • sample_with_replacement: Whether to sample with replacement. Default: False
  • portion_of_islands_to_reset: The portion of islands to reset. Default: 0.5
  • template (dict): The template to use for a program. Default: {"preface": ""}

Input Interface:

  • operation (str): The operation to perform. It can be one of the following: ["register_program","get_prompt","get_best_programs_per_island"]

Output Interface:

  • retrieved (Any): The retrieved data. It can be one of the following:
    • If the operation is "get_prompt", it can be a dictionary with the following keys
      • code (str): The code of the prompt
      • version_generated (int): The version of the prompt generated
      • island_id (int): The id of the island that generated the prompt
      • header (str): The header of the prompt
    • If the operation is "register_program", it can be a string with the message "Program registered" or "Program failed to register"
    • If the operation is "get_best_programs_per_island", it can be a dictionary with the following keys:
      • best_island_programs (List[Dict[str,Any]]): A list of dictionaries with the following keys:
        • rank (int): The rank of the program (1 is the best)
        • score (float): The score of the program
        • program (str): The program
        • island_id (int): The id of the island that generated the program

Citation:

@Article{FunSearch2023, author = {Romera-Paredes, Bernardino and Barekatain, Mohammadamin and Novikov, Alexander and Balog, Matej and Kumar, M. Pawan and Dupont, Emilien and Ruiz, Francisco J. R. and Ellenberg, Jordan and Wang, Pengming and Fawzi, Omar and Kohli, Pushmeet and Fawzi, Alhussein}, journal = {Nature}, title = {Mathematical discoveries from program search with large language models}, year = {2023}, doi = {10.1038/s41586-023-06924-6} }

set_up_flow_state 

def set_up_flow_state()

This method sets up the state of the flow and clears the previous messages.

get_prompt 

def get_prompt()

This method gets a prompt from an island. It returns the code, the version generated and the island id.

reset_islands 

def reset_islands()

This method resets the islands. It resets the worst islands and copies the best programs to the worst islands.

register_program 

def register_program(program: AbstractArtifact, island_id: int,
                     scores_per_test: ScoresPerTest)

This method registers a program in an island. It also updates the best program if needed.

Arguments:

  • program (AbstractArtifact): The program to register
  • island_id (int): The id of the island to register the program
  • scores_per_test (ScoresPerTest): The scores per test of the program

get_best_programs 

def get_best_programs() -> List[Dict[str, Any]]

This method returns the best programs per island.

run 

def run(input_message: FlowMessage)

This method runs the flow. It performs the operation requested in the input message.

SamplerFlowModule

SamplerFlowModule.SamplerFlow

SamplerFlow Objects 

class SamplerFlow(ChatAtomicFlow)

This class implements a SamplerFlow. It is a flow that queries a LLM to generate a response to a given input. This class is a child of ChatAtomicFlow. and expects the same parameters as ChatAtomicFlow (see https://huggingface.co/aiflows/ChatFlowModule).

Configuration Parameters:

  • name (str): The name of the flow. Default: "SamplerFlowModule"
  • description (str): A description of the flow. Default: "A flow that queries an LLM model to generate prompts for the Sampler flow"
  • backend Dict[str,Any]: The backend of the flow. Used to call models via an API. See litellm's supported models and APIs here: https://docs.litellm.ai/docs/providers. The default parameters of the backend are all defined at aiflows.backends.llm_lite.LiteLLMBackend (also see the defaul parameters of litellm's completion parameters: https://docs.litellm.ai/docs/completion/input#input-params-1). Except for the following parameters who are overwritten by the ChatAtomicFlow in ChatAtomicFlow.yaml:
    • model_name (Union[Dict[str,str],str]): The name of the model to use. Default: "gpt-4" When using multiple API providers, the model_name can be a dictionary of the form {"provider_name": "model_name"}. E.g. {"openai": "gpt-3.5-turbo", "azure": "azure/gpt-3.5-turbo"} Default: "gpt-3.5-turbo" (the name needs to follow the name of the model in litellm https://docs.litellm.ai/docs/providers).
    • n (int) : The number of answers to generate. Default: 1
    • max_tokens (int): The maximum number of tokens to generate. Default: 2000
    • temperature float: The temperature of the generation. Default: 1.0
    • top_p float: An alternative to sampling with temperature. It instructs the model to consider the results of the tokens with top_p probability. Default: 0.4
    • frequency_penalty (number): It is used to penalize new tokens based on their frequency in the text so far. Default: 0.0
    • presence_penalty (number): It is used to penalize new tokens based on their existence in the text so far. Default: 0.0
    • stream (bool): Whether to stream the response or not. Default: false
  • system_message_prompt_template (Dict[str,Any]): The template of the system message. It is used to generate the system message. Default: See SamplerFlow.yaml for default.
  • init_human_message_prompt_template (Dict[str,Any]): The prompt template of the human/user message used to initialize the conversation (first time in). It is used to generate the human message. It's passed as the user message to the LLM. Default: See SamplerFlow.yaml for default.
  • human_message_prompt_template (Dict[str,Any]): The prompt template of the human/user message (message used everytime the except the first time in). It's passed as the user message to the LLM. Default: See SamplerFlow.yaml for default.
  • previous_messages (Dict[str,Any]): Defines which previous messages to include in the input of the LLM. Note that if first_kand last_k are both none, all the messages of the flows's history are added to the input of the LLM. Default:
    • first_k (int): If defined, adds the first_k earliest messages of the flow's chat history to the input of the LLM. Default: 1
    • last_k (int): If defined, adds the last_k latest messages of the flow's chat history to the input of the LLM. Default: 1

Input Interface Initialized (Expected input the first time in flow):

  • header (str): A header message to include in prompt
  • code (str): The "example" samples to generate our new sample from.

Input Interface (Expected input the after the first time in flow):

  • header (str): A header message to include in prompt
  • code (str): The "example" samples to generate our new sample from.

Output Interface:

  • api_output (str): The output of the API call. It is the response of the LLM to the input.
  • from (str): The name of the flow that generated the output. It's always "SamplerFlow"

Citation:

@Article{FunSearch2023, author = {Romera-Paredes, Bernardino and Barekatain, Mohammadamin and Novikov, Alexander and Balog, Matej and Kumar, M. Pawan and Dupont, Emilien and Ruiz, Francisco J. R. and Ellenberg, Jordan and Wang, Pengming and Fawzi, Omar and Kohli, Pushmeet and Fawzi, Alhussein}, journal = {Nature}, title = {Mathematical discoveries from program search with large language models}, year = {2023}, doi = {10.1038/s41586-023-06924-6} }

run 

def run(input_message)

This method calls the backend of the flow (so queries the LLM). It calls the backend with the previous messages of the flow.

Returns:

Any: The output of the backend.

EvaluatorFlowModule

EvaluatorFlowModule.EvaluatorFlow

EvaluatorFlow Objects 

class EvaluatorFlow(AtomicFlow)

This class implements an EvaluatorFlow. It is a flow that evaluates a program (python code) using a given evaluator function. This code is an implementation of Funsearch (https://www.nature.com/articles/s41586-023-06924-6) and is heavily inspired by the original code (https://github.com/google-deepmind/funsearch)

Configuration Parameters:

  • name (str): The name of the flow. Default: "EvaluatorFlow"
  • description (str): A description of the flow. This description is used to generate the help message of the flow. Default: "A flow that evaluates code on tests"
  • py_file (str): The python code containing the evaluation function. No default value. This MUST be passed as a parameter.
  • function_to_run_name (str): The name of the function to run (the evaluation function) in the evaluator file. No default value. This MUST be passed as a parameter.
  • test_inputs (Dict[str,Any]): A dictionary of test inputs to evaluate the program. Default: {"test1": None, "test2": None}
  • timeout_seconds (int): The maximum number of seconds to run the evaluation function before returning an error. Default: 10
  • run_error_score (int): The score to return if the evaluation function fails to run. Default: -100
  • use_test_input_as_key (bool): Whether to use the test input parameters as the key in the output dictionary. Default: False

Input Interface:

  • artifact (str): The program/artifact to evaluate.

Output Interface:

  • scores_per_test (Dict[str, Dict[str, Any]]): A dictionary of scores per test input.

Citation:

@Article{FunSearch2023, author = {Romera-Paredes, Bernardino and Barekatain, Mohammadamin and Novikov, Alexander and Balog, Matej and Kumar, M. Pawan and Dupont, Emilien and Ruiz, Francisco J. R. and Ellenberg, Jordan and Wang, Pengming and Fawzi, Omar and Kohli, Pushmeet and Fawzi, Alhussein}, journal = {Nature}, title = {Mathematical discoveries from program search with large language models}, year = {2023}, doi = {10.1038/s41586-023-06924-6} }

load_functions 

def load_functions()

Load the functions from the evaluator py file with ast parsing

run_function_with_timeout 

def run_function_with_timeout(program: str, **kwargs)

Run the evaluation function with a timeout

Arguments:

  • program (str): The program to evaluate
  • kwargs (Dict[str, Any]): The keyword arguments to pass to the evaluation function

Returns:

Tuple[bool, Any]: A tuple (bool, result) where bool is True if the function ran successfully and result is the output of the function

evaluate_program 

def evaluate_program(program: str, **kwargs)

Evaluate the program using the evaluation function

Arguments:

  • program (str): The program to evaluate
  • kwargs (Dict[str, Any]): The keyword arguments to pass to the evaluation function

Returns:

Tuple[bool, Any]: A tuple (bool, result) where bool is True if the function ran successfully and result is the output of the function

analyse 

def analyse(program: str)

Analyse the program on the test inputs

Arguments:

  • program (str): The program to evaluate

Returns:

Dict[str, Dict[str, Any]]: A dictionary of scores per test input

run 

def run(input_message: FlowMessage)

This method runs the flow. It's the main method of the flow.

Arguments:

  • input_message (FlowMessage): The input message
Downloads last month
0
Unable to determine this model's library. Check the docs .