Package: fameio
FAME-IoPython scripts for FAME models, generation of protobuf input files and conversion of protobuf output files.
Please visit theFAME-Wikito get an explanation of FAME and its components.
The package is also formerly known asFAME-Py.InstallationWe recommend installingfameiousing PyPI:pip install fameioYou may also usepipx. For detailed information please refer to the officialpipxdocumentation.pipx install fameiofameiois currently developed and tested for Python 3.8 or higher.
See thesetup.pyfor a complete listing of dependencies.UsageFAME-Io currently offers two main scriptsmakeFameRunConfigandconvertFameResults.
Both are automatically installed with the package.
The first one creates a protobuf file for FAME applications using YAML definition files and CSV files.
The latter one reads output files from FAME applications in protobuf format and converts them to CSV files.You may use theexample dataprovided for theAMIRISmodel which can be used to simulate electricity markets inGermany,Austria, and a simpleproof-of-concept model.Make a FAME run configurationDigests configuration files in YAML format, combines them with CSV data files and creates a single input file for FAME applications in protobuf format.
Call structure:makeFameRunConfig -f <path/to/scenario.yaml>You may also specify any of the following arguments:CommandAction-lor--logSets the logging level. Default isinfo. Options aredebug,info,warning,warn,error,critical.-lfor--logfileSets the logging file. Default isNone. IfNoneis provided, all logs get only printed to the console.-oor--outputSets the path of the compiled protobuf output file. Default isconfig.pb.This could look as follows:makeFameRunConfig -f <path/to/scenario.yaml> -l debug -lf <path/to/scenario.log> -o <path/to/config.pb>You may also call the configuration builder from any Python script withfromfameio.scripts.make_configimportrunasmake_configmake_config("path/to/scenario.yaml")Similar to the console call you may also specify custom run config arguments and add it in a dictionary to the function call.fromfameio.scripts.make_configimportrunasmake_configfromfameio.source.cliimportOptionsrun_config={Options.LOG_LEVEL:"info",Options.OUTPUT:"output.pb",Options.LOG_FILE:"scenario.log",}make_config("path/to/scenario.yaml",run_config)Scenario YAMLThe "scenario.yaml" file contains all configuration options for a FAME-based simulation.
It consists of the sectionsSchema,GeneralProperties,AgentsandContracts, all of them described below.SchemaThe Schema is used to validate the inputs of the "scenario.yaml".
Since the Schema should be valid across multiple scenarios, it is recommended to defined it in a separate file and include the file here.Currently, the schema specifies:which type of Agents can be createdwhat type of input attributes an Agent useswhat type of Products an Agent can send in Contracts.The Schema consists of the sectionsHeaderandAgentTypes.HeaderScientific applications often evolve, and so do their required input parameters.
Therefore, the header specifies information what FAME-based application the schema is corresponding to.
In this way a schema.yaml is tied to a specific version an application, ensuring a match between the inputs required by the application, and those provided by the files created with FAME-Io.Header:Project:MyProjectNameRepoUrl:https://mygithosting.com/myProjectCommitHash:abc123Projectname of your project / FAME-based applicationRepoUrlURL of your projectCommitHashhash of the commit / version of your projectAgentTypesHere, each type of agent that can be created in your FAME-based application is listed, its attributes and its available Products for Contracts.
The structure of this sectionAgentTypes:MyAgentType:Attributes:MyAttribute:...MyOtherAttribute:...Products:['Product1','Product2','Product3']MyOtherAgentWithoutProductsOrAttributes:MyAgentTypeJava's simple class name of the Agent typeAttributesindicates that beginning of the attribute definition section for this Agent typeMyAttributeName of an attribute as specified in the corresponding Java source code of this Agent type (annotated with "@Input")MyOtherAttributeName of another attribute derived from Java source codeProductslist of Products that this Agent can send in Contracts; derived from Java source code of this Agent type (annotated with "@Product")MyOtherAgentWithoutProductsOrAttributesan Agent type that requires neither Attributes nor ProductsBoth Attributes and Products are optional - there could be useful Agents that require neither of them.
In the above example attribute definition was not shown (indicated by...).
The next example provides details on how to define an attribute:MySimpleAttribute:AttributeType:enumMandatory:trueList:falseValues:['AllowedValue1','AllowedValue2']Default:'AllowedValue1'Help:'Myhelptext'MyComplexAttribute:AttributeType:blockNestedAttributes:InnerAttributeA:AttributeType:integerInnerAttributeB:AttributeType:doubleMySimpleAttribute,MyDoubleList,MyComplexAttributeNames of the attributes as specified in the Java enum annotated with "@Input"AttributeType(required) data type of the attribute; see options in table belowMandatory(optional - true by default) if true: the attribute is required for this agent and validation will fail if the attribute is missing in the scenarioandno default is providedList(optional - false by default)AttributeType: time_seriescannot be trueAttributeType: blockif true: any nested element in the scenario must be part of a list element and thus can appear multiple timesif false: any nested element in the scenario can only appear onceany other AttributeType: the attribute is interpreted as list, i.e. multiple values can be assigned to this attribute in the scenarioNestedAttributes(required only ifAttributeType: block, otherwise disallowed) starts an inner Attribute definition block - defined Attributes are sub-elements ofMyComplexAttributeValues(optional - None by default): if present defines a list of allowed values for this attributeDefault(optional - None by default): if present defines a default value to be used in case the scenario does not specify itHelp(optional - None by default): if present defines a help text to you attributeAttributeTypevalueintegera 32-bit integer valuedoublea 64-bit floating-point value (integers also allowed)longa 64-bit integer valuetime_stampeither a FAME time stamp string or 64-bit integer valuestringany stringenumany string, however, usually tied to a set of allowedValuestime_serieseither a path to a .csv-file or a single 64-bit floating-point value; does not supportList: trueblockthis attribute has no value of its own but hosts a group of nested Attributes; impliesNestedAttributesto be definedGeneralPropertiesSpecifies FAME-specific properties of the simulation. Structure:GeneralProperties:RunId:1Simulation:StartTime:2011-12-31_23:58:00StopTime:2012-12-30_23:58:00RandomSeed:1Output:Interval:100Process:0Parameters:RunIdan ID that can be given to the simulation; use at your discretionStartTimetime stamp in the format YYYY-MM-DD_hh:mm:ss; first moment of the simulation.StopTimetime stamp in the format YYYY-MM-DD_hh:mm:ss; last moment of the simulation - i.e. simulation terminates after passing that time stampRandomSeedseed to initialise random number generation; each value leads to a unique series of random numbers.Intervalnumber of simulation ticks in between write-to-disk events; may be used for performance optimisations;Processid of process that performs write-to-disk operations; leave at 0 to be compatible with single-processes;AgentsSpecifies all Agents to be created in the simulation in a list. Each Agent has its own entry.
Structure:Agents:-Type:MyAgentWithInputsId:1Attributes:MyEnum:SAME_SHARESMyInteger:2MyDouble:4.2MyTimeSeries:"./path/to/time_series.csv"-Type:MyAgentWithoutInputsId:2Agent Parameters:TypeMandatory; Java's simple class name of the agent to be createdIdMandatory; simulation-unique id of this agent; if two agents have the same ID, the configuration process will stop.AttributesOptional; if the agent has any attributes, specify them here in the format "AttributeName: value"; please see attribute table aboveThe specifiedAttributesfor each agent must match the specifiedAttributesoptions in the linked Schema (see above).
For better structure and readability of thescenario.yaml,Attributesmay also be specified in a nested way as demonstrated below.Agents:-Type:MyAgentWithInputsId:1Attributes:Parent:MyEnum:SAME_SHARESMyInteger:2Parent2:MyDouble:4.2Child:MyTimeSeries:"./path/to/time_series.csv"In case Attributes are defined withList: trueoption, lists are assigned to an Attribute or Group:Attributes:MyDoubleList:[5.2,4.5,7,9.9]MyListGroup:-IntValueA:5IntValueB:42-IntValueA:7IntValueB:100Here,MyDoubleListandMyListGroupneed to specifyList: truein the corresponding Schema.
The shorter[]-notation was used to assign a list of floating-point values toMyDoubleList.
Nested itemsIntValueAandIntValueBofMyListGroupare assigned within a list, allowing the specification of these nested items several times.ContractsSpecifies all Contracts, i.e. repetitive bilateral transactions in between agents.
Contracts are given as a list.
We recommend moving Contracts to separate files and to use the!includecommand to integrate them in the scenario.Contracts:-SenderId:1ReceiverId:2ProductName:ProductOfAgent_1FirstDeliveryTime:-25DeliveryIntervalInSteps:3600-SenderId:2ReceiverId:1ProductName:ProductOfAgent_2FirstDeliveryTime:-22DeliveryIntervalInSteps:3600Attributes:ProductAppendix:valueTimeOffset:42Contract Parameters:SenderIdunique ID of agent sending the productReceiverIdunique ID of agent receiving the productProductNamename of the product to be sentFirstDeliveryTimefirst time of delivery in the format "seconds after the January 1st 2000, 00:00:00"DeliveryIntervalInStepsdelay time in between deliveries in secondsAttributescan be set to include additional information asint,float,enumordictdata typesDefinition of Multiple Similar ContractsOften, scenarios contain multiple agents of similar type that also have similar chains of contracts.
Therefore, FAME-Io supports a compact definition of multiple similar contracts.SenderIdandReceiverIdcan both be lists and support One-to-N, N-to-One and N-to-N relations like in the following example:Contracts:# effectively 3 similar contracts (0 -> 11), (0 -> 12), (0 -> 13)# with otherwise identical ProductName, FirstDeliveryTime & DeliveryIntervalInSteps-SenderId:0ReceiverId:[11,12,13]ProductName:MyOtherProductFirstDeliveryTime:100DeliveryIntervalInSteps:3600# effectively 3 similar contracts (1 -> 10), (2 -> 10), (3 -> 10)# with otherwise identical ProductName, FirstDeliveryTime & DeliveryIntervalInSteps-SenderId:[1,2,3]ReceiverId:10ProductName:MyProductFirstDeliveryTime:100DeliveryIntervalInSteps:3600# effectively 3 similar contracts (1 -> 11), (2 -> 12), (3 -> 13)# with otherwise identical ProductName, FirstDeliveryTime & DeliveryIntervalInSteps-SenderId:[1,2,3]ReceiverId:[11,12,13]ProductName:MyThirdProductFirstDeliveryTime:100DeliveryIntervalInSteps:3600Combined with YAML anchors complex contract chains can be easily reduced to a minimum of required configuration.
The following example is equivalent to the previous one and allows a quick extension of contracts to a new couple of agents e.g. (4;14):Groups:-&agentList1:[1,2,3]-&agentList2:[11,12,13]Contracts:-SenderId:0ReceiverId:*agentList2ProductName:MyOtherProductFirstDeliveryTime:100DeliveryIntervalInSteps:3600-SenderId:*agentList1ReceiverId:10ProductName:MyProductFirstDeliveryTime:100DeliveryIntervalInSteps:3600-SenderId:*agentList1ReceiverId:*agentList2ProductName:MyThirdProductFirstDeliveryTime:100DeliveryIntervalInSteps:3600CSV filesTIME_SERIES inputs are not directly fed into the Scenario YAML file.
Instead, TIME_SERIES reference a CSV file that can be stored some place else.
These CSV files follow a specific structure:They must contain exactly two columns.The first column must be a time stamp in formYYYY-MM-DD_hh:mm:ssThe second column must be a numerical value (either integer or floating-point)The separator of the two columns is a semicolonExemplary content of a valid CSV file:2012-01-01_00:00:00;400
2013-01-01_00:00:00;720.5
2014-01-01_00:00:00;650
2015-01-01_00:00:00;99.27772
2016-01-01_00:00:00;42
2017-01-01_00:00:00;0.1Please refer also to the detailed article aboutTimeStampsin theFAME-Wiki.Split and join multiple YAML filesThe user may include other YAML files into a YAML file to divide the content across files as convenient.
We explicitly recommend using this feature for theSchemaandContractssections.
Otherwise, the scenario.yaml may become crowded.Command: !IncludeTo hint YAML to load the content of another file use!include "path/relative/to/including/yaml/file.yml".
You can concatenate !include commands and can use !include in the included file as well.
The path to the included file is always relative to the file using the !include command.
So with the following file structurefile-structurea.yaml
folder/b.yaml
folder/c.yaml
folder/deeper_folder/d.yamlthe following !include commands workin a.yamlToBe: !include "folder/b.yaml"
OrNot: !include "folder/deeper_folder/d.yaml"in b.yamlThatIs: !include "c.yaml"
TheQuestion: !include "deeper_folder/d.yaml"Provided thatin c.yamlOr: maybed.yamlnot: "?"the resulting file would look like this:THe Joined file a.yamlToBe:
  ThatIs:
    Or: maybe
  TheQuestion:
    not: "?"
OrNot:
  not: "?"You may also specify absolute file paths if preferred by starting with a "/".When specifying only a file path, the complete content of the file is assigned to the given key.
You always need a key to assign the !include command to.
However, you cannot combine the value returned from !include with other values in the same key.
Thus, the following combinations do not work:caveats.yml!include "file.yaml" # no key assigned

Key:
  Some: OtherItem
  !include "file.yaml" # cannot join with other named items

List:
  - an: entry
  !include "file.yaml" # cannot directly join with list items, even if !include returns a listIntegrate specific nodes of YAML filesInstead of includingallcontent in the included file, you may also pick a specific node within that file.
For this use!include [<relative/path/to/file.yaml>, Path:To:Field:In:Yaml].
Here,:is used in the node-specifying string to select a sequence of nodes to follow - with custom depth.
Consider the following two files:file_to_be_included.yamlSet1:Subset1:Key:ValueSet2:OtherKey:OtherValueincluding_file.yaml-Type:MyAgentWithInputsId:1Attributes:!include_node[file_to_be_included.yaml,Set1:Subset1]Compiling "including_file.yaml" results inresulting_file.yaml-Type:MyAgentWithInputsId:1Attributes:Key:ValueLoad multiple filesUsing wildcards in the given path (e.g. "path/to/many/*.yaml") will lead to loading multiple files and assigning their content to the same key.
You can make use of this feature with or without specifying a node selector.
However, the elements to be joined across multiple files must be lists.
These lists are then concatenated into a single list and then assigned to the key in the file calling !include.
This feature is especially useful for Contracts: You can split the Contracts list into several files and place them in a separate folder.
Then use !include to re-integrate them into your configuration. An example:my_contract1.yamlContracts:
 - ContractA
 - ContractBmy_contract2.yamlContracts:
 - ContractC
 - ContractD
 - ContractEincluding_file.yamlContracts: [!include "my_contract*.yaml", "Contracts"]results inresult.yamlContracts:
 - ContractA
 - ContractB
 - ContractC
 - ContractD
 - ContractEIgnoring filesFiles that have their name start with "IGNORE_" are not included with the !include command.
You will see a debug output to notify you that the file was ignored.
Use this to temporarily take files out ouf your configuration without deleting or moving them.Read FAME resultsTakes an output file in protobuf format of FAME-based applications and converts it into files in CSV format.
An individual file for each type of Agent is created in a folder named after the protobuf input file.
Call structure:convertFameResults -f <./path/to/protobuf_file.pb>You may also specify any of the following arguments:CommandAction-lor--logSets the logging level. Default isinfo. Options aredebug,info,warning,warn,error,critical.-lfor--logfileSets the logging file. Default isNone. IfNoneis provided, all logs get only printed to the console.-aor--agentsIf specified, only a subset of agents is extracted from the protobuf file. Default is to extract all agents.-oor--outputSets the path to where the generated output files are written to. If not specified, the folder's name is derived from the input file's name. Folder will be created if it does not exist.-seor--single-exportEnables export of individual agents to individual files, when present. If not present (the default) one file perAgentTypeis created.-mor--memory-savingWhen specified, reduces memory usage profile at the cost of runtime. Use only when necessary.-ccor--complex-columnDefines how to deal with complex indexed output columns (if any).IGNOREignores complex columns.MERGEsquashes all data from complex columns in one big string entry.SPLITcreates a separate file for each complex indexed output column.-tor--timeOption to define conversion of time steps to given format (default=UTC) by-t/--time {UTC, INT, FAME}Additionally, you may merge TimeSteps of a certain range of steps in the output files to
i) associate multiple time steps with a common logical time in your simulation
ii) reduce number of lines in output filesFor this, add the optionmerge-timesand specify the arguments as follows:CommandAction-fpor--focal-pointTimeStep on whichsteps-beforeearlier andsteps-afterlater TimeSteps are merged on-sbor--steps-beforeRange of TimeSteps before thefocal-pointthey get merged to-saor--steps-afterRange of TimeSteps after thefocal-pointthey get merged toThis could look as follows:convertFameResults -f <./path/to/protobuf_file.pb> -l debug -lf <path/to/output.log> -a AgentType1 AgentType2 -o myCsvFolder -m -cc SPLIT merge-times -fp 0 -sb 1799 -sa 1800Make sure that in the range of time steps you specify for merging there is only one value per column in the merged time range.
If multiple values per column are merged values will get concatenated and might yield unexpected results.You may also call the conversion script from any Python script with:fromfameio.scripts.convert_resultsimportrunasconvert_resultsconvert_results("./path/to/protobuf_file.pb")Similar to the console call you may also specify custom run config arguments and add it in a dictionary to the function call.fromfameio.scripts.convert_resultsimportrunasconvert_resultsfromfameio.source.cliimportOptionsrun_config={Options.LOG_LEVEL:"info",Options.LOG_FILE:"scenario.log",Options.OUTPUT:"Output",Options.AGENT_LIST:['AgentType1','AgentType2'],Options.MEMORY_SAVING:False,Options.SINGLE_AGENT_EXPORT:False,Options.RESOLVE_COMPLEX_FIELD:"SPLIT",Options.TIME:"INT",Options.TIME_MERGING:{},}convert_results("./path/to/protobuf_file.pb",run_config)Cite FAME-IoIf you use FAME-Io for academic work, please cite as follows.Bibtex entry:@article{fameio2023joss,
  author  = {Felix Nitsch and Christoph Schimeczek and Ulrich Frey and Benjamin Fuchs},
  title   = {FAME-Io: Configuration tools for complex agent-based simulations},
  journal = {Journal of Open Source Software},
  year    = {2023},
  doi     = {doi: https://doi.org/10.21105/joss.04958}
}Available SupportThis is a purely scientific project by (at the moment) one research group.
Thus, there is no paid technical support available.
However, we will give our best to answer your questions and provide support.If you experience any trouble with FAME-Io, you may contact the developers viafame@dlr.de.
Please report bugs and make feature requests by filing issues following the provided templates (see alsoContribute).
For substantial enhancements, we recommend that you contact us viafame@dlr.defor working together on the code in common projects or towards common publications and thus further develop FAME-Io.ContributePlease read the Contributors License Agreementcla.md, sign it and send it tofame@dlr.debefore contributing.You will also find templates forbug reports,feature requests, andpull requests.We kindly ask you to read the Contributors License Agreementcla.md, sign it, and send it tofame@dlr.debefore contributing. Please see ourconventions of contributionwhich are described in the Wiki.Testing changes locallyOnce some changes have been performed on the local git clone, use the following command to override your local installation with your modified copy in order to test the result:python3setup.pybdist_wheel&&pip3install--force-reinstall--no-dependencies./dist/*.whlCode styleWe use the code formatting libraryblack.
The maximum line length is defined as 120 characters.
Therefore, before committing, runblack --line-length 120 .Pre-Commit hooksFAME-Io uses several pre-commit hooks to ensure high code quality.
To use them, installdevpackages and then initialise pre-commit in FAME-Io's base folder:pip install fameio[dev]
    pre-commit install -t pre-commit -t pre-push