## 2 Materials and Methods  
### Mission Architecture  
For the purpose of generalizing the autonomy evaluation procedure, a generic architecture for defining the test missions is proposed. The authors maintain that any autonomous mission may be subdivided into a set of generic _tasks_ that autonomous missions often incorporate (e.g., transit, conduct survey, station keeping, etc.). Each task may be divided into generic subtasks (for the remaining document, _task_ will refer to both tasks and subtasks). These tasks are separated by time and order via the mission-scripted programming but also by _events_ that represent time-based changes in the environment (e.g., a newly discovered obstacle, a sudden change in sea state, etc.) or within the autonomous platform itself (e.g., low battery, faulty on-board sensor, etc.). Depending upon how/when these events occur during the tasks being performed, specific mission _behaviors_ should (or must) take place in order to complete a specific task (or mission) or to prevent catastrophic system failure. In other words, behaviors consist of actions and/or reactions conducted by the autonomous vehicle during the execution of a task to aid in completion of the task. An important aspect of decomposing a mission into tasks and events is that at the lowest level of the decomposition, there could be significant overlap of tasks and reactions to events for a wide variety of missions. This enables one to make inferences about future autonomous capability from the evaluation performed in this environment for a set of canonical missions. Flexibility in mission definitions to capture a range of operational scenarios (as opposed to testing individual scenarios) during evaluation is highly desirable [26]. Describing a mission with tasks and associated events form the foundations for creating the corresponding scenarios and metrics. In the proposed architecture, a _mission_ (\(M\)) is defined as a set of tasks that must be completed to realize a specific goal or goals. _Tasks_ (\(T\)) are a set of data structures that define quantifiable starting and ending states (as a function of time and space) and that can be combined with other tasks to realize a mission goal. Tasks should be described through simple language (e.g., travel to a desired waypoint), and they are conditionally dependent within a specific mission such that relations between the order of tasks can be represented as a directed graph. _Events_ (\(E\)) are defined as a set of possible time-based changes (both internal and external to the given autonomous vehicle) that occur during and/or as a result of a specific task. _Behaviors_ (\(B\)) are defined as the actions necessary to perform a given task (e.g., avoid known obstacle, no event occurring, etc.) or occur in response to an event (e.g., avoid a new obstacle, perform station keeping, etc.). Example parameters used for performance evaluation may include elements such as the vehicle's total time and distance traveled to complete a task, the closest point of approach (CPA), and the total energy consumed. _Scenarios_ become instances of a given mission and are defined such that \(S\) denotes a space of all possible scenarios that can be\"realized\" with respect to the mission and the platform being used to accomplish the mission. Each element \(S_{i}\in S\) is selected by choosing values for a set of scenario parameters that are derived from key aspects of the mission description. Scenario parameters may encompass items such as survey areas (box size and location), launch/recovery points, time of day, and starting battery charge. A metric function, \(\mu\), maps elements of the scenario space \(S_{i}\) to \(\Re\) values:  
\[\forall S_{ik}\in S_{i}\,\exists\,\mu:\mu(S_{ik})\in\Re \tag{1}\]  
thereby combining many aspects for evaluating performance of the autonomous vehicle as it reacts to events and accomplishes tasks as it executes the scenario. Essentially, this metric produces a score for every scenario (a realized instance of the mission).