A computer-implemented method comprising: receiving , from a user, a task for an excavator and associated task parameter values; given the task, the associated task parameter values, at least some of perception-related data received from a perception module, using a high-level task planner module comprising one or more rules-based models to divide the task into one or more base move sub-tasks and one or more material removal sub-tasks, wherein the one or more base move sub-tasks comprise a sequence of route points and the one or more material removal sub-tasks comprise a sequence of sub-task regions and a goal for each sub-task region; given the sequence of route points and states information wherein at least some of the states information is obtained from the perception module, using a base move sub-task planner that comprises one or more models to generate a set of one or more waypoints between consecutive pairs of route points to guide the excavator between route points; given the sequence of sub-task regions, associated goals, and states information wherein at least some of the states information is obtained from the perception module, using a material removal sub-task planner comprising one or more planner models to determine motion primitives and related parameter values for each sub-task region to accomplish the goal for that sub-task region; using the motion primitives, related parameters, excavator parameters, and perception module data as inputs to a joint move primitive module to generate joint trajectories and using waypoints, excavator parameters, and perception module data as inputs to a base move primitive module to generate base trajectories; and providing the joint trajectories and base trajectories to joint controller module(s) and base controller module(s), respectively, to generate controller commands to control devices of the excavator. receiving , from a user, a task for an excavator and associated task parameter values; given the task, the associated task parameter values, at least some of perception-related data received from a perception module, using a high-level task planner module comprising one or more rules-based models to divide the task into one or more base move sub-tasks and one or more material removal sub-tasks, wherein the one or more base move sub-tasks comprise a sequence of route points and the one or more material removal sub-tasks comprise a sequence of sub-task regions and a goal for each sub-task region; given the sequence of route points and states information wherein at least some of the states information is obtained from the perception module, using a base move sub-task planner that comprises one or more models to generate a set of one or more waypoints between consecutive pairs of route points to guide the excavator between route points; given the sequence of sub-task regions, associated goals, and states information wherein at least some of the states information is obtained from the perception module, using a material removal sub-task planner comprising one or more planner models to determine motion primitives and related parameter values for each sub-task region to accomplish the goal for that sub-task region; using the motion primitives, related parameters, excavator parameters, and perception module data as inputs to a joint move primitive module to generate joint trajectories and using waypoints, excavator parameters, and perception module data as inputs to a base move primitive module to generate base trajectories; and providing the joint trajectories and base trajectories to joint controller module(s) and base controller module(s), respectively, to generate controller commands to control devices of the excavator. <EOS>