Patent Description:
Unmanned Ground Vehicles (UGVs) are mobile machines that can move and perform work without having a human operator onboard. Some UGVs may operate autonomously thereby following a path and performing work in accordance with one or more job parameters.

Unmanned Ground Vehicles offer many advantages over manned systems for a variety of industries. First, they may operate more safely and precisely. Second, their sensing abilities are not limited to human senses. Third, they are not subject to human limitations, such as needing to sleep or eat, fatigue, human error, et cetera.

As UGVs begin to find use in such industries as agriculture, forestry, and construction, it is important that they interact with operators and the environment around them in a manner that is safe and efficient. Unlike hobby drones or residential lawn mowers, these massive machines have the ability to cause significant loss or damage if their activities are misplaced or performed erroneously. Further, industry adoption of such UGVs requires enough trust in the platform and system to incur the expense of equipment acquisition and then further trust to operate the UGVs on their jobs. Finally, given the significant increase in complexity over manned systems, it is important for UGVs to provide interfaces and workflows that are easy and intuitive for operators.

Precision agriculture and smart farming offer potential solutions to meet the rising food needs of an ever-expanding human population. One aspect of smart farming uses unmanned ground vehicles to perform field operations. Such UGVs can or will soon perform tasks related to fertilizing before seeding, seeding, spraying crops, and harvesting.

<CIT>, considered as generic, describes a platform allowing telemetric control of agricultural machines with a mobile device and activation of automatic functions of the machine with the mobile device.

The invention is defined by the independent claims <NUM> and <NUM>.

A mobile device includes a processor and memory, coupled to the processor, which memory contains instructions that when executed cause the processor to provide an autonomy management application. The autonomy management application includes an autonomy approval function. The autonomy approval function provides a user interface configured to display at least one aspect of an autonomy job to a user and receive user acceptance of the autonomy job. The autonomy approval function is configured to instruct the user to move within a pre-determined distance from an autonomous work machine that will execute the autonomy job and provide a machine-verified indication of user proximity within the pre-determined distance based on data of the mobile device and/or the work machine. The autonomy approval function includes a begin mission user interface element displayable after verification that a user has moved within the pre-determined distance of the autonomous work machine, which begin mission user interface element, when selected, causes the autonomy approval function to initiate motion. A computer-implemented method is also provided.

A mobile device includes a processor and memory, coupled to the processor, which memory contains instructions that when executed cause the processor to provide an autonomy management application. The autonomy management application includes an autonomy approval function. The autonomy approval function providing a user interface configured to display at least one aspect of an autonomy job to a user and receive user acceptance of the autonomy job. The autonomy approval function is configured to instruct the user to move about a perimeter of an autonomous work machine that will execute the autonomy job and provide a machine-verified indication of user movement. The autonomy approval function includes a begin mission user interface element displayable after verification (based on data of the mobile device and/or the work machine) that a user has moved about the perimeter of the autonomous work machine, which begin mission user interface element, when selected, causes the autonomy approval function to initiate motion.

The mobile device can be a tablet computer.

User proximity within the pre-determined distance can be detected using electromagnetic radiation.

The autonomy approval function can provide a display indicative of at least one system check performed by the autonomous work machine.

The display can provide an indication of a perception system check.

The display can provide an indication of a positioning system check.

The display can provide an indication of a tractor subsystem check.

The display can provide an indication of an implement subsystem check.

The display can provide a countdown timer indicating time until the autonomous work machine will begin motion.

The display can provide an indication that motion is initiating.

The autonomy approval function can provide a halt user interface element that, when actuated, halts motion of the autonomous work machine.

The autonomy management application can provide a user interface element that, when selected, allows the user to edit the autonomy job.

The begin mission user interface element can require the user to slide the user interface element.

A computer-implemented method of providing autonomy approval for an autonomous work machine can include receiving user approval of an autonomy job for an autonomous work machine and instructing the user to ensure that the user is within a pre-determined distance from the autonomous work machine. The method includes detecting user proximity within the pre-determined distance from the autonomous work machine. The method also includes providing a begin mission user interface element based on detecting user proximity within the pre-determined distance. Upon actuation of the begin mission user interface element, engaging motion of the autonomous work machine.

Detecting user proximity can include detecting electromagnetic radiation relative to at least one of the user and the autonomous work machine.

<FIG> is a block diagram of a control system of an UGV with which embodiments described herein are particularly useful. Control system <NUM> includes controlled vehicle system <NUM> coupled to or including robotic controller <NUM>. Robotic controller <NUM> receives or generates one or more ground operations for the unmanned ground vehicle and provides control signals to controlled system <NUM> to operate the vehicle.

Controlled system <NUM> includes a number of subsystems that operate various functions of the vehicle. Steering subsystem <NUM> provides directional control for the vehicle. In examples where there is no provision for a human driver, the steering subsystem <NUM> may directly control hydraulic or mechanical elements to cause movement elements (e.g. wheels or tracks) to follow a particular course. In another example, when the ground vehicle is equipped with provision for a human operator, steering subsystem <NUM> may include the human operable steering mechanism as well as electromechanical system for engaging the human operably steering system.

Controlled system <NUM> also includes throttle subsystem <NUM> for controlling the speed or power of the prime mover of the system. In examples where the vehicle uses an internal combustion engine, throttle subsystem <NUM> may control the amount of air provided to the internal combustion engine. In examples where the vehicle uses an electric drive, throttle subsystem may provide a control signal that specifies an amount of power to be applied to the electric drive motor(s).

Controlled system <NUM> also includes drive engagement system <NUM>. Drive engagement system <NUM> selectively couples the prime mover to the movement elements. As such, drive engagement system <NUM> may include a clutch and/or hydraulic circuit. Transmission control subsystem <NUM> is also provided and selects different dears or drive ranges. In examples where UGV is a semi-tractor, transmission control subsystem <NUM> may select among up to <NUM> gears.

Controlled system <NUM> may also include a light subsystem <NUM> that may provide lights for illumination and/or annunciation. Such lights may include front illumination lights, rear illumination lights as well as side illumination lights. The illumination lights may provide illumination in the visible spectrum and/or in a spectrum that is detectable by sensors <NUM> coupled to robotic controller <NUM>.

Controlled system <NUM> also may include hydraulics system <NUM> that has a hydraulic pump operably coupled to the prime mover as well as one or more hydraulic valves that control hydraulic fluid flow to one or more actuators of the vehicle. Hydraulics system <NUM> may be used to lift or position objects, such as a bucket or work implement.

Controlled system <NUM> may also include or be coupled to power take-off (PTO) subsystem <NUM>. PTO subsystem <NUM> controls engagement between the prime mover and a PTO output shaft or coupling, which is used by towed implements for power.

Controlled system <NUM> also includes a number of sensors <NUM> germane to the operation of the controlled system. Where controlled system <NUM> includes an internal combustion engine, sensors <NUM> may include an oil pressure sensor, an oil temperature sensor, an RPM sensor, a coolant temperature sensor, one or more exhaust oxygen sensors, a turbine speed sensor, et cetera. In examples where the controlled system includes a hydraulic system, sensors <NUM> can include a hydraulic pressure sensor. In examples where the controlled system is an agricultural harvester, sensors <NUM> can include such sensors as a moisture sensor, mass flow sensor, protein content sensor, et cetera.

Controlled system <NUM> may include one or more additional subsystems as indicated at reference numeral <NUM>.

Robotic controller <NUM> is coupled to controlled system <NUM> as indicated at reference numeral <NUM>. This connection may include one or more system busses out connections such that robotic controller can issue command signals to the various subsystems of controlled system <NUM>. Robotic controller <NUM> may include one or more processors or logic modules that enable robotic controller <NUM> to execute a sequence of instructions stored within memory (either within robotic controller <NUM> or coupled thereto) to enable robotic controller <NUM> to perform automatic or semi-automatic control functions. In one embodiment, robotic controller <NUM> is a microprocessor.

Robotic controller <NUM> is coupled to wireless communication module <NUM> to allow control system <NUM> to communicate wirelessly within one or more remote devices. Wireless communication can take various forms, as desired. Examples of suitable wireless communication include, without limitation, Bluetooth (such as Bluetooth Specification <NUM> rated at Power Class <NUM>); a Wi-Fi specification (such as IEEE <NUM>. a/b/g/n); a known RFID specification; cellular communication techniques (such as GPRS/GSM/CDMA/<NUM> NR); WiMAX (IEEE <NUM>), and/or satellite communication.

Robotic controller <NUM> is also coupled to position detection system <NUM>, which allows robotic controller <NUM> to determine the geographical location of control system <NUM>. In one example, position detection system <NUM> includes a GPS receiver that is configured to receive information from GNSS satellites and calculate the device's geographical position.

Robotic controller <NUM> is also coupled to one or more sensors <NUM> that provide respective signals that allow robotic controller to move the machine to perform a given task. Examples of sensors <NUM> include camera systems that view the area around the machine. These cameras may operate in the visible spectrum and/or above and/or below it. Sensors <NUM> may also include one or more LIDAR sensors that provide a direct indication of distance from the machine to an object in the environment. Sensors <NUM> can also include other types of sensors that provide information indicative of a distance to objects around the machine, such as RADAR and/or ultrasonic sensors. Certainly, combinations of the various sensors set forth above can also be used. Sensors <NUM> may also include audio sensors such that robotic controller can detect sounds (such as nominal operation of the machine, abnormal conditions, and/or human voice).

Robotic controller is also coupled to user interface module <NUM>, which may be coupled to one or more displays in a cab of the machine as well as one or more user input mechanisms, such as buttons, pedals, joysticks, knobs, et cetera. Additionally, or alternatively, user interface module <NUM> may be configured to generate a user interface on a remote user device, such as a laptop computer, smartphone, or tablet. In such instance, user interface module <NUM> may provide user interface information in any suitable format including HTML.

In accordance with various embodiments herein, a method, workflow, and system are provided for an approval sequence which safely and intuitively transfers operational control from a human (supervisor or operator) to an autonomous work machine. This can include a specific workflow that transfers control of an agricultural machine from a supervisor in or out of the cab to the autonomy systems (robotic controller) of agricultural machine. In one example, the system includes an application executing on a mobile device held by a supervisor or operator of the work machine in proximity to the work machine. The application and user interface provide steps to safely and intuitively allow autonomous control to begin. Various aspects include the particular checks and interaction with the user interface of the mobile device and the autonomous machine in an understandable method and/or flow that keeps the supervisor/operator informed and in control, while allowing the autonomous work machine to perform safely at the start of a mission.

<FIG> is a diagrammatic view of an exemplary mobile device <NUM> executing an autonomy management application in accordance with one example. As can be seen, screen <NUM> of mobile device <NUM> is currently showing an active tab "map" <NUM> which, when selected, displays a map of the surrounding area, with an autonomous work machine, illustrated diagrammatically as a tractor <NUM> located on the map. While the mobile device <NUM> illustrated in <FIG> is a smartphone, it is expressly contemplated that any suitable mobile device having wireless communication, and a user input/output mechanism, can be used. For example, mobile device <NUM> can be a laptop computer, a notepad, a PDA, or a dedicated hardware system. The application also includes a number of other inactive tabs including Home tab <NUM>, Plan tab <NUM>, and Analyze tab <NUM>. When in the map mode, the view can be changed to display or not display certain elements of interest. For example, as shown in <FIG>, elements include Fields <NUM>, Equipment <NUM>, Flags <NUM>, and Recent Activity <NUM>. Additionally, as shown in <FIG>, using the application executing upon the mobile device, the user can specifically add a flag to the map by pressing user input mechanism <NUM>.

When the user of mobile device <NUM> selects Plan tab <NUM>, the application transitions to the view shown in <FIG>. In <FIG>, the plan for a <NUM> worklist is shown for a hypothetical farm entitled ACME Farm. In the illustrated plan, various agricultural operations can be selected via tabs <NUM>, <NUM>, <NUM>, and <NUM>. As shown, tab <NUM> displays tillage work for the plan while tab <NUM> will display seeding work for the <NUM> worklist plan. Similarly, tab <NUM> will display application operations for the worklist plan, while tab <NUM> will display harvest operations. In the illustrated example, tab <NUM> has been selected and a number of fields are displayed to the user which require tillage operations. Further, the user has selected field ACME <NUM>, illustrated as having <NUM> acres as indicated at reference numeral <NUM>.

When the user selects a particular field, such as field <NUM> as shown in <FIG>, the application executing on the mobile device transitions to the view set forth in <FIG>. As can be seen, further details with respect to the tillage operation for the selected field are provided in <FIG>. Specifically, the type of operation is a rip till, as indicated at reference numeral <NUM>. Additionally, the tillage depth is set at <NUM> as indicated at reference numeral <NUM>. The work group for the field is illustrated in field <NUM> while the number of autonomous equipment assets assigned to the operation is indicated at field <NUM>. Additionally, guidance field <NUM> indicates guidance relative to the till operation. Should the user wish to edit any of the parameters of the till operation, a user interface element <NUM> can be selected which allows the user to edit the till operation. When the user is satisfied with the parameters for the operation, the user selects user interface element <NUM> indicated as "begin startup" to initiate the autonomous tillage operation. When this occurs, the autonomy startup method and techniques using the user's mobile device begin, and the display transitions to that shown in <FIG>.

<FIG> shows step <NUM> of an autonomy startup procedure for a selected operation in accordance with one embodiment. As shown, a display of the selected field is shown on a map. In the illustrated example, this is a satellite map. Further, the pathing of the autonomous machine is overlaid upon the map view. This pathing information may be provided to robotic controller <NUM>, but is generally considered a priori information relative to the autonomy startup. In the illustrated example, a start location for the operation is indicated at reference numeral <NUM> with a stop location indicated at reference numeral <NUM>. Additionally, some of the pathing information may be color-coded in order to indicate to the user certain types of passes. For example, such passes may include Field Row passes, End Turns, Headland passes, Transitions, and Passables, for example. Additionally, the application may include a further details user interface element <NUM>, that, when actuated, slides panel <NUM> upwardly illustrating additional details relative to the tillage operation. For example, such details may include path details which may further include total length, engaged length, total time, end turns, and area covered. Additionally, details may include machine details and/or autonomy settings. In this view, the user may make changes to machine details and/or autonomy settings as desired. When satisfied with the settings and pathing, the user selects user interface element <NUM> to move to the next step.

<FIG> is a view of the autonomy application executing upon a mobile device in accordance with one embodiment. As shown, the mobile application has transitioned once the user selected user interface element <NUM> (shown in <FIG>) to autonomy startup step <NUM> of <NUM>. In this step, a diagrammatic view of the work machine is shown. In this particular example, the work machine is a tractor <NUM>. It is important to ensure reasonable user proximity to the work machine before motion is initiated. This helps ensure that work machine motion is not started without a responsible user in the proximity of the work machine. In one example, autonomy startup includes determining that a user is within a pre-determined distance of the work machine. This detection of user proximity is preferably done using data of the mobile application and/or the work machine. In a first example, GPS coordinates from the mobile device (obtained using a GPS sensor in the mobile device) are compared with GPS coordinates from the work machine (obtained from position detection system <NUM>) to determine a distance between the two device. If the determined distance is within a pre-determined distance, such as <NUM> feet, the user is deemed sufficiently close to the work machine to begin work machine motion. Another way the user proximity can be detected, is determining whether a WiFi signal from the work machine is detectable by the mobile device. Thus, in this example, if the mobile device can detect the Wifi signal of the work machine, the user is deemed to be sufficiently close to the work machine to begin work machine motion. In yet another example, one or more cameras on the work machine detect the user either standing by the work machine, or walking around it. When the user is detected by a camera of the work machine, the user is deemed sufficiently close to the work machine to begin work machine motion.

In the example shown in <FIG>, the application instructs the user to "walk around the tractor. " When the user is ready to begin this task, the user selects user interface element <NUM>. When this occurs, the view of the application transitions to that shown in <FIG>.

In <FIG>, the user has initiated the perimeter machine check and is walking around the tractor and implement. As the user does this with the mobile device, the robotic controller tracks the user's position using one or more sensors <NUM>. In one example, cameras on the work machine provide images to robotic controller <NUM>, which processes the images to identify the user in the images as the user walks around the work machine. Additionally, the display of the mobile device is preferably updated in substantially real-time to show the user's progress walking around the perimeter. In the example shown in <FIG>, the progress is <NUM>%. Once the user has completed the perimeter check, the autonomy startup sequence transitions to that shown in <FIG>.

Once the perimeter check is complete, the autonomy startup method transitions to the display shown in <FIG>. In this display, one or more front images <NUM> from cameras on the machine are displayed to the user along with one or more implement or rear images <NUM>. Based on the perimeter check and the review of the displayed images, if the user continues to believe that the machine is ready to begin autonomous operation, the user engages input mechanism <NUM> in order to initiate the autonomy mission. In the illustrated example, this input mechanism is a "slide to begin mission" mechanism that requires the user to place his or her finger on element <NUM>, and to slide the element in the direction by arrow <NUM>. This particular input type is helpful to prevent the user from inadvertently from pressing a soft button. Thus, the specific user input to engage autonomy is preferably a more complicated input than a simple button press. When the user engages slide to being user input <NUM>, the autonomy startup method transitions to that shown in <FIG>.

<FIG> is a diagrammatic view of an application executing upon a mobile device during autonomy startup in accordance with one embodiment. In the illustrated example, a diagrammatic view of the work machine is shown, as indicated at reference numeral <NUM>. Additionally, the status of the autonomy systems check is provided to the user as well as a countdown to autonomy startup. In the example shown in <FIG>, the robotic controller is checking the various perception systems of the work machine. These are the sensors and/or systems that allow robotic controller <NUM> to sense the area around it. As set forth above, this may include one or more cameras that operate in the visible spectrum, ultrasonic sensors, LIDAR sensors, RADAR sensors, et cetera. During this test, all such perception sensors are tested to determine if they are functional and working appropriately. Additionally, the application shows diagrammatic views of the various perception sensors that are being tested. For example, <FIG> illustrates a front view <NUM> is being tested along with left and right-side views <NUM>, <NUM>, respectively, and an implement view <NUM>. When all such perception systems have passed the check, a check mark indicated at reference numeral <NUM>. If any of the perception systems did not pass the check, the application indicates an alert, such as an exclamation point or stop sign by the perception text on the screen, as well as an indication of the perception system or systems that did not pass the check. Once the perception system check is completed, robotic controller begins testing the guidance system. When this occurs, the display on the mobile device transitions to that shown in <FIG>.

<FIG> is a diagrammatic view indicating that robotic controller <NUM> is checking guidance systems. Additionally, a countdown timer to autonomy startup is updated to show time left before autonomy begins. The guidance systems include any system that provides a positional signal to the robotic controller. As described in <FIG>, this position detection system <NUM> can take various forms including using GPS information, or other suitable guidance information. In the example shown in <FIG>, all guidance systems have passed the check, and the autonomy checking proceeds to the display shown in <FIG>.

<FIG> has updated the countdown timer to autonomy startup and includes an indication that autonomy systems include checking the various tractor systems. These are the subsystems of controlled system <NUM>, illustrated in <FIG>. Such checks include determining that the prime mover is available (e.g., an internal combustion engine is running) and operating within appropriate parameters. Further, tractor system check may include determining that suitable hydraulic pressure is available, et cetera. If all systems of the controlled system pass the relevant checks, the autonomy startup sequence transitions from that shown in <FIG>, to that shown in <FIG>.

In <FIG>, robotic controller <NUM> checks one or more systems of the implement. This may include determining whether actuators on the implement are actuatable or any other subsystems of the implement. If the implement subsystems check out appropriately, the display on the mobile device changes to that of <FIG>.

<FIG> is a view of an application executing upon a mobile device in accordance with one embodiment. As shown in <FIG>, when the autonomous work machine passes all of the autonomy system checks illustrated above, the work machine will begin initiating motion. This state is indicated to the user on mobile device <NUM> at reference numeral <NUM>. Additionally, the application shows the work machine as flashing or otherwise engaging a number of lights <NUM> on the work machine. Preferably, these lights also flash on the real world work machine. Further, the mobile device also provides a user interface element <NUM> that, when selected, causes the machine to halt operation. For example, the user may determine that some aspect of the operation is incorrect, or may wish to make changes, the user can simply press user interface element <NUM> to halt machine operation. When this occurs, changes can be made to the operation and/or autonomous machine. However, in order for autonomy to begin again, the autonomy approval sequence must be performed again.

<FIG> is a flow diagram of a computer-implemented method of performing autonomy approval in accordance with one embodiment. Method <NUM> begins at block <NUM> where the autonomy equipment is placed in a start position and the implement is setup and ready for work. The start position may be a pre-defined start position indicated on a map, such as position <NUM> (shown in <FIG>). Additionally, status box <NUM> indicates that while block <NUM> is executing, that the autonomy system is disabled. Next, at block <NUM>, the user presses an autonomy switch to enable. This switch may be a user interface element, such as user interface element <NUM> shown in <FIG> relative to a particular field operation.

Once the autonomy switch to enable has occurred, control passes to block <NUM>. Additionally, status box <NUM> indicates that the autonomy system is enabled, and preferably an enable light on a switch is lit. However, as indicated at block <NUM>, the autonomy system is not yet ready and is not in motion. At block <NUM>, the supervisor application executing on the user's mobile device returns a notification that autonomy is enabled and waiting for approval. This is indicated in <FIG> at autonomy startup heading shown at the top of <FIG>. At this step, the user or supervisor clicks or enters the work list displayed therein. For example, the user may view the work list and the map and pathing in order to verify. Further, as indicated at block <NUM>, the user also verifies various work items. At block <NUM> the user has verified the various work items, and the method transitions to the next step, as indicated in <FIG> by pressing soft button <NUM>. When this occurs, method <NUM> transitions to block <NUM> where the user completes the autonomy ready check list.

The first step of the autonomy ready check list is to perform the perimeter check. As set forth above, the user proximity can be detected by instructing the user to walk around the autonomous machine and any implement connected thereto. Preferably, this check is actively tracked by the robotic controller and feedback on the progress on perimeter check is provided in real-time to the user. When the check is complete, the user may get into the cab of the autonomous vehicle or simply walk away, and then send autonomy approval. This may be done by sliding a "slide to begin" mission user interface element, such as element <NUM> shown in <FIG>. When the autonomy approval has been received, robotic controller <NUM> transitions to the state shown at block <NUM> where the autonomy system is ready but not yet in motion. When this occurs, robotic controller <NUM> takes over and reports status back to the user relative to various system checks, as indicated at block <NUM>. For example, the system checks can include perception, guidance, tractor, and/or implement. Robotic controller <NUM> does not initiate any motion whatsoever until all checks are complete. When all checks are complete, and all checked systems provide satisfactory results, then the autonomy system can transition to a ready (motion) state and begin its operation.

<FIG> are diagrammatic views of an exemplary mobile device executing an autonomy management application autonomy startup procedure in accordance with another embodiment. In accordance with the alternate embodiment shown in <FIG>, when a user wishes to begin startup (such as by pressing user interface element <NUM> - shown in <FIG>) of an autonomy operation, a display executing on a user's mobile device may switch to that shown in <FIG>. The display <NUM> of the mobile device depicts an overhead view of the work area with the autonomous work <NUM> centered on the display. In one example, the overhead view is a satellite view obtained from a remote server based on the location of the mobile device and/or autonomous work machine. Additionally, the location of the user is also preferably indicated on the display, as shown at reference numeral <NUM>. As can be seen, in one example, the display informs the user that a "machine walk-around" is required, as indicated at reference numeral <NUM>. When the user selects user interface element <NUM>, the display transitions to that shown in <FIG>.

<FIG> is a diagrammatic screen view of an autonomy approval process running on a mobile device of a user in accordance with one embodiment. As shown, display <NUM> continues to depict the autonomous work machine in the center of the map while diagrammatic versions of fields of view for various cameras are shown, as indicated at reference numeral <NUM>. Additionally, the position of the user is also preferably shown as indicated at reference numeral <NUM>. The application informs the user that he or she must walk around the tractor and implement ensuring that all cameras see the user, as indicated at reference numeral <NUM>. When the user is ready to begin the walk-around, the user selects user interface element <NUM> and display <NUM> transitions to that shown in <FIG>.

<FIG> is a diagrammatic screen view of an autonomy approval process running on a mobile device of a user in accordance with one embodiment. As can be seen in <FIG>, as the user has moved from the position shown in <FIG>, to the position shown in <FIG> (indicated at reference numeral <NUM>), the fields of view <NUM> for the cameras in which the user's image has been detected have been changed. In one example, the fields of view where the user has been detected can have their color changed (e.g. to green, for example). In other examples, the depiction of the field of view could be made to flash or simply indicate "ok. " Regardless, embodiments generally provide substantially real-time feedback of the walk-around process to the user. Additionally, as indicated at reference numeral <NUM>, the user is instructed to "walk around the entire machine. " This ensures two things, first the user will view the entire area surrounding the machine and ensure that the area and autonomous work machine is safe for autonomy to begin. Second, the process also ensures that all cameras of the autonomous work machine are functioning properly. Once all cameras have detected the user during the walk-around, display <NUM> transitions to that shown in <FIG>.

<FIG> is a diagrammatic screen view of an autonomy approval process running on a mobile device of a user in accordance with one embodiment. <FIG> displays after w walk-around process has completed successfully and shows the operator the projected initial path <NUM> of the autonomous work machine. Additionally, the work machine has also completed a self-diagnostic of its internal systems and provides the results of the diagnostic at reference numeral <NUM>. In the illustrated example, all such internal system have passed the diagnostic and thus the work machine is ready for motion. In order to begin motion, the user swipes user interface element <NUM> to the right.

The present discussion has mentioned processors and servers. In one embodiment, the processors and servers include computer processors with associated memory and timing circuitry, not separately shown. They are functional parts of the systems or devices to which they belong and are activated by, and facilitate the functionality of the other components or items in those systems.

<FIG> is a block diagram of an autonomous work machine <NUM> communicating with elements in a remote server architecture <NUM>. In an example embodiment, remote server architecture <NUM> can provide computation, software, data access, and storage services that do not require end-user knowledge of the physical location or configuration of the system that delivers the services. In various embodiments, remote servers can deliver the services over a wide area network, such as the internet, using appropriate protocols. For instance, remote servers can deliver applications over a wide area network and they can be accessed through a web browser or any other computing component. Software or components shown in <FIG> as well as the corresponding data, can be stored on servers at a remote location. The computing resources in a remote server environment can be consolidated at a remote data center location or they can be dispersed. Remote server infrastructures can deliver services through shared data centers, even though they appear as a single point of access for the user. Thus, the components and functions described herein can be provided from a remote server at a remote location using a remote server architecture. Alternatively, they can be provided from a conventional server, or they can be installed on client devices directly, or in other ways.

<FIG> also depicts another embodiment of a remote server architecture. <FIG> shows that it is also contemplated that some elements of <FIG> are disposed at remote server location <NUM> while others are not. By way of example, remote storage (e.g. data store <NUM>) can be disposed at a location separate from location <NUM> and accessed through the remote server at location <NUM>. Regardless of where they are located, they can be accessed directly by the autonomous work machine, through a network (either a wide area network or a local area network), they can be hosted at a remote site by a service, or they can be provided as a service, or accessed by a connection service that resides in a remote location. Also, the data can be stored in substantially any location and intermittently accessed by, or forwarded to, interested parties. For instance, physical carriers can be used instead of, or in addition to, electromagnetic wave carriers. In such an embodiment, where cell coverage is poor or nonexistent, another mobile machine (such as a fuel truck) can have an automated information collection system.

It will also be noted that the elements of <FIG>, or portions of them, can be disposed on a wide variety of different devices. Some of those devices include servers, desktop computers, laptop computers, tablet computers, or other mobile devices, such as palm top computers, cell phones, smart phones, multimedia players, personal digital assistants, etc..

<FIG> is a simplified block diagram of one illustrative embodiment of a handheld or mobile computing device that can be used as a user's or client's hand held device <NUM>, in which the present system (or parts of it) can be deployed. <FIG> are examples of handheld or mobile devices.

<FIG> provides a general block diagram of the components of a client device <NUM> that can run some components shown in <FIG>, to interact with the autonomous work machine. In mobile device <NUM>, a communications link <NUM> is provided that allows the handheld device to communicate with other computing devices and under some embodiments provides a channel for receiving information automatically, such as by scanning. Examples of communications link <NUM> include allowing communication though one or more communication protocols, such as wireless services used to provide cellular access to a network, as well as protocols that provide local wireless connections to networks.

Under other embodiments, applications can be received on a removable Secure Digital (SD) card that is connected to an interface <NUM>. Interface <NUM> and communication links <NUM> communicate with a processor <NUM> along a bus <NUM> that is also connected to memory <NUM> and input/output (I/O) components <NUM>, as well as clock <NUM> and location system <NUM>.

I/O components <NUM>, in one embodiment, are provided to facilitate input and output operations. I/O components <NUM> for various embodiments of the device <NUM> can include input components such as buttons, touch sensors, optical sensors, microphones, touch screens, proximity sensors, accelerometers, orientation sensors and output components such as a display device, a speaker, and or a printer port. Other I/O components <NUM> can be used as well.

Memory <NUM> stores operating system <NUM>, network settings <NUM>, applications <NUM> (such as the autonomy management application), application configuration settings <NUM>, data store <NUM>, communication drivers <NUM>, and communication configuration settings <NUM>. Memory <NUM> can include all types of tangible volatile and non-volatile computer-readable memory devices. It can also include computer storage media (described below). Memory <NUM> stores computer readable instructions that, when executed by processor <NUM>, cause the processor to perform computer-implemented steps or functions according to the instructions. Processor <NUM> can be activated by other components to facilitate their functionality as well.

<FIG> shows one embodiment in which device <NUM> is a tablet computer. In <FIG>, that tablet computer is shown with user interface display screen, which can be a touch screen or a pen-enabled interface that receives inputs from a pen or stylus. It can also use an on-screen virtual keyboard. Of course, it might also be attached to a keyboard or other user input device through a suitable attachment mechanism, such as a wireless link or USB port, for instance. The tablet computer can also illustratively receive voice inputs as well.

<FIG> provides an additional example of devices <NUM> that can be used, although others can be used as well. In <FIG>, a smart phone or mobile phone <NUM> is provided as the device <NUM>.

<FIG> is one embodiment of a computing environment in which elements of <FIG>, or parts of it, (for example) can be deployed. With reference to <FIG>, an exemplary system for implementing some embodiments includes a general-purpose computing device in the form of a computer <NUM>. Components of computer <NUM> may include, but are not limited to, a processing unit <NUM> (which can comprise processor <NUM>), a system memory <NUM>, and a system bus <NUM> that couples various system components including the system memory to the processing unit <NUM>. The system bus <NUM> may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. Memory and programs described with respect to <FIG> can be deployed in corresponding portions of <FIG>.

By way of example only, <FIG> illustrates a hard disk drive <NUM> that reads from or writes to non-removable, nonvolatile magnetic media, a magnetic disk drive <NUM>, an optical disk drive <NUM>, and nonvolatile optical disk <NUM>. The hard disk drive <NUM> is typically connected to the system bus <NUM> through a non-removable memory interface such as interface <NUM>, and magnetic disk drive <NUM> and optical disk drive <NUM> are typically connected to the system bus <NUM> by a removable memory interface, such as interface <NUM>.

For example, and without limitation, illustrative types of hardware logic components that can be used include Field-programmable Gate Arrays (FPGAs), Program-specific Integrated Circuits (e.g., ASICs), Program-specific Standard Products (e.g., ASSPs), System-on-a-chip systems (SOCs), Complex Programmable Logic Devices (CPLDs), etc..

The computer <NUM> is operated in a networked environment using logical connections (such as a local area network - LAN, or wide area network WAN) to one or more remote computers, such as a remote computer <NUM>.

It should also be noted that the different embodiments described herein can be combined in different ways. That is, parts of one or more embodiments can be combined with parts of one or more other embodiments. All of this is contemplated herein.

Claim 1:
A combination comprising an autonomous work machine and a mobile device (<NUM>, <NUM>), the
mobile device (<NUM>, <NUM>) comprising:
a processor (<NUM>);
memory (<NUM>), coupled to the processor (<NUM>), the memory (<NUM>) containing instructions that when executed cause the processor (<NUM>) to provide an autonomy management application, wherein the autonomy management application includes an autonomy approval function;
the autonomy approval function providing a user interface configured to display at least one aspect (<NUM>, <NUM>) of an autonomy job to a user and receive user acceptance (<NUM>) of the autonomy job;
characterized in that the autonomy approval function is configured to instruct the user to move within a pre-determined distance from of the autonomous work that will execute the autonomy job, and provide, based on data of the mobile device (<NUM>, <NUM>)
and/or the work machine, a machine-verified indication of user proximity within the pre-determined distance; and
that the autonomy approval function includes a begin mission user interface element (<NUM>) displayable after verification that a user has moved within the pre-determined distance of the autonomous work machine, which begin mission user interface element (<NUM>), when selected, causes the autonomy approval function to initiate motion.