Embodiments of the present disclosure provide a system, method, apparatus and computer-readable medium for teleoperation. An exemplary system includes a robot machine having a machine body, at least one sensor, at least one robot processor, and at least one user processor operable to maintain a user simulation model of the robot machine and the environment surrounding the robot machine, the at least one user processor being remote from the robot machine. The system further includes at least one user interface comprising a haptic user interface operable to receive user commands and to transmit the user commands to the user simulation model, a display operable to display a virtual representation of the user simulation model.

BACKGROUND OF THE INVENTION

Field of the Invention

Exemplary embodiments of the present disclosure relate to a system, method, apparatus, and computer-readable medium for teleoperation. The present disclosure relates more specifically to teleoperation of a remote device.

Description of Related Art

Teleoperation generally includes the operation and use of a system or machine from a distance. Teleoperation is commonly associated with the remote control of robotics and mobile robots. However, teleoperation can be applied to a range of circumstances in which a device or machine is operated by a person from a distance.

BRIEF SUMMARY OF THE INVENTION

In view of the foregoing, it is an object of the present disclosure to provide a system, method, apparatus, and computer-readable medium for teleoperation.

A first exemplary embodiment of the present disclosure provides a system for teleoperation. The system includes a robot machine having a machine body, at least one sensor operable to sense data regarding the machine body and an environment surrounding the robot machine, at least one robot processor operable to maintain a robot simulation model of the robot machine and the environment surrounding the robot machine, and the robot processor operable to communicate with the at least one sensor and to maneuver the machine body in the environment surrounding the robot machine as a function of the sensed data from the at least one sensor. The system further includes at least one user processor operable to maintain a user simulation model of the robot machine and the environment surrounding the robot machine, the at least one user processor being remote from the robot machine, and the at least one user processor operable to communicate with the robot machine. The system still further includes at least one user interface including a user control operable to receive user commands and to transmit the user commands to the user simulation model, a display operable to display a virtual representation of the user simulation model, wherein the user simulation model receives user commands from the user interface and outputs virtual representation updates to the user interface based on the received user commands, and wherein the user simulation model receives sensed data from the at least one sensor and outputs robot virtual representation updates to the user interface based on the received sensed data. The system further includes wherein the robot simulation model receives sensed data from the at least one sensor and outputs robot commands to the machine body based on the received sensed data, wherein the robot simulation model receives user commands from the user interface and outputs robot user commands to the machine body based on the received user commands, wherein the user simulation model receives robot simulation updates from the robot simulation model and transmits user simulation updates to the robot simulation model, and wherein the robot simulation model receives user simulation updates from the user simulation model and transmits robot simulation updates to the user simulation model.

A second exemplary embodiment of the present disclosure provides an apparatus for teleoperation. The apparatus includes at least one processor and a memory storing computer program instructions executable by the at least one processor, wherein the memory with the computer instructions and the processor are configured to cause the apparatus to at least maintain, by the memory, a simulation model, and sense, by at least one sensor, data corresponding to a body of the apparatus and an environment surrounding the apparatus. The apparatus is further configured to transmit, by the at least one processor, the sensed data, and receive, by the at least one processor, a new simulation model, the new simulation model comprising a description of the body of the apparatus and the environment surrounding the apparatus. The apparatus is still further configured to determine, by the at least one processor, an updated simulation model, the updated simulation model comprising a combination of the new simulation model and the sensed data, and moving the apparatus in response to the determined updated simulation model.

A third exemplary embodiment of the present disclosure provides an apparatus for teleoperation. The apparatus includes at least one processor and a memory storing computer program instructions executable by the at least one processor, wherein the memory with the computer instructions and the processor are configured to cause the apparatus to at least maintain, by the memory, a simulation model, and transmit, by the at least one processor, the simulation model to a user interface and a robot machine. The apparatus is further configured to receive, by the at least one processor, sensed data from the robot machine and user inputs from the user interface, and determine, by the at least one processor, an updated simulation model comprising a combination of the received sensed data, the received user inputs, and the simulation model. The apparatus is still further configured to transmit, by the at least one processor, the updated simulation model to the robot machine and the user interface.

A fourth exemplary embodiment of the present disclosure provides an apparatus for teleoperation. The apparatus includes a user interface, at least one processor and a memory storing computer program instructions executable by the at least one processor, wherein the memory with the computer instructions and the processor are configured to cause the apparatus to at least maintain, by the memory, a user simulation model, and display, by the user interface, the user simulation model as a virtual representation of a robot machine and an environment surrounding the robot machine, the virtual representation based on the simulation model. The apparatus is further configured to receive, by the user interface, user inputs to the virtual representation, and change, by the at least one processor, the displayed virtual representation based on the received user inputs. The apparatus is further configured to transmit, by the at least one processor, the user inputs, to receive, by the at least one processor, an updated user simulation model, and to display, by the user interface, the updated user simulation model.

A fifth exemplary embodiment of the present disclosure provides a non-transitory computer-readable medium tangibly storing computer program instructions which when executed on a processor of an apparatus causes the apparatus to at least maintain a simulation model, and sense data corresponding to a body of the apparatus and an environment surrounding the apparatus. The apparatus is further cause to transmit the sensed data, and receive a new simulation model, the new simulation model comprising a description of the body of the apparatus and the environment surrounding the apparatus. The apparatus is still further caused to determine an updated simulation model, the updated simulation model comprising a combination of the new simulation model and the sensed data, and moving the apparatus in response to the determined updated simulation model.

A sixth exemplary embodiment of the present disclosure provides a non-transitory computer-readable medium tangibly storing computer program instructions which when executed on a processor of an apparatus causes the apparatus to at least maintain a simulation model, and transmit the simulation model to a user interface and a robot machine. The apparatus is further caused to receive sensed data from the robot machine and user inputs from the user interface, and determine an updated simulation model comprising a combination of the received sensed data, the received user inputs, and the simulation model. The apparatus is still further caused to transmit the updated simulation model to the robot machine and the user interface.

A seventh exemplary embodiment of the present disclosure provides a non-transitory computer-readable medium tangibly storing computer program instructions which when executed on a processor of an apparatus causes the apparatus to at least maintain a user simulation model, and display the user simulation model as a virtual representation of a robot machine and an environment surrounding the robot machine, the virtual representation based on the simulation model. The apparatus is further caused to receive user inputs to the virtual representation, and change the displayed virtual representation based on the received user inputs. The apparatus is further caused to transmit the user inputs, to receive an updated user simulation model, and to display the updated user simulation model.

The following will describe embodiments of the present disclosure, but it should be appreciated that the present disclosure is not limited to the described embodiments and various modifications of the invention are possible without departing from the basic principles. The scope of the present disclosure is therefore to be determined solely by the appended claims.

DETAILED DESCRIPTION OF THE INVENTION

Embodiments of the present disclosure present a user interface operable to provide a user with a visual representation of a remote device, and a remote device operable to sense its surroundings and communicate with the user interface and a server. Embodiments further include server that maintains a simulation model operable to simulate the remote device, receive sensor data from the remote device, receive user inputs from the user interface, and transmit the simulation model to the user interface. Embodiments of the present disclosure allow a user to remotely control the movement of a remote device in a user interface environment that has nearly zero latency between user inputs and the user's perception that the user inputs have been executed. Embodiments of the present disclosure provide a user interface which includes a virtual representation or simulation model of a remote device operable to control a virtual representation of the remote device and operable to communicate the user inputs to the remote device to be performed by the remote device. Embodiments of the present disclosure provide a remote device operable to move in response to commands from a robot simulation model and in response to sensor data, which overrides commands from the robot simulation in response to a safety threshold.

Embodiments of the present disclosure provide an apparatus, system and method for dynamic teleoperation. Embodiments of the present disclosure provide the ability to correctly compensate for communications delays, allowing a user to drive a representation of a robot machine in real-time.

Aspects of the invention allow a remote user to interact with a virtual representation and control a robot machine in the real world. Embodiments of the virtual representation contain both a visual displayed scene of the robot machine and the environment surrounding the robot machine. Embodiments of the virtual representation also can include data fed back to the user interface from the robot machine. Embodiments of the user interface include any combination of interfaces that correctly mimic the robot machine, including motion bases, haptic feedback, and visual representations. The virtual representation is a visual depiction of a simulation model or mathematical model of the robot machine and the environment surrounding the robot machine which includes the robot machine, robot machine sensors, and physical objects surrounding the robot machine.

Referring toFIG.1, shown is a signaling diagram of an exemplary system suitable for use in practicing exemplary embodiments of this disclosure. Illustrated inFIG.1is teleoperation system100including robot machine102, server104, user interface106, and communication link108. Teleoperation system100includes three reference frames—a virtual representation110, the simulation model112, and the robot machine model114. Each reference frame includes a standard, six degree-of-freedom (6DOF) coordinate system (three angular positions, velocities, and accelerations and three linear positions, velocities, and accelerations) in which kinematics and/or dynamics of an N-DOF, multi-body robot machine are described. In addition, each reference frame includes an N-DOF contact load model to describe environment interaction via a single contact point or multiple contact points. Note that these coordinate systems may be abstracted to arbitrary coordinate systems in order to map between the robot machine model114and the virtual representation110, which do not need to be the same for user workload reduction reasons.

The robot machine model114reference frame describes the physical robot machine102as well as the environment surrounding the robot machine and any physical objects surrounding the robot machine102. The robot machine is operable to perform user-controlled functions and contains a plurality of sensors116operable to sense the orientation and location of the robot machine and the physical environment surrounding the robot machine102. Exemplary sensors116include those found in self-driving cars and the typical construction site geosystems.

The simulation model112reference frame describes a mathematical model of the robot machine102and its surrounding environment. These mathematical models maybe be based on both physics and other phenomenological effects. An optimal state estimator118accompanies the simulation models112to provide real-time system state estimation and to mitigate drift between the simulation model112and the robot machine model114. An exemplary optimal state estimator118includes an extended Kalman filter (EKF).

The virtual representation110reference frame is driven by the simulation model112in the simulation model112reference frame. The virtual representation110provides a display of a “virtual” robot machine and its “virtual” surrounding environment. A user interacts with the virtual representation110with user interface106through user inputs to the user interface106. Embodiments of user interface106includes a display120, user controls122and haptic feedback device124. Embodiments of user interface106through user inputs are operable to maneuver the virtual robot machine in virtual representation110in real-time and to transmit those maneuvers to the robot machine102and the server104. Embodiments of user inputs include the user physically or verbally manipulating or instructing user interface106in a fashion to control forces, torques, motions (linear and rotational), and operational mode settings. The user inputs at the user interface106are transmitted to both server104and robot machine102. The user inputs in combination with the simulation model112and the robot machine model114determine movement of the robot machine102. The transmitted user inputs will travel to robot machine102through communication link108. The time it takes the user inputs to travel to robot machine102through communication link108will be different (e.g., greater than) the time that a user will perceive the maneuvers on the virtual robot machine in the virtual representation110and the time to server104.

The virtual representation110can include an optional motion base and 3D virtual reality similar to that found in modern gaming systems. The displayed virtual representation110includes of virtual objects of the robot machine102and the surrounding environment with full physical feature descriptions, such as trees, rocks, roads, people, and landscapes. As with modern mixed reality systems, the virtual representation110combines live images with simulated 3D objects in a seamless manner. Embodiments of virtual representation110can be displayed in one or more television displays or in an immersive, wrap-around screens would be a set of virtual reality googles with head and eye tracking.

The user interface106includes at least one processor, at least one memory operable to store computer program instructions. User interface106can also include user controls122, a display120, and a haptic feedback device124. The user controls122and the haptic feedback device124can be any combination of joysticks, pedals, or application-specific interfaces (including finger and hand interfaces as in surgical robots). Embodiments of user interface106are operable to allow a user to manipulate the virtual representation110of the robot machine102and also is operable to provide haptic feedback to the user through haptic feedback device124based on movement of the virtual representation110of robot machine102. User inputs from user interface106are transmitted as user command data140to server104and through communications link108to robot machine102.

Embodiments of robot machine102are operable to move and interact with its environment. Robot machine102can include at least one robot motor126, at least one robot processor128, at least one robot memory130operable for storing computer program instructions, a plurality of sensors116, and at least one robot device132. The plurality of sensors116generate sensor feedback data142regarding the environment surrounding robot machine102, the location of robot machine102relative to the surrounding environment and the state, status, and location of the at least one robot device132. The sensor feedback data142may be of the robot machine102itself or of the surrounding environment and may come from any combination of measurement devices, such as Global Positioning System (GPS) antennas, inertial measurement units (IMUs), imaging sensors, and load sensors. Embodiments of the at least one robot device132can include any combination of moveable arms, claws, digging devices, holding devices, gripping devices, tools and the like. Robot machine102can also include propulsion elements134, which includes any combination of wheels, treads, axles and the like that are operated by the at least one robot motor126and are operable to move the robot machine102. The robot machine102further includes at least one robot transmitter136and at least one robot receiver138operable to transmit and receive communications to and from the user interface106and the server104.

The at least one robot memory130is operable to store computer program instructions including the robot machine model114and any received data from the plurality of sensors116, the user interface106, and the server104. The at least one robot processor128when executing the data stored on the at least one robot memory130is operable to cause the at least one robot motor126, the propulsion element134and the at least one robot device132to move and perform as directed.

The sensor feedback data from the plurality of sensors116is transmitted from robot machine102to the server104via communication link108. The data transfer through communication link108results in a time delay between when the data is transmitted by robot machine102and when it is received by server104. After passing through communications link108, the sensor feedback data interacts with the simulation model112. The simulation model112and the sensor feedback data142are processed by the optimal state estimator (OSE)118, which is operable to recursively estimate the “state information” of the slave robot and its surrounding environment given the noisy, time-delayed data through communication link108. Embodiments of the “state information” of the robot machine102includes any set of variables that provide a complete representation of robot machine102at an instant in time. Embodiments of “state information” includes both time-varying quantities (positions, velocities, accelerations, loads, time delays) and constant quantities (model parameters, geometric parameters, inertial parameters). The OSE118is operable to be used to avoid quasi-steady-state drift between the simulation machine model114and the simulation model112.

User command data140, sensor feedback data142, and state estimation data144drive the simulation model112including the simulation model112robot and the environment surrounding the robot in the simulation model112reference frame. The OSE118estimates all positions, velocities, and accelerations for the given models, as well as the communications time delays from communications link108. It should be noted that any other pertinent quantities of robot machine102may be estimated during this process as well, such as higher-order time derivatives, constant parameters, model parameters, geometric parameters, inertial parameters, forces, positions, velocities, accelerations, loads, time delays, and torques acting on or from robot machine102. In addition, the OSE118removes any extraneous data or noise present in the sensor feedback data142and provides state information about the robot machine102that may not be directly measured by the plurality of sensors116. This results in several “virtual sensors” or extra data not necessarily present in the sensor feedback data142for robot machine102, which can be used to provide enhanced situational awareness to the user through virtual representation110. The OSE118is operable to update the virtual representation110in real-time depending on information such as the desired robot machine102tasks, known environment, or sensor noise levels and dynamics.

As the simulation model112describing both the robot machine102and its surrounding environment are recursively updated via sensor feedback data142and the OSE118, simulation model112information is transmitted to the user interface106. The simulation model112may undergo a coordinate system mapping in order to reduce the processing required by the user interface106to create the virtual representation110based on the simulation model112. This results in the virtual representation110including a virtual robot machine data and virtual environment data with which the user can interact with through user interface106.

In addition, a set of learning system data146may be transmitted to the user interface106to supplement the simulation model112. The learning system data146includes a set of historical and task-dependent data related to the current tasks robot machine102in the current surrounding environment. The information from the simulation model112may be compared to this historical data set and pertinent behavioral or status data may be communicated to the user to assist and coach in the operation of robot machine102. For example, the learning system data146can assist the user in understanding the difficulties associated with operating a robot machine102in the presence of a time delay between when a user inputs their commands into a user interface106and when they are actually carried out by robot machine102. Indicators such as haptic feedback124or visual cues from display120may be used to communicate this information to the user at user interface106.

In an alternative embodiment, robot machine102may include one or a plurality of robots that can coordinate and share their data from their sensors in a network to supplement the information in the simulation model112reference frame. All of the sensor data from the plurality of robots can be combined with the learning system data146to create a network that can benefit multiple users simultaneously that are operating multiple robot machines in a similar or the same physical environment.

The learning system data146and the simulation model112, which include a mathematical model of the robot machine102and its surrounding environment are used as the basis for the virtual representation110that the user receives and interacts with at the user interface106. The virtual representation110that is displayed by display120and felt by the user through haptic feedback124allow the user to feel as though they are present on the physical robot machine102. Communications delay effects from communications link108are mitigated in teleoperation system100as the OSE118estimates and compensates for the time delays. The time delays between transmitted data between the robot machine102, server104and user interface106can be estimated by OSE118. Also, the higher-order motion derivatives of robot machine102can be estimated as well by OSE118. This provides a “lead prediction” capability to the user, giving the user a sense for how the robot machine102is moving as time evolves. The ability to predict where the robot machine102will be at the next time step is critical to compensating for communications delays. Note that this information may also be used in a feed-forward sense, allowing the user to compensate for known disturbances in the system and environment.

As the user interacts with the virtual representation110with minimal delay or no delay between when the user makes a user input and when the user perceives the execution of the user input in the virtual representation110, the user is able to user command data140to the robot machine102such that the robot machine102can successfully perform tasks and interact with its environment.

Additionally, the robot machine102contains local intelligence in the form of its own robot machine model114, which is a mathematical representation of the robot machine102and the environment surrounding the robot machine102. This on-board mathematical model includes sensor feedback data142and a version of the simulation model112, and performs the same tasks as the simulation model112. This robot machine model114serves as a supervisory system and does not suffer the delays of the communications link108since no data needs to be transmitted over the communications link108. The simulation model112and the robot machine model114are constantly synchronized over the communications link108. This synchronization monitors the time evolutions of the simulation model112and the robot machine model114and can contain additional information beyond what the user controls can provide, such as feed-forward data, time delay data, or virtual sensor data.

The robot machine102robot machine model114also assists in avoiding unintended collisions and other unintended effects with the surrounding environment. This relies on annotated user commands that contain information about the virtual representation110and the intended effect of the user commands in when executed by the robot machine102. For example, the virtual representation110indicates contact with the environment that enables the robot machine102to continue to move past a sensed contact, impending or actual, depending on the sensors used to control safety. Embodiments include wherein computationally complex cognitive functions, such as 3D world mapping and scene annotations, are performed by one or both of the server104or robot machine102. Embodiments also include that the portion of the simulation model112and/or the robot machine model114encompassing the environment surrounding the robot machine102include information from other sources such as a robot network or past learning data.

Embodiments of the present disclosure provide a system or apparatus wherein the user does not feel the effects of the communications channel latency and operates in a feature rich virtual user interface. The richness of the virtual representation is possible because it does not rely on real-time sensor back from a remote robot machine that would be limited in content by communications link bandwidth. Rather, the user interacts with the virtual user interface and operates the virtual representation of the robot machine in real-time. Additionally, the bandwidth of the sensors on the robot machine themselves is not a factor in the visual, auditory, inertial, and load feedback the user receives at the user interface. These are locally synthesized with virtually zero lag. The end result is that the user can operate the representation of the robot machine at the peak of his/her dynamic capabilities unencumbered by the communications link lag.

In practice, a user interacts with the virtual representation110through user interface106and operates a virtual representation of simulation model112, which is a virtual representation of robot machine102and its surrounding environment. When the user inputs a user command (e.g., movement of the virtual representation of robot machine102), the virtual representation110in real-time moves the virtual representation of robot machine102as instructed by the user. Simultaneously, the user command is transmitted over communication link108via communication line148to robot machine102. The user command is also sent to server104via communication line150. The robot machine102combines the robot simulation model114, sensor feedback data142from the plurality of sensors116having information regarding the status of the robot machine102and the surrounding environment to determine an updated robot machine model114. The robot machine102with the at least one robot processor128and at least one robot memory130will determine whether there is a difference between the status of the robot machine102and the determined updated robot machine model114. If there is a difference, the at least one robot processor128will cause the robot machine102to move such that the robot machine102is configured the same as the updated robot machine model114and no difference exists.

The robot machine102will also transmit the sensor feedback data142from the plurality of sensors116to server104through communications link108via communication line152. The robot machine102also transmits a copy of the robot machine model114(including updates to the robot machine model114) to server104via communication line152through communications link108.

Upon receipt of the robot machine model114and sensor feedback data142, server104will, in combination with the OSE118, combine the simulation model112, the sensor feedback data142, and user commands to determine an updated simulation model112. The updated simulation model112will be transmitted to the robot machine102through communications link108via communication line154. The simulation model112is also sent to the user interface106via communication line156. The received sensor feedback data142is sent from server104to user interface106via communication line158.

Reference is now made toFIG.2, which illustrates a simplified block diagram of the devices suitable for use in practicing exemplary embodiments of this disclosure. InFIG.2, server204is adapted for communication over communications link208with robot machine202. Similarly, server204is adapted for communication with user interface206directly. Server204can communicate with user interface206and robot machine202through wired connections, wireless connections, or a combination of both. Embodiments of server204include a single server or a plurality of servers.

Server204includes its own processing means such as at least one processor256, storing means such as at least one computer-readable memory258storing at least one computer program260. Server204also includes communicating means such as a transmitter262and a receiver264for bidirectional wired or wireless communication with other devices shown inFIG.2. Server204may communicate directly with user interface206through communication line266. Server204is operable to communicate with robot machine202through communication link208via wired and/or wireless connections through communication lines268and270.

User interface206includes its own processing means such as at least one processor207, storing means such as least one computer-readable memory209storing at least one computer program211. User interface206also includes communicating means such as transmitter272and a receiver274for bidirectional wireless or wired communication with server204through communication line266. User interface206is also operable for communication to robot machine202through communication line276, communication line270, and communications link208. User interface206includes a display220operable for displaying a virtual representation and user controls222for interacting with and manipulating the virtual representation. User interface206is operable to provide haptic feedback to a user through haptic feedback device224.

Robot machine202includes its own processing means such as at least one processor228, storing meanings such as at least one computer-readable memory230storing at least one computer program233, and communicating means such as transmitter236and receiver238for bidirectional wireless communications with server204. It should be appreciated that embodiments include both wireless and wired communication. Robot machine202is also operable to receive communications from user interface206through communication line276,270and communications link208. Robot machine202includes at least one robot device232. Embodiments of the at least one robot device232includes any combination of moveable arms, claws, digging devices, holding devices, gripping devices, tools and the like. Robot machine202includes a plurality of sensors216operable to sense the location and status/state of the physical robot machine202body and the environment surrounding the robot machine202. Robot machine202includes a motor226and propulsion elements234such as any combination of wheels, treads, tracks, jet engines, and propellers. Motor226is operable to provide power to and move propulsion elements234and robot device232. In some embodiments, motor226may include at least two motors.

Computer program instructions211,260,233are assumed to include program instruction that, when executed by the associated processor207,256,228enable each device to operate in accordance with embodiments of the present disclosure, as detailed above. In these regards, embodiments of this disclosure may be implemented at least in part by computer software stored on computer-readable memories258,209,230which is executable by processors207,256,228, or by hardware, or by a combination of tangibly stored software and hardware (and tangibly stored firmware). Electronic devices implementing these aspects of the disclosure need not be the entire devices as depicted inFIG.1andFIG.2, but embodiments may be implemented by one or more components of same such as the above described tangibly stored software, hardware, firmware and processor, or a system on a chip, an application specific integrated circuit ASIC or a digital signal processor.

Various embodiments of the computer readable memory209,258,230include any data storage technology type which is suitable to the local technical environment, including but not limited to semiconductor based memory devices, magnetic memory devices and systems, optical memory devices and systems, fixed memory, removable memory, disc memory, flash memory, DRAM, SRAM, EEPROM and the like. Various embodiments of the processors207,256,228include but are not limited to general purpose computers, special purpose computers, microprocessors, digital signal processors (DSPs) and multi-core processors.

Reference is now made toFIG.3, which presents a logic flow diagram in accordance with an embodiment of a user interface as described herein. An exemplary user interface begins at block302by receiving a simulation model from a server. Next, at block304the user interface displays a virtual representation of the simulation model. At block306, the user interface receives user inputs. The user inputs can be received through user controls for a user's hands or through oral commands. At block308, the user interface manipulates the virtual representation in response to the user inputs. Embodiments provide that the user interface is operable to provide haptic feedback to the user in response to the user inputs. For example, the virtual representation may display a virtual robot machine and a rock in the environment surrounding the robot machine. The manipulation of the virtual representation may cause the robot machine to contact the rock displayed by the user interface. In this instance, the user interface is operable to provide haptic feedback to the user in response to the robot machine contacting the rock. Embodiments of haptic feedback include the user interface applying forces, vibrations, or motions on the user through the user controls or in a user seat.

Next, at block308the user interface manipulates the virtual representation in response to the user inputs. Embodiments of manipulating includes moving the virtual robot machine throughout the virtual environment and/or moving portions of the virtual robot machine (e.g., a robot tool, shovel, hoe, etc.). At block310, the user interface transmits the user inputs to a robot machine and to a server. At block312, the user interface receives an update simulation model from the server. The process is then repeated at block304except for the fact that the user interface now displays the updated simulation model rather than the simulation model. It should be appreciated that embodiments of the present disclosure provide that user interface receive an updated simulation model at any point during the process set forth inFIG.3. In this regard, block312can occur between blocks304and206, between blocks306and308, and between blocks308and310.

Referring now toFIG.4, shown is a logic flow diagram in accordance with an exemplary server. Beginning at block402the server receives user inputs from a user interface and receives robot machine sensor information from a robot machine. It should be appreciated that embodiments provide that the server either receive the user inputs and the sensor information simultaneously or at different times. Next at block404, the server compares the user inputs and the robot machine sensor information to a simulation model to determine an updated simulation model. It should be understood that embodiments of server are operable to maintain a simulation model that can be amended, changed, and/or updated by the server. At block406, the server transmits the updated simulation model to the user interface and to the robot machine. Following block406, the process can be repeated by beginning again at block402.

Referring toFIG.5, shown is an exemplary logic flow diagram in accordance with an exemplary robot machine. Beginning at block502, the robot machine receives user inputs from a user interface and a simulation model from a server. It should be appreciated that embodiments provide that the robot machine can either receive the user inputs and the simulation model simultaneously or at different times. It should also be appreciated that embodiments of robot machine are operable to maintain a robot simulation model, which is a mathematical representation of the body of the robot machine and the surrounding environment. Next at block504, the robot machine determines a difference between the user inputs, a simulation model and the robot simulation model. At block506, the robot machine moves in response to the determined difference. At block508, the robot machine sensors sense a location and status of the body of the robot machine and the environment surrounding the robot machine. If should be appreciated that embodiments include block508occurring at any point in the process depicted inFIG.5. Next at block510, the robot machine transmits the sensor information and the determined difference to a server. The process can then be repeated by returning to block502.

It should be appreciated that embodiments of the present disclosure provide that the processes described inFIGS.3,4, and5can occur simultaneous but not necessarily at the same rate. For example, the processes set forth inFIG.3can occur 2 or more times faster than the processes set forth inFIG.5.