REMOTE OPERATION OF UNMANNED VEHICLE USING HOSTED WEB SERVER

A system for remote operation of an unmanned vehicle (UV) includes a control station having a communication link to the UV and executing a web browser for (1) receiving web pages of a UV control web application, (2) rendering the web pages on the control station, (3) receiving control input from an operator via rendered web pages, and (4) generating messages on the communication link based on the control input. A hosted server system is coupled to the control station via the communication link and to internal subsystems of the UV for flight control, mission control, navigation, and system monitoring, and (1) executes a web server serving the web pages of the UV control web application to the control station for the above operations of the web browser, and (2) provides back-end controls to the internal subsystems based on the control input from the operator.

BACKGROUND

The invention relates to the field of remote operation of unmanned vehicles, including but not limited to unmanned aerial vehicles. The present description focuses on the UV application in particular, but those skilled in the art will appreciate the applicability to other types of unmanned vehicles.

SUMMARY

In known arrangements for remotely operating an unmanned vehicle (UV), specific UV control software is installed on a ground device that controls the UV. Traditional remote-control technologies provide a low-level messaging format that must be sent outside of an air vehicle boundary for processing and display in proprietary software. It is also necessary to support configuration management on the control software, to provide for updating as functions or configurations of the target UV(s) change.

In a disclosed approach, a “hosted” Web Server processing unit is used, i.e., one that is located away from the ground-based control device. In one example, the web server is an embedded server installed into the UV. The control software is installed on the embedded webserver that interfaces with the air vehicle systems (e.g., flight controller, aircraft subsystems, mission systems, payload and external interfaces). A remote operator with network connection has access to the control software via a Web Browser and has the ability to fly and execute autonomous mission sets.

This solution shifts the paradigm from hosting the human machine interface control software on a remote controller to hosting the control software elsewhere in the system, such as on the controlled vehicle. Other example hosting arrangements are also described.

The Web Server manages protocol services for communicating with the air vehicle subsystems by utilizing Input Output (I/O) protocols. In one embodiment, the Web Server obtains real time status feedback and tasking capability over the air vehicle subsystems from its protocol service interactions. The Web Server allows for an operator to connect to the system using a web browser through the data link. Once the operator's web browser is connected, the embedded web server publishes air vehicle subsystem telemetry allowing real time status updates within the operator's browser. The web server defines an application interface (API) for tasking state and command data to the air vehicle. Operator web browser interactions transmitted to the server are processed and routed as I/O commands to the autopilot and subsystems.

The following are potential advantages of the disclosed technique:It provides remote control capability from any device with a web browser. There is no need for custom ground control software to be installed on an operator's computing device. Traditional remote-control technologies provide a low-level messaging format that must be sent outside of the air vehicle boundary for processing and display in proprietary software.Because the UV hosts the control software, the remote operator computer software does not need to updated as functions or configurations change. This is because the full capability of the UV control software in enabled by the web browser. Because of this there is no requirement to support configuration management on the ground control software.The webserver is able to define a complete API set based on available aircraft subsystem services at a single connection point.The webserver may adopt standards, such as W3C Open web platform, that provide rich interactive experiences on any device.Remote control from any computer device with a web browser is possible. Loading proprietary software on the computer device is not required.

The solution is disclosed in the context of an unmanned air vehicle system, but the concept may be more broadly applicable to remotely operated unmanned vehicles generally. Additionally, although control of the UV (e.g., flight control) is a key aspect of operation, the solution can also be used for other functions such as monitoring, maintenance activities, payload control, etc.

DETAILED DESCRIPTION

In one aspect, a system is disclosed for remote operation of an unmanned vehicle (UV), such as an unmanned aerial vehicle. The system includes a control station used by an operator and having a communication link for controlling operation of the UV, the control station executing a web browser capable of (1) receiving web pages of a UV control web application, (2) rendering the web pages on the control station, (3) receiving control input from the operator via rendered web pages, and (4) generating corresponding messages (e.g., hypertext messages) on the communication link to the UV based on the control input. The system further includes a hosted server system coupled to the control station via the communication link and to internal subsystems of the UV for control. The hosted server system (1) executes a web server serving the web pages of the UV control web application to the control station for the rendering, receiving and generating operations of the web browser, and (2) provides back-end controls to the internal subsystems based on the control input from the operator via the web application.

FIG.1shows a system block diagram. Overall, the system includes a control station10(typically ground-based) coupled to a hosted server system12via a first communications link14, and a set of UV subsystems16(of a UV, not depicted) coupled to the hosted server system12via a second communications link18. The UV subsystems16of the UV may include for example an autopilot, engine speed control (ESC), engine control unit (ECU), microcontroller, GPS, and radio. As indicated, in one embodiment the hosted server system employs a web server located on the UV, i.e., co-located with the UV subsystems16, in which case the second link18is typically a localized communications link of the UV (e.g., a LAN). Alternatively, the web server of the hosted server system12may be located elsewhere, e.g., in a data center or other ground location. In one embodiment the web server is hosted in a so-called “cloud” server, i.e., one of a collection of servers used with other computing infrastructure to provide a generalized computing service to external client-type systems such as the control station10. These two arrangements are described in additional detail below.

FIG.2shows a more detailed block diagram according to a first general arrangement in which the hosted server system12is deployed partly in a ground data terminal (GDT)20interposed between the control station10and a UV22. The control station10executes a browser24which may be of a generally known type (e.g., Chrome® or Edge®), and in operation the browser24handles both UV control (CNTL) content26as well as GDT control content28. The GDT20includes an embedded computer (E-COMP)30having a hosted web server32and a proxy client34, along with GDT subsystems (SUB-SYSs)36such as datalinks and antenna pointing components. The UV22is shown as including an autopilot (A/P)38, UV subsystems40, and an embedded computer42executing a proxy server44.

In operation, the web server32of GDT30serves browser compatible content such as JavaScript, HTML, static files, etc. to the browser24of the control station10using browser compatible protocols such as HTTP/HTTPS, Web sockets, WebRTC, etc., for local rendering to a user/operator at the control station10. The UV control content26served by the web server32can include pages for mission planning, launch, mission control, navigation control, health/status monitoring, and landing. Served GDT control content28can include antenna pointing controls and datalink management. The operator uses control features of rendered pages (e.g., buttons, selection menus, etc.) to issue UV control commands and GDT control commands without the need for any proprietary software in the control station10. Control-related messages from the browser24are provided to the proxy client34which reissues them to the proxy server44of UV22, which in turn translates the commands from the web API into component specific commands for the UV subsystems40. Telemetry and responses from the UV subsystems40are translated back via the proxy server44, proxy client34and web server32to be provided to the browser24. Example UV subsystems40include flight controls, mission controls, and navigation controls.

It will be appreciated that the control station10and E-COMPs30,42are all computerized devices generally including one or more processors, memory, and I/O interface circuitry on one or more system buses, along with local nonvolatile secondary storage, with communications-type interfaces for the external communications links and other interfaces or specialized I/O circuitry as may be needed. The computerized devices store and execute computer program instructions to realize functionality as described herein.

FIG.3shows a detailed block diagram according to a second general arrangement in which the hosted server system12is located entirely on the UV22itself, being in direct communication with the control station10as well as with the local UV subsystems40(i.e., no proxy client34and proxy server44are employed). Overall functionality may be similar to that ofFIG.2as described above, i.e., serving and use of web pages, etc., but done more directly and omitting the proxying operations of that arrangement.

FIG.4is a screenshot of a main page (home page)50of the web application served by the web server32. It includes various components overlaid onto a page background52, which may be for example a map of an operating area of the UV22. The components can include controls for UV commands54and automated wizards56for guiding users through aircraft activities such as maintenance, launching, landing, and reconfiguration. The application can also support built in testing (BIT), alert monitoring58, telemetry and status60, as well as tasking subsystem components such as powering aircraft payloads62.

FIGS.5A and5Bshow other example pages70,72of the web application. The page70may be a specialized page used for troubleshooting system issues, containing a set of alerts74and troubleshooting indicators/controls76. The page72displays mapping tiles and digital elevation information78also served to the browser.

FIG.6shows an example mission planning page80providing an ability to plan and execute flight plans82and execute vehicle specific commands. TheFIGS.84,86and88are planned flight trajectories/paths drawn by an operator (over an operating area appearing in the page background, not depicted) in the flight planning process.

FIG.7illustrates an operation example. The UV22is shown as including the web server32as well as a VTOL engine speed control (ESC) service90and a VTOL motor ESC92. Enumerated process steps include the following:1. A user at control station10requests a page: e.g., Vertical Takeoff & Landing (VTOL) System Page942. The webserver32serves browser supported content (HTML, JavaScript, etc.) via browser supported protocols (e.g., HTTP/HTTPS/WebSocket/WebRTC)3. The user interacts with the browser to send a command (e.g., Arm VTOL motors)4. The web server32passes command to system specific service using inter-process communication or internal web server service: e.g., Translated VTOL motor command sent over DDS5. Corresponding service translates the Web API command to a specific format of the target system component: e.g., Arm motor CAN message sent to the VTOL motor engine speed controllers (ESCs)90,926. System component response with vender specific API: e.g., VTOL motor state via a CAN message7. Interfacing software service translates the vender specific format and communicates with the web server32via the webserver API, inter-process communication, or function calls as an internal service: e.g., VTOL motor state telemetry via DDS8. The webserver32sends the response from the command from the UV22to the user via browser supported protocols (HTTP/HTTPS/WebSocket/WebRTC). Served web content updates the user interface page54with the updated system state: e.g., Displaying VTOL motors “ARMED”