Abstract:
A system performs stable control of moving devices (such as a helicopter or robot) with attached camera(s), providing live imagery back to a head-mounted computer (HMC). The HMC controls the moving device. The HMC user specifies a desired path or location for the moving device. Camera images enable the user-specified instructions to be followed accurately and the device&#39;s position to be maintained thereafter. A method of controlling a moving device with a headset computer includes analyzing, at the headset computer, at least one image received from the moving device to form an indication of change in position of the moving device. The method displays to a user of the headset computer the indication of change in position of the moving device. The method can additionally include enabling the user to control the moving device.

Description:
RELATED APPLICATIONS 
     This application claims the benefit of U.S. Provisional Application No. 61/749,196, filed on Jan. 4, 2013 and is a continuation-in-part of U.S. application Ser. No. 13/468,207, filed May 10, 2012. The entire teachings of the above applications are incorporated herein by reference. 
    
    
     BACKGROUND 
     Mobile computing devices, such as notebook PCs, smart phones, and tablet computing devices, are now common tools used for producing, analyzing, communicating, and consuming data in both business and personal life. Consumers continue to embrace a mobile digital lifestyle as the ease of access to digital information increases with high-speed wireless communications technologies becoming ubiquitous. Popular uses of mobile computing devices include displaying large amounts of high-resolution computer graphics information and video content, often wirelessly streamed to the device. While these devices typically include a display screen, the preferred visual experience of a high-resolution, large format display cannot be easily replicated in such mobile devices because the physical size of such device is limited to promote mobility. Another drawback of the aforementioned device types is that the user interface is hands-dependent, typically requiring a user to enter data or make selections using a keyboard (physical or virtual) or touch-screen display. As a result, consumers are now seeking a hands-free high-quality, portable, color display solution to augment or replace their hands-dependent mobile devices. 
     SUMMARY 
     Recently developed micro-displays can provide large-format, high-resolution color pictures and streaming video in a very small form factor. One application for such displays can be integrated into a wireless headset computer worn on the head of the user with a display within the field of view of the user, similar in format to either eyeglasses, audio headset or video eyewear. A “wireless computing headset” device includes one or more small high-resolution micro-displays and optics to magnify the image. The WVGA microdisplay&#39;s can provide super video graphics array (SVGA) (800×600) resolution or extended graphic arrays (XGA) (1024×768) or even higher resolutions. A wireless computing headset contains one or more wireless computing and communication interfaces, enabling data and streaming video capability, and provides greater convenience and mobility through hands dependent devices. For more information concerning such devices, see co-pending patent applications entitled “Mobile Wireless Display Software Platform for Controlling Other Systems and Devices,” U.S. application Ser. No. 12/348,648 filed Jan. 5, 2009, “Handheld Wireless Display Devices Having High Resolution Display Suitable For Use as a Mobile Internet Device,” PCT International Application No. PCT/US09/38601 filed Mar. 27, 2009, and “Improved Headset Computer,” U.S. Application No. 61/638,419 filed Apr. 25, 2012, each of which are incorporated herein by reference in their entirety. 
     Embodiments of the present invention provide a way to implement stable remote control of moving devices (such as a helicopter or robot) that have one or more cameras attached, providing live imagery back to the controlling device, a head-mounted or headset computer (HMC or HSC) or other computer device. The user specifies how he wants the moving device to move. This invention uses analysis of the camera images to enable those instructions (user specified) to be followed accurately and the moving device&#39;s position to be maintained thereafter. 
     In one embodiment, a method of controlling a moving device with a headset computer includes analyzing, at the headset computer, at least one image received from the moving device to form an indication of change in position of the moving device. The method further can include displaying to a user of the headset computer the indication of change in position of the moving device. The method can additionally include enabling the user to control the moving device. 
     In another embodiment, the method can enable the user to control the moving device by user input. The user input can be head movement, hand gesture, voice command, or a digital command. 
     In another embodiment, the method can include transmitting at least one directional or angular command to the moving device. 
     In another embodiment, analyzing the at least one image further can include transmitting the at least one image to a host computer to perform the analysis and also include receiving the analysis from the host computer. The method can further include coupling the headset computer and the host computer for communication over a wireless transport. In another embodiment, the method can include coupling the headset computer and the moving device for communication over a wireless transport. 
     In another embodiment, the method can further include displaying, to the user, the at least one images from the moving device. The method can also include overlaying the indication of change of the moving device on the at least one images. 
     In another embodiment, enabling the user to control the device can further include analyzing user input, comparing the user input to a limit of the moving device, and reducing movement commands to the movement device to be within the limit. 
     In another embodiment, enabling the user to control the device can further include sending commands to an autopilot of the motion device. 
     In another embodiment, a system for controlling a moving device with a headset computer can include an analysis module configured to analyze, at the headset computer, at least one image received from the moving device to form an indication of change in position of the moving device. The system can further include a display configured to display to a user of the headset computer the indication of change in position of the moving device. The system can additionally include a control module configured to enable the user to control the moving device. 
    
    
     
       BRIEF DESCRIPTION OF DRAWINGS 
       The foregoing will be apparent from the following more particular description of example embodiments of the invention, as illustrated in the accompanying drawings in which like reference characters refer to the same parts throughout the different views. The drawings are not necessarily to scale, emphasis instead being placed upon illustrating embodiments of the present invention. 
         FIGS. 1A-1B  are schematic illustrations of a headset computer cooperating with a host computer (e.g., Smart Phone, laptop, etc.) according to principles of the present invention. 
         FIG. 2  is a block diagram of flow of data and control in the embodiment of  FIGS. 1A-1B . 
         FIG. 3  is a diagram  3000  showing an example embodiment of a front facing camera view of a motion device (e.g., a helicopter) hovering in front of window. 
         FIG. 4  is a block diagram  4000  illustrating an example embodiment of a method employed by the headset computer. 
         FIG. 5  illustrates a computer network or similar digital processing environment in which the present invention may be implemented. 
         FIG. 6  is a diagram of the internal structure of a computer (e.g., client processor/device or server computers) in the computer system of  FIG. 5 . 
     
    
    
     DETAILED DESCRIPTION 
       FIGS. 1A and 1B  show example embodiments of a wireless computing headset device  100  (also referred to herein as a headset computer (HSC) or head mounted computer (HMC)) that incorporates a high-resolution (VGA or better) microdisplay element  1010 , an image analysis system (analyzer) and other features described below. HSC  100  can include audio input and/or output devices, including one or more microphones, input and output speakers, geo-positional sensors (GPS), three to nine axis degrees of freedom orientation sensors, atmospheric sensors, health condition sensors, digital compass, pressure sensors, environmental sensors, energy sensors, acceleration sensors, position, attitude, motion, velocity and/or optical sensors, cameras (visible light, infrared, etc.), multiple wireless radios, auxiliary lighting, rangefinders, or the like and/or an array of sensors embedded and/or integrated into the headset and/or attached to the device via one or more peripheral ports (not shown in detail in  FIG. 1B ). Typically located within the housing of headset computing device  100  are various electronic circuits including, a microcomputer (single or multicore processors), one or more wired and/or wireless communications interfaces, memory or storage devices, various sensors and a peripheral mount or mount, such as a “hot shoe.” 
     Example embodiments of the HSC  100  can receive user input through sensing voice commands, head movements,  110 ,  111 ,  112  and hand gestures  113 , or any combination thereof. Microphone(s) operatively coupled or preferably integrated into the HSC  100  can be used to capture speech commands which are then digitized and processed using automatic speech recognition techniques. Gyroscopes, accelerometers, and other micro-electromechanical system sensors can be integrated into the HSC  100  and used to track the user&#39;s head movement to provide user input commands. Cameras or other motion tracking sensors can be used to monitor a user&#39;s hand gestures for user input commands. Such a user interface overcomes the hands-dependant formats of other mobile devices. 
     The headset computing device  100  can be used in various ways. It can be used as a remote display for streaming video signals received from a remote host computing device  200  (shown in  FIG. 1A ). The host  200  may be, for example, a notebook PC, smart phone, tablet device, or other computing device having less or greater computational complexity than the wireless computing headset device  100 , such as cloud-based network resources. The host may be further connected to other networks  210 , such as the Internet. The headset computing device  100  and host  200  can wirelessly communicate via one or more wireless protocols, such as Bluetooth®, Wi-Fi, WiMAX, 4G LTE or other wireless radio link  150 . (Bluetooth is a registered trademark of Bluetooth Sig, Inc. of 5209 Lake Washington Boulevard, Kirkland, Wash. 98033.) In an example embodiment, the host  200  may be further connected to other networks, such as through a wireless connection to the Internet or other cloud-based network resources, so that the host  200  can act as a wireless relay. Alternatively, some example embodiments of the HSC  100  can wirelessly connect to the Internet and cloud-based network resources without the use of a host wireless relay. 
       FIG. 1B  is a perspective view showing some details of an example embodiment of a headset computer  100 . The example embodiment HSC  100  generally includes, a frame  1000 , strap  1002 , rear housing  1004 , speaker  1006 , cantilever, or alternatively referred to as an arm or boom  1008  with a built in microphone, and a micro-display subassembly  1010 . 
     A head worn frame  1000  and strap  1002  are generally configured so that a user can wear the headset computer device  100  on the user&#39;s head. A housing  1004  is generally a low profile unit which houses the electronics, such as the microprocessor, memory or other storage device, along with other associated circuitry. The microprocessor is configured for image analysis used to control (e.g. remotely control) a moving vehicle or object according to principles of the present invention. Speakers  1006  provide audio output to the user so that the user can hear information. Microdisplay subassembly  1010  is used to render visual information to the user. It is coupled to the arm  1008 . The arm  1008  generally provides physical support such that the microdisplay subassembly is able to be positioned within the user&#39;s field of view  300  ( FIG. 1A ), preferably in front of the eye of the user or within its peripheral vision preferably slightly below or above the eye. Arm  1008  also provides the electrical or optical connections between the microdisplay subassembly  1010  and the control circuitry housed within housing unit  1004 . 
     According to aspects that will be explained in more detail below, the HSC or HMC display device  100  allows a user to select a field of view  300  within a much larger area defined by a virtual display  400 . The user can typically control the position, extent (e.g., X-Y or 3D range), and/or magnification of the field of view  300 . 
       FIGS. 1A-1B  show an example embodiment of a monocular microdisplay presenting a single fixed display element supported on the face of the user with a cantilevered boom. Other mechanical configurations for the remote control display device  100  are possible. 
       FIG. 2  is a block diagram  2000  showing additional details of an example embodiment of the HSC  100 , host  200  and the data that travels between them. The HSC  100  receives vocal input from the user via the microphone, hand movements  2002  or body gestures via positional and orientation sensors, the camera or optical sensor(s), and head movement inputs  2004  via the head tracking circuitry such as 3 axis to 9 axis degrees of freedom orientational sensing. These are translated by software in the HSC  100  into keyboard and/or mouse commands  2008  that are then sent over the Bluetooth or other wireless interface  150  to the host  200 . The host  200  then interprets these translated commands in accordance with its operating system/application software to perform various functions. Among the commands is one to select a field of view  300  within the virtual display  400  and return that selected screen data to the HSC  100 . Thus, it should be understood that a very large format virtual display area might be associated with application software or an operating system running on the host  200 . However, only a portion of that large virtual display area  400  within the field of view  300  is returned to and actually displayed by the micro display  1010  of HSC  100 . 
     A moving object (for a nonlimiting example of a vehicle) of interest is coupled to the HMC  100  or other computer device  200  (the connection type is not important provided camera images from the object can be streamed to the HMC in real time). The moving object sends images from at least one camera to the HMC  100  or other computer device  200 . The camera may be that of another HMC or HSC  100  worn by a passenger of the moving vehicle, for example, instead of the HMC remotely controlling the vehicle. 
     The HMC  100  or other computer device  200  runs software or otherwise implements an image analyzer that monitors changes in the position of objects in the camera images to calculate whether the moving object/vehicle is correctly following a desired course or maintaining position if no desired course is provided. Where discrepancies are detected between the desired course and the moving object&#39;s current position, the instructions to the moving object/vehicle may be adjusted to compensate and bring the moving object/vehicle into the desired position. 
       FIG. 3  is a diagram  3000  showing an example embodiment of front facing camera views of a motion device (e.g., a helicopter) hovering in front of window. The operator is instructing the helicopter to remain stationary. The HMC  100  or computer host  200  executing the invention software processes the received real time images  304 ,  314  and  324  to detect horizontal and vertical edges, as shown in the analyzed images  306 ,  316 , and  326 . The HMC  100  selects clearly detectable edges and monitors them for movement from frame to frame to determine movement of the helicopter. 
     When the camera is in the correct position  302 , the analyzed image  306 , corresponding with the real time image  304 , shows detected horizontal and vertical edges. The HSC  100  continually monitors the edges in subsequent frames to determine whether the helicopter has moved. 
     After the camera has dropped too low  312 , the detected edges have moved upwards in the real time image  314 , shown by the dotted line and arrow in analyzed image  316 . In response, the analysis system (generally at  100 ,  200 ) causes the helicopter system (or operator) to increase rotor power for a short time to bring the helicopter back up to the required level. 
     After the camera has rotated to the right  322 , the detected edges shift to the left in real time image  324 , shown by the dotted line and arrow in analyzed image  326 . In response, the system/analyzer  100 ,  200  enables the helicopter operator to adjust the tail fin/rotor to rotate the helicopter back to the correct position  302 . 
     If the operator wants to move the helicopter by a small amount, he can specify horizontal and vertical deltas, and in response, the autopilot software of the helicopter adjusts the desired edge positions accordingly. The operator can specify the horizontal and vertical deltas by manual entry (e.g., with a keyboard and mouse) or by hand gesture, head tracking, body gesture, etc. Then the auto-adjustment process (e.g., the image analyzer feeding to the vehicle autopilot system) causes the helicopter to move to the new position to match the desired edge positions. 
     If the operator wants to move the helicopter to a new location entirely, he can temporarily suspend the automatic position control (analyzer of the present invention coupled to helicopter auto pilot) and fly the helicopter using direct control (e.g., a joystick, manual entry, body movement, head tracking, or hand gestures) until the helicopter reaches the desired position. The helicopter can re-engage automatic position control to fine tune the desired position. Further, automatic position control can remain enabled during direct control to maintain orientation and prevent stalling and/or crashing of the helicopter. 
     In one embodiment the HSC  100  may take the form of the HSC described in a co-pending U.S. Patent Publication Number 2011/0187640 which is hereby incorporated by reference in its entirety. 
     In another embodiment, the invention relates to the concept of using a Head Mounted Display (HMD)  1010  in conjunction with an external ‘smart’ device  200  (such as a smartphone or tablet) to provide information and control to the user hands-free. The invention requires transmission of small amounts of data, providing a more reliable data transfer method running in real-time. 
     In this sense therefore, the amount of data to be transmitted over the connection  150  is small and includes instructions on how to lay out a screen, text to display, and other stylistic information such as drawing arrows, background color(s), or images to include, etc. 
     Additional data can be streamed over the same  150  or another connection and displayed on screen  1010 , such as a video stream if required by the Controller  200 . 
     The user can control the motion device by moving his or her head. The HSC tracks these movements using head tracking technology. For example, a user that turns his or her head to the right causes the motion device to turn to the right a proportional amount of degrees. The HSC moves the desired path/position/course in the analyzed image to the right, and then allows the auto-course adjustment to take place as described above. Similarly, the HSC can adjust the course if the user looks up or down. Other movements, such as moving forward or backward, can accelerate or decelerate the motion device. As such, the HSC combines analysis of head tracking movements with images received from the motion device to provide an intuitive, motion based, hands free control of the motion device. 
       FIG. 4  is a block diagram  4000  illustrating an example embodiment of a method employed by the headset computer. The method first analyzes images received from the moving device to form an indication of the change of position of the moving device ( 402 ). The method then displays, to the user of the headset computer, the indication of the change ( 404 ). Then, the method enables the user to control the device by user input ( 406 ). 
       FIG. 5  illustrates a computer network or similar digital processing environment in which the present invention may be implemented. 
     Client computer(s)/devices  50  and server computer(s)  60  provide processing, storage, and input/output devices executing application programs and the like. Client computer(s)/devices  50  can also be linked through communications network  70  to other computing devices, including other client devices/processes  50  and server computer(s)  60 . Communications network  70  can be part of a remote access network, a global network (e.g., the Internet), a worldwide collection of computers, Local area or Wide area networks, and gateways that currently use respective protocols (TCP/IP, Bluetooth, etc.) to communicate with one another. Other electronic device/computer network architectures are suitable. 
       FIG. 6  is a diagram of the internal structure of a computer (e.g., client processor/device  50  or server computers  60 ) in the computer system of  FIG. 5 . Each computer  50 ,  60  contains system bus  79 , where a bus is a set of hardware lines used for data transfer among the components of a computer or processing system. Bus  79  is essentially a shared conduit that connects different elements of a computer system (e.g., processor, disk storage, memory, input/output ports, network ports, etc.) that enables the transfer of information between the elements. Attached to system bus  79  is I/O device interface  82  for connecting various input and output devices (e.g., keyboard, mouse, displays, printers, speakers, etc.) to the computer  50 ,  60 . Network interface  86  allows the computer to connect to various other devices attached to a network (e.g., network  70  of  FIG. 5 ). Memory  90  provides volatile storage for computer software instructions  92  and data  94  used to implement an embodiment of the present invention (e.g., analyzer module, display module, and control module code detailed above). Disk storage  95  provides non-volatile storage for computer software instructions  92  and data  94  used to implement an embodiment of the present invention. Central processor unit  84  is also attached to system bus  79  and provides for the execution of computer instructions. 
     In one embodiment, the processor routines  92  and data  94  are a computer program product (generally referenced  92 ), including a computer readable medium (e.g., a removable storage medium such as one or more DVD-ROM&#39;s, CD-ROM&#39;s, diskettes, tapes, etc.) that provides at least a portion of the software instructions for the invention system. Computer program product  92  can be installed by any suitable software installation procedure, as is well known in the art. In another embodiment, at least a portion of the software instructions may also be downloaded over a cable, communication and/or wireless connection. In other embodiments, the invention programs are a computer program propagated signal product  107  embodied on a propagated signal on a propagation medium (e.g., a radio wave, an infrared wave, a laser wave, a sound wave, or an electrical wave propagated over a global network such as the Internet, or other network(s)). Such carrier medium or signals provide at least a portion of the software instructions for the present invention routines/program  92 . 
     In alternate embodiments, the propagated signal is an analog carrier wave or digital signal carried on the propagated medium. For example, the propagated signal may be a digitized signal propagated over a global network (e.g., the Internet), a telecommunications network, or other network. In one embodiment, the propagated signal is a signal that is transmitted over the propagation medium over a period of time, such as the instructions for a software application sent in packets over a network over a period of milliseconds, seconds, minutes, or longer. In another embodiment, the computer readable medium of computer program product  92  is a propagation medium that the computer system  50  may receive and read, such as by receiving the propagation medium and identifying a propagated signal embodied in the propagation medium, as described above for computer program propagated signal product. 
     Generally speaking, the term “carrier medium” or transient carrier encompasses the foregoing transient signals, propagated signals, propagated medium, storage medium and the like. 
     While this invention has been particularly shown and described with references to example embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the scope of the invention encompassed by the appended claims.