Patent Description:
User interaction with applications executed on mobile electronic devices such as smartphones, tablets, notebooks, or the like generally occurs by a user using a finger or stylus to select keyboard characters displayed on a touchscreen display or as buttons on a standalone computer keyboard. Modern personal computers can be configured with a graphical user interface (GUI), speech-to-text, time of flight (ToF) camera for gesture recognition, and/or other peripheral or input/output devices for non-voice modes, e.g., keyboard, touchscreen, mouse. Such computer accessories can provide users with different computer input modes. As automobiles are increasingly deployed with computer technology, it is desirable for the computer technology to be accessible by a combination of touchscreen computers, driver monitoring camera sensors for eye gaze tracking, ToF cameras for gesture recognition, voice-activated commands, and/or other modes of interaction.

The document <CIT> discloses a method of input to a computer in a vehicle.

The invention is defined by the method of claim <NUM>.

In various embodiments, the display of the second electronic device includes a heads-up display, touch-sensitive windshield, or other device for projecting or displaying information on a windshield or other display medium of a vehicle.

In various embodiments, the first input mode is a first touchpad computer input in response to a receipt of a first amount of pressure applied to the touch-sensitive display of the first electronic device and the at least one second input mode includes a second touchpad computer input in response to a second amount of pressure applied to the touch-sensitive display of the first electronic device greater than the first amount of pressure.

In various embodiments, the multimodal application homologically maps positions on a display of the first electronic device to a virtual keyboard displayed by the second electronic device at a windshield or other display medium of the vehicle.

In various embodiments, the method further comprises performing a swipe typing operation that includes establishing a starting point on the second electronic device by activating a location on the display of the first electronic device that corresponds to the starting point on the second electronic device.

In various embodiments, the first input mode is an input from a touch-sensitive keyboard or display of a touchpad computer.

In various embodiments, controlling the display of the first electronic device comprises: displaying a list of items or group of objects on the display of the second electronic device; activating a cursor at the display of the second electronic device; moving the cursor to an item of interest of the list of items or an object of interest of the group of objects in response to a first input to the first electronic device; and selecting the item of interest or object of interest in response to a second input to the first electronic device.

In various embodiments, the first electronic device includes two electronic devices, wherein a first of the two electronic devices receives the first input and in response provides a first output of the first input mode to the second electronic device, and wherein the second of the two electronic devices receives the second input and in response provides a second output of the first input mode to the second electronic device.

In various embodiments, the first electronic device includes a touch-sensitive display where a swipe typing operation is performed for moving the cursor to the item of interest.

In various embodiments, the first electronic device captures an address and the multimodal application processes the address for use by a navigation system, which in response outputs navigation information related to the address to the second electronic device.

In various embodiments, the first input mode is a camera that provides a digital photograph, text, or graphics from a document to an analysis system to determine the address.

In various embodiments, the first electronic device is at least one of a camera, touchpad computer, a speech interface, an eye gaze detection device, or a tactile input device.

In various embodiments, the method further comprises identifying objects in a field of view of the user, and determining by the multimodal application the address from the identified objects.

The invention is also defined by the subject-matter of the independent system claim <NUM>.

In various embodiments, the multimodal input processing system further includes a processor that executes the multimodal application to homologically map positions on a display of the first electronic device to a display by the second electronic device at a windshield or other display medium of the vehicle.

In various embodiments, the multimodal input processing system further includes a processor that identifies a starting point on the second electronic device by detecting an activated location on the display of the first electronic device that corresponds to the starting point on the second electronic device.

In various embodiments, the multimodal input processing system further comprises a processor that controls the display of the first electronic device by activating a cursor at the display of the second electronic device; moving the cursor to a location of interest in response to a first input to the first electronic device; and selecting the item of interest or object of interest in response to a second input to the first electronic device.

In various embodiments, the multimodal input processing system further comprises a processor that receives an address captured by the first electronic device and processes the address for use by a navigation system, which in response outputs navigation information related to the address to the second electronic device.

In various embodiments, the multimodal input processing system further comprises a processor that receives data from the first electronic device that identifies objects in a field of view of the user; and determines the address from the identified objects.

The present invention will become more apparent in view of the attached drawings and accompanying detailed description. The embodiments depicted therein are provided by way of example, not by way of limitation, wherein like reference numerals refer to the same or similar elements. The drawings are not necessarily to scale, emphasis instead being placed upon illustrating aspects of the invention. In the drawings:.

Various aspects of the inventive concepts will be described more fully hereinafter with reference to the accompanying drawings, in which some exemplary embodiments are shown. The present inventive concept may, however, be embodied in many different forms and should not be construed as limited to the exemplary embodiments set forth herein.

These terms are used to distinguish one element from another, but not to imply a required sequence of elements. For example, a first element can be termed a second element, and, similarly, a second element can be termed a first element, without departing from the scope of the present invention. The term "or" is not used in an exclusive or sense, but in an inclusive or sense.

Spatially relative terms, such as "beneath," "below," "lower," "above," "upper" and the like may be used to describe an element and/or feature's relationship to another element(s) and/or feature(s) as, for example, illustrated in the figures. It will be understood that the spatially relative terms are intended to encompass different orientations of the device in use and/or operation in addition to the orientation depicted in the figures. For example, if the device in the figures is turned over, elements described as "below" and/or "beneath" other elements or features would then be oriented "above" the other elements or features. The device may be otherwise oriented (e.g., rotated <NUM> degrees or at other orientations) and the spatially relative descriptors used herein interpreted accordingly.

Exemplary embodiments are described herein with reference to cross-sectional illustrations that are schematic illustrations of idealized exemplary embodiments (and intermediate structures). Thus, exemplary embodiments should not be construed as limited to the particular shapes of regions illustrated herein but are to include deviations in shapes that result, for example, from manufacturing.

To the extent that functional features, operations, and/or steps are described herein, or otherwise understood to be included within various embodiments of the inventive concept, such functional features, operations, and/or steps can be embodied in functional blocks, units, modules, operations and/or methods. And to the extent that such functional blocks, units, modules, operations and/or methods include computer program code, such computer program code can be stored in a computer readable medium, e.g., such as non-transitory memory and media, that is executable by at least one computer processor.

In accordance with aspects of the inventive concepts, electronic devices of a vehicle and/or personal computing devices otherwise in communication with a vehicle computer are constructed and arranged to control an output of data, information, graphics, audio, video, or the like on a vehicle display, e.g., a heads-up display (HUD), by executing an multimodal input application at a special-purpose computer processor, e.g., speech-to-text conversion or other voice-based control, on-screen keyboard, touchscreen or touchpad, gesture recognition, biometrics, and so on, or a combination thereof, and establishing a data exchange between various input devices and the vehicle display. In various embodiments, the HUD, can be or include a projection on a visor or windshield, e.g., as a windshield display, as examples, or other display medium of a vehicle.

<FIG> is a network diagram of an embodiment of an architecture <NUM> within which data is input by the driver of a vehicle <NUM> when a conventional computer keyboard is unavailable, not feasible, or impractical for purposes of entering data for processing by a vehicle's electronic device, in accordance with aspects of the inventive concepts.

In the overall architecture <NUM> of <FIG>, a variety of systems and devices can communicate via one or more wired and/or wireless communication systems, networks, and/or channels, collectively represented by a network <NUM>, generally referred to as a "cloud. " The communication systems, networks, or the like can include, but need not be limited to, wide area and/or local area networks, such as the Internet, Global Positioning System (GPS), cellular networks, Wi-Fi, Bluetooth, and so on. Various electronic devices of the architecture <NUM>, such as a multimodal input processing system <NUM>, personal computing device <NUM> such as a driver or passenger smartphone, and so on can exchange data with the remote computer <NUM>, remote database <NUM>, and/or external application systems <NUM> via the network cloud <NUM>. Remote computer <NUM>, remote database <NUM>, and/or external application systems <NUM> can provide navigation, music or other audio, program content, marketing content, internet access, speech recognition, cognitive computing, artificial intelligence, and so on via network cloud <NUM> to the vehicle <NUM>. Remote computer <NUM>, remote database <NUM>, and/or external application systems <NUM> may include one or more computer hardware processors coupled to one or more computer storage devices for performing steps of one or more methods, for example, shown in <FIG>.

In <FIG>, vehicles <NUM> are generally represented as automobiles, but the inventive concepts are not limited thereto. While embodiments of the present invention are generally described herein in the context of automobiles, the inventive concepts can be applied to any type of vehicle, e.g., trucks, busses, motorcycles, planes or other aircraft, bicycles, trains, watercraft, military vehicles, industrial vehicles, emergency vehicles, and the like. Additionally, the inventive concepts can, in some embodiments, be applied to autonomous, partially autonomous, self-driving, driverless, or driver-assisted vehicles.

As shown in <FIG> and <FIG>, the various electronic devices of the vehicle <NUM> can include computer and communication systems that receive user inputs in the form of text, voice commands, or a combination thereof. A multimodal input processing system <NUM> can be part of or hosted by the vehicle <NUM>. The multimodal input processing system <NUM> stores and executes a multimodal application that supports an interaction in accordance with one or more voice and non-voice modes, and more specifically, combines or otherwise permits the operational coexistence of multiple user input modes or channels in the same interaction allowing a vehicle driver, passenger, or other user to interact with a vehicle display <NUM>, such as a vehicle heads-up display (HUD) <NUM>, through multiple input modes or channels. In some embodiments, the multimodal input processing system <NUM> can be part of a vehicle head unit responsible for processing multiple data input modes, which are provided by user or machine inputs to peripheral or input/output devices of the various computer devices of the vehicle <NUM> and/or personal computing devices <NUM> of a user of the vehicle <NUM>, such as mobile phones, tablets, phablets, personal computers, laptop computers, televisions (including smart televisions) and so on. In some embodiments, the remote computer <NUM>, database <NUM> and/or external application systems <NUM> can cooperatively communicate with a local applications processor <NUM> (see <FIG>) of the multimodal input processing system <NUM> to carry out functionality of the various computer devices of the vehicle <NUM>. In other embodiments, the local applications processor <NUM> facilitates data exchanges between electronic devices of the vehicle <NUM>, such as touchpad <NUM> and a personal computer <NUM>, and HUD <NUM> or other projection or display device. In some embodiments, a personal computing device <NUM> can be in physical or wireless communication with other computer elements of the vehicle <NUM>, such as the HUD <NUM> of the vehicle <NUM>, via a wireless interface of the multimodal input processing system <NUM>.

As shown in <FIG>, in some embodiments, the multimodal input processing system <NUM> includes at least one processor <NUM> coupled to at least one computer storage device or memory <NUM>, e.g., via at least one bus (BUS). In various embodiments, the multimodal application is configured to utilize one or more processor(s) <NUM> and computer storage <NUM> of a vehicle computer or system, and to share these assets with other electronic devices of the vehicle <NUM>. The computer storage <NUM> can take the form of any now known or later developed computer storage device, including, but not limited to, a hard drive, random access memory in its various forms, read only memory in its various forms, and/or a compact disc, as examples. The computer storage <NUM> can store program code, instructions, and data necessary for the multimodal input processing system <NUM> to carry out its tasks, including performing one of more methods of <FIG>.

At least one communication interface module <NUM> can be configured to enable the vehicle <NUM> and/or multimodal input processing system <NUM> to communicate and exchange information and data with systems and subsystems external to the vehicle <NUM> and local to or onboard the vehicle. The communication interface module <NUM> can also enable communication with the external applications <NUM>.

The multimodal input processing system <NUM> can include one or more input and output devices for exchanging data processed by the system <NUM> with other elements of the environment of <FIG>, and specifically the vehicle <NUM> of <FIG>. Examples may include, but are not limited to, navigation system outputs (e.g., time sensitive directions for a driver), incoming text messages converted to output speech, vehicle status outputs, and the like, e.g., output from a local or onboard storage medium or system. In some embodiments, the communication interface module <NUM> provides input/output control functions with respect to one or more electronic devices, such as the HUD <NUM>, touchpad <NUM>, and/or personal computer <NUM> of the driver or passenger. An example of an input/output control function may include managing the buffering and output of data and the storage of data in computer storage <NUM> for subsequent use by other elements of the multimodal input processing system <NUM>. Other input devices providing at least one of a first input mode and/or a second mode may include, but not be limited to, a computer keyboard, image scanner, microphone, speech recognition module or elements, pointing device, graphics tablet, mouse, optical device, trackball, refreshable braille display, optical character recognition (OCR) technology, 3D modeling computer, sensors, such as infrared sensors or the like, or a combination thereof.

In some embodiments, the multimodal input processing system <NUM> further comprises a keyboard mapping module <NUM> that applies a homology algorithm or the like to establish a same or similar relative position of keyboard characters of the touchpad <NUM> or related touch-sensitive input device and those of the virtual keyboard <NUM> displayed on the vehicle display medium <NUM>, e.g., a windshield or visor. In some embodiments, the vehicle display medium <NUM> may be a touchable display medium for receiving data signal inputs for performing a swipe typing operation or other touch-related operation at the vehicle display medium <NUM>. In other embodiments, the keyboard <NUM> is projected near the top of the vehicle display medium <NUM>, e.g., a windshield or visor, for example, within reach of the user for providing input to the keyboard <NUM>.

For example, referring to <FIG>, character <NUM> of touchpad <NUM> can be homologically mapped to a character <NUM> displayed at virtual keyboard <NUM>, so that when character <NUM> is touched or otherwise selected at touchpad <NUM>, character <NUM> of keyboard can be displayed as being selected, for example, by being highlighted, enlarged, or otherwise displayed in a manner so that a viewer can recognize that the character <NUM> was selected, for example, by proxy from the touchpad <NUM>. In some embodiments, a swipe typing operation may be performed. Here, multiple characters forming a word or other arrangement of objects, characters, or the like may be created by moving a finger or object such as a stylus over the keyboard. In these embodiments, this movement is mapped homologically, and the starting point of the swipe typing operation is selected by establishing the relationship between a location on the virtual keyboard <NUM> and the location on the touchpad <NUM> where the finger or object directly or indirectly contacts the display of the touchpad <NUM>.

In some embodiments, the multimodal input processing system <NUM> further comprises a device signal translator <NUM>, for example, that converts signals received by a first electronic device, such as the touchpad <NUM>, to data for controlling the HUD <NUM>. For example, the touchpad <NUM> may communicate with a driver monitoring camera <NUM> in the vehicle <NUM> that monitors driver or passenger eye movements by sensing light, for example, infrared or related radiation reflected from the pupil(s). This captured data can be processed by the device signal translator <NUM> to translate the eye movement data received from the touchpad <NUM> to movements of the cursor <NUM> displayed at the HUD <NUM>.

In some embodiments, the multimodal input processing system <NUM> further comprises a pressure sensor processor <NUM> that receives data from the touchpad <NUM> that distinguishes pressures or related forces applied to the display screen of the touchpad <NUM> and/or other electronic device in communication with the multimodal input processing system <NUM>. For example, a user may use a finger or stylus to apply a light touch or low amount of pressure to the touchpad <NUM>. The pressure sensor processor <NUM> can, in response, instruct the HUD <NUM> to display an object, keyboard character, and so on in response to the activated light pressure. The pressure sensor processor <NUM> may receive a signal having a different amount of pressure, for example, greater force or greater amount of time that the force is applied to the display screen. A different command may be generated to execute the object in response to this greater or longer force, for example, activate a keyboard character when a greater amount of pressure is applied and/or the pressure is applied for a greater amount of time.

In some embodiments, the multimodal input processing system <NUM> further comprises a mode controller <NUM> that facilitates one or more input modes with respect to the HUD <NUM> and/or other computer display controlled by various modes, e.g., touchpad, computer mouse, gaze technology, and so on. In particular, the mode controller <NUM> controls data input to the HUD <NUM> from various input modes, such as gesture, speech, tactile, gaze, augmented reality, monitoring cameras, navigation devices, and so on. For example, the mode controller <NUM> can monitor eye gaze tracking information and determine when an eye gaze is directed to the virtual keyboard <NUM> so that the touchpad <NUM> can be activated to output commands to the HUD <NUM> to select characters, icons, display buttons, and so on identified and/or determined from the directed eye gaze.

<FIG> is an embodiment of a method <NUM> of input to a computer that can be performed by the system of <FIG>, in accordance with aspects of the inventive concepts. In describing method <NUM>, reference is made to elements of <FIG>. As described herein, text and/or other data can be input by a vehicle driver in a driving car, parked car, or autonomous vehicle in situations where a keyboard or related peripheral computer input/output component is not available or not feasible during use, for example, a virtual keyboard of a smartphone while driving the vehicle <NUM>.

In step <NUM>, an electronic communication is formed, for example, by the multimodal input processing system <NUM>, between a first electronic device and a display of a second electronic device, for example, HUD <NUM>, touch-sensitive windshield, and/or related device for displaying data on the windshield, console, or region of interest in the vehicle. In some embodiments, the first electronic device is a touchpad <NUM> at a vehicle console <NUM> as shown in <FIG>. In other embodiments, the first electronic device is an eye-driven communication system, including a camera that receives light reflections from a user's pupil(s), which is processed to translate the movement of the user's eyes into mouse cursor movements or the like for output to the second electronic device <NUM>. In other embodiments, the first electronic device is a voice or audio input device.

In step <NUM>, a first input mode is provided by a first electronic device, such as the touchpad <NUM> or other electronic device, to the HUD <NUM>. In some embodiments, the first input mode includes a touchpad computer input, for example, so that the HUD <NUM> can receive and process commands or the like for controlling a movement of a cursor <NUM> displayed on the HUD <NUM> for selecting characters of the virtual keyboard <NUM>. In other embodiments, the first input mode includes a display-less touchpad computer input, for example, the touchpad <NUM> includes a touch-sensitive keyboard but includes no display. Here, the user input can be a swiping motion or other human gesture received by the keyboard, but the display is external to the touchpad device <NUM>. In other embodiments, the first input mode is a mode in which a camera and/or other sensor for detecting eye movements is used by the multimodal input processing system <NUM> to control functions of the HUD <NUM>. In other embodiments, other input devices may provide various input modes to control functions of the HUD <NUM>, such as gesture recognition, speech recognition, augmented reality, monitoring cameras, navigation devices, and so on, which provide inputs to the multimodal input processing system <NUM> for communicating with the HUD <NUM> according to various input modes, either individually or simultaneously.

In step <NUM>, the multimodal input processing system <NUM> executes a multimodal application that permits multiple input modes, for example, a combination of the first input mode in step <NUM> and at least one other input mode, referred to as a second input mode, to interact with the display of the second electronic device. In some embodiments, the first input mode can include multiple inputs, for example, provided by a microphone to enter speech commands such as "drive me there" in conjunction with a camera <NUM> providing data determined from a vehicle driver's gaze in a direction of a destination of interest. The second input mode can provide an output or update on the display of the second electronic device, such as update a map that indicates the destination of interest (see, for example, <FIG>).

In step <NUM>, the vehicle display, e.g., HUD <NUM> or other device providing a display on the windshield or other vehicle display medium, is controlled by the output of the executed multiple modal application, which is determined by the combination of first input mode of step <NUM> and the at least one second input mode. In some embodiments, a first input mode may be a signal output in response to a user applying a slight amount of pressure in the form of a light touch by a stylus or other pointed object or by finger to the touchpad <NUM> to activate a cursor <NUM> displayed on the vehicle display medium <NUM>, e.g., windshield. When activated, the cursor <NUM> moves to various regions of the HUD <NUM> commensurate with movement of a finger or stylus about the display of the touchpad <NUM>. This is achieved by the touchpad <NUM> outputting commands to the HUD <NUM> to move the cursor <NUM> accordingly.

The second input mode may be a signal output to the HUD <NUM> from the same or different electronic device than the touchpad <NUM>. In cases where the same electronic device, e.g., touchpad <NUM>, is used for both the first and second input modes, the second input mode may be provided in response to the user applying a greater amount of pressure or applying a similar amount of force for a greater amount of time to the touchpad <NUM> than that provided by the first input mode. This action could be used, for example, to control the cursor <NUM> positioned on the screen at or near a character <NUM> to activate the character <NUM> displayed at a HUD keyboard <NUM> so that the character is entered as input to a computer and/or is highlighted or otherwise indicated as being activated.

In another example, a second input mode may pertain to a different electronic device than that of the first input mode. For example, a second input mode may be provided by a driver monitoring camera <NUM>, which can control the cursor <NUM> displayed at the vehicle display medium <NUM>, e.g., a windshield <NUM>.

In another example, a user may provide an input mode directly from a touchable vehicle display medium, e.g., windshield display, and/ or from a displayed keyboard that is sensitive to touch. Here, the user can enter information on the vehicle display medium, e.g., on the windshield, before starting the vehicle. Other first and/or second input modes may be available in connection with an input mode via the vehicle display medium, e.g., windshield.

<FIG> is an embodiment of a method <NUM> of input to a computer that can be performed by the system of <FIG>, in accordance with aspects of the inventive concepts.

In step <NUM>, a virtual keyboard <NUM> is displayed, for example, as part of a display at the HUD <NUM>, on a windshield <NUM> of vehicle <NUM> shown in <FIG>. Other embodiments of the invention provide display locations of use to passengers, or of use to the driver on a periodic basis only. The virtual keyboard <NUM> permits a driver, passenger, and/or other viewer to see the keyboard. As described herein, the multimodal input processing system <NUM> permits the viewer(s) to see the keyboard <NUM> relative to their fingers while they perform typing from the touchpad <NUM>.

In step <NUM>, positions on the touchpad <NUM> are mapped homologically to the virtual keyboard <NUM> on the display, for example, by the multimodal input processing system <NUM>. For example, a displayed image of the virtual keyboard <NUM> can cornprise an illustration of representation of individual input components touched, such as a letter key (e g. , M, N, O, etc.) or other keyboard character <NUM> touched on a typing keyboard, a note key (e.g., middle C) touched on a piano keyboard, a number pushed on a cell phone, and so on. In doing so, the virtual keyboard <NUM> displays an indication of the corresponding selected character <NUM>, for example, by moving cursor <NUM> to the corresponding selecting character <NUM>. This permits a dashboard <NUM>, or console or other control panel of the vehicle <NUM> to be operated without undue concentration by a user, e.g., a vehicle driver, on visually reading the display of the touchpad <NUM> while working the controls, e.g., selecting keys, characters, or the like on the touchpad display while viewing the selection on the windshield <NUM>, which is achieved by the selection of the starting location for the cursor <NUM> with respect to characters, words, objects, and so on.

In step <NUM>, the touchpad <NUM> is accessed, for example, by a user lightly touching the touchpad. In response, in step <NUM>, a virtual cursor is shown at an equivalent position on the displayed virtual keyboard <NUM>. For example, when character <NUM> is touched on touchpad <NUM>, a virtual cursor <NUM> is dynamically displayed at the HUD <NUM> that identifies the touched input of the character <NUM>. In some embodiments, every touchpoint on the touchpad <NUM> is directly mapped to a location of the HUD <NUM>. A feature includes an amount of force, pressure, or the like with respect to a user touching the touchpad <NUM>, using a finger or object for physically applying a force to the surface of the touchpad <NUM>. A threshold amount of force applied to the touchpad keyboard may activate the system to display the virtual cursor <NUM>.

In step <NUM>, the cursor <NUM> can be moved to another region of the virtual keyboard <NUM>. In response to a selection of a starting point, the cursor <NUM> may be moved by performing a swipe typing operation at the touchpad <NUM>.

In step <NUM>, a movement of the cursor is activated by a different amount of force, pressure, or the like than that applied by the user in order to display the virtual cursor <NUM> in step <NUM>. For example, the user can perform a typing operation on a keyboard of the touchpad <NUM> by performing a user gesture such as the swiping movement on a touchscreen or keyboard of the touchpad <NUM>, referred to as swipe typing. The touchpad <NUM> may provide related features, such as predictive text, where the computer learns from user habits when performing a typing operation of the keyboard.

Although features of method <NUM> are described with reference to the touchpad <NUM>, the method steps can equally apply to other electronic input devices, such as a vehicle driver or passenger smartphone <NUM>. Here, the multimodal input processing system <NUM> establishes communications between the smartphone <NUM> and the HUD <NUM> so that user actions, such as text application entries to the smartphone <NUM>, can be viewed on the HUD <NUM>. Method steps of other methods, for example, described with reference to <FIG>, can also equally apply to other electronic devices.

<FIG> is an embodiment of another method <NUM> of input to a computer that can be performed by the system of <FIG>, in accordance with aspects of the inventive concepts. The method <NUM> when executed permits a touchpad <NUM> to be used to select items from a list displayed on a vehicle display medium <NUM>, such as a windshield. Other display object techniques may similarly apply. For example, in a method shown herein, one display object may be selected when establishing a starting point of the operation by the multimodal input processing system <NUM> in response to a user-initiated action at the touchpad <NUM>, then extended with a "lasso" to select multiple objects. A lasso-based selection may be performed by gesture, touch-based, gaze, or other input modes, for example, described herein.

In step <NUM>, a second electronic device, such as HUD <NUM>, displays a list of items such as music titles, a group of objects, or other arrangement of icons, objects, listings, and so on. The arrangement of a list displayed on the HUD <NUM> or related projection on the windshield <NUM> may be predetermined and preprogrammed in a vehicle computer by an original equipment manufacturer (OEM) or other entity, for display by the HUD <NUM> according to a desired arrangement. In some embodiments, as shown in <FIG>, a map <NUM> is displayed by the HUD <NUM>. In some embodiments, a user can select a location on the map <NUM>, which is used as an input for navigation applications, for example, described at least in <FIG> and <FIG>. In some embodiments, the list of items is constrained to possible matches to nearby locations, frequently visited locations, or other predetermined criteria.

In step <NUM>, the touchpad <NUM> is accessed, for example, by a user lightly touching the touchpad. In response, in step <NUM>, a virtual cursor <NUM> is shown at an equivalent position on the displayed virtual keyboard <NUM>, for example, similar to steps <NUM>, <NUM> of <FIG>. This can be performed by the mapping module <NUM> and/or pressure sensor processor <NUM> of the multimodal input processing system <NUM>.

In step <NUM>, the cursor <NUM> can be moved to the desired list item. The multimodal input processing system <NUM> can send an instruction to the HUD <NUM> to move the cursor <NUM> to a displayed object mapped to a counterpart object displayed on the touchpad <NUM>, smartphone <NUM>, or other electronic device used by the vehicle driver or passenger to enter information.

At step <NUM>, the desired list item is selected. The multimodal input processing system <NUM> may receive a signal indicative of a greater amount of force, pressure, or the like applied to a region of the touchpad <NUM>, smartphone <NUM>, or the like that the force, pressure, touch, or the like performed in step <NUM>.

In other embodiments, another input mode may be detected and processed, for example, a voice command instructing the system to "select this one. " Other input modes may equally apply, such as tactile, voice, and/or biometric inputs.

In other embodiments, when a map <NUM> (see <FIG>) is displayed and a location on the map is selected, a navigation process may be executed, for example, whereby a Unix desktop environment (UDE) is provided for performing at least one step in the navigation process.

<FIG> is an embodiment of another method <NUM> of input to a computer that can be performed by the system of <FIG>, in accordance with aspects of the inventive concepts.

In step <NUM>, the HUD <NUM> displays a virtual keyboard <NUM>, for example, similar to step <NUM> of <FIG>. In step <NUM>, positions on the touchpad <NUM> are mapped homologically to the virtual keyboard <NUM> on the display, for example, similar to step <NUM> of the method <NUM> of <FIG>. In step <NUM>, a gaze or related eye movement detection device, such as but not limited to a driver monitoring camera <NUM>, is activated to monitor a user's eye movement, such as a vehicle driver or passenger. In response, the multimodal input processing system <NUM> receives and processes the collected eye movement data to determine if the user is directing a gaze at the virtual keyboard <NUM>, or more specifically, to a particular region of the virtual keyboard <NUM> where a character <NUM>, or key, button, icon, graphical representation, and so on is displayed. The virtual cursor <NUM> may be displayed at this region where the user's gaze is determined by the driver monitoring camera <NUM>.

In step <NUM>, a second electronic device is activated. In some embodiments, when the user touches the touchpad <NUM>, the HUD <NUM> is activated, whereby the virtual cursor <NUM> remains at the position determined by the gaze in step <NUM> so that swipe typing or the like can be performed on the touchpad <NUM> to control the movement of the virtual cursor <NUM>. In other embodiments, a gesture detection device determines the location of the virtual cursor <NUM>, for example, at a position on the windshield <NUM> where the user directs one or both eyes, i.e., gazes at a starting character at the virtual keyboard <NUM> displayed on the windshield <NUM>. In these other embodiments, a swipe typing operation or the like can be performed on the touchpad <NUM> to control the movement of the virtual cursor <NUM> and to establish a starting point for a swipe typing operation.

<FIG> is an embodiment of another method <NUM> of input to a computer that can be performed by the system of <FIG>, in accordance with aspects of the inventive concepts. In describing method <NUM>, reference is made to elements of <FIG>.

In step <NUM>, the HUD <NUM> displays a list of items such as music titles, a group of objects, or other arrangement of icons, objects, listings, and so on, for example, for example, similar to step <NUM> of <FIG>.

In step <NUM>, a virtual cursor <NUM> of the HUD <NUM> is activated by a first input mode. In some embodiments, the first input mode includes eye gaze technology that detects an eye movement of the driver, for example, gazes at a location of the arrangement displayed on the windshield <NUM>, whereby in step <NUM> the multimodal input processing system <NUM> moves the virtual cursor <NUM> to the location of the directed eye movement. In other embodiments, the first input mode includes gesture recognition technology such as a ToF camera <NUM> in <FIG> that detects a user gesture such as finger pointing. In response, in step <NUM> the multimodal input processing system <NUM> moves the virtual cursor <NUM> to a desired object, listed item, or the like at the location identified by the gesture. The virtual cursor <NUM> may be moved according to a different second input mode than the first input mode used for activating the virtual cursor <NUM>. For example, eye gaze technology may be used to both activate and move the virtual cursor <NUM> to a desired location. In another example, gesture recognition technology may be used to activate the virtual cursor <NUM>, while eye gaze technology may be used to move the virtual cursor <NUM>. The mode controller <NUM> of the multimodal input processing system <NUM> may permit multiple input modes to be used to activate and move the virtual cursor <NUM>.

In step <NUM>, the desired object, listed item, or the like of interest may be selected by a third input mode, which may be the same as or different than the first and second input modes of steps <NUM> and <NUM>, respectively. For example, after the virtual cursor <NUM> is moved in step <NUM> to a location of interest, the object or the like at the location of interest can be selected by a speech command, tactile input, or other predetermined input mode.

As described herein, one of the input modes to the multimodal input processing system <NUM> is a voice assistant, or digital processor that uses voice recognition, speech synthesis, natural language processing (NLP), and the like to provide a service through an application executed for communicating via the multimodal input processing system <NUM> with the window display such as HUD <NUM>. A well-known application for voice assistants relates to navigation, for example, where the driver interacts with the voice assistant by orally submitting a destination address, request for directions, and so on.

Conventional navigation applications determine a destination either by processing an address, point of interest (POI), or category search. However, in some embodiments of the inventive concepts, additional features such as gazing at a target, reusing a place of interest that is not specified in a typed address or other location information to a navigation system, but is instead determined from other sources, such as a recent interaction between driver and passenger for a current navigation, or a radio station which mentions the location name, or written on a piece of paper.

Referring again to <FIG>, in some embodiments, the multimodal input processing system <NUM> further comprises a navigation controller <NUM> that outputs data from a vehicle navigation system <NUM> to the second electronic device, e.g., HUD <NUM>, and allows a display of the navigation data to be controlled from multiple input modes, for example, provided by a combination of touchpad, keyboard-mapping, gesture recognition, speech recognition, augmented reality, sensing and/or monitoring cameras.

Referring again to <FIG>, illustrated is an embodiment of the computer architecture <NUM> that includes data flow paths between elements of the architecture, in accordance with aspects of the inventive concepts. A feature according to some embodiments is to permit a user to enter street addresses or other destination location data to a navigation system in a safe and efficient manner by using different input modes than typing text into a keyboard of a navigation device. In some embodiments, the method <NUM> can be applied to autonomous or partially autonomous vehicles, for example, autonomous ground vehicles, driverless or driver-assisted automobiles, and so on where various inputs regarding a requested navigation destination are received and executed.

A camera <NUM>, scanner, or other electronic detection device that is used for destination inputs by processing an image from the contents of a document <NUM>. In some embodiments, the document <NUM> is a piece of paper with handwritten or typed text. In other embodiments, the document <NUM> is a photograph, printed hotel reservation, or the like that includes address-related text and/or other information from which a desired destination location can be determined, for example, the OCR software <NUM> and/or other related information extraction technology <NUM>.

In flow path 912a, the OCR software <NUM> and/or other related information extraction technology <NUM> determines the address of interest from the image of the document <NUM>. In particular, the OCR software <NUM> can extract text, characters, keywords, or the like such as street numbers, street, city, or state names, zip codes, and so on. In some embodiments, a camera <NUM> or related sensor is programmed with or otherwise in communication with a special-purpose computer that stores and executes the OCR software <NUM> or related technology to scan the contents of the document <NUM> and to parse or otherwise distinguish and recognize an address among the contents. One or more sensors such as camera <NUM> as with other embodiments may be located at the windshield <NUM> as shown in <FIG> and/or positioned at other locations of the vehicle such as the vehicle door, back seat, and so on. In some embodiments, the camera <NUM> may be a mobile camera that captures the contents and/or other destination inputs, photographs in the document, and so on for output to the system <NUM>. In other embodiments, the OCR software <NUM> is on a standalone computer that communicates indirectly with the camera <NUM> via the multimodal input processing system <NUM>, or communicates directly with the camera <NUM>. The determined address can be stored at the multimodal input processing system <NUM> or other data repository for subsequent retrieval and processing.

In some embodiments where the OCR software <NUM> cannot identify an address or other useful information from the extracted text or the like, the method <NUM> proceeds to flow path 912b where the extracted data is analyzed by an information extraction technology <NUM>, which may include cognitive computing, artificial intelligence, analytics, or the like to determine an address. The address determined from the OCR software <NUM> or information extraction technology <NUM> can be stored for subsequent recognition by its characteristic form, e.g., dependent on language, country, and so on.

In other embodiments where the OCR software <NUM> cannot identify an address or other useful information from the extracted text or the like, the method <NUM> proceeds to flow path 912c where the extracted data is output directly to the navigation system <NUM> or search engine or the like to determine from the text in the extracted data an address, or a listing of possible addresses. The navigation system <NUM> may be an application of a mobile electronic device <NUM> or may be part of a remote computer in communication with the system <NUM> via a network <NUM>. In some embodiments, the navigation system <NUM> may be part of an autonomous or semi-autonomous vehicle for providing directions to an intended destination.

In flow path <NUM>, a user such as the vehicle driver, passenger, or other user can utter into a microphone <NUM> or other audio input device an oral command such as "drive me there. In flow path <NUM>, the system <NUM> outputs the address, e.g., parsed by the OCR software <NUM> and/or information extraction technology <NUM> or otherwise determined from the document <NUM> and/or speech uttered via the microphone <NUM> and converted to data, to the vehicle navigation system <NUM>, which processes the address accordingly and provides turn-by-turn directions and/or other navigation information. In some embodiments, the multimodal input processing system <NUM> receives via flow path <NUM> a navigation result from the navigation system <NUM> such as current location, status, street maps, and so on.

In flow path <NUM>, the received navigation data can be displayed by the system <NUM> in real-time on the windshield <NUM>, for example, executed by HUD <NUM>, for updating the displayed map <NUM>. In other embodiments, a mobile electronic device <NUM> such as a driver or passenger smartphone may receive the navigation data in addition to or instead of the HUD <NUM>. In some embodiments, in flow path <NUM>, the navigation system <NUM> provides multiple results from a received address, in which case a list, for example, list <NUM> shown in <FIG> is displayed so that a user can select an address from the displayed list of address options in the list <NUM>.

In a related application, as shown in <FIG>, a first flow path <NUM>, <NUM> extends between the microphone <NUM> and a speech recognition system <NUM>, for example, configured to perform Voice Activity Detection (VAD), Automatic Speech Recognition (ASR), Natural Language Understanding (NLU), or a combination thereof. The speech recognition system <NUM> can be executed in response to flow path <NUM> to detect and process any speech occurring in a vehicle, for example, between a driver and passenger. In some embodiments, the speech recognition system <NUM> processes the detected speech by employing models and analyzing audio data, for example, the microphone <NUM> detecting speech provided by a driver and/or passenger. Other information extraction tools may equally apply such as named entity recognition (NER), automatic speech recognition (ASR), natural language understanding (NLU), and so on, for recognizing the address of interest. Information extraction tools such as the speech recognition system <NUM> may be one of the external applications <NUM> described in <FIG>.

Accordingly, when a vehicle driver, passenger, or other user speaks a command such as "drive me there," the speech recognition system <NUM> or the like in flow path <NUM> can in response output a result such as a selected last memorized address. Alternatively, instead of flow path <NUM> a spoken command received via microphone <NUM>, a list <NUM> that includes the address can be displayed at the windshield <NUM> for example, via HUD <NUM>, for selection by the user, for example, using gaze detection, gesture technology, speech recognition, multi-modal text input such as XT9® technology, and/or other input mode described herein. Other details regarding the selected address may be displayed in real-time on the windshield <NUM> and/or selected electronic device <NUM>. In this alternative embodiment, in flow path <NUM> a command regard the selection by the user is output to the system <NUM> for subsequent processing.

In some embodiments, shown in flow path <NUM>, the multimodal input processing system outputs the selected address received from flow path <NUM>, <NUM>, or <NUM> to the vehicle navigation system <NUM> which processes the address accordingly. In response, the multimodal input processing system <NUM> receives via flow path <NUM> navigation data such as current location, status, street maps, turn-by-turn directions and/or other navigation information and so on for displaying in real-time in flow path <NUM> on the windshield <NUM> and/or display of an electronic device such as the user's smartphone <NUM>.

In a related application, as shown in <FIG>, the first flow path <NUM> extends between a gaze detection device, for example, camera <NUM> (also described in <FIG>), and multimodal input processing system <NUM> (instead of or in addition of the microphone <NUM>) for providing an input establishing where a user, e.g., driver, is looking. A second flow path <NUM> extends between a 3D modeling computer <NUM> and the multimodal input processing system <NUM> for providing a 3D model of a region, e.g., city, town, or other geography in proximity of the vehicle, and within the field of view of the driver's gaze. In some embodiments, the navigation system <NUM>, search engine, and/or other data provider provides 3D model information for processing the collected images collected by the input device, e.g., camera or the like, which is processed as destination entry data. In doing so, the multimodal input processing system <NUM> receives the inputs via flow paths <NUM> and <NUM>.

In some embodiments, an image <NUM> of the 3D model is displayed in flow path <NUM> on the windshield <NUM>. In flow path <NUM>, the multimodal input processing system <NUM> outputs data regarding the gaze detection and 3D model to an external application <NUM>, for example, an analysis tool <NUM> which performs a visibility analysis to identify buildings, landmarks, and/or related objects viewed by the user and detected by the gaze detection device <NUM>. A result of the visibility analysis is provided in flow path <NUM> to the multimodal input processing system <NUM>, which also receives in flow path <NUM> a command via a microphone <NUM> to drive the user to the building, landmark, or the like identified from the visibility analysis.

In some embodiments, shown in flow path <NUM>, the multimodal input processing system outputs the selected address to the vehicle navigation system <NUM> which processes the address accordingly. In some embodiments, the multimodal input processing system <NUM> receives in response navigation data such as current location, status, street maps, and so on for displaying in real-time on the windshield <NUM>. Various techniques may be used to display such information is displayed on the windshield. Although a HUD <NUM> is described as providing one technique, embodiments herein are not limited to use of the HUD <NUM> and therefore other projection or display devices may equally apply.

In a related application, as shown in flow path <NUM> of <FIG>, a camera <NUM>, scanner, or other electronic detection device processes an image <NUM> from the photograph of a building, landmark, or other indicator of a location of interest. In some embodiments, the image <NUM> may include additional location information such as nearby restaurants, hotels, stores, and so on. In flow path <NUM>, the camera <NUM> outputs the image contents to the multimodal input processing system <NUM> for processing.

In flow path <NUM>, the multimodal input processing system <NUM> can determine whether the image <NUM> can be used to determine a location for navigation or other location identification purposes. In some embodiments, an external application system <NUM> can include a database that stores a listing of locations. The identified location in the image can be compared by the multimodal input processing system <NUM> to the records of the other locations in the database. In flow path <NUM>, comparison data such as images and analysis results are exchanged.

In some embodiments, multiple results are generated. Here, the results may be output in flow path <NUM> for display in a list <NUM>, for example, on the windshield <NUM>. The list can include pictures for display instead of or in addition to addresses or other text information. In some embodiments, shown in flow path <NUM>, the multimodal input processing system outputs a selected address to the vehicle navigation system <NUM> which processes the address accordingly. In some embodiments, the multimodal input processing system <NUM> receives navigation data such as current location, status, street maps, and so on for displaying in real-time on the windshield <NUM>.

Claim 1:
A method of input to a computer in a vehicle, characterized in that the method comprises:
forming an electronic communication between a first electronic device (<NUM>) and a display (<NUM>) of a second electronic device that is viewable by a user from a location in the vehicle (<NUM>), wherein forming the electronic communication includes providing by the first electronic device (<NUM>) a first input mode for the second electronic device (<NUM>);
executing a multimodal application that permits the user to interact with the display of the second electronic device by a combination of the first input mode and at least one second input mode for the second electronic device; and
controlling, by the combination of the first input mode and the at least one second input mode, the display of the second electronic device (<NUM>).