On-demand user control

A system and method for on-demand user control may detect a user's intend to interact with a currently undisplayed user control. The user's intent may include, for example, a hand approaching a touch-sensitive screen device. In response to the detected intent, the currently undisplayed user control may be displayed. The choice and location of the displayed user control may be derived from a location associated with the detected intent. The user control may be displayed progressively giving the appearance of morphing, emerging or being extruded from the display screen. The user may interact with the displayed user control using, for example, gestures on the touch-sensitive screen device.

BACKGROUND

Technical Field

The present disclosure relates to the field of human-machine interfaces. In particular, to a system and method for on-demand user control.

Related Art

Touch-sensitive display devices may be used to provide a human-machine interface (HMI; a.k.a. user interface (UI) or graphical user interface (GUI)) that receives user inputs sensed by the device (e.g., touch gestures on the display surface). The touch-sensitive display devices may represent display content that includes visual metaphors (a.k.a. soft controls; e.g., soft keys) that represent tactile controls (e.g., buttons) that provide visual prompts or enhance familiarity to ease the user's interactions with the HMI.

In some systems the touch-sensitive display device's display area (a.k.a. screen real-estate) is limited due to considerations including: size/portability of the system, bill-of-material (BOM) cost, graphics processing limitations, and other similar factors. In these systems the use of soft controls may be compromised by the desire to include multiple soft controls on the display, each being sufficiently large for ease of user interaction, while also wanting to preserve display area for other visual content and status information.

DETAILED DESCRIPTION

A system and method for on-demand user control may detect a user's intent to interact with a currently undisplayed user control. The user's intent may include, for example, a hand approaching a touch-sensitive screen device. In response to the detected intent, the currently undisplayed user control may be displayed. The choice and location of the displayed user control may be derived from a location associated with the detected intent. The user control may be displayed progressively giving the appearance of morphing, emerging or being extruded from the display screen. The user may interact with the displayed user control using, for example, gestures on the touch-sensitive screen device.

FIGS. 1A-1Dare a time series representation of an on-demand user control.FIG. 1Aillustrates an example touch-sensitive display device102having two visual content elements, a desired temperature104and a fan speed setting106. The touch-sensitive device102may be, for example, a component of an infotainment system installed in a vehicle such as an automobile. InFIG. 1Ba user's hand108is illustrated approaching the touch-sensitive display. A system for on-demand user control may detect a user's intent to interact with a currently undisplayed user control by, for example, detecting the approaching hand of the user108. In response to detecting the user's intent, the system may begin to display a user control110. The user control110may be a visual representation on the touch-sensitive display device102of a tactile (e.g., physical) control element that may be familiar to the user. The user control110may take the place of, or displace, an existing visual content element (e.g., the desired temperature104). Alternatively, the user control110may be placed in an area not previously occupied by another visual content element other than a background or other ornamentations.

FIG. 1Cillustrates the user's hand108getting closer to the touch-sensitive display device102. As the user's hand108gets closer the system may display more of the user control110. The user control110may appear to be extruded from the surface of the touch-sensitive display device102. InFIG. 1Dthe user's hand108is in contact with the surface of the touch-sensitive display device102and user control110has been rendered into a final desired appearance. The user control110may be created by visually transitioning (a.k.a. morphing) from an existing visual content element or background to the final desire appearance for the user control110. Morphing may include initially applying one or more surface textures from an existing visual content element or background to the user control110and then transitioning to one or more final desired surface textures for the user control110.

FIG. 7is a schematic representation of a user's hand108interacting with a rotary control type of user control110. The illustration on the left shows the user's forefinger and thumb proximate the top and bottom, respectively, of the user control110(proximate the 12 and 6 o'clock positions using a clock face metaphor). The illustration on the right shows the user's forefinger and thumb after they have transitioned (e.g., rotated) to proximate the 3 and 9 o'clock positions respectively. The interaction of the user with the user control110may be characterized as the user control110having been turned clockwise a quarter turn or 90 degrees. Alternatively, the interaction may be characterized as the user control110having been turned from, for example, the 12 o'clock to the 3 o'clock positions. In a further alternative, the interaction may be characterized as the user control110having been turned clockwise (or to the right) by, for example, eight steps for a user control having 32 steps (a.k.a. detents or clicks) per revolution. Similarly, the user's interactions with the other types of user controls, such as those illustrated inFIGS. 5A-5D and 6A-6Cand describe below, may be characterized in any way that is appropriate to the type of user control. The interactions with, or operations of, the various types of user controls may be expressed in either relative (e.g., eight steps to the right) or absolute (e.g., turned to the 3 o'clock position) terms as is appropriate for the type of user control being represented. The characterization of the user's interaction with the user control110may be sent to other processing elements such as, for example, one or more applications, system software, an operating system or other similar processing elements for further processing.

FIG. 2is a schematic representation of functional components of a system for on-demand user control. The example system200may comprise functional modules for intent detection202, location detection204, gesture detection206, user control rendering208and user control blending210. The intent detection module202may detect a user's intent to interface with a user control110by, for example, detecting that the user's hand108is approaching a touch sensitive display214. Alternatively or in addition, the intent detection module202may detect that the user's hand108is close to (e.g., proximate) or in contact with the display214. Any mechanism that allows detection of the user's hand108in relation to the display214including, without limitation, capacitance, thermal, resistive and optical sensing mechanisms may be used to provide user input216to the intent detection module202, the location detection module204and the gesture detection module206. The intent detection module202may provide to the user control rendering module208any one or more of: an indication that the user's hand108has been detected, a position or distance relative to the display214and a direction of movement of the user's hand108(e.g., approaching the display). The providing of the indications may be event driven, periodic, on-demand or a combination of these.

The location detection module204may receive user input216as describe above from which a location of the user's hand108relative to the display214may be determined. The location may be expressed, for example, using Cartesian coordinates (e.g., X,Y) relative to the display214. The location may be determined to be proximate to a status indicator or information element (e.g.,104and106) displayed on the display214or proximate to one of one or more designated areas on the display. A user control, of one or more user controls, may be associated with each status indicator, information element and designated area. The determined location and identification of an associated user control may be provided to the user control rendering module208.

The gesture detection module206may receive user input216as described above from which a gesture by the user's hand108can be determined. The gesture may be interpreted taking into account a type of user control that is displayed approximate the location of the gesture. Different types of gestures may be associated with each type of user control including, for example, one-dimensional translations (e.g., sliding or toggling), two-dimensional translations (e.g., sliding or rotation) and touches (e.g., pressing). The interpreted gesture may be associated with a particular action of the user control such as, for example, turning a knob, toggling a switch or pressing a button. In addition, the interpreted gesture may be associated with relocating, modifying or dismissing the user control. The gesture and/or the associated user control action may be provided to the user control rendering module208.

The user control rendering module208may render a user control (e.g.,110) and provide for having it displayed on the display214. The user control rendering module208may be triggered to render the user control based on one or more indications received from the intent detection module202. The location for displaying the user control and the type of user control to render may be determined from the location and identification of the user control received from the location detection module204. Further, animation of the user control (e.g., the appearance of extruding of the control from the display surface, rotating a knob or toggling a switch) may be based on any combination of indications and information received from the intent detection module202, the location detection module204and the gesture detection module206.

The user control rendering module208may cause the rendered user control to be displayed on the display214. The user control blending module210may, optionally, blend the appearance of the user control provided by the user control rendering module208with a background image212before the user control is displayed on the display214. The background image212may include one or more visual content elements (e.g., the desired temperature104or the fan speed setting106) and/or may include areas represented using surface textures and/or colors. The user control blending module210may mix textures and colors of the background image with textures and colors of the rendered user control, may superposition the user control over the background image using alpha blending, or use any other graphics compositing mechanisms. The user control blending module210may be used in both displaying and animating the user control on the display214.

The system200optionally may include a mechanism to provide a haptic feedback to the user. The haptic feedback may include, for example, deforming or otherwise modifying the surface of the display102or214in the area proximate where the user control110is displayed such that the user may be able to feel a presence of the user control110. The haptic feedback mechanism may be integral or external to the display102or214. The haptic feedback may be modified or changed when the user provides a gesture to operate the user control110. For example, the shape, size and/or location of the haptic feedback may change. Alternatively, or in addition, transient feedback may be provided to simulate detents (clicks), key depressions, key rebounds or control resistance. The haptic feedback mechanism may include, without limitation, piezoelectric, piezo-polymer, vibration generating or any other haptic feedback generating technologies.

FIGS. 5A-5Dare schematic representations of example appearances for a user control.FIGS. 6A-6Care schematic representations of further example appearances for a user control. The user control may appears as: a joystick502, a toggle switch504, a rotary control (a.k.a. a jog dial or knob)506, a group of radio buttons508, a slider602, a push button604, or a set of piano keys606. The example appearances of user controls inFIGS. 5A-5D and 6A-6Care illustrative only and not intended to be limiting in any way. The user control may take on the appearance of any physical control type that lends itself to being visually represented on the display device and interacted with by the user.

FIG. 4is a representation of a method for on-demand user control. The method400may be, for example, implemented using either of the systems200and300described herein with reference toFIGS. 2 and 3. The method400includes the act of detecting an intent, of a user, to interact with an undisplayed user control402. Detecting the intent may include detecting that the user's hand108is approaching or is proximate to the display214. Detecting the intent may further comprise determining a location associated with the user's action and associating one of one or more user controls with the determined location. Association of the determined location with one or more user controls may be based on information (e.g., declared regions on the display) provided by a system component or by an application internal or external to the system200. Alternatively, or in addition, association of the determined location with one or more user controls may be based on a response to query send to interface, system function or application internal or external to the system200.

The associated user control, which was previously undisplayed, may be displayed responsive to detecting the user's intent404. The displaying of the user control may be progressive where, for example, the user control appears to emerge from (e.g., be extruded or morphed from) the display214in response to the user's hand108getting closer to the display214. When the user control is displayed, an interaction by the user with the displayed control may be received406. The received interaction may be in the form of a gesture. In response to receiving the user interaction, the display of the user control may be updated to reflect a change in the appearance of the user control resulting from the interaction (e.g., the flipping of a switch or depressing of a button). Further, in response to receiving the user interaction, a command or control operation may be communicated to an application, other components of the system or to another system. The command or control operation may be expressed as a function of the user control type and/or a control function associated with the location of the user control.

FIG. 3is a further schematic representation of a system for on-demand user control. The system300comprises a processor302, memory304(the contents of which are accessible by the processor302) and an I/O interface306. The memory304may store instructions which when executed using the process302may cause the system300to render the functionality associated with the intent detection module202, the location detection module204, the gesture detection module206, the user control rendering module208and the user control blending module210as described herein. In addition the memory304may store the background image212.

The processor302may comprise a single processor or multiple processors that may be disposed on a single chip, on multiple devices or distributed over more that one system. The processor302may be hardware that executes computer executable instructions or computer code embodied in the memory304or in other memory to perform one or more features of the system. The processor302may include a general purpose processor, a central processing unit (CPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a digital signal processor (DSP), a field programmable gate array (FPGA), a digital circuit, an analog circuit, a microcontroller, any other type of processor, or any combination thereof.

The memory304may comprise a device for storing and retrieving data, processor executable instructions, or any combination thereof. The memory304may include non-volatile and/or volatile memory, such as a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), or a flash memory. The memory304may comprise a single device or multiple devices that may be disposed on one or more dedicated memory devices or on a processor or other similar device. Alternatively or in addition, the memory304may include an optical, magnetic (hard-drive) or any other form of data storage device.

The memory304may store computer code, such as the intent detection module202, the location detection module204, the gesture detection module206, the user control rendering module208and the user control blending module210as described herein. The computer code may include instructions executable with the processor302. The computer code may be written in any computer language, such as C, C++, assembly language, channel program code, and/or any combination of computer languages. The memory304may store information in data structures including, for example, the background image212.

The I/O interface306may be used to connect devices such as, for example, a mechanism to detect the user's hand108, a virtual keyboard interface, touch-sensitive display102or214, and to other components of the system300.

All of the disclosure, regardless of the particular implementation described, is exemplary in nature, rather than limiting. The systems200and300may include more, fewer, or different components than illustrated inFIGS. 2 and 3. Furthermore, each one of the components of systems200and300may include more, fewer, or different elements than is illustrated inFIGS. 2 and 3. Flags, data, databases, tables, entities, and other data structures may be separately stored and managed, may be incorporated into a single memory or database, may be distributed, or may be logically and physically organized in many different ways. The components may operate independently or be part of a same program or hardware. The components may be resident on separate hardware, such as separate removable circuit boards, or share common hardware, such as a same memory and processor for implementing instructions from the memory. Programs may be parts of a single program, separate programs, or distributed across several memories and processors.

The functions, acts or tasks illustrated in the figures or described may be executed in response to one or more sets of logic or instructions stored in or on computer readable media or non-transitory media. The functions, acts or tasks are independent of the particular type of instructions set, storage media, processor or processing strategy and may be performed by software, hardware, integrated circuits, firmware, micro code and the like, operating alone or in combination. Likewise, processing strategies may include multiprocessing, multitasking, parallel processing, distributed processing, and/or any other type of processing. In one embodiment, the instructions are stored on a removable media device for reading by local or remote systems. In other embodiments, the logic or instructions are stored in a remote location for transfer through a computer network or over telephone lines. In yet other embodiments, the logic or instructions may be stored within a given computer such as, for example, a CPU.

While various embodiments of the system and method for on-demand user control have been described, it will be apparent to those of ordinary skill in the art that many more embodiments and implementations are possible within the scope of the present invention. Accordingly, the invention is not to be restricted except in light of the attached claims and their equivalents.