English:
[0001] This application relates generally to computing device user interfaces, and more particularly to user interfaces suitable for touchscreen equipped mobile devices.
[0002] Mobile computing devices equipped with touchscreen displays have enabled a new generation of personal electronic devices. Touchscreen displays provide both a graphical display and a flexible a user interface, and thus enable devices to eliminate physical keys and buttons in favor of a larger display. Such mobile devices typically include graphical user interface systems which enable users to enter commands by touching icons presented on the display. A common form of user interface presents a virtual keyboard or keypad with a number of small key icons arranged in the manner of a traditional physical keyboard or keypad. While virtual keyboard user interfaces are intuitive and useful, the limited size of the touchscreen display in typical mobile devices requires the virtual keyboard key icons to be limited in size, making it difficult for users to accurately touch without mistyping.
[0003] The various embodiments enable user interfaces suitable for use on computing devices with touchscreen displays that can compensate for the size of a user's fingers by locally increasing the size of a portion of the display image by tracking a user's gaze with a digital camera. In an embodiment, a mobile device includes a touchscreen display and a digital camera positioned so that it can image a user's face while the user is viewing the touchscreen display. By tracking the user's eyes in digital images, the mobile device may be configured to determine where the user is looking, and if the user's gaze is directed to a location on the touchscreen display, a portion of the displayed image in the vicinity of where the user is looking may be increased in size. When the displayed image includes a virtual keyboard or keypad, key icons in the vicinity of the user's gaze may be increased in size commensurate with the dimensions of the user's finger tip. In an embodiment, the mobile device may be calibrated to a user's finger size and eyes in a training routine. While the embodiments may be particularly useful for virtual keyboard user interface displays, the eye- tracking localized magnification functionality may also be useful for other applications, as well as other forms of computing devices, including those with large touchscreen displays.
[0005] FIG. 1 is a frontal view of a mobile device showing a conventional user interface for a messaging application on a touchscreen display.
[0006] FIG. 2 is a frontal view of a mobile device showing an embodiment user interface for a messaging application.
[0007] FIGs. 3 and 4 are frontal views of a mobile device illustrating operation of an embodiment user interface.
[0008] FIGs. 5 and 6 are frontal views of a mobile device illustrating a calibration operation of an embodiment user interface.
[0009] FIG. 7 is a process flow diagram of an example method for implementing an embodiment user interface.
[0010] FIG. 8 is a process flow diagram of an embodiment method for calibrating an embodiment user interface.
[0011] FIG. 9 is a process flow diagram of an embodiment method for implementing another embodiment user interface.
[0012] FIG. 10 is a component block diagram of a mobile device suitable for use in an embodiment.
[0013] The various embodiments will be described in detail with reference to the accompanying drawings. Wherever possible, the same reference numbers will be used throughout the drawings to refer to the same or like parts. References made to particular examples and implementations are for illustrative purposes, and are not intended to limit the scope of the invention or the claims.
[0015] Computing devices equipped with a touchscreen user interface/display provide great flexibility in terms of user inputs and graphical representations of material. finger will typically encompass more than one key icon. This is illustrated in FIG. 1 which shows a mobile device 1 with a touchscreen 2 on which is displayed a virtual keyboard 4. In a typical virtual keyboard layout as illustrated in FIG. 1, a user's finger 5 attempting to press a single key (e.g., Ή" key) will also touch surrounding key icons, such as the keys for "G," Ή," "V" and "B" as illustrated. As a result, typing errors on such virtual keyboards and keypads are common and can frustrate users.
[0016] The various embodiments provide a solution for this problem with small mobile device displays by providing selective enhancement of the displayed image or icons in response to a user's gaze. In an embodiment, the user may calibrate their computing device to their own finger size and train the device to recognize the location of the user's gaze based on images taken of their eyes. By determining where the user is looking on the touchscreen and enhancing the portions of a virtual keyboard near where the user is looking in an amount based upon a finger size calibration factor, the mobile device can provide a user interface keyboard or keypad that fits within the real estate of the touchscreen display but provides key icons large enough for the user's fingers. In a further embodiment, the method of enhancing the portion of the display within the user's gaze can be applied to other applications, such as displayed maps, photographs, text, and other visual material. Also, the various embodiments may be implemented on any computing device including a touchscreen display, even those with large touchscreen displays, since the localized image enhancement functionality can enable users to accurate press particular locations on a display without having to magnify the entire image.
[0017] FIG. 2 illustrates an embodiment operating on a mobile device 1 with a touchscreen 2. In this embodiment, the user's eyes are imaged by a digital camera 3 which is positioned on the mobile device 1 so that it can image the user's face while the user is interacting with the touchscreen 2. Based upon where the user is looking, the mobile device 1 increases the size of key icons in the virtual keypad virtual keyboard 4 so that they are compatible with the user's finger 5. As illustrated, a user attempting to press the virtual key for the letter Ή" is presented with an icon that is as large as the user's finger tip. In this manner, the chance of pressing the wrong key or confusing the touchscreen interface with a touch event encompassing multiple key icons is reduced.
[0018] This embodiment is further illustrated in FIGs. 3 and 4 which show how the digital camera 3 images a user's eyes 5 to determine the location of user's gaze 6 on the touchscreen display 2. This may be accomplished by using image recognition software to locate the user's eyes and, more particularly, the location of the user's pupils. Determining the location of the user's gaze may be accomplished by applying image analysis rules to the image of the user's eyes which may take into account the orientation and position of the user's face with respect to the device touchscreen display as well as the location of the user's pupils within the image, and applying trigonometric calculations to estimate where the user is looking. As the mobile device determines the location of the user's gaze 6 from images obtained by the digital camera 3, the key icons of the virtual keypad 4 close to the center of the user's gaze 6 are enhanced in size. Thus, as the user's gaze 6 shifts from the left-hand side of the virtual keyboard 4, as shown in FIG. 3, to the right hand side, as shown in FIG. 4, those icons of the virtual keypad 4 within the center of the user's gaze 6 are made larger based on the calibration factor for the user's finger. FIGs. 3 and 4 illustrates how the embodiments enable a virtual keyboard 4 to fit within the limited space of a mobile device touchscreen 2 while also providing key icons large enough to accommodate a user's finger.
[0019] While FIGs 2-4 show a virtual keyboard 4 that is enhanced in response to a user's gaze, the embodiments may also be applied to any virtual keypad, such as a telephone keypad, a calculator keypad or a game-specific keypad. Thus, the images of a virtual keyboard 4 are provided merely as examples and not as limitations on the scope of the embodiments.
[0020] To calibrate the mobile device 1 to a user's finger size, the mobile device may require the user to initialize or calibrate their device to their fingers before usage. FIG. 5 shows an example user interface display that might be presented on a touchscreen 2 in a calibration process. In such a calibration process, a user may be prompted to touch a virtual key 8 on the screen with the finger or fingers normally used during texting. In this embodiment, the user merely has to touch the displayed key icon to enable the mobile device 1 to determine the dimensions of the surface area of the touchscreen 2 contacted by the user's finger tip. Using the measured dimensions, a device processor can calculate an appropriate user fmger size calibration factor that may be stored in memory for use in resizing key icons as illustrated in FIGs. 2-4.
[0021] To calibrate the mobile device 1 to a user's eyes to enable it to accurately determine where a user is looking, the mobile device 1 may require the user to train their device to track their eyes in a training routine. FIG. 6 shows a simple user interface display that might be presented on a touchscreen 2 in such a training routine. For example, a user may be prompted to follow a moving icon on the touchscreen 2, such as an image of a bee 9 as it moves about on the touchscreen display 2. At each instant, the mobile device 1 compares the image of the user's eyes obtained from the digital camera 3 to the location of the moving icon 9 on the display 2 to learn how to correlate the image of the user's eyes to a particular location on the display 2. Since the device processor knows where the image is on the touchscreen, the processor can calculate a difference or correction factor, or otherwise adjust the image analysis rules to bring the determined location and the actual location into alignment. Such a calibration/learning process may be used to develop a set of rules for analyzing user facial images to determine a focus of gaze, such as by using well known machine learning techniques and well known image recognition techniques.
[0022] The calibration and training processes illustrated in FIGs. 5 and 6 may be repeated multiple times in order to provide a statistically averaged fmger sized calibration and eye tracking rules. Further, the eye-tracking training may continue for a predetermined period of time or until the error between the actual location of the displayed image and the determined location at which the user is looking is reduced below a maximum threshold value.
[0023] An example method 10 that may be implemented on a computing device according to an embodiment is illustrated in FIG. 7. In method 10, the eye tracking/display enhancement functionality may be activated when a virtual keypad is displayed on a touchscreen, step 12. As part of activating the eye tracking/display enhancement functionality, a processor within the computing device may access calibration values and eye tracking rules from memory in step 14. In step 16, the processor may activate the digital camera 3 and begin imaging the user's face and eyes. In step 18, the processor may analyze an image of the user's face to determine the location of the user's gaze. As mentioned above, this may be accomplished by using image recognition software to locate the location and orientation of the user's pupils. Determining the location of the user's gaze may be accomplished by applying the image analysis rules obtained from memory to the image of the user's eyes. Such image analysis rules may be determined or adjusted during a user training routine as discussed below with reference to FIG. 8.
[0024] When the direction or location of the user's gaze is determined, the processor may compare that location to the location of the displayed virtual keypad to determine whether the user is looking at the keypad in determination step 20. If the processor determines that the user is not presently looking at the virtual keyboard (i.e., determination step 20 = "No"), the processor may continue to obtain and analyze images of the user's eyes from the digital camera by returning to step 18. If the processor determines that the user is presently looking at the virtual keyboard (i.e., determination step 20 = "Yes"), the processor may use the finger size calibration factor to enhance key icons near the center of the user's gaze in step 22. The processor then continues to receive and analyze images from the digital camera by returning step 18. This process may continue so long as the virtual keyboard is displayed on the touchscreen display or until the functionality is otherwise terminated by the user.
[0025] In addition to enlarging the size of virtual keys within the vicinity of where the user is looking, the processor may also adjust the display coordinates mapped to each key icon to match the size of the icons for purposes of recognizing a touch to the touchscreen display as a user input. Graphical user interface functions map the location of user interface icons (e.g., key icons) in the displayed image to coordinate locations or boundaries on the touchscreen interface so that the device processor can recognize when a particular icon is being touched by a user. Therefore, in step 22 the processor also may adjust the mapping of key icons to display coordinates to match the dimensions of the icons presented on the display. In an embodiment, the resizing of keyboard icons may be accomplished in two steps. First, the processor may determine the new size for each key icon based upon the location of the user's gaze and a fmger calibration factor. Then, in a second step, the processor may adjust the mapping of icons to touchscreen sensor coordinates to match the adjusted dimensions of the various key icons.
[0026] FIG. 8 illustrates an example method 30 which may be implemented on a computing device for training and calibrating the device to a user's fmger size and eye gaze. While method 30 illustrates both calibration/training operations together, the fmger size calibration and the eye-tracking training may also be accomplished separately. In method 30 in step 32, a processor within the computing device may activate the calibration/training application, such as in response to a user menu selection or as part of a larger device initialization routine. In step 34, the processor may generate a display prompting the user to touch a particular spot on the touchscreen, such as illustrated in FIG. 5. In step 36, the processor may receive signals or coordinate information from the touchscreen display and measure the size and shape of the area contacted by the user's fmger. In step 38, the processor may use the measured size and shape of the contact area to calculate a fmger size calibration factor that can be stored in memory. This calibration factor may be a dimensional value (e.g., an area, diameter or radius) or a magnification factor (e.g., a percentage or numerical value) that may be applied to an icon's dimensions in order to determine an icon size that is suitable for the user's fingers.
[0027] In step 40, the processor may activate the digital camera and image the user's face and eyes. In step 42, the processor may analyze an obtained image to locate the user's face within the image, recognize the user's eyes and obtain dimensional measurements of the user's face within the image area. This information may be used to generate or adjust image recognition rules that can recognize and track the user's eyes when the computing device is held at various distances from the user's face. The distance between the computing device and a user's face will likely vary each time the functionality is activated and while the user is interacting with the computing device. Thus, part of the calibration may involve obtaining measurements to enable the computing device processor to compensate for different imaging distances. Further, the training process may be repeated at different face-to-camera distances to enable the processor to generate recognition rules that can compensate for differences in such separation distances.
[0028] In step 44, the processor may display a message prompting the user to follow a moving cartoon or icon on the display. In step 46 the processor may move the cartoon or icon an incremental distance on the display, and in step 48, the processor may obtain and analyze an image of the user's face to determine the location of the user's gaze. The cartoon or icon may be moved in step 46 continuously, incrementally, or in large steps (e.g., to each of the corners and center sequentially). The first time that the eye-tracking training is accomplished, the gaze determination operation in step 48 may employ default rules for translating images of the user's eyes to a gaze location on the computing device. In step 50, the computing device may adjust the gaze determination image analysis rules so the determined location matches the actual cartoon or icon location on the display. This eye-tracking rules calibration process may involve a simple linear adjustment or transformation of the rules used in step 48 to determine the user gaze location so that the determined location matches the actual location of the cartoon or icon on the display. In determination step 52, the processor may determine whether the training process is finished. The processor may continue the training routine by moving the cartoon or icon on the display for a predetermined amount of time, through a predetermined number of locations, or until the difference between the determined location of the gaze and the actual location of the cartoon or icon is consistently within a maximum threshold value (i.e., an acceptable error). So long as the processor determines that the training is not finished (i.e., determination step 52 = "No"), the processor may continue moving the cartoon or icon on the display, step 46, determining the location of the user's gaze, step 48, and calibrating the gaze-to-icon determination rules, step 50. Once the processor determines that the training is finished (i.e., determination step 52 = "Yes"), the processor may store the calibration settings and eye tracking image analysis rules in memory, step 54.
[0029] The various embodiments may also be applied to other types of applications which present visual displays that may benefit from preferential visual enhancement. For example, the eye-tracking localized magnification functionality may be useful in map or navigation programs so that users can see local map details within the context of a larger map display. Similar to a virtual keypad, a map or navigation application which permits a user to select a particular location for further information by touching a touchscreen interface may locally magnify the map in an amount that depends upon a fmger size calibration factor. In this manner, a map or navigation application can receive user inputs via a fmger touch to a particular location with reduced chance that the size of the user's fmger tip will inadvertently select a wrong location.
[0030] Another practical application of the various embodiments would be to use the eye-tracking localized magnification functionality to enhance hyperlinks, Internet web links, and other "clickable" text displayed on a touchscreen display. Due to the small size of touchscreen displays on most mobile devices, the size of displayed web page icons and text is typically smaller than the fmger tip of most users. Thus, touching the display screen to "click on" hyperlinks in the form of icons or hypertext can be difficult. The various embodiments can locally magnify web pages and documents containing hyperlinks where the user is looking so that the hyperlink icon or text is large enough to enable the user to accurately touch a desired hyperlink without inadvertently touching an incorrect hyperlink.
[0031] FIG. 9 illustrates an example method 60 for implementing such functionality on a computing device. In step 62, a computing device processor may activate the visual tracking and enhancement functionality in response to the activation of a map or other application. As part of activating the eye tracking/display enhancement functionality, a processor with in the computing device may access in memory calibration values and eye tracking rules, step 64. In step 66, the processor may activate the digital camera and begin imaging the user's face and eyes. In step 68, the processor may analyze an image of the user's face to determine the location of the user's gaze. To accomplish this analysis, the processor may apply image analysis rules determined during a user training routine as discussed above with reference to FIG. 8. When the location of the user's gaze is determined, the processor may compare that location to the location of the touchscreen display to determine whether the user is looking at the computing device display in determination step 70. If the processor determines that the user is not presently looking at the display (i.e., determination step 70 = "No"), the processor may continue to obtain and analyze images from the digital camera by returning to step 68. If the processor determines that the user is presently looking at the display (i.e., determination step 70 = "Yes"), the processor may enhance a portion of the keyboard image near the center of the user's gaze in step 72. The processor then continues to obtain and analyze images from the digital camera by returning step 68. This process may continue so long as the particular application is activated or a particular type of image is displayed on the touchscreen display, or until the functionality is terminated by a user.
[0032] Typical mobile devices 1 suitable for use with the various embodiments will have in common the components illustrated in FIG. 10. For example, a mobile receiver device 1 may include a processor 101 coupled to internal memory 102, a touchscreen display 103, and a digital camera 3 positioned so that it can image a user's eyes while the user is looking at the display 103. Additionally, the mobile device 1 may have an antenna 104 for sending and receiving electromagnetic radiation that is connected to a wireless data link and/or cellular telephone transceiver 105 coupled to the processor 101. Mobile devices typically also include menu selection buttons or rocker switches 107 for receiving user inputs. While FIG. 10 illustrates a mobile computing device, other forms of computing devices, including personal computers and laptop computers, will typically also include a processor 101 coupled to internal memory 102, a touchscreen display 103, and a digital camera 3 positioned so that it can image a user's eyes while the user is looking at the display 103. Further, the digital camera 3 may be built into the computing device as shown in FIG. 10 or configured as a separate device electronically coupled to the processor 101, such as by means of a cable (not shown) or wireless data link (not shown). Thus, FIG. 10 is not intended to limit the scope of the claims to a mobile computing device in the particular illustrated form factor.
[0033] The processor 101 may be any programmable microprocessor, microcomputer or multiple processor chip or chips that can be configured by software instructions (applications) to perform a variety of functions, including the functions of the various embodiments described herein. In some mobile devices, multiple processors 101 may be provided, such as one processor dedicated to wireless communication functions and one processor dedicated to running other applications. Typically, software applications may be stored in the internal memory 102 before they are accessed and loaded into the processor 101. In some mobile devices, the processor 101 may include internal memory sufficient to store the application software instructions. In some mobile devices, the secure memory may be in a separate memory chip coupled to the processor 101. In many mobile devices 1 the internal memory 102 may be a volatile or nonvolatile memory, such as flash memory, or a mixture of both. For the purposes of this description, a general reference to memory refers to all memory accessible by the processor 101, including internal memory 102, removable memory plugged into the mobile device, and memory within the processor 101 itself.
[0034] The foregoing method descriptions and the process flow diagrams are provided merely as illustrative examples and are not intended to require or imply that the steps of the various embodiments must be performed in the order presented. As will be appreciated by one of skill in the art the order of steps in the foregoing embodiments may be performed in any order. Words such as "thereafter," "then," "next," etc. are not intended to limit the order of the steps; these words are simply used to guide the reader through the description of the methods. Further, any reference to claim elements in the singular, for example, using the articles "a," "an" or "the" is not to be construed as limiting the element to the singular.
[0035] The various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
[0036] The hardware used to implement the various illustrative logics, logical blocks, modules, and circuits described in connection with the embodiments disclosed herein may be implemented or performed with a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general-purpose processor may be a microprocessor, but, in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Alternatively, some steps or methods may be performed by circuitry that is specific to a given function.
[0037] In one or more exemplary embodiments, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or transmitted over as one or more instructions or code on a computer- readable medium. The steps of a method or algorithm disclosed herein may be embodied in a processor-executable software module executed which may reside on a computer-readable medium. Computer- readable media includes both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. A storage media may be any available media that may be accessed by a computer. By way of example, and not limitation, such computer-readable media may comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that may be used to carry or store desired program code in the form of instructions or data structures and that may be accessed by a computer. Also, any connection is properly termed a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. Disk and disc, as used herein, includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media. Additionally, the operations of a method or algorithm may reside as one or any combination or set of codes and/or instructions on a machine readable medium and/or computer-readable medium, which may be incorporated into a computer program product.
[0038] The preceding description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the following claims and the principles and novel features disclosed herein.

中文:
[0001]本申请案大体上涉及计算装置用户接口,且更明确地说,涉及适合于配备有触摸屏的移动装置的用户接口。
[0002]配备有触摸屏显示器的移动计算装置已经实现新一代个人电子装置。触摸屏显示器提供图形显示器和柔性用户接口两者,且因此使得装置能够消除物理按键和按钮以有利于较大的显示器。此类移动装置通常包括图形用户接口系统,其使得用户能够通过触摸呈现在显示器上的图标来输入命令。常见形式的用户接口呈现具有多个小按键图标的虚拟键盘或小键盘,其以传统物理键盘或小键盘的方式来布置。尽管虚拟键盘用户接口是直观且有用的,但典型移动装置中的触摸屏显示器的有限大小要求虚拟键盘按键图标在大小上受到限制,从而使得用户难以在没有键入错误的情况下准确地触摸。
[0003]各种实施例实现适合于在具有触摸屏显示器的计算装置上使用的用户接口,其可通过经由用数码相机跟踪用户注视而局部地增加显示图像的一部分的大小来补偿用户手指的大小。在一实施例中,移动装置包括触摸屏显示器和数码相机,所述数码相机经定位以使得其可在用户正查看触摸屏显示器时对用户面部成像。通过在数字图像中跟踪用户眼睛,移动装置可经配置以确定用户正观看的位置,且如果用户注视指向触摸屏显示器上的位置,那么可在大小上增大所显示图像的在用户正观看的位置附近的一部分。当所显示图像包括虚拟键盘或小键盘时,可与用户指尖的尺寸相称地在大小上增大在用户注视附近的按键图标。在一实施例中,可在训练例程中将移动装置校准到用户的手指大小和眼睛。尽管所述实施例可尤其对虚拟键盘用户接口显示有用,但眼睛跟踪局部化放大功能性还可对其它应用程序以及其它形式的计算装置(包括具有大触摸屏显示器的计算装置)有用。
[0005]图1为移动装置的正视图,其展示触摸屏显示器上的用于消息接发应用程序的常规用户接口。
[0006]图2为移动装置的正视图,其展示用于消息接发应用程序的实施例用户接口。
[0007]图3和4为移动装置的正视图,其说明实施例用户接口的操作。
[0008]图5和6为移动装置的正视图,其说明实施例用户接口的校准操作。
[0009]图7为用于实施实施例用户接口的实例方法的过程流程图。
[0010]图8为用于校准实施例用户接口的实施例方法的过程流程图。
[0011]图9为用于实施另一实施例用户接口的实施例方法的过程流程图。
[0012]图10为适合于在实施例中使用的移动装置的组件框图。
[0013]将参看附图来详细地描述各种实施例。只要可能,将在整个图式中使用相同的参考标号来指代相同或相似的部分。对特定实例和实施方案做出的参考是出于说明性目的,且不希望限制本发明或权利要求书的范围。
[0015]配备有触摸屏用户接口/显示器的计算装置在用户输入和材料的图形表示方面提供很大灵活性。然而,此类用户接口并不是没有限制的。举例来说,当常规键盘按照虚拟键盘或小键盘来在典型的移动装置触摸屏显示器上表示时,必须以充分小而足以配合在显示器的宽度和高度内的大小来显示按键图标。此类小按键图标可能非常难以用手指来准确地按压,因为手指所触摸的表面面积将通常涵盖一个以上按键图标。这在图1中说明,图1展示具有上面显示虚拟键盘4的触摸屏2的移动装置1。在如图1中所说明的典型虚拟键盘布局中,试图按压单个按键(例如,“H”按键)的用户手指5还将触摸到周围的按键图标,例如如所说明的用于“G”、“H”、“V”和“B”的按键。因而,此类虚拟键盘和小键盘上的键入错误是常见的,且可能使用户感到沮丧。
[0016]各种实施例通过响应于用户注视而提供所显示图像或图标的选择性增强来提供对小移动装置显示器所具有的此问题的解决方案。在一实施例中,用户可将其计算装置校准到其自身的手指大小且训练所述装置基于对其眼睛拍摄的图像来辨识用户注视的位置。通过确定用户正在触摸屏上观看的位置且基于手指大小校准因数来将虚拟键盘的在用户正观看的位置附近的部分增强一定量,移动装置可提供配合在触摸屏显示器的不动产内但提供对于用户手指来说足够大的按键图标的用户接口键盘或小键盘。在另一实施例中,可将增强显示器的在用户注视内的部分的方法应用于其它应用,例如所显示的地图、照片、文字和其它视觉材料。而且,各种实施例可在包括触摸屏显示器的任何计算装置上实施,甚至是具有大触摸屏显示器的计算装置,因为局部化图像增强功能性可使得用户能够准确按压显示器上的特定位置而不必放大整个图像。
[0017]图2说明在具有触摸屏2的移动装置1上进行操作的实施例。在此实施例中,通过数码相机3对用户眼睛进行成像,数码相机3定位于移动装置1上,使得其可在用户正与触摸屏2交互时对用户面部进行成像。基于用户正观看的位置,移动装置1增大虚拟小键盘虚拟键盘4中的按键图标的大小,使得其与用户手指5相容。如所说明,向试图按压用于字母“H”的虚拟按键的用户呈现与用户指尖一样大的图标。以此方式,按压错误按键或用涵盖多个按键图标的触摸事件将触摸屏接口搞乱的机会得以降低。
[0018]此实施例在图3和4中进一步说明,图3和4展示数码相机3如何对用户眼睛5进行成像以确定触摸屏显示器2上的用户注视6的位置。这可通过使用图像辨识软件定位用户眼睛,且更明确地说,定位用户瞳孔的位置来完成。可通过对用户眼睛的图像应用图像分析规则(其可考虑到用户面部相对于装置触摸屏显示器的定向和位置以及用户瞳孔在所述图像内的位置)且应用三角计算估计用户正观看的位置来完成确定用户注视的位置。随着移动装置根据数码相机3所获得的图像确定用户注视6的位置,在大小上增强虚拟小键盘4的靠近用户注视6的中心的按键图标。因此,随着用户注视6从虚拟键盘4的左手侧(如图3中所示)移位到右手侧(如图4中所示),基于用户手指的校准因数而使虚拟小键盘4的在用户注视6的中心内的那些图标更大。图3和4说明所述实施例如何使得虚拟键盘4能够配合在移动装置触摸屏2的有限空间内,同时还提供足够大以适应用户手指的按键图标。
[0019]尽管图2到4展示响应于用户注视而增强的虚拟键盘4,但所述实施例还可应用于任何虚拟小键盘,例如电话小键盘、计算器小键盘或游戏专用小键盘。因此,提供虚拟键盘4的图像仅仅作为实例而不作为对所述实施例的范围的限制。
[0020]为了将移动装置1校准到用户手指大小,移动装置可要求用户在使用之前将其装置初始化或校准到其手指。图5展示可能在校准过程中在触摸屏2上呈现的实例用户接口显示。在此校准过程中,可提示用户用在文本编辑期间通常使用的手指触摸所述屏幕上的虚拟按键8。在此实施例中,用户仅仅必须触摸所显示的按键图标来使得移动装置1能够确定触摸屏2的由用户指尖接触的表面面积的尺寸。使用所测量的尺寸,装置处理器可计算恰当的用户手指大小校准因数,所述用户手指大小校准因数可存储在存储器中以供用于调整按键图标的大小中,如图2到4中所说明。
[0021]为了将移动装置1校准到用户眼睛以使得其能够准确地确定用户正观看的位置,移动装置1可在训练例程中要求用户训练其装置以跟踪其眼睛。图6展示可能在此类训练例程中在触摸屏2上呈现的简单用户接口显示。举例来说,可提示用户随着触摸屏2上的移动图标(例如蜜蜂9的图像)在触摸屏显示器2上到处移动而跟随所述移动图标。在每一时刻,移动装置1将从数码相机3获得的用户眼睛的图像与移动图标9在显示器2上的位置进行比较,以学习如何使用户眼睛的图像与显示器2上的特定位置相关。由于装置处理器知道图像在触摸屏上的位置,所以处理器可计算差异或校正因数,或者另外调整图像分析规则以使所确定位置与实际位置对准。此类校准/学习过程可用以(例如)通过使用众所周知的机器学习技术和众所周知的图像辨识技术来形成用于分析用户面部图像以确定注视焦点的一组规则。
[0022]图5和6中所说明的校准和训练过程可重复多次以便提供统计上平均的手指大小的校准和眼睛跟踪规则。另外,眼睛跟踪训练可继续进行预定时间周期或直到所显示图像的实际位置与用户正观看的所确定位置之间的误差降低到最大阈值以下为止。
[0023]图7中说明根据实施例的可在计算装置上实施的实例方法10。在方法10中,可当在触摸屏上显示虚拟小键盘时激活眼睛跟踪/显示增强功能性(步骤12)。作为激活眼睛跟踪/显示增强功能性的一部分,计算装置内的处理器可在步骤14中从存储器存取校准值和眼睛跟踪规则。在步骤16中,处理器可激活数码相机3且开始对用户面部和眼睛成像。在步骤18中,处理器可分析用户面部的图像以确定用户注视的位置。如上文所提及,这可通过使用图像辨识软件定位用户瞳孔的位置和定向来完成。确定用户注视的位置可通过将从存储器获得的图像分析规则应用于用户眼睛的图像来完成。此些图像分析规则可在用户训练例程期间确定或调整,如下文参看图8所论述。
[0024]当确定了用户注视的方向或位置时,处理器可在确定步骤20中将所述位置与所显示的虚拟小键盘的位置进行比较以确定用户是否正观看小键盘。如果处理器确定用户目前不在观看虚拟键盘(即,确定步骤20=“否”),那么处理器可通过返回到步骤18来继续从数码相机获得用户眼睛的图像并对其进行分析。如果处理器确定用户目前正在观看虚拟键盘(即,确定步骤20=“是”),那么处理器可在步骤22中使用手指大小校准因数来增强在用户注视的中心附近的按键图标。处理器接着通过返回步骤18来继续从数码相机接收图像并对其进行分析。此过程可继续进行,只要在触摸屏显示器上显示虚拟键盘,或直到所述功能性以其它方式由用户终止为止。
[0025]除了放大用户正观看的位置附近的虚拟按键的大小之外,处理器还可调整映射到每一按键图标的显示器坐标以匹配于图标的大小,以用于将对触摸屏显示器的触摸辨识为用户输入的目的。图形用户接口功能将所显示图像中的用户接口图标(例如,按键图标)的位置映射到触摸屏接口上的坐标位置或边界,使得装置处理器可辨识何时特定图标正被用户触摸。因此,在步骤22中,处理器还可调整按键图标到显示器坐标的映射以匹配于呈现在显示器上的图标的尺寸。在一实施例中,调整键盘图标的大小可在两个步骤中完成。首先,处理器可基于用户注视的位置和手指校准因数来确定每一按键图标的新大小。接着,在第二步骤中,处理器可调整图标到触摸屏传感器坐标的映射以匹配于各种按键图标的经调整尺寸。
[0026]图8说明可在计算装置上实施以用于训练所述装置并将其校准到用户手指大小和眼睛注视的实例方法30。尽管方法30一起说明校准/训练操作两者,但还可分开完成手指大小校准和眼睛跟踪训练。在方法30中,在步骤32中,计算装置内的处理器可激活校准/训练应用程序,例如响应于用户菜单选择或作为较大装置初始化例程的一部分。在步骤34中,处理器可产生提示用户触摸所述触摸屏上的特定点的显示,例如图5中所说明。在步骤36中,处理器可从触摸屏显示器接收信号或坐标信息且测量用户手指所接触的区域的大小和形状。在步骤38中,处理器可使用接触区域的所测量大小和形状来计算可存储在存储器中的手指大小校准因数。此校准因数可为尺寸值(例如,面积、直径或半径)或放大因数(例如,百分比或数字值),其可应用于图标尺寸以便确定适合于用户手指的图标大小。
[0027]在步骤40中,处理器可激活数码相机且对用户面部和眼睛进行成像。在步骤42中,处理器可分析所获得的图像以在所述图像内定位用户面部,辨识用户眼睛,且获得图像区域内的用户面部的尺寸测量值。此信息可用以产生或调整图像辨识规则,所述图像辨识规则可在将计算装置固持在距用户面部各种距离处时辨识并跟踪用户眼睛。计算装置与用户面部之间的距离将有可能在每当激活所述功能性时以及在用户正与计算装置交互的同时改变。因此,校准的一部分可涉及获得测量值以使得计算装置处理器能够补偿不同成像距离。另外,可在不同面部-相机距离处重复所述训练过程以使得处理器能够产生可补偿此些分隔距离的差异的辨识规则。
[0028]在步骤44中,处理器可显示提示用户跟随显示器上的移动卡通或图标的消息。在步骤46中,处理器可将所述卡通或图标在显示器上移动某一递增的距离,且在步骤48中,处理器可获得用户面部的图像并对其进行分析以确定用户注视的位置。可在步骤46中连续地、递增地或以大步长移动所述卡通或图标(例如,依序移动到角落和中心中的每一者)。当第一次完成眼睛跟踪训练时,步骤48中的注视确定操作可采用默认规则将用户眼睛的图像翻译为计算装置上的注视位置。在步骤50中,计算装置可调整注视确定图像分析规则,使得所确定的位置匹配于显示器上的实际卡通或图标位置。此眼睛跟踪规则校准过程可涉及对在步骤48中用来确定用户注视位置的规则的简单线性调整或变换,使得所确定的位置匹配于所述卡通或图标在显示器上的实际位置。在确定步骤52中,处理器可确定训练过程是否已完成。处理器可通过在显示器上移动所述卡通或图标持续预定时间量、经过预定数目的位置,或直到注视的所确定位置与卡通或图标的实际位置之间的差异始终处于最大阈值内(即,可接受的误差)为止来继续所述训练例程。只要处理器确定训练未完成(即,确定步骤52=“否”),处理器就可继续在显示器上移动卡通或图标,步骤46,确定用户注视的位置,步骤48,且校准注视-图标确定规则,步骤50。一旦处理器确定完成训练(即,确定步骤52=“是”),处理器就可将校准设置和眼睛跟踪图像分析规则存储在存储器中,步骤54。
[0029]各种实施例还可应用于可受益于优先视觉增强的呈现视觉显示的其它类型的应用程序。举例来说,眼睛跟踪局部化放大功能性可在地图或导航程序中有用,使得用户可在较大地图显示的上下文内看到局部地图细节。类似于虚拟小键盘,准许用户通过触摸触摸屏接口来选择特定位置以获得进一步信息的地图或导航应用程序可将地图局部地放大取决于手指大小校准因数的量。以此方式,地图或导航应用程序可经由对特定位置的手指触摸接收用户输入,其中用户指尖的大小将无意地选择错误位置的机会得以降低。
[0030]各种实施例的另一实用应用将是使用眼睛跟踪局部化放大功能性来增强触摸屏显示器上所显示的超链接、因特网网络链接和其它“可点击的”文字。归因于大多数移动装置上的触摸屏显示器的大小较小,所显示的网页图标和文字的大小通常小于大多数用户的指尖。因此,触摸显示屏以“点击”呈图标或超文本形式的超链接可较困难。各种实施例可局部地放大含有用户正观看的超链接的网页和文档,使得超链接图标或文字足够大以使得用户能够准确地触摸所要的超链接而不会无意地触摸不正确的超链接。
[0031]图9说明用于在计算装置上实施所述功能性的实例方法60。在步骤62中,计算装置处理器可响应于地图或其它应用程序的激活而激活视觉跟踪和增强功能性。作为激活眼睛跟踪/显示增强功能性的一部分,计算装置内的处理器可在存储器中存取校准值和眼睛跟踪规则,步骤64。在步骤66中,处理器可激活数码相机且开始对用户面部和眼睛成像。在步骤68中,处理器可分析用户面部的图像以确定用户注视的位置。为了完成此分析,处理器可应用在用户训练例程期间所确定的图像分析规则,所述用户训练例程如上文参看图8所论述。当确定了用户注视的位置时,处理器可在确定步骤70中将所述位置与触摸屏显示器的位置进行比较以确定用户是否正观看计算装置显示器。如果处理器确定用户目前不在观看显示器(即,确定步骤70=“否”),那么处理器可通过返回到步骤68来继续从数码相机获得图像并对其进行分析。如果处理器确定用户目前正在观看显示器(即,确定步骤70=“是”),那么处理器可在步骤72中增强键盘图像的在用户注视的中心附近的部分。处理器接着通过返回步骤68来继续从数码相机获得图像并对其进行分析。此过程可继续进行,只要特定应用程序被激活或在触摸屏显示器上显示特定类型的图像,或直到所述功能性由用户终止为止。
[0032]适合于与各种实施例一起使用的典型移动装置1将共同具有图10中所说明的组件。举例来说,移动接收器装置1可包括耦合到内部存储器102的处理器101、触摸屏显示器103和经定位以使得其可在用户正观看显示器103时对用户眼睛成像的数码相机3。另外,移动装置1可具有连接到无线数据链路的用于发送和接收电磁辐射的天线104和/或耦合到处理器101的蜂窝式电话收发器105。移动装置还通常包括菜单选择按钮或摇臂开关107以用于接收用户输入。虽然图10说明移动计算装置,但其它形式的计算装置(包括个人计算机或膝上型计算机)将通常也包括耦合到内部存储器102的处理器101、触摸屏显示器103和经定位以使得其可在用户正观看显示器103时对用户眼睛成像的数码相机3。另外,数码相机3可构建到计算装置中,如图10所示,或配置为例如借助于电缆(未图示)或无线数据链路(未图示)电耦合到处理器101的单独装置。因此,图10不希望将权利要求书的范围限于呈特定所说明形状因数的移动计算装置。
[0033]处理器101可为任何可编程的微处理器、微型计算机或多处理器芯片,其可通过软件指令(应用程序)配置以执行多种功能,包括本文中所描述的各种实施例的功能。在一些移动装置中,可提供多个处理器101,例如一个处理器专用于无线通信功能,且一个处理器专用于运行其它应用程序。通常,在软件应用程序被存取且被加载到处理器101中之前,其可存储在内部存储器102中。在一些移动装置中,处理器101可包括足以存储应用程序软件指令的内部存储器。在一些移动装置中,安全存储器可在耦合到处理器101的单独存储器芯片中。在许多移动装置1中,内部存储器102可为易失性或非易失性存储器(例如快闪存储器),或两者的混合物。出于本描述的目的,对存储器的一般参考指代可由处理器101存取的所有存储器,包括内部存储器102、插入到移动装置中的可装卸式存储器以及在处理器101自身内的存储器。
[0034]前述方法描述和过程流程图仅作为说明性实例而提供,且不希望要求或暗示必须以所呈现的次序执行各种实施例的步骤。如所属领域的技术人员将了解,前述实施例中的步骤次序可以任何次序执行。例如“此后”、“接着”、“接下来”等词不希望限制步骤的次序;这些词仅仅用以引导读者阅读对方法的描述。另外,例如使用冠词“一”或“所述”以单数形式对权利要求元素的任何参考不应解释为将元素限于单数形式。
[0035]结合本文所揭示的实施例来描述的各种说明性逻辑块、模块、电路和算法步骤可实施为电子硬件、计算机软件或两者的组合。为了清楚地说明硬件与软件的此可互换性,上文已大体上关于其功能性而描述了各种说明性组件、块、模块、电路和步骤。将所述功能性实施为硬件还是软件取决于特定应用和施加于整个系统的设计约束。熟练的技术人员可针对每一特定应用以不同方式来实施所描述的功能性,但所述实施方案决策不应被解释为会导致脱离本发明的范围。
[0036]用以实施结合本文中所揭示的实施例而描述的各种说明性逻辑、逻辑块、模块和电路的硬件可用通用处理器、数字信号处理器(DSP)、专用集成电路(ASIC)、现场可编程门阵列(FPGA)或其它可编程逻辑装置、离散门或晶体管逻辑、离散硬件组件或其经设计以执行本文中所描述的功能的任何组合来实施或执行。通用处理器可以是微处理器,但在替代方案中,处理器可以是任何常规处理器、控制器、微控制器或状态机。处理器还可实施为计算装置的组合,例如,DSP与微处理器的组合、多个微处理器、结合DSP核心的一个或一个以上微处理器或任何其它此类配置。或者,一些步骤或方法可由专用于给定功能的电路执行。
[0037]在一个或一个以上示范性实施例中,所描述的功能可以硬件、软件、固件或其任何组合来实施。如果以软件来实施,那么可将所述功能作为一个或一个以上指令或代码存储在计算机可读媒体上或经由计算机可读媒体传输。本文中所揭示的方法或算法的步骤可在所执行的处理器可执行软件模块中体现,所述处理器可执行软件模块可驻留在计算机可读媒体上。计算机可读媒体包括计算机存储媒体与通信媒体两者,所述通信媒体包括促进将计算机程序从一处传送到另一处的任何媒体。存储媒体可为可由计算机存取的任何可用媒体。举例来说而非限制,所述计算机可读媒体可包含RAM、ROM、EEPROM、CD-ROM或其它光盘存储装置、磁盘存储装置或其它磁性存储装置,或可用以运载或存储呈指令或数据结构形式的所要程序代码且可由计算机存取的任何其它媒体。同样,恰当地将任何连接称作计算机可读媒体。举例来说,如果使用同轴电缆、光纤电缆、双绞线、数字订户线(DSL)或例如红外线、无线电和微波等无线技术从网站、服务器或其它远程源传输软件,那么同轴电缆、光纤电缆、双绞线、DSL或例如红外线、无线电和微波等无线技术包括于媒体的定义中。如本文中所使用,磁盘和光盘包括压缩光盘(CD)、激光光盘、光学光盘、数字多功能光盘(DVD)、软盘和蓝光光盘,其中磁盘通常以磁性方式再现数据,而光盘使用激光以光学方式再现数据。以上各项的组合也应包括在计算机可读媒体的范围内。另外,方法或算法的操作可作为代码和/或指令中的一者或任何组合或集合而驻留在机器可读媒体和/或计算机可读媒体上,所述媒体可并入到计算机程序产品中。
[0038]提供所揭示实施例的先前描述是为了使所属领域的任何技术人员能够制作或使用本发明。所属领域的技术人员将容易明白对这些实施例的各种修改,且在不脱离本发明的精神或范围的情况下,本文所界定的一般原理可应用于其它实施例。因此,本发明不希望限于本文中所展示的实施例,而是应被赋予与所附权利要求书以及本文中所揭示的原理和新颖特征一致的最广范围。