Source: https://patents.google.com/patent/EP2357556A1/en
Timestamp: 2019-04-21 03:17:00+00:00

Document:
A method and system for displaying an on-screen keyboard. An on-screen keyboard and a plurality of input fields are displayed on the touch-sensitive display. Each input field has an associated input interface and the on-screen keyboard is associated with a first input field. A subsequent contact with the touch-sensitive display that corresponds to a second input field is detected. The second input field is associated with a second input interface that is distinct from the on-screen keyboard and the second input interface is compatible with text input form the on-screen keyboard. The second input interface is concurrently displayed with the on-screen keyboard in response to the detected subsequent contact with the second input field.
The disclosed embodiments relate generally to electronic devices with touch-sensitive surfaces, and more particularly, to electronic devices with touch-sensitive surfaces that display on-screen keyboards.
An exemplary user interface object is an on-screen keyboard that allows a user to enter text using a touch screen display. Typically, an on-screen keyboard is displayed when a user selects an input field that requires text entry. Existing methods of displaying and hiding an on-screen keyboard are distracting to a user. Consider a screen with numerous text fields. When a user sequentially selects each text field, the on-screen keyboard is displayed, hidden and then re-displayed. The appearance and disappearance of the on-screen keyboard as the user selects each text field can become very distracting to the user.
Accordingly, there is a need for computing devices with faster, more efficient methods and interfaces for displaying and hiding an on-screen keyboard and other types of on-screen user input interfaces. Such methods and interfaces may complement or replace conventional methods for displaying and hiding on-screen keyboards. Such methods and interfaces reduce the cognitive burden on a user and produce a more efficient human-machine interface. For battery-operated computing devices, such methods and interfaces conserve power and increase the time between battery charges.
In accordance with some embodiments, a method is provided to automatically display and hide an on-screen keyboard and other types of on-screen user input interfaces. The system and method determine where and when the on-screen keyboard is displayed. To prevent distracting the user, after an on-screen keyboard is displayed, the on-screen keyboard continues to be displayed for as long as possible in accordance with some embodiments. The on-screen keyboard continues to be displayed as a user selects other input fields that have associated input interfaces that are compatible with the on-screen keyboard. A user interface is compatible with the on-screen keyboard if the user interface is capable of being controlled by or manipulated by text input. Compatible input interfaces are concurrently displayed with the on-screen keyboard. The on-screen keyboard is enabled to control the compatible input interface while the input interface is displayed. The on-screen keyboard is hidden when a user selects an input field that has an associated input interface that is not compatible with the on-screen keyboard. Such method reduces the distraction to a user and produces a more efficient human-machine interface.
In accordance with some embodiments, a computer-implemented method is performed at an electronic device including one or more processors, memory and a touch-sensitive surface. The method includes displaying an on-screen keyboard and a plurality of input fields on the touch-sensitive display. Each input field has an associated input interface and the on-screen keyboard is associated with a first input field. The method also includes detecting a contact with the touch-sensitive display that corresponds to a second input field. The second input field is associated with a second input interface that is distinct from the on-screen keyboard and the second input interface is compatible with text input from the on-screen keyboard. The method further includes, in response to the detected subsequent contact with the second input field, displaying the second input interface concurrently with the on-screen keyboard.
In accordance with some embodiments, a computer-implemented method is performed at an electronic device including a processor, memory and a touch-sensitive surface. The method includes displaying a program window with a plurality of input fields. The method also includes detecting user contact with a first one of the input fields. The method further includes, if the first input field is associated with text input and an on-screen keyboard is not currently displayed, display the on-screen keyboard. The method further includes, if the first input field is associated with text input and an on-screen keyboard is currently displayed, continue to display the on-screen keyboard. The method further includes, if the first input field is not associated with text input and the on-screen keyboard is currently displayed, remove the on-screen keyboard.
In accordance with some embodiments, a computer-implemented method is performed at an electronic device including one or more processors, memory and a touch-sensitive display. The method includes displaying a first input interface and a plurality of input fields on the touch-sensitive display. Each input field has an associated input interface and the first input interface is associated with a first input field. The method further includes detecting a subsequent contact with the touch-sensitive display that corresponds to a second input field. The second input field is associated with a second input interface that is distinct from the first input interface and the second input interface is compatible with the first input interface. The method further includes, in response to the detected subsequent contact with the second input field, displaying the second input interface concurrently with the first input interface.
In accordance with some embodiments, an electronic device includes a touch-sensitive display, one or more processors, memory, and one or more programs; the one or more programs are stored in the memory and configured to be executed by the one or more processors and the one or more programs include instructions for performing the operations of any of the methods described above. In accordance with some embodiments, a graphical user interface on an electronic device with a touch-sensitive display, a memory, and one or more processors to execute one or more programs stored in the memory includes one or more of the elements displayed in any of the methods described above, which are updated in response to inputs, as described in any of the methods above. In accordance with some embodiments, a computer readable storage medium has stored therein instructions which when executed by an electronic device with a touch-sensitive display, cause the device to perform the operations of any of the methods described above. In accordance with some embodiments, an electronic device includes: a touch-sensitive display; and means for performing the operations of any of the methods described above. In accordance with some embodiments, an information processing apparatus, for use in a electronic device with a touch-sensitive display, includes means for performing the operations of any of the methods described above.
In accordance with some embodiments, an electronic device includes a processing unit and a touch sensitive display unit coupled to the processing unit, the processing unit being configured to: display an on-screen keyboard and a plurality of input fields on the touch-sensitive display unit, each input field having an associated input interface, wherein the on-screen keyboard is associated with a first input field; detect a contact with the touch-sensitive display unit that corresponds to a second input field, wherein the second input field is associated with a second input interface that is distinct from the on-screen keyboard and wherein the second input interface is compatible with text input from the on-screen keyboard; and in response to the detected subsequent contact with the second input field, display the second input interface concurrently with the on-screen keyboard on the touch-sensitive display unit.
In accordance with some embodiments, an electronic device includes a processing unit and a touch sensitive display unit coupled to the processing unit, the processing unit being configured to: display a program window with a plurality of input fields on the touch sensitive display unit; detect a user contact with a first one of the input fields; if the first input field is associated with text input and an on-screen keyboard is not currently displayed, display the on-screen keyboard on the touch-sensitive display unit; if the first input field is associated with text input and an on-screen keyboard is currently displayed, continue to display the on-screen keyboard on the touch-sensitive display unit; and if the first input field is not associated with text input and the on-screen keyboard is currently displayed, remove the on-screen keyboard from display on the touch-sensitive display unit.
In accordance with some embodiments, an electronic device includes a processing unit and a touch-sensitive display unit coupled to the processing unit, the processing unit being configured to: display a first input interface and a plurality of input fields on the touch-sensitive display, each input field having an associated input interface and wherein the first input interface is associated with a first input field; detect a subsequent contact with the touch-sensitive display that corresponds to a second input field, wherein the second input field is associated with a second input interface that is distinct from the first input interface and wherein the second input interface is compatible with the first input interface; and in response to the detected subsequent contact with the second input field, display the second input interface concurrently with the first input interface.
Thus, multifunction devices with touch-sensitive displays are provided with faster, more efficient methods and interfaces for displaying and hiding an on-screen keyboard and other types of on-screen user input interfaces, thereby increasing the effectiveness, efficiency, and user satisfaction with such devices. Such methods and interfaces may complement or replace conventional methods for displaying and hiding on-screen keyboards.
Figures 5A-5G illustrate exemplary user interfaces for displaying and hiding an on-screen keyboard in accordance with some embodiments.
Figures 6A-6C are flow diagrams illustrating methods of for hiding and displaying an on-screen keyboard in accordance with some embodiments.
Figures 6D and 6E are flow diagrams illustrating a method of hiding and displaying an on-screen user input interface in accordance with some embodiments.
Figure 7 is a functional block diagram of an electronic device with a processing unit and a touch-sensitive display unit coupled to the processing unit in accordance with some embodiments.
U.S. Patent Application No. 61/210,331 , "Smart Keyboard Management for a Multifunction Device with a Touch Screen Display" filed March 16, 2009, is incorporated by reference herein in its entirety.
The user interfaces may include one or more soft keyboard or on-screen embodiments. As used herein, "soft keyboard" and "on-screen keyboard" are used interchangeably. The soft keyboard embodiments may include standard (QWERTY) and/or non-standard configurations of symbols on the displayed icons of the keyboard, such as those described in U.S. Patent Applications 11/459,606 , "Keyboards For Portable Electronic Devices," filed July 24, 2006, 11/459,615 , "Touch Screen Keyboards For Portable Electronic Devices," filed July 24, 2006, and 61/210,331 , "Smart Keyboard Management for a Multifunction Device with a Touch Screen Display," filed March 16, 2009 the contents of which are hereby incorporated by reference in their entireties. The keyboard embodiments may include a reduced number of icons (or soft keys) relative to the number of keys in existing physical keyboards, such as that for a typewriter. This may make it easier for users to select one or more icons in the keyboard, and thus, one or more corresponding symbols. The keyboard embodiments may be adaptive. For example, displayed icons may be modified in accordance with user actions, such as selecting one or more icons and/or one or more corresponding symbols. One or more applications on the device may utilize common and/or different keyboard embodiments. Thus, the keyboard embodiment used may be tailored to at least some of the applications. In some embodiments, one or more keyboard embodiments may be tailored to a respective user. For example, one or more keyboard embodiments may be tailored to a respective user based on a word usage history (lexicography, slang, individual usage) of the respective user. Some of the keyboard embodiments may be adjusted to reduce a probability of a user error when selecting one or more icons, and thus one or more symbols, when using the soft keyboard embodiments.
I/O subsystem 106 couples input/output peripherals on device 100, such as touch screen 112 and other input control devices 116, to peripherals interface 118. I/O subsystem 106 may include display controller 156 and one or more input controllers 160 for other input or control devices. The one or more input controllers 160 receive/send electrical signals from/to other input or control devices 116. The other input control devices 11G may include physical buttons (e.g., push buttons, rocker buttons, etc.), dials, slider switches, joysticks, click wheels, and so forth. In some alternate embodiments, input controller(s) 160 may be coupled to any (or none) of the following: a keyboard, infrared port, USB port, and a pointer device such as a mouse. The one or more buttons (e.g., 208, Figure 2) may include an up/down button for volume control of speaker 111 and/or microphone 113. The one or more buttons may include a push button (e.g., 206, Figure 2). A quick press of the push button may disengage a lock of touch screen 112 or begin a process that uses gestures on the touch screen to unlock the device, as described in U.S. Patent Application 11/322,549 , "Unlocking a Device by Performing Gestures on an Unlock Image," filed December 23, 2005, which is hereby incorporated by reference in its entirety. A longer press of the push button (e.g., 206) may turn power to device 100 on or off. The user may be able to customize a functionality of one or more of the buttons. Touch screen 112 is used to implement virtual or soft buttons and one or more soft keyboards.
Touch screen 112 may use LCD (liquid crystal display) technology, LPD (light emitting polymer display) technology, or LED (light emitting diode) technology, although other display technologies may be used in other embodiments. Touch screen 112 and display controller 156 may detect contact and any movement or breaking thereof using any of a plurality of touch sensing technologies now known or later developed, including but not limited to capacitive, resistive, infrared, and surface acoustic wave technologies, as well as other proximity sensor arrays or other elements for determining one or more points of contact with touch screen 112. In an exemplary embodiment, projected mutual capacitance sensing technology is used, such as that found in the iPhoned® and iPod Touch® from Apple Inc. of Cupertino, California.
A touch-sensitive display in some embodiments of touch screen 112 may be as described in the following applications: (1) U.S. Patent Application No. 11/381,313 , "Multipoint Touch Surface Controller," filed May 2, 2006; (2) U.S. Patent Application No. 10/840,862 , "Multipoint Touchscreen," filed May 6, 2004; (3) U.S. Patent Application No. 10/903,964 , "Gestures For Touch Sensitive Input Devices," filed July 30, 2004; (4) U.S. Patent Application No. 11/048,2G4 , "Gestures For Touch Sensitive Input Devices," filed January 31, 2005; (5) U.S. Patent Application No. 11/038,590 , "Mode-Based Graphical User Interfaces For Touch Sensitive Input Devices," filed January 18, 2005; (6) U.S. Patent Application No. 11/228,758 , "Virtual Input Device Placement On A Touch Screen User Interface," filed September 16, 2005; (7) U.S. Patent Application No. 11/228,700 , "Operation Of A Computer With A Touch Screen Interface," filed September 16, 2005; (8) U.S. Patent Application No. 11/228,737 , "Activating Virtual Keys Of A Touch-Screen Virtual Keyboard," filed September 16, 2005; and (9) U.S. Patent Application No. 11/367,749 , "Multi-Functional Hand-Held Device," filed March 3, 2006. All of these applications are incorporated by reference herein in their entirety.
Operating system 126 (e.g., Darwin, RTXC, LINUX, UNIX, OS X, WINDOWS, or an embedded operating system such as VxWorlcs) includes various software components and/or drivers for controlling and managing general system tasks (e.g., memory management, storage device control, power management, etc.) and facilitates communication between various hardware and software components.
Contact/motion module 130 may detect contact with touch screen 112 (in conjunction with display controller 156) and other touch sensitive devices (e.g., a touchpad or physical click wheel). Contact/motion module 130 includes various software components for performing various operations related to detection of contact, such as determining if contact has occurred (e.g., detecting a finger-down event), determining if there is movement of the contact and tracking the movement across the touch-sensitive surface (e.g., detecting one or more finger-dragging events), and determining if the contact has ceased (e.g., detecting a finger-up event or a break in contact). Contact/motion module 130 receives contact data from the touch-sensitive surface. Determining movement of the point of contact, which is represented by a series of contact data, may include determining speed (magnitude), velocity (magnitude and direction), and/or an acceleration (a change in magnitude and/or direction) of the point of contact. These operations may be applied to single contacts (e.g., one finger contacts) or to multiple simultaneous contacts (e.g., "multitouch "/multiple finger contacts). In some embodiments, contact/motion module 130 and display controller 156 detects contact on a touchpad. In some embodiments, contact/motion module 130 and controller 160 detects contact on a click wheel.
As discussed in greater detail in the discussion of Figure 3, the on-screen keyboard display module 190 determines where and when to display an on-screen keyboard.
As discussed in greater detail in the discussion of Figure 3, the interface display module 195 determines where and when to display a standard user object.
In some embodiments, the device 100 is a device where operation of a predefined set of functions on the device is performed exclusively through a touch screen 112 and/or a touchpad. By using a touch screen and/or a touchpad as the primary input control device for operation of the device 100, the number of physical input control devices (such as push buttons, dials, and the like) on the device 100 may be reduced.
The predefined set of functions that may be performed exclusively through a touch screen and/or a touchpad include navigation between user interfaces. In some embodiments, the touchpad, when touched by the user, navigates the device 100 to a main, home, or root menu from any user interface that may be displayed on the device 100. In such embodiments, the touchpad may be referred to as a "menu button." In some other embodiments, the menu button may be a physical push button or other physical input control device instead of a touchpad.
In some embodiments, application 136-1 includes a plurality of event handlers 190 and one or more application views 191, each of which includes instructions for handling touch events that occur within a respective view of the application's user interface. Each application view 191 of the application 136-1 includes one or more event recognizers 180. Typically, a respective application view 191 includes a plurality of event recognizers 180. In other embodiments, one or more of event recognizers 180 are part of a separate module, such as a user interface kit (not shown) or a higher level object from which application 136-1 inherits methods and other properties. In some embodiments, a respective event handler 189 includes one or more of: data updater 176, object updater 177, GUI updater 178, and/or event data 179 received from event sorter 170. Event handler 189 may utilize or call data updater 176, object updater 177 or GUI updater 178 to update the application internal state 192. Alternatively, one or more of the application views 191 includes one or more respective event handlers 189. Also, in some embodiments, one or more data updater 176, object updater 177, and GUI updater 178 are included in a respective application view 191.
Event comparator 184 compares the event information to predefined event or sub-event definitions and, based on the comparison, determines an event or sub-event, or determines or updates the state of an event or sub-event. In some embodiments, event comparator 184 includes event definitions 186. Event definitions 186 contain definitions of events (e.g., predefined sequences of sub-events), for example, event 1 (187-1), event 2 (187-2), and others. In some embodiments, sub-events in an event 187 include, for example, touch begin, touch end, touch movement, touch cancellation, and multiple touching. In one example, the definition for event 1 (187-1) is a double tap on a displayed object. The double tap, for example, comprises a first touch (touch begin) on the displayed object for a predetermined phase, a first lift-off (touch end) for a predetermined phase, a second touch (touch begin) on the displayed object for a predetermined phase, and a second lift-off (touch end) for a predetermined phase. In another example, the definition for event 2 (187-2) is a dragging on a displayed object. The dragging, for example, comprises a touch (or contact) on the displayed object for a predetermined phase, a movement of the touch across touch-sensitive display 112, and lift-off of the touch (touch end). In some embodiments, the event also includes information for one or more associated event handlers 189.
In some embodiments, event definition 186 includes a definition of an event for a respective user-interface object. In some embodiments, event comparator 184 performs a hit test to determine which user-interface object is associated with a sub-event. For example, in an application view in which three user-interface objects are displayed on touch-sensitive display 112, when a touch is detected on touch-sensitive display 112, event comparator 184 performs a hit test to determine which of the three user-interface objects is associated with the touch (sub-event). If each displayed object is associated with a respective event handler 189, the event comparator uses the result of the hit test to determine which event handler 189 should be activated. For example, event comparator 184 selects an event handler associated with the sub-event and the object triggering the hit test.
In some embodiments, a respective event recognizer 180 activates event handler 189 associated with an event when one or more particular sub-events of an event are recognized. In some embodiments, a respective event recognizer 180 delivers event information associated with the event to event handler 189. Activating an event handler 189 is distinct from sending (and deferred sending) sub-events to a respective hit view. In some embodiments, event recognizer 180 throws a flag associated with the recognized event, and event handler 189 associated with the flag catches the flag and performs a predefined process.
In some embodiments, data updater 176 creates and updates data used in application 136-1. For example, data updater 176 updates the telephone number used in contacts module 137, or stores a video file used in video player module 145. In some embodiments, object updater 177 creates and updates objects used in application 13G-1. For example, object updater 176 creates a new user-interface object or updates the position of a user-interface object. GUI updater 178 updates the GUI. For example, GUI updater 178 prepares display information and sends it to graphics module 132 for display on a touch-sensitive display.
Figure 3 is a block diagram of an exemplary multifunction device with a display and a touch-sensitive surface in accordance with some embodiments. The device 300 need not be portable. In some embodiments, the device 300 is a laptop computer, a desktop computer, a tablet computer, a multimedia player device, a navigation device, an educational device (such as a child's learning toy), a gaming system, or a control device (e.g., a home or industrial controller). The device 300 typically includes one or more processing units (CPU's) 310, one or more network or other communications interfaces 360, memory 370, and one or more communication buses 320 for interconnecting these components. The communication buses 320 may include circuitry (sometimes called a chipset) that interconnects and controls communications between system components. The device 300 includes an input/output (I/O) interface 330 comprising a display 340, which is typically a touch screen display. The I/O interface 330 also may include a keyboard and/or mouse (or other pointing device) 350 and a touchpad 355. Memory 370 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM or other random access solid state memory devices; and may include non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid state storage devices. Memory 370 may optionally include one or more storage devices remotely located from the CPU(s) 310. In some embodiments, memory 370 stores programs, modules, and data structures analogous to the programs, modules, and data structures stored in the memory 102 of portable multifunction device 100 (Figure 1), or a subset thereof. In some embodiments, memory 370 or the computer readable storage medium of memory 370 store the following programs, modules and data structures, or a subset thereof including operation system 126, communication module 128, graphics module 132, text input module 134, on-screen keyboard display module 190 and input interface display module 195.
The graphics module 132 displays graphics on the touch screen 112 or other display. The graphics include various user interfaces such as virtual game controllers, sliders, pickers, wheels and virtual knobs. The user interfaces may be standard user interfaces or custom user interfaces specific to an application programs.
In conjunction with the graphics module 132 and the text input module 134, the on-screen keyboard display module 190 determines where and when an on-screen keyboard is displayed. The on-screen keyboard display module 190 also determines when to hide a displayed on-screen keyboard. In some embodiments, the graphics module 132 and the text input module 134 request display instructions from the on-screen keyboard display module 190. In some embodiments, the on-screen keyboard display module 190 sends display instructions to the graphics module 132 and display instructions to the text input module 134. In some embodiments, the on-screen keyboard display module 190 includes display behavior rules 396 for the on-screen keyboard. Each rule 397 specifies an event and the corresponding display action. For example, a rule 397-1 may specify when the event of rotating the device occurs, the corresponding display action is updating the orientation of the displayed on-screen keyboard.
In some embodiments, the on-screen keyboard display module 190 (or an on-screen input interface display module 195) in conjunction with the graphics module 132, the text input module 134 (or an user input module) and the contact/motion module 130 (Figure 1) perform similar functions in relation to hiding and displaying other types of on-screen user input interfaces and virtual controllers associated with respective input fields displayed on a touch screen display, including determining when those different user input interfaces are compatible. Therefore, the descriptions provided herein with respect to hiding and displaying an on-screen keyboard shall be understood to also apply to hiding and displaying other types of user input interfaces on a touch-sensitive display. In any of these situations a goal is to improve effectiveness of a user interface by displaying on-screen user input interfaces for as long as possible to avoid the distraction of hiding and then redisplaying the same user interface, especially when a plurality of input fields with compatible data types are displayed in a single view on the touch-sensitive display. This includes in some embodiments displaying the compatible on-screen user input interfaces in their same positions to avoid disruption in the appearance of the displayed application.
The on-screen keyboard display module 190 determines whether a user interface is compatible with an on-screen keyboard. As discussed in further detail below, a user interface is compatible with an on-screen keyboard if the user interface is capable of accepting text input. For example, a font select box is compatible with an on-screen keyboard because a font selection box is capable of accepting text input. In this example, the on-screen keyboard can be used to specify or narrow the list of fonts in the font selection box. Based on the comparison, the on-screen keyboard display module 190 determines whether to hide the on-screen keyboard concurrently display the user interface and the on-screen keyboard. In conjunction with the graphics module 132 and the text input module 134, the on-screen keyboard display module 190 determines where to display the on-screen keyboard and the user interface. Other pairs of user input interfaces are considered to be compatible when each interface in the pair is configured to input at least some input values associated with the other interface for their associated input fields.
The on-screen keyboard display module 190 configures user interfaces to accept input from an on-screen keyboard. In some embodiments, the on-screen keyboard module 190 configures the on-screen keyboard to direct user input to the user interfaces. For example, a font selection box may be configured to accept input from an on-screen keyboard and the on-screen keyboard is configured to send user input to the font selection box.
The on-screen keyboard display module 190 determines and maintains user interface state data 380. User interface state information 380 includes information related to the active application(s) 382, active input field(s) 384 or icon(s) and the currently displayed user interface(s) 386. Active applications data 382 includes the name and identifiers of all of the active applications. In some embodiments, the active application data includes the current application view. Active input field data 384 includes the input fields that a user has selected and that are still active. Currently displayed user interface data 386 includes the user interfaces that are currently displayed. The interface state information 380 is used by the on-screen keyboard display module 190 to determine when to hide or when to display the on-screen keyboard. For example, if the currently displayed user interfaces data 396 indicates that the on-screen keyboard is currently displayed and the active field data 384 indicates that a input field corresponding to a volume slider is selected, then the on-screen keyboard display module 190 will hide or instruct a module (e.g., the graphics module 132 and the text input module 134) to hide the on-screen keyboard. The on-screen keyboard module 190 also detects changes in the user interface state data 380 and determines whether to display or hide the on-screen keyboard based on detected changes. For example, when the active input field 384 data or the active application data 382 indicates that a user has exited or switched to a different application or application view, the keyboard display module 190 will hide or instruct a module (e.g., the graphics module 132 and the text input module 134) to hide the on-screen keyboard.
The input interface display module 195 determines where and when an interface object is displayed. The input interface module 195 performs similar operations to the on-screen keyboard display module 190 but is not limited to displaying the on-screen keyboard. The input interface module 195 also maintains and uses user interface state data 388 and display behavior rules 398 in a similar way to the on-screen keyboard display module 190. The input interface display module 195 includes user interface state data 388. User interface state data includes active application data 390, active input field data 392 and current displayed user interfaces data 394. The input interface display module 195 includes display behavior rules 398 for each interface object controlled by the input interface display module 195. The display behavior for each interface includes a one or more display behavior rules 399 that define events and correspond display action.
Figures 5A-5G illustrate exemplary user interfaces for automatically displaying and hiding an on-screen keyboard in accordance with some embodiments. The user interfaces in these figures are used to illustrate the processes described below, including the processes in Figures 6A and 6B. It is noted that the device 100 may be used in either a portrait orientation or a landscape orientation.
Figure 5A illustrates a portable multifunction device 100 displaying multiple exemplary input fields 513 on a display 550. Each field is associated with a user interface object. For example, Font field 506 is associated with a "text/picker" and Volume field 508 is associated with a slider. The user interface associated with the field is displayed in response to user selection of the field. The user interfaces may be custom user interfaces specific to an application or standard user interfaces supplied by the device 100. In some embodiments, the input fields 513 are displayed in response to execution of one or more applications 136 (Figure 1B) or other applications. For example, the fields 513 could be associated with a settings menu associated with a particular application, such as a media player application or a email application. In Figure 5A, the dashed circle 540 corresponds to a finger contact with the touch sensitive display. Figure 5B shows the display 550 when a user selects a "Name" field 502 from Figure 5A. As shown in Figure 5B, a "User Name" field 515, a "Password" field 516 and an on-screen keyboard 520 are displayed in response to selection 540 of the "Name" field 502. The on-screen keyboard 520 may be used to input one or more text characters into the fields S15 and 516.
Figure 5C shows the resulting display 550 when a user selects the "Font" field 506 from the interface state shown in Figure 5B. In response to the user selection 541, the "User Name" field 515 (Figure 5B) and the "Password" field 516 (Figure 5B) disappear and a font selection box 514 is displayed. As shown in Figure 5C, the on-screen keyboard 520 remains. As discussed in further detail below, the on-screen keyboard 520 is displayed while the user makes subsequent selections of input fields that have input interfaces that are compatible with the on-screen keyboard 250. Since both fields are text fields, the on-screen keyboard remains displayed. Continually displaying, hiding and re-displaying the on-screen keyboard is distracting to a user. A user may interact directly with the font selection box 514 to select a font. It is noted that the font selection box 514 is compatible with the on-screen keyboard 520 and is configured to accept text input from the on-screen keyboard 520. As shown in Figure 5C, a user has pressed "A" and "R" as shown by the dashed circles 517 and 518. The font selection box 514 includes font suggestions 521 that start with "AR." A user may also completely type out the name of a font using the on-screen keyboard 520. Some users may prefer to use the on-screen keyboard 520 to control other interfaces such as the font selection box 514.
Figure 5D shows the resulting display 550 when a user selects an input field (i.e., "Volume" field 508) having an associated input interface that is incompatible with the displayed on-screen keyboard 520 shown in Figure 5C. In this case, the "volume" field 508 has a type of user interface (i.e., slider) which is incompatible with a text field. As shown in Figure 5D, when a user selects the "Volume" field 508, the font selection box 514 (Figure 5B) and the on-screen keyboard 520 (Figure 5B) disappear and a volume slider 522 is displayed. As discussed in further detail below, the on-screen keyboard 520 (Figure 5B) is hidden because it is not compatible with the volume slider 522. In particular, a slider does not accept text input. In some embodiments, an on-screen keyboard that includes arrows is compatible with the volume slider 522 and would be concurrently displayed with the volume slider 522. For example, in such an embodiments, by pressing a respective arrow on the on-screen keyboard, the knob 523 on the volume slider 522 would move a direction corresponding to the direction of the respective arrow.
Figure 5E shows a user selection 543 of the "Network" field 512. Figure 5F shows the resulting display 550 when a user selects the "Network" field 512 in Figure 5E. As shown in Figure 5F, the volume slider 522 disappears (Figure 5E), a "Network Name" icon 524 is displayed, a "Password" field 515 is displayed and an on-screen keyboard 520 is displayed. The volume slider 522 disappears because it is incompatible with the on-screen keyboard 520. In some embodiments, only the on-screen keyboard is displayed persistently while the other interfaces are displayed or hidden depending the active field.
Figure 5G shows the resulting display 550 when a user selects the "Icon" field 510 in Figure 5F. As shown in Figure 5F, the "Network Name" field 524, the "Password" field 526 and an on-screen keyboard 520 disappear. An icon input field 528 and an icon selector 530 are displayed. The on-screen keyboard 520 from Figure 5E is hidden because it is incompatible with the icon selector 540. In Figure 5G, the icon field 528 requires icons which the on-screen keyboard 520 does not have.
Figures 6A and 6B are flow diagrams illustrating a method 600 of automatically displaying and hiding an on-screen keyboard in accordance with some embodiments. The method 600 is performed at a electronic device (e.g., device 300, Figure 3, or portable multifunction device 100, Figure 1) with a display and a touch-sensitive surface. In some embodiments, the display is a touch screen display and the touch-sensitive surface is on the display. In some embodiments, the display is separate from the touch-sensitive surface. Some operations in method 600 may be combined and/or the order of some operations may be changed.
As described below, the method 600 provides an intuitive way to display and hide an on-screen keyboard. The method 600 also determines where to display the on-screen keyboard so it does not conflict with the displayed fields associated with the application. In addition, after the on-screen keyboard is displayed, it is displayed in the same position for as long as possible based on compatibility of the input interfaces associated with other fields of the same application window. For example, when used in conjunction with an application (such as web page) with multiple input fields, the on-screen keyboard continues to be displayed as the user selects other input fields until the user selects an input field that has an associated input interface that is incompatible with the on-screen keyboard. The method reduces the distraction to the user when a user selects multiple input fields, thereby creating a more efficient human-machine interface.
In accordance with some embodiments, a computer-implemented method is performed at an electronic device including one or more processors, memory and a touch-sensitive display. Display an on-screen keyboard and plurality of input fields on the touch-sensitive display (602). For example, the input fields may be text fields or drop down menus. In some embodiments, the input fields are icons (e.g., 513, Figure 5A). The on-screen keyboard may correspond to any language and any layout. For example, the on-screen keyboard 520 may be presented as a QWERTY, AZERTY or Dvorak keyboard. Each input field has an associated input interface and the on-screen keyboard is associated with a first input field (602). The user may select one or more soft keys of the on-screen keyboard to enter text into the first input field. The input interface is selected from a group consisting of one or more virtual wheels, selection boxes, icon selectors, pickers, sliders, virtual knobs, virtual game controllers and a plurality of on-screen keyboards (604). For example, in Figure 5A, the "Name" field 502 is associated with text entry interface and the "Volume" field 508 is associated with a slider interface. Each on-screen keyboard corresponds to a respective language or layout (604). For example, there may be a on-screen keyboard for each written and spoken language such as English, French and German. In some embodiments, the input fields are displayed by the graphics module 132.
In some embodiments, the on-screen keyboard is displayed in response to a contact with the touch-sensitive display that corresponds to the first input field (606). In some embodiments, the first input field is a text input field that is associated with an on-screen keyboard. In some embodiments, the first input field is an icon that when selected displays one or more input fields. For example, as shown in Figure 5B, additional input fields (e.g., text field 515, text field 516, Figure 5B) appear in response to user selection of the "Name" field 502.
In some embodiments, the on-screen keyboard is displayed: at a size that is smaller than the area of the touch-sensitive display, at an orientation that corresponds to the orientation of the touch-sensitive display and in a portion of the touch-sensitive display that is determined to minimally interfere with the active application. In some embodiments, the on-screen keyboard is displayed at the bottom of the touch-sensitive display and at an orientation that corresponds to the orientation of the touch-sensitive display. In some embodiments, when the orientation of the touch-sensitive display changes (e.g., from portrait to landscape), the on-screen keyboard is repositioned at the bottom of the touch-sensitive display and at an orientation corresponding to the orientation of the touch-sensitive display. In some embodiments, the on-screen keyboard display module 190 determines where the on-screen keyboard is displayed.
In some embodiments, the on-screen keyboard is not displayed when a hardware keyboard is connected to the electronic device (610). An on-screen keyboard may not be necessary when a hardware keyboard is attached to the electronic device and the on-screen keyboard may unnecessarily occupy display screen space. In some embodiments, when the first input field requires input from the on-screen keyboard, the on-screen keyboard is displayed even though a hardware keyboard is attached to the electronic device. The on-screen keyboard may include one or more symbols that the hardware keyboard does not have. For example, the input field may require Greek symbol input and the on-screen keyboard includes Greek symbols while the hardware keyboard may not. In some embodiments, the on-screen keyboard display module 190 determines whether the on-screen keyboard is displayed.
Detect a subsequent contact with the touch-sensitive display that corresponds to a second input field (612). The second input field is associated with a second input interface that is distinct from the on-screen keyboard and the second input interface is compatible with text input from the on-screen keyboard (612). An input interface that is compatible with the on-screen keyboard is an interface that is capable of accepting input from an on-screen keyboard. In some embodiments, an input interface is compatible with an on-screen keyboard if the input interface is capable of accepting text input. For example, as shown in Figure 5C, a font picker (514, Figure 5C) is compatible with an on-screen keyboard (520, Figure 5C). A user may use the font selection box (514, Figure 5C) interface to select a font. The user may also use the on-screen keyboard (520, Figure 5C) to specify a font or narrow the list of font selections (521, Figure 5C). In another example, a color picker that displays a range of colors is not compatible with an on-screen keyboard because the color picker does not accept text input. In some embodiments, an on-screen keyboard may include one or more keys that allow the on-screen keyboard to be compatible with a greater range of user interfaces. For example, in some embodiments, the on-screen keyboard includes directional arrows that allow the on-screen keyboard to control a knob on a slider, the rotational direction of a virtual wheel and a virtual joystick. Thus, the compatibility of the on-screen keyboard and the user interface will in part depend on the language and layout of the on-screen keyboard. In some embodiments, the on-screen keyboard display module 190 determines whether the input interface and the on-screen keyboard are compatible.
Concurrently display the second input interface with the on-screen keyboard in response to the detected subsequent contact with the second input field (614). In other words, both the on-screen keyboard and the second interface are displayed at the same time. In some embodiments, the second input interface is displayed out of the second input field. In some embodiments, the second input interface is displayed next to or in close proximity to the second input field. As shown in Figure 5C, the on-screen keyboard 520 is concurrently displayed with the font selection box 514. In some embodiments, the on-screen keyboard module 190 determines the how the user interface and the on-screen keyboard are displayed.
In some embodiments, while the second input field is active, enable the user to control the second input interface via the on-screen keyboard (616). In other words, while the second input interface is concurrently displayed with the on-screen keyboard, the on-screen keyboard is configured to control the second input interface. For example, as shown in Figure 5C, the on-screen keyboard can specify or narrow the list of fonts 521 in the font selection box 514. As shown in Figure 5C, when a user selects "A" and then "R" the fonts 521 in the selection box 514 start with "AR."
In some embodiments, while the on-screen keyboard is displayed, subsequent selections with the touch-sensitive display that correspond to input fields having associated input interfaces that are compatible with the on-screen keyboard are detected. In other words, while the on-screen keyboard is displayed, the user selects icons or input fields that have associated input interfaces that are compatible with the on-screen keyboard. In some embodiments, the icons or input fields are text input fields requiring input from an on-screen keyboard. As the user selects each subsequent input field or icon, the associated input interfaces, that are compatible with the on-screen keyboard, are displayed or hidden while the on-screen keyboard is continually displayed. In other words, the on-screen keyboard is displayed as long as the user selects input fields that are associated with input interfaces that are compatible with the on-screen keyboard. For example, Figures 5B shows an on-screen keyboard 520 and text fields 515 and 516. Figure 5C shows the resulting screen when a user selects the "Font" field 506 from interface state shown in Figure 5B. As shown in Figure 5C, the fields 515 and 516 from Figure 5B disappear, a font selection box 514 appears and the on-screen keyboard 520 remains displayed. If the user subsequently selects the "Set Date & Time" field 504 of the user interface state shown in Figure 5C, the font selection box 514 will disappear, new input interfaces corresponding to the "Set Date & Time" field 504 will be displayed and the on-screen keyboard 520 will remain displayed. In summary, after an on-screen keyboard is displayed and as the user makes subsequent selections of input fields, the on-screen keyboard is displayed as long the input interface associated with the subsequently selected input fields are compatible with the on-screen keyboard. Continually hiding and redisplaying the on-screen keyboard is distracting to the user. Some users may prefer to use the on-screen keyboard to control other input views.
In some embodiments, while the second input filed is active and the second input interface is displayed, detect a contact with the touch-sensitive display that corresponds to a third input field (620). The third input field is associated with a third input interface that is not compatible with the on-screen keyboard (620). Display the third input interface and hide the on-screen keyboard in response to the detected contact with the third input field (620). The third input interface is not compatible with the on-screen keyboard when the third input interface is from a group consisting of one or more color pickers, sliders, virtual knobs and virtual game controllers (622). For example, as shown in Figure 5C, while the font selection box 514 and the on-screen keyboard 520 are concurrently displayed, the user selects the "Volume" field 508. After the selection, as shown in Figure 5D, the font selection box 514 disappears, a volume slider 522 is displayed and the on-screen keyboard 520 is hidden. In some embodiments, the on-screen keyboard includes keys corresponding to a first language (624). The third input interface includes keys corresponding to a second language and the third input interface is not compatible with the on-screen keyboard when the second language is different than the first language (624). Thus, when a user selects an input field requiring an on-screen keyboard that corresponds to a different language than the language of the displayed on-screen keyboard, the displayed on-screen keyboard is hidden. In some embodiments, the on-screen display module 190 determines whether the third interface is compatible with the on-screen keyboard and determines whether the on-screen keyboard should hidden.
In some embodiments, when the first electronic device is in a first interface state, detect a transition of the electronic device to a second user-interface state (626). Hide the on-screen keyboard in response to the detecting (626). In some embodiments, the state information includes information about the active application, the active icon or field and the displayed user interfaces. For example, the on-screen keyboard is displayed in a first screen and when a user navigates to a different screen or application (e.g., by selecting an icon or button), the on-screen keyboard is hidden. When a user navigates away from a screen displaying an on-screen keyboard, the on-screen keyboard is no longer necessary and therefore does not need to be displayed. In some embodiments, the user-interface state information is stored in the on-screen keyboard display module 190 as user interface state data 191. In some embodiments, the on-screen keyboard display module 190 collects user interface state data 191 and determines changes in user-interface state.
Figure 6C is a flow diagram illustrating a method 630 of automatically displaying and hiding an on-screen keyboard in accordance with some embodiments. The method 630 is performed at an electronic device (e.g., device 300, Figure 3, or portable multifunction device 100, Figure 1) including one or more processors, memory, a display and a touch-sensitive surface. In some embodiments, the display is a touch screen display and the touch-sensitive surface is on the display. In some embodiments, the display is separate from the touch-sensitive surface. Some operations in method 630 may be combined and/or the order of some operations may be changed.
In some embodiments, display a program window with a plurality of input fields (631). Detect a user contact with a first one of the input fields is detected (632). Display the on-screen keyboard if the first input field is associated with text input and an on-screen keyboard is not currently displayed (634). Continue displaying the on-screen keyboard if the first input field is associated with text input and an on-screen keyboard is not currently displayed (636). Remove the on-screen keyboard if the first input field is not associated with text input and the on-screen keyboard is currently displayed (638).
Figure 6D and 6E are flow diagrams illustrating a method 640 of automatically displaying and hiding an on-screen user input interface in accordance with some embodiments. The method 640 is performed at an electronic device (e.g., device 300, Figure 3, or portable multifunction device 100, Figure 1) including one or more processors, memory, a display and a touch-sensitive surface. In some embodiments, the display is a touch screen display and the touch-sensitive surface is on the display. In some embodiments, the display is separate from the touch-sensitive surface. Some operations in method 640 may be combined and/or the order of some operations may be changed.
Display a first input interface and a plurality of input fields on the touch-sensitive display (642). Each input field having an associated input interface and the first input interface is associated with a first input field (642). In some embodiments, the first input interface is displayed in response to a contact with the touch-sensitive display that corresponds to the first input field (644).
Detect a subsequent contact with the touch-sensitive display that corresponds to a second input field (646). The second input field is associated with a second input interface that is distinct from the first input interface and the second input interface is compatible with the first input interface (646). In some embodiments, the second input interface is compatible with the first input interface due to the first input interface and the second input interface both being configured to input at least some input values associated with both the first and second input fields (648). For example, the first and second input values may be configured to accept text and the first input interface and the second input interface are configured to input text. In other words, the first and second input interfaces have compatible input values. For example, an on-screen keyboard and a font selection box both accept text. In some embodiments, the input values associated with the first and second input fields can be represented by text input (650). In some embodiments, the first input interface is compatible with text input and the second input interface is an on-screen keyboard (652).
In some embodiments, the first and second input interfaces are respective touch-sensitive graphical interface elements configured to input values with similar respective data types in response to user interaction with the respective first and second touch-sensitive graphical interface elements (654). In some embodiments, the first and second touch-sensitive graphical interface elements are selected from the set comprising: a picker, a virtual wheel, a slider virtual, a virtual game controller, and a virtual knob (656). In some embodiments, the first and second graphical interface elements are selected from the set comprising: a selection box and an icon selector (658).
In some embodiments, while the second input field is active, enable the user to input values for the second input field via the first input interface (660). For example, when a user may use an on-screen keyboard to specify or narrow font selections in a font selection box.
The steps in the information processing methods described above may be implemented by running one or more functional modules in information processing apparatus such as general purpose processors or application specific chips. These modules, combinations of these modules, and/or their combination with general hardware (e.g., as described above with respect to Figures 1A, 1B, 1C and 3) are all included within the scope of protection of the invention.
In accordance with some embodiments, Figure 7 shows a functional block diagram of an electronic device 700 configured in accordance with the principles of the invention as described above. The functional blocks of the device may be implemented by hardware, software, or a combination of hardware and software to carry out the principles of the invention. It is understood by persons of skill in the art that the functional blocks described in Figure 7 may be combined or separated into sub-blocks to implement the principles of the invention as described above. Therefore, the description herein may support any possible combination or separation or further definition of the functional blocks described herein.
As shown in Figure 7, an electronic device 700 includes a processing unit 702 and a touch-sensitive display unit 704 coupled to the processing unit 702. In some embodiments, the processing unit includes an on-screen keyboard displaying unit 706, contact detecting unit 708, a second input interface displaying unit 710, a user input enabling unit 712, a third input interface displaying unit 714, a transition detecting unit 716, and a program window displaying unit 718.
In some embodiments, the processing unit 702 is configured to: display an on-screen keyboard and a plurality of input fields on the touch-sensitive display unit 704 (e.g., with the on-screen keyboard displaying unit 706), each input field having an associated input interface, wherein the on-screen keyboard is associated with a first input field; detect a contact with the touch-sensitive display unit 704 (e.g., with the contact detecting unit 708) that corresponds to a second input field, wherein the second input field is associated with a second input interface that is distinct from the on-screen keyboard and wherein the second input interface is compatible with text input from the on-screen keyboard; and in response to the detected subsequent contact with the second input field, display the second input interface concurrently with the on-screen keyboard on the touch-sensitive display unit 704 (e.g., with the second input interface displaying unit 710).
In some embodiments, the input interface is selected from a group consisting of one or more virtual wheels, selection boxes, pickers, sliders, virtual knobs, virtual game controllers and a plurality of on-screen keyboards, each on-screen keyboard corresponding to a respective language or layout.
In some embodiments, the processing unit 702 is further configured to: while the second input field is active, enable a user of the electronic device to control the second input interface via the on-screen keyboard (e.g., with the user input enabling unit 712).
In some embodiments, the processing unit 702 is further configured to: detect a contact with the touch-sensitive display unit 704 (e.g., with the contact detecting unit 708) that corresponds to a third input field, wherein the third input field is associated with a third input interface that is not compatible with the on-screen keyboard; and in response to the detected contact with the third input field, on the touch sensitive display unit 704, display the third input interface (e.g., with the third input interface displaying unit 714) and hide the on-screen keyboard (e.g., with the on-screen keyboard displaying unit 706).
In some embodiments, the third input interface is not compatible with the on-screen keyboard when the third input interface is from a group consisting of one or more color pickers, sliders, virtual knobs and virtual game controllers.
In some embodiments, the on-screen keyboard includes keys corresponding to a first language; the third input interface includes keys corresponding to a second language; and the third input interface is not compatible with the on-screen keyboard when the second language is different than the first language.
In some embodiments wherein the electronic device is in a first user-interface state, the processing unit 702 is further configured, while the electronic device is in the first user-interface state, to: detect a transition of the electronic device to a second user-interface state (e.g., with a transition detecting unit 716); and in response to the detecting, hide the on-screen keyboard (e.g., with the on-screen keyboard displaying unit 706).
In some embodiments, the on-screen keyboard is not displayed by the processing unit 702 when a hardware keyboard is attached to the electronic device.
In some embodiments, the on-screen keyboard is displayed by the processing unit 702 on the touch-sensitive display unit 704 (e.g., with the on-screen keyboard displaying unit 706) in response to a contact with the touch-sensitive display unit that corresponds to the first input field.
In some embodiments, the processing unit 702 is configured to: display a program window with a plurality of input fields on the touch sensitive display unit 704 (e.g., with the program window displaying unit 718); detect a user contact with a first one of the input fields (e.g., with the contact detecting unit 708); if the first input field is associated with text input and an on-screen keyboard is not currently displayed, display the on-screen keyboard on the touch-sensitive display unit 704 (e.g., with the on-screen keyboard displaying unit 706); if the first input field is associated with text input and an on-screen keyboard is currently displayed, continue to display the on-screen keyboard on the touch-sensitive display unit 704 (e.g., with the on-screen keyboard displaying unit 706); and if the first input field is not associated with text input and the on-screen keyboard is currently displayed, remove the on-screen keyboard from display on the touch-sensitive display unit 704 (e.g., with the on-screen keyboard displaying unit 706).
In some embodiments, the processing unit 702 is configured to: display a first input interface and a plurality of input fields on the touch-sensitive display unit 704 (e.g., with the first input interface displaying unit 720), each input field having an associated input interface and wherein the first input interface is associated with a first input field; detect a subsequent contact with the touch-sensitive display unit that corresponds to a second input field (e.g., with the contact detecting unit 708), wherein the second input field is associated with a second input interface that is distinct from the first input interface and wherein the second input interface is compatible with the first input interface; and in response to the detected subsequent contact with the second input field, display the second input interface concurrently with the first input interface (e.g., with the second input interface displaying unit 710).
In some embodiments, the second input interface is compatible with the first input interface due to the first input interface and the second input interface both being configured to input at least some input values associated with both the first and second input fields.
In some embodiments, the input values associated with the first and second input fields can be represented with text input.
In some embodiments, the first input interface is an on-screen keyboard and the second input interface is compatible with text input.
In some embodiments, the first input interface is compatible with text input and the second input interface is an on-screen keyboard.
In some embodiments, the first and second input interfaces are respective touch-sensitive graphical interface elements configured to input values with similar respective data types in response to user interaction with the respective first and second touch-sensitive graphical interface elements.
In some embodiments, the first and second touch-sensitive graphical interface elements are selected from the set comprising: a picker, a virtual wheel, a slider virtual, a virtual game controller, and a virtual knob.
In some embodiments, the first and second graphical interface elements are selected from the set comprising: a selection box and an icon selector.
In some embodiments, the processing unit is configured to: while the second input field is active: enable the user to input values for the second input field (e.g., with the user input enabling unit 712) via the first input interface.
In some embodiments, the first input interface is displayed by the processing unit 702 (e.g., with the first input interface displaying unit 720) in response to a contact with the touch-sensitive display unit 704 that corresponds to the first input field.
in response to the detected subsequent contact with the second input field, displaying the second input interface concurrently with the on-screen keyboard.
The method of claim 1, wherein the input interface is selected from a group consisting of one or more virtual wheels, selection boxes, pickers, sliders, virtual knobs, virtual game controllers and a plurality of on-screen keyboards, each on-screen keyboard corresponding to a respective language or layout.
enabling a user of the electronic device to control the second input interface via the on-screen keyboard.
in response to the detected contact with the third input field, displaying the third input interface and hiding the on-screen keyboard.
The method of claim 4, wherein the third input interface is not compatible with the on-screen keyboard when the third input interface is from a group consisting of one or more color pickers, sliders, virtual knobs and virtual game controllers.
the third input interface is not compatible with the on-screen keyboard when the second language is different than the first language.
in response to the detecting, hiding the on-screen keyboard.
The method of claim 1, wherein the on-screen keyboard is not displayed when a hardware keyboard is attached to the electronic device.
The method of claim 1, wherein the on-screen keyboard is displayed in response to a contact with the touch-sensitive display that corresponds to the first input field.

References: Application No. 61
 Application No. 11
 Application No. 10
 Application No. 10
 Application No. 11
 Application No. 11
 Application No. 11
 Application No. 11
 Application No. 11
 Application No. 11