Source: http://www.google.es/patents/US9311115
Timestamp: 2017-12-12 20:23:36
Document Index: 513503196

Matched Legal Cases: ['Application No. 2013200021', 'Application No. 4249', 'Application No. 10', 'Application No. 10', 'Application No. 10', 'Application No. 2012']

Patente US9311115 - Pushing a graphical user interface to a remote device with display rules ... - Google Patentes
A graphical user interface (“GUI”) can be presented on a remote control accessory device that has user input and display devices. The portable media device can provide the accessory with a GUI image to be displayed. The accessory can specify the configuration of the remote GUI image and send that...http://www.google.es/patents/US9311115?utm_source=gb-gplus-sharePatente US9311115 - Pushing a graphical user interface to a remote device with display rules provided by the remote device
Número de publicación US9311115 B2
Número de solicitud US 13/161,339
También publicado como US20110246891
Número de publicación 13161339, 161339, US 9311115 B2, US 9311115B2, US-B2-9311115, US9311115 B2, US9311115B2
Inventores Emily Schubert, Shyam Toprani, Thomas M. Alsina, Szu-Wen Huang
Citas de patentes (115), Otras citas (36), Clasificaciones (3), Eventos legales (1)
US 9311115 B2
A graphical user interface (“GUI”) can be presented on a remote control accessory device that has user input and display devices. The portable media device can provide the accessory with a GUI image to be displayed. The accessory can specify the configuration of the remote GUI image and send that information to the portable media device. The portable media device can generate a GUI image based on the configuration and send the GUI image to the accessory. The accessory can display the GUI image on one part of its display and other information on another part of its display. The portable media device can define different GUI image types for use with different types of input controls of the accessory. Based on which input control is currently active, the portable media device can send the corresponding GUI image to the accessory for display on the accessory display.
1. A method for operating a portable media device, the method comprising, by the portable media device:
establishing communication with an accessory having a display, wherein the accessory and the portable media device are different devices;
receiving a request from the accessory for communication of a remote graphical user interface to replace a local graphical user interface displayed on the display, the request including dimensions of the local graphical user interface;
generating the remote graphical user interface using information and control elements that fit within the dimensions included with the request; and
communicating the remote graphical user interface to the accessory,
wherein the local graphical user interface is generated by program code of the accessory and the remote graphical user interface is generated by program code stored on the portable media device, the program code of the accessory being separate and distinct from the program code of the portable media device.
2. The method of claim 1 wherein the request further comprises one or more of:
size of a character to be displayed in the remote graphical user interface; and
number of lines of text to be displayed in the remote graphical user interface.
3. The method of claim 1 wherein the information used to generate the remote graphical user interface includes a video signal.
4. The method of claim 1 wherein the request defines a location of the remote graphical user interface within a display area of the display of the accessory.
5. The method of claim 1 wherein the information used to generate the remote graphical user interface includes text data.
6. The method of claim 1 wherein when a value associated with the request for the remote graphical user interface is below a predetermined minimum value, defining, by the portable media device, the remote graphical user interface using default parameters.
an accessory interface coupled to the processor, wherein an accessory and the portable media device are different devices,
receive, from the accessory, via the accessory interface, screen information for defining a display window for displaying a remote graphical user interface on a display of the accessory;
generate the remote graphical user interface including a number of graphical elements selected to fit within the display window using program code stored on the portable media device; and
send the remote graphical user interface to the accessory,
wherein the remote graphical user interface replaces at least a portion of a local graphical user interface generated by program code of the accessory when the remote graphical user interface is sent to the accessory, the program code of the accessory being separate and distinct from the program code stored on the portable media device.
8. The portable media device of claim 7 wherein the screen information comprises one or more of:
information about a location of the display window in relation to the display of the accessory;
information about a screen resolution of the display of the accessory; and
information about dimensions of a font to be used for displaying text within the display window.
9. An accessory comprising:
program code that when executed by the controller generates a local graphical user interface configured to control operation of the accessory independent of a portable media device; and
a media device interface coupled to the controller,
wherein the controller in conjunction with the media device interface is configured to:
send information to the portable media device that includes a specification for a display window to be placed on the display, wherein the accessory and the portable media device are different devices;
receive a remote graphical user interface including a number of graphical elements based at least in part on the information sent to the portable media device;
render the display window on the display; and
display the remote graphical user interface within the display window.
10. The accessory of claim 9 wherein the specification further comprises
a number of lines of text to be displayed in the display window.
11. The accessory of claim 9 wherein the controller is further configured to place another display window adjacent to the display window provided by the portable media device and to display information from a second source other than the portable media device within the other display window.
12. The accessory of claim 11 wherein the second source is the accessory.
13. The accessory of claim 11 wherein the other display window occupies a remaining portion of the display of the accessory not occupied by the display window provided by the portable media device.
14. A method executed by an accessory, the method comprising:
displaying a local graphical user interface generated by program code executed by a processor of the accessory;
receiving a user request for display of a remote graphical user interface in place of the local graphical user interface;
sending configuration information for the remote graphical user interface to a portable media device, the configuration information including information specifying a size for the graphical user interface and a location of the graphical user interface within a display area of a display of the accessory, wherein the accessory and the portable media device are different devices;
receiving from the portable media device, the remote graphical user interface, the remote graphical user interface being generated by a processor of the portable media device that is separate and distinct from the accessory and including a number of graphical elements conforming to the configuration information; and
displaying the remote graphical user interface on the display of the accessory at the specified location.
displaying, by the accessory, another graphical user interface adjacent to the graphical user interface received from the portable media device.
16. A method for operating an accessory that is operable using a control mechanism, the method comprising:
requesting a remote user interface from a portable media device in response to a user request for remote content received at a local graphical user interface of the accessory, the accessory;
receiving from the portable media device, a remote user interface associated with the control mechanism,
wherein the accessory and the portable media device are different devices, the local user interface being generated by program code of the accessory and the remote user interface being generated by program code stored on the portable media device, the program code of the accessory being separate and distinct from the program code of the portable media device.
17. The method of claim 16 wherein the control mechanism is selected from the group consisting of a mechanical control device, a touchscreen, and a voice-based input device.
18. A method carried out by a portable media device, the method comprising:
receiving an accessory identifier from an accessory, the accessory identifier being stored on the accessory and defining at least a type of display associated the accessory;
sending to the accessory, a graphical user interface image associated with an input control mechanism of the accessory, the graphical user interface image being stored on the portable media device and associated with the accessory identifier received from the accessory, wherein the accessory and the portable media device are different devices;
receiving a first command from the accessory in response to a first input provided to the accessory using the input control mechanism in cooperation with the graphical user interface image;
based on the received information, sending to the accessory, an updated graphical user interface image associated with the input control mechanism; and
receiving a second command from the accessory, the second command being in response to a second input provided to the accessory using the input control mechanism and the updated graphical user interface image,
wherein the accessory identifier of the accessory is stored separately from the graphical user interface image stored on the portable media device.
19. The method of claim 18 wherein the portable media device has multiple stored graphical user interfaces images associated with other accessory identifiers.
20. An accessory comprising;
a first graphical user interface configured to control accessory functionality made available by content provided by the accessory and independent of a portable media device;
a control mechanism associated with the first graphical user interface;
a media device interface; and
a controller coupled with the control mechanism, the display, and the media device interface, wherein the controller is configured to:
send a request to the portable media device for a second graphical user interface in accordance with a user input received at the first graphical user interface,
receive a second graphical user interface from the portable media device, the second graphical user interface being associated with the control mechanism and configured to control accessory functionality made available by content provided by the portable media device, wherein the accessory and the portable media device are different devices;
receive an input via the control mechanism;
send a command to the portable media device in response to the input; and
receive an update to the second graphical user interface from the portable media device in response to the command, the updated second graphical user interface being associated with the control mechanism.
21. The accessory of claim 20 further comprising a sensor associated with the control mechanism,
receive a signal from the sensor indicative of activation of the control mechanism by a user.
22. The accessory of claim 21 wherein the sensor comprises one or more of: a pressure sensor, a touch sensor, a proximity sensor, or an acoustic sensor.
23. A portable media device comprising;
an accessory interface coupled with the processor,
receive information from an accessory indicating receipt of a user input at a local graphical user interface of the accessory requesting a remote graphical user interface conforming with one or more parameters, the local graphical user interface being provided by the accessory independent of the portable media device; and
send the remote graphical user interface conforming with the one or more parameters to the accessory, wherein the accessory and the portable media device are different devices and the remote graphical user interface sent to the accessory is different than a native graphical user interface of the portable media device.
24. The portable media device of claim 23 wherein the one or more parameters includes a size of the remote graphical user interface on the display and a size of any text included in the remote graphical user interface.
This application is a continuation-in-part of U.S. application Ser. No. 13/032,614 filed Feb. 23, 2011, which is a continuation-in-part of U.S. application Ser. No. 12/683,218 filed Jan. 6, 2010, entitled “Pushing a User Interface to a Remote Device,” which is a continuation-in-part of U.S. application Ser. No. 12/119,960 filed May 13, 2008, entitled “Pushing a User Interface to a Remote Device.” The respective disclosures of all the applications are incorporated by reference herein in their entirety.
The present disclosure relates generally to accessories for portable media devices and in particular to allowing a portable media device to provide a user interface for controlling the media device to an accessory.
Embodiments of the present invention relate to providing a graphical user interface (“GUI”) on a remote control accessory device, where the GUI can be defined and managed by a portable media device rather than the accessory device. The accessory device can provide a combination of user input and visual feedback devices, such as a video screen for presenting information and feedback to a user, along with buttons, knobs, touchscreen and/or touchpad for receiving user input. The portable media device can provide the accessory with an image (referred to herein as a “remote” GUI image) to be displayed on the video screen; the image can include various user interface elements that can resemble or replicate a “native” GUI provided directly on the portable media device. The accessory can send information to the portable media device indicative of a user action taken in response to the displayed image; such information can indicate, for example, that a particular button was pressed or that a particular portion of a touch-sensitive display screen was touched by the user. The portable media device can process this input to identify the action requested by the user and take the appropriate action. The action may include providing to the accessory an updated GUI image to be displayed, where the updated GUI image reflects the user action. In some embodiments, the accessory can provide access to various third party applications resident of the portable media device by interacting with the display of the accessory. In some embodiments, where the display of the accessory is touch-sensitive, the user can directly or indirectly touch a portion of the screen associated with an identifier for a third party application in order to access the corresponding third party application.
FIG. 10 is a flow diagram of a process usable to generate a remote GUI image according to an embodiment of the present invention.
FIG. 11 illustrates character size and spacing parameters that can be specified by a display rule according to an embodiment of the present invention.
FIG. 12 illustrates determination of character size based on an angular size parameter that can be specified by a display rule according to an embodiment of the present invention.
FIG. 13 is a schematic illustrating a display window according to an embodiment of the present invention.
FIG. 14 illustrates an accessory displaying two remote GUI images according to an embodiment of the present invention.
FIG. 15 is a flow diagram of a process for generating a remote GUI image according to an embodiment of the present invention.
FIG. 16 is a flow diagram of a process for receiving a remote GUI image according to an embodiment of the present invention.
FIG. 17 is a schematic illustrating two types of remote GUI images associated with two different input control mechanisms according to an embodiment of the present invention.
FIG. 18 is a block diagram of an accessory according to an embodiment of the present invention.
FIG. 19 is a flow diagram of a process for switching remote GUI images based on currently active input control mechanism according to an embodiment of the present invention.
FIG. 20 is a flow diagram of a process for switching remote GUI images based on currently active input control mechanism according to another embodiment of the present invention.
In accordance with an embodiment of the present invention, when PMD 102 of FIG. 1A is connected to accessory 104 of FIG. 1B, accessory 104 can enter a “remote GUI” operating mode. In this operating mode, PMD 102 can provide a GUI image to accessory 104. Accessory 104 can display the image on display 114. This remote GUI image can be displayed as received, without modification by accessory 104, thereby allowing PMD 102 to control the look of the interface. In some embodiments, PMD 102 provides a remote GUI image that replicates the native GUI of PMD 102. In other embodiments, PMD 102 can provide a remote GUI image that conforms to a modified version of the native GUI. For example, the modified GUI may use different fonts or color schemes, or may include additional or different control options. Certain aspects of the remote GUI image, such as sizes of image elements and accessible functionality can be modified based on commands from the accessory, as described below.
At different times, PMD 102 of FIG. 1A can be coupled to different accessories. Thus, for example, at one time PMD 102 can be coupled to accessory 104 of FIG. 1B and can provide a remote GUI suitable for use with an interface where the user input devices are buttons 116 a-h. At a different time, PMD 102 can be coupled to accessory 150 of FIG. 1C and can provide a remote GUI suitable for use with a touchscreen interface. Because the remote GUI is under the control of PMD 102, a user can interact with PMD 102 directly or through various accessories and experience a generally consistent interface from one configuration to the next. Such consistency can help to make the use of different interfaces, as well as the transition from one accessory to another, more intuitive for the user. In some embodiments, each type accessory interface can have a remote GUI associated with it. Thus, if a single accessory has multiple input interfaces, then each of the input interfaces can have a remote GUI image type associated with it. Thus, a single accessory can have multiple remote GUI image types associated with it.
Storage device 206 may be implemented, e.g., using disk, flash memory, or any other non-volatile storage medium. In some embodiments, storage device 206 can store a database of media assets (also referred to herein as “tracks”), such as audio, video, still images, or the like, that can be played by PMD 202, together with metadata descriptive of each track. Metadata can include, e.g., a media type (audio track, video track, audio book, still image, etc.); an asset title; a name of an artist or performer associated with the asset; composer or author information; asset length; chapter information; album information; lyrics; information about associated artwork or images; description of the asset; and so on. Other information, including programs to be executed by processor 204 (e.g., a remote GUI program) and/or configuration information to be used in executing such programs (e.g., various display rules for a remote GUI image as described below), can be stored in storage device 206.
Accessory I/O interface 214 of PMD 202 and PMD I/O interface 226 of accessory 220 allow PMD 202 to be connected to accessory 220 and subsequently disconnected from accessory 220. As used herein, PMD 202 and accessory 220 are “connected” whenever a communication channel between accessory I/O interface 214 and PMD I/O interface 226 is open and are “disconnected” whenever the communication channel is closed. Connection can be achieved by physical attachment (e.g., between respective mating connectors of PMD 202 and accessory 220), by an indirect connection such as a cable, or by establishing a wireless communication channel. Similarly, disconnection can be achieved by physical detachment, disconnecting a cable, powering down accessory 220 or PMD 202, or closing the wireless communication channel. Thus, a variety of communication channels may be used, including wired channels such as USB, FireWire, or universal asynchronous receiver/transmitter (“UART”), or wireless channels such as Bluetooth. In some embodiments, PMD 202 can be connected to multiple accessories concurrently using any combination of wired and/or wireless communication channels.
In some embodiments, a SetDisplayInfo command can be sent by accessory 220 to PMD 202 to provide information about display 232. In one embodiment, the information can include any or all of: the display dimensions (e.g., in pixels or inches or both) of a display device, color depth information for the display device (e.g., whether the display is color or black and white, the number of distinct color values supported, etc.); the display format of the display device (e.g., analog or digital input and signal formats); and the refresh rate of the display device. In some embodiments, accessory 220 may provide multiple display devices, and the information provided to PMD 202 may include the number of display devices as well as separate configuration information for each display device.
In some embodiments, a SetDisplayRules command can be sent by accessory 220 to PMD 202 to specify desired parameters for remote GUI images. For example, in some embodiments, accessory 220 may be in an environment (e.g., an automobile) where safety concerns indicate that displayed information should be sized and organized such that it can be quickly and easily read by a user (e.g., a driver). Accordingly, in some embodiments, accessory 220 can provide display “rules,” such as a minimum character size (in absolute dimensions such as inches or centimeters or in pixels), minimum character pitch or space between lines (also in absolute dimensions or pixels), maximum number of lines of text per screen, preferred font, minimum size of graphical objects (e.g., icons, streets on a map), or the like. PMD 202 can apply these rules in generating remote GUI images for accessory 220.
In some embodiments, e.g., for automobiles, display rules can be based on national or regional safety regulations. PMD 202 can be programmed with a lookup table or other data structure that associates applicable display rules with various nations or regions, and accessory 220 can provide a nation or region identifier via the SetDisplayRules command. In response to receiving a nation or region identifier, PMD 202 can look up the applicable display rules.
In some embodiments, a SetControlInfo command can be sent by accessory 220 to PMD 202 to provide information about the type, number and location of user input controls 222. In one embodiment, the command can first identify the number of controls. Then, for each control, the command can identify the type (e.g., button, knob, touchscreen) and approximate location of the control. The command can also provide additional parameters describing the control. For example, in the case of a touchscreen, the command can specify the cell size or resolution of the touchscreen.
In some embodiments, an AccessorySettings command can be sent by accessory 220 to PMD 202 to provide information about its characteristics. The AccessorySettings command can be a general-lingo command for providing information about the accessory's capabilities, preferences, and/or settings; in some embodiments, this command is also usable to send information unrelated to remote GUI operations. The payload of the AccessorySettings command can include a token identifying the type of information being provided (e.g., information about display capabilities, information about audio capabilities, etc.) and a descriptor associated with the token that carries the specific information. For example, in the case of a display token, the associated display descriptor can include information indicating properties of the accessory's display screen. In one embodiment, properties can include the vertical and horizontal physical screen size (e.g., in inches or centimeters); the vertical and horizontal screen resolution (e.g., in number of pixels); the vertical and horizontal dimensions of the portion of the screen allocated to the remote GUI (e.g., in number of pixels); the screen's color depth (e.g., in number of bits or simply an indicator of whether the screen is color or black and white); the screen's gamma (color correction) factor; and other features such as whether the display is touch-sensitive, touch resolution (e.g., number of cells, cell size, size of touch-sensitive area).
In some embodiments, the AccessorySettings command can be sent with a “display rules” token, indicating that the associated descriptor identifies display rules, or desired parameters for remote GUI images. As described above, display rules can be based on national or regional safety standards for particular types of accessories, and the display rules token can simply provide an identifier of the applicable nation or region. In other embodiments, the display rules descriptor can provide specific display rules such as minimum character size, minimum line spacing or pitch, maximum number of lines per screen, preferred font, minimum size of graphical objects, etc.
In some embodiments, the AccessorySettings command can also be used to provide information about user input controls that are to be associated with the remote GUI. For example, a token associated with input control information can be defined. The associated descriptor can contain information about the controls, such as the number of controls, the type of each control (e.g., rotational, button, or touchscreen), the approximate location of each control, control parameters (e.g., touchscreen cell size or touch resolution parameters) etc., similarly to the SetControlInfo command described above. The accessory can also assign each control a unique identifier and include the assigned identifier in the control descriptor. They accessory can subsequently use the assigned identifier to specify to the PMD which control was operated.
The EnterRemoteGUIMode and ExitRemoteGUIMode commands described above can allow accessory 220 to switch PMD 202 into or out of remote GUI mode. In some embodiments, PMD 202 may have more than two user interface (“UI”) modes. For example, PMD 202 can have a standard operating mode in which it presents its native GUI on its own display, a remote GUI mode in which it delivers GUI images to accessory 220, and an external UI mode in which an accessory provides a local GUI for controlling PMD 202. In some embodiments, PMD 202 can disable its native GUI when operating in remote GUI mode or external UI mode; in other embodiments, PMD 202 can simultaneously provide a native GUI and a remote GUI or external UI.
At block 506, accessory 220 can provide configuration, environment and/or status information to PMD 202. In one embodiment, any combination of the SetDisplayInfo, SetDisplayRules, SetControlInfo, SetEnvInfo, AccessorySettings, and SetStatusInfo commands described above can be used. Accessory 220 can also provide an accessory-specific (or environment-specific) image element to PMD 202, e.g., using the SetEnvImage command described above. In some embodiments, some or all of the configuration and environment information for accessory 220 can be pre-stored in PMD 202 as described above, and block 506 may include PMD 202 using the accessory identification obtained during block 504 to access the pre-stored configuration and environment information. Accessory 220 can also supplement or override any pre-stored information with new information using the commands described above.
At block 508, PMD 202 generates an initial remote GUI image using the information provided at block 506. In one embodiment, PMD 202 can apply various rules (including display rules received from accessory 220) to adapt the native GUI image to the configuration of accessory 220 and/or to the status or environment of accessory 220. Such rules can be incorporated into a control program executed by processor 204 to generate remote GUI images.
In some embodiments, the remote GUI can be adapted to conform to display rules specified by accessory 220. For example, a display rule may specify a minimum allowable character size, spacing between lines of text, minimum size and/or spacing for graphical objects (e.g., icons) or the like. PMD 202 can modify the native GUI image to conform to the design rules. This can include changing the amount and/or arrangement of information included in a single remote GUI image. Examples are described below.
Additionally, as noted above, accessory 220 may supply an image element to PMD 202. PMD 202 can determine a size and position for the accessory-supplied image element within the remote GUI image. The element might be small, such as a logo in a corner of the display area, or it can be larger as desired.
In another example, the native GUI of PMD 202 may use a light background with dark text and highlighting. This can provide good visibility in daylight, but when driving at night, the resulting brightness might be distracting. In some embodiments PMD 202 can adapt the color scheme depending on whether accessory 220 is in day or night mode—e.g., selecting bright text and dark backgrounds for nighttime use, or dark text and bright backgrounds for daytime use. Day or night mode can be selected based on status information provided by accessory 220; for example, as noted above accessory 220 can use a SetStatusInfo command to indicate when vehicle headlights are turned on or off, and PMD 202 can switch to day night or day mode depending on the state of the headlights. In other embodiments, the remote GUI can incorporate a user-settable option to select day or night mode. In still other embodiments, PMD 202 can select between day or night mode based on a BrightnessAdjust command received from accessory 220; brightness below (or above) a certain threshold can be interpreted as a request for night (or day) mode.
After successful identification and authentication, process 700 can proceed to block 706. At block 706, PMD 202 can provide information related to remote GUI options to accessory 220. In some embodiments, the information can be provided in response to a request from accessory 220; for instance, PMD 202 can receive the GetPMDOptions command described above and provide the options using the RetPMDOptions command described above. At block 708, PMD 202 can receive instructions from the accessory regarding functionalities to be exposed in the remote GUI; for example, PMD 202 can receive a SetPMDOptions command from accessory 220. In some embodiments, at block 708, PMD 202 can also receive a SetDisplayRules command (or other information specifying display rules) from accessory 220.
Generation of the remote GUI image can be similar to that described above with reference to FIG. 5. In some embodiments, the remote GUI image for any particular operating condition is based on the PMD's native GUI image. The native GUI image can be modified to take into account characteristics of the accessory's display, such as size (in pixels or physical dimensions), aspect ratio, color depth, and the like. An image element supplied by the accessory can also be incorporated into the remote GUI image. In some embodiments, the image can be modified to reflect the location of accessory input controls; however, as described above, this is not required. The remote GUI image can also be adapted to the operating environment of a particular accessory (e.g., making GUI elements larger and/or easier to read if the accessory is in an automobile or other vehicle), to display rules specified by the accessory (e.g., using the SetDisplayRules command), and/or to the operating status of the accessory (e.g., day or night mode). The selection of which functionality to expose can be based on environment and/or status information, or the selection can be based on explicit instructions from the accessory (e.g., a SetPMDOptions command).
In the processes described above, a PMD generates a remote GUI image that is displayed by the accessory. As noted, it can be desirable for the PMD to adapt the remote GUI image to the environment of a particular accessory, e.g., complying with safety standards for an automobile. FIG. 10 is a flow diagram of a process 1000 for generating a remote GUI image that can be implemented in a PMD according to an embodiment of the present invention. Process 1000 can be incorporated into any of the processes described above.
At block 1002, a PMD can establish a connection to an accessory. As described above, establishing a connection provides a communication channel, and the connection can be wired or wireless (or a combination thereof) as desired.
At block 1004, the PMD can receive configuration information from the accessory, including a display descriptor and a display rule. The configuration information can also include an instruction to enter remote GUI mode; in some embodiments, the PMD waits to enter remote GUI mode until the accessory instructs it to do so.
The display descriptor can include any information descriptive of a display of the accessory, such as height and width of the screen (or a portion thereof that has been allocated to remote GUI images) in absolute dimensions, resolution of the screen (or relevant portion thereof) in pixels, color depth, signal format, and the like.
Other information useful in assessing the appearance of an image, such as distance from the screen to the user, can also be provided in a display descriptor. In some embodiments, such as automobiles, the user can be a driver who sits in relatively fixed relation to an in-dash console. The distance from the screen to the user can be specified as a range (e.g., based on the range of seat positions provided by the automobile), a minimum distance (e.g., corresponding to seat in its forward-most position), or a maximum distance (e.g., corresponding to seat in its rearward-most position). In some embodiments, the automobile may be able to determine the current seat position for a given driver and provide a current distance based on the current seat position.
A display rule received at block 1004 can specify particular features or qualities that a displayed image should have. Examples of display rules include minimum character size (height and/or width) and minimum spacing or pitch between lines. FIG. 11 illustrates one implementation of defining character height for uppercase characters (distance indicator 1102) and lowercase characters (distance indicator 1104), and line pitch, i.e., distance between adjacent lines (distance indictor 1106). Also illustrated are examples for defining character width (distance indicator 1108) and horizontal pitch or kerning (distance indicator 1110). Any or all of these attributes can be defined by display rules. Minimum sizes can be specified in absolute units (e.g., inches or centimeters), and size can be defined based on on-screen measurements or apparent size (e.g., angular size) based on the user's position relative to the screen.
Other display rules related to formatting of on-screen text can also be specified, e.g., requiring a certain amount of space between lines of text, maximum number of lines per screen, maximum number of characters per line, particular fonts to be used, or the like.
In addition to or instead of character-size rules, other display rules can also be provided. Examples include rules specifying a minimum size of graphical objects such as icons (e.g., the icons shown in FIG. 9A) or on-screen control elements (e.g., arrows 153 and 155 in FIG. 1C), and rules specifying color schemes or minimum contrast requirements. Display rules can be presented using any suitable data structure or format (e.g., key-value pairs).
In some embodiments, a set of display rules can be defined and stored in the PMD in association with a rule identifier. For example, in the case of automobiles, different nations or regions can have different safety standards that restrict the amount or density of information that can be presented on a console screen, the fonts or size of text that can be used, or the like. Such rules can be stored in association with a region code identifying the nation or region in which these rules apply. At block 1004, the PMD can receive a region code and retrieve the associated display rules from its internal storage.
Based in part on the display descriptor and display rules, the PMD can generate a remote GUI image (block 1006). Block 1006 can be executed after the PMD is instructed by the accessory to enter remote GUI mode, or portions of block 1006 can be executed in preparation for such an instruction.
In some embodiments, the remote GUI image presents information similar to that which can be presented on the PMD's own display (e.g., as shown in FIGS. 1A-1C and FIGS. 8 and 9A-9B), with adaptations based on the display descriptor and display rules. For example, display rules can be used in determining the amount of text and/or number of icons to be displayed on the screen. Referring again to FIG. 10, in some embodiments, at block 1008, the PMD can calculate a character size in pixels based on the display descriptor and display rules. In one such embodiment, the display descriptor can include the display height in centimeters and vertical resolution in pixels, and the display rule can specify a minimum character height in centimeters, as measured at the screen. To compute a character height in pixels, the PMD can divide the display height by the vertical resolution to determine a pixel height (in centimeters in this embodiment). The PMD can then divide the minimum character height specified in the design rule by the calculated pixel height to determine a character height in pixels. Similar calculations can be done for horizontal character size in pixels. (In some embodiments, the horizontal size is determined by the vertical size and a font selected by the PMD or specified by the display rules.)
At block 1010, the PMD can calculate a line pitch in pixels, based on the display descriptor and a display rule that specifies the minimum line pitch in centimeters, as measured at the screen. The minimum character pitch in pixels can be computed by dividing the minimum character pitch in centimeters by the pixel height. (Kerning, or horizontal character pitch, can be determined similarly.)
FIG. 11 illustrates character size parameters that can be specified by display rules according to an embodiment of the present invention. Menu 1100 consists of a number of selectable items that include image and text elements. As illustrated, display rules can specify any or all of an uppercase character height 1102, lowercase character height 1104, line pitch 1106, character width 1108, and character spacing (or kerning) 1110. In some embodiments, the size and pitch parameters specified by the design rule can be treated as minimum values, and the PMD can use larger values (thus producing larger text or larger spacing) but not smaller values. It is to be understood that it is not necessary to specify all of the parameters illustrated in FIG. 11. For example, it may suffice to specify uppercase character height 1102 and line pitch 1106, allowing other parameters to be determined based on the font used. The font itself can be selected by the PMD or specified in a display rule.
In some embodiments, the display rule can specify a minimum apparent character size and/or line pitch as measured at the user's viewing location. For instance, as shown in FIG. 12, the display rule can specify a minimum angular size δ. The actual size depends on the distance D from the viewer (point 1202) to the display device. Thus, for a display device 1204 at distance D1, the actual size required to produce angular size δ is h1; for a display device 1206 at a larger distance D2, the actual size would be h2, which is larger than h1. The actual size h to produce a desired angular size δ at distance D can be determined using the formula h=2*D*tan(δ/2). Once the actual size h is known, the PMD can use the techniques described above to determine the size in pixels that will produce an image with actual size h.
Referring again to FIG. 10, at block 1012, using the character size and pitch information in combination with the known display size or resolution, the PMD can determine limits on the number of characters per line and/or number of lines per screen. For example, using a line pitch in pixels (determined at block 1010) and a vertical display resolution in pixels (specified in the display descriptor), the PMD can compute the maximum number of lines per screen. Similarly, using character pitch in pixels and a horizontal display resolution in pixels provided in the display descriptor, the PMD can compute the maximum number of characters per line. In other embodiments, limits on lines per screen and/or characters per line can be directly specified in a display rule.
At block 1014, the PMD can determine a scrolling speed based in part on the maximum number of lines per screen. The scrolling speed can determine, e.g., how fast to update the remote GUI image when a user operates a scrolling control such as arrows 153 or 155 in FIG. 1C. For example, slower scrolling speeds may be desirable where fewer lines appear on the screen. Other parameters related to updating can also be determined.
At block 1016, the PMD can select information and control elements to be displayed in the remote GUI image. The selection of information elements can depend in part on the current state of the PMD, which can depend on prior user input. Thus, for example, in one state, the remote GUI image can display a list of album titles; if the user selects an album, the remote GUI image can be updated to display a list of track titles for tracks from the selected album.
The selection of information elements can also depend in part on the amount of available space. One aspect of selecting information elements can include determining how many media assets or other options to represent on a single screen, and this can be determined, e.g., based on the maximum number of lines per screen. Another aspect relates to selecting information about each media asset or other option. For instance, the local GUI of some PMDs can display a list of tracks with the album name and or artist name under each track in smaller print. Where the amount of information that can be displayed is more limited, the remote GUI image can include less information; for example, just the track title might be included, or the track title followed by the artist name on the same line might be included.
Control elements can be selected based in part on the user input controls, to the extent that information about user input controls is provided to the PMD. Thus, for example, if a touchscreen control is available, touchable control elements for navigating a menu or playing a track (e.g., as shown in FIG. 1C) can be included. The size of such elements can be determined based on properties of the touchscreen as specified by the accessory (e.g., using the AccessorySettings command described above).
At block 1018, the PMD can arrange the information and control elements into a screen image. In some embodiments, this can include populating an image buffer or other data-storage array with pixel values representing the image as a color for each pixel. Where text is being displayed, pixel colors can be determined based on characters to be displayed and pre-stored patterns associated with a particular font that can be scaled to the desired size. For a graphical element such as an icon, a pre-stored icon can be scaled to the desired size. Pixel color can be further influenced by environment information (e.g., in the case of a vehicle, whether to operate in day or night mode).
At block 1020, the PMD can generate a video signal based on the screen image. A particular signal format or technique for converting pixel color data to a video signal is not required, and any signal format that is recognizable to the accessory can be used.
It will be appreciated that process 1000 is illustrative and that variations and modifications are possible. Steps described as sequential may be executed in parallel, order of steps may be varied, and steps may be modified, combined, added or omitted. For example, while the process is described with reference to characters and text information, graphical elements such as icons and/or images can also be included in a remote GUI image. Accessory-provided display rules can be used in determining the size, number, and arrangement of graphical elements. In addition, the PMD can select information or control elements to include based on accessory-provided information indicating supported functionalities (e.g., as described above).
To update a remote GUI image, the PMD can repeat portions of process 1000, e.g., selection and arrangement of information elements (and control elements if desired) at blocks 1016 and 1018. The PMD generally does not need to recompute characters per line, lines per screen or other parameters that can affect information selection, unless the accessory sends a new display descriptor or display rule. In some embodiments, the accessory is not permitted to change its display descriptors or display rules during remote GUI operation; for example, the PMD can require that display descriptors and display rules be sent prior to entering remote GUI mode and can ignore any further display descriptors or display rules that are sent while the PMD is operating in remote GUI mode.
In some embodiments, the accessory can define the size of the remote GUI image that is displayed on the display of the accessory. In addition, the accessory can also define where within the display area of the accessory, is the remote GUI image to be displayed. In other words, the accessory can specify what part of the accessory display and how much of the accessory display area is to be occupied by the remote GUI image.
In operation, the accessory can send information defining the remote GUI image size and location to the PMD. Upon receiving this information, the PMD can generate an appropriate remote GUI image that conforms to the provided information and send the generated remote GUI image to the accessory for displaying on the accessory display. In other words, the accessory can define a display window to be displayed on the accessory display. Once the display window is defined, the PMD can generate video information that can fit within the defined display window and send the video information to the accessory. In addition, as described above, the accessory can also specify the type and amount of information that can be displayed in the display window. This provides the accessory with greater control of the look and feel of the remote GUI image.
In some embodiments, the accessory can send information defining the display window to the PMD, e.g., using the AccessorySettings command and the display descriptor token described above or using the SetDisplayInfo and SetDisplayRules commands. In some embodiments, the payload of these commands can include size of a font (e.g., in millimeters or centimeters) to be displayed in the display window, width and height of the display of the accessory (e.g., in millimeters and/or pixels), and one or more offsets associated with the display window (e.g., in millimeters and/or pixels). The offsets can define the size of the display window and a location of the display window within the display of the accessory.
FIG. 13 illustrates a display window/remote GUI image displayed on a display of the accessory according to an embodiment of the present invention. As illustrated in FIG. 13, display window 1302 can be displayed on the display 1304 of accessory 1300. In order to define the location and size of display window 1302, accessory 1300 may send information about the offsets X1-X4 to the PMD. In some embodiments, the offsets X1-X4 can define the bounds of display window 1302 and also define a location of display window 1302 within display 1304 of accessory 1300. In some embodiments, offsets X1-X4 can define a distance from the left, bottom, right, and top edges, respectively, of display 1304. In some embodiments, offsets X-X4 can have a value defined in millimeter, centimeter, percentage, etc. In other embodiments, offsets X1-X4 can define a pixel value by which display window 1302 is to be offset from the edges of accessory display 1304. In this instance, offsets X1-X4 can have a measurement unit/value defined by number of pixels. Some embodiments of the present invention may use both types of measurement units/values or a combination of the measurement units/values described above. By providing the offsets X1-X4, the accessory can define a location on display 1304 where display window 1302 can be located. In some embodiments, accessory 1300 may not provide values for all of the offsets X1-X4 to the PMD. In such instances, the PMD can consider the value of the offsets not provided by the accessory as 0 (zero) or some other predetermined “default” value. It is to be noted that display window 1302 can be placed anywhere on display 1304 by providing the requisite offset values for some or all of offsets X1-X4. In some embodiments, the accessory may only provide values for offsets X1 and X2 and based on these values and the display dimensions of accessory display 1304, the PMD can determine the other offsets X3 and X4 for display window 1302. In this manner, the accessory can control the placement of display window 1302 on accessory display 1304.
In some embodiments, the portion of accessory display 1304 not occupied by display window 1302 can be used by accessory 1300 to display an additional GUI image 1406, as illustrated in FIG. 14. In some embodiments, additional GUI image 1406 can be generated by accessory 1300. For instance, accessory 1300 may wish to display navigation information in addition to the information from the PMD. In such an instance, the accessory can define an appropriate size for display window 1302 to display the information received from the PMD and use the rest of the accessory display 1304 or some portion of the remaining accessory display 1304 to display the navigation information. In some embodiments, additional GUI image 1406 image can be received from an external device other than the PMD. As described earlier, accessory 1300 can arrange display window 1302 and additional GUI image 1406 in any manner it wishes within display 1304. In some embodiments, the accessory can display more than one additional GUI image in the portion of display 1304 not occupied by display window 1302.
In some embodiments, the accessory can send information defining the display window during the identification and authorization process. As described above, the information sent by the accessory can include one or more of: a width and height of the accessory display (in millimeters and/or pixels), a height/size of the font to be used for displaying text in the display window, and one or more offsets associated with the display window. In some embodiments, the offsets can also ensure that the entire remote GUI image provided by the PMD will be displayed on the accessory screen. In some embodiments, depending on the quality and resolution of the accessory display, image information located in certain areas of the accessory display may not be visible to a user. Thus, any text or image placed in this area will be lost to the user. In order to avoid this problem, accessory 1300 can define a “safe” area for its display. The safe area can have offsets such that any information displayed within the safe area is visible and generally without distortion. In some embodiments, the safe area can be defined has having offsets of between 3% and 10% from the edges of accessory display 1304 based on the dimensions of accessory display 1304. For example, if accessory display 1304 has dimensions of 7 inches (width)×5 inches (height), the safe area can be defined as having dimensions of 6.79 inches (width)×4.85 inches (height), which is a 3% offset. Any information displayed within the safe area can be expected to appear without distortion. In other words, the “safe” area can be defined using offsets that are between 3% and 10% on each side of the accessory display.
The accessory can specify the safe area for the accessory display. In some embodiments, the safe area can be specified using offsets X1-X4 as described above. In some embodiments, if the accessory does not provide the offsets X1-X4 or is not capable of providing the offsets, the PMD can either request that the accessory provide its “safe” area offsets or use a default offset of e.g., between 3% and 10%, which can be selected based on display type of the accessory. The display window can be defined to be located within the safe area that is defined based on the accessory-provided or default offsets.
It is to be noted that the default offsets can be different for different displays based on size and resolution of the display. Selection of the default offsets in the absence of instructions by the accessory ensures that the information provided by the PMD will be completely visible on the accessory display without some information being lost or cut-off. In some embodiments, the PMD can determine the default offsets associated with a particular accessory display based on information received from the accessory about the accessory display. For example, the PMD can have a database storing information associating various accessory displays with default offsets. In other embodiments, the PMD can obtain this information from an external entity. In still other embodiments, the accessory can provide its default offsets information to the PMD and the PMD can store the information for subsequent use (e.g., when the accessory reconnects).
In some embodiments, the PMD can define a minimum value for each dimension/offset for the display window. In this instance, if the offset values received by the PMD result in a display window that has dimensions smaller than the minimum values, the PMD can generate the display window using the default offsets or communicate a error message indicating that the display window cannot be generate based on the received inputs.
FIG. 15 is a flow diagram of a process 1500 for operating a PMD according to an embodiment of the present invention. Process 1500 can be performed, e.g., by PMD 202 of FIG. 2.
At block 1502, the PMD can establish a connection to the accessory. At block 1504, the PMD can authenticate and identify the accessory and verify that the accessory supports the remote GUI option. As part of the authentication and identification process or thereafter, the PMD can receive information from the accessory defining a display window for the remote GUI image to be sent by the PMD (block 1506), e.g., using the AccessorySettings command and the display rules and display descriptor tokens. As described above, the information defining the display window can include offset information with respect to the dimensions of the display of the accessory, minimum font size for the characters to be displayed within the display window, resolution of the remote GUI image to be sent to the accessory, etc. Thereafter, at block 1508 the PMD can receive a command from the accessory, e.g., the EnterRemoteGUIMode command, indicating that the PMD should start sending a remote GUI image to the accessory. The PMD can then generate a remote GUI image conforming to the instructions received at block 1506 (block 1510). Thereafter, the PMD can send the remote GUI image to the accessory at block 1512. As described above, in some embodiments, the remote GUI image may include a video signal. It is to be understood that the PMD can continue to operate in the remote GUI mode indefinitely, generating and sending remote GUI images conforming to the instructions.
FIG. 16 is a flow diagram of a process 1600 for displaying a remote GUI image on a display of an accessory according to an embodiment of the present invention. In some embodiments, process 1600 can be performed, e.g., by accessory 220 of FIG. 2.
At block 1602, the accessory can establish a connection to the PMD. At block 1604, the accessory can send information to the PMD defining a display window that is to be used to display a remote GUI image sent by the PMD to the accessory. As described above, the information for defining the display window can include information specifying where within the display area of the accessory (i.e. location) should the display window be shown, the dimensions of the display window, characteristics of text to be displayed in the display window, etc. Thereafter, at block 1606, the accessory can send instructions to the PMD requesting the PMD to initiate the remote GUI mode, e.g., using the EnterRemoteGUIMode command. At block 1608, the accessory can receive a video signal conforming to the display window from the PMD. At block 1610, the accessory can render the display window on its display and present an image corresponding to the received video signal within the display window.
It will be appreciated that processes 1500 and 1600 are illustrative and that variations and modifications are possible. Steps described as sequential may be executed in parallel, order of steps may be varied, and steps may be modified, combined, added or omitted. For example, in some embodiments, in addition to displaying the remote GUI image received from the PMD, the accessory may display another remote GUI image from another source device in a portion of the accessory display not occupied by the remote GUI image received from the PMD. Thus, the accessory can display multiple remote GUI images adjacent to each other. In some embodiments, a portion of the accessory display not used for the remote GUI image can be used to operate an interface to the accessory's functionality such as navigation, climate control, sound system, or the like.
In some embodiments, the remote GUI image can be based in part on the type of accessory and the display capabilities of the accessory. In other embodiments, for a single accessory, multiple types of GUI images can be defined based on the type of input control being used to manipulate the accessory. For instance, based on the type of input control used, information can be presented differently on the accessory display in order to make it easy and convenient for a user to operate the selected input control. Thus, in some embodiments, each type of input control mechanism can have its own associated remote GUI image. In some embodiments, the PMD can determine which GUI image to generate and send to the accessory based on information received about the type of accessory input control mechanism currently being used.
In some embodiments, the accessory may have more than one input control mechanism and the user of the accessory may switch between using one or the other input control mechanism. In some embodiments, each of the input control mechanisms may have an associated remote GUI image type. For example, consider that the accessory is an in-vehicle entertainment system with one or more mechanical input control such as a wheel, a button, a knob, etc. and a display that has touchscreen capabilities. In this instance, the user can interact with the accessory using either the touchscreen or any one of the mechanical input controls. However, depending on the input mechanism used, the information displayed on the display of the accessory can be presented differently to make it easy for the user to operate the accessory. In some embodiments, the accessory can define the remote GUI image that is to be displayed by the accessory for each type of input control mechanism. In other words, if the touchscreen input control of the accessory is being used, the accessory may instruct the PMD to display information in a certain manner. Alternatively, if the user is using one of the mechanical input controls for controlling the accessory, the accessory may instruct the PMD to display information in a different manner. For example, when the touchscreen control is activated, the PMD can send a first remote GUI image as defined by the accessory. Then if a mechanical input control, e.g., a wheel-based control, is activated, the layout of the information in the remote GUI image can be changed as defined by the accessory and the PMD can send a second remote GUI image corresponding to the wheel-based control. In some embodiments, the remote GUI image associated with each input control mechanism is defined to enable ease of use for the user and in some instances, to satisfy certain government rules and regulations.
In some embodiments, the PMD can be preprogrammed with information about which type of remote GUI image is to be presented for each type of input control mechanism. In this instance, once the PMD knows which input control mechanism is being used on the accessory, it can generate and send an appropriate remote GUI image associated with that input control mechanism.
FIG. 17 illustrates two different remote GUI images associated with two different accessory input control mechanisms according to an embodiment of the present invention. In some embodiments, remote GUI image 1702 can be associated with the touchscreen input mechanism of the accessory and can include one or more identifiers 1704. Identifiers 1704 can be arranged in a manner so that a user can easily select any of the identifiers and initiate an action associated with the identifier by touching the screen 1702 at the location of the desired identifier. In some embodiments, identifiers 1704 can include icons, text, images, etc. A scroll bar 1706 can be used to navigate to additional information, e.g., additional identifiers, if any, that may be of interest to the user. In some embodiments, the accessory can control or limit what information is to be presented in remote GUI image 1702 and how the information is to be arranged on the display of the accessory, as described above.
Remote GUI image 1720 can be associated with a different input control of the accessory, e.g., a mechanical control mechanism such as a button or a knob. Remote GUI image 1720 can be different from remote GUI image 1702 in order to facilitate the use of the associated input control mechanism. As illustrated, in this instance, identifiers 1704 are arranged in a list format with additional information 1710 for each identifier 1704 being displayed adjacent to the identifier. A selection window 1708 can be moved up or down the list to select one or more identifiers 1704. For example, if the input control mechanism associated with remote GUI image 1720 is a knob or a wheel-based control mechanism, it may be easy to turn the knob or wheel to move selection window 1708 to select an item 1704 in the list.
Thus, there can be multiple remote GUI image types, each associated with a particular input control of the accessory. Each remote GUI image type can be designed such that it is easy for the user to interact with the accessory using the associated input control mechanism. It is to be noted that the remote GUI images illustrated in FIG. 17 are exemplary and one skilled in the art will realize that many other configurations of remote GUI images are possible and can be implemented depending on type of input control mechanism. For instance, in case of voice-based control, in addition to a remote GUI image, the accessory or the PMD may also audibilize the selections available to the user and also confirm a selection by providing an audio confirmation to the user. Different image types can be different in amount of information presented, arrangement of items, presence or absence of visual highlighting of a current selection, etc.
FIG. 18 is a block diagram of an accessory 1820 according to an embodiment of the present invention. Controller 1824, PMD I/O 1826, audio output 1828, and display 1836 can have similar functionality as the corresponding components described with reference to FIG. 2 above.
Accessory 1820 can also include a first input control mechanism 1822 with an associated sensor 1834 and a second input control mechanism 1830 with an associated sensor 1832. First input control mechanism 1822 can be used to interact with and control accessory 1820. First input control mechanism 1822 can be, for e.g., a touchscreen mechanism, a mechanical control such as a knob, a button, etc., or a voice-based control mechanism. Sensor 1834 can detect activation of first input control mechanism 1822 and send a signal indicating activation of first input control mechanism 1822 to the PMD via PMD I/O 1826. For example, if the first input control mechanism is a wheel-based control, once a user touches, turns, or grabs the wheel, sensor 1834, which can be a pressure sensor, can detect that the user has touched or grabbed the wheel and send a signal to the PMD indicating that the user has activated first input control mechanism 1822. Sensor 1834 can depend on the type of first input control mechanism 1822. For example, sensor 1834 can be any sensor that can detect activation of first input control mechanism 1822. In some embodiments, sensor 1834 can be a pressure sensor, a strain gauge, a sound/acoustic sensor, a touch sensor, a proximity sensor, etc.
In some embodiments, second input control mechanism 1830 can also be, for e.g., a touchscreen mechanism, a mechanical control such as a knob, a button, etc., or a voice-based control mechanism. In some embodiments, sensor 1832 can be a pressure sensor, a strain gauge, a sound/acoustic sensor, touch sensor, a proximity sensor, etc. Sensor 1832 can detect activation of second input control mechanism 1830 and send a signal indicating the activation to the PMD. Thus, sensor 1832 (or sensor 1834) can be any type of sensor that can detect activation of an input control mechanism of the accessory and can generate an appropriate a signal indicating the activation and send the signal to the PMD.
While accessory 1820 is described herein with reference to particular blocks, it is to be understood that these blocks are defined for convenience of description and are not intended to imply a particular physical arrangement of component parts. Further, the blocks need not correspond to physically distinct components. Embodiments of the present invention can be realized in a variety of devices including electronic devices implemented using any combination of circuitry and software. In some embodiments, accessory 1820 can have more than two input control mechanisms. For example, in some embodiments, accessory 1820 may have a touchscreen-based input control, a mechanical input control, and a voice-based input control. In some embodiments, a single sensor can be used to detect activation of the first and the second input control mechanisms.
In operation, the PMD can send a remote GUI image to the accessory based in part on which input control of the accessory is currently in use. In some embodiments, the PMD can dynamically switch remote GUI images that it sends to the accessory depending on which input control of the accessory is currently being used. FIG. 19 is a flow diagram of a process 1900 for providing remote GUI images to an accessory according to an embodiment of the present invention. Process 1900 can be performed, e.g., by PMD 202 of FIG. 2.
At block 1902, the PMD can determine whether it is connected to the accessory. At block 1904, the PMD can send a first GUI image to the accessory based on the current input control mechanism of the accessory being used. In some embodiments, the PMD may send a remote GUI image associated with an input control mechanism that is designated as the “default” input control mechanism. For example, in some embodiments, the mechanical input control mechanism may be designated as the default input control either by the accessory or the PMD and a remote GUI image suited for mechanical control input (e.g., image 1720 of FIG. 17) can be sent. At block 1906, the PMD can receive information indicating that a second or different input control mechanism of the accessory was activated. Based on that information, the PMD can generate a second GUI image that is associated with the second input control mechanism (block 1908) and send the second GUI image to the accessory at block 1910. For example, a touch input may be activated and PMD can switch to image 1702 of FIG. 17.
It will be appreciated that process 1900 is illustrative and that variations and modifications are possible. Steps described as sequential may be executed in parallel, order of steps may be varied, and steps may be modified, combined, added or omitted. For example, prior to block 1904, the PMD may request information, from the accessory, about the currently active input control mechanism of the accessory or the accessory may send this information without any request from the PMD, e.g., using the AccessorySettings command. In some embodiments, the PMD may receive instructions from the accessory to send a remote GUI image prior to block 1904, e.g., using the EnterRemoteGUIMode command.
FIG. 20 is a flow diagram of a process 2000 for controlling an accessory according to an embodiment of the present invention. Process 2000 can be performed, e.g., by accessory 1820 of FIG. 18.
At block 2002, the accessory can determine that it is connected with a PMD. At block 2004, the accessory can receive a first remote GUI image from the PMD that is associated with a first input control mechanism of the accessory. Thereafter, the accessory can detect activation of a second input control mechanism of the accessory at block 2006. Upon detection, the accessory can send a signal to the PMD that the second input control mechanism was activated (block 2008). Based on this information, the PMD can generate a second remote GUI image that is associated with the second input control mechanism. The accessory can receive the second remote GUI image at block 2010 and display the second remote GUI image on a display associated with the accessory (block 2012).
It will be appreciated that process 2000 is illustrative and that variations and modifications are possible. Steps described as sequential may be executed in parallel, order of steps may be varied, and steps may be modified, combined, added or omitted. For example, prior to receiving the first remote GUI image from the PMD, the accessory may inform the PMD about the type of the first input control mechanism so that the accessory may receive the correct remote GUI image from the PMD at block 2004. In some embodiments, instead of receiving the first GUI image associated with the first input control mechanism, the accessory may receive a remote GUI image associated with a “default” input control mechanism regardless of which input control mechanism is currently active. In some embodiments, the accessory can simply report what input control is currently being operated, e.g., using commands described above. Based on the information received from the accessory, the PMD can determine whether the user has switched to a different control and switch the remote GUI image type accordingly. Thus additional commands or signals indicating a change of controls may not be needed.
In some embodiments, the default input control mechanism can be the first input control mechanism, the second input control mechanism or any other input control mechanism. In some embodiments, the default input control mechanism can be designated by the accessory. In other embodiments, the default input control mechanism can be designated by the PMD. In some embodiments, if the accessory is displaying a first remote GUI image associated with a first input control mechanism and it detects activation of a second input control mechanism, the second remote GUI image received from the PMD in response to the activation of the second input control mechanism can include substantially the same information as in the first remote GUI image provided that the user has not operated any of the controls of the second input control mechanism subsequent to the activation.
In some embodiments, the user can perform a first action using the first input control mechanism followed by a second action using the second input control mechanism. The PMD can recognize this and adjust the remote GUI image accordingly. For example, consider that the accessory is currently displaying a remote GUI image (e.g., image 1720) that shows a list of songs and the currently active input control mechanism is the mechanical input control. As soon as the user touches any part of the accessory display, the accessory can detect that the touchscreen mode has been activated and send that information to the PMD. The PMD in response can send a different remote GUI image, e.g., image 1702, that is associated with the touchscreen input control and the accessory can display that remote GUI image on the display. Consider further that the user selects a song from the list by touching the display at a location where a song identifier associated with that song is displayed. Thereafter, consider that the user issues a voice command “play.” The accessory can detect that the voice-control is now activated and inform the PMD accordingly. The PMD can generate and send another remote GUI image associated with the voice-control mechanism to the accessory. In some embodiments, the newly received remote GUI image can also include the results of execution of the command “play.” Thus, the PMD can dynamically adjust the remote GUI images based on which input control mechanism of the accessory is activated and also seamlessly transition among various types of remote GUI images.
The use of accessory-supplied display rules can allow the accessory (or its manufacturer) to retain at least some control over the appearance of the remote GUI image, even though the image itself is generated by the PMD. Thus, an accessory manufacturer can assure compliance with applicable safety rules and/or provide a more user-friendly interface (e.g., by avoiding small type that may be difficult for a user to read).
In some embodiments, the accessory can verify the identity of the PMD and/or the version of the remote GUI software that is in use by the PMD prior to instructing the PMD to enter remote GUI mode. For example, the accessory can store a valid digital certificate associated with a public key. Prior to entering remote GUI mode, the accessory can require the PMD to present a matching certificate and/or encode a random challenge using a private key associated with the public key in the certificate.
In some embodiments, the accessory can also store a list of allowable versions of the remote GUI software. Prior to entering remote GUI mode, the accessory can determine the version of the remote GUI software supported by the PMD and verify that the supported version corresponds to an allowable version. (In some embodiments, a PMD can support multiple versions of the remote GUI software, and the accessory can select an allowable version from among the supported versions.)
US4414576 25 Sep 1981 8 Nov 1983 Vicon Industries, Inc. Housing assembly for electrical apparatus
US5231872 21 Feb 1991 3 Ago 1993 Ttc/Truck Tech Corp. Tire monitoring apparatus and method
US5995106 19 Ago 1997 30 Nov 1999 Sun Microsystems, Inc. Graphical user interface for displaying and navigating in a directed graph structure
US6160551 20 Mar 1995 12 Dic 2000 Sun Microsystems, Inc. Graphical user interface for displaying and manipulating objects
US6219653 15 Sep 1998 17 Abr 2001 Forest Products International Exchange, Inc. Freight calculation system and method of operation
US6480378 6 Abr 2001 12 Nov 2002 Dean Chang Multi-angle placement rack of personal digital assistant for use in an automobile
US6556219 18 May 1999 29 Abr 2003 Gateway, Inc. Method and system for peripheral device user interface construction
US6772212 8 Mar 2000 3 Ago 2004 Phatnoise, Inc. Audio/Visual server
US6978424 15 Oct 2001 20 Dic 2005 General Instrument Corporation Versatile user interface device and associated system
US6983171 28 Feb 2003 3 Ene 2006 Motorola, Inc. Device and method for communicating teletype information in a vehicle communication system
US7043572 6 Jun 2003 9 May 2006 Microsoft Corporation Method and system for providing a peripheral service to a host computing device
US7302465 22 Oct 2001 27 Nov 2007 Comverse, Inc. Distributed multimedia transfer
US7307573 8 Abr 2005 11 Dic 2007 Ohsung Electronics Co., Ltd. Remote control system and information process system
US7834758 31 Oct 2007 16 Nov 2010 General Motors Llc In-vehicle entertainment method and system for executing the same
US8196044 5 Ene 2004 5 Jun 2012 Microsoft Corporation Configuration of user interfaces
US20010019367 18 Dic 2000 6 Sep 2001 Walton Anthony Stephen Control system
US20010038392 26 Jun 2001 8 Nov 2001 Samsung Electronics Co., Ltd. Browser based command and control home network
US20030041189 3 Jun 2002 27 Feb 2003 Samsung Electronics Co., Ltd. Computer system and method of indicating operating states of peripheral devices thereof
US20030095211 21 Nov 2001 22 May 2003 Satoshi Nakajima Field extensible controllee sourced universal remote control method and apparatus
US20030154291 18 Dic 2002 14 Ago 2003 International Business Machines Corporation Mobile modular computer
US20040028295 7 Ago 2002 12 Feb 2004 Allen Ross R. Portable document scan accessory for use with a wireless handheld communications device
US20040049475 6 Sep 2002 11 Mar 2004 Toshiba Tec Kabushiki Kaisha System and method for globally providing document access history information
US20040179848 11 Mar 2003 16 Sep 2004 Mikhail Belenkii Scintillation free laser communication system
US20040242269 * 3 Sep 2003 2 Dic 2004 Apple Computer, Inc. Automatically updating user programmable input sensors to perform user specified functions
US20050034158 4 Ago 2004 10 Feb 2005 Delavega Gabriel Streaming video selection system and method
US20050146616 15 Feb 2005 7 Jul 2005 Parulski Kenneth A. Printing system and method having a digital printer that uses a digital camera image display
US20050222875 2 Abr 2004 6 Oct 2005 Lordeman Frank L System and method for interlinking medical-related data and payment services
US20050237821 11 Feb 2005 27 Oct 2005 Dekker Gerard J Method and system of external data storage
US20060280447 26 May 2006 14 Dic 2006 Eriko Ozaki Information processing apparatus and method of controlling same
US20070043550 16 Ago 2005 22 Feb 2007 Tzruya Yoav M System and method for providing a remote user interface for an application executing on a computing device
US20070049197 31 Ago 2005 1 Mar 2007 Andre Klein Control device for audio players
US20070201705 27 Feb 2006 30 Ago 2007 Apple Computer, Inc. Media delivery system with improved interaction
US20070225048 29 Sep 2006 27 Sep 2007 Fujitsu Limited Communication method
US20070240029 15 Jun 2007 11 Oct 2007 Hiroshi Kitada System, computer program product and method for selecting an application service provider
US20080089508 10 Jul 2007 17 Abr 2008 Lee Michael C Call features for automatic call distribution system
US20080184269 31 Ene 2007 31 Jul 2008 Halliburton Energy Services, Inc. Remotely controlling and viewing of software applications
US20080244426 27 Mar 2007 2 Oct 2008 Amulet Technologies, Llc Smart Peripheral Architecture for Portable Media Players
US20090195513 5 Feb 2008 6 Ago 2009 Delphi Technologies, Inc. Interactive multimedia control module
US20090248802 1 Abr 2008 1 Oct 2009 Microsoft Corporation Systems and Methods for Managing Multimedia Operations in Remote Sessions
US20090254839 4 Abr 2008 8 Oct 2009 Microsoft Corporation Techniques to remotely manage a multimedia conference event
US20090254960 16 Mar 2006 8 Oct 2009 Videocells Ltd. Method for a clustered centralized streaming system
US20090284476 13 May 2008 19 Nov 2009 Apple Inc. Pushing a user interface to a remote device
US20100088634 * 23 Ene 2008 8 Abr 2010 Akira Tsuruta Multi-window management apparatus and program, storage medium and information processing apparatus
US20100293462 6 Ene 2010 18 Nov 2010 Apple Inc. Pushing a user interface to a remote device
US20110145863 22 Feb 2011 16 Jun 2011 Apple Inc. Pushing a graphical user interface to a remote device with display rules provided by the remote device
US20110265003 1 Jul 2011 27 Oct 2011 Apple Inc. Pushing a user interface to a remote device
EP1698518A2 3 Mar 2006 6 Sep 2006 Ira M. Marlow Multimedia device integration system for vehicles
JP3131798U Título no disponible
JP2000082039A Título no disponible
JP2001142618A Título no disponible
JP2002215483A Título no disponible
JP2004030631A Título no disponible
JP2004121592A Título no disponible
JP2004172695A Título no disponible
JP2005074211A Título no disponible
JP2006115098A Título no disponible
JP2006236323A Título no disponible
JP2006321470A Título no disponible
JPH113314A Título no disponible
JPH1120556A Título no disponible
JPH06275059A Título no disponible
JPH11260045A Título no disponible
KR20040094347A Título no disponible
KR20050072069A Título no disponible
WO2006040951A1 26 Sep 2005 20 Abr 2006 Toyota Jidosha Kabushiki Kaisha In-vehicle handsfree telephone system
WO2009036366A1 12 Sep 2008 19 Mar 2009 Panasonic Avionics Corporation Media device interface system and method for vehicle information systems
WO2009140095A2 4 May 2009 19 Nov 2009 Apple Inc. Pushing a user interface to a remote device
1 "Parrot Launches New Premium Line of Hands-Free Bluetooth, Music Kits with iPod/iPhone Integration Now, Drive and Talk Hands-Free While Listening to your MP3s Through the Car's Sound System", Retrieved from the Internet on Jun. 8, 2009 at http://press.parrot.com/press.php?id-communique=83&langue=usa, 11 pages.
2 Combined Search and Examination Report of Aug. 13, 2009 for Great Britain Application No. GB0907592.0, 6 pages.
3 Examination Report mailed Jan. 30, 2015 in Australian Patent Application No. 2013200021, 3 pages.
4 Examination Report mailed Nov. 26, 2015 in Indian Patent Application No. 4249/KOLNP/2010, 2 pages.
5 Final Office Action mailed on May 18, 2015 for U.S. Appl. No. 12/683,218, 11 pages.
6 Final Office Action of Oct. 3, 2011 for U.S. Appl. No. 12/119,960, 13 pages.
7 International Preliminary Report on Patentability with Written Opinion of Dec. 23, 2009 for PCT Patent Application No. PCT/US2009/042728, mailed on Nov. 17, 2010, 8 pages.
8 International Search Report of Dec. 23, 2009 for PCT Patent Application No. PCT/US2009/042728, 6 pages.
9 JP Office Action (English Translation] mailed Dec. 24, 2013 in JP2012-202681, 4 pages.
10 MAXTech Technology Ltd., "MaxTech's full range of MP3 Digital Audio Products (Portables, Stand-alone Players and Car HiFi System) created big interest during the show Jan. 6-9, 2000", CES 2000/Las Vegas, 2 pages. Retrieved from the Internet at http://web.archive.org/web/20000930170634/www.maxtech.com/hk/t-details.htm>on Sep. 26, 2008, 2 pages.
11 MAXTech Technology Ltd., DA-CS15G, Retrieved from the Internet at http://web.archive.org/web/20010223230441/www.maxtech.com.hk/g-p06.htm>on Sep. 23, 2008, 2 pages.
12 Neuros: Product Detail, Neuros 128 MB MP3 Digital Audio Computer, Retrieved from http://www.neurosaudio.com on Apr. 9, 2003, 6 pages.
13 Non-Final Office Action and Interview Summary of Jun. 8, 2011 for U.S. Appl. No. 12/119,960, 14 pages.
14 Non-Final Office Action mailed on Nov. 4, 2015 for U.S. Appl. No. 12/119,960, 11 pages.
15 Non-Final Office Action of Jul. 13, 2011 for U.S. Appl. No. 12/683,218, 11 pages.
16 Non-Final Office Action of Mar. 14, 2011 for U.S. Appl. No. 12/119,960, 13 pages.
17 Notice of Allowance mailed on Apr. 27, 2015 for U.S. Appl. No. 14/039,557, 15 pages.
18 Notice of Allowance mailed on Nov. 5, 2015 for U.S. Appl. No. 14/226,705, 14 pages.
19 Notice of Allowance mailed on Nov. 6, 2015 for U.S. Appl. No. 14/222,302, 11 pages.
20 Notice of Allowance mailed on Sep. 30, 2015 for U.S. Appl. No. 14/224,830, 13 pages.
21 Office Action (English Translation) mailed Mar. 22, 2015 in Korean Patent Application No. 10-2012-7031480, 6 pages.
22 Office Action mailed Apr. 18, 2014 in U.S. Appl. No. 13/175,581, 8 pages.
23 Office Action mailed Dec. 1, 2014 in U.S. Appl. No. 13/175,581, 11 pages.
24 Office Action mailed Dec. 10, 2014 in U.S. Appl. No. 14/222,302, 34 pages.
25 Office Action mailed Jan. 2, 2014 in U.S. Appl. No. 13/032,614, 30 pages.
26 Office Action mailed Jul. 1, 2014 in U.S. Appl. No. 14/222,302, 29 pages.
27 Office Action mailed Jul. 22, 2014 in Korean Patent Application No. 10-2012-7031480, 5 pages.
28 Office Action mailed Jun. 19, 2014 in U.S. Appl. No. 13/032,614, 24 pages.
29 Office Action mailed Mar. 15, 2013 in U.S. Appl. No. 12/683,218, 11 pages.
30 Office Action mailed Mar. 26, 2014 in U.S. Appl. No. 12/119,960, 12 pages.
31 Office Action mailed Nov. 20, 2014 in U.S. Appl. No. 12/119,960, 13 pages.
32 Office Action mailed Nov. 27, 2014 in Korean Patent Application No. 10-2014-7014708, 7 pages.
33 Office Action mailed Oct. 30, 2013 in U.S. Appl. No. 13/175,581, 8 pages.
34 Office Action mailed Sep. 22, 2014 (English Translation) in Japanese Patent Application No. 2012-202681, 4 pages.
35 Restriction Requirement of Apr. 28, 2011 for U.S. Appl. No. 12/683,218, 4 pages.
36 Restriction Requirement of Feb. 7, 2011 for U.S. Appl. No. 12/119,960, 4 pages.
Clasificación internacional G06F3/01, G06F9/44
Clasificación cooperativa G06F9/4445
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:SCHUBERT, EMILY;TOPRANI, SHYAM;ALSINA, THOMAS M.;AND OTHERS;SIGNING DATES FROM 20110610 TO 20110614;REEL/FRAME:026463/0660