Remote display graphics

Images displayed on a source device may be displayed on a target device. The source device may send the target device graphical commands and information to be processed natively at the target device, rather than duplicating the screen of a source device at the target device. Graphical elements to be used by the target device may be sent with the graphical commands or may be sent during out-of-band configuration exchanges between the source and target devices to reduce latency during display sharing.

BACKGROUND

A wide variety of media devices are available for consumers to use for the consumption of an ever growing selection of content. These media devices may include televisions, tablet computers, personal computers, electronic book readers, gaming consoles, set-top boxes, media players, in-vehicle entertainment systems, portable media players, smartphones, and so forth. The content presented by these media devices may include audio, video, electronic books, games, and so forth. The content may be downloaded or streamed from a content provider to the media device.

Users may add or remove different media devices into an environment over time. For example, users may enter a living room with a smartphone or tablet computer, add a Blu-ray or other media player to a home entertainment system, and so forth.

Certain implementations and embodiments will now be described more fully below with reference to the accompanying figures, in which various aspects are shown. However, various aspects may be implemented in many different forms and should not be construed as limited to the implementations set forth herein. Like numbers refer to like elements throughout.

DETAILED DESCRIPTION

Media devices may display audio/visual content for display to a user. These media devices may include televisions, tablet computers, personal computers, electronic book readers, gaming consoles, set-top boxes, media players, in-vehicle entertainment systems, portable media players, tablet computers, smartphones, and so forth. The content may include audio, video, electronic books, games, and so forth. Sometimes a user may wish to take audio/visional content that is generated on one device, such as a mobile phone or tablet, and display that content on another device, such as a large screen television providing a display or home audio video receiver (“AVR”) providing sound output. Copying the output of the display of one device to another device may be referred to as screen or display sharing or mirroring. In display sharing, what is seen on the display of one device (called a source device) is sent to another device (called a target device) for simultaneous display on the target device. In certain configurations the source device and target device may display different content, such as in screen extension or other configurations.

A number of techniques have been developed to share content between devices wirelessly without the use of a cord or other physical connector. Miracast and WiDi (wireless display) are two protocols that enable display sharing between devices. Traditional display sharing techniques are typically designed with viewing content with a relatively low level of user interaction, such as streaming a video or movie that is playing on a mobile device to a television or larger display. Such techniques typically focus on true “mirroring”, that is taking the video display output of one device and copying that onto another device. Thus, traditional display mirroring first involves generating display data to be shown on a display of a source device prior to sending the display data to a target device.

This focus suffers from a number of drawbacks. First, traditional display mirroring is processor intensive. Traditional mirroring may involve capturing the screen display of a source device, encoding rasterized video display data (to reduce its size for transmission), packetizing the encoded data, and sending the packetized data over a wireless connection. Second, traditional display mirroring may suffer from high latency, as the time needed for processing and wireless transmission may be high. Third, traditional mirroring may result in low quality images on a target device due to both the data encoding (which reduces fidelity) and due to differences between screen size and quality between the target device and source device as data rasterized at a fixed resolution (e.g., 720p) cannot be scaled up or down without a loss of quality. For example display data formatted for a small screen may not look good when displayed on a larger (and potentially higher resolution) screen. Other drawbacks also exist. For display mirroring applications that are sensitive to latency, quality, etc., such as video games, web browsing, or other applications, improved display sharing techniques are desired.

Offered is a such an improved display sharing system and method. Instead of transmitting the processed display data of a source device to be mimicked on a target device (which simply copies the image of the source display onto the target display), the source device may transmit higher level display or graphical instructions/commands to the target device. The target device may then execute those graphical instructions natively at the target device, thus rendering on the target device graphics that are configured for the target device, rather than rendering on the target device graphics that are configured for the source device. For example, traditional display mirroring captures display data to be output on a source device and sends it to a target device. Here, higher level graphical commands are captured and sent to a target device. As the graphical commands require significantly less processing for transmission than display data (for example, encoded video data), processing resources may be freed for other tasks. Further, as the graphical commands are generated earlier in the processing timeline than display data, the graphical commands may be sent to the target device earlier, thus allowing for more flexibility in synchronizing the source and target displays when experiencing transmission delays, or the like. Skipping encoding also improves image quality by bypassing lossy compression which results in loss of image data during the encoding process. Also, the graphic instructions are significantly smaller in terms of data size than encoded video data, thus reducing bandwidth consumption for display sharing and reducing latency.

FIG. 1Aillustrates display sharing according to one aspect of the present disclosure. A user102operates a source media device104(S). After a connection and initialization process, described in more detail below, the source media device104(S) connects over a network150to a target media device104(T). The source media device104(S) prepares graphical commands (also called graphical instructions), as shown in block160. The graphical commands may be commands such as calls between an application and a graphics library. The graphical commands may also be instructions sent from a graphics library for execution by a processor, such as a graphics processor. Graphical commands are further discussed below. The graphical commands are transmitted from the source media device104(S) to the target media device104(T), as shown in block162. The graphical commands are then executed by the source media device104(S) to render images on a display of the source media device104(S), as shown in block164. The target media device104(T) receives the graphical commands, as shown in block170, and executes the graphical commands to display images on a display of the target media device104(T), as shown in block172. By sharing graphical instructions in the manner described herein, wireless display sharing may be improved.

Described below is a system of media devices that may be configured to share graphical information to improve display sharing.

Each media device may include a communication module. The communication module is configured to discover other media devices and the services they support and provide a framework for establishing and using connections with the other media devices. The connections may be authenticated, encrypted, and so forth.

The communication module may be configured to operate in conjunction with a content direct sharing module which sends a media stream to a receiving media device. For example, the content direct sharing module uses connections maintained by the communication module to send audio and video data from a tablet computer to a television for display.

The connection module may be implemented as a software stack executing on the media devices in the system. The connection module on a first media device may differ from the connection module on a second media device. For example, the first media device may include a transport module configured to provide secure sockets over Internet Protocol while the second media device may not. The media devices may include the content direct sharing module, the device controller module, or both.

By providing the communication module, the content direct sharing module, and the device controller module on media devices, the system enhances the user experience. Media devices entering the environment with their associated services may be readily added or removed, and content may be easily shared among those media devices.

FIG. 1Bis an illustrative system100for providing a multiple media device infrastructure in which various services of the media devices may be discovered and shared. As used in this disclosure, services are modules or groups of instructions which, when executed by a processor, provide a particular set of functionalities. For example, a video streaming service may be configured to participate in a streaming session with a content provider and output video suitable for presentation on a display.

A user102is depicted with several media devices104(1),104(2), . . . ,104(D). While a single user102is shown, more than one user102may be present. The media devices104are configured to present, store, manipulate, or otherwise participate in the consumption of content106. The media devices104may include televisions, tablet computers, personal computers, electronic book readers, gaming consoles, set-top boxes, media players, in-vehicle entertainment systems, portable media players, smartphones, and so forth.

The media devices104may execute one or more services. These services may be configured to interact with the content106by receiving and processing the content106for presentation, generating a stream of content for presentation, providing various functionality, and so forth. The content106may include audio, video, electronic books, games, and so forth. The media devices104are discussed below in more detail with regard toFIG. 2.

The media devices104include a communication module108. The communications module108is configured to support a framework within which availability of different services on different media devices104may be distributed and connections between those media devices104may be established. These connections may be used to send, receive, control, or otherwise interact with the content106. The communication module108is discussed in more detail below with regard toFIG. 3.

These connections maintained by the communication module108may use one or more networks. For example, the network(s)150illustrated inFIG. 1Amay include local area networks (“LANs”)110and wide area networks (“WANs”)112as illustrated inFIG. 1B. The LANs110may be supported using an access point topology, ad-hoc peer to peer topology, and so forth. The LANs may be wired, wireless, or a combination. The LANs110may be implemented using Ethernet, Wi-Fi, Bluetooth, ZigBee, and so forth. Within the system100, several different LANs110may coexist. For example, media device104(1) and104(6) may be connected to one another using Wi-Fi, while the media device104(2) and104(5) may be interconnected using Bluetooth. The connections may be between media devices104which are on the same or different local area networks110.

The LAN(s)110may connect to one or more WANs112. The WAN112may include one or more public networks such as the Internet, private networks, cellular data networks, or a combination thereof. The WAN112may in turn couple to one or more servers114(1),114(2), . . . ,114(S). The servers114may exchange information with the one or more media devices104. While the servers114are depicted as single servers, in some implementations the servers114or the functions attributed to the servers114may be provided by a plurality of devices. For example, the server114may be implemented as a virtualized server executing across a plurality of physical servers.

In one implementation the server114or another device coupled to the WAN112may provide a proxy service to transfer information from a media device104(1) to another media device104(2) using the WAN112. These media devices104may be on the same or different LANs. In another implementation the media devices104on different LANs110may establish connections with one another over the WAN112without using the proxy service.

The media device104may include other modules such as a content direct sharing module116. The content direct sharing module116provides a media stream118from a first media device104to a second media device104. Certain media devices may be configured differently from other media devices. For example, as illustrated here a tablet computer media device104(1) may be configured to receive content106which may be protected using one or more digital rights management (“DRM”) schemes, while the television media device104(6) may not be configured in this manner. A content direct sharing module116executes on both the tablet media device104(1) and the television media device104(6). The content direct sharing module116(1) of the media device104(1) generates a media stream118which is sent, using the LAN110, to the media device104(6) which presents at least a portion of the media stream118. In one aspect, the media stream118may include coded content106on the source media device104. In another aspect, the media stream118may include graphical commands or calls to be executed by a processor at a target media device to create images for display.

In some implementations the content direct sharing module116which is receiving the media stream118may be disallowed from storing the media stream in permanent or semi-permanent memory. For example, the content direct sharing module116may be configured to store no more than 300 frames of media stream118data at any given time and may also be configured to display received frames in a streaming manner. The content direct sharing module116may also be configured to establish a secured pathway to the display of the media device104executing the content direct sharing module116, to reduce the likelihood of interception of the media stream118by other applications or services executing on the media device104.

The content direct sharing module116may include modules such as a virtual screen module, a virtual audio device module, or both. The virtual screen module may be configured to acquire graphical command data which is designated for presentation on the display of the media device104. The virtual screen module may also be configured to receive graphical command data, such as the media stream118from another media device104. The media stream118may include graphical commands and associated graphical data. The virtual screen module may send the graphical commands and graphical data to a processor of the media device for rendering and display on a display of the media device104. This communication may be facilitated by the communication module108. The virtual audio device module may be configured to acquire audio data which is designated for presentation on speakers of the media device104. Likewise, the virtual audio device module may be configured to receive audio data, such as from the media stream118and present that audio data on the speakers of the media device104.

The content direct sharing module116may be configured to send media streams118to multiple media devices104, either as a multicast or as separate streams. For example, the media device104(1) may provide the media stream118to the media devices104(5) and104(6) for presentation using a multicast. In another example, the media device104(1) may provide a first media stream118(1) to the media device104(5) and a second media stream118(2) to the media device104(6).

The content direct sharing module116may be configured to mirror presentation of content106, or provide different information to different media devices104. In one implementation first content106may be presented on the media device104providing content direct sharing while different second content is sent to another media device104. Continuing the example, the media device104(1) may present a user interface for controlling presentation of the content106, while a media stream118(1) comprising video is sent to the media device104(6) for presentation on a display and a media stream118(2) comprising audio is sent to the media device104(5) for presentation by the speakers.

The media device104may also include a device controller module120. The device controller module120may be configured to generate media presentation data. In one aspect, the media presentation data may include an address and content identifier, which when processed by the receiving media device, initiates a transfer from a server114(1) to the receiving media device over the LAN110and/or WAN112. The address and content identifier may include, for example, a uniform resource locator (“URL”) which directs the target media device to content on the server for retrieval and display on the target media device. Content may be shared from the server114to a media device104through the transmission of graphical commands as described below.

FIG. 2illustrates a block diagram200of the media device104configured to support the multiple media device infrastructure. The media device104may include one or more processors202configured to execute one or more stored instructions. The processors202may comprise one or more cores. The media device104may include one or more input/output (“I/O”) interface(s)204to allow the processor202or other portions of the media device104to communicate with other devices. The I/O interfaces204may comprise inter-integrated circuit (“I2C”), serial peripheral interface bus (“SPI”), Universal Serial Bus (“USB”) as promulgated by the USB Implementers Forum, RS-232, one or more media device interfaces such as High Definition Multimedia Interface (“HDMI”) as promulgated by HDMI Licensing LLC, TOSLINK as promulgated by Toshiba Corp., analog video, analog audio, IEEE 1394 as promulgated by the Institute for Electrical and Electronics Engineers, and so forth.

The I/O interface(s)204may couple to one or more I/O devices206. The I/O devices206may include input devices such as one or more of a camera, a microphone, a touch sensor, a button, and so forth. The I/O devices206may also include output devices such as one or more of a display, audio speakers, haptic output device and so forth. In some embodiments, the I/O devices206may be physically incorporated with the media device104or may be externally placed.

The media device104may also include one or more communication interfaces208. The communication interfaces208are configured to provide communications between the media device104and other devices, such as other media devices104, routers, access points, the servers114, and so forth. The communication interfaces208may include personal area networks, wired and wireless local area networks (“LANs”)110, wide area networks (“WANs”)112, and so forth. For example, Ethernet, Wi-Fi, Bluetooth, ZigBee, and so forth.

The media device104may also include one or more busses or other internal communications hardware or software that allow for the transfer of data between the various modules and components of the media device104.

As shown inFIG. 2, the media device104includes one or more memories210. The memory210comprises one or more computer-readable storage media (“CRSM”). The CRSM may be any one or more of an electronic storage medium, a magnetic storage medium, an optical storage medium, a quantum storage medium, a mechanical computer storage medium and so forth. The memory210provides storage of computer readable instructions, data structures, program modules and other data for the operation of the media device104.

The memory210may include at least one operating system (OS) module212. The OS module212is configured to manage hardware resource devices such as the I/O interfaces204, the I/O devices206, the communication interfaces208, and provide various services to applications or modules executing on the processors202. Also stored in the memory210may be one or more of the following modules. These modules may be executed as foreground applications, background tasks, daemons, and so forth.

A user interface module214is configured to provide a user interface to the user102using the I/O devices206and accept inputs received from the I/O devices206. The user interface may include one or more visual, audible, or haptic elements. For example, the user interface may be configured to provide a graphic user interface, an audible user interface, and so forth.

One or more application modules216may be stored in the memory210. The one or more application modules216provide functionality which interacts with the user102. For example, an application module216may be a game which is playable by the user.

One or more service modules218may be stored in the memory210. The service modules218provide one or more services comprising one or more data processing functionalities to other service modules218, application modules216, or other modules. The one or more functionalities may be associated with presentation of the content106on one of the media devices104. For example, the service modules218may be configured to stream audio or video from a server for presentation, transfer and present images, transfer files, and so forth. Other service modules218may provide functionality such authentication to confirm information such as the ownership or affiliation of a particular media device104.

As described above, the communication module108is configured to establish and support communications between the media device104other media devices104, servers114, and other devices. The communication module108may provide an abstraction to allow the application modules216, service modules218, and so forth to readily interact with corresponding modules on other media devices104. The communication module108may access the communication interfaces208to exchange information between the devices. The communication module108is discussed in more detail below with regard toFIG. 3.

As described above, the content direct sharing module116provides a media stream118from a first media device104to a second media device104. In one implementation the content direct sharing module116may provide for concurrent presentation between the first media device104and the second or more media devices104. In another implementation, the content direct sharing module116may suppress or discontinue presentation on the first media device104while presentation continues on the second or more media devices104. For example, a media presentation application module216on the tablet media device104(1) may be receiving a stream of a movie. The content direct sharing module116may request from the communication module108connections to the pair of television media device104(6) and104(7). The communication module108may establish these connections and the movie may appear simultaneously on the television media devices104(6) and104(7), while a user interface allowing for control of the media presentation application module216may remain on the tablet media device104(1) to facilitate the user's102control of the movie playback. The movie may be stored and streamed from one of the media devices104(such as the tablet media device104(1)) or may be stored at another location, such as a server114, or other location.

A development module220may be stored in the memory210of the media device104. The development module220may include various libraries, modules, functions, and so forth such as may be present in a software development kit or other toolset. The development module220may implement functions such as “default service implementation” in which an initial set of basic service housekeeping issues are defined. The development module220may also implement a “transport manager” configured to select and manage particular transport modules for use based on parameters such as access level associated with the service, what transport module the service was discovered over, and so forth. The transport modules are discussed in more detail below with regard toFIG. 3.

A digital rights management module222may provide support for presenting or processing content106which is protected using one or more digital rights management schemes. Other modules224may also be present. For example, a speech recognition module may be present and used to accept spoken input from the user102as received from a microphone I/O device206.

The memory210may also include a datastore226to store information. The datastore226may use a flat file, database, linked list, tree, or other data structure to store the information. In some implementations, the datastore226or a portion of the datastore226may be distributed across one or more other devices including servers, network attached storage devices and so forth.

The datastore226may store content106, either in its entirety or a portion. One or more route maps228may be stored. The route maps228provide information about services on other media devices104and the routing to access those services.

Service description objects (“SDO”)230may be stored. The SDO230provides information such as service identifiers, access information, encryption information, and so forth, associated with a particular service as provided by a service module218. The service identifiers may be used to indicate a particular service which is available. The access information may include details on account names, passwords, supported transports, and so forth. The encryption information may include data such as encryption keys, protocols, and so forth.

Device objects (“DOs”)232may also be stored in the memory210. These may include a service description array indicating available services local to a particular media device104, universally unique identifier (“UUID”) associated with the particular media device104, friendly name for the particular media device104, transport information, and so forth. Transport modules and the associated information are discussed below with regard toFIG. 3.

The memory210may also store one or more graphics libraries238. The graphics library238may be configured as an implementation of an application programming interface (API) to coordinate interactions between applications and a graphics processor when generating graphics. An application programming interface may include a set of functions or routines, executed in software and/or hardware, to specify how computing components should interact with each other. An API may be in the form of a library that includes specifications for routines, data structures, object classes, variables, etc. A graphics library238may include graphical information for use in generating graphics. The one or more graphics libraries238may be configured for particular applications, operating systems, hardware, etc. As used here, the term graphics library may include the API and an instantiation of the API. In one example, one or more of the graphics libraries238may be an API implementation that interfaces between an application and lower levels according to the Open Graphics Library (OpenGL), Open GL|ES, EGL, or other standards.

Other data234may also be stored. For example, the other data234may include user preferences, configuration files, and so forth.

FIG. 3illustrates a block diagram300of the communication module108configured to support the multiple media device infrastructure. In some implementation the communication module108and the subordinate modules may be implemented as a background process. The communication module108may include modules including one or more transport modules302, a discovery manager304, one or more device explorers306, an authentication manager308, a security manager310, a service registrar312, a connection manager314, and so forth.

The transport modules302are configured to provide functionality at a transport layer and establish and maintain communication channels that transfer bits from one media device104to another device. In one implementation the transport modules302may operate at layer 4 of the Open Systems Interconnection (“OSI”) model. The media device104may have multiple transport modules302available contemporaneously.

The communication module108provides a simplified abstraction to the application modules216, service modules218, and other modules. A module requiring communication services may use this simplified abstraction to readily initiate communication, without the module knowing the details of the particular transport module and type of connection involved. For example, a requesting service module218(1) on the media device104(1) may initiate a request to communicate with the corresponding service module218(2) on the media device104(5). This communication may use one or more of several different transport modules302, but this complexity may be hidden from the requesting service module218(1) and from the receiving service module218(2).

Due to the modular nature of the communication module108, additional transport modules302may be easily added without requiring re-engineering of the communication module108. For example, a transport module302supporting an infrared optical transport mechanism may be added without disrupting other existing transport modules302or other operational aspects of the communication module108.

The transport modules302may include a secured Wi-Fi Display module302(1) and a Wi-Fi Display socket302(2). These transport modules302(1)-(2) may be compliant with the Miracast standard promulgated by the Wi-Fi Alliance, the Intel Wireless Display (“WiDi”) standard developed by Intel Corporation, or both. The Wi-Fi Display transport module allows for peer-to-peer exchange of media information. This exchange may be encrypted in the case of the secured transport, or unencrypted in the case of the unsecured transport. The transport modules302may include transport modules configured for the exchange of graphical data and instructions, as described below.

Transport modules302which are based on Internet Protocol (“IP”) server sockets may be provided, such as a secure server socket302(3) and an unsecure server socket302(4). These transport modules302(3)-(4) may support wired or wireless communication.

Transport modules302may also support the Bluetooth standard as promulgated by the Bluetooth Special Interest Group. A Bluetooth module which is secured302(5) may be provided, as well as a Bluetooth module302(6) which is unsecured.

Other transport modules302may also be provided. For example, a transport module may provide transport functionality over a wireless WAN, such as LTE, 3G, 4G, and so forth. The transport modules302may implement a portion of the standard, specification, or protocol. For example, the transport modules associated with Bluetooth may omit application level functions in the Bluetooth specification, such as profiles.

A discovery manager304manages the device explorers306. The discovery manager304provides an interface which abstracts the mechanism provided by the device explorers306to search for media devices104and discover the services associated with those media devices. The discovery manager304provides to other modules the results of the discovery, while concealing from other modules consuming these results the complexity of the search and discovery processes which may differ from transport module302to transport module302.

The device explorers306are modules which are configured to gather device object232data using one or more of the transport modules302. The device explorers306may process the device objects232to generate one or more route maps228. The device explorers306may also be configured to respond to the device explorers306on other devices to distribute device objects232, route maps228, or other information.

The device explorers306may operate in “active” or “passive” modes. The active mode requires some interaction, such as user102approval, before a service is advertised as being available. The active mode may be time limited, such that the associated service is only discoverable for a particular period, so long as the user102is activating a particular command or button, and so forth. For example, the service module218providing output for the tablet media device104(1) may be placed in active mode, such that the display and speakers of that media device104(1) are only discoverable when the user102explicitly makes them so.

In comparison, the device explorers306may operate in a passive mode. In the passive mode, the service modules218are available without user interaction. For example, the AVR media device104(5) and the television media device104(6) may be configured to operate in passive mode such that any of the media devices104in the system100may see the resources they have available to share.

Depicted here are device explorers306including for simple service discovery protocol (“SSDP”)306(1), Wi-Fi Display306(2), Bluetooth306(3), Avahi306(4), and so forth. The SSDP306(1) allows for advertisement and discovery of services over an IP-based network using user datagram protocol (“UDP”) packets for transfer using one or more of the transport modules302. The Avahi explorer306(4) may implement a version of Avahi as developed by Lennart Poettering and others. Avahi provides a zero configuration protocol using least a portion of the multicast domain name service (“DNS”), DNS-based service discovery (“DNS-SD”), or both to discover and configure an IP-based network without user intervention.

The authentication manager308is configured to authenticate the media devices104in the system100and authorize communications with one or more of those media devices104. This authentication may be accomplished within the LAN110, or may involve external resources such as the servers114. In one implementation, the authentication may involve parsing a device identifier received from another media device104to determine a manufacturer or model number. Previously determined devices104may then be allowed to communicate. In another implementation, a first media device104may interrogate the server114to authenticate a second media device104. For example, the tablet media device104(1) may discover the presence of the television media device104(6). Continuing the example, the tablet media device104(1) may send data about the television media device104(6) to the server114, to confirm that connectivity is permitted. The authentication manager308may manage this exchange.

In some implementations the authentication manager308, or another module, may be configured to authenticate the identity of the user102. In one implementation, access levels to services available on the LAN110may be based at least in part on user identity. For example, the user102(1) who lives in a home environment where the system100is implemented may have “family” access when using the device of another user102(1) in that environment.

A security manager310is configured to manage use of encryption and other techniques to prevent unauthorized use of the media device104. For example, the security manager310may be configured to require a predetermined encryption for administrative access to the communication module108. The security manager310may work in conjunction with the authentication manager308to manage authentication tokens granted to other media devices104. The security manager310may permit different access levels. These different access levels permit connectivity and operation with different restrictions.

In one implementation, the access levels may include levels such as “family,” “friend,” “first party devices,” and “guest.” The family level allows connections only from another media device104which is registered or otherwise associated with a common account. The friend level allows access to a specifically “trusted” media device104. The first party device level indicates that the media device104shares a common manufacturer, distributor, or other attributes. The guest level is similar to friend level access, but has a limited duration in time. For example, the media device104may be considered a friend device when on the same LAN110. Particular service modules218may define the access levels they will operate with.

The service router312is configured to process incoming connections. The service router312may route the connection to the authentication manager308for processing. Once authenticated and authorized, the service router312may direct the traffic to the appropriate service module218.

The connection manager314manages the active connections in the media device104. The user interface module214may be configured to present to the user102various prompts or information, such as whether to permit a particular connection, whether to seek out a particular connection, and so forth.

When one or more media devices desire to exchange content (either as a source device or as a target device), they may enter a discover mode where each device makes its presence known to other devices. In this manner a source device may detect a target device and vice-versa. A communication connection may then be established between the source device and target device. The communication connection may be established through the exchange of messages including instructions to connect the devices for purposes of exchanging information, such as graphical information. Graphical information may include graphical commands and/or graphical data, as described below. Following the establishment of a communication connection, the devices may exchange initialization information and beginning a display sharing session. A display sharing session is a session during which graphical commands for execution and display are sent from the source media device to the target media device.

Devices that wish to connect to each other after device discovery may engage in a handshake and connection setup that may be controlled by various components discussed above with regard toFIGS. 2 and 3. As part of device discovery or connection setup between two or more devices, a device may advertise its capabilities, including available services, communication protocols and constraints, hardware capabilities, operating OS, and the like.

When a source media device connects to a target media device for purposes of display sharing, an initialization process may be executed by the devices. The initialization process may occur prior to establishing a display sharing session. During this initialization process various graphical data may be sent to the target device from the source device. Graphical data may include data to be used or referred to when executing a graphical command to generate display data. For example, graphical data may include data used by a graphics processor. Examples of graphical data include graphical assets such as geometric information, vectorized drawings, vector graphics, geometric graphics, vectorized geometries, vectorized geometric primitives, textures, images, or the like. Graphical data may include data to be stored in a graphics library. As part of the initialization process, graphical assets may be sent and preloaded into one or more graphics libraries of the target prior to a display sharing session.

Other information such as configuration information may be exchanged between the source media device and target media device either during initialization or as part of device discovery/connection setup. Configuration information may include information regarding settings, capabilities, or other configurations of the respective devices. For example, configuration information may include display settings, graphics parameters, frame rates, bit rates, or other communication parameters may also be exchanged during this initialization. Configuration information may also include information regarding the different resources (both hardware and/or software) used by the different devices to allow format-based conversion of information from one device to another. For example, the source and target media devices may exchange information about each other's operating system, hardware drivers, processor capabilities, or other device settings. The source and target media devices may also exchange information about the display configurations of each other's displays, such as the respective display's make/model, resolution, refresh rate, or the like. Other types of configuration information may also be exchanged.

Such out-of-band exchanges of graphical data, configuration information, etc. may continue over a back channel separate from any display sharing sessions while the source media device and target media device remain capable of communicating with each other. Further, additional data that may be needed by the target media device to properly display the content may also be sent by the source in-band during a display sharing session as needed. Such data may be sent as part of a display sharing communications, during lulls in display sharing communications, or otherwise.

If communication conditions between the source media device and target media device are such that the target media device is unable to receive all desired graphical elements prior to commencement of a display sharing session, certain graphical elements may be prioritized over others for transmission to the target media device. Further, different version of certain graphical elements (such as low resolution textures, etc.) may initially be sent to the target media device while more complete versions (such as high resolution textures, etc.) may be sent later as communication conditions allow.

The source media device may keep a record of what graphical information it has sent to the target media device or may otherwise be aware of what graphical information is available to the target media device (such as being informed of the target media device's graphics library contents during initialization). Thus the source media device may send the target media device only the graphics assets the target media device does not already have, thereby reducing the amount of bandwidth used for the display sharing operations.

FIG. 4illustrates an example configuration for generating graphical commands and display data in a source media device104(S). An application module216, such as a software application operating on a device, generates graphical commands for processing by lower levels to create images for a display206(D). The application module216may generate a graphical command such as a call to a graphics library238. The graphics library238may be configured for the particular hardware of the media device, such as a graphics library238including graphical data configured for the graphics processor202(G) of the media device. The graphics library238may receive the call from the application module216and convert that call to an instruction that may be executed by a driver, graphics processor (such as graphics processor202(G)), or other component to generate display data (such as to be displayed on display206(D)). As part of generating the instruction to the lower level driver or graphics processor, the graphics library238may use graphical data such as the graphical assets discussed above. AlthoughFIG. 4illustrates the graphics library238as communicating directly with the graphics processor202(G), a driver (not pictured) for the graphics processor202(G) may be located in between the graphics library238and the graphics processor202(G). In that situation, the graphics library238may generate graphical commands for operation by the driver. A display driver (also not pictured) may also be located between the graphics processor202(G) and the display206(D).

The graphics library238may include an application interface, such as an API or instantiation of an API, to interface between the application and the lower level driver/processor. The graphics library238may operate according to the OpenGL standard. OpenGL describes a language independent API for generating commands for execution by a graphics processor. As shown inFIG. 4, a graphics library wrapper240or binder may process calls from the application module216prior to sending them to the graphics library238. The graphics library wrapper240may provide a different interface to convert calls from the language of the application module216to the language of the graphics library238. For example, Java OpenGL is a wrapper that allows OpenGL to be used in the Java programming language. The graphics library wrapper240may itself be an API. Calls to the graphics library wrapper240or graphics library238may be called graphics calls.

As illustrated inFIG. 4, during operation, an application, such as an application operated by application module216, may make an graphics call that is sent to the graphics library wrapper240. That graphics library wrapper240may then process the call, for example by translating the call, and send the resulting translation, which is itself a call to the graphics library238. The graphics library238may then process the call to create an instruction, such as a command in driver code, or the like to a driver, graphics processor202(G) or other processor to create rendered images to be sent to the display206(D) of the media device.

In one example, an application, such as a video game, is operated by the appropriate application module216. During operation of the game, the application module216makes a call to the graphics library wrapper240. The graphics library wrapper240then determines the appropriate graphics library238call for executing the desired graphics operation. The graphics library238then passes an instruction to the graphics processor202(G). The graphics processor202(G) then creates display data that may be sent to a display driver or display206(D) to actually create and show images on the display206(D) of the device.

To operate display sharing, the calls to the graphics library238or graphics library wrapper240, or the graphical instructions from the graphics library238to the graphics processor202(G), or other graphical commands, of a source media device may be duplicated and routed to a target media device for native graphics generation on the target media device to perform display mirroring as described above. As shown inFIG. 5, in one aspect, graphics calls from a graphics library wrapper240are sent to another module of a media device, such as the content direct sharing module116, or other module. The content direct sharing module116may then duplicate the graphics calls from the graphics library wrapper240and send them and/or other graphical information to a communication interface208to be sent to a different, i.e. target, media device. The communication module108(not shown) may mediate exchanges between the content direct sharing module116and the communication interface208. In this configuration the graphics library wrapper240may intercept and manipulate calls from the application module216prior to the calls being processed by the graphics library238or sent to the content direct sharing module. The content direct sharing module116may also send the target media device graphics calls to the graphics library wrapper240or graphical instructions to the graphics processor202(G).

In one aspect, the present system differs from prior display mirroring techniques in that the present system is configured to send graphical commands from a source media device to a target media device. Prior display mirroring techniques are configured to send display data between devices. The difference between graphical commands and display data is that graphical commands are higher level instructions for executed by higher layer computing resources such as an API or processor and display data is instructions and data used by lower layer computing resources, such as instructions configured for a monitor or display. Graphical commands may include commands like clear screen, set vertex buffer, draw elements, set texture, and the like. Graphical commands may operate on graphical data such as vector data used to draw lines and shapes (including indices, vertices, lines, points, curves, shapes, polygons, color, tangent, etc.), non-vector data (such as textures, etc.), graphical data address information, or other data used by a processor to render graphics.

Graphical commands may operate on pre-rasterized data. Rasterization describes a process for mapping shapes to pixels for display on a display device. Rasterization typically occurs at the lower levels of a graphics pipeline, such as that shown inFIG. 4. Pre-rasterized data may be data that is described in a vector graphics (i.e., shape) format rather than in a raster image (i.e., pixels or dots) format. Such pre-rasterized data may include the graphical data and graphical assets mentioned above, including vector data, non-vector data, and the like. Graphical commands may also operate on pre-rendered data. Rendering describes a process for generating an image from a model. Pre-rendered data may be data from which a pixel image may be generated. Such pre-rendered data may include, for example, model or scene files including geometry data, viewpoint data, texture data, lighting data, shading data, and the like. Display data may include post-rendered or post-rasterized image data such as bitmap data, video data, or pixel data for use by a display. Pixel data may include data to generate images for particular points on a display or image. Pixel data may include, for one or more pixels, color, brightness, and/or location. Graphical commands generally do not include display data. Graphical commands generally involve further execution or further processing in order to generate displayable data.

Display data may typically be configured or formatted for the particular display the images are to be sent to. For example, display data may be created in a format designed for a 720i resolution display, where pixel information is generated for a high-definition screen with 1280×720 pixels and where display data is output in a manner designed for an interlaced display. In another example, if display data is configured for a screen with a refresh rate of 60 Hz, the display data may include information to refresh the screen 60 times per second. In a traditional display mirroring system, this display data (i.e., formatted for a 720i, 60 Hz screen) would be created and sent both to the screen of a source device and to a target device for display. Such display data may include, for example, encoded video data such as H.264/MPEG-4 encoded video, as may be used by traditional display mirroring system. If, however, the target device had a screen that was configured differently from the source screen (for example, the target device had a 1020p, 120 Hz screen, or a 4K screen), the result would be the target device displaying display data configured for the source screen that was inferior to display data had it been formatted for the native target screen. In the present system, by sending graphical commands from the source media device to the target media device, the target media device may natively execute the graphical commands to create display data configured and formatted for the screen of the target device.

The differences between graphical commands and display data may be further explained in reference again toFIG. 4. As illustrated inFIG. 4, graphical commands may include the graphics calls from an application module216to an API such as the graphics library wrapper240or the graphics library238. Graphical commands may also include API calls from the graphics library wrapper240to the graphics library238. Graphical commands may also include instructions sent to the driver or graphics processor202(G), such as those sent from the graphics library238. Display data may include processed images, such as rendered or rasterized images created by a processor such as a graphics processor202(G) and output to a frame buffer or a display. Commands from the application module216through the graphics processor202(G) are generally considered graphical commands and data from the graphics processor202(G) to the display206(D) are generally considered display data. In prior display sharing systems, display data is sent to a target media device104(T). In the present system, graphical commands are sent to the target media device104(T). The graphical commands sent to the target media device104(T) may also include other graphical information such as graphical data as explained herein, such as graphical resources to generate images for a display.

In another aspect, as shown inFIG. 5, a source media device may be configured with a special graphics library238′. The special graphics library238′ may act as the main graphics library in a device, allowing it to capture graphical commands from any application or system process. As shown inFIG. 5, the special graphics library238′ may intercept graphical commands which are then sent to the graphics library238of the source media device. The special graphics library may exist on both the source media device104(S) and on the target media device104(T). The special graphics library238′ may be configured to assist with display sharing. That is, the special graphics library238′ may be an implementation of an API configured in a manner designed to efficiently exchange graphical commands and graphical data between the source media device104(S) and the target media device104(T), for example by using a standard set of instructions or resources that may be shared among devices that are configured for display sharing as described herein.

The special graphics library238′ may be located between the graphics library wrapper240and the graphics library238. In one aspect, the content direct sharing module116, which may also coordinate communications between devices may be located between the graphics library wrapper240and the special graphics library238′. In this manner graphics calls (such as those from the graphic library wrapper240, or elsewhere) or other graphical information may be reformatted or otherwise manipulated by the content direct sharing module116(or other module) to be compatible with the special graphics library238′.238′. For example, certain graphical assets may be referred to by a particular memory address in the graphics library238but may be at a different location in the special graphics library238′. Thus, the memory address in a call from the graphics library wrapper240may be changed prior to being sent to a target device. In another example, the special graphics library238′ may be written in another language from the graphics library wrapper (which itself is typically written to interact with the graphics library238rather than the special graphics library238′). Thus the content direct sharing module116(or other module) may change the language of the call prior to passing the call to the special graphics library238′. In this situation the content direct sharing module116(or other module) may mediate calls between the graphics library wrapper240and the graphics library238. In another example, if the special graphics library238′ may be configured to operate as a mediator between calls between the graphics library wrapper240and the graphics library238or other interface with the hardware drivers.

In one aspect, all graphics and images appearing on the display of the source media device104(S) may be processed through the special graphics library238′. This may make display sharing easier when the special graphics library238′ is also be replicated on one or more different target media devices104(T). For example, graphics calls of the source device that are manipulated into the format of the special graphics library238′ may be more easily handled by the target device when the target device is also configured with a special graphics library238′ configured to interact with the graphics library of the target device. Similarly, certain graphical resources (i.e. graphical assets) may be referred to in the same way in the special graphics library238′ of a source media device as in the special graphics library238′ of the target media device. In this manner, calls to a special graphics library238′ of the source device may be sent directly to the target media device for execution at the target media device. Thus, after receiving graphics calls from the graphics library wrapper240, the content direct sharing module116may manipulate the graphics calls according to the configurations of the special graphics library238′ and send the manipulated graphics calls to both the special graphics library238′ for processing on the source media device and to the communication interface208to be sent to a target media device.

If a source media device104(S) is configured with a special graphics library238′, but no target media device is detected and/or enabled, graphical commands will not be forwarded and may be processed by the source media device104(S) as normal. If a target media device is detected and display sharing is activated, graphical information may otherwise be manipulated prior to sending to the target media device to assist in processing the graphical information by the target media device. This manipulation may also include altering or removing certain unnecessary graphical commands at the source media device prior to sending to the target media device in order to conserve bandwidth and otherwise improve display sharing efficiency. For example, the content direct sharing module116(or other appropriate module) may detect and/or remove redundant state changes or no-effect commands. The content direct sharing module116may also detect and/or removed duplicated or overlapping memory blocks in resources. The content direct sharing module116may also perform compression (potentially lossless compression) of certain resources to reduce transmission payload size. The content direct sharing module116may also coordinate asynchronous transmission of resources that are not needed immediately, such as textures, etc. as described in reference to the out-of-band initialization and configuration processes.

The graphical commands, which may initially be configured for the source device (such as based on the source device's OS, or other setting) may be reconfigured prior to being sent to the target device. Graphical data, which may initially be configured for the source device may also be reconfigured prior to being sent to the target device. If graphical data is reconfigured, corresponding graphical command(s) may also be reconfigured so the graphical command sent to the target device is configured to expect and operate on the new graphical data. For example, graphical commands may be translated into a language that may be more easily understood at the target side, the graphical commands may be reformatted based on a configuration of the target device (such as based on the target device's OS, or other setting), may be reformatted based on the special graphics library238′ of the target device, or otherwise. In another example, vector data for operation on by a graphical command may be transformed to match a screen area of the target device by scaling the geometries of the vector data up or down. In another example, a graphics command may be changed to alter the drawing area and resolution to match the target device. In another example a graphical command (or group of graphical commands) may be changed to another graphical command (or group of graphical command) if a target device does not support the original graphical command(s) and/or if the substituted graphical command(s) would otherwise perform the desired task. For example, shaders may be used to process vertices, geometries, etc. In another example, a pre-rendered image referred to by a graphical command may be converted to a format or configuration that the target device can read and use. In another example, output frame buffers may be reconfigured to match the target device display and to handle gamma issues to adjust for brightness differences.

FIG. 6illustrates an example of an exchange of graphical information between a source media device104(S) and a target media device104(T) for display mirroring. While the source media device104(S) and target media device104(T) may each include other components, only those used in this example exchange are illustrated. An application module216at the source media device104(S) generates graphical commands for the graphics library wrapper240. The graphics library wrapper240generates further graphical commands, such as calls to a graphics library and sends those commands to the content direct sharing module116(S) of the source media device104(S). The content direct sharing module116(S) then manipulates the graphical commands based on a format of the special graphics library238′(S) and sends the manipulated graphical instructions to the special graphics library238′(S) and to the communication interface208(S). The source media device104(S) may then generate the appropriate graphics for its own display by passing instructions from the special graphics library238′ (S) to the graphics library238(S) and to the graphics processor202(G)(S) which will then execute the instructions and use data from one or more graphics libraries to generate the graphics for the display206(D)(S).

The manipulated graphical commands are sent wirelessly over the network150to the communication interface208(T) of the target media device104(T). The received commands may then be passed to an application or module running on the target media device104(T) that is capable of processing the instructions. As illustrated, this module is the content direct sharing module116(T), however a different application or module may be used. The manipulated graphical instructions are then sent to the special graphics library238′(T) of the target media device104(T) which then processes the instructions for communication with the graphics library238(T) and the graphics processor202(G)(T) for eventual execution and rendering on the display206(D)(T) of target media device104(T). Thus, referring again toFIG. 1B, the media stream118sent from a source media device to a target media device may take the form of graphical commands to be executed at the target media device. In prior systems, the media stream118would be display data, such as specific pixel instructions (such as video data or rasterized pixel data output from graphics processor202(G)(T)) to be shown on display206(D)(S). By sending graphical commands from an API level layer instead of display data, such as pixel data, the target device may render its own graphics based on the configurations of its processor(s), display, etc. AlthoughFIG. 6illustrates a graphical command from the graphics library wrapper240being sent from the source media device104(S) to the target media device104(T), any graphical command generated in the source media device104(S) may be sent to the target media device104(T) for display sharing.

In another configuration of the system, a special graphics library238′ may not be used by the source media device and/or the target media device. In this configuration, graphical information at the API level (i.e., created by the application module216(S)), graphical information output from the graphics library wrapper240(S) and/or graphical information output from the graphics library238(S) may be sent from the source media device104(S) to the target media device104(T). The sent graphical information is then used by the target media device104(T) to natively generate graphics and images by the processor(s) of the target media device for rendering on the display206(D)(T) of the target media device104(T). The graphics libraries of the target media device104(T) may store graphical data sent by the source media device104(S) during the display sharing session or sent as part of out-of-band communications such as the initialization process discussed above. In one aspect, a source media device may also send timing information or instruction to one or more target devices to coordinate display of the images among the devices (source and/or target(s)) so that images are synchronized and/or displayed at the correct time when viewed by a user.

The display sharing system may also allow for display sharing among more than two devices. For example, as shown inFIG. 7, a source media device104(S) may send graphical commands across the network150to multiple target media devices104(T). The individual target devices may then process the graphical commands natively for each respective display of the target media devices104(T), thereby sharing the display of the source media device104(S) among multiple targets. In another example, as shown inFIG. 8, a target media device104(T) may receive content from multiple source media devices104(S)-1,104(S)-2, and104(S)-3. The target media device104(T) may divide its own display to show the content from the respective source media devices (content106-1,106-2, and106-3) on different portions of the target's display, possibly leaving some of the target's display for other content not received from a separate device. The content sent by the source media devices to the target media device is represented by graphical commands generated by the individual source media devices as illustrated inFIG. 6and described above.

FIG. 9illustrates display sharing operations as performed by a source device according to one aspect of the present disclosure. A source media device104(S) may detect a target device, as shown in block902. The source media device104(S) may then initialize the target device, as shown in block904. The initialization may comprise exchanging configuration information with the target device, sending the target device graphical data, or other processes. The source media device104(S) may then initiate a display sharing session with the target device, as shown in block906. The source media device104(S) may then generate graphical commands from an API, as shown in block908. The graphical commands may be executable by a graphics processor and my reference information contained in a graphics library. The source media device104(S) may then send the graphical command to the target device, as shown in block910. The source media device104(S) may then execute the graphical command to create rendered image data, as shown in block912. The rendered image data may be formatted for the display of the source media device104(S). The source media device104(S) may then display an image corresponding to the rendered data, as shown in block914.

FIG. 10illustrates display sharing operations as performed by a target device according to one aspect of the present disclosure. A target media device104(T) may detect a source device, as shown in block1002. The target media device104(T) may receive initialization information, as shown in block1004. The target media device104(T) may also send its own configuration information to the target media device104(T), not shown. The target media device104(T) may then initiate a display sharing session, as shown in block1006. The display sharing session may be initiated in response to an instruction to do so from the source device. The target media device104(T) may receive graphical data from the source device, as shown in block1008. The graphical data may be received as part of the initialization process, as part of the display sharing session, or as part of an out-of-band communication that is not part of the display sharing session. The target media device104(T) may receive a graphical command from the source device, as shown in block1010. The target media device104(T) may also receive information regarding when the graphical command is to be executed and/or when a corresponding image should be displayed by the target media device104(T). The target media device104(T) may then execute the graphical command to create rendered image data, as shown in block1012. The rendered image data may be formatted for the display of the target media device104(T). The target media device104(T) may then display an image corresponding to the rendered data, as shown in block1014. The image may correspond to a similar image being displayed on the source media device.

Those having ordinary skill in the art will readily recognize that certain steps or operations illustrated in in the figures above can be eliminated or taken in an alternate order. Moreover, the methods described above may be implemented as one or more software programs for a computer system and are encoded in a computer readable storage medium as instructions executable on one or more processors.

The computer readable storage medium can be any one of an electronic storage medium, a magnetic storage medium, an optical storage medium, a quantum storage medium and so forth. Separate instances of these programs can be executed on or distributed across separate computer systems. Thus, although certain steps have been described as being performed by certain devices, software programs, processes, or entities, this need not be the case and a variety of alternative implementations will be understood by those having ordinary skill in the art.

Additionally, those having ordinary skill in the art readily recognize that the techniques described above can be utilized in a variety of devices, environments and situations.

Although the present disclosure is written with respect to specific embodiments and implementations, various changes and modifications may be suggested to one skilled in the art and it is intended that the present disclosure encompass such changes and modifications that fall within the scope of the appended claims.