Cross-platform media redirection and interoperability

A media controller application that enables a user to select source and destination services and devices for media playback that provides an interoperability layer between different service and device ecosystems, and a media redirection hardware adapter that connects to a host device using a physical connector and receives audio and then retransmits at least a portion of the audio wirelessly.

CROSS-REFERENCE TO RELATED APPLICATIONSApplication No.Date FiledTitleCurrentHerewithCROSS-PLATFORM MEDIAapplicationREDIRECTION ANDINTEROPERABILITYIs a National PhaseEntry (371) filing of:PCT/US18/30801May 3, 2018CROSS-PLATFORM MEDIAREDIRECTION ANDINTEROPERABILITYwhich is a PCT filing of and claimsbenefit of, and priority to:62/501,037May 3, 2017CROSS-PLATFORM MEDIAREDIRECTION ANDINTEROPERABILITYthe entire specification of each of which is incorporated herein by reference.

BACKGROUND

Field of the Art

The disclosure relates to the field of computing devices, and more particularly to the field of media computing devices.

Discussion of the State of the Art

Many different manufacturers offer over-the-top (“OTT”) services and hardware devices for delivering media to users in various ways (for example, SONOS™, ROKU™, PANDORA™, GOOGLE PLAY™, and other media services and devices). In many cases, proprietary hardware is needed, and each device operates within an isolated manufacturer ecosystem that may not be capable of interacting with services or devices from other manufacturers (often utilizing proprietary formats or interfaces). Moreover, many hardware and software vendors are actively developing their own branded media content, and these vendors often try to limit playback of their own content to their own or their partners' systems. The emergence of “walled gardens” and the multiplication of proprietary hardware devices and interface protocols has tended to “balkanize” home entertainment systems, and to make their use frustratingly complex for consumers. Moreover, many consumers use wireless extensively today, for example by using sophisticated wireless speaker systems; it is crucial and increasingly difficult to deliver high-quality, tightly synchronized audio and video in these conditions, while minimizing latency—this is usually difficult to accomplish in wireless environments in any case, but becomes much more so when multiple vendors using various protocols and technologies all try to make use of the same wireless infrastructure in home environments.

What is needed then, is a cross-platform media redirection solution that can bring interoperability to hardware and software from various vendors regardless of their out-of-the-box capabilities.

SUMMARY

Accordingly, the inventor has conceived and reduced to practice, systems and methods for cross-platform media redirection and interoperability, that enables hardware-based audio redirection for any device with audio output capability, as well as software-based media redirection between vendor services and devices using a unified interface.

According to one aspect, a media controller comprising: a processor; a memory; a plurality of wireless network interfaces; and a plurality of programming instructions stored in the processor and operating on the memory, and configured to: provide an interactive user interface; configured to select at least a portion of a plurality of devices visible on a network based at least in part on a received user interaction; identify a plurality of device characteristics of a selected device via one of the plurality of wireless network interfaces; direct the operation of at least a portion of the plurality of media processing components based on at least a portion of the identified device characteristics, is disclosed.

According to another aspect, a media redirection hardware adapter, comprising: a processor; a memory; an audio controller operating an audio interface; a plurality of wireless network interfaces; an audio redirector comprising at least a plurality of programming instructions stored in the memory and operating on the processor, and configured to: connect to a host device using at least the audio interface; receive audio media signals from an audio source; process at least a portion of the audio media signals, the processing comprising at least a de-multiplexing operation that produces a plurality of audio channels; and send at least a portion of the de-multiplexed audio channels to a plurality of external audio playback devices via at least one of the plurality of wireless network interfaces, is disclosed.

According to another aspect, a method for cross-platform media redirection and interoperability, comprising the steps of: connecting, using a media redirection hardware adapter comprising a processor, a memory, a plurality of wireless network interfaces, and an audio interface, to a host device using at least the audio interface; receiving a plurality of audio media signals from an audio source; processing, using an audio redirector comprising at least a plurality of programming instructions stored in the memory and operating on the processor, at least a portion of the audio media signals, the processing comprising at least a de-multiplexing operation that produces a plurality of audio channels; and sending at least a portion of the de-multiplexed audio channels to a plurality of external audio playback devices via at least one of the plurality of wireless network interfaces, is disclosed.

DETAILED DESCRIPTION

The inventor has conceived, and reduced to practice, systems and methods for cross-platform media redirection and interoperability, that enables hardware-based audio redirection for any device with audio output capability, as well as software-based media redirection between vendor services and devices using a unified interface.

Conceptual Architecture

FIG. 1is a block diagram illustrating an exemplary system architecture100for a cross-platform media redirection system, according to one aspect. According to the aspect, media controller101may comprise a media controller software application operating on a user's device (for example, a smartphone, laptop, or tablet computing device), that may operate software-based networking and multimedia capabilities such as including (for example, and not limited to) file sharing via a number of protocols (for example, SMB, NFS, FTP, HTTP/S, etc), streaming media content from a variety of providers and devices and using a variety of transmission methods, file types, or media encoding schema (for example, MPEG audio and/or video, WAV audio, audio encoded using various lossless audio codecs such as FLAC or ALAC audio file types, MKV multimedia container files, etc.), media transcoding or up/down scaling to adapt a particular media stream for retransmission to another destination, or other capabilities useful for viewing, interacting with, or modifying media over a network. These capabilities provide a means to receive media from a variety of sources, perform any necessary conversion or other processing on at least a portion of the media, and then retransmit at least a portion of the media for playback at another device over the network. This provides a cross-platform interoperability regardless of the particular capabilities of any one source or destination device or service, without the need for any custom configuration of network devices or alterations to a user's media library.

Media controller101may receive raw media inputs from a variety of input sources, for example including Internet-based media streaming services such as PANDORA™105, NETFLIX™106, and other similar services107via the Internet102. Media hub101may also receive raw media from a plurality of OTT devices110such as including, but not limited to, APPLE TV™111, ROKU™112, AMAZON FIRE TV™113, GOOGLE CHROMECAST™114, and other AMAZON™ devices such as ECHO™115, or any of a variety of additional or alternate devices that are available or may be introduced. Some of these devices110may communicate with media controller101wirelessly such as using a Wi-Fi local area network (LAN), and some may be directly connected via one or more physical ports on wireless media hub101. Media controller101may send high-definition video wirelessly to a video dongle130which may in turn be inserted into an HDMI or other port (for example, DISPLAYPORT™ or USB™ ports) of display device140(which may be a conventional television, a smart television, a computer display monitor, or another media display device). Media controller101may also stream audio wirelessly to one or more audio playback devices synchronously, such as (but not limited to) headphones144, wireless speakers141,142, wireless sound bar143, and so forth. In some arrangements, media controller101may send audio and video to a connected multimedia dongle130, which may then play video via a connected display device140and audio via connected audio devices such as speakers141connected to the display device140(for example, in a home theater setup with multiple speakers connected to a television). In some other arrangements, media controller101may be used in conjunction with an audio redirection hardware adapter200(as shown below, referring toFIG. 2andFIG. 7) connected to a media device, to utilize the network or audio redirection capabilities of the hardware adapter200to expand the capabilities of the media arrangement.

Examples of how a media controller101may be used are many. For example, a user may select a movie from a cloud-based service such as NETFLIX™ or AMAZON PRIME™ and direct the video via an HDMI output or a wireless video dongle to a television, while sending the audio wirelessly to a set of speakers to provide a rich audio experience. The user may select how audio is distributed, may select what is to be viewed, and may control viewing settings, all from within an application operating on their device, retaining the familiarity with their particular device's software and hardware interaction experience.

FIG. 2is a block diagram illustrating an exemplary system architecture for a cross-platform media redirection hardware adapter200, showing two alternate hardware configurations, according to one aspect. In a first hardware arrangement210, a media redirection hardware adapter (MRHA)200may comprise a system-on-a-chip215further comprising a memory11and a processor12(as described below, referring toFIG. 4), configured to operate without integral hardware display or input devices (such as, for example, a computer monitor or keyboard as are commonly present in computing devices), and configured to perform audio interception and redirection using an audio redirector216that may be used to intercept and redirect incoming audio streams received via audio connector205and provided by audio controller201. Some audio connectors205provide power along with the audio signal to connected devices (for example, to operate headphones that do not have a power source of their own), however if this power supply is insufficient a dedicated power supply230may be used to plug into another power source such as a wall outlet or USB port, to provide the level of power needed to operate the full functionality of MRHA200.

Audio intercepted by audio redirector216may be modified to separate the audio channels present (generally at least two, but any number may be possible) and then optionally provide each individual channel to a wireless interface220for retransmission to network-connected destination devices (such as wireless speakers, such that each speaker will play a single audio channel such as to provide a multichannel immersive audio arrangement, or for enabling positional audio using individual speakers to isolate sounds to particular physical locations within the room or media hardware setup), or these channels may be used to enhance the audio prior to retransmission, such as to duplicate and/or modify audio channels to produce more channels than were present in the source audio. For example, a standard tip-ring-sleeve (TRS) audio connector, regardless of physical size (common sizes include 2.5 mm, 3.5 mm, and ¼″ diameter connectors), can only provide two discrete audio channels via hardware (as the third contact is used for the ground connection), and any additional channels must be provided through software multiplexing. This may be used to provide software-emulated surround or positional audio capability using an audio redirection hardware adapter, by intercepting a stereo (two-channel) audio signal from a source device, and then using software processing to produce a number of additional channels to expand the audio as needed, then retransmitting the original audio and any new channels produced for output via wireless interface220.

In an alternate hardware arrangement250, a plurality of integral speakers260a-nmay be utilized to enable MRHA200to render some or all audio channels without the need for external destination devices. This may be used, for example, to provide a number of additional output devices to increase the number of available channels for use in audio processing and redirection as described above, or to provide native audio output capability to a device that does not have such capability, such as some personal music players that are designed without speakers and rely on an audio connector205to play audio through a connected output device, such as a user's headphones.

Many media devices such as speakers or media players are capable of both transmitting and receiving audio via a hardware audio connector205. When connected to an audio input on a host device, MRHA200may be used to receive audio via wireless interface220, process via audio redirector216as described above, and optionally provide a number of audio channels as output via audio connector205for playback on the connected host device. For example, this may be used to connect audio redirector to an input-only audio port on (for example) a speaker that is designed to receive audio but is not designed to provide audio for playback over a hardware port. In such an arrangement, the audio redirection capabilities of MRHA200may still be used, and rather than receiving audio from audio connector205and retransmitting via wireless interface220, audio is received via wireless interface220and retransmitted via a combination of audio connector205and wireless interface220. In this way it can be seen that audio may be intercepted, processed, and retransmitted in a multi-directional manner, adapting to the capabilities of the devices and connections used in a user's media setup. Another exemplary use may be to connect to an audio port and use only the power provided by the host device, while audio is both received and transmitted exclusively through wireless interface220. This may be used to power MRHA200without the use of a power cable or external power source, by connecting to an audio port on a device that is not part of a media playback arrangement, or by connecting to a port that provides only power and no data signals.

Operation of MRHA200may be controlled or configured wirelessly using a media controller application101(as described previously inFIG. 1), for example so that a user can select how they want audio channels to be processed and retransmitted, or so that a user can select whether to use integral speakers260a-n, or other configuration. Additionally, if the hardware capabilities of a host device allow it, the host device's specific capabilities may be identified via the hardware connector205and exposed via wireless interface220, providing networking capabilities to audio devices that may not natively have them. For example, a hardware speaker with no networking capability may be used as a host device in this manner, and the MRHA200now provides networking capability for the speaker while also optionally providing the audio redirection and retransmission capabilities described above. This can be used to extend the functionality of a media hardware setup, as well as to enable devices with no network capabilities of their own to be used in a network-based media arrangement, furthering the cross-platform capabilities of the system regardless of the hardware capabilities of any individual device.

It should be noted that, while an audio connector is used in the example ofFIG. 2and is discussed throughout, other similar media interfaces may be used in place of an audio connector, according to various arrangements. For example, HDMI or DISPLAYPORT™ interfaces may be used; in such arrangements, audio controller201and audio interface205would be instead an HDMI or DISPLAYPORT™ controller201and interface205; similarly, other newly emerging media interfaces may be used in various embodiments of the invention. When using alternate physical connectors, additional or alternate processing capabilities may be enabled such as video processing and redirection (for example, to stream video to one device and audio to others), or to draw power from a mobile high-definition link (MHL) compliant port such as an HDMI port.

Detailed Description of Exemplary Aspects

FIG. 3is a flow diagram illustrating an exemplary method300for cross-platform media redirection using a media controller application, according to one aspect. According to the aspect, in an initial step301a third-party device connects to media controller101over a network. In a next step302, media controller101may identify a connected device's characteristics, including vendor information and the device's software ecosystem (for example, any proprietary file or communication formats that may be utilized, or device-specific media types or sources such as Internet streaming sources available as channels on the device). In a next step303, media controller101may configure processing operations based on identified device ecosystem and other characteristics, for example so that media may be processed to convert or encapsulate proprietary information into “device agnostic” or universal types for compatibility with devices from other vendors or that do not conform to a particular ecosystem. In a next step304, media is received and then in a next step305it may be processed to convert any proprietary formatting (such as media protocols or file container formats) into a form that is compatible with a destination device or that is not device-specific for broad compatibility. In a next step306, media may optionally be processed to conform to a particular destination ecosystem, for example for transmission to a device that has specific configuration parameters or rules for receiving media, or to take advantage of features offered by a particular ecosystem on a destination device (such as specific audio formats such as lossless compression, or specific software interface features that may depend on or interact with specific media characteristics, such as looking up information on actors present in a scene using embedded metadata, or other such uses), and then in a final step307media is transmitted to destination devices in formats appropriate for their configuration. Additionally, some processing operations for bridging media between device ecosystems may rely on emulating a media source or destination device to the third-party OTT device, for example media controller101may identify itself as a television to a device that is configured only to provide direct media output to a destination, such as a media streaming stick or APPLE TV™. In this manner, devices from different vendors may be used in conjunction with each other regardless of software or hardware limitations, for example media may be selected for streaming from an APPLE TV™ device and processed for transmission to a CHROMECAST™ device as a destination, when ordinarily these devices are not designed to interact with each other and are configured to interact directly with a television or other destination or output device directly. In this manner, media controller101may be utilized in a many-to-many configuration, serving as a central hub for a wide variety of device and ecosystem types for broad compatibility and interaction.

FIG. 4is a flow diagram illustrating an exemplary method for cross-platform audio redirection using a hardware adapter, according to one aspect. According to the aspect, a media redirection hardware adapter (MRHA)200may be connected to a host device401, for example via an audio connector or HDMI port. MRHA200may then determine the host capabilities402, for example the connected port may be optionally used for input and/or output according to the particular configuration or capabilities of the port or host device. For example, an audio port may be configured by the host device for output of audio to play on external devices, and therefore used as input by MRHA200, or it may be configured for use as input such as from a microphone, and therefore used as an output connection by MRHA200. MRHA may then receive audio content403from a source device, for example either the host device via the connected hardware port or from a network device via a wireless interface220, according to a particular configuration or use case. Received audio may then be processed404using an audio redirector216operating on MRHA200, for example to transcode or remux audio channels as described above with reference toFIG. 2. Processed audio may then be used as output, with at least a portion of the audio being retransmitted405via wireless interface220for playback on a plurality of network devices such as wireless speakers or other media playback devices. Additionally, at least a portion of the audio may optionally be provided as output via the connected hardware port to the host device, for example if the host device is configured to receive audio input via the port and is configured to be used as a destination device in the current arrangement (for example, by a user directing the operation of the MRHA200using a media controller application101, as described previously inFIG. 1).

FIG. 5is an illustration of an exemplary usage arrangement, illustrating the use of a media redirection hardware adapter with several wireless speakers. According to the arrangement shown, an MRHA200may be connected to a media device501such as (for example) a personal media player such as an MP3 player or a user's phone. Depending on the specific configuration of the media device501, MRHA200may draw power directly from the host device connection without the need for additional cables or connection to a power source such as a wall socket, and MRHA200may optionally be used to provide audio output capabilities that the host device501may not ordinarily have (for example, many MP3 players do not have internal speakers and are designed to function with an external output device connected). MRHA200may then intercept audio produced by the host device501, process at least a portion of the audio using an audio redirector component216(as described previously inFIG. 2), and then retransmit at least a portion of the audio to each of a plurality of network devices such as (for example, including but not limited to) wireless speakers502a-n.

FIG. 6is an illustration of an exemplary usage arrangement, illustrating the use of a media controller application on a mobile device as well as a media redirection hardware adapter within a home media setup. According to the arrangement shown, a user may use their mobile device601to interact with a media controller101, selecting media for playback and devices to be used during playback. For example, a user may select media for streaming from a variety of online media streaming sources610such as (for example, including but not limited to) NETFLIX™ or PANDORA™. This media may then be transmitted to a MRHA200connected to a host device such as a television602. MRHA200may use an audio redirector216(as described previously inFIG. 2) to retransmit channels from the audio portion of the streaming media to a plurality of wireless speakers603a-n, while providing a video portion for playback on the host device602, and optionally providing at least a portion of the audio for playback such as using connected speakers604a-ncontrolled by the host device602. In another example, a user may select media stored locally on their device601for playback, which may then be streamed to MRHA200for rebroadcasting to wireless speakers603a-nand playback via TV602. In this manner, it can be seen that combining the media library interoperability of a media controller101and the redirection and retransmission capabilities of a MRHA200, a user may enjoy media from a wide variety of sources using a wide variety of playback devices and arrangements, regardless of the configuration of any particular device or media source involved as the media controller101and MRHA200adapt to the capabilities of the sources, devices, and arrangement used and provide for full cross-platform compatibility in a manner that is fully transparent to the user, requiring no additional configuration or setup to use media and devices the user already has in an arrangement of their choosing.

Hardware Architecture