Patent Description:
<CIT> discloses systems and methods are described for aggregating media content from multiple sources for viewing on a local display or for place-shifting to a remote display. The aggregation system responds to instructions received from a remote device via the digital network to provide the video output incorporating the media content received from any of the media sources to the locally-connected display.

<CIT> discloses a system and method for mixing video streams in a collaborative mobile environment. In an embodiment, method for mobile video mixing includes receiving a plurality of video streams at a mobile device corresponding to one or more cameras capturing an event.

<CIT> discloses an audio-video processing method, including receiving multiple video streams from distinct sources and preparing multiple distinct composited video streams from the multiple video streams.

<CIT> discloses a system comprising multiple cameras feeding video streams to a video board. A user interface is provided on the video board showing a mosaic of all the video streams received from the cameras. The user can select one video stream using the user interface and said video stream is then output for transmission.

A media processing system including a media studio allows the user to bring the studio along to live events. The user may preview multiple media input items, generate a media output item based on the multiple media input items, share the media output item, and store the media output item for future viewing, all via a remote operator console.

Referring to <FIG>, a media processing system <NUM>, including a media studio <NUM>, is operable to receive media input data from one or more media devices <NUM>. The media input data may be received via one or more wired and wireless networks <NUM> and one or more wired or wireless direct connections. Receiving media data may include receiving media data sent from a media device <NUM>, e.g., a camera <NUM>, or retrieving data, e.g., from a storage device <NUM>. The media studio <NUM> is further operable to be controlled by and receive media input data from an operator console <NUM>, which may be remote to the media studio <NUM>.

The media studio <NUM> is further operable to generate media output data and provide the media output data to media devices <NUM>, e.g., a viewer <NUM>. The system <NUM> allows a user to produce media output data at a location where the recording (or a portion of the recording) is taking place. Controlling the media studio <NUM> via a remote operator console <NUM> provides the user the freedom to move about the recording location while maintaining oversight and control over the production process.

Referring now to <FIG> in more detail, the media processing system <NUM> includes a media studio <NUM>, and one or more media devices <NUM>. The one or more media devices <NUM> include the operator console <NUM>, and may further include one or more cameras <NUM>, one or more viewers <NUM> and one or more media storage devices <NUM>. The media devices <NUM> may be remote or local to the media studio <NUM> and may be coupled to the media studio <NUM> via at least one of a network <NUM> and a direct wired or wireless connection. A server <NUM> may also be communicatively coupled to the media studio <NUM> and the media devices <NUM> via the networks <NUM>. Additionally, the media processing system <NUM> may include a computer or television monitor <NUM> communicatively coupled to the media studio <NUM>.

The media studio <NUM> is operable to receive multiple media input items, and to combine them into a multiview media item for viewing via the operator console <NUM>. The media input items are sometimes referred to herein as individual media items. The multiview media item is an aggregated media content item that includes one or more of the media input items, arranged to be displayed substantially simultaneously on a user display, such as a touch screen. The media input items may be arranged such that they appear side-by-side, in rows, in a picture-in-picture format, etc. within the user display. In addition, the multiview media item may include, for example, a graphical user interface which is displayed on the user display and accepts user inputs. As described below, the media studio <NUM> may compress the multiview media item prior to providing it to one of the media devices <NUM> such as the operator console <NUM> and/or the storage devices <NUM>.

The media input items may be visual and/or audio data such as videos captured by a video camera, sounds captured by a microphone, scenes from a security camera, etc. The media input items may be, e.g., streamed data or static data such as single digital photographs. The media studio <NUM> is further operable to receive commands from the operator console <NUM>, and to generate a media output item according to the received commands. The media output item may include data from one or more of the media input items.

As an example, the media studio <NUM> may receive four media input items. The media studio <NUM> may generate a multiview media item including each of the four media input items, and transmit the multiview media item to the operator console <NUM>. A user of the operator console <NUM> may select, via a user interface, one of the four views in the multiview media item to be included in the media output item. Based on a command received from the operator console <NUM>, the media studio <NUM> may generate the media output item that includes the selected media input item. The media output item may be transmitted via a network <NUM> to be shared with viewers or stored in a storage device <NUM>.

In addition to selecting one or more media input items to be included in the media output item, the media studio <NUM> may perform various media processing functions. The media processing functions may be performed based on commands received from the operator console. A nonlimiting list of processing functions that may be performed by the media studio <NUM> includes scaling, mixing, morphing, compositing, adding overlays, etc. In addition, the media studio may perform functions such as object tracking, image stabilization, etc. The operation of the media studio <NUM> will be discussed in greater detail below.

Communications between the media studio <NUM> and the media devices <NUM> may occur via the network <NUM> and via direct connections <NUM>. In general, the network <NUM> represents one or more mechanisms for delivering media content between the media studio <NUM> and the media devices <NUM>. Accordingly, the network <NUM> may be one or more of various wired or wireless communication mechanisms, including any desired combination of wired (e.g., cable and fiber) and/or wireless (e.g., cellular, wireless, satellite, microwave, and radio frequency) communication mechanisms and any desired network topology (or topologies when multiple communication mechanisms are utilized). Exemplary communication networks include wireless communication networks, local area networks (LAN) <NUM> such as a WiFi network or Ethernet, and/or wide area networks (WAN) <NUM>, such as the Internet, etc..

In addition to the one or more networks <NUM>, one or more wired or wireless direct connections <NUM> may be used to connect the media studio <NUM> to the media devices <NUM>. Direct connections <NUM> may include e.g., Bluetooth, Universal Serial Bus (USB), high-definition multimedia interfaces (HDMI), custom serial interfaces, etc..

In particular, one or more high-definition multimedia interfaces (HDMI) may be used to transfer data between a media device <NUM> and the media studio <NUM>, or from the media studio <NUM> to the monitor <NUM>. HDMI is a well-known, proprietary audio/video interface for transferring uncompressed video data and compressed or uncompressed digital audio data from an HDMI-compliant source device such as the media device <NUM> to a digital media processing device such as the media studio <NUM> or to the compatible computer monitor, e.g., the monitor <NUM>.

The server <NUM> may be communicatively coupled to the media studio <NUM> and the media devices <NUM> via the network <NUM>. The server <NUM> may include a communications circuit for communications via the network <NUM>, and may further include memory and one or more processors configured to perform programs, i.e., sets of computer-executable instructions, stored in the memory. The server <NUM> may, e.g., receive media output items and store the media output items for future use.

Media content, such as the media input items, media output items, and/or multiview media items, is generally delivered via the network <NUM> in a digital format, e.g., as compressed audio and/or video data, and may include media data and metadata. For example, MPEG refers to a set of standards generally promulgated by the International Standards Organization/International Electrical Commission Moving Picture Experts Group (MPEG). <NUM> refers to a standard promulgated by the International Telecommunications Union (ITU). Accordingly, by way of example and not limitation, media content may be provided in a format such as the MPEG-<NUM>, MPEG-<NUM>, or the H. <NUM>/MPEG-<NUM> Advanced Video Coding standards (AVC) (H. <NUM> and MPEG-<NUM> at present being consistent), HEVC/H. <NUM>, or according to some other standard or standards. For example, media content could be audio data formatted according to standards such as MPEG-<NUM> Audio Layer III (MP3), Advanced Audio Coding (AAC), etc. Further, the foregoing standards generally provide for including metadata.

Media devices <NUM> include the operator console <NUM>, and may include one or more cameras <NUM>, one or more viewers <NUM> and one or more storage devices <NUM>. The operator console <NUM> may be used to control the operation of the media studio <NUM>, and in some cases, may also perform the function of a media input device <NUM> such as a camera <NUM>, as described below.

The camera <NUM> captures media data, i.e., visual and sound data such as photographs and videos, and transmits the media data, e.g., via the network <NUM>, to the media studio <NUM>. Examples of cameras <NUM> include portable devices such as smartphones, tablets, laptops, digital cameras, etc., including one or more data collectors for collecting media data. The collected media data may include visual data and audio data. Examples of cameras <NUM> may further include security cameras, traffic cams, cameras transported by drones, etc..

The camera <NUM> may collect and transmit the media data, e.g., via the network <NUM>, or via the direct connection <NUM>, to the media studio <NUM>. The collected media data may be, e.g., full high-definition (Full HD) data, providing an image resolution of <NUM> by <NUM> pixels. Data formats with other resolutions may also be used.

The viewer <NUM> may be used to display media output data received from the media studio <NUM>, and may include a display such as a liquid crystal display (LCD) or plasma display. The media data may be received, for example, via the network <NUM> or via the direct connection <NUM>. Examples of the viewer <NUM> include mobile devices such as mobile phones, tablets, and laptops and may further include devices such as digital televisions. The viewer <NUM> may receive, e.g., Full HD data, providing an image resolution of <NUM> by <NUM> pixels. Data formats with other resolutions may also be used.

The storage device <NUM> may store media data and provide an interface to allow the media studio <NUM> to access the data via the network <NUM> or via the direct connection <NUM>. The media storage device may include one or more types of data storage such as read only memory (ROM), random access memory (RAM), flash memory, electrically programmable memory (EPROM), electrically programmable and erasable memory (EEPROM), embedded MultiMediaCard (eMMC), secure digital (SD) card, a hard drive, etc. Further, the media storage device <NUM> may include a processor, programmed, for example, to receive commands from the media studio <NUM>. The processor may be further programmed, based on the commands, to retrieve media data items from data storage, and send the media data items to the media studio <NUM>.

Communications between the storage device <NUM> and the media studio <NUM> may be performed via the network <NUM>. Additionally or alternatively, communications may be performed via the direct connection <NUM>. For example, the storage device <NUM> may be connected to the media studio <NUM> via a Universal Serial Bus (USB) port, or other wired or wireless interface.

The operator console <NUM> may be used to control the operation of the media studio <NUM>. As shown in <FIG>, the operator console <NUM> may include a memory <NUM> and a processor <NUM>, the memory <NUM> storing program code, i.e., computer-executable instructions, executable by the processor <NUM>. The processor <NUM> may be communicatively coupled to a user interface <NUM>, a data collector <NUM>, and a communications circuit <NUM>.

The user interface <NUM> is communicatively coupled to the processor <NUM> and may include one or more input devices such as a microphone, buttons, a touchscreen display, a mouse, a keyboard, a gesture-recognition device, switches, etc., for receiving input from the user. The user interface <NUM> may further include one or more output devices such as a display, lamps, speakers, etc. for communicating information to the user. All or a portion of the user interface <NUM> may be physically separate from the operator console <NUM>. For example, the operator console <NUM> may be a tablet computer which projects its output to another screen, (e.g., air-play), while the operator continues to control the media studio <NUM> from the tablet computer.

The data collector <NUM> may be, for example, a digital camera as is known that may collect media data. The media data may include visual data such a still photographs and video recordings and may further include audio data such as a sound recording or soundtrack. The data collector <NUM> may, e.g., output the media data to the processor <NUM>.

The communications circuit <NUM> is communicatively coupled to the processor <NUM>, and is configured to communicate with the media studio <NUM> via, e.g., the network <NUM> and/or the direct connections <NUM>.

The communications circuit <NUM> may include a radio frequency (RF) transceiver for WiFi communications (typically <NUM> or <NUM> bands). The RF transceiver may communicate, for example, directly with a RF receiver included in the media studio <NUM>. Additionally or alternatively, the communications circuit <NUM> may include, e.g., an Ethernet interface, a Universal Serial Bus (USB) interface, a Bluetooth transceiver, a high-definition multimedia interface (HDMI), etc..

Alternatively, the communications circuit <NUM> may communicate with the media studio <NUM> indirectly. For example, the communications circuit <NUM> may communicate with hotspot, i.e., a communications circuit including a router and client providing a connection to a wide area network (WAN) <NUM> such as the Internet. The media studio <NUM> may receive the communications via the WAN <NUM>.

The processor <NUM> is communicatively coupled to each of the user interface <NUM>, the data collector <NUM>, and the communications circuits <NUM>.

The processor <NUM> is programmed to control the operation of the media studio <NUM> based on inputs received from a user via the user interface <NUM>. More specifically, the processor <NUM> is programmed to receive a media content item, also called an aggregated media item, e.g., a multiview media item including one or more views from one or more cameras <NUM>, and to display the multiview media item via the user interface <NUM>. The processor <NUM> is further programmed to receive input from the user via the user interface <NUM>. For example, the user may view the multiview media item, and select one of the views to be included in a media output item generated by the media studio. The processor <NUM> may send a command to the media studio <NUM> to include the selected view in the media output item.

In addition to commands related to selecting media input items for display in the media output item, commands from the operator console <NUM> may include instructions to perform functions such as scaling, mixing, morphing, compositing, adding overlays, etc. Further, commands from the operator console <NUM> may include instructions to perform functions such as object tracking, image stabilization, etc..

An exemplary media studio <NUM> is shown in <FIG>. The media studio <NUM> includes an internal memory <NUM> and a processor <NUM>, the memory storing program code, i.e., computer-executable instructions, executable by the processor <NUM>. The processor <NUM> is communicatively coupled to a user interface <NUM>, a network interface <NUM>, and an auxiliary interface <NUM>.

The internal memory <NUM> may be, e.g., read only memory (ROM), random access memory (RAM), flash memory, electrically programmable memory (EPROM), electrically programmable and erasable memory (EEPROM), embedded MultiMediaCard (eMMC), a hard drive, etc., and may be used to store programs executable by the processor <NUM>, as well as to store, for example, data representing inputs from the user, instructions received from the operator console <NUM>, media data received from the remote media device <NUM>, media metadata, etc..

The user interface <NUM> is communicatively coupled to the computer <NUM>, and may include one or more output devices such as a display, lamps, speakers, etc. for communicating information to the user. The user interface <NUM> may further include one or more input devices such as buttons, a microphone, a touch screen display, a mouse, a keyboard, a gesture-recognition device, switches, etc., for receiving input from the user.

The network interface <NUM> includes one or more interfaces to the network <NUM>. For example, the network interface <NUM> may include a hotspot, such as is known, for WiFi communications. The hotspot may include a router. The router may include a radio frequency (RF) transceiver for WiFi communications (typically <NUM> or <NUM> bands) and may be operable to receive multiple transmissions substantially simultaneously. The router may connect media devices <NUM> with the processor <NUM> of the media studio <NUM>.

Additionally, the network interface <NUM> may include an Internet client. The Internet client is a mechanism for connecting to and communicating with the Internet, via, e.g., satellite communications or a cable network. The Internet client may include a transceiver and antenna for satellite communications, e.g., in the Ka band (<NUM>-<NUM>). The internet client may receive, via the network <NUM>, Internet protocol (IP) communications from, e.g., media devices <NUM>.

The network interface <NUM> may further include hardware, software, and firmware that support communications with, e.g., portable and mobile devices in the cellular frequency bands, and that may operate, for example, according to the <NUM> LTE protocols as defined by the 3GPP standards body, or another suitable wireless communications protocol.

In addition to connecting the media devices <NUM> with the processor <NUM> of the media studio <NUM>, the router and the Internet client may be used in combination to provide Internet access for media devices <NUM>.

The auxiliary interface <NUM> may include one or more wired or wireless interface circuits that may be used, for example, to connect to one or more media devices <NUM>. The media devices <NUM> may include, e.g., one or more storage devices <NUM>. The auxiliary interface <NUM> may include a universal serial bus (USB) interface circuit to communicate with external USB devices, for example a memory stick or memory back-up device. As another example, the auxiliary interface <NUM> may include a MicroSD interface, as is known, to store data on and retrieve data from a MicroSD data card. Further, the auxiliary interface <NUM> may include, e.g., a Bluetooth interface for wireless connection to a media device <NUM>.

Additionally, the media studio <NUM> may include a high-definition media interface (HDMI) for connecting to a media device <NUM>, such as a camera <NUM>.

The media studio <NUM> processor <NUM> is generally programmed to receive one or more media input items from one or more media devices <NUM>. The media studio <NUM> processor <NUM> may, in some cases, generate a multiview media item. The multiview media item may include, for example, a picture-in-picture (PIP) display, wherein two or more of the media input items are displayed simultaneously, e.g., side-by-side, within the multiview media item. The media studio <NUM> may transmit the multiview media item via the network <NUM> to one or more of the media devices <NUM>, such as the operator console <NUM> or the storage devices <NUM>.

The media studio <NUM> processor <NUM> is further programmed to receive commands from the operator console <NUM>. Based on the commands, the media studio <NUM> may generate a media output item. The media studio <NUM> may select data from one or more of the media input items to include in the media output item. The media studio <NUM> may further, based on commands received from the operator console <NUM>, perform various media processing functions such as scaling, mixing, morphing compositing, adding overlays, tracking of specific people or objects, smart tagging etc. related to the generation of the media output item.

In addition, the media studio <NUM> may perform media processing functions based on predetermined rules for generating the media output item. Examples of rules are discussed below.

The media studio <NUM> processor <NUM> may output the media output item e.g., to viewers <NUM>, to the operator console <NUM>, and/or to other display devices. Additionally or alternatively, the media studio <NUM> may output the media output item, e.g., to a server <NUM>, or to storage devices <NUM>, where the media output item may be stored for future use. Exemplary implementations for the media studio <NUM> are described below.

A first exemplary implementation is shown in <FIG>. The media studio <NUM> receives first, second, and third media input items 80a, 80b, 80c, respectively, from first, second, and third cameras 20a, 20b, 20c. Additionally, the media studio <NUM> receives a fourth media input item 80d from the operator console <NUM>. Each of the first, second, third, and fourth media input items 80a, 80b, 80c, 80d, may be, for example, Full High Definition (Full-HD) and may include a picture resolution of <NUM> by <NUM> pixels. Data formats with other resolutions may also be used.

As described above, the media input items 80a, 80b. 80c, 80d may be received by the network interface <NUM> included in the media studio <NUM>. The network interface <NUM> may include a router to support WiFi communications directly with the cameras <NUM> and operator console <NUM>. Additionally or alternatively, the network interface <NUM> may include, e.g., an Internet client. The media studio <NUM> may receive media input items <NUM> from the cameras <NUM> and operator console <NUM> indirectly, via the wide area network <NUM>, e.g., the Internet. The media input items <NUM> received via the network interface <NUM> may be media data which has been compressed, according to, e.g., H. <NUM>/MPEG-<NUM> Advanced Video Coding standards (AVC), or another compression standard as is known. Audio media data may be compressed, e.g., according to Advanced Audio Coding (AAC).

Additionally, the media studio <NUM> may receive a fifth media input item <NUM> via high-definition multimedia interface (HDMI). HDMI is a wired interface which may be used to transfer the fifth media input item <NUM> from, for example, a camera <NUM> to the media studio <NUM>. The fifth media input item <NUM> received via the HDMI may be uncompressed data.

An input multiplexer (mux) <NUM> may be a portion of the processor <NUM> and/or a separate electronic circuit, and may be implemented as hardware, firmware, software, or a combination of any or all of hardware, firmware, or software. In the first exemplary implementation, the input mux <NUM> is communicatively coupled to the network interface <NUM>, first, second, third, and fourth decoders 72a, 72b, 72c, 72d, third encoder 78c, and the auxiliary interface <NUM>. The input mux <NUM> receives the first, second, third, and fourth media input items 80a, 80b, 80c, 80d (collectively media input items <NUM>) from the network interface, and may additionally receive the fifth media input item <NUM>. The input mux <NUM> is programmed to route the media input items <NUM> and additional media input item <NUM>, e.g., to the decoders <NUM>, or the encoder 78c for further processing.

For example, as further shown in <FIG>, the input mux <NUM> may route the first, second, third, and fourth media input items 80a, 80b, 80c, 80d respectively to decoders 72a, 72b, 72c, 72d. The input mux <NUM> may further route the media input item <NUM>, received via HDMI, to the third encoder 78c.

The foregoing is only an example of how media input items <NUM> may be routed. In principle, any of the input media items <NUM>, <NUM> may be routed to any of the decoders <NUM>. In some cases, the routing may be indirect. For example, the media input item <NUM>, may, as discussed above, be received as uncompressed media data. In order for this data to be in a similar or same format to the other media input items <NUM>, the media input item <NUM> may be compressed. Following compression, for example according to H. <NUM>/MPEG-<NUM> AVC, the media input item <NUM> may also be routed to one of the decoders <NUM>.

The decoders <NUM> generally decompress media input items, i.e., receive compressed media input items and generate, based on the received media input items, decompressed media input items. In order to compensate for variation in latency between multiple media input items, the decoders <NUM> may be programmed to time synchronize the media input items during decompression. The decoders <NUM> may be included in the processor <NUM> and/or may be one or more separate electronic circuits, and may be implemented as hardware, firmware, software, or a combination of any or all of hardware, firmware, or software.

For example, still referring to <FIG>, the first decoder 72a may receive the media input item 80a from the input mux <NUM> compressed according to the standard H. <NUM>/MPEG4 AVC. The decoder 72a may decompress the media input item 80a and generate the decompressed media input item 80a' in Full-HD and may further provide the decompressed media input item 80a' to a video processor <NUM> and the PIP generator <NUM>.

In a similar manner, each of the second, third, and fourth decoders 72b, 72c, 72d may receive, respectively, second, third, and fourth media input items 80b, 80c, 80d as compressed data, e.g., according to the standard H. <NUM>/MPEG4 AVC, and generate respective decompressed media input items 80b', 80c', 80d'. Each of the decompressed media input items 80a', 80b', 80c', 80d' may be provided to the video processor <NUM> and the PIP generator <NUM>. The decompressed media input items 80a', 80b', 80c', 80d' may be referred to collectively as decompressed media input items <NUM>' herein.

The decoders <NUM> may decompress media input items compressed according to a variety of compression standards. A nonlimiting of example compression standards includes H. <NUM>/MPEG4, MPEG2, MPEG1, Advanced Audio Coding (AAC), etc. Other compression standards, including future compression standards, may be used.

The encoders <NUM> generally compress media input items, i.e., receive uncompressed media input items, e.g., in Full-HD, and generate, based on the received media input items, compressed media input items. The compression may be, e.g., according to the standard H. <NUM>/MEG4 AVC. Other compression standards may be used. The encoders <NUM> may be included in the processor <NUM> and/or may be one or more separate electronic circuits, and the encoders <NUM> may be implemented as hardware, firmware, software, or a combination of any or all of hardware, firmware, or software.

The media studio <NUM> may further include one or more transcoders (not shown). The transcoders may be included in the processor <NUM> and may perform, e.g., digital-to-digital conversion of one encoding to another, such as for video files (e.g., MPEG2, MPEG4) and audio files (e.g., MP3, WAV).

As shown in <FIG>, a first encoder 78a may receive a media output item 84a from the video processor <NUM>. The media output item 84a may be decompressed.

The first encoder 78a may generate a media output item 84a'. The media output item 84a' may be compressed according to, e.g., H. <NUM>/MPEG4 AVC. The first encoder 78a may output the compressed media output item 84a', for example, via the network <NUM> to one or more viewers <NUM>. The compressed media output item 84a' may be generated according to a Standard Quality (SQ) format. The standard quality format may include a relatively low resolution or bit rate to accommodate, e.g., viewers <NUM> that are compatible with the low resolution or bit rate. Additionally, the first encoder 78a may, via an auxiliary interface <NUM>, store the compressed media output item 84a' to the storage devices <NUM>.

A second encoder 78b may receive a media output item 84b from the video processor <NUM>. The media output item 84b may be generated according to a High Quality (HQ) format. The high quality format may include, e.g., Full-high definition (HD) with a higher resolution and bit rate in comparison to the standard quality format. The media output item 84b, as output from the video processor <NUM>, may be decompressed.

The second encoder 78b may generate the media output item 84b'. The media output item 84b' may be compressed according to, e.g., H. <NUM>/MPEG4 AVC. The second encoder 78b may output the compressed media output item 84b', for example, via the network <NUM> to one or more viewers <NUM>. The compressed media output item 84b' may be generated, e.g., in Full high-definition (HD) and may be output to viewers <NUM> configured to accommodate Full-HD. Further, the compressed media output item 84b' may be stored, for example in a storage device <NUM> for future use.

The fourth encoder 78d may receive a multiview media item <NUM> from the PIP Generator <NUM> which may be uncompressed and formatted, e.g., in Full-HD. The fourth encoder 78b may generate the multiview media item <NUM>' compressed according to H. <NUM>/MPEG4 AVC. The fourth encoder 78d may output the compressed multiview media item <NUM>', via the network <NUM>, to the operator console <NUM>.

Further, as shown in <FIG>, the third encoder 78c may receive the media input item <NUM> in an uncompressed format, and output the media input item <NUM>' compressed according to H. <NUM>/MPEG4 AVC to the storage device <NUM> via the auxiliary interface <NUM>.

The media studio <NUM> includes a video processor <NUM> which generates first, second, and third media output items 84a, 84b, 84c. The video processor <NUM> may be programmed, e.g., to process high-definition media data. The video processor <NUM> may be included in the processor <NUM> and/or may be one or more separate electronic circuits, and may be implemented as hardware, firmware, software, or a combination of any or all of hardware, firmware, or software.

The video processor <NUM> is programmable to receive commands from the operator console <NUM> and perform video processing functions such as switching, mixing, compositing, scaling, morphing, etc. based on the commands. The video processor <NUM> may further be programmable to perform video processing functions based on one or more predetermined rules, described below. The first, second, and third media output items 84a, 84b, 84c may include data from one or more of the decompressed media input items 80a', 80b', 80c', 80d', <NUM>. As a first example, based on a first command from the operator console <NUM>, the video processor <NUM> may generate the first, second, and third media output items 84a, 84b, 84c to include one of the decompressed media input items <NUM>', e.g., the decompressed media input item 80b'. Upon receiving a second command from the operator console <NUM>, the video processor <NUM> may switch to generating the first, second, and third media output items 84a, 84b, 84c to include a different one of the decompressed media input items <NUM>', e.g., the media input item 80d'.

The video processor <NUM> may perform more complex video processing functions such as superimposing a first media input item <NUM>, e.g., 80a', on a background from a second media input item, e.g., 80b', displaying two decompressed media input items <NUM>' side-by-side, fading from one decompressed media input <NUM>' to a second decompressed media input item <NUM>', displaying video from a first decompressed media input item <NUM>' together with audio from a second decompressed media input item <NUM>', etc..

In addition to being programmed to perform video processing functions based on commands received from the operator console <NUM>, the video processor <NUM> may be programmed to perform video processing functions according to predetermined rules. A nonlimiting list of example rules that may be used in relation to generating the media output item include:.

For example, according to an example time protocol, the video processor <NUM> may switch sequentially between each of four different media input items <NUM>', with each of the four media input items <NUM>' being displayed for a predetermined time period, e.g., <NUM> seconds. For an example motion-based selection rule, the video processor <NUM> may identify one of the media input items <NUM>' with a highest amount of motion, or an amount of motion greater than a predetermined motion threshold, for display in the media output item. For an example signal-strength selection rule, the video processor <NUM> may identify, e.g., one or more media input items <NUM>' with a signal strength above a predetermined signal strength threshold, and display the identified one or more input items <NUM>' sequentially. For an example of person or object tracking, a person or object may be identified, for example, by a command from the operator console <NUM>. The video processor <NUM> may, based on the identified person or object, select one of the media input items <NUM>' which includes the identified person or object. According to an example voice or noise level selection rule, the video processor <NUM> may select a media input item associated, e.g., with a highest voice or noise level.

The video processor <NUM> may further be programmed to perform media processing operations based on downloaded applications. The applications may be event specific. For example, an application may be specific for baseball games. A first camera <NUM> may be used to capture the pitcher, a second camera <NUM> to capture the batter, and third and fourth cameras <NUM> may be used manually by operators to track play in the field. The rule may, based on, e.g., current game conditions, select between the various media input items. The video processor <NUM> may further be programmed to accept commands from the operator console <NUM> to override the selection of the rule.

The media studio <NUM> further includes a picture-in-picture (PIP) generator <NUM>. The PIP generator <NUM> may be included in the processor <NUM> and/or may be one or more separate electronic circuits, and may be implemented as hardware, firmware, software, or a combination of any or all of hardware, firmware, or software.

The PIP generator <NUM> is programmable to receive one or more decompressed media input items <NUM>', <NUM> and to combine them into a multiview media item <NUM>. The multiview media item <NUM> may include, for example, four media input items <NUM>', arranged in two rows of two images each. The PIP generator <NUM> may output the multiview media item <NUM> to, e.g., the fourth encoder 78d for compression. The fourth encoder 78d may generate a compressed version of the multiview media item <NUM>', which may be transmitted, e.g., via the network <NUM> to the operator console <NUM>.

According to the first exemplary implementation, as shown in <FIG>, the media studio <NUM> may receive, via the network <NUM>, four media input items <NUM>, compressed, e.g., according to the standard H. <NUM>/MPEG4 AVC. The four compressed media input items <NUM> may be passed, via the input mux <NUM>, to four decoders <NUM>, respectively, where the media input items <NUM> are decompressed, resulting in four HD media input items <NUM>'. The four HD media input items <NUM>' may be provided to the PIP generator <NUM> and the video processor <NUM>.

The PIP generator may generate a high-definition multiview media item <NUM> that includes all four media input items <NUM>', arranged such that the may be viewed simultaneously. The high definition media item <NUM> may be compressed, e.g., according to the standard H. <NUM>/MPEG4 AVC by the second encoder <NUM>, generating a compressed multiview media item <NUM>'. The compressed multiview media item <NUM>' may be transmitted via the network <NUM> to the operator console <NUM>.

As a next step, a user of the operator console <NUM> may view the multiview media item <NUM>' on the user interface <NUM> (<FIG>) of the operator console <NUM>. The user may, e.g., select one of the four media input items <NUM>' to be output by the media studio <NUM>. For example, the user interface <NUM> may include a touchscreen display, and the user may tap the image to be displayed on the touchscreen display. As described above, the user may additionally select one or more other operations such as scaling, mixing, morphing, providing overlays, etc., in generating the media output item. Based on the input from the user, the operator console <NUM> may send one or more commands to the media studio <NUM> indicating operations to be performed in preparing the output media item.

The video processor <NUM> may be programmed, upon receiving the one or more commands from the operator console <NUM>, to include the selected media input item <NUM>' in the first, second and third media output items 84a, 84b, 84c, and to perform any additional media processing indicated by the commands. The video processor <NUM> may output the first media output item 84a to the first encoder 78a for compression, e.g., according to the standard H. <NUM>/MPEG4. The encoder 78a may output the compressed version of the media output item 84a', for example, via the network <NUM> to one or more viewers <NUM>. A second, e.g. high-definition media output item 84b may be output to the encoder 78b. The encoder 78b may output a compressed media output item 84b in high-definition format for high-definition viewing. A third version of the media output item 84c may be output via an HDMI to, e.g., the monitor <NUM>.

Additionally or alternatively to outputting the media output item 84b' for viewing, the media studio <NUM> may store the media output item 84b', and one or more of the media input items <NUM>, <NUM> in the storage device <NUM>, via the auxiliary interface <NUM>.

A second exemplary implementation is shown in <FIG>. In the second implementation, the first, second, and third media input items 80a, 80b, 80c are provided to the operator console <NUM> directly, for example, via one or more wired or wireless connections, instead of indirectly via the PIP generator <NUM>. The media input items 80a, 80b, 80c may be provided as low resolution stream inputs.

In other aspects, the second exemplary implementation operates in a like manner to the first exemplary implementation. The video processor <NUM> receives the media input items <NUM>', <NUM> via the input mux <NUM> and decoders <NUM> as described above. The video processor <NUM> may further receive commands from the operator console <NUM> providing instructions for the generation of the first and third media output items 84a, 84c. The first media output item 84a may be, e.g., encoded to generate compressed media output item 84a' which may be transmitted to one or more viewers <NUM>. The second media output item, may be, for example, transmitted to the monitor <NUM>. Further, one or more of the media input items <NUM>', <NUM>, and/or the media output item 84a', may be stored to storage devices <NUM>.

A third exemplary implementation is shown in <FIG>. In the third implementation, the media studio <NUM> may be programmed to receive, e.g., first, second, and third media input items 80a, 80b, 80c, and directly output first, second, and third media output items 80a", 80b", 80c". The first, second, and third media output items 80a", 80b", 80c" may be representations respectively of media input items 80a, 80b, 80c, following decompression via the first, second, and third decoders 72a, 72b, 72c and recompression via the first, third, and fourth encoders 78a, 78c, 78d. Each of the first, second, and third media input items 80a", 80b", 80c" may be, e.g., transmitted via the network <NUM> to storage in a remote server, such as the server <NUM> (<FIG>). In this manner, the first, second, and third media input items 80a", 80b", 80c" may be available for further processing at a later time.

In each of the example implementations described above, one or more of the media input items <NUM> may be stored data that is retrieved from a storage device <NUM>, or received, for example, from the server <NUM>. Further, any of the media input items <NUM>, <NUM> may be included in the media output item <NUM>. For example, the media input item <NUM> may additionally or alternatively be included in the media output item <NUM> in any of the first and second implementations, or one of the media output items of the third implementation. Still further, any or all of the media input items <NUM>, <NUM> and the media output item <NUM> may be stored in the storage device <NUM>. The media input items <NUM>, <NUM> may be stored as compressed data or as decompressed data.

The implementations above are described based on use of the H. <NUM>/MPEG4AVC compression standard. This is only intended as an example. Other compression standards such as MPEG1, MPEG2, etc. may also be used as required for compatibility, e.g., with media data sources and media display devices such as the media devices <NUM>.

<FIG> is a diagram of an exemplary process <NUM> to generate a media output item <NUM> based on one or more media input items <NUM>, <NUM> under control of the remote operator console <NUM>. The process <NUM> may be carried out according to computer-executable instructions, e.g., in the media studio <NUM>. The process <NUM> starts in a block <NUM>.

In the block <NUM>, the media studio <NUM> recognizes a trigger event for initiating the process <NUM>. For example, the media studio <NUM> may receive an input from a user via the user interface <NUM>. Additionally or alternatively, the media studio <NUM> may receive an input from the user via the operator console <NUM>. Still further, the media studio <NUM> may receive an input, e.g., from the remote server <NUM> indicating that the user would like to initiate generation of one or more media output items. The process <NUM> continues in a block <NUM>.

In the block <NUM>, the media studio <NUM> processor <NUM> receives one or more media input items <NUM>. As described above, the media studio <NUM> processor <NUM> may receive media input items from cameras <NUM>, the operator console <NUM>, and/or from storage devices <NUM>. The media studio <NUM> processor <NUM> may receive the media input items via the network <NUM> or via the direct connection <NUM>. The direct connection <NUM> may include a high-definition multimedia interface (HDMI). The process <NUM> continues in a block <NUM>.

In the block <NUM>, the media studio <NUM> processor <NUM> generates a multiview media item <NUM> for the operator console <NUM>. As described above, the picture-in-picture (PIP) generator <NUM> (<FIG>) may receive one or more decompressed media input items <NUM>', <NUM>. The PIP generator <NUM> may generate the multiview media item <NUM> to include one or more of the media input items <NUM>', <NUM> displayed simultaneously. For example, the multiview media item <NUM> may arrange four media input items <NUM>, <NUM> to be viewed simultaneously in a single display. The process <NUM> continues in a block <NUM>.

In the block <NUM>, the media studio <NUM> processor <NUM> transmits the multiview media item <NUM> to the operator console <NUM>. The multiview media item <NUM> may, e.g., be compressed by the encoder <NUM>, and transmitted via the network <NUM> as compressed media data. The operator console <NUM> processor <NUM> may receive and display the multiview media item <NUM> on the user interface <NUM>. The process <NUM> continues in a block <NUM>.

In the block <NUM>, the media studio <NUM> processor <NUM> receives commands from the operator console <NUM> processor <NUM>. For example, the user may select, via the user interface <NUM>, one of the media input items <NUM>', <NUM> for generation of the media output item <NUM>. The user may further select one or more processing functions such as scaling, mixing, morphing, overlaying graphics, etc. for generation of the media output item <NUM>. The operator console <NUM> processor <NUM> may transmit one or more commands based on the input from the user, which may be received by the media studio <NUM> processor <NUM>. The process <NUM> continues in a block <NUM>.

In the block <NUM>, the media studio <NUM> processor <NUM> generates the media output item <NUM> based on the command received from the operator console <NUM> processor <NUM>. For example, the media studio <NUM> processor <NUM> may receive the command to include the media input item 80c from the third camera 20c in the media output item. The media studio <NUM> may further receive one or more commands to perform other media processing operations. Based on the commands, the video processor <NUM> may generate a media output item <NUM> that includes the media input item 80c from the third camera 20c processed according to the selected media processing operations. The process <NUM> continues in a block <NUM>.

In the block <NUM>, the media studio <NUM> processor <NUM> transmits the media output item <NUM>. The media output item may be transmitted via the network <NUM> to one or more viewers <NUM>. Additionally or alternatively, the media output item <NUM> may be transmitted via the network <NUM> to a remote server <NUM> for storage. Still further, the media output item <NUM> may be stored via the auxiliary interface <NUM> in one or more storage devices <NUM>. The process continues in a block <NUM>.

In the block <NUM>, the media studio <NUM> processor <NUM> determines whether an event has occurred to trigger an end of the process <NUM>. For example, the media studio <NUM> may receive an input from the user interface <NUM> or from the operator console <NUM> processor <NUM> indicating that the process should end. Alternatively, the media studio <NUM> may determine that no new data and/or no new commands have been received for a predetermined time, e.g., <NUM> seconds, and may determine that the process <NUM> should end. Still further, the media studio <NUM> processor <NUM> may receive an input from the user interface <NUM> or from the operator console <NUM> processor <NUM> indicating that the media studio <NUM> should generate the media output item according to a predetermined rule. In this case, the media studio <NUM> processor <NUM> may end the process <NUM> and initiate the process <NUM>, described below. If no trigger event is detected, the process <NUM> may continue in the block <NUM>.

<FIG> is a diagram of an exemplary process <NUM> for generating a media output item <NUM> based on one or more media input items <NUM>, <NUM> based on a predetermined rule. The process <NUM> may be carried out according to computer-executable instructions, e.g., in the media studio <NUM>. The process <NUM> starts in a block <NUM>, and, as described below, may in some cases be initiated subsequent to block <NUM> of the process <NUM>.

In the block <NUM>, the media studio <NUM> processor <NUM> recognizes a trigger event for initiating the process <NUM>. For example, the media studio <NUM> processor <NUM> may receive an input from a user via the user interface <NUM>. Additionally or alternatively, the media studio <NUM> processor <NUM> may receive an input from the user from the operator console <NUM> processor <NUM>. Still further, the media studio <NUM> may receive an input, e.g., from the remote server <NUM> indicating that the user would like to initiate generation of one or more media output items <NUM> according to a predetermined rule. The input may be received from the user interface <NUM> or from the operator console <NUM> processor <NUM>, as described above with reference to the block <NUM> of process <NUM>. Alternatively, the input may be received, e.g., upon turning on the media studio <NUM>. The process <NUM> continues in a block <NUM>.

In the block <NUM>, the media studio <NUM> processor <NUM> receives one or more media input items. As described above, the media studio <NUM> processor <NUM> may receive media input items <NUM> from cameras <NUM> and/or from the operator console <NUM> via the network <NUM>. Additionally or alternatively, the media studio <NUM> processor <NUM> may receive a media input item <NUM> via the direct connection <NUM> such as a high definition multimedia interface (HDMI). The process <NUM> continues in a block <NUM>.

In the block <NUM>, the media studio <NUM> processor <NUM> generates a media output item <NUM> from one or more of the media input items <NUM>, <NUM> according to a predetermined rule. For example, the predetermined rule may specify that the media studio <NUM> processor <NUM> should sequentially switch each of the available media input items <NUM>, <NUM>, at a first predetermined time period, to the media output item <NUM>. For example, the media studio <NUM> processor <NUM> may switch, every three seconds, from one media input item <NUM>, <NUM> to a next media input item <NUM>, <NUM>, and continue to loop through the media input items <NUM>, <NUM>. Other predetermined rules may be used, as described above. The process <NUM> continues in a block <NUM>.

In the block <NUM>, the media studio <NUM> processor <NUM> transmits the media output item <NUM>. The media output item <NUM> may be transmitted via the network <NUM> to one or more viewers <NUM>. Additionally or alternatively, the media output item <NUM> may be transmitted via the network <NUM> to a remote server <NUM> for storage. Still further, the media output item <NUM> may be stored via the auxiliary interface <NUM> in one or more storage devices <NUM>. The process continues in a block <NUM>.

In the block <NUM>, the media studio <NUM> processor <NUM> determines whether an event has occurred to trigger an end of the process <NUM>. For example, the media studio <NUM> processor <NUM> may receive an input from the user interface <NUM> or from the operator console <NUM> indicating that the process should end. Alternatively, the media studio <NUM> processor <NUM> may determine that a timeout has occurred, i.e., that the rule is intended to run for second predetermined time period, e.g., <NUM> minutes, and the second predetermined time period has been reached. Still further, the media studio <NUM> processor <NUM> may receive an input indicating that the user of the operator console <NUM> would like to manually control the generation of the media output item <NUM> via the operator console <NUM>. In this case, the process <NUM> may end, and the media studio may (re)initiate the process <NUM>, starting in the block <NUM>. If no trigger event is detected, the process <NUM> may continue in the block <NUM>.

As used herein, the adverb "substantially" means that a shape, structure, measurement, quantity, time, etc. may deviate from an exact described geometry, distance, measurement, quantity, time, etc., because of imperfections in materials, machining, manufacturing, etc..

The term "exemplary" is used herein in the sense of signifying an example, e.g., a reference to an "exemplary widget" should be read as simply referring to an example of a widget.

Networked devices such as those discussed herein generally each include instructions executable by one or more networked devices such as those identified above, and for carrying out blocks or steps of processes described above. For example, process blocks discussed above may be embodied as computer-executable instructions.

Computer-executable instructions may be compiled or interpreted from computer programs created using a variety of programming languages and/or technologies, including, without limitation, and either alone or in combination, Java™, C, C++, Visual Basic, Java Script, Perl, HTML, etc. In general, a processor (e.g., a microprocessor) receives instructions, e.g., from a memory, a computer-readable medium, etc., and executes these instructions, thereby performing one or more processes, including one or more of the processes described herein. Such instructions and other data may be stored and transmitted using a variety of computer-readable media. A file in a networked device is generally a collection of data stored on a computer readable medium, such as a storage medium, a random access memory, etc..

A computer-readable medium includes any medium that participates in providing data (e.g., instructions), which may be read by a computer. Such a medium may take many forms, including, but not limited to, non-volatile media, volatile media, etc. Non-volatile media include, for example, optical or magnetic disks and other persistent memory. Volatile media include dynamic random access memory (DRAM), which typically constitutes a main memory. Common forms of computer-readable media include, for example, a floppy disk, a flexible disk, hard disk, magnetic tape, any other magnetic medium, a CD-ROM, DVD, any other optical medium, punch cards, paper tape, any other physical medium with rules of holes, a RAM, a PROM, an EPROM, a FLASH-EEPROM, any other memory chip or cartridge, or any other medium from which a computer can read.

In the drawings, the same reference numbers indicate the same elements. Further, some or all of these elements could be changed. With regard to the media, processes, systems, methods, etc. described herein, it should be understood that, although the steps of such processes, etc. have been described as occurring according to a certain ordered sequence, such processes could be practiced with the described steps performed in an order other than the order described herein. It further should be understood that certain steps could be performed simultaneously, that other steps could be added, or that certain steps described herein could be omitted. In other words, the descriptions of processes herein are provided for the purpose of illustrating certain embodiments, and should in no way be construed so as to limit the claimed invention.

Accordingly, it is to be understood that the above description is intended to be illustrative and not restrictive. Many embodiments and applications other than the examples provided would be apparent to those of skill in the art upon reading the above description. The scope of the invention should be determined, not with reference to the above description, but should instead be determined with reference to the appended claims.

It is anticipated and intended that future developments will occur in the arts discussed herein, and that the disclosed systems and methods will be incorporated into such future embodiments. In sum, it should be understood that the invention is capable of modification and variation and is limited only by the following claims.

Claim 1:
A system (<NUM>) comprising a computing device (<NUM>), and one or more media input devices (<NUM>, <NUM>), wherein the one or more media input devices include an operator console (<NUM>) remote from the computing device (<NUM>), wherein the operator console (<NUM>) is configured to capture AV media items and control the computing device (<NUM>), and wherein the computing device (<NUM>) includes a processor (<NUM>) and a memory (<NUM>), the memory (<NUM>) storing instructions executable by the processor (<NUM>) such that the processor (<NUM>) is programmed to:
decompress a plurality of individual AV media items (<NUM>) using at least a plurality of decoders (<NUM>) to generate a plurality of decompressed AV media input items (<NUM>'), each of the plurality of individual AV media items (<NUM>) received from the respective one or more media input devices, at least one of the plurality of individual AV media items (<NUM>) received from the operator console (<NUM>) remote from the computing device;
process the plurality of decompressed AV media input items (<NUM>') to generate a first AV media output item (<NUM>) including data from one or more of the plurality of decompressed media input items;
output the first AV media output item (<NUM>) to one or more remote devices for viewing and/or storing of the first AV media output item;
generate a multiview media item (<NUM>) that includes multiple of the decompressed AV media input items (<NUM>'), the multiview media item (<NUM>) combining the one or more individual media items (<NUM>') for a common display;
send, to the operator console (<NUM>) remote from the computing device, the multiview media item (<NUM>);
receive one or more commands from the operator console (<NUM>), the one or more commands identifying at least one of the multiple of the AV media input items included in the multiview media item (<NUM>) to be included in a second AV media output item (<NUM>); and
generate the second AV media output item (<NUM>) based at least in part on the identified at least one of the multiple media input items; and
output the second AV media output item (<NUM>)to one or more remote devices for viewing and/or storing of the second AV media output item.