Patent Description:
The present disclosure relates generally to audio/video systems and more specifically to creating a tiled video display, and controlling the tiled video display.

As time progresses, users in both home and commercial settings have access to increasing amounts of video content provided as video streams from a variety of video source devices. Through cable television boxes and satellite television receivers, users may access a huge variety of televisions channels. Through digital media receivers, users may access an ever expanding number of individual programs and movies that are streamed over the Internet. Through Blueray™ and other disc players, users may access a massive collection of recorded video content. Still further video content may be obtained from surveillance cameras, video conferencing systems, computer systems, gaming systems, and the like. Given all this video content, it is not surprising that users often desire to browse multiple video streams from multiple video source devices simultaneously.

To address this issue, a variety of video processing systems have been developed that take video streams from different video source devices, and simultaneously display the video streams in windows shown on a screen of a display device. For example, a processing system may take <NUM> different video streams, and simultaneously display their video content in <NUM> video windows arranged in a matrix on the screen of the display device.

<CIT> discloses a display apparatus controlled by the touch screen of a separate mobile device, both displaying the same mosaic of available video sources as a tiled layout. On the touch screen, the user can resize a particular tile and thereby modify the tiled layout both on the display apparatus and on the touch screen of the mobile device.

<CIT> discloses a user interface having zoom functionality, wherein a user interface is displayed having representations of a plurality of content.

While existing video processing systems that use video windows may aid the user in browsing video content, they generally suffer a number of shortcomings.

First, they are generally difficult to control. Some video processing systems rely upon physical selection buttons, disposed on either a front panel of the video processing system itself, or a button-centric remote control. To select a video window, and to them perform operations on the window, for example, to resize the video window to obtain a larger view of the video content shown therein, the user may have to actuate specific physical buttons in a particular order. It may be difficult for a user to remember which buttons correspond to which video windows and which operations. Incorrect button presses may lead to undesired results and frustration. Other video processing systems rely upon a graphical user interface displayed on the screen of the display device to control the windows. The graphical user interface may be navigated via a button-centric remote control or a pointing device (e.g., a mouse). While such a graphical user interface may offer some advantages over physical buttons, it may still be difficult to operate. Often, navigating the graphical user interface is cumbersome. The user may need to repeatedly scroll through various selected video windows to select a desired one, or move an often-difficult to see cursor to a particular location to select a particular video window. Such operations may time consuming and frustrating.

Second, the number of different video steams many existing video processing systems can display simultaneously is inadequate for some applications. Existing video processing systems are typically limited to simultaneously displaying some modest fixed maximum number of different video streams, for example, <NUM>, <NUM>, <NUM>, etc. different video streams. Yet a user may have access to a far greater number of different video streams. A user may desire to view and rapidly change focus between dozens or even hundreds of videos, for example, to navigate through a video library, categorize video content, or other purposes that involve "long tail" video content. However, this may be difficult given the limitations of existing video processing systems.

Accordingly, there is a need for improved systems that may address some, or all, of these shortcomings.

Preferred embodiments are subject matter of the dependent claims.

The invention description below refers to the accompanying drawings, of which:.

<FIG> is a block diagram of an example architecture <NUM> for a video tiling system. The example architecture includes a matrix switcher/controller <NUM> and a host controller <NUM>. The matrix switcher/controller <NUM> is configured to switch signals between, control, and otherwise interoperate with a variety of devices, for example, providing N x N switching, multi-zone audio and video processing, device control, and other capabilities. The host controller <NUM> is configured to control and monitor operations of the matrix/switcher controller <NUM>, as well as to provide user interface interpretation and high-level control functions.

The matrix switcher/controller <NUM> may be coupled to a variety of video source devices, such as disc players <NUM>, cable television boxes <NUM>, satellite television receivers <NUM>, digital media receivers <NUM>, surveillance cameras, video conferencing systems, computer systems, gaming systems, and the like. At least some of the video source devices may also operate as audio sources, providing audio streams that, for example, accompany video streams provided by the devices. The matrix switcher/controller <NUM> may also be coupled to dedicated audio source devices (not shown), such as compact disc (CD) players, digital music players, radio receivers, etc. Likewise, the matrix switcher/controller <NUM> may be coupled to a variety of display devices, for example, display device <NUM>. The display devices may be televisions, monitors, projectors or other devices that can display video content. Display devices, including display device <NUM>, may also operate as audio output devices, and include speakers for playing audio streams that, for example, accompany video content. The matrix switcher/controller <NUM> may also be coupled to dedicated audio output devices (not shown). Further, the matrix switcher/controller <NUM> may be coupled to a variety of other types of devices, including lighting devices, heating ventilation and air conditioning (HVAC) devices, telephony devices, etc., either directly, or through one or more intermediate controllers.

The host controller <NUM> is coupled to the matrix switcher/controller <NUM> and may also be coupled to a data switch <NUM>. A wireless access point <NUM> may be coupled to the data switch <NUM>, or integrated into the data switch <NUM>. Alternatively, the host controller <NUM> may include its own wireless network interface.

The host controller <NUM> may wirelessly communicate with a variety of different types of user interface devices, such as remote controls, in-wall keypads, dedicated touch panels, and the like. In particular, the host controller <NUM> may communicate with one or more wireless mobile devices <NUM> having touch-sensitive screens. As used herein, the term "wireless mobile device" refers to an electronic device that is adapted to be transported on one's person, and includes wireless data communication capabilities. Devices such as tablet computers (e.g., the iPad® tablet available from Apple, Inc. ), smartphones (e.g., the iPhone® multimedia phone available from Apple, Inc. ), and portable media players (e.g., such as the iPod® touch available from Apple, Inc. ), are considered wireless mobile devices.

<FIG> is a block diagram of an example matrix switcher/controller <NUM>. The programmable multimedia controller <NUM> may include a general-purpose computer <NUM> having a processor <NUM> and a memory <NUM>. The memory <NUM> comprises a plurality of storage locations for storing software and data structures. The processor <NUM> includes logic configured to execute the software and manipulate data from the data structures. A general-purpose operating system <NUM>, portions of which are resident in memory <NUM> and executed by the processor <NUM>, may functionally organize the general-purpose computer <NUM>. The general-purpose operating system may be an OS X® Unix-based operating system, available from Apple, Inc. , or another type of operating system. A management process <NUM>, executed by the processor <NUM>, may operate to, among other things, manage the construction of the tiled video output stream, in conjunction with software executing on the host controller <NUM>, and on a wireless mobile device <NUM>. A microcontroller <NUM> may be interconnected to the general purpose computer <NUM>, and configured to implement low-level management of switching and device control operations for the matrix switcher/controller <NUM>. A device control interface <NUM> may operate under the direction of the microcontroller <NUM>, to communicate with, and provide control commands to, devices coupled to the matrix switcher/controller <NUM>. Further, an audio switch <NUM> and a video switch <NUM> may be controlled by the microcontroller <NUM>. The audio switch <NUM> and the video switch <NUM> may be N x N crosspoint switches. While shown as separate switches, the functionality of the audio switch <NUM> and the video switch <NUM> may be integrated into a single component in some implementations.

The matrix switcher/controller <NUM> may have a modular design, and a plurality of input/output slots may accommodate removable service modules. A mid plane <NUM> may interconnect the general purpose computer <NUM>, the audio switch <NUM>, the video switch <NUM> and other components of the matrix switcher/controller <NUM> to the input/output slots. Service modules may be disposed in the input/output slots. These modules include video input modules and may further include audio input modules, audio output modules, video output modules, and combined modules (collectively audio and/or video input and/or output modules <NUM>), audio processing modules, video processing modules, and combined processing modules (collectively audio and/or video processing modules <NUM>), as well as other types of modules (not shown) that perform different types of functions.

To implement a video tiling system that allows a user to view video content of multiple video streams from multiple video source devices simultaneously, one or more multi-window video processing output modules <NUM> are populated in the input/output slots of the matrix switcher/controller <NUM>. In one implementation, each multi-window video processing output module <NUM> includes two scalars, two frame buffers, two mixers, and other components for supporting two video tiles of a tiled video display. In other implementations, differing numbers of components may be included for supporting differing numbers of video tiles. For example, in another implementation an individual multi-window video output module <NUM> may support four video tiles of a tiled video display, or eight video tiles of a tiled video display.

The multi-window video processing output modules <NUM>, and their internal components, may be coupled to each other in a daisy chain arrangement, and operate to build a tiled video output stream through cascading operation. <FIG> is a block diagram of components <NUM> of two example multi-window video processing output modules coupled in a daisy chain arrangement. The host controller <NUM> may supply video, for example, graphics that represent an on screen display (OSD) to be shown in conjunction with the video tiles. The video may be provided to a pattern/color key generator <NUM> that changes the color of certain pixels within each frame of the video to a particular color or a particular pattern of colors, to produce a "key" that is later used as a condition by mixers. Certain details regarding the creation and use of such a key may be found in <CIT> The pattern/color key generator <NUM> may also, in some implementations, provide certain video timing data.

The first multi-window video processing output module may take a first video stream from a video source device <NUM>, <NUM>, <NUM>, or <NUM> that has been received by the matrix switcher/controller <NUM> and switched through video switch <NUM>. The first video stream may be subject to a first scalar <NUM>, and a video tile created therefrom stored in a first frame buffer <NUM>. The locally stored video tile may be passed from the first frame buffer <NUM> to a first mixer <NUM>, per the video timing data received, for example, from the pattern/color key generator <NUM>. The first mixer <NUM> may include, among other things, pattern/color keying logic and blending logic. The pattern/color keying logic of the first mixer <NUM> looks for the key, for example, the pixels of the particular color or having the particular pattern of colors, within frames of incoming video from the pattern/color key generator <NUM>, and determines whether pixels from the incoming video or those of the local video tile should be passed, for example, passing the local video tile where the key is present. In one implementation, the pattern/color keying logic of the first mixer <NUM> may look for the key within a programmable mixing region, e.g., a rectangular region having a size and offset within the frame. When the tile is to be resized, the local video tile may be passed within the mixing region regardless of the presence of a key, as the scalar <NUM> scales the video tile to the new size and the programmable mixing region expands or contracts accordingly. The blending logic of the first mixer <NUM> combines the pixels of the incoming video with those of the local video tile, as directed by the pattern/color keying logic of the first mixer <NUM> according to the key, or simply the programmable mixing region, as the case may be. The first multi-window video processing output module may then output a video stream, including the first video tile and graphics, on a first output port of the first multi-window video processing output module.

The first output port may be coupled (e.g., daisy chained) to a second input port of the first multi-window video processing output module, so that the video stream is fed back to the first multi-window video processing output module. The first multi-window video processing output module may take a second video stream from a video source device <NUM>, <NUM>, <NUM>, or <NUM>. The second video stream may be subject to a second scalar <NUM>, and a video tile created therefrom stored in a second frame buffer <NUM>. The locally stored video tile may be passed from the second frame buffer <NUM> to a second mixer <NUM>, per the video timing data received, for example, from the pattern/color key generator <NUM>. Pattern/color keying logic of the second mixer <NUM> looks for the key, within frames of incoming video coming in on the second input port, and determines whether pixels from the incoming video or those of the local video tile should be passed. Alternatively, if the tile is being resized, the local video tile may be passed within a mixing region regardless of the presence of a key. The blending logic of the second mixer <NUM> combines the pixels of the incoming video with those of the local video tile, as directed by the pattern/color keying logic of the second mixer <NUM> according to the key, or simply the programmable mixing region, as the case may be. In this manner, the second video tile may be combined with the video stream including the first video tile and the graphics, to produce a video stream having two video tiles. This video stream is output on a second output port of the first multi-window video processing output module. The second output port may be coupled (daisy chained) to an input port of a second multi-window video processing output module, and the technique repeated to add a third video tile, using a third scalar <NUM>, a third frame buffer <NUM>, and a third mixer <NUM>, and again repeated to add a fourth video tile using a fourth scalar <NUM>, a fourth frame buffer <NUM>, and a fourth mixer <NUM>, and again repeated. etc., to build a tiled video output stream having a desired number of video tiles. Eventually, an output port may be coupled to a display device <NUM>, and the tiled video output stream provided to the display device <NUM> for display to a user. Accompanying audio may also be provided via the output port.

<FIG> is an exterior view of an example display device <NUM>, showing a tiled video display, produced from a tiled video output stream. In this example, the tiled video display includes six video tiles <NUM>-<NUM> on a screen <NUM>, each showing video content of a different video stream, arranged according to what may be referred to as a "<NUM> /Large Upper Left" tiling layout. However, it should be understood that a different number of video tiles may be shown, arranged according to a variety of different tiling layouts. The video streams whose content is shown in the video tiles <NUM>-<NUM> may be scaled by the multi-window video processing output modules <NUM>, while aspect ratios are maintained. Also, a frame or border <NUM> having programmable properties (e.g., presence, color, witch, etc.) may be rendered, and shown around each of the tiles, and the entire collection of tiles, to provide visual definition of their boundaries.

<FIG> is series of diagrams of example preset tiling layouts <NUM> that may be supported by the video tiling system. The previously mentioned"<NUM> /Large Upper Left" tiling layout may be an example of one of these supported tiling layouts. These preset tiling layouts, as well as custom tiling layouts, may be stored as one or more editable files, for example, extensible markup language (XML) files. It should be understood that a variety of other preset tiling layouts, and user-defined tiling layouts, may be supported. The user-defined tiling layouts may be defined in XML by the user to meet particular applications and user-specific needs.

The tiled video display is controlled from a user interface shown on a touch-sensitive screen a wireless mobile device <NUM>. <FIG> is a block diagram of an example wireless mobile device <NUM>. The mobile device <NUM> may include a processor <NUM>, a memory <NUM>, a wireless network interface <NUM>, and a touch-sensitive screen <NUM>, among other components. The processor <NUM> includes logic configured to execute software and manipulate data from data structures. The memory <NUM> comprises a plurality of storage locations for storing the software and the data structures. The wireless network interface <NUM> may communicate with other devices, for example, the host controller <NUM>. The touch-sensitive screen <NUM> receives gestures (e.g., multi-touch gestures) from a user.

A general purpose operating system <NUM>, portions of which are resident in memory <NUM>, may functionally organize the wireless mobile device <NUM>. The general-purpose operating system <NUM> may be an IOS® operating system available from Apple, Inc. , or another type of operating system. A control application (app) <NUM> may be executed in conjunction with the operating system <NUM>. The control app <NUM> may display a user interface (UI) on the touch sensitive screen <NUM>, upon which gestures may be received to control the video tiling system. In response to input received in the UI, the control app <NUM> may communicate with the host controller <NUM>, which may in turn pass along information to the management process <NUM> executing on the matrix switcher/controller <NUM>.

<FIG> is an exterior view of an example wireless mobile device <NUM> showing an example UI. The UI includes a plurality of UI tiles <NUM>-<NUM> arrange arranged on a virtual display screen <NUM>. UI tiles <NUM>-<NUM> are each a graphical representation of a corresponding video tile <NUM>-<NUM>, and are arranged in a tiling layout that corresponds to the arrangement of the video tiles <NUM>-<NUM> on the screen <NUM> of the display device <NUM>. The virtual display screen <NUM> is a graphical representation of the screen space of the screen <NUM> of the display device <NUM>. The UI tiles <NUM>-<NUM> may include static images, or in some implementations, may show the same video content being shown in the corresponding video tiles <NUM>-<NUM> on the display device <NUM>.

A respective video source device that provides the content of each video tile <NUM>-<NUM> may be indicated within the respective UI tile <NUM>-<NUM>, for example, via text labels. Further, a particular one of the video tiles <NUM>-<NUM> for which related audio is to be played on an audio output device is indicated by a sound icon <NUM>. Audio volume may be adjusted by a volume control <NUM>. A layouts menu may be accessed by a layouts icon <NUM>. Other functionality may be accessed by other icons and interface elements.

Using gestures on the touch sensitive screen <NUM> of the wireless mobile device <NUM>, a user resizes and/or rearrange the UI tiles <NUM>-<NUM> on the virtual display screen <NUM>. For example, a user may use gestures to expand a particular UI tile, so that it encompasses a greater portion, or all, of the virtual display screen <NUM>, or contract a particular UI tile, so that it encompasses a smaller portion of the virtual display screen <NUM>. Similarly, a user may use gestures to swap a particular UI tile with another UI tile, so their respective positions are exchanged. In response to resizing or rearranging UI tiles, the host controller <NUM> may cause the matrix switcher/controller <NUM>, and its multi-window video processing output modules <NUM>, to change the tiled video output stream, such that the video tiles shown on the display device <NUM> are resized and/or rearranged in a corresponding manner.

Resizing effectively transitions between different preset tiling layouts, according to a defined expansion or contraction progression. <FIG> is a diagram of an example expansion progression and an example contraction progression. Beginning with a "2x3" tiling layout <NUM>, there are a plurality of tiles <NUM>-<NUM>, <NUM>-<NUM> (representing both video tiles and corresponding UI tiles). If a first tile <NUM> is expanded, it may be transitioned to be the primary tile of a "<NUM> Left <NUM> Right" tiling layout, while nearby tiles <NUM>, <NUM> become the secondary tiles, according to a defined progression. If the first tile <NUM> is expanded again, it may be transitioned to be the single tile of a "Fullscreen" tiling layout, according to the defined progression. Similarly, if a second tile <NUM> is being shown in a "Fullscreen" tiling layout and it is contracted, it may be transitioned to be the primary tile of a "<NUM> Left <NUM> Right" tiling layout, while other tiles <NUM>, <NUM> become the secondary tiles, according to a defined progression. Likewise, if the second tile <NUM> is contracted again, is may be shown as part of the "2x3" tiling layout <NUM>, along with additional tiles <NUM>, <NUM>, and <NUM>. During resizing, associations between touch locations and the UI tiles may be constantly updated so that the UI correctly understands user input. Further, any occlusions created by foreground tiles upon background tiles may be dynamically accounted for.

The gestures entered on the touch sensitive display screen <NUM> of the wireless mobile device <NUM> to resize the UI tiles <NUM>-<NUM> may include multi-touch gestures. <FIG> is a diagram <NUM> illustrating example expand and pinch multi-touch gestures, to expand and contract UI tiles, and corresponding video tiles. At frame <NUM>, a user enters an expand gesture upon the touch sensitive screen <NUM> over a particular UI tile <NUM>, by touching two points <NUM>, <NUM> and moving apart. At frame <NUM>, as the particular UI tile <NUM> expands, a visual indicator <NUM>, for example, a red outline, may signal a "next size up" available for the tile according to an expansion progression. At frame <NUM>, the user releases from touching the touch sensitive screen <NUM>, and the particular UI tile <NUM> (and corresponding video tile) is set to an appropriate size according to the expansion progression.

At frame <NUM>, a user enters a pinch gesture upon the touch sensitive screen <NUM> over the particular UI tile, by touching two points <NUM>, <NUM> and moving together. At frame <NUM>, as the particular UI tile <NUM> contracts, a visual indicator <NUM>, for example, a red outline, may signal available area to place the particular UI tile <NUM> according to a contraction progression. At frame <NUM>, the user releases from touching the touch sensitive screen <NUM>, and the particular UI tile <NUM> (and corresponding video tile) is set to an appropriate size according to the contraction progression.

Similar to resizing, rearranging may use gestures entered on the touch sensitive screen <NUM> of the wireless mobile device <NUM>. <FIG> is a diagram <NUM> illustrating example drag and drop gestures to rearrange UI tiles, and corresponding video tiles. At frame <NUM>, a user touches over a particular UI tile <NUM> on the touch sensitive screen <NUM>, and begins to drag across the screen. The UI tile <NUM> may be decreased in size and shown as a representation <NUM> that moves with the location <NUM> of the user's touch. When the representation <NUM> is moved over another one of the UI tiles <NUM>, a visual indicator <NUM>, for example, a red outline, may indicate the particular UI tile can be swapped with that UI tile <NUM>. At frame <NUM>, when the user releases from touching the touch sensitive screen <NUM>, the UI tiles <NUM>, <NUM> (and corresponding video tiles) are swapped.

Further, in addition to rearranging and resizing, gestures entered on the touch sensitive screen <NUM> of the wireless mobile device <NUM> is used to change the particular one of the video tiles <NUM>-<NUM> for which related audio is played on an audio output device. Using a drag and drop gesture, the user may select the sound icon <NUM> and drag it from one UI tile to another UI tile. In response, the host controller <NUM> may cause the matrix switcher/controller <NUM> and its audio switch <NUM> to direct audio for the corresponding video tile to an audio output device.

Still further, gestures in conjunction with menus may be used on the sensitive screen <NUM> of the wireless mobile device <NUM> to configure and change properties of video tiles, and video source devices. In response to a touch and hold over a particular UI tile on the touch sensitive screen <NUM> of the wireless mobile device <NUM>, a control options menu may be shown in the UI. <FIG> depicts an example control options menu <NUM> that may be shown in the UI. In region <NUM>, a user may select a particular video source device (e.g., a disc player <NUM>, a cable television box <NUM>, a satellite television receiver <NUM>, a digital media receiver <NUM>, a surveillance camera, a video conferencing system, a computer system, a gaming system, etc.) for the video tile corresponding to the particular UI tile. A particular video stream (e.g., a channel) offering particular video content from that video source device may be selected via additional controls (not shown). In region <NUM>, the user may select whether the corresponding video tile is the one whose related audio is to be played on an audio output device. Further, in region <NUM>, the video source device that provides the video stream may be controlled, or powered off.

A user may save a current tile arrangement and configuration as a custom tiling layout. <FIG> depicts an example layouts menu <NUM> that may be shown in the UI. The layouts menu <NUM> may be shown in response to selection of the layouts icon <NUM>. By selecting an interface element (e.g., a button) <NUM>, the current video tiling layout may be saved as a custom tiling layout. Upon saving, the custom tiling layout may be displayed in a region <NUM>, for later rapid selection. Preset tiling layouts also may be displayed for selection, in another region <NUM>.

<FIG> is a flow diagram of an example sequence of steps <NUM> for browsing video content of multiple video streams using gestures on the touch sensitive screen <NUM> of a wireless mobile device <NUM>. At step <NUM>, a control app <NUM> executing on the wireless mobile device <NUM>, working in conjunction with the host controller <NUM>, displays a UI that includes a plurality of UI tiles <NUM>-<NUM> arranged on virtual display screen <NUM>, each UI tile being a graphic representation of a corresponding video tile shown on the display device <NUM>. At step <NUM>, a gesture (e.g., a multi touch gesture) is detected on the touch sensitive screen <NUM> over a particular one of the UI tiles. At step <NUM>, a type of the gesture is determined.

If the gesture is of a first type (e.g., an expand or pinch multi-touch gesture), at step <NUM>, the control app <NUM> expands the particular UI tile, so that it encompasses a greater portion or all of the virtual display screen <NUM>, or contract the particular UI tile so that it encompasses a smaller portion of the virtual display screen <NUM>. Further, at step <NUM>, the host controller <NUM> causes the matrix switcher/controller <NUM> and its multi-window video processing output modules <NUM> to change the tiled video output stream, such that a corresponding video tile is expanded or contracted in a corresponding manner.

If the gesture is of a second type (e.g., a drag and drop gesture), at step <NUM>, the control app <NUM> may move a representation of the particular UI over another UI tile in response to the gesture, and upon release swap the two UI tiles. Further, at step <NUM>, the host controller <NUM> may cause the matrix switcher/controller <NUM> and it multi-window video processing output modules <NUM> to change the tiled video output stream, such that corresponding video tiles are swapped in a corresponding manner.

Further, if the gesture is of a third type (e.g., a drag and drop over a sound icon <NUM>), at step <NUM>, the control app <NUM> moves the sound icon from the particular UI tile to another UI tile in response to the gesture. At step <NUM>, the host controller <NUM> causes the matrix switcher/controller <NUM>, and it audio switch <NUM>, to direct audio for the corresponding video tile to an audio output device.

Further, if the gesture is of a fourth type (e.g., a touch and hold), at step <NUM>, the control app <NUM> may display a control options menu <NUM> in the UI, which may be used to configure and change properties of the corresponding video tile, and the video source device that provides the video stream for that video tile. At step <NUM>, the host controller <NUM> may cause the matrix switcher/controller <NUM> to change properties as indicated in the control options menu <NUM>.

The video tiling system discussed above may provide for the display of an effectively unlimited number of video streams through a recursive tiling technique. Rather than show a single video stream from a particular video source device, one or more of the video tiles may be configured to show a tiled video output stream, such that multiple nested video tiles are shown within the confines of the video tile.

<FIG> is a diagram of an example tiled video display including nested video tiles, created through a recursive tiling technique. In this example, the video tiling system uses six video tiles at each level. However, it should be understood that different numbers of video tiles may be used in different embodiments. Of the six primary video tiles <NUM>-<NUM>, some of the tiles <NUM>, <NUM>, <NUM>, <NUM> may show video content of a single video stream. Other tiles <NUM>, <NUM> may include a plurality of nested video tiles. The nesting may continue for a user-selectable number of levels. For example, a first video tile <NUM> may include one level of nesting, including within its confines six first level nested video tiles <NUM>-<NUM>, each of which shows video content of a single respective video stream. Similarly, a second video tile <NUM> may include two levels of nesting. The second video tile <NUM> may include within its confines six first level nested video tiles <NUM>-<NUM>, five of which <NUM>-<NUM> show video content of a single respective video stream. The sixth first level nested video tile <NUM> may include within its confines six further nested video tiles <NUM>-<NUM>, each of which shows video content of a single respective video stream. It should be understood that this pattern may be repeated, with additional levels of nesting, for a configurable number of levels. Further, it should be understood that some or all of the video tiles at any particular level of nesting may include nested video tiles.

To provide for nested video tiles, tiled video output streams may be fed back through the video switch <NUM> to the multi-window video processing output modules <NUM>, and used to construct further tiled video output streams. <FIG> is a flow diagram of an example sequence of steps <NUM> for creating a tiled video display including nested video tiles, through a recursive tiling technique. At step <NUM>, a first tiled video output stream is produced by one of the multi-window video processing output modules <NUM>, using the techniques discussed above. At step <NUM>, the multi-window video processing output module feeds the first tiled video output stream back through the video switch <NUM>. At step <NUM>, the video switch <NUM> switches the first tiled video output stream to one of the multi-window video processing output modules <NUM>. At step <NUM>, the first tiled video output stream is used in place of a video stream from a video source device by a receiving multi-window video processing output module, and a second tiled video output stream is eventually constructed. In the second tiled video output stream, the video tiles of the first tiled video output stream are shown as nested video tiles. At step <NUM>, a decision is made whether there is to be an additional level of nesting. If not, execution may proceed to step <NUM>, where the second tiled video output stream is provided as a final video output stream to be shown on a display device <NUM>. If not, execution proceeds back to steps <NUM>-<NUM>, where the second tiled video output stream is fed back again (now being used as the first tiled video output stream), and a new second video output stream is produced therefrom, having another level of nesting. The process may be repeated to eventually to produce the final video output stream that shows the desired number of video streams.

The user may use the UI shown on the touch sensitive screen <NUM> the wireless mobile device <NUM> to navigate through the layers of nesting. In addition to there being UI tiles that correspond to each video tile, there may be nested UI tiles that correspond to each nested video tile. Using gestures (e.g., multi-touch gestures) on the touch sensitive display <NUM> of the wireless mobile device <NUM>, a user may expand a UI tile having nested UI tiles to encompasses some, or all, of the virtual display screen <NUM>. This may cause a corresponding change of the video tiles on the screen of the display device <NUM>. The user may then, using similar gestures, expand one of the nested UI tiles, again causing corresponding video tile changes, to proceed to a lower level. Alternatively, the user may contract nested UI tiles, and corresponding nested video tiles, to progress up to a higher level. In this manner, the user may navigate between the video content of a large number of video streams.

For example, while is discussed above that a tiled video display may be shown upon a single display device, in alternative embodiments the tiled video display may be shown on a plurality of display devices. In some implementations, the tiled video display may be shown in its entirety on each display device of the plurality. In other implementations, portions of the tiled video display may be shown on each display device of the plurality, which may be arranged side-by-side as a video wall. In such an implementation, a plurality of different video output streams may be generated, each corresponding to a respective display device, and distributed appropriately.

Claim 1:
A method comprising:
displaying a tiled video display on a screen (<NUM>) of a display device (<NUM>), the tiled video display including a plurality of video tiles that each show video content of a different video stream of a plurality of video streams, the video tiles arranged on the screen according to a tiling layout;
displaying a user interface, UI, on a touch sensitive screen (<NUM>) of a wireless mobile device (<NUM>) separate from the display device, the UI showing a virtual display screen including a plurality of UI tiles that each corresponds to a respective video tile of the tiled video display, the UI tiles arranged on the virtual display screen according to the tiling layout of the video tiles, the UI tiles each including a static image or video content corresponding to the video content being shown in the respective video tile of the tiled video display, a particular one of the plurality of UI tiles having a sound icon disposed thereon;
playing audio associated with the video tile to which the particular UI tile corresponds on an audio output device;
detecting a gesture over at least a portion of the particular UI tile on the touch sensitive screen (<NUM>);
in response to the gesture, modifying both the particular UI tile on the virtual display screen and a corresponding video tile on the screen (<NUM>) of the separate display device (<NUM>), wherein the modifying includes resizing the particular UI tile and the corresponding video tile, wherein the resizing of the particular UI tile and the corresponding video tile transitions the tiling layout to a different tiling layout that includes a different number of tiles, the transition being according to a predefined expansion or contraction progression that defines a series of transitions between tiling layouts;
detecting a further gesture on the touch sensitive screen (<NUM>);
in response to the further gesture, moving the sound icon from the particular UI to another UI tile on the virtual display screen; and
playing audio associated with a video tile corresponding to the another UI tile on the audio output device.