Selection and display of adaptive rate streams in video security system

Selection and display of adaptive rate video streams in a video security system in which user devices such as mobile computing devices mix and display multiple streams concurrently from security cameras on the user devices. A client application running on the user devices determines available buffer resources on the user devices, and enables selection of one or more video streams from a grid displayed on a display of the user devices. In response to the determined resources, the client application obtains higher bit rate video streams for the selected video streams and lower bit rate video streams for the non-selected video streams. The client application then displays the higher bit rate streams in visually distinct focus panes that attract the attention of an operator and displays the lower bit rate streams in less visually distinct periphery panes. In a preferred embodiment, operators can select a stream from the periphery panes to display as higher bit rate stream in a focus pane.

BACKGROUND OF THE INVENTION

Manufacturers of security systems provide security devices for public and private institutions, commercial businesses, schools, hospitals and government institutions to list a few examples. The security devices are connected to security networks installed in the customer premises. One of the most important security devices in the security network is the video camera, or security camera. Video data streams from security cameras have increasingly become the focus of security-related activities such as loss-prevention and evidence collection, in addition to surveillance.

Client devices run applications for managing and viewing video data streams from the security networks. The client devices have typically been desktop-based systems or workstations that include high-performance CPUs and provide large memory footprints. The applications on the client devices leverage the large amount of available memory and processing power in order to display and manage multiple full-rate video data streams from one or more security cameras concurrently.

The recent proliferation of inexpensive user devices for displaying video streams over public and cellular data networks has had a profound impact on the way security professionals and first responders consume and share video data from private security networks. In examples, first responders at an accident scene can capture and share video streams in real time with one another on mobile computing devices such as smart phones and police dispatchers can stream surveillance video data of a suspect to mobile computing devices of law enforcement personnel in the field.

Each of the video streams is typically displayed within a separate pane in a grid that displays on the user device. However, the limited processing power, memory, and functionality of user devices impose significant restrictions on the display of video data streams on user devices. In practice, most tablet and smart phone user devices can typically decode and display only one video data stream at a time. This presents a problem for security personnel, who typically require the ability to display multiple video data streams concurrently on the client, such as in a 2×2 grid layout. In addition, the main application that runs on these user devices is a web browser. Web browsers are more limited in their capabilities than the custom applications installed on security workstations for viewing video streams from security cameras.

Current approaches to solving the problem of displaying multiple video data streams from security cameras on client mobile computing devices have focused on server-side optimization. Typically, streaming servers within the security network perform the majority of the mixing and processing of the video data streams from the security cameras.

In a typical example of current approaches, the server decodes the video data streams (“video streams”) from all security cameras, mixes them into a display grid, and then compresses the video streams into a single video data stream. The server then sends the grid including the compressed video data stream to the client mobile computing device. The client device uncompresses and displays the video streams in the display grid. This process repeats for each compressed stream transmission from the server.

SUMMARY OF THE INVENTION

While these approaches greatly reduce the complexity of tasks that the client must perform, they also have problems. One problem is that it increases complexity on the server. Existing server-based solutions require significant processing and memory resources on the streaming server for mixing the video data streams from the security cameras and compressing the streams prior to transmission to the client. This complexity increases cost and can impact time-to-market readiness.

Another problem is that media streaming servers within security networks typically have limited available bandwidth. As the number of security cameras on the security network increase, the streaming server can become a throughput bottleneck in the security network, impacting performance of the security network and the display of the video data streams.

The present invention utilizes concepts of Adaptive Bit Rate (ABR) video streaming, also known as HTTP Adaptive Streaming (HAS) between the user devices and the streaming server to overcome the server-side complexity and bottlenecks for selecting and mixing video streams of existing systems and methods. This present invention can be used to shift the burden for selecting and mixing the video streams with different bit rates to the client user devices.

Because user devices typically have fewer memory and processing resources than server-side devices, the present invention focuses the majority of the available buffer resources on the user device to display a narrow subset of the video streams at higher bit rates. The remaining video streams are displayed at lower bit rates. Preferably, in response to the available buffer resources on the user device, the client application obtains a higher bit rate video stream for a selected video stream on the client application, and obtains lower bit rate video streams for the remaining or non-selected video streams.

The decision to allocate the available buffer resources on user devices for displaying video streams in this manner is based on typical behavior patterns of security system operators. The operators typically focus their gaze on one video stream of a multi-pane display grid at a time, and only periodically glance at the remaining streams in the other panes. Accordingly, the present invention can take advantage of this behavior by displaying the video stream that is the focus of operator interest at a higher bit rate within in a focus pane, and displaying the remaining video streams at lower bit rates in periphery panes. The client application utilizes a HAS-style communication protocol between a streaming server for obtaining different video streams for the same content at different bit rates. This provides the ability to mix and display the video streams from the security cameras using client applications on the user devices without creating processing and resource bottlenecks on the client devices, thereby overcoming the problems and limitations of existing systems and methods. This solution also reduces complexity of the streaming server and places fewer resource requirements on the streaming server than existing solutions and methods.

Embodiments utilize concepts of ABR/HAS video streaming to gauge available data buffer resources on the user devices prior to selecting and displaying the different video streams at different bit rates. The client applications obtain different bit rates for the video streams displayed on the user devices in response to the determined resources on the user devices. The available buffer resources are typically determined in conjunction with current CPU utilization and available bandwidth resources on the user devices.

In one example, the system uses different bandwidth optimization techniques on a streaming server in response to video stream bit rate selection on the client user devices. The techniques utilize a HAS-style communications protocol between the client application and the streaming server.

The streaming server accepts and/or provides multiple video streams for the same content at different or varying bitrates, also known as alternate bit rate streams. The video streams typically include live video data from security cameras. When the security cameras cannot provide the video streams at the bit rates selected by the user devices, the streaming server transcodes the video streams into new video streams with the different bit rates/encoding. The streaming server builds HAS-compatible sub-streams from the different video streams, and provides the sub-streams within one or more HAS streams, also known as HAS multiplexes. The streaming server then provides the HAS multiplex including the sub-streams for the video streams to the user devices for decoding and display.

In general, according to one aspect, the invention features a video security system for selecting and displaying content on user devices and typically mobile computing devices such as tablets and smart phones, which also typically have cellular data connections. The video security system includes one or more security cameras that provide the content in video streams over a security network, and client applications executing on the user devices that display the video streams, and select a bit rate for each of the video streams. In addition, the video security system includes a streaming server that accepts the video streams from the security cameras, and provides the client applications with the video streams having different bit rates determined by the client applications.

Each of the client applications of the video security system display each of the video streams within panes of a grid on a display of the user devices, the panes including one or more periphery panes for displaying lower bit rate video streams and one or more focus panes for displaying higher bit rate video streams. The focus panes include focus pane indicators that enable the focus panes to be visually distinct from the periphery panes.

In response to selection on the client applications of video streams of the periphery panes, the client applications request the streaming server to provide higher bit rate video streams for the selected video streams and lower bit rate video streams for the video streams of the current focus panes, and display the video streams for the current focus panes in the periphery panes and the selected video streams in the focus panes.

The streaming server preferably includes a HAS transcoder, a load balancer, and a media service. The HAS transcoder accepts the video streams from the security cameras, and generates one or more HAS multiplexes that include the video streams as sub-streams of the HAS multiplexes. The load balancer operates in conjunction with the HAS transcoder to transcode the sub-streams into new sub-streams having the bit rates selected by the client applications, and places the new sub-streams within the HAS multiplexes. The media service accepts the HAS multiplexes from the HAS transcoder, generates video segments for the sub-streams of the HAS multiplexes, and provides the HAS multiplexes to each of the client applications.

In general, according to another aspect, the invention features a user device that provides selection and display of video streams from a video security system. The user device comprises a client application and a display. The client application communicates with a streaming server to obtain different video streams at different bit rates. The display displays the video streams from the client application within panes, wherein the client application enables the selection and display of the video streams within the panes.

Preferably, the panes include one or more focus panes for displaying higher bit rate video streams, and one or more periphery panes for displaying lower bit rate video streams. In examples, the focus panes are larger than the periphery panes and include focus pane indicators to enable the focus panes to be visually distinct from the periphery panes.

In a preferred embodiment, the client application, in response to selection of a video stream of any of the periphery panes, requests the streaming server to provide a higher bit rate video stream for the selected video stream and a lower bit rate video stream for the video stream for the current focus pane, and displays the video stream for the current focus pane in a periphery pane and the selected video stream in the focus pane.

In general, according to yet another aspect, the invention features a method for selecting and displaying content on user devices in a video security system. The method comprises providing the content in video streams from one or more security cameras over a security network, displaying the video streams, and selecting a bit rate for each of the video streams on client applications executing on the user devices. The method additionally comprises accepting the video streams from the security cameras and providing the client applications with the video streams having different bit rates determined by the client applications from a streaming server.

In general, according to yet another aspect, the invention features a method for selecting and displaying video streams from a video security system on a user device. The method comprises communicating with a streaming server to obtain different video streams at different bit rates from a client application running on the user device, and displaying the video streams from the client application within panes on a display of the user device, wherein the client application enabling the selecting and displaying of the video streams within the panes.

Preferably, in response to selecting a video stream of any of the periphery panes on the client application, the client application requests the streaming server to provide a higher bit rate video stream for the selected video stream and a lower bit rate video stream for the video stream for the current focus pane, and displays the video stream for the current focus pane in a periphery pane and the selected video stream in the focus pane.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

As used herein, the term “and/or” includes any and all combinations of one or more of the associated listed items. It will be further understood that the terms: includes, comprises, including and/or comprising, when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. Further, it will be understood that when an element, including component or subsystem, is referred to and/or shown as being connected or coupled to another element, it can be directly connected or coupled to the other element or intervening elements may be present.

FIG. 1shows a video security system100that includes security cameras103that transmit video streams310over security network136to user devices106. Other devices that communicate over the security network136typically include a video network recorder130, a security video analytics system132, a security system workstation120, and a streaming server140. These devices communicate over the security network136under the control of a video security control system114. The devices communicate over the security network136via network interfaces134.

User devices106include fixed and mobile computing devices, such as cellular phones, smart phones, and/or tablet devices communicate with the streaming server140over a network cloud110that typically include various segments including enterprise networks, service provider networks, and cellular data networks.

The user devices106each include a display150. Operators interact with the user devices106using user input devices126such as a keyboard, computer mouse, tablet pens, and touch-enabled display screens, in examples.

The streaming server140includes a load balancer142, a HAS transcoder144, and a media service146, in one implementation.

Operators configure the security cameras103and other devices on the security network136via the security system workstation120. Operators utilize user input devices126such as a keyboard, mouse, or pointing device for performing configuration tasks on the security system workstation120. Operators interact with graphical displays such as user interfaces on the system display124for entering the configuration information and displaying results.

Network video recorders130record and save video streams310from the security cameras103. The security video analytics system132performs analysis upon frames of both live and stored/recorded video streams310for selected features of interest and then typical adds metadata to the stored streams.

User devices106issue requests through the network cloud110to connect to the streaming server140. The security cameras103typically provide multiple streams of type H.264/AVC, but can also include MJPEG, MPEG4, or H.264/SVC, in examples.

The HAS transcoder144accepts the different video streams310from the security cameras103. The HAS transcoder144(or encoder) converts the different video streams310from each of the security cameras103into corresponding HAS sub-streams324with different bit rates. The HAS transcoder144then creates a HAS multiplex322associated with the sub-streams322created for each of the security cameras103. Preferably, when the security cameras103cannot provide video streams310at the bit rates requested by the user devices106, the streaming server140transcodes the video streams310into new video streams310having the requested bit rates.

The media service146accepts the HAS multiplex322including the sub-streams324from the HAS transcoder, generates video segments for the sub-streams324, and provides the video segments for the client application162to mix and display on a display150of the user devices106. The client applications162download the video segments for the streams310/sub-streams324, and mix and display the streams310on a display150of the user devices106.

The load balancer142handles providing high-bitrate H.264 and low bitrate MJPEG2000 (MJPEG) streams, in examples. Operators select video streams310on the user devices106to request higher bit rate streams for the same content in response to security objectives.

FIG. 2Ashows an exemplary security camera103that communicates over the security network136via interface134. In one example, the interface134is an Ethernet interface on an IP-enabled security camera103that communicates using IP protocols over the security network136. Each security camera103provides different video streams310at different bit rates for the same content. The video streams310are typically of types H.264/AVC and H.264/SVC, MPEG4, and MJPEG, in examples.

FIG. 2Bshows an exemplary HTTP Adaptive Stream (HAS) multiplex322that includes one or more alternate bit rate sub-streams324. The HAS transcoder144accepts the video streams with different bit rates from each of the security cameras103as input, and preferably generates one HAS stream322or multiplex per security camera103as output. The HAS transcoder144generates sub-streams324for each of the video streams310received from the security cameras103. The HAS transcoder144includes the sub-streams324created for the video streams310of each of the security cameras within an associated HAS multiplex322. Current HAS protocol implementations include Apple HLS, Microsoft Smooth Streaming, and MPEG-DASH, in examples.

Each HAS multiplex322includes the sub-streams324or segments created for each of the alternate bitrate video streams310. The alternate bit rate sub-streams324are typically H.264/AVC, but can be of any type. In one implementation, a HAS multiplex322includes one low bit rate MJPEG sub-stream324, one high bit rate H.264 sub-stream324, and one audio sub-stream324. Preferably, the media service146of the streaming server146provides each HAS multiplex322to the client media application162(client application) on the user devices106.

In examples, depending on available buffer resources on the streaming server140, the HAS transcoder144can additionally create new sub-streams324from the sub-streams324of the HAS multiplex322. The HAS transcoder144creates the new sub-streams324when the bit rates of the current sub-streams324do not provide the bit rates for the video streams requested by the client application. In conjunction with the HAS transcoder144, the load balancer146creates the new sub-streams324by transcoding existing higher bit rate sub-streams324of type H.264 to lower resolution, reduced frame rate, and/or decreased image quality sub-streams324. Alternatively, the HAS transcoder144creates the new sub-streams324by transcoding existing lower bit rate sub-streams324of type MPEG4 or MJPEP to higher resolution, increased frame rate, and/or increased image quality sub-streams324.

FIGS. 3A and 3Bshow different embodiments of a client media player application162, or client application, for displaying and selecting video streams310within panes306of a grid312on a display150of a user device106. InFIG. 3A, the client application162is a stand-alone application, such as a native application or Java application using the Java programming language, in examples. Java is a registered trademark of Oracle, Inc. In a typical example, the client application162communicates with the streaming server140over network cloud110using a proprietary application protocol that sits on top of a TCP/IP transport layer.

The client media player application162accepts one or more HAS streams/HAS multiplexes322from the media service146. The client application162preferably includes a mixer152that accepts and mixes the alternative bit rate sub-streams324of each HAS multiplex322. The mixer152also creates a pane306for each of the video streams308, and a grid312to display each of the sub-streams324within its own pane306of the grid312. Each of the panes306is typically associated with one HAS multiplex322, where each of the panes display and enable selection of the sub-streams324of their associated HAS multiplex322.

FIG. 3Bshows the client application162running within a web browser164. In examples, the client application162and its mixer152and grid312are implemented as part of a Java applet or as a script using Javascript. The web browser164communicates with the streaming server140using the HTTP protocol. Commands from the client application162are encapsulated in HTTP messages.

FIG. 4Ashows one embodiment of the client application162for displaying video streams310within panes306of grid312. The grid312in this example is a 2×2 matrix that at any time includes three lower bitrate video streams310, such as MJPEG streams, labeled stream2, stream3, and stream4, displayed in periphery panes308-1,308-2, and308-3, respectively, and a higher bit rate video stream310displayed within a focus pane402. The grid312has a top edge314, bottom edge316, left edge318, and right edge320. In this example, the focus pane402and the periphery panes308are the same size, and the focus pane402is adjacent to the left edge318and the top edge314of the grid312.

Focus pane402downloads video segments from higher bitrate sub-streams324of the HAS multiplex322associated with the focus pane402. The client application162displays the grid312including the periphery panes308and their video streams310/sub-streams324, and the focus pane(s)402and their video streams310/sub-streams324on the display150of the user devices106.

The focus pane indicator406highlights or otherwise visually sets apart the focus pane402from the periphery panes308. In one implementation, the client application162provides a colored selection rectangle around the focus pane(s)402in order to indicate to the operator which of the panes306include the focus video stream310. When there is more than one focus pane402, the client application162can provide the operator with the ability to prioritize the focus (e.g. most recently selected).

The periphery panes308download video segments from lower bitrate sub-streams324of their associated HAS multiplexes322. This scheme saves bandwidth on the client media player application162.

In one example, the frame rate of the video streams310downloaded to the periphery panes308is 24 fps or lower, which is the minimum frequency required to eliminate the perception of moving frames for the human eye. Setting the video streams of the periphery panes308to use a lower threshold fps such as 24 fps provides more bandwidth for the video stream of the focus pane402. The client application162then requests a higher available bitrate for the stream of the focus pane402, determined by an available buffer and/or bandwidth resources of the user device106.

In a preferred embodiment, all periphery panes308automatically download lower-bit rate MJPEG streams of the same low frame rate, such as 16 frames per second (fps) or less, and the focus panes402download higher-bit rate H.264 streams in response to available buffer resources on the user devices106.

Though the display of the video streams310in the periphery panes308may experience “flicker” or pauses between frame updates at lower frame rates using this scheme, their video streams are of much less importance to the operator than the stream of the focus pane402. This enables concurrent display and mixing of multiple video streams on the client media player application162while providing the higher bitrate of the stream in the focus pane402in response to bandwidth and buffer resources available on the user device162.

FIG. 4Bshows another embodiment of the client application162for displaying its grid312of video streams310. The exemplary grid312is a 3×3 matrix that at any given time includes four low bit rate MJPEG streams310, labeled stream2, stream3, stream5, and stream6. Streams2through5are displayed in periphery panes308-1,308-2, and308-3, and308-4, respectively. In this embodiment, segments for higher bit rate sub-streams324, labeled as stream1and stream4, are downloaded by the client application162and displayed in first focus pane402-1and second focus pane402-2.

As in the exemplary grid312ofFIG. 4A, the focus panes402and the periphery panes308inFIG. 4Bare the same size. Focus panes402-1and402-2are placed adjacent to one another and with respect to the top314of the grid312. Focus pane indicators406-1and406-2associated with focus panes402-1and402-2highlight or otherwise visually set apart the focus panes402from the periphery panes308. This embodiment illustrates the ability to apportion the majority of the available buffer resources on the user device106between two higher bit rate streams310displayed in the first focus pane402-1and the second focus pane402-2, respectively.

FIG. 5Ashows another embodiment of the client application162for its grid312of video streams310. The exemplary grid312shows an oversized focus pane402relative to the size of the periphery panes308for displaying stream7as a higher bit rate stream310. The grid312shows periphery panes308-1through308-6for displaying lower bit rate video stream310stream1through stream6, respectively. The lower bit rate video streams310are typically MJPEG streams. In this example, the oversized focus pane402is centered with respect to the left edge318and the right edge320of the grid312, and the relatively smaller periphery panes308surround the focus pane402at the left edge318and the right edge320of the grid312. In addition, focus pane indicator406highlights or otherwise visually sets apart the focus pane402from the periphery panes308.

Experimentation has shown that lower frame rate, low-resolution video frame images of high quality can be constructed from higher bit rate video data sub-streams324using the HAS transcoder144on the streaming server140with minimal performance impact to the streaming server140. In one example, streaming server140can fabricate a set of MJPEG individual JPEG images or “thumbnails” from a higher bit rate sub-stream324such as an H.264 encoded stream. The client application162successively pulls and displays the MJPEG lower bit rate frames or segments in sequence.

Alternatively, the HAS transcoder144can also fabricate high-resolution sub-streams324from low frame rate sub-streams324such as MJPEG frames with minimal performance impact to the streaming server140.

Also inFIG. 5A, an operator on the client application162selects stream1in periphery pane308-1to display as a higher bitrate stream310in the focus pane402. Reference404indicates the selection.FIG. 5Bshows how the client application162updates the display of the video streams310in response to the selection, within the panes306of the grid312on the user devices106.

InFIG. 5B, stream7, the video stream310previously displayed in focus pane402ofFIG. 5A, is now displayed as a lower bit rate stream in periphery pane308-1. Selected video stream404, stream1fromFIG. 5A, is now displayed in the focus pane402ofFIG. 5Bas a higher bit rate stream310.

FIG. 6is a flow diagram showing a method for selecting and displaying video streams310with different bit rates on user devices106. The method preferably utilizes HTTP Adaptive Streaming (HAS) between the client application162and the streaming server140.

In step602, each security camera103provides different video streams at different bit rates for the same content. Preferably, each security camera103provides one or more streams310of higher bit rate H.264 streams and one or more lower bit rate MJPEG streams/images. Each security camera also provides a list to the streaming server140that includes the types of streams and optional quality/bitrate settings available for its provided streams310. In step604, the streaming server140combines the lists from each security camera103into a master list or “adaptive palette” of available streams310at different bitrates/frame rates from all security cameras103on the security network136.

According to step606, an operator using client application162on a user device106selects a low-bitrate MJPEG stream310in a periphery pane308of the grid312to display as a high bit rate stream in the focus pane402. In step608, the client application sends the selection404to the streaming server140.

In step610, the streaming server140creates a HAS multiplex322that includes different video streams at different bit rates for the selected stream404.

Then, in step614, the client application162communicates with the streaming server using a HAS-style communications protocol. The client application162communicates with the streaming server140to obtain a higher bit rate sub-stream for the selected sub-stream in response to available buffer resources on the user device106.

In step616, the streaming server140provides a higher bit rate H.264 sub-stream for the selected sub-stream and lower bit rate streams/images for the non-selected sub-streams. Finally, in step616, the client application162mixes and updates the streams in the grid312on the display150. The client application162downloads and displays video segments for the higher bit rate H.264 sub-stream in the focus pane402and downloads and displays video segments for the lower bit rate sub-streams324of the non-selected video streams310in the periphery panes308.

Note that the client/server interaction provided by the HAS-style communications protocol between the client application162and the streaming server140of step612differs from the behavior of standard HAS protocols such as HLS, Smooth Streaming, and MPEG-DASH. Typically, these protocols include a streaming server that is ‘dumb’ or ‘unintelligent’ because in these protocols, the client performs all the decision making for which streams310to select and the bit rates to request for the streams310. In contradistinction, the streaming server140is more intelligent, providing the ability for the client application162and the streaming server140to ‘talk’ or ‘negotiate’. In examples, the negotiation between the client application162and the streaming server140can include asking for additional sub-streams324, or indicating to the streaming server what streams it is using. This allows the streaming server140to reduce the number of alternate bitrate streams310to include only those used by each client application162on each user device106.

In step702, the streaming server140accepts the stream selection404from the client application162. The streaming server140in steps704and708checks the stream type of the selected stream404in order to create different sub-streams at different bit rates based on the stream type of the selected stream404. The security cameras103provide higher bit rate video streams of type H.264 such as H.264/AVC and H.264/SVC, and lower bit rate streams/images of types MJPEG and MPEG4, in examples.

In step704, the streaming server140determines if the stream type of the selected stream404is of type MJPEP or MJPEG4. If the result is true, in step706, the HAS transcoder144of the streaming server140transcodes the selected stream into different bit rate sub-streams of type H.264, providing multiple higher bit-rate sub-streams of type H.264 and one lower bit-rate sub-stream of H.264. Upon the conclusion of step706, the method transitions to step716, which creates a HAS Multiplex322and includes the created sub-streams324within the HAS multiplex322.

If the result of step704is not true, the method transitions to step708. In step708, the streaming server140determines if the stream type of the selected stream404is of type H.264. If this is not the case, the method transitions to step710to indicate an error condition associated with an unsupported stream type. Otherwise, the method transitions to step712to assemble the different video streams at different bit rates of type H.264 from their associated security camera103and create sub-streams324for each. Then, in step714, the HAS transcoder144decodes one of the H.264 video streams and then encodes it into a new MJPEG2000 sub-stream in order to provide low fps, low resolution, but high quality frame images for the non-selected video streams. Finally, in step716, the streaming server140creates a HAS Multiplex322and includes the created sub-streams324within the HAS multiplex322.

FIG. 8provides detail for method step612ofFIG. 6. InFIG. 8, an exemplary HAS-style communications protocol provides communication between the client application162and the streaming server140for obtaining a higher-bit rate video stream on the client application162in response to available buffer resources on the user device106.

In step802, the client application downloads a HAS Stream descriptor for a HAS multiplex322that includes the selected stream404to determine the different sub-streams324with different bit rates that are available for the selected stream404. In step804, the client application162downloads one of the segments of the selected sub-stream404/324to gauge the buffering capabilities of the user devices106. In step806, the client application162determines if it can download the segments of the selected sub-stream faster than their playback time. In step808, the client application162buffers segments of the selected sub-stream in order to provide smoother playback.

Then, according to step812, the client application162first determines if frequent rebuffering is occurring by testing for buffer underruns/underflows. This typically occurs when the buffers of the client application162are fed data at a lower speed than their playback speed. If buffer underruns are occurring, the client application162transitions to step810to request a lower bit rate stream for the selected sub-stream404/324to address the rebuffering events. Step810transitions to step804to complete this process in response to the available buffer resources.

If buffer underruns are not occurring, step812transitions to step814to test if the buffers are full or nearing full capacity. When the buffers are full or nearly at full capacity and there are ample available CPU and/or bandwidth resources still available on the user devices106, this allows the client application162in step816to request a higher bit rate stream for the selected sub-stream404. Step814transitions to step816to complete this process in response to the available buffer resources.

When the condition of step814is not met, indicating that a sufficient alternate bit rate sub stream has been determined for the selected stream in response to available buffer resources on the user device, step814transitions to step818to return the alternate bit rate sub-stream.