Video throttling based on individual client delay

A communication interface is configured to send video frames of a video sequence for delivery to a plurality of clients, wherein the video frames include key frames, and to receive feedback from a delayed client of the plurality of clients indicating that the delayed client requires a slower transmission rate for the video frames. A processing system is configured to, in response to the feedback from the delayed client, direct the communication interface to send subsequent video frames to the plurality of clients except to the delayed client, determine whether the delayed client can receive a next full key frame, and if the delayed client can receive the next full key frame, direct the communication interface to resume transmission of the video sequence to the delayed client starting with the next full key frame in the subsequent video frames that are presently being sent to the other clients.

TECHNICAL BACKGROUND

In the field of video analysis, software is developed and employed to process video feeds provided by video cameras or similar imaging devices in order to generate information that is useful for any of a variety of applications, such as inventory tracking, traffic analysis, and queue control. The video feeds may need to be accessed by various parties or clients in various areas or regions. Typically, a public or private communication network is used to provide the video feed to the various parties. Unfortunately, communications networks often use transmission protocols that may be undesirable for transmission of video feeds.

Overview

A method of throttling a video sequence is disclosed herein. The method comprises sending video frames of the video sequence for delivery to a plurality of clients, wherein the video frames include key frames. The method further comprises receiving feedback from a delayed client of the plurality of clients indicating that the delayed client requires a slower transmission rate for the video frames. The method further comprises, in response to the feedback from the delayed client, sending subsequent video frames to the plurality of clients except to the delayed client. The method further comprise determining whether the delayed client can receive a next full key frame. The method further comprises, if the delayed client can receive the next full key frame, resuming transmission of the video sequence to the delayed client starting with the next full key frame in the subsequent video frames that are presently being sent to the other clients.

In an embodiment, a computer-readable medium has stored thereon program instructions for throttling a video sequence, wherein the program instructions, when executed by a video source system, direct the video source system to send video frames of the video sequence for delivery to a plurality of clients, wherein the video frames include key frames. The program instructions further direct the video source system to receive feedback from a delayed client of the plurality of clients indicating that the delayed client requires a slower transmission rate for the video frames. The program instructions further direct the video source system to, in response to the feedback from the delayed client, send subsequent video frames to the plurality of clients except to the delayed client. The program instructions further direct the video source system to determine whether the delayed client can receive a next full key frame. The program instructions further direct the video source system to, if the delayed client can receive the next full key frame, resume transmission of the video sequence to the delayed client starting with the next full key frame in the subsequent video frames that are presently being sent to the other clients.

In an embodiment, a video source system comprises a communication interface and a processing system. The communication interface is configured to send video frames of the video sequence for delivery to a plurality of clients, wherein the video frames include key frames, and to receive feedback from a delayed client of the plurality of clients indicating that the delayed client requires a slower transmission rate for the video frames. The processing system is configured to, in response to the feedback from the delayed client, direct the communication interface to send subsequent video frames to the plurality of clients except to the delayed client. The processing system is further configured to determine whether the delayed client can receive a next full key frame, and if the delayed client can receive the next full key frame, direct the communication interface to resume transmission of the video sequence to the delayed client starting with the next full key frame in the subsequent video frames that are presently being sent to the other clients.

In an embodiment, sending the video frames of the video sequence for delivery to the plurality of clients comprises sending the video frames of the video sequence simultaneously for delivery to the plurality of clients.

In an embodiment, the feedback from the delayed client indicating that the delayed client requires the slower transmission rate for the video frames comprises an indication that a receive buffer in the delayed client has exceeded a capacity threshold.

In an embodiment, the feedback from the delayed client indicating that the delayed client requires the slower transmission rate for the video frames comprises an indication that the delayed client cannot process the video frames at a current transmission rate.

In an embodiment, sending the subsequent video frames to the plurality of clients except to the delayed client comprises skipping transmission of the subsequent video frames of the video sequence to the delayed client while continuing transmission of the subsequent video frames to the other clients.

In an embodiment, the feedback from the delayed client is derived from protocol information.

In an embodiment, the protocol information comprises messages received from the delayed client in a transmission control protocol.

DETAILED DESCRIPTION

The following description and associated drawings teach the best mode of the invention. For the purpose of teaching inventive principles, some conventional aspects of the best mode may be simplified or omitted. The following claims specify the scope of the invention. Some aspects of the best mode may not fall within the scope of the invention as specified by the claims. Thus, those skilled in the art will appreciate variations from the best mode that fall within the scope of the invention. Those skilled in the art will appreciate that the features described below can be combined in various ways to form multiple variations of the invention. As a result, the invention is not limited to the specific examples described below, but only by claims and their equivalents.

Certain transmission protocols are widely used in communication networks that provide video feeds or media streams to multiple clients. However, in most cases, these protocols perform flow control and retransmissions. Flow control and retransmissions are unsuitable for delivery of “live” real-time video and audio to multiple clients simultaneously because when one client of a media stream is too slow or busy, an encoder or data producer has to either block or buffer the media stream. Blocking the media stream affects all of the clients of the same data source, while buffering the media stream may be unfeasible because of limited resource or unsuitable because of the transmission delay.

Advantageously, a system and method for throttling a video sequence or media stream to an individual client is provided. In some examples, the protocol flow control is used to detect slow clients. When a slow client is detected, the slow client receives gracefully degraded media content. For example, transmission to this client may be skipped or paused until its transmission buffer can receive a complete key frame.

FIG. 1illustrates video system100for throttling a video sequence or media stream to an individual client of a plurality of clients. Video system100includes video source102, video source system110, communication network150, client system160, client system170, and client system180. Video source102is coupled to video source system110. Video source system110is coupled to communication network150. Communication network150is coupled to client systems160,170, and180. One video source is shown for simplicity; those skilled in the art will appreciate that video system100may include numerous video sources.

The connections between video source102and video source system110, video source system110and communication network150, and communication network150and client systems160,170, and180may use various communication media, such as air, metal, optical fiber, or some other signal propagation path—including combinations thereof. The connections may be direct links, or they might include various intermediate components, systems, and networks.

In operation, video source system110receives the video feed or media stream from video source102. Those skilled in the art will appreciate that, in some cases, multiple video sources may provide video source system110with multiple video feeds simultaneously. Those skilled in art will also appreciate that while shown separately, video source system110may comprise one or more video sources.

Video source system110receives the video feed comprising video frames and sends the video frames to a plurality of clients over communication network150. Communication network150may include intermediate networks, systems, or devices. Once the video feed is received, video source system110processes the video frames and provides the video frames to a plurality of clients over communication network150. Client systems160,170, and180may comprise a server or any computer system capable of receiving the video frames. Client systems160,170, and180may also display the video feed for one or more operators.

FIG. 2illustrates an operation200of operating video system100for throttling a video feed transmitted to one or more of a plurality of client devices. To begin, video source102provides video source system110with a digital video feed comprising video frames. Video source system110receives the video feed comprising video frames and sends the video frames including key frames to a plurality of clients (202). For example, video source system110may send video frames to client systems160,170, and180. In some examples, video source system110may perform operations on or otherwise process the video feed. For example, video source system110may compress the video feed, merge several feeds together, and/or throttle one or more of the video feeds. Those skilled in the art will appreciate that, in some examples, video source102can provide video source system110with a compressed video feed. Data compression is particularly useful in communications because it enables video source system110to transmit or store the same media content using fewer bits.

In a video feed or media stream, individual video frames of pictures are grouped together (called a group of pictures, or GOP) and played back so that the viewer registers the video's spatial motion. In some instances, these video frames may be compressed using standards such as, for example, the Moving Picture Experts Group (MPEG) standard. The MPEG standard includes various types of frames including predicted frames (P-frames) and intraframes (I-frames).

I-frames, also called key frames, are single frames of digital content that the compressor examines independent of the frames that precede and follow it. Each I-frame includes all of the data needed to display that frame. P-frames follow I-frames and contain only data that has changed from the preceding or forthcoming I-frame (such as color or content changes). Typically, I-frames are interspersed with P-frames (including bi-directional predictive frames or B-frames) in a compressed video. I-frames comprise more bits than P-frames, and thus take up more space on storage mediums such as transmit and/or receive buffers. In most cases, the video quality typically improves as the number of I-frames in the video stream increases. However, those skilled in the art will appreciate that additional I-frames may not result in better video quality, despite the greater number of bits comprising an I-frame relative to a P-frame.

Client systems160,170, and180can include receive buffers to store received video frames. Client systems160,170, and180can provide feedback to video source system110indicating that the client requires a slower transmission rate for the video frames. Thus, video source system110could receive feedback from a delayed client indicating that the delayed client requires a slower transmission rate for the video frames (204). For example, client system170and client system160may send feedback to video source system110indicating that they are delayed in receiving the video feed from video source102. The delay may be caused because, for example, a receive buffer in the client system is at or near capacity. For example, the feedback from the delayed client could indicate that the delayed client requires a slower transmission rate for the video frames by indicating that a receive buffer in the delayed client has exceeded a capacity threshold. Alternatively or additionally, the client system may not be processing the received frames fast enough. For example, the feedback from the delayed client could indicate that the delayed client requires a slower transmission rate for the video frames by indicating that the delayed client cannot process the video frames at a current transmission rate. Those skilled in the art will appreciate that messages sent by the client may indicate the client's ability to process frames rather than its inability to process frames.

In some examples, client systems160,170, and180may not provide direct feedback to video system110. Instead, the feedback may be derived from protocol information. For example, if communication network150operates using the Transmission Control Protocol (TCP) then video source system110may derive the feedback information based on the TCP protocol. However, as discussed, in some cases TCP may be unsuitable for delivery of “live” real-time video and audio to multiple clients because it performs flow control and retransmissions.

TCP is one of the two core protocols for transferring information over the Internet, complementing Internet Protocol (IP), and thus the entire suite is commonly referred to as TCP/IP. TCP provides the service of exchanging data directly between two network hosts. IP provides addressing and routing messages across one or more networks. More specifically, TCP provides reliable, ordered delivery of a stream of bytes from a program on one computer to another program on another computer. TCP is the typical protocol utilized by most Internet applications such as web servers that host websites that are accessible by personal computer systems and other client devices. Other applications which do not require reliable data stream service may use the User Datagram Protocol (UDP), which provides a datagram service that emphasizes reduced latency over reliability.

In some examples, the TCP flow control is used to detect slow clients. When a slow client is detected, transmission to this client is skipped until its transmission buffer can receive a complete key frame. This way, other clients are un-affected by the slow client. The slow client receives gracefully degraded media content.

Video source system110receives feedback from the plurality of clients indicating whether or not they are delayed. In response to the feedback from a delayed client indicating that the delayed client requires a slower transmission rate for the video frames, video source system110continues to send subsequent video frames to the plurality of clients except to the delayed client (206). For example, client system170may provide feedback via the TCP protocol indicating that a receive buffer is at or near capacity. Video source system110then continues to send subsequent video frames to client system180and client system160. Client system160may subsequently provide feedback to video source system110indicating that it too is delayed. In this case, video source system110continues transmitting only to client system180.

Video source system110determines whether the delayed client(s) can receive a next full key frame (208). Video source system110may make this determination based on the feedback received from the plurality of clients. For example, one or more of the client systems may indicate through feedback that it can now accept a full key frame. In this example, client system170was first delayed, followed by client system160. Either client system160or170may indicate that it is capable of resuming reception of the video frames at any time.

If a delayed client is able to receive the next full key frame, video source system110resumes transmission of the video sequence to the delayed client starting with the next full key frame in the subsequent video frames that are presently being sent to the other clients (210). For example, client system160and/or client system170may clear out the contents of their receive buffers. At this point, the client systems may provide feedback indicating their ability to resume the sequence of video frames starting with the next key frame in the sequence. In this manner, video source system110effectively throttles back transmission of the subsequent video frames to the delayed client.

FIG. 3illustrates an example of operation of video system300. In this example, a frame stream is provided to client systems360,370, and380. The frame stream comprises key frames and P-frames. Key frames are designated with a “K” followed by a sequential numeral. Likewise, P-frames are designated by the letter “P” followed by a number in a sequence. Each P-frame and sequential numeral is then followed by a “-” and the associated key frame. Those skilled in the art will appreciate that video system300can include more frame streams and more or fewer client systems.

In this example, the sequence of video frames begins with K1 and continues until the first P-frame after the nthkey frame, n being a variable representing the sequential numeral associated with its respective P-frame. The P-frame is denoted “P1-Kn.” As shown, client systems360,370, and380have already received the first and second key frames and the associated P-frames. Client systems360,370, and380continue to receive video frames from the frame stream until client system370indicates through feedback that it is delayed in receiving the video frames. A video source system receives the feedback and pauses transmission of additional frames to client system370. The video source system continues to monitor the feedback of client system370while providing client systems360and380with video frames from the frame stream.

When the video source system determines that client system is capable of receiving at least a full key frame, the video source system makes a note to resume transmission of the frame stream to client system370at the next key frame in the frame stream sequence. In this case, the frame stream transmission to client system370is resumed at key frame “Kn.”

FIG. 4illustrates an example of video source system400, such as video source system110ofFIG. 1. Video source system400includes communication interface411and processing system401. Processing system401is linked to communication interface411through a bus. Processing system401includes a processor402and memory devices403that store operating software410.

Communication interface411includes a network interface412and input ports413. Communication interface411includes components that communicate over communication links, such as network cards, ports, radio frequency (RF) transceivers, processing circuitry and software, or some other communication devices. Communication interface411may be configured to communicate over metallic, wireless, or optical links. Communication interface411may be configured to use Time Division Multiplexing (TDM), Internet Protocol (IP), Ethernet, optical networking, wireless protocols, communication signaling, or some other communication format—including combinations thereof.

Network interface412is configured to connect to external devices over a communication network. In some examples, these external network devices may include video sources416and video storage systems. Input ports413are configured to connect to video sources416. Input devices, such as a keyboard, mouse, or other user input devices may also be connected to input ports413. If output ports are included, the output ports may be configured to connect to output devices, such as a display, a printer, or other output devices. Those skilled in the art will appreciate that, alternatively or additionally, some or all of video sources416may be connected to video source system400over a communication network via network interface412.

Processor402includes a microprocessor and other circuitry that retrieves and executes operating software from the memory devices403. Memory devices403include random-access memory (RAM)404, read-only memory (ROM)405, hard drive406, and/or any other memory apparatus. Memory devices403comprise non-transitory computer readable storage media, such as disk drives, flash drives, data storage circuitry, or some other hardware memory system. Memory devices403may comprise a single device or could be distributed across multiple devices—including devices in different geographic areas. Memory devices403may be embedded in various types of equipment.

Operating software410includes computer programs, firmware, or some other form of machine-readable processing instructions. In this example, the operating software410includes an operating system407, applications408, and modules and data409. The operating software410may include other software or data as required by any specific embodiment. When executed by the processor402, the operating software410directs processing system401to operate video source system400as described herein.

FIG. 5illustrates an example of video source500, such as video source102ofFIG. 1and video sources416ofFIG. 4. The video source500includes a lens502, a sensor504, a processor506, memory508, and a communication interface510.

Lens502is configured to focus an image of a scene or field of view on sensor504. Lens502may be any type of lens, pinhole, zone plate, or the like that is able to focus an image on sensor504. Sensor504digitally captures video of the scene and passes the resulting video images to processor506. Processor506is configured to store some or all of the video in memory508. In some examples, processor506could be configured to process the video and send the processed video to video source system400ofFIG. 4via communication interface510.

Although shown external to video source system400, video source system400may comprise video source500. Likewise, processing system401of video source system400may comprise some or all of the components and/or functionality described herein for video source500.