Patent Description:
Modern video capture devices make use of a variety of different types of technologies for capturing, processing, and transmitting or otherwise transferring captured video data. A video capture device may be designed to capture primarily visible light video, depth-based video, infrared-based video, or a combination thereof. However, in some cases, the technologies used by video capture devices to process and/or transfer the captured video data limit the quality of the resulting video stream and/or negatively affect the user experience of watching the resulting video stream. Device clocks of video capture devices may drift, causing temporal distortion or "jitter" in the video stream, video processing operations may cause the video stream to skew out of sync, and/or a communication bus or other medium may provide less reliable data transfer than necessary to provide a high-quality video stream to users on the other end of the transfer. For example, use of universal serial bus (USB) technology to transfer frames of video stream may present a variety of challenges associated with accurate timing of the frames in the video stream after transfer.

These challenges are compounded when multiple video streams are combined to form three-dimensional video streams, video streams with multiple points of view, or the like. Synchronizing each video stream requires extremely accurate timing of each frame, and each video stream may be captured by different video capture devices, each with a unique set of technologies that must be considered in order for the video streams to be correlated to a shared timeframe. Providing combined video streams for viewing without significant temporal distortion, jitter, or other issues that cause a negative user experience is a challenging task.

<CIT> describes that a system recovers a local media clock from a master media clock based on time-stamped packets received from a transmitter. The packets may include audio, video, or a combination of both, sampled at a rate determined by the master media clock at the transmitter. Timestamps in the packets may be based on values of a remote real-time counter at the transmitter that is synchronized with a local real-time counter at a receiver. The local media clock may be syntonized with the master media clock through the clock periods. The clocks may be synchronized by syntonizing the clocks and adjusting the phase of the local media clocks based on timestamps and a real-time counter.

<CIT> describes that two clocks may be synchronized by calculating skew and offset values that may be determined from several correlation events. A correlation event may be the passing of messages in both directions between the two devices. The skew and offset values may be used to determine the time of non-correlated events. The clock synchronization may be performed on a real time basis or may be performed on a post processing basis. One method for calculating the skew and offset may use inequalities within a solution space to refine a solution set with multiple sets of correlation events.

<CIT> describes computer-based methods and apparatuses, including computer program products, for canonical scheduling for heterogeneous content delivery. A content stream of bits is preprocessed by dividing the content stream of bits into data packets and assigning a timestamp to each data packet. The preprocessed content stream of bits is transmitted upon request from a receiver. A second timestamp based on the assigned timestamp from at least a portion of the respective data packets is established. The data packets are scheduled for transmission based on the second timestamp. The data packets are transmitted based on the schedule.

A computerized method comprises receiving a frame of a video stream from a capture device via a bus interface, the frame including a first start frame timestamp and a first end frame timestamp based on a device clock associated with the capture device. A second start frame timestamp and a second end frame timestamp associated with the received frame are predicted based on a plurality of previously received frames of the video stream. The second start frame timestamp and second end frame timestamp are further based on a system clock associated with a display system. A skew value of the received frame is calculated based on a difference between the first start frame timestamp and the second start frame timestamp, and a difference between the first end frame timestamp and the second end frame timestamp. Upon the calculated skew value exceeding a skew threshold, the received frame is corrected to correlate to the second start frame timestamp and the second end frame timestamp. The corrected frame is then provided for display.

In <FIG>, the systems are illustrated as schematic drawings. The drawings may not be to scale.

The systems and methods described herein are configured to correlate video stream frames received from video capture devices with a system clock of a receiving computing device and to provide the correlated video stream frames for display. The frames, including timestamps based on the device clock of the video capture device, are received by the computing device via a communication bus interface. The computing device predicts the expected timestamps of the received frame based on the system clock of the computing device and on the timestamps of previously received video stream frames. If a difference between the predicted timestamps and the timestamps received with the frame exceeds a defined skew threshold, the timestamps of the frame are corrected to correlate with the predicted timestamps. The correlated frame is then provided for display (e.g., via one or more user applications executing on the computing device).

Correlating and/or synchronizing frames as described herein is effective at eliminating significant differences in the timestamps of frames of one or more video streams, which reduces temporal distortion, smooths "jitter" of the video streams, and generally improves the user experience of viewing the video streams. In some examples, the operations for correlating the frames as described herein is an unconventional use of clocks, processors, memory, and other conventional computing resources. The accurate synchronization of multiple combined video streams provided by the described methods may be used to improve the quality of three-dimensional video streams and/or other types of video streams that rely on the use of multiple video stream inputs. Further, the performance of USB and other similar bus technology as communication media for high-quality live video streams is enhanced through use of the described systems and methods. In this manner, the disclosure improves the functioning of the computing device performing operations as described herein.

<FIG> is an exemplary block diagram illustrating a system <NUM> including a computing device <NUM> and video capture devices <NUM> configured to capture video streams and correlate the frames <NUM> of the video streams according to an embodiment. The computing device <NUM> includes a correlator application <NUM> that is configured to receive the video stream frames <NUM> from the video capture devices <NUM> and correlate the video stream frames <NUM> using the system clock <NUM> as described herein. The video stream frames <NUM> are received via a universal serial bus (USB) interface <NUM>. The correlator application <NUM> provides correlated frames to one or more applications <NUM>, which may use the correlated video stream frames <NUM> in various ways (e.g., display, further processing, image analysis, etc.).

The video capture devices <NUM> each include a device clock <NUM> and a USB interface <NUM>. Captured video stream frames <NUM> are timestamped by the video capture devices <NUM> based on the device clock <NUM> and sent to the computing device <NUM> using the USB interface <NUM>. A USB bus <NUM> connects the USB interface <NUM> to the USB interface <NUM>.

The computing device <NUM> may be a personal computer, a server device, a laptop, a tablet, a mobile phone, a game system, a wearable device, or the like. The computing device <NUM> may include one or more processors, memory, interfaces, and/or other computer components. The correlator application <NUM> may be stored in memory of the computing device <NUM> and configured to correlate the timestamps of frames <NUM> as described herein. In some examples, the correlator application <NUM> includes driver software, firmware, or the like, and may execute at the user level or the kernel level. The system clock <NUM> of the computing device <NUM> is configured to provide accurate measurement of the passage of time and the correlation of the frames <NUM> may be based on time measurements made using the system clock <NUM>.

The USB interface <NUM> may comprise a conventional USB interface as would be understood by a person of ordinary skill in the art. This may include, for instance, at least one physical USB port configured to admit a USB cable, physical connections to other portions of the computing device <NUM>, firmware and/or software configured to enable communication over the USB interface <NUM> using USB protocols, or the like.

In some examples, the computing device <NUM> comprises a plurality of interfaces, including the USB interface <NUM>, enabling the computing device <NUM> to interact with a variety of other entities. For instance, the computing device may include network interfaces beyond the USB interface <NUM> (e.g., wired network interfaces such as Ethernet interfaces, wireless network interfaces such as Wi-Fi interfaces or cellular network interfaces, user interfaces such as displays, touchscreens, speakers, microphones, keyboards, mice, game controllers, etc.). It should be understood that the computing device <NUM> may include any combination of interfaces that enables the computing device <NUM> to perform the operations described herein without departing from the description.

The video capture devices <NUM> may include one or more visible light-based cameras such as red-green-blue (RGB) cameras, depth-based cameras, infrared-based cameras, or other types of cameras. The video capture devices <NUM> may each be configured to collect video data, buffer or store the video data at defined levels of quality (e.g., 720p, 1080p, <NUM>, etc.), and send the video data to the computing device <NUM> using a USB interface <NUM>. The device clock <NUM> of each video capture device <NUM> is configured to provide time measurements that are used by the video capture devices <NUM> to timestamp frames <NUM> of the captured video data as described herein.

Each video capture device <NUM> may be connected to and/or in communication with the computing device <NUM> via a bus <NUM>. In some examples, the bus <NUM> may include wireless communication alone or in combination with wired communication. Each video capture device <NUM> may be connected to the computing device <NUM> via a separate bus <NUM>, multiple video capture devices <NUM> may be connected to the computing device <NUM> via a single bus <NUM>, or a combination thereof. In some examples, the bus <NUM> may be a direct connection between the video capture devices <NUM> and the computing device <NUM>. Alternatively, the bus <NUM> may be an indirect connection that includes one or more hubs, switches, or other network components, etc. without departing from the description herein.

<FIG> is an exemplary block diagram <NUM> illustrating a frame <NUM> being transferred from a video capture device to a computing device according to an embodiment. The frame <NUM> is generated by the video capture device <NUM> based on captured video data. The frame <NUM> includes a header <NUM> storing metadata associated with the frame <NUM> and the frame data <NUM> including the captured video data of the frame. Upon generation of the frame <NUM>, the video capture device <NUM> generates a start frame timestamp <NUM> representing a moment in time when capture of the video data associated with the frame <NUM> began and an end frame timestamp <NUM> representing a moment in time when the capture of the video data associated with the frame <NUM> ended relative to the device clock <NUM>. The start frame timestamp <NUM> and the end frame timestamp <NUM> are included in the header <NUM> of the frame <NUM>. Alternatively, the timestamps <NUM> and <NUM> may be associated with the frame <NUM> in different ways without departing from the description herein (e.g., timestamps <NUM> and <NUM> may be stored separately and mapped to frame <NUM> based on a frame ID, etc.). For example, the applications <NUM> in <FIG> may query the video capture device <NUM> using an application programming interface (API) to obtain the timestamps <NUM> and <NUM>.

In some examples, the frame <NUM> may be broken into multiple frame packets <NUM> to be sent to the computing device <NUM> (e.g., via a USB bus <NUM>, etc.). The number and size of each of the frame packets <NUM> may be determined based on the communication protocol used and the size of the frame <NUM> (e.g., the USB protocol may have a maximum packet size of <NUM> kilobyte (KB) and, if frame <NUM> is ~<NUM> megabytes (MB), the frame <NUM> may be broken into ~<NUM> frame packets <NUM>, etc.). Each frame packet <NUM> includes a header <NUM> for metadata associated with the frame <NUM> and/or the specific frame packet <NUM> and a frame data portion <NUM> of the frame data <NUM> (e.g., a frame identifier, a frame packet identifier, frame sequence information, etc.). In some examples, the header <NUM> of each frame packet <NUM> may include the start frame timestamp <NUM> and end frame timestamp <NUM> of the frame <NUM>.

In some examples, the video capture device <NUM> may do further processing of the frame <NUM> prior to sending it to the computing device. Such processing may include compression, noise reduction, and/or other image processing. Such processing may increase the time required between the capture of the frame <NUM> the transfer of the frame <NUM> to the computing device <NUM>.

The frame packets <NUM>, including a first frame packet <NUM> and a last frame packet <NUM>, are sent serially to the computing device <NUM> (e.g., via USB bus <NUM>, etc.). Upon receiving the first frame packet <NUM>, a system start frame timestamp <NUM> for the frame <NUM> may be generated using time as measured by the system clock <NUM> of the computing device <NUM> (e.g., system timestamps may be generated based on QueryPerformanceCounter (QPC) functionality of the operating system (OS) associated with the computing device <NUM>, etc.). Upon receiving the last frame packet <NUM>, a system end frame timestamp <NUM> may be generated, also using the time as measured by the system clock <NUM> of the computing device <NUM>. The first frame packet <NUM> may be detected based on the packet being the first packet received associated with the frame <NUM> (e.g., packet metadata may include a frame identifier, etc.) and/or packet sequence data indicating that the packet is the first packet. The last frame packet <NUM> may also be detected based on metadata of the packet (e.g., a last frame flag is set, the metadata includes a final packet sequence value and the last frame packet <NUM> includes a sequence value that matches the final packet sequence value, etc.). Alternatively, or additionally, the frame data portion <NUM> of the last frame packet <NUM> may include end-of-frame or end-of-file (EOF) data that is identified by the computing device <NUM>.

The system start frame timestamp <NUM> and system end frame timestamp <NUM> are provided to the correlator application <NUM> of the computing device <NUM>. Other data associated with the frame <NUM> may also be provided to the correlator application <NUM> (e.g., a frame identifier, a video stream identifier, the start frame timestamp <NUM> and the end frame timestamp <NUM> of the frame <NUM>, etc.). The computing device <NUM> and correlator application <NUM> are configured to generate a correlated frame <NUM>, including a header <NUM> and frame data <NUM>, from the received frame packets <NUM> of the frame <NUM>, wherein the correlated frame <NUM> has been correlated to the timing of the system clock <NUM> and other frames that have also been received by the computing device <NUM>. The correlator application <NUM> generates correlated start frame timestamp <NUM> and correlated end frame timestamp <NUM> for inclusion in the header <NUM> of the correlated frame <NUM>. The correlated timestamps <NUM> and <NUM> are based on the start frame timestamp <NUM> and end frame timestamp <NUM> of the original frame <NUM>, the system start frame timestamp <NUM> and system end frame timestamp <NUM> based on the system clock <NUM>, and/or on timestamps associated with other received frames of the same video stream.

The correlator application <NUM> includes a frame buffer <NUM>, predicted timestamps <NUM>, a calculated skew value <NUM> associated with the frame <NUM>, and a defined skew threshold <NUM> that are used in correlating the timestamps <NUM> and <NUM> of the correlated frame <NUM>. In some examples, the frame buffer <NUM> includes previously received frames of the video stream with which the frame <NUM> is associated. In other examples, the frame buffer <NUM> of the correlator application <NUM> includes multiple frame buffers associated with multiple video streams, enabling the correlator application <NUM> to correlate and/or synchronize frames of multiple video streams as described below with respect to <FIG>. The frame buffer <NUM> may be configured as a circular buffer or linked list, such that the buffer retains frame data for a defined number of frames and, when frame data of a new frame is added to the frame buffer <NUM>, the oldest frame data is removed and/or overwritten by the new frame data. The frame data stored in the frame buffer <NUM> may include, for instance, a mapping of start frame timestamps and/or end frame timestamps of the previously received frames of the associated video stream. The start and end timestamps of the previously received frames (e.g., start frame timestamps and end frame timestamps that have previously been correlated to the system clock <NUM> as described herein mapped to associated frame IDs, frame sequence IDs, etc.) enable the correlator application <NUM> to analyze the timestamp data of multiple frames to detect patterns and/or predict timestamps <NUM> of received frames of the video stream (e.g. frame <NUM>, etc.).

The correlator application <NUM> is configured to calculate an average clock differential value for the video stream of the frame <NUM> based on the timestamp data of the frame buffer <NUM>. The differences between the start frame and end frame timestamps of a frame based on the device clock <NUM> and the system start frame and system end frame timestamps of the frame based on the system clock <NUM> are averaged across some or all of the previously received frames in the frame buffer <NUM> (e.g., averaged across the previous few minutes of frames), resulting in an average clock differential value indicating the average time required for a frame to reach the computing device <NUM> after being captured by the video capture device <NUM> and/or average time difference between the device clock <NUM> and the system clock <NUM>.

Additionally, or alternatively, the correlator application <NUM> may be configured to calculate an average framerate and/or frame length based on the frame data of the frame buffer <NUM>. The differences in the correlated start frame timestamps and correlated end frame timestamps of the frames of the frame buffer <NUM> may be combined and then the result is divided by the number of frames in the frame buffer <NUM> (e.g., if the sum of the frame lengths in the frame buffer <NUM> is <NUM> seconds and the frame buffer <NUM> includes <NUM> frames, the average frame length is <NUM> milliseconds (ms), etc.). The calculated average frame length may be used when generating predicted timestamps <NUM> for the frame <NUM> as described herein. Further, the average framerate may be calculated based on a total number of frames received in the frame buffer <NUM> divided by a defined time period (e.g., if the frame buffer includes frame data of <NUM> received frames and the difference between the timestamps of the first frame and the last frame is <NUM> seconds, the average framerate may be calculated as <NUM> frames per second (fps), etc.).

Other values may also be calculated based on the frame data of the frame buffer <NUM> for use in generating predicted timestamps <NUM>. For instance, the correlator application <NUM> may calculate an average time between frames (e.g., the time between the correlated end frame timestamp of a frame and the correlated start frame timestamp of the next frame in sequence, etc.), which can be used to predict a timestamp of a start of an incoming frame based on an end of a recently received frame.

In some examples, the correlator application <NUM> may obtain information from the USB interface and/or bus driver software, such as a current bandwidth of the USB communication and/or timestamps based on when the frame was initially placed onto the bus at the video capture device <NUM>. The USB start frame timestamp may be synchronized to the system clock <NUM>, enabling the correlator application <NUM> to generate predicted timestamps <NUM> with accuracy. A USB end frame timestamp that is similarly synchronized to the system clock <NUM> may also be added to the final packet of a frame and used by the correlator application <NUM> to correlate the frame to the system clock <NUM> and the other frames of the video stream.

The correlator application <NUM> is configured to generate predicted timestamps <NUM> based on the collected and/or calculated values described above. For instance, a predicted start frame timestamp may be generated based on an average clock differential value between the device clock <NUM> and the system clock <NUM> and/or a USB-based start frame timestamp. The predicted start frame timestamp may account for processing time at the video capture device <NUM>, duration or latency associated with the transfer of the frame <NUM> from the video capture device <NUM> and the computing device <NUM>, duration or latency associated with processing of the frame at the video capture device <NUM>, and time differences between the system clock <NUM> and device clock <NUM>, etc. Further, the predicted start frame timestamp may be generated to correlate to the previously received frames in the frame buffer <NUM> based on calculated average framerate, calculated average frame length, and/or calculated average time between frames, etc. (e.g., at an average framerate of <NUM> frames per second, each frame is -<NUM> long and a predicted start frame timestamp may be generated to be <NUM> after the correlated start frame timestamp of the immediately previous frame in the frame buffer <NUM>, etc.). Generating the predicted end frame timestamp of the predicted timestamps <NUM> may be based on the predicted start frame timestamp and a calculated average frame length based on the received frames of the frame buffer <NUM>, such that the predicted timestamps <NUM> are correlated closely with the timestamps of the frames of the frame buffer <NUM>.

Upon calculating the predicted timestamps <NUM>, the correlator application <NUM> is configured to calculate a skew value <NUM> of the received frame <NUM>. The skew value <NUM> is based on the comparison of the predicted timestamps <NUM> to the measured timestamps of the frame <NUM> (e.g., the start frame timestamp <NUM>, end frame timestamp <NUM>, system start frame timestamp <NUM>, system end frame timestamp <NUM>, etc.). In some examples, the predicted timestamps <NUM> are generated to be relative to the capture of the frame <NUM> on the video capture device <NUM> (e.g., relative to the start frame timestamp <NUM> and end frame timestamp <NUM>, etc.). The skew value <NUM> is calculated as the difference between predicted timestamps <NUM> and the measured timestamps of the frame <NUM>. The skew value <NUM> may represent that the measured timestamps differ from the predicted timestamps <NUM> by an amount of time (e.g., the entire frame <NUM> is skewed early or late, etc.) and/or that the time difference between the predicted timestamps <NUM> differs from the time difference between the measured timestamps (e.g., the frame length of the frame <NUM> is longer or shorter than a predicted frame length, etc.). The skew value <NUM> may include one or more values representing the various timestamp differences detected (e.g., differences between the predicted start frame timestamp and the measured start frame timestamp, differences between the predicted end frame timestamp and the measured end frame timestamp, etc.).

The correlator application <NUM> is configured to compare the skew value(s) <NUM> to at least one defined skew threshold <NUM>. In some examples, when the skew value <NUM> exceeds the skew threshold <NUM>, the predicted timestamps <NUM> are used as the correlated start frame timestamp <NUM> and correlated end frame timestamp <NUM> for the correlated frame <NUM>. When the skew value <NUM> does not exceed the skew threshold <NUM>, the measured timestamps of the frame <NUM> may be used for the correlated start frame timestamp <NUM> and correlated end frame timestamp <NUM>.

In some examples, the skew threshold <NUM> includes a skew threshold associated with start frame timestamp differences and a skew threshold associated with end frame timestamp differences. If one of the skew thresholds is exceeded by the associated skew value and the other skew threshold is not exceeded, the correlator application <NUM> may replace the timestamp (e.g., start frame or end frame, etc.) associated with the exceeded skew threshold with the predicted timestamp as the correlated timestamp in the correlated frame <NUM> while the other measured timestamp is maintained as the correlated timestamp in the correlated frame <NUM>. Alternatively, if one or both skew thresholds are exceeded by the associated skew values, the correlator application may replace both measured timestamps of the frame <NUM> with the predicted timestamps <NUM> as the correlated timestamps <NUM> and <NUM> of correlated frame <NUM>.

Alternatively, or additionally, the skew threshold <NUM> may be configured to apply to frame skew that represents the timing of the entire frame <NUM> being skewed early or late, or to frame skew that represents the measured length of the frame <NUM> being longer than the predicted frame length by the skew threshold <NUM>. In the case of the entire frame <NUM> being skewed by more than the skew threshold <NUM>, the correlated timestamps <NUM> and <NUM> may be corrected to match the predicted timestamps <NUM>. Alternatively, in the case of the measured frame length exceeding the predicted frame length by more than the skew threshold <NUM> (e.g., the measure end frame timestamp of the frame <NUM> is later than the predicted end frame timestamp, etc.), the correlated end frame timestamp <NUM> may be corrected to match the predicted end frame timestamp of the predicted timestamps <NUM>, effectively "clipping" the frame <NUM> to create the correlated frame <NUM>. If the measured frame length is less than the predicted frame length, the difference may be ignored, maintaining the measured timestamps of the frame <NUM>.

The correlator application <NUM> may be further configured to add the frame data, including the correlated timestamps <NUM> and <NUM>, of the correlated frame <NUM> to the frame buffer <NUM> for future use by the correlator application <NUM>. In some examples, the oldest frame data of the frame buffer <NUM> may be removed or otherwise overwritten by the new frame data.

<FIG> is an exemplary flow chart <NUM> illustrating operation of a computing device (e.g., computing device <NUM>, etc.) to receive a frame (e.g., frame <NUM>, etc.) and correlate the frame based on predicted timestamps (e.g., predicted timestamps <NUM>, etc.) according to an embodiment. The operations of flow chart <NUM> may be performed by one or more software applications configured to do so (e.g., correlator application <NUM>, etc.) on a computing device of a display system. At <NUM>, a frame of a video stream is received from a capture device (e.g., video capture device <NUM>, etc.) via a bus interface (e.g., USB interface <NUM>, etc.), the frame including a first start frame timestamp (e.g., start frame timestamp <NUM>, etc.) and a first end frame timestamp (e.g., end frame timestamp <NUM>, etc.) based on a device clock (e.g., device clock <NUM>, etc.) associated with the capture device. In some examples, the frame is received in a series of packets as described above, each packet including the first start frame timestamp and the first end frame timestamp in a header portion of the packet. Upon receipt of the frame packets, they may be used to reconstruct the frame during the process described in flow chart <NUM>. The received frame may be one in a series of frames of the video stream previously received from the capture device. The bus interface may include a USB interface and/or other types of bus interfaces that may be used without departing from the description herein.

At <NUM>, a second start frame timestamp and a second end frame timestamp are predicted (e.g., predicted timestamps <NUM>, etc.) based on a plurality of previously received frames of the video stream (e.g., frames in the frame buffer <NUM>, etc.), the second start frame timestamp and the second end frame timestamp being based on a system clock (e.g., system clock <NUM>, etc.) associated with the display system. The second start frame timestamp and second end frame timestamp may be predicted based on one or more collected and/or calculated values associated with the previously received frames of the video stream and/or the bus interface and associated aspects of the bus, protocol, etc. (e.g., average framerate, average frame length, bus bandwidth, bus-based timestamps, clock differential values, etc.) as described above.

At <NUM>, a skew value of the received frame is calculated based on a difference between the first start frame timestamp and the second start frame timestamp, and a difference between the first end frame timestamp and the second end frame timestamp. In some examples, the calculated skew value is in the form of a discrete number of time units (e.g., <NUM>, <NUM>, etc.). Alternatively, or additionally, the calculated skew value may include values relative to the received frame and other frames of the video stream. For instance, a skew value may indicate that the received frame has a <NUM>% longer frame length than the average frame length for frames of the associated video stream, or a skew value may indicate that the measured timestamps of the received frame is skewed <NUM>% out of sync with the predicted timestamps of the received frame (e.g., the received frame length matches the predicted frame length at <NUM>, but the measured start frame and end frame timestamps of the received frame are <NUM> later than the predicted start frame and end frame timestamps, etc.), etc..

If, at <NUM>, the calculated skew value exceeds a skew threshold, the received frame is corrected to correlate to the second start frame timestamp and the second end frame timestamp at <NUM>. After the received frame has been corrected, it is provided for display at <NUM>. In some examples, correcting the received frame includes correcting the timestamps of the received frame to match the predicted timestamp values as described herein.

Alternatively, if the calculated skew value does not exceed the skew threshold at <NUM>, the received frame is provided for display at <NUM>.

At <NUM>, providing the frame for display may include displaying, on a user interface, the frame in sequence with other frames of the video stream immediately as live video. Alternatively, the frame may be combined with other frames of the video stream into a stored video file that may then be played at a later time.

In some examples, the received frame is provided for display via the applications <NUM> in <FIG>, which may first perform additional processing on the received frame prior to display.

In some examples, the provided frame is further used to update a frame buffer for use in correlating received frames in the future. Predicted frames are based on the frame data of the frame buffer and, because the measured timestamps of frames are used unless they are skewed too far from the predicted timestamps, the calculated average data values may change over time in response to the frame buffer being updated with the measured timestamps of received frames. Thus, the predicted timestamps may change dynamically to reflect changing characteristics of the video capture device, the communication bus and/or interfaces, or the like.

<FIG> is an exemplary flow chart <NUM> illustrating operation of a computing device (e.g., computing device <NUM>, etc.) to receive frames (e.g., frame <NUM>, etc.) of multiple video streams, correlate the received frames, and combine the multiple video streams into a combined video stream according to an embodiment. As in <FIG>, the operations of flow chart <NUM> may be performed by one or more software applications configured to do so (e.g., correlator application <NUM>, etc.) on a computing device of a display system. At <NUM>, frames from a first video stream and a second video stream are received. The frames from the multiple streams may be received via a single bus interface or over multiple interfaces. Each frame may include an identifier for the associated video stream enabling the frames to be sorted by the computing device. The frames may be processed in order as they are received and/or stored in a queue in order until they can be correlated as described herein. In some examples, the frames may be received continuously during the operations described in the flow chart <NUM>, such that the described operations may occur repeatedly for as long as frames are received.

At <NUM>, the next received frame is selected by the computing device (e.g., computing device <NUM> and/or specifically correlator application <NUM>, etc.) to be correlated. At <NUM>, at least one timestamp of the selected frame is predicted based on a plurality of previously received frames of the video stream associated with the selected frame. In some examples, the at least one timestamp includes a start frame timestamp and an end frame timestamp as described herein. Predicting the at least one timestamp may include accessing a frame buffer (e.g., frame buffer <NUM>, etc.) that is specific to the video stream of the selected frame and generating at least one predicted timestamp as described above with respect to <FIG>.

At <NUM>, a skew value of the selected frame is calculated based on a difference between a received at least one timestamp of the selected frame and the predicted at least one timestamp. For instance, if the selected frame includes a start frame timestamp, a predicted start frame timestamp is generated and compared to the received start frame timestamp, resulting in a calculated skew value of the received start frame timestamp. In examples where a start frame timestamp and an end frame timestamp are received, the skew value(s) may be calculated based on predicted start frame and predicted end frame timestamps as described herein.

At <NUM>, if the calculated skew value exceeds a skew threshold, the selected frame is corrected to correlate to the predicted at least one timestamp at <NUM>. Alternatively, if the calculated skew value does not exceed the skew threshold at <NUM>, the selected frame is included into a combined video stream at <NUM> as described below. In some examples, the skew threshold may be defined specifically for one of the multiple video streams, such that skew values of frames associated with each video stream are compared to stream-specific skew thresholds (e.g., the first video stream may have a skew threshold of <NUM> and the second video stream may have a skew threshold of <NUM>, etc.). Alternatively, the skew threshold may be identical for all video streams.

At <NUM>, the selected frame is corrected to correlate to the predicted at least one timestamp. In some examples, correcting the selected frame includes setting timestamps of the selected frame to correlated timestamp values (e.g., correlated start frame timestamp <NUM> and correlated end frame timestamp <NUM>, etc.). Correlating the timestamp values may include correcting the received timestamps of the selected frame to match the predicted timestamps due to the excessive skew values of the received timestamps.

At <NUM>, the selected frame is included into the combined video stream. The combined video stream includes correlated frames from each of the first and second video streams, combined in such a way to form a single video stream. For instance, the combined video stream may be a three-dimensional video stream based on combining the two video streams. The two video streams may include two RGB-based video streams from two different angles, an RGB-base video stream and a depth-based video stream, an RGB-based video stream and an infrared (IR) video stream, etc. Each frame of the first video stream may be matched with a frame of the second video stream based on correlated timestamps to synchronize the two video streams. Any methods for combining frames of the now-correlated multiple video streams into a single video stream understood by a person of ordinary skill in the art may be used without departing from the description herein. Further, it should be understood that more than two video streams may be processed, correlated, and combined as described with respect to <FIG>.

At <NUM>, if received frames remain that have not been processed, the next frame is selected at <NUM>. Alternatively, if the no received frames remain to be processed, the combined video stream is provided for display at <NUM>. Further, the combined video stream may be provided for display at <NUM> while the operations of flow chart <NUM> are being performed on remaining frames, such that frames of the combined video stream that have already been processed may be displayed while the remaining frames are processed as described. Viewing the combined video stream may include viewing a three-dimensional representation of a scene or room based on the multiple video streams, viewing an area from multiple locations based on the multiple video streams, or the like.

Aspects of the disclosure enables various scenarios, such as next described.

In an example, a user of a computing device is viewing a live video stream from a USB-based camera device connected to the computing device via a USB cable. A frame of the video stream is timestamped by the camera device using the associated device clock as it is captured. The timestamps include a start frame timestamp and an end frame timestamp. The camera device performs a compression operation on the frame and transfers the compressed frame to the computing device over the USB cable. Transfer over the USB cable includes dividing the frame data up into multiple packets and including the timestamps of the frame in a header of each of the packets. Further, USB protocol-based timestamps based on when the start frame and end frame packets are transferred via the USB interface may be included on the packet headers as well. Upon receiving the frame packets, the computing device uses a correlator application to generate predicted timestamps for the frame. The predicted start frame and end frame timestamps are generated based on the received USB protocol-based timestamps, a calculated average bandwidth of the USB interface, and a calculated average frame length based on a plurality of previously received frames stored in a frame buffer associated with the correlator application. A skew value is calculated by comparing the camera device clock-based start frame and end frame timestamps to the predicted start frame and end frame timestamps. The calculated skew value exceeds a defined skew threshold, so the correlator application corrects the timestamps of the frame to match the predicted timestamps, rather than the measured timestamps applied by the camera device. The frame is added to the live video stream and provided for viewing by the use in sequence with other frames of the video stream. Further, the frame is added to the frame buffer of the correlator application, replacing the oldest frame of the buffer.

In another example, a user is using a virtual reality device to view a three-dimensional representation of a room. An RGB-based camera and a depth-based camera are placed in the room in order to capture the video data necessary to generate the three-dimensional video stream for display to the user. Both cameras are transferring video stream frame data to the virtual reality device via USB-based wireless communication interfaces. Frames from each camera are transmitted in streams of packets to the virtual reality device as described herein. The virtual reality device includes a correlator application that receives and correlates the frames from both cameras based on characteristics specific to each video stream in order to synchronize both video streams to the system clock of the virtual reality device. The correlator application includes a frame buffer specific to each video stream and generates predicted timestamps for a received frame based on the previously received frames of the frame buffer specific to the received frame. Skew values for each frame are compared to defined skew thresholds that are also specific to the associated video stream and, when a skew threshold is exceeded by the skew value of a frame, the predicted timestamps for the frame are used to correlate the frame timestamps with the system clock of the virtual reality device as described herein. Once frames from each video stream are correlated to the system clock, they are in sync with each other and may be combined to create a high-quality three-dimensional video stream for viewing by the user.

The present disclosure is operable with a computing apparatus according to an embodiment as a functional block diagram <NUM> in <FIG>. In an embodiment, components of a computing apparatus <NUM> may be implemented as a part of an electronic device according to one or more embodiments described in this specification. The computing apparatus <NUM> comprises one or more processors <NUM> which may be microprocessors, controllers or any other suitable type of processors for processing computer executable instructions to control the operation of the electronic device. Platform software comprising an operating system <NUM> or any other suitable platform software may be provided on the apparatus <NUM> to enable application software <NUM> to be executed on the device. According to an embodiment, correlation of frames of video streams using a system clock may be accomplished by software.

Computer executable instructions may be provided using any computer-readable media that are accessible by the computing apparatus <NUM>. Computer-readable media may include, for example, computer storage media such as a memory <NUM> and communications media. Computer storage media, such as a memory <NUM>, include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or the like. Computer storage media include, but are not limited to, RAM, ROM, EPROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information for access by a computing apparatus. In contrast, communication media may embody computer readable instructions, data structures, program modules, or the like in a modulated data signal, such as a carrier wave, or other transport mechanism. As defined herein, computer storage media do not include communication media. Therefore, a computer storage medium should not be interpreted to be a propagating signal per se. Propagated signals per se are not examples of computer storage media. Although the computer storage medium (the memory <NUM>) is shown within the computing apparatus <NUM>, it will be appreciated by a person skilled in the art, that the storage may be distributed or located remotely and accessed via a network or other communication link (e.g. using a communication interface <NUM>).

The computing apparatus <NUM> may comprise an input/output controller <NUM> configured to output information to one or more output devices <NUM>, for example a display or a speaker, which may be separate from or integral to the electronic device. The input/output controller <NUM> may also be configured to receive and process an input from one or more input devices <NUM>, for example, a keyboard, a microphone or a touchpad. In one embodiment, the output device <NUM> may also act as the input device. An example of such a device may be a touch sensitive display. The input/output controller <NUM> may also output data to devices other than the output device, e.g. a locally connected printing device. In some embodiments, a user <NUM> may provide input to the input device(s) <NUM> and/or receive output from the output device(s) <NUM>.

The functionality described herein can be performed, at least in part, by one or more hardware logic components. According to an embodiment, the computing apparatus <NUM> is configured by the program code when executed by the processor <NUM> to execute the embodiments of the operations and functionality described. For example, and without limitation, illustrative types of hardware logic components that can be used include Field-programmable Gate Arrays (FPGAs), Application-specific Integrated Circuits (ASICs), Program-specific Standard Products (ASSPs), System-on-a-chip systems (SOCs), Complex Programmable Logic Devices (CPLDs), Graphics Processing Units (GPUs).

Although some of the present embodiments may be described and illustrated as being implemented in a smartphone, a mobile phone, or a tablet computer, these are only examples of a device and not a limitation. As those skilled in the art will appreciate, the present embodiments are suitable for application in a variety of different types of devices, such as portable and mobile devices, for example, in laptop computers, tablet computers, game consoles or game controllers, various wearable devices, etc..

At least a portion of the functionality of the various elements in the figures may be performed by other elements in the figures, or an entity (e.g., processor, web service, server, application program, computing device, etc.) not shown in the figures.

Examples of well-known computing systems, environments, and/or configurations that may be suitable for use with aspects of the disclosure include, but are not limited to, mobile computing devices, personal computers, server computers, hand-held or laptop devices, multiprocessor systems, gaming consoles, microprocessor-based systems, set top boxes, programmable consumer electronics, mobile telephones, mobile computing and/or communication devices in wearable or accessory form factors (e.g., watches, glasses, headsets, or earphones), network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like. Such systems or devices may accept input from the user in any way, including from input devices such as a keyboard or pointing device, via gesture input, proximity input (such as by hovering), and/or via voice input.

wherein combining the video stream with the second video stream creates a three-dimensional correlated video stream.

The embodiments illustrated and described herein as well as embodiments not specifically described herein but within the scope of aspects of the claims constitute exemplary means for receiving a frame of a video stream from a capture device via a bus interface, the frame including a first start frame timestamp and a first end frame timestamp based on a device clock associated with the capture device; means for predicting, based on a plurality of previously received frames of the video stream, a second start frame timestamp and a second end frame timestamp associated with the received frame, the second start frame timestamp and second end frame timestamp being based on a system clock associated with the display system; means for calculating a skew value of the received frame based on a difference between the first start frame timestamp and the second start frame timestamp, and a difference between the first end frame timestamp and the second end frame timestamp; means for correcting the received frame to correlate to the second start frame timestamp and the second end frame timestamp upon the calculated skew value exceeding a skew threshold; and means for providing the corrected frame for display. The illustrated one or more processors <NUM> together with the computer program code stored in memory <NUM> constitute exemplary processing means for generating predicted timestamps based on analysis of receive frame timestamps and combining correlated frames into video streams, as described herein.

In some examples, the operations illustrated in the figures may be implemented as software instructions encoded on a computer readable medium, in hardware programmed or designed to perform the operations, or both. For example, aspects of the disclosure may be implemented as a system on a chip or other circuitry including a plurality of interconnected, electrically conductive elements.

Claim 1:
A computerized display system (<NUM>) for correlating timestamps of video frames (<NUM>) from a capture device (<NUM>) comprising:
at least one processor (<NUM>);
at least one memory (<NUM>) comprising computer program code, the at least one memory (<NUM>) and the computer program code configured to, with the at least one processor (<NUM>), cause the at least one processor (<NUM>) to:
receive, by a correlator application (<NUM>), a frame (<NUM>) of a video stream from a capture device (<NUM>) via a bus interface (<NUM>), the frame (<NUM>) including a first start frame timestamp (<NUM>) and a first end frame timestamp (<NUM>) based on a device clock (<NUM>) associated with the capture device (<NUM>);
predict, by the correlator application (<NUM>), based on a plurality of previously received frames (<NUM>) of the video stream, a second start frame timestamp (<NUM>) and a second end frame timestamp (<NUM>) associated with the received frame, the second start frame timestamp (<NUM>) and second end frame timestamp (<NUM>) being based on a system clock (<NUM>) associated with the display system (<NUM>) wherein predicting the second start frame timestamp includes maintaining a mapping of start frame timestamps of the plurality of previously received frames of the video stream, and wherein predicting the second start frame timestamp and the second end frame timestamp includes determining an average frame length of the plurality of previously received frames of the video stream, wherein the predicted second start frame timestamp and second end frame timestamp are separated by the determined average frame length;
calculate, by the correlator application (<NUM>), a skew value (<NUM>) of the received frame (<NUM>) based on a difference between the first start frame timestamp (<NUM>) and the second start frame timestamp (<NUM>), and a difference between the first end frame timestamp (<NUM>) and the second end frame timestamp (<NUM>);
upon the calculated skew value (<NUM>) exceeding a skew threshold (<NUM>), correct, by the correlator application (<NUM>), the received frame (<NUM>) to correlate to the second start frame timestamp (<NUM>) and the second end frame timestamp (<NUM>); and
provide the corrected frame (<NUM>) for display.