System, device, and method for user customizable streaming video playback

A system for customizable video playback includes: a customizable video playback server including a video database and a playback manager; and a plurality of customizable video playback devices, each including a dynamic index file and a playback controller, wherein the customizable video playback device enables a user to select a source video from the video database, and select a list of variable length video chunks from the source video, such that the variable length video chunks are sequenced by the dynamic index file, according to sequencing input from the user, to produce a customized video. Also disclosed is a method for customizable video playback, including selecting a source video, selecting variable-length video chunks, sequencing the customized video using a dynamic index file, and playing the customized video.

FIELD OF THE INVENTION

The present invention relates generally to the field of streaming of audiovisual information, and in particular HTTP adaptive bitrate streaming protocols such as HTTP live streaming (HLS) or The Moving Picture Experts Group (MPEG)-dynamic adaptive streaming over HTTP (MPEG-DASH), and more particularly to methods and systems for expanding the functionality of these protocols to allow for dynamic playback of multiple customized temporal versions of a given video.

BACKGROUND OF THE INVENTION

Streaming is a way of delivering video and audio data over the Internet, enabling a receiving playback device to start displaying the data before it fully loads. This allows the client to start playing the video without downloading the entire video.

Recent years have seen the rapid adoption of Adaptive Bitrate (ABR) streaming over Hypertext transfer protocol (HTTP), used in Video on Demand (VoD), Over the Top (OTT), Pay TV and other internet-based media delivery services, including mobile devices.

HTTP is a layer 7 protocol for internet communication. Web applications use HTTP to send data back and forth in a way that devices at both ends will be able to interpret.

Adaptive bitrate streaming is a method of video streaming over HTTP wherein the source content is segmented into small equal-length video chunks which are sequenced by an M3U8 index file. Multiple identical sets of video chunks are created at various bit rates, each set is sequenced by a similar index file. The system measures the internet bandwidth available to each receiving device, prompting a switch to a different bit rate when bandwidth conditions change. This makes it possible to stream videos at different quality levels, and to seamlessly switch in the middle of a video from one quality level to another.

HTTP Live Streaming (HLS) is an adaptive bitrate streaming protocol introduced by Apple in 2009. HLS is similar to MPEG-DASH, another streaming protocol, in that it breaks videos down into smaller chunks and encodes those chunks at different quality levels.

Both HLS and MPEG-DASH use HTTP, the advantage being that most of the Internet already uses HTTP. With HTTP, the stream uses a standard port (port 80 or 443) that is almost always open. This ensures that the stream is rarely blocked by a firewall, which can block streaming protocols that use specialized or unusual ports. Both protocols expect the video files to be stored in smaller chunks of equal length, typically 10 seconds each for HLS and 6 seconds each for MPEG-DASH.

The above-mentioned adaptive bitrate streaming protocols were an important development in video streaming. The existing method of splitting the master video into multiple equal-length chunks that are sequenced by a static index file, enables ability to switch from one set of chunks to other duplicate chunk sets with different bit rates. This solves major streaming problems associated with buffering and seamless bit rate adaptation.

Furthermore, the existing method also solves the problem of keyframe management by using a keyframe at the head of each chunk so that when the chunk switch occurs, there are no compression artifacts of unresolved frames. This keyframe management makes the whole method easily scalable since there is no additional server-side processing required to stitch the chunks back together.

However, existing streaming methods and systems do not allow for simultaneous streaming of different streams that are customized from the same source video, and they do not allow for variable-length chunking of video streams.

As such, considering the foregoing, it may be appreciated that there continues to be a need for novel and improved devices and methods for customizable video streaming.

SUMMARY OF THE INVENTION

The foregoing needs are met, to a great extent, by the present invention, wherein in aspects of this invention, enhancements are provided to the existing model of video streaming.

In an aspect, a system for customizable video playback can include:

In a related aspect, a customizable video playback server can include:

In another related aspect, a customizable video playback device can include:

DETAILED DESCRIPTION

Before describing the invention in detail, it should be observed that the present invention resides primarily in a novel and non-obvious combination of elements and process steps. So as not to obscure the disclosure with details that will readily be apparent to those skilled in the art, certain conventional elements and steps have been presented with lesser detail, while the drawings and specification describe in greater detail other elements and steps pertinent to understanding the invention.

The following embodiments are not intended to define limits as to the structure or method of the invention, but only to provide exemplary constructions. The embodiments are permissive rather than mandatory and illustrative rather than exhaustive.

The customizable video playback system 100 described herein addresses the challenge of obtaining various content representations from a single video source based on user-defined settings. The disclosed algorithm introduces a dynamic approach, allowing users to extract specific elements or generate summaries tailored to their preferences.

More specifically, the customizable video playback system 100 provides a method and system built on top of HTTP adaptive bitrate streaming protocols such as HLS and MPEG-DASH, wherein the protocols are characterized by comprising a plurality of small equal-length video chunks which are sequenced by a static M3U8 playlist file.

In contrast to the conventional HTTP-based adaptive bitrate streaming protocols, the customizable video playback system 100 introduces novel improvements that allow a multitude of individual separate viewers to initiate the simultaneous streaming of custom re-edited temporal versions of the same video, while still maintaining the adaptive bitrate functionality of the original protocols. These improvements include:

In the following, we describe the structure of an embodiment of a customizable video playback system 100 with reference to FIG. 1, in such manner that like reference numerals refer to like components throughout; a convention that we shall employ for the remainder of this specification.

In an embodiment, as shown in FIGS. 1, 2, 3, and 5C, a system for customizable video playback 100 can include:

In a related embodiment, as shown in FIGS. 2 and 6A, a data structure 610 for a source video 214 can include an ordered sequence 620 of segments 580, wherein each corresponding segment 580 can include:

In a further related embodiment, each video chunk 592 can include a variable number of video frames 626, such that the ordered sequence of corresponding video chunks 591 are variable length video chunks 591, such that the variable length video chunks 591 are each separately configurable with an independent variable length 595, which is based on the segment length of a corresponding segment 580, such that an aggregated length 597 of all video chunks 591 in the ordered sequence 590 of video chunks 591, 592, 594 is equal to the segment length 587;

In a related embodiment, as shown in FIGS. 2, 6A, and 6C, a customizable video playback server 102 can include:

In a related embodiment, as shown in FIGS. 3 and 6C, a customizable video playback device 104 can include:

In various related embodiments, as shown in FIGS. 2, 3, 4A, 5C, and 6C, the customizable video playback system 100 described herein introduces important novel functionality to conventional adaptive bitrate streaming protocols, introducing a method and system facilitating customizable video playback of alternate temporal versions such as summarized versions, character-weighted versions, and concept-weighted versions. In related embodiments, the customizable video playback system 100 can:

In one embodiment, the end user may be watching a summarized version of a video program defined according to a summarization algorithm, such that the compression ratio of which may be changed many times by the end user throughout the playback of the video program, such that:

In a first example embodiment 401, as shown in FIGS. 1, 4B and 4C, the customizable video playback system 100 can be implemented such that the customizable video playback device 104 in a first example embodiment is configured as a segmentizer player 491, wherein:

In a second example embodiment 402, as shown in FIGS. 4D and 4E, the customizable video playback system 100 can be implemented as a web-based player 492, such that:

In a third example embodiment, the customizable video playback system 100 can be implemented as an enhanced web-based player, such that:

In a fourth example embodiment, as shown in FIGS. 4F and 4G, the customizable video playback system 100 can be implemented as a further enhanced web-based player 494, such that:

In related embodiments, further Improvements of the video playback device 104 of the customizable video playback system 100 can include:

Finally, in a fifth example embodiment, the customizable video playback system 100 can be implemented as a yet further enhanced web-based player, such that:

In a related embodiment, have an HLS-based web player can serve video in different qualities (e.g., 320p, 480p, 720p) or different compression levels (e.g., 20%, 30%, etc.). Rather than recalculating every segment each time the user changes settings, the system dynamically builds or reuses existing HLS manifests based on information stored in a cache, such as a the REDIS™ session database 409. Main function of the HLS-based web player can include:

In a related embodiment, a segmentizer player can be configured such that:

In a further related embodiment, with respect to generation of video chunks 591:

In another further related embodiment, with respect general rules for all types of segments:

In another further related embodiment, with respect to use of WASM (WebAssembly):

In various embodiments, potential applications and use cases of the customizable video playback system 100 can include:

The adaptability of the customizable video playback system 100 makes it suitable for a wide range of applications, revolutionizing how users interact with and extract meaningful content from videos across diverse industries.

In a related embodiment, as shown in FIG. 5A, the dynamic index file 310 can be configured as a dynamic index file data structure 501, such that:

In a related embodiment, as shown in FIG. 5A, a video timeline 510 can be separated into DX segments 512, 514, 516 and SHOT segments 522, 524. The start frames 518 of each segment 512, 514, 516, 522, 524 are displayed on the right side of FIG. 5A.

In a further related embodiment, the summarization algorithm can be configured to calculate a significance value for each segment based on the content metrics. Dialogue significance is based on the calculation of significance values for the speaking people or characters in the video, word statistics of the sentences, concept statistics mined from the sentences, and other factors. There is usually visual information occurring during DX segments 512, 514, 516, but the transcribed sentence audio of the DX segment 512, 514, 516 takes priority over the visual information in the calculation of the significance value of the DX segment 512, 514, 516. The visual information of a DX segment 512, 514, 516 may assist in the identification of the speaker of each sentence which, in turn, assists in the calculation of character significance values. The visual information also assists in solving pronoun/antecedent relationships which, in turn, assists in the calculation of character significance.

In another further related embodiment, the customizable video playback device 104 can be configured to calculate a shot significance is based on the calculated significance value for the people or characters and their appearance in the shots, and other visual factors. There may usually be audio information occurring during a SHOT segment, but the visual information takes priority over the audio information in the calculation of the significance value of the SHOT segment. The audio information of a SHOT segment may assist in the calculation of the shot significance value. The audio information may assist in the identification of the most significant portions of a shot, which determines how much to trim the less significant portions of the shot.

In a related embodiment, as shown in FIG. 5B, the preparation of the streaming video chunks begins with the embedding of frame accurate reference in the overall long-play video. The frame boundaries of each segment 522 are searched and located. In the case of DX segments, the video is chunked based on the start frame location and the end frame location of each defined DX segment. In the case of SHOT segments, the start frame location and end frame locations are found, and each SHOT segment is divided and chunked into multiple sub-shot chunks 530.

In a further related embodiment, the streaming viewer may select or change a parameter that prompts the summarization algorithm to change the length of the overall video by skipping certain segments. The algorithm populates the dynamic index file 310 with a list of chunk numbers that will be played, the unprovided chunk numbers will therefore be skipped. Because the first frame of each chunk is a keyframe, all chunk transitions will be smooth and maintain the integrity of the interframe prediction video compression.

In another further related embodiment, FIG. 5C shows a sequence 503 of DX and SHOT segments. To the right of each segment is a line of segment data that includes the start and end frame, segment significance value, and whether the segment is included in the customized streaming view. SHOT segment 1627 and DX segments 1623, 1628, and 1631 and are not included in the view because their significance values fall below a threshold.

In this example embodiment, SHOT segments 1625 and 1630 are included in the view, but some of the sub SHOT segments are not included because the shots are being trimmed of less significant information based on significance threshold filtering, wherein:

Thus, in related embodiments, the variable length video chunks 591 can be configured with unique and meaningful time-length based segmentation boundaries of visual and audio events contained in the selected source video 214, such as dialogue segment boundaries, shot segment boundaries, and sub-shot segment boundaries.

Further, in various related embodiments, characteristics of the system for customizable video playback 100 can include:

In summary, in a related embodiment, as shown in FIGS. 5C and 6C, a dynamic index file 310 can include:

In a further related embodiment, the customizable video playback device 104 can be configured to play each corresponding segment 580 in the ordered segment list 570 in a segment order of the ordered segment list 570 (i.e., the segments 580 are played in sequence according to the ordering/sequencing of the ordered segment list 570), such that the video playback device 104 for each corresponding segment 580 is configured to play each corresponding variable length video chunk 591 corresponding to a corresponding chunk number 582 in the ordered chunk list 560 in a chunk order of the ordered chunk list 560 (i.e., for each segment 580, the video chunks 591 are played in sequence according to the ordering/sequencing of the ordered chunk list 560).

In some embodiments, the corresponding variable length video chunk 591 can be stored in the video database 210 and accessed by reference via the index chunk number 582, or other cases a copy of the corresponding variable length video chunk 591 may be stored locally on the customizable video playback device 104 in a local database (which can be a partial replicate of the video database 210) or in the dynamic index file 310.

In a further related embodiment, the ordered segment list 570 of segments 580 of the dynamic index file 310 can be a truncation and/or reordering of the ordered sequence 620 of segments 580 of the source video 214; and the ordered chunk list 560 of at least one index chunk number 582 can be a truncation and/or reordering of the ordered sequence 590 of corresponding video chunks 591, 592, 594 of the source video 214, which can facilitate an editing and resequencing of the source video 214 into the customized video 314. Thus, in an alternative description, the ordered segment list 570 of segments 580 of the dynamic index file 310 can be a permutation of the entire ordered sequence 620 of segments 580 of the source video 214 or a permutation of a subset thereof; and the ordered chunk list 560 of at least one index chunk number 582 can be a permutation of the entire ordered sequence 590 of corresponding video chunks 591, 592, 594 of the source video 214 or a permutation of a subset thereof.

In a further related embodiment, the selected video chunks 591 can each be separately configured with an independent variable length 595, which is based on the segment length of a corresponding segment 580, such that an aggregated length 597 of each video chunk 591 in the ordered chunk list 560 of at least one chunk number 582 is equal to the segment length 587;

In another further related embodiment, the dynamic index file 310 can be configured such that:

In a related embodiment, as shown in FIG. 6B, an object data structure 650 for an object 224 in the object database 220 can include:

In a further related embodiment, as shown in FIG. 8, a segment significance calculation process 800 for calculating a segment significance 588 of a segment 580 in a source video 214 can include:

In further related embodiments, a dynamic weighting can be applied to a calculated segment significance 588, for example based on configurable or user determined dynamic criteria.

In yet a further related embodiment, the playback manager 230 can be configured to calculate a segment significance 588 for each segment 580 in the source video 214, such that the segment significance 588 can be calculated as an aggregated sum of object significances 655 for each corresponding object 224, such as a person or character, appearing in corresponding variable length video chunks 591 associated with the segment, wherein the corresponding object 224 is stored in the object database 220; such that the playback manager 230 can be configured to execute an object recognition algorithm 231, wherein:

In another yet further related embodiment, a preceding object detection stage may be omitted, such that the playback manager 230 can be configured to execute an image recognition algorithm 234 to recognize a set of recognized image objects with a matching object image sample 656 in the corresponding variable length video chunks 591 associated with the segment 580, such that a corresponding object significance 655 is added to the aggregated sum 588 of object significances 655 for the segment, for each recognized image object 224.

In yet further related embodiment, the playback manager 230 can be further configured such that:

In yet further related embodiments, lines of dialogue can be cross referenced with scripts or subtitles to identify speakers; and for objects that are of type concepts (i.e., wherein the object type 654 is a concept), they can be detected in the spoken dialogue of a video, via the speaker diarization algorithm 236, but can also be supported by visuals.

In a yet further related embodiment, the segment included flag 589 can be set to true, if the segment significance 588 exceeds a predetermined significance threshold; and the segment included flag 589 can be set to false, if the segment significance 588 does not exceed the predetermined significance threshold. The predetermined significance threshold can for example be defined in a configuration file and can further be user adjustable by a user 122.

In a further related embodiment, the customizable video playback device 104 can be configured to play each corresponding segment 580 in the ordered segment list 570 in a segment order of the ordered segment list 570 for all segments with the segment included flag 589 set to true (i.e., the included segments 580 are played in sequence according to the ordering/sequencing of the ordered segment list 570), such that the video playback device 104 for each corresponding included segment 580 is configured to play each corresponding variable length video chunk 591 corresponding to a corresponding chunk number 582 in the ordered chunk list 560 in a chunk order of the ordered chunk list 560 (i.e., for each included segment 580, the video chunks 591 are played in sequence according to the ordering/sequencing of the ordered chunk list 560).

In an embodiment, as shown in FIG. 9, of a customizable video live streaming process 900, video and audio can be recorded for live streaming and the following actions/process steps can occur almost instantly and at a high rate of speed as to be perceived as “real time”, comprising:

Thus, in various related embodiments, the system for customizable video playback 100, including a customizable video playback server 102 and a customizable video playback device 104 configured to using a dynamic index file 310 to process a video summarization algorithm, which can also be referred to as a dynamic content extraction algorithm, which offers a transformative solution to the challenges in video processing. By dividing videos into segments and implementing a sophisticated algorithm within a web player, the customizable video playback system 100 brings about numerous possibilities across various industries. The capability to customize video playback, generate concise summaries, and enhance accessibility opens doors to improved learning experiences, enriched content creation, and interactive storytelling.

An embodiment of the system for customizable video playback 100 can include:

In related embodiments, the customizable video playback device 104 can include configurations as:

It shall be understood that an executing instance of an embodiment of the system for customizable video playback 100, as shown in FIG. 1, can include a plurality 114 of customizable video playback devices 104, which are each tied to one or more users 122.

An executing instance of an embodiment of the system for customizable video playback 100, as shown in FIG. 1, can similarly include a plurality of customizable video playback servers 102.

In an embodiment, as illustrated in FIG. 7, a method for customizable video playback 700, can include:

FIGS. 1, 2, and 3 are block diagrams and flowcharts, methods, devices, systems, apparatuses, and computer program products according to various embodiments of the present invention. It shall be understood that each block or step of the block diagram, flowchart and control flow illustrations, and combinations of blocks in the block diagram, flowchart and control flow illustrations, can be implemented by computer program instructions or other means. Although computer program instructions are discussed, an apparatus or system according to the present invention can include other means, such as hardware or some combination of hardware and software, including one or more processors or controllers, for performing the disclosed functions.

In this regard, FIGS. 1, 2, and 3 depict the computer devices of various embodiments, each containing several of the key components of a general-purpose computer by which an embodiment of the present invention may be implemented. Those of ordinary skill in the art will appreciate that a computer can include many components. However, it is not necessary that all of these generally conventional components be shown in order to disclose an illustrative embodiment for practicing the invention. The general-purpose computer can include a processing unit and a system memory, which may include various forms of non-transitory storage media such as random access memory (RAM) and read-only memory (ROM). The computer also may include nonvolatile storage memory, such as a hard disk drive, where additional data can be stored.

FIG. 1 shows a depiction of an embodiment of the system customizable video playback 100, including the customizable video playback server 102, and the customizable video playback device 104. In this relation, a server shall be understood to represent a general computing capability that can be physically manifested as one, two, or a plurality of individual physical computing devices, located at one or several physical locations. A server can for example be manifested as a shared computational use of one single desktop computer, a dedicated server, a cluster of rack-mounted physical servers, a datacenter, or network of datacenters, each such datacenter containing a plurality of physical servers, or a computing cloud, such as AMAZON EC2™ or MICROSOFT AZURE™.

It shall be understood that the above-mentioned components of the customizable video playback server 102 and the customizable video playback device 104 are to be interpreted in the most general manner.

For example, the processors 202 302 can each respectively include a single physical microprocessor or microcontroller, a cluster of processors, a datacenter or a cluster of datacenters, a computing cloud service, and the like.

In a further example, the non-transitory memory 204 and the non-transitory memory 304 can each respectively include various forms of non-transitory storage media, including random access memory and other forms of dynamic storage, and hard disks, hard disk clusters, cloud storage services, and other forms of long-term storage. Similarly, the input/output 206 and the input/output 306 can each respectively include a plurality of well-known input/output devices, such as screens, keyboards, pointing devices, motion trackers, communication ports, and so forth.

Furthermore, it shall be understood that the customizable video playback server 102 and the customizable video playback device 104 can each respectively include a number of other components that are well known in the art of general computer devices, and therefore shall not be further described herein. This can include system access to common functions and hardware, such as for example via operating system layers such as WINDOWS™, LINUX™, and similar operating system software, but can also include configurations wherein application services are executing directly on server hardware or via a hardware abstraction layer other than a complete operating system.

An embodiment of the present invention can also include one or more input or output components, such as a mouse, keyboard, monitor, and the like. A display can be provided for viewing text and graphical data, as well as a user interface to allow a user to request specific operations. Furthermore, an embodiment of the present invention may be connected to one or more remote computers via a network interface. The connection may be over a local area network (LAN) wide area network (WAN), and can include all of the necessary circuitry for such a connection.

In a related embodiment, the customizable video playback device 104 communicates with the customizable video playback server 102 over a network 106, which can include the general Internet, a Wide Area Network or a Local Area Network, or another form of communication network, transmitted on wired or wireless connections. Wireless networks can for example include Ethernet, Wi-Fi, BLUETOOTH™, ZIGBEE™, and NFC. The communication can be transferred via a secure, encrypted communication protocol.

In various related embodiments, as shown in FIGS. 2 and 3, components of the customizable video playback server 102, and the customizable video playback device 104 can include:

Typically, computer program instructions may be loaded onto the computer or other general-purpose programmable machine to produce a specialized machine, such that the instructions that execute on the computer or other programmable machine create means for implementing the functions specified in the block diagrams, schematic diagrams or flowcharts. Such computer program instructions may also be stored in a computer-readable medium that when loaded into a computer or other programmable machine can direct the machine to function in a particular manner, such that the instructions stored in the computer-readable medium produce an article of manufacture including instruction means that implement the function specified in the block diagrams, schematic diagrams or flowcharts.

In addition, the computer program instructions may be loaded into a computer or other programmable machine to cause a series of operational steps to be performed by the computer or other programmable machine to produce a computer-implemented process, such that the instructions that execute on the computer or other programmable machine provide steps for implementing the functions specified in the block diagram, schematic diagram, flowchart block or step.

Accordingly, blocks or steps of the block diagram, flowchart or control flow illustrations support combinations of means for performing the specified functions, combinations of steps for performing the specified functions and program instruction means for performing the specified functions. It will also be understood that each block or step of the block diagrams, schematic diagrams or flowcharts, as well as combinations of blocks or steps, can be implemented by special purpose hardware-based computer systems, or combinations of special purpose hardware and computer instructions, that perform the specified functions or steps.

As an example, provided for purposes of illustration only, a data input software tool of a search engine application can be a representative means for receiving a query including one or more search terms. Similar software tools of applications, or implementations of embodiments of the present invention, can be means for performing the specified functions. For example, an embodiment of the present invention may include computer software for interfacing a processing element with a user-controlled input device, such as a mouse, keyboard, touch screen display, scanner, or the like. Similarly, an output of an embodiment of the present invention may include, for example, a combination of display software, video card hardware, and display hardware. A processing element may include, for example, a controller or microprocessor, such as a central processing unit (CPU), arithmetic logic unit (ALU), or control unit.

Here has thus been described a multitude of embodiments of the customizable video playback system 100, including the customizable video playback server 102, the customizable video playback device 104, and methods related thereto, which can be employed in numerous modes of usage.

The many features and advantages of the invention are apparent from the detailed specification, and thus, it is intended by the appended claims to cover all such features and advantages of the invention, which fall within the true spirit and scope of the invention.

For example, alternative embodiments can reconfigure or combine the components of the customizable video playback server 102 and the customizable video playback device 104. The components of the customizable video playback server 102 can be distributed over a plurality of physical, logical, or virtual servers. Parts or all of the components of the customizable video playback device 104 can be configured to operate in the customizable video playback server 102, whereby the customizable video playback device 104 for example can function as a thin client, performing only graphical user interface presentation and input/output functions. Alternatively, parts or all of the components of the customizable video playback server 102 can be configured to operate in the customizable video playback device 104. For example, parts or all of the video database 210 and the object database 220 can be replicated to the customizable video playback device 104, for temporary, long-term, or permanent storage on the customizable video playback device 104.

Many such alternative configurations are readily apparent, and should be considered fully included in this specification and the claims appended hereto. Accordingly, since numerous modifications and variations will readily occur to those skilled in the art, the invention is not limited to the exact construction and operation illustrated and described, and thus, all suitable modifications and equivalents may be resorted to, falling within the scope of the invention.