Patent ID: 12249149

DETAILED DESCRIPTION

FIG.1schematically illustrates a combat sports event100, in particular a boxing match in a boxing ring where a first combatant101and a second combatant102perform a combat. In the boxing ring, a referee103is located. Around the boxing ring, three video cameras200are positioned at different viewpoints to capture video sequences201. In particular, each of the video cameras200provides a respective video sequence201. This can for example be a continuous video stream in real-time.

Performing one particular exemplary embodiment of the method according to the present invention for automated video production, initially, three video sequences201are captured by respective video cameras200. The three video sequences201are received by a processing entity, e.g., by a data processing apparatus which is configured to perform the method according to embodiments of the present invention. Based on the processing step, a desired output video sequence300may be generated. The processing step is described in further detail in the following.

Processing the video sequences201may comprise determining, by an object detection algorithm, at least one region of interest202in the video sequence201, wherein the at least one region of interest202is associated with one of the first or second combatants101,102of the combat sports event100, preferably with at least one body part of the first or second combatant101,102. Determining the at least one region of interest202may be performed automatically, in particular based on a machine learning model, as described in detail above.

Further, processing the video sequences201may comprise determining a quality-of-view indicator of the at least one determined region of interest202in the video sequence201based at least in part on a positional relationship between the first combatant101and the second combatant102of the combat sports event100. Determining the quality-of-view indicator may be performed automatically, in particular based on a machine learning model, as described in detail above.

Examples for regions of interest may include the first or second combatant or their body parts, in particular their fist, forearm, elbow, foot, shin, knee, head, torso, shoulder and/or hip. Further, a region of interest may be an image section being equal to an entire video sequence captured by a video camera and/or may be a virtual camera view constructed from other region of interests in combination.

Steps of the method according to embodiments of the present invention are preferably performed by a plurality of machine learning models, in particular separately operated and/or trained machine learning models. In embodiments of the present invention, performing the method includes utilizing a sequence of at least two machine learning models. Alternatively or additionally, performing the method according to embodiments of the present invention includes utilizing at least three machine learning models, wherein outputs of at least two of the at least three machine learning models are used as inputs for at least one of the at least three machine learning models, thus creating kind of a tree structure of machine learning models. Each of the machine learning models may be adapted to their specific task, i.e., being trained to perform their task in an optimal manner.

Alternatively, according to embodiments of the present invention, the method may be performed by one singe machine-learning model configured to perform processing of video sequences201and to generate the output video sequence300.

Prior to processing the video sequences201, preprocessing one or more of the three captured video sequences201may be performed for reducing the processing load that is needed for performing processing. It is preferred that preprocessing is performed based on a preprocessing machine learning model that may be one of the above-mentioned plurality of machine learning models. In embodiments of the present invention, preprocessing can include splitting the video sequences201with respect to time sections and/or cropping the video sequences201with respect to image sections. Alternatively or additionally, preprocessing may include filtering out noise, adjusting lighting or contrast, masking and/or padding irrelevant sections of the video sequences201, introducing labels to video sequences201, marking sections of video sequences201as sections of main combat actions, and/or reducing the resolution or frame rate for simplicity. These adjustments assist in reducing redundant and/or irrelevant information, thus streamlining the input that will be processed by the subsequent stages of the method in order to, for example, reduce the need for processing resources.

In embodiments of the present invention, automatically determining the quality-of-view indicator is performed based on an evaluation machine learning model. Said evaluation machine learning model may be one of the above-mentioned plurality of machine learning models. Further, automatically determining a quality-of-view indicator may be performed based on and/or considering a viewpoint of a respective video camera200and/or based on relative positional aspects of the combatants101,102and optionally the referee103.

Additionally or alternatively, further relative positional aspects, as discussed in detail above, may be considered, for example relative positional aspects between body parts of a combatant101,102and/or relative positional aspects with respect to the referee103and/or relative positional aspects with respect to the viewpoint(s) of video camera(s). The viewpoint can be the location of a respective video camera200relative to the combat sports event, i.e., the ring in case of a combat sports type carried out in a ring. Further, the viewpoint may include a respective angle of view, more preferably a respective zoom factor and/or further settings of the respective video camera200. Thus, in other words, the viewpoint of a video camera200may include positioning and orientation of a respective video camera200and optionally other settings of the respective video camera200.

In embodiments of the present invention, the method may include generating the output video sequence300including determining a viewpoint of a virtual camera based on which the output video sequence300is generated. Determining the viewpoint of the virtual camera may be performed based on a viewpoint machine learning model that may be one of the above-mentioned plurality of machine learning models. Alternatively or additionally, determining the viewpoint of the virtual camera may include selecting one of the viewpoints of the plurality of video sequences201based at least in part on the determined quality-of-view indicators. Alternatively or additionally, determining the viewpoint of the virtual camera may include performing a zoom-in operation or a zoom-out operation, in particular on one or more of the at least one video sequence201. Alternatively or additionally, determining the viewpoint of the virtual camera may include performing a crop operation, in particular on one or more of the at least one video sequences201. Alternatively or additionally, determining the viewpoint of the virtual camera may include adjusting settings of one of the video cameras200.

Regarding the output video sequence300that is generated by embodiments of the present invention, the output video sequence300may include highlight scenes of the combat sports event and/or statistics of the combat sports event. The highlight scenes and/or statistics may be selected based on individual viewer preferences that may preferably be set by viewers, e.g., by providing viewer input to a frontend device which may be in particular capable of displaying the generated output video sequence300. Prior to generating the output video sequence300, determining highlight scene proposals may be performed. This may include detecting actions such as punches, kicks, take-downs. Determining highlight scene proposals may be, in embodiments of the present invention, performed based on a machine learning model which may be one of the above-mentioned plurality of machine learning models. Alternatively or additionally, highlight scene proposals are rated based on user preferences, wherein a selection of highlight scenes is performed. This may be performed by means of the above-mentioned machine learning model. Additionally or alternatively, generating the output video sequence300may include at least part of the determined statistics and proposed highlight scenes. The proposed and selected highlight scenes may be included in the output video sequence300as overlay graphics, in particular during breaks of the combat sports event100. The proposed and selected statistics may be included in the output video sequence300as overlay information, in particular during breaks of the combat sports event100.

InFIG.2, an exemplary output video scene300is illustrated. As can be seen in said figure, a first combatant201and a second combatant202are performing a boxing event. InFIG.3, the above-mentioned three video sequences201from the three video cameras200are illustrated. The video sequences201may preferably be preprocessed adding labelling information for facilitating smooth and efficient further processing. Further, as can be seen in the figure, regions of interest202are marked by bounding boxes that are observed. InFIG.4, a set of proposed highlight scenes is illustrated.

FIGS.5aand5bshow a flowchart explaining one embodiment of the present invention in different words. First of all, “record fight using a camera system with N cameras places around the ring” is performed. Then, “Extract audio and find synchronization time offset between camera videos” is performed. Then, “Split processing of N camera videos” is performed. Then, “Detect fighters and referee positions” is performed. Then, “Quantify view angles and view quality” is performed. Then, “Update virtual camera positions” is performed. Then, “Extract stats and generate highlight proposals” is performed. Then, “Select active virtual camera and perform crop to produce video stream” is performed. Then, “Do highlight selection and fusion of stats” is performed. Then, “Add relevant overlay graphics and insert highlights during round breaks” is performed. Then, “Transfer processed video stream to user” is performed.

FIG.6shows a flowchart describing parts of embodiments of the present invention in different words. The figure relates to a process of updating a virtual camera viewpoint in an embodiment of the present invention. The flowchart includes the step of “Get fighter and referee detections”. Further, the flowchart includes the step of “Calculate target view coordinated”. Further, the flowchart includes the step of “Update current virtual camera coordinates”. Further, the flowchart includes the step of “Crop and extract virtual camera view”.

FIGS.7aand7binclude a flowchart describing parts of embodiments of the present invention in different words, including in particular a live real-time generation of round highlights. With respect to the content of the flowchart, reference is made to the figures for the sake of clarity because the flowchart comprises several ramifications that cannot be reflected concisely in text form.

With respect to the different wording ofFIGS.5a,5b,6,7aand7b, the following expressions may be understood as follows: The expression “fight” may be understood as species to the generic expression “combat sports event”. The expression “camera system” may be understood as species to the generic expression “at least one video camera”. The expression “camera” may be understood as genus expression to the species expression “video camera”. The expressions “camera video” and “clip” may be understood as species to the generic expression “video sequence and/or output video sequence”. The expression “fighter” may be understood as species to the generic expression “combatant”. The expression “view angle” and “view quality” may be understood as species expressions in the context of the above-described generic expression “quality-of-view indicator”. The expressions “camera position” and “view” may be understood as species to the generic expression “viewpoint”. The expression “stats” may be understood as an abbreviation of the expression “statistics”. The expression “video stream” may be understood as species to the generic expression “output video sequence”. The expression “target view” may be understood as species to the generic expression “viewpoint of a video camera and/or virtual camera”.

Certain embodiments of the invention may be based on using a machine learning model or machine learning algorithm. Machine learning may refer to algorithms and statistical models that computer systems may use to perform a specific task without using explicit instructions, instead relying on models and inference. For example, in machine learning, instead of a rule-based transformation of data, a transformation of data may be used that is inferred from an analysis of historical and/or training data. For example, the content of images may be analyzed using a machine learning model or using a machine learning algorithm. In order for the machine learning model to analyze the content of an image, the machine learning model may be trained using training images as input and training content information as output. By training the machine learning model with a large number of training images and/or training sequences (e.g. words or sentences) and associated training content information (e.g. labels or annotations), the machine learning model “learns” to recognize the content of the images, so the content of images that are not included in the training data can be recognized using the machine learning model. The same principle may be used for other kinds of sensor data as well: By training a machine learning model using training sensor data and a desired output, the machine learning model “learns” a transformation between the sensor data and the output, which can be used to provide an output based on non-training sensor data provided to the machine learning model. The provided data (e.g., sensor data, meta data and/or image data) may be preprocessed to obtain a feature vector, which is used as input to the machine learning model.

Machine learning models may be trained using training input data. The examples specified above use a training method called “supervised learning”. In supervised learning, the machine learning model is trained using a plurality of training samples, wherein each sample may comprise a plurality of input data values, and a plurality of desired output values, i.e., each training sample is associated with a desired output value. By specifying both training samples and desired output values, the machine learning model “learns” which output value to provide based on an input sample that is similar to the samples provided during the training. Apart from supervised learning, semi-supervised learning may be used. In semi-supervised learning, some of the training samples lack a corresponding desired output value. Supervised learning may be based on a supervised learning algorithm (e.g., a classification algorithm, a regression algorithm or a similarity learning algorithm). Classification algorithms may be used when the outputs are restricted to a limited set of values (categorical variables), i.e., the input is classified to one of the limited set of values. Regression algorithms may be used when the outputs may have any numerical value (within a range). Similarity learning algorithms may be similar to both classification and regression algorithms but are based on learning from examples using a similarity function that measures how similar or related two objects are. Apart from supervised or semi-supervised learning, unsupervised learning may be used to train the machine learning model. In unsupervised learning, (only) input data might be supplied and an unsupervised learning algorithm may be used to find structure in the input data (e.g. by grouping or clustering the input data, finding commonalities in the data). Clustering is the assignment of input data comprising a plurality of input values into subsets (clusters) so that input values within the same cluster are similar according to one or more (pre-defined) similarity criteria, while being dissimilar to input values that are included in other clusters. Reinforcement learning is a third group of machine learning algorithms that may be used to train the machine learning model. In reinforcement learning, one or more software actors (called “software agents”) are trained to take actions in an environment. Based on the taken actions, a reward is calculated. Reinforcement learning is based on training the one or more software agents to choose the actions such, that the cumulative reward is increased, leading to software agents that become better at the task they are given (as evidenced by increasing rewards).

Furthermore, some techniques may be applied to some of the machine learning algorithms. For example, feature learning may be used. In other words, the machine learning model may at least partially be trained using feature learning, and/or the machine learning algorithm may comprise a feature learning component. Feature learning algorithms, which may be called representation learning algorithms, may preserve the information in their input but also transform it in a way that makes it useful, often as a pre-processing step before performing classification or predictions. Feature learning may be based on principal components analysis or cluster analysis, for example.

In some examples, anomaly detection (i.e., outlier detection) may be used, which is aimed at providing an identification of input values that raise suspicions by differing significantly from the majority of input or training data. In other words, the machine learning model may at least partially be trained using anomaly detection, and/or the machine learning algorithm may comprise an anomaly detection component.

In some examples, the machine learning algorithm may use a decision tree as a predictive model. In other words, the machine learning model may be based on a decision tree. In a decision tree, observations about an item (e.g., a set of input values) may be represented by the branches of the decision tree, and an output value corresponding to the item may be represented by the leaves of the decision tree. Decision trees may support both discrete values and continuous values as output values. If discrete values are used, the decision tree may be denoted a classification tree, if continuous values are used, the decision tree may be denoted a regression tree.

Association rules are a further technique that may be used in machine learning algorithms. In other words, the machine learning model may be based on one or more association rules. Association rules are created by identifying relationships between variables in large amounts of data. The machine learning algorithm may identify and/or utilize one or more relational rules that represent the knowledge that is derived from the data. The rules may e.g. be used to store, manipulate or apply the knowledge.

Machine learning algorithms are usually based on a machine learning model. In other words, the term “machine learning algorithm” may denote a set of instructions that may be used to create, train or use a machine learning model. The term “machine learning model” may denote a data structure and/or set of rules that represents the learned knowledge (e.g., based on the training performed by the machine learning algorithm). In embodiments, the usage of a machine learning algorithm may imply the usage of an underlying machine learning model (or of a plurality of underlying machine learning models). The usage of a machine learning model may imply that the machine learning model and/or the data structure/set of rules that is the machine learning model is trained by a machine learning algorithm.

For example, the machine learning model may be an artificial neural network (ANN). ANNs are systems that are inspired by biological neural networks, such as can be found in a retina or a brain. ANNs comprise a plurality of interconnected nodes and a plurality of connections, so-called edges, between the nodes. There are usually three types of nodes, input nodes that receiving input values, hidden nodes that are (only) connected to other nodes, and output nodes that provide output values. Each node may represent an artificial neuron. Each edge may transmit information, from one node to another. The output of a node may be defined as a (non-linear) function of its inputs (e.g., of the sum of its inputs). The inputs of a node may be used in the function based on a “weight” of the edge or of the node that provides the input. The weight of nodes and/or of edges may be adjusted in the learning process. In other words, the training of an artificial neural network may comprise adjusting the weights of the nodes and/or edges of the artificial neural network, i.e. to achieve a desired output for a given input.

Alternatively, the machine learning model may be a support vector machine, a random forest model or a gradient boosting model. Support vector machines (i.e., support vector networks) are supervised learning models with associated learning algorithms that may be used to analyze data (e.g., in classification or regression analysis). Support vector machines may be trained by providing an input with a plurality of training input values that belong to one of two categories. The support vector machine may be trained to assign a new input value to one of the two categories. Alternatively, the machine learning model may be a Bayesian network, which is a probabilistic directed acyclic graphical model. A Bayesian network may represent a set of random variables and their conditional dependencies using a directed acyclic graph. Alternatively, the machine learning model may be based on a genetic algorithm, which is a search algorithm and heuristic technique that mimics the process of natural selection.

Although some aspects have been described in the context of an apparatus, it is clear that these aspects also represent a description of the corresponding method, where a block or device corresponds to a method step or a feature of a method step. Analogously, aspects described in the context of a method step also represent a description of a corresponding block or item or feature of a corresponding apparatus.

Some or all of the method steps may be executed by (or using) a hardware apparatus, such as a processor, a microprocessor, a programmable computer or an electronic circuit. Depending on certain implementation requirements, embodiments of the invention can be implemented in hardware or in software. The implementation can be performed using a non-transitory storage medium such as a digital storage medium, for example a floppy disc, a DVD, a Blu-Ray, a CD, a ROM, a PROM, and EPROM, an EEPROM or a FLASH memory, having electronically readable control signals stored thereon, which cooperate (or are capable of cooperating) with a programmable computer system such that the respective method is performed. Therefore, the digital storage medium may be computer readable.

Some embodiments of the invention provide a data carrier having electronically readable control signals, which are capable of cooperating with a programmable computer system, such that one of the methods described herein is performed.

Generally, embodiments of the invention can be implemented as a computer program (product) with a program code, the program code being operative for performing one of the methods when the computer program product runs on a computer. The program code may, for example, be stored on a machine-readable carrier. Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine-readable carrier. In other words, an embodiment of the present invention is, therefore, a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.

A further embodiment of the invention provides a storage medium (or a data carrier, or a computer-readable medium) comprising, stored thereon, the computer program for performing one of the methods described herein when it is performed by a processor. The data carrier, the digital storage medium or the recorded medium are typically tangible and/or non-transitionary. A further embodiment of the present invention is an apparatus as described herein comprising a processor and the storage medium.

A further embodiment of the invention provides a data stream or a sequence of signals representing the computer program for performing one of the methods described herein. The data stream or the sequence of signals may, for example, be configured to be transferred via a data communication connection, for example, via the internet.

A further embodiment of the invention provides a processing means, for example, a computer or a programmable logic device, configured to, or adapted to, perform one of the methods described herein.

A further embodiment of the invention provides a computer having installed thereon the computer program for performing one of the methods described herein.

A further embodiment of the invention provides an apparatus or a system configured to transfer (e.g., electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may, for example, be a computer, a mobile device, a memory device, or the like. The apparatus or system may, for example, comprise a file server for transferring the computer program to the receiver.

In some embodiments, a programmable logic device (for example, a field programmable gate array) may be used to perform some or all of the functionalities of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor in order to perform one of the methods described herein. Generally, the methods are preferably performed by any hardware apparatus.

REFERENCE SIGNS

100combat sport[s] event101first combatant102second combatant103referee200video camera[s]201video sequence[s]202region[s] of interest300output video sequence[s]