Patent Description:
In a soccer game an offside situation of an offensive player is an important problem and at the same time it is difficult for linesmen to make a correct decision whether an offensive player is offside or not. The linesman has to check that at the moment when an offensive player forwards the ball no other offensive player is closer to the goal line than the defensive player who is closest to the goal line. Of course the goalkeeper is excluded from this consideration. Apparently, the linesman has to observe simultaneously two or even more players who are very likely moving dynamically during an attack. Among other things the judgment is made complicated by players being obscured by other players or by a parallax error of the linesman if he is not on the same line parallel to the goal line of the closest defensive player. In the context of the present disclosure it is not necessary to elaborate further the difference between active and passive offside situation. It is sufficient to understand that the judgment of offside situations is challenging. The same is true for viewers watching a soccer game on TV because the perspective of the broadcast camera capturing the game is many times not well-suited to verify a potential offside situation. In consequence, viewers watching the game on a TV may judge a potential offside event differently than the linesman and think that the linesman has made a wrong decision. This may cause a lot of frustration on the side of the viewers that might even spoil the joy of watching the soccer game.

In view of this problem and in order to enhance the viewer experience of soccer games it is known to insert offside lines in an image captured by a broadcast camera. The inserted offside line assists a viewer in making his or her own judgment. For example, from EVS broadcast Equipment S. a product named EPSIO Live ® is commercially available and enables displaying an offside line in a broadcast image of a soccer game. However, today available products for inserting an offside line require a considerable amount of manual interventions of an operator for inserting the offside line at the right position.

<CIT> discloses a system for determining the position of all players on a playing field. The system is capable of determining, in real time, from optical data a position of a playing ball and positions of players from an offensive team and from a defensive team. Two surveillance instruments, such as cameras, survey the playing field and are connected to a computer. The computer algorithms identify objects, colors and shapes and thus determine in real time the positions of the ball and the players of the offensive and defensive team.

<CIT> discloses a method of calculating a transform matrix for transforming points in a model of an object to an image of the object captured by a camera. This method is applied in particular to images of a football match.

<CIT> describes a method for detecting and classifying offside events in a soccer game. The method builds on ball tracking and detection as well as on determining ball dynamics. In addition to that, players are detected. For implementing the ball tracking feature it is suggested to utilize an artificial neural network, which is trained to recognize the ball. For illustrating a potential off-side position a bar parallel to the goal line is positioned manually by an operator.

Nevertheless, there remains a desire to facilitate the insertion of graphical objects in a broadcast image.

According to a first aspect the present disclosure suggests a method for automatically mapping a reference view of a playing field containing field lines with a camera image.

In general terms, the proposed technique automates the estimation of the homography, which is a kind of geometric transformation between pairs of two-dimensional images. Homography maps a reference view of a sports field, as seen from a given, known a priori, viewpoint (e.g. a "top" view) to images acquired by a camera pointed at the same sport field, and vice-versa. The method is applicable to one or more camera images corresponding to one or more camera views each one covering the complete playing field or only a fraction of the playing field.

The suggested method enables automatically detecting predefined positions on the playing field and thus enables to calculate the geometric transformation of the playing field caused by a specific camera view. This geometric transformation determines the way how an additional auxiliary line like an offside line has to be inserted into the camera image such that the user has the impression that the line is painted on the grass of the playing field. The advantage of the suggested method is that the classification of the predefined positions on the playing field in the calculation of the geometric transformation is performed automatically without human intervention. The operator of a live broadcast covering for example sports events is significantly unburdened by this automatic process.

In most of the ball games there are many intersections of field lines, in particular on a soccer playing field. Therefore, it is a convenient approach to take benefit of the intersections of field lines for determining the geometric transformation. Just for the sake of clarity it is noted that three to five predefined positions are sufficient to calculate a first estimate of the geometric transformation. However, the more predefined positions are taken into account the more precisely the geometric transformation can be calculated.

The detected field lines are very useful for an assessment of the quality of the geometric transformation because a comparison of the detected field lines and calculated field lines enables the assessment of the fidelity of the geometric transformation.

An embodiment takes benefit from having two sets of field lines available and further comprises adapting parameters of the geometric transformation for minimizing the differences between the detected field lines and the calculated field lines. In this way a refinement of the geometric transformation is achievable. Once it is no longer possible to further minimize the differences between the detected field lines and calculated field lines the corresponding parameters set of the geometric transformation is used to calculate the position and orientation of the additional line that is to be inserted into the camera image.

For instance in a soccer game, the inserted line can be an offside line. Since the geometric transformation, which is caused by the camera perspective, is known, it is possible to insert the line such that a viewer in front of a TV screen has the impression that the line is painted on the grass of the playing field. All necessary connections and calculations are performed automatically real-time.

According to a second aspect of the present disclosure an apparatus is suggested for automatic mapping a reference view of a playing field containing field lines with a camera image of a live broadcast production covering a ballgame on a playing field with field lines.

According to a third aspect the present disclosure suggests a software containing program code, which, when executed by a processor, performs a method according to the first aspect of the present disclosure.

Exemplary embodiments of the present disclosure are illustrated in the drawings and are explained in more detail in the following description. In the figures the same or similar elements are referenced with the same or similar reference signs.

Reference herein to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one implementation of the disclosure.

The embodiments described below comprise separate devices to facilitate the understanding of different functional group of the present disclosure. However, it is to be understood that some devices may very well integrated in a single device.

Most generally speaking the present disclosure relates to mapping of a reference view of a playing field for sports such as soccer, basketball, US football, tennis and others. The reference view is a view on the playing field from a known view point, for instance from a top view. Under a first aspect the present disclosure suggests a method for mapping the reference view to images acquired by a camera pointed at the same sport field, and vice-versa. The mapping permits to determine an estimate for a geometric transformation also known as homography between the pairs of the two dimensional images, namely the reference view and the camera image. Once the geometric transformation is known, it enables the insertion of graphical objects into the camera image in compliance with the perspective of the camera view, e.g. it is possible to display an offside line in a soccer game, player trajectories, advertising, scoreboard, etc. as if these objects were "painted on the grass" of the playing field.

The principles of the method are are explained with reference to <FIG>.

In <FIG> an attack situation of a soccer team is shown. The goal of the defensive team is on the right side in <FIG>. An offensive player <NUM> just has the ball in the situation shown in <FIG>. A second attacking player <NUM> is marked by two defenders <NUM>, <NUM>. From the view shown in <FIG> it is difficult to say whether this second attacking player <NUM> is offside or not, i.e. whether he is closer to the goal line of the defensive team than any other defender with the exception of the goalkeeper of the defensive team. In the foreground of <FIG> a linesman <NUM> runs together with the defender <NUM>, who is closest to the goal line, in order to be able to identify an offside situation. The linesman tries to be on the same level as the defender <NUM> with regard to the goal line to have a view on the position of the defender which is not falsified by a parallax error. If the linesman <NUM> falls behind the position of the defender <NUM> or advances with regard to that position there is a risk that the linesmen makes an erroneous decision.

For viewers watching the game on the television screen it is even more difficult to make a correct judgment of a potential offside event. If, for instance, the linesman decides differently than what the viewers have seen on the TV screen this may cause frustration on the side of the viewers and even spoil the joy of watching the game. In order to assist the viewers, a line is inserted between offside and fair play. The line is shown as a straight line <NUM> in <FIG>. The offside line <NUM> is displayed on a screen of a TV as if it was painted on the grass of the playing field. It is not straightforward to achieve this effect because the vanishing point of the camera image depends on the perspective of the camera on the playing field. In other words: In the camera image the playing field and the field lines thereon undergo geometric transformation that per se is not known but must be extracted from the camera image itself. To this end an operator selects some locations on the playing field e.g. intersections of field lines. In <FIG> some of the locations are shown as circles <NUM>. He then identifies the corresponding locations in a top view of the playing field that is shown in <FIG>. With this information the applicable geometric transformation can be calculated and an offside line can be manually inserted into the camera image. The line complies with the geometric transformation and is properly displayed on the grass as the line <NUM> parallel to the goal line. This can be seen best in the top view shown in <FIG> where the line <NUM> is parallel to the goal line <NUM> and perpendicular to side lines <NUM>. With reference to <FIG> it is evident that such an offside line is very instrumental to the viewer to make his own decision about an offside situation.

From the description of the conventional approach for creating an offside line it becomes evident that this process requires a lot of manual interventions by the operator who is already loaded with other tasks involved in the preparation of an image that is eventually going on air. Loading the operator with the additional task of inserting an offside line brings on a considerable risk that the broadcast operator gets overloaded and the program output signal going on air is compromised in one way or another.

The present disclosure, therefore, aims at replacing manual interventions by automatic processes when inserting an offside line in the camera image.

<FIG> shows a block diagram of an apparatus <NUM> for achieving this goal. A camera <NUM> captures images in a football stadium. For the sake of simplicity only a single camera <NUM> is shown in <FIG>. But in reality, there are for example <NUM> to <NUM> cameras distributed around the playing field to make sure that there is always one camera that has a good shot at an interesting scene anywhere on the playing field. The camera images of multiple cameras are processed in parallel.

The camera <NUM> feeds the image data into a data connection <NUM> that transmits the data to a video processor <NUM>, a first neural network (AI1) <NUM> and a second neural network (AI2) <NUM>. The block diagram is a simplification of the broadcast equipment for producing life broadcast programs. The broadcast equipment may also include storage devices, routers, digital video effect devices, character generators etc. All these devices are not shown in <FIG> because they are not relevant to understand the present disclosure. The data connection <NUM> transmits the image frames from the camera <NUM> in synchronized fashion to the video processor <NUM> and to neural networks <NUM> and <NUM>, i.e. every image frame arrives at the same time at all three devices.

The first neural network <NUM> automatically classifies each pixel of the camera image into a plurality of classes, wherein the classes correspond to predefined locations on the playing field. In one embodiment of the present disclosure the predefined locations are intersections of field lines on the playing field. <FIG> shows intersections of field lines on a soccer field. Each intersection is marked with a circle having an index number <NUM> to <NUM> in the circle.

The first neural network <NUM> has been trained to detect corresponding locations in the camera image as it is shown in <FIG>. The first neural network <NUM> generates for each pixel in the camera image a triplet composed of the geometric position of the pixel in the image and a class identifying whether the pixel corresponds to one of the predefined locations: (x,y,class). These triplets are output to the video processor <NUM> which calculates a first estimate of a geometric transformation that transforms the top view of the playing field into the playing field captured by the camera. The known geometric transformation enables the video processor <NUM> to calculate field lines in the camera image.

The second neural network <NUM> has been trained to detect the field lines in the camera image and outputs the coordinates of the field lines to the video processor <NUM>. The result is shown in <FIG> as a composite image where field lines captured by the camera <NUM> are shown as solid lines <NUM> and the field lines calculated according to the first estimate of the geometric transition are shown as dashed lines <NUM>. As it can be seen from <FIG> there are deviations between the field lines <NUM> and <NUM>. For better illustration the deviations are exaggerated, i.e. in reality the deviations are smaller.

The video processor <NUM> compares the positions of the detected field <NUM> lines in the camera image with the calculated field lines <NUM> and refines the geometric transformation to minimize the differences and/or deviations between the two sets of field lines.

The geometric transformation has eight degrees of freedom (parameters). Finding a refined geometric transformation consists in finding the set of parameter values that causes the calculated lines <NUM> in the composite image to overlap as good as possible the field lines <NUM> detected by neural network <NUM>. According to the approach of the present disclosure the first estimate mentioned above is used as a starting point. Subsequently, the <NUM>-dimensional parameter space is explored. This exploration is conducted in an iterative fashion. At each iteration one parameter in the parameter space is incremented. At each iteration, a step of fixed (given) extent in the <NUM>-dimensional parameter space is made. The orientation chosen for this step is the one that increases the most the overlap between the calculated field lines <NUM> and the detected field lines <NUM>. This process stops when it is no longer possible to make a step that increases the overlap. Such procedure is known in the literature as "iterative gradient descent".

Finally, the offside line is computed and inserted into the camera image by the video processor <NUM> using the refined geometric transformation. In this embodiment the offside line is still manually positioned by an operator, because the operator decides at which distance from the goal line of the defensive team the offside line <NUM> is drawn in the camera image. However, all preparation steps enabling generating an offside line in the correct perspective for each camera image is performed automatically. In fact commercially available processors are fast enough to perform all necessary computations in real time for example within <NUM>, which is the equivalent of processing <NUM> frames per second. For each and every frame the correct geometric transformation is available in real time and an offside line can be inserted into each camera frame in compliance with the right perspective. The only remaining task for the operator is to position the offside line.

In a further embodiment of the present disclosure the operator is also unburdened from positioning the offside line.

The component that is required to perform this additional functionality is shown in the block diagram in <FIG> with dashed lines. A third neural network <NUM> (AI3) receives the camera images in synchronicity like the neural networks <NUM>, <NUM>. The neural network <NUM> is trained for detecting players on the playing field. This includes distinguishing the players of the two teams and their respective positions. Therefore, the neural network <NUM> is capable of detecting the position of the defence player who is closest to the goal line of his team. The position data is output as coordinates of the camera image to the video processor <NUM>. The video processor <NUM> utilizes this information to position the offside line as illustrated in <FIG>. The defence player who is closest to the goal line of his team is referenced as <NUM>. The hatched area on the left side of line <NUM> in <FIG> indicates the offside in the moment of the game shown in <FIG>. Consequently, at any moment the right position and perspective of the offside line is available in real-time and can be inserted into every camera image frame. However, the offside line is only interesting in some scenes of the game and rather disturbing during other scenes. Therefore, in one embodiment the operator controls the insertion of the offside line by activating a button <NUM> on a control panel <NUM> shown in <FIG> without the need to provide any further inputs because the video processor <NUM> of apparatus <NUM> has already all necessary information to position and orient the offside line such that the viewer gets the desired impression that the offside line is drawn on the grass of the playing field.

<FIG> exhibits a flow diagram illustrating the steps of the method according to the present disclosure. In step S1 camera images are received and transmitted via a data connection or network <NUM>. In step S2 a neural network <NUM> classifies each pixel of each camera image to determine whether the pixel belongs to a predefined position such as an intersection of field lines on a playing field. The result of processing step S2 is a data triplet containing the geometric position of the pixel in the camera image and the corresponding class information. The data triplet is provided to a video processor <NUM> that uses in step S3 the data triplet to calculate the geometric transformation which is associated with the camera perspective capturing the playing field. In step S4 the video processor <NUM> receives information about a position of the offside line. In one embodiment the information is provided as manual input by an operator. In another embodiment the information about the position of the offside line is automatically determined or by the neural network <NUM>, which automatically determines the positions of all players on the playing field. Thus, the neural network <NUM> can easily detect an offside position by applying a simple rule. In either case the video processor <NUM> uses the information about the position of the offside line to insert the offside line <NUM> at the right position into the camera image in compliance with the camera perspective. The user gets the impression that the offside line is drawn with paint on the grass of the playing field.

The offside line is only exemplary for other graphical objects, which are inserted into the camera image and which are of interest to the viewer such as player trajectories, player names, advertising, scoreboard, etc..

In its most general implementation the method according to the present disclosure only performs steps S1 to S3, i.e. calculates the geometric transformation to make it available for other purposes as it is explained in greater detail further below.

As mentioned above, the method of the present disclosure, and more particularly the geometric transformation works in both directions between the reference view and the camera image, i.e. it is possible to map a position in a camera image with the corresponding position in the reference view. This aspect is interesting e.g. for live broadcasts of sports events when a plurality of cameras are installed in a stadium. Obviously, the cameras have different views on the scenes on the playing field. It occurs that in an important scene a single camera image does not display all key players but only in two or more camera images shot by different cameras having different views or perspectives on the playing field. The suggested method allows reconciling observations made by multiple cameras pointed at the same event by projecting all of camera images on the reference view. The reference view represents a single tactical view including the position of players, balls etc. The projection is performed by video processor.

This feature is particularly useful for an automated player detection system. The neural network <NUM> is capable of tracking all players on the playing field in the received camera images. The video processor <NUM> receives this information and is provided with an automated player detection functionality, which is configured to indicate the positions of all players in the reference view. Thus, the apparatus <NUM> also implements an automated player detection system.

The output of the automated player detection system is displayed in <FIG> showing a top view on the playing field. A starting point of a player at the beginning of the important scene is indicated with a cross <NUM>. A line <NUM> shows the trajectory of the player to an end point <NUM> of the player at the end of the important scene. For the sake of clarity only the trajectory of a single player is shown in <FIG>. However, the operator can select on the control panel <NUM> additional players if he finds their trajectories are also interesting for analysing the important scene.

It is to be noted that a reverse transformation, i.e. from the reference or tactical view into a camera view or camera image is equally feasible. Hence, any information that is easily represented in the tactical view (e.g. the offside line) can be automatically represented in the original camera view.

In <FIG> illustrate the reversibility of the geometric transformation. <FIG> shows a camera image or camera view on a playing field. <FIG> displays players <NUM> of an offensive team, players of a defensive team <NUM>, an offside line <NUM> and some predefined locations <NUM>.

<FIG> shows a top view or tactical view corresponding to the camera view of <FIG>. The positions of the players <NUM>, <NUM>, the offside line <NUM> and the predefined locations <NUM> are shown at their corresponding positions in the top view. Lines <NUM> visualize the aperture of a camera that has taken the image shown in <FIG>. Arrows <NUM> symbolize the reversible geometric transition between the views. The offside line <NUM> is an example of a graphical object that can easily be positioned in the top view in <FIG>. From there it is transferred in the right perspective into the camera image shown in <FIG>. At the same time the operator can check if graphical objects to be inserted into the camera view are obstructed by players. If necessary, he can move the graphical object to a better suited position.

Generally speaking, the method and apparatus of the present disclosure enable augmented reality in live broadcasts, in particular in live broadcasts of sports events.

Individual components or functionalities of the present disclosure are described in the exemplary embodiments as software or hardware solution. This does not mean, however, that a functionality described as a software solution cannot be implemented in hardware as well, and vice versa. Likewise, mixing solutions are also conceivable for one skilled in the art, in which components and functionalities are partially implemented in software and hardware at the same time.

In the claims, the word "comprise" does not include other elements or steps, and the undefined article "a" does not exclude a plurality.

A single unit or device may perform the functions of several elements set forth in the claims. The fact that individual functions and elements are listed in different dependent claims does not mean that a combination of these functions and elements could not be used advantageously.

Claim 1:
Method for automatically mapping a reference view of a playing field containing field lines with a camera image, wherein the method comprises
- determining in the reference view predefined locations on the playing field;
- receiving the camera image from a camera (<NUM>);
- classifying intersections of the field lines with a first neural network (<NUM>) at predefined positions on the playing field for automatically classifying each pixel of the camera image into a plurality of classes, wherein the classes correspond to the predefined locations in the reference view on the playing field;
- detecting the field lines (<NUM>) in the camera image with a second neural network (<NUM>);
- receiving the output of the first neural network (<NUM>) and a known geometry of the playing field and the field lines thereon for calculating a geometric transformation defining how the geometry of the playing field is transformed by the perspective of the camera image and for calculating field lines in the camera view;
- refining the geometric transformation by minimizing deviations between the calculated field lines and the detected field lines;
- detecting positions of players and distinguishing their respective teams with a third neural network (<NUM>);
- automatically positioning an offside line (<NUM>) in the camera image by using the refined geometric transformation according to the detected position of one of the players (<NUM>), wherein the offside line is displayable in the camera image.