Patent Description:
The increasing number of vision-based tracking systems deployed in production have necessitated fast, robust camera calibration. In the domain of sport, for example, the majority of current work focuses on sports where lines and intersections are easy to extract, and appearance is relatively consistent across venues. As relevant prior art in the field one can find patent documents <CIT>, <CIT> and non-patent document "<NPL>.

Further aspects of the present invention will become apparent from the following description with reference to the attached drawings. In some embodiments, a method of calibrating a broadcast video feed is disclosed herein. A computing system retrieves a plurality of broadcast video feeds for a plurality of sporting events. Each broadcast video feed includes a plurality of video frames. The computing system generates a trained neural network, by generating a plurality of training data sets based on the broadcast video feed by partitioning the broadcast video feed into a plurality of frames and learning, by the neural network, to generate a homography matrix for each frame of the plurality of frames. The computing system receives a target broadcast video feed for a target sporting event. The computing system partitions the target broadcast video feed into a plurality of target frames. The computing system generates for each target frame in the plurality of target frames, via the neural network, a target homography matrix. The computing system calibrates the target broadcast video feed by warping each target frame by a respective target homography matrix.

In some embodiments, a system is disclosed herein. The system includes a processor and a memory. The memory has programming instructions stored thereon, which, when executed by the processor, performs one or more operations. The one or more operations include retrieving a plurality of broadcast video feeds for a plurality of sporting event. Each broadcast video feed includes a plurality of video frames. The one or more operation further include generating a trained neural network, by generating a plurality of training data sets based on the broadcast video feed by partitioning the broadcast video feed into a plurality of frames and learning, by the neural network, to generate a homography matrix for each frame of the plurality of frames. the one or more operations further include receiving a target broadcast further include for each target frame in the plurality of target frames, generating, by the neural network, a target homography matrix. The one or more operations further include calibrating the target broadcast video feed by warping each target frame by a respective target homography matrix.

In some embodiments, a non-transitory computer readable medium is disclosed herein. The non-transitory computer readable medium includes one or more sequences of instructions that, when executed by one or more processors, causes a computing system to perform one or more operations. The computing system retrieves a plurality of broadcast video feeds for a plurality of sporting events. Each broadcast video feed includes a plurality of video frames. The computing system generates a trained neural network, by generating a plurality of training data sets based on the broadcast video feed by partitioning the broadcast video feed into a plurality of frames and learning, by the neural network, to generate a homography matrix for each frame of the plurality of frames. The computing system receives a target broadcast video feed for a target sporting event. The computing system partitions the target broadcast video feed into a plurality of target frames. The computing system generates for each target frame in the plurality of target frames, via the neural network, a target homography matrix. The computing system calibrates the target broadcast video feed by warping each target frame by a respective target homography matrix.

Camera calibration is an important task for computer vision applications, such as tracking systems, simultaneous localization and mapping (SLAM), and augmented reality (AR). Recently, many professional sports leagues have deployed some version of a vision-based tracking system. Additionally, AR applications (e.g., Virtual <NUM> in NBA®, First Down Line in NFL®) used during video broadcasts to enhance audience's engagement have become commonplace. All of these applications require high-quality camera calibration systems. Presently, most of these applications rely on multiple pre-calibrated fixed cameras or the real-time feed of pan-tilt-zoom (PTZ) parameters directly from the camera. However, as the most widely available data source in the sports domain is broadcast videos, the ability to calibrate from a single, moving camera with unknown and changing camera parameters would greatly expand the reach of player tracking data and fan-engagement solutions. Calibration of a single moving camera remains a challenging task as the approach should be accurate, fast, and generalizable to a variety of views and appearances. The one or more techniques described herein allows for a computing system to determine camera homography of a single moving camera given the frame and the sport.

Current approaches to camera calibration mainly follow a framework based on field registration, template matching (i.e., camera pose initialization), and homography refinement. Most of these approaches focus on sports where semantic information (e.g., key court markings) is easy to extract, the field appearance is consistent across stadiums (e.g., green grass and white lines), and motion of the camera is relatively slow and smooth. These assumptions, however, do not hold in more dynamic sports, such as basketball, where players occlude field markings, the field appearance varies wildly from venue to venue, and the camera moves quickly.

Furthermore, most existing works consist of multiple standalone models that are trained or tuned separately. As a result, they cannot achieve a global optimal for such an optimization task. This issue further limits the performance of those methods in more challenging scenarios as error propagates through the system, module to module.

The one or more techniques described herein relate to a brand new end-to-end neural network used for camera calibration. Through use of the end-to-end neural network, the present system is able to handle more challenging scenarios involving motion blur, occlusion and large transformations - scenarios existing systems are simply unable to account for or address. In some embodiments, the present system implements area-based semantics rather than lines for camera calibration, thus providing a more robust approach for dynamic environments and those environments with highly variable appearance features. In some embodiments, the present system incorporates a spatial transformation network for large transform learning, which aids in reducing the number of required templates for calibration purposes. In some embodiments, the present system implements an end-to-end architecture for camera calibration, which allow for joint training and inference homography much more efficiently.

<FIG> is a block diagram illustrating a computing environment <NUM>, according to example embodiments. Computing environment <NUM> may include camera system <NUM>, organization computing system <NUM>, and one or more client devices <NUM> communicating via network <NUM>.

Network <NUM> may be of any suitable type, including individual connections via the Internet, such as cellular or Wi-Fi networks. In some embodiments, network <NUM> may connect terminals, services, and mobile devices using direct connections, such as radio frequency identification (RFID), near-field communication (NFC), Bluetooth™, low-energy Bluetooth™ (BLE), Wi-Fi™, ZigBee™, ambient backscatter communication (ABC) protocols, USB, WAN, or LAN. Because the information transmitted may be personal or confidential, security concerns may dictate one or more of these types of connection be encrypted or otherwise secured. In some embodiments, however, the information being transmitted may be less personal, and therefore, the network connections may be selected for convenience over security.

Network <NUM> may include any type of computer networking arrangement used to exchange data or information. For example, network <NUM> may be the Internet, a private data network, virtual private network using a public network and/or other suitable connection(s) that enables components in computing environment <NUM> to send and receive information between the components of environment <NUM>.

Camera system <NUM> may be positioned in a venue <NUM>. For example, venue <NUM> may be configured to host a sporting event that includes one or more agents <NUM>. Camera system <NUM> may be configured to capture the motions of all agents (i.e., players) on the playing surface, as well as one or more other objects of relevance (e.g., ball, referees, etc.). In some embodiments, camera system <NUM> may be an optically-based system using, for example, a plurality of fixed cameras. For example, a system of six stationary, calibrated cameras, which project the three-dimensional locations of players and the ball onto a two-dimensional overhead view of the court may be used. In another example, a mix of stationary and non-stationary cameras may be used to capture motions of all agents on the playing surface as well as one or more objects of relevance. As those skilled in the art recognize, utilization of such camera system (e.g., camera system <NUM>) may result in many different camera views of the court (e.g., high sideline view, free-throw line view, huddle view, face-off view, end zone view, etc.). Generally, camera system <NUM> may be utilized for the broadcast feed of a given match. Each frame of the broadcast feed may be stored in a game file <NUM>.

Camera system <NUM> may be configured to communicate with organization computing system <NUM> via network <NUM>. Organization computing system <NUM> may be configured to manage and analyze the broadcast feed captured by camera system <NUM>. Organization computing system <NUM> may include at least a web client application server <NUM>, pre-processing engine <NUM>, a data store <NUM>, and a camera calibrator <NUM>. Pre-processing engine <NUM> and camera calibrator <NUM> may include one or more software modules. The one or more software modules may be collections of code or instructions stored on a media (e.g., memory of organization computing system <NUM>) that represent a series of machine instructions (e.g., program code) that implements one or more algorithmic steps. Such machine instructions may be the actual computer code the processor of organization computing system <NUM> interprets to implement the instructions or, alternatively, may be a higher level of coding of the instructions that is interpreted to obtain the actual computer code. The one or more software modules may also include one or more hardware components. One or more aspects of an example algorithm may be performed by the hardware components (e.g., circuitry) itself, rather as a result of the instructions.

Data store <NUM> may be configured to store one or more game files <NUM>. Each game file <NUM> may include the broadcast data of a given match. For example, the broadcast data may be a plurality of video frames captured by camera system <NUM>.

Camera calibrator <NUM> may be configured to calibrate the cameras of camera system <NUM>. For example, camera calibrator <NUM> may be configured to project players detected in the trackable frames to real world coordinates for further analysis. Because cameras in camera systems <NUM> are constantly moving in order to focus on the ball or key plays, such cameras are unable to be pre-calibrated. Camera calibrator <NUM> may be configured to generate a homography matrix that can register a target ground-plane surface of any frame from the broadcast video with a top view field model. For example, camera calibrator <NUM> may implement a single neural network to find a homograph matrix H that can register the target ground-plane surface of any frame I from a broadcast video with a top view field model M. In some embodiments, the standard objective function for computing homography with point correspondence may be: <MAT> where xi represents the (x,y) location of pixel i in the broadcast image l and <MAT> is the corresponding pixel location on the model "image" M and χ represents a set of point correspondences between the two images I and M.

Client device <NUM> may be in communication with organization computing system <NUM> via network <NUM>. Client device <NUM> may be operated by a user. For example, client device <NUM> may be a mobile device, a tablet, a desktop computer, or any computing system having the capabilities described herein. Users may include, but are not limited to, individuals such as, for example, subscribers, clients, prospective clients, or customers of an entity associated with organization computing system <NUM>, such as individuals who have obtained, will obtain, or may obtain a product, service, or consultation from an entity associated with organization computing system <NUM>.

Client device <NUM> may include at least application <NUM>. Application <NUM> may be representative of a web browser that allows access to a website or a stand-alone application. Client device <NUM> may access application <NUM> to access one or more functionalities of organization computing system <NUM>. Client device <NUM> may communicate over network <NUM> to request a webpage, for example, from web client application server <NUM> of organization computing system <NUM>. For example, client device <NUM> may be configured to execute application <NUM> to access content managed by web client application server <NUM>. The content that is displayed to client device <NUM> may be transmitted from web client application server <NUM> to client device <NUM>, and subsequently processed by application <NUM> for display through a graphical user interface (GUI) of client device <NUM>.

<FIG> are block diagrams illustrating neural network architecture <NUM> of camera calibrator <NUM>, according to example embodiments. As discussed, camera calibrator <NUM> may utilize a single neural network, which takes in a video frame as input, and outputs a homography matrix of that frame. For example, camera calibrator <NUM> may utilize a single neural network for a single moving camera calibration given unknown camera internal parameters across a variety of sports (e.g., basketball, soccer, football, hockey, etc.). Neural network architecture <NUM> may include three modules: semantic segmentation module <NUM>, camera pose initialization module <NUM>, and homography refinement module <NUM>. Each of the three modules <NUM>-<NUM> are integrated into a single neural network architecture, such as that shown by neural network architecture <NUM>. Because all three module <NUM>-<NUM> are connected, neural network architecture <NUM> is capable of end-to-end training.

Semantic segmentation module <NUM> may be configured to identify features of the playing surface (e.g., basketball court, soccer field, etc.). For example, semantic segmentation module <NUM> may be configured to extract key features and remove irrelevant information from an input image, l (reference numeral <NUM>). Such output may result in a venue agnostic appearance Y (reference numeral <NUM>) that may be used to determine the point correspondences. Thus, the objective function H from above may be rewritten as: <MAT> where θH represents a vector of the eight homography parameters, W(; θ) represents the warping function with transform parameters θ, and L() represents any loss function that measures the difference between two images, in this case the predicted semantic map Y and the warped overhead model M.

Semantic segmentation module <NUM> may conduct area-based segmentation on a playing surface by dividing the playing surface into one or more regions. By dividing the playing surface into one or more regions, semantic segmentation module <NUM> may transform the overhead field model M into a multi-channel image. Given the multi-channel image, semantic segmentation module <NUM> may classify each pixel in I into one region of the one or more regions. To generate area-based semantic labels of each image, semantic segmentation module <NUM> may warp the overhead model with the associated ground truth homography, thus providing ground truth semantic labels for training.

<FIG> is a block diagram illustrating one or more images <NUM>-<NUM> of a basketball playing surface, according to example embodiments. As shown, image <NUM> may correspond to a top-down view field model for a basketball playing surface. Semantic segmentation module <NUM> may divide the basketball playing surface into four regions, resulting in a <NUM>-chanel image. For example, region <NUM> may correspond to a first channel, region <NUM> correspond to a second channel, region <NUM> may correspond to a third channel, and region <NUM> may correspond to a fourth channel. In operation, semantic segmentation module <NUM> may utilize image <NUM> to classify each pixel in an input image (e.g., in I) into one of regions <NUM>-<NUM>.

Image <NUM> may illustrate semantic labels applied to an incoming image. Semantic segmentation module <NUM> may generate image <NUM> by warping the field model M (e.g., image <NUM>) using the ground truth homography. These images (e.g., image <NUM> and image <NUM>) may then be used to train semantic segmentation module <NUM>.

Image <NUM> may illustrate a polygonal area of image <NUM> from a top-down perspective, illustrating the fraction of the field model in the camera view.

Referring back to <FIG> and <FIG>, for the segmentation task, semantic segmentation module <NUM> may implement a Unet style auto-encoder <NUM> (hereinafter "Unet <NUM>"). Unet <NUM> may take, as input, image I <NUM> and output a semantic map Y <NUM> as needed by θH. In some embodiments, cross-entropy loss may be used to train Unet <NUM>. For example: <MAT> where C may represent the set of classes, and <MAT> may represent the ground truth label, and <MAT> may represent the likelihood of pixel i belonging to class c.

Camera pose initialization module <NUM> may be configured to select an appropriate template from a set of templates using a semantic map. Camera pose initialization module <NUM> may use a Siamese network to determine the best template for each input semantic image. Siamese network may be a convolutional encoder that computes a hidden representation for a semantic image, which may be the output of Unet <NUM> or any semantic template image. In some embodiments, the similarity between two images may be the L<NUM> norm between their hidden representations. In some embodiments, each image may be encoded to a <NUM>-length vector for similarity calculation.

For a PTZ camera, the projective matrix P may be expressed as: <MAT> where Q and S are decomposed from rotation matrix R, K are the intrinsic parameters of a camera in camera system <NUM>, I is a <NUM> × <NUM> identity matrix, and C is the camera translation. The matrix S may describe the rotation from the world coordinate to the PTZ camera base, and Q represents the camera rotation due to pan and tilt. For example, S may be defined to rotate around world x-axis by about -<NUM>° so that the camera looks along the y-axis in the world plane. In other words, the camera is level and its projection is parallel to the ground.

In some embodiments, for each image, camera calibrator <NUM> may assume a center principle point, square pixels, and no lens distortion. In some embodiments, six parameters may be identified. For example, the six parameters may be the focal length, three-dimensional camera location, pan and tilt angles.

In some embodiments, pre-processing engine <NUM> may initialize intrinsic camera matrix K, camera location C, and rotation matrix R. With this initialization, pre-processing engine <NUM> may identify the optimal focal length, three-dimensional camera location, and rotation angles. For example, pre-processing engine <NUM> may use the Levenberg-Marquardt algorithm to find the optimal focal length, three-dimensional camera location, and rotation angles. Once pre-processing engine <NUM> determines K, C, R, and S, pre-processing engine <NUM> may generate Q. In some embodiments, pre-processing engine <NUM> may generate the pan and tilt angles given Q. For example, pre-processing engine <NUM> may generate the pan and tile angles by applied the Rodrigues formula to Q. Thus, from the above, camera pose initialization module <NUM> may generate the <NUM>-dimensional camera configuration (pan, tilt, zoom, and three-dimensional camera location), λ.

After pre-processing engine <NUM> estimates the camera configuration λ for each training image, pre-processing engine <NUM> may generate a dictionary of possible camera poses Λ.

In some embodiments, pre-processing engine <NUM> may generate the dictionary of possible camera poses Λ by uniformly sampling from the range of possible camera poses. For example, pre-processing engine <NUM> may determine the ranges of pan, tilt, focal length, and camera location from training data and uniformly sample the poses from a <NUM>-dimensional grid. Such method is able to cover all camera poses, even if the training set is small. Further, using a smaller grid may simplify the homography refinement since the maximum scale of the transformation needed is on the scale of the grid size.

In some embodiments, pre-processing engine <NUM> can learn the possible camera poses Λ directly from the training data using clustering. Such process may be beneficial, for example, when the training set has sufficient diversity. For example, pre-processing engine <NUM> may treat Λ as a multi-variant normal distribution and apply a Gaussian Mixture model (GMM) to build the camera pose set. In some embodiments, mixing weights π may be fixed as equal for each component. In some embodiments, covariance matrix Σ may be fixed for each distribution. In such embodiments, the characteristic scale of Σ may set the scale of the transformations that are handled by homography refinement module <NUM>. In contrast with traditional GMMs, instead of setting the number of components K, the GMM learning algorithm implemented by pre-processing engine <NUM> may find the number of components K and the mean µk of each distribution given the mixing weights π and covariance matrix Σ. Identical Σ and π for each component may ensure that the GMM components are sampled uniformly from the manifold of the training data.

In some embodiments, the GMM learning algorithm may be:
<IMG>.

Because pre-processing engine <NUM> may fix Σ, camera pose initialization module may only update µ during the maximization step. Pre-processing engine <NUM> may gradually increase K until the stopping criteria are satisfied. The stopping criteria may aim to generate enough components so that every training example is close to the mean of one component in the mixture. Pre-processing engine <NUM> may generate the camera pose dictionary A utilizing all components [µ<NUM>,.

Given the dictionary of camera poses Λ, camera pose initialization module <NUM> may compute the homography for each pose and use the A to warp the overhead field model M. Accordingly, a set of image templates <IMG> = [T<NUM>,. , Tk] and their corresponding homography matrices <MAT> may be determined and used camera pose initialization module <NUM>.

Given the semantic segmentation image Y and a set of template images <IMG>, camera pose initialization module <NUM> may use a Siamese network to computer the distance between each input and template pair (Y,Tk). In some embodiments, the target/label for each pair may be similar or dissimilar. For example, for a grid sampled camera pose dictionary, a template Tk may be similar to the image if its pose parameters are the nearest neighbor in the grid. For the GMM-based camera pose dictionary, a template Tk may be labeled as similar to an image if the corresponding distribution of the template <IMG>(;µk, Σ ) gives the highest likelihood to the pose parameters λ of the input image. This procedure may generate a template similarity label for every image in the training set.

Once the input semantic image Y and the template images <IMG> are encoded (after FC1), camera pose initialization module <NUM> may use the latent representations to compute the L2 distance between the input image and each template. A selection module <NUM> may find the target camera pose index k and may retrieve its template image Tk and homography <MAT> as output according to: <MAT> where f() may represent the encoding function of the Siamese network.

In some embodiments, camera pose initialization module <NUM> may use contrastive loss to train the Siamese network. For example, <MAT>.

where a may represent the binary similarity label for the image pair (Y,Tk) and m may represent the margin for contrastive loss.

Homography segmentation module <NUM> may be configured to refine the homography by identifying the relative transform between the selected template and the input image. For example, homography segmentation module <NUM> may implement a spatial transformer network (STN) that allows for the handling of large non-affine transformation and use of a smaller camera pose dictionary. For example, given the input image and a selected template, the two images may be stacked and provided as input to STN. STN may be used to regress the geometric transformation parameters. In some embodiments, residual blocks may be used in convolutional encoder to preserve the salient features for deformation prediction. In some embodiments, ReLU may be used for all hidden layers, while the output layer of STN may use a linear activation.

To compute the relative transform between input semantic image Y and the selected template image Tk, homography segmentation module <NUM> may stack the images into an n-channel image (e.g., <NUM>-channel image), forming the input to the localization layers of the STN. In some embodiments, the output of the localization layers may be the parameters (e.g., <NUM>-parameters) of the relative homography H that maps the semantic image Y to the template Tk.

In some embodiments, homography segmentation module <NUM> may initialize the last of the localization layers (e.g., FC3), such that all elements in the kernel are zero and the bias is to the first n values (e.g., <NUM> values) of a flattened identity matrix. Therefore, at the start of the training, the input may be assumed to be identical to the template, providing an initialization for the STN optimization. Therefore, the final homography may be <MAT>.

Once H is computed, transformer <NUM> of homography refinement module <NUM> may warp the overhead model M to the camera perspective or vice versa, which allows camera calibrator <NUM> to compute the loss function. For example, homography refinement module <NUM> may us a Dice coefficient loss: <MAT> where U, V may represent semantic images, C may represent the number of channels, ∘ may represent the element-wise multiplication, and ∥·∥ may represent the sum of pixel intensity in an image. Here, for example, the intensity of each channel may be the likelihood that the pixel belongs to a channel c. One of the major advantages of using area-based segmentation, as opposed to line-based segmentation, is that it is robust to occlusions and makes better use (i.e., more efficient use) of the network capacity because a larger fraction of image pixels may belong to a meaningful class.

A limitation, however, of intersection-of-union (IoU) based loss is that as the fraction of the field of view in the image decreases, the IoU loss may become sensitive to segmentation errors. For example, if the playing surface occupied a tiny portion of the image, a small transform could reduce the IoU dramatically. Therefore, homography refinement module <NUM> uses the Dice loss on the warped playing surface in both perspectives - a high occupancy perspective can achieve coarse registration, while a low occupancy perspective can provide strong constraints on fine-tuning. Thus, the loss functions may be defined as: <MAT> where Y may represent the ground truth semantic image and M' may represent the masked overhead field model so that loss is only computed for the area shown in the image. Losses from the two perspectives may be weighted by δ, where the weight for the lower occupancy fraction perspective is always higher.

Because each module <NUM>-<NUM> may use the output of other modules as input, the three modules <NUM>-<NUM> may be connected into a single neural network (i.e., neural network architecture <NUM>). As such, the total loss of the network may become: <MAT> where α, β ∈ [<NUM>,<NUM>).

Camera calibrator <NUM> may train the entire neural network architecture <NUM> incrementally, module-by-module, so that the Siamese network and STN may start training with reasonable inputs. For example, training may start with a <NUM>-epoch warm-up for the Unet; the Siamese network training may be turned on with a α = <NUM> and β = <NUM>. After another <NUM> epochs, for example, the STN may be turned on with α = <NUM> and β = <NUM>. Neural network architecture may continue to undergo join training until convergence.

<FIG> is a flow diagram illustrating a method <NUM> of generating a fully trained calibration model, according to example embodiments. Method <NUM> may begin at step <NUM>.

At step <NUM>, organization computing system <NUM> may retrieve one or more data sets for training. Each data set may include a plurality of images captured by a camera system <NUM> during the course of a game.

In some embodiments, the data set may be created from thirteen basketball games. Those skilled in the art recognize that more than thirteen games or less than thirteen games may be used for training purposes. For example, ten games may be used for training and the remaining three games may be used for testing. Those killed in the art recognize that more than ten games or less than ten games may be used for training and more than three games or less than three games may be used for testing. The aforementioned number of games for training purposes is exemplary only and is not meant to limit the foregoing discussion. Different games may have different camera locations, with each game being played in a unique venue. As such, the playing surface appearance for each game may be very different from game-to-game. For each game, <NUM>-<NUM> frames may be selected for each annotation with a high camera pose diversity. Professional annotators may have clicked four to six point correspondences in each image to compute the ground truth homography. These annotations may have produced <NUM> images for training and <NUM> images for testing. In some embodiments, the training data may be further enriched by flipping the images horizontally, which may generate <NUM> training examples in total.

In some embodiments, the data set may be created from twenty soccer games. For example, the twenty soccer games were held in nine different stadiums during day and night, and the images may consist of different perspectives and lighting conditions. Accordingly, the data set may include <NUM> training images collected from <NUM> games and <NUM> testing images collected from the other <NUM> games.

At step <NUM>, organization computing system <NUM> may generate a plurality of camera pose templates from the one or more data sets. For example, based on the retrieved one or more data sets for training, camera calibrator <NUM> may generate camera pose templates for training. In some embodiments, camera calibrator <NUM> may generate the camera pose templates using the GMM-based method discussed above, provided that the one or more data sets is adequately large and diverse. In some embodiments, one or more data sets may be considered adequately large and diverse when a complete and relatively clean overhead playing surface image is achieved. In such embodiments, camera calibrator <NUM> may set the standard deviation for pan, tilt, focal length, and camera locations (x, y, z). In some embodiments, camera calibrator <NUM> may further set the threshold for stopping criteria and warping loss, <IMG>δ.

Continuing with the first example referenced above, using the basketball data set, camera calibrator <NUM> may use the GMM-based method to generate camera pose templates from <NUM> training images. In such example, camera calibrator <NUM> may set the standard deviation for pan, tilt, focal length, and camera locations (x, y, z) to <NUM>°, <NUM>°, <NUM> pixels, and <NUM> feet respectively. The non-diagonal elements may be set to zero, as camera calibrator <NUM> assumes that those camera configurations are independent of each other. The threshold for the stopping criteria may be set to <NUM> and the clustering algorithm may generate <NUM> components. For the warping loss, <IMG>δ may be set to <NUM> because the camera perspective may have a lower field occupancy rate than the top view perspective.

In some embodiments, camera calibrator <NUM> may generate the camera pose templates using a high grid resolution if, for example, the one or more data sets has an insufficient number of examples. In such embodiments, camera calibrator <NUM> may set the resolution of pan, tilt, and focal length.

Continuing with the second example referenced above, using the soccer data set, camera calibrator <NUM> may use a high grid resolution approach to generate the camera pose templates. In such examples, camera calibrator <NUM> may set the resolution of pan, tilt, and focal length to <NUM>°, <NUM>°, and <NUM> pixels, respectively. In some embodiments, the camera locations may be fixed at, for example, <NUM>, <NUM>, and <NUM>, yards relative to the top left corner of the field. Because the soccer data set has an insufficient number of examples to use the GMM-based camera pose estimation, camera calibrator <NUM> may use a uniform sampling for this data set with estimated pan, tilt, and focal length range ([-<NUM>°, <NUM>°], [<NUM>°, <NUM>°], [<NUM>, <NUM>] pixels respectively), which generates <NUM> templates for camera pose initialization.

As those skilled in the art recognize, although basketball and soccer are discussed in the current examples, such methodologies may be extended to the video broadcast of any sport.

At step <NUM>, organization computing system <NUM> may learn, based on the one or more training data sets, how to calibrate a single moving camera. For example, neural network of camera calibrator <NUM> may learn how to calibrate a single moving camera based on the one or more training data sets. In some embodiments, each module of neural network architecture <NUM> may be trained simultaneously. For example, because each module <NUM>-<NUM> of neural network architecture <NUM> uses the output of other modules as input, the three modules <NUM>-<NUM> may be connected into a single neural network. As such, the total loss of the network may become: <MAT> where α, β ∈ [<NUM>,<NUM>).

In some embodiments, one or more modules of modules <NUM>-<NUM> may be "warmed up" with synthesized data. For example, due to the small number of training examples in the above referenced soccer data sets, camera calibrator <NUM> may use synthesized data to warm up camera pose initialization module <NUM> and homography refinement module <NUM>. Apart from Unet in semantic segmentation module <NUM>, the rest of neural network architecture <NUM> uses the semantic images as input so that camera calibrator <NUM> can synthesize an arbitrary number of semantic images to pre-train parts of the network. Using a specific example, <NUM> semantic images may be generated by uniformly sampling the pan, tilt, and focal length parameters. For each synthesized image, their ground truth homography is known, and the template assignment can be easily found by down sampling the grid. Thus, camera pose initialization module <NUM> and the STN may be pre-trained individually. Once camera pose initialization module <NUM> and homography refinement module <NUM> are warmed up, camera calibrator <NUM> may train neural network with real data.

At step <NUM>, organization computing system <NUM> may output a fully trained prediction model. For example, at the end of the training and testing processes, camera calibrator <NUM> may have a fully trained neural network architecture <NUM>.

<FIG> is a flow diagram illustrating a method <NUM> of calibrating a broadcast camera, according to example embodiments. Method <NUM> may begin at step <NUM>.

At step <NUM>, organization computing system <NUM> may receive (or retrieve) a broadcast feed for an event. In some embodiments, the broadcast feed may be a live feed received in real-time (or near real-time) from camera system <NUM>. In some embodiments, the broadcast feed may be a broadcast feed of a game that has concluded. Generally, the broadcast feed may include a plurality of frames of video data. Each frame may capture a different camera perspective.

At step <NUM>, organization computing system <NUM> may input each frame into neural network architecture <NUM>. For example, camera calibrator <NUM> may identify a first frame in a received broadcast feed and provide that frame to neural network architecture <NUM>.

At step <NUM>, organization computing system <NUM> may generate a homography matrix H for each frame. For example, semantic segmentation module <NUM> may identify the court features Y in each frame. The output from semantic segmentation module <NUM> may be the semantic map Y generated by the Unet. The semantic map Y may be provided as input to camera pose initialization module <NUM>. Camera pose initialization module <NUM> may select the appropriate template Tk from a set of templates using semantic map Y. Camera pose initialization module <NUM> may further identify the target camera pose index k and retrieve its template image Tk and homography <MAT> using selection module <NUM>. Camera calibrator <NUM> may pass, as input to homography refinement module <NUM>, both Tk and Y concatenated and <MAT>. Homography refinement module <NUM> may then predict the relative homography H between the template and the semantic map by passing the concatenated item Tk and Y to the STN. Homography refinement module <NUM> may then generate the homography matrix H based on the relative homography H and <MAT> using matrix multiplication, i.e., <MAT>.

At step <NUM>, organization computing system <NUM> may warp each frame by its respective homography matrix H.

<FIG> illustrates a system bus computing system architecture <NUM>, according to example embodiments. System <NUM> may be representative of at least a portion of organization computing system <NUM>. One or more components of system <NUM> may be in electrical communication with each other using a bus <NUM>. System <NUM> may include a processing unit (CPU or processor) <NUM> and a system bus <NUM> that couples various system components including the system memory <NUM>, such as read only memory (ROM) <NUM> and random access memory (RAM) <NUM>, to processor <NUM>. System <NUM> may include a cache of high-speed memory connected directly with, in close proximity to, or integrated as part of processor <NUM>. System <NUM> may copy data from memory <NUM> and/or storage device <NUM> to cache <NUM> for quick access by processor <NUM>. In this way, cache <NUM> may provide a performance boost that avoids processor <NUM> delays while waiting for data. These and other modules may control or be configured to control processor <NUM> to perform various actions. Other system memory <NUM> may be available for use as well. Memory <NUM> may include multiple different types of memory with different performance characteristics. Processor <NUM> may include any general purpose processor and a hardware module or software module, such as service <NUM><NUM>, service <NUM><NUM>, and service <NUM><NUM> stored in storage device <NUM>, configured to control processor <NUM> as well as a special-purpose processor where software instructions are incorporated into the actual processor design. Processor <NUM> may essentially be a completely self-contained computing system, containing multiple cores or processors, a bus, memory controller, cache, etc. A multicore processor may be symmetric or asymmetric.

To enable user interaction with the computing device <NUM>, an input device <NUM> may represent any number of input mechanisms, such as a microphone for speech, a touch-sensitive screen for gesture or graphical input, keyboard, mouse, motion input, speech and so forth. An output device <NUM> may also be one or more of a number of output mechanisms known to those of skill in the art. In some instances, multimodal systems may enable a user to provide multiple types of input to communicate with computing device <NUM>. Communications interface <NUM> may generally govern and manage the user input and system output. There is no restriction on operating on any particular hardware arrangement and therefore the basic features here may easily be substituted for improved hardware or firmware arrangements as they are developed.

Storage device <NUM> may be a non-volatile memory and may be a hard disk or other types of computer readable media which may store data that are accessible by a computer, such as magnetic cassettes, flash memory cards, solid state memory devices, digital versatile disks, cartridges, random access memories (RAMs) <NUM>, read only memory (ROM) <NUM>, and hybrids thereof.

Storage device <NUM> may include services <NUM>, <NUM>, and <NUM> for controlling the processor <NUM>. Other hardware or software modules are contemplated. Storage device <NUM> may be connected to system bus <NUM>. In one aspect, a hardware module that performs a particular function may include the software component stored in a computer-readable medium in connection with the necessary hardware components, such as processor <NUM>, bus <NUM>, display <NUM>, and so forth, to carry out the function.

<FIG> illustrates a computer system <NUM> having a chipset architecture that may represent at least a portion of organization computing system <NUM>. Computer system <NUM> may be an example of computer hardware, software, and firmware that may be used to implement the disclosed technology. System <NUM> may include a processor <NUM>, representative of any number of physically and/or logically distinct resources capable of executing software, firmware, and hardware configured to perform identified computations. Processor <NUM> may communicate with a chipset <NUM> that may control input to and output from processor <NUM>. In this example, chipset <NUM> outputs information to output <NUM>, such as a display, and may read and write information to storage device <NUM>, which may include magnetic media, and solid state media, for example. Chipset <NUM> may also read data from and write data to RAM <NUM>. A bridge <NUM> for interfacing with a variety of user interface components <NUM> may be provided for interfacing with chipset <NUM>. Such user interface components <NUM> may include a keyboard, a microphone, touch detection and processing circuitry, a pointing device, such as a mouse, and so on. In general, inputs to system <NUM> may come from any of a variety of sources, machine generated and/or human generated.

Chipset <NUM> may also interface with one or more communication interfaces <NUM> that may have different physical interfaces. Such communication interfaces may include interfaces for wired and wireless local area networks, for broadband wireless networks, as well as personal area networks. Some applications of the methods for generating, displaying, and using the GUI disclosed herein may include receiving ordered datasets over the physical interface or be generated by the machine itself by processor <NUM> analyzing data stored in storage <NUM> or <NUM>. Further, the machine may receive inputs from a user through user interface components <NUM> and execute appropriate functions, such as browsing functions by interpreting these inputs using processor <NUM>.

It may be appreciated that example systems <NUM> and <NUM> may have more than one processor <NUM> or be part of a group or cluster of computing devices networked together to provide greater processing capability.

Claim 1:
A method (<NUM>) of generating a trained neural network (<NUM>) for calibrating a broadcast video feed captured by a single movable camera, comprising:
retrieving (<NUM>), by a computing system, a plurality of broadcast video feeds for a plurality of sporting events, each broadcast video feed comprising a plurality of video frames;
generating (<NUM>-<NUM>), by the computing system, a trained neural network, by:
generating (<NUM>) a plurality of training data sets based on the plurality of broadcast video feeds by partitioning the plurality of broadcast video feeds into a plurality of frames; and
learning, by the neural network, to generate a homography matrix for each frame of the plurality of frames;
characterized in that, learning to generate a homography matrix for each frame of the plurality of frames comprises:
learning, by a semantic segmentation module (<NUM>) of the neural network, to generate a venue agnostic appearance for each frame of the plurality of frames;
learning, by a camera pose initialization module (<NUM>) of the neural network, to compute a distance between each input received from the semantic segmentation module and a set of template images corresponding to multiple overhead field model (M) of the venue warped according to a dictionary of camera poses to each of which a homography matrix is associated;
learning, by the camera pose initialization module of the neural network, to identify a template homography matrix associated with the semantic segmentation module and the set of template images;
learning, by a homography refinement module (<NUM>) of the neural network, to generate a relative homography matrix based on a concatenated input comprising the venue agnostic appearance for each frame and a template image; and
learning, by the homography refinement module of the neural network, to generate the homography matrix based on the relative homography matrix and the template homography matrix;
wherein each of the semantic segmentation module, the camera pose initialization module, and the homography refinement module are trained simultaneously.