Patent Description:
The present disclosure generally relates to system and method for player reidentification in broadcast video.

Player tracking data has been implemented for a number of years in a number of sports for both team and player analysis. Conventional player tracking systems, however, require sports analytics companies to install fixed cameras in each venue in which a team plays. This constraint has limited the scalability of player tracking systems, as well as limited data collection to currently played matches. Further, such constraint provides a significant cost to sports analytics companies due to the costs associated with installing hardware in the requisite arenas, as well as maintaining such hardware.

XP30081004A1 proposes a method for athlete identification by combing the segmentation, tracking and recognition procedures into a coarse-to-fine scheme for jersey number (digital characters on sport shirt) detection. Firstly, image segmentation is employed to separate the jersey number regions with its background. And size/pipe-like attributes of digital characters are used to filter out candidates. Then, a K-NN (K nearest neighbor) classifier is employed to classify a candidate into a digit in "<NUM>-<NUM>" or negative. In the recognition procedure, Zernike moment features are used, which are invariant to rotation and scale for digital shape recognition. Synthetic training samples with different fonts are used to represent the pattern of digital characters with non-rigid deformation. Once a character candidate is detected, a SSD (smallest square distance)-based tracking procedure is started. The recognition procedure is performed every several frames in the tracking process. After tracking tens of frames, the overall recognition results are combined to determine if a candidate is a true jersey number or not by a voting procedure.

<CIT> generally relates to a system and method for performing analysis of events that appear in live and recorded video feeds, such as sporting events. In particular, the application relates to a system and methods for enabling spatiotemporal analysis of component attributes and elements that make up events within a video feed, such as of a sporting event, systems for discovering, learning, extracting, and analyzing such events, metrics and analytic results relating to such events, and methods and systems for display, visualization, and interaction with outputs from such methods and systems.

In accordance with a first aspect of the present invention there is provided a method of re-identifying players in a broadcast video feed as defined in claim <NUM>. Optional and/or preferable features are defined in the dependent claims.

In accordance with a second aspect of the present invention there is provided a system for re-identifying players in a broadcast video feed as defined in claim <NUM>. Optional and/or preferable features are defined in the dependent claims.

In accordance with a third aspect of the present invention there is provided a non-transitory computer readable medium including one or more sequences of instructions that, when executed by one or more processors, perform operations as defined in claim <NUM>. Optional and/or preferable features are defined in the dependent claims.

It is to be noted, however, that the appended drawings illustrated only typical embodiments of this disclosure and are therefore not to be considered limiting of its scope, for the disclosure may admit to other equally effective embodiments, so long as they fall within the scope of the appended claims.

In the last decade, vision-based tracking systems have been widely deployed in many professional sports leagues to capture the positional information of players in a match. This data may be used to generate fine-grained trajectories which are valuable for coaches and experts to analyze and train their players. Although players can be detected quite well in every frame, an issue arises in conventional systems due to the amount of manual annotation needed from gaps between player trajectories. The majority of these gaps may be caused by player occlusion or players wandering out-of-scene.

One or more techniques disclosed herein improve upon conventional systems by providing a system capable of generating fine-grained player trajectories despite player occlusion or the player wandering out-of-scene. For example, one or more techniques disclosed herein may be directed to operations associated with recognizing and associating a person at different physical locations over time, after that person had been previously observed elsewhere. The problem solved by the techniques described herein are compounded in the domain of team sports because, unlike ordinary surveillance, players' appearances are not discriminative. The techniques described herein overcome this challenge by leveraging player jersey (or uniform) information to aid in identifying players. However, because players constantly change their orientations during the course of the game, simply using jersey information is not a trivial task. To accurately identify player trajectories, the present system may identify frames of video with visible jersey numbers and may associate identities using a deep neural network.

<FIG> is a block diagram illustrating a computing environment <NUM>, used by embodiments. Computing environment <NUM> includes organization computing system <NUM>, and may include camera system <NUM>, and one or more client devices <NUM> communicating via network <NUM>.

Network <NUM> may be of any suitable type, including individual connections via the Internet, such as cellular or Wi-Fi networks. Network <NUM> may connect terminals, services, and mobile devices using direct connections, such as radio frequency identification (RFID), near-field communication (NFC), Bluetooth™, low-energy Bluetooth™ (BLE), Wi-Fi™, ZigBee™, ambient backscatter communication (ABC) protocols, USB, WAN, or LAN. Because the information transmitted may be personal or confidential, security concerns may dictate one or more of these types of connection be encrypted or otherwise secured. Sometimes, the information being transmitted may be less personal, and therefore, the network connections may be selected for convenience over security.

Network <NUM> may include any type of computer networking arrangement used to exchange data or information. For example, network <NUM> may be the Internet, a private data network, virtual private network using a public network and/or other suitable connection(s) that enables components in computing environment <NUM> to send and receive information between the components of environment <NUM>.

Camera system <NUM> may be positioned in a venue <NUM>. For example, venue <NUM> may be configured to host a sporting event that includes one or more agents <NUM>. Camera system <NUM> may be configured to capture the motions of all agents (i.e., players) on the playing surface, as well as one or more other objects of relevance (e.g., ball, referees, etc.). Camera system <NUM> may be an optically-based system using, for example, a plurality of fixed cameras. For example, a system of six stationary, calibrated cameras, which project the three-dimensional locations of players and the ball onto a two-dimensional overhead view of the court may be used. In another example, a mix of stationary and non-stationary cameras may be used to capture motions of all agents on the playing surface as well as one or more objects of relevance. As those skilled in the art recognize, utilization of such camera system (e.g., camera system <NUM>) may result in many different camera views of the court (e.g., high sideline view, free-throw line view, huddle view, face-off view, end zone view, etc.). Generally, camera system <NUM> may be utilized for the broadcast feed of a given match. Each frame of the broadcast feed may be stored in a game file <NUM>.

Camera system <NUM> may be configured to communicate with organization computing system <NUM> via network <NUM>. Organization computing system <NUM> may be configured to manage and analyze the broadcast feed captured by camera system <NUM>. Organization computing system <NUM> may include at least a web client application server <NUM>, a data store <NUM>, an auto-clipping agent <NUM>, a data set generator <NUM>, a camera calibrator <NUM>, a player tracking agent <NUM>, and an interface agent <NUM>. Each of auto-clipping agent <NUM>, data set generator <NUM>, camera calibrator <NUM>, player tracking agent <NUM>, and interface agent <NUM> may be comprised of one or more software modules. The one or more software modules may be collections of code or instructions stored on a media (e.g., memory of organization computing system <NUM>) that represent a series of machine instructions (e.g., program code) that implements one or more algorithmic steps. Such machine instructions may be the actual computer code the processor of organization computing system <NUM> interprets to implement the instructions or, alternatively, may be a higher level of coding of the instructions that is interpreted to obtain the actual computer code. The one or more software modules may also include one or more hardware components. One or more aspects of an example algorithm may be performed by the hardware components (e.g., circuitry) itself, rather as a result of the instructions.

Data store <NUM> may be configured to store one or more game files <NUM>. Each game file <NUM> may include the broadcast data of a given match. For example, the broadcast data may a plurality of video frames captured by camera system <NUM>.

Auto-clipping agent <NUM> may be configured parse the broadcast feed of a given match to identify a unified view of the match. In other words, auto-clipping agent <NUM> may be configured to parse the broadcast feed to identify all frames of information that are captured from the same view. In one example, such as in the sport of basketball, the unified view may be a high sideline view. Auto-clipping agent <NUM> may clip or segment the broadcast feed (e.g., video) into its constituent parts (e.g., difference scenes in a movie, commercials from a match, etc.). To generate a unified view, auto-clipping agent <NUM> may identify those parts that capture the same view (e.g., high sideline view). Accordingly, auto-clipping agent <NUM> may remove all (or a portion) of untrackable parts of the broadcast feed (e.g., player close-ups, commercials, half-time shows, etc.). The unified view may be stored as a set of trackable frames in a database.

Data set generator <NUM> may be configured to generate a plurality of data sets from the trackable frames. In some embodiments, data set generator <NUM> may be configured to identify body pose information. For example, data set generator <NUM> may utilize body pose information to detect players in the trackable frames. Data set generator <NUM> may be configured to further track the movement of a ball or puck in the trackable frames. Data set generator <NUM> may be configured to segment the playing surface in which the event is taking place to identify one or more markings of the playing surface. For example, data set generator <NUM> may be configured to identify court (e.g., basketball, tennis, etc.) markings, field (e.g., baseball, football, soccer, rugby, etc.) markings, ice (e.g., hockey) markings, and the like. The plurality of data sets generated by data set generator <NUM> may be subsequently used by camera calibrator <NUM> for calibrating the cameras of each camera system <NUM>.

Camera calibrator <NUM> may be configured to calibrate the cameras of camera system <NUM>. For example, camera calibrator <NUM> may be configured to project players detected in the trackable frames to real world coordinates for further analysis. Because cameras in camera systems <NUM> are constantly moving in order to focus on the ball or key plays, such cameras are unable to be pre-calibrated. Camera calibrator <NUM> may be configured to improve or optimize player projection parameters using a homography matrix.

Player tracking agent <NUM> may be configured to generate tracks for each player on the playing surface. For example, player tracking agent <NUM> may leverage player pose detections, camera calibration, and broadcast frames to generate such tracks. Player tracking agent <NUM> may further be configured to generate tracks for each player, even if, for example, the player is currently out of a trackable frame. For example, player tracking agent <NUM> may utilize body pose information to link players that have left the frame of view.

Interface agent <NUM> may be configured to generate one or more graphical representations corresponding to the tracks for each player generated by player tracking agent <NUM>. For example, interface agent <NUM> may be configured to generate one or more graphical user interfaces (GUIs) that include graphical representations of player tracking each prediction generated by player tracking agent <NUM>.

Client device <NUM> may be in communication with organization computing system <NUM> via network <NUM>. Client device <NUM> may be operated by a user. For example, client device <NUM> may be a mobile device, a tablet, a desktop computer, or any computing system having the capabilities described herein. Users may include, but are not limited to, individuals such as, for example, subscribers, clients, prospective clients, or customers of an entity associated with organization computing system <NUM>, such as individuals who have obtained, will obtain, or may obtain a product, service, or consultation from an entity associated with organization computing system <NUM>.

Client device <NUM> may include at least application <NUM>. Application <NUM> may be representative of a web browser that allows access to a website or a stand-alone application. Client device <NUM> may access application <NUM> to access one or more functionalities of organization computing system <NUM>. Client device <NUM> may communicate over network <NUM> to request a webpage, for example, from web client application server <NUM> of organization computing system <NUM>. For example, client device <NUM> may be configured to execute application <NUM> to access content managed by web client application server <NUM>. The content that is displayed to client device <NUM> may be transmitted from web client application server <NUM> to client device <NUM>, and subsequently processed by application <NUM> for display through a graphical user interface (GUI) of client device <NUM>.

<FIG> is a block diagram illustrating a computing environment <NUM>, used by embodiments. As illustrated, computing environment <NUM> includes auto-clipping agent <NUM>, data set generator <NUM>, camera calibrator <NUM>, and player tracking agent <NUM> communicating via network <NUM>.

Auto-clipping agent <NUM> may include principal component analysis (PCA) agent <NUM>, clustering model <NUM>, and neural network <NUM>. As recited above, when trying to understand and extract data from a broadcast feed, auto-clipping agent <NUM> may be used to clip or segment the video into its constituent parts. Auto-clipping agent <NUM> may focus on separating a predefined, unified view (e.g., a high sideline view) from all other parts of the broadcast stream.

PCA agent <NUM> may be configured to utilize a PCA analysis to perform per frame feature extraction from the broadcast feed. For example, given a pre-recorded video, PCA agent <NUM> may extract a frame every X-seconds (e.g., <NUM> seconds) to build a PCA model of the video. PCA agent <NUM> may generate the PCA model using incremental PCA, through which PCA agent <NUM> may select a top subset of components (e.g., top <NUM> components) to generate the PCA model. PCA agent <NUM> may be further configured to extract one frame every X seconds (e.g., one second) from the broadcast stream and compress the frames using PCA model. PCA agent <NUM> may utilize PCA model to compress the frames into <NUM>-dimensional form. For example, PCA agent <NUM> may solve for the principal components in a per video manner and keep the top <NUM> components per frame to ensure accurate clipping.

Clustering model <NUM> may be configured to the cluster the top subset of components into clusters. For example, clustering model <NUM> may be configured to center, normalize, and cluster the top <NUM> components into a plurality of clustersFor clustering of compressed frames, clustering model <NUM> may implement k-means clustering. Clustering model <NUM> may set k = <NUM> clusters. K-means clustering attempts to take some data x = {x<NUM>, x<NUM>,. , xn} and divide it into k subsets, S = {S<NUM>, S<NUM>,. Sk} by optimizing: <MAT> where µj is the mean of the data in the set Sj. In other words, clustering model <NUM> attempts to find clusters with the smallest inter-cluster variance using k-means clustering techniques. Clustering model <NUM> may label each frame with its respective cluster number (e.g., cluster <NUM>, cluster <NUM>,. , cluster k).

Neural network <NUM> may be configured to classify each frame as trackable or untrackable. A trackable frame may be representative of a frame that includes captures the unified view (e.g., high sideline view). An untrackable frame may be representative of a frame that does not capture the unified view. To train neural network <NUM>, an input data set that includes thousands of frames pre-labeled as trackable or untrackable that are run through the PCA model may be used. Each compressed frame and label pair (i.e., cluster number and trackable/untrackable) may be provided to neural network <NUM> for training.

Neural network <NUM> may include four layers. The four layers may include an input layer, two hidden layers, and an output layer. Input layer may include <NUM> units. Each hidden layer may include <NUM> units. Output layer may include two units. The input layer and each hidden layer may use sigmoid activation functions. The output layer may use a SoftMax activation function. To train neural network <NUM>, auto-clipping agent <NUM> may reduce (e.g., minimize) the binary cross-entropy loss between the predicted label for sample ŷJ and the true label yj by: <MAT>.

Accordingly, once trained, neural network <NUM> may be configured to classify each frame as untrackable or trackable. As such, each frame may have two labels: a cluster number and trackable/untrackable classification. Auto-clipping agent <NUM> may utilize the two labels to determine if a given cluster is deemed trackable or untrackable. For example, if auto-clipping agent <NUM> determines that a threshold number of frames in a cluster are considered trackable (e.g., <NUM>%), auto-clipping agent <NUM> may conclude that all frames in the cluster are trackable. Further, if auto-clipping agent <NUM> determines that less than a threshold number of frames in a cluster are considered untrackable (e.g., <NUM>% and below), auto-clipping agent <NUM> may conclude that all frames in the cluster are untrackable. Still further, if auto-clipping agent <NUM> determines that a certain number of frames in a cluster are considered trackable (e.g., between <NUM>% and <NUM>%), auto-clipping agent <NUM> may request that an administrator further analyze the cluster. Once each frame is classified, auto-clipping agent <NUM> may clip or segment the trackable frames. Auto-clipping agent <NUM> may store the segments of trackable frames in database <NUM> associated therewith.

Data set generator <NUM> may be configured to generate a plurality of data sets from auto-clipping agent <NUM>. As illustrated, data set generator <NUM> may include pose detector <NUM>, ball detector <NUM>, and playing surface segmenter <NUM>. Pose detector <NUM> may be configured to detect players within the broadcast feed. Data set generator <NUM> may provide, as input, to pose detector <NUM> both the trackable frames stored in database <NUM> as well as the broadcast video feed. Pose detector <NUM> may implement Open Pose to generate body pose data to detect players in the broadcast feed and the trackable frames. Pose detector <NUM> may implement sensors positioned on players to capture body pose information. Generally, pose detector <NUM> may use any means to obtain body pose information from the broadcast video feed and the trackable frame. The output from pose detector <NUM> may be pose data stored in database <NUM> associated with data set generator <NUM>.

Ball detector <NUM> may be configured to detect and track the ball (or puck) within the broadcast feed. Data set generator <NUM> may provide, as input, to ball detector <NUM> both the trackable frames stored in database <NUM> and the broadcast video feed. Ball detector <NUM> may utilize a faster region-convolutional neural network (R-CNN) to detect and track the ball in the trackable frames and broadcast video feed. Faster R-CNN is a regional proposal based network. Faster R-CNN uses a convolutional neural network to propose a region of interest, and then classifies the object in each region of interest. Because it is a single unified network, the regions of interest and the classification steps may improve each other, thus allowing the classification to handle objects of various sizes. The output from ball detector <NUM> may be ball detection data stored in database <NUM> associated with data set generator <NUM>.

Playing surface segmenter <NUM> may be configured to identify playing surface markings in the broadcast feed. Data set generator <NUM> may provide, as input, to playing surface segmenter <NUM> both trackable frames stored in database <NUM> and the broadcast video feed. Playing surface segmenter <NUM> may be configured to utilize a neural network to identify playing surface markings. The output from playing surface segmenter <NUM> may be playing surface markings stored in database <NUM> associated with data set generator <NUM>.

Camera calibrator <NUM> may be configured to address the issue of moving camera calibration in sports. Camera calibrator <NUM> may include spatial transfer network <NUM> and optical flow module <NUM>. Camera calibrator <NUM> may receive, as input, segmented playing surface information generated by playing surface segmenter <NUM>, the trackable clip information, and posed information. Given such inputs, camera calibrator <NUM> may be configured to project coordinates in the image frame to real-world coordinates for tracking analysis.

Keyframe matching module <NUM> may receive, as input, output from playing surface segmenter <NUM> and a set of templates. For each frame, keyframe matching module <NUM> may match the output from playing surface segmenter <NUM> to a template. Those frames that are able to match to a given template are considered keyframes. Keyframe matching module <NUM> may implement a neural network to match the one or more frames. Keyframe matching module <NUM> may implement cross-correlation to match the one or more frames.

Spatial transformer network (STN) <NUM> may be configured to receive, as input, the identified keyframes from keyframe matching module <NUM>. STN <NUM> may implement a neural network to fit a playing surface model to segmentation information of the playing surface. By fitting the playing surface model to such output, STN <NUM> may generate homography matrices for each keyframe.

Optical flow module <NUM> may be configured to identify the pattern of motion of objects from one trackable frame to another. Optical flow module <NUM> may receive, as input, trackable frame information and body pose information for players in each trackable frame. Optical flow module <NUM> may use body pose information to remove players from the trackable frame information. Once removed, optical flow module <NUM> may determine the motion between frames to identify the motion of a camera between successive frames. In other words, optical flow module <NUM> may identify the flow field from one frame to the next.

Optical flow module <NUM> and STN <NUM> may work in conjunction to generate a homography matrix. For example, optical flow module <NUM> and STN <NUM> may generate a homography matrix for each trackable frame, such that a camera may be calibrated for each frame. The homography matrix may be used to project the track or position of players into real-world coordinates. For example, the homography matrix may indicate a <NUM>-dimensional to <NUM>-dimensional transform, which may be used to project the players' locations from image coordinates to the real world coordinates on the playing surface.

Player tracking agent <NUM> may be configured to generate a track for each player in a match. Player tracking agent <NUM> may include neural network <NUM> and re-identification agent <NUM>. Player tracking agent <NUM> may receive, as input, trackable frames, pose data, calibration data, and broadcast video frames. In a first phase, player tracking agent <NUM> may match pairs of player patches, which may be derived from pose information, based on appearance and distance. For example, let <MAT> be the player patch of the fth player at time t, and let <MAT> be the image coordinates <MAT>, the width <MAT>, and the height <MAT> of the jth player at time t. Using this, player tracking agent <NUM> may associate any pair of detections using the appearance cross correlation <MAT> and <MAT> by finding: <MAT> where I is the bounding box positions (x, y), width w, and height h; C is the cross correlation between the image patches (e.g., image cutout using a bounding box) and measures similarity between two image patches; and L is a measure of the difference (e.g., distance) between two bounding boxes I.

Performing this for every pair may generate a large set of short tracklets. The end points of these tracklets may then be associated with each other based on motion consistency and color histogram similarity.

For example, let vi be the extrapolated velocity from the end of the ith tracklet and vj be the velocity extrapolated from the beginning of the jth tracklet. Then cij = vi · vf may represent the motion consistency score. Furthermore, let p(h)i represent the likelihood of a color h being present in an image patch i. Player tracking agent <NUM> may measure the color histogram similarity using Bhattacharyya distance: <MAT>.

Recall, tracking agent <NUM> finds the matching pair of tracklets by finding: <MAT>.

Solving for every pair of broken tracklets may result in a set of clean tracklets, while leaving some tracklets with large, i.e., many frames, gaps. To connect the large gaps, player tracking agent may augment affinity measures to include a motion field estimation, which may account for the change of player direction that occurs over many frames.

The motion field may be a vector field that represents the velocity magnitude and direction as a vector on each location on the playing surface. Given the known velocity of a number of players on the playing surface, the full motion field may be generated using cubic spline interpolation. For example, let <MAT> to be the court position of a player i at every time t. Then, there may exist a pair of points that have a displacement <MAT>. Accordingly, the motion field may then be: <MAT> where G(x, <NUM>) may be a Gaussian kernel with standard deviation equal to about five feet. In other words, motion field may be a Gaussian blur of all displacements.

Neural network <NUM> may be used to predict player trajectories given ground truth player trajectories. Given a set of ground truth player trajectories, Xi, the velocity of each player at each frame may be calculated, which may provide the ground truth motion field for neural network <NUM> to learn. For example, given a set of ground truth player trajectories Xi, player tracking agent <NUM> may be configured to generate the set V̂(x, λ), where V̂(x, λ) may be the predicted motion field. Neural network <NUM> may be trained, for example, to minimize <MAT>. Player trajectory agent may then generate the affinity score for any tracking gap of size λ by: <MAT> where <MAT> is the displacement vector between all broken tracks with a gap size of λ.

Re-identification agent <NUM> may be configured to link players that have left the frame of view. Re-identification agent <NUM> may include track generator <NUM>, conditional autoencoder <NUM>, and Siamese network <NUM>.

Track generator <NUM> may be configured to generate a gallery of tracks. Track generator <NUM> may receive a plurality of tracks from database <NUM>. For each track X, there may include a player identity label y, and for each player patch I, pose information p may be provided by the pose detection stage. Given a set of player tracks, track generator <NUM> may build a gallery for each track where the jersey number of a player (or some other static feature) is always visible. The body pose information generated by data set generator <NUM> allows track generator <NUM> to determine a player's orientation. For example, track generator <NUM> may utilize a heuristic method, which may use the normalized shoulder width to determine the orientation: <MAT> where l may represent the location of one body part. The width of shoulder may be normalized by the length of the torso so that the effect of scale may be eliminated. As two shoulders should be apart when a player faces towards or backwards from the camera, track generator <NUM> may use those patches whose Sorient is larger than a threshold to build the gallery. After this stage, each track Xn, may include a gallery: <MAT>.

Conditional autoencoder <NUM> may be configured to identify one or more features in each track. For example, unlike conventional approaches to re-identification issues, players in team sports may have very similar appearance features, such as clothing style, clothing color, and skin color. One of the more intuitive differences may be the jersey number that may be shown at the front and/or back side of each jersey. In order to capture those specific features, conditional autoencoder <NUM> may be trained to identify such features.

Conditional autoencoder <NUM> may be a three-layer convolutional autoencoder, where the kernel sizes may be 3x3 for all three layers, in which there are <NUM>, <NUM>, <NUM> channels respectively. Those hyper-parameters may be tuned to ensure that jersey number may be recognized from the reconstructed images so that the desired features may be learned in the autoencoder. f(Ii) may be used to denote the features that are learned from image i.

Use of conditional autoencoder <NUM> improves upon conventional processes for a variety of reasons. First, there is typically not enough training data for every player because some players only play a very short time in each game. Second, different teams can have the same jersey colors and jersey numbers, so classifying those players may be difficult.

Siamese network <NUM> may be used to measure the similarity between two image patches. For example, Siamese network <NUM> may be trained to measure the similarity between two image patches based on their feature representations f(I). Given two image patches, their feature representations f(Ii) and f(Ij) may be flattened, connected, and input into a perception network. L<NUM> norm may be used to connect the two sub-networks of f(Ii) and f(Ij). Perception network may include three layers, which include may <NUM>, <NUM>, and <NUM> hidden units, respectively. Such network may be used to measure the similarity s(Ii, Ij) between every pair of image patches of the two tracks that have no time overlapping. In order to increase the robustness of the prediction, the final similarity score of the two tracks may be the average of all pairwise scores in their respective galleries: <MAT>.

This similarity score may be computed for every two tracks that do not have time overlapping. If the score is higher than some threshold, those two tracks may be associated.

<FIG> is a block diagram <NUM> illustrating parts of the operations discussed above and below in conjunction with <FIG> and <FIG>, according to embodiments. Block diagram <NUM> may illustrate the overall workflow of organization computing system <NUM> in generating player tracking information. Block diagram <NUM> may include set of operations <NUM>-<NUM>. Set of operations <NUM> may be directed to generating trackable frames (e.g., Method <NUM> in <FIG>). Set of operations <NUM> may be directed to generating one or more data sets from trackable frames (e.g., operations performed by data set generator <NUM>). Set of operations <NUM> may be directed to camera calibration operations (e.g., Method <NUM> in <FIG>). Set of operations <NUM> may be directed to generating and predicting player tracks (e.g., Method <NUM> if <FIG> and Method <NUM> in <FIG>).

<FIG> is a flow diagram illustrating a method <NUM> of generating player tracks, according to embodiments. Method <NUM> may begin at step <NUM>.

At step <NUM>, organization computing system <NUM> may receive (or retrieve) a broadcast feed for an event. The broadcast feed may be a live feed received in real-time (or near real-time) from camera system 102The broadcast feed may be a broadcast feed of a game that has concluded. Generally, the broadcast feed may include a plurality of frames of video data. Each frame may capture a different camera perspective.

At step <NUM>, organization computing system <NUM> may segment the broadcast feed into a unified view. For example, auto-clipping agent <NUM> may be configured to parse the plurality of frames of data in the broadcast feed to segment the trackable frames from the untrackable frames. Generally, trackable frames may include those frames that are directed to a unified view. For example, the unified view may be considered a high sideline view. In other examples, the unified view may be an endzone view. In other examples, the unified view may be a top camera view.

At step <NUM>, organization computing system <NUM> may generate a plurality of data sets from the trackable frames (i.e., the unified view). For example, fata set generator <NUM> may be configured to generate a plurality of data sets based on trackable clips received from auto-clipping agent <NUM>. Pose detector <NUM> may be configured to detect players within the broadcast feed. Data set generator <NUM> may provide, as input, to pose detector <NUM> both the trackable frames stored in database <NUM> as well as the broadcast video feed. The output from pose detector <NUM> may be pose data stored in database <NUM> associated with data set generator <NUM>.

Ball detector <NUM> may be configured to detect and track the ball (or puck) within the broadcast feed. Data set generator <NUM> may provide, as input, to ball detector <NUM> both the trackable frames stored in database <NUM> and the broadcast video feed. Ball detector <NUM> may utilize a faster R-CNN to detect and track the ball in the trackable frames and broadcast video feed. The output from ball detector <NUM> may be ball detection data stored in database <NUM> associated with data set generator <NUM>.

Accordingly, data set generator <NUM> may generate information directed to player location, ball location, and portions of the court in all trackable frames for further analysis.

At step <NUM>, organization computing system <NUM> may calibrate the camera in each trackable frame based on the data sets generated in step <NUM>. For example, camera calibrator <NUM> may be configured to calibrate the camera in each trackable frame by generating a homography matrix, using the trackable frames and body pose information. The homography matrix allows camera calibrator <NUM> to take those trajectories of each player in a given frame and project those trajectories into real-world coordinates. By projection player position and trajectories into real world coordinates for each frame, camera calibrator <NUM> may ensure that the camera is calibrated for each frame.

At step <NUM>, organization computing system <NUM> may be configured to generate or predict a track for each player. For example, player tracking agent <NUM> may be configured to generate or predict a track for each player in a match. Player tracking agent <NUM> may receive, as input, trackable frames, pose data, calibration data, and broadcast video frames. Using such inputs, player tracking agent <NUM> may be configured to construct player motion throughout a given match. Further, player tracking agent <NUM> may be configured to predict player trajectories given previous motion of each player.

<FIG> is a flow diagram illustrating a method <NUM> of generating trackable frames, according to embodiments. Method <NUM> may correspond to operation <NUM> discussed above in conjunction with <FIG>. Method <NUM> may begin at step <NUM>.

At step <NUM>, organization computing system <NUM> may receive (or retrieve) a broadcast feed for an event. The broadcast feed may be a live feed received in real-time (or near real-time) from camera system <NUM>. The broadcast feed may be a broadcast feed of a game that has concluded. Generally, the broadcast feed may include a plurality of frames of video data. Each frame may capture a different camera perspective.

At step <NUM>, organization computing system <NUM> may generate a set of frames for image classification. For example, auto-clipping agent <NUM> may utilize a PCA analysis to perform per frame feature extraction from the broadcast feed. Given, for example, a pre-recorded video, auto-clipping agent <NUM> may extract a frame every X-seconds (e.g., <NUM> seconds) to build a PCA model of the video. Auto-clipping agent <NUM> may generate the PCA model using incremental PCA, through which auto-clipping agent <NUM> may select a top subset of components (e.g., top <NUM> components) to generate the PCA model. Auto-clipping agent <NUM> may be further configured to extract one frame every X seconds (e.g., one second) from the broadcast stream and compress the frames using PCA model. Auto-clipping agent <NUM> may utilize PCA model to compress the frames into <NUM>-dimensional form. For example, auto-clipping agent <NUM> may solve for the principal components in a per video manner and keep the top <NUM> components per frame to ensure accurate clipping. Such subset of compressed frames may be considered the set of frames for image classification. In other words, PCA model may be used to compress each frame to a small vector, so that clustering can be conducted on the frames more efficiently. The compression may be conducted by selecting the top N components from PCA model to represent the frame. In some examples, N may be <NUM>.

At step <NUM>, organization computing system <NUM> may assign each frame in the set of frames to a given cluster. For example, auto-clipping agent <NUM> may be configured to center, normalize, and cluster the top <NUM> components into a plurality of clusters. For clustering of compressed frames, auto-clipping agent <NUM> may implement k-means clustering. Auto-clipping agent <NUM> may set k = <NUM> clusters. K-means clustering attempts to take some data x = {x<NUM>, x<NUM>,. , xn} and divide it into k subsets, S = {S<NUM>, S<NUM>,. Sk} by optimizing: <MAT> where µj is the mean of the data in the set Sj. In other words, clustering model <NUM> attempts to find clusters with the smallest inter-cluster variance using k-means clustering techniques. Clustering model <NUM> may label each frame with its respective cluster number (e.g., cluster <NUM>, cluster <NUM>,. , cluster k).

At step <NUM>, organization computing system <NUM> may classify each frame as trackable or untrackable. For example, auto-clipping agent <NUM> may utilize a neural network to classify each frame as trackable or untrackable. A trackable frame may be representative of a frame that includes captures the unified view (e.g., high sideline view). An untrackable frame may be representative of a frame that does not capture the unified view. To train the neural network (e.g., neural network <NUM>), an input data set that includes thousands of frames pre-labeled as trackable or untrackable that are run through the PCA model may be used. Each compressed frame and label pair (i.e., cluster number and trackable/untrackable) may be provided to neural network for training. Accordingly, once trained, auto-clipping agent <NUM> may classify each frame as untrackable or trackable. As such, each frame may have two labels: a cluster number and trackable/untrackable classification.

At step <NUM>, organization computing system <NUM> may compare each cluster to a threshold. For example, auto-clipping agent <NUM> may utilize the two labels to determine if a given cluster is deemed trackable or untrackable. If auto-clipping agent <NUM> determines that a threshold number of frames in a cluster are considered trackable (e.g., <NUM>%), auto-clipping agent <NUM> may conclude that all frames in the cluster are trackable. If auto-clipping agent <NUM> determines that less than a threshold number of frames in a cluster are considered untrackable (e.g., <NUM>% and below), auto-clipping agent <NUM> may conclude that all frames in the cluster are untrackable. Still further, if auto-clipping agent <NUM> determines that a certain number of frames in a cluster are considered trackable (e.g., between <NUM>% and <NUM>%), auto-clipping agent <NUM> may request that an administrator further analyze the cluster.

If at step <NUM> organization computing system <NUM> determines that greater than a threshold number of frames in the cluster are trackable, then at step <NUM> auto-clipping agent <NUM> may classify the cluster as trackable.

If, however, at step <NUM> organization computing system <NUM> determines that less than a threshold number of frames in the cluster are trackable, then at step <NUM>, auto-clipping agent <NUM> may classify the cluster as untrackable.

<FIG> is a block diagram <NUM> illustrating parts of the operations discussed above in conjunction with method <NUM>, according to embodiments. As shown, block diagram <NUM> may include a plurality of sets of operations <NUM>-<NUM>.

At set of operations <NUM>, video data (e.g., broadcast video) may be provided to auto-clipping agent <NUM>. Auto-clipping agent <NUM> may extract frames from the video. Auto-clipping agent <NUM> may extract frames from the video at a low frame rate. An incremental PCA algorithm may be used by auto-clipping agent to select the top <NUM> components (e.g., frames) from the set of frames extracted by auto-clipping agent <NUM>. Such operations may generate a video specific PCA model.

At set of operations <NUM>, video data (e.g., broadcast video) may be provided to auto-clipping agent <NUM>. Auto-clipping agent <NUM> may extract frames from the video. Auto-clipping agent <NUM> may extract frames from the video at a medium frame rate. The video specific PCA model may be used by auto-clipping agent <NUM> to compress the frames extracted by auto-clipping agent <NUM>.

At set of operations <NUM>, the compressed frames and a pre-selected number of desired clusters may be provided to auto-clipping agent <NUM>. Auto-clipping agent <NUM> may utilize k-means clustering techniques to group the frames into one or more clusters, as set forth by the pre-selected number of desired clusters. Auto-clipping agent <NUM> may assign a cluster label to each compressed frames. Auto-clipping agent <NUM> may further be configured to classify each frame as trackable or untrackable. Auto-clipping agent <NUM> may label each respective frame as such.

At set of operations <NUM>, auto-clipping agent <NUM> may analyze each cluster to determine if the cluster includes at least a threshold number of trackable frames. For example, as illustrated, if <NUM>% of the frames of a cluster are classified as trackable, then auto-clipping agent <NUM> may consider the entire cluster as trackable. If, however, less than <NUM>% of a cluster is classified as trackable, auto-clipping agent may determine if at least a second threshold number of frames in a cluster are trackable. For example, is illustrated if <NUM>% of the frames of a cluster are classified as untrackable, auto-clipping agent <NUM> may consider the entire cluster trackable. If, however, less than <NUM>% of the frames of the cluster are classified as untrackable, i.e., between <NUM>% and <NUM>% trackable, then human annotation may be requested.

<FIG> is a flow diagram illustrating a method <NUM> of calibrating a camera for each trackable frame, according to embodiments. Method <NUM> may correspond to operation <NUM> discussed above in conjunction with <FIG>. Method <NUM> may begin at step <NUM>.

At step <NUM>, organization computing system <NUM> may retrieve video data and pose data for analysis. For example, camera calibrator <NUM> may retrieve from database <NUM> the trackable frames for a given match and pose data for players in each trackable frame. Following step <NUM>, camera calibrator <NUM> may execute two parallel processes to generate homography matrix for each frame. Accordingly, the following operations are not meant to be discussed as being performed sequentially, but may instead be performed in parallel or sequentially.

At step <NUM>, organization computing system <NUM> may remove players from each trackable frame. For example, camera calibrator <NUM> may parse each trackable frame retrieved from database <NUM> to identify one or more players contained therein. Camera calibrator <NUM> may remove the players from each trackable frame using the pose data retrieved from database <NUM>. For example, camera calibrator <NUM> may identify those pixels corresponding to pose data and remove the identified pixels from a given trackable frame.

At step <NUM>, organization computing system <NUM> may identify the motion of objects (e.g., surfaces, edges, etc.) between successive trackable frames. For example, camera calibrator <NUM> may analyze successive trackable frames, with players removed therefrom, to determine the motion of objects from one frame to the next. In other words, optical flow module <NUM> may identify the flow field between successive trackable frames.

At step <NUM>, organization computing system <NUM> may match an output from playing surface segmenter <NUM> to a set of templates. For example, camera calibrator <NUM> may match one or more frames in which the image of the playing surface is clear to one or more templates. Camera calibrator <NUM> may parse the set of trackable clips to identify those clips that provide a clear picture of the playing surface and the markings therein. Based on the selected clips, camera calibrator <NUM> may compare such images to playing surface templates. Each template may represent a different camera perspective of the playing surface. Those frames that are able to match to a given template are considered keyframes. Camera calibrator <NUM> may implement a neural network to match the one or more frames. Camera calibrator <NUM> may implement cross-correlation to match the one or more frames.

At step <NUM>, organization computing system <NUM> may fit a playing surface model to each keyframe. For example, camera calibrator <NUM> may be configured to receive, as input, the identified keyframes. Camera calibrator <NUM> may implement a neural network to fit a playing surface model to segmentation information of the playing surface. By fitting the playing surface model to such output, camera calibrator <NUM> may generate homography matrices for each keyframe.

At step <NUM>, organization computing system <NUM> may generate a homography matrix for each trackable frame. For example, camera calibrator <NUM> may utilize the flow fields identified in step <NUM> and the homography matrices for each key frame to generate a homography matrix for each frame. The homography matrix may be used to project the track or position of players into real-world coordinates. For example, given the geometric transform represented by the homography matrix, camera calibrator <NUM> may use his transform to project the location of players on the image to real-world coordinates on the playing surface.

At step <NUM>, organization computing system <NUM> may calibrate each camera based on the homography matrix.

<FIG> is a block diagram <NUM> illustrating aspects of operations discussed above in conjunction with method <NUM>, according to embodiments. As shown, block diagram <NUM> may include inputs <NUM>, a first set of operations <NUM>, and a second set of operations <NUM>. First set of operations <NUM> and second set of operations <NUM> may be performed in parallel.

Inputs <NUM> may include video clips <NUM> and pose detection <NUM>. Video clips <NUM> may correspond to trackable frames generated by auto-clipping agent <NUM>. Pose detection <NUM> may correspond to pose data generated by pose detector <NUM>. As illustrated, only video clips <NUM> may be provided as input to first set of operations <NUM>; both video clips <NUM> and post detection <NUM> may be provided as input to second set of operations <NUM>.

First set of operations <NUM> may include semantic segmentation <NUM>, keyframe matching <NUM>, and STN fitting <NUM>. At semantic segmentation <NUM>, playing surface segmenter <NUM> may be configured to identify playing surface markings in a broadcast feed. Playing surface segmenter <NUM> may be configured to utilize a neural network to identify playing surface markings. Such segmentation information may be performed in advance and provided to camera calibration <NUM> from database <NUM>. At keyframe matching <NUM>, keyframe matching module <NUM> may be configured to match one or more frames in which the image of the playing surface is clear to one or more templates. At STN fitting <NUM>, STN <NUM> may implement a neural network to fit a playing surface model to segmentation information of the playing surface. By fitting the playing surface model to such output, STN <NUM> may generate homography matrices for each keyframe.

Second set of operations <NUM> may include camera motion estimation <NUM>. At camera flow estimation <NUM>, optical flow module <NUM> may be configured to identify the pattern of motion of objects from one trackable frame to another. For example, optical flow module <NUM> may use body pose information to remove players from the trackable frame information. Once removed, optical flow module <NUM> may determine the motion between frames to identify the motion of a camera between successive frames.

First set of operations <NUM> and second set of operations <NUM> may lead to homography interpolation <NUM>. Optical flow module <NUM> and STN <NUM> may work in conjunction to generate a homography matrix for each trackable frame, such that a camera may be calibrated for each frame. The homography matrix may be used to project the track or position of players into real-world coordinates.

<FIG> is a flow diagram illustrating a method <NUM> of tracking players, according to embodiments. Method <NUM> may correspond to operation <NUM> discussed above in conjunction with <FIG>. Method <NUM> may begin at step <NUM>.

At step <NUM>, organization computing system <NUM> may retrieve a plurality of trackable frames for a match. Each of the plurality of trackable frames may include one or more sets of metadata associated therewith. Such metadata may include, for example, body pose information and camera calibration data. Player tracking agent <NUM> may further retrieve broadcast video data.

At step <NUM>, organization computing system <NUM> may generate a set of short tracklets. For example, player tracking agent <NUM> may match pairs of player patches, which may be derived from pose information, based on appearance and distance to generate a set of short tracklets. For example, let <MAT> be the player patch of the jth player at time t, and let <MAT> be the image coordinates <MAT>, the width <MAT>, and the height <MAT> of the jth player at time t. Using this, player tracking agent <NUM> may associated any pair of detections using the appearance cross correlation <MAT> and <MAT> by finding: <MAT>.

Performing this for every pair may generate a set of short tracklets. The end points of these tracklets may then be associated with each other based on motion consistency and color histogram similarity.

For example, let vi be the extrapolated velocity from the end of the ith tracklet and vj be the velocity extrapolated from the beginning of the fth tracklet. Then cij = vi · vj may represent the motion consistency score. Furthermore, let p(h)i represent the likelihood of a color h being present in an image patch i. Player tracking agent <NUM> may measure the color histogram similarity using Bhattacharyya distance: <MAT>.

At step <NUM>, organization computing system <NUM> may connect gaps between each set of short tracklets. For example, recall that tracking agent <NUM> finds the matching pair of tracklets by finding: <MAT>.

Solving for every pair of broken tracklets may result in a set of clean tracklets, while leaving some tracklets with large, i.e., many frames, gaps. To connect the large gaps, player tracking agent <NUM> may augment affinity measures to include a motion field estimation, which may account for the change of player direction that occurs over many frames.

The motion field may be a vector field which measures what direction a player at a point on the playing surface x would be after some time λ. For example, let <MAT> to be the court position of a player i at every time t. Then, there may exist a pair of points that have a displacement <MAT>. Accordingly, the motion field may then be: <MAT> where G(x, <NUM>) may be a Gaussian kernel with standard deviation equal to about five feet. In other words, motion field may be a Gaussian blur of all displacements.

At step <NUM>, organization computing system <NUM> may predict a motion of an agent based on the motion field. For example, player tracking system <NUM> may use a neural network (e.g., neural network <NUM>) to predict player trajectories given ground truth player trajectory. Given a set of ground truth player trajectories Xi, player tracking agent <NUM> may be configured to generate the set V̂(x, λ), where V̂(x, λ) may be the predicted motion field. Player tracking agent <NUM> may train neural network <NUM> to reduce (e.g., minimize) <MAT>. Player tracking agent <NUM> may then generate the affinity score for any tracking gap of size λ by: <MAT>
where <MAT> is the displacement vector between all broken tracks with a gap size of λ. Accordingly, player tracking agent <NUM> may solve for the matching pairs as recited above. For example, given the affinity score, player tracking agent <NUM> may assign every pair of broken tracks using a Hungarian algorithm. The Hungarian algorithm (e.g., Kuhn-Munkres) may optimize the best set of matches under a constraint that all pairs are to be matched.

At step <NUM>, organization computing system <NUM> may output a graphical representation of the prediction. For example, interface agent <NUM> may be configured to generate one or more graphical representations corresponding to the tracks for each player generated by player tracking agent <NUM>. For example, interface agent <NUM> may be configured to generate one or more graphical user interfaces (GUIs) that include graphical representations of player tracking each prediction generated by player tracking agent <NUM>.

In some situations, during the course of a match, players or agents have the tendency to wander outside of the point-of-view of camera. Such issue may present itself during an injury, lack of hustle by a player, quick turnover, quick transition from offense to defense, and the like. Accordingly, a player in a first trackable frame may no longer be in a successive second or third trackable frame. Player tracking agent <NUM> may address this issue via re-identification agent <NUM>.

At step <NUM>, organization computing system <NUM> may identify a subset of short tracks in which a player has left the camera's line of vision. Each track may include a plurality of image patches associated with at least one player. An image patch may refer to a subset of a corresponding frame of a plurality of trackable frames. Each track X may include a player identity label y. Each player patch I in a given track X may include pose information generated by data set generator <NUM>. For example, given an input video, pose detection, and trackable frames, re-identification agent <NUM> may generate a track collection that includes a lot of short broken tracks of players.

At step <NUM>, organization computing system <NUM> generates a gallery for each track. For example, given those small tracks, re-identification agent <NUM> may build a gallery for each track. Re-identification agent <NUM> may build a gallery for each track where the jersey number of a player (or some other static feature) is always visible. The body pose information generated by data set generator <NUM> allows re-identification agent <NUM> to determine each player's orientation. For example, re-identification agent <NUM> may utilize a heuristic method, which may use the normalized shoulder width to determine the orientation: <MAT> where l may represent the location of one body part. The width of shoulder may be normalized by the length of the torso so that the effect of scale may be eliminated. As two shoulders should be apart when a player faces towards or backwards from the camera, re-identification agent <NUM> may use those patches whose Sorient is larger than a threshold to build the gallery. Accordingly, each track Xn, may include a gallery: <MAT>.

At step <NUM>, organization computing system <NUM> matches tracks using a convolutional autoencoder. For example, re-identification agent <NUM> may use conditional autoencoder (e.g., conditional autoencoder <NUM>) to identify one or more features in each track. For example, unlike conventional approaches to re-identification issues, players in team sports may have very similar appearance features, such as clothing style, clothing color, and skin color. One of the more intuitive differences may be the jersey number that may be shown at the front and/or back side of each jersey. In order to capture those specific features, re-identification agent <NUM> may train conditional autoencoder to identify such features.

Conditional autoencoder may be a three-layer convolutional autoencoder, where the kernel sizes may be 3x3 for all three layers, in which there are <NUM>, <NUM>, <NUM> channels respectively. Those hyper-parameters may be tuned to ensure that jersey number may be recognized from the reconstructed images so that the desired features may be learned in the autoencoder. f(Ii) may be used to denote the features that are learned from image i.

Using a specific example, re-identification agent <NUM> may identify a first track that corresponds to a first player. Using conditional autoencoder <NUM>, re-identification agent <NUM> may learn a first set of jersey features associated with the first track, based on for example, a first set of image patches included or associated with the first track. Re-identification agent <NUM> may further identify a second track that may initially correspond to a second player. Using conditional autoencoder <NUM>, re-identification agent <NUM> may learn a second set of jersey features associated with the second track, based on, for example, a second set of image patches included or associated with the second track.

At step <NUM>, organization computing system <NUM> measures a similarity between matched tracks using a Siamese network. For example, re-identification agent <NUM> may train Siamese network (e.g., Siamese network <NUM>) to measure the similarity between two image patches based on their feature representations f(I). Given two image patches, their feature representations f(Ii) and f(Ij) may be flattened, connected, and fed into a perception network. L<NUM> norm may be used to connect the two sub-networks of f(Ii) and f(Ij). Perception network may include three layers, which include <NUM>, <NUM>, and <NUM> hidden units, respectively. Such network is used to measure the similarity s(Ii, Ij) between every pair of image patches of the two tracks that have no time overlapping. In order to increase the robustness of the prediction, the final similarity score of the two tracks may be the average of all pairwise scores in their respective galleries: <MAT>.

Continuing with the aforementioned example, re-identification agent <NUM> may utilize Siamese network <NUM> to compute a similarity score between the first set of learned jersey features and the second set of learned jersey features.

At step <NUM>, organization computing system <NUM> associates the tracks if their similarity score is higher than a predetermined threshold. For example, re-identification agent <NUM> may compute a similarity score for every two tracks that do not have time overlapping. If the score is higher than some threshold, re-identification agent <NUM> associates those two tracks.

Continuing with the above example, re-identification agent associates first track and the second track if, for example, the similarity score generated by Siamese network <NUM> is at least higher than a threshold value. Assuming the similarity score is higher than the threshold value, re-identification agent <NUM> determines that the first player in the first track and the second player in the second track are indeed one and the same.

<FIG> is a block diagram <NUM> illustrating aspects of operations discussed above in conjunction with method <NUM>, according to embodiments.

As shown block diagram <NUM> may include input video <NUM>, pose detection <NUM>, player tracking <NUM>, track collection <NUM>, gallery building and pairwise matching <NUM>, and track connection <NUM>. Block diagram <NUM> illustrates a general pipeline of method <NUM> provided above.

Given input video <NUM>, pose detection information <NUM> (e.g., generated by pose detector <NUM>), and player tracking information <NUM> (e.g., generated by one or more of player tracking agent <NUM>, auto-clipping agent <NUM>, and camera calibrator <NUM>), re-identification agent <NUM> may generate track collection <NUM>. Each track collection <NUM> may include a plurality of short broken tracks (e.g., track <NUM>) of players. Each track <NUM> may include one or more image patches <NUM> contained therein. Given the tracks <NUM>, re-identification agent <NUM> may generate a gallery <NUM> for each track. For example, gallery <NUM> may include those image patches <NUM> in a given track that include an image of a player in which their orientation satisfies a threshold value. In other words, re-identification agent <NUM> may generate gallery <NUM> for each track <NUM> that includes image patches <NUM> of each player, such that the player's number may be visible in each frame. Image patches <NUM> may be a subset of image patches <NUM>. Re-identification agent <NUM> then can pairwise match each frame to compute a similarity score via Siamese network. For example, as illustrated, re-identification agent <NUM> may match a first frame from track <NUM> with a second frame from track <NUM> and feed the frames into Siamese network.

Re-identification agent <NUM> may then connect tracks <NUM> based on the similarity scores. For example, if the similarity score of the two frames exceeds some threshold, re-identification agent <NUM> will connect or associate those tracks.

<FIG> is a block diagram illustrating architecture <NUM> of Siamese network <NUM> of re-identification agent <NUM>, according to embodiments. As illustrated, Siamese network <NUM> may include two sub-networks <NUM>, <NUM>, and a perception network <NUM>.

Each of two sub-networks <NUM>, <NUM> may be configured similarly. For example, sub-network <NUM> may include a first convolutional layer <NUM>, a second convolutional layer <NUM>, and a third convolutional layer <NUM>. First sub-network <NUM> may receive, as input, a player patch I<NUM> and output a set of features learned from player patch I<NUM> (denoted f(I<NUM>)). Sub-network <NUM> may include a first convolutional layer <NUM>, a second convolutional layer <NUM>, and a third convolutional layer <NUM>. Second sub-network <NUM> may receive, as input, a player patch I<NUM> and may output a set of features learned from player patch I<NUM> (denoted f(I<NUM>)). The output from sub-network <NUM> and sub-network <NUM> may be an encoded representation of the respective player patches I<NUM>, I<NUM>. The output from sub-network <NUM> and sub-network <NUM> may be followed by a flatten operation, which may generate respective feature vectors f(I<NUM>) and f(I<NUM>), respectively. Each feature vector f(I<NUM>) and f(I<NUM>) may include <NUM> units. The L2 norm of f(I<NUM>) and f(I<NUM>) may be computed and used as input to perception network <NUM>.

Perception network <NUM> may include three layers <NUM>-<NUM>. Layer <NUM> may include <NUM> hidden units. Layer <NUM> may include <NUM> hidden units. Layer <NUM> may include <NUM> hidden units. Perception network <NUM> may output a similarity score between image patches I<NUM> and I<NUM>.

<FIG> illustrates a system bus computing system architecture <NUM>, used with embodiments. System <NUM> may be representative of at least a portion of organization computing system <NUM>. One or more components of system <NUM> may be in electrical communication with each other using a bus <NUM>. System <NUM> may include a processing unit (CPU or processor) <NUM> and a system bus <NUM> that couples various system components including the system memory <NUM>, such as read only memory (ROM) <NUM> and random access memory (RAM) <NUM>, to processor <NUM>. System <NUM> may include a cache of highspeed memory connected directly with, in close proximity to, or integrated as part of processor <NUM>. System <NUM> may copy data from memory <NUM> and/or storage device <NUM> to cache <NUM> for quick access by processor <NUM>. In this way, cache <NUM> may provide a performance boost that avoids processor <NUM> delays while waiting for data. These and other modules may control or be configured to control processor <NUM> to perform various actions. Other system memory <NUM> may be available for use as well. Memory <NUM> may include multiple different types of memory with different performance characteristics. Processor <NUM> may include any general purpose processor and a hardware module or software module, such as service <NUM><NUM>, service <NUM><NUM>, and service <NUM><NUM> stored in storage device <NUM>, configured to control processor <NUM> as well as a special-purpose processor where software instructions are incorporated into the actual processor design. Processor <NUM> may essentially be a completely self-contained computing system, containing multiple cores or processors, a bus, memory controller, cache, etc. A multi-core processor may be symmetric or asymmetric.

To enable user interaction with the computing device <NUM>, an input device <NUM> may represent any number of input mechanisms, such as a microphone for speech, a touch-sensitive screen for gesture or graphical input, keyboard, mouse, motion input, speech and so forth. An output device <NUM> may also be one or more of a number of output mechanisms known to those of skill in the art. In some instances, multimodal systems may enable a user to provide multiple types of input to communicate with computing device <NUM>. Communications interface <NUM> may generally govern and manage the user input and system output. There is no restriction on operating on any particular hardware arrangement and therefore the basic features here may easily be substituted for improved hardware or firmware arrangements as they are developed.

Storage device <NUM> may be a non-volatile memory and may be a hard disk or other types of computer readable media which may store data that are accessible by a computer, such as magnetic cassettes, flash memory cards, solid state memory devices, digital versatile disks, cartridges, random access memories (RAMs) <NUM>, read only memory (ROM) <NUM>, and hybrids thereof.

Storage device <NUM> may include services <NUM>, <NUM>, and <NUM> for controlling the processor <NUM>. Other hardware or software modules are contemplated. Storage device <NUM> may be connected to system bus <NUM>. In one aspect, a hardware module that performs a particular function may include the software component stored in a computer-readable medium in connection with the necessary hardware components, such as processor <NUM>, bus <NUM>, display <NUM>, and so forth, to carry out the function.

<FIG> illustrates a computer system <NUM> having a chipset architecture that may represent at least a portion of organization computing system <NUM>. Computer system <NUM> may be an example of computer hardware, software, and firmware that may be used to implement the disclosed technology. System <NUM> may include a processor <NUM>, representative of any number of physically and/or logically distinct resources capable of executing software, firmware, and hardware configured to perform identified computations. Processor <NUM> may communicate with a chipset <NUM> that may control input to and output from processor <NUM>. In this example, chipset <NUM> outputs information to output <NUM>, such as a display, and may read and write information to storage device <NUM>, which may include magnetic media, and solid state media, for example. Chipset <NUM> may also read data from and write data to RAM <NUM>. A bridge <NUM> for interfacing with a variety of user interface components <NUM> may be provided for interfacing with chipset <NUM>. Such user interface components <NUM> may include a keyboard, a microphone, touch detection and processing circuitry, a pointing device, such as a mouse, and so on. In general, inputs to system <NUM> may come from any of a variety of sources, machine generated and/or human generated.

Chipset <NUM> may also interface with one or more communication interfaces <NUM> that may have different physical interfaces. Such communication interfaces may include interfaces for wired and wireless local area networks, for broadband wireless networks, as well as personal area networks. Some applications of the methods for generating, displaying, and using the GUI disclosed herein may include receiving ordered datasets over the physical interface or be generated by the machine itself by processor <NUM> analyzing data stored in storage <NUM> or <NUM>. Further, the machine may receive inputs from a user through user interface components <NUM> and execute appropriate functions, such as browsing functions by interpreting these inputs using processor <NUM>.

It may be appreciated that example systems <NUM> and <NUM> may have more than one processor <NUM> or be part of a group or cluster of computing devices networked together to provide greater processing capability.

Claim 1:
A method of re-identifying players in a broadcast video feed, comprising:
retrieving, by a computing system (<NUM>), a broadcast video feed for a sporting event, the broadcast video feed comprising a plurality of video frames;
identifying, by the computing system (<NUM>), a plurality of image patches associated with a plurality of players across the plurality of video frames, each image patch of the plurality of image patches being a subset of the corresponding frame of the plurality of video frames;
generating, by the computing system (<NUM>), a plurality of tracks for the plurality of players, each tracking of the plurality of tracks associated with a corresponding player of the plurality of players, wherein each track of the plurality of tracks comprises a subset of image patches associated with a corresponding player; for each track, generating, by the computing system (<NUM>), a gallery of image patches wherein a jersey number of each player is visible in each image patch of the gallery;
matching, by the computing system (<NUM>) via a convolutional autoencoder, tracks across galleries;
measuring, by the computing system (<NUM>) via a neural network (<NUM>), a similarity score for each matched track;
determining, by the computing system (<NUM>), that two tracks have a similarity score above a threshold value; and
responsive to determining that the two tracks have the similarity score above the threshold value, associating, by the computing system (<NUM>), the two tracks.