Active Learning of Ordinal Embeddings: A User Study on Football Data
Authors: Christoffer Löffler
Kion Fallah
Stefano Fenu
Dario Zanca
Bjoern Eskofier
Christopher John Rozell
Christopher Mutschler
Venue: Accepted by TMLR ()

Abstract: Humans innately measure distance between instances in an unlabeled dataset using an unknown similarity function. Distance metrics can only serve as proxy for similarity in information retrieval of similar instances. Learning a good similarity function from human annotations improves the quality of retrievals. This work uses deep metric learning to learn these user-defined similarity functions from few annotations for a large football trajectory dataset.
We adapt an entropy-based active learning method with recent work from triplet mining to collect easy-to-answer but still informative annotations from human participants and use them to train a deep convolutional network that generalizes to unseen samples. 
Our user study shows that our approach improves the quality of the information retrieval compared to a previous deep metric learning approach that relies on a Siamese network. Specifically, we shed light on the strengths and weaknesses of passive sampling heuristics and active learners alike by analyzing the participants' response efficacy. To this end, we collect accuracy, algorithmic time complexity, the participants' fatigue and time-to-response, qualitative self-assessment and statements, as well as the effects of mixed-expertise annotators and their consistency on model performance and transfer-learning.


Published in Transactions on Machine Learning Research (04/2023)
Active Learning of Ordinal Embeddings: A User Study on Football Data
Christoffer Löffler
christoffer. dorio. sanca zanca@fou. de bjoern. eskofier@for de
Dario Zanca²
Björn Eskofier
FFriddrch Alezaader- Universität (llangnn-Nürbbee Machine Learaing and Data Analytics (MaD) Lab,
Carl- Siiershh-Straße 2b, 91052 Erlangen, Germany
Kion Fallah
kion &gattch edu sfenus @gatech. edu crozell@gatecl edu
Stefano Fenu
Christopher Rozell
School of Electrical and Computer Engineering
School of Interactive Computing
Georgia Institute of Technology, Atlanta, GA
Christopher Mutschler
christopher deristopher.muusshheeeiiiii fww.:hhheeesessss.abbboooaaa
Precise Positioning and Analytics
Prounhofer Institute for Integrated Circuits (IIS)
Nuremberg, Germang
Reviewed on OpenReview: https: //tp:::///peeeeveeee net/ forum? id- og3t=5kim
Abstract
Humans innately measure the distance between instances in an unlabeled dataset using an unknown similarity function. Distance metrics only serve proxy for similarity in information retrieval of similar instances. Learning good similarity function from human annotations improves the quality of retrievals. This work uses deep metric learning to learn these user-defined similarity functions from few annotations for large football trajectory dataset. We adapt an entropy- based active learning method with recent work from triplet mining collect casy-to-answer but still informative annotations from human participants and use them to train deep convolutional network that generalizes to unseen samples. Our user study shows that our approach improves the quality of the information retrieval compared to previous deep metric learning approach that relics on Siamese network. Specifically, we shed light on the strengths and weaknesses of passive sampling heuristics and active learners alike by analyzing the participants' response efficacy. To this end, we collect accuracy, algorithmic time complexity, the participants' fatigue, time-to ressoose,, qualitative self-assessment and statements, as well as the effects of mixod-oxportise annota- tors and their consistency on model performance and transfer learning.
1 Introduction
Position tracking of persons, vehicles objects is ubiquitous and enables various trajectory data mining tasks iZheng 2015 such as match or performance analysis in popular sports like football (Löffler et al. 2021), hockey CCen el al. (205)) and basketball (Sha et al. 2016), or (public) transport planing and mobility analysis (Fernández et al. 2017 Shen et al., 2019 Yadamjay et al. 2020). With each additional moving agent, the data bocome increasingly complex due its often unstructured and high-dimensional nature. For example, football has 23 trajectories (players and ball) that are typically tracked via multi-camera video systems Löfiler et al. (021 This affects tasks such similarity-based information retrieval, which

---

Published in Transactions on Machine Learning Research (04/2023)
queries similar occurrences to a given query scene (Sha et al. 2016 Here, scene is an ensemble of agents'
trajectories in a window of time.
In such large and complex datasets, information retrieval requires two expensive steps. First, the unstruc- tured trajectories are optimally assigned, e.g., using the Hungarian algorithm Kuhn (1955). Second, the pairwise distance between the matched pairs of trajectories is computed on the raw trajectory data. Already by themselves, the two steps do not scale well to more realistic datasets that can have both a large number of samples and high dimensionality.
Recently, convolutional Siamese networks were leveraged by Löffler et al. (001 | to learn approximations of both the trajectory assignment and distance metric. The resulting diver-dimensoonal representation enables the scaling up of Euclidean distance-based information retrieval. However, there are two limitations. First, the Euclidean distance in itself is limiting, e.g., it does not weigh subjectively more important trajectories higher, and is affected by the data's high dimensionality (Canessa ct al.) (020). Sccond, following directly from estimating the Euclidean distance of high dinensional data, the embedding captures global ordinal structures better than local ones (Löffler et al. 2021
This work proposes to actively learn a distance function from human annotations. This learned similarity function is independent of the data's dimensionality and leads to a lower-dimension dimensional ordinal embedding that more closely matches human perception. We address the associated costs of the human-in-the-lowe by adapting an Active Learning (AL) sampler. We hypothesize that human annotations preserve more relevant information than the Euclidean distance and that a neural network can learn this perceived similarity from few annotations. We pay special attention to mixed-expertise annotators.
Our method learns rank-ordering from relative comparisons of tuple of data instances using active query selection method. In our experiments, we choose InfoTuple (Canal et al., 2020) over Crowd Kernel Learning (Tamuz et al.) 2011 as it generalizes triplet queries to arbitrary tuple size and queries the oracle with more informative tuples. This has clear benefits: due to the tuples' larger size, they provide more context. and are easier to annotate, and are also more sample efficient. We conduct a user study to evaluate the Active Learning sampling and use a well-suited football trajectory dataset for it. The study is centered around challenging relative comparison queries to human annotators. The query tuple composition impacts the efficiency of annotators, who may skip hard queries (see pre-study in Appendix A.1p. Furthermore, while participants may have different similarity functions in mind, we can use a proxy similarity function for pre-training (Sha et al. 2016 Löftle et al. (021). Crucially, the study allows broader evaluation than accuracy per labeled sample. It also evaluates practically important questions such as the impact of skipped responses for different tuple composition algorithuss, and it reports the annotators' intra-rater consistency and inter-rater reliability. We use a questionnaire for qualitative self-assessment of expertise and perceived similarity. These properties make our study suitable for the evaluation.
How consistent and reliable are the participants? RQ2: What is the best bascling hcuristic to generate tuple queries? RQ5: Does active sampling perform better than non-active sampling? Our experiments then also answer which methods are the most suitable with respect to pure improvement in effective accuracy, relative time efficiency (i.e., user response time), or sampling efficiency (i.e., number of tuples skipped). We conclude with a study on how users' response consistency affects the learned metric and the capacity for transfer learning. We summarize our contributions as follows
We pose three broader research questions. RQ
We reduce the computational complexity of the InfoTuple active learner.
We experimentally analyze the efficiency of our method with a user study on active learning met on real-world data, analyzing strengths and limitations.
We answer the research questions and show the adapted InfoTuple active sampling leads to higher triplet accuracy than (uon) active sampling methods but falls behind slightly in sampling efficiency. In addition, participants can form relatively consistent groups of user-specific similarity functions.
We release a simple but effective web app for active learning experiments[]
Code available at http:////tttteeeecccrrr//s////tt Ofddinnaa-------o -nnbeddinnn
2

---

Published in Transactions on Machine Learning Research (04/2023)
The rest of this article is structured follows. Section 2 formulates the problem and presents AL methods and adaptations. Section explains our experimental design, including the procedure, metries, participants, and dataset. We present the results in Section 4 and discuss them in Section Section concludes.
Query
sample
Query Scene Similar Scenes Scanes
Ordinal Embedding
?
finetune
(a) Left: We collect participants' responses to relative similarity queries. In our study, use tuple size of nine. Right: We finetune a nenral network to learn an ordinal embedding. We use different (active)
sampling methods generate queries. show 3D UMAP (Melnnes embedding purely for visualization.
2018 plot of the learned
consistency
non-active
consistency
active
Warm- up
Repeat #1
Repeat #2
Active (NN)
Rnd
Rnd/NN
NN
InfoTuple
RQ₁
RQ2
RQ₁
RQ3
(b) We ask participants annotate InfoTuples in sequential phases. start with a warm-up, then measure consistency in two repeated phases (RQ:). The main research questions are answered in a non- active (RQ2) and in an active phase (RQa). The query tuple composition heuristics RQ2 are Random (Rnd), Nearest Neighbor (NN), and their combination (Rnd/NN).
The study consists of several different annotation phases. annotations and to finetune and sample from the learned embedding. (b) shows the multi-phase study design that compares different sampling strategies and evaluates model performance with respect to efficiency and effectiveness.
Figure 1:
aa shows the procedure to collect
2 Method
Tuple composition and sample selection are important for learning ordinal embedding specific to human's similarity function. We formulate the problem first as pairwise similarity learning using a Siamese network like Lötfeer et al. (2021), which initially approximates assignments through metric learning (Section 2.1 This delivers a meaningful embedding to warm-start tuple selection strategies. Next, Scction 2.2 extends the objective triplet-base based learning to generate an ordinal embedding. Section 2.3then explains optimizations of InfoTuple, and adaption to select the most informative samples from meaningful candidate set..
3

---

Published in Transactions on Machine Learning Research (04/2023)
2.1 Problem Formulation
We first investigate calculating pairwise distances of spatio-temporal matrices E of dimensionality S T:
2,,
1122
T1,
1
12,2
22,7
T
3S,1
45.
EST
where the number of the dimensions of the trajectory S is the spatial dimension (S = 2 for 2D positional tracking), and the number of time steps T is the temporal dimension (T = 125 for 5s tracking at 25 Hz). A trajectory may correspond to an agent like a football player. A row vector T represents a single dimension 8 over time T. Positional tracking in sports may be calculated from multi-perspective video feeds Löffler et al. (2021). For 5s scenes of one player sampled at 25 Hz, this produces trajectories # of the dimensionality 2 x 125.
In multi-agent tracking, scene X consists of N different trajectories such that X = (I(),F2 Z(N) Löf- fier et al. 2021). See Fig. la for three scenes. Hence, X is of dimensionality N x S x T. For scene with all 22 players and ball, this results in size of 23 x 2 x 125. However, the ordering of the dimension N is unknown, because agents may follow different strategies in each scene and can adapt their role in a game as needed Bialkowski et al. (2014). This problematic for calculating the distance between matrices (Löffler iet al. 2021). Specifically, given a pair of matrices X and X2, the respective pairwise assignment of vectors 200) and (05) from each matrix unknown.
Hence, to calculate the matrices' pairwise distance, we first compute the optimal row-wise assignment in the dimension N of Xı and X₂ using the Hungarian algorithm (Kuhn) 1955), that minimizes the sum of pairwise distances between the matrices in On²²). For this we calculate the distance between two trajectories H and a as the average Euclidean distance over the spatial dimensions at each point in time (Löffler al. 202
T
d(z
=
| - J'eel2
The distance between ensembles of trajectorics (X,, X2) is simply the sum of distances between trajecto- ries (as in [ of matrices X1 and X2, that are optimally assigned along dimension N (using the Hungarian algorithm). In terms of sports tracking data, this procodure maps players from one scene optimally onto players of another scene such that the distance between scenes is minimal.
We then further follow Löffler et al. [2021 and use Deep Siamese Metric Learning with Temporal Convo- lutional Network (Lea et al.) 2016) with a Resnet architecture (He et al. 2016) f to learn an approximate assignment and lower dimensional, distance-preserving embedding. The observations are pairs of matrices X1 and X₂ sampled uniformly at random from all possible permutations of the dataset. Our goal is to find an embedding that preserves the distance dens X2) (X,, X2) where dens is the Euclidean distance between the learned lower dimensional representations f(X1) X1 and f(X2) X2. Hence, the learning objective can be formulated as
C(X, X²) = (()) - f(X2) dens (X,, (x))
(2)
We use two Lg regularization terms. The first || [|f(())l22 + +I| centers learned representations and the second |[0]] performs weight regularization on the network f's parameters 0. dens is the Euclidean distance. It is simple to implement but still measures play similarity just as well as, e.g, Dynamic Time Warping, Fréchet distance or Lx distance (Sha et al. 2016).
X1 and X₂ contain trajectories of multiple agents which leads a computationally expensive assignment problem (Sha et al. 2017) between two sets of trajectories describing two different scenes. This is because role
4

---

Published in Transactions on Machine Learning Research (04/2023)
assignments are not fixed and often even disjoint between teams. Sha et al. (2017) build a tree structure to perform the assignment, that others subsequently use to dynamically set role- based assignments independent of priori positions (Di et al. 2018). In contrast, we estimate these assignments similar to Löffler et al. (2021) (based on trajectory data and agnostic of roles) who compare three approaches: random assignments, exploiting additional meta-information such as roles, or inferring spatial proximity from data. The last variant uses a fixed grid template and assigns rows to channels, but this introduces sparsity in these inputs, which become overdetermined, as there are fewer players than entries in the grid template. Instead, we propose improved template matching algorithm: we center circular template with as many available positions as rows over the spatial center of X. Then, we fit the template's variance in each dimension S to match X. For a pair X1 and X2, the template fitting and matching produces representations, with that we can estimate pairwise distances with lower error than Löfler et al. (2021), but without introducing sparsity in the network's inputs. See Appendix A.4 for the experimental comparison of the variants.
In summary, this approximates and reduces the computational complexity for s scenes from O(s n²) down to O(s (() where m is the size of the network's embedding.
2.2 Active Metric Learning
Learning metric from human annotators is costly. Active Learning helps reduce the amount of labeled relative comparisons by querying the oracle with informative samples (Settles) 2009. Houlsby et al. 2011 Active Learners use model state, an acquisition function, and a query format.
Initial model state. Performing active sampling on a non-random model state is called warm-starting. Since there are no ground-truth labols available, we can use the Icarned Euclidean embodding to warm start AL methods, as inspired by (Simo-Serra et al. (2015). The Euclidean embedding is independent of the participants' similarity function and can serve as a more general early estimate of similarity. Following that, we may then finc-tune this pre-trained model to an initial set of annotations before actively selecting queries.
Acquisition function. For learning from relative comparisous, the acquisition function needs to construct the query Q from the most suitable candidate samples from the dataset. Following the notation from Canal ct al. 2020). a tuplewise query Qn at time step n of the AL procedure has a 'hoad' object a, and an un-ordered 'body' of objects 8" = (bf, bg. ..., b) Now On = (a,; B") denotes the th tuple query, and a (R1 (Qn), ..., (Q=)(Qn)) which is a permutation of B" rank-ordered by similarity such that (Q,) < R2 (Q,), if i < j. This indicates that the oracle ranks the object RQ(Qn) more similar to the anchor a, than the object (Q,) (Canal et al. 2020
participant's ranking response R(Q,))
Next, we follow a principled approach for an initial Active Learning heuristic. Given the head Xa: we categorize closer samples in the Euclidean embedding as either positive Xp or negative Xn, Xuan et al. (2020) further break down neighbors based on their distance to X,, Hard negatives are close to the head but dissimilar, whereas easy negatives are far away from it and least similar. Of these two types, the hard samples are most beneficial for learning, whereas casy ones produce no useful gradient (Xuan et al. (020). Conversely, hard positives are highly similar samples, that are far away from X., and thus mining these for tuples is difficult. Easy positives on the other hand are naturally available in appropriately pre-trained embedding. Xuan et al. (2020) show that selecting easy positives keeps intra-class variance and helps to avoid over-clusterngg of the embedding.
We adapt the triplet. mining concepts to Active Learning. Query tuples Qn composed of X, 's's nearest neighbors can contain both easy positives and hard negatives. The other samples with higher distance from X₂ are hard positives and easy negatives and are least useful when learning similarity. Nadagoudal et al. 2022 recently proposed similar NN acquisition function to collect both similarity and classification responses, see Sec. 2.4
Query format. Generally, an ordinal embedding can be learned from tuples of size s 2 3. A triplet loss (Chechik et al. 2010) over the three samples X., X,, X, is defined as follows:
L(X,, Xp: Xx) max((I r(xx)l12 [|x())) r(x,))l2 11,
(3)
5

---

Published in Transactions on Machine Learning Research (04/2023)
Canal et al. 2020 show that human annotators benefit from larger tuple sizes than three. Query tuples On or arbitrary size greater than three can provide more context for considering the similarity and can be more sample-efficiett. We can simply decompose response R(Qx)) into triplets t = (Xar Xp, and use triplet loss. In this work, we determine the tuple size experimentally via a pre-study.
To summarize; we establish Active (NN) as an Active Learning heuristic, that constructs queries of arbitrary size from the nearest neighborhood of an anchor sample in an embedding.
2.3 Informative Queries
The selection of easy positives and hard nogatives may not reliably construct the most informative queries. InfoTuple (Canal et al. 2020) improves upon this by maximizing the information gain of new queries. Given probabilistic embedding, and previous and candidate queries, InfoTuple then selects that query that maxi- mizes the mutual information that a response provides. It then uses a d-dimensional probabilistic embedding, such as t-Stochastic Triplet Embedding (tSTE) (Van Der Maaten & Weinberger 2012) probabilistic Multi- Dimensional Scaling (MDS) (Tamuz et al., 2011), to embed the dataset X' of size N. Next, Canal et al. (2020) use the datasets's embedding M E Raxx to define a similarity matrix K = M M. From this similarity, the authors calculate a N x N pairwise distance matrix D for X. Given the queries Q1, Q2, Qn-1 and their responses R(Q1), R(Q2), .... R(Qn-1), they select the next query from a set of possible queries using the conditional entropy (-I-)) of the next possible reply R(Qn):
arg min (BR(Q
(((Q.))) K, -((())))))R)))))))(((()))))))), R,(Qx--)))
(4)
Q
The equation trades of two terms: the first term selects for uncertain queries provided previous responses R(Qn-1) while the second lerm selects for unambiguous responses R(Qn) that can be encoded in the simi- larity matrix K by responses R(Qn-1). Balancing these two measures selects queries with high entropy, but that can be consistently annotated by the oracle.
Next, Canal et al. 22200) develop simplifying assumptions the joint statistics of query and embedding to enable tractable estimation via Monte Carlo sampling. Importantly, they assumee Gaussian distribution inter-object distances, and only need to sample the distribution for the current query Qn, proposed by Lohaus et al. (2019). With these simplifications Eq. and Eq. 6 depend only on the distance matrix D:
H(R(()))R(QQ~II)) H
E D On
((((QN))
(5)
RIQ P
H(P(H(Qn))
(6)
They slightly abuse the notation by using H(X) (x(()) for a probability mass function p of random variable X, and N to represent the Gaussian distribution " We refer to Canal et al. (2020) for further details.
2.4 Adaption
In this work, we use landem of a generalizing neural network for learning similarity and Info Tupless probabilistic model for selecting queries via Active Learning. We adapt InfoTTuple to increase its effectiveness, leading to less ambiguous queries, improved sampling efficiency, and reduction of the required time to select informative tuples.
Effectiveness, Following the assumptions in Sec. 2.2 we use the neural network's embedding to generate sample candidates to form the body from the neighborhood of a head a. This differs from the approach of Nadagouda et al. (2022), which selects the most informative nearest neighbor query from all possible queries. We do this because problem domain suffers from sparse similarity. For instance, successful goals are

---

Published in Transactions on Machine Learning Research (04/2023)
typically rare but passes in midfield are much more common, leading to long-tailed, imbalanced distribution of sample similarity with sparsity in the tail (Wang et al. 2010). Thus, queries may often provide no positive and informative candidates. This leads to high rate of skipped queries, which in turn leads to fatigue of the human annotators. We discuss this phenomenon in Sec. S This pre-sclection from the Neural Network's
embedding space increases the likelihood of finding similar candidates and reduces the subjective difficulty of the annotation task. Based upon a sampling of m candidates, InfoTuple then selects the most informative query Qn. The set of candidates larger than the tuple size of Qn and is a hyperparameter (see Sec. 3.5
Efficiency. The temporal complexity of the selection algorithm is primarily bound in the Monte Carlo style computation of Eq. 5 and Eq. First, reducing the available candidate samples for greater effectiveness also reduces computational complexity. It leads to a smaller set of candidate samples available for constructing queries from. Second, we downsample the remaining m samples like Canal et al. (2020). which further reduces the number of constructed queries Q that have to be evaluated.
3
Experimental Design
In this scction, we design a user study to answer the initial rescarch questions.
3.1 Procedure
No scene is simlar.
You've answered queries.
Figure 2:
The web-based query page asks annotators to compare eight samples with an anchor and to either select the most similar sample skip the query. Each sample shows the team in ball possession (blue), the defending team (green), and the ball (orange).
The study is designed in several phases and continuously collects annotations from 18 participants, see Fig. [bb] A smaller pre-study with participants provided preliminary insights. With its help, we set the tuple size of TnfoTuple one anchor and cight samples for its body. The number of required samples for learning a meaningful ordinal embedding was about 250 300 triplets. Hence, the user study collects an equivalent number of tuples. Furthermore, chose to train user-specific models, the pre-study showed

---

Published in Transactions on Machine Learning Research (04/2023)
that learned similarity is highly individual and does not generalize well. We include an evaluation of this notion in Sec. 4.4
Task description. The task is to annotate an InfoeTuple such as in Fig. 2 Participants compare an anchor with eight other samples, and then either choose the most similar sample, based on their own function of similarity or skip the query.
Phases The user study is segmented into five parts. The first is an introduction with description of the data and the mechanics of the annotation process. The second is short warm-up of five queries, that familiarizes participants with the mechanics of the annotation website. Next, the split phase RQ1 occurs before and after RQ2. Finally, the active learning RQ; concludes. RQ1 addresses intra-/inter-ae We repeat a fixed set of InpoT as Fig. [b] shows. We denote these sets as repeated set and highlight the phases as RQ1 in the figure. The repetition takes place after other phases, and the location of samples on the website is randomized, in order break up any potentially lingering memorics of the spatial layout of repeated queries. This way, we collect at least 20 InfoTuple but increase the set for each skipped sample. Hence, the repeated set may also grow in size as it also repeats skipped samples. Next, for the second research question, we collect annotations with different compositions of InfoTTuple. The figure highlights the three segments as RQ2: Rnd, Handom/NN and NN. We collect 20 Info Tuple to answer the research question and 10 more for testing. Finally, the last phase labeled RQ3 executes two different, active learning strategies and trains neural networks with the annotations collected from participants. The first method is the Nearest Neighbor active learner, and the second is the adapted InfoTuple algorithm. We collect 20 InfoTuple with an acquisition size of one.
metrics.
Warm start. We use a warm-start strategy for active learners. Starting from a pre-trained Siamese network, we finetune model for each participant with their replies to the repeat #1 phase.
Annotation tool. The annotation tool is implemented in Python as a Flask web app. It uses the PyTorrh framework for deep learning and integrates InfoTuple X The user study was performed on an AMD Ryzen 3800X (64GB RAM) and an NVidia Geforce RTX 2070 Super (8GB VRAM), and participants used their clients to access a web frontend. Each participant had the same computing time and resources available.
3.2 Survey
We ask participants to self-assess in order to experiment with extensions from individual models toward generalized models of user-defined similarity. This serves as a qualitative, parallel approach to the intra- and inter-rater metrics.
We provide simple questionnaire to inquire about the participants' self-assessment of their expertise and a qualitative description of their individual similarity function. We specifically ask them about their definition of similarity before and after the participation. This allows for qualitatively estimating the clustering of participants into groups that employ similar notions.
The questions were the following:
1. What is
your expertise in football on scale of (novice) to 6 (expert)?
2. What will you look for when you compare football scenes' similarity?
3. Did your definition of similarity change?
4. What did you look for when you compare football scenes' similarity?
The first two questions were answered when looking at the tutorial. To capture any changes, the last two questions were answered after finishing all queries of the procedure. See Appendix A.2 for an analysis.
8

---

Published in Transactions on Machine Learning Research (04/2023)
3.3 Metrics
Triplet accuracy is defined the agreement, between two triplets (a, (1,b1,b) b2) and (a', 11, for set of annotated triplets. In this study, we generate triplets from annotated tuples of arbitrary size.
Intra-rater consistency C for a participant P; measures the agreement of annotations and skipped samples belween the two phases Repeat #1 and Repeat #2:
C(P:) = aggnggsgreemmntt (P:)
(7)
ratings (P:)
Pairwise inter-rater reliability R for two participants P. and Pj includes numbers of all annotations and skips. It is the fraction of ratings in agreement over the total number of ratings:
ratingSngreement (P..P)
RP,PP
(8)
ratings (P,, Pş)
Response effectiveness [Bernard et. al. More effective sampling methods have higher ratio relative to others.
2018 E is the measure of accuracy per time spend on response.
triplet accuracy
E
(9)
response time
We additionally evaluate the total effectiveness TE of AL samplers, which includes the time spent sampling and training a model. While this depends on the choice of model and available resources for sampling and training, we consider it a relevant performance factor
triplet accuracy
TE response time + computation time
(10)
A high number of skipped responses may lead to participants' fatigue. Hence, we track the methods' label effectiveness LE. The accuracy relative to the number of skipped annotations provides a measure that extends the concept of elficiency (Bernard al. 2018), which only measures the number of labeled instances over time.
LE = triplet accuracy
(11)
skipped responses
3.4 Dataset
We use dataset from the German Bundesliga from season 2014/15, that consists of trajectories sampled 25 Hz of 304 games, extracted from melti-perspective video feeds.
For our user study, we choose one game and extract scenes of 5s fixed length with 50% overlap of 2.5 We further pre-process the data to simplify the implementation: we transform the data so that the team in ball possession players from left to right, we only use active (not paused) scenes, where one team has more ball possession than the other, and we require that no players are missing. This yields 1, 005 samples for the game. We consider the experiment size be minimal but still representative. allows us control side effects that we can see and compare the actual performance of the different methods. This helps to answer the research questions and still tests the proposed method.
We pre-train the bascling Siamese network similarly to Löffler et al. 2021 For this, we extract about 1,200,000 scenes from 304 games with a smaller overlap of 18, and use the same pre-processing. Then, we split the data into train/valiit /van/iiiiiaioo/eestttt splits 40/10/100
As we cannot process all possible combinations of scene pairs, we instcad randomly sample 10 million pairs for training and million otherwise. Then, we train using the regularized Siamese loss (see Eq. 2 and the ellipse assignment method (see Sec. 2.1). We refer to Löffler al. (2021) for additional details the baseline's training and optimizer configuration.
9

---

Published in Transactions on Machine Learning Research (04/2023)
Test dataset. There is no ground-truth dataset for user-specific similarity functions. A full set of compar- isons would be unfeasible, given the combinatorial complexity, further motivating the use of Active Learning. Hence, for the test dataset, we collect additional hold-out sets of each participant's annotations. The in- consistency of annotations then impacts the possible accuracy score and introduces a baseline test error. Especially a low intra-rater consistency poses an upper limit achievable performance.
Additionally, the composition of tuples itself may have large impact on the test scores, as they represent different embedding neighborhoods. Annotations sampled at random for testing would provide information the ordinal embedding's ' global order and are less biased. Test tuples from the anchor's neighborhood are based on the pre-trained Siamesc embedding and thus are biasod. However, they may provide some insights into the embedding's ' fine structure.
3.5 Experimental Setup
We use a Temporal Convolutional Network Lea et al. 2016) with a Resnet architecture (He et al. 2016) like Bai et al. 2018 and Löftler et al. (021). For its training, we use the triplet loss described earlier with an Adam optimizer with a learning rate of 0.001 and batch size of 32 for 10 epochs after each acquisition. We implement the experiments in PyTorch.
The InfoTTpple method uses the tSTE to fit a probabilistic embedding. Furthermore, we select the hyper- parameters for InfoTuple follows. We sub-sample the candidates to 100 neighboring samples, that is about 10% of the whole dalasel, and generale 10 random permulations as possible queries Qn. We set the number of Monte Carlo passes to 10 and sub-sample the factorial of the 8-tuples with the factor 0.1.
4 Experiments
This section first presents the fundamental intra- and inter-rater metrics in Sec. 4.1 With this context, we give the sampling efficiency of the evaluated methods in Sec. 1.2 and specifically show how the quality of annotators' replics affects the learning of their similarity functions in Sec. 4.2.1 Next, analyze response times and the amounts of skipping in Sec. 4.3 before we conclude with a study on clustering of highly agreeing annotators in Sec. 4.4 We asked 18 participants (aged 20 to 35, one female) to take part in the user study.
4.1 Intra- and Inter-Rater Metrics
As a first step, which will allow us to better understand the results, we evaluate the annotators' intra-rater consistency and their inter-rater reliability. This allows us lo analyze similarity functions based on the annotator's noise as well as discover possible groups of similarity functions.
Intra-rater reliability. We determine participants' reply consistency that we consider the noise level of their replies in later analysis.
The procedure contains two identical sets of queries (Repeat # and Repeat #2) that we compare to calculate consistency as defined in Eq. 7 Repeat #1 consists of at least 20 queries, with additional queries for each skipped reply. Repeat #2 then repeats the exact same queries per user. We shuffle the samples the UT avoid memory effects and repeat the set only after other annotation tasks.
Fig. 3a shows each participant's consistency from low of 0.25 to high of 0.75. The typical reply consistency clusters of 0..998 (#0.11), see Fig, 3b We see two outliers, one to the low (p6) and to the high end (pl1). The consistency agrees with the participants' solf-assessmentt of their expertise as 1 for p6 and 6 for pl1 on a scale of 1 (novice) to 6 (expert). Next, we normalize the self-assessment between and 1, and show the difference to measured consistency Fig. [e|| The self-assessed expertise typically matches consistency within step up or down on the scale. This validates the quantitatively measured consistency. Furthermore, this score may help decide whether to collect annotations from a participant, or whether the generic bascline model is the better-performing alternative (see Sec. 4.2.1
10

---

Published in Transactions on Machine Learning Research (04/2023)
0.6
Consistency
0.4
0.2
p6 p14 p7 p9
p3
p18 p1 p4 p15p12 p8 p5 p10p16 p2 p13p17p11 Participants
(a) Intra-rater consistency repeated queries is from 0.25% to 0.75% (random baseline at 12.5%).
0.2
0.4
0.8
-0.4
-0.3
-0.1
0.0
0.1
0.2
0.3
Intra-rater consistency
Discrepancy of expertise
(b) The intra-rater consistency centers around (c) Survey self-assessment minus their mea- 49.8% (+0.11) with two outliers at the ex- sured consistency shows the small difference between qualitative and quantitative expertise.
tremes.
Figure The intra-rater consistency in dal shows how reliable participants determine similarity in fixed and repeated set of queries, and show their distribution in shows that participants self-assessed expertise in the survey differs from measured consistency by 20% 1.2 points the scale of 6.
These results show that the participants answer queries inconsistently, as the mean consistency to find the same similar samples from cight in total is 0.49(=0.112. Additionally, the consistency measures identify unreliable oracles which help understand algorithmic performance better.
5.17
0.42
0.6
25
56
290
0.30
0.30
4.45
0.330
34
4:43
0.3
0.305
83
0.3403
0.39
0.3
9.3
0.5
46.0 42
9.4
9.20
04
04
0.45
0.5
218
03
0.33
AB 022 ntr 24
3833300.5
2.31
489.5
0.430.4
0.380.20
26035
34
0.2
0.37
0.32 27
02002 0333
0.5
0.36
0.6
0.6
0.5
0.520.47
0.6
0.99
0.2
54
9.4
0.3
03600260 49
9.380.34
0.32
0.49
22T
3.555.44
6.400.34
23 pio 210
0.5
9.290.62
0.470.35
2833
0.22
0.190.3
24
02
0.80.53
p11 927 p13
0:33
0.52 0.2
0.41
0.36
0.36
8.59
0.48
0.39
2.300..4
0.41
0.8
52.
33
42 08.0888055
3:20
02.02
0
0:360.2
Figure 4: The inter-rater reliability on fixed, repeated dataset shows whether participants form clusters of similarity functions, or whether they have orthogonal notions of similarity.
Inter-rater reliability. The subjective similarity functions may be part of clusters. Similar metrics could be learned from annotations participants concepts of similarity are close enough. The identification of such groups may be beneficial for analysis and fine-tuning.
The repeated sets of queries (Repeat #1) are identical for all participants. Only the actual number of replies may differ depending skipped replies. Hence, it allows to calculate the inter-rater reliability
11

---

Published in Transactions on Machine Learning Research (04/2023)
0.72
0.71
Triple accuracy
0.70
0.69
0.60
Active
0.67
Active (InfoTuple)
0.66
Random/NN
0.65
No. acquisitions
Figure 6: Triplet accuracy averaged over all users on the test set. We initialize the training data with the first Fized dataset and then perform 20 acquisitions of size one. We train user-individual models and show the averages.
defined by Eq. This way, can identify sets of participants whose similarity functions overlap sufficiently to be considered as cluster members. We then use the reliability the pairwise distance for hierarchical bottom-up clustering with maximum linkage.
We show a heat map of the pairwise reliability of the raters in Fig. and a dendrogram of hi- erarchical clusters in Fig. 5 In the heatmap, we 0.8
Hieerrrciicll Clustering Dendrogram
sort participants by their consistency from low high. Higher inter-rater reliability is highlighted with brighter colors. Participants with lower con- sistency also show lower inter-rater reliability.
The intra- and inter-rater scores appear to be sim- ilarly ranged distributions. That means that the replies of some participants may be similar enough cluster them together, allowing us treat the entire cluster one individual. The dendrogram shows hierarchical clusters of agrecing participants. We search for clusters of inter-rater reliability that have similar linkage as the intra-rater consistency, and highlight clusters with a linking distance equivalent to 0.37 or above, e.g., the four participants [p5, p10, p12, p17] form one such cluster.
0.0
p12
p17
05
p10
p4
p3 pi1 p18 p7
p15 p1 p9
62
p13
p16
P6
P8 P14
Participants are leaf nodes.
Figure 5: Hierarchial clustering dendrogram based on inter-rater reliability as pairwisc distance.
We can determine clusters of similarity functions from these results. Still, the noise replies can be in similar magnitude as the reliability between participants, which corroborates the need for an analysis of transfer learning within such clusters. We will conduct this in Sec. 4.4
4.2 Triplet Accuracy
The gains in predictive accuracy per annotated sample demonstrate whether a tuple composition is informa- tive for the model. The accuracy represents the usefulness of the model for the application. We hypothesize that an active learning sampling of tuples most efficient in increasing accuracy.
For this, we calculate the triplet accuracy on user-specific test datasets over 20 acquisition steps. For each participant, first fine-tune the pre-trained network on 20 annotated Info InfoTuples from their Repeat #1 phases (equivalent to 140 triplets). We evaluate three non-active tuple composition methods as basclines (Rnd, Random/NN, NN) in comparison to two active learning methods (Active-NN, InfoTTule)) The base- lines are constructed from replies to fixed queries, the active learners were not fixed but interactively adapted to users. The test sets are user-specific sets
annotated InfoTTuple that were sampled randomly.
The test in Fig. 0 sees the active learners ahead, with InfoTuple on top. Purely randomly composed tuples compare well, especially in the first half of the acquisition phase, but level out around 69% triplet. accuracy.
12

---

Published in Transactions on Machine Learning Research (04/2023)
The remaining composition methods, which were constructed at least partially from the pre-trained network's embedding neighborhood stagnate for most of the duration of the experiment. These results for triplet accuracy are for 40 InfoTTuples (280 triplets) in total. We further investigate the ceiling of the triplet accuracy, which is likely due to the noisy annotations and inconsistent replics, by training and testing user- individual models with all 120 IufoTuples (or 840 triplets) from the different annotation phases. The average triplet accuracy then reaches 73.7% (+0.01), which only marginally higher than previously. The low gains with three times as many annotations show the diminishing returns of larger data collections.
Given the limited consistency of replies and the lack of ground truth data, there is an upper limit to the model's ability to learn. However, the relatively higher scores of Info aupoe are consistent and lead to increased model accuracy, whereas the NN active learner comes in second but similarly to the baselines. In terms of efficiency, Info Tuple is clearly preferable.
4.2.1 Noisy Oracles
Consistency of replies affects triplet. accuracy and influences the network's ability to learn user-specific similarity functions. lower consistency is harder to train with, and a pre-trained network performs better on test data than finc-tuned one. More consistent annotations, however, enable fine-tuning. The InfoTuple aclive learner benefits from more consistent annotations.
We compare the triplet accuracy for the pre-trained model with all fine-tuned models for each user. We again use the users' self-annotated test sets Rnd and compare the accuracy. Additionally, we split the participants into groups of lower and higher consistency. For each group, we compare the pre-trained model with one trained using the InfoTuple active learner to test fine-tuning with more consistent annotations.
We show the effect of consistency on the InfoTuple algorithm in Fig. 7 For annotators with low consistency, the pre-trained network often performs similarly or even better than the finetuned variant. The pre-trained model is strong bascline for inconsistent annotators. Fig. 7b shows that consistent annotations allow the active learner to learn similar or better model than the pre-trained baseline, with 74% for the finetuned case vs 69.2% pre-trained on the test set. Appendix A.3 reports dis-aggregated triplet accuracy per user, supporting the results.
Generally, if the pre-trained network already performing well on a lest set, the fine-tuning with inconsistent data tends to decrease test. triplet accuracy. It depends on the consistency of the similarity function used for annotations if fine-tur tine-tuning is beneficial. For consistent participants, that also tend be self-assessed experts, better user-specific similarity functions can be learned.
Furthermore, with Tuple as the AL sampler, we see strong increases in triplet accuracy over the pre- trained baseline (see Fig. 7). The most consistent & participants lead to more stable learning and higher triplet accuracy, which is likely due to their more consistent replies or clearer notion of similarity (see Fig. 7b
0.9
0.9
Triplet aocuracy
0.8
0.8
0.5
0.5
Pre-trained
Fine-luned
mre-rrined
Fine- taned
Training variant
Training variant
(a) Least consistent. on Rud. (b) Most consistent & on Rud.
Figure 7: This figure shows the effectiveness of fine-tuning via Info active learning in comparison to the pre-trained baseline for different levels of consistency. Participants are grouped according to their consistency, into a least consistent third and the most consistent two-thirds. Relative gains are greater for more consistent replies.
13

---

Published in Transactions on Machine Learning Research (04/2023)
Rel. triplet acouraty per time
0.003
0.002
10
Active (NK)
0.000
Active (intoTuple)
10
Random/NN
-0.001
MN
Rod
10
15
20
acquisitions
(a) Response Effectiveness over the number (b) The response times per phase, here as box plots over all users, show relatively slable around 10s for each phase of the procedure.
of acquisitions.
Figure 8: We present the triplet accuracy after applying Eq. 9 in (al and show response times in bb
4.3
Considering Time and Sampling Efficiency
We consider three different perspectives that shed some light on the relationship between triplet accuracy, time, and skipping. These metrics are relevant for determining the efficiency of the time spent, well the potential fatigue of participants.
4.3.1 Response Effectiveness
The relative return in accuracy on the invested amount of response time is a relevant criterion when choosing a sampling method. The time that users spend analyzing samples and forming decisions may also be indicative of how difficult query is to reply to. This excludes the time that methods require to propose queries.
We evaluate the response time for all users by dividing the relative gains in predictive accuracy by the participant's average response time. This yields a relative improvement in accuracy per time spend on a response.
We normalize triplet accuracy using Eq. and report the results Fig. g There, we see the highest increase in triplet accuracy for InfoTuupl, followed by the NN active learner. The Random baseline follows closely. Interestingly, the strongest method Info is also the one that is replied to quickest. The response times of the methods in our study are: Random with 9.7s, Random/NN with 10.98 s, NN with 10.35 and the active samplers NN with 11.19s and InfoTuple with 9,48s, see Fig. 8b for an overview and Fig 14 in Appendix A.3 for user-specific details.
The relative return accuracy should not be bought with exorbitant amounts time. Our results show that both active learners perform comparably well due to their higher relative gains accuracy compared to the baselines, which speaks for them tuple composition methods.
4.3.2 Total Effectiveness
While the response time itself is important for choosing a suitable sampling method, the overall time spent may also include an active method's computational overhead. The analysis similar to Sec. 4.3.1 but also includes the time of methods to compute queries on top.
The algorithm execution times cause minor decreases in total effectiveness for the two Active Learning methods. The Nearest Neighbor active learner computes for 2.8 (+ 0.59 and the InfoTuple active learner needs 6.11s (# 0.56s). However, this depends mainly on the computational power and implementation efficiency. In this work, we optimized InfoTuple to decrease its complexity while still providing benefits over others.
Computation, such as the training of the neural network or the calculations performed by Info impacts the eccuracy-pe--time efficiency. However, while consider the cognitive load for participants to be taxing,
14

---

Published in Transactions on Machine Learning Research (04/2023)
the waiting time between queries may be less demanding. In user study, the additional waiting time of the active methods was still reported as tough by several participants and should be considered when selecting a method.
4.3.3 Label Effectiveness
We consider the number of skips during the labeling process a potential source of participants' fatigue, because it takes longer reach the set number of annotations. Hence, participants are queried repeatedly, Higher label effectiveness is to be preferred, see Eq. 11
We present the least, the median, and the most consistent participants timeline of skipped and annotated queries in Fig. to highlight the prolonging effect, that large number of skipped queries has on the duration of the experiment, and thus also on the participants' fatigue. In addition, we use Eq. 11 to calculate the label effectiveness of the different sampling methods. We report the details in Fig. 14 Appendix A.3 and discuss the findings here.
The NN active learner combines an apparently easier-to-answe query composition with the benefits of active learning and increases accuracy (by the number of skipped queries) with LE = 1.95%, followed by non-active with LE = 1.39%. InfoTuple follows closely with LE = 1.26% due to its raw gains of accuracy. Even though more queries were skipped, responses were more informative overall. Random and Random /NN sampling score worst due to the high amount of skips or lower gains in accuracy with LE 0.75% and LE 0.35% respectively. Furthermore, we may infer that skipping unsuitable queries may be necessary to annotate more consistency, see p6 with p15 or p11 in Fig. 9 The mean percentage of skipped samples is 22.22% for NN queries, closely followed by Random/ /NN /NN sampling (22.36%) and the NN active learner (24.53%). Both methods compose queries from close neighbors of the query anchor, i.e., likely similar samples. InfoTuple querics are skipped 50.1% of the time and random tuples form the rear with 53.97%.
The methods that primarily focus on NN sampling perform best due to the higher likelihood of positive samples in a query. InfoTuple samples from larger pool of most informative samples, which may cause participants to skip more queries. However, its queries tend to be more informative that InfoTuple keeps up with the other methods.
Rispores thne in poconds [cl
30
arrocated Hippes
Despona secrede [e]
aiippe arnoisted
Naroo do time in socunds Pil
80
Pogina
Rmd
Arietti
NN
Roa
880
IodoToo Torooi
b
Depibit #1
Rndl
Rosnw
Arthe-
Inlo Lanie
30
20
20
10
152
200
250
1.50
202
300
query
query
query
(a) Skips vs responses p6. (b) Skips vS responses p15.
(c) Skips vs responses p11.
Figure 9: All response times for the least, the median, and the most consistent participant. We highlight the annotation phases.
4.4 Data Augmentation with Annotations
Participants' similarity functions may aligu well enough to profit from combining their annotations in order to augment their training dataset. This would increase sampling efficiency, as a larger collection of annotations would be available for warm-starting the training. Due to the comparable levels of intra-rater consistency and inter-rater reliability, such clusters seem realistic.
Our analysis selects the pair of participants with the highest inter-rater reliability. We exclude skipped samples and we select the pair with more consistent participants. we then initialize the training dataset with the combined set of both participants' Repeat #1 replics, fine-tune one model for each participant on their annotations selected by InfoTuple, and test the participants' own test sets. Recall that since the queries,
15

---

Published in Transactions on Machine Learning Research (04/2023)
0.70
0.65
0.60
0.35
0.90
P11
211 group))
(group)
Figure 10: The effectiveness of grouping the training data from participants with similar consistency and notions of similarity. Tt can be beneficial to combine their warm-up training datasets to increase triplet accuracy.
that make up the Repeat #1 data are identical, the combined annotations augment the training with replies to skipped queries and can even contain different replies.
We select the participants p3, pll, and p18 according to the hierarchical clustering Fig. 5 because pl1 is the most consistent participant overall. The results in Fig. I0 show the respective participant's differences the test set. Two of the participants benefit from additional annotations, the third sees a further decrease in accuracy.
Augmenting the warm-start dataset with the annotations of similar annotators can be beneficial. Our results for a triplet of participants, which also includes the highest inter-rater reliability, confirm it. This suggests that similarity functions may be clustered and used for transfer learning. However, our additional experiments with larger clusters yield diminishing returns and confirm our pre-study's 's findings, likely due to lower inter-rater reliability.
4.5
Discussion
RQ:: : How consistent and reliable are the participants?
Our evaluation shows that the participants are able to respond to the queries with relatively high quality, their intra-rater consistency is in the range of 0.498 (+.0111) average. Their self-assessment in the range of to 6 is relatively consistent with a divergence of 20%, which we show indicates whether a user-specific data collection is beneficial. Furthermore, we found that noisy oracles affect the network's ability to learn user-specific similarity functions. However, even though the more consistent group benefited more from fine-tuning, the tendency does also points towards smaller increase for less consistent. annotators.
RQ2: What is the best bascline heuristic to generate tuple queries?
The composition of tuple queries has implications beyond the learned model's triplet accuracy. Participants may take longer to respond or even find no response at all, which ultimately leads to fatigue. We show in Fig. 6 that the heuristics Random and NN yield a very similar triplet accuracy with Random/NN as a close third. However, once we account for the average response time, the gap between NN and Random on the one hand, and Randon/NN on the other increases. We see two possible explanations. First, the Nearest Neighbor sampling generates queries from the relatively sparse Euclidean embedding, that are more likely to contain easy positives that participants select more quickly. Conversely, queries generated by the Random heuristic contain easy negative samples and are often quickly skipped, Fig. The skipping renders Random sampling less efficient, as the procedure's sampling phases require more queries to collect the same number of annotations, which leads to higher fatigue. Finally, the combination of Random/ NNN mixes easy positives and negatives and seems to require closer inspection by participants and is less sample efficient because of this, see Fig. Sb.
Overall, the NN heuristic balances the three metrics triplet accuracy, time spent on responses, and the number of skips and fatiguing of participants better than the other baselines Random and Random/NN.
16

---

Published in Transactions on Machine Learning Research (04/2023)
RQ3:
Does active sampling method perform better than baselines?
We show that amnotation can be more valuable due to Active Learning, as the differences between ActiveNN and the baseline NN heuristics show. Furthermore, selecting informative samples with InfoTTple increases sample efficiency compared to ActiveNN, because a purely NN-query may not be maximally informative due to the sparse similarity of our problem domain. Response times are comparable to the heuristic baselines. Interestingly, the increases in accuracy per skipped sample are favor of the ActiveNN sampler compared to Inpol while the absolute triplet accuracy of models trained using InfoTuple is higher. Additionally, we show that the gained accuracy per time justifies the computational complexity of AL. Overall, Active Learning and specifically InfoTuple perform better than heuristic baselines.
Quality of Annotations
The quality of annotations is an important factor in machine learning. Specifically for Active Learning, applications typically deal with expensive domain experts, that may perform more reliably, but whose time is more expensive. Alternatively, non-expert annotators may seem more cost-cffective but are likely to be less reliable (Wu et al. 2020). In our study, we address the scenario of mixed expertise and evaluate the deep metric learning objective with respect to quantitative and self aelf-assessment. We show that self-assessmeen can be predictive of the usefulncss of Active Learning compared to strong bascline metric Icarner. We show that transfer learning can be applied successfully using more consistent, and potentially also semantically similar annotations.
Broader Applicability of the Proposed Method
The ideas proposed in this paper are twofold. First, the proposed deep active learning method may be applied to other tasks. Learning similarity of trajectory data is common problem, e.g., in other team sports like basketball (Sha et al. 2016), hockey (Chen et al. 2005 or Australian football (Alexander et al. 2022 in animal farming CChen et al. 2005), or in personal mobility, traffic or public transportation (Shen et al. 2019 Yadamjav et al. 2020 Fernández et al.) 2017). Still, the propose method uses flexible learner that may be adapted for other data than trajectories, e.g., learning similarity in image datasets such as Food73 (Wilber et al.) or ranking the age of laces (Liu et al.) and learning similarity between lexts (Neculoiu el al.).
Generally, the proposed method may be considered with the aim to train (or fine-tune) generalizing Metric Learner from few relative comparisons of large tuple size with the help of a human-in-the-llop.
Broader Evaluation of Active Learning
The second idea that we propose is our call for a broader evaluation of new active learning algorithms. In Active Learning, the predominant performance metric seemss to be accuracy per labeled sample. There are, however, additional practically important considerations besides accuracy. Experimental platforms, such as NEXT (Jamieson et al., 2015). capture detailed statistics on, e-g., network response time, timing data that helps to estimate human fatigue, and also the quality of annotations, ie., via intra-rater agreement. More recent AL studies Canal et al. (2020) additionally analyze the tasks' difficulty and response time. Hence, in our work, we do not only rank methods by accuracy. Our analysis of effective accuracy over (total) time spent on responding, and the efficiency of accuracy by skipped queries support the results. The adapted InfoTTuple method, which evaluates the Nearest Neighbors sampled from the neural network embedding, shows to be less fatiguing as well.
5 Related Work
Many approaches for deep metric learning use triplets to learn generalizing representations (Hoffer & Ailon 2015. Xuan et al. 2020). Recently, Xuan et al. (2020) introduced the idca of sampling triplets with casy positive examples. This addresses the issue of over-clustering of the learned embedding, and it is volerant of high-class variance. This is highly relevant to our study because it motivates the composition of tuple queries. Our study evaluates the effect of easy positive and negative samples with a human-in-the-loop
Active learning uses insights into the data distribution or model views to select the sample(s) that best improves the model's fit of the underlying data distribution. We can group approaches for deep learning into different categorics. Some estimate model uncortainty to select the most informative (Houlsby et al. 2011 Kirsch et al. (209)) or uncertain samples (Gal et al. 2017 Beluch et al. (2018), others select representative combine diverse and uncertain selections AAsh
samples using covering set. (Sener & Savarese 2018)
17

---

Published in Transactions on Machine Learning Research (04/2023)
et al. 2020 Nadagouda et al. 2022 use information theory to perform active metric learning and active classi fication on deep probabilistie model that uses Monte Carlo dropout sampling (Gal & Ghahramani 2016) as an approxination of uncertainty. They sample the most informative NN queries from all possible ones. Similarly, Canal et al. (2020) learn an ordinal embedding by maximizing the mutual information of queries to an oracle. Their work is based on a probabilistic model, fitted to relative comparisons, that learns the ordinal embedding. Both Nadagouda et al. (2022) and Canal al. (2020) use larger tuple sizes than three, which improves sample efficiency and also provides more context for easier annotations.
Information retrieval from multi-agent trajectory datasets has to first solve the assignment problem of these
2016 Löffler et al. 2020), and second, provide a quickly scarchable Di et al. 2018 Löffler et al., 2020). Di et al. (2018) additionally learn ranking from humans. They estimate the assignment of agents (players) within a tree-structure (Sha et al. 2017] and then train convolutional autoencoder on trajectory plots to extract features. In contrast, we train one neural network Löffler et al. (2020), that learns to solve the assignment problem and produces lower dimensional embedding, such that additional tree structures are redundant and search is accelerated. Di et al. (2018) learn ranking from participants of a tick-through study, that collects only pairwisc comparisons, and trains lincar rankSVM on these tuples and the extracted features of the autoencoder. This approach differs from this paper, as we optimize the network's embedding directly.
agents' trajectories (Sha et al. 2017
representation (Sha et al., 2016 2017
However, the embedding may be used for on-top learning to rank approaches DDi et al. 2018 or even active embedding search (Canal et al, 2019
6 Conclusion
In this work, we studied how to learn unknown similarity functions, that humans innately use to measure similarity between instances in an unlabeled, unstructured dataset. To learn these metrics from few amno- lations, adapted mining easy positive triplets from query sample's neighborhood in a neural network's embedding space and applied InfoGain to construct the most meaningful queries from the sparse similarity of a football trajectory dataset. Our user study shows the benefits of this mothod and provides a manccd evaluation with respect to accuracy and the practical considerations of the effectiveness of Active Learning with a human-in -ntta-----he--oopp response, total and label effectiveness. An accompanying survey sheds light on the mixed expertise of participants, their diverse notions of similarity, and their relation to a strong metric learning bascline. Future work may leverage the user-specific ordinal embedding to perform iterative active search, that can further improve information retrieval.
Author Contributions
The main author Christoffer Löffler implemented algorithms, designed and conducted experiments, wrote and revised the mamuscript. Kion Fallah and Stefano Fenu contributed to the theoretical framework and experimental design, offering insights. Kion Fallah and Dario Zanca provided manuscript feedback. Christo- pher Rozell guided the rescarch, including experimental and algorithmic design. Christopher Mutschler supervised and helped refine the experiments and methodology, and provided manuscript feedback. Björn Eskofier supervised the project and provided manuscript feedback.
Acknowledgments
We would like to acknowledge support for this project from the IFI programme of the German Academic Ex- change Service (DAAD), the Bavarian Ministry of Economic Affairs, Infrastructure, Energy and Technology as part the Bavarian project Leistungszentrum Elektroniksysteme (LZE) and the Center for Analytics- Data-Applications (ADA-Center) within the framework of "BAYERN DIGITAL II". We thank the study participants for their valuable contributions. Special thanks Nicolas Witt and Robert Marzilger for their support in organizing the study.
18

---

Published in Transactions on Machine Learning Research (04/2023)
References
Jeremy P. Alexander, Karl B. Jackson, Timothy Bedin, Matthew A. Gloster, and Sam Robertson. Quanti- fying congestion with player tracking data in Australian football. 17(8):o0276637, 2022. ISSN 1932-6203.
doi: 10. 10.337///urraal..ooe.07777
Jordan T. Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal. Deep batch active learning by diverse, uncertain gradient lower bounds. In International Conference on Learning Representations (ICLR), Virtual Conference, Formerly Addis Ababa Ethiopia, 2020.
Shaojic Bai, J Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXi preprint arXiv:1803.011 2018.
William H Beluch, Tim Genewein, Andreas Nürnberger, and Jan M Köhler. The power of ensembles for active learning in image classification. In IEEE Conference on Compuler Vision and Pallern Recognation (CVPR), pp. 9368 9377, Salt Lake City, UT, 2018.
Jurgen Bernard, Marco Hutter, Matthias Zeppelzauer, Dieter Feliner, and Michael Sedlmair. Comparing Visual-Interactive Labeling with Active Learning: An Experimental Study. IEEE Transactions on Visu- alization and Computer Graphics, 24(1):298-31) 308, January 2018. ISSN 1077-2626, 1941-0506, 2160-9306. doi: 10. /0110//VCCG201..2277.2
Alina Bialkowski, Patrick Lucey, Peter Carr, Yisong Yue, Sridha Sridharan, and lain Matthews. Large- scale analysis of soccer matches using spatiotemporal tracking data. In International Conference on Data Mining, pp. 725 730, Los Alamitos, CA, USA, 2014. IEEE, IEEE Computer Society.
Gregory Canal, Andy Massimino, Mark Davenport, and Christopher Rozell. Active embedding search via noisy paired comparisons. In International Conference on Machine Learning, pp. 902-911. PMLR, 2019.
Gregory Canal, Stefano Fenu, and Christopher Rozell. Active ordinal querying for tuplewise similarity learning. Proceedings of the AAAI Conference on Artificial Intelligence, 33(04))33322-33 33333333333-330000 Apr. 2020. doi: (0660///aai.334114....
Enrique Canessa, Sergio E Chaigneau, Sebastián Moreno, and Rodrigo Lagos. Informational content of cosine and other similaritics calculated from high-dimensional conceptual property norm data. Cognâtive Processing, 21(4):601 614, 2020.
Gal Chechik, Varun Sharma, Uri Shalit, and Samy Bengio. Large scale online learning of image similarity through ranking. Journal of Machine Learning Research, 11(3), 2010.
Lei Chen, M. Tamer Özsu, and Vincent Oria. Robust and fast similarity search for moving object trajectories. In Proceedings of the 2005 ACM SIGMOD International Conference on Management of Data SIGMOD doi: 10.1145/106615711662111
'05, pp. 491. ACM Press, 2005. ISBN 978-1-59593-0660
Mingyang Di, Diego Klabjan, Long Sha, and Patrick Lucey. Large-scale adversarial sports play retrieval with learning to rank. ACM Transactions on Knowledge Discovery from Data (TKDD), 12(6))1-18, 2018.
Esther Calvo Fernéndez, José Manuel Cordero, George Vouros, Nikos Pelekis, Theocharis Kravaris, Harris Georgiou, Georg Fuchs, Natalya Andrienko, Gennady Andrienko, Enrique Casado, et al. Dart: machine- learning approach to trajectory prediction and demand-capacity balancing. SESAR Innovation Days, Belgrade, pp. 30, 2017.
Yarin Gal and Zoubin Ghahramani. Dropout a bayesian approximation: Representing model uncertainty in deep learning. In Maria Florina Balcan and Kilian Q. Weinberger (eds.), Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pp. 1050 1059, New York, New York, USA, 20 22 Jun 2016. PMLR.
Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data. In 34th International Conference on Machine Learning (ICML), Sydney, Australia, 2017.
19

---

Published in Transactions on Machine Learning Research (04/2023)
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the Conference on Compuler Vision and Pollern Recognition, pp. 770-778, 2016.
Elad Hoffer and Nir Ailon. Deep metric learning using triplet network. In International workshop on similarity-based pattern recognition, pp. 84-92. Springer, 2015.
Neil Houlsby, Ferene Huszár, Zoubin Ghahramani, and Máté Lengyel. Bayesian active learning for classifi- cation and preference learning. arXiv preprint arxix:1112, 11..5111..77, 2011.
Kevin Jamieson, Lalit Jain, Chris Fernandez, Nicholas Glattard, and Rob Nowak. Next: A system for real-world development, evaluation, and application of active learning. Advances in neural information processing systems, 28, 2015.
Andreas Kirsch, Joost van Amersfoort, and Yarin Gal. Batchbald: Efficient and diverse batch acquisition for deep bayesian active learning. In Advances in Neural Information Processing Systems (NewrIPS), pp. 7026 7037. Vancouver, Canada, 2019.
Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly, 2 (1-2):83 97, 1955.
Colin Lea, Rene Vidal, Austin Reiter, and Gregory D Hager. Temporal convolutional networks: A unified approach to action segmentation. In European conference on computer vision, pp. 47-54. Springer, 2016.
Hao Liu, Jiwen Lu, Jianjiang Feng, and Jie Zhou. Ordinal Deep Feature Learning for Facial Age Estimation. In 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017), pp. 157-164. IEEE. ISBN 978-1-5090-40233 doi: 10.1109/FG.2011.225
Christoffer Löffler, Christian Nickel, Christopher Sobel, Daniel Dzibel, Jonathan Braat, Benjamin Gruhler, Philipp Woller, Nicolas Witt, and Christopher Mutschler. Automated quality assurance for hand-held tools via embedded classification and automl. In Joint European Conference on Machine Learning and Knowledge Discovery in Dalabases (BCML-PKDD), Ghent, Belgium, 2020.
Christoffer Löffler, Luca Reeb, Daniel Dzibela, Robert Marzilger, Nicolas Witt, Björn M. Eskofier, and Christopher Mutschler. Deep siamese metric learning: highly scalable approach searching unordered sets of trajectories. ACM Trans. Intell. Syst. Technol., 13(1,, nov 2021. ISSN 2157-6904. doi: 10.1145/ 3465057.
Michael Lohaus, Philipp Hennig, and Ulrike von Luxburg. Uncertainty estimates for ordinal embeddings. arXiv preprint Xrxi:: 11655, 2019.
Leland Mclnnes, John Healy, Nathaniel Saul, and Lukas Grossberger. Umap: Uniform manifold approxima- tion and projection. The Journal of Open Source Software, 3(29):861, 2018.
Namrata Nadagouda, Austin Xu, and Mark A. Davenport. Active metric learning and classification using similarity queries. arziv preprint arXiw:2202. 0,1523,0 2022. doi: 00.48550/ARXIV..O 2202.01953.
Paul Neculoiu, Maarten Versteogh, and Mihai Rotaru. Learning Text Similarity with Siamese Recurrent Net- works. In Proceedings of the 1st Workshop on Representation Learning for NLP, pp. 148 157. Association for Compulational Linguisties. doi: 10.18653/v1 /111661667..
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. In International Conference on Learning Representations (ICLR), Vancouver, Canada, 2018.
Burr Settles. Active learning literature survey. Technical report, University of Wisconsin-Madison Depart- ment of Computer Sciences, 2009.
Long Sha, Patrick Luccy, Yisong Yue, Peter Carr, Charlic Rohlf, and lain Matthews. Chalkboarding: A new spatiotemporal query paradigm for sports play retrieval. In Proceedings of the 21st International Conference on Intelligent User Interfaces, pp. 2016.
20

---

Published in Transactions on Machine Learning Research (04/2023)
Long Sha, Patrick Lucey, Stephan Zheng, Tachwan Kim, Yisong Yue, and Sridha Sridharan. Fine-grained retrieval of sports plays using tree-based alignment of trajectories. aiv preprinl orx: 711.022555, 2017.
Zhi-Hao Shen, W. Du, X. Zhao, and Jianhua Zou. Retrieving similar trajectories from cellular data at city scale. ArXio, //////9977.1771 2019.
Edgar Simo-Serra, Eduard Trulls, Luis Ferraz, Jasonas Kokkinos, Pascal Fua, and Francesc Moreno-Noguer. Discriminative Learning of Deep Convolutional Feature Point Descriptors. In 2015 IEEE International Conference on Computer Vision (ICCV), pp. 118 126, Santiago, Chile, December 2015. IEEE. ISBN 978- 978-1-4673-8891-2 doi: 00.1099/ICCV.201522.
Omer Tamuz, Ce Liu, Serge Belongie, Ohad Shamir, and Adam Tauman Kalai. Adaptively learning the crowd kernel. arXiv preprint arXiv:1105.1033 2011.
Laurens Van Der Maaten and Kilian Weinberger. Stochastic triplet embedding. In 2012 IEEE International Workshop on Machine Learning for Signal Processing, pp. 1-6. IEEE, 2012.
Gang Wang, David Forsyth, and Derek Hoiem. Comparative object similarity for improved recognition with few or no examples. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 35322. IEEE, 2010.
Michael J Wilber, Uljung S Kwak, and Serge J Belongie. Cost-Effective HIT's for Relative Similarity Com- parisons.
Jian Wu, Victor S Sheng, Jing Zhang, Hua Li, Tetiana Dadakova, Christine Leon Swisher, Zhiming Cui, and Pengpeng Zhao. Multi-label active learning algorithms for image classification: Overview and future promise. ACM Computing Surveys (CSUR), 53(2):1-33 2020.
Hong Xuan, Abby Stylianou, and Robert Pless. Improved Embeddings with Easy Positive Triplet Mining. In 2020 IEFE Winter Conference Applications of Computer Vision (WACV), pp. 2463 2471, Snowmass Village, CO, USA, March 2020. IEEE. ISBN 978-1-72816-553-0 doi: N0.1109/WACYV WWACCV45572.200000909333
Munkh-Erdene Yadamjav, Zhifeng Bao, Baihua Zheng, Farhana M. Choudhury, and Hanan Samot. Querying recurrent convoys over trajectory data. ACM Trans. Intell. Syst. Technol., 11(5), August 2020. ISSN 2157- 6904.
Yu Zheng. Trajectory Data Mining: An Overview. 6(3):1 41, 2015. ISSN 2157-6904, 2157-6912. doi: 10.1145/2743025.
A Appendix
A.1 Pre-Study
We conducted pre-study to set the size of InfoTuple and also the number of required annotations for user- individual model training. We furthermore have discovered that model would not generalize well from the complete, diverse group to one hold-out user. We also collected feedback on the introduction (Lutorial) and query format, which helped to improve it and to avoid confusion among participants.
Size of InfoTuple. The pre-study was conducted with 7 participants, that were split into two groups A with 4 participants and B with participants. Group A was queried with tuples of size 7 (1 head, 6 body), and group B with smaller tuples of size 4 (1 head, 3 body). We monitored response times and found that both groups responded within a similar amount of time. Furthermore, participants of group B reported that they often could not select a similar sample. Hence, we determined that a larger tuple size would not negatively impact response times, but would provide additional context that would help respond to the query, consistent with Canal et al. Canal et al. 2020
Count of InfoTuple. Second, we set the required number of annotated triplets by, first, collecting 500 annotated triplets per participant, and second, fine-tuning model for each in increasing sub-sets sizes, that
21

---

Published in Transactions on Machine Learning Research (04/2023)
group
notion
example
semantic
Leam movement.
direction (attack or defense); dynamic or slatic pass or shot; short or long; at rest
ball movement
scmantic category
standards (kickoff); situations (pressuring, passing, shot); consistency of calegory
team formation area
compact; shifting; offensive or defensive
which third; which flank
proximily
player and ball directions
Are there players near the ball?
direction of ball trajectory
length of trajectory distances
spatial proximity similar candidate scenes
Table 1: Qualitative clustering of responses to questionnaire into semantic and prorimity.
randomly sampled from the total pool. The increase in accuracy converged over the increasing number of training data. Hence, we chose to collect 20 Info InfoTuple of size 8 (140 triplets) per method, more showed diminishing returns.
Individual or general model. Lastly, we observed that we were not able to train one model by combining the training data from all participants into one large pool. We used leave-one-out cross-validation to train on all but one participant and then tested on the remaining participant as a hold-out test dataset. The general model did not achieve good results training an individual model. Hence, we designed the user study such that individual models were trained.
A.2 Notions of Similarity
We hypothesize that different notions of similarity may either already be captured by the Euclidean distance metric, be learned better from annotations. Recall that have collected additional metadata participants' notion of similarity using questionnaire, see Sec. 3.2 This may help distinguish participant groups.
We test this hypothesis by analyzing the participants' responses to the questionnaire and extract two sets of rules, that we categorize as either semantic or prozimity, Semantic rules describe the scenes' semantics whereas proximity rules largely ignore such interpretations. They instead pay closer attention to proximity only to subsets of trajectories. To reduce bias, ask two raters to assign participants to either group and then have the raters resolve their disagreements through analytical discussion. Here, we compare the triplct accuracy of the agroed-upon group compositions.
1.0
0.9
0.9
Triplet accurscy
0.8
0.7
0.6
0.5
6.5
trained
trained
Fine
Training veriant
variant
(a) prosimity notion on Rnd.
(b) semantic notion on Rnd.
Figure 11: This figure distinguishes two types of similarity and shows the effectiveness of fine-tuning via InfoTuple active learning in comparison the pro-trained bascline. Participants are grouped by their qualitative description of similarity, that is either proximity semantic.
We show the extracted groups in Table each group's triplet accuracy in Fig. We assign participants to either group if they describe specific notion provide examples that match it. The resulting prozimity
22

---

Published in Transactions on Machine Learning Research (04/2023)
group contains 8 participants and the semantic group 8, too. We excluded participant that did not describe their notion and another that did not fit. into any. We observe that high self-assessed expertise correlates with an assignment to the semantic group. Furthermore, we see an increase in triplet accuracy from pre-trained models to fine-tuned models for both groups in Fig.
The increase triplet accuracy is visible for both the semantic and the proximity groups. Hence, we can not show a difference from qualitative clustering of notions of similarity. However, fine-tuning can outperform the Siamese bascline in both cases, because the Euclidean similarity metric does not adequately capture either group's similarity function. In future work, alternative interpretations of qualitative and quantitative data may find more suitable (sub) group segmentation.
A.3 Additional Results
Fig. 2 shows the effectiveness of fine-tuning compared to the Siamese baseline per participant Fig. shows every participant's response times for each phase. Finally, Fig. [4 shows the adapted triplet accuracy for three variants of effectiveness: response, total and label effectiveness.
Togelt acouaty
0.25
0.36
0.38
0.41
0.44
0.47
0,47
0.48
0-48
0.50
0.52
0.55
0.55
0.55
0.57
0.02
0.64
0.75
Consistency
Figure 12: The effectiveness of all fine-tuning variants (violin plot) in comparison to the pre-trained Siamese
network (horizontal bar) shows the impact of inconsistent oracles and a performance ceiling for the relatively noisy train and test data.
50
40
in
10
Particpants
Figure 13: We show all users' response times for each phase ordered by consistency.
23

---

Published in Transactions on Machine Learning Research (04/2023)
Rall Iripee arrurery per tate time [s]
Wel. smplect acouracy per shupping
0.003
0.03
0.032
0.002
0.02
0.033
Fettim (MK)
Activn 1900
Arttien (MRO
0.020
Fondom
Activc frdellspe)
0.000
Actios Tisla Tup io ticoom Ra 1ocommoto
Random
Activv (ntoTTpte)
Ramddom/MN
-0.032
10.001
arquisitions
acqulsitions
No. acqubitions
(a) Accuracy per response time.
(b) Accuracy per total time.
(c) Accuracy per skipped samples.
Figure 14: We average the results for all users on the test dataset. (a) shows the triplet accuracy per response time spent. (b) shows the triplet accuracy per time including compute time, which depends on compute resources and may limit the experiment's 's scale. (c) shows the triplet accuracy over the number of skips, which can cause fatigue.
A.4 Evaluation of Ellipse Template
We present the experimental comparison our proposed ellipse template matching algorithm from Sec. 2 with the assignment methods proposed by Löffler et al. (2021). Their best-performing method uses an over- determined role position assignment, that introduces sparsity in the inputs of the Siameso network because there are more positions in the template than trajectories in the scene. Our proposed ellipse template has the same number of positions as there are trajectories, and we fit the template to each scene, see Sec. 2 for details.
We reproduce the results of Löftler el al. 2021 and compare them with our proposed template matching algorithm. For that, we use a Siamese network with an embedding size of 64, pre-train on 304 games (see Section 3.4), and perform an evaluation of ours and their method on the same subset the dataset that we use in our main study. We follow their evaluation scheme and calculate the structural correspondence between the original data and the learned representations. They define the Mean Absolute Percentage Error (MAPE) the expected absolute error relative to the ground truth, as
D
dens (Xi,1, X1,2 - dens (f(X.1), f(Xi2)) dens X X.2)
MAPE
(12)
D
where they sample and compare pairs of matrices X,, and X from a dataset matrix pairs D. We adapt our notation from Sec. 2 slightly, such that X,, refers to the first matrix of the ith pair and X to the second matrix. Likewise, f(X;,1) refers the learned representation of X,,1.
For their best performing method role position, Löffter et al. (2021) report a MAPE of 2.66% on their validation and 2.68% on their test set. Paired with the embedding size of 64, this also achieves speedy retrieval due to the lower dimensionality. We reproduce similar MAPE of 2.669% on our dataset, which is in line with their results. In comparison, our ellipse template achieved a MAPE of 2.31% with the same hyperparameters. This clearly shows the benefits of our proposed method, as its error is 0.359% lower, and at the same time, eliminates input sparsity.
A.5 Instructions
The user study uses a web-based annotation tool, that begins with a landing page with a short introduction. Participants are first presented with an introduction that familiarizes them with the data, the task, and the mechanics of the annotation tool. Fig. 15 shows the explanation of the data and description of details. Fig. 16 then describes the participant's task in this study. Finally, Fig. shows example of query and once more explains the mechanics of the annotation process. After this participants begin annotating the warm-up samples with button press.
24

---

Published in Transactions on Machine Learning Research (04/2023)
Introduction
This data collection aimms to learn what you perceive as similaniy for football scenes.
How does the data look like?
You will see plots of nort scenes [5s] with two teamss and ball.
the bal. The gama's direction is alweys from left right, Le,, the team in ball passession plays from left to right tracking data,
Blue teamn (left)
is opponent.
trajectories
again
end
beginring
Trajectories move towands the marker,
8:50 035
Figure 15: The landing page begins with an introduction to the dataset.
Your Task
You will compare an ANCHOR scene with other coenes.
has yellow background:
decide which other locks simiar click
If none looks similar to you, please click the large green buttor:
No scene similar,
Figure 16: The next part describes the task and familiarizes participants with their options.
25

---

Published in Transactions on Machine Learning Research (04/2023)
Thase us the shae scieec fatt cal choies coor di Foats
Figure 17: An example concludes the explanation and previews the annotation task that follows.
26


Reviews:
[{"review_id": "Atqlr8ahWT", "reviewer": "Reviewer_UmLB", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n- The motivation for active learning of ordinal embeddings is reasonable and interesting.\n - A user study on football data is performed to verify the proposed method.\n\nWeakness:\n- Introduction describes less about the user study on football data. It would be better to motivate why this user study is suitable for the active learning method.\n- Some notations are unclear or wordy. For example, in Equation 1, there seems no need to expand all elements in X; In Equation 2, what is the difference between $X_2$ and $X’_2$? I cannot find the definition of $X’_2$.\n- The dataset is small. Any overparameterization risk? It means that the number of learnable parameters is much larger than the number of training samples. The authors should analyze the influence of this risk.\n- There are some typos. For example, on page 3, “multiple agents which leads to” should be “… which lead to”.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work presents an active learning method to learn a distance function from few human annotations. In particular, the authors emphasize the tuple composition, sample selection, and a real case on the football dataset. Experiments are done to verify the proposed method.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "1. Careful proofreading\n2. Clear Clarification on notations\n3. Analysis of overparameterization risk on the small dataset or Empirical results on larger datasets", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "aqMw4CJWeM", "reviewer": "Reviewer_1z6h", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- This paper studies an interesting problem of ordinal embedding on football data\n- The authors conduct extensive experiments\n\nWeakness:\n- The technical novelty of the paper is rather limited. It simply adopts InfoTuple for football data. The authors might want to submit to some application oriented journal.\n- The dataset used is very small and one one dataset is used. The authors might need to conduct experiments on more larger datasets\n- The authors should also include more state-of-the-art network embedding approaches as baselines", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper adapts InfoTuple for active learning of ordinal embeddings on football data. The main contribution is the application of InfoTuple on football data.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "1. The dataset used is very small and one one dataset is used. The authors might need to conduct experiments on more larger datasets\n2. The authors should also include more state-of-the-art network embedding approaches as baselines", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "gclWUNMiTW", "reviewer": "Reviewer_wkRi", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- empirically solid and provides a detailed description of the proposed system\n- have abundant results to support the proposed methods and conducted real human study\n\nWeakness:\n- proposed method is very specific to the task and unclear if how the submission will be relevant to the general audience of TMLR\n- The writing is not super clear for audience who are not already very familiar with the task of multi-agent trajectory tracking", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper describes a pipeline of active learning for the task of football trajectories matching. It describes a compreshensive description on how to design this system, which includes learning a convolutional network, sampling hard examples for human annotators (i.e. active learning), and other important details such as active learning. The submission also presents a detailed description of actual human study and shows that the proposed method performs better than other baselines such as popular prior method of matching the trajectories by eucledian distance or non-actively learning. Overall, I think the submission is technically solid. \n\nThe downside is that the prescribed methods are very much specific to the application of trajectory matching. Also, perhaps given my unfamiliarity with the application, I find the current draft hard to follow at times.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "My main request is around writing. I think in the first few paragraphs of the intro, the paper should concisely define the task of \"multi-agent position tracking\". The paper should also motivate why this is an interesting and important task. Then in section 2.1, the paper should give concrete examples to the technical notations. For example, what is a \"trajectory\"? How is the data in X collected?\n\nI also encourage the author to think about the general implication of their proposed method. Can some of the ideas here be applicable to other tasks as well?", "broader_impact_concerns": "I do not see negative societal impact from this paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "7CljzcoJst", "reviewer": "Reviewer_zZvF", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well written in general. Authors address an important problem with an interesting application. The research questions posed in the paper also highlight several practically important challenges. The proposed methods and questions were examined in experimental analyses.\n\nMain weaknesses of the paper are as follows:\n\n1. Mathematical statements and notation: Some notation such as the definition of B^n has typo. It can be also incorrect since you apply set union on B^n while it is defined as a list. There are several unclear notation and statements like this.\n\n2. There are some over-claims such as “We prove the efficiency of our method with a user study on active learning methods on real-world data, analyzing strengths and limitations.”. Indeed, this claim is experimentally analyzed.\n\n3. Following (2) and similar claims, experimental analyses of the paper should be improved. In addition, mathematical justification of the claims should be provided.\n\n4. In the analyses, some simple CNNs have been employed. The results should be examined using different types of deep neural networks, or an ablation study with different CNNs should be provided to explore generalization of the results for different models.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work addresses the problem of learning user-defined similarity functions from few annotations. To this end, an entropy-based active learning method was employed to obtain annotations from human participants and use them to train a convolutional neural network (CNN). \nThe experimental results show that the proposed method improves the quality of the information retrieval compared to a baseline metric learning method. The strengths and weaknesses of passive sampling heuristics and active learners are examined in experimental analyses.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Three major improvements are required:\n1. Proof-reading paper by fixing mathematical notation and statements.\n2. Either providing mathematical justification of claims, or revising the claims.\n3. Improvement of the analyses using additional ablation studies with different deep neural networks.", "broader_impact_concerns": "A broader evaluation of methods was given and some concerns were discussed.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an entropy-based active learning algorithm to do football trajectory matching. The reviewers agree that this is an interesting application with solid experimental results and a user study conducted by the authors. However, the common consensus is that the paper can be strengthened by showing the potential impact of the proposed method with more large-scale datasets and by adding more extensive baseline methods. There were several comments on the presentation of the paper, some of which have already been addressed. At this point, I recommend acceptance with a minor revision, where the authors address all presentational and clarification questions raised by the reviewers.
Decision: Accept (poster)