An information-theoretic approach to unsupervised keypoint representation learning
Authors: Ali Younes
Simone Schaub-Meyer
Georgia Chalvatzaki
Venue: Submitted to ICLR 2023 (2023)
Keywords: representation learning
keypoint discovery
unsupervised learning

Abstract: Extracting informative representations from videos is fundamental for the effective learning of various downstream tasks. Inspired by classical works on saliency, we present a novel information-theoretic approach to discover meaningful representations from videos in an unsupervised fashion. We argue that local entropy of pixel neighborhoods and its evolution in a video stream is a valuable intrinsic supervisory signal for learning to attend to salient features. We, thus, abstract visual features into a concise representation of keypoints that serve as dynamic information transporters. We discover in an unsupervised fashion spatio-temporally consistent keypoint representations that carry the prominent information across video frames, thanks to two original information-theoretic losses. First, a loss that maximizes the information covered by the keypoints in a frame. Second, a loss that encourages optimized keypoint transportation over time, thus, imposing consistency of the information flow. We evaluate our keypoint-based representation compared to state-of-the-art baselines in different downstream tasks such as learning object dynamics. To evaluate the expressivity and consistency of the keypoints, we propose a new set of metrics. Our empirical results showcase the superior performance of our information-driven keypoints that resolve challenges like attendance to both static and dynamic objects, and to objects abruptly entering and leaving the scene.

Under review as a conference paper at ICLR 2023
AN INFORMATION-THHORETII APPROACH TO UNSUPER- VISED KEYPOINT REPRESENTATION LEARNING
Anonymous authors Paper under double-bling review
ABSTRACT
Extracting informative representations from videos is fundamental for effective learning of various downstream tasks. Inspired by classical works on saliency, we present novel coormation-theoretie approach to discover meaningful repre- sentations from videos in an unsupervised fashion. We argue that local entropy of pixel neighborhoods and its evolution in a video stream is valuable intrinsic supervisory signal for learning to attend to salient features. We, thus, abstract visual features into a concise representation of keypoints that serve as dynamic formation Iransporters. We discover in an unsupervised fashion spatio-temporally consistent keypoint representations, thanks to two original losses. First, a loss that maximizes the information covered by the keypoints in a frame. Second, a loss that optimizes transportation over time, imposing consistency of the information flow. We compare our keypoint-based representation to state-of- the-art baselines in different downstream tasks such as learning object dynamics. To evaluate the expressivity and consistency of the keypoints, propose a new set of metrics. Our empirical results showcase the superior performance of our information-driven keypoints that resolve challenges like attendance both static and dynamic objects, and to objects abruptly entering and leaving the scene."
-
INTRODUCTION
Humans are remarkable for their ability to form representations of essential visual entities and store information to effectively learn downstream tasks from experience (Cooper, 1990; Radulescu et al., 2021). Research evidence shows that the human visual system processes visual information in two stages; first, it extracts sparse features of salient objects (Bruce Tsotsos, 2005); second, it discovers the interrelations of local features for grouping them find correspondences (Marr, 2010; Kadir & Brady, 2001). For videos with dynamic entities, humans not only focus on dynamic objects, but also on the structure of the background scene if this plays key role in the information flow (Riche et al., 2012: Borji et al., 2012). Ideally, we want a learning algorithm to extract similar abstractions that can be useful to various downstream tasks. Notable research works in Computer Vision (CV) and Machine Learning (ML) have proposed different feature representations from pixels for challenging downstream tasks (Szeliski, 2010; Harris al., 1988; Lowe, 2004; Rublee et al., 2011: 1; Rosten Drummond, 2006; Mur-Artal et al., 2015). Recent efforts focus on deep learning representations of Points of Interest (Pol) for tasks like localization and pose estimation (DeTone et al., 2018; Florence et al., 2018; Sarlin et al., 2020; Ono et al., 2018; Sarlin et al., 2019; Dusmanu et al., 2019).
Keypoints stand out as Pol with semantic interpretation (Jiang et al., 2009; Alexe et al., 2010), e.g., representing objects (Xiongwei et al., 2020) or the joints of a human pose (Kreiss et al., 2019) and can represent structure useful for learning control (Xiong et al., 2021). Many keypoint detection methods are trained in supervised fashion, relying on annotations (Cao et al., 2017). Unsupervised and self-supervised learning methods can compensate for the need for expensive human annotations (Wang et al., 2019; 2020; Minaee et al., 2021; Kim et al., 2019; Yang al., 2020; Gopalakrishnan et al., 2020; Chen et al., 2019). Current state-of-the-art methods for unsupervised keypoint discovery mainly focus on dynamic entities in a video (Kulkarni et al., 2019; Minderer et al., 2019), not effectively representing the scene's 's static and dynamic entities. Namely, these methods are trained to
Project website: https: //si tes. gooal.goollee..

---

Under review as a conference paper at ICLR 2023
reconstruct differences between frames and cannot easily disambiguate occlusions consistently represent randomly appearing-therr then dieen-ppppe objects in a video stream.
This work introduces Maximum Information keypoiNTs (MINT), an information thhooretc treatment of keypoint--asse representation learning by considering keypoints as the "transporters" of prominent information in a frame and subsequently through video stream. Our proposed method relies on local entropy computed in neighborhoods (patches) around candidate keypoints. We argue that image entropy, and its changes over time, provide strong inductive bias for training keypoints represent salient objects, as early works in saliency detection pointed out (Kadir & Brady, 2001; Bruce & Tsotsos, 2005). To compute the entropy. we introduce a novel, efficient entropy layer that operates locally on image patches. MINT maximizes both the image entropy coverage by the keypoints and the conditional entropy coverage across frames. To do so, MINT relies on an original formulation of unsupervised keypoint representation learning with loss functions to maximize the represented image entropy and the information transportation across frames by the keypoints, imposing spatio-temporal consistency of the represented entities.
We provide qualitative and quantitative empirical results on four different video datasets that allow us to unveil the representation power of MINT against strong baselines of unsupervised keypoint discovery. Unsupervised keypoint representation learning is challenging to benchmark due to the absence of designated metries and datasets. We, therefore, provide new set of metrics with downstream task in the domain of multiple object detection and tracking, based on CLEVRER (Yi al., 2019). Moreover, we provide results two challenging datasets (Sharma al., 2018; Memmesheimer et al., 2019) that contain interesting dynamic scenes of various difficulties (close-up frames with dynamic interactions vs. high-res wide frames with clutter). We show that MINT economizes the use of keypoints, deactivating excessive ones when the information is well contained, and dynamically activates them to represent new entities entering the scene temporarily. Finally, to demonstrate the suitability of MINT as a representation for control, we devise an imitation learning downstream task based on the toy environments of the MAGICAL. benchmark (Toyer et al., 2020).
To summarize, our contributions are: (!) an original information- -hheoreticc approach to unsupervised keypoint representation learning that uses local image entropy as a training inductive bias, seeking to maximize the represented information in videos; (2) entropy layer to compute local image entropy in patches; (3) an unsupervised way for learning to represent a variable number entities in video streams by activating/deaaciivtnng keypoints to cover the necessary information: (4) a new set of evaluation metrics in a simple and intuitive downstream task for benchmarking the performance of unsupervised keypoint discovery methods.
2 MAXIMUM INFORMATION KEYPOINTS
We propose an unsupervised method for keypoint discovery in videos based on eeformationnthhoreee principles. Keypoints should adequately represent the scene and dynamic changes in it. Starting from the original assumption that keypoint represents patch of information on the image, we leverage local image entropy to measure the representation power of keypoints in terms of their transmitted amount of information. Consequently, we argue that keypoints should cover areas in the image that are rich in information, while the number of keypoints should dynamically adapt to represent new information. Finally, keypoints should consistently represent the same information pattern spatiotemporally in video. With this motivation, we propose maximizing the information covered by the keypoint representation in a video through the introduction of two novel losses based on informationn-hhorrtic measures. (1) An information maximization loss encourages the keypoints to cover areas with high entropy in a single frame. (2) An information transportation loss enables the keypoints to represent the same entity over subsequent frames. In the following, we present these losses and theoretical analyses supporting their design.
2.1 PIXEL INFORMATION & LOCAL IMAGE ENTROPY
Our information-theone theoretti approach for unsupervised keypoint discovery requires quantifying the amount of information each pixel location in single frame carries. We measure the information of a pixel via Shannon's 's entropy definition (Shannon, 2001), based on the probability of each pixel. Images can be considered lattices, with pixels being the random variables (Li, 2009). We compute the
2

---

Under review as a conference paper at ICLR 2023
(T)
K
Entropy
Layer
Σ
B(4)
Masked entropy
M,
Figure 1: The architecture of our keypoint model (b4)) (Sec. 2.2) and the masked entropy (Sec. 2.2.1). 1). For an input image It model (T+) outputs K feature maps f.") for each keypoint k. f f1, K}. A heatmap h,") is generated for each keypoint, while the active keypoints aggregated to form the mask M, The entropy layer computes the entropy of the image E(4). ). Our masked entropy (ME) loss maximizes the percentage of the entropy in the masked entropy image. Red arrows show the backward gradient flow. Only part encircled by the dashed line used during inference.
discrete probability density of pixel using the statistics of the color intensities in its neighborhood, represented by normalized histogram of the neighboring pixel values (Sabuncu, 2006). In order to compute these histograms efficiently and to derive the final entropy image, we developed computationally optimized entropy layer as detailed in Appendix B.
Our entropy layer estimates the pixel-wise entropy image E(T) for RGB input image I " x with H being the height and W the width of an image frame with 3 color channels. E(I) consists of the local entropies E[I (xx, y)) computed at each pixel location (x, y) by estimating the entropy of the neighborhood region R(x, y) centered at (ær, y), using a normalized histogram- discrete probability function p(b, Rx, y)) for each color value b in the region R(x, y) summed and normalized over the color channels (details in Appendix B). The final per-pixel local entropy is
FI( w))
Σ bE(0,255]
p(b, R(x, v)) log(p(h, R(x,y)))
(1)
Computing the entropy for all pixels results in the pixel-wise entropy image E()).
2.2 INFORMATION-TIEORTTT KEYPOINT DISCOVERY
We consider the keypoints as a compact representation of images, which attend to salient entities in scene (Szeliski, 2010). In our method, keypoints represent distinctive information patterns overlaid on a set of neighboring pixels an image frame. We explicitly treat the keypoint as a transporter of information in its surrounding patch, and develop an end-to-end approach for unsupervised keypoint discovery in videos, based on ceemmationnthooree principles.
We define keypoint discovery model (T,) (cf. Fig. 1), which is deep neural network that discovers (1.....K], in an input color image at time t. It outputs K feature maps r!", each corresponding one keypoint. The coordinates (I,,, y4)x of the respective keypoint k,()) obtained with a spatial soft-argmax (Levine et al., 2016). Besides predicting the coordinates, the s {0, 1} per keypoint. The activation status determines whether keypoint active (s = 1) or not (s = 0) in specific frame, allowing the network to learn to decide the ideal number of active keypoints. Overall, keypoint is defined by its coordinates and the activation score k(d) = (z,, V/, (()) Additionally, we define a differentiable heatmap X E T HW for each th keypoint by thresholding distance- Gaussian G centered at the coordinates of the keypoint (details Appendix A.3). The heatmaps localize the keypoint information coverage. We want to maximize the coverage of information by the keypoints spatio- temporally in the video stream. For that, we need to ensure that both the per frame and intra-frame information is sufficiently transmitted by the keypoints. Inspired by information theory, derive novel losses that allow us to learn information driven keypoint representations, while providing error bounds that theoretically justify the design of those losses (Sabuncu, 2006; Yu et al., 2021).
K keypoints kg¹)
model also assigns activation status
3

---

Under review as a conference paper at ICLR 2023
I1-1
B(4-1)
B 1)
Masked Conditional Entropy
B(L)
M,
Masked conditional entropy (MCE) computation. Given two consecutive images 7--1 and I., we extract their entropy images F1) and E(T,). The conditional entropy F(T((T+-1) depends on the entropy image of hoth images. Multiplying the conditional entropy image by the aggregated mask M, gives the MCE image. The masked conditional entropy (MCE) loss maximizes pereentage the masked conditional entropy.
Figure 2:
2.2.1 MAXIMIZATION OF KEYPOINT INFORMATION
With information maximization, we encourage keypoints to represent image regions rich informa- tion. We leverage local image information entropy (Sec. 2.1) to define losses for keypoint discovery. We want to enforce maximum collective information coverage by the keypoints in a frame for repre- senting all entities in a scene. For that, we introduce two losses; the masked entropy (ME) loss and the masked conditional entropy (MCE) loss.
The ME loss encourages the information coverage by the keypoints in single frame. We use the heatmap ( of each keypoint k(()) to retrieve the local image information at time t, We filter out inactive keypoints by multiplying the heatmap with the activation status (1) Aggregating all
heatmaps gives the aggregated mask M, min(E; h(()) @ (()) .))
Considering keypoints as channels of information, we arrive at the following theorem that bounds information loss by the keypoints' masking of the original image. The bound follows Fano's inequality (Sabuneu, 2006; Scarlett & Cevher, 2019). and proves that maximizing the keypoints' masked entropy lowers the probability of error of the information loss by this keypoint representation. Theorem 1. Let TAM be the masked image at time obtained by the operation I I @ Mt. where @ denotes the Hadamard (i.e., element-wise) product. Let B be the "vocabulary' of pixel intensities, and we assume that every pixel location (x, y) is uniform B. The average error probability Ps over all pixels N H x W of the information approximated by 1 w.r.t. to the original image 1, can be lower bounded by
CZ (x,)))) N log [B]
log 2
P 2
log [
Proof Appendix C. We can assume that the upper bound for the error probability remains because of the activation of the keypoints, there is probability that the masked image is "empty", i.e., all keypoints inactive. From Eq. (2) we can see that the ME maximization lowers the probability of error. This motivates the practical implementation of the ME loss (ME(1)) that optimizes the percentage of the masked entropy over the all pixel locations V), Ex, E(I,) @ M, w.r.t. the total image entropy En.y E(I.)
[ME(I))=1 =1 Eny E(I) M, =1 Eny E(T)) @ min (in Σ, E(I)
1 o o (0,1)
Σxy E(I)
The MCE loss encourages the keypoints to pay special attention dynamic entities when the available number of keypoints are insufficient for covering the information in sequence of frames. measures
The conditional entropy of an image the information change of pixels, indicating moved objects. Optimizing the conditional entropy 4-1) in a sequence of images encourages the keypoint detector to attend to moving objects (cf. Fig. 2). The pixel-wise conditional entropy of an image can be computed by the joint entropy of two images subtracting the reference image entropy I-11) = E(I, I--1) E(I-1)). where
time given reference image J,-1 at time
E(I, I-1) max E(I-1)) following Lemma 1.
Lemma 1. The joint entropy of two images I, and In can approximated by E(I, I2) s
mas (TTT)) ((()))
4

---

Under review as a conference paper at ICLR 2023
Accordingly, bound the information loss by the keypoints in sequence of frames.
Corollary 1. Following Theorem 1. we can bound the average probability of error Pemu of the conditioned masked images in time as
(z. 1)
log 2 log [B]
prond
N log [3]
Following Eq. (4), we observe that the MCE maximization lowers the probability of error of the conditional information loss between frames, leading to the practical implementation of the MCE loss, similarly to the ME loss, LMCE( [ 1t-1) that maximizes the percentage of total masked conditional entropy Ees 1 @ M, to the total conditional entropy Σx.y 1
LmC I--1) = EzN B( ④ M Eos ((1(1-11)
(5)
2.2.2 MAXIMIZATION OF KEYPOINT INFORMATION TRANSPORTATION
Keypoints should transmit information about the same entity over time. Temporal consistency means aligning each keypoint to the same information pattern across its occurrences. Parting from this idea and inspired by feature transportation (Kulkarni al., 2019), we propose infor- mation transportation (IT), relying on image entropy and waiving the need for image reconstruction.
B(L-)
B(4)
B(S
In a temporal sequence of frames, we can perform key- point information transportation by reconstructing the im- age information entropy of the current frame E(T:) using information from previous frame E(I--1) (ef. Fig. 3). The conditional image entropy of the two frames E 411,,, represents the amount of pixel-wise information needed to describe the information of E E(T)) given E(I,1). Let's consider the 5th keypoint at time step (coordinates are omitted for avoiding verbosity). Its associated heatmap ((()) represents the spatial information covered by the 7th keypoint. We can construct source information image F(S(()) by subtracting the information carried by the key- point in frames t 1 and f from the information image E(I,1),,i.. i.e., E(s!"))
B(R())
B(L)
I(G, R
Figure 3: Information transportation for a kcypoint k(),, The source is the entropy of the previous image E(It-1) and the target is the entropy of the current frame E(I;). Re- moving the patch around the keypoint posi- tion time t- 1 and gives the source infor- mation E(S( The sum of the conditional entropy B([,([t-1) and the patch around the kcypoint at time f- gives the target informa- tion ((T The reconstructed information after transportation is the sum of target and source information E F(R(")). The objective to maximize the mutual information between the reconstructed entropy and the entropy of the current frame I I(1,, R
((I-1))(1-A1")) (0 hk))).
(1-hk)))) (1))
Implanting the keypoint information represented by h,))
onto the conditional image entropy ((1)) that con- tains all conditional information except for the one of the 7th keypoint forms the target pixel-wise information E(T(()) E(L) @ (h!") (E(I,)ee-1) @ (1 - n1 ) The reconstruction of the image information E(I,) results from the sum of the source and target image information E(R! = E(S(()) + ()T This reconstruction pro- cess leads us to the following theorem, that shows that maximizing the mutual information (MI) between the per keypoint reconstructed information and the original image entropy, lowers the probability of error due to information loss.
Theorem 2. Following Fano's 's inequality (Sabuncu, 2006; Scarlett & Cevher, 2019). we prove that the average error probability of the transportation for the 7th keypoint (f) is lower bounded by
Ez,y I(Ie(z, (2, v), (2) (x.v))
log log B
( > 1
(6)
N Nlog B
"The factor < encourages the network concentrate on transportation more than reconstruction.
5

---

Under review as a conference paper at ICLR 2023
Considering all transportations independent, we can bound the joint average eroor probability, by summing the per keypoint error probability
E Ein [(I,(1, (), (x,
K log log B
pi (oint) 2 K
(7)
N log [
Proof in Appendix C. From Eq. (6), we deduce that for optimizing the 8th keypoint's information transportation, we should maximize the mutual information (MI) I(I,, R, This motivates our practical implementation of the information transportation (IT) loss for all keypoints. We construct the IT loss through the difference E(4) [[,, R normalized by the area of the heatmap A,
(equal for all keypoints). Minimizing E(I,) [[,,, R(()) maximizes MI, as dictated by Theorem 2. We found that normalizing with A,, helps in practice have a better scale. We, also, regularize the excessive keypoints' movement by minimizing the norm of the distance traveled by each keypoint
d't) [|(2;, v:)e (zi, v:):-1ll (scaled by weight m The final IT loss becomes
K E(I) I(I,, R(O)
Cr
+ md de
(8)
A
2.2.3 THE MINT LOSS AUXILIARY LOSSES
The overlapping loss provides an auxiliary supervisory signal that spreads the keypoint over the im- age, encouraging them to cover distinctive regions. The sum of the Gaussians G( (ef. Appendix A.3) (Et K k²²)) 3, ()/)) K. The overlapping loss minimizes the maximum of the aggregation of the Gaussian normalized by the number keypoints K with a lower bound /3 to allow occlusion and avoid over-penalization.
around the keypoints x,(1) help estimate their overlap, L. min (max
The active status loss encourages the model to deactivate unnecessary keypoints, by minimizing the normalized sum of active keypoints while maximizing the ME. The interplay of the losses allows the method to eventually reach a trade-off between the number of active keypoints and the covered entropy. The active status loss optimizes L, = E st )/K.
The overall MINT loss LMINT is a weighted combination of all losses (with dedicated weight À loss), with the weight of the status loss reversed to schedule it according to the percentage of ME
LMITT AMECME + AMcE + Air + XoC, + (1 CME)
Further information about the hyperparameters available in Appendix E.2.
EXPERIMENTS
We evaluate MINT on four datasets ranging from videos of synthetic objects CLEVRER (Yi et al., 2019) and MAGICAL (Toyer et al., 2020) to realistic human video demonstrations - -IME (Sharma et al., 2018) and SIMITATE (Memmesheimer et al., 2019). Our experiments show the efficacy of our method as a representation for different tasks, and we provide quantitative results w.r.t. evaluation metries (for object detection and tracking on CLEVRER) for several downstream tasks (learning dynamics on CLEVRER, imitation learning on MAGICAL), and qualitative results on the challenging datasets of MIME and SIMITATE. We compare against baselines for unsupervised end-to-ent keypoint detection Transporter (Kulkarni et al., 2019), Transporter-modifee modified version with smaller receptive field, Video Structure (Minderer al., 2019), an ablations of MINT; MINT w/o Reg., i.e., MINT without the regularization terms, and an end-to-end CNN- based feature extraction. We report statisties for quantitative results over 5 seeds. extensive ablation study of MINT is provided in Appendix E. and baselines are discussed in Appendix E.3.
i.e.,
Downstream task I: Object detection and tracking. Capturing scene structure requires detecting all objects in an image, while object tracking is essential for representing the scene's dynamics. MINT can successfully train a tempo-temmporlly consistent keypoint representation on videos, leading to its natural application for object (static/dynamic appearing/disappearing detection and tracking.
We use CLEVRER (Yi et al., 2019), a dataset for visual reasoning with complete object annotations, containing videos with static and dynamic objects, with good variability in the scenes, testbed. To

---

Under review as a conference paper at ICLR 2023
Video Structure
MINT (ours)
Transporter
Figure 4: Qualitative results on CLEVRER dataset for Task (object detection and tracking) and Task (learning dynamics). Our method is able to assign keypoints to all objects, independent of whether they move or not, and follows their trajectory. The number of keypoints is dynamically adjusted to the number of objects. Future states are transparent, as well as the predicted keypoint and trajectories.
Table I: Quantitative evaluation of keypoint detection and tracking on CLEVRER eYi al., 2019).
Method
DOP +
TOP
UAK
RAK
MINT w/o Reg. (ours)
0.918 + 0.073
0.897 0.078
6.793 + 1.956
2.478 + 0.865
MINT (ours)
0.855 + 0.118
0,838 = 0.121
0.889 + 0.639
1.123 + 0.448
Transporter
0.787 +: 0.113
0,745 + 0.119
18.417 1.639
1.157 0.323
Transporter-modiffied
0.832 + 0.107
0.794 0.114
16.267 2.349
1.764 0.671
Video Sirattuee
0.567 + 0.256
0,543 + 0.253
18,104 3,538
1.922 + 0.652
Table Prediction success rate on CLEVRER (Yi et al., 2019).
Method
1-step prediction
2-steps prediction
3-steps prediction
MINT (ours)
0.844 + 0.116
0.827 +0.126
0.811=0.132
Transpurter
0.746 = 0.116
0.716 = 0.120
0.692 + 0,122
Transporter-moddfieed
0.814 = 0.099
0.791 = 0.106
0.769 + 0.110
Video Structure
0.734 + 0.124
0.719 = 0.125
0.699 + 0.127
quantitatively assess the performances of MINT, we developed evaluation metrics for CLEVRER. We propose the percentage of the detected object (DOP) and the percentage of tracked objects (TOP) as two metrics, with higher values corresponding to better keypoint detection and tracking. keypoint detects an object it lies on its mask, and tracks it, if detects the same object in two consecutive frame. Assigning keypoints to areas already represented by other keypoints or empty spaces signals bad keypoint detection. To evaluate these cases, we define two additional metries for the redundant keypoint assignment (RAK) and unsuccessful keypoint assignment (UAK), with lower values corresponding to better detection. The metrics are described in detail in Appendix D. We train all keypoint detectors 100. The train-test split emulates a low-data regime and tests the methods' generalization abilities. As seen in Table 1, MINT w/o Reg. detects more objects (DOP) and tracks them better (TOP), showing the benefit of our information-theorrticc losses. The proposed MINT model exhibits the best trade-off between superior performance against the baselines on all metrics, and better handling of keypoint assignment (UAK and RAK) than MINT w/o Reg. See more discussion about the ablations in Appendix 1. These results are supported by the visual comparisons in Fig. 4, and videos on the project website?.
subset of 20 videos from the CLEVRER dataset and test them on
Downstream task II: Learning dynamics. Proper object detection allows us to learn the underlying dynamics that evolve scene. We test the representation power of the discovered keypoints by training a prediction model using the pre-trained keypoint detectors from Task I (using the best seed for each method). The prediction model treats the keypoints graph nodes in an Interaction Network (IN) (Battaglia et al., 2016) to model the relational dynamics (cf. Appendix E.4). We train the prediction model to forecast the future positions of the keypoint given history window of four-time steps. We compare the prediction against the ground truth position of the object in the predicted frame using CLEVRER (Yi et al., 2019). We report in Table 2 the ratio of successfully predicted objects (i.e., predicted keypoint lying on the same object in the next frame) to the ground truth number of objects in the next time step. The comparison demonstrates that keypoints detected by our method represent the scene better than the baselines and help to predict the next state. Fig. 4 shows the prediction performance using different keypoint detectors.
Downstream task III: Object discovery in realistic scenes. Our method addresses challenging aspects beyond synthetic datasets. We evaluate the keypoint detectors on two additional datasets; (1) MIME (Sharma et al., 2018): a collection of close-up videos of human hands manipulating objects, and (2) SIMITATE (Memmesheimer et al., 2019): video dataset of humans performing manipulation
Videos https: / / /s//sittss.ooo com/ /vm http://iitiiioo ill or in the supplemental material.

---

Under review as a conference paper at ICLR 2023
25
Number of keypoints
20
15-
Transporter
Video structure
MINT
5
60
Timestep
Figure 5: Come-and-2 go scenario. In manipulation video from MIME, the hand enters after the start of the video and departs before the end. We plot the number of active keypoints w.r.t timesteps. Transporter has fixed number keypoints. Video structure increases the number active keypoints when the hand appears, struggles when disappears. MINT activates and deactivates the suitable number of keypoints.
tasks in wide-view cluttered scenes. For training, we use a subset of 80 videos from various tasks, and for testing 100, and since no annotations are provided, we rely qualitative analysis. In MIME, the human hand enters and leaves the scene abruptly, allowing the evaluation of MINT in come-and- go scenarios shown in Fig. 5. MINT activates only the required number of keypoints, while Transporter (Kulkarni et al., 2019) uses a static number of keypoints, and Video structure (Minderer al., 2019) fails to deactivate the excessive keypoints when the hand disappears. Fig. shows the number of active keypoints over time, revealing our method's 's superior performance, both in keypoints number economy and in the qualitative representation of the objects in the scene.
The qualitative results for SIMITATE, on the other hand, in Fig. 6 show that only MINT can disambiguate between static and dynamic objects, tracking the human movement, while maintaing the structure of the keypoints relatively constant over the static objects. The baselines rely on reconstructing the movement, failing represent the scene's 's structure. The qualitative results in SIMITATE reveal the need for the conditional entropy loss (forcing attention on moving objects when number of available keypoints restricted) and the information transportation loss (ensuring spatio-temporal consistency)- see also Fig. 10 in Appendix.
Downstream task IV: Imitation learning. Imitation learning from video frames a stag-standing challenge for control. Keypoints can define low-dimensional representation, that could reduce the computational burden considerably. In this experiment, we investigate representation for control tasks, like imitation learning in MAGICAL (Toyer et al., 2020). We first pretrain MINT on 24 demonstration vidcos from different tasks. Then, we fix the keypoint detector and train agent mimic the demonstrated actions, using IN (Battaglia et al., 2016), followed by a fully-connected layer that decodes the actions (cf. Appendix E.4). The agent uses as input the observed keypoints from four frames. We also found useful to predict the next state as an auxiliary task. We compare the MINT-based agent against an agent that uses CNN to extract features directly from pixels. The CNN agent is trained from scratch for each environment (cf. Appendix E.5). We consider three environments with different levels of difficulty; MoveToRegion: move agent to specific region, only the agent is involved (easy). MoveToCornerr move an object the top- corner, one object and the agent are involved (medium). Makel place multiple objects in line, four objects and the agent are involved (hardest). We evaluate the learned policy on environment instances from demonstrations (Demo) and randomly initialized (TestJitter). The results in Table 3 reveal that pretrained keypoint model with MINT suitable for control, achieving comparable or superior performance to task-specific ONN-based agent (cf. Appendix E.5 for more details).
suitability of our keypoint
Limitations. Our method relies on local image entropy after filtering high-frequency color changes. As result, the method has difficulties in recognizing transparent objects and objects with the same color as the background. We plan to investigate the integration of implicit representation learning to counterpart this issue. Another limitation the interpretation of the keypoints in the three-dimensional
8

---

Under review as a conference paper at ICLR 2023
Frame 0
Frame 10
Frame 30
Frame 40
MINT (ours)
Video Structure Transporter
Figure 6: Crowded scenes. video from SIMITATE dataset with human moving in All the methods can track the human successfully, but only MINT (ours) can keep the keypoints on the static ohjects consistently (green ellipses), while the baselines lose track of them (red ellipses).
Table 3: Average score for imitation learning on MAGICAL (Toyer et al., 2020). Higher values are better.
Method
MoReTonn
MoveTaCorner
MakelLine
Demo
TestJitter
Demo
Testlitter
Demo
Testlitter
MINT (ours)
1.00
0.00
0.86 = 0.31
1.00
0.00
0.80
0.34
0.2 0.22
0.06 0.14
CNN
1.00
0.00
0.84 = 0.32
0.74 = 0.35
0.30 = 0.38
0.00 = 0.00
0.01 = 0.06
space. The current method operates images and does not provide 3D information. Adding depth information or extending to multi-view setting are options for future improvements.
4 RELATED WORK
Our work draws inspiration from classical approaches for saliency detection in images, using local information detect salient objects (Kadir & Brady, 2001; Bruce & Tsotsos, 2005; Fritz et al., 2004; Renninger et al., 2004; Borji & Itti, 2012). The idea of extracting sparse feature representations of high-dimensional visual data is dominant in computer vision (Harris al., 1988; Lowe, 2004;
DeTone et al., 2018). Traditional geometric computer vision methods relied on the extraction of hand-crafted feature descriptors (Lowe, 2004; Rublee et al., 2011; Schmid et al., 2000; Mur-Artal et al., 2015). Recently, CNN architectures have proven superior (Yi et al., 2016; De'Tone et al., 2018; Song et al., 2020; Zheng et al., 2017) performance. Keypoints represent class of Pol that have semantic entity (Duan et al., 2019; Cao et al., 2017; McNally et al., 2022), but most methods rely on explicit annotations of kcypoint locations for training. Related to our work are methods that rely on image reconstruction for detecting keypoints in video streams in unsupervised fashion (Jakab et al., 2018; Zhang et al., 201 2018b; Minderer et al., 2019; Kulkarni et al., 2019; Gopalakrishnan et al., 2020). An extended related work discussion is provided in Appendix G.
CONCLUSION
We presented MINT, an anforeition-theortticc approach for unsupervised keypoint discovery from videos. Treating keypoints as transporters of information, we defined two losses inspired by in- formation theory; information maximization loss and information transportation loss. The losses promote covering areas with high information while ensuring spatio-temporal consistency. Additionally, the method learns to activate keypoints upon necessity and reduces redundant keypoints thanks to auxiliary losses. The experimental evaluation demonstrated the superior performance of our method for various downstream tasks, ranging from object detection to dynamics prediction and imitation learning. Moreover, we showed qualitatively that our method can tackle many challenges for keypoint detection in realistic scenarios, such as attending to static and dynamic objects and handling come-and-pe situations in realistic videos. Overall, we demonstrated how to learn reasonable keypoint representations purely unsupervised from videos, with promising results for future applications.

---

Under review as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
We aim to guarantee the reproducibility of our work by providing additional details of the method and the conducted experiments in the appendix. Moreover, we provide the code in the supplementary material, and we will make it public upon acceptance.
REFERENCES
Bogdan Alexe, Thomas Deselaers, and Vittorio Ferrari. What an object? In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 73-80, 2010.
Mor Avi-Aharon, Assaf Arbelle, and Tammy Riklin Raviv. DeepHist: Differentiable joint and color histogram layers for image-to-image translation. arXiv preprint arXiv:2005.03999, 2020.
Axel Barroso-Laguna, Edgar Riba, Daniel Ponsa, and Krystian Mikolajezyk. Key.Net: Keypoint detection by handcrafted and learned cnn filters. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5836-5844, 2019.
Peter Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, et al. Interaction networks for learning about objects, relations and physics. Advances in Neural Information Processing Systems, 29, 2016.
Ali Borji and Laurent Itti. State-of-the-art in visual attention modeling. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(1):185-207 2012.
Ali Borji, Dicky N Sihite, and Laurent Itti. Quantitative analysis human-model agreement in visual saliency modeling: A comparative study. IEEE Transactions on Image Processing, 22(1):55-60 69, 2012.
Neil Bruce and John Tsotsos. Saliency based on information maximization. Advances in Neural Information Processing Systems, 18, 2005.
Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. Realtime multi-person person 2d pose estimation using part affinity fields. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7291-7299, 2017.
Po-Yi Chen, Alexander H Liu, Yen-Cheng Liu, and Yu-Chiang Frank Wang. Towards scene un- derstanding: Unsupervised monocular depth estimation with semantic-awar representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 2624-2632, 2019.
Lynn A Cooper. Mental representation of diree-dimensional objects in visual problem solving and recognition. Journal of Experimental Psychology: Learning, Memory, and Cognition, 16(6):1097, 1990.
Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. SuperPoint: Self-supervised interest point detection and description. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition workshops, pp. 224-236, 2018.
Andrea Dittadi, Samuele Papa, Michele De Vita, Bernhard Schölkopf, Ole Winther, and Francesco Locatello. Generalization and robustness implications in object-centric learning, arXiv preprint arXiv:2107.00637, 2021.
Kaiwen Duan, Song Bai, Lingxi Xie, Honggang Qi, Qingming Huang, and Qi Tian. CenterNet: Keypoint triplets for object detection. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 6569-6578 2019.
Mihai Dusmanu, Ignacio Rocco, Tomas Pajdla, Mare Pollefeys, Josef Sivic, Akihiko Torii, and Torsten Sattler. D2-Net: A trainable enn for joint description and detection of local features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8092-8101, 2019.
10

---

Under review as a conference paper at ICLR 2023
Marco Ewerton, Angel Martinez-Gonzzzee and Jean-Mare Odobez. An efficient image-to-image translation hourglass gourglas--based architecture for object pushing policy learning. JEEE/RSJ Interna- tional Conference on Intelligent Robots and Systems, pp. 3478 -484. IEEE, 2021.
Mario Ferraro, Giuseppe Boccignone, and Terry Caelli. Entropy-basee based representation of image information. Pultern Recognition Letters, 23(12):1391-) 1391-1398, 2002.
Peter Florence, Lucas Manuelli, and Russ Tedrake. Self-supervised correspondence in visuomotor policy learning. IEEE Robotics and Automation Letters, 5(2):492-499, 2019.
Peter R Florence, Lucas Manuelli, and Russ Tedrake. Dense object nets: Learning dense visual object descriptors by and for robotic manipulation. In Conference on Robot Learning, pp. 373-385. PMLR, 2018.
Gerald Fritz, Christin Seifert, Lucas Paletta, and Horst Bischof. Attentive object detection using an information theoretic saliency measure. In International Workshop on Attention and Performance in Computational Vision, pp. 29-41. Springer, 2004.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249-256 JMLR Workshop and Conference Proceedings, 2010.
Irving John Good. Rational decisions. In Breakthroughs in statistics, pp- 365-377. Springer, 1992.
lan Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
Anand Gopalakrishnan, Sjoerd van Steenkiste, and Jürgen Schmidhuber: Unsupervised object keypoint learning using local spatial predictability. In International Conference Learning Representations, 2020.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference machine learning, pp. 1861-1870. PMLR, 2018.
Shugo Hamahashi, Shuichi Onami, and Hiroaki Kitano. Detection of nuclei 4d nomarski dic microscope images of early caenorhabditis elegans embryos using local image entropy and object tracking. BMC bioinformatics, 6(1):1-15 1-15, 2005.
Chris Harris, Mike Stephens, et al. A combined corner and edge detector. In Proceedings of the Alvey Vision Conference. volume 15, pp. 10-5244. Alvety Vision Club, 1988.
Stephen Hausler, Sourav Garg, Ming Xu, Michael Milford, and Tobias Fischer. Patch-NetVLAD: Multi-scale fusion of locally-global descriptors for place recognition. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14141-14152, 2021.
R Devon Hjelm, Alex Fedorov, Samuel Marriie-Marchldonn, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In International Conference on Learning Representations, 2018.
Franko Hržić, Ivan Štajduhar, Sebastian Tschauner, Erich Sorantin, and Jonatan Lerga. Local-entropy based approach for x-ray image segmentation and fracture detection. Entropy, 21(4):338, 2019.
Wei-Chih Hung, Varun Jampani, Sifei Liu, Pavlo Molchanov, Ming-Hsuar Yang. and Jan Kautz. Scops: Self-supervised co-part segmentation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pottern Recognition, pp. 869-878, 2019.
Tomas Jakab, Ankush Gupta, Hakan Bilen, and Andrea Vedaldi. Unsupervised learning of object landmarks through conditional image generation. Advances in Neural Information Processing Systems, 31, 2018
Yu-Gang Jiang, Jun Yang, Chong Wah Ngo, and Alexander G Hauptmann. Representations of keypoint-basee semantic concept detection: A comprehensive study. IEEE Transactions on Multimedia, 12(1):42-53, 2009.
Cho
11

---

Under review as a conference paper at ICLR 2023
Timor Kadir and Michael Brady. Saliency, scale and image description. International Journal of Computer Vision, 45(2):83-105, 2001.
Yunji Kim, Seonghyeon Nam, In Cho, and Seon Joo Kim. Unsupervised keypoint learning for guiding class-conditional video prediction. Advances Neural Information Processing Systems, 32, 2019.
Lingpeng Kong, Cyprien de Masson d' A'ttum, Lei Yu, Wang Ling, Zihang Dai, and Dani Yo- gatama. A mutual information maximization perspective of language representation learning, In International Conference on Learning Representations, 2019.
Sven Kreiss, Lorenzo Bertoni, and Alexandre Alahi. PifPaf: Composite fields for human pose estima- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Paltern Recognition. pp. 11977-11986, 2019.
Tejas D Kulkarni, Ankush Gupta, Catalin Ionescu, Sebastian Borgeaud, Malcolm Reynolds, Andrew Zisserman, and Volodymyr Mnih. Unsupervised learning of object keypoints for perception and control. Advances in Neural Information Processing Systems, 32, 2019.
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 17(1):1334-1377 2016.
Stan Li. Markov random field modeling in image analysis. Springer Science & Business Media, 2009.
Yunzhu Li, Antonio Torralba, Anima Anandkumar, Dieter Fox, and Animesh Garg. Causal discovery in physical systems from videos. Advances in Neural Information Processing Systems, 33:9180- 9192, 2020.
Ralph Linsker. Self-organization in perceptual network. Computer, 21(3):105-117, 1988.
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. Advances in Neural Information Processing Systems, 33: 1555-11155,, 2020.
David G Lowe. Distinctive image features from ceale-invariant keypoints. International Journal of Computer Vision, 60(2)911-10, 2004.
David Marr. Vision: A computational investigation into the human representation and processing of visual information. MIT press, 2010.
William McNally, Kanav Vats, Alexander Wong, and John McPhee. Rethinking keypoint represen- tations: Modeling keypoints and poses as objects for multi-person human pose estimation. In European Conference on Computer Vision, pp. 37-54. Springer, 2022.
Raphael Memmesheimer, Ivanna Kramer, Viktor Seib, and Dietrich Paulus. Simitate: A hybrid imitation learning benchmark. In IEEE/RSJ International Conference on Intelligent Robots and Systems, pp- 5243-52499 IEBE, 2019.
Shervin Minaee, Yuri Boykov, Fatih Porikli, Antonio Plaza, Nasser Kehtarnavaz, and Demetri Terzopoulos. Image segmentation using deep learning: A survey, IEEE Transactions Pattern Analysis and Machine Intelligence, 2021.
Matthias Minderer, Chen Sun, Ruben Villegas, Forrester Cole, Kevin P Murphy, and Honglak Lee. Unsupervised learning object structure and dynamics from videos. Advances in Neural Information Processing Systems, 32, 2019.
David Minnen, George Toderici, Saurabh Singh, Sung Jin Hwang, and Michele Covell. Image- dependent local entropy models for learned image compression. In 2018 25th IEEE International Conference on Image Processing (ICIP), pp. 430-434. IEEE, 2018.
Raul Mur-Artal, Jose Maria Martinez Montiel, and Juan D Tardos. ORB-SLAM: a versatile and accurate monocular slam system. IEEE Transactions Robotics, 31(5): 116-1163,, 2015.
12

---

Under review as a conference paper at ICLR 2023
Kevin P Murphy. Probabilistic machine learning: introduction. MIT press, 2022.
Alejandro Newell, Kaiyu Yang, and Jia Deng. Stacked hourglass networks for human pose estimation. In European Conference Computer Vision, pp. 483-499. Springer, 2016.
Yuki Ono, Eduard Trulls, Pascal Fua, and Kwang Moo Yi. LF-Net: Learning local features from images. Advances in Neural Information Processing Systems, 31, 2018.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv; 180700378, 2018.
Leandro Pardo. Statistical inference based divergence measures. Chapman and Hall/CRC, 2018.
Emanuel Parzen. On estimation of a probability density function and mode. The annals of mathemat- ical statistics, 33(3):1065-1076, 1962.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. PyTorch: An imperative style, high-pe sech-performance deep learning library. Advances in Neural Information Processing Systems, 32. 2019.
Jan Peters, Katharina Mulling, and Yasemin Altun. Relative entropy policy search. In Twenty-Fourth AAAI Conference Artificial Intelligence, 2010.
Angela Radulescu, Yeon Soon Shin, and Yael Niv. Human representation learning. Annual Review of Neuroscience, (4(1):253-21)) 2021.
Kate Rakelly, Abhishek Gupta, Carlos Florensa, and Sergey Levine. Which mutual-information representation learning objectives are sufficient for control? Advances in Neural Information Processing Systems, 34:26345-26336 2021.
Laura Renninger, James Coughlan, Preeti Verghese, and Jitendra Malik. An information maximization model of eye movements. Advances Neural Information Processing Systems, 17, 2004.
Nicolas Riche, Matei Mancas, Dubravko Culibrk, Vladimir Crnojevic, Bernard Gosselin, and Thierry Dutoit. Dynamic saliency models and human attention: A comparative study videos. In Asian Conference on Computer Vision, pp. 586-598. Springer, 2012.
Edward Rosten and Tom Drummond. Machine learning for high-speed corner detection. In European Conference on Computer Vision, pp. 430-443. Springer, 2006.
Ethan Rublee, Vincent Rabaud, Kurt Konolige, and Gary Bradski. ORB: An efficient alternative to sift or surf. In Proceedings of the IEEE International Conference on Computer Vision, pp. IEEE, 2011.
Mert Rory Sabuncu. Entropy-bass based image registration. Princeton University, 2006.
Paul-Edouard Sarlin, Cesar Cadena, Roland Siegwart, and Marcin Dymezyk. From coarse to fine: Robust hierarchical localization large scale. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 22716-12725 2019.
Paul-Edouard Sarlin. Daniel DeTone. Tomasz Malisiewiez, and Andrew Rabinovich. SuperGlue: Learning feature matching with graph neural networks. In Proceedings of the IEEF/CV Confer- ence on Computer Vision and Pattern Recognition, pp. 1998-49473 2020.
Jonathan Scarlett and Volkan Cevher. An introductory guide to fano's inequality with applications in statistical estimation. Technical report, Cambridge University Press, 2019.
Cordelia Schmid, Roger Mohr, and Christian Bauckhage. Evaluation interest point detectors. International Journal of Computer Vision, 37(2)151-172.. 2000.
Claude Elwood Shannon. A mathematical theory of communication. ACM SIGMOBILE Mobile Computing and Communications Review, 5(1):3-55 2001.
13

---

Under review as a conference paper at ICLR 2023
Pratyusha Sharma, Lekha Mohan, Lerrel Pinto, and Abhinav Gupta. Multiple interactions made easy (mime): Large scale demonstrations data for imitation. Conference on Robot Learning, pP- 906-915. PMLR, 2018.
Gautam Singh, Fei Deng, and Sungjin Ahn. Illiterate dall-e learns to compose. In International Conference Learning Representations, 2021.
Yafei Song, Ling Cai, Jia Li, Yonghong Tian, and Mingyang Li. SEKD: Self-evolving keypoint detection and description. arXiv preprini arXiv:2006.05077, 2020.
Richard Szeliski. Computer vision: algorithms and applications. Springer Science & Business Media, 2010.
Rashish Tandon, Karthikeyan Shanmugam, Pradeep Ravikumar, and Alexandros G Dimakis.
On the information theoretic limits of learning ising models. Advances in Neural Information Processing Systems, 27, 2014.
Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. arXiv preprint physicx/0004057, 2000.
Sam Toyer, Rohin Shah, Andrew Critch, and Stuart Russell. The MAGICAL benchmark for robust imitation. Advances in Neural Information Processing Systems, 33: 18:18884-1822 18295, 2020.
Tianlu Wang, Jieyu Zhao, Mark Yatskar, Kai-Wei Chang, and Vicente Ordonez. Balanced datasets not enough: Estimating and mitigating gender bias deep image representations. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5310-5319, 2019.
Zeyu Wang, Klint Qinami, Ioannis Christos Karakozis, Kyle Genova, Prem Nair, Kenji Hata, and Olga Russakovsky. Towards fairness in visual recognition: Effective strategies for bias mitigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
8919 8928, 2020.
Yue Wu, Yicong Zhou, George Saveriades, Sos Agaian, Joseph P Noonan, and Premkumar Natarajan. Local shannon entropy measure with statistical tests for image randomness. Information Sciences, 222:323-344 342, 2013.
Haoyu Xiong, Quanzhou Li, Yun-Chun Chen, Homanga Bharadhwaj. Samarth Sinha, and Animesh Garg. Learning by watching: Physical imitation of manipulation skills from human videos. In IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 7827-7834. IEEE, 2021.
Wu Xiongwei, Doyen Sahoo, and HOI Steven. PolarNet: Learning to optimize polar keypoints for keypoint based object detection. In International Conference Learning Representations, 2020.
Tianhan Xu and Wataru Takano. Graph stacked hourglass networks for 3d human pose estimation. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp.
16105-16114, 2021.
Ge Yang, Amy Zhang, Ari Morcos, Joelle Pineau, Pieter Abbeel, and Roberto Calandra. Plan2 Unsupervised representation learning by latent plans. Learning for Dynamics and Control, pp. 935-946. PMLR, 2020.
Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua Tenenbaum. Clevrer: Collision events for video representation and reasoning. In International Conference on Learning Representations, 2019.
Kwang Moo Yi, Eduard Trulls, Vincent Lepetit, and Pascal Fua. LIFT: Learned invariant feature transform. European Conference on Computer Vision, pp. 467-483. Springer, 2016.
Lantao Yu, Yang Song, Jiaming Song, and Stefano Ermon. Training deep eneryy-based models with f-divergence minimization. In International Conference on Machine Learning, pp. 10957-10967. PMLR, R. 2020.
14

---

Under review as a conference paper at ICLR 2023
Shujian Yu, Luis G Sánchez Giraldo, and José C Principe. cotommttion-thooetiic methods in deep neural networks: Recent advances and emerging opportunities. In IJCAI, pp. 6669-4675, 2021.
Quanshi Zhang, Ying Nian Wu, and Song-Chun Zhu. Interpretable convolutional neural networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 8827-8836, 20
Yuting Zhang, Yijie Guo, Yixin Jin, Yijun Luo, Zhiyuan He, and Honglak Lee. Unsupervised discovery of object landmarks as structural representations. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2694-2703, 2018b.
Liang Zheng, Yi Yang, and Qi Tian. SIFT meets CNN: A decade survey of instance retrieval. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(5):1224-1244, 2017.
Wangjiang Zhu, Shuang Liang, Yichen Wei, and Jian Sun. Saliency optimization from robust background detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 28 2814-2821, 2014.
15

---

Under review as a conference paper at ICLR 2023
APPENDIX
The appendix provides additional details on the architecture, the entropy layer, proofs, the description of evaluation metrics, experimental details with an ablation study, additional results and discussions, as well as comments on the used code. Moreover, we provide video results for better visualization and the code the supplemental material.
Architecture
17
A.I Keypoint model
17
A.2 Feature map to keypoint
17
A.3 Keypoints to heatmaps
18
Entropy layer
18
B.1 Entropy module
18
B.2 CUDA extension
19
Proofs
20
Proof of Theorem
20
C.2 Proof of Lemma
C,3 Proof of Theorem 2
Evaluation metrics for object detection and tracking
22
D.I Percentage of the detected object (DOP).
22
D.2 Percentage of tracked objects (TOP).
22
D.3 Unsuccessful keypoint assignment (UAK).
23
D.4 Redundant keypoint assignment (RAK).
23
Additional experimental analysis
23
E.1
Ablation study
23
E.2
Hyperparameters
25
E.3
Baselines
26
E.4
Interaction network architecture
28
E.5
Imitation learning results
28
E.6
Additional video results
29
F
Code
30
G Extended Related Work discussion
30
16

---

Under review as a conference paper at ICLR 2023
A ARCHITECTURE
This section contains additional information about the model architecture in Sec. 2.2, implementation details on how to get the keypoint coordinates from the feature map, and the heatmaps for keypoints to ensure reproducibility.
A.1 KEYPOINT MODEL
Keypoints provide a cow-dimeniional representation of figh-dimensional RGB images. Therefore, many keypoint detection techniques are inspired by the autoencoder architecture (Goodfellow et al., 2016) which uses the bottleneck consolidate the information into reduced dimensionality for keypoint extraction (Minderer et al., 2019; Kulkarni et al., 2019). Instead, we suggest taking an hourglass architecture (Newell al., 2016) which upscales the compressed information again and outputs several feature maps with high activation in places with eminent information (Ewerton et al., 2021; Xu & Takano, 2021: Newell et al., 2016). This allows the network to predict the information at the original image size yielding finer resolution and correspondence to the original pixels.
Our keypoint detector consists of an hourglass convolutional neural network with three convolutional layers, with kernel sizes of 5,3, 3, 3 and strides of 3, 2, respectively. The upsampling part of the model consists of three transposed convolutional layers, with kernel sizes of 3, 2,33, and strides of 1, 2, 2. The number of input and output channels for each layer depends on the number of keypoints K and the number of input image channels C, see Table 4. The result is passed through softplus layer ensure the positivity of the feature maps. Lastly, append spatial softargmax layer (see Appendix A.2) to get the coordinates of the keypoints from the feature maps ft. We initialize all the convolutional layers with Xavier's normal initialization (Glorot & Bengio, 2010) and add leaky ReLU activation and a batch normalization layer after each of them. The total number of the parameters is 58, 725 for the input of size x20x420. We normalize the input the range [-0.5, 0.5].
Table 4: Architecture details for RGB-image of 320 480 and layer and BatchNorm2d layer (50 parameters) after each convolutional layer.
keypoints, there is Icaky ReLU
layer (type)
Input channels
Output channels
kernel size
stride
Output shape
num params
Normalize-1
C
C
[3, 320, 480)
0
Conv2d-1
C
K
5
3
[25, 79, 106]
1,900
Conv2d-2
K
K
[50, 26, 35]
5,650
Conv2d-3
K
2K
11,300
Conv Transpose2d-1
2K
2K
3
53, 71]
22,550
TrawTraasposc2d-22
2K
K
3
2
[25, 107. 143]
11,275
Conv Transpose
K
K
3
2
[25, 107, 143)
5,650
Softplus- 14
K
K
0
SpatialSSttarmmaxLLyee//
K
K
[25. 2]
Final output
[25.3]
total 58,725
A.2 FEATURE MAP TO KEYPOINT
The argmax operator is not differentiable, an alternative to extract the keypoints coordinates from the feature maps. The spatial soft argmax (Levine et al., 2016) takes 2D feature maps (6), flattens the feature maps and compute for each of them the weights Wi
opted to use differentiable spatial soft argmax as
e f00 m
Wi Σ edl (0-mmaa min(
(10)
Before applying the softmax, we subtract the maximum value from the input, which does not change the output of the softmax but helps for numerical stability. In order to map the weights to coordinates, generate mesh grid (o (Zgrid Vgrid) of and y coordinates, with the same size as the input image.
We flatten the mesh grid and compute the expected coordinates [2, ds] as the weighted sum of the coordinate grid with wi This process is visualized in Fig. 7.
17

---

Under review as a conference paper at ICLR 2023
Input feature map f(t)
Flattened f in blue 10*weights in red
Expected coordinates
00
40
00
20-
10-
100
170
0
2000
4020
6000
8000
10000
12000
14000
16200
120
125
25
100
Figure 7: Feature map to keypoint (spatial-sof argmax)
Distances
Gaussian
Heatmap
(x,, y) (I (x+,,:)
(+ (zi, yi)
h (zi, yi)
Process the Gaussian
G t G( -
G < 1, ReLU ((0))
(6) -mx (G
Figure 8: Keypoints to heatmap (Gaussian around keypoints)
A.3 KEYPOINTS TO HEATMAPS
The heatmap (t) generation for keypoint takes coordinates as pair of real numbers (I,, Vi) R². We start by generating a pixel-coordinates array with the same width and height as the original image H W x 2, where 2 denotes the coordinates of each pixel (æ, y) E NH* the squared distance between the input and all the pixels [|(x, (æ;, 1 2. We use the squared distance to generate a Gaussian distribution around the input coordinates (( (I;, yi) with a standard deviation ggt))
Then
compute
The heatmap defines the area and weighting of information belonging to each keypoint. The heatmap should be around the center of the keypoint, as the keypoint covers the information in this point completely, and descend gradually to 0 representing information out of reach of the keypoint. We achieve that by thresholding and clamping the Gaussian. We use a threshold and scaling factor for the thresholded Gaussian to get the final heatmap hç. Table 6 for more details on the scale of these hyperparameters. The process visualized Fig.
B ENTROPY LAYER
The entropy layer (Sec. 2.1) the one of the main modules our method. For an input image, the entropy layer outputs the entropy image. In this section, we provide additional implementation details. We split the explanation into two subsections: (/) the entropy module definition in PyTorch (Paszke al., 2019), and, (2) the CUDA extension for the parallel execution.
B. ENTROPY MODULE
The input of the entropy module is an RGB image E Hx x3, The input image is processed to remove high-frequeccy color changes. The processing is completely vectorized to allow efficient execution using PyTorch during training. The processing of the input image starts by blurring the image using an average blur layer, followed by sharpening the result, and finally dividing the sharp image by the smooth image. Fig. reflects example of the processing inside the entropy layer.
18

---

Under review as a conference paper at ICLR 2023
Input image
blurred image (smooth)
Sharpened image (sharp)
rre-processed image (division)
Output entropy image
Region histogram h(,, R( y))
R(,,)))
E(I(,,)) y))
R(x, ((x,27 y)
Figure Entropy layer with the prepossessing
Before forwarding the processed image into the entropy function, we generate the neighborhood region R(x, y) for each pixel location (z, y). These regions have square shape with the corresponding pixel (Ir, y) being the center. Instead of iterating over all the pixels with for loops, we use the stridden operation to factorize the extraction of the regions. Using strides to extract the regions aligns with how the image is stored in the memory and does not create overhead. These tricks are essential for the efficiency of our entropy layer.
B.2 CUDA EXTENSION
We need the histogram of the color values to compute the entropy. However, the naive implementation via vectorizing the code requires computing a pairwise distance-matrix between each pixel with every histogram bin, which means multiplying the number of the possible regions by 256. This causes exploding GPU memory requirements (more than 50GB). Motivated by this observation, we present in this work an efficient entropy layer based on kernel density estimation.
To estimate the value for each histogram bin inside a region R(z, y) centered on pixel at location (3r, y), we the kernel density estimator
II Un,) KC(
b bb)
f(b, y)
Σ
(11)
B
(it ((R(())))))
where (In, yn) is the pixel value at location (In, Vo) in the region R(x, 3) and B is the bandwidth. used as smoothing parameter.
We follow (Avi-Aharon al., 2020) and use the derivative of the logistic regression function, the Sigmoid function ((), as kernel KC..), that is for variable v
K(v)= du ( ((v)a(-u).
(12)
The integral of the function ab,, y) defined in Eq. (11) over the region gives the histogram value of the bin color channel c:
a Io(Ens Un) - L/2, - o IolWns Un) + L/2))
hist, (b, R(x, y))
Σ
B
(13)
where L 1/256 is the bin size, so that each bin represents color value. We get the probability of each color value by dividing the sum of the histogram values by the size of the region | and the number of channels C
1
p(T(z, v)) = p(b, R(x, v))
hist. (b. R(x,y))
(14)
C
(T.g.b)
19

---

Under review as a conference paper at ICLR 2023
The entropy of the pixel in the center of the patch is
E(I(x,y)) =
Σ
p(I( vn)) log(p(I( Bol))
(15)
(am, Je R(2.9)
Σ
R(x, R(l,,)))oogpp( R(x,
(16)
25(0
The entropy module uses our entropy function, implemented as an autograd function in PyTorch, to realize the CUDA extension of the entropy computation. The input the entropy function is the regions of the processed images R(x, y). The CUDA extension allocates a GPU block for each region, hence, the grid size equals to the number of all possible regions for all images in the batch. The block size is 256 threads, i..., a thread for each bin b. Each thread iterates over the whole region and computes the histogram of its corresponding bin value according to Eq. (13). Then, normalizes the result by the region size and the number of channels to get the probability Eq. (14), And finally, each thread computes the entropy of the pixel Eq. (15), which is equivalent to the sum of the entropy of the histogram bins Eq. (16).
C PROOFS
C.I PROOF OF THEOREM 1
We want to minimize the average probability error of how well the masked image I,M represents the information of the actual image 4. First, will work pixel level bound the probability of сиroг in the intensity of the th pixel in location x, y in images I,M and 1,, denoted as p!!)) = F(4(I,)) y) * I, (z, (x, v)). Images can in general be considered lattices, with pixels being the random variables over intensities B (in our case these are the number of bins in the histogram as described in Appendix B).
Since the error event for the nh pixel is a binary event, it follows that p(n)) a binary probability. Therefore, the average error probability over all pixels N of the image can be computed
P.
1 N
Σ P(n)
(17)
where N = H x W is the total number of pixels, computed as the product of the height H and width W of the image.
On pixel-level, following Fano's inequality (Sabuncu, 2006; Scarlett & Cevher, 2019; Tandon et al., 2014) and assuming that the th pixel in position x, y an image can take value uniformly on B, we get
((T (x, v)) < ( p!n) log(1B) - 1),
(18)
where E2(a) = a log + (1-a) a) log T-a is the binary entropy function. Eq. (18) can be further bounded as (p(")) < 2, and |B 1 < [B] Morcover, since a pixel is uniform on [B] its
entropy can be considered F(T, ()) = log | Therefore, we can further bound Eq. (18) as
E(I(2,,)) (x, (1 log
p!() log(1B))
(19)
E(([(z 444 v)) E(1 v) < log 2 p!n) log((B)) - E(L v)
(20)
+I(I( y), TM v)) 2 (1 - P! log( - log
(21)
p(()) 21 [(4((z v), 1IM (x, w)) + log
(22)
log [
The mutual information of two random variable is upper bounded by the minimum entropy of the marginals, therefore. ((1 1 v)) < min(E(1 y)), E (z, v))). But the masked image will by definition represent less information than the original image, therefore,
"The entropy layer is part of the submitted code, that will he open-sourced upon acceptance.
20

---

Under review as a conference paper at ICLR 2023
we have (z, 1 (z, 2)) < E(I, (z, y)). We can take the worse case scenario and assume I14(2, (z, = E (I.y)) (Murphy, 2022). Therefore, Eq. (22) becomes
p!n) 21 E (x. v)) log log [B]
(23)
which bounds the enror the information carried by pixel in the masked image.
To acquire the bound of the average error probability Eq. (17), we sum Eq. (23) for all pixels and divide by N to get
N
1 z.y ELI (x, 8)) + Σn N
log2
P(a)
(24)
N
N
N log B
ΣN EII (In, Bn))
log 2
Pe 2
(25)
N log [
log [
C.2 PROOF OF LEMMA
Let X, Y be two discrete random variables, and the respective entropy is lower-bounded B(X) 2 0,
E(Y) 2 The joint entropy between the random variables can be expressed
E(X,y, Y) = E(X) + E(Y = E(Y) + E(X
(26)
As any conditional entropy is greater or equal to zero, we get E(X,Y) 2 E(X), and similarly E(X,Y) 2 EYY). If E(X) 2 E then
E(X,Y) E(X) 2 B(Y).
(27)
If EYY 2 E(X) then
E(X,Y) E(Y) E(X)
(28)
Therefore, for discrete random variables, we get the bound of the joint entropy
E(X,Y) 2 max(F(X); FI()))
(29)
Following the previous derivation, when considering two images I1 In whose pixels are discrete random variables over intensities B, and assuming that they are independent such that (2 = E(I)), we can consider that the joint image entropy is approximated by the pixel-wise maximum of the two marginal entropies
(21,,12)(1,)) = (Ex)
(30)
Approximating the joint entropy by the lower bound gives a lower bound of the conditional entropy as well. Given that we maximize the masked conditional entropy in the CME loss, then maximizing the lower bound is acceptable approximation. Meanwhile in the IT loss, using the lower bound of the joint entropy gives an upper bound for the mutual information, equivalent to its general upper bound of the minimum entropy of the marginals. Accordingly, maximizing the upper bound gives higher probability of good reconstruction, and the approximation of the joint entropy acceptable.
C.3 PROOF OF THEOREM 2
Due the process of information transportation of the keypoints, we try to reconstruct the information each keypoint carries. Therefore, we can again leverage Fano's 's inequality (Scarlett & Cevher, 2019). to provide a lower bound for the average error probability of information transportation per keypoint.
We formalize our error probability of information transportation of the th keypoint as the per-pixel error event (T))) (((( 9) # R (z, v)). Therefore, from Fano's incquality similarly Eq. (18)
we have for the per
((T((.))) (s. (), & (2(!((()) + (IT()) log([B] -1).
(31)
21

---

Under review as a conference paper at ICLR 2023
And following similar manipulations in Appendix C.1, end up in the equivalent version of Eq. (22)
I(I(2, u). 11 (2, w)) 2 (1 per(()) (B) log 2 +>
(32)
p! 2 [[),(, y), R1, (x; v)) - log
(33)
log [B]
Similarly to Appendix C.I. we obtain the average error probability of information transportation of the hh keypoint by summing the per-pixel probabilities of Eq. (33) and averaging over number of pixels N
Ele,s) [[le y), R1 (x,
log 2
pIT 2 1
(34)
N log [
log [
Finally, considering all keypoint transportations as independent events, allows us to compute the joint average error probability for all keypoints as per (joint) K (((())
leading to the bound
Σ Ezv 2. [[I( v). (x, (x,
K log 2 log B
piT (joint)
K
(35)
N log [B]
D EVALUATION METRICS FOR OBJECT DETECTION AND TRACKING
Each keypoint should provide representation of feature in object, and keypoints should be distinctive and distributed over the shallow scene. Keypoints assigned to empty spaces are considered unsuccessfully assigned. To judge the performance of method, developed metrics that the object masks provided by CLEVRER (Yi et al., 2019) over set of test videos V, each of which of length T.
D.1 PERCENTAGE OF THE DETECTED OBJECT (DOP).
We consider an object detected if there is at least one keypoint on its mask Morj. At each time frame, we count the percentage detected objects with respect the ground truth (GT) number of objects and average these values over the whole video. We get the final result by averaging the value over all the videos the test dataset. Better detection corresponds higher percentage of detected objects.
Nateettdd
Mpor V ==1t=1
(36)
NGr
where dettected is the number of detected objects (at least one keypoint lies in the object mask)
K
Naetocted
Σ
Σ
[((., vi) E Mons)
> 0.
(37)
objEO
and NoT is the ground truth number of objects and O is the set of all objects in the scene.
D.2 PERCENTAGE OF TRACKED OBJECTS (TOP).
We consider an object tracked if there is at least one keypoint on its mask in the current and the previous timeframe. At each time frame, we count the percentage of tracked objects with respect to the ground truth (GT) number of objects and average these values over the whole video. We get the final result by averaging the value over all the videos the test dataset. Better detection corresponds to higher percentage of tracked objects.
V
T
Ntracked NcT
MTOP V TT
(38)
22

---

Under review as a conference paper at ICLR 2023
Where Nrracked is the number of tracked objects (at least one keypoint lies in the object mask in time frames and t-1)
Ntracked
Σ
Σ
[((2,, vi), M [I((,;,, Vi). M 00.
(39)
objEO
D.3 UNSUCCESSFUL KEYPOINT ASSIGNMENT (UAK).
keypoint is unsuccessfully assigned time frame it does not belong to any object. We average the number of unsuccessful keypoint over the whole video, and then over test videos to get global value over the testset
MUKA V
Nuk
(40)
where Nu is the number of unsuccessful keypoints (does not belong to the sum of the masks)
Nuk
I((x4,))) &
Σ Mobj)
(41)
objEO
lower unsuccessful keypoint assignment metric MUKA corresponds better keypoints activation.
D.4 REDUNDANT KEYPOINT ASSIGNMENT (RAK).
Assigning keypoints to areas already represented by other keypoints signals bad keypoint detection. The RAK metric accounts for the number of keypoints over the area of the object. The number of keypoints an object mask should be proportional to its area Aabj. We assume keypoint can represent some area of pixels Ag. If the keypoints cover the object, the RAK metric will have a value of O, with higher values if more or fewer keypoints were assigned to that object.
MRAK O ΣΣ
V
T
A
AgNotll
(42)
Anbj
objEO
where Ag is the representation area of a keypoint (e.g. average object areas in the dataset) Aobs is the area of the object's mask and naby is the number of keypoints assigned the object
Nobj = >> I((2, y:) E Morj)
for each obj E O.
(43)
The lower the value of MRAK, the better, because more efficient, the distribution of the keypoints.
The metrics collectively judge the efficacy of keypoint detection and tracking methods, where only detected objects can tracked, so DOP metric an upper bound for TOP metric. The value of the metric RAK will go to in the case of not detecting any object, but can go higher in case of assigning redundandant keypoints to the same object. Following the previous observation, we recommend judging the the value of the RAK metric jointly with the value of the DOP metric.
E ADDITIONAL EXPERIMENTAL ANALYSIS
E.
ABLATION STUDY
Our method for unsupervised kcypoint discovery in video streams uses a collection information- theoretic losses and some regularizers. In the ablation study, we investigate the role of each component and we discuss our design choices. In the following, we analyze different design choices like the entropy region size, the conditional entropy in the information transportation loss, and the regularizers.
Ablation analysis Using the proposed evaluation metrics, we analyzed several aspects of MINT on CLEVRER (Yi et al., 2019). We report the results in Table 5.
23

---

Under review as a conference paper at ICLR 2023
Since we compute the local entropy using the probability of the pixel value in its neighborhood region, we investigated the effect of the region size on the performance by varying the region size while using the information maximization (IM) (i.e., ME and MCE from Sec. 2.2.1) loss alone. The results show that a region of size 5 x 5 gives the highest values for the DOP and TOP metrics. We observe also that increasing the region size led to increase in UAK, with a decrease in RAK; we hypothesize this is due to an over-smoothing effect of the bigger region, which leaks some information outside the objects. We noticed, on the other hand, an increase the order of 30 minutes in the training time (50% of the training time) of one seed when increasing the region size by 2. Given the marginal improvement and the need for more resources, we adopted a region size of 3 x 3 for all of our experiments.
Then, we examined the information-theored additional hyperparameter k to set the contribution of the conditional entropy in the information transportation (IT) loss. The results prove that adding conditional information improves the keypoint detection, with K = 0.5 giving the best results for DOP and TOP followed by K = 0.9. The value of RAK increases with lower K, because keypoints seek the same areas of high information to reconstruct much information as possible, leading to the redundant assignment. The introduction of conditional entropy in the IT loss, as describe in Sec. 2.2.2, helps mitigate this behavior by lowering the reconstruction error outside the transportation regions, i.e., the keypoint position in the current and the previous time frame. We highlight two values from this part; with k 0.5 we get the best scores for DOP and TOP, while K- = 0.9 trades off well all of the metrics (we call this model MINT w/o Reg. highlighted in light blue, that is also referenced in Table 1).
losses without regularization, where we have an
Next, we investigate the regularization terms proposed in our method: (1) the movement loss controlled by the weight my in the information transportation loss, (2) the overlapping loss (O), and (3) the active status loss (S). We experimented with all possible combinations of those regularizers. We can observe that the movement regularizer helps decrease the UAK metric, this regularizer stabilizes the keypoint movement and constraints the keypoints from jumping into the background. The overlapping loss reduces the RAK value by almost half (from 3. 3.982 2.079), but this comes with a higher UAK. The status loss pushes the UAK lower but comes at the cost of lower DOP and TOP. Introducing the overlapping and the status loss together allows better overall performance, where the overlapping loss pushes the DOP higher. We achieved the best trade-off across all metrics by setting x = 0.9 while using all the regularizers (highlighted in light green). We adopt this option for our method MINT, and proved to outperform the baselines both in the synthetic dataset (quantitatively proved in Table 1, and qualitatively shown in Fig. 4) and for realistic scenarios (Figs. 5 and 6).
Finally, we investigated the performance of the losses that work for single images, mainly the masked entropy loss (ME) with the regularizers: the active status loss (S) and the overlapping loss (O). This combination of losses does not use any temporal information, hence, it can operate on static images. We train this combination of losses on CLEVRER, operating on single images. We can observe that the model learned to track objects despite being trained on single images. However, we argue that, due to the structure of our training process using samples of sequence of images, biased the model towards reducing the movement of the keypoints while attending to features, leading to good tracking performance. We call this ablation MINT w/o Temp. and we discuss it further later.
We show that if we have enough knowledge about the environment and we can decide the suitable number of keypoint (e.g.. K=10 keypoints for CLEVRER), then the IM loss alone is enough to get good performance (last row in Table 5), with low UAK and RAK, as the keypoint assignment is easier.
We further discuss the two major ablations MINT w/o Reg. and MINT w/o Temp. in detail regard- ing their performance qualitatively on the realistic d
MINT w/o Reg. Our method MINT without the regularization terms, i.e., (1) removing the regular- ization for the keypoints' movement m. = 0 in the information transportation loss, (2) removing the overlapping loss X, = O, and (3) removing the active status loss X, = O. Besides the quantitative results in Table 5, which show that the information-theoretie losses can detect and track objects better than the basclines (outperforming all of the baselines in the DOP and TOP metrics), we provide qualitative evidence of the performance of the proposed cnarmation-thoored losses, where MINT w/o Reg. can detect and track the object in synthetic Fig. 12 and realistic scenes Figs. 10 and 11.
VVideo results for the ablation study: https://sites wooeees..ooleeee www.vvewwmmntt-iiddee ablations
24

---

Under review as a conference paper at ICLR 2023
Table 5: Ablation study on MINT losses. We report the statistics of the metric values over 5 seeds. IM stands the information maximization losses (ME + MCE), the contribution of the conditional entropy in the IT loss, ma is the movement regularizer weight in the IT loss, is the overlapping loss and S is the active status loss.
information transportation, x decides
The ablations picked for MINT w/o reg. red, and light green consequently. The weight scales used for all the ablations (ΛмE = AMCE = 100, ArT = 20, X = 10, Ao 30, K=25).
MINT w/o Temp. and MINT are highlighted with light blue, light
The near the method's 's name indicates longer training time.
Method
DOP f
TOP f
UAK 4
RAK 4
(3x3)
0,951 + 0.042
0.929 0.048
6.777 + 1.369
3,885 + 1.090
IM (5x5)*
0.956 + 0.036
0.932 + 0.043
8.276 = 1.428
3.660 + 1.083
IM (7x7)*
0.951 + 0.041
0.926 1 0.048
9.946 = 1.593
3.098 1 0,959
IM+IT (ma 0,k=0)
0.917 = 0.072
0.897 = 0.077
3,543 = 1,529
5.096 = 1.587
IM+IT (ma 0.k=0.5)
0.935 0.058
0.916 0.063
4.754 = 1.463
3,982 = 1.226
IM+IT (mg = 0,x=0.9)
0.918 + 0.073
0.897 0.078
6.793 1.956
2.478 0.865
IM+IT (md = 0,x=1)
0.916 + 0.073
0.895 + 0.078
5.645 = 1.873
2.336 + 0.768
IM+IT (ma = 1)
0.883 = 0.097
0.865 = 0.102
1.665 = 0.954
1.896 = 0,706
IM+IT (ma 0)+0
0,921 = 0.066
0.898 0,073
7.769 = 1.880
2.079 = 0.604
IM+IT (ma 1)0
0,879 = 0.102
0.861 0.105
2.196 = 1.228
1.705 = 0.582
IM+IT (my = 0)SS
0,851 + 0.114
0.830 0.118
1.057 = 0.666
1.159 - 0.455
IM+IT (ud = 1)48
0.842 + 0.116
0.823
1.060 = 0.735
1.180 + 0.475
IM+IT (m. 0)+++0
0.859 + 0.112
0.840 +
1.130 = 0.710
1.324 + 0.508
IM+IT (ma 055
0.844 0.120
0.826 - 0.123
1.100 0.716
1.121 = 0.451
IM+IT (m. 1x=00.)+++
0.855 + 0.118
0.838 0.121
0.889 = 0.639
1.123 0.448
ME4S+0
0.849 0.115
0.826
0.119
0.938
0.615
1.142 0.446
MINT w/o Reg.
0,918 1 0.073
0.897 1 0.078
6.793 = 1,956
2.478 1 0.865
MINT w/o Temp.
0.849 + 0.115
0.826 1 0.119
0.958
0.615
1.142 1 0.446
MINT (ours)
0.855 = 0.118
0.838
0.121
0.889
0.639
1.123 = 0.448
K=10
IM (3x3)
0.879 = 0.085
0.847 + 0.095
1.662 = 0.781
1.536 = 0.554
On the other hand, the experiments justify the role of regularization in stabilizing keypoint detection and removing excessive keypoints. Fig. 12 from CLEVRER and Fig. 10 from SIMTATE shows a better distribution of keypoint when using MINT over MINT w/o Reg. Fig. 11 depicts the contribution of the regularization losses for economizing the number of used keypoints in the come-and-go situation, allowing MINT outperform the other models.
MINT w/o Temp. our method MINT operating on single images without the losses that operate
temporally over two images. MINT w/o Temp. requires (1) removing the masked conditional entropy loss AMCE 0, and (2) the information transportation loss Årr 0.
This ablation proposes good performance for keypoint detection on static images when using the losses that operate on single image. Table 5 shows that MINT w/o Temp. can detect 85% of the objects in the scene while distributing the keypoint reasonably. Fig. 12 shows that the MINT w/o Temp. assigns keypoints to objects in the scene successfully. Fig. 10 shows that MINT w/o Temp. can detect the human and objects in the background, but due to the lack of temporal information, doesn't 't concentrate on the moving objects (e.g. the hand of the human).
Overall, the full MINT model (ef. Table 5) trades off the need for good detection and tracking performance, but with a reasonable distribution of keypoints, to adequately represent the informa- tion in the video when minimizing our information theoretic losses (i.e., maximizing the covered information entropy spatiotemporally). as dictated by Theorems and 2.
E.2 HYPERPARAMETERS
Table 6 provides the hyperparameters used for CLEVRER (Yi et al., 2019) in our experiments. We the same values for all other datasets, i.e.also for MIME (Sharma et al., 2018), SIMITATE (Memmesheimer et al., 2019), and MAGICAL (Toyer et al., 2020). The only exceptions are the activation threshold 7, the std for heatmap oa, and the threshold of the heatmap where
25

---

Under review as a conference paper at ICLR 2023
Frame
Frame 10
Frame 30
Frame 40
MINT w/o Temp. MINT wio Reg. MINT (ours) (ours)
Bililllss Miile de
(ours)
Figure 10: Crowded scenes. video from SIMITATE dataset with human moving room. compare MINT with its ablations MINT w/o Reg. and MINT w/o Temp. from Appendix E.
Table 6: Hyperparameters
Parameter
value
Parameter name
value
Parameter
value
learning rate epochs activation threshold
0,001
clip value
10.0
weight decay
0.00001
100
num keypoints K
25
number of stacked frames
3
15
entropy region size /
std for heatmap dr.
9.0
Threshold for heatmap
0.1
Thresholded heatmap scale
3.5
CE contribution (IT)
0.5
movement weight (IT)
1.0
ME weight AME
100
MCE weight AMICE
100
IT weight An
20
active status weight X
10
overlapping weight X
30
values depending the size of the input image (i.e., 7 15, oc, 9.0, 0.1 for MIME, 7 10,
OG, = 9.0, = 0.5 for SIMITATE, = 10, aG. = 7.0, T 0.3 for MAGICAL). Our method requires sequence of 2 frames for the loss computation, and found that the batch size does not affect the training and can be chosen based on the available GPU resources.
E.3 BASELINES
Video Structure (Minderer al., 2019) is unsupervised method for learning keypoint- based representation from videos. Video structure learn keypoints detector ddet (dc)(v)) = I₃ for video sequence 1/ that captures the spatial structure of the objects in each frame in set of keypoints z;- Additionally, it learns a reconstruction model 6eee that reconstructs frame ii from its keypoint representation H, and the first frame of the sequence vi. An additional skip connection from the first frame to the reconstruction model output changes its actual task to predict v2 Uı; hence
2t V1 ( xz).
The keypoint detector is trained to optimize three losses: (1) 1.2 image reconstruction loss
Limage Σ
Ilu
(44)
where v the true and is the reconstructed image. (2) Temporal separation loss penalizes the overlap between trajectories within Gaussian radius dsep
dek
Coop ΣΣ expl
(45)
Tsep
where des- Σ3 |1 Fx) keypoints and K. (3) Sparsity loss adds 1.1 penalty on the keypoint intensity y (the value of the corresponding
(0
I is the distance between the trajectories of )11
26

---

Under review as a conference paper at ICLR 2023
20
Number of keypoints
15
Transporter
Video structure
MINT w/o Reg.
MINT
5
-20
20
40
60
80
Figure 11: Come-and-go scenario. In manipulation video from MIME, the hand enters after the start of the video and departs before the end. We plot the number of active keypoints timesteps. The results show the of regularization improving resource assignment economizing the number active keypoints.
feature map) to encourage keypoints to be sparsely active
Caparse Σ
|
(46)
a keypoint active if the intensity is higher than a specific threshold th the threshold a hyperpa- rameter that has to be tuned depending the video.
Transporter (Kulkarni et al., 2019) is neural network architecture for discovering keypoint represen- tations in an unsupervised manner by transporting learned image features between video frames using the keypoint bottleneck. During training, spatial feature maps Φ(z) and keypoint coordinates (fx)) are predicted for source frame v, and target frame x, using a ConvNet and KeyNet (Barroso-Laguna et al., 2019). The keypoint coordinates are transformed into Gaussian heatmaps heça)-
trasmported features map ò(r, 8t) generated by suppressing both sets of keypoint location in ((x,) and composing into the feature maps around the keypoints from Ny;
Ó(I.. st)
h (1 hep=)))) Φ(x.) + hypta))) 0(xx).
(47)
An additional refiner net learns to map the transported features maps into an image ă, The learning objective is reconstructing the target image 2, from the process. Hence, the Transporter optimizes the L. reconstruction error || h
Transporter-modifes is modified version of the transporter baseline (Kulkarni et al.. 2019). The original implementation of the method has two potential bottlenecks; (1) the feature maps o(s) have receptive field of size 24 for each position, for input of size 128x128 (2) the resolution of the
27

---

Under review as a conference paper at ICLR 2023
MINT (ours)
Transporter
Video Structure
MINT w/o Reg. (ours)
Transporter-modified
MINT w/o Temp. (ours)
Qualitative results CLEVRER dataset for Task (object detection and tracking) and Task II (learning dynamics). We include additional results for all baselines Appendix E.3.
Figure 12:
feature maps between which the features are transported is 32x32. For fair comparison to our method, which uses an entropy region of size 3x3, we modified the network architecture of ConvNet ((x) to have (1) receptive field of 7 and (2) feature map of size 122x122. We call the new architecture Transporter-modiin modiife.d
The experimental results show that the Transporter-modified model outperforms the original Trans- porter in the quantitative evaluation on CLEVRER dataset (Yi et al., 2019) (cf. Table 1). An interesting observation from visual results in Fig. 12, one (top middle image) are more stable than those the Transporter-mode modified (bottom middle image). We argue this behavior is due the smaller receptive field leading the model to assign keypoints to features instead of objects, and thus keypoints jump to similar features in different objects.
keypoints of the original Trasnporter
E.4 INTERACTION NETWORK ARCHITECTURE
The interaction network (Battaglia et al., 2016) is a model developed for learning the interaction relations between physical objects to infer the physics of the environment. The interaction network treats the objects as nodes of a graph, with the relations as edges. In our case, we use the keypoints object nodes, with the coordinates, status, and positional encoding as features. We form fully connected graph of the keypoints, with no relational features for edges.
The interaction network used in our experiments has two sub- models; relational model and an object model. The relational model uses the relational information and object attributes to predict the effects of all interactions. The object model the effects update the features of the object. We encode node features before passing them to the interaction network. After one pass through the interaction network, we decode the features into coordinates for the prediction task, and we add another prediction head for the action decoding in the imitation learning task.
E.5 IMITATION LEARNING RESULTS
The CNN agent is trained from scratch for every environment. Note that the state space for the CNN agent, the image pixels, is order of magnitude higher than the keypoints' features. For fair comparison, we train the CNN agent longer (twice the epochs used for training the MINT-bbased agent to counteract for MINT's pretraining).
CNN-agent:
The CNN feature extractor consists of convolution blocks, each consisting 2D convolutional layer with ReL.U activation function and batch normalization layer. The input to the model is
28

---

Under review as a conference paper at ICLR 2023
MoveToRegion
MoveToCorner
MakeLine
Figure 13: Rollouts from MINT agent in MAGICAL (Toyer et al., 2020) datasct.
sequence of 4 color images stacked over the channel axis; hence, the input size is 12 96 96. The layers have 64, 128, 128, 128, and 128 filters, with a kernel size of 3 and stride of 2, except the initial layer, which has kernel size of and stride of 1. The output of the last block is flattened and passed to linear layer to provide the final features. A policy model uses the feature to infer the actions. The policy model is multi-layer perceptron with linear layers of sizes 128, 64, 32, and 32. The output matches the action dimension of MAGICAL environment which is 18.
We, also, provide visualizations of the learned policies of the MINT-based agent on the three environments of MAGICAL in Fig. 13. The visualization shows that MINT can assign reasonable keypoints for the agent and all the objects in the environment. The imitation agent can solve the first two tasks MoveToRegion and MoveToCorme agent receives a score of 1.0 when sorts all objects one line, while it gets score of 0.5 for putting out of in one line. Our imitation agent could (occasionally) sort only 3 out of 4 in the depicted environment (which led to an overall 0,2 mean score over seeds - Table 3), despite being able to assign keypoints to all objects. The results suggest that there is problem in encoding the relational features between keypoints, hindering the agent to reason upon getting the right locations. We argue that further investigation of the appropriate model pool information from the keypoints is necessary to solve this harder task, but this is out of scope of the current work.
but it struggles with the last task MakeLine. The
E.6 ADDITIONAL VIDEO RESULTS
We provide additional video results both as supplemental material and on the website our project: http:///teeggoggg...../viviiiiii
29

---

Under review as a conference paper at ICLR 2023
F CODE
We include our code in the supplemental material, and we plan to publish our code under an open- source license to accompany the paper upon acceptance. We provide instructions run the code, with sample datasets to reproduce the results in the paper.
The implementation of video structure (Minderer al., 2019) 6 uses outdated libraries. Due to compatibility reasons, we reimplemented their code in PyTTrcc with our best effort.
We adapted the implementation of Transporter from Li et al. (2020) into the codebase of MINT
G EXTENDED RELATED WORK DISCUSSION
Image information entropy. Our work draws inspiration from classical approaches in saliency detection in images that use local information to detect salient objects (Kadir & Brady, 2001; Bruce & Tsotsos, 2005; Fritz et al., 2004; Renninger al., 2004; Borji & Itti, 2012). We use the entropy as a measure of saliency. Bruce & Tsotsos (2005) proposes that regions with high self-information typically correspond to salient objects, and Alexe al. (2010) quantified objectiveness by self- information approximated via center-surrount feature differences. Gopalakrishnan et al. (2020) used local predictability as measure of saliency learn keypoints.
However, image information entropy has applications in different computer vision problems, apart for saliency detection. Sabuncu (2006) uses entropy and other pntommtiion-theoreer measures for image registration. Ferraro et al. (2002) propose an entropy-basee representation of image information with application to pattern recognition and active vision. Other applications are medical image analysis (Hržić et al., 2019), nuclear detection (Hamahashi et al., 2005), image compression (Minnen et al., 2018), and measuring image randomness ( (Wu et al., 2013). Our method proposes information-Heee -eemmation-theortiic losses based on the local entropy as measure of information.
Representation learning. The idea of extracting sparse feature representations of visual data is dominant in computer vision and machine learning research (Harris et al., 1988; Lowe, 2004), and connects to the functioning of the human visual system (Marr, 2010). Such sparse representations are generally known as Pol, which are 2D locations that are stable and repeatable under various lighting conditions and viewpoints (DeTooee et al., 2018). Traditional geometric computer vision methods relied on the extraction of hand-crafted feature descriptors (Lowe, 2004; Rublee et al., 2011) 1) for tasks like localization (Schmid et al., 2000: Mur-Artt Artal et al., 2015). In the deep learning era, CNN architectures have proven superior to handcrafted features (Yi al., 2016; De'one et al., 201 8; Song et al., 2020; Zheng et al., 2017). Deep approaches extract clouds of Pol that are useful for correspondence searching visual place recognition from different viewpoints posc-estimation for control (Florence et al., 2019).
(Hausler et al., 2021),
Related to our method are object-centric approaches Singh et al. (2021); Locatello et al. (2020); Dittadi et al. (2021), which aim learn abstract representations for objects in scene. Our approach to keypoint discovery, alongside our metrics, are directed to learning and evaluating keypoints as an object-centric representation. In that direction, (Hung et al., 2019; Zhang et al., 2018a) finetune pre-trained ImageNet features such that each filter represent an object part. SCOPS (Hung et al., 2019) uses way of computing saliency based local connextivity of superpixels, as defined in (Zhu et al., 2014), and poses a saliency constraint in part discovery. Differently from SCOPS, we use local image entropy quantify saliency, and provide method that can not only detect important parts in static frames, but can also represent temporal change in information, due to changes in video. ICNN (Zhang et al., 2018a) optimizes mutual information and spatial entropy for interpretability. In particular, the authors try to enforce part template matching to single target object part, that should be interpretable.
Differently to these methods, we can represent scenes with multiple objects in videos, and not in static images with single objects, in which the information dynamically changes, optimizing for information coverage and temporal consistency. Moreover, our method learns from scratch and uses feature maps to locate keypoints representing part features. The keypoints in our approach information transporters of their local information (in the heatmap around them), while we can
https://gituub.com/gog www.://wwwchooooooooooeeeeeteeee
"https///github.. httpttuww.cc//p/aiaaaaaa com/pairlab/v-ccie
"hhe baselines are part of the submitted code, that will be open-sourced upon acceptance.
30

---

Under review as a conference paper at ICLR 2023
economize the use of keypoints to adequately represent the video information, with minimum number of keypoints.
Keypoint learning. Keypoints represent class of Pol that have a semantic entity, e.g., representing objects (Duan et al., 2019), or human joints (Cao et al., 2017: McNally et al., 2022). but most methods rely explicit annotations of keypoint locations. Related to work methods that rely image reconstruction for detecting keypoints in video streams in an unsupervised fashion (Jakab et al., 2018; Zhang al., 201 8b; Minderer al., 2019; Kulkarni al., 2019; Gopalakrishnan al., 2020). Jakab et al. (2018) use an autoencoder architecture with a differentiable keypoint bottleneck trained on the difference between a source and a target image, trying to restrict the information flow. MINT also uses a differentiable keypoint representation, but it operates on the output of an hourglass architecture. Our results suggest that learning to redistribute the information after compression is beneficial for keypoint discovery (Newell et al., 2016). Minderer et al. (2019) use a similar architecture as Jakab al. (2018) but operate on video sequences for detecting keypoints, using the intensity of the bottleneck heatmap as an indicator of the importance of a keypoint. Setting up threshold on the intensity is challenging and domain-specific.. Contrarily, we learn a binary classification of active/inactive keypoints and optimize the number of keypoints used in every frame, as we showed in our results. Kulkarni et al. (2019) propose feature transportation in the keypoint bottleneck of Jakab al. (2018) before reconstruction. Inspired by this idea, MINT performs information transportation and waives the need for image reconstruction, which would require an additional appearance encoder and a reconstruction decoder from keypoints. Gopalakrishnan et al. (2020) devised three-stage architecture that first learns a spatial feature embedding, then solves a local spatial prediction task related object permanence, and finally converts error maps into keypoints. While this method employs local information, it trains three architectural modules separately, unlike (Kulkarni et al., 2019; Minderer al., 2019) and MINT that use single end-to-end pipeline for keypoint discovery.
principles proved advantageous training and understanding machine learning models (Yu et al., 2021). Different information measures aim to describe random variable's 's behavior due to probability density function. The probability density function is normally unknown, and machine lcarning methods usually estimate it (Pardo, 2018; Avi-Aharon et al., 2020).In method, we use kernel density estimation (KDE) (Parzen, 1962) to estimate the probability density function for a region of pixels. Various coomatiionthhoreete quantities were used in machine learning for different applications; examples are the cross-entropy loss for classification (Good, 1992; Goodfellow et al., 2016), maximum entropy regularization in reinforcement learning (Peters al., 2010; Haarnoja al., 2018), mutual information for self-supervised learning and interpretability (Rakelly et al., 2021; Zhang et al., 201 8a), and KL divergence for training deep energy models (Yu et al., 2020). Our approach uses Shannon's 's definition of entropy (Shannon, 2001) to compute the local image entropy. With image entropy. we estimate joint entropy, conditional entropy, and mutual information and develop our information- theoretic losses.
Information-theorede approaches in Machine Learning.
The information bottleneck is an iiformationn-hhoreee in deep neural networks (Tishby et al., 2000). The information maximization principle (InfoMax) (Linsker, 1988) treats the neural network as an information channel and aims to maximize the information transferred through the network. Recent methods in computer vision and natural language processing use the InfoMax principle for self-supervised learning (Oord et al., 2018; Hjelm et al., 2018; Kong al., 2019). Our method adopts the treatment of the neural network as information channel in the information maximization loss and extends it to treat the keypoints as transmitters of information, while being completely unsupervised.
framework that analyzes learning dynamics
31


Reviews:
[{"review_id": "ozHiZy5TsKK", "reviewer": "Reviewer_srpG", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an unsupervised keypoint discovery method, especially for videos. To this end, authors propose to build an entropy layer to measure the information, and a deep learning model to predict keypoint. The model is learned in an unsupervised manner, by maximizing the information in a single frame and across frames. Authors demonstrate that the keypoints can be used for many downstream vision tasks such as detection, tracking and dynamic learning.", "summary_of_the_review": "Overall I feel the paper above the acceptance bar, but there are still a bit more can be discussed and investigated, see in \"Weaknesses\".", "strength_and_weaknesses": "### Strength\n* Unsupervised learning of keypoints is an important and difficult task. It avoids the requirement of extensive labels and can help many downstream tasks like matching, tracking etc.\n* The proposed method learns reliable keypoint on both static and moving objects, and the number of active keypoints is compact.\n* The proposed method works well on both synthetic objects and realistic complex scenarios. \n### Weaknesses\n* The novelty is not very strong. Some similar works on unsupervised saliency/keypoint learning have already been proposed and need to be discussed in this paper. For example, [a] has a segmentation network to predict masks (instead of keypoints) training in an unsupervised way, and [b] proposed to learn object parts in an unsupervised way. These two papers are not identical this the submission but share some similarities.\n* The ME loss, the overlapping loss and the active status loss seem to work on single images instead of videos. I am curious how well it works on static images.\n\n\n[a] SCOPS: Self-Supervised Co-Part Segmentation\n[b] Interpretable Convolutional Neural Networks", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "nsJpdozPBIx", "reviewer": "Reviewer_PH7P", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new keypoint representation learning method through novel losses inspired by information theory. One of the losses encourages the keypoints to represent the scene without information loss, and the other guides the keypoints to represent well the dynamics of the video through conditional entropy and the keypoints in the previous frame. In addition, they proposed overlapping loss to prevent the multiple keypoints from representing near regions and active status loss to represent the scene with sparse keypoints.\n\nThey empirically show their method outperforms baselines (Transporter [1] and Video Structure [2]). In particular, they show their method can represent the scene with a smaller number of keypoints and better transportation (e.g., when some of the objects in the scene move, only the relevant keypoints track). \n\n[1] Tejas D Kulkarni, Ankush Gupta, Catalin Ionescu, Sebastian Borgeaud, Malcolm Reynolds, Andrew Zisserman, and Volodymyr Mnih. Unsupervised learning of object keypoints for perception and control. Advances in Neural Information Processing Systems, 32, 2019.\n\n[2] Matthias Minderer, Chen Sun, Ruben Villegas, Forrester Cole, Kevin P Murphy, and Honglak Lee. Unsupervised learning of object structure and dynamics from videos. Advances in Neural Information Processing Systems, 32, 2019.", "summary_of_the_review": "This paper proposes a new method for keypoint representation learning inspired by information theory. Through the proposed objective, they show better detection and prediction performance. Moreover, they also show the sparsity of using the keypoint representation and more stable transportation on the video dataset. However, at least for me, the explanation for the transportation loss is not clear, and the absence of the ablation studies of each loss makes me less understand their method.", "strength_and_weaknesses": "Strength\n- Well-designed model and losses. I could agree with why they propose and applies the losses.\n- Good visualization, especially for transportation and sparse keypoints usage in Figures 5 and 6.\n- Investigating Imitation learning. Usually, keypoint representation learning or similar representation learning papers, such as Object-Centric Learning, validate their method for detection, segmentation, or prediction [3,4,5]. Still, they evaluate their approach, not just detection and prediction, but also imitation learning.\n- Mentioning their limitations because their model uses local image entropy.\n\n[3] Locatello, Francesco, et al. \"Object-centric learning with slot attention.\" Advances in Neural Information Processing Systems 33 (2020): 11525-11538.\n\n[4] Dittadi, Andrea, et al. \"Generalization and robustness implications in object-centric learning.\" arXiv preprint arXiv:2107.00637 (2021).\n\n[5] Singh, Gautam, Fei Deng, and Sungjin Ahn. \"Illiterate dall-e learns to compose.\" International Conference on Learning Representations. 2021.\n\nWeaknesses\n- Overall, the paper is well-written and easy to follow, but section 2.2.2 is not easy, at least for me. \n    - In Figure 3, you mentioned The target information $E(T_t)$ is the sum of the conditional entropy E(I_t|I_{t-1}) and the patch around the keypoint at time $t-1$, but in paragraph, $E(T_t)$ is described differently. It looks like the summation of the patch around the keypoint at time $t$ and the new information not represented through the keypoint at time $t$.\n    - In Figure 3, $E(S_t)$ looks like many regions with high entropy are not represented through keypoints. What is the original scene? From the entropy figure, it is not easy to understand.\n    - It is hard for me to follow that \"In a temporal sequence of frames, optimized keypoint information transport means that we can reconstruct the information of the current frame $E(I_t)$ using information from previous frames $E(I_{t−1})$ and the keypoint information from both frames.\"\n    - You mentioned being inspired by [1]. When comparing with the loss in [1], you apply the new term, $E(I_t | I_{t-1}) \\bigodot (1-h_i^{(t)})$. It can be analyzed like the new information (dynamics) but not represented through keypoints at time $t$. Why must it be applied to be the target information?\n- They propose a set of multiple losses, but they didn't do the ablation studies on what happens if one or some of the losses are not used. It can be interesting and show the role of each loss empirically, so I hope to be updated in the revised version.\n- The main baselines are Transporter [1] and Video Structure [2]. Could you make an additional section to discuss the difference in detail?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "SjvhHjuD2A", "reviewer": "Reviewer_Nich", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes an information-based unsupervised keypoint discovery method. Unlike other works that transport the CNN features between image frames, this work proposes to transport the information represented by the entropy of pixel color computed in a local region, and the authors argue that it provides a better inductive bias for unsupervised keypoint detection. To verify the hypothesis, they provide a quantitative and qualitative evaluation of synthetic and real-world datasets and demonstrate improvements over baselines on all tasks. ", "summary_of_the_review": "The problem this paper is addressing is of significance to the community. The proposed method is novel, and the evaluation results also show improvement over the baselines. The main concern is the lack of certain key experiments and ablation studies that are required to justify the claim and the advantage over existing works. The score could be increased if the above concerns are properly addressed.\n\n---\n## Post-rebuttal updates (edited):\n\nI'm grateful to the authors for the comprehensive feedback and extra experiments in the revision. \n\n1. The extra experiments on the additional loss terms for keypoint detection and tracking reveal that the information contributes significantly to the performance. In fact, in some cases, the model performs better without the additional loss terms, which is interesting. It would be nice to include this setting in other metrics and experiments as well. \n2. I also appreciate the additional findings on Transporter-modified. The results indicate that Transporter performance could be enhanced by fine-tuning its architecture, which seems to imply that a deeper investigation into the model design is required to reveal the cause of the performance difference. \n\nAdditionally, after reviewing the other reviewers' comments, I concur with Reviewer NNTd that the paper is theoretically overstated. For example, it is unclear to me how the proof in Appendix C.2 can justify the approximation $E\\left(I_t, I_{t-1}\\right)=\\max \\left(E\\left(I_t\\right), E\\left(I_{t-1}\\right)\\right)$. Neither the assumption that two consecutive frames are independent nor the statement that the joint will be equal to the max of two marginal are theoretically sound. It appears to be an empirical discovery rather than a result of mathematical analysis. \n\nDespite the writing being theoretical sugar coating, the performance difference between the model and baselines demonstrated in the experiments and the website videos still clearly indicate that this is a promising approach in unsupervised keypoint detection. Therefore, I am leaning towards acceptance and have increased the score to 6.", "strength_and_weaknesses": "Strengths:\n\n- The research topic of unsupervised keypoint detection is of significance to the community.\n- The method is original, and the evaluation results do demonstrate a consistent improvement over baselines on several tasks.\n- The proposed parallel entropy layer is novel and could be useful for future works.\n- The evaluation is well-designed. It includes a range of tasks including object detection and tracking, dynamic prediction, real-world object discovery, and imitation learning, which thoroughly demonstrate the quality of learned keypoints.\n- The authors also introduce a new set of metrics to evaluate object detection and tracking quality, which can be considered one contribution.\n- The additional video material provided in the anonymous link and supplementation are much appreciated, they provided a clear insight into the benefits of the method.\n\nWeakness:\n\nThe main weakness of the paper is the lack of some key ablation studies, which make the claim that the method represents the scene better than the baselines less convincing. \n\n- In addition to using local pixel entropy as input, the method also introduces several regularization terms in the training objective. It is possible that the improved performance comes from these new loss terms which are not included in the baseline models, they include:\n    1. Minimizing the norm of the distance traveled by each keypoint in the IT loss to regularizing excessive keypoints’ movement\n    2. The overlapping loss that encourages the keypoint to spread over the image\n    3. The additional activation status variable, and the corresponding active status loss that control (reduce if possible) the number of active keypoint\n    \n    These additional terms could be considered contributions of the paper, but they should be separated from the main claim that the information-based method is better than the CNN-based method. Therefore, two sets of experiments can be considered in the paper, (1) the ablation on the proposed method to show the effect of each regularization term, or (2) incorporating these terms in the baseline models when comparing them with the proposed method.\n    \n- The qualitative results show a clear advantage of the proposed method which is very impressive. That is, for the proposed method, the movement of an object only affects the keypoints attached to its body, but for the baseline models, it also causes the movement of many keypoints surrounding the object. However, it is unclear if this behavior is due to architectural design, namely the receptive filed and transporting resolution.\n    1. Receptive filed. In the baseline model, e.g., transporter, the CNN features that the model learns to transport has a receptive field of 24 (input size is 64x64) for each position (computed based on the implementation the paper use [1]), so each pixel contains the information of a large area of the image. This might explain why the surrounding keypoints will also follow the movement of the object - they capture part of the object’s information. On the other hand, the image entropy is designed to represent local information, the entropy region size is three. Is it possible that the baselines can enjoy similar improvement with a careful receptive field design?\n    2. Resolution. Based on transporter implementation [1], the resolution of the feature map to transport is 16x16, and the input image resolution is 64x64, both are smaller than the counterpart of the proposed method and could inevitably cause loss of information. Please provide some investigation into this factor.\n\n[1] [https://github.com/pairlab/v-cdn](https://github.com/pairlab/v-cdn)", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "SfF-4qTAh53", "reviewer": "Reviewer_NNTd", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a novel method for learning to represent images with keypoints, that can be used for many downstream tasks, like object detection and tracking and learning dynamics.\n- A fixed number of keypoints are extracted from a neural network from each frame.\n- An information transporter (similar to Kulkarni et al., 2019) is trained so the keypoints reconstruct local entropy patterns of neighboring frames\n- novel performance metrics are introduced and the proposed method outpuerforms the competition on these metrics as well as on the standard metrics.", "summary_of_the_review": "The paper is a mixed bag: strong results and bad presentation. There are two possibilities:\n1) There is an theory, from which the losses can be derived. Then this is not presented, and the explanations are insufficient.\n2) The local entropy is just a saliency feature. In this case the explanation is incorrect.\n\nIn both scenarios the paper is not ready for publication.\n", "strength_and_weaknesses": "Strengths:\n\nThe method well on multiple tasks and performs better than the competing methods.\n\nWeaknesses:\n\nThe title exaggerates the contributions: \"An information-theoretic approach to unsupervised keypoint representation learning \"\n- The reader expects a theory that explains something not yet known, provide some optimality etc. There is no theorem, nor proof.\n- When entropy is minimised or maximised, one wants to achieve some property for the associated random variable, and there is an explanation why that property is desired for the task. Here this is not the case the \"local entropy\" acts as a hand crafted saliency feature.\n\nThe explanation about the entropy is a forced analogy\n- The losses and equations are not wrong, but the equations would make more sense if they described reconstructing a saliency pattern. E.g. CE(I_t|I_{t−1}) = max(E(I_t)-E(I_{t−1}), 0) could be interpreted as the novel saliency on the next frame and there is no need to approximate the joint entropy with the maximum.\n- From a saliency perspective using heat maps make sense, the keypoints are used to reconstruct the pattern. Where do they come from an information theoretic perspective? All losses and explicite functional relationships (e.g. heatmaps) should be derived top-down from the theory and assumptions.\n\nSome important content is delegated to the appendix, it could be better placed in the main text\n- (E) Ablation study of the losses\n- (C) Evaluation metrics. As this is listed as a contribution, this should be part of the main text\n\nThe ablation studies are limited. It would be important to see some results with different design choices, e.g. number of keypoints\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject