Active Acquisition for Multimodal Temporal Data: A Challenging Decision-Making Task
Authors: Jannik Kossen
Cătălina Cangea
Eszter Vértes
Andrew Jaegle
Viorica Patraucean
Ira Ktena
Nenad Tomasev
Danielle Belgrave
Venue: Accepted by TMLR ()

Abstract: We introduce a challenging decision-making task that we call active acquisition for multimodal temporal data (A2MT). In many real-world scenarios, input features are not readily available at test time and must instead be acquired at significant cost. With A2MT, we aim to learn agents that actively select which modalities of an input to acquire, trading off acquisition cost and predictive performance. A2MT extends a previous task called active feature acquisition to temporal decision making about high-dimensional inputs. We propose a method based on the Perceiver IO architecture to address A2MT in practice. Our agents are able to solve a novel synthetic scenario requiring practically relevant cross-modal reasoning skills. On two large-scale, real-world datasets, Kinetics-700 and AudioSet, our agents successfully learn cost-reactive acquisition behavior. However, an ablation reveals they are unable to learn adaptive acquisition strategies, emphasizing the difficulty of the task even for state-of-the-art models. Applications of A2MT may be impactful in domains like medicine, robotics, or finance, where modalities differ in acquisition cost and informativeness.

Published in Transactions on Machine Learning Research (06/2023)
Active Acquisition for Multimodal Temporal Data: A Challenging Decision-Making Task
Jannik Kossen 1+1
Cätálina Cangea2 21
Eszter Vértes?
Andrew Jaegle?
Viorica Patraucean?
Ira Ktena
Nenad Tomasey²
Danielle Belgrave?
OATML, Department of Computer Science, University of Oxford Coogle DeepMind
Reviewed on OpenReview:
https: // openreview. net/ forfor?m? id= Gbu.bHQEQEE
/ttps://openrevieu
Abstract
We introduce challenging decision-makinn task that we call active acquisition for multi- modal temporal data (A2MT). In many real-world scenarios, input features are not readily available at lest time and must instead be acquired at significant cost. With A2MT, we aim to learn agents that actively select which modalities of an input to acquire, trading off acquisition cost and predictive performance. A2MT extends a previous task called active feature acquisition to temporal decision making about bigh-dimenssonal inputs. We propose a method based on the Perceiver TO architecture address A2MT in practice. Our agents are able solve a novel synthetic scenario requiring practically relevant cross-modal rea- soning skills. On two large-scale, rcal-world datasets, Kinetics-700 and AudioSet, agents successfully learn cost-reactive acquisition behavior. However, ablation reveals they are unable to learn adaptive acquisition strategies, emphasizing the difficulty of the task even for state-of-the-art models. Applications of A2MT may be impactful in domains like medicine, robotics, or finance, where modalities differ in acquisition cost and informativeness.
1 Introduction
In making clinical diagnosis, the medical professional must carefully choose specific set of tests to diagnose the patient quickly and correctly. It is of crucial importance to choose the right test at the right time, and tests should only be performed when useful, as they may otherwise cause unnecessary patient discomfort or financial expense. Recently, largo-scale datasets of medical treatment records have become available (Hyland ltt al. 2020 Johnson et al. 2020). They may potentially facilitate improvements in medical domain knowledge and patient care, for example by allowing us to learn which tests to perform when. While prior work has demonstrated that machine learning can be used to inform complex diagnoses from simple measurements, see, for example, Liotta et al. (2003); Diaz-Pinto et al. 2002)). treatment records present a significant modelling challenge as they contain temporally sparse observations from high-dimensional modalities, e.g. X-Rays, MRIs, blood tests, or genetic data.
Prior work in active feature acquisition (AFA) (Greiner al.) 2002 Melville al. 2004 Ling et al, 2004 Zubek al. 2004] Sheng & Ling) 2006; Saar- Teechansy el al. 2009) has similarly considered the cost of feature acquisition
per-datum basis: given test input for which all features are missing initially, which features should one acquire to best trade off predictive performance and the cost of feature acquisition? This is different to active learning (Settles) 2010),, which minimizes the number of label acquisitions needed for model training. Concurrent methods in AFA often use a Bayesian experimental design approach (Lindley) 1956 Chaloner & Verdinelli, 1995 Sebastiani Wynn 2000), acquiring features
test time
that maximise the expected information gain with respect. to the prediction Ma el. al. 2018 Li & Oliva
"Work doue while interning Google DespMind.
i "Coresppnndecee to jannik. kossen Stc..ox....ll and ccange@Ckleppmind.com.

---

Published in Transactions on Machine Learning Research (06/2023)
Past Future
(b)
Modalities
What
project
Predictive Loss
Images
DMAGRRO]
L
Numbers
(MAGGGD)
1,73205
200000
2,23905
2449488
Timesteps
Agent
A2MT Objective
(a)
ralax)
Acquisition Cost
In many practical applications, features not available priori at test time and have to be acquired at a real-world cost lo allow for the prediction of associated label. In Aclive Acquisilion for Multimodal Temporal Data, we aim to learn agents that efficiently acquire for multimodal temporal inputs: (a) at each timestep, the agent decides which modalitics the input it acquires, paying a per-modality acquisition cost; (b) then, a separate model predicts given the sparse sequence of observations; (c) lastly, the agent gets rewarded for low prediction loss and small acquisition cost.
Figure 1:
2021
Lewis et al. 2021
Alternatively, AFA can be phrased as reinforcement learning task where agents
optimize the trade-off objective directly (Shim et al. 2018 Kachuee et al. 2019 Zannone et al. 2019 Janisch ct al. 2019 2020 Yin et al. 2020). Notably, prior work in AFA assumes static data: although acquisitions are sequential, feature values do not evolve along a temporal dimension. Furthermore, the features themselves usually correspond low-dimensional obscrvations, i.c. single values in a tabular datasct.
In this work, propose active acquisition for multimodal temporal data (A2MT). Taking the above medical setup motivation, we extend the familiar setting of AFA in two key ways: (1) We assumne that inputs are sequences that evolve temporally. Our agents will need to Iearn not only which features to acquire, but also when to acquire them. (2) We no longer assume that inputs are unimodal and low-dimensional. Instead, we assume that each input comprises a collection of high-dimensional modalities, and that acquisitions are made for entire modalitics at each timestep. With these extensions, A2MT gencralizes AFA and reduces the gap to practical applications that are often both temporal and multimodal. A2MT can also find application outside the medical domain (cf. $
To study A2MT in controlled environment, propose a set of synthetic scenarios of increasing difficulty that are temporal and multimodal, both key requirements for A2MT. Further, we propose to study A2MT on audio-visual datasets, concretely AudioSet (Gemmeke et al. 2017 and Kinetics-700 2020 (Smaira et al. 2020). These provide challenging testbed for A2MT and avoid some the complications of working with medical data. We propose a method based on Perceiver TO (Jaegle al. 2021] a modality-agnostic architecture that can be applied directly a large variety of real-world inputs and we explore different reinforcement learning techniques to train the agent. Our method is able to solve subset of the synthetic lasks we propose and provides reasonable performance on the real-world datasets. However, further inves- tigation reveals that our method can ultimately be outperformed by a non-adaptive ablation, highlighting the difficulty of the A2MT task and, consequently, opportunities for future work.
In summmary, our contributions are:
We introduce the active acquisition for maltimodal Lemporal dala (A2MT) scenario ($2)
We suggest both synthetic and real-world datasets that motivate A2MT and allow for convenient benchmarking of methods ($3).
We proposc a novel method based on Perceiver IO to tackle A2MT provide a thorough empirical study (55 and discuss key areas of improvement for future work
2

---

Published in Transactions on Machine Learning Research (06/2023)
2 Active Acquisition for Multimodal Temporal Data
With A2MT (fig. D. we study strategies for cost-sensitive acquisition when data is both multimodal and temporal. We train an agent policy x that makes a binary acquisition decision per modality and timestep. After a fixed number of timesteps, a model makes a sequence-level prediction given the sparsely acquired observations. The agent then observes reward that consists of two terms: (1) The agent gets rewarded in proportion to the negative predictive loss given the observations. (2) Acquisitions come at a (modality- specific) cost and the agent is penalized for cach acquisition made. Term (1) compels the agent to acquire often, as additional observations should improve the quality of the predictions of f. However, this increases the penalty incurred from term (2). The agent therefore needs to trade off acquisition cost and prediction reward, learning which modalities in the input are worth acquiring and when is worth acquiring them.
To make the most of a limited acquisition budget, we aim to learn agents with individualized acquisition strategies that adapt to the sequences at hand. For example, agents may learn to make use of interactions between modalities: a past observation in one modality (e.g. suspicious value in a blood test) may lead to the acquisition of another modality (e.g. a more specialised test), achieving both accuracy and cost-efficeency. A2MT requires models that can successfully accommodate sparse, multimodal, and temporal data. This is challenging for state-of-the-art models, even without the additional complexities of active selection required for successful A2MT.
In A2MT (Alg- D. we assume access to fully observed training set of input-output pairs ((x1, MI)..... (æN, WN)). For classification tasks, for example, y: are labels, y4 E (l,...,C)) Each input D; is a sequence of observations into M modalities Ts,t- = (It,t.1..... a. each modality may be high-dimensional, T.a.m E Rd= For example, Li,., could be a single frame in a vidco where we collapse the image height H, width W, and color channels C into a single axis with dimensionality dm = H- W C. We focus on a single sample and drop the leading axis, x = 2₁, to avoid notational clutter. We use colons to indicate 'slices' of inputs along particular axis, e.g. Fiit- (x 1#1...... f...))
(Z.,1....,
At each timestep t, the observation Ti,t decomposes
At each timestep t E T). we obtain set of acquisition decisions across modalities, or actions, as = (r..), by sampling from the agent pol- icy, a ~ (11::==-,,41-1:0) binary indicator of whether modality m was acquired at time We write ø instead of to highlight the fact that the inputs may contain missing entries, and O are the trainable agent parameters. Here, x gives the joint probability over all possible acquisition decisions at that timestep, including extremes such as acquiring all or none of the modalities, iee T (d. 11:-11: O) E 1] At each timestep and for each modality m, we acquire xm, only if Om, 1. We summarize the set of all actions across timesteps as a (ai,...,0, @T).
A Algorithm 1 A2MT
Inputs: Test input æ, agent n, model f. 1: for t
T do
Here, atm E (0,1) is a
Sample at (--1-1--11 (1:-1;;;
for = 1 to M do
4:
if atm = then Acquire: ãim + Zt,m. clsc
5:
6:
7:
Do not acquire: Temm + 0.
8:
end if
end for
10: end for
11: Return prediction ((
Lastly, we assume access to model f which makes a prediction & given an input a. After completing the acquisition process for a given test sample, we use this model to predict [(ă1r) = ÿ. Consequently, we require that f can predict for multimodal inpuls with features missing arbitrarily across time and modalities.
We train agents to maximize the following reward:
R= E (-C(a) ((((rrr),))) where C(a)= = ((a)=-"
T
M
m=1 Cm at.
Here, the expectation is over the data (a. y) and actions a from the policy; C(a) gives the total cost of acquisition along the sequence; cm is modality-speciffe cost, and ((f(#rr).)) is log likelihood loss. Equation ① summarizes the problem of A2MT: trading off acquisition costs against predictive performance
3

---

Published in Transactions on Machine Learning Research (06/2023)
for multimodal and temporal imputs. In $ detail how optimize this objective using reinforcement learning. We refer to @ for a discussion of A2MT in terms of Markov Decision Processes.
3 Datasets for A2MT
While the medical domain acts as important practical motivalion for A2MT, medical datasets, such Hyland al. (2020); Johnson al. 2020 usually require significant domain expertise. And while the eventual application of A2MT methods in this domain is crucial, we believe that mix of synthetic and real non-medical data can serve as widely accessible testbed for the machine learning community to develop robust methodology. We therefore introduce datasets suitable for developing A2MT methods.
Digit
3 0
4
3
4
3
Raw
Counter
2 0
3 2
1
0
1
0
Digit
1 3 0
4
3 4
4
3 0
v2: Images Counter
2 0
3
2
0
2
Image Audio Counter Digit
0 3 0
4
3
4
H
3
3.1 Synthetic Datasets
We introduce three scenarios with varying levels of difficulty and clear optimal acquisition strategy, designed to lest the cross-modal reasoning capabilities of the agents.
Figure 2: The synthelic scenarios allow for sparse acquisition while keeping perfect accuracy. This re- quires agents capable of cross-modal reasoning. (Tabel is in the above.)
Concretely, we create a dataset (cf. algorithm B.1 of fixed-length se- quences with two modalities: counter and digit. The digit modal- ity is a sequence of random numbers of length T, e.g. 0131043443. For the counter modality, we draw random 'starting values' uni- formly from the set of valid numbers. For each starting value, create sequence counting down to 0 from that starting value, and then concatenate all countdown se- quences, Lee when drawing the starting values 2, 3, and 2, we generate the sequences 210, 3210, and 210, and the final counter modality is their concatenation, 2103210210. We cut sequences to length T. The label attached to each sequence the sum of the digit modality at all timesteps where the counter modality is zero, c.g. 3+3+3=9 this case.
To increase the complexity of this lask, can replace the sequence of raw numbers with sequence of images, for example replacing numbers with matching MNIST digits [LeCun] 1998) for both modalities. We also consider replacing the numbers of the counter modality with audio sequences from the spoken digit dataset (Jackson 2017). Figure 2 shows these synthetic dataset variants. Further, one can increase task complexity by adjusting the sequence length and the number of unique symbols for the modalities.
To solve the synthetic task, agents need to reason about interactions between modalities; key feature of our practical motivation for A2MT. Each modality in the synthetic scenario offers an ideal strategy to save acquisition cost without sacrificing predictive accuracy. (1) The agent can learn acquire the digit modality only if the counter modality is zero, because only then is digit relevant for the label. (2) The agent further learn to skip observations in the counter modality, becanse of the regular pattern they follow.
3.2 Audio-Visual Datasets
Additionally, we propose to apply large-scale audio-visual classification datasets, such as AudioSet (Gemmeke) ct al. (017)) or Kinetics-700-2022 (Smaira et al. (020),, the A2MT setting. These datascts are multimodal and temporal, each input being sequence of sound and images. We divide each input video into set of temporally aligned images and audio segments. Compared to the synthetic scenarios, these datasets offer the complexity and noise of real-world data, idcally allowing A2MT agents to optimize the trade-off objective in interesting ways.
In fig. we illustrate the variety in inputs for the AudioSet dataset: (a) shows an example where both modalities are informative towards the label, (b) the change in the image modality could be a signal for the agent to revisit the audio modality, (c) shows an example where the image modality is seemingly unrelated to the sound-based label, and in (d) all image frames are identical.
4

---

Published in Transactions on Machine Learning Research (06/2023)
(b)
Images
Audio
6
(d)
Images
Audio
Time
Time
Figure 3: Four example sequences from the AudioSet training set. The labels associated with these inputs are (a) Music, (b) Music, (c) Speech, and (d) ronic music. The audio signal often more informative of the label than the images for AudioSet. Inputs are downsampled in the above visualization.
We observe empirically that model predictions degrade consistently we mask out parts of the input at eval- uation time (ef. 55) Further, we find that models trained on inputs which span longer durations (increasing the temporal stride to keep the input size constant) perform better. These results suggest there temporal variety in these datasets and that additional observations improve model predictions. This supports the idea that A2MT-style selection of the right inputs is possible.
4 Perceiver 10 for A2MT
We use Perceiver IO (Jacgle et al. 2021 for both the predictive model f and the agent Perceiver TO is suitable archilecture for A2MT as it is modality-agnostic and scales well to large input sizes. Further, Perceiver 10 gracefully handles missing data: mask tokens are used to signal where features are missing in the input, cf. Jaegle et al. (2021). For the model f, we make no changes to the standard Perceiver TO architecture. For the agent т, we condition on previous actions by appending them to the decoder input of the Perceiver. We propose two variants of our approach: Inputs for the synthetic datasets tend to be somewhat smaller, and we can afford a computationally more involved routine. In contrast, for the real world datasets, we use a computationally leaner sctup that allows for sequential application of the Perceiver and training of the agent at scale. We give full details in C
4.1 Variant 1: Small Data Regime
For the synthetic scenarios, the agent π and the model f are two fully separate Perceiver 10 models that do not share any parameters. They are trained jointly: the agent acquires the input data features for the classifier. For policy training, we can make use of the straight-through Gumbel-Softmax estimator (Jang et al. 2017) due to the simple unmasking effect of actions in A2MT. The Gumbel trick (Jang et al. 2017 Maddison et al. 2017) allows for backpropagation through the discrete action variables and is generally considered low-variance estimator in comparison to alternatives such as REINFORCE (Williams 1992
Concretely, joint training directly follows algorithm I for each input in a batch of training samples: we iteratively apply the agent म, unmasking modalities as given by the sampled actions a. We then predict with the model f given the partially observed input sequence #1.T. We can compute Monte Carlo sample of the reward, eq. [1], given the observed prediction loss L(f( y) and cost of acquisition C(a), where L is log- li-likklihood loss. Due to the Gumbel parameterization of the actions, can directly apply gradient- based optimization to the agent parameters O. We simultaneously train the model f by minimizing the loss L. in the objective with respect to the parameters of f.
4.2 Variant 2: Large Data Regime
We apply the Perceiver-based agent repeatedly during A2MT, i.c up to T 25 times in the scenarios that we study. This increases computational cost, particularly for gradient computation during training. To
5

---

Published in Transactions on Machine Learning Research (06/2023)
save compute for the large scale imputs of real-world datasets, share Perceiver 10 encoders between the predictive model ʃ and agent x; unlike in the small data regime, ʃ and are no longer fully separate models. We further pre-train the encoder and the decoder of f. In other words, in the large data regime, the classifier f is no longer trained on inputs acquired by the policy æ. Lastly, when training the agent decoder parameters, we fix both the (shared) encoder parameters as well as the decoder parameters of
To ensure that the classifier and, in particular, the shared encoder is suitable for inputs encountered later during agent training, we propose to use a particular masking setup for the imputs during pre-training of the classifier. Concretely, for each imput, (M1) we drop modalities at timestep by sampling masks with fixed per-modality probability, Om, ~ Bernoull((pm): (M2) we randomly draw mmax ~ Unif(O, T) and mask out all inputs at & > max, iee. am,t = 0 VI > 1max; (M3) we randomly drop entire modalities from the inputs with fixed per-modality rates am,1 = dm Vt, with dm ~ Bernoull((p:)) The masking mechanisms (M1-M3) are applied only during pre-training, and we use the same fixed values for pm and dm in all experiments, see $C.4 for further details.
This masking procedure exposes the encoder to sparse input distributions during pre-training that are equiv- alent to those created by randomly acting agent. This leads to versatile encoder representations that are useful during agent training. In addition to helping agent training, we observe that the masking procedures affect the test. set performance of model predictions positively, and we observe the highest performance when all methods (M1-M9) are applied simultaneously (cf. $5). This fits with similar results on how Transformer- based architectures benefit from masking at training time (Devlin et al. 2018
Dosovitskiy
et al. 2021 Feichtenhofer al.) 2022).
Following related work in active feature acquisition, such as Li & Oliva 2021 ; we additionally condition the agent directly on model predictions (( at each timestep l. Concretely, we concatenate both pre- dicted class probabilities, (y(((11.), as well as associated predictive entropies, Epy(s/e..)l-oogg Py (v @1:)). to the latent representation of the Perceiver IO agent. While this incurs additional computational cost from predicting with the model at each timestep, we believe the additional information will be useful in inform- ing agent behavior. For example, this allows the agent to be aware the uncertainty altached to model predictions at each timestep. This way, the agent could, for example, learn to stop acquiring when model predictions sufficiently confident.
Inspired by reward shaping Ng et al. 1999 Li & Oliva 2021,, we optionally add an intermediate reward term to the objective eq. ! 1=-a (C(f(t1e),)) -C((((re-)) where a is a hyperparameter, and y is the discount factor. This term directly incentivizes the agent to decrease the predictive loss of f. which helps with credit, assignment, as otherwise hard for the agent to learn which particular acquisition actually led to loss reduction.
The Gumbel-Softmax estimator becomes prohibitively expensive in the large-scale scenario, as it requires full backpropagation through repeated application of both f and We therefore rely on the advantage actor critic (A2C) (Sutton & Barto 2018) policy gradient method, which approximates gradients of eq. (1) with respect to the agent parameters via Monte Carlo samples of the score function estimator. A2C uses an additional baseline model that reduces variance during optimization, and we implement this baseline as a separate Perceiver 10 decoder head.
5
Experiments
We next give experimental results on synthotic and real data and refer to C for further details.
5.1 Synthetic Scenario
We begin by exploring the performance of our small-scale variant on the synthetic datasets. We use sequences of length T= 10 and three distinct values (0, 1, 1,2) per modality such that the input shape is (T, Rdd)) = (10,3) per modality when using a one-hot encoding for the values.
6

---

Published in Transactions on Machine Learning Research (06/2023)
Input Sequeace 1
I
Iaput Sequence 2
Input Sequence 3
Digit
[2 0
Q
2
2
2
0
1]
[2 1
2
2
0
1
1
1]
[1 2
1
1 2 0
1
0
1]
Actions Digit
[o
1
0
1
0
01
[0
1
1
0
0
1
01
[0
1
0
0
1
0
Counter
[1
0
2
0
12
0
[1
2
Actiona Counter
[1
1
1
1
D
1
1
0]
[0
1
1
1
0
1
0
1
0]
[1
1
1
1
D
1
01
Label
True
Pred.:
True
4
Pred.:
True
Pred.:
Figure 4: Acquisition behavior of the Perceiver 10 agent on simple synthetic scenario. 'Digit' and 'Counter' give ground truth values for the Counter and Digit input modalities. 'Actions Digit' and 'Actions Counter' mark when the agent did (1) or did not acquire (0) for each of the modalities. The agent successfully learns a sparse acquisition strategy: it (almost always) acquires the Digit modality only if the Counter modality is 0, and further learns to skip some acquisitions the Counter modality.
For raw digits as inputs our agent learns to acquire an av- erage of 46.2% of the digit modality, 68.8% of the counter modality, and achieves an accuracy of 92.4% on the test set. Clearly, the agent learns a selective acquisition procedure for both modalities without sacrificing predictive accuracy. We further compute the optimal acquisition rates for the synthetic scenario as 27.2% for the digit and 39.3% for the counter modality. These highlight that, particular for the counter modality, there is a gap relative to optimal behavior for our agent, cf. table [ In $ we further compare the agent's acquisition strategy optimal behavior. We display examples of learned agent behavior individual test set inputs in fig. For the digit modality, the agent mostly follows the ideal strategy and acquires only whenever the counter is zero. For the counter modality, the agent learns to skip acquisitions at some of the non-informative timesteps. In fig. A.I we display training dynamics: the agent initially shows high acquisition rates for both modalities, which quickly leads to increases in model predictive accuracy. Then, the agent learns to discard irrelevant parts of the input; this happens more quickly for the digit than the counter modality.
Table 1: Results on the synthetic task.
Metric
Agent
Oracle
Label Prediction Accuracy
92.4%
100.0%
Digit Acquisition Rate
46.2%
37.2%
Counter Acquisition Rate
68.8%
39.3%
Figure A.2 shows training curves for early results on the image and image/audio versions of the synthetic scenario. The agents overfit to the training set and their behavior does not generalize. We suspect that our method struggles to learn acquisition strategies and representations simultaneously.
5.2 Audio-Visual Datasets
Next, we investigate the performance of the large scale variant of our approach on the AudioSet and Kinetics datasets. We split the training set of each dataset into a subset used for model pre- training and a subset used exclusively for agent training, taking up 80% and 20% of the original training set respectively. Note again that model and agent share the encoder, which learned dur- ing model pre-training, and then fixed during agent training. For both datasots, cach input sam- ple consists of audio-visual input with 250 frames of images and a raw audio signal spanning 10 seconds. For images, take every 10th frame as input, obtaining total of 25 input frames. We pre-process the audio signal to mel spectrograms, and then divide the signal into 25 segments. This leads to an input shape of (25, 40 128) for the audio modality, and (25, 200 200 33) for the image modality, where we collapse additional dimensions beyond time into the last axis as described in [2 We first discuss insights from pro-training before moving on to results for A2MT agents.
Table 2: Masking at training time proposed by (M1-M3) improves per- formance on the (unmasked, fully ob- served) AudioSet lest set.
5.2.1 Model Pretraining
Variant
mAP
No Masking
0.178
Masking Variants. Table 2 gives results of different pre-training variants for Perceiver 10 on AudioSet. We observe that masking sig-
(M1) Random Masking
0.230
+ (M2) Max. Timestep
0.262
nificantly improves mcan average precision (mAP: higher is better)
+ (M3) Modality Dropping
0.344
on the AudioSet validation set, going from 0.178 mAP without any
+ Couv. Downsampling
0.370
masking to 0.344 mAP for the complete masking setup (MI-M3)
7

---

Published in Transactions on Machine Learning Research (06/2023)
Table Impact of dropping entire modalities at evaluation time on the AudioSet and Kineties test sets. For both datasets, performance is best when no modalities are dropped (fully observed). For Au- dioSet, audio is more informative than images, as 'audio only' performance trumps 'image only'; for Kinetics, images are more informative.
Table 4: Gradually masking out the more informa- live modality (audio for AudioSet, images for Ki- neties) at evaluation time. Performance degrades the 'mask rate', the fraction of randomly selected timesteps at which we mask inputs, increases. The weak modality (images for AudioSet, audio for Ki- netics) is not masked here.
Variant
AudioSet (mAP)
Kinetics (top-1)
Mask Rate
AudioSet (mAP)
Kineties (top-1)
Fully Observed
0.370
0.396
0.0
0.370
0.396
Audio Only
0.302
0.087
0.3
0.369
0.397
Image Only
0.137
0.305
0.8
0.331
0.365
0.9
0.292
0.316
1.0
0.137
0.087
(cf. 44.2 We observe that cach the masking procedures (M1-M3) progressively improves evaluation metrics. See $C.4 for details on masking rates. We further add a single strided convolutional pre-processing layer per modality, which further improves performance. Note that, here, we evaluate the model on fully observed data. For A2MT we are particularly interested in the predictive performance of f for sparse data.
Sparsity at Evaluation Time. In order for A2MT agent training to be successful, the model f needs to be capable of extracting information from sparse inputs. Table 3 reports model performance when dropping entire modalitics during evaluation. We observe that the model relies on both modalities for prediction, as the performance on fully observed data is best. As expected, the audio modality is more informative for AudioSet and images are stronger for Kinetics. Table | shows how predictions degrade when increasing the masking rate across time for the stronger modality. Predictions are best on fully observed data and deteriorate significantly 90% missing inputs. Evidently, the model learns to make use of additional data in the input at low masking rates while also predicling reasonably for sparse inpuls.
In summary, the proposed masking routines (M1-M3) help learn models which use all modalities in the input and degrade as inputs bocome increasingly sparse. These results suggest that our pre-trained Perceiver IO models are good candidates for use in A2MT, which we investigate next.
5.2.2 Agent Training
Agents React to Cost. We follow the setup described in $4.2 and train agents using A2C without intermodiate rewards. We set a nonzero acquisition cost only for the more informative modality of cach dataset. Table 5 shows agent. behavior for acquisition costs on the audio modality of the AudioSet dataset and table gives results for image cost on Kinetics. We observe that our agents generally react to increased acquisition costs by decrcasing the number of acquisitions, lcarning how many acquisitions are 'worth it' for given cost. Further, we find that agents hold on to single acquisition (1/25 imputs frames, i.e. acquisition rate 0.04) even at relatively high costs. This makes sense, as performance deteriorates drastically when all of the informative modality is dropped. Conversely, agents readily learn to not acquire large portions of the informative imput modality, as we have observed this has only a small effect on predictive scores. For the Kinetics dataset, the agents discover that an acquisition rate of about 0.2 optimizes the objective across a varicty of low to medium costs. In we discuss results when imposing costs on the weak modality.
Random Ablations. We are ultimately interested in learning agents which display adaptive behavior that meaningfully adjusts to the information in each input. Therefore, we compare the performance of our agents against two ablations that we call 'random-ratc' and 'random-lhot: For both, we first compute the average acquisition rate of the agent per modality, i.e. the average fraction of timesteps at which it acquires. For the 'randomrrate ablation, we acquire modalities with a fixed Bernoulli probability per timestep equal to the average acquisition rate of the agent. Additionally, we construct the 'random-Ihot ablation by acquiring at a fixed number of timesteps per modality that are equidistantly spread across the sequence. The number of acquisitions chosen such that we match the average number of agent acquisitions per modality. (Usually
8

---

Published in Transactions on Machine Learning Research (06/2023)
Table 5: The agent acquisition rate appropriately reduces for the audio modality on the AudioSet test set as acquisition costs increase; the agent acquisition rate the average fraction of timesteps at which the agent acquires. We impose costs only on the audio modality and compare against two ablations: (a) Random-Rate which acquires at each timestep with fixed probability matching the agent acquisition rate, and (b) Random- HHot, which acquires at a fixed set of equidistant timesteps matching the acquisition rate of the agent. The agent outperforms the rate-hatched ablation at high costs, but does not improve over the discrete 1-hot ablation. See the main text for further discussion. Standard deviations are 5-fold repetitions on the test set.
Cost per Audio Acquisition
1 x 10-6
1 x 10-5
x 10-4
2.5 x 10-1
1 x 10-3
Agent Acquisition Rate
0.98
0.83
0.23
0.11
0.04
Agent (mAP)
0.3724 + 0.0010
0.3697 + 0.0025
0.3359 + 0.0117
0.3116 + 0.0108
0.2580 + 0.0010
Raudom-Ratte (mAP)
0.3728 + 0.0005
0.3719
0.0007
0.3365 + 0.0016
0.2946 1 0.0014
0.2295 + 0.0009
Raudom- IHot (maPP
0.3732 + 0.0001
0.3731 + 0.0002
0.3472 + 0.0003
0.3165 1 0.0006
0.2660 0.0004
Table 6: The agent acquisition rates generally reduce as costs increase for the image modality of the Kinetics test set; the agent acquisition rate is the average fraction of timesteps at which the agent acquires. We do not impose any costs for acquisitions of the less-informative audio modality. We report standard deviations over 5-ld repeated application on the test set. See main text or table for details.
Cost per Image Acquisition
1 x 10-3
5 x 10-3
1 x 10-2
x 10-1
5 x 10-1
Agent Acquisition Rate
0.19
0.22
0.21
0.09
0..4
Agent (top1)
0.3370
0.0009
0.3336 + 0.0034
0.3326 + 0.0016
0.3309 + 0.0011
0.2722 + 0.0005
Hanndom-Rater (top1)
0.2804 - 0.0019
0.2856
0.0019
0.2830 + 0.0006
0.2378 + 0.0012
0.1721 + 0.0014
Random- LHot (top1)
0.3038 = 0.0003
0.3067
0.0003
0.3063 + 0.0002
0.2957 + 0.0007
0,2089
0.0002
(a) Agent
(b) Random-Rate
(c) Random-1Hot
Acquisition Rate
1.0
0.0
10
15
20
10
15
10
15
Timestep
Timestep
Timestep
Figure 5: Comparing learned acquisition patterns of an agent on AudioSet to the patterns of the random ablations. Our agent learns a set of fixed timesteps for which it always acquires, similar to the random- baseline. In (a) and (c), acquisition rates are close to zero and too small to be visible for some timesteps.
the number of agent acquisitions is not an integer, and so we remove some acquisition probability for the last of the fixed timesteps.) See fig. 5 for an illustration of the ablations.
These are ablations rather than baselines, as they use the per-modality acquisition rates found by the agent, and thus incur the same cost as the agent. However, potentially unlike the agent, the ablations do not act adaptively: they acquire with the same fixed probabilities for each sequence. If we find that our agent can consistently outperform both ablations, this is supporting evidence for adaptive behavior the agent, adjusting its acquisitions to the information in cach input.
For AudioSet, the agent does not consistently outperform either ablation at low acquisition costs. However, the agent does tend to outperform the random-rate, but not the random-lhed -hot, ablation at medium-to to-hh acquisition costs. Figure 5 displays the average temporal acquisition pattern of the agent and ablations for the audio modality of AudioSet at high acquisition costs. The agent learns a discrele pattern of acquisitions across timesteps, similar to the random- maatom-lhot baseline. This is advantageous in comparison to the fixed low rates of the random-rate bascline: due to the small value of the acquisition probabilitics, the resulting Binomial distribution over acquisitions has significant mass at acquisitions. Therefore, the random-rate bascline sometimes does not acquire anything at all, which has a large negative effect its average performance.
9

---

Published in Transactions on Machine Learning Research (06/2023)
It is encouraging that the agents learn discrete acquisition behavior, avoiding the drawbacks of fixed small acquisition rates. However, this also shows that the agents do not learn individualized predictions, and instead acquire at fixed timesteps, ignoring individual inputs for decision making. There is almost no variance in acquisition behavior for different test samples. For scenarios with higher Icarned acquisition rates- where the variance of lixed-rate acquisition is less disadvantageous we find some agents do not learn 1-hot style acquisition patterns and instead behave more similarly to random-rate.
For Kinetics, we observe that our agents are able to outperform the random-rate and sandom-lhot ablations. However, instead of indi- vidualized bchavior, figs. 0 and A.3 show that the agents learn a static pattern where they never acquire both modalities simultaneously. While this is interesting behavior that optimizes the objective better than the random ablations (presumably because at least one modality is always acquired), falls short of our goal of adaptive acquisition.
Audio Rate Image
Intermediate Rewards. We investigate the effect of intermediate rewards agent behavior. Figure 7 (a) displays 100 random samples of agent behavior from the AudioSet test set when using intermediate rewards. Intermediate rewards lead to agents that show variable ac- quisition behavior between samples. also seems that intermediate reward which encourages the agent to reduce predictive loss with each action leads to greedy behavior in the agents, where acquisitions at earlier timesteps are preferred. Further, fig. (b) shows the number of acquisitions per sample is now correlated to the entropy, which expresses certainty or uncertainty of the predictions. However, fig. (c) shows that model entropy and loss practically uncorrelated here. (We expand on this in our discussion in 7 Comparing to the random ablations, we find that, while our AudioSet agents do learn adaptive behavior, their performance can still be matched by the non-adaptive ablations, cf. table A.5 For Kinetics, we observe similar behavior (excopt that we outperform the random ablations with the same caveat as above), cf. table A.6
0
10
20
Timesteps
Figure 6: Acquisition patterns on Ki-
netics at 0.001 cost per image.
6 Related Work
Additional Applications of A2MT. A2MT-like problems can be found in a variety of application domains. For example in wearable devices, the energy cost of sensor activation is significant (Possas et al., 2018 and a policy that roduces sensor usage without sacrificing prodictive accuracy is desirable. Literature in AFA further mentions compuler security or fraud detection as areas of application. Given that both these areas naturally may have temporal or multimodal components, these applications transfer to A2MT well. Lastly, the acquisition procedure in A2MT can be a way to reduce input size and thus computational cost of predictions.
Active Perception. Active perception (Bajosy 1988 Aloimonos el al. 1988 Bajesy et al., 2018) models
the cost associated with visual attention in the context of embodied agents. In the language of A2MT: we cannot observe all visual input features at once, we need to decide where to look actively and iteratively. For
(a) Agent Samples for Audio
1e-2
(b) Correlation: 0.36
1e-2
(c) Correlation: -0.15
Acquisiion Rate
1.5
7
Entropy
1.0
0
20
0.2
0.4
0.6
0.5
1.5
Timestep
Acquisition Rate
Entropy
1e-2
Figure 7: Behavior for agents trained with intermediate reward on 100 random samples from the AudioSet test set. (a) The acquisition behavior displays significant variance, and agents prefer acquisition at earlier timesteps. (Samples in grey, mean j and standard deviation a in blue.) (b) The per-sample acquisition rates are correlated to the entropy of model predictions. (c) Correlation between entropy and predictive loss is poor.
10

---

Published in Transactions on Machine Learning Research (06/2023)
example, Mnih et al. (2014) Haque et al. (2016); Jayaraman & Grauman 2018 solve visual classification or reconstruction problems by learning agents that iteratively attend to parts of the input. Relatedly, active sensing, e.g. SStsangi et al. (2018); Hero & Cochran (2011), studies how to actively select a subset of sensors
to observe to reduce uncortainty over latent variables. Satsangi et al. 2020 connect model uncertainty and
prediction reward in active perception.
Efficient Video Classification. Related work in efficient video classification has sought to reduce the computational cost of video classification by selecting small set of salient frames for each input YYung
et al.,
2016 Wu et al. 2019b; Korbar et al., 2019; Wu et al. 2019a Gao et al.) 2020 Zheng et al.. 2020;
Wang et al. 2021 Ghodrati et al.; 2021] Panda et al. 2021 Gowda et al. 2021 Yang et al.
2022 For
computationally cheap policies, these methods can achieve computational cost-savings compared to methods
that rely on all timesteps as input. This is related to so-called anytime prediction approaches (Grubb &
Bagnell 2012 Zilberstein 1996, Horvitz 1987) which explicitly trade off computation cost and predictive accuracy. In contrast to all of the above, A2MT assumes that there a real-world cost associated with the acquisilion of the input modalities, e-g. the cost. of performing an MRI scan, and we can completely neglect any computational costs. Also, A2MT requires temporally causal acquisition, which is something not respected by most efficient video classification approaches. These difference in motivation lead to methods which (in all cases we know of) are not applicable to the A2MT scenario.
7 Discussion
While our agents did react sensibly to acquisition cost on the complex audio-visual datasets in $5.2.2 they learned static behavior and did not adjust acquisitions sensibly to individual sequence. In this section, we offer a discussion of possible reasons for this negative result: Figure (c) suggest that our models may not be suitable for learning adaptive behavior. We would expect the uncertainty (entropy) of the predictions to correlate with the predictive loss. Then, entropy could be a useful signal to guide agent behavior: for samples where the model is certain about the prediction (low entropy) the agent can stop acquisitions early; when the model is uncertain, the agent may acquire more reduce uncertainty and therefore loss. We do not see this correlation in fig. (c), and so it may be hard for the agent to learn a policy that optimizes the reward, which depends on the pro-trained predictive model.
One possible explanation for the lack of correlation is that our Perceiver 1O models currently underfit the AudioSet and Kinetics dataset. However, as we are not aware of prior work studying the entropies of Perceiver IO predictions, we cannot exclude the possibility that the entropics of Perceiver IO generally do not conform expectations. Therefore, both training the initial model longer and additional model archilectures may be worth exploring. Lastly, future work should not exclude the possibility that subtle distribution shift in masking distributions between pre-training and agent training further inhibits policy learning.
Alternatively, AudioSet and Kinetics may be ill-suited for application to A2MT: while we have found that there is some signal diversity in AudioSet and Kinetics, different sections of a given clip often look similar, presumably making it difficult for the agents to learn adaptive behavior. Subsequent work in A2MT could consider datasets with longer clip durations and more content diversity per sequence, e.g. ActivityNet (Fabian) Caba Heilbron & Niebles, 2015). Lastly, there are interesting variations of the A2MT setup that future work could explore, e.g. other reward formulations such as fixed budget per sample a global budget across samples, acquisition costs that change with time, actions that affect state evolution.
8 Conclusion
We have introduced the task of active acqusstiion for multimodal temporal data (A2MT), extending prior work in active feature acquisition to multimodal and temporal inputs. We have further proposed a Per- cciver basbd reinforcement lcarning approach to tackle A2MT problems. On novel synthetic scenarios our agents successfully learn to use cross-modal reasoning to optimize the trade-off between feature acquisi- tion cost and predictive accuracy. We further adapt Kinetics and AudioSet, two large scale video classifica- tion datasots, to application to A2MT: here, the agents appropriately react to modality-specific acquisition costs. However, ablations reveal they do not adapt acquisitions to individual inputs. We believe A2MT is a challenging and practically relevant task, and we would be excited for the community to join our efforts.
11

---

Published in Transactions on Machine Learning Research (06/2023)
Broader Impact Statement
Our paper proposes a sequential decision making task, as well as a method to approach this task. We believe that this method can have useful societal and economic impact in domains such as robolics, finance, and healtheare. However, one of the main limitations of this paper that we have focused on synthotic data scenarios. Although this synthetic data is motivated by real-world scenarios, we do not recommend direct deployment of our method to practical scenarios. Work on automated decision making should always be carried out in close collaboration with domain experts, while proactively taking into account safety and ethical considerations.
Acknowledgments
We thank Yujia Li, Timothy Lillicrap, Andrew Brock, Adrià Recasens, João Carreira, Lucas Smaira, Isabela Albuquerque, Joost van Amersfoort, Ali Eslami, our action editor, Martha White, and the anonymous reviewers for helpful leedback and interesting discussions that have led to numerous improvements of the paper.
References
John Aloimonos, Isaac Weiss, and Amit Bandyopadhyay. Active vision. International journal of computer vision, 1(4):333 356, 1988.
Ruzena Bajesy. Active perception. Proceedings of the IEEE, 76(8))966-100 -0099666-005,,, 1988.
Ruzena Bajesy, Yiannis Aloimonos, and John K Tsotsos. Revisiting active perception. Autonomous Robots, 2018.
Kathryn Chaloner and Isabella Verdinelli. Bayesian experimental design: A review. Statistical Science, pp. 273-304, 1995.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. 1rXiv:1810.04580 2018.
Andres Diaz-Pinto, Nishant Ravikumar, Rahman Attar, Avan Suinesiaputra, Yitian Zhao, Eylem Levelt, Erica Dall' Armellina, Marco Lorenzi, Qingyu Chen, Tiarnan DL Keenan, al. Predicting myocardial infarction through retinal scans and minimal personal information. Nature Machine Intelligence, 4(1): 55-61, 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Rep- resentations, 2021.
Bernard Ghanem Fabian Caba Heilbron, Victor Escorcia and Juan Carlos Niebles. Activitynet: large- scale video benchmark for human activity understanding. In Conference on Computer Vision and Pattern Recognition, pp. 961-976101111111991 2015.
Christoph Feichtenhofer, Haoqi Fan, Yanghao Li, and Kaiming He. Masked autoencoders spatiotemporal learners. 2rX 2022.
Ruohan Gao, Tae- Hyun Oh, Kristen Grauman, and Lorenzo Torresani. Listen to look: Action recognition by previewing audio. In Conference on Computer Vision and Pallern Recognition, pp. 10457-10467, 2020.
Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled datasct for audio events. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 776-780. TEEF, 2017.
12

---

Published in Transactions on Machine Learning Research (06/2023)
Amir Ghodrati, Babak Ehteshami Befnordi,, and Amirhossein Habibian. Framecxit: Conditional early exiting for efficient video recognition. In Conference Computer Vision and Poltern Recognation, pp. 15608- 15618, 2021.
Shreyank N Gowda, Marcus Rohrbach, and Laura Sevilla-Lara, Smart frame selection for action recognition. In AAAI Conference on Artificial Intelligence, volume 35, pp. 1451-1459, 2021.
Russell Greiner, Adam J Grove, and Dan Roth. Learning cost-sensitive active classifiers. Artificial Intelli- gence, 12911))137 174, 2002.
Alex Grubb and Drew Bagnell. Speedboost: Anytime prediction with uniform near-optimality. In Artificial Intelligence and Statistics, 2012.
Albert Haque, Alexandre Alahi, and Li Fei-Fei. Recurrent attention models for depth-based person identifi- cation. In Conference on Computer Vision and Pattern Recognition, 2016.
Alfred O Hero and Douglas Cochran. Sensor management: Past, present, and future. Sensors Journal, 2011.
Eric Horvitz. Reasoning about beliefs and actions under computational resource constraints. In Conference on Uncertainty in Artificial Intelligence, 1987.
Stephanie T, Hyland, Martin Faltys, Matthias Hüser, Xinrui Lyu, Thomas Gumbsch, Cristóbal Esteban, Christian Bock, Max Horn, Michael Moor, Bastian Rieck, et al. Early prediction of circulatory failure in the intensive care unit using machine learning. Nature medicine, 26(3):364 373, 2020.
Z. Jackson. free spoken- digein-ddiiit-tttaae 2017.
Free spoken digit dataset (Isdd).
Andrew Jaegle, Sebastian Borgeaud, Jean- Caan-Baptiste Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, et al. Perceiver io: A general architecture for structured inputs & outputs. In International conference on machine learning, 2021.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. In Interna- lional Conference on Learning Representations, 2017.
Jaromír Janisch, Tomáš Pevný, and Viliam Lisy. Classification with costly features using deep reinforcement learning. In AAAI Conference on Artificial Intelligence, volume 33, pp. 3966, 2019.
Jaromír Janisch, Tomáš Pevuý, and Viliam Lisý. Classification with costly features as a sequential decision- making problem. Machine Learning, 10915)1157711615 2020.
Dinesh Jayaraman and Kristen Grauman. Learning to look around: Intelligently exploring unseen envi- ronments for unknown tasks. Conference on computer vision and pattern recognition, pp. 1238 1247, 2018.
Alistair Johnson, Lucas Bulgarelli, Tom Pollard, Steven Horng, Leo Authony Celi, and Roger Mark. Mimic-iv (version 2.0). PhysioNet, 2020.
Mohammad Kachuee, Orpaz Goldstein, Kimmo Karkkainen, Sajad Darabi, and Majid Sarrafzadeh. Op- portunistic learning: Budgeted cost-sensitive learning from data streams. In International Conference Learning Representations, 2019.
Bruno Korbar, Du Tran, and Lorenzo Torresani. Sesampler: Sampling salient clips from video for efficient 2019.
action recognition. In International Conference on Computer Vision, pp.
Yann LeCun. The mnist database of handwritten digits.
1998.
Sarah Lewis, Tatiana Matejovicova, Yingzhen Li, Angus Lamb, Yordan Zaykov, Miltiadis Allamanis, and Cheng Zhang, Accurate imputation and efficient data acquisitionwith transformer-based vaes. In NeurIPS 2021 Workshop Deep Generalive Models and Downstream Applications, 2021.
13

---

Published in Transactions on Machine Learning Research (06/2023)
Yang Li and Junier Oliva. Active feature acquisition with generative surrogate models. In International Conference on Machine Learning, pp. 8450-6459 PMT.R, 2021.
Dennis Lindley. On a measure of the information provided by an experiment. The Annals of Mathematical Statistics, 27(4):986-1000) 1956.
Charles X Ling, Qiang Yang, Jianning Wang, and Shichao Zhang. Decision trees with minimal costs. International conference on Machine learning, pp. 69, 2004.
Lance A Liotta, Mauro Ferrari, and Emanuel Petricoin. Clinical proteomics: written in blood. Nature, 425 (6961):905-9055 2003.
Chao Ma, Sebastian Tschiatschek, Konstantina Palla, José Miguel Hernández-Lobato Sebastian Nowozin, and Cheng Zhang. Eddi: Efficient dynamic discovery of high-value information with partial vac.
arXiv:1809.114 2018.
Chris J Maddison, Andriy Mnih, and Yee Whye Tch. The concrete distribution: A continuous relaxation of discrete random variables. In International Conference Learning Represenlations, 2017.
Prem Melville, Maytal Saar-Tsechane Tsechansky, Foster Provost, and Raymond Mooney, Active Peaturevvlute acqui- sition for classifier induction. In Fourth IEFE International Conference on Data Mining (CCDM'04), pp. 483 486. IEEE, 2004.
Volodymyr Mnih, Nicolas Heess, Alex Graves, et al. Recurrent models of visual attention. Advances in neural information processing systems, 27, 2014.
Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In Ieml, volume 99, pp. 278 287, 1999.
Rameswar Panda, Chun-Fu Richard Chen, Quanfu Fan, Ximeng Sun, Kate Sacnko, Aude Oliva, and Rogerio Feris. Adamml: Adaptive multi-modal learning for efficient video recognition. In International Conference on Computer Vision, pp. 7576-7585, 2021.
Rafael Possas, Sheila Pinto Caceres, and Fabio Ramos. Egocentric activity recognition on budget. In Conference on Computer Vision and Pattern Recognition, pp. 5967-5976, 2018.
Maytal Saar- Tsechassyh,,,,,, Prem Melville, and Foster Provost. Active feature-value acquisition. Monagement Science, 55(4):664-684, 2009.
Yash Satsangi, Shimon Whiteson, Frans Oliehoek, and Matthijs TJ Spaan. Exploiting submodular value functions for scaling up active perception. Autonomous Robots, 2018.
Yash Satsangi, Sungsu Lim, Shimon Whiteson, Frans Oliehoek, and Martha White. Maximizing information gain in partially observable environments via prodiction reward. International Conference on Autonomous Agents and Mulliagent Systems, 2020.
Paola Sebastiani and Henry P Wymn. Maximum entropy sampling and optimal bayesian experimental design.
Journal of the Royal Statistical Sociely: Series B (Statistical Methodology), (2(1):144511) 157, 2000.
Burr Settles. Active Learning Literature Survey. Machine Leorning, 2010.
Victor Sheng and Charles X Ling, Feature value acquisition testing: sequential batch test algorithm. In International conference on Machine learning, pp. 109-816, 2006.
Hajin Shim, Sung Ju Hwang, and Eunho Yang. Joint active feature acquisition and classification with variable-size set encoding. Advances in neural information processing systems, 31, 2018.
Lucas Smaira, João Carreira, Eric Noland, Ellen Clancy, Amy Wu, and Andrew Zisserman. A short note on the 70002200-7002020 human action dataset. orXiv:2010.10861, 2020.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
14

---

Published in Transactions on Machine Learning Research (06/2023)
Yulin Wang, Zhaoxi Chen, Haojun Jiang, Shiji Song, Yizeng Han, and Gao Huang. Adaptive focus for efficient video recognition. In International Conference Computer Vision, pp. 16249-16258, 2021.
Ronald Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3):229 256, 1992.
Wenhao Wu, Dongliang He, Xiao Tan, Shifeng Chen, and Shilei Wen. Multi-agent reinforcement learning based frame sampling for effective untrimmed video recognition. In International Conference on Computer Vision, pp. 1222-6231, 2019a.
Zuxuan Wu, Caiming Xiong, Chih- Yao Ma, Richard Socher, and Larry $ Davis. Adaframe: Adaptive frame selection for fast video recognition. In Conference on Compuler Vision and Pollern Recogration, pp- 1278 1287, 2019b.
Mingyu Yang, Yu Chen, and Hun-Seok Kim. Efficient deep visual and inertial odometry with adaptive visual modality selection. arXiv:2205.061 061187, 2022.
Serena Yeung, Olga Russakovsky, Greg Mori, and Li Fei-Fei. End-to-end learning of action detection from frame glimpses in vidcos. In computer vision and pattern recognition, pp. 2678-2687, 2016.
Haiyan Yin, Yingzhen Li, Sinno Jialin Pan, Cheng Zhang, and Sebastian Tschiatschek. Reinforcement
learning with efficient active feature acquisition. arXiv preprint arXiv:2011.00825, 2020.
Sara Zannone, José Miguel Hernández-Lobato, Cheng Zhang, and Konstantina Palla. Odin: Optimal dis- covery of high-value information using model-based deep reinforcement learning. In ICML Real-world Sequential Decision Making Workshop, 2019.
Yin-Dong Zheng, Zhaoyang Liu, Tong Lu, and Limin Wang. Dynamic sampling networks for efficient action recognition in videos. Transactions on Image Processing, 29:7970-7988, -983, 2020.
Shlomo Zilberstein. Using anytime algorithms in intelligent systems. Al magazine, 17(3):73-77 73, 1996.
Valentina Bayer Zubek, Thomas Glen Dietterich, et al. Pruming improves heuristic search for cost-sensitive learning. 2004.
15

---

Published in Transactions on Machine Learning Research (06/2023)
A
Additional Results
A.J we display the training dynamics for the experiments the synthetic scenario with raw inputs, fig. A.2 gives performance for the MNIST and the MNIST + Spoken Digit version of the synthetic experiment.
In fig.
In tables and A.2 we report confusion matrices comparing our agent's acquisition strategy to oracle be- havior on the raw version the synthetic data experiment. Our test set has 10 sequences of length = 10, so there are a total of 10 limesteps per modality for which we evaluate agent acquisitions. For the digit modal- ity, optimal behavior is to acquire only when the counter modality zero. Table A.l]shows the agent largely follows this optimal bchavior, acquiring in 98.6% of cases where the counter is zcro (true positives), and when the counter is not zero, the agent does not acquire in 84.8% of cases (Irue negatives). For the counter modal- ity, oracle behavior is to acquire the first element of each countdown, as all values following as well as the start of the next countdown can be inferred. Because countdowns have minimum length of 2, the counter does not need to be acquired when a new countdown starts at the last timestep. Table shows that agent behav- ior less ideal for the counter modality, acquiring 71 71.6% of starting values (true positives) but not acquiring only for 33.0% of negatives (true negatives). In other words, at timesteps where the agent needs not acquire (negatives), the agent unnecessarily acquires 67 97.0% of the time (false positives). These extra acquisitions may help the agent better discover zeros in the counter modality when it misses a countdown starting value.
Table A.3 and table A.4 give results of our agents on AudioSet and Kinetics respectively when costs are imposed on the weaker input modality. For AudioSet, the agents learn lo acquire R 1/25 image per sequence across a magnitude of acquisition costs. Surprisingly, the performance at 1 image frame is only about 1 p.p. mAP lower than the performance on fully obscrved data, which explains why makes sense for the agent to quickly drop lo such low acquisition rate on the image modality. At high costs of 0.01 per acquisition, the agent longer acquires any images, which finally does hurt mAP by about 4 p.p. For Kinetics, we observe that the number of acquisitions for the weaker audio modality drops steadily cost is increased, and, correspondingly, so does the topl-accuracy. This indicates that, for Kinetics, the model makes better use of the weaker audio modality.
Figure A.3 shows the agent never acquires both modalities on the Kinetics dataset across a variety of costs.
Table A.5 and table A.6 give agent behavior with intermediate rewards enabled for AudioSet and Kinetics. They do not lead to significantly improved agent performance relative to the non-adaptive ablations.
B Synthetic Sequence Generation
Algorithm B.1 gives Python code for generating the sequences for the synthetic dataset (cf. $3.1
Algorithm B.1 Generate Synthetic Sequence
def drax_sequene (langth, digit_lox, digit_ counter_ 1ov, counter, counter_bigh) high):
digits, counter, iaportant_digita O. O), []
while (digit))s) length:
cou_t randon. randint (counter pooo 1, counter_ 1)
counter_i np. arange (nax_count, counter_ counter. axtend (counter_s)
digite_i = up. randon. Mellon.readdia randiat (digit_ _ow,, digit_higa 1, len (ccouater_)) if len (digittts__ len (digits) length:
important_dee petprrttt__dggitet
(digita_i [-1])
digits digits [
counter counter [:length]
label - sun sun (saportant_ddggit)
return digits, counter, label
16

---

Published in Transactions on Machine Learning Research (06/2023)
Table A.1: Confusion matrix for agent acquisitions of the digit modality compared to oracle behavior on the test set of the raw syuthetic scenario.
Table A.2: Confusion matrix for agent acquisitions of the counter modality compared oracle behavior on the test set of the raw syuthetic scenario.
Oracle
Oracle
False
True
False
True
Falsc
53247
9575
Falsc True
19987
40665
Agent
Agent
True
528
36650
11171
28177
-Tain Reward
0.6
Train Accuracy
1.0
Mww M
My
Test Accuracy
Acquisitions Digit
Acquisitions Counter
0.4
0.9
0.5
0.5
0.5
0.2
0.8
0.0
0
0
2
0
2
0
2
2
1e5
1e5
1e5
1e5
1e5
Training Step
Training Step
Training Step
Training Step
Training Step
Figure A.1: Performance of the Perceiver 1O-basod model and agent when applied the raw-digit version of the synthetic dataset. The model learns quickly solve the task, and the agent slowly reduces the number of acquired datapoints. Note that the 'Aquuisitions' plots give acquisition rates averaged across time, s.t. 'I' corresponds to the entire modality being acquired.
C Experiment Details
C.1 Synthetic Experiments: Raw Inputs
C.1.1 Dataset
We generate synthetic dataset with training set of size 50 x 103 and test set of size 10 x 10a. For the 'raw' version, the input modalities have shape (10, 3), i.e. we use sequences of length 10 and there are three distinct values (2, 1) per modality. We set the acquisition cost to 0.0005 per modality and timestep.
C.1.2 Architecture
We train using a batch size 256. We use the ADAM optimizer with initial learning rate 3 10-4, weight decay of 1 10-6, and cosine anncaling schodule. For the Perceiver IO encoder we use single cross-attend block with self-attention operations per Perceiver I0 block; we use 128 queries, and the hidden dimension is 128. For the Perceiver TO decoder, for total of 2 10 steps. We set the discount factor to =
single head with 128 queries and hidden dim of 128. We train
C.1.3 Choosing Acquisition Costs
In this section, we detail how we arrived at selection of acquisition costs for the results in $5.222 For both AudioSet and Kinetics, we found valid cost ranges through experinentation. As rule of thumb, we aimed for acquisition costs such that T times the cost is smaller than the observed predictive loss on the fully unmasked data. If this were not the case, is too attractive to not acquire any input samples. Note that predictive losses between AudioSet and Kinetics are significantly different; AudioSet multi-class (per instance) problem and Kinetics is not. As costs are directly traded off with predictive loss in our objective, ef. eq. (1) this explains the different cost magnitudes bet the datasets.
17

---

Published in Transactions on Machine Learning Research (06/2023)
(8)
2.0
1.0
1.0
1.0
Acquistions Counter
1.0
TTeain .Teinn Reward
Accuracy
Acquietions Digit
1.5
0.5
0.5
0.5
0.5
1.0
Test
0.0
0.0
1e6
1e6
1e6
1e6
1e6
Training Step
Training Step
Training Step
Training Step
Training Step
1.0
1.0
1.0
1.0
p-10
-Tran Reward
1.5
Accuracy
Test Accurany
Acqusitiona Digit
Acquisitios Counter
p
0.5
0.5
0.5
1e6
1e6
Training Step
Training Step
Training Step
Training Step
Training Step
Figure A.2: Performance of the Perceiver 10 based model when applied to the (a) MNIST and (b) ALNIST/Spoket Digit version of the synthetic dataset. The agents are unable to learn prediction strategies that generalize to the test set for this scenario. The different plots show sweep over dropout probabilities in [o, 0.3, 0.5, 0.8] for the input tokens: while we do observe a regularizing effect for dropout here, does not lead to improved generalization performance. Acquisition cost is set to 0 for these runs. Note that the 'Acquisitions' plots give acquisition rates s.t. '1' corresponds the entire modality being acquired. We apply running mean smoothing with kernel size 10 for the plots of the first two columns.
Table A.3: Acquisition behavior for the agent and ablations costs increase for the image modality of the AudioSet datasct. Standard deviations over 5 repctitions over the test set.
Cost Image
0.00001
0.00010
0.00100
Agent Acquisition Rate Agent (mAP)
0.07
0.05
0.01
0.3683 + 0.0060
0.3627
0.0017
0.3289 + 0.0072
Random-Ratt Rate (mAP)
0.3505 + 0.0005
0.3470 + 0.0010
0.3238 + 0.0008
Random-1Hor (mAP)
0.3681 + 0.0003
0.3650 + 0.0004
0.3275 = 0.0004
Table A.4: Acquisition behavior for the agent and ablations as costs increase for the audio modality of the Kinetics dataset. Standard deviations over 5 repetitions over the test set.
Cost Audio
0.001
0.010
0.100
Agent Acquisition Rate
0.79
0.43
0.05
Agent (top1)
0.3313 + 0.0041
0.3253 + 0.0036
0.2672
0.0015
Handom-Rate (top1)
0.2825 + 0.0012
0.2840 + 0.0020
0.2524 + 0.0013
Random- Moadomm-HHtt (topl)
0.3063 + 0.0003
0.2930 + 0.0003
0.2638 + 0.0004
18

---

Published in Transactions on Machine Learning Research (06/2023)
Image Cost 0.0010
Image Cost 0.0050
Image Cost 0.0100
Image
Image
Image
Audio
Audio
Audio
10
20
0
10
20
10 Timesteps
20
Timesteps
Timesteps
Image Cost = 0.1000
Image Cost = 0.5000
Image
Image
0
Audio
Audio
10
20
10
20
Timesteps
Timesteps
Figure A.3: Learned patterns on Kinetics at a variety of image cost. The y-axis gives the average acquisition rate of the agent for modality and timestep. As the image cost increases, the mumber of acquisitions decreases. Further, the agents learn a static pattern where at least one modality is always acquired.
Table A.5: Agent behavior with intermediate rewards on AudioSet with different audio acquisition costs and trade-off parameters See $4.2 for an explanation of intermediate rewards and a. Standard deviations over repetitions over the test set.
Cost Audio Trade-off a
0.0005
0.0005
0.00010
0.00010
0.1
0.5
0.1
0.5
Agent Acquisition Rate Agent (mAP)
0.05
0.13
0.04
0.06
0.2648 + 0.0063
0.3104 = 0.0120
0.2671 + 0.0009
0.2668 + 0.0075
Random- Rate (mAP) Random-1 (mAP)
0.2439 + 0.0012
0.3043
0.0013
0.2323 + 0.0014
0.2570 + 0.0014
0.2750 + 0.0006
0.3238
0.0004
0.2664 + 0.0003
0.2838 + 0.0008
Table A.6: Agent behavior with intermediate rewards on Kinetics with different image acquisition costs and trade-off parameters a. See $4.2 for an explanation of intermediate rewards and a. Standard deviations over repetitions over the test set.
Cost Image Trade-off
0.01
0.01
0.10
0.10
0.1
0.5
0.1
0.5
Agent Acquisition Rate Agent (top1)
0.19
0.27
0.15
0.18
0.3260 = 0.0018
0.3234 + 0.0012
0.3225 + 0.0027
0.3162
0.0008
Random- Maadoom-Rateee (topl)
0.2895 + 0.0004
0.2673 + 0.0011
0.2768 + 0.0011
0.2807 + 0.0009
Random- -aallom-HHot Hot (top1)
0.3015 + 0.0003
0.2798 + 0.0004
0.3042 + 0.0004
0.3037 + 0.0003
19

---

Published in Transactions on Machine Learning Research (06/2023)
Once we found valid cost value, increased and decreased costs to find range of costs that leads to varied agent behavior. First, we increased costs until the agent (almost) did not make any acquisitions anymore. Tables 5 and G show this requires adjusting the costs across multiple magnitudes. Note that acquisition rates of about 0.04 correspond to acquiring a single segment for our total of = 25 segments. As the agent already acquires only single segment, additional cost. increases are therefore not that interesting from the perspective of A2MT. We also decreased costs until agent behavior became stagnant. For AudioSet at the lowest cost, the agent acquires (almost) all segments of the audio signal, so further decreasing the cost would not change behavior. For Kinetics, agent acquisition rates are already constant for the three lowest acquisition costs, and it is unlikely that further cost decreases would change this. It. seems that the agent refuses to learn to acquire more than 20% of the image modality, regardless of how low the cost is. In table | we observed that predictive performance on Kinetics is almost unchanged until mask rates are increased above 80%, which points towards the repetitive nature of the image modality in the Kinetics dataset. It is therefore likely that the agent behavior sensible here, as there is simply no benefit to increasing acquisitions above 20% for the image modality of Kineties for the Perceiver 10 predictive model.
C.2 Synthetic Experiments: MNIST and SpokenDigit Versions
For all hyperparameters not mentioned in the below, we use the same settings as for the 'raw' version of the synthetic dataset. We use a batch size of 128 and train for x 106 steps. The input shape of the MNIST images is (10, 28, 28), and the shape of the audio snippets is (10, 39, 80) after mel spectrogram pre-processing.
C.3 Audio-Visual Datasets
C.3.1 Dataset
The AudioSet dataset has training set of size 1 771 873, an evaluation set of size 17748, and 632 classes. We use the unbalanced version of the dataset. The Kinetics dataset has training set of size 545 793, a test set of size 67 858, and 700 classes. For both datascts, each input sample consists of audio-visual input with 250 frames of images and raw audio signal spanning 10 seconds. For images, we take every 10th frame as input, obtaining a total of 25 input frames. We pre-process the audio signal to mel spectrograms, and then divide the signal into 25 segments. The audio modality has input shape (25, 40, 128), and the image modality has input shape (25, 200, 200,3).
C.4 Architecture
Perceiver IO. For the Perceiver TO encoder we use a single cross-attend block with self-attention oper- ations per Perceiver 10 block; we use 1024 queries, and the hidden dimension is 1024. For the Perceiver 10 decoder, i.e. the policy head, we use single cross-allend head with 1024 queries and hidden dim of 1024. The A2C baseline network is also Perceiver I0 decoder module with the same configuration as the policy,
Pre- Training. We use the ADAM optimizer with initial learning rate of 3 x 10-4, weight decay of x 10-6 and a cosine annealing schedule. We train for total of 100 epochs using batch size of 512. We use the masking settings detailed below.
For the masking variants M1-M3, we report results with the following settings. For M1, we keep inputs at given modality and timestep with Pm = 0.2. For M2, we set Pm = 0.4, which accounting for the ionalional 'max-timestep' masking mechanism (which masks out half the input on average), yields the same expected unmasking rate of 0.2. For M3, we keep Pm 0.4 and additionally drop modalitics with dm 0.5. In all of the above, probabilities are the same across modalities m. These settings performed best in preliminary experiments.
Agent Training. We train agents for a total of 20 x 10a steps. We re-use the optimizer configuration from pre-training for training of the agent and the A2C baseline network. We set the discount factor to y 1.
20

---

Published in Transactions on Machine Learning Research (06/2023)
D A2MT as a Markov Decision Process
Here, we attempt to formally connect A2MT to Markov Decision Processes (MDPs) which are the main paradigm for framing problems in the reinforcement learning literature (Sutton & Barto 2018 Concretely, we believe that a Partially Observable MDP (POMDP) is best suited to the A2MT scenario. We refer to 2020 for a similar treatment the (PO)MDP action space for active
Yin et al. [2020]: ; Janisch et al. 2019 feature acquisition problems.
POMDPs are defined by the 6-tuple (S,,,,,,,,,,,), At each timestep 1, the enviromment is in some state S and the agent selects an action E A. This leads to new state 8++1 P(- ar) and reward r. R(s;, az). In a POMDP, the agent never observes states directly, and instead has access only to observations E S2 depending on the unobserved state, 0r ~ (-1ss). To select actions, the agent uses a policy depending only on the current observation a; ~ =+-oo))
Using the notation roduced above and in L we can align the A2MT framework with a POMDP by (L...., are binary acquisition decisions per modality, at AB, ,t E (I,...,T) T). After the sequence is consumed, at T + 1, we insert an additional timestep at which no action is laken, AP =0. 0. Formally, the action space A is the union of both spaces, A=A² U A", (2) For t E (I...., T). the state 8, contains the unmasked sequence of the modalities until time f, i.e. the sequence of observations Di:t: as well as a vector of all previous agent actions, aii Here, the modalities p(=,/=<+1) evolve according to the data generating distribution, c.g. the algorithm for the synthetic datasel creation in $3.1or or the generative model underlying AudioSet/ /Kinetics video data. At time T + 1, the state additionally contains the label associated with the multimodal sequence, i.e. a sample from the conditional distribution p(y ((T) of the generating process. (3) The observation kernel O(o, (s)) emits sequence of (partially) masked modalities according to the acquisition pattern of the agent up to time 1, æ1:- Note that because of our construction in (2), the state contains all necessary information to assemble this sequence at each timestep. No observation is emitted at 0T+1- (4) For & E T)), the reward function is defined as (s; ax) = (a,) = Σn, Cmar,m, ie. the reward is the negative modality- specific acquisition cost at that timestep, which does not depend on the state, cf. eq- ① For == 7 1, the reward is given by the negative loss achieved by the classifier R(ST+1, ar+1) R(s++1) = -(f((rr)))))) y) Note that, instead of modelling the prediction (( 3) an action, we include ʃ as part of the global environment state as it is not trained by reinforcement learning. When f is fixed, e.g. in large scale experiments, the reward from the classification at T+1 depends only on the acquisitions of the agent; f and its parameters are fixed across sequences. When f is trained jointly with the policy, the parameters of f are updated between- but not within episodess, making the reward and thus the POMDP nonstationary.
making the following identifications: (1) Actions for
21


Reviews:
[{"review_id": "I5iQlh7CiD", "reviewer": "Reviewer_Xafx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n* The problem in the paper is ambitious, and the model chosen is a relatively new attention-based architecture that aims to decouple the input length from the model’s residual stream dimension.\n* The problem is well-defined, and its relevance is described compellingly.\n* The difficulty of datasets scales well from more simple to more complex.\n* The paper provides some interesting observations, including a change from overfitting to generalization for the simple synthetic dataset, the comparison against random agents, the 'negative' result concerning adaptivity (see also Weakness paragraph), and the relationship between loss and entropy.\n\n**Weaknesses:**\n* The datasets should be clearly described in the main body of the paper. In several places, it is pretty hard to follow what are the shapes of the data or what part of the data the evaluations are performed on (e.g., Table 1-3).\n* The paper's clarity would improve if the tables’ and figures’ captions were self-explanatory. One must find information scattered across the paper to understand better what is happening. For instance, in Table 4-5, it not entirely clear what 'Agent Acquisition Rate' is (it is hard to find a definition of that quantity, with a caption of Figure A.1 being some proxy of that), what are the Random baselines (page 8), why is only one cost per acquisition (page 7), what dataset is used for the calculations for Kinetics.\n* The paper structures A2MT as a reinforcement learning problem by adding a temporal aspect, actions, sparse rewards, and using an RL algorithm for training (A2C). However, the exact setup is not clearly defined (typically, one describes the underlying MDP or a similar object). Some deviations from the standard setup include the reward being defined by a performance of another learned model or the policy depending on the whole trajectory (and sometimes more).\n* The description of the synthetic dataset could be improved. For instance, “the counter modality repeatedly counts down from a randomly drawn staring value” is somewhat confusing. The lack of clear description here makes the analysis of Figure 4 hard to follow.\nDescribing Gumbel softmax as an RL algorithm is perhaps an abuse of notation.\n* Random agents (baselines) are unclear: how to compute the acquisition rate for 'Random-Rate' agent, or how is the “fixed set of equidistantly spaced timesteps” chosen?\n* The paper provides a 'negative' result by showing that the learned agent does not learn a policy that adapts to inputs. This is interesting (as mentioned in the Strengths paragraph); however, the justification of the Authors’ “there might not be enough signal from the model predictions for the agent to anticipate how acquisitions affect the reward” or “method struggles to learn acquisition strategies and representations simultaneously”. This leaves the reader slightly unsatisfied and unsure whether the Authors’ made enough effort to clarify the issue. Some of the reasons could include (1) Some specific features of the data (e.g., periodicity); (2) Too short learning; (3) Difficulty in learning a policy to match a pre-trained predictive model; (4) Distribution shift in masking from pre-training to policy training; (5) Not enough regularization (the Authors’ mention the model’s overfitting), etc.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper has three contributions. First, it introduces a so-called A2MT task, which abstractly allows the agent to query temporal and multimodal data, e.g., video or audio stream. At each timestep, the agent can select some modalities (possibly none or all) and view their entry corresponding to the given timestep, e.g., a given frame in the video stream. The goal is to acquire enough data to predict some labels attached to the temporal and multimodal data (possibly some function of the data). Each query has its cost, possibly dependent on the modality, which creates a tug-of-war dynamics, where the agent wants to increase the chances of predicting the label, and at the same time, minimize the number of queries. The paper's second contribution is the actual instance of A2MT consisting of synthetic data (sort of a guessing numbers game) and the datasets based on AudioSet and Kinetics, introduced in other papers. Finally, the paper proposes a deep-learning system composed of a prediction model (here based on Perceiver IO architecture) and a policy, which makes a decision on which modality to query at each step. The paper includes experimental analysis, including positive and negative results. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I like the paper: it sets an interesting problem, proposes a learning system that aims to solve the tasks, and discusses some interesting experimental phenomena. However, I would like the Authors to improve the clarity, the technical details, and discuss some of the alternative reasons why the policy does not learn adaptive behavior (for details, see the Weakness section). If the Authors decide to follow up with this, I am willing to set \"Claims And Evidence\" to \"Yes\".", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "La9NTiMTDW", "reviewer": "Reviewer_5VUY", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n- The paper proposes an interesting task and datasets on which to evaluate it. The task generalizes some existing tasks that either assume that the acquirable features do not vary in time or that the acquirable features are low-dimensional.\n- The paper evaluates relative to some very important ablations and presents findings despite their \"nonsignificance\" in many settings, which informs readers that there is significant room for improvement on the methods side.\n- The paper evaluates on a tightly controlled synthetic regime, which is also itself a good regime for further experimentation on varying some of the fundamental characteristics of the data and evaluation.\n\n## Weaknesses\n### 1: Presentation weaknesses\n- [W1.1] would be significantly improved with an architecture diagram of the policy and the classifier. \n- [W1.2] S3.1: It is unclear if the randomly drawn starting value for the counter is drawn once or multiple times. The example sequence given: “2103210210” appears to be only realizable if it is drawn multiple times, assuming values: [2, 3, 2]. Please clarify this in the paper.\n- [W1.3] The experiment presented in Table 3 is unclear. A mask rate of 0 results in AudioSet mAP of 0.370, which appears to correspond to the 0.370 values in Table 2 and Table 1. However, the 0.370 value in Table 1 includes masking (M1,M2,M3). It is unclear what a “Mask Rate” of 0.0 (Table 3 Row 1) and masking of M1, M2, and M3 combined are doing, because masking with 0.0 probability seems like a contradiction with applying the masking of M1, M2, and M3. Is the mask rate the input to M1, M2, and M3? This would make sense if it were used as the parameter of the Bernoulli distribution, so it would work for M1 and M3, but not M2. My main guess here is that the Mask Rate in Table 3 is the rate of masking the test data, not the rate of masking the training data. Please clarify the paper so that others are less likely to have to guess at this. If my guess is correct, I suggest using terminology that clarifies the difference between masking when it is used as model optimization hyperparameter, vs. when it is used as a (test/eval) dataset parameter.\n- [W1.4] S5.1 would be improved by presenting the metrics in a table, rather than inline, in order to increase their visibility.\n- [W1.5] The related work section could be improved by including discussion of the “anytime prediction” framework and approaches (e.g., but not limited to, [A,B,C,D] in the References section below), in which a learner’s output can be computed at “any time” and improves with the addition of more inputs. The proposed A2MT framework, in which a single prediction is made after a fixed number of timesteps with costs on using certain inputs, could be considered a degenerate version of anytime prediction, which also imposes costs on using certain inputs, but requires approaches to be capable of producing an output at every timestep. \n### 2: Experimental weaknesses\n- [W2.1] The experiments in S5.1 and Fig A.1 could be improved with more metrics of counter acquisition that are normalized relative to the optimal number of acquisitions and the precise timesteps an optimal agent would attend to the counter. The optimal number of acquisitions is determined by the counter variable(s) — the digits need only be attended to by an optimal agent where the counter is 0, so observing the counter values after each 0 is sufficient. The paper states that 68.6% of the counter modality is attended to, but it is unclear how much an optimal agent should be attending to the test data. Furthermore, matching the optimal attention rate to counter variables is insufficient for attending to the correct counter variables — the analysis could be further improved with a metric of accuracy of attention to the “necessary” counter variables — scoring true positives for attention to the counter values that succeed 0, false negatives for failing to attend to those values, and false positives for counter modality attention elsewhere. These same metrics could be applied to the digit modality. These types of accuracy metrics are particularly valuable for this synthetic dataset because they’re easily computable relative to exactly computing or estimating them for the Audio-Visual datasets.\n- [W2.2] The experiments in Table 4 and Table 5 would be significantly improved if they included prior work on these same classification tasks, both from approaches that are cost-insensitive (e.g. any prior classification approaches, which I’m fairly confident exist) and those that are cost-sensitive, if they exist on these dataset (I don’t know such exist on these datasets). The former type of approaches would ground the proposed methods potential improvement over cost-insensitive approaches, which is perhaps the main value proposition of any A2MT-based method. \n- [W2.3] It is unclear how the ranges of cost values investigated in Table 4 and Table 5 were chosen. The paper would be improved with additional experiments here. The cost ranges appear to differ significantly: in [1e-6, 1e-3] for audio, vs. [1e-3, 5e-1] for images. It is possible that the effect of outperforming the random ablations is only present in the lower range [1e-3, 5e-1] for both modalities, and that the effect of underperforming the ablations is only present in the higher range [1e-6, 1e-3]. If the experiments presented results that used the same dense ranges for both modalities, we could draw stronger conclusions.\n### References\n- [A] Zilberstein, Shlomo. \"Using anytime algorithms in intelligent systems.\" AI magazine 17.3 (1996): 73-73.\n- [B] Horvitz, Eric J. \"Reasoning about beliefs and actions under computational resource constraints.\" arXiv preprint arXiv:1304.2759 (2013).\n- [C] Shani, Guy, Joelle Pineau, and Robert Kaplow. \"A survey of point-based POMDP solvers.\" Autonomous Agents and Multi-Agent Systems 27 (2013): 1-51.\n- [D] Grubb, Alex, and Drew Bagnell. \"Speedboost: Anytime prediction with uniform near-optimality.\" Artificial Intelligence and Statistics. PMLR, 2012.\n ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a framework for classification in which costs are assigned to observing different multimodal inputs (e.g. images or audio), and implements an approach to perform this classification with a Perceiver-IO type of architecture that takes actions that determine whether it attends to each of the different inputs. Experiments on a controlled synthetic dataset and several larger datasets evaluate the methods performance relative to its ablations. The paper finds that in some settings on some datasets, the method outperforms ablations in which the attention to the inputs is set at a fixed rate and at a fixed frequency. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Critical weaknesses to address: W1.1, W2.2 (cost-insensitive prior work), W2.3.\nThe remaining weaknesses are important (including (W2.2 cost-sensitive prior work)) but noncritical.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "bZCRblLbHs", "reviewer": "Reviewer_sDJ9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The motivation of the task formulation is clear and evident, and one can clearly see could have potential applications in various fields like finance, and healthcare.\n- The formulated problem is novel and in time, which is an up-to-date formulation for the classic Active Feature Acquisition (AFA) problem.\n- The empirical findings, especially those in Section 5.2.2, are of interest to the TMLR community.\n- Overall, the delivery and presentation of the work are mostly clear and easy to follow, with some minor formatting issues that need to be addressed.\n\nWeaknesses:\n- The discussion of related work can improve, in particular, the authors should consider situating or differentiating their work against the line of work in multimodal active learning.\n- Based on my understanding, the synthetic scenario used in the paper is not a natural abstraction of the temporal decision-making problem. Especially, I am not sure if the feature values evolve.\n> \"Notably, prior work in AFA assumes static data: although acquisitions are sequential, feature values do not evolve along a temporal dimension.\"\n>\nMore specifically, the synthetic scenario involves a series of random counting sequences, in which the starting number does not depend on the previous sequences. \n> \"The counter modality repeatedly counts down from a randomly drawn starting value, e.g. 2103210210. The label attached to each sequence is the sum of the digit modality at all timesteps where the counter modality is zero, e.g. 3+3+3=9 in this case.\"\n>\nThe presented problem is more of a reasoning problem over sequences than a temporal decision-making problem, to my understanding. For a more natural abstraction, a Markovian (or more complicated temporally dependent sequence) sequence would make more sense. I would love to hear more about the design choice of this experiment from the authors.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "- The paper introduces a novel task formulation, i.e., the Active Acquisition for Multimodal Temporal Data (A2MT) problem.\n- The authors have designed synthetic scenarios and validated on real-world datasets, AudioSet and Kinetics.\n- The paper introduces a reinforcement learning approach based on Perceiver IO to tackle the proposed problem with a thorough empirical\nstudy. The agents' appropriate response to modality-specific acquisition costs demonstrates the potential of the proposed method for real-world applications.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Major:\n- (See weakness for details) I would really like to see how the presented method works on a more continuous time-dependent synthetic scenario, e.g., a simple Markovian counter sequence.\n\nMinor/Formatting:\n- In Section 2, the notation $c$ has been used to denote both color channels and cost. Please consider differentiating them.\n- Figure 5 and 7 are too small to read. Please consider enlarging them.", "broader_impact_concerns": "Not applicable.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper is well-written and introducing an interesting problem. You have already addressed many of the reviewers concerns, so I won’t reiterate those here. The decision is Accept with Minor Revisions, where I (the action editor) can check that a few small things are implemented. 

The reviewers largely seem happy with the changes. A few requests based on the reviewers reviews:
1. One reviewer asked for an architecture diagram of the policy and classifier. I suspect the reason for this request was that it is not clear how the classifier and the policy were jointly trained. Presumably, they were simply trained in parallel, with the policy producing the inputs and then that being handed to the classifier for training. I think this concern can be addressed by more clearly highlighting this training procedure, and stating what you said that you have two independent perceiver IO models. It would also be useful to explain why the Perceiver IO model easily handles missing data, since when searching in the original paper for the word “missing”, nothing seems to come up. A diagram of this might be useful way to explain this, but is not strictly necessary.

2. There was a nice request to formalize the MDP underlying this RL problem, and you did so in the appendix. However, this could use a bit of improvement. Its mostly ok except for the treatment of f (which makes this a nonstationary POMDP). Since f is not trained using RL, I wonder if it would be better to consider f as part of the MDP. In the simplest setting, f is fixed (somehow trained ahead of time) and the RL agent’s job is just to figure out what inputs to acquire to minimize cost while still getting good predictions. If f is changing, then this f could be part of the POMDP state (not visible to the agent). 

3. One reviewer asked to include anytime prediction. I see how this is related, but quite different; its appreciated that you discussed it. However, the related work section is too sparse. Active perception and attention are large areas. For example, see a reasonably large survey in this work (I do apologize that it is my own, but of course, its the easiest one for me to think of, you can find other maybe more pertinent citations in it):
“Maximizing Information Gain in Partially Observable Environments via Prediction Rewards” Satsangi et al., AAMAS 2020

These should be very simple writing changes, thus the Accept with Minor Revisions.

Decision: Accept (poster)