Title: Multi-model Ensemble Conformal Prediction in Dynamic Environments
Venue: NeurIPS 2024
Accepted: yes
Keywords: uncertainty quantification

conformal prediction

dynamic environments
Abstract: Conformal prediction is an uncertainty quantification method that constructs a prediction set for a previously unseen datum, ensuring the true label is included with a predetermined coverage probability. Adaptive conformal prediction has been developed to address data distribution shifts in dynamic environments. However, the efficiency of prediction sets varies depending on the learning model used. Employing a single fixed model may not consistently offer the best performance in dynamic environments with unknown data distribution shifts. To address this issue, we introduce a novel adaptive conformal prediction framework, where the model used for creating prediction sets is selected ‘on the fly’ from multiple candidate models. The proposed algorithm is proven to achieve strongly adaptive regret over all intervals while maintaining valid coverage. Experiments on both real and synthetic datasets corroborate that the proposed approach consistently yields more efficient prediction sets while maintaining valid coverage, outperforming alternative methods.

Multi-model Ensemble Conformal Prediction in Dynamic Environments

Abstract

Conformal prediction is an uncertainty quantification method that constructs a prediction set for a previously unseen datum, ensuring the true label is included with a predetermined coverage probability. Adaptive conformal prediction has been developed to address data distribution shifts in dynamic environments. However, the efficiency of prediction sets varies depending on the learning model used. Employing a single fixed model may not consistently offer the best performance in dynamic environments with unknown data distribution shifts. To address this issue, we introduce a novel adaptive conformal prediction framework, where the model used for creating prediction sets is selected 'on the fly' from multiple candidate models. The proposed algorithm is proven to achieve strongly adaptive regret over all intervals while maintaining valid coverage. Experiments on real and synthetic datasets corroborate that the proposed approach consistently yields more efficient prediction sets while maintaining valid coverage, outperforming alternative methods.

1 Introduction

Most machine learning algorithm designs aim to enhance label prediction accuracy. Nevertheless, a significant challenge persists as many models demonstrate limitations in predicting labels with high certainty, falling short of achieving the desired levels of accuracy and other critical evaluation metrics. In applications such as medical diagnosis, it is sometimes more efficient to predict a subset of labels rather than a single label [Levy et al., 2021, Straitouri et al., 2023]. This necessitates predicting a set of candidate labels with a valid coverage probability, rather than limiting to a single label.

One of the most widely used frameworks for set prediction is conformal prediction [Vovk et al., 2005]. Conventional conformal prediction algorithms can achieve the desired coverage assuming the exchangeability of data [Balasubramanian et al., 2014]. However, in many real-world online problems, the distribution of data shifts over time, making the exchangeability assumption no longer applicable. Consequently, adaptive conformal prediction algorithms have been developed [Gibbs and Candes, 2021], where prediction sets are constructed in a time-varying manner. Despite these advancements, the efficiency (e.g., prediction set size or regret) of previous conformal prediction methods in online settings with distribution shifts heavily depends on the model employed, and

a single model may not consistently perform well across various distribution shifts. Creating an efficient prediction set size is as important as obtaining the desired coverage. Trivial cases may arise where the prediction set alternates between an empty set and the full set of labels, with the full set being created with a probability equal to the desired coverage probability, and empty sets otherwise. This approach achieves the desired coverage but leads to impractical prediction sets [Bhatnagar et al., 2023]. To address this limitation, our proposed algorithm incorporates multiple learning models simultaneously. It dynamically selects the suitable model based on the performance of each model with the most recently received data.

Related work: Conformal prediction [Vovk et al., 2005, Shafer and Vovk, 2008, Vovk, 2015] is an effective method for uncertainty quantification that has been widely used to predict a set of candidate labels for income data. It treats the learning model as a black box and provides a prediction set for new test data. Conformal prediction is applicable in both Classification [Ding et al., 2024, Shi et al., 2013, Romano et al., 2020] and Regression [Romano et al., 2019, Papadopoulos et al., 2011, Boström et al., 2017] problems. In dynamic environments where data distribution shifts over time, using vanilla conformal prediction algorithms may not lead to the desired coverage performance. To cope with this challenge, conformal prediction in dynamic environments has been studied recently. [Tibshirani et al., 2019] explored conformal prediction for dynamic settings using a reweighting approach, but their method requires prior information about the data dependency structure; [Barber et al., 2023] resolved this dependency by requiring weights to be fixed. Incorporating time-varying coverage probability was introduced in [Gibbs and Candes, 2021], but determining the appropriate step size remains a significant challenge. One way to address dynamic environments is adopting learning with expert advice [Cesa-Bianchi et al., 1997, Vovk, 1995, Littlestone and Warmuth, 1994]. [Zaffran et al., 2022] suggested that tuning the step size based on expert (base learner) aggregation could be an effective strategy; however, this method causes each expert to receive equal impact from all historical data, which makes the algorithm unable to adapt to sharp distribution shifts. Furthermore, [Gibbs and Candès, 2022] introduced an approach that employs multiple experts, each assigned a distinct step size from a pool of candidate sizes, resulting in varying coverage probabilities at each time t . While [Gibbs and Candès, 2022] successfully demonstrated adaptive regret across time intervals of a fixed width, [Bhatnagar et al., 2023] points out that this approach fails to achieve suitable regret across varying widths of arbitrary time intervals simultaneously. [Bhatnagar et al., 2023] addressed this limitation by establishing strongly adaptive regret [Daniely et al., 2015] across any arbitrary time interval width. Their proposed methodology assigns a specific time interval to each expert. Despite achieving sublinear strongly adaptive regret, the efficacy of this approach depends on the hyper-parameter selection that determines each expert's lifetime. The proposed method in our study also employs experts who operate within specific time intervals. However, each expert consists of multiple learning models, aiming to select the appropriate model according to the specific data distribution during its operation, resulting in more efficient prediction sets.

Contributions. Overall, our contributions can be summarized as follows:

I) We introduce a novel adaptive conformal prediction algorithm, S trongly A daptive M ultimodel Ensemble O nline C onformal P rediction (SAMOCP), designed for dynamic environments with unknown distribution shifts. This algorithm incorporates multiple models and dynamically selects a model based on its performance in previous time steps.

II) We demonstrate that SAMOCP exhibits strongly adaptive regret for any arbitrary time interval while ensuring valid coverage.

III) Through experimental tests on classification tasks subject to distribution shifts, we demonstrate that SAMOCP outperforms existing methods by constructing more efficient prediction sets while also achieving a coverage probability closely aligned with the target value.

2 Preliminaries

This section explains standard conformal prediction and adaptive online conformal predictions, where data is collected sequentially. We begin with outlining standard conformal prediction. Given a miss coverage probability α , a learning model m , a historical dataset { ( X τ , Y true τ ) } t -1 τ =1 , and a new data X t ∈ X , the objective is to construct a prediction set C m α ( X t ) ⊆ Y := { 1 , 2 , . . . , K } , where K denotes the total number of classes, such that C m α ( X t ) contains the true label Y true t with probability 1 -α . In the online setting, historical dataset is updated to { ( X τ , Y true τ ) } t τ =1 at the end of each time t when true label Y true t for input data X t is observed. In this scenario, conformal prediction treats

the historical dataset as a calibration dataset, which is utilized to determine whether a candidate label Y ∈ Y should be included in the prediction set. Consequently, in an online manner, conformal prediction relies on an evolving calibration dataset, which contains all the historical data 'on the fly' to decide which candidate labels should be included in the prediction set. Non-conformity scores { S m ( X τ , Y true τ ) } t -1 τ =1 are introduced. Specifically, each non-conformity score S m ( X τ , Y true τ ) assesses the disagreement between the ground-truth label Y true τ and predicted label ˆ f m ( X τ ) . Upon obtaining a new datum X t , the standard conformal prediction algorithm constructs the prediction set for X t as C m α ( X t ) = { Y ∈ Y | S m ( X t , Y ) ≤ ˆ q m α } , where the threshold ˆ q m α is obtained as

The Quantile function sorts all non-conformity scores of the historical data and then identifies the ⌈ t (1 -α ) ⌉ th smallest score as ˆ q m α . Note that 1 -α is fixed, hence conformal prediction cannot readily cope with potential data distribution shifts in dynamic environments. Adaptive conformal prediction algorithms have been developed to address this issue, allowing the miss coverage probability to vary at each time t and thereby enabling the algorithm to dynamically adapt to potential shifts in the distribution. In such a scenario, ˆ q m α can be obtained by replacing α with α t in (1). Then at each time step t , α t is updated after observing Y true t .

Recent studies on online conformal prediction with distribution shifts have incorporated adaptive miss coverage probabilities to address dynamic environments. However, methods based on a single learning model may not achieve consistently reliable performance in dynamic environments. This underscores the necessity for employing multiple models and adaptive strategies to determine the appropriate model for each time t . To this end, in this work, we introduce a novel adaptive multimodel online conformal prediction algorithm designed to identify the suitable learning model at each time t within dynamic environments. At time slot t , the goal is to construct a prediction set for the new data X t , based on the historical dataset { ( X τ , Y true τ ) } t -1 τ =1 , such that the true label is included in the prediction set with probability 1 -α . The proposed algorithm for dynamic settings achieves strongly adaptive regret while ensuring valid coverage.

3 Methodology

In this section, two adaptive algorithms are developed for static and dynamic environments respectively. Subsection 3.1 develops the M ultimodel Ensemble O nline C onformal P rediction (MOCP) algorithm to identify the suitable learning model among M distinct candidates in a static environment. Subsequently, in Subsection 3.2, we propose SAMOCP, an adaptation of MOCP tailored for dynamic environments with unknown distribution shifts.

3.1 Multi-model Conformal Prediction in Static Environments

Note that the non-conformity score S m ( X τ , Y true τ ) depends on the learning model. Such dependency leads to a model-specific ordering of non-conformity scores, yielding different prediction sets for each model. After observing Y true t , the adaptive miss coverage probability α t must be updated for time t +1 to cope with distribution shifts effectively. Given that different models achieve different prediction sets, assigning and updating the same α t for different models would be inadequate. Instead, at each time t , we assign a specific miss coverage probability to each model m ∈ [ M ] , denoted as α t m , and update it based on the corresponding prediction set. Consequently, for M learning models, there are M candidates for miss coverage probability α t at each time t . Each candidate is updated according to a distinct rule. These M update rules operate in parallel, with each one updating the corresponding miss coverage probability upon observing the true label. Next, the update procedure for α t m will be examined, followed by a detailed explanation of how each instance of MOCP selects the appropriate miss coverage probability from M distinct options at each time step.

To update miss coverage probability α t m , we adopt the pinball loss [Koenker and Bassett, 1978], which can be written as

where

is the best possible value of miss coverage probability for model m at time t which constructs the smallest prediction set that covers Y true t .The miss coverage probability α t m +1 can be updated via SF-OGD [Orabona and Pál, 2018] as

where η is the learning rate and

with err t m := I [ Y true t / ∈ C m α t m ] equals 1 if the predicted set does not contain the true label Y true t , and 0 otherwise. According to the updating rule outlined in equation (4), the adjustment of α t m at each time t is governed by the ∇ α t m L (¯ α t m , α t m ) , as detailed in equation (5). When err t m = 1 , it signals that the coverage probability 1 -α t m is too small, resulting in a prediction set that can not encompass Y true t . Consequently, there's a necessity to enlarge the coverage probability, effectively achieved by reducing α t m , which would be facilitated by (4); given that the denominator in the second term is always positive and the gradient will be positive in this scenario. On the other hand, when ¯ α t m > α t m , 1 -α t m leads to a prediction set that covers Y true t but also includes unnecessary labels Y ′ := { Y ′ ∈ Y | ˆ q m ¯ α t m < S m ( X t , Y ′ ) ≤ ˆ q m α t m } . In such cases, optimization necessitates increasing α t m to avoid including unnecessary labels and output a more efficient prediction set. This adjustment is facilitated by the update rule (4).

Additionally, the weight w t m is assigned to each model m ∈ [ M ] , which influences the selection of its corresponding miss coverage probability α t m . MOCPlearns which model to select over time based on the performance of each model over previous time steps, as reflected in w t m . The algorithm updates the weight associated with each model after revealing the true label Y true t . This update is performed with respect to the loss function of the corresponding miss coverage probability. Specifically, w t m can be updated by

where 0 < ϵ < 1 is the step size. At each time t , upon receiving new data X t , MOCP first calculates the normalized weights, denoted as { ¯ w t m } M m =1 . For any m ∈ [ M ] , ¯ w t m = w t m ∑ j M =1 w j t ensures that ¯ w t m ∈ [0 , 1] and represents the likelihood of selecting miss coverage probability α t m . Then, the algorithm selects miss coverage probability α ˆ t m , where ˆ m ∈ [ M ] , according to PMF ¯ w t := ( ¯ w t m ) M m =1 , i.e., each miss coverage probability α t m is selected with probability proportional to the corresponding normalized weight ¯ w t m . The prediction set for X t is constructed according to the threshold in (1), by replacing α and m with α ˆ t m and ˆ m respectively. After receiving Y true t , each weight w t m and miss coverage probability α t m are updated according to (6) and (4), respectively. This entire process is detailed in Algorithm 1. The MOCP algorithm achieves a runtime of O ( T ) when the number of models M is constant.

Given that the environment is static, there exists a miss coverage probability that can minimize the loss function for each model m ∈ [ M ] over [ T ] , denoted as α m . The best miss coverage probability among { α m } M m =1 can be obtained by

The following theorem demonstrates that MOCP achieves sublinear regret (See proof in A.2).

Theorem 1 Algorithm 1 achieves the following regret bound in a static environment

3.2 Multi-model Ensemble Conformal Prediction In Dynamic Environments

Through Algorithm 1, we demonstrated how to select the suitable miss coverage probability of a model that has achieved lower loss compared to other models over previous time steps. However,

Select one of the miss coverage probabilities { α t m } M m =1 according to PMF ¯ w t = ( ¯ w t m ) M m =1 . Observe true label Y true and compute optimal value ¯ α ∀ m ∈ [ M ] with (3).

Time

Expert

1

2

3

4

5

1

2

3

4

5

this approach assumes a static environment that does not change over time, which may limit the algorithm's effectiveness in dynamic environments where data distribution shifts occur. In addition, the selection of stepsize ϵ in (6) critically affects the performance. In environments with unknown distribution shifts, a large ϵ indicates faster adaptation to abrupt changes, whereas a small ϵ is more suitable for environments with less variability. Thus, the efficient choice of ϵ depends on the variability of the environment, which poses a challenge in scenarios with unknown distribution shifts.

To address this limitation, we introduce Strongly Adaptive (SA)MOCP. Specifically, each instance of MOCP is treated as an 'expert', and multiple experts are created at distinct time steps with specific step sizes and lifetimes to cope with potential distribution shifts. At the end of its lifetime, the expert becomes inactive, ensuring that it no longer affects the decision-making process, refer to Figure 1 for an illustration. This strategy prevents outdated experts from contributing to the selection of the suitable miss coverage probability in dynamic environments. Subsequently, SAMOCP dynamically selects the appropriate expert for each time t and utilizes its chosen miss coverage probability to construct the prediction set. The lifetime of each expert is determined by the specific time t at which it was created and hyperparameter g ∈ N , as [Bhatnagar et al., 2023].

The active interval of the expert created at time t is defined as [ t, t + λ ( t ) -1] . Consequently, experts considered active at any given time τ are those whose active intervals include τ . In a dynamic setting with unknown distribution shifts, the best model may vary across different distributions. This variability results in scenarios where, at each time t , some active experts might not have adapted to the current data distribution yet, and thus they rely on different models compared to more recently established active experts. In such cases, the miss coverage probability of model ˆ m ∈ [ M ] chosen by expert n at time t is represented as α ˆ mn t , and its best possible value is denoted as ¯ α ˆ mn t . To select the suitable miss coverage probability among different experts, we assigned distinct weights to each,

with specific initialization and step sizes for updates. Specifically, this weight for expert n at time t is denoted as h n t , and its step size is defined as ϵ n := min( ϵ, σ √ λ ( n ) ) , where σ > 1 is a constant and

λ ( n ) is the lifetime for expert n that obtained by (9). Given that n th expert is activated at t = n , the initialization and update rule for h n t is as follows:

where r n t = L (¯ α ˆ m ˆ n t , α ˆ m ˆ n t ) -L (¯ α ˆ mn t , α ˆ mn t ) represents the loss of the n th expert relative to the loss of the learner who selects expert ˆ n . We denote the set of active experts at each time t as A ( t ) . The learner selects the suitable miss coverage probability among all active experts according to the PMF ¯ h t := ( ¯ h n t ) n ∈A ( t ) , where each ¯ h n t represents the normalized version of the weight h n t , calculated as ¯ h n t = h n t ∑ i ∈A ( t ) h i t , ensuring that ¯ h n t ∈ [0 , 1] . Algorithm 2 summarizes the SAMOCP method. It can be observed from (9) that the maximum number of active experts (MOCP instances) at each time t is g ⌊ log 2 t ⌋ . Hence, the complexity of SAMOCP is of order O ( T log 2 T ) .

Algorithm 2 Strongly Adaptive Multi-model Ensemble Online Conformal Prediction (SAMOCP)

Create new expert n (where n = t ) by Algorithm 1( α ˆ m ˆ n t -1 , η, ϵ n ).

Remove experts whose lifetime has been finished.

Every active expert selects miss coverage probability from M options.

Select one miss overage probability from active experts according to PMF ¯ h t = ( ¯ h n t ) n ∈A ( t ) . Construct prediction set for X t using selected miss coverage probability.

Observe true label Y true t .

for n ∈ A ( t ) do

Obtain learner loss L (¯ α ˆ m ˆ n t , α ˆ m ˆ n t ) .

Update every parameters assigned to each model for expert n via Algorithm 1.

Update h n t +1 with (10).

end for end for

Let CovE ( T ) := ∣ ∣ ∣ 1 T ∑ T t =1 E [ err t ] -α ∣ ∣ ∣ represent the coverage error. In a dynamic setting where multiple experts are incorporated, each including M miss coverage probabilities, the expected error is calculated as E [ err t ] = ∑ t n =1 ∑ M m =1 ¯ h n t ¯ w mn t err mn t , where mn represents the m th model by expert n . Using the two following theorems, we prove that SAMOCP has bounded coverage error and achieves strongly adaptive regret across any time interval of arbitrary width (Proofs can be found A.3 and A.4).

Theorem 2 For any T ≥ 1 and any γ ∈ ( 1 2 , 1 ) , Algorithm 2 achieves the coverage error bound

where β γ ( T ) measures the smoothness of model weights within experts and the cumulative gradient norm for each model within experts. The definition of β γ ( T ) is provided in detail in equation (31) in the Appendix A.3. If there exists a γ ∈ ( 1 2 , 1 ) such that β γ ( T ) ≤ ˜ O ( T θ ) where θ < 1 -γ , then the coverage bound (11) will be CovE ( T ) ≤ ˜ O ( T -min ( 1 2 -γ,γ -1+ θ ) ) = o T (1) .

Theorem 3 Algorithm 2 achieves strongly adaptive regret over any interval I ⊆ [ T ] and positive constants A, B, as follows

where

Note that in equation (13) , α m ∗ n represents the miss coverage probability assigned to the best model for expert n , as obtained by equation (7) .

The miss coverage probability α m ∗ n ∗ in (13), is related to specific interval I , which can vary across different intervals with distinct distributions in a dynamic environment. In such settings, there is no fixed miss coverage probability α m ∗ n ∗ that can be consistently applied over various time intervals. This necessitates establishing that SAMOCP has bounded regret in dynamic environments with respect to the time-varying benchmark in (16), as demonstrated by the following lemma. The proof is provided in A.5.

Lemma 1 By defining the variation of the loss function to be

We establish the following bound for the dynamic regret of Algorithm 2

where ˜ O suppresses positive constants and polylogarithmic factors, e.g., log T . Also the best miss coverage probability at each time t can be obtained by

Lemma 1 establishes that the dynamic regret of SAMOCP (15) depends on the variation of the loss functions (14). In addition, it can be obtained from (15) that SAMCOP achieves sublinear regret if the variation of the loss function is also sublinear, i.e., V ( L ( . ) T t =1 ) = o ( T ) .

4 Experiments

In this section, the performance of the proposed method, SAMOCP, is assessed within the context of classification tasks. We conduct a comprehensive comparison with recently proposed methods in online conformal prediction for dynamic environments within classification tasks. The section begins with a detailed explanation of the experimental settings, followed by a discussion of the results. Note that throughout the experiments in this section, the desired miss coverage probability α is 0 . 1 . All experiments were performed on a workstation with NVIDIA RTX A4000 GPU. Codes are available at hyperrefhttps://github.com/erfanhajihashemi/Multi-model-Ensemble-Conformal-Predictionin-Dynamic-Environments.

Dataset: We utilize corrupted versions of CIFAR-10 and CIFAR-100 [Krizhevsky, 2009], known as CIFAR-10C and CIFAR-100C [Hendrycks and Dietterich, 2019]. These datasets consist of 15 generated corruptions spanning 5 distinct levels of severity. The evaluation encompasses two settings: sudden and gradual distribution shifts. For both settings, the data sequence is split into batches of 500 data samples each. The severity of corruption changes (increases or decreases) after each batch of data. In the sudden shifts, the severity level alternates between the version of the data without any corruption (severity level 0 ) and the most severe corruption (severity level 5 ). In the gradual setting, severity starts at level 0 and increases one by one after each batch until it reaches level 5 . After reaching level 5 , the severity decreases one by one and goes back to level 0 in subsequent batches. This cycle of increasing and decreasing severity continues throughout the experiment. Also, additional experiments on TinyImageNet-C [Hendrycks and Dietterich, 2019] and synthetic data are provided in the Appendix, Section B.

Baselines and experimental settings: We employ ResNet-50, ResNet-18 [He et al., 2016], GoogLeNet [Szegedy et al., 2015], and DenseNet-121 [Huang et al., 2017] as candidate learning models. Each active expert consists of all these learning models and needs to select the appropriate model

during its active interval. The proposed method is compared with the most recent adaptive conformal prediction algorithms designed for dynamic environments, including FACI [Gibbs and Candès, 2022], ScaleFreeOGD [Bhatnagar et al., 2023], and SAOCP [Bhatnagar et al., 2023]. FACI employs a fixed number of active experts over all time steps, with each expert assigned one of the candidate learning rates for updating the miss coverage probability. ScaleFreeOGD reduces the learning rate based on the cumulative norms of gradients [Orabona and Pál, 2018]. SAOCP allows each expert to have its own active interval, within which it operates similarly to ScaleFreeOGD. In order to show how SAMOCP results in more efficient sets compared to SAOCP in a multi-model setting, we developed a multi-model ensemble version of SAOCP, denoted as SAOCP(MM), where each expert consists of M update rules, each corresponding to a different learning model. This approach follows our multi-model approach but employs a similar rule to SAOCP for updating weights. To determine the value of g , we employed a grid search approach within the candidates { 4 , 8 , 16 , 24 , 32 , 48 , 64 } . The one that led to the smallest prediction set size (Avg Width) while maintaining reasonable coverage and runtime was selected, which was g = 8 . While the hyperparameter g is set to 8 for both SAMOCP and SAOCP(MM), it is set to 32 for SAOCP, as in [Bhatnagar et al., 2023]. Since 4 learning models are incorporated in this section, the maximum number of updates at each time t in SAMOCP, Mg ⌊ log 2 t ⌋ , is equal to that in SAOCP and SAOCP(MM), which is 32 ⌊ log 2 t ⌋ . Meanwhile, note that randomness might be undesirable in practice, the predicted miss coverage in SAMOCP is calculated in a deterministic fashion, i.e., α t = ∑ n ∈A ( t ) ∑ M m =1 ¯ h n t ¯ w mn t α mn t . For every experiment conducted on the synthetic dataset, CIFAR-10C, CIFAR-100C, parameters ϵ , σ , and η were selected through grid search, with values of 0 . 9 , 140 , and 0 . 05 , respectively.

Score Functions: We utilized the nonconformity score defined as in [Angelopoulos et al., 2020] to construct prediction sets. Let

where ˆ f m Y ( X ) denotes the probability of predicting label Y for input X by model m , and U t is a random variable sampled from a uniform distribution over the interval [0 , 1] . The term k Y := |{ Y ′ ∈ Y | ˆ f m Y ′ ( X ) ≥ ˆ f m Y ( X ) }| denotes the number of labels that have a higher or equal predicted probability than label Y according to the model's output probability distribution, e.g., the softmax output. ρ ( X,Y ) := ∑ K Y ′ =1 ˆ f m Y ′ ( X ) I [ ˆ f m Y ′ ( X ) > ˆ f m Y ( X )] sums up the probabilities of all labels that have a higher predicted probability than label Y . The hyperparameters ξ and k reg are set to 0 . 02 and 5 for CIFAR-100C, and 0 . 1 and 1 for Cifar-10C, respectively.

Evaluation Metrics: Coverage measures the percentage of instances where the true label is included in the prediction sets outputted by the conformal prediction algorithm over the period [ T ] . Avg Width refers to the average size of these prediction sets. Adaptive regret is calculated for time intervals of length 100. The metric Avg Regret represents the average of these regret values across the entire time horizon [ T ] . Run Time indicates the time required to complete each iteration of the algorithm. Lastly, Single Width measures the probability that prediction sets contain exactly one element while accurately covering the true label, highlighting cases that are most informative for predictions.

4.1 Results

Table 1 presents the performance of SAMOCP for the classification task on the CIFAR-100C dataset under a gradual shift setting, where each method receives 8 , 550 data points sequentially. It is evident that the performance of previous methods, particularly in terms of prediction set size and singlewidth prediction sets, depends on the learning model employed. The proposed method SAMOCP outperforms existing methods by creating smaller prediction sets, lower regret, and more single-width prediction sets that correctly cover the true label, while also achieving coverage close to the targeted level. It is noteworthy that SAMOCP surpasses every variant of previous methods in these aspects. Furthermore, SAMOCP is faster than SAOCP and SAOCP(MM), despite having the same maximum number of updates at each time t .

In dynamic environments, data distribution does not necessarily shift gradually. Instead, we may encounter abrupt distribution shifts, with significant differences between data distributions in two successive time slots. To demonstrate how SAMOCP behaves in such environments, another experiment was conducted, in which SAMOCP can successfully track these sharp transitions and select a suitable learning model for creating a prediction set. Experimental results on CIFAR-10C are detailed in

Table 2, where the proposed algorithm again outperforms previous methods in terms of prediction set size, regret, and single width prediction sets that accurately cover the true labels while maintaining coverage close to the target value.

To demonstrate that SAMOCP achieves the lowest regret over different intervals compared to existing methods, we illustrate the regret for various interval sizes in Figure 2. For each existing method, there are 4 different regrets corresponding to the 4 learning models used and the lowest regret is depicted. The results show that our method consistently leads to lower regret than the best version of each previous method across different learning models. Note that a lower regret implies that the algorithm adapts faster to changes. The regret calculated over different time intervals indicates the algorithm's adaptivity in capturing the distribution shift at different time scales. Therefore, Figure 2 indicates that the SAMOCP can adapt faster to distribution shifts compared to benchmarks in various time scales. Furthermore, we include experiments on synthetic data and another real dataset, TinyImageNet-C, using different sets of learning models that do not necessarily contain 4 models in the Appendix, Section B. This demonstrates how SAMOCP can rely on a mixture of learning models over the period [ T ] and select the appropriate one for each distribution setting.

100

200

300

400

500

Interval size

0

10

20

30

40

4

SAMOCP

SAOCP(MM)

FACI

ScalefreeOGD

SAOCP

(a) CIFAR-10C

100

200

300

400

500

Interval size

0

10

20

30

40

50

4

SAMOCP

SAOCP(MM)

FACI

ScalefreeOGD

SAOCP

(b) CIFAR-100C

Conclusion

In this study, we introduced a novel conformal prediction algorithm designed for online environments undergoing distribution shifts. Recognizing that the selection of baseline models affects the efficiency of conformal prediction, our algorithm incorporates multiple models simultaneously. For each expert, the contribution of each model is dynamically adjusted based on its time-evolving weight. We demonstrated that our proposed method SAMOCP achieves strongly adaptive regret across any time interval of arbitrary width and maintains valid coverage. Experimental results in environments with both gradual and sudden distribution shifts indicated that our algorithm produces more informative prediction sets and achieves a coverage rate close to the target value, compared to those created by previous methods using their best baseline models.

References

A.N. Angelopoulos, S. Bates, M. Jordan, and J. Malik. Uncertainty sets for image classifiers using conformal prediction. In International Conference on Learning Representations , 2020.

Balasubramanian, S.S. Ho, and V. Vovk. Conformal Prediction for Reliable Machine Learning: Theory, Adaptations, and Applications . Newnes, 2014.

R.F. Barber, E.J. Candès, A. Ramdas, and R.J. Tibshirani. Conformal prediction beyond exchangeability. The Annals of Statistics , 51(2):816-845, 2023.

Besbes, Y. Gur, and A. Zeevi. Non-stationary stochastic optimization. Operations research , 63(5): 1227-1244, 2015.

Bhatnagar, H. Wang, C. Xiong, and Y. Bai. Improved online conformal prediction via strongly adaptive online learning. In International Conference on Machine Learning , pages 2337-2363. PMLR, 2023.

Boström, H. Linusson, T. Löfström, and U. Johansson. Accelerating difficulty estimation for conformal regression forests. Annals of Mathematics and Artificial Intelligence , 81:125-144, 2017.

Cesa-Bianchi, Y. Freund, D. Haussler, D.P. Helmbold, R.E. Schapire, and M.K. Warmuth. How to use expert advice. Journal of the ACM (JACM) , 44(3):427-485, 1997.

Daniely, A. Gonen, and S. Shalev-Shwartz. Strongly adaptive online learning. In International Conference on Machine Learning , pages 1405-1411. PMLR, 2015.

Ding, A. Angelopoulos, S. Bates, M. Jordan, and R.J. Tibshirani. Class-conditional conformal prediction with many classes. In Advances in Neural Information Processing Systems , volume 36, 2024.

John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research , 12(7):2121-2159, 2011.

Gibbs and E. Candes. Adaptive conformal inference under distribution shift. Advances in Neural Information Processing Systems , 34:1660-1672, 2021.

Gibbs and E. Candès. Conformal inference for online prediction with arbitrary distribution shifts. arXiv preprint arXiv:2208.08401 , 2022.

Elad Hazan, Alexander Rakhlin, and Peter Bartlett. Adaptive online gradient descent. In Advances in Neural Information Processing Systems , volume 20, 2007.

He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Hendrycks and T. Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations , 2019.

Huang, Z. Liu, L. Van Der Maaten, and K.Q. Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4700-4708, 2017.

Koenker and G. Bassett. Regression quantiles. Econometrica: journal of the Econometric Society , pages 33-50, 1978.

Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.

Le and X. Yang. Tiny imagenet visual recognition challenge. Technical Report 7, CS 231N, 2015.

Levy, M. Agrawal, A. Satyanarayan, and D. Sontag. Assessing the impact of automated suggestions on decision-making: Domain experts mediate model errors but take less initiative. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems , 2021.

Li, F. Liu, W. Yang, S. Peng, and J. Zhou. A survey of convolutional neural networks: analysis, applications, and prospects. IEEE Transactions on Neural Networks and Learning Systems , 33 (12):6999-7019, 2021.

Littlestone and M.K. Warmuth. The weighted majority algorithm. Information and Computation , 108(2):212-261, 1994.

Orabona and D. Pál. Scale-free online learning. Theoretical Computer Science , 716:50-69, 2018.

Papadopoulos, V. Vovk, and A. Gammerman. Regression conformal prediction with nearest neighbours. Journal of Artificial Intelligence Research , 40:815-840, 2011.

Romano, E. Patterson, and E. Candes. Conformalized quantile regression. In Advances in Neural Information Processing Systems , volume 32, 2019.

Romano, M. Sesia, and E. Candes. Classification with valid and adaptive coverage. Advances in Neural Information Processing Systems , 33:3581-3591, 2020.

Shafer and V. Vovk. A tutorial on conformal prediction. Journal of Machine Learning Research , 9 (3), 2008.

Shi, C.S. Ong, and C. Leckie. Applications of class-conditional conformal predictor in multi-class classification. In 2013 12th International Conference on Machine Learning and Applications , volume 1, pages 235-239. IEEE, 2013.

Straitouri, L. Wang, N. Okati, and M.G. Rodriguez. Improving expert predictions with conformal prediction. In International Conference on Machine Learning , pages 32633-32653. PMLR, 2023.

Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 1-9, 2015.

Tan and Q. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In Proceedings of the International Conference on Machine Learning , pages 6105-6114. PMLR, 2019.

R.J. Tibshirani, R. Foygel Barber, E. Candès, and A. Ramdas. Conformal prediction under covariate shift. In Advances in Neural Information Processing Systems , volume 32, 2019.

Vovk. Cross-conformal predictors. Annals of Mathematics and Artificial Intelligence , 74:9-28, 2015.

Vovk, A. Gammerman, and G. Shafer. Algorithmic learning in a random world , volume 29. Springer, New York, 2005.

V.G. Vovk. A game of prediction with expert advice. In Proceedings of the eighth annual conference on Computational learning theory , pages 51-60, 1995.

Zaffran, O. Féron, Y. Goude, J. Josse, and A. Dieuleveut. Adaptive conformal predictions for time series. In International Conference on Machine Learning , pages 25834-25866. PMLR, 2022.

A Proofs

Lemma 2 For every α ∈ [0 , 1] and learning rate η > 0 , adaptive miss coverage probability α t m for any model m ∈ [ M ] and t ∈ [ T ] is bounded as

A.1 Proof of lemma 2

Based on equation (4), we have

We prove this lemma using a contradiction. Suppose there exists a ˆ t such that α m ˆ t / ∈ [ -η, 1 + η ] , where ˆ t ≥ 2 is the smallest such time index. We first prove the case of violating the upper bound; by contradiction, assume α m ˆ t > 1+ η . According to equation (18), this would necessitate that α m ˆ t -1 ≥ 1 . Given that we assumed ˆ t is the smallest time index to violate the upper bound, it should follow that α m ˆ t -1 ≤ 1 + η . However, α m ˆ t -1 > 1 > ¯ α m ˆ t -1 implies err m ˆ t -1 = 1 . By equation (4) we have

which contradicts our assumption that α m ˆ t -1 > 1 + η . Next, assume α m ˆ t < -η . By equation (18), we have α m ˆ t -1 < 0 . Given that ˆ t is the smallest index that violates the lower bound of the lemma, it must hold that α m ˆ t -1 ≥ -η . Considering α m ˆ t -1 < 0 < ¯ α m ˆ t -1 , we deduce that err m ˆ t -1 = 0 . Therefore, by equation (4), we have

which contradicts our initial assumption that α m ˆ t < -η .

A.2 Proof of Theorem 1, Regret for MOCP:

Algorithm 1 has the following regret bound

Where α m ∗ can be obtained via (7). To prove the theorem, we first introduce and prove the following two lemmas

Lemma 3 for miss coverage probability assigned to any model ˜ m ∈ [ M ] , we have the following bound

where α ˜ m = arg min α ˜ t m ∑ T t =1 L (¯ α ˜ t m , α ˜ t m ) .

Proof: We first begin with

Then define adaptive learning rate η t [Duchi et al., 2011, Hazan et al., 2007] as

So we have

Therefore,

Since the loss function (2) is convex, we have the following inequality

By summing (19) over t ∈ [ T ] we have

where (i) used α t m ∈ [ -η, 1 + η ] by Lemma 2.

Lemma 4 For miss coverage probability assigned to any model ˜ m ∈ [ M ] we have the following bound

Proof: Referring to the definition of ¯ w t m in Subsection 3.1 and defining W t := ∑ M m =1 w t m , we have

where W 1 = 1 , ( i ) follows from the inequality exp( -ϵx ) ≤ 1 -ϵx + ϵ 2 x 2 for | ϵ | ≤ 1 , and ( ii ) follows from 1 + x ≤ e x . On the other hand, we have

where w ¯ m 1 = 1 M . By combining (21) and (22) we have

By taking the logarithm on both sides we have

which leads to

Now, we define the best model in the static environment as

Then, we replace ˜ m with best model m ∗ in Lemma 3 and 4.By setting ϵ = 1 √ T and summing results of two lemmas we have:

A.3 Proof of Theorem 2, Coverage error for SAMOCP:

We first define expected miss coverage error as

The proof of this theorem is based on a grouping argument. So we first divide T into ⌈ T 1 -γ ⌉ group for γ ∈ ( 1 2 , 1) , and write the k th group as

where | G k | ≤ ⌈ T γ ⌉ . We also define a new variable, H mn n : t , assigned to m th update rule of n th expert as follows

So the update rule in (4) can be written as:

For k th group where 2 ≤ k ≤ ⌈ T 1 -γ ⌉ , by using (26) we have

Since at each time t we activate an expert with lifetime as defined in (9), the n th expert will be activated at time t = n . Consequently, ¯ h n t will be 0 for t < n . Therefore, we have

Note that according to (26), H mn n : t ( α mn t -α mn t +1 ) ≤ η . By summing (27) over t ∈ G k we have

For G 1 we have

By summing over all group we have

We define β γ ( T ) as:

Then we have:

A.4 Proof of Theorem 3, static regret for SAMOCP

We can write the regret as

where I ˜ n denotes the time interval during which expert ˜ n is active, starting at time t = ˜ n . The third and fourth terms in the expression are analogous to the regret experienced by expert ˜ n , as established in Theorem 1. To evaluate the regret for the first and second terms, we employ Lemma 4. The main difference is in the number of experts considered: For a looser bound, assuming that experts remain active beyond their designated lifetime, the maximum number of experts at each time step t would be gt . Thus, we derive the following bound for the first and second terms

By summing (33) with regret bound of Theorem 1 we have

Until this point, the regret bound we've established applies solely to intervals that start with expert ˜ n , where each interval's length corresponds to the lifetime of expert ˜ n . However, to derive a regret bound for any arbitrary time interval I , we need to partition the interval into subintervals in a suitable manner. As proposed in [Daniely et al., 2015], we can divide an interval I into two sequences of non-overlapping and consecutive intervals, denoted as ( I -p , ..., I 0 ) and ( I 1 , ..., I q ) , such that | I r +1 | | I r | ≤ 1 2 for all r ∈ (1 , q -1) , and | I r | | I r +1 | ≤ 1 2 for all r ∈ ( -p, -1) . Subsequently, by employing

the inequality ∑ ∞ r =1 √ 2 -r T 0 ≤ 4 √ T 0 , and by replacing ˜ n with n ∗ using (13), we have

Given that A and B are positive constant variables, we have determined the regret bound for our problem, demonstrating sublinear regret for Algorithm 2.

A.5 Proof of Lemma 1, Dynamic regret for SAMOCP:

To prove the regret in a dynamic environment we adopt a method which was first proposed by [Besbes et al., 2015]. So we can write the dynamic regret in our problem as

where α m ∗ n ∗ t is obtained by (16). The first two terms in (36) represent the static regret as defined in Theorem 3 and have been shown to be bounded. Consequently, to establish the overall regret bound, it is sufficient to find the upper bounds for the third and fourth terms in (36). We begin by dividing the total time interval T into sub-intervals I r indexed by r = 1 , ..., [ T/ | I | ] where | I | is the length of each interval. so we can rewrite (36) as

For the second two terms we have

where V ( L ( . ) t ) is the variability of environment [Besbes et al., 2015] defined in (14). So the regret in (36) for any arbitrary | I | will be

Since (35) holds for any interval I ⊆ [ T ] , By selecting | I | = ∣ ∣ ∣ ( T V ( L ( . ) T t =1 ) ) 2 3 ∣ ∣ ∣ we have

B Additional Experiments

B.1 Synthetic Dataset

In this subsection, we present our analysis using synthetic data to compare our proposed method, SAMOCP, with recent adaptive conformal prediction methods designed for dynamic settings. We analyze our experiments with a new set of learning models to demonstrate how our proposed method can effectively utilize a mixture of learning models to cope with distribution shifts in a dynamic environment with unknown distribution changes. Additionally, experiments with synthetic data also confirm that SAMOCP maintains its advantages when varying the number of learning models. Specifically, we conducted experiments for cases with 2 and 3 learning models.

Data Generation: To generate synthetic data that mimics real-world scenarios, we employ two distinct transformation sequences. The first transformation sequence introduces visual noise and blur effects through the application of Gaussian blur and random Gaussian noise. This approach aims to subtly degrade image clarity, simulating real-life challenges such as camera focus issues or atmospheric conditions like fog or mist. The Gaussian blur is applied with moderate settings, while random noise is incorporated to simulate sensor noise or digital compression artifacts commonly encountered in digital imagery. The second transformation sequence focuses on color manipulation. By adjusting image attributes such as brightness, contrast, saturation, and hue in minor increments, we challenge the models to perform reliably under varying lighting conditions and color settings-typical variations that occur due to different times of the day or inconsistencies in camera settings. Additionally, a random conversion of some images to grayscale is employed to further challenge the models' dependency on color information.

In this experiment, two distinct datasets each containing 3000 images are generated from each transformation type. These datasets are designed with a fixed number of 20 classes and hyperparameters ξ and k reg are set to 0 . 1 and 4 , respectively. The variations between the datasets are due to random elements introduced during image processing, such as differences in which pixels are affected by noise or how color properties are altered. This randomness ensures each dataset contains unique instances, even though they stem from the same transformation principles. By concatenating images from the two datasets, the experiment simulates both gradual and sudden distribution shifts. Gradual shifts are seen within the datasets from a single transformation, while sudden shifts occur when switching between datasets from different transformations.

We conducted experiments using a distinct set of learning models. As shown in Table 3, we incorporated two learning models, Efficientnet_b0 and GoogLeNet, where we achieved the smallest prediction set size with coverage close to the target. The SAOCP for GoogLeNet obtained an average width close to our method; however, it is noteworthy that the maximum number of updates in SAOCP is twice that of SAMOCP, demonstrating that our algorithm achieved this result with lower computational costs. Additionally, its regret is almost five times larger than our method's. We also provide synthetic data analysis using another set of learning models consisting of GoogLeNet, DenseNet-121, and EfficientNet-B0 [Tan and Le, 2019], as detailed in Table 4 where our method again was able to construct smaller prediction sets. It should also be noted that, due to severely corrupted data in our synthetic dataset, none of the models were able to produce single width prediction sets that cover true labels.

B.2 TinyImageNet Dataset

Here, we conduct the experiment on a new dataset involving a gradual distribution shift using a new real dataset, TinyImageNet-C, which is a corrupted version of the TinyImageNet [Le and Yang, 2015] dataset that features 200 distinct classes. We have also incorporated a new mixture of learning models-GoogLeNet, DenseNet-121, EfficientNet-B0, and MobileNet-V2 [Li et al., 2021] to demonstrate the performance of our algorithm. In Table 5, we detail our proposed method's comparison with previous methods, where, once again, our algorithm is able to achieve smaller prediction set sizes while maintaining coverage close to the target value of 1 -α . It is reasonable to study prediction sets in situations where we achieve coverage close to the desired level. The SAOCP(MM) method achieves coverage of less than 85% , which is significantly below the target value. Therefore, we do not consider its prediction sets in our comparison. As demonstrated in the table, SAMOCP obtains smaller prediction sets and operates faster than SAOCP. For the TinyImageNet-C dataset, the parameters η , ξ and k reg are set to 0 . 025 , 0 . 01 and 20 , respectively. Other parameters remain consistent with previous experiments. All real datasets are downloaded from the Zenodo repository.

B.3 SAMOCP vs. SAOCP (equal lifetimes)

We compared our method with a specific version of SAOCP where the hyperparameter g in equation (9) is set to 8, giving both SAOCP and SAMOCP experts the same lifetime. We conduct experiments identical to those described in section 4, addressing both sudden and gradual shifts for CIFAR-10C and CIFAR-100C, respectively. Our results are demonstrated in Table 6 for CIFAR-10C and in Table 7 for CIFAR-100C. For both cases, we observe the advantage of utilizing multiple learning models instead of a single model, as our proposed method, SAMOCP, significantly outperforms SAOCP across all four learning models. SAMOCP achieved better results in terms of coverage, average width, average regret, and single width compared to SAOCP.

B.4 SAMOCP vs. MOCP

To validate the advantage of SAMOCP versus MOCP in dynamic environments, we conducted experiments for both sudden and gradual distribution shifts. see Tables 8 and 9. In both tables, SAMOCP outperforms MOCP in terms of Average Width and Single Width metrics.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We clearly point out our contribution in the abstract and introduction.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The result tables (sections 4 and B) demonstrate that our method does not achieve the best coverage and runs slower than single models.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: For each theorem and lemma in section 3.1 and 3.2 we explained our assumptions and provide clear proof in section A.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide explanation of our experiment procedure in section 4 and also provide full explanation of generating synthetic data set in section B.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: A link to the GitHub repository, containing the full implementation of the method described in the paper, is provided in Section 4

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: In sections 4 and B, we determine the values of the hyperparameters.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We did every experiment for 10 different random seeds.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: In section B, we mentioned that we used an NVIDIA RTX A4000 GPU, and demonstrated the run time for each iteration of our algorithm compared to previous algorithms.

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: We do not expect any direct societal impact in our work.

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: In section 4 we cited every datasets we employed.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: In section B we mentioned that real datasets are downloaded from Zenodo respiratory.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers conformal prediction in the online setting with multiple fitted models, which is very interesting and in line with real applications. As it remains an active question regarding how to aggregate multiple models, the proposed approach adopts importance sampling to choose the 'best' model at each step. The performance is justified with comprehensive simulations and comparisons with baseline approaches are presented.\n\nSTRENGTHS:\nThe problem studied in this paper is well-defined and is of interest in the literature. The theoretical guarantee and numerical performance of the proposed approach are well presented.\n\nWEAKNESSES:\nPlease see questions in the following section.\n\nQUESTIONS:\n1. [Choice of weights] The weights defined in (6) follow exponential decay. It would be interesting to explore other choices of the weights: (1) in terms of theoretical guarantee, how will the coverage/accuracy guide the choice of w? (2) choices of w can also be compared via numerical simulations.\n\n2. [Thm 1] Average regret is considered in Thm 1. It remains unknown how will the MOCP approach outperform FACI in terms of the coverage guarantee.\n\n3. [MOCP vs SAMOCP] MOCP would also be capable of the tasks presented in the simulation section, then it could be more informative to compare SAMOCP with MOCP to validate the advantage of SAMOCP.\n\n4. [From adversarial setting to stochastic setting] It would be interesting to study the theoretical performance of the proposed approach under the stochastic setting where observations are i.i.d. to understand the gains of model aggregation. Please see references e.g. (1) Duchi et al. (Adaptive Subgradient Methods for Online Learning and Stochastic Optimization); (2) Hazan et al. (Adaptive Online Gradient Descent); etc.\n\nLIMITATIONS:\nLimitations are discussed in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-27"}

comments: SUMMARY:
This paper proposes an online multimodal conformal prediction method named SAMOCP, developed to address data distribution shifts in dynamic environments. Specifically, the method selects the best model from multiple candidates to create prediction sets via strongly adaptive online learning.

STRENGTHS:
1. The method aims to expand single-modal online conformal prediction to multimodal.
2. The method achieves better efficiency through the selection from multiple models.

WEAKNESSES:
1. This paper appears to be a variant of SAOCP [1], utilizing many techniques from [1] to construct candidate sets with multiple models. By optimizing the weight of each model, the final prediction set is obtained from the best model. However, there is insufficient evidence to validate the superiority of SAMOCP compared to SAOCP, both theoretically and empirically.
2. The authors claim that the method is devised for a multimodal setting. However, both the methodology and experiments seem to involve multiple experts for a unimodal dataset using different deep neural networks, which contradicts the introduction of the method.
3. There are several concerns regarding the theoretical analyses. What is the meaning of Theorems 1 and 3? Are they intended to demonstrate that the pinball loss will increase with the addition of the optimal single model? What is the strength of these theoretical analyses for decision-making in an online multimodal setting?
4. It is suggested to provide a Complexity Analysis of the overall model.

QUESTIONS:
See in the Weaknesses.

LIMITATIONS:
The limitations of the authors have been pointed out in the Conclusion.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-12

comments: SUMMARY:
This paper investigates online conformal prediction within dynamic environments. The authors introduce a novel adaptive conformal prediction framework that leverages multiple candidate models. The proposed algorithm achieves sublinear regret, and its effectiveness is demonstrated through both real and synthetic datasets.

STRENGTHS:
The proposed method is novel, with a clear presentation and comprehensive literature review. It demonstrates good performance compared to benchmark methods.

WEAKNESSES:
The intuition behind the proposed algorithm in the dynamic environment (Section 3.2) is not sufficiently developed.

QUESTIONS:
1. In Algorithm 1, how should \(\epsilon\) be chosen?

2. How does this multi-model approach perform compared to retraining the non-conformal score after each update?

3. How quickly can this method adapt to distribution shifts compared to benchmarks in real data studies? Rapidly capturing distribution shifts and making valid prediction sets is crucial in practical applications.

4. Could the authors provide intuition behind the design of the algorithm in a dynamic environment? Specifically, why are many experts retained when \(t\) is large, despite having a total of \( M\) models?

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes the Strongly Adaptive Multimodal Online Conformal Prediction (SAMOCP) methodology, which constructs adaptive conformal prediction sets by integrating information from multiple learning models in dynamic environments. This is accomplished by creating multiple experts at each time step, where each expert is an online learning algorithm that dynamically updates the weights of each model and has a finite active time interval. The appropriate miscoverage probability at time $t$ for constructing the prediction set in the subsequent step is determined by the miscoverage probability associated with the selected expert among the many active experts.\n\nSTRENGTHS:\n1. The concept of developing a Conformal Prediction framework for dynamic environments using information from multiple learning models is novel. \n\n2. The paper is well-written and well-structured. \n\n3. The performance of the proposed method is analyzed with theoretical justifications and has demonstrated advantages through extensive experiments compared to existing benchmarks.\n\nWEAKNESSES:\n1. As already pointed out in this paper, this new method is much slower compared to other benchmarks. Therefore, it would be beneficial if the authors could conduct a comprehensive computational complexity analysis, in addition to reporting the empirical runtime in the experiments. This would help practitioners understand how well this method can scale to more complex datasets with a larger number of learning models.\n\nQUESTIONS:\n1. Did you try to implement the method on time series data where consecutive data can potentially exhibit continuous distributional shifts? For example, on stock price data, with the response output being 1 if the price goes up and 0 if it goes down.\n\n2. Can you explain how exactly to ‘select one of the miscoverage probabilities ${\\alpha_t^m\\}$ according to the pmf …’? Is it simply selecting the $\\alpha_t^m$ with the highest normalized weight?\n\n3. In the numerical experiments, if my understanding is correct, the sudden shifts setting involves having intact data before a certain threshold and fully corrupted data after that threshold. How exactly does it work for the gradual shifts? For instance, if there are five distinct levels of severity, do you assign all images with level 0 in the first 1/5 of the horizon, level 1 in 1/5-2/5 of the horizon, and so on?\n\n4. How is the number of experts chosen?\n\nMinor:\nThere seems to be a typo in line 134: the miscoverage probability $\\alpha_{t+1}^m$ can be updated via ..., instead of $\\alpha_{t}^m$. \n\nFor completeness and better clarity, I would suggest adding a line in the algorithm between ‘select one miscoverage probability…’ and ‘Observe true $Y_{t+1}$' that says ‘construct prediction set for $Y_{t+1}$ using the selected miscoverage probability.'\n\nLIMITATIONS:\nThe authors addressed the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-17"}