Continuous Mean-Covariance Bandits

Abstract: Existing risk-aware multi-armed bandit models typically focus on risk measures of individual options such as variance. As a result, they cannot be directly applied to important real-world online decision making problems with correlated options. In this paper, we propose a novel Continuous Mean-Covariance Bandit (CMCB) model to explicitly take into account option correlation. Specifically, in CMCB, there is a learner who sequentially chooses weight vectors on given options and observes random feedback according to the decisions. The agent's  objective is to  achieve the best trade-off between reward and risk, measured with option covariance. To capture different reward observation scenarios in practice, we consider three feedback settings, i.e., full-information, semi-bandit and full-bandit feedback. We propose novel algorithms with optimal regrets (within logarithmic factors), and provide matching lower bounds to validate their optimalities. The experimental results also demonstrate the superiority of our algorithms.  To the best of our knowledge, this is the first work that considers option correlation in risk-aware bandits and explicitly quantifies how arbitrary covariance structures impact the learning performance.The novel analytical techniques we developed for exploiting the estimated covariance to build concentration and bounding the risk of selected actions based on sampling strategy properties can likely find applications in other bandit analysis and be of independent interests. 


Reviews and author responses:

 Thank you very much for your time and effort in reviewing our paper!

 Thank you very much for your increased score and effort in reviewing our paper!

The paper presents a novel set of methodologies to deal with the minimization of a regret where the goal is to maximize the revenue and, at the same time, not to have large risk on the reward realization. The authors analyses three interesting setting, the full information one, the semi-bandit information one and the bandit one. Three algorithms have been proposed, analysed an experimented empirically.  I think that the paper is well written, propose a novel framework and some interesting solutions, as well as insights and comments on the provided theoretical results. The experiments are convincing, even if they are on a synthetically generated dataset. I think that overall the paper is a solid contribution to the MAB field.


I would like to have an estimates and/or an analysis of the computational complexity of the optimization procedure required at each time step by the algorithms. Indeed, it is common that algorithms which are optimal in terms of regret requires long processing time in the full feedback setting.

I think that the experiments have been commented in a short and concise way. More insights on what is shown by them could have added a more value to the paper.
Finally, i would have expected the author to also test their algorithm on a real-world dataset about financial assets. This would have further strenghten the validity of the contribution you provided.

Post rebuttal: 
I acknolwledge I read the rebuttal. My ratings stays the same. I do not see any drect societal impact of this work. Maybe it might be used to reduce the risk in the investment, therefore it might provide a positive impact on the field of portfolio optimization.

The focus of this paper is the management of risk in stochastic multi-armed bandits, specifically when the reward distributions of the arms are correlated (as can happen in practical portfolio management scenarios). This problem requires a solution that takes into account covariances. The authors devise solutions under three different forms of feedback, and provide algorithms, lower and upper bounds on the risk-incorporated regret in each case (except for a lower bound in the CMCB-FB setting). They also validate their algorithms empirically on a suite of synthetic problem instances.  The problem is well-motivated. The paper is well-written. The contributions are significant. My only critical comment is that the paper does not test the proposed algorithms on real data sets. Given the setting, it seems conceivable that stock prices or other such time-series data from the financial domain will exhibit the characteristics (correlated signals) that are explicitly being addressed by the paper. It will reinforce the foundational assumption of the paper to show real data on which traditional notions of risk (based on variance alone) are unsatisfactory. Moreover, it would make a compelling case for the paper to demonstrate on the same data set that the proposed approaches indeed deal much better with the correlations.

Post-rebuttal: I appreciate that the authors have provided a new set of results, as requested. The submission becomes stronger; I change my score from 6 to 7.
 I do not see any substantive negative societal impact.

 Thank you very much for your time and effort! We will certainly incorporate these descriptions in our revision.

Overall, this paper introduces continuous mean-covariance bandits, a new variant of risk-aware bandit problem in which the arms are taken from continuous action spaces and share high levels of correlation in between. The paper considers 3 respective types of feedback settings in the model: full-information, semi-bandit and full-bandit, providing respective algorithms for each and deriving corresponding matching upper and lower regret bounds for the first two cases as well as an upper regret bound for the third case under additional assumptions. It is the first to explore the covariance structures on learning performances in risk-aware bandits.     Strengths:
1. The paper is well-written, with most of the sections and contents well-presented. The authors are the first to define and explore the problem of covariance arising in bandits with continuous settings, which finds applications in many settings.

2. All the algorithms in different settings have been clearly presented, with clear descriptions and steps stated clearly. The sketches of proofs provided are adequate for understanding of the overall analysis, and the upper bounds provided by the Upper Confidence Bound-motivated algorithms are order-optimal to the information theoretical lower bounds.  

3. Proper experiments have been conducted, and the outcomes seem to corroborate with the theoretical conclusions.  

Weaknesses:
1. The motivation of the CMCB problem is unclear. Other than brief descriptions of possible application, it is unclear why the covariance factor should be a part of risk-measure. Why is such construction of interest? Are there any prior works justifying definition (1)? 

2. Some terms are not properly explained after their introduction. Exactly what is meant by full-information/semi-bandit/full-bandit feedback? Why are these setups relevant? The upper bounds for CMCB-FI and CMCB-SB seem to be of the same order, and it is not clear how much additional insight CMCB-SB carries compared to the case of FI.

3. The choices of bandit algorithms to solve respective cases seem a bit arbitrary. Is there any motivation for using the MC-Empirical/UCB/ETE algorithms in the three respective cases? Would an adaptation of algorithms built on UCB-principles be just as effective? Better explanations on motivations of algorithmic choice would be more helpful. 

Overall, my recommendation would be ‘5, marginally below threshold’. I’d be happy to revise my score, if the authors address my concerns accordingly.

---------------

I'm now raising my recommendation to '6' as the authors have addressed my questions in their review. It would be ideal if the final version of the paper could contain more detailed descriptions on the motivations of the algorithms. 
 Limitations: One limitation is that the authors have not completed the analysis of CMCB in three respective cases, as the lower bounds for CMCB-FB algorithms remain unknown. 

Societal Impact: Not applicable. The paper is largely algorithmic and centers on theoretical justification of an abstract model. If any, the societal impact would be beneficial as it provides an insightful model for risk-aware decision making in the case of continuous action spaces. 


 Thank you for the explanations. It would be ideal if these descriptions are included into the paper. I would raise my score to '6'. 

 Thank you very much for these helpful suggestions. We appreciate your positive comments.

1 (Computational Complexity)

Since the optimization in our algorithm is a convex program, one can solve it using the standard projected gradient descent method. For our $d$-dimensional simplex, the projections can be computed in $\tilde{O}(d)$ time [Hazan et al. 2007]. Thus, our algorithm can be performed efficiently.

Reference:  
[1] Elad Hazan, Amit Agarwal, and Satyen Kale. Logarithmic regret algorithms for online convex optimization. Machine Learning, 2007.

2 (Experiments)  

(1) (Experiments on Real-world Financial Data)

Following the reviewer's suggestion, we conduct experiments on real-world data, i.e., "US Funds dataset from Yahoo Finance". This is an open dataset on Kaggle, which provides financial data of 1680 ETF funds in 2010-2017. We select five funds and generate a stochastic distribution from their data of returns (since we study a stochastic bandit problem). We set $d=5$, $\rho=1$ and $T=10000$, and each algorithm is run for $50$ independent simulations.

The following tables present the regret results for the full-information, semi-bandit and full-bandit feedback settings, respectively.

|                     | Full-information Feedback|
|:-------------------:|:----------------:|
| MC-Empirical (ours) | 3.63*10^2        |
| OGD  [15]           | 2.08*10^4        |
| LinearFI            | 9.05*10^5        |

|                     | Semi-bandit Feedback |
|:-------------------:|:----------------:|
| MC-UCB (ours)       | 2.31\*10^3       |
| MC-UCB-$\Gamma$     | 3.56\*10^4       |
| OLS-UCB-C [11]      | 6.51\*10^5       |

|                     | Full-bandit Feedback |
|:-------------------:|:----------------:|
| MC-ETE (ours)      | 6.58\*10^5        |
| OGD-ETE            | 8.66\*10^5        |
| LinearFB           | 9.76\*10^5        |



The experimental results on the real-world dataset show that our proposed algorithms enjoy lower regrets than their baselines in all settings considered, which demonstrates the superiority of our algorithms in handling the risk-aware decision making problem with correlated options.

We will incorporate more experimental results on real-world datasets in our revision.


Below we give some insights in our experiments.

For the FI setting, the experimental results show that our MC-Empirical significantly outperforms Online Gradient Descent (which has the same regret order as ours but cannot capture the covariance information). This demonstrates that our sample strategy sufficiently utilize the observed information than conventional gradient descent based policy. 

For the SB setting, our MC-UCB outperforms prior OLS-UCB algorithm [11] which uses a universal covariance upper bound in confidence region. The results demonstrate the tightness of our constructed confidence region which take into account the actual covariance structure.

For the FB setting, our MC-ETE achieves the lowest regret among all compared algorithms, owing to its effective covariance estimators for limited feedback.




 Thank you very much for your time and effort in reviewing our paper. 

1 (Motivation)

(1) (Motivation of Mean-covariance)

The mean-covariance metric was first proposed in a celebrated portfolio selection model, called Markowitz's model [Markowitz 1952], and has been extensively studied for several decades in the fields of finance [Gasser et al. 2017], management science [Chiu et al. 2016] and production economics [Dragicevic et al. 2016]. 

The covariance term models the objective of diversifying the risk incurred due to the correlation structures of options, e.g., stocks in the same sector possibly have similarly trends of ups and downs, and impels the optimal policy to shift from only pursuing high rewards to targeting the best trade-off between rewards and risk. This well characterizes the demands of many risk-sensitive applications, e.g., investment, clinical trials and supply chain management, where the correlation of options is critical to performance. The learners in this case want to not only obtain high returns but also minimize the risk due to option correlations.

In addition, from the perspective of risk-aware bandit literature, the mean-covariance metric is a natural extension of the popular mean-variance problem, e.g., [Sani et al. 2012, Zhu and Tan 2020], from the discrete decision space to continuous decision space.


(2) (Prior Works on Mean-covariance)

There are a number of papers/books in different fields studying and discussing this metric in details, e.g., [Markowitz 2000, Gasser et al. 2017] in finance, [Warmuth et al. 2012] in online optimization, and [Chiu et al. 2016] in supply chain management.



References:  
[1] A. Dragicevic, A. Lobianco, A. Leblois. Forest planning and productivity-risk trade-off through the Markowitz mean-variance model. Forest Policy and Economics, 2016.  
[2] HM Markowitz, GP. Todd. Mean-variance analysis in portfolio choice and capital markets. Wiley, 2000.  
[3] SM. Gasser, M. Rammerstorfer, K. Weinmayer. Markowitz revisited: Social portfolio engineering, European Journal of Operational Research, 2017.  
[4] CH. Chiu, TM. Choi. Supply chain risk analysis with mean-variance models: A technical review. Annals of Operations Research, 2016.  
(The unlisted references can be found in our paper.)

2 (Clarity)  

(1) (Definitions of Three Feedback Settings)

The full-information (FI) setting means that after sampling an action (a d-dimensional weight vector), the leaner can observe all entries of the random reward vector. 

The semi-bandit (SB) setting refers to the case where after sampling a weight vector, the learner can only observe entries of the random reward vector which are placed positive weights.

The full-bandit (FB) setting denotes the case where after sampling a weight vector, the leaner can only observe the weighted sum of the entries of random reward vector.

We presented the formal definition and explanation of FI, SB and FB in the first paragraphs of their corresponding sections, i.e., Sections 4,5,6, respectively. Following the reviewer's suggestions, we will also add explanation after their introduction.

(2) (Relationship of Three Feedback Settings)

The FI, SB and FB settings are three different feedback structures under a common continuous mean-covariance bandit model. They capture different important observation scenarios in practice, which span from seeing rewards of all options, to only receiving rewards of the selected options, to only observing a weighted sum of option rewards. In fact, full-information, semi-bandit and full-bandit are standard terminology in the bandit literature, which represent different degrees of information observability and are widely used in prior works, e.g., [Dani et al. 2007, Degenne and Perchet 2016, Rejwan and Mansour 2020].

(3) (Results/Insights of CMCB-SB Compared to CMCB-FI)

The regrets for the FI and SB settings are different with respect to dimension $d$. The $\tilde{O}(d\sqrt{dT})$ regret bound for the SB setting has a heavier dependence on $d$ than that of $\tilde{O}(\sqrt{T})$ for the FI setting. This matches the fact that the SB setting only reveals  rewards of the queried dimensions, and provides less information than FI in terms of observable dimensions. 

Similarly, the regret lower bound for the SB setting $\Omega(\sqrt{cdT})$ also has additional dependence on $d$ than that of $\Omega(\sqrt{T})$ for the FI setting, which also validates that the SB setting is harder than FI, and the extra hardness comes from the restriction on observable dimensions. In addition, when $c \leq \frac{1}{d}$ in the SB setting, i.e., the observable dimensions can achieve $d$, the feedback structure of SB reduces to FI, and in this case our SB lower bound $\Omega(\sqrt{cdT})$ degenerates to the FI lower bound $\Omega(\sqrt{T})$. Hence, both our upper and lower bounds capture the insight that the SB setting requires more costs on $d$ than the FI setting, due to more limited feedback structure with respect to observable dimensions.

References:  
[1] Dani, Varsha, Thomas P. Hayes, and Sham Kakade. The price of bandit information for online optimization. Advances in Neural Information Processing Systems, 2007.  
[2] Degenne, R. and Perchet, V. Combinatorial semi-bandit with known covariance. Advances in Neural Information Processing Systems, 2016.  
[3] Rejwan, Idan, and Yishay Mansour. Top-k combinatorial bandits with full-bandit feedback. Algorithmic Learning Theory. PMLR, 2020.

3 (Algorithmic Choice)  

(1) (Motivation of Algorithmic Choice)

The choice of algorithmic types depends on the feedback setting.

In the FI setting, since one can observe random rewards in all dimensions (no matter what action $\boldsymbol{w}_t$ we take), the optimal policy is to fully utilizes the observed information and follows the empirical estimates. Thus, we choose MC-Empirical that always selects the best empirical action (in this case, a UCB-type algorithm has the same regret order but a larger constant factor than MC-Empirical). 

The SB setting has a typical bandit feedback type, where only random rewards of  the selected dimensions can be observed. In this case, the empirical-based algorithm may get stuck in well-behaved but sub-optimal actions and miss a potentially better (optimal) action due to the lack of exploration, while a UCB-type algorithm can avoid such over-exploitation problem by giving priority to actions that were rarely sampled before. Hence, for CMCB-SB, we design MC-UCB with a novel and tight confidence region that can adapt to the covariance structure of options. It enables us to full characterize the impacts of option correlations in regret bounds. 

In the FB setting, the feedback structure is severely limited, where one can only observe an aggregate reward (the weighted sum of random rewards) involving an integrated noise of $d$ sub-noises from different dimensions. Yet, we need to estimate the covariance for every pair of the $d$ sub-noises. In this case, the empirical and UCB type algorithms do not work, due to the significant restriction of observable information. Thus, we choose an ETE type algorithm which plays a set of well-designed and pre-determined actions and can effectively estimate the whole covariance matrix. The ETE policy is adopted in many bandit problems with significantly limited observations, e.g., partial monitoring games [Lin et al. 2014] and contextual bandits with volatile arms and submodular reward [Chen et al. 2018]. 

(2) (Comparison to a UCB-based Adaption)

An adaption of algorithms based on UCB principles will suffer higher regrets than our algorithms, and cannot capture the impacts of covariance structures on learning performance in regret bounds as ours.

Specifically, for the FI setting, a UCB-based adaption algorithm will suffer a larger constant factor of regret than our MC-Empirical and perform worse in practice. For the SB setting, a naive UCB-based adaption that uses a loose universal confidence region (rather than a covariance-adapting one as in our MC-UCB) will have a regret bound depending on the loose universal covariance upper bound and cannot capture the covariance information as ours. For the FB setting, to our best knowledge, there is no available covariance estimator for non-fixed actions under such severely limited feedback in Statistics. As a result, a UCB-based adaption cannot effectively estimate the whole covariance matrix and will suffer a linear regret incurred by the covariance-based risk.


References:  
[1] Lin, T., Abrahao, B., Kleinberg, R., Lui, J., & Chen, W. Combinatorial partial monitoring game with linear feedback and its applications. International Conference on Machine Learning, 2014.  
[2] Chen, Lixing, Jie Xu, and Zhuo Lu. Contextual combinatorial multi-armed bandits with volatile arms and submodular reward. Advances in Neural Information Processing Systems, 2018.

 Thank you very much for your time and effort in handling our paper. We appreciate your positive comments. 

Following the reviewer's suggestion, we conduct experiments on real-world data, i.e., "US Funds dataset from Yahoo Finance". This is an open dataset on Kaggle, which provides financial data of 1680 ETF funds in 2010-2017. We select five funds and generate a stochastic distribution from their data of returns (since we study a stochastic bandit problem). We set $d=5$, $\rho=1$ and $T=10000$, and each algorithm is run for $50$ independent simulations.

The following tables present the regret results for the full-information, semi-bandit and full-bandit feedback settings, respectively.

|                     | Full-information Feedback|
|:-------------------:|:----------------:|
| MC-Empirical (ours) | 3.63*10^2        |
| OGD  [15]           | 2.08*10^4        |
| LinearFI            | 9.05*10^5        |

|                     | Semi-bandit Feedback   |
|:-------------------:|:----------------:|
| MC-UCB (ours)       | 2.31\*10^3       |
| MC-UCB-$\Gamma$     | 3.56\*10^4       |
| OLS-UCB-C [11]      | 6.51\*10^5       |

|                     | Full-bandit Feedback |
|:-------------------:|:----------------:|
| MC-ETE (ours)      | 6.58\*10^5        |
| OGD-ETE            | 8.66\*10^5       |
| LinearFB           | 9.76\*10^5        |


The experimental results on the real-world dataset show that our proposed algorithms enjoy lower regrets than their baselines in all settings considered, which demonstrates the superiority of our algorithms in handling the risk-aware decision making problems with correlated options.

We will incorporate more experimental results on real-world datasets in our revision.

Meta-review: The paper considers the problem of regret minimization on bandits with continuous arms. The key idea is to use the correlation between arms. It shows matching upper and lower regret bounds for full information and semi-bandit settings, and upper bound for the full bandit setting.

Three expert reviewers considered the strengths and weaknesses of the paper. In the author rebuttal, the authors additionally provided evidence on a non-synthetic dataset, in response to two reviewers' comments. The authors engaged with reviewer questions, and the reviewers appreciated this engagement, with two scores increasing. A brief discussion post rebuttal confirmed that all reviewers thought that the paper is novel and provides interesting results. Therefore it is great pleasure for me to recommend that the paper is accepted for publication at NeurIPS.

Decision: accept