Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper introduces a new method for studying the value of hedging on uncertain variables in decision-making problems. This is done by augmenting the reward with the value of the hedge. That way, gains/losses associated with a particular hedge can be computed so that alternative scenarios such as no hedging or hedging on another uncertainty can be evaluated and compared.
strengths_and_weaknesses: ### Strengths
* The paper is clearly written and gives many examples illustrating the method and showing the different cases that may occur: hedging for a fixed policy, hedging with policy re-optimization.

* It also provides theoretical results showing how the calculations for the VoPH when the rewards are unknown can be done efficiently using graphical models and BN inference methods.

Overall, I think this is a strong paper that will be valuable for the community. A list of my main concerns is provided below. I would like the authors to comment on them in their rebuttal and revise the paper if needed. 

### Weaknesses

* The paper seems to assume that the reader is familiar with concepts from finance such as hedging, risk aversion, or utility functions. Given that this is an ML conference it would be good to introduce all these more carefully (see my comment under Questions for more details). The authors may save some space by shortening section 3.2 introducing BNs and IDs which should be familiar concepts for the ML audience. 

* The terminology is confusing at times, the paper uses the terms reward, value and utility interchangeably.

* There is a big gap in the related work section. The paper ignores all the works on reward shaping in RL. I will not make a list of papers here to avoid favoritism but a simple search for "reward shaping in RL" should give the authors plenty of references. Although the objectives in most of these papers (e.g. encourage exploration (intrinsic motivation) or ease learning) are different in nature from the objective in this paper (studying the value of hedging), the means to accomplish these goals are similar. In fact, it is difficult to assess how different this paper is from all these works without the proper context.

* The paper also fails to discuss the assumptions and limitations of the method. Such as the availability of a graphical model of the problem. Full knowledge of the dynamics is a strong assumption that should be mentioned in the paper.


questions: Aside from addressing the concerns listed under weaknesses I would like the authors to answer the following questions:

1. As mentioned under weaknesses, a proper description of certain financial concepts is much needed. For instance:

     * How should the utility function be chosen? Is this given? How do the different utility functions affect the final policy/decision? To be more specific, I found Section 3.1 hard to follow. I suggest the authors expand it with examples or intuitions illustrating these concepts.

     * The notion of certain equivalents is also unclear. Can the authors give an intuition on why it is the inverse of the utility function? 

     * The last sentence of the second paragraph in Section 4 is confusing: "The agent’s certain equivalent (CE) for their (decision) situation (represented by the ID/BN) is the certain amount, in units of value V, at which they are indifferent between this amount and their situation."

2. The paper discusses how to compute VoPH for a particular policy when the rewards are unknown (Theorem 10). However, equation 4 includes an inner maximization to find the optimal policy, which itself depends on the augmented reward. Can the authors explain how equation 4 can be solved? 


limitations: N/A
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper studies hedging within the framework of causal influence diagrams (CID). Roughly speaking, a hedge is an opportunity for a risk-averse agent to increase its expected utility, by creating anti-correlated sources of value. This paper formalises several concepts relevant to hedging within the CID-framework, and proves a number of theorems that follow from these definitions. These theorems can then be used to identify potential hedges in an uncertain decision situation, or reason about the incentives of an agent. This paper builds on recent work on CIDs, as developed by Tom Everitt and others.
strengths_and_weaknesses: Originality:

There is, to the best of my knowledge, no previous work studying hedging within a similar framework.


Quality:

The presented results are of good quality. The maths is sound, and the experiments serve their purpose.


Clarity:

I found the presentation and exposition in this paper to be exceptionally clear.


Significance:

My impression is that the contribution made by this paper is somewhat marginal. It studies a relevant problem from a promising direction, but the results themselves could be more substantial. The first two theorems (Theorem 6 and 9) are fairly straightforward. Theorem 10 seems to be more substantive, but its implications are not discussed in the paper in any great detail. Of course, for a paper like this, the definitions and framework constitute a contribution by themselves, but it is not clear to me that this is enough to make the paper impactful in this case. I would have liked to see a more complete treatment of the problem, either with more theoretical results, or with case-studies that demonstrate how these results could help us to solve or better understand some class of problems (be they practical or theoretical). Alternatively, it would also suffice to provide a compelling argument for why such extensions would be very difficult to provide (thus arguing that there is a sense in which the provided results are complete).


questions: I have the following questions:
- What is the main (practical or theoretical) significance of the results presented in the paper? 
- What problems can the presented results help us to solve, or gain more insight into (as compared to a naive analysis)? 
- What further theoretical results or extensions do you anticipate that it will be possible to build on the results you have provided?
- Is there a theoretical reason to expect the provided results to be exhaustive (in some relevant sense)?

I believe that a better understanding of these points could compel me to increase my rating.

limitations: I find that the authors have addressed the potential negative societal impact of their work in a fully satisfactory way. Their discussion of the limitations of their work is also adequate (although dispersed throughout the text).
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors propose a modified Bayesian Network formalism incorporating the hedging information that can be useful in the decision making process.
strengths_and_weaknesses: The main weakness I see is an insufficient contextualization of the problem. The Markov Decision Process, the control as probabilistic inference, the risk-aversion Reinforcement Learning, etc. are neither mentioned nor discussed.

There isn't presented any alternative approach to solve the problems presented and so any comparison is missing.
questions: Fig. 1 - Some variables are not defined. This is a problem present in many figure where the caption is not so informative. 

row 90 --> what is the meaning of "numeraire"?

row 122  --> $\mathbf{V_k}$ should be substituted by $V_k$ since we refer to a particular Value node.

row 131 --> what is the alternative set?

limitations: There isn't a mention neither to the Markov Decision Process nor to the Control as Probabilistic Inference (I will not make a list of papers here to avoid favoritism but a simple search for "Control as Probabilistic Inference" should give the authors plenty of references). 
The difference with these approaches should be described because it is not clear what is the advantage of the proposed approach. 

The Reinforcement Learning (RL) literature on this topic is not considered (e.g. risk-aversion RL). RL is cited only regarding the risk in the reward manipulation.
review_title: null

Meta-review: metareview: This paper proposes a decision-theoretic view of hedging within the framework of probabilistic graphical models augmented with a reward.
After reading each other's reviews and the authors' feedback, the reviewers have solved most of their concerns and agree that the paper deserves publication.
However, the authors need to seriously consider the reviewers' suggestions for making their paper clearer in the camera-ready version.
confidence: Less certain
Decision: Accept