Hedging as Reward Augmentation in Probabilistic Graphical Models
Authors: Debarun Bhattacharjya
Radu Marinescu
Venue: NeurIPS 2022 Accept (2022)
Keywords: hedging
influence diagram
bayesian network
reward augmentation

Abstract: Most people associate the term `hedging' exclusively with financial applications, particularly the use of financial derivatives. We argue that hedging is an activity that human and machine agents should engage in more broadly, even when the agent's value is not necessarily in monetary units. In this paper, we propose a decision-theoretic view of hedging based on augmenting a probabilistic graphical model -- specifically a Bayesian network or an influence diagram -- with a reward. Hedging is therefore posed as a particular kind of graph manipulation, and can be viewed as analogous to control/intervention and information gathering related analysis. Effective hedging occurs when a risk-averse agent finds opportunity to balance uncertain rewards in their current situation. We illustrate the concepts with examples and counter-examples, and conduct experiments to demonstrate the properties and applicability of the proposed computational tools that enable agents to proactively identify potential hedging opportunities in real-world situations.

Hedging as Reward Augmentation in Probabilistic Graphical Models
Debarun Bhattacharjya
Radu Marinescu
IBM Research debarunblus ibm. com
IBM Research
radu. www.ummiinescccie.....m. ibm. com
Abstract
Most people associate the term 'hedging' exclusively with financial applications, particularly the use financial derivatives. We argue that hedging is an activity that human and machine agents should engage in more broadly, even when the agent's value is not necessarily in monetary units. In this paper, we propose a deeision-theoretic view of hedging based on augmenting probabilistic graphical model specifically Bayesian network or an influence diagram with reward. Hedging therefore posed as particular kind of graph manipulation, and can be viewed as analogous to control/intervention and information gathering related analysis. Effective hedging occurs when risk-averse agent finds opportunity balance uncertain rewards in their current situation. We illustrate the concepts with examples and counter-examples, and conduct experiments demonstrate the properties and applicability of the proposed computational tools that enable agents to proactively identify potential hedging opportunities real-world situations.
Introduction
Hedging is mostly synonymous with the financial industry. Wikipedia defines (financial) hedge as "an investment position intended to offset potential losses gains that may be incurred by companion investment." Various relevant financial instruments are noted, including stocks, exchange- traded funds, insurance, forward contracts, swaps, options, etc. The notion of a hedge is however also occasionally used in other contexts; for instance, a hedge linguistics refers to phrase that signifies caution uncertainty, or has intended effect of dampening the strength of claim. The fundamental hedging idea that is so widely applied in finance can however be generalized. Informally, hedging can be viewed minimizing the 'risk" that arises from uncertainty in existing potential gains/losses along some measure of value. It may be possible and indeed prudent for a risk-averse decision maker to try to balance the uncertainty in their existing situation in some fashion.
Behavioral experiments indicate that people are occasionally risk-secking while making decisions, particularly when losses are involved (22). However, such behavior is often associated with contextual cognitive biases that they are typically unaware [26] when they truly understand the implications of risk-seeking behavior and its meaning in utility theory opposed common parlance, most people are quick to reject it and adopt lisk-averse behavior instead. Thus, from prescriptive perspective, most real-world situations should involve risk-averse decision makers, and therefore there can be real practical benefits around investigating hedging opportumities. Indeed, we argue that hedging is an activity that human and machine agents should engage in more broadly, in the appropriate situations, regardless of whether the agent's value is monetary units.
Probabilistic graphical models such Bayesian networks (35 and influence diagrams 21 effective at representing wide variety real-world uncertain situations where the latter model explicitly represents decisions by decision maker and they are powerful modeling paradigms in practice since they can be learned through a combination of expert opinion and data [17]. In
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

D
(a)
(b)
(c)
Figure I: Graphical manipulations in an illustrative 5 node influence diagram, where X is an uncer- tainty (random variable) of interest. D is decision and V represents reward/value: (a) Information X observed before making decision D. (b) Control Intervening on through decision D'. (c) Hedging Augmenting reward V to X's states.
this paper, we propose a general decision-thcoreti approach and associated perspective for hedging related valuations, showcasing the required computations in real-world situations that are represented as probabilistic graphical models. The primary focus is around formalizing hedging as an analysis tool for probabilistic graphical models in general, where decision maker can study potential benefits of hedging on uncertainties in their decision situation so as to explore opportunities for creating value by reducing their risk.
Contributions. Our contributions are summarized as follows:
We pose hedging as reward augmentation in decision maker's situation as represented by Bayesian network or influence diagram, and demonstrate how to evaluate hedging opportunities given model. This view of hedging graph manipulation is analogous to information gathering intervening in a decision situation.
We highlight number of results around valuing hedging, using several examples to illustrate the concepts. Some results in particular could be helpful for agents to proactively seek out hedging as way to improve their situation.
We conduct experiments to demonstrate applications of hedging related computations as applied to probabilistic graphical models.
Note that in this work, the model is assumed to be known a-priori, possibly through a combination of statistical learning and/or causal discovery from data as well as assessments from domain experts and/or the decision maker(s). This makes the proposed analyses techniques indifferent to how models are learned and/or elicited.
2 Related Work
Hedging in Finance. Since hedging is an activity that is centuries old, believed to have originated with futures market for rice in Japan in the late 17th century, much has obviously been written about financial hedging. This includes literature around the extent to which hedging is used by corporations [15] and the degree to which corporate hedging is effective 33 Instruments for hedging are studied in basic texts on finance [28). and machine learning algorithms are widely adopted by the financial industry in general, including for hedging (5) 6. Although some of this literature in economics and finance is tangentially relevant to our work, it does not consider hedging in the context of probabilistic graphical models for representing uncertain situations.
Graph Manipulations in Bayesian Networks Influence Diagrams. The closest related work is around graph manipulations in Bayesian networks (both acausal and causal) and influence diagrams. In the influence diagram literature, this includes studying the benefits from information gathering, i.c. observing uncertainties before making decision, and having the ability to control uncertainties and set them to desired states. The former manipulation is shown in the illustrative 5-node influence diagram in Figure Tax [a), where informational are is added from uncertainty to decision D (the uncertainty must not be downstream of the decision in general, to avoid cycles). This informs the value of information about X prior to deciding 20 There is significant literature on this subject,
including around efficient computation (4) 24 The latter manipulation, shown in Figure ((b) where
associated with the value of control [31 27 30 41. This is particularly popular in causal networks, forming the
ares from X's parents are removed and augmented decision node /'' sets the state of X,

---

basis for interventions in Pearl's do-calculus [36 well as work that studies such causal semantics through a decision- -heisiionthhooretic lens [8] 8 Hedging has been previously formulated as a buying price in prior work (39 4 here we expand upon this work by extending this notion to general value measures (not just monetary units), which vastly increases applicability of the concepts and tools in AI Furthermore, we position hedging as a new kind of graph manipulation augmenting a reward to some uncertainty X analogous to information and control, as illustrated in Figure [c) and described later in Section 4
Other Related Work (Reward Shaping, AT Safety, Robust MI. etc.) Since we frame hedging as reward augmentation, we briefly note some other related work, particularly around the implications of manipulating rewards as it pertains to reinforcement learning:
There is long line of research reward shaping, which is method where additional rewards are provided to guide a learning agent [T, 29] 34]. The goal in reward shaping is to make easier to learn using supplemental rewards; in contrast, we frame hedging as an attempt to explore opportunities for minimizing risk as suitable for the decision maker in an already specified model. There may also be unintended and potentially harmful behavior emerging from poor design of real- world artificial intelligence systems, as explored in the literature on Al safety [ 255, One of the concerns is around reward hacking/tamperingg/ htt.r//arrnngggnnngoorrrrrrn gaming/corruption. such as where agent finds a way to give itself high reward without completing its intended goals 2 19] There is ongoing
work on methods alleviate negative side effects of misspecified reward functions [16] [2]
We note that hedging is pertinent for risk-averse decision makers and the subject of risk sensitivity has long been of interest reinforcement learning [32] 37,, with more recent application to problem
settings where safe policies are desired [4].
Hedging-like strategies can also be found in robust machine learning settings including decision-
theoretic online learning [13
3 Value, Utility and Graphical Models
We use the framework of probabilistic graphical models - in particular, Bayesian networks and influence diagrams. Since the notions of value/reward and utility are crucial define hedging, we begin with some relevant background.
3.1 Value/Reward and Utility Functions
Preference modeling is widely studied in AI, economics, management science, behavioral decision making, marketing and related fields. In this paper, we follow a preference modeling approach from decision analysis that has also been used for valuing information (0 40 4. Specifically, we assume the decision maker's 's preferences are decomposed into measurable value function v(.) [1 23
over domain R. as well as a one-dimensional utility function u(.) : IR + ER that is monotonic over the value measure 43 to(-) is said to be 'riskless' because it captures strength of preference for situations with no uncertainty, whereas u(-) captures the decision maker's attitude towards risk, i.e. lotteries that involve uncertainty in the value measure.
The main motivation for using measurable value functions that enables one to work with concepts that involve magnitudes. For example, an ordinal measure which only provides rankings and no other quantitative implications may be insufficient for modeling measures of interest to a decision maker, such as monetary units, lives saved, number of customers, quality-adjuster life years, lead time, etc. Crucially for our work, the preference decomposition ensures hedging valuations are in units that are meaningful to the decision maker.
The decision maker's risk aversion coefficient the ratio of the second derivative and first derivative of (w)) 3 The concavity (convexity) of the utility function provides an indication of the decision maker's 's risk aversion (risk seeking behavior). Some parametric forms for utility functions are popular in practice due to their computational properties. The (monotonically increasing) exponential utility function follows b (p) exp (-x/p), where and b > 0 are
the utility function u(-): 7(v) ("()))
Readdrs familiar with reinforcement learning (RL) should view this as a 'reward function' and not what is typically know as 'value function'' in RL. Here we use these terms interchangeably and often use them both together through the term 'value/reward" means of clarification.
3

---

Effects
Game Outcome
V
V
Game Choice #1
O.P.
Toss
Flip
f
92: indpencentt
A.P.
Gold
Effects
Toss
Choice #2
V
Game Outcome N2
Supplier
Effects
(e)
(b)
Figure 2: Examples of Bayesian networks (BNs) and influence diagrams (IDs): (a) Oil wildcatter ID with a potential augmented reward on airline price (A.P.) [3)]. (b) BN for (two versions of) the Game example. v1: Flipped second game; v2: An independent coin toss for second game. Hedging explored on Game Outcome #2. (c) Gold mining BN with two potential augmented rewards [42
constants and sgn(p) is the sign of the risk tolerance 00. The power utility function is of the form + b sgg(()) sgn(2) ()) |x]"", with constants and > O, where sgn(a) and sgn(x) are the signs of a * 0 and (38). The power function exhibits discontinuous behavior around 0 for < 0 [44). The limiting cases for the exponential and power functions are the linear and logarithmic functions, respectively. Many of our results apply only for linear and exponential utility functions, because they are the only ones that satisfy constant risk aversion, i.e. where ((v) a constant 3. Note that a linear utility function signifies risk neutrality, which is the most widely used assumption in many applications (sometimes implicitly), for instance most (but not all) models for reinforcement learning. Exponential utility functions also popular; the risk tolerance p in these utility functions is the reciprocal of the (constant) risk aversion coefficient 7
3.2 Bayesian Networks and Influence Diagrams
Bayesian network (henceforth BN) B represents probabilistic dependencies between random variables through directed acyclic graph. It consists of graph with node for each uncertain variable, along with a conditional probability distribution (CPD) for each node. We denote uncertain nodes as X (X1.... X NN) }. such that each X; has domain Dom(X:) and is associated with CPD P(X, Pa( x))) where Pa(X;) are the parents of X; in
While it is not traditional include additional types of nodes in BN, we also include value/reward nodes since hedging requires value. Thus, we assume that the BN also includes value nodes V (Vi,... Vx} which are sinks and only have parents from X. Each value node V2 is assumed to be a scalar in R, and associated with a value/reward function (.(()) ) Dom (Pa((P)))))))))))) - RR The total value is aggregated function over the individual value nodes, V a(V). Summation and multiplication are common modes of aggregation. Most of the results presented here apply for additive value aggregation. In many applications (including but not limited to finance), V is in monetary units, which is a natural measure by which many real-world decision situations are valued, but other measures also possible and indeed used in examples in this paper.
Even though there are no explicit decisions for the decision maker in this Bayesian network model augmented with value nodes, we still refer to their situation simplicity but also in recognition the fact that there could be potential future opportunities that the decision maker may wish to include explicitly as decisions.
decision situation. This is both for
influence diagram (henceforth ID) I is a graphical model of decision maker's decision situation under uncertainty and can be viewed as an extension and generalization of a BN. It can represent sequential decision situations such as Markov decision processes (MDPs) and partially observable MDPs (POMDPs). Along with uncertainties X' and value nodes V. they include decisions The domain of decision Dom(D, is the set of alternatives (also known as actions) that the decision maker take. Ares into decision nodes have special semanties they
D (D....., Dar).

---

are informational ares, indicating that the parents of the decision have been observed before it is made. Ares into other nodes are probabilistic or deterministic, like in BNs. We make the standard assumptions: total ordering of decisions as well as no forgetting, i.e. parents for a decision are also parents of subsequent decisions. Figure 2 shows examples of BNs and IDs, with decisions (rectangles), uncertainties (ovals concentric if deterministic) and values/rewards (diamonds).
Similar to what was described for BNs, the value nodes of an ID are aggregated into total value, V = a(V). The decision maker's preferences under risk are modeled using one-dimensional utility function u(-), which is applied on measurable value V with domain R and is denoted by f(V) - u(a (a((((V)) (V)). A policy for I is a list of decision rules II = (II1,... ..l....... II
where each rule II; is associated with a decision D, and is mapping from the domain of its parents to an alternative in the domain of D,, II, d( (P))) ). such that d(-)
(om((P((D))) - Dom (D;). The optimal policy of Z maximizes the expected utility, II* = argmax11 Ex (x,n(/())). An ID a fixed policy can be drawn in policy form, also known as policy diagram (... where decisions have a deterministic or probabilistic relationship with its parents, thus effectively converting the IDD to BN.
4 Hedging as Reward Augmentation
We describe general notion of hedging, posed as reward augmentation in probabilistic model. We will generally refer to the model as an ID, calling out situations where there are no decision nodes.
Consider an agent whose decision situation is represented by an ID I with uncertain nodes X, value nodes V. and optionally decision nodes D. When there are no decision nodes, this BN B. Since an ID in policy form is equivalent to a BN, all its corresponding results apply to BNs as well. The agent's s certain equivalent (CE) for their (decision) situation (represented by the ID/BN) the certain amount, in units of value V, at which they are indifferent between this amount and their situation. For instance, if the agent has utility function u(-) and faces two potential outcomes of value wl (with probability p) and v2 (with probability 1 p) respectively, then the following must hold: (CCE) = p * u(v1)= + (1 - p) u(v2), implying that C'E = u "p=(p x w(vl) + (1 1 p) u(r2)).
We first define a decision maker's s prior certain equivalent, i.e. without any augmentation to their prior situation:
Definition 1. For an agent with BN B, their prior certain equivalent C. = = (Fx((())))))
Definition 2. For an agent with ID L, their prior certain equivalent at policy IT CF. (Ex,x[((( Their (optimal) prior certain equivalent is C.E = uu" = (f
The key difference between the two CE expressions for IDs is that one is at fixed policy II whereas the other involves an optimization over policies. We now formalize a hedge as reward augmentation:
Definition hedge in a BN or ID is an augmented reward r'(:) for uncertainties Xs C X. This is deterministic function that introduces new value node V with parents X in the agent's 's situation, s.I. V" r' s)).. ). hedge is simple if the associated uncertainty set X x is a single node in the graph. It is balanced for additive value aggregation if Ely' s)] 0, and for multiplicative
value aggregation if sX)) = =1
Example 1 (Oil Wildcatter). Consider an oil wildcatter whose decision situation is represented by the ID in Figure 2 a) 39 Their profit depends their drilling decision (D), oil price (O.P.) and amount of oil (A) in the reservoir. The wildcatter has also included an uncertain node for airline stock price (A.P.), since it is negatively correlated with the price of crude oil. Augmenting their situation with a reward that is a deterministic function of any of the three uncertain nodes would result in a simple hedge. Hedging on node A.P. a classic case of corporate financial hedging.
4.1 The Value of a Hedge
There is a natural important question to address: how much does a hedge balance out the uncertainty in the rewards from the current situation, for a feciisoo-thhooeeie decision maker? The hedging value of the augmented reward is defined as the additional improvement the agent's certain equivalent from including it, beyond the certain equivalent of the augmented reward alone. We distinguish between the case for when an ID is fixed at policy vs. when re-optimization is allowed, post reward augmentation. The former case is identical to that for a BN. Formally, they are defined as:
5

---

Definition 4. The value of hedging of augmented reward r' over nodes X at policy for additive value aggregation is:
-
(I)
where V is the new value node from the augmented reward and C = u (Exn((())))
Definition 5. The value of hedging of augmented reward r') over nodes X for additive value aggregation is:
VoH ( (x, V')) - FFg u (max Ex,n(
(2)
where V is the new value node from the augmented reward and CEg u- (maxu Ex,ulf(V))-
Remark. The reason for including the third term in both VoH definitions is to distinguish the value of the hedge from the incremental value of the reward augmentation alone, which would only include the first two terms. This can help reveal the specific effect that probabilistic dependence between rewards has on the value to the decision maker, beyond the value from the augmented reward by itself. When only the first two terms are included, we refer to the computation as value of augmented reward (V AR). Also, for multiplicative value aggregation, we replace subtraction with division. Note that VoH can be positive or negative or zero
We illustrate the value of hedging with the following examples. For BN, both versions of VoH (V "!!" and V *)H*) are identical since there is no policy to optimize.
Example (The Game). Consider a deal between two players who will toss a coin to determine game play: either fencing or golf. The loser must pay $1000 the winner. Suppose both players believe that the probabilities of player#1 winning at fencing and golf are 0.9 and 0.3 respectively. Suppose also that both players are risk-averse with exponential utility functions, with the same risk tolerance p = $3000. The players' (prior) certain equivalents are $35.72 and - -$300.63 respectively. Note that if the players were risk-neutral, their certain equivalents would be $200 and -200 respectively; since they are risk-averse, their C'Es are lower. Computational details for all examples are provided in Appendix [
Figure 2 b) (v1) models this situation from either player's 's perspective as a BN, where concentric ovals indicate deterministic relation. This figure also includes the possibility of playing second game of the other type. Thus, players would play both types of games regardless of the coin flip outcome. What is the value of hedging of the second game outcome to both players, where the payoffs are the same as the first one? For 1, VoHl can be computed as $130.3; for #2ayer this is $100.22 (see Appendix B.I V oH is positive for both players in this example, implying that the dependence between the games is helping balance the uncertainty for both.
The value of hedging a fixed policy has the following desirable properties for utility functions that satisfy constant risk aversion (linear and exponential):
Theorem 6. For additive value function aggregation:
i For risk-neutral decision maker. VoH for any augmented reward is O.
ii For an exponential utility function, if X and value nodes V are independent under policy II, IlHlll for any augmented reward is 0.
Example 3 (The Game). Consider Figure 2b) (v2) where the players are considering playing
second game, but this time they will flip the coin again, therefore the are between game choices is absent. Here the value of hedging of the second game outcome is 0 for both players, this is $(-70) 701.2112 ..6 + 350.63 + 350.63) = $0 (see Appendix B. Given that 2 has already struck deal to play the first game, they would rather not play again in either of the versions in Figure bb)) - but if they had to choose, they would choose the hedge from vl and they would be willing pay up the Voll of $100.22 (-701.26 vs. -601.04). This is the hedging benefit the player- they would rather play both games than their weaker game (fencing) twice!
for player#
The afore-mentioned results around zero hedging value for resk-nlutral decision maker or for uncertainties that are irrelevant to value are perhaps consistent with the reader's intuitive understanding for how hedging should work. However, these results do not extend in general to other cases, such as for other utility functions, as the next example illustrates, or when the policy can be re-optimized.

---

Example 4 (The Game). We revisit the version of the game in Figure 2 b) with a second independent coin toss (v2). Suppose player #1 has a logarithmic utility function of the form u(v) = log(U + w). where w is their initial wealth, assumed to be $5000. Here VoH = $(283.15 18 - 101.7 - 101.7) = $79.78 (see Appendix B.T). We consider this be a counter-example: even though the second coin toss is independent from the first and associated uncertain rewards, there is still value from hedging
due to 'wealth effects' arising from utility functions that do not satisfy constant risk aversion.
4.2 The Value of a Perfect Hedge
Thus far, we have described situations where the agent is considering hedges where the rewards based some uncertainties are known a-priori. While such potential rewards may be known in some applications, there are other situations where this may not be the case. How can an agent with graphical model to represent their decision situation pro-actively seek out hedging opportunities without knowing the actual rewards? The notion of a perfect hedge is formalized, again both at a fixed policy and when re-optimization is allowed.
Definition 7. The value of a perfect hedge on uncertainty X policy II for additive value aggregation is the simple and balanced augmented reward on X that optimizes the increase in certain equivalent at policy II, with and w/o the hedge:
VoPH = max (((ER(X)
(E='(XxI((O)VV1) - -C
(3)
where V is the new value node from the balanced hedge, R(X) denotes the set of balanced rewards for X, and CE =u- (Ex (f)((.
Definition 8. The value of perfect hedge for an uncertainty X that is not downstream of any decision, for additive value aggregation, the simple and balanced augmented reward on X that optimizes the increase in certain equivalent, with and w/o the hedge:
max VoPH ((()E((X)
(u''max ([ -CE.
(4)
where V is the new value node from the balanced hedge, R(X) denotes the set of balanced rewards for X, and CF2 = u "mmxx (xV))
Recall that balanced reward for additive value aggregation is one whose expected value is zero Thus, for additive aggregation, the perfect hedge is the one with zero expected reward that maximizes the certain equivalent, both at policy and otherwise, given this hedge.
Remark. A perfect hedge is intended for when the agent is trying to identify which uncertainties
in their situation are most amenable for hedging. We make a few observations: 1) Unlike in the definition(s) for Voll, VoP does not depend on r(-) by design, since it is unknown; 2) The third term from the Voll definition(s) is discarded here since the focus is on incremental value from the best possible balanced hedge; 3) When re-optimization is allowed, it may be possible for a decision to have an impact on downstream uncertainty. i.c. different policies may change the distribution of the augmented reward V". Thus we add the caveat for VoPH H" to disallow any invalid uncertainty.
The following intuitive results around VoPH H hold for some important and practical special cases:
Theorem 9. For additive value function aggregation:
i For a risk-neutral decision maker and any valid uncertainty X, VoPH = 0 and VoPH"(() = for any policy II. ii For any valid uncertainty X, VoPH"(() > 0 and VoPH Z 0, A 11.
üi For a risk-averse decision maker with an exponential utility function, and any uncertainty X, the value of any hedge r(-) on X at policy II is bounded above by V (X) Ex,nlV'], where V is the new value node from the hedge.
The last result above is important as it can help the agent reject any potential hedges based on VoPH. This is analogous to value of information analysis where some information gathering sources can be rejected immediately if they cost more than the value of perfect information, i.e. the additional value to decision maker when uncertain variable is observed before making decision (20).
7

---

Example 5 (Oil Wildcatter). Let us revisit the oil wildcatter example in Figure 2 a) with model parameters as provided in Appendix B.2 33 With these numbers, is optimal for the oil wildcatter to drill a-priori, with certain equívalent of $38.99M. If we fix the policy at drill (d), the value of perfect hedging for each uncertainty can be computed: VoPH = $7 686, (oPH((OP) = $3. $3.26M, (oPH)(AP) = $1.61M. Hedging on the amount of oil seems comparatively most lucrative, which might be realizable by finding a dependent reservoir nearby, preferably one with a common source with the current reservoir. Hedging on airline price could occur through purchase of airline stocks.
VoPH" on airline price is lower than that on oil price, since the latter is directly involved in the determination of the monetary prospects. Even though the dependence between these uncertainties is very high, there is a marked difference in potential value from hedging VoPH quantifies these differences. According to Theorem iii), the wildcatter should not be willing to pay more than (oP = $1.61M (once they have decided to drill) for any deal whose prospects are determined solely by the states of AP (high and low) and whose mean equals Lower VoPH means that the decision maker can eliminate more potential deals before getting into specifics.
It may seem intractable to compute VoPH, based on its definition(s), which involves an optimization over rewards. However, VoPH for a fixed policy is a difference of expectations which can be computed efficiently [4).
Theorem 10. For additive value function, an exponential utility function, and any uncertainty X, the perfect hedge (augmented reward) and oPPH for X at policy 11 are:
r' (X) Ex (X) (z): VoPH = Ex(CE" - CCE
where (x) is the certain equivalent given state x of X is observed, Ex(CE" is its expecta- tion over all states of X, and C is the prior certain equivalent at policy II.
Remark. The intuition behind the above result is described in the proof sketch in Appendix A but importantly shows that VoPH can be computed efficiently for any uncertain node X. By converting the ID to policy form, one can apply standard BN inference algorithms to compute the required certain equivalents after setting evidence for X at its various states. Computing VoPH* is harder problem though, deserving further future attention.
5 Experiments
Hedging related computations graphical models are holistic in that they leverage the entire model, including the utility function, thus they do not necessarily conform to measures that solely rely on dependence/correlln
to identify hedging opportunities. This is where concepts such as VoH and VoPH are useful in practice, as they account for all modeling aspects together. We conduct experiments with both synthetic and real data to demonstrate applicability of hedging valuations well as some complexities that non-trivial and potentially counter-intuitive. Since our contributions in this paper are primarily conceptual and apply to any BN or ID, we note that the required computations, as formalized by equations in Definitions 4 5 and [ can be performed using any of the existing state-of-the-ae BN inference methods and ID) solvers.
5.1
Auto MI.
Figure 3 shows an ID where an automated machine learning agent must choose setting (S) which affects its uncertain performance on different tasks. For instance, this could be an auto Al pipeline decision for some future tasks. Value is measured by multiplicative positive metric such as % performance improvement compared to a baseline algorithm. The agent is assumed to be risk-averse with utility function u(E) (og(v). The agent is currently committed to two tasks (PI and P2) but is considering an opportunity to hedge on a third task (P3). Since tasks are conditionally independent given the setting, one may not expect hedging play role.
The setting (S) is assumed to be binary with states high (h) and low (I). Each task i's performance in setting is lognormally distributed with mean parameter ju and common standard deviation parameter a. The expected performance for any task Pe is related to the mean and standard deviation parameter
of the lognormal distribution as: P = +o"/02/2 compactness.
We denote = [u u4] and P. = [P! PP] for
8

---

Consider the following numerical case: = 1, expected perfor- mances for tasks: Pi [2. 11. P2 3. 2]. P3 [1. With these numbers, policy л = h is optimal a-priori.
D
In this example, all computations assumptions, and we obtain value of hedging Vol** = 1, Voll" 0.33, and value of augmented reward VoAR 1.42. Re-optimization changes Voff and V VoAR in this case because the setting is so much better for P3 that it is chosen after reward augmentation. Note that fixing policy π h underestimates the actual value from augmentation, since the agent would prefer the setting if it could also accrue value from P3 (V oooRRis is 0.61 vs. 1.42). Interestingly, although augmentation is desirable here again due intrinsic value from P3, there is reduction in value from hedging itself, since VoH* < 1. (Since value multiplicative here, VoH 1 is equivalent to VoHf = 0 for additive aggregation.) This example illustrates that allowing for decision re-optimization and generic utility function besides the linear or exponential utility function such as logarithmic this case can impact hedging valuations unexpected ways.
be done analytically due to the
S
0.61, VoARt
P1
P2
P3
V1
V
Figure 3: AutoML ID, with potential augmented reward. D is a decision, S is the ting and P1,P2, P3 are perfor- mance tasks.
5.2 Gold Mining Stocks
Probabilistic graphical models are able to represent an uncertain situation by combining expert opinion and data. Figure[[c) shows an example BN about three gold mining stocks: ABX, BGO and AEM 42 Stock j's monthly return is modeled through linear regression:
R = 3o + Bas M + BaG + S
where M and G are the market and gold-industry specific indices, {.9. 3M. Sc}} are regression coefficients, and E, is Gaussian supplier; the effect of the conditions at the supplier is modeled through the error terms. The numbers based on real data from 1996 to 1998 (see Appendix C.T).
term for stock-specific effects. ABX and AEM have a common
We assume the decision maker is only considering the gold mining industry, for whatever reason. They have committed to investing $ $1000 in ABX and are now considering potential hedging opportunity by investing an additional $1000 in either BGO or AEM. Intuition suggests that BGO will be relatively better hedge since ABX and AEM have a common supplier Pearson correlation coefficients for (ABX, AEM) and (ABX, BGO) are 0.41 and 0.31 respectively.
Table shoows VoH for varying risk tolerance. Unsurprisingly. all hedging valuations are negative since stocks belong to the same industry. However, AEM is the more appropriate choice between the two, mainly because of the higher variance in BGO returns. Note VoHl -> 0 as risk tolerance increases. Computations for this example were done using Monte Carlo simulation with 10 runs of 100K samples each.
5.3 Supply Chain Risk
We consider a real-world supply chain risk BN model for a company's 's line of product manufactured in South-east Asia and shipped to USA 19). The company is interested in managing the lead time in the supply chain. We simplified the original BN slightly, considering 18 uncertain nodes for different types of risks including disasters, contractor related risks, customs, etc. and value nodes for lead time (in units days) at sourcing, sub-assembly, transport and for other reasons due to disasters. Value is additive and u(-) is monotonically decreasing (less lead time is preferred), assumed to be exponential with risk tolerance The structure as well as all parameters of this BN are included in Appendix
Table 2 shows VoPH computed for 7 key uncertainties for various levels of risk tolerance p. using the result in Theorem I0 Computations were done using exact Bayesian inference available from the pgmpy package. The table reveals the most relevant sources uncertainty for hedging purposes,
hutps//eittnbbcom//mgm/////pp

---

Table I: Value of hedging (Vof) on BGO and AEM stocks in dollars given the decision maker's current investment in ABX in the Gold Mining BN (Figureßic) with varying risk tolerance (p). Monte Carlo error for computations is also indi- cated based on 10 runs of 100K samples each.
Table 2: Value of perfect hedging (VoPH) in units of lead time (days) for variables in the Supply Chain Risk BN (Figure Appendix C22 with varying risk tolerance (p). The two most valuable hedges are for natural disaster (ND) and geo-political instability (GI).
Variahle
p 100
200
500
p
1K
Variable
p 10
P 50
100
BGO
18.6
-9.4
-3.7
-1.9
EDI
0.001
10.47
10.12
10.06
10.02
CSP
0.078
0.015
0.007
AEM
- 13.5
-6.7
-2.7
-133
CMB
0.373
0.049
0.023
1..14
#0.05
+0.02
#0.01
DL
0.003
ND
6,004
0.758
0.353
LS
0.033
0.007
0.003
GI
1.91
0.171
0.078
which in this case would primarily arise from insurance or contracts in practice. The company should clearly consider insurance against natural disasters (ND) and geo-political instability (GI) in particular, but there are other surprisingly important risks, such as contract manufacturer backruptcy (CMB) and to a lesser extent component sourcing problems (CSP). As the decision maker gets closer to being risk neutral, VoPH - 0 for all uncertain nodes.
This application illustrates the primary purpose of VoPH analysis: to identify potential hedging opportunities without necessarily having specific information about rewards and the cost of hedging the time of the analysis. This is similar to pursuing information gathering opportunities before having details about the accuracy of such tests [20]. The actual valuation of any hedge here (such as insurance) would course depend on the specifies of the reward, but VoPH provides guidance around directions for the finance team to pursue, before they have identified specific hedges. Thus the goal is to provide initial support towards improving the decision situation by potentially reducing risk, similar to how value of information and control can provide guidance around potential information sources or intervening on uncertainties wherever possible. In general, VoPH could be an important tool in an analyst's 's toolbox for improving their situation represented by BN ID.
6 Conclusions
We have presented hedging as reward augmentation a new kind of graph manipulation in BNs and IDs that represent decision maker's situation, where preferences are represented by one- dimensional utility function over a measurable value/reward. Most results apply for special cases such additive value aggregation, and for decision makers with exponential utility functions. There are computational reasons why it is hard to generalize beyond point, explaining why additive reward aggregation is pervasive in domains that compute optimal policies, but the fundamental ideas are more general and there may be opportunities to explore other cases for select problem classes.
There are open problems for future work, such around computing the value of perfect hedging when re-optimization is allowed; since this is a difficult optimization problem, approximations may be necessary. We also stress that there may be risks of reward augmentation, for instance Al safety. Side effects can be crucial for hedging, which if ignored during modeling could lead to undesirable effects. For instance, a company that seen to be actively hedging may sacrifice shareholder confidence. Thus, while hedging may bring opportunities, there are potential risks around reward augmentation that should not be neglected. Hedging however also provides analysis opportunities for human and machine agents, as we have argued in this paper. Additional investigation of real- worldwrldd problems is necessary to better understand potential benefits of hedging in AI systems and could lead to further methodological research progress.
Acknowledgements
We thank Ross Shachter for several helpful discussions around decision-theoreee hedging as well as three anonymous reviewers for valuable feedback and suggestions.
10

---

References
[1] D. Amodei, C. Olah, J. Steinhardt, P. Christiano, J. Schulman, and D. Mane. Concrete problems ArXiV, 2016.
in Al safety. Technical Report
[2] Armstrong. Motivated value selection for artificial agents. In Proceedings of the Artificial Intelligence and Ethics Workshop at the AAAI Conference on Artificial Intelligence, pages 12-20, 2015.
[3] K. J. Arrow. The theory of risk aversion. In Aspects of the Theory of Risk-Bearing. Lecture 2, Yrjo Jahnssonin Saatio: Helsinki, 1965.
[4] D. Bhattachariya and R. Shachter. Three new sensitivity analysis methods for influence diagrams. In Proceedings of Conference on Uncertainty in Artificial Intelligence (UAI), pages 56-64, 2010.
[5] H. Buehler, K. Gonon, J. Teichmann, B. Wood, B. Mohan, and J. Kochems. Deep hedging: Hedging derivatives under generic market frictions using reinforcement learning. Technical Report 19-80, Swiss Finance Institute Research Paper, 2019.
[6] J. Cao, J. Chen, J. Hull, and Z. Poulos. Deep hedging of derivatives using reinforcement learning. Journal of Financial Data Science, 3(1):10-27, 2021.
[7] G. Cooper. A method for using belief networks as influence diagrams. In Proceedings of Conference Uncertainty in Artificial Intelligence (UAI), pages 55-63, 1988.
[8] A. Dawid. Influence diagrams for causal modelling andinferencc. International Statistical Review / Revue Internationale de Statistique, 70(2):161-189, 2002.
[9] L. Deleris and Erhun. Quantitative risk assessment in supply chains: A case study based on engineering risk analysis concepts. In K. Kempf K., P. Keskinocak, and R. Uzsoy, editors, Planning Production and Inventories in the Extended Enterprise, pages 105-131. Springer, New York, NY, 2011.
[10] M. Dorigo and M. Colombetti. Robot shaping: Developing autonomous agents through learning. Artificial Intelligence, 71(2):321-370, 1994.
[11] J. S. Dyer and R. K. Sarin. Mcasurable multiattribute value functions. Operations Research, 27(4):810 822, 1979.
[12] T. Everitt, Krakovna, L. Orseau, M. Hutter, and S. Legg. Reinforcement learning with cor- rupted reward signal. In Proceedings of International Joint Conference Artificial Intelligence (IJCAI), pages 4705-4713, 2017.
[13] Y. Freund and R. F. Schapire. A decision-theoretic generalization of online learning and an application to boosting. Journal of Computer and System Sciences, 55(1):119-139, 1997.
[14] J. García and F. Fernández. A comprehensive survey on safe reinforcement learning. J. Mach. Learn. Res., 16:1437-1480, 2015.
[15] W. R. Guay and S. Kothari. How much do firms hedge with derivatives? Journal of Financial Economics, 10::4334611) 2003.
[16] D. Hadfield-Menell. S. Milli, S. Russell, P. Abbeel, and A. D. Dragan. Inverse reward design.
In Advances in Neural Information Processing Systems (NeurIPS), pages 6765-6774, 2017.
[17] D. Heckerman, D. Geiger, and D. M. Chickering. Learning Bayesian networks: The combination of knowledge and statistical data. Machine Learning, 20(3):197-24,, 1995.
[18] D. Heckerman and R. D. Shachter. Decision- foundations for causal reasoning. Journal of Artificial Intelligence Research, 3:405-430, 1995.
[19] B. Hibbard. Model-based utility functions. Journal of Artificial General Intelligence, 3(1):1224,
2012.

---

[20] R. Howard. Information value theory. IEEE Transactions on Systems Science and Cybernetics, 2(1):22-26, 1966.
[21] R. Howard and J. Matheson. Influence diagrams. In R. Howard and J. Matheson, editors, The Principles and Applications of Decision Analysis, volume II. Strategic Decisions Group, Menlo Park, CA, 1984.
[22] D. Kahneman and A. Tversky. Prospect theory: An analysis of decision under risk. Economet- rica: Journal of the Econometric Society, pages 263-291, 1979.
[23] R. L. Keeney and H. Raiffa. Decisions with Multiple Objectives: Preferences and Value Trade-Offs. Wiley, New York, 1976.
[24] A. Krause and C. Guestrin. Optimal value of information in graphical models. Artificial Intelligence Research, 35:557-591, 2009.
[25] Leike, M. Martic, V. Krakovna, P. Ortega, T. Everitt, A. Lefrancq, L. Orseau, and Legg. AI safety gridworlds. Technical Report 111.11711.0983.. ArXiV, 2017.
[26] S. Lichtenstein and P. Slovic. The Construction of Preference. Cambridge University Press, Cambridge, U.K., 2006.
[27] T. Lu and M. J. Druzdzel. Causal models, value of intervention, and search for opportunities. In Proceedings of the First European Workshop on Probabilistic Graphical Models (PGM), page 108-116, 2002.
[28] D. G. Luenberger. Investment Science. Oxford University Press, New York, 1997.
[29] M. J. Mataric. Reward functions for accelerated learning. In Proceedings of the International Conference on Machine Learning (ICML), page 181-189, 1994.
[30] D. Matheson and J. E. Matheson. Describing and valuing interventions that observe control decision situations. Decision Analysis, 2(3): 2005.
[31] J. Matheson. Using influence diagrams to value information and control. In R. M. Olivier and J. Q. Smith, editors, Influence Diagrams, Belief Nets and Decision Analysis, pages 25-48. Wiley, Chichester, 1990.
[32] R. Mihatsch, O. Neuneier. Risk-sensitive reinforcement learning. Machine Learning. 49:267-290, 2002.
[33] D. R. Nance, C. W. Smith, and C. W. Smithson. On the determinants of corporate hedging policy. Journal of Finance, 48(1):267-284, 1993.
[34] A. Y. D. Harada, and S. J. Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In Proceedings of the International Conference on Machine Learning (ICML), page 278-287, 1999.
[35] J. Pearl. Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Infernnce. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 1988.
[36] J. Pearl. Causality: Models, Reasoning and Inference. Cambridge University Press, New York, NY, USA, 2nd edition, 2009.
[37] M. Petrik and D. Subramanian. An approximate solution method for large risk-averse Markov decision processes. In Proceedings of Conference on Uncertainty in Artificial Intelligence (UAI), page 805-814, 84,, 2012.
[38] J. Pratt. Risk aversion in the small and in the large. Econometrica, 32: 1964.
[39] T. Seyller. The Value of Hedging. PhD thesis, Stanford University, 2008
[40] R. D. Shachter. Efficient value of information computation. In Proceedings of Conference on Uncertainty in Artificial Intelligence (UAI), pages 594-601, 1999.
12

---

[41] R. D. Shachter and D. Heckerman. Pearl causality and the value of control. In H. Geffner, R. Dechter, and J. Y. Halpern, editors, Heuristics, Probability and Causality: A Tribute to Judea Pearl, pages 431 131-447. 2010.
[42] C. Shenoy and P. P. Shenoy. Bayesian network models of portfolio risk and return. In Y. S. Abu-Mostafa, B. LeBaron, A W. Lo, and A. S. Weigand, editors, Computational Finance 1999, pages 87-106. MIT Press, Cambridge, MA, 2000.
[43] J. von Neumann and O. Morgenstem. Theory of Games and Economic Behavior. 2nd edition, 1947.
[44] P. т. Wakker. Explaining the characteristics of the power (CRRA) utility family. Health Economics, 17(12)1129-1344, 2008.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed reproduce the main perimental results (either in the supplemental material or as a URL)? [No] All the information necessary to conduct the experiments provided in the paper itself.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [N/A]
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [No] The experiments do not require extensive compute power.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're
using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A] We used publicly available data without
identifiable information or offensive content.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
(b) Did you deseribe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "sjKoB_vWbo", "reviewer": "Reviewer_XnHK", "summary": "The authors propose a modified Bayesian Network formalism incorporating the hedging information that can be useful in the decision making process.", "questions": "Fig. 1 - Some variables are not defined. This is a problem present in many figure where the caption is not so informative. \n\nrow 90 --> what is the meaning of \"numeraire\"?\n\nrow 122  --> $\\mathbf{V_k}$ should be substituted by $V_k$ since we refer to a particular Value node.\n\nrow 131 --> what is the alternative set?\n", "limitations": "There isn't a mention neither to the Markov Decision Process nor to the Control as Probabilistic Inference (I will not make a list of papers here to avoid favoritism but a simple search for \"Control as Probabilistic Inference\" should give the authors plenty of references). \nThe difference with these approaches should be described because it is not clear what is the advantage of the proposed approach. \n\nThe Reinforcement Learning (RL) literature on this topic is not considered (e.g. risk-aversion RL). RL is cited only regarding the risk in the reward manipulation. ", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The main weakness I see is an insufficient contextualization of the problem. The Markov Decision Process, the control as probabilistic inference, the risk-aversion Reinforcement Learning, etc. are neither mentioned nor discussed.\n\nThere isn't presented any alternative approach to solve the problems presented and so any comparison is missing.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "O7qIJR-JPjc", "reviewer": "Reviewer_fGwk", "summary": "This paper introduces a new method for studying the value of hedging on uncertain variables in decision-making problems. This is done by augmenting the reward with the value of the hedge. That way, gains/losses associated with a particular hedge can be computed so that alternative scenarios such as no hedging or hedging on another uncertainty can be evaluated and compared.", "questions": "Aside from addressing the concerns listed under weaknesses I would like the authors to answer the following questions:\n\n1. As mentioned under weaknesses, a proper description of certain financial concepts is much needed. For instance:\n\n     * How should the utility function be chosen? Is this given? How do the different utility functions affect the final policy/decision? To be more specific, I found Section 3.1 hard to follow. I suggest the authors expand it with examples or intuitions illustrating these concepts.\n\n     * The notion of certain equivalents is also unclear. Can the authors give an intuition on why it is the inverse of the utility function? \n\n     * The last sentence of the second paragraph in Section 4 is confusing: \"The agent’s certain equivalent (CE) for their (decision) situation (represented by the ID/BN) is the certain amount, in units of value V, at which they are indifferent between this amount and their situation.\"\n\n2. The paper discusses how to compute VoPH for a particular policy when the rewards are unknown (Theorem 10). However, equation 4 includes an inner maximization to find the optimal policy, which itself depends on the augmented reward. Can the authors explain how equation 4 can be solved? \n\n", "limitations": "N/A", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n* The paper is clearly written and gives many examples illustrating the method and showing the different cases that may occur: hedging for a fixed policy, hedging with policy re-optimization.\n\n* It also provides theoretical results showing how the calculations for the VoPH when the rewards are unknown can be done efficiently using graphical models and BN inference methods.\n\nOverall, I think this is a strong paper that will be valuable for the community. A list of my main concerns is provided below. I would like the authors to comment on them in their rebuttal and revise the paper if needed. \n\n### Weaknesses\n\n* The paper seems to assume that the reader is familiar with concepts from finance such as hedging, risk aversion, or utility functions. Given that this is an ML conference it would be good to introduce all these more carefully (see my comment under Questions for more details). The authors may save some space by shortening section 3.2 introducing BNs and IDs which should be familiar concepts for the ML audience. \n\n* The terminology is confusing at times, the paper uses the terms reward, value and utility interchangeably.\n\n* There is a big gap in the related work section. The paper ignores all the works on reward shaping in RL. I will not make a list of papers here to avoid favoritism but a simple search for \"reward shaping in RL\" should give the authors plenty of references. Although the objectives in most of these papers (e.g. encourage exploration (intrinsic motivation) or ease learning) are different in nature from the objective in this paper (studying the value of hedging), the means to accomplish these goals are similar. In fact, it is difficult to assess how different this paper is from all these works without the proper context.\n\n* The paper also fails to discuss the assumptions and limitations of the method. Such as the availability of a graphical model of the problem. Full knowledge of the dynamics is a strong assumption that should be mentioned in the paper.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "L2qx-TIuf_o", "reviewer": "Reviewer_GBff", "summary": "This paper studies hedging within the framework of causal influence diagrams (CID). Roughly speaking, a hedge is an opportunity for a risk-averse agent to increase its expected utility, by creating anti-correlated sources of value. This paper formalises several concepts relevant to hedging within the CID-framework, and proves a number of theorems that follow from these definitions. These theorems can then be used to identify potential hedges in an uncertain decision situation, or reason about the incentives of an agent. This paper builds on recent work on CIDs, as developed by Tom Everitt and others.", "questions": "I have the following questions:\n- What is the main (practical or theoretical) significance of the results presented in the paper? \n- What problems can the presented results help us to solve, or gain more insight into (as compared to a naive analysis)? \n- What further theoretical results or extensions do you anticipate that it will be possible to build on the results you have provided?\n- Is there a theoretical reason to expect the provided results to be exhaustive (in some relevant sense)?\n\nI believe that a better understanding of these points could compel me to increase my rating.\n", "limitations": "I find that the authors have addressed the potential negative societal impact of their work in a fully satisfactory way. Their discussion of the limitations of their work is also adequate (although dispersed throughout the text).", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality:\n\nThere is, to the best of my knowledge, no previous work studying hedging within a similar framework.\n\n\nQuality:\n\nThe presented results are of good quality. The maths is sound, and the experiments serve their purpose.\n\n\nClarity:\n\nI found the presentation and exposition in this paper to be exceptionally clear.\n\n\nSignificance:\n\nMy impression is that the contribution made by this paper is somewhat marginal. It studies a relevant problem from a promising direction, but the results themselves could be more substantial. The first two theorems (Theorem 6 and 9) are fairly straightforward. Theorem 10 seems to be more substantive, but its implications are not discussed in the paper in any great detail. Of course, for a paper like this, the definitions and framework constitute a contribution by themselves, but it is not clear to me that this is enough to make the paper impactful in this case. I would have liked to see a more complete treatment of the problem, either with more theoretical results, or with case-studies that demonstrate how these results could help us to solve or better understand some class of problems (be they practical or theoretical). Alternatively, it would also suffice to provide a compelling argument for why such extensions would be very difficult to provide (thus arguing that there is a sense in which the provided results are complete).\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept