A Unified Causal Framework for Auditing Recommender Systems for Ethical Concerns

## Abstract

Abstract As recommender systems become widely deployed in different domains,
they increasingly influence their users’ beliefs and preferences.
Auditing recommender systems is crucial as it not only ensures the continuous improvement of recommendation algorithms but also safeguards against potential \llreplace pitfallsissues like biases and ethical concerns.
In this paper, we view recommender system auditing from a causal lens and
provide a general recipe for defining auditing metrics.
Under this general causal auditing framework,
we categorize existing auditing metrics
and identify gaps in them— \vsedit notably, the lack of metrics for auditing user agency while accounting for the \vsedit multi-step dynamics of the recommendation process.
We leverage our framework
and propose two classes of such metrics:
future- and past-reacheability and stability \lldelete \sgcomment add a sentence briefly explaining the metrics. Something like:. \sgedit \lldelete We present two classes of metrics to quantify user agency:
reacheability and stability, that measure the ability of a user to influence their own and other users’ recommendations, respectively.
We provide both a gradient-based and a black-box approach
for computing these metrics,
allowing the auditor to compute them
under different levels of access to the recommender system.
In our experiments,
we demonstrate the efficacy of methods for computing the proposed metrics
and inspect the design of recommender systems through these proposed metrics.

## 1 Introduction

Recommender systems actively shape people’s online experience by moderating which news, music, social media posts, products, and job postings are most visible to them.
These socio-technical systems have the potential to influence individuals’ choices and beliefs as well as public opinion at large which can have tangible ethical impact [, , ].
Monitoring recommender systems for the potential of \llreplaceadverseharmful effects is a difficult task requiring careful design of evaluation metrics and auditing frameworks.

Traditional metrics for evaluating recommender systems are observational in nature and only measure the recommender system’s ability to recommend items that it perceives to be relevant to the user at a given time, without considering how the dynamic between the user and recommender system evolves over time as both entities continuously influence each other with every interaction. Recommender systems effectively act as information filters and as such, they guide, to large extent, the information a user is exposed to. This opens them up to being agents of harm as users can be exposed to targeted propaganda, misinformation and inappropriate content that is only aggravated over time as user’s continue to interact with the recommender system.
Traditional metrics for evaluating recommender systems are often correlational in nature, focusing on the system’s ability to recommend content perceived as relevant by the users based on their past behavior. However, these metrics tend to overlook critical ethical concerns, particularly how these systems interact with and shape user beliefs and preferences over time.
Recommender systems act as powerful information filters, determining much of the content users encounter, raising concerns about user agency and the extent to which users control their own online experiences.

We argue that the instances of harm in recommendation settings discussed above are *inherently causal* and require a causal framework to be addressed.
User agency is inherently *causal* because measuring it involves answering *what-if* questions.
For example, the question “Do users have agency over their recommendations?” requires causal reasoning on the behavior of the recommender system under behavior shift of the users [, , ]. Instead of a pure prediction setting, we are interested in the *effect of interventions*.

Causal modeling has gained increasing attention in the ethical AI space in recent years [, , , , , ].
\llreplaceYet, iIn the recommendation setting, researchers have primarily focused on simulation and agent-based modeling to study long-term ethical effects of certain design interventions [, , , ].
\lldeleteWhile causal approaches to recommender systems exist, they frequently focus on estimating the direct effect of single-step recommendations [, ] or propose new causality-based recommender systems [zhu2023causal]. \llcommentI have moved this to related work. It’s a bit out of the place here.
\llreplaceTo the best of our knowledge, there has been no previous effort to develop a general evaluation framework for using causal metrics to audit recommender systems.
While some causal auditing metrics have been proposed [, ], a principled framework for reasoning about them in recommender system audits is still lacking.

In this work, we present a causal graphical model to capture the
dynamics of a general class of recommender systems that includes
matrix factorization as well as neural network-based models
(Section 3.1).
This causal model allows us to provide a general recipe for defining causal metrics for auditing recommender systems,
and categorize existing auditing metrics based on
the \lldeletetargets of the interventions and the outcomes of interest (Section 3.2).

Under this unifying perspective,
we identify the lack of metrics \llreplaceinfor measuring user \llreplaceautonomyagency.
We leverage the causal framework to formalize \lldeleteand extend two \lleditclasses of metrics of user agency: past- and future-*reachability* (Section 4.1), which extends the one defined in [] as well as past- and future-*stability* (Section 4.2).
Reachability measures the ability of a user to
be recommended (or *reach*) some desired item at a given time under modifications to their future or historical ratings.
Stability measures how sensitive a user’s recommendations are to edits made by adversarial users to their \vseditown future/past ratings.

Computing these user-agency metrics requires us to solve different optimization problems.
We provide both a gradient-based and a black-box approach for solving them.
When the recommender system is learned through matrix factorization, under mild assumptions, we show that the corresponding \lleditoptimization objectives \lleditfor these metrics have a special structure \lldelete(concavity of (2), quasi-convexity of (4)) \llcommentWe don’t need to get into these technical details here. that can be leveraged to obtain the optimum efficiently (Section 5).
Using the proposed metrics,
we inspect how the stochasticity of recommendations and the nature of the recommender system (sequence-dependent versus not) may influence user agency.
We find, on average, an increase in the stability of a user’s recommendations but smaller upgrades in reachability as the stochasticity of the system decreases; and that a standard matrix factorization based recommender system is less stable but facilitates better reachability for users as compared to a deep recurrent recommender network (Section 6).
We summarize our contributions as follows:

- •
We provide a general causal framework for defining new causal metrics and categorizing existing metrics for auditing recommender systems in a principled manner (Section 3);
- •
Using our proposed framework, we develop two classes of metrics for measuring user agency while accounting for the dynamics of the recommendation process: past- and future-reachability and stability (Section 4). We provide effective ways to compute the metrics, allowing the auditor to have different levels of access to the systems (Section 5).
- •
Empirically, we investigate two common classes of recommender systems in terms of our proposed user agency metrics and found that higher stochasticity in a recommender system will help with stability but harm reachability (Section 6).

## 2 Related Work

##### Structural Causal Models (SCMs).

In this work, we use the SCM framework [] to model causal relationships.
An SCM is a tuple $\mathcal{M}=(\mathbf{U},\mathbf{V},\mathcal{F},\mathbb{P}(\mathbf{U}))$,
where $\mathbf{U}$ and $\mathbf{V}$
are the sets of exogenous and endogenous nodes.
An SCM implies a directed acyclic graph (DAG)
$\mathcal{G}$ over the nodes $\mathbf{V}$ with $\text{Pa}_{i}$ denoting
the parents of node $V_{i}\in\mathbf{V}$.
Each $V_{i}\in\mathbf{V}$ is generated
using the structural equation
$V_{i}:=f_{i}(\text{Pa}_{i},U_{i})$,
where $\text{Pa}_{i}\subset\mathbf{V}$,
$U_{i}\in\mathbf{U}$, and $f_{i}\in\mathcal{F}$.
The joint distribution $\mathbb{P}(\mathbf{U})$ induces
the observational distribution $\mathbb{P}(\mathbf{V})$.
An interventional distribution
is generated by the (modified)
SCM $\mathcal{M}_{x}=(\mathbf{U},\mathbf{V},\mathcal{F}_{x},\mathbb{P}(\mathbf{U}))$,
where $\mathcal{F}_{x}$ is the set of
functions obtained by replacing the structural equation
for $X\in\mathbf{V}$ with $X:=x$.
Informally, it represents the state of the system
after intervening on $X$ and setting it to $x$,
denoted by $\text{do}(X=x)$.
SCMs also allow us to compute counterfactual quantities
that express what would have happened,
had we set the node $X$ to $x$,
given that the (factual) event $E$ occurred.
Counterfactuals distributions are generated
using the SCM $\mathcal{M}_{\text{cf}}:=(\mathbf{U},\mathbf{V},\mathcal{F}_{x},\mathbb{P}(
\mathbf{U}|E))$.

##### Recommender System Evaluation.

In recent years, government organizations and academics have called for more ethical scrutiny in evaluating recommender systems, which has led to an active—but fragmented—area of research.
\vseditPrior work on ethic\vsedital \lleditissues in recommender systems identified different areas of concerns including fairness, inappropriate content, privacy, autonomy and personal identity, opacity, and wider social effects []. Past works have proposed a variety of metrics for these areas of concern including measures of fairness [, ], moral appropriateness of content [], stability [], and diversity [, , ]. The majority of these metrics exclusively rely on observational quantities under a fixed recommendation policy and thereby neglect potential effects caused by the users’ behaviors, preferences, or the recommender system itself.
According to Pearl’s three-layer causal hierarchy [], these metrics are association-based, as they only consider the joint distribution of the recommendations and user feedback without accounting for the underlying causal mechanisms.
\vseditStudying a phenomenon like user agency requires answers to causal questions pertaining to how a recommender would respond to interventions made in the user’s behavior over a period of time, which cannot be answered by association-based metrics alone.
\lldeleteBy contrastTherefore, the metrics we formalize are interventional and counterfactual.
Interventional metrics quantify the effect of
hypothetical changes to specific parts of the recommender system.
Counterfactual distributions further encode how
the system would behave if interventions were performed in hindsight
while taking into account what actually happened.

##### Auditing Recommender Systems.

Recently there have been efforts to measure and mitigate (demographic) bias [] and user agency (e.g., notions of reachability [, ]). These works take a step towards interventional metrics by considering the effect of specific interventions on the recommender system’s behavior.
Schnabel et al. [] view recommendation systems as a mechanism that introduces selection bias,
and answer questions concerning how (aggregated) user feedback will adapt under changes to the recommendation policy.
These metrics fall under the interventional layer of Pearl’s causal hierarchy.
Most interventional (or counterfactual) metrics defined for recommender systems choose the give intervention to be the recommendations themselves,
and the outcome of interest in this case is commonly the user feedback [].
Dean et al. [], Curmei et al. [] take the intervention to be users’ own feedback and the outcome of interest to be the recommendation they receive,
allowing their proposed metric to answer questions concerning user agency.
However, existing causal metrics are often limited in the kind of interventions they consider, with few works tapping into new types of interventions and ethical concerns.
In addition, most are one-step metrics, ignoring the dynamics of the recommender-user interaction over time.
\lledit
Separately, another line of work integrates causal approaches into recommender systems, often proposing new causality-based models that focus on estimating the direct effect of recommendations on user engagement [, ].

## 3 A Causal Perspective on Auditing Recommender Systems

We first provide a general recommender system setup (Section 3.1), its corresponding causal graph (Figure 1), and an example illustrating the general setup.
We then contextualize existing metrics for auditing recommender systems using this causal framework and provide a general recipe for one to develop new auditing metrics (Section 3.2).

### 3.1   Recommender System Setup

We consider the setting where there are $n$ users and a set of items $\mathcal{V}$ to recommend.
Each user $i\in[n]$
has an unobserved vector $\mathbf{o}^{\star}_{i}\in\mathbb{R}^{|\mathcal{V}|}$ indicating the user’s initial (true) preference (i.e., ratings) for all items $\mathcal{V}$.
At each time step $t\in[T]$,
a recommender system presents a recommendation (set) $\mathbf{A}_{i,t}\subseteq\mathcal{V}$ to user $i$.
In turn, the user provides their feedback/ratings $\mathbf{O}_{i,t}\in(\mathbb{R}\cup\{\text{unrated}\})^{|\mathcal{V}|}$ for the recommendation where $\mathbf{O}_{i,t}[k]$ is the $k$-th entry of user’s rating vector, $\mathbf{O}_{i,t}[k]=\text{unrated}$ for $k\notin\mathbf{A}_{i,t}$, and $\mathbf{O}_{i,t}[k]\in(\mathbb{R}\cup\{\text{unrated}\})$ indicates user’s choice of rating or not rating for item $k\in\mathbf{A}_{i,t}$.
The user-recommender interaction history is denoted by $\mathbf{H}_{i,t}=(\mathbf{A}_{i,1},\mathbf{O}_{i,1},\cdots,\mathbf{A}_{i,t-1},
\mathbf{O}_{i,t-1})$
and we use $\mathbf{D}_{t}=\{\mathbf{H}_{i,t}\}_{i\in[n]}$ to denote the interaction history for all users, or in other words, the training dataset for the recommender at time $t$.
It is worth noting that both $\mathbf{A}_{i,t}$ and $\mathbf{O}_{i,t}$ are random variables depending on previous user-recommender interactions.
More specifically,
the recommendation $\mathbf{A}_{i,t}$ depends on the training set $\mathbf{D}_{t}$;
the user rating $\mathbf{O}_{i,t}$ is a function of the user’s true initial preference $\mathbf{o}^{\star}_{i}$, their interaction history $\mathbf{H}_{i,t}$, and possibly other users’ feedback in $\mathbf{D}_{-i,t}$.
Finally, we let $\mathbf{D}_{-i,t}$ denote $\mathbf{D}_{t}\setminus\mathbf{H}_{i,t}$ and use $\mathbf{O}_{-i}$ to refer to user ratings for users $[n]\setminus\{i\}$.
When clear from the context, we omit the subscript $i$ or $t$ for discussing the corresponding quantity for all users or time steps.
For any variable or function $y$, we use $y_{t_{1}:t_{2}}$ to refer to $y_{t_{1}},\cdots,y_{t_{2}}$.

Figure: Figure 1: The figure above depicts the causal graph representing a general recommender system at time $t$, specifically pertaining to a user $i$’s interactions with the system. Here, $D_{-i,t}$ denotes the interaction history of all users besides $i$ upto $t$, $A_{i,t}$ denotes the set of recommendations $i$ receives at $t$, $H_{i,t}$ denotes $i$’s interaction history upto $t$, $O_{i,t}$ denotes $i$’s feedback at this time and $o_{i}^{*}$ denotes $i$’s true preference.

We use the Structural Causal Model (SCM) framework [] to capture the causal relationships
among these variables.
In an SCM, each node $X$ is generated according to a
structural equation of the form $X:=f_{X}(\text{Pa}_{X},\epsilon_{X})$,
where $f_{X}$ is some deterministic function,
$\text{Pa}_{X}$ denotes the parents of $X$,
and $\epsilon_{X}$ is an exogenous noise term.
An SCM also encodes interventional distributions,
denoted by $\mathbb{P}(Y|\text{do}(X=x))$, which
represents the distribution of $Y$ when the variable
$X$ is set to $x$, i.e., its structural equation
is replaced with $X:=x$ [, Defn. 3.2.1].
The causal graph for the recommender system is given in Fig. 1.
We provide a concrete example below.

###### Example 1 .

In the basic form of a score-based recommender system,
the initial preference vector $\mathbf{o}_{i}^{\star}$ is the user’s true ratings for item set $\mathcal{V}$, the recommendation $\mathbf{A}_{i,t}$ is a singleton(^3^33With some abuse of notation, we say $\mathbf{A}_{i,t}\in\mathcal{V}$ in this case.),
and the user rating is given by $\mathbf{O}_{i,t}=\mathbf{o}^{\star}_{i}[\mathbf{A}_{i,t}]$.
To determine the recommendation $\mathbf{A}_{i,t}$,
a recommendation algorithm first predicts a score per user-item pair $(i,j)$, indicating the predicted user preference on item $j\in\mathcal{V}$ for user $i\in[n]$.
To learn the scoring function, one rely on the dataset $\mathbf{D}_{t}$ and may use approaches like matrix factorization (more details in Section 5).
Finally, given the score,
a recommendation algorithm may recommend differently, e.g., deterministically according to the highest score on unseen items for the user, or stochastically among the top scoring items.

From an auditor’s perspective,
one may be interested in defining metrics to measure how the ratings among users correlate with each other (associational metrics),
how the recommendation algorithm may cause the popularity of an item (interventional metrics),
or how much change a user can make over their recommendations had they behaved differently under the same historical recommender (counterfactual metrics).
Hereafter, we provide a discussion of associational, interventional, and counterfactual metrics for auditing a recommender system, categorizing existing metrics using this causal framework, and illustrating how one can use this unifying perspective to define new metrics.

###### Example 1 .

### 3.2   Causal Perspective on Auditing Recommender Systems

Existing metrics for auditing recommender systems often focus on the *association* between variables concerning different qualities of recommendations under the current system. Metrics such as the diversity of the recommended items, and the average ratings or popularity of items are of this kind since they are defined over the observational distribution
$\mathbb{P}(\{\mathbf{A}_{i,t},\mathbf{O}_{i,t}\})$ (e.g., [, ]).

When defining *interventional* and *counterfactual* metrics,
one needs to be explicit about the intervention to apply,
the outcome one cares about, and the distributions involved in obtaining the metric.
More specifically, to define a metric of causal nature, one needs to specify the following three quantities of interests:
(i) the intervention;
(ii) the (random) outcome of interest;
and (iii) a metric (or in other words, a functional) that maps the random outcome of interest to a real value.
This process allows us to categorize existing causal metrics used for auditing recommender systems based on the nature of the intervention and outcome of interest:

- •
Recommendation $\mathbf{A}_{i}$ as intervention: In this case, the auditor interrogates
the effect of changing the recommendation algorithm.
Often, the outcome of interest is user ratings $\mathbf{O}_{i}$, and the corresponding interventional distribution is $\mathbb{P}(\mathbf{O}_{i}|\text{do}(\mathbf{A}))$ (e.g., exposure bias []).
This quantity is often at the core of a recommendation algorithm and is studied the most, as recommender systems use predicted user preferences to make recommendations, through optimizing the recommendations against user interests, e.g., $\max_{\mathbf{a}_{i}}\mathbb{P}(\mathbf{O}_{i}|\text{do}(\mathbf{A}=\mathbf{a}
_{i}))$(^4^44On a separate note, an active line of work has centered around using observational data to estimate this quantity, e.g., Schnabel et al. [].).
- •
Other user’s data $\mathbf{D}_{-i}$ as intervention: Here, the auditor may want to investigate under the current recommender system, what would happen to a particular user (or a particular group of users) if other users have changed their behavior. For example, in measuring conformity bias (i.e., how other users’ ratings may influence the user’s reaction to the same item) [], the auditor inspects whether the probability of the user’s own rating change as one intervenes on other user’s historical ratings/data. The corresponding interventional distribution is $\mathbb{P}(\mathbf{O}_{i}|\text{do}(\mathbf{D}_{-i}))$.
- •
User’s own reaction $\mathbf{O}_{i}$ (or $\mathbf{H}_{i}$) as intervention: The auditor may be interested in inspecting the amount of change a user’s own reaction may cause in their recommendations. In this case, The outcome of interest, as we will discuss in Section 4, can be the user’s recommendation [], i.e., the distribution of interest is $\mathbb{P}(\mathbf{A}_{i}|\text{do}(\mathbf{O}_{i}))$.

With this perspective and the proposed three-step recipe for defining new metrics,
an auditor can identify gaps in existing metrics and define new ones.
As an example, in Section 4,
we define a suite of user-centric causal metrics using this framework.
Finally, there is often less discussion on counterfactual metrics in auditing recommender systems, since it involves inspecting quantities that are hard (if not impossible) to obtain in practice.
We provide more discussion on this in Section 4.
We also note that the aforementioned metrics are only example metrics that one can categorize using this framework. There are other metrics that fit in this framework (e.g., selection bias and position bias []) that we do not go into details.

## 4 \llreplace User-Centric Causal MetricsUser-Centric Causal Metrics for Auditing User Agency

In the following, we focus on introducing user-centric causal metrics in auditing a recommender system.
\vseditWhile much attention has been given to \lldeletevarious aspects of recommender system performanceauditing various ethical concerns of recommender systems, there has been comparatively little work conducted on measuring user \lldeleteautonomy and agency.
This gap is particularly notable given the rich line of qualitative work emphasizing the importance of user agency [].
\llreplaceThis work conceptualizes user agency asUser agency is a user’s power over their own recommendations compared to recommendations being driven by external forces like other users’ behaviors or algorithmic profiling [].
\llreplaceUser agencyIt can be compromised in various ways, several of which we target with the metrics we propose.
First, recommender systems can enforce filter bubbles that restrict users from diverse content feeds and amplify biases [].
Second, recommendation algorithms can be vulnerable to strategic behavior and adversarial attacks that alter recommendations for unrelated users [].
For example, consider content duplication attacks on e-commerce platforms [].
In this setting, providers game the recommendation system by duplicating item listings with little to no changes to maximize the probability of recommendation. Maintaining user agency over their own recommendations in this scenario requires recommendation stability.
\lldeleteAs such, user agency ties directly to core ethical concerns of recommender systems. \llcommentWe made this point pretty clear already.
The user-centric causal metrics we define in this section are designed with the intent of quantifying \lldeletethis notion of user agency from \lleditthese two separate viewpoints: \lleditreachability measures the extent to which a user can escape filter bubbles (Section 4.1), while stability assesses the influence that other strategic users can have on this user’s recommendations (Section 4.2).

When defining these metrics, we take the interventions as user’s feedback $\mathbf{O}_{i}$ or other user’s feedback $\mathbf{O}_{-i}$, and the outcome of interests as their recommendations $\mathbf{A}_{i}$.
We call these metrics user-centric since the interventions are user feedback (actions that users can take),
and the outcome of interest is the user’s own recommendations (that are consumed by the user).
In Section 4.1, we extend existing metrics for auditors to measure the autonomy an user can have regarding their own recommendations;
In Section 4.2,
we develop new metrics for auditors to measure the influence a user’s choice can have on other users’ recommendations.
Throughout, we provide both interventional and counterfactual metrics and discuss how these metrics connect to the dynamics of the recommendation process.

### 4.1 \llreplace Metrics on User Agency:Past- and Future-Reachability

A recent line of work develops metrics for inspecting user agency, with the most relevant work introducing the concept of (maximum) stochastic recheability [, ].
Conceptually, stochastic reachability measures the maximum change in recommendation probability on item $j$ for user $i$ upon modifying their ratings for a predefined set of items $\mathbf{a}$, e.g., historical recommendations or unseen items for the user.
Under our framework, the original maximum stochastic recheability for a user-item pair $(i,j)$ at time $t$ can be rewritten as: \llreplace

$$ $\displaystyle\max_{f\in\mathcal{F}}\mathbb{P}(\mathbf{A}_{i,t+1}=j|\mathbf{A}_ {i,t}=\mathbf{a},\text{do}(\mathbf{O}_{i,t}=f(\mathbf{a}))),$ $$

$$ $\displaystyle\max_{f\in\mathcal{F}}\mathbb{E}_{\mathbf{A}_{i,t}}\left[\mathbb{ P}(\mathbf{A}_{i,t+1}=j|\text{do}(\mathbf{O}_{i,t}=f(\mathbf{A}_{i,t})))\right],$ $$

SG: Can we rewrite this as follows (makes it more consistent with the future-k notation)?:

$$ $\displaystyle\max_{f\in\mathcal{F}}\mathbb{P}(\mathbf{A}_{i,t+1}=j|\text{do}( \mathbf{O}_{i,t}=f(\mathbf{A}_{i,t}))),$ $$

makes sense!
where $\mathcal{F}$ is the set of all measurable functions that map a recommendation set $\mathbf{a}$ to a user rating vector $\mathbf{o}$,
and $\mathbf{a}$ is a fixed set of (recommendation) items that can be randomly generated from the unseen or historically recommended set.
By rewriting the original stochastic reachability using our framework,
we identify that the original metric allows the auditor to inspect user agency for a single time step
and the intervention is specific to user ratings for a particular set of predefined recommendations $\mathbf{a}$ (that are not determined by the dynamics of the recommendation process).
In does not allow an auditor to inspect user agency under the natural evolution of a recommender system as users apply interventions over time.
To this end, we introduce the following (maximum) future-$k$ reacheability.

###### Definition 4.1 (future- $k$ reachability) .

For any user $i\in[n]$ and item $j\in\mathcal{V}$, at time $t\in\mathbb{N}_{+}$,
their maximum future-$k$ reachability for $k\in\mathbb{N}_{+}$ is given by

$$ $\max_{f_{1:k}\in\mathcal{F}^{k}}\mathbb{E}_{\mathbf{A}_{i,t:t+k-1}}\left[ \mathbb{P}(\mathbf{A}_{i,t+k}=j|\text{do}(\mathbf{O}_{i,t}=f_{1}(\mathbf{A}_{i ,t})),\ldots,\text{do}(\mathbf{O}_{i,t+k-1}=f_{k}(\mathbf{A}_{i,t+k-1}))\right],$ (1) $$

where the expectation is over the recommendation trajectory
$\mathbf{A}_{i,t:t+k-1}$ and
$\mathcal{F}^{k}=\mathcal{F}\times\cdots\times\mathcal{F}$.

In this case, the intervention $f_{t^{\prime}}(\mathbf{A}_{i,t+t^{\prime}-1})$
would affect the distribution of $\mathbf{A}_{i,t+t^{\prime}}$, the recommendations in the next time step.
This metric formalizes a notion of reachability where
the user can arbitrarily change the rating of
the recommended item, but the item trajectory \vsedit(the sequence of items they choose to rate) follows
the (future) dynamics of the recommender system itself.

In parallel, we define another metric that quantifies reachability
under edits to the history over the past $k$ time steps,
thus allowing the auditor to inspect user agency retrospectively.
Consider a recommender system at time $t$, that we call
the *factual* recommender system.
We denote the *factual* trajectory for a user
$i$ over the past $k$ time steps by
${\mathbf{h}}^{\star}_{i,t-k:}=(\mathbf{a}^{\star}_{i,t-k},\mathbf{o}^{\star}_{
i,t-k},\cdots,\mathbf{a}^{\star}_{i,t-1},\mathbf{o}^{\star}_{i,t-1})$.
Now, consider a *counterfactual* recommender system
that is retrained at time $t$ by modifying the ratings in
${\mathbf{h}}^{\star}_{i,t-k:}$ to a counterfactual history
$\mathbf{h}_{i,t-k:}$,
while keeping everything else the same as the *factual*
recommender system.

###### Definition 4.2 (Past- $k$ reachability) .

For any user $i\in[n]$ and item $j\in\mathcal{V}$, at time $t\in\mathbb{N}_{+}$,
their maximum past-$k$ reachability for $k\in\mathbb{N}_{+}$ is given by

$$ $\displaystyle\max_{f_{1:k}\in\mathcal{F}^{k}}\mathbb{E}_{{\mathbf{h}}^{\star}_ {i,t-k:}}\left[\mathbb{P}(\mathbf{A}_{i,t}=j|\text{do}({\mathbf{H}}_{i,t-k:}= \mathbf{h}_{i,t-k:}))\right],$ (2) $$

where $\mathbf{h}_{i,t-k:}=(\mathbf{a}^{\star}_{i,t-k},f_{1}(\mathbf{a}^{\star}_{i,t-
k}),\cdots,\mathbf{a}^{\star}_{i,t-1},f_{k}(\mathbf{a}^{\star}_{i,t-1}))$,
$\mathbf{A}_{i,t}$ denotes the counterfactual recommendation under the edited history
$\mathbf{h}_{i,t-k:}$.
The expectation is taken over the ‘factual’ history ${\mathbf{h}}^{\star}_{i,t-k:}$
and the edited history $\mathbf{h}_{i,t-k:}$
depends on the factual history ${\mathbf{h}}^{\star}_{i,t-k:}$.

This metric is a counterfactual quantity [, Ch. 7]
as it involves random variables from both the factual and counterfactual
recommender systems.
Informally, in past-$k$ reachability, we maximize the counterfactual
recommendation probability where the item trajectory is fixed
to the factual trajectory, but the ratings can be arbitrarily edited.
By contrast, in the future-$k$ metric, the trajectory
follows the recommender dynamics
(instead of being set to the factual trajectory).

###### Definition 4.1 (future- k 𝑘 k italic_k reachability) .

###### Definition 4.2 (Past- k 𝑘 k italic_k reachability) .

### 4.2 \llreplace Metrics on User Influence:Past- and Future-(In)stability

We introduce a new class of metrics that allow the auditor to measure the influence other users have on a particular user’s recommendations.
In addition to their own feedback and the recommendation algorithm itself,
a user’s autonomy is also influenced by other users’ feedback to the recommender systems.
The class of (in)stability metrics we define below allows auditors to inspect user agency through this angle.

###### Definition 4.3 (Future $k$ -(In)stability) .

Given user $i_{1},i_{2}\in[n]$ such that $i_{1}$ is the user of interest and $i_{2}$ is the user who can update their feedback,
at time $t\in\mathbb{N}_{+}$,
user $i_{1}$’s future-$k$ maximum instability with respect to user $i_{2}$ for $k\in\mathbb{N}_{+}$ is given by

$$ $\displaystyle\max_{f_{1:k}\in\mathcal{F}^{k}}$ $\displaystyle\mathbb{E}_{\mathbf{A}_{i_{2},t:t+k-1}}\bigg{[}d(\mathbb{P}( \mathbf{A}_{i_{1},t+k}|$ $\displaystyle do(\mathbf{O}_{i_{2},t}=f_{1}(\mathbf{A}_{i_{2},t})),\ldots,do( \mathbf{O}_{i_{2},t_{0}+k-1}=f_{k}(\mathbf{A}_{i_{2},t+k-1}))),\mathbb{P}( \mathbf{A}_{i_{1},t+k}))\bigg{]},$ (3) $$

where $d:\Delta\times\Delta\to\mathbb{R}_{\geq 0}$ measures the distance between two probability distributions.

In this case, an auditor seeks to intervene on the ratings given by user $i_{2}$ and observe how these interventions affect the recommendations received by user $i_{1}$, following the dynamics of the recommendation process.
The metric is defined as the deviance between the recommendation distribution before and after the intervention.
This allows the auditor to measure how unstable (or in another way, how manipulatable) the recommendation for user $i_{1}$ can be with respect to user $i_{2}$’s feedback.

Similar to the past-$k$ reachability metric (see Defn. 4.2),
we now define a counterfactual past-$k$ stability metric
which quantifies the stability of user $i_{1}$
under edits to the history of user $i_{2}$.
Informally, we seek to maximally change the recommendation
probability of item $j$ for user $i_{1}$ by editing the ratings of user $i_{2}$,
while keeping their item trajectory to the factual one.

###### Definition 4.4 (Past $k$ -(In)stability) .

Let ${\mathbf{h}}^{\star}_{i_{2},t-k:}=(\mathbf{a}^{\star}_{i_{2},t-k},\mathbf{o}^{
\star}_{i_{2},t-k},\cdots,\mathbf{a}^{\star}_{i_{2},t-1},\mathbf{o}^{\star}_{i
_{2},t-1})$
denote the *factual* recommendation history for user $i_{2}$.
We define user $i_{1}$’s past-$k$ maximum instability with respect to user $i_{2}$ for $k\in\mathbb{N}_{+}$ as

$$ $\displaystyle\max_{f_{1:k}\in\mathcal{F}^{k}}\mathbb{E}_{{\mathbf{h}}^{\star}_ {i_{2},t-k:}}\left[d\left(\mathbb{P}(\mathbf{A}_{i_{1},t}=j|\text{do}(\mathbf{ H}_{i_{2},t-k:}=\mathbf{h}_{i_{2},t-k:})),\mathbb{P}(\mathbf{A}_{i_{1},t}=j) \right)\right],$ (4) $$

where $\mathbf{h}_{i_{2},t-k:}=(\mathbf{a}^{\star}_{i_{2},t-k},f_{1}(\mathbf{a}^{
\star}_{i_{2},t-k}),\cdots,\mathbf{a}^{\star}_{i_{2},t-1},f_{k}(\mathbf{a}^{
\star}_{i_{2},t-1}))$ is the
edited counterfactual history
(which depends on the factual history ${\mathbf{h}}^{\star}_{i_{2},t-k:}$),
and $\mathbf{A}_{i_{1},t}$ is the counterfactual recommendation.

While both interventional and counterfactual metrics measures different aspects of user agency, there is a distinction in their scope:

- •
Past-/counterfactual metrics focus on how user behavior (e.g., the items a user chose to rate) contributes to the recommendations they receive in the present. For example, consider a social media user who primarily receives recommendations for cat videos in the present. Counterfactual metrics help us understand whether the narrowness in the recommendations can be attributed to the recommendation system, or the user’s behavior in the past, which imply vastly different conclusions in terms of user agency. If engaging with cat videos unfavorably in the past would have led to more diverse recommendations in the present, the observed narrow recommendations do not imply a violation of user agency.
- •
Future-/interventional metrics focus on the user’s radius of influence on recommendations in the future. Assume the user who likes cat videos adjusts their preferences and now shows interest in dog videos. Interventional metrics help us understand the ability of changed user behavior to steer upcoming recommendations. If alterations in behavior (engaging with more dog videos) do not lead to diversification of recommendations away from only cat content, we can conclude that there is a violation of user agency for their own recommendations.

Both reachability and stability can be conceptualized as measures of user agency, though the two metrics focus on different aspects of itThough both reachability and stability are measures of user agency, the two metrics focus on different aspects of it: stability focuses on the influences of behavior changes of other users, while reachability focuses on influences of the individual at hand.
\lleditEmpirically, in our experiments with embedding-based recommender systems, we observe a trade-off between these two metrics (Section 6). Further investigation into this trade-off is an interesting direction for future work.
\lldeleteFor the embedding-based recommender systems we experiment with in this paper, intuitively, higher reachability suggests that users have more control over their embeddings (through adjusting their ratings). When the item embeddings are relearned using these user embeddings or ratings, recommendations for other users will be influenced, suggesting lower stability of the system and a tradeoff between a model’s performance on these two classes of metrics. \lldeleteTraditional observational metrics used to evaluate recommender systems like accuracy aim to benchmark a model’s ability to learn the user’s static preferences but cannot quantify user agency or the control a user has over their own recommendations, necessitating this new regime of metrics.

In the following, we provide an operationalized procedure for computing these user-centric metrics; the exact implementation details can be found in Section 6 and Appendix C.

###### Definition 4.3 (Future k 𝑘 k italic_k -(In)stability) .

###### Definition 4.4 (Past k 𝑘 k italic_k -(In)stability) .

## 5 Operationalized Procedure for Computing \llreplace User-CentricUser Agency Metrics

When computing the user-centric causal metrics,
we need to consider two aspects:
(i) whether the auditor can intervene
on the recommender system, i.e.,
if they have access to the interventional distribution; and
(ii) whether they can obtain gradients of
the objective functions in (1)-(4).
By obtaining the gradients, the metrics can be computed through methods like gradient descent.
Similar to Dean et al. [],
we consider cases where the auditor can apply intervention on the recommender system,
thus addressing concerns around (i).
We provide methods that allow auditors to have different levels of access to the internals of the system (e.g., they may not directly have access to the required gradients).
Before delving into our methods,
we first provide a high-level discussion on the required gradients.
Consider the case where the auditor aims to compute (1) when $k=1$,
we need

$$ $\displaystyle\nabla_{f_{1}}\mathbb{E}_{\mathbf{A}_{i,t}}\left[\mathbb{P}( \mathbf{A}_{i,t+1}=j|\text{do}(\mathbf{O}_{i,t}=f_{1}(\mathbf{A}_{i,t})))\right]$ $\displaystyle=$ $\displaystyle\sum_{\mathbf{a}_{i,t}}\mathbb{P}(\mathbf{A}_{i,t}=\mathbf{a}_{i, t})\nabla_{f_{1}}\mathbb{P}(\mathbf{A}_{i,t+1}=j|\text{do}(\mathbf{O}_{i,t}=f_ {1}(\mathbf{a}_{i,t}))),$ $$

where $\nabla_{f_{1}}$ refers to gradient with respect to the parameters of $f_{1}$
and the equality holds because of Fubini’s theorem.
In this case, the auditor needs to effectively compute $\nabla_{f_{1}}\mathbb{P}(\mathbf{A}_{i,t+1}=j|\text{do}(\mathbf{O}_{i,t}=f_{1}
(\mathbf{a}_{i,t})))$ for all $\mathbf{a}_{i,t}\subset\mathcal{V}$.
In the following,
we first discuss cases where
the auditor has access to the actual gradients.
We show that under mild conditions, for matrix factorization based recommender systems,
the objective in (2) is concave
and (4) is quasi-convex and has its optimum at extreme points (Section 5.1).
For cases where the auditor only has access to zeroth-order information of the system, we provide an effective approach for approximating the required gradient (Section 5.2).
\vseditWe show simplified form of the gradients we require access to for future-$k$ reachability/stability in Appendix F, similar to the one step gradient above.

### 5.1   White-box Access

In the most desirable setting, the auditor has what we term as white-box access to the recommender system.
In this case, the auditor has access to the gradient of the interventional distribution of interest.
We consider two types of score-based systems where the recommendation probability is given by

$$ $\displaystyle\mathbb{P}(\mathbf{A}_{i,t+1}=j|\text{do}(\mathbf{O}_{i,t}=f_{1}( \mathbf{a}_{i,t})))\propto\exp(\beta\mathbf{p}_{i}^{\top}\mathbf{q}_{j}),$ (5) $$

where $\beta>0$ controls the stochasticity of the system,
the user embedding $\mathbf{p}_{i}$ and item embedding $\mathbf{q}_{j}$ depend on the intervention $\mathbf{O}_{i,t}$ (and $\mathbf{O}_{-i,t})$.
The two types of systems learn $\mathbf{p}_{i}$ and $\mathbf{q}_{j}$ differently—one uses matrix factorization and the other uses an LSTM-based approach.
For more details, we refer the readers to Appendix A.1.

To obtain the gradient $\nabla_{f_{1}}\exp(\beta\mathbf{p}_{i}^{\top}\mathbf{q}_{j})$,
we need to know how the systems updates their user and item embeddings when user ratings change.
For matrix factorization models, when computing reachability (i.e., when users change their own ratings),
we assume item embedding $\{\mathbf{q}_{j}\}_{j\in\mathcal{V}}$ are fixed and user embedding $\mathbf{p}_{i}$ is updated according to the matrix factorization objective.
Effectively, we are assuming that the user’s own ratings only affects their own embedding.
The closed form of the update can be found in Appendix A.2.
Under this assumption, we have the below result.

###### Proposition 5.1 .

When $\{\mathbf{p}_{i}\}_{i\in[n]}$ and $\{\mathbf{q}_{j}\}_{j\in\mathcal{V}}$ are learned through matrix factorization,
the past-$k$ reachability objective (2) is concave in the parameters of $f_{1:k}$
if item embeddings $\{\mathbf{q}_{j}\}_{j\in\mathcal{V}}$ are fixed and $\mathbf{p}_{i}$ is updated according to the matrix factorization objective when the ratings $\mathbf{O}_{i}$ are changed.

For computing stability (i.e., when other users change their ratings),
we assume the user embedding $\{\mathbf{p}_{i}\}_{i\in[n]}$ are fixed and item embedding $\mathbf{q}_{j}$ is updated according to the matrix factorization objective.
Effectively, we are assuming that other users affect user $i$’s recommendation through item embeddings.
The closed form of the update can be found in Appendix A.2.
Under this assumption, we have that past-$k$ stability is obtained at boundary points.

###### Proposition 5.2 .

When $\{\mathbf{p}_{i}\}_{i\in[n]}$ and $\{\mathbf{q}_{j}\}_{j\in\mathcal{V}}$ are learned through matrix factorization,
the past-$k$ stability objective (4) is quasi-convex in the parameters of $f_{1:k}$ and achieves its optimal value at a boundary point of the domain,
if user embeddings $\{\mathbf{p}_{i}\}_{i\in[n]}$ are fixed and item embeddings $\{\mathbf{q}_{j}\}_{j\in\mathcal{V}}$ are updated according to the matrix factorization objective when user $i_{2}$’s ratings $\mathbf{O}_{i_{2}}$ are changed, $d$ is the $L_{2}$ distance.

We note that the definitions of the proposed metrics don’t require any specific assumptions on how the recommender is trained and we only consider one fixed embedding here for ease of operationalization, borrowing this approach from [, , ]. In our experiment, we also consider user embedding $\mathbf{p}_{i}$ and item embedding $\mathbf{q}_{j}$ to be the output of an LSTM that takes in user and item history, respectively. In this case, if the auditor has the corresponding LSTM, they may obtain gradient of $\mathbf{p}_{i}$ and $\mathbf{q}_{j}$ with respect to user history.
We provide more details on this in Section 6.

###### Proposition 5.1 .

###### Proposition 5.2 .

### 5.2   Black-box Access

In certain cases, the auditor can only obtain the output of a recommender through querying the system with different inputs,
but no direct access to the recommendation mechanism itself (thus no access to the required gradients).
In this case, we leverage the traditional toolkit to estimate the gradient using finite difference methods (e.g., []):

$$ $\displaystyle\nabla_{\theta}\mathbb{P}(\mathbf{A}_{i,t+1}=j|\text{do}(\mathbf{ O}_{i,t}=f_{\theta}(\mathbf{a}_{i,t})))$ $\displaystyle\approx$ $\displaystyle\frac{1}{2\epsilon}|\mathbb{P}(\mathbf{A}_{i,t+1}=j|\text{do}( \mathbf{O}_{i,t}=f_{\theta+\epsilon\mathbf{z}}(\mathbf{a}_{i,t})))-\mathbb{P}( \mathbf{A}_{i,t+1}=j|\text{do}(\mathbf{O}_{i,t}=f_{\theta-\epsilon\mathbf{z}}( \mathbf{a}_{i,t})))|,$ $$

where $\theta\in\mathbb{R}^{d}$, $\epsilon>0$ is a perturbation scale, and $\mathbf{z}\sim\mathcal{N}(0,\mathbf{I}_{d})$.
For more details on using zeroth-order information to perform optimization for a neural network, we refer the readers to [].

## 6 Experiments

We conduct experiments to show how our proposed metrics of reachability and stability vary for different recommender systems, across different time horizons and as the stochasticity of the recommender system changes.
\llcommentLeqi should change the next sentence.
Since these metrics are meant to be proxies of user agency, we show how our the aforementioned experimental parameters affect user agency, by observing how these measurements change as we change these parameters.

### 6.1   Experimental Setup

##### Dataset.

Our experiments are based on the publicly available MovieLens-1M Dataset [] which contains 1,000,209 total ratings given by 6,040 users on 3,706 movies
(see Table I in the Appendix for the dataset summary statistics).
The ratings are integer values between 1 and 5, both inclusive.

##### Recommender Systems.

We perform experiments with a Matrix Factorization (MF) recommender system [] from Surprise[]
and a Recurrent Recommender Network []
(additional details about these recommender systems and implementation details on both reachability and stability are included in Appendix C).
For future-facing metrics, we use a *deterministic* item selection
policy that always recommends the top-$1$ item.
For past-facing metrics, we use the *stochastic* softmax policy in (5).

##### Experimental Parameters.

There are two main parameters for the metrics:

- •
Time Horizon $t$: For our experiments, we show a comparison between $t=1$ and $t=5$ to illustrate the point that, with a longer time horizon, users are more likely to ‘reach’ an item and adversaries are more likely to manipulate their recommendations. We performed these experiments with a time horizon of $t=1,2,3,4,5$ but chose to show only t=1 vs t=5 to show the effect of user agency under short vs long time horizon.
- •
Stochasticity ($\beta$):For our experiments, we show a comparison between $\beta=0.8$ and $\beta=5$ to illustrate the point that, with a longer time horizon, users are more likely to ‘reach’ an item and adversaries are more likely to manipulate their recommendations. We performed these experiments with other values of $\beta$ but chose to show only $\beta=0.8$ vs $\beta=5$ to show the effect of user agency under short vs long time horizon.

Additional plots with different experimental parameters can be found in the Appendix E.

### 6.2   Comparison among metrics and algorithms

##### Algorithm comparisons.

We demonstrate that zeroth order optimization techniques with Black Box Access as described in Section 5 can serve as a viable optimization strategy.
We use the ratio between the final probability of an item being recommended to a user after intervention and the initial probability of the item being recommended to the user before any intervention to measure the gain in reachability.
This has been referred to as “Lift” in [], \vseditwhile the initial probability of the item being recommended to the user before any intervention is termed “baseline reachability.”
We compare the mean values of Lift obtained for both past-$5$ and future-$5$ reachability and stability
using gradient descent with those obtained using black box access (see Figure 3(a) for these comparisons).
For reachability, gradient descent (GD) outperforms black box access, but black box access still obtains a solution with Lift $>1$.
For stability, black box access converges to the same solution as GD.
For stability, we also plot the value obtained by
exhaustively searching the extreme points of the
rating domain, represented by “Oracle” in 3(b).

Figure: Figure 2: Scatterplot of Past-5-Reachability for a Matrix Factorization based recommender as $\beta$ varies
Refer to caption: extracted/5867308/images/log3.png

Figure: (a) Mean value of reachability Lift
Refer to caption: extracted/5867308/images/mezo1.png

##### Metric comparisons.

Allowing both past and future edits for longer time horizons lead to higher gains in reachability compared to shorter time horizons
as the user has more freedom to change their ratings.
Meanwhile, allowing edits over a longer time horizon decreases the stability of a user’s recommendations on average. We observe that most user-adversary pairs have instability values close to 0 or 1, indicating a duality where a user’s recommended list is either heavily affected by the actions of an adversary or is minimally affected by them.
Figure 4 shows these results.

Figure: (a) Scatterplot of Future Reachability for different time horizons (Matrix Factorization)
Refer to caption: extracted/5867308/images/future_reach_mf.png

### 6.3   Investigating the Design of Recommender Systems

##### Stochasticity.

A higher $\beta$ leads to less stochastic recommendations.
The plot in Figure 2 shows the past-$5$ reachability of multiple user-item pairs for $\beta=0.8$ and $\beta=5$.
We observe higher values of Lift (gain in reachability) for lower stochasticity.
Moreover, these values are higher for items with high baseline reachability values in both cases, as the points move away from the $y=x$ line for higher baseline reachability. For $\beta=5$, we observe multiple Lift values of the order of $10^{3}$ but for $\beta=0.8$, we only see lift values slightly greater than 1 for the most part, or of the order of $10$ at best.

As we increase $\beta$, or decrease the stochasticity of the system, user recommendations tend to become more stable, as shown in Figure 5(a) where we plot mean values of instability for multiple user-adversary pairs and varying $\beta$ from $0.2$ to $6$ in discrete steps.
The intuition for this is that a lower stochasticity implies that the user is pushed more towards deterministic recommendations, and in the most extreme case, their recommendation list is perfectly stable, with the entire probability distribution of recommendation being concentrated on one item.
We also plot histograms for instability values (Figure 5(b)) at $\beta=0.8$ and $\beta=5$ and see that a greater percentage of items for $\beta=5$ have their instability values concentrated around 0 than do for $\beta=0.8$, which matches the prior observation.

Figure: (a) Past Stability values (MF) as $\beta$ varies
Refer to caption: extracted/5867308/images/stab_linspace.png

##### Nature of Recommender Systems

We observe that the Matrix Factorization based recommender system has higher reachability than the Recurrent Recommender Network when averaged over a large number of user-item pairs. On the other hand, the Recurrent Recommender Network consistently has higher stability than the Matrix Factorization based system when averaged over a large number of (user-user) pairs.
The differences in Figures 6 and 4 illustrate this, as the reachability scale is lower than that for MF in \lleditFigure 4(a) and there is no concentration of instability values around $1$ unlike in \lleditFigure 4(b). \vseditWe infer that the Matrix Factorization based system facilitates more reachability but is less stable than it’s Recurrent Recommender Network counterpart\lldelete, indicating that the more complex deep learning based model may allow less user agency in this setting \llcommentNot sure if we can conclude this..

Figure: (a) Scatterplot of Future Reachability for different time horizons for a recurrent recommender network
Refer to caption: extracted/5867308/images/future_reach_rrn.png

## 7 Conclusion

This paper presents a causal framework to formalize
interventional and counterfactual metrics to audit
and evaluate recommender systems in a principled manner.
Addressing the problem of user agency in recommender systems,
we present the metrics of *reachability*
and *stability* to quantify a user’s agency
over their recommendations under \lldeletecounterfactual edits to their own or a different user’s history.
\vseditIn our experiments, we explicitly investigated the implication of user agency defined by reachability and stability with respect to two design choices of recommender systems: stochasticity in the system and the recommendation algorithm. We find that with more stochasticity, users’ agency towards their own recommendation increases as indicated by the increase of reachability. However, higher stochasticity also allows other users’ to potentially have more control over the systems, indicated by lower stability of the system. Comparing recommendation algorithms, we observed that Matrix Factorization has higher reachability than the Recurrent Recommender Network, suggesting that more complex models may allow less user agency \lleditin some aspects. This could be attributed to the fact that small changes to a dataset have low influence on large-scale deep learning systems. An interesting future step is to \lleditinvestigate the tradeoff between reachability and stability across systems, and implement the proposed metrics in real-world systems, further confirming their validity in practice.

## Appendix A Details for Section 5

### A.1   Score-based Recommender Systems

Given data of $T$ time steps,

$$ $\displaystyle\min_{f\in\mathcal{F},g\in\mathcal{G}}\sum_{t\in[T]}\sum_{i\in[n] }(\mathbf{o}_{i,t}[j]-f(\mathbf{h}_{i,t})^{\top}g(\mathbf{h}^{v}_{j,t}))^{2}+ \Omega(f,g),$ (6) $$

where $\mathbf{h}^{v}_{j,t}$ is the rating history for item $j$ so far.
In matrix factorization,
the user representation $f(\mathbf{h}_{i,t})=\mathbf{p}_{i}\in\mathbb{R}^{d}$ and item representation $g(\mathbf{h}^{v}_{j,t})=\mathbf{p}_{j}\in\mathbb{R}^{d}$ are learnable constant vectors.
Another setting could be that $f,g$ are both neural networks, e.g., LSTMs, that map user and item history to a $d$-dimensional vector.
In both cases, the regularizer $\Omega(f,g)$ are the norms of the user and item vectors.
Once a score $\hat{s}_{i,j,t}=\hat{f}(\mathbf{h}_{i,t})^{\top}\hat{g}(\mathbf{h}^{v}_{j,t})$ is learned using the dataset $\mathbf{d}_{t}$, the recommendation probability $\mathbb{P}(\mathbf{A}_{i,t}=j|\text{do}(\mathbf{D}_{t}=\mathbf{d}_{t}))$ for example can be a pointmass $\mathbb{I}\{j=\operatorname*{arg\,max}_{j\in\mathcal{V}}\hat{s}_{i,j,t}\}$, or follows a softmax policy $e^{\beta\hat{s}_{i,j,t}}/(\sum_{l}e^{\beta\hat{s}_{i,l,t}})$ for $\beta\geq 0$ controlling how stochastic one wants the recommender to be.

### A.2   Proofs

For the proofs, we consider a matrix factorization model with learned user factors given by $P$ $\in$ $\mathbb{R}^{n\times d}$ and learned item factors given by $Q\in\mathbb{R}^{m\times d}$.

###### Proof of Proposition 5.1 .

For a fixed factual recommendation trajectory ${\mathbf{h}}^{\star}_{i,t-k:}=(\mathbf{a}^{\star}_{i,t-k},\mathbf{o}^{\star}_{
i,t-k},\cdots,\mathbf{a}^{\star}_{i,t-1},\mathbf{o}^{\star}_{i,t-1})$,
the optimization problem in (2) is

$$ $\displaystyle\max_{\mathbf{o}_{1:k}\in\mathbb{R}^{k}}\mathbb{P}(\mathbf{A}_{i, t}=j|\mathbf{A}_{i,t-k}^{\star}=\mathbf{a}_{i,t-k}^{\star},\text{do}(\mathbf{O }_{i,t-k}=\mathbf{o}_{1}),\ldots,\mathbf{A}_{i,t-1}^{\star}=\mathbf{a}_{i,t-1} ^{\star},\text{do}(\mathbf{O}_{i,t-1}=\mathbf{o}_{k})),$ $$

In this setting, the user $i$ modifies the ratings $\mathbf{o}_{1:k}$
for the items in the factual trajectory $\mathbf{a}_{i,t-k:t-1}^{\star}$.
Since retraining the entire recommender system is not feasible after
every user interaction,
we make the following simplifying assumption for how the user vector
is updated after a rating $o_{k}$ is modified.

Assumption:
After each rating user $i$ modifies, the user vector $P_{i}$ (the $i^{\text{th}}$ row of $P$)
is updated but $Q$ is kept unchanged.
The objective of matrix factorization is to solve following expression, where $R$ is the user-item rating matrix:

$$ $\min_{P,Q}\|PQ^{T}-R\|_{2}^{2}$ $$

Under this assumption, the updated user vector after every interaction is given by:

$$ $P_{i}=\arg\min_{p^{\prime}}\sum_{v\in V}(p^{\prime T}Q_{v}-R_{iv})^{2}$ $$

where $V$ is the subset of items the user $i$ has interacted with. Every interaction adds an additional element to $V$ and populates $R_{iv}$.
After $k$ timesteps(from $t-k$ to $t-1$), this optimization problem has a simple closed-form solution given by:

$$ $P_{i}=(Q_{\text{rated}}^{T}Q_{\text{rated}})^{-1}Q_{\text{rated}}^{T}R_{\text{ rated}},$ $$

where

$$ $Q_{\text{rated}}=\begin{bmatrix}Q_{\mathbf{a}_{i,1}^{\star}}\\ Q_{\mathbf{a}_{i,2}^{\star}}\\ \vdots\\ Q_{\mathbf{a}_{i,t-1}^{\star}}\end{bmatrix},\quad R_{\text{rated}}=\begin{ bmatrix}R_{i,1}\\ \vdots\\ R_{i,t-k-1}\\ \mathbf{o}_{1}\\ \mathbf{o}_{3}\\ \vdots\\ \mathbf{o}_{k}\end{bmatrix}.$ $$

$Q_{\text{rated}}\in\mathbb{R}^{(t-1)\times d}$ is a matrix whose rows are the item vectors of the items rated by the user upto time $t$
and $R_{\text{updated}}$ is a column vector of size $t_{0}-1$ which represents all the corresponding ratings given to these items by the user $i$ up to time $t$ with the last $k$ ratings being replaced by $\mathbf{o}_{1},\mathbf{o}_{2},\dots\mathbf{o}_{k}$ as discussed above.

We assume a stochastic $\beta$-softmax selection rule given by:

$$ $\mathbb{P}(\mathbf{A}_{i,t_{0}}=j)=\frac{e^{\beta s_{ij}}}{\sum_{k\in V}e^{ \beta s_{ik}}}$ $$

where $s_{ij}=P_{i}^{T}Q_{j}$ is the predicted rating for the interaction between user $i$ and item $j$.
In this case,

$$ $\mathbb{P}(\mathbf{A}_{i,t}=j|\mathbf{A}_{i,t-k}^{\star}=\mathbf{a}_{i,t-k}^{ \star},\text{do}(\mathbf{O}_{i,t-k}=\mathbf{o}_{1}),\ldots,\mathbf{A}_{i,t-1}^ {\star}=\mathbf{a}_{i,t-1}^{\star},\text{do}(\mathbf{O}_{i,t-1}=\mathbf{o}_{k} ))=\frac{e^{\beta P_{i}^{T}Q_{j}}}{\sum_{k\in V}e^{\beta P_{i}^{T}Q_{k}}}$ $$

Maximizing this is equivalent to maximizing the logarithm of this quantity, given by:

$$ $\beta P_{i}^{T}Q_{j}-\underset{\text{k $\in$ $V$}}{\text{LSE}}(\beta P_{i}^{T} Q_{k})$ $$

where LSE denotes the log-sum-exponential.
We can rewrite the objective as:

$$ $\max_{\mathbf{o}_{1,\ldots k}\in\mathbb{R}^{k}}\beta P_{i}^{T}Q_{j}-\underset{ \text{k $\in$ $V$}}{\text{LSE}}(\beta P_{i}^{T}Q_{k})=\min_{\mathbf{o}_{1, \ldots k}\in\mathbb{R}^{k}}-\beta P_{i}^{T}Q_{j}+\underset{\text{k $\in$ $V$}} {\text{LSE}}(\beta P_{i}^{T}Q_{k})$ $$

We can see that $P_{i}^{T}Q_{j}$ is a linear function in all $\mathbf{o}_{1},\dots\mathbf{o}_{k}$ since every $Q_{j}$ is independent of $\mathbf{o}_{1},\mathbf{o}_{2}\dots\mathbf{o}_{k}$.

Let $\mathbf{o}$ = $[\mathbf{o}_{1},\dots\mathbf{o}_{k}]^{T}$. Then, for some $b_{k}$ $\in$ $\mathbb{R}^{t}$ and $c_{k}\in\mathbb{R}\forall k\in V$, the objective becomes:

$$ $\min_{\mathbf{o}_{1,\ldots k}\in\mathbb{R}^{k}}\underset{\text{k $\in$ $V$}}{ \text{LSE}}\left(\beta(b_{k}^{T}\mathbf{o}+c_{k})\right)-\beta(b_{j}^{T} \mathbf{o}+c_{j})$ $$

This is convex because log-sum-exp is a convex function, affine functions are convex, and the composition of a convex and an affine function is convex.
∎

###### Lemma A.1 .

Let $f:\mathbb{R}^{n}\rightarrow\mathbb{R}$ be a convex function with only one root, i.e., $f(\mathbf{x}_{0})=0$ for some $\mathbf{x}_{0}\in\mathbb{R}^{n}$. Then, $g(\mathbf{x})=[f(\mathbf{x})]^{2}$ is a quasi-convex function.
Proof:
To prove that $g(\mathbf{x})$ is quasi-convex, we need to show that for any $\alpha\in\mathbb{R}$, the sublevel set $S_{\alpha}=\{\mathbf{x}\in\mathbb{R}^{n}|g(\mathbf{x})\leq\alpha\}$ is a convex set.

- 1.
If $\alpha<0$, the sublevel set $S_{\alpha}$ is empty because $[f(\mathbf{x})]^{2}\geq 0$ for all $\mathbf{x}\in\mathbb{R}^{n}$. An empty set is convex by definition.
- 2.
If $\alpha=0$, the sublevel set $S_{0}=\{\mathbf{x}_{0}\}$ because $f(\mathbf{x}_{0})=0$ and $[f(\mathbf{x})]^{2}>0$ for all $\mathbf{x}\neq\mathbf{x}_{0}$. A singleton set is convex.
- 3.
If $\alpha>0$, consider the sublevel set $S_{\alpha}=\{\mathbf{x}\in\mathbb{R}^{n}|[f(\mathbf{x})]^{2}\leq\alpha\}$. We can rewrite this as:
$S_{\alpha}=\{\mathbf{x}\in\mathbb{R}^{n}|-\sqrt{\alpha}\leq f(\mathbf{x})\leq
\sqrt{\alpha}\},$
which is a convex set for any convex $f$.

###### Proof of Proposition 5.2 .

For a fixed factual recommendation trajectory
${\mathbf{h}}^{\star}_{i_{2},t-k:}=(\mathbf{a}^{\star}_{i_{2},t-k},\mathbf{o}^{
\star}_{i_{2},t-k},\cdots,\\
\mathbf{a}^{\star}_{i_{2},t-1},\mathbf{o}^{\star}_{i_{2},t-1})$,
the optimization problem in (4) can be written as:

$$ $\max_{\mathbf{o}_{1,\ldots k}\in\mathbb{R}^{k}}d(\mathbb{P}(\mathbf{A}_{i_{1}, t}|\mathbf{A}_{i_{2},t-k}^{\star}=\mathbf{a}_{i_{2},t-k}^{\star},\text{do}( \mathbf{O}_{i_{2},t-k}=\mathbf{o}_{1}),\ldots,\mathbf{A}_{i_{2},t-1}^{\star}= \mathbf{a}_{i_{2},t-1}^{\star},\text{do}(\mathbf{O}_{i_{2},t-1}=\mathbf{o}_{k} )),\mathbb{P}(\mathbf{A}_{i_{1},t}))$ $$

In this setting, the user $i_{2}$ modifies the ratings
of the factual item trajectory $\mathbf{a}_{i_{2},t-k:t-k+1}^{\star}$.
Since retraining the entire recommender system is not feasible after
every user interaction,
we make the following simplifying assumption for how the item vector
is updated after a rating $o_{k}$ is modified.

Assumption: After $i_{2}$ rates an item $j$, the item vector $Q_{j}$(the $j^{\text{th}}$ row of $Q$) is updated but $P$ is kept unchanged.
Similar to the proof past-$k$ reachability, the updated item vector for an item $j$ rated by $i_{2}$ is given by:

$$ $Q_{j}=(P_{\text{rated}}^{T}P_{\text{rated}})^{-1}P_{\text{rated}}^{T}R_{\text{ rated}},$ $$

where

$$ $P_{\text{rated}}=\begin{bmatrix}P_{\mathbf{\mu}^{*}_{j,1}}\\ \vdots\\ P_{i_{2}}\\ \vdots\\ P_{\mathbf{\mu}^{*}_{j,t-1}}\end{bmatrix},\quad R_{\text{rated}}=\begin{ bmatrix}R_{j,1}\\ \vdots\\ \mathbf{o}_{t}\\ \vdots\\ R_{j,t-1}\end{bmatrix}.$ $$

Here, $P_{\text{rated}}$ is a matrix $\in\mathbb{R}^{(t-1)\times d}$ whose rows are the user vectors of the users that rated the item $j$ up to $t$, that is represented by $\mathbf{\mu}^{*}_{j,t}$;
with user $i_{2}$ rating it at $t-k+t-1$ and $R_{\text{rated}}$ is a column vector of size $t-1$ that represents all the corresponding ratings given by these users to $j$, with the rating at time $t+k^{\prime}-1$ being replaced with $\mathbf{o}_{k^{\prime}+k}$.

We assume a stochastic $\beta$-softmax selection rule given by:

$$ $\mathbb{P}(\mathbf{A}_{i,t_{0}}=j)=\frac{e^{\beta s_{ij}}}{\sum_{k\in V}e^{ \beta s_{ik}}}$ $$

where $s_{ij}=P_{i}^{T}Q_{j}$ is the predicted rating for the interaction between user $i$ and item $j$.

The $L_{2}$ distance metric is defined as the defined as the $L_{2}$ distance between the discrete probability probability distributions of $i_{1}$’s next recommended item before and after $i_{2}$’s modifies their ratings.

$$ $f(\mathbf{o})=\sum_{j\in V}\Bigg{(}\mathbb{P}\Big{(}\mathbf{A}_{i_{1},t_{0}}=j \mid\mathbf{A}_{i_{2},t_{0}-k}^{\star}=\mathbf{a}_{i_{2},t_{0}-k}^{\star}, \text{do}\left(\mathbf{O}_{i_{2},t_{0}-k}=\mathbf{o}_{1}\right),\ldots,\mathbf {A}_{i_{2},t_{0}-1}^{\star}=\mathbf{a}_{i_{2},t_{0}-1}^{\star},\\ \text{do}\left(\mathbf{O}_{i_{2},t_{0}-1}=\mathbf{o}_{k}\right)\Big{)}-\mathbb {P}\left(\mathbf{A}_{i_{1},t_{0}}=j\right)\Bigg{)}^{2}$ $$

We note that the predicted rating of a user-item interaction only changes when the item is one of the items for which the user $i_{2}$ modifies their ratings. In this case, we can see that the predicted rating given by $i_{1}$ on an item $i_{2}$ rates $\mathbf{o}_{k}$ is a linear function in $\mathbf{o}_{k}$. Our objective can be written as $f(\mathbf{o})=\sum_{j\in V}g_{j}(\mathbf{o})$, where

$$ $g_{j}(\mathbf{o})=\left(\frac{e^{\beta(c_{i_{1}j}\mathbf{o}_{k}+d_{i_{1}j})}}{ \sum_{k\in V}e^{\beta(c_{i_{1}k}\mathbf{o}_{k}+d_{i_{1}k})}}-\mathbb{P}( \mathbf{A}_{i_{1},t_{0}}=j)\right)^{2}$ $$

We can rewrite this as:

$$ $g_{j}(r)=\left(\frac{e^{\beta(c_{i_{1}j}\mathbf{o}_{k}+d_{i_{1}j})}}{\sum_{k \in V}e^{\beta(c_{i_{1}k}\mathbf{o}_{k}+d_{i_{1}k})}}-C_{j}\right)^{2}$ $$

where $C_{j}=\mathbb{P}\left(\mathbf{A}_{i_{1},t_{0}}=j\right)$ is a constant.
Now, let’s consider the function:

$$ $h_{j}(\mathbf{o})=\frac{e^{\beta(c_{i_{1}j}\mathbf{o}_{k}+d_{i_{1}j})}}{\sum_{ k\in V}e^{\beta(c_{i_{1}k}\mathbf{o}_{k}+d_{i_{1}k})}}$ $$

This function is the softmax function, which is known to be convex (Boyd and Vandenberghe, 2004).
Next, consider the function:

$$ $l_{j}(\mathbf{o})=h_{j}(\mathbf{o})-C_{j}$ $$

The difference of a convex function and a constant is also convex (Boyd and Vandenberghe, 2004). Therefore, $l_{j}(\mathbf{o})$ is convex.
Finally, let’s look at:

$$ $g_{j}(\mathbf{o})=(l_{j}(\mathbf{o}))^{2}$ $$

Each $l_{j}$ is a monotonic convex function with exactly one root.
By Lemma A.1, each $g_{j}(\mathbf{o})$
is quasiconvex.
Now, we can write the stability objective function as:

$$ $f(\mathbf{o})=\sum_{j\in V}g_{j}(\mathbf{o})$ $$

The sum of quasiconvex functions is also quasiconvexconvex (Boyd and Vandenberghe, 2004).
Since $f(r)$ is quasiconvex, in practice,
if we optimize each $o_{k}$ in the interval $[a,b]\subset\mathbb{R}$,
the maximum value of $f(r)$ is attained at an extreme point of $[a,b]$ (Boyd and Vandenberghe, 2004).
In other words, the stability objective is quasiconvex and achieves its optimal value at a boundary point of the domain.
∎

###### Proof of Proposition 5.1 .

###### Lemma A.1 .

###### Proof of Proposition 5.2 .

## Appendix B Algorithm Details

- •
Past Reachability: Details provided in 1. We use a model trained up to time $t_{0}-k$, and then aim to optimize the user’s ratings for the factual next $k$ items with the objective of maximizing the probability of recommending to item to be reached after $k$ steps, with a stochastic recommendation policy given by:
$\mathbb{P}(\mathbf{A}_{i,t_{0}}=j)=\frac{e^{\beta s_{ij}}}{\sum_{k\in V}e^{
\beta s_{ik}}}$
where $s_{ij}$ is the predicted rating for the interaction between user $i$ and item $j$.
This is a stochastic selection rule controlled by a parameter of stochasticity $\beta$ to select items after the recommender system ranks them. There are $k$ parameters of optimization, where $k$ is the time horizon, corresponding to the user’s action at each time step.
- •
Future Reachability: Details provided in 2. We use a model trained up to time $t_{0}$ and the user follows the recommendations given by a deterministic recommender system that returns the top-1 item that the user hasn’t already rated based on predicted score, with the same objective mentioned above. There are $k\times|V|$ parameters of optimization here, where $|V|$ is the size of the item vocabulary. The ($k\cdot m+t$)th parameter represents the user’s action if they were to be recommended item $m$ at time $t_{0}+t$.
- •
Past Stability: Details provided in 3. We use a model trained up to time $t_{0}-k$, and then aim to optimize the adversary’s ratings for the factual next $k$ items with the objective of maximally changing the user’s recommended list, with the same stochastic recommendation policy mentioned above. We use Hellinger Distance as our distance metric. There are $k$ parameters of optimization, where $k$ is the time horizon, corresponding to the adversary’s action at each time step.
- •
Future Stability: Details provided in 4. We use a model trained up to time $t_{0}$ and the adversary follows the recommendations given by a deterministic recommender system that returns the top-1 item that the user hasn’t already rated based on predicted score, with the same objective mentioned above. There are $k\times|V|$ parameters of optimization here, where $|V|$ is the size of the item vocabulary. The ($k\cdot m+t$)th parameter represents the adversary’s action if they were to be recommended item $m$ at time $t_{0}+t$.

Note: We use a deterministic selection rule for computing future based metrics because it does not suffer from too much variation across epochs, unlike the stochastic selection rule, which would require an even larger parameter space to account for every possible item sequence.

Figure: Algorithm 1 past-$k$ reachability

Figure: Algorithm 2 future-$k$ reachability

Figure: Algorithm 3 past-$k$ stability

Figure: Algorithm 4 future-$k$ stability

## Appendix C Experimental Details

##### Dataset Summary Statistics

The table below shows some dataset summary statistics.

**Table I: Statistics and performance metrics for the ML 1M dataset.**
| Data set | ML 1M |
| --- | --- |
| Users | 6040 |
| Items | 3706 |
| Ratings | 1000209 |
| Density (%) | 4.47% |

##### Additional Recommender System Details:

We choose $100$ as the size of both user and item latent vectors for matrix factorization.

For the Recurrent Recommender Network, we train two LSTMs in parallel: one that takes a user’s item history as input and outputs a user vector and another that takes an item’s user history as input and outputs an item vector.

The user’s item history is a list of the form [($\text{item$_$id}_{k}$, $\text{rating}_{k}$)] where $k$ varies from $1$ to $n$, if $n$ is the number of interactions the user has had upto this point.

Each LSTM has input size $2$ and hidden size $100$, and we choose the last hidden state as the user/item vector.

Similar to general factorization based recommenders, the rating for a particular interaction is given by the dot product of the user and item vector for the (user,item) pair involved in the interaction.

For training, we first sort every interaction in the dataset by timestep and attempt to predict the rating for the next interaction based on interactions that have already taken place.

We train three versions of each recommender system:
one with complete data, one with the last item for each user left out and one with the last 5
items rated by each user left out in order to run our experiments.

Updating the recommender system after every interaction: We acknowledge that re-training the recommender system after every new user-item interaction is not feasible and is not done in practice in commercial systems[]. Both matrix factorization and recurrent recommender networks are inherently factorization based and create latent user and item representations that are updated whenever the model retrains. Instead of retraining the model, for computing reachability based metrics, we operate under the assumption that the item latent vectors remain fixed while we update the user latent vectors by solving a closed form equation given in 7 for matrix factorization. Similarly, for computing stability based metrics, we operate under the assumption that the user latent vectors remain fixed while we update the item latent vectors by solving a closed form equation given in 8 for matrix factorization. We borrow this approach from [].

$$ $p=\underset{p^{\prime}}{\operatorname{argmin}}\sum_{v\in\mathcal{V}_{\text{ seen }}}\left(p^{\prime T}q_{v}-r_{v}\right)^{2}$ (7) $$

$$ $q=\underset{q^{\prime}}{\operatorname{argmin}}\sum_{u\in\mathcal{U}_{\text{ seen }}}\left(p_{u}^{T}q^{\prime}-r_{u}\right)^{2}$ (8) $$

Here, $p$ denotes the user latent vector and $q$ denotes the item latent vector. $\mathcal{V}_{\text{seen }}$ is the list of items the user has rated, $r_{v}$ is the rating given to item $v$. Similarly, $\mathcal{U}_{\text{seen }}$ is the list of users the item has been rated by and $r_{v}$ is the rating given by user $u$.

For the recurrent recommender network, this is even more straightforward as we leave the LSTM as is and simply query it with the edited user/item history to obtain new user/item vectors

##### Settings for plots:

Regarding the plots in 6,

- •
For reachability based metrics, we randomly choose 10$\%$ of users as set $A$ and 10$\%$ of items as set $B$ and compute the reachability of every (user,item) pair in $A\times B$.
- •
For stability based metrics, we randomly choose 20$\%$ of users and divide them equally among set $A$ and set $B$ to serve as the primary user set and adversary set respectively. We then compute the stability of every combination of (user,adversary) pair in $A\times B$.

##### Compute Requirements

We make use of 40G A6000 GPUs.

## Appendix D Metrics for Aggregated Groups

As an additional experiment, we group items by their popularity and attempt to see if more popular items are more reachable than less popular ones.

For this, we use the number of interactions an item is a part of in the training set as a proxy for it’s popularity among users. We consider two groups of items:

- •
Set A consists of the 30 most popular items.
- •
Set B consists of items with intermediate popularity (30 items between 200th to 300th in popularity).

We then randomly choose 10$\%$ of the total users as set C and compute the future reachability for every (user,item) pair in $C\times A$ and compare it to the future reachability of every (user,item) pair in $C\times B$.

Figure: Figure 7: Future Reachability(MF) for popular items vs items with intermediate popularity
Refer to caption: extracted/5867308/images/agg_future_reach.png

**Table II: Future Reachability with Popular vs Intermediate Items**
| Item Type | Popular item | Intermediate Item |
| --- | --- | --- |
| Mean Lift | 1.3735 | 1.1846 |
| Lower Confidence Interval | 1.0500 | 1.1361 |
| Upper Confidence Interval | 1.6971 | 1.2330 |

Observations:
The baseline values for future reachability for popular and intermediate items were observed to be similar on average.
The results in Table II then show us that popular items have higher values of max future reachability on average as compared to items with intermediate popularity. We note that this difference is not as prominent when measuring past reachability instead of future reachability.

We also group users by their activity and attempt to see if active users act as better adversaries on average than inactive users, i.e., whether active users can cause a larger change in a random user’s recommendation list than inactive users.

Using the number of ratings given by users as a proxy for their activity, we consider two groups of users:

- •
Set A consists of the 30 most active users.
- •
Set B consists of users with intermediate activity (30 users between 200th to 300th in activity).

We then randomly choose 10$\%$ of the remaining users as set C, to act as primary users and compute the stability for every (user,adversary) pair in $C\times A$ and compare it to the reachability of every (user,item) pair in $C\times B$. Figure 8 shows the resulting histogram for future instability.

Figure: Figure 8: Future Instability(MF) for Active vs Intermediate users
Refer to caption: extracted/5867308/images/agg_future_stab.png

**Table III: Future Instability with Active vs Intermediate Adversaries**
| Adversary | Active User | Intermediate User |
| --- | --- | --- |
| Mean | 0.2962 | 0.1938 |
| Lower Confidence Interval | 0.2395 | 0.1549 |
| Upper Confidence Interval | 0.3527 | 0.2327 |

Observations:
These results show us that active users introduce more instability on average as compared to users with intermediate levels of activity. We note that this effect is not as prominent when measuring past stability instead of future stability.

We perform both these experiments on the Matrix Factorization based recommender system.

## Appendix E Selected figures with additional parameter values

1) Scatterplot for reachability with additional time horizons, for comparison
$\beta=0.8$

Figure: Figure 9: Future Reachability: Time horizon = 1 vs Time horizon = 2 vs Time horizon = 5
Refer to caption: extracted/5867308/addn/future125reach.png

Figure: Figure 10: Future Reachability: Time horizon = 1 vs Time horizon = 3 vs Time horizon = 5
Refer to caption: extracted/5867308/addn/future135reach.png

Figure: Figure 11: Past Reachability: Time horizon = 1 vs Time horizon = 2 vs Time horizon = 5
Refer to caption: extracted/5867308/images/image2.png

Figure: Figure 12: Past Reachability: Time horizon = 1 vs Time horizon = 3 vs Time horizon = 5
Refer to caption: extracted/5867308/images/image1.png

2) Scatterplots for past stability with different stochasticities

Figure: Figure 13: $\beta=$ 0.4 vs $\beta=$ 0.8
Refer to caption: extracted/5867308/addn/pastreachbeta0408.png

3) Mean Past Instability as a function of time horizon

Figure: Figure 14: Mean Past Instability vs Time Horizon
Refer to caption: extracted/5867308/addn/instabfutk.png

4) Past stability for different time horizons
$\beta=0.8$

Figure: Figure 15: Time horizon=1 vs Time Horizon=2
Refer to caption: extracted/5867308/addn/past12stab.png

Figure: Figure 16: Time Horizon=2 vs Time Horizon=4
Refer to caption: extracted/5867308/addn/past24stab.png

## Appendix F Gradient Expressions

In this section, we write out the gradient expressions for both reachability and stability for multiple timesteps to give a better idea of the gradients one might need access to in a whitebox setting.

### F.1   Reachability Gradient

#### F.1.1 2 timesteps

- •
Gradient with respect to $f_{1}$
$\displaystyle\nabla_{f_{1}}\mathbb{E}_{A_{i,t},A_{i,t+1}}\left[P(A_{i,t+2}=j|
\text{do}(O_{i,t}=f_{1}(A_{i,t})),\text{do}(O_{i,t+1}=f_{2}(A_{i,t+1})))\right]$
$\displaystyle=\sum_{a_{i,t}}P(A_{i,t}=a_{i,t})\nabla_{f_{1}}\sum_{a_{i,t+1}}
\left[P(A_{i,t+1}=a_{i,t+1}|\text{do}(O_{i,t}=f_{1}(a_{i,t})))\right.$
$\displaystyle\quad\left.P(A_{i,t+2}=j|\text{do}(O_{i,t+1}=f_{2}(a_{i,t+1})),
\text{do}(O_{i,t}=f_{1}(a_{i,t})))\right]$
- •
Gradient with respect to $f_{2}$
$\displaystyle\nabla_{f_{2}}\mathbb{E}_{A_{i,t},A_{i,t+1}}\left[P(A_{i,t+2}=j|
\text{do}(O_{i,t}=f_{1}(A_{i,t})),\text{do}(O_{i,t+1}=f_{2}(A_{i,t+1})))\right]$
$\displaystyle=\sum_{a_{i,t}}P(A_{i,t}=a_{i,t})\sum_{a_{i,t+1}}P(A_{i,t+1}=a_{i
,t+1}|\text{do}(O_{i,t}=f_{1}(a_{i,t})))$
$\displaystyle\quad\nabla_{f_{2}}P(A_{i,t+2}=j|\text{do}(O_{i,t+1}=f_{2}(a_{i,t
+1})),\text{do}(O_{i,t}=f_{1}(a_{i,t})))$

#### F.1.2 3 timesteps

- •
Gradient with respect to $f_{1}$
$\displaystyle\nabla_{f_{1}}\mathbb{E}_{A_{i,t},A_{i,t+1},A_{i,t+2}}[$
$\displaystyle P(A_{i,t+3}=j|\text{do}(O_{i,t}=f_{1}(A_{i,t})),$
$\displaystyle\text{do}(O_{i,t+1}=f_{2}(A_{i,t+1})),\text{do}(O_{i,t+2}=f_{3}(A
_{i,t+2})))]$
$\displaystyle=\sum_{a_{i,t}}P(A_{i,t}=a_{i,t})$
$\displaystyle\nabla_{f_{1}}\sum_{a_{i,t+1}}[P(A_{i,t+1}=a_{i,t+1}|\text{do}(O_
{i,t}=f_{1}(a_{i,t})))$
$\displaystyle\sum_{a_{i,t+2}}P(A_{i,t+2}=a_{i,t+2}|\text{do}(O_{i,t+1}=f_{2}(a
_{i,t+1})),\text{do}(O_{i,t}=f_{1}(a_{i,t})))$
$\displaystyle P(A_{i,t+3}=j|\text{do}(O_{i,t+2}=f_{3}(a_{i,t+2})),$
$\displaystyle\text{do}(O_{i,t+1}=f_{2}(a_{i,t+1})),\text{do}(O_{i,t}=f_{1}(a_{
i,t})))]$
- •
Gradient with respect to $f_{2}$
$\displaystyle\nabla_{f_{2}}\mathbb{E}_{A_{i,t},A_{i,t+1},A_{i,t+2}}\left[P(A_{
i,t+3}=j|\text{do}(O_{i,t}=f_{1}(A_{i,t})),\text{do}(O_{i,t+1}=f_{2}(A_{i,t+1}
)),\text{do}(O_{i,t+2}=f_{3}(A_{i,t+2})))\right]$
$\displaystyle=\sum_{a_{i,t}}P(A_{i,t}=a_{i,t})\sum_{a_{i,t+1}}P(A_{i,t+1}=a_{i
,t+1}|\text{do}(O_{i,t}=f_{1}(a_{i,t})))$
$\displaystyle\quad\nabla_{f_{2}}\sum_{a_{i,t+2}}\left[P(A_{i,t+2}=a_{i,t+2}|
\text{do}(O_{i,t+1}=f_{2}(a_{i,t+1})),\text{do}(O_{i,t}=f_{1}(a_{i,t})))\right.$
$\displaystyle\quad\left.P(A_{i,t+3}=j|\text{do}(O_{i,t+2}=f_{3}(a_{i,t+2})),
\text{do}(O_{i,t+1}=f_{2}(a_{i,t+1})),\text{do}(O_{i,t}=f_{1}(a_{i,t})))\right]$
- •
Gradient with respect to $f_{3}$
$\displaystyle\nabla_{f_{3}}\mathbb{E}_{A_{i,t},A_{i,t+1},A_{i,t+2}}\left[P(A_{
i,t+3}=j|\text{do}(O_{i,t}=f_{1}(A_{i,t})),\text{do}(O_{i,t+1}=f_{2}(A_{i,t+1}
)),\text{do}(O_{i,t+2}=f_{3}(A_{i,t+2})))\right]$
$\displaystyle=\sum_{a_{i,t}}P(A_{i,t}=a_{i,t})\sum_{a_{i,t+1}}P(A_{i,t+1}=a_{i
,t+1}|\text{do}(O_{i,t}=f_{1}(a_{i,t})))$
$\displaystyle\quad\sum_{a_{i,t+2}}P(A_{i,t+2}=a_{i,t+2}|\text{do}(O_{i,t+1}=f_
{2}(a_{i,t+1})),\text{do}(O_{i,t}=f_{1}(a_{i,t})))$
$\displaystyle\quad\nabla_{f_{3}}P(A_{i,t+3}=j|\text{do}(O_{i,t+2}=f_{3}(a_{i,t
+2})),\text{do}(O_{i,t+1}=f_{2}(a_{i,t+1})),\text{do}(O_{i,t}=f_{1}(a_{i,t})))$

#### F.1.3 T timesteps, gradient wrt f k subscript 𝑓 𝑘 f_{k} italic_f start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT

$$ $\displaystyle\nabla_{f_{k}}\mathbb{E}_{A_{i,t},...,A_{i,t+T-1}}[P(A_{i,t+T}=j| \text{do}(O_{i,t}=f_{1}(A_{i,t})),...,\text{do}(O_{i,t+T-1}=f_{T}(A_{i,t+T-1}) ))]$ $\displaystyle=\sum_{a_{i,t}}P(A_{i,t}=a_{i,t})\sum_{a_{i,t+1}}P(A_{i,t+1}=a_{i ,t+1}|\text{do}(O_{i,t}=f_{1}(a_{i,t})))\cdots$ $\displaystyle\quad\sum_{a_{i,t+k-1}}P(A_{i,t+k-1}=a_{i,t+k-1}|\text{do}(O_{i,t +k-2}=f_{k-1}(a_{i,t+k-2})),...,\text{do}(O_{i,t}=f_{1}(a_{i,t})))$ $\displaystyle\quad\nabla_{f_{k}}\sum_{a_{i,t+k}}[P(A_{i,t+k}=a_{i,t+k}|\text{ do}(O_{i,t+k-1}=f_{k}(a_{i,t+k-1})),...,\text{do}(O_{i,t}=f_{1}(a_{i,t})))$ $\displaystyle\quad\quad\cdots$ $\displaystyle\quad\quad\sum_{a_{i,t+T-1}}P(A_{i,t+T}=j|\text{do}(O_{i,t+T-1}=f _{T}(a_{i,t+T-1})),...,\text{do}(O_{i,t}=f_{1}(a_{i,t})))]$ $$

### F.2   Stability Gradient

#### F.2.1 1 timestep

Gradient wrt $f_{1}$

$$ $\displaystyle\nabla_{f_{1}}\mathbb{E}_{\mathbf{A}_{i_{2},t}}\bigg{[}d(\mathbb{ P}(\mathbf{A}_{i_{1},t+1}|\text{do}(\mathbf{O}_{i_{2},t}=f_{1}(\mathbf{A}_{i_{ 2},t}))),\mathbb{P}(\mathbf{A}_{i_{1},t+1}))\bigg{]}$ $\displaystyle=\sum_{a_{i_{2},t}}P(\mathbf{A}_{i_{2},t}=a_{i_{2},t})\nabla_{f_{ 1}}\bigg{[}d(\mathbb{P}(\mathbf{A}_{i_{1},t+1}|\text{do}(\mathbf{O}_{i_{2},t}= f_{1}(a_{i_{2},t}))),\mathbb{P}(\mathbf{A}_{i_{1},t+1}))\bigg{]}$ $$

#### F.2.2 2 timesteps

- •
Gradient wrt $f_{1}$
$\displaystyle\nabla_{f_{1}}\mathbb{E}_{\mathbf{A}_{i_{2},t:t+1}}\bigg{[}d(
\mathbb{P}(\mathbf{A}_{i_{1},t+2}|\text{do}(\mathbf{O}_{i_{2},t}=f_{1}(\mathbf
{A}_{i_{2},t})),\text{do}(\mathbf{O}_{i_{2},t+1}=f_{2}(\mathbf{A}_{i_{2},t+1})
)),\mathbb{P}(\mathbf{A}_{i_{1},t+2}))\bigg{]}$
$\displaystyle=\sum_{a_{i_{2},t}}P(\mathbf{A}_{i_{2},t}=a_{i_{2},t})\nabla_{f_{
1}}\sum_{a_{i_{2},t+1}}\bigg{[}P(\mathbf{A}_{i_{2},t+1}=a_{i_{2},t+1}|\text{do
}(\mathbf{O}_{i_{2},t}=f_{1}(a_{i_{2},t})))$
$\displaystyle\quad d(\mathbb{P}(\mathbf{A}_{i_{1},t+2}|\text{do}(\mathbf{O}_{i
_{2},t}=f_{1}(a_{i_{2},t})),\text{do}(\mathbf{O}_{i_{2},t+1}=f_{2}(a_{i_{2},t+
1}))),\mathbb{P}(\mathbf{A}_{i_{1},t+2}))\bigg{]}$
- •
Gradient with $f_{2}$
$\displaystyle\nabla_{f_{2}}\mathbb{E}_{\mathbf{A}_{i_{2},t:t+1}}\bigg{[}d(
\mathbb{P}(\mathbf{A}_{i_{1},t+2}|\text{do}(\mathbf{O}_{i_{2},t}=f_{1}(\mathbf
{A}_{i_{2},t})),\text{do}(\mathbf{O}_{i_{2},t+1}=f_{2}(\mathbf{A}_{i_{2},t+1})
)),\mathbb{P}(\mathbf{A}_{i_{1},t+2}))\bigg{]}$
$\displaystyle=\sum_{a_{i_{2},t}}P(\mathbf{A}_{i_{2},t}=a_{i_{2},t})\sum_{a_{i_
{2},t+1}}P(\mathbf{A}_{i_{2},t+1}=a_{i_{2},t+1}|\text{do}(\mathbf{O}_{i_{2},t}
=f_{1}(a_{i_{2},t})))$
$\displaystyle\quad\nabla_{f_{2}}\bigg{[}d(\mathbb{P}(\mathbf{A}_{i_{1},t+2}|
\text{do}(\mathbf{O}_{i_{2},t}=f_{1}(a_{i_{2},t})),\text{do}(\mathbf{O}_{i_{2}
,t+1}=f_{2}(a_{i_{2},t+1}))),\mathbb{P}(\mathbf{A}_{i_{1},t+2}))\bigg{]}$

#### F.2.3 T timesteps. gradient wrt f k subscript 𝑓 𝑘 f_{k} italic_f start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT

$$ $\displaystyle\nabla_{f_{k}}\mathbb{E}_{\mathbf{A}_{i_{2},t:t+T-1}}\bigg{[}d( \mathbb{P}(\mathbf{A}_{i_{1},t+T}|\text{do}(\mathbf{O}_{i_{2},t}=f_{1}(\mathbf {A}_{i_{2},t})),\ldots,\text{do}(\mathbf{O}_{i_{2},t+T-1}=f_{T}(\mathbf{A}_{i_ {2},t+T-1}))),\mathbb{P}(\mathbf{A}_{i_{1},t+T}))\bigg{]}$ $\displaystyle=\sum_{a_{i_{2},t}}P(\mathbf{A}_{i_{2},t}=a_{i_{2},t})\sum_{a_{i_ {2},t+1}}P(\mathbf{A}_{i_{2},t+1}=a_{i_{2},t+1}|\text{do}(\mathbf{O}_{i_{2},t} =f_{1}(a_{i_{2},t})))\cdots$ $\displaystyle\quad\sum_{a_{i_{2},t+k-1}}P(\mathbf{A}_{i_{2},t+k-1}=a_{i_{2},t+ k-1}|\text{do}(\mathbf{O}_{i_{2},t+k-2}=f_{k-1}(a_{i_{2},t+k-2})),\ldots,\text {do}(\mathbf{O}_{i_{2},t}=f_{1}(a_{i_{2},t})))$ $\displaystyle\quad\nabla_{f_{k}}\sum_{a_{i_{2},t+k}}\bigg{[}P(\mathbf{A}_{i_{2 },t+k}=a_{i_{2},t+k}|\text{do}(\mathbf{O}_{i_{2},t+k-1}=f_{k}(a_{i_{2},t+k-1}) ),\ldots,\text{do}(\mathbf{O}_{i_{2},t}=f_{1}(a_{i_{2},t})))$ $\displaystyle\quad\cdots\sum_{a_{i_{2},t+T-1}}d(\mathbb{P}(\mathbf{A}_{i_{1},t +T}|\text{do}(\mathbf{O}_{i_{2},t}=f_{1}(a_{i_{2},t})),\ldots,\text{do}( \mathbf{O}_{i_{2},t+T-1}=f_{T}(a_{i_{2},t+T-1}))),\mathbb{P}(\mathbf{A}_{i_{1} ,t+T}))\bigg{]}$ $$

### F.3   Reachability gradient computation for deterministic (top-1) item choice

For simplicity, we assumeIn our experiments, we set the user \lleditto always select\lldeletes the top item recommended by the system. While sampling items from a user’s preference distribution (e.g., using Gumbel-softmax) is possible, it introduces more variability as user’s now interact with a more diverse variety of item sequences, which necessitates additional parameter updates since our parameter space consists of a separate parameter for every (item, timestep) tuple. Additionally, we demonstrate below how top-1 selection affects gradient computation, with the subgradient of the item selection term reducing to 0, which is not the case for Gumbel-softmax selection.
Example of 2-step gradient computation with respect to $f_{1}$:
From F.1.1,

$$ $\displaystyle\nabla_{f_{1}}\mathbb{E}_{A_{i,t},A_{i,t+1}}\left[P(A_{i,t+2}=j| \text{do}(O_{i,t}=f_{1}(A_{i,t})),\text{do}(O_{i,t+1}=f_{2}(A_{i,t+1})))\right]$ $\displaystyle=\sum_{a_{i,t}}P(A_{i,t}=a_{i,t})\nabla_{f_{1}}\sum_{a_{i,t+1}} \left[P(A_{i,t+1}=a_{i,t+1}|\text{do}(O_{i,t}=f_{1}(a_{i,t})))\right.$ $\displaystyle\quad\left.P(A_{i,t+2}=j|\text{do}(O_{i,t+1}=f_{2}(a_{i,t+1})), \text{do}(O_{i,t}=f_{1}(a_{i,t})))\right]$ $$

We focus on the terms dependent on $f_{1}$.

$$ $\displaystyle\nabla_{f_{1}}\sum_{a_{i,t+1}}\left[P(A_{i,t+1}=a_{i,t+1}|\text{ do}(O_{i,t}=f_{1}(a_{i,t})))\right.\left.P(A_{i,t+2}=j|\text{do}(O_{i,t+1}=f_{ 2}(a_{i,t+1})),\text{do}(O_{i,t}=f_{1}(a_{i,t})))\right]$ $$

This can be written as:

$$ $\displaystyle\nabla_{f_{1}}\sum_{k\in[n]}(\prod_{l\neq k}\mathbb{I}_{g(k,t) \geq g(l,t)})m(f_{1},f_{2})$ $$

Here $[n]$ denotes the list of item ids(from 1 to $n$) and $g$ is a function that takes an item and timestep as arguments and returns the user’s preference score for that item at that timestep.
The condition specified by the product of indicator functions is simply that $k$ is the item that the user has the highest preference score for, in other words, $k$ is the top-1 recommended item. We have also written $P(A_{i,t+2}=j|\text{do}(O_{i,t+1}=f_{2}(a_{i,t+1})),\text{do}(O_{i,t}=f_{1}(a_
{i,t})))$ as a function of $f_{1}$ and $f_{2}$, $m(f_{1},f_{2})$.
Using the product rule on this expression, we get,

$$ $\displaystyle\sum_{k\in[n]}(\prod_{l\neq k}\mathbb{I}_{g(k,t)\geq g(l,t)} \nabla_{f_{1}}m(f_{1},f_{2})+m(f_{1},f_{2})\nabla_{f_{1}}(\prod_{l\neq k} \mathbb{I}_{g(k,t)\geq g(l,t)}))$ $$

This reduces to:

$$ $\displaystyle\sum_{k\in[n]}(\prod_{l\neq k}\mathbb{I}_{g(k,t)\geq g(l,t)} \nabla_{f_{1}}m(f_{1},f_{2}))$ $$

as the 0 is a subgradient of the product of indicator functions with respect to $f_{1}$. Therefore, in making the simplification to users only choosing the top-1 item, we do away with the gradient propagation path through the item choice itself.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. How does the computational complexity scale with the number of users, items and time horizon? What are recommended approaches for large-scale recommender systems?

2. What are the practical implications of assuming static embeddings during gradient computation? How would the results change with full retraining?

3. Could the framework be extended to handle more complex recommendation scenarios like slate recommendations or contextual bandits?
soundness: 3
strengths: 1. The technical claims and methodology are very well-supported. The causal framework is rigorously developed with clear mathematical formulations. The empirical evaluation is comprehensive, with well-designed ablation studies showing impact of various stochasticity levels, time horizon lengths and model architecture choices.

2. Novel formalization of reachability and stability metrics presented capture both immediate and long-term effects, handle multi-step recommendation dynamics and account for both user and adversary perspectives.

3. The paper is generally well-written and logically structured. The causal framework is presented clearly with helpful examples.
summary: The paper presents a unified causal framework for auditing recommender systems with focus on user agency. The authors make three main contributions:
1. A general causal framework that formalizes interventional and counterfactual metrics for auditing recommender systems.
2. Two novel classes of metrics - reachability and stability, to measure user agency while accounting for recommendation dynamics.
3. Efficient computational methods for these metrics under different levels of access to the recommender system.

The framework is evaluated empirically using both matrix factorization and recurrent neural network based recommenders, showcasing interesting trade-offs between stability and reachability.
weaknesses: 1. The assumption of static user/item embeddings during gradient computation could be better justified. Additional experiments showing impact of this simplification would be valuable.

2. The empirical evaluation focuses on movie recommendations - testing on other domains (e.g. social media, e-commerce, etc.) would strengthen the framework's generalizability claims.

3. The choice of distance metrics for stability measures (L2 distance) could be better justified. Adding discussion of metric sensitivity to adversarial perturbations and analysis of the relationship between local and global notions of reachability would be useful.

4. The paper presents limited discussion of computational complexity and scalability analysis, particularly for large-scale recommender systems. The paper could analyze how the methods scale with number of users, items and time horizon.

---

confidence: 3
contribution: 3
initial_rating: 3
presentation: 2
questions: What are the differences and impacts of applying this model to various recommendation models?
soundness: 3
strengths: The causal approach offers a novel way to address ethical issues, providing a structured method for defining and calculating user-centric metrics.

Offering both gradient-based and black-box methods for metric computation enables broader application
summary: The paper proposes a unified causal framework for auditing recommender systems, specifically to address ethical concerns such as user agency, stability, and reachability. It categorizes auditing metrics from a causal perspective and introduces two key metrics, past- and future-reachability, and stability, which measure a user’s ability to influence recommendations. The empirical studies evaluate the metrics on different recommender models, highlighting the trade-offs between user influence on recommendations and system stability.
weaknesses: The framework’s reliance on specific causal assumptions and models,this may reduce its generalizability across diverse recommender systems.

The paper lacks a discussion about the differences between recommendation systems.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 2
questions: Please see them in the weaknesses.
soundness: 2
strengths: S1- This paper provides comprehensive details on the background of the problem.

S2- The authors give detailed experiment settings which improves the reproducibility.
summary: In this work, authors pay attention to recommender system auditing from a causal perspective, and point out the lack of metrics for auditing user agency for the recommendation process. Therefore, two metrics are proposed, including future- and past-reachability and stability, which can measure the impact of users on their own and other users. To calculate these metrics, the authors also design a gradient-based and a black box approach.
weaknesses: W1-The motivation of this paper is not quite clear. For example, what’s the actual relationship between user agency and ethical concerns? 

W2-The experiments are only conducted on ML-1M, which are insufficient to explain the universality of the conclusions since the recommendation senarios are diverse. Experiments on at leasr one dataset from other recommendation senarios are needed.

W3- In Figure 3 for the distribution of past instability values, for MF, Past-5 shows lower proportion of 0.0 than Past-1, but for RRN, Past-5 presents higher proportion of 0.0 than Past-1. Could you please explain the reason for this contrary result?

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 3
questions: In addition to **Q1** to **Q5** mentioned in the Weaknesses, I have several other questions:

- **Q6**: What is the relationship between the proposed metrics and recommendation performance? Does a stronger recommendation model perform better according to these metrics?

- **Q7**: The metric comparisons in Figure 3 are described but lack corresponding explanations. For instance, why do some items show "a user’s recommended list is either heavily affected by the actions of an adversary or is minimally affected by them"?

- **Q8**: Is the time horizon parameter in the experimental parameters equivalent to $k$ in Definitions 4.1 and 4.2? If not, how is $k$ set in the experiments?

I am happy to engage in further discussion, and if these issues are addressed, I am willing to reconsider the score.
soundness: 2
strengths: - Auditing recommender systems is a highly meaningful area of study, and the paper contributes valuable insights.
- The article is well-written and clearly articulated, making complex concepts accessible.
- It provides methods for auditing from both white-box and black-box perspectives, catering to different levels of system access.
summary: In this work, the authors adopt a causal perspective on recommender system auditing and present a general method for defining auditing metrics. Within this overarching causal auditing framework, they categorize existing audit metrics. Leveraging their framework, they propose two types of metrics: future-/past-reachability and stability, which respectively measure a user's ability to influence recommendations for themselves and for other users. Additionally, they introduce a gradient-based method and a black-box method for calculating these metrics, allowing auditors to assess them at various levels of system access. Empirically, the authors demonstrate the effectiveness of their proposed metrics and use them to examine the design of recommender systems.
weaknesses: - **W1**: **Ambiguity in Definitions**: The definitions in the article lack detailed explanations, which may lead to ambiguity. For example:
  - **Q1-1**: In Definitions 4.1 and 4.2, the authors consider only the intervention on $O_{i,t}$ without accounting for its effect on $A_{i,t+1}$. Why was this setting chosen?
  - **Q1-2**: Are Definitions 4.1 and 4.2 consistent? Specifically, does past-$k$ at time $t+k$ equal future-$k$ at time $t$? It would be helpful if the authors could address this question both intuitively and formally.

- **W2**: **Limited Analysis Scope**: The analysis in Section 5 is confined to $k=1$, representing only a special case of the broader definitions provided.
  - **Q2**: Please describe how the corresponding white-box and black-box methods would operate when $k > 1$.

- **W3**: **Practical Applicability Concerns**: There is a gap between the theoretical propositions and practical scenarios.
  - **Q3**: Proposition 5.1 requires fixing item embeddings, while Proposition 5.2 requires fixing user embeddings. Since these conditions are difficult to meet in real recommender systems, how does this gap affect practical auditing?

- **W4**: **Lack of Experimental Rationale**: Certain experimental setups lack clear justification.
  - **Q4-1**: Section 6.1 mentions different policies for future and past metrics. Why was this setup chosen? Please explain the rationale behind this decision.

- **W5**: **Incomplete Experimental Validation**:
  - **Q5-1**: The use of a single dataset limits the experimental scope and generalizability of the findings.
  - **Q5-2**: The current experiments focus on analyzing existing models within the proposed framework but do not clarify why this framework or these metrics are more valid than existing auditing methods. Additional experiments, such as straightforward case studies, are needed to further validate the framework.

Meta-review: This paper presents a causal framework for auditing recommender systems with a focus on ethical considerations such as user agency, stability, and reachability. It introduces two novel classes of metrics: future- and past-reachability and stability, which measure a user’s influence over their own recommendations and others’. The authors also propose two computational approaches, gradient-based and black-box, for calculating these metrics. Empirical evaluations demonstrate the utility of these metrics using matrix factorization and recurrent recommender network models, revealing insights into the trade-offs between stability and reachability.

The paper has several strengths. It tackles an important and underexplored problem in auditing recommender systems from a causal perspective, offering a framework that has the potential to generate new insights into user-centric evaluation metrics. The proposed metrics are novel, and the framework itself is methodologically rigorous and well-motivated. Additionally, the authors provide detailed explanations and experiments, improving the paper’s clarity and reproducibility.

However, the paper has notable weaknesses that undermine its potential contribution. First, the motivation for focusing on user agency as an ethical concern, while mentioned, lacks a compelling connection to practical scenarios and actionable insights. The authors do not sufficiently justify the real-world impact of their metrics, especially when tied to the ethical concerns they aim to address. Second, the experimental evaluation is narrow in scope, relying solely on the ML-1M dataset, which limits the generalizability of the findings across diverse recommendation domains. Reviewers consistently raised concerns about the lack of validation on other datasets, as this significantly impacts the credibility of the conclusions. Additionally, the framework relies on several assumptions, such as static user and item embeddings, that simplify the computational challenges but reduce the practical applicability of the methods for real-world, large-scale systems. This gap between theoretical contributions and practical implications is a significant drawback. Finally, there are ambiguities in the definitions of the proposed metrics (e.g., the relationship between past and future reachability), and some experimental results lack sufficient explanation or justification.

The most critical reasons for the recommendation to reject this paper are the lack of experimental diversity, insufficient justification of the practical implications of the metrics, and the significant gap between theoretical assumptions and real-world applicability. While the paper demonstrates a well-developed theoretical framework, its limitations in empirical validation and practical relevance prevent it from making a strong case for acceptance.

Decision: Reject