Title: Improved Analysis for Bandit Learning in Matching Markets
Venue: NeurIPS 2024
Accepted: yes
Keywords: Bandits

Matching markets

Stable matching

Stable regret

Exploration and exploitation

$\alpha$-condition
Abstract: A rich line of works study the bandit learning problem in two-sided matching markets, where one side of market participants (players) are uncertain about their preferences and hope to find a stable matching during iterative matchings with the other side (arms). The state-of-the-art analysis shows that the player-optimal stable regret is of order $O(K\log T/\Delta^2)$ where $K$ is the number of arms, $T$ is the horizon and $\Delta$ is the players' minimum preference gap. However, this result may be far from the lower bound $\Omega(\max\{N\log T/\Delta^2, K\log T/\Delta\})$ since the number $K$ of arms (workers, publisher slots) may be much larger than that $N$ of players (employers in labor markets, advertisers in online advertising, respectively). In this paper, we propose a new algorithm and show that the regret can be upper bounded by $O(N^2\log T/\Delta^2 + K \log T/\Delta)$. This result removes the dependence on $K$ in the main order term and improves the state-of-the-art guarantee in common cases where $N$ is much smaller than $K$. Such an advantage is also verified in experiments. In addition, we provide a refined analysis for the existing centralized UCB algorithm and show that, under $\alpha$-condition, it achieves an improved $O(N \log T/\Delta^2 + K \log T / \Delta)$ regret.

Improved Analysis for Bandit Learning in Matching Markets

Abstract

A rich line of works study the bandit learning problem in two-sided matching markets, where one side of market participants (players) are uncertain about their preferences and hope to find a stable matching during iterative matchings with the other side (arms). The state-of-the-art analysis shows that the player-optimal stable regret is of order O ( K log T/ ∆ 2 ) where K is the number of arms, T is the horizon and ∆ is the players' minimum preference gap. However, this result may be far from the lower bound Ω(max { N log T/ ∆ 2 , K log T/ ∆ } ) since the number K of arms (workers, publisher slots) may be much larger than that N of players (employers in labor markets, advertisers in online advertising, respectively). In this paper, we propose a new algorithm and show that the regret can be upper bounded by O ( N 2 log T/ ∆ 2 + K log T/ ∆) . This result removes the dependence on K in the main order term and improves the state-of-the-art guarantee in common cases where N is much smaller than K . Such an advantage is also verified in experiments. In addition, we provide a refined analysis for the existing centralized UCB algorithm and show that, under α -condition, it achieves an improved O ( N log T/ ∆ 2 + K log T/ ∆) regret.

1 Introduction

The two-sided matching market problem has been extensively studied in the literature due to its wide range of applications like labor market, school admission, house allocation, and online advertising [26, 9, 1]. There are two sides of participants in the market, such as the employers and workers in the labor market, advertisers and publishers in online advertising. Each participant on the one side has a preference ranking over the other side. The concept of stability, which characterizes the equilibrium state of the market where no participant wants to break up the current matching relationship and find another partner, has attracted great interest from researchers [26]. Achieving stability is critical for ensuring the long-term viability of the market.

A rich line of works [9, 15, 25] study how to find a stable matching in the market. Most of them assume the preference ranking of each market participant is known beforehand, which we refer to as the offline setting. However, in real applications, the knowledge of the preferences may be uncertain. For example, in the labor market, employers usually do not know the working abilities of workers before being matched, and advertisers also do not know the exact conversion rate of placing the advertisement in a publisher slot. This makes the traditional algorithms unavailable to

Zilong Wang

find an exact stable matching. With the emergence of online market platforms such as the online labor market UpWork and TaskRabbit as well as online advertising platforms where employers or advertisers have many similar tasks, market participants are able to learn their unknown preferences during iterative matchings with the other side of agents.

Multi-armed bandit (MAB) is a classic framework that characterizes the learning process during iterative interactions [3, 19]. It considers the setting with single player on one side and multiple arms on the other side. At each round, the player selects an arm and receives a reward. The player has unknown preferences over arms and would learn this knowledge based on the collected rewards. To accumulate as many rewards as possible, the player faces the dilemma of exploration and exploitation. The former selects arms with less observations while the latter focuses on arms with better historical performances. How to balance the exploration and exploitation trade-off is the key of bandit algorithm design. The upper confidence bound (UCB) [3], Thompson sampling (TS) [14, 2], and explore-then-commit (ETC) [10] are common strategies in MAB to achieve this objective.

Liu et al. [20] introduce the bandit learning problem in matching markets and try to provide theoretical guarantees. Two sides of agents in the market can be modeled as players and arms. Without loss of generality, denote N and K as the number of players and arms, respectively. It is worth noting that this work and all of the following works assume N ≤ K to ensure each player has a chance to be matched. In this problem, the objective is to find a stable matching and minimize the stable regret for each player, which is defined as the difference between the reward of the stable arm and that the player receives during the horizon. Since there may be more than one stable matching, they mainly focus on the players' most preferred one corresponding to the player-optimal stable matching and the least preferred one corresponding to the player-pessimal stable matching. Note that players receive more rewards in the player-optimal stable matching and thus the former objective is the most desirable. Liu et al. [20] first study a centralized setting where a central platform would compute allocations for players to avoid conflicts. Both ETC and UCB-type algorithms are proposed for this setting. The former achieves a player-optimal stable regret guarantee with prior knowledge of players' minimum preference gap ∆ and the latter can only ensure to reach the playerpessimal stable matching. Motivated by real applications where the central platform may not always exist, a rich line of works then study the decentralized case where no platform coordinates players' behavior [21, 28, 4, 18, 22]. This line of works again only achieve guarantees for player-pessimal stable regret [21, 18, 28, 4, 22]. Table 1 compares settings and regrets among these works. Until recently, Zhang et al. [31] and Kong and Li [16] independently derive algorithms that have polynomial player-optimal stable regret and show the upper bound is O ( K log T/ ∆ 2 ) . However, this result may be still far from the lower bound Ω(max { N log T/ ∆ 2 , K log T/ ∆ } ) [28] since K is usually much larger than N such as that the number of workers (publisher slots) is usually much larger than that of employers in labor markets (advertisers in online advertising, respectively).

In this paper, we try to provide more efficient algorithms and improve the results over existing works. The detailed contribution can be summarized as follows: (1) We propose an algorithm named adaptively explore-then-Gale-Shapley (AETGS) with elimination. State-of-the-art works [31, 16] explicitly separate the exploration and exploitation processes, which can lead to unnecessary regret, as exploring certain preference rankings may not contribute to the exploitation process. To avoid excessive exploration, our AETGS with elimination algorithm integrates the players' learning process into the GS steps. Players adaptively switch between exploration and exploitation and promptly eliminate sub-optimal arms. (2) We prove that the player-optimal stable regret of AETGS with elimination can be upper bounded by O ( N 2 log T/ ∆ 2 + K log T/ ∆) . This is the first result that removes the dependence on K in the main regret order term and improves existing works in common cases where N is much smaller than K . We also conduct experiments to show the advantages of the algorithm. (3) We refine the analysis of the centralized UCB algorithm in Liu et al. [20] for markets satisfying the α -condition. By investigating the preference hierarchy structure of the α -condition, we demonstrate that the stable matching converges sequentially from player 1 to player N . Through inductive analysis over players, we establish an O ( N log T/ ∆ 2 + K log T/ ∆) regret upper bound, which improves the original result for this algorithm in this specific market.

2 Related Work

The problem of bandit learning in matching markets is first introduced by Das and Kamenica [8]. They study the special case where both sides of agents have the same preferences and propose

some empirical methods to solve the problem. Liu et al. [20] first theoretically formulate this problem and provide an upper bound for the stable regret of players. They propose a centralized explore-then-commit (ETC) algorithm and upper confidence bound (UCB) algorithm, which obtain an O ( K log T/ ∆ 2 ) player-optimal stable regret and O ( NK log T/ ∆ 2 ) player-pessimal stable regret, respectively. It is worth noting that the former ETC algorithm requires knowledge about ∆ to ensure the algorithmic operation. Due to the generality, the following works focus on the decentralized setting. Liu et al. [21] and Kong et al. [18] propose the UCB and TS-type algorithm for general decentralized markets, respectively. Such a setting is much more challenging and both of them only achieve O ( exp( N 4 ) N 5 K 2 log 2 ( T ) / ∆ 2 ) upper bound for the player-pessimal stable regret.

To improve the stable regret guarantee, a line of research studies some special markets with unique stable matching in which case the player-optimal stable matching is equivalent to the playerpessimal one. Sankararaman et al. [28] propose the UCB-D3 algorithm based on the assumption of serial dictatorship, i.e., all arms share the same preferences, and obtain an O ( NK log T/ ∆ 2 ) regret upper bound. To investigate the problem hardness, they also derive a lower bound Ω ( max { N log T/ ∆ 2 , K log T/ ∆ }) under this assumption. Basu et al. [4] consider more general α -condition setting for unique stable matching. They propose the UCB-D4 algorithm and also achieve the O ( NK log T/ ∆ 2 ) regret bound. Later, Maheshwari et al. [22] study the market satisfy-

ing α -reducible condition and proposes a communication-free algorithm. Their regret bound has an exponential dependence on the number of market participants. Recently, researchers have developed algorithms that can achieve player-optimal stable regret guarantees without assuming unique stable matching. Both Zhang et al. [31] and Kong and Li [16] propose ETC-type algorithms that achieve O ( K log T/ ∆ 2 ) player-optimal stable regret. Wang and Li [29] studies the matching markets with serial dictatorship and obtains the O ( N log T/ ∆ 2 + K log T/ ∆ ) regret. Table 1, compares our proposed algorithm with these related works in terms of their corresponding settings and theoretical guarantees.

There are also other works considering unknown preferences in matching markets. Wang et al. [30] study a many-to-one market where an arm can accept multiple players. Jagadeesan et al. [12] consider online matching markets with monetary transfers. Min et al. [23] investigate Markov matching markets where state transitions occur during the matching process and players' rewards depend on the current state. Other studies have focused on non-stationary rewards, such as Muthirayan et al. [24], Ghosh et al. [11], who propose robust algorithms to mitigate the impact of reward disturbances. Additionally, several studies have explored the problem of offline matching market learning. Dai and Jordan [6, 7] propose approaches that leverage historical data to design optimal matching or recommend participants on both sides.

3 Setting

This paper considers the problem of bandit learning in two-sided matching markets. Denote N = { p 1 , p 2 , . . . , p N } as the player set and K = { a 1 , a 2 , . . . , a K } as the arm set. Let N and K be the number of players and arms, respectively. To ensure that each player can be matched with an arm, we follow previous works and assume N ≤ K [20, 21, 28, 4, 18, 31, 16].

̸

For each player p i ∈ N , its preference towards arm a j can be portrayed by an absolute utility µ i,j ∈ (0 , 1] . For any pair of arms a j and a j ′ , µ i,j > µ i,j ′ indicates that player p i prefers arm a j over a j ′ . Following previous works for matching markets [9, 20, 21, 28, 4, 18, 31, 16], players are assumed to have distinct preferences over different arms, i.e., µ i,j = µ i,j ′ for any a j = a j ′ . In practice, players' preferences which correspond to workers' abilities and the publisher's conversion rates are typically unknown and can be learned through the interactive matching process. On the other side, each arm a j also has a fixed and distinct preference utility π j,i over each player p i ∈ N , and π j,i > π j,i ′ means that arm a j prefers player p i over p i ′ . As in labor markets where workers usually know their preferences over employers based on the payments and task types, the preferences of arms are assumed to be known beforehand [20, 21, 18, 28, 4, 31, 16].

̸

At each round t = 1 , 2 , . . . , each player p i proposes to an arm A i ( t ) . For each arm a j , denote A -1 j ( t ) = { p i : A i ( t ) = a j } as the set of players who selects arm a j at round t . When more than one player selects a j , it accepts its most-preferred one in A -1 j ( t ) , i.e. a j will match with p i ∈ arg max p i ∈ A -1 j ( t ) π j,i . If a player p i is successfully matched with arm A i ( t ) , it will receive a random reward X i ( t ) characterizing its matching experience, which we assume is a 1 -subgaussian random variable with expectation µ i,A i ( t ) . Otherwise, p i is rejected by its proposed arm and only gets reward X i ( t ) = 0 . Denote ¯ A i ( t ) as the final matched arm of player p i at round t . Then ¯ A i ( t ) = A i ( t ) if p i is accepted by the arm A i ( t ) and we simply set ¯ A i ( t ) = ∅ if p i is rejected.

Stability is a key property of a matching in two-sided markets [9, 27, 25]. A matching ¯ A ( t ) = { ( i, ¯ A i ( t )) : i ∈ [ N ] } is stable if no market participant wants to break up its current matching relationship and find a new partner. Formally speaking, there is no player-arm pair ( p i , a j ) such that µ i,j > µ i, ¯ A i ( t ) and π j,i > π j, ¯ A -1 j ( t ) . It is worth noting that there may be multiple stable matchings in the market. Denoted M = { m : m is stable } as the set of all stable matchings. It is shown that there exists a stable matching m ∗ ∈ M such that all players are matched with their most preferred stable arm [9], i.e., µ i,m ∗ i ≥ µ i,m i for any m ∈ M,i ∈ [ N ] . Given a specified horizon T , the learning objective is to minimize the player-optimal stable regret for each player p i which is defined as the difference between the cumulative reward received by being matched with m ∗ i and the cumulative reward received by p i over T rounds:

Here, the expectation is taken over by the randomness of the reward generation and the randomness inherent in the player's strategy.

For completeness, we introduce the procedure of the offline Gale-Shapley (GS) algorithm, which would be useful when describing the algorithmic details. The offline GS algorithm is a classic algorithm to find the player-optimal stable matching when both sides of the market participants know their exact preference rankings. Following offline GS, each player proposes to the arm one by one based on its preference ranking. Until no rejection happens, the final matching is exactly the player-optimal stable matching [9]. Specifically, at the first step, all players propose to their most preferred arm. Arms would accept their most preferred player among those who propose to it and reject others. Then players who are rejected at previous steps would then propose to their next preferred arm. And arms still reject the players who propose to it except for their most preferred one. Such a process continues until no rejection happens.

For convenience, we also define some useful notations that quantify the hardness of the learning problem in matching markets and are used in the later analysis.

̸

Definition 3.1. For each player p i , denote σ i as p i 's preference ranking and let σ i,k as p i 's the k -th preferred arm in its ranking. With a little abuse of notation, let σ i ( a j ) represent the rank of arm a j in p i 's preference. For each player p i and arm a j = a j ′ , let ∆ i,j,j ′ = | µ i,j -µ i,j ′ | be the preference gap of p i between a j and a j ′ . Define ∆ = min i,k ∈ [ σ i ( m ∗ i )] ∆ i,σ i,k ,σ i,k +1 as the minimum preference gap between the arm ranked the first ( σ i ( m ∗ i ) + 1) -th among all players. Further, define ∆ N = min i,k ∈ [ N ] ∆ i,σ i,k ,σ i,k +1 as the minimum preference gap between the arm ranked the first ( N +1) -th among all players.

4 Algorithm for General Markets

In this section, we propose an algorithm called adaptively explore-then-Gale-Shapley (AETGS) with elimination. For simplicity, we present the centralized version of the algorithm in Algorithm 1 from view of player p i . The discussion on how to extend it to a decentralized version is deferred to later subsections.

In general, AETGS with elimination is an adaptive version of the GS algorithm. Since players do not know their preference rankings, they need to learn this knowledge by exploring arms (Line 4). To reduce the regret during exploration, players would adaptively eliminate sub-optimal arms (Line 6-8). Until they find their most preferred arm among available arms, they will stop exploration and focus on this arm (Line 9-11). And once the player finds this arm is occupied by a more preferred player, it would re-start exploration to find the next preferred arm (Line 12-19).

Specifically, each player p i still maintains ˆ µ i,j ( t ) and T i,j ( t ) to represent the empirical mean and the number of observations on each arm a j at the end of round t . To determine whether an arm is more preferred than another, it maintains a confidence interval for each arm a j with upper bound UCB i,j ( t ) := ˆ µ i,j ( t ) + √ 6 log T/T i,j ( t ) and lower bound LCB i,j ( t ) := ˆ µ i,j -√ 6 log T/T i,j ( t ) . When T i,j = 0 , they will be initialized as + ∞ and -∞ , respectively. And once the confidence intervals of the two arms are disjoint, it can regard the arm with a higher empirical mean to be more preferred (Line 1). To be consistent with the offline GS, each player maintains D i to represent the set of arms that have rejected p i during previous steps. In the beginning, it is initialized as an empty set. And we use A i to represent the available arms with the potential to be the stable arm of p i , which is initialized as K \ D i . For convenience, denote E i as the exploration status of player p i . E i = True means that p i still needs to explore arms in A i to determine its most preferred arm. And E i = False means that p i already finds its most preferred arm and now focuses on this arm (Line 2).

To reduce the regret suffered during exploration, players would update A i and eliminate sub-optimal arms in real-time (Line 6-8). Here to avoid collision during round-robin exploration, we would maintain A i such that it contains no less than N arms if p i still has not determined its most preferred one. Thus the union of the available arm set over all players with E i = True contains more than N arms and the round-robin exploration over A i for each such player p i can be carried out without collisions. For completeness, we defer how to arrange players' explorations in later discussions. And once there exists an arm in A i that can be regarded to be optimal, p i will set the exploration status E i to be False and update the exploration arm set A i to only contain this optimal arm. For convenience, denote A i as this arm (Line 9-11).

̸

The update of the available arm set should not only depend on p i 's own observations but also on the other market participants. Specifically, if a player p i ′ determines A i ′ as its most preferred arm, then the final stable player of A i ′ would be the same as or more preferred than p i ′ . So if A i ′ prefers p i ′ than p i , then A i ′ would not be the stable arm of p i and there is no need for p i to explore A i ′ anymore. In this case, p i deletes arm A i ′ from its available set and update A i (Line 12-19). It is worth noting that this operation may incorporate the eliminated arms again in A i . This is reasonable as the previously eliminated arm may be more preferred than the current arms in A i after the deletion operation. And if the deleted arm is p i 's current most preferred arm, it will mark E i as True and restart exploration to find the next most preferred one (Line 15-17).

4.1 Theoretical Results.

Theorem 4.1. Following Algorithm 1, the player-optimal stable regret for each player p i satisfies

Due to the space limit, the proof of Theorem 4.1 is deferred to Appendix A. The following are discussions on the detailed implementation as well as the novelty of the result.

Arrangement of the round-robin exploration process. Recall that the number of available arms of each player is always larger than N based on Line 6 and we assume players can explore their available arms in a round-robin manner without conflict. We now propose an arrangement by letting players explore the available arms in units of N to guarantee this property. Specifically, in every 2 N rounds, each player selects the N available arms with the fewest observations (randomly breaks ties) and explores them in a round-robin way. It can be shown by contradiction that there exists an assignment such that each player can successfully match with their respective N arms once during these 2 N rounds (Lemma B.2 in Appendix), which only doubles the original regret without influencing the regret order. This guarantees that after every 2 N rounds, the observation count difference among all available arms is at most 1 . Players would perform arm elimination and optimal arm identification (Line 6 and 9) in the end of each 2 N rounds. Therefore, compared to the timely eliminating/deleting of arms, this approach ensures that each player will select each arm at most one additional time during each exploration cycle before the player finds the optimal one. Since each player may restart exploration (Line 12-13) up to N 2 times (each of N players can focus on N arms), this scenario leads to an additional O ( N 2 K ) constant regret and does not influence the final regret order.

Extension to the decentralized setting. For simplicity, we present the algorithm in a centralized manner. It can also be extended to the decentralized version where no central platform coordinates players' selections. Specifically, we divide the total horizon into several phases and the length of each grows exponentially, i.e., the lengths of phases are 2 , 4 , 8 , · · · . At the end round of each phase, players would decide whether to eliminate sub-optimal arms as Line 6-8, whether to determine one arm as the most preferred one and update the exploration status as Line 9-11. After the end of each phase, players would communicate their current exploration status, update their deletion set and available arm set, re-update the exploration status as Line 12-19, and then communicate their updated available arm set to determine the round-robin exploration process in the next phase. If there exists a player whose exploration status becomes True from False during communication, the phase length would re-start from 2 and grows exponentially since new arms may be required for exploration. If the final exploration status of all players is False and their optimal arms are different, the next phase would continues until the end of the interaction. The detailed implementation of communication is deferred to the next paragraph. Based on the communication, players with E i = True can have a pre-agreed protocol to explore arms in their available arm set in a round-robin manner without collision as discussed in the last paragraph. Within each phase, they just roundrobin explore arms and collect observations but do not make any decision on arms' optimality. If L observations on a sub-optimal arm are enough to decide its sub-optimality in the centralized version, then this arm would be eliminated at the end of the corresponding phase with the selected time to be at most 2 L due to the exponentially increasing phase length. So the regret in this decentralized version is at most two times as that suffered in the centralized version.

This paragraph describes the implementation of the communication procedure. Recall that players need to communicate their exploration status and available arm set (calculated by subtracting the deletion and eliminating set from K ) at the end of each phase. For the phase length, recall that it grows exponentially until a player's exploration status becomes True from False and a player updates E i from False as True only when its most preferred arm is occupied by a higher-priority player (Line 15). As shown by Lemma A.3, each player may occupy N arms, so such event happens at most N 2 times. And when all players find their unique optimal arm which requires O ( N 2 log T/ ∆ 2 ) times, the phase would continue until the end of the interaction. Above all, the total number of phases is of order O ( N 2 log ( N 2 log T/ ∆ 2 )) . For the detailed communication procedure, as phase 1 in Kong and Li [16], players can first estimate their unique indices and we assume the matching results are public as [16, 18, 21]. During the communication block of each phase, players sequentially transmit their data based on their indices, received by others through matching outcomes. Specifically, in the corresponding round, player p i selects the focused arm if E i = False and nothing otherwise, incurring an O ( N 3 log ( N 2 log T/ ∆ 2 )) cost for status communication in all phases. For deletion (eliminating) sets, p i first selects the arm with index k to indicate it will transmit k arms and then sequentially selects these k arms. The communication cost on the arm set size is O ( N 3 log ( N 2 log T/ ∆ 2 )) . Recall that players delete arms only when a higher-priority player focuses on this arm, so N players focus on at most N arms before reaching stability and each player deletes up to N arms. Also, each player can eliminate up to K -N arms during each exploration and would re-start exploration for at most N 2 times. Thus the communication cost on the deletion (eliminating) arms is O ( N 3 K ) and the total communication cost is O ( N 3 log ( N 2 log T/ ∆ 2 ) + N 3 K ) , which is not the main order of the regret.

Key idea of removing the dependence on K . Balancing the exploration-exploitation trade-off is the key to achieving low regret. Previous efforts were devoted to addressing pessimal stable regret [20, 21, 18] and uniqueness assumptions [28, 4] using classic UCB and TS strategies. Until recently, Zhang et al. [31] and Kong and Li [16] show that ETC-type strategies better fit this problem. Specifically, players first uniformly explore arms to learn the complete preference ranking of the top N arms, and then use the GS procedure for exploitation to find the player-optimal stable matching. However, such a method may over-explore and cause unnecessary regret. The reason is that to learn the first N -ranked arms, each sub-optimal arm a j must be selected O (log T/ ∆ 2 i,σ i,N ,j ) times to be distinguished from the N -ranked arm. And each time selecting this arm, the player pays ∆ i,m ∗ i ,j regret. The mismatch between the paid regret and the difference to be figured out results in O ( K log T/ ∆ 2 N ) regret.

In contrast to the existing approach, we present a more adaptive perspective that integrates the learning process into each GS step. To avoid additional regret, players do not need to estimate their complete preference. Instead, they would start exploitation once the optimal available arm is identi-

fied. And if this arm is occupied by a higher-priority player, this player would restart exploration to find the next preferred one. To avoid the additional cost while exploring the optimal arm, we design a more efficient way to let players promptly eliminate K -N sub-optimal arms and only maintain the remaining N arms to guarantee no collision. For the eliminated arm a j , the reward difference to be figured out is at least ∆ i,m ∗ i ,j , which matches the regret when selecting this arm. So the regret caused by these eliminated arms is O ( K log T/ ∆) , avoiding dependence on K in the main order.

Recall Kong and Li [17] propose an adaptively-explore-then-deferred-acceptance (AETDA) algorithm for more general many-to-one markets with responsiveness. Compared with their realization in the one-to-one setting, our algorithm shares the same idea of balancing exploration and exploitation but further introduces the elimination operation to avoid unnecessary selections. Compared with their O ( N 2 K log T/ ∆ 2 ) result in the reduced one-to-one setting, our Theorem 4.1 removes the dependence on K in the main order.

Discussion on the definition of gaps. Recall that our ∆ is defined as the minimum preference difference among arms that ranked in the first ( σ i ( m ∗ i ) + 1) -th positions ( gap 4 ), while the lower bound in Sankararaman et al. [28] for markets with serial dictatorship depends on ∆ that is defined as the minimum preference difference between the arm ranked σ i ( m ∗ i ) and the arm ranked σ i ( m ∗ i ) + 1 ( gap 1 in Table 1, respectively). It is an open problem whether the lower bound should depend on our gap 4 in general markets. Here we would like to discuss that the knowledge of gap 4 is important to learn the true stable matching. Consider a market with 4 players and 5 arms. The preference rankings of players are p 1 : a 1 > a 2 > a 3 > a 4 > a 5 ; p 2 : a 2 > a 3 > a 1 > a 4 > a 5 ; p 3 : a 3 > a 1 > a 2 > a 4 > a 5 ; p 4 : a 1 > a 2 > a 3 > a 4 > a 5 and the preference rankings of arms are a 1 : p 2 > p 3 > p 4 > p 1 ; a 2 : p 3 > p 4 > p 1 > p 2 ; a 3 : p 4 > p 1 > p 2 > p 3 ; a 4 : p 1 > p 2 > p 3 > p 4 ; a 5 : p 1 > p 2 > p 3 > p 4 . In this market, the player-optimal stable matching is { ( p 1 , a 4 ) , ( p 2 , a 1 ) , ( p 3 , a 2 ) , ( p 4 , a 3 ) } . However, if player p 1 has collected enough observations to identify gap 1 but not collected enough observations to identify gap 4 and wrongly estimate the first σ i ( m ∗ i ) ranked arms, i.e., p 1 wrongly estimate the preference ranking as p 1 : a 1 > a 2 > a 4 > a 3 > a 5 . Then the computed player-optimal stable matching under this preference ranking is { ( p 1 , a 4 ) , ( p 2 , a 3 ) , ( p 3 , a 1 ) , ( p 4 , a 2 ) } , which is not stable in the original market as player p 1 and arm a 3 form a blocking pair. This example shows that player p 1 must identify the gap among the first σ i ( m ∗ i ) ranked arms to find a stable matching in the market, which further illustrates the crucial role of gap 4 in learning the true stable matching. We leave the lower bound in general markets as an important future direction.

5 Experiments

In this section, we compare our Algorithm 1 (abbreviated as AETGS-E) with baselines ETGS [16], ML-ETC [31] and Phased ETC [4] which also enjoy guarantees for player-optimal stable regret in general decentralized one-to-one markets. To better illustrate the advantages of our algorithm, especially when N is much smaller than K , we set N = 3 and K = 10 . The preference rankings for both players and arms are generated as random permutations. The preference gap between any adjacent ranked arms is set as 0 . 1 . The feedback X i,j ( t ) for player p i on arm a j at time t is drawn independently from the Gaussian distribution with mean µ i,j and variance 1 . We report the maximum cumulative player-optimal stable regret among all players and the cumulative player-optimal instability in Figure 1 (a) and (b), respectively. Here the cumulative player-optimal unstability is defined as the number of matchings that are not the player-optimal stable one. All algorithms run for T = 100 k rounds and all results are averaged over 50 independent runs. The error bars represent standard errors, which are computed as standard deviations divided by √ 50 .

As shown in the figure, our AETGS-E algorithm, which only conducts necessary explorations over unknown preferences and promptly eliminates sub-optimal arms, achieves the least cumulative regret and cumulative player-optimal unstability among all baselines. The ML-ETC and ETGS algorithms need to sufficiently explore K arms to estimate the full preference ranking, requiring more exploration time to find the player-optimal stable matching. The PhasedETC algorithm has not yet converged within the displayed rounds due to the cold start problem.

0

20k

40k

60k

80k

100k

Round

t

0

5k

10k

15k

20k

25k

Maximum Cumulative Regret

(a) Cumulative optimal stable regret

AETGS-E

ETGS

PhasedETC

ML-ETC

0

20k

40k

60k

80k

100k

Round

t

0

10k

20k

30k

40k

50k

Player-optimal Unstability

(b) Cumulative player-optimal unstability

AETGS-E

ETGS

PhasedETC

ML-ETC

6 Centralized UCB Algorithm for Markets with α -condition

In this section, we provide a new analysis for the centralized UCB algorithm in markets satisfying α -condition. The algorithm is first introduced by Liu et al. [20]. For completeness, we present the full algorithm in Algorithm 2. At each round, players submit their UCB rankings to the centralized platform (Line 3). The platform runs the GS algorithm (based on players' submitted rankings) and returns the partner to each player (Line 4).

Algorithm 2 centralized UCB

Input: N,K .

1: Initialize: ˆ µ i,j (0) = 0 , T i,j (0) = 0 , UCB i,j (1) = ∞ , ∀ i ∈ [ N ] , j ∈ [ K ] .

2: for round t = 1 , 2 , . . . , T do

3: Receive rankings ˆ σ := { ˆ σ i } i ∈ [ N ] according to the decreasing order of { UCB i,j ( t ) } j ∈ [ K ] , ∀ i ∈ [ N ] ;

4: A i ( t ) ← Gale-Shapley ( ˆ σ, π ) for each player p i ;

5: Observe X i ( t ) , and update ˆ µ i,j ( t ) , T i,j ( t ) , UCB i,j ( t +1) for each p i , a j ;

6: end for

In the following, we introduce the α -condition. Conditions guaranteeing the unique stable matching have been widely studied in the offline setting [5, 13] and also the online setting to improve the learning efficiency [28, 4, 22]. Among these conditions, the α -condition is shown to be the weakest sufficient one [13] and incorporate the conditions studied in existing works [28, 4, 22].

Let β denote a pair of permutations of [ N ] and [ K ] . Then [ N ] β = { Q ( β ) 1 , . . . , Q ( β ) N } and [ K ] β = { q ( β ) 1 , . . . , q ( β ) K } denote permutations of the ordered sets [ N ] and [ K ] , respectively. The j -th player in [ N ] β is the Q ( β ) j -th player in [ N ] , and the k -th arm in [ K ] β is the q ( β ) k -th arm in [ K ] . Then we can define the α -condition below.

Definition 6.1. The α -condition is satisfied if there is a stable matching ( j ∗ , i ∗ ) , a left-order of players and arms s.t. ∀ i ∈ [ N ] l , ∀ j > i, j ∈ [ K ] l : µ i,j ∗ i > µ i,j where j ∗ i is the partner of player p i in stable matching ( j ∗ , i ∗ ) , and a (possibly different) right-order of players and arms s.t. ∀ j < i ≤ N,q j ∈ [ K ] r , Q i ∈ [ N ] r : π q j ,Q i ∗ q j > π q j ,Q i . Here similarly, i ∗ q j is the partner of arm a q j in stable matching ( j ∗ , i ∗ ) .

Without loss of generality, we consider the identity of players and arms is just the left order, i.e., [ N ] = [ N ] l and [ K ] = [ K ] l . Thus we only deal with player order Q ( r ) i = Q i and arm order q ( r ) j = q j , for i ∈ [ N ] , j ∈ [ K ] in the rest of the paper. Under α -condition, it is easy to inductively verify that for any i ∈ [ N ] , the player p i is matched with arm a i , and the player p Q i is matched with the arm a q i in the unique stable matching [4].

6.1 Theoretical Results

We analyze the regret for the centralized UCB algorithm under α -condition.

Theorem 6.2. When preferences of participants satisfy α -condition, following Algorithm 2, the stable regret for each player p i satisfies

The centralized UCB algorithm is proposed by Liu et al. [20] and shown to have O ( NK log T/ ∆ 2 ) player-pessimal stable regret for general markets. We provide a new analysis for markets satisfying α -condition which removes the dependence of K in the regret. Due to the space limit, we discuss the key idea of the proof below and defer the detailed proof to Appendix C.

We investigate the preference structure of α -condition to obtain the improved analysis. For player p i , its regret is due to selecting sub-optimal arm a k with µ i,k < µ i,i . Arm a k will be selected by p i when its UCB value is higher than p i 's stable matched arm a i , which time is bounded by O (log T/ ∆ 2 i,i,k ) , and one ∆ i,i,k on the denominator can be eliminated when multiplying ∆ i,i,k to compute regret. This contributes O ( K log T/ ∆) regret since there are at most K -1 sub-optimal arms. It is worth noting that arm a k will also be selected by p i if p i is rejected by a i in the GS algorithm. Recall that under α -condition, there is a right order Q i ′ = i ∈ [ N ] r for player p i , such that ∀ i ′′ > i ′ , Q i ′′ ∈ [ N ] r : π q i ′ ,Q i ′ > π q i ′ ,Q i ′′ , which means arm a q i ′ = a i can only prefer players p Q 1 , p Q 2 , · · · , p Q i ′ -1 than player p Q i ′ = p i . Thus p i is rejected by a i only when these players select a i , and a i is sub-optimal for those players. To bound the regret of p i when being rejected, we just need to bound the exploration times of these players p Q 1 , p Q 2 , · · · , p Q i ′ -1 on arm a i . However, the exploration time of a single player p Q ℓ with 1 ≤ ℓ ≤ i ′ -1 on a i can not be trivially bounded by O (log T/ ∆ 2 N ) since p Q ℓ may have to select arm a i after rejected by its stable arm a q ℓ in offline GS, where a q ℓ might be selected by p Q 1 , · · · , p Q ℓ -1 . This leads to a recursion form. We control this term using the fact that when a player is rejected by its stable matched arm in the GS, it can date back to a higher right-order player wrongly over-estimate its preference for a sub-optimal arm. This key observation and the definition of ∆ make it possible to derive the final O ( N log T/ ∆ 2 N ) bound.

7 Conclusion

In this paper, we investigate the problem of whether a tighter bound can be derived for the bandit learning problem in two-sided matching markets. For the general one-to-one matching markets, we try to improve the learning efficiency of the existing algorithms. By integrating the offline GS procedure into the online learning process and carefully designing the elimination strategy, we show that the player-optimal stable regret can be upper bounded by O ( N 2 log T/ ∆ 2 + K log T/ ∆) . This result removes the dependence on K in the main order term of existing works and improves the stateof-the-art result [31, 16] in common cases where the number of players is much smaller than that of arms. An experiment is conducted to verify its advantage over other baselines in such markets. We also present a novel analysis for the centralized UCB algorithm in markets satisfying α -condition and derive an improved O ( N log T/ ∆ 2 N + K log T/ ∆) regret upper bound.

One significant future direction is to investigate the optimality of algorithms. Although the dependence on N,K,T in Theorem 6.2 matches the lower bound, the definition of ∆ differs. It remains unclear how the upper bound changes with the same ∆ . Furthermore, since the lower bound provided by [28] applies only to special markets, and the learning problem in general markets is more challenging due to the complex preference structure, determining whether an algorithm can perform better in general markets is still an open problem.

Acknowledgments and Disclosure of Funding

The corresponding author Shuai Li is supported by National Key Research and Development Program of China (2022ZD0114804) and National Natural Science Foundation of China (62376154). Fang Kong is supported by the Baidu Scholarship.

We thank Yuhao Zhang and Wenqian Wang for valuable discussions and suggestions on the proof of Lemma B.2.

References

Atila Abdulkadiro˘ glu and Tayfun S¨ onmez. House allocation with existing tenants. Journal of Economic Theory , 88(2):233-260, 1999.

Shipra Agrawal and Navin Goyal. Further optimal regret bounds for thompson sampling. In Proceedings of the 16th International Conference on Artificial Intelligence and Statistics , pages 99-107, 2013.

Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning , 47(2):235-256, 2002.

Soumya Basu, Karthik Abinav Sankararaman, and Abishek Sankararaman. Beyond log 2 ( t ) regret for decentralized bandits in matching markets. In International Conference on Machine Learning , pages 705-715, 2021.

Simon Clark. The uniqueness of stable matchings. Contributions in Theoretical Economics , 6(1), 2006.

Xiaowu Dai and Michael Jordan. Learning in multi-stage decentralized matching markets. In Advances in Neural Information Processing Systems , volume 34, pages 12798-12809, 2021.

Xiaowu Dai and Michael I Jordan. Learning strategies in decentralized matching markets under uncertain preferences. Journal of Machine Learning Research , 22(1):11806-11855, 2021.

Sanmay Das and Emir Kamenica. Two-sided bandits and the dating market. In International Joint Conference on Artificial Intelligence , pages 947-952, 2005.

David Gale and Lloyd S Shapley. College admissions and the stability of marriage. The American Mathematical Monthly , 69(1):9-15, 1962.

Aur´ elien Garivier, Tor Lattimore, and Emilie Kaufmann. On explore-then-commit strategies. In Advances in Neural Information Processing Systems , volume 29, pages 784-792, 2016.

Avishek Ghosh, Abishek Sankararaman, Kannan Ramchandran, Tara Javidi, and Arya Mazumdar. Decentralized competing bandits in non-stationary matching markets. arXiv preprint arXiv:2206.00120 , 2022.

Meena Jagadeesan, Alexander Wei, Yixin Wang, Michael Jordan, and Jacob Steinhardt. Learning equilibria in matching markets from bandit feedback. In Advances in Neural Information Processing Systems , volume 34, 2021.

Alexander Karpov. A necessary and sufficient condition for uniqueness consistency in the stable marriage matching problem. Economics Letters , 178:63-65, 2019.

Emilie Kaufmann, Nathaniel Korda, and R´ emi Munos. Thompson sampling: An asymptotically optimal finite-time analysis. In International Conference on Algorithmic Learning Theory , pages 199-213. Springer, 2012.

Alexander S Kelso Jr and Vincent P Crawford. Job matching, coalition formation, and gross substitutes. Econometrica: Journal of the Econometric Society , pages 1483-1504, 1982.

Fang Kong and Shuai Li. Player-optimal stable regret for bandit learning in matching markets. In Proceedings of the 2023 Annual ACM-SIAM Symposium on Discrete Algorithms (SODA) . SIAM, 2023.

Fang Kong and Shuai Li. Improved bandits in many-to-one matching markets with incentive compatibility. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 13256-13264, 2024.

Fang Kong, Junming Yin, and Shuai Li. Thompson sampling for bandit learning in matching markets. In International Joint Conference on Artificial Intelligence , 2022.

Tor Lattimore and Csaba Szepesv´ ari. Bandit algorithms . Cambridge University Press, 2020.

Lydia T Liu, Horia Mania, and Michael Jordan. Competing bandits in matching markets. In International Conference on Artificial Intelligence and Statistics , pages 1618-1628. PMLR, 2020.

Lydia T Liu, Feng Ruan, Horia Mania, and Michael I Jordan. Bandit learning in decentralized matching markets. Journal of Machine Learning Research , 22(211):1-34, 2021.

Chinmay Maheshwari, Eric Mazumdar, and Shankar Sastry. Decentralized, communicationand coordination-free learning in structured matching markets. In Advances in Neural Information Processing Systems , 2022.

Yifei Min, Tianhao Wang, Ruitu Xu, Zhaoran Wang, Michael Jordan, and Zhuoran Yang. Learn to match with no regret: Reinforcement learning in markov matching markets. In Advances in Neural Information Processing Systems , volume 35, pages 19956-19970, 2022.

Deepan Muthirayan, Chinmay Maheshwari, Pramod Khargonekar, and Shankar Sastry. Competing bandits in time varying matching markets. In Learning for Dynamics and Control Conference , pages 1020-1031. PMLR, 2023.

Alvin E Roth and Marilda Sotomayor. Two-sided matching. Handbook of game theory with economic applications , 1:485-541, 1992.

Alvin E Roth. The evolution of the labor market for medical interns and residents: a case study in game theory. Journal of political Economy , 92(6):991-1016, 1984.

Alvin E Roth. Stability and polarization of interests in job matching. Econometrica: Journal of the Econometric Society , pages 47-57, 1984.

Abishek Sankararaman, Soumya Basu, and Karthik Abinav Sankararaman. Dominate or delete: Decentralized competing bandits in serial dictatorship. In International Conference on Artificial Intelligence and Statistics , pages 1252-1260. PMLR, 2021.

Zilong Wang and Shuai Li. Optimal analysis for bandit learning in matching markets with serial dictatorship. Theoretical Computer Science , 1010:114703, 2024.

Zilong Wang, Liya Guo, Junming Yin, and Shuai Li. Bandit learning in many-to-one matching markets. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management , pages 2088-2097, 2022.

Yirui Zhang, Siwei Wang, and Zhixuan Fang. Matching in multi-arm bandit with collision. In Advances in Neural Information Processing Systems , 2022.

A Proof of Theorem 4.1

Define F = { ∃ 1 ≤ t ≤ T, i ∈ [ N ] , j ∈ [ K ] : | ˆ µ i,j ( t ) -µ i,j | > √ 6 log T T i,j ( t ) } as the failure event that the estimated reward is far from the expected reward at some time and some player-arm pair. The regret can be upper bounded as follows.

where Eq. (1) holds based on Lemma A.1, A.2, and A.4.

Lemma A.1.

Proof. Recall that player p i would update the available set A i when other players p i ′ sets E i ′ = False and π A i ′ ,i ′ > π A i ′ ,i as Line 14. Denote t s as the round index when this operation happens for the s -th time. Without loss of generality, let t 0 = 1 .

Recall that at a high level, each time another player p i ′ sets E i ′ as False , it means that p i ′ learns its most preferred arm in current available set. Combined with F and Lemma A.6, the determined arm of players during each exploration would be truly their most preferred one. Thus the AETGS algorithm is an online version of GS and the s ′ -th time player p i ′ sets E i ′ as False is equivalent to that p i ′ proposes its s ′ -th most preferred arm in the offline GS. According to Lemma A.3, at most N -1 arms are proposed by all players before reaching stability. Thus for player p i , the operation in Line 14 would happen for at most N -1 times.

Recall that for each s , during time t s to t s +1 , player p i would explore all available arms in a roundrobin manner, eliminate sub-optimal arms until N arms are in the set, and focuses on the best one among these N when it is identified. For convenience, denote R s as the set of the remaining N arms that p i explored in A i in a round-robin manner until condition Line 9 is satisfied, D s as the set of arms that p i eliminated due to condition Line 6, and j s as the arm that p i focuses from the time it sets E i as False to time t s +1 -1 . Then it holds that

where Eq. (2) is due to that, based on Lemma A.6 and the offline GS algorithm, the arm j s before GS stops would be better than m ∗ i , thus the regret caused by selecting these arms is less than 0 . For the first term in Eq. (2), we have

where Eq. (4) is due to Lemma A.5, Eq. (5) holds since R s contains no more than N arms according to the elimination condition (Line 6).

We now analyze the second term in Eq. (2). For any arm a j and s ∈ { 0 , ..., N -1 } , denote T i,j,s as the value of T i,j at the end of the round t s +1 -1 . For s ≥ 1 and arm a j ∈ D s , if T i,j,s -1 ≤ 96 log T/ ∆ 2 i,j s -1 ,j , it must hold that T i,j,s := ∑ s ′ ≤ s ( T i,j,s ′ -T i,j,s ′ -1 ) ≤ 96 log T/ ∆ 2 i,j s ,j to ensure arm a j is eliminated from A i at step s based on Lemma A.5. On the other hand, if T i,j,s -1 > 96 log T/ ∆ 2 i,j s -1 ,j , based on Lemma A.5, it holds that T i,j,s -T i,j,s -1 ≤ 96 log T/ ∆ 2 i,j s ,j -96 log T/ ∆ 2 i,j s -1 ,j when a j is eliminated. For any arm a j , denote s j, 1 := max 0 ≤ s ≤ N -1 { s : T i,j,s ≤ 96 log T/ ∆ 2 i,j s ,j } as the last step when the number of observation times on a j is less than that threshold. Then the second term in Eq. (2) satisfies

where the second last line is due to the definition of s j, 1 and the above analysis.

Above all,

Lemma A.2.

Proof. Based on the AETGS algorithm, when E i = True , the central platform would assign arms in A i to player p i in a round-robin manner. Since the number of arms |∪ i :E i =True A i | to be explored is larger than the number ∑ i 1 { E i = True } of players with E i = True based on the elimination condition in Line 6, we can assume that there is no collision in the exploration phase as discussed in Section 4. So the regret caused by collision only occurs during time with E i = False .

Denote t s and t s as the round index when p i sets E i as False for the s -th time and as True for the s +1 -th time, respectively. Recall that when E i = False , p i will always select arm A i . Here we use j s to represent the arm that is selected by p i from time t s to t s .

Further, recall that in the AETGS algorithm, each time an arm is added into D i (Line 13), the eliminated arms may be contained into A i again. And only when other players focus on their currently most preferred arm, such operation of adding arms to D happens. Based on Lemma A.3, such an operation happens for at most N times. For any player p i ′ , denote t i ′ ,r as the round index when p i ′ adds arms to D i ′ (Line 13) for r -th time. Then { t i ′ ,r } r ∈ [ N ] further divide { [ t s , t s ] } s ∈ [ N ] into at most 2 N slices. We use t ′ s , t ′ s to represent the start round and end round index of the s -th slice, where s ∈ [2 N ] . Based on Lemma A.5, p i ′ and p i would select the same arm for at most 96 log T/ ∆ 2 times within each slice.

Then the regret satisfies

̸

Lemma A.3. In the offline GS algorithm, at most N -1 arms have been proposed by players before the algorithm stops.

̸

̸

Proof. Based on the offline GS algorithm, once an arm is proposed, it has a temporary player. By contradiction, once N arms have been proposed, it means that N players are occupied. In this case, each player has a partner and the algorithm stops.

Lemma A.4.

Proof.

where the second last inequality is due to Lemma B.1.

Lemma A.5. For any player p i , let ¯ T i = 96log T/ ∆ 2 . For any two arms j, j ′ with µ i,j > µ i,j ′ and σ i ( a j ) ∈ [1 , σ i ( m ∗ i )] , if T i ( t ) := min { T i,j ( t ) , T i,j ′ ( t ) } > ¯ T i , we have UCB i,j ′ ( t ) < LCB i,j ( t ) conditioned on ⌝ F .

Proof. By contradiction, suppose UCB i,j ′ ( t ) ≥ LCB i,j ( t ) . According to ⌝ F and the definition of LCB and UCB , we have

We can then conclude ∆ i,j,j ′ = µ i,j -µ i,j ′ ≤ 4 √ 6 log T T i ( t ) , which implies that T i ( t ) ≤ 96 log T ∆ 2 i,j,j ′ ≤ 96 log T ∆ 2 . This contradicts the fact that T i ( t ) > ¯ T i .

Lemma A.6. Conditioned on ⌝ F , at any time t , UCB i,j ( t ) < LCB i,j ′ ( t ) implies µ i,j < µ i,j ′ .

Proof. According to the definition of LCB and UCB , we have

where two inequalities comes from ⌝ F . Thus if UCB i,j ( t ) < LCB i,j ′ ( t ) , there would be

The lemma can thus be proved.

B Technical Lemmas

Lemma B.1. (Corollary 5.5 in Lattimore and Szepesv´ ari [19]) Assume that X 1 , X 2 , . . . , X n are independent, σ -subgaussian random variables centered around µ . Then for any ε > 0 ,

Lemma B.2. (Arrangement of players' round-robin exploration) Suppose there are N players who need to explore their respective N arms. There exists an assignment such that during 2 N rounds, each player can match with each of its arm for once.

Proof. Without loss of generality, let's assume that players assign their respective N arms in 2 N rounds one by one, based on the players' and arms' indices, aiming to ensure that no arm is assigned to more than one player at the same round. By contradiction, suppose when player p i assigns its j -th arm, there is no available round to make this assignment due to conflicting constraints. Given that player p i is currently assigning the j -th arm, it implies that there are 2 N -j +1 rounds where no arm is assigned to player p i . Since none of these rounds satisfy the conflict constraint, it means that the previous i -1 players assigned arm j in these 2 N -j +1 rounds. This creates a contradiction since the first i -1 players can only occupy i -1 rounds when selecting arm j , where i -1 < 2 N -j +1 with i ≤ N,j ≤ N .

C Proof of Theorem 6.2

In this section, we analyze the regret of the centralized-UCB algorithm under α -condition. Recall that F = { ∃ 1 ≤ t ≤ T, i ∈ [ N ] , j ∈ [ K ] : | ˆ µ i,j ( t ) -µ i,j | > √ 6 log T T i,j ( t ) } is the failure event that the estimated reward is far from the expected reward at some time and some player-arm pair.

For any player p i with i ∈ [ N ] , we know that its stable arm is a i under α -condition. Thus its regret can be decomposed as

The first term is the number of selections for sub-optimal arms. The second term is the regret caused by the bad events.

̸

For the arm a k such that it is sub-optimal for player p i , i.e., µ i,k < µ i,i , it will be selected because the preference for arm a k of player p i is estimated higher than its stable matched arm a i , or player p i is rejected by arm a k in the GS algorithm. Note that under α -condition, there is a right order Q i ′ = i ∈ [ N ] r for player p i , such that ∀ i ′ < i ′′ ≤ N,Q i ′′ ∈ [ N ] r : π q i ′ ,Q i ′ > π q i ′ ,Q i ′′ , which means arm a q i ′ = a i can only prefer players p Q 1 , p Q 2 , · · · , p Q i ′ -1 than player p Q i ′ = p i . Denote The right-order mapping for α -condition for player p i is lr ( i ) so that Q lr ( i ) = i with Q i defined in Definition 6.1, and lr ( i ) ≤ N . For player p i , denote G t,i := {∀ 1 ≤ i ′ ≤ lr ( i ) -1 , ¯ A Q i ′ ( t ) = i } as the event all players preferred by arm a i do not select p i at time t . Then the number of selections for sub-optimal arm a k can be decomposed as

The last inequality is from Lemma C.1.

For the second term in the RHS of the last inequality, E [ ∑ T t =1 1 { ¯ A i ( t ) = k, ⌝ G t,i , ⌝ F } ] , we can sum over all sub-optimal arms and it turns out to be

Hence,

where the third inequity is from the Lemma C.2. The fourth inequality is from the definition of ∆ N . Above all, the stable regret of player i can be bounded by

where the second inequality is based on Lemma A.4.

Lemma C.1. Conditioned on ⌝ F , under the traditional single-player UCB algorithm with single player p i , the expected number of times at which the UCB index of arm a j ′ exceeds that of the better arm a j , is at most 24 log( T ) / ∆ 2 i,j,j ′ by round T .

Proof. Conditioned on ⌝ F , for any i, j, t we have,

Recall that the UCB index is:

The event that arm a j ′ is successfully selected for player p i rather than the better arm a j at time t implies that

which leads to

where ∆ i,j,j ′ is the reward difference between the µ i,j ′ and µ i,j .

Lemma C.2. For any player p i with right order Q lr ( i ) , the following inequality holds:

Proof. For player p i with right order Q lr ( i ) , from α -condition we have that its stable matched arm a i may prefer p Q 1 , p Q 2 , · · · , p Q lr ( i ) -1 than player p Q lr ( i ) . For any i ′ < lr ( i ) , we know that the number of times player p Q i ′ selects arm a i is decomposed as by

The event ⌝ G t,i ′ implies that there exists another player p Q i ′′ with i ′′ < i ′ that selects the stable arm of p Q i ′ . This leads to a recursion form. But it is easy to verify that every event ⌝ G t,i ′ happens only when there exists two players p Q u ′ , p Q u ′′ with u ′ < u ′′ ≤ lr ( i ′ ) , such that player p Q u ′ explores the stable matched arm a q u ′′ of p Q u ′′ , i.e., p Q u ′ selects a q u ′′ conditioned on G t,u ′ . And thus it holds that

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Our abstract and introduction clearly describe the scope and outline our main contributions.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Section 7 discusses the limitation of this paper.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate 'Limitations' section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We provide techniques clearly in the paper, and their detailed proofs are in Appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Section 5 provides the settings and results of our experiments carefully.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The codes are uploaded in supplementary material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Section 5 provides the settings and results of our experiments carefully.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We provide the error bar in the results of experiments in Section 5.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer 'Yes' if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [No]

Justification: We do not provide specific information on the computer resources used, as most experiments on multi-armed bandits are lightweight.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper complies with NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: There is no societal impact of the work performed.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: The paper does not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
This paper considers two-sided matching bandit problems, where the goal is to minimize regret against a player's optimal stable matching. The state-of-the-art algorithms achieve a regret bound of  $KlogT/\Delta^2$.  The authors suggest an algorithm using an adaptive online Gale-Shapley to achieve a regret bound of $N^2\log T/\Delta^2$ under $N<K$. Under $\alpha$-condition, they provide a new analysis for the previous algorithm to achieve a regret bound of $N\log T/\Delta^2$.

STRENGTHS:
1. The authors suggest an adaptive online Gale-Shapley algorithm.
2. The suggested algorithms outperform the previous algorithms when $N$ is much smaller than $K$.
3. They demonstrate this result using synthetic experiments.

WEAKNESSES:
1. Without $\alpha$-condition, the upper bound has $N^2$ instead of $N$, which may not be tight with respect to $N$.
2. Even though they provide a new algorithm, it may be hard to see the technical novelty.

QUESTIONS:
Without $\alpha$-condition, is the lower bound linear with respect to $N$?

LIMITATIONS:
The authors have discussed the limitations in the conclusion.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-06-28

comments: SUMMARY:
The paper studies the bandit learning problem in two-sided matching markets and provide improved regret analysis that nearly match the lower bound (although with slightly different definition of instance-dependent gaps). The bound is particularly useful when the number of players is much smaller than the number of arms. Numerical experiments show significant improvement under some special cases. The paper also improve the regret analysis for an existing UCB algorithm under an additional $\alpha$-condition.

STRENGTHS:
1. The paper is very well-written. The flow is clear and easy-to-follow.
2. Strong technical novelty as well as solid theoretical results. The paper gives new insights on how to balance between exploration and exploitation in matching markets.
3. Numerical experiments show significant improvement compared to existing policies.

WEAKNESSES:
Although the dependence on $N$, $K$, $T$ is optimal, the main weakness lies in the difference of defining the instance-dependent gap $\Delta$.

QUESTIONS:
I would suggest the authors add a paragraph in the introduction to explcitly point out the contributions in this paper. In particular, I would like to see the main algorithmic design insight be emphasized from this work. For example, why previous algorithms fail to obtain low regret (due to over exploration)?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-12

comments: SUMMARY:
The paper proposed a new algorithm for bandit learning in matching markets and showed new results on regret bounds. However, the combination of bandits with matching markets is wired. Is there any evidence that practitioners would like to use bandits for learning in markets? The novelty of the theory is also unclear.

STRENGTHS:
The paper proposed a new algorithm for bandit learning in matching markets and showed new results on regret bounds.

WEAKNESSES:
The combination of bandits with matching markets is wired. Is there any evidence that practitioners would like to use bandits for learning in markets?

QUESTIONS:
1. The key novelty in the paper is unclear. What are the new techniques developed in the paper for proving the upper bound? 

2. Do you have any empirical markets that could use the proposed algorithms?

3. How to generalize the techniques to decentralized markets?

LIMITATIONS:
The combination of bandits with matching markets is wired. Is there any evidence that practitioners would like to use bandits for learning in markets?
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-14

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studies the problem of bandit learning in two-sided matching markets, where the number of players $N$ is smaller than the number of arms $K$. The players' preferences $\\mu\\_{i,j}$ are unknown but the arms' utility preferences $\\pi\\_{i,j}$ are known. Two algorithms with improved regret bounds are proposed. The first one is adaptive online Gale-Shapley (AOGS) for general markets, where multiple stable matchings exist. In AOGS, each player follows an adaptive explore-then-commit strategy to identify the best available arm based on the confidence bounds of each arm reward estimates and whether an arm is better matched to other players. The regret bound of AOGS is stated in terms of a new notion of gap. The second algorithm is centralized UCB combined with offline Gale-Shapely for markets with $\\alpha$-condition, where a unique stable matching exists.\n\nSTRENGTHS:\n- Novelty: The following contributions of the work seem novel: the new definition of the gap, the exploration strategy in Algorithm 1, and the improved analysis of Algorithm 2.\n- Significance: All algorithms appear to be rigorously analyzed. No proofs are missing. The bound in Theorem 6.2 have optimal dependency on $K, N$ and $log(T)$ compared to a known lower bound.\n- Writing quality: The paper is generally well-written.\n\nWEAKNESSES:\nThe contributions are rather incremental. Details are below:\n- The new bound in Theorem 4.1 depends on a new notion of gap. Given that the existing literature in Table 1 already have at least three different notions of gaps, it is not clear in what sense another type of gap is better than the existing ones. Does the new gap capture the difficulty of practical scenarios better? This was not sufficiently discussed in the paper. \n- A lower bound result for the new gap is missing. As the paper acknowledged, comparing with the lower bound in Sankararaman et al is not entirely meaningful because the lower bound there uses a different kind of gap.\n- In general, a comprehensive comparison with existing results is missing. For example, do any of the existing results in Table 1 imply anything about the new results in the current paper? Another thing is Algorithm 2 was proposed by Liu et al for $\\mathrm{gap}\\_2$, while the paper uses the same algorithm for $\\mathrm{gap}\\_3$ under $\\alpha$-condition, so it is not clear why the analysis in this paper is considered an improved version of the one in Liu et al (the goal is different). Maybe if the proposed analysis implies that Algorithm 2 obtain optimal bounds on multiple types of gaps *simultaneously*, then it would be a significant result.\n\nQUESTIONS:\nPlease see the questions in the Weakness section above. One more question below:\n- The assumption that $\\pi\\_{i,j}$ are known seem strong and has been exploited thoroughly in previous works. How are the results impacted if this assumption does not  hold?\n\nLIMITATIONS:\n- The paper relies on a strong assumption that $\\pi\\_{i,j}$ must be fully known and does not discuss how the approach and results change if this assumption is violated. In particular, it is not clear whether a weaker assumption would work, such as only the top $k$ $\\pi\\_{i,j}$ are known for $k < \\min(N, K)$.\n- The theory in this paper concerns a practical setting that might have some potential societal impacts, none of which is significant.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}