The 5G Cellular Backhaul Management Dilemma: To Cache or to Serve

To reap the benefits of cache-enabled small cell networks, new backaul management mechanisms are needed to prevent the predicted files that are downloaded at the small base stations (SBSs) for caching purposes from jeopardizing the urgent requests that need to be served via the backhaul. Such mechanisms must account for the heterogeneity of the backhaul that will encompass both wireless backhaul links (at various frequency bands) and a wired backhaul component. In this paper, the heterogeneous backhaul management problem is formulated as a minority game in which each SBS has to define the number of predicted files to download, without affecting the required transmission rate of the current requests. For the formulated game, it is shown that a unique fair proper mixed Nash equilibrium (PMNE) exists. A self-organizing reinforcement learning algorithm is then proposed and shown to converge to a unique Boltzmann-Gibbs equilibrium, which approximates the desired PMNE. Simulation results show that the performance of the proposed approach can be close to that of the ideal optimal algorithm while it outperforms a centralized greedy approach in terms of the amount of data that is cached without jeopardizing the quality-of-service of current requests.

Related papers:

title: What Will 5G Be?
abstract: What will 5G be? What it will not be is an incremental advance on 4G. The previous four generations of cellular technology have each been a major paradigm shift that has broken backward compatibility. Indeed, 5G will need to be a paradigm shift that includes very high carrier frequencies with massive bandwidths, extreme base station and device densities, and unprecedented numbers of antennas. However, unlike the previous four generations, it will also be highly integrative: tying any new 5G air interface and spectrum together with LTE and WiFi to provide universal high-rate coverage and a seamless user experience. To support this, the core network will also have to reach unprecedented levels of flexibility and intelligence, spectrum regulation will need to be rethought and improved, and energy and cost efficiencies will become even more critical considerations. This paper discusses all of these topics, identifying key challenges for future research and preliminary 5G standardization activities, while providing a comprehensive overview of the current literature, and in particular of the papers appearing in this special issue.

title: Inter-Operator Resource Management for Millimeter Wave Multi-Hop Backhaul Networks
abstract: In this paper, a novel framework is proposed for optimizing the operation and performance of a large-scale multi-hop millimeter wave (mmW) backhaul within a wireless small cell network having multiple mobile network operators (MNOs). The proposed framework enables the small base stations to jointly decide on forming the multi-hop, mmW links over backhaul infrastructure that belongs to multiple, independent MNOs, while properly allocating resources across those links. In this regard, the problem is addressed using a novel framework based on matching theory composed of two, highly inter-related stages: a multi-hop network formation stage and a resource management stage. One unique feature of this framework is that it jointly accounts for both wireless channel characteristics and economic factors during both network formation and resource management. The multi-hop network formation stage is formulated as a one-to-many matching game, which is solved using a novel algorithm, that builds on the so-called deferred acceptance algorithm and is shown to yield a stable and Pareto optimal multi-hop mmW backhaul network. Then, a one-to-many matching game is formulated to enable proper resource allocation across the formed multi-hop network. This game is then shown to exhibit peer effects and, as such, a novel algorithm is developed to find a stable and optimal resource management solution that can properly cope with these peer effects. Simulation results show that, with manageable complexity, the proposed framework yields substantial gains, in terms of the average sum rate, reaching up to 27% and 54%, respectively, compared with a non-cooperative scheme in which inter-operator sharing is not allowed and a random allocation approach. The results also show that our framework improves the statistics of the backhaul sum rate and provides insights on how to manage pricing and the cost of the cooperative mmW backhaul network for the MNOs.

title: Dynamic Backhaul Resource Allocation: An Evolutionary Game Theoretic Approach
abstract: The recently deployed 4G access technology promises to satisfy the increasing demand of bandwidth consuming applications by providing high network capacity, low latency and seamless mobility. Towards this direction, concept solutions concerning the integration of wireless and optical networks have been proposed. However, the majority of these approaches assume the conventional fixed commitment of resources to the base stations, an inefficient and costly process, especially in case the Passive Optical Network (PON) belongs to a different operator. As a result, new, more dynamic backhaul resource allocation approaches are required. In this paper, the authors study the problem of committing resources of the backhaul network to a base station by employing evolutionary game theory in order to model the interactions between the subscribers and the base station. The asymptotic stability of the proposed scheme is proven under the replicator dynamics model. Finally, the impact of time delay in the proposed scheme is also investigated.

title: Millimeter-Wave Enhanced Local Area Systems: A High-Data-Rate Approach for Future Wireless Networks
abstract: Wireless data traffic is projected to skyrocket 10 000 fold within the next 20 years. To tackle this incredible increase in wireless data traffic, a first approach is to further improve spectrally efficient systems such as 4G LTE in bands below 6 GHz by using more advanced spectral efficiency techniques. However, the required substantial increase in system complexity along with fundamental limits on hardware implementation and channel conditions may limit the viability of this approach. Furthermore, the end result would be an extremely spectrally efficient system with little room for future improvement to meet the ever-growing wireless data usage. The second approach is to move up in frequency, into an unused nontraditional spectrum where enormous bandwidths are available, such as at millimeter wave (mmWave). The mmWave option enables the use of simple air interfaces since large bandwidths can be exploited (e.g., 2 GHz) to achieve high data rates rather than relying on highly complex techniques originally aimed at achieving a high spectral efficiency with smaller bandwidths. In addition, mmWave systems will easily evolve to even higher system capacities, because there will be plenty of margin to improve the spectral efficiency as data demands further increase. In this paper, a case is made for using mmWave for a fifth generation (5G) wireless system for ultradense networks by presenting an overview of enhanced local area (eLA) technology at mmWave with emphasis on 5G requirements, spectrum considerations, propagation and channel modeling, air-interface and multiantenna design, and network architecture solutions.

title: Femtocaching and device-to-device collaboration: A new architecture for wireless video distribution
abstract: We present a new architecture to handle the ongoing explosive increase in the demand for video content in wireless networks. It is based on distributed caching of the content in femtobasestations with small or non-existing backhaul capacity but with considerable storage space, called helper nodes. We also consider using the wireless terminals themselves as caching helpers, which can distribute video through device-todevice communications. This approach allows an improvement in the video throughput without deployment of any additional infrastructure. The new architecture can improve video throughput by one to two orders-of-magnitude.

title: Exploiting user mobility for wireless content delivery
abstract: We consider the problem of storing segments of encoded versions of content files in a set of base stations located in a communication cell. These base stations work in conjunction with the main base station of the cell. Users move randomly across the space based on a discrete-time Markov chain model. At each time slot each user accesses a single base station based on it's current position and it can download only a part of the content stored in it, depending on the time slot duration. We assume that file requests must be satisfied within a given time deadline in order to be successful. If the amount of the downloaded (encoded) data by the accessed base stations when the time deadline expires does not suffice to recover the requested file, the main base station of the cell serves the request. Our aim is to find the storage allocation that minimizes the probability of using the main base station for file delivery. This problem is intractable in general. However, we show that the optimal solution of the problem can be efficiently attained in case that the time deadline is small. To tackle the general case, we propose a distributed approximation algorithm based on large deviation inequalities. Systematic experiments on a real world data set demonstrate the effectiveness of our proposed algorithms.

title: Optimal geographic caching in cellular networks
abstract: In this work we consider the problem of an optimal geographic placement of content in wireless cellular networks modelled by Poisson point processes. Specifically, for the typical user requesting some particular content and whose popularity follows a given law (e.g. Zipf), we calculate the probability of finding the content cached in one of the base stations. Wireless coverage follows the usual signal-to-interference-and noise ratio (SINR) model, or some variants of it. We formulate and solve the problem of an optimal randomized content placement policy, to maximize the user's hit probability. The result dictates that it is not always optimal to follow the standard policy “cache the most popular content, everywhere”. In fact, our numerical results regarding three different coverage scenarios, show that the optimal policy significantly increases the chances of hit under high-coverage regime, i.e., when the probabilities of coverage by more than just one station are high enough.

title: Exploiting Base Station Caching in MIMO Cellular Networks: Opportunistic Cooperation for Video Streaming
abstract: We propose a novel MIMO cooperation framework called the cache-induced opportunistic cooperative MIMO (Coop-MIMO) for video streaming in backhaul limited multicell MIMO networks. By caching a portion of the video files, the base stations (BSs) opportunistically employ Coop-MIMO transmission to achieve MIMO cooperation gain without expensive payload backhaul. We derive closed form expressions of various video streaming performance metrics for cache-induced opportunistic Coop-MIMO and investigate the impact of BS level caching and key system parameters on the performance. Specifically, we first obtain a mixed fluid-diffusion limit for the playback buffer queueing system. Then we derive the approximated video streaming performance using the mixed fluid-diffusion limit. Based on the performance analysis, we formulate the joint optimization of cache control and playback buffer management as a stochastic optimization problem. Then we derive a closed form solution for the playback buffer thresholds and develop a stochastic subgradient algorithm to find the optimal cache control. The analysis shows that the video streaming performance improves linearly with the BS cache size BC, the transmit power cost decreases exponentially with BC, and the backhaul cost decreases linearly with BC.

title: Approximation Algorithms for Mobile Data Caching in Small Cell Networks
abstract: Small cells constitute a promising solution for managing the mobile data growth that has overwhelmed network operators. Local caching of popular content items at the small cell base stations (SBSs) has been proposed to decrease the costly transmissions from the macrocell base stations without requiring high capacity backhaul links for connecting the SBSs with the core network. However, the caching policy design is a challenging problem especially if one considers realistic parameters such as the bandwidth capacity constraints of the SBSs that can be reached in congested urban areas. We consider such a scenario and formulate the joint routing and caching problem aiming to maximize the fraction of content requests served locally by the deployed SBSs. This is an NP-hard problem and, hence, we cannot obtain an optimal solution. Thus, we present a novel reduction to a variant of the facility location problem, which allows us to exploit the rich literature of it, to establish algorithms with approximation guarantees for our problem. Although the reduction does not ensure tight enough bounds in general, extensive numerical results reveal a near-optimal performance that is even up to 38% better compared to conventional caching schemes using realistic system settings.

title: Caching at the edge: A green perspective for 5G networks
abstract: Endowed with context-awareness and proactive capabilities, caching users' content locally at the edge of the network is able to cope with increasing data traffic demand in 5G wireless networks. In this work, we focus on the energy consumption aspects of cache-enabled wireless cellular networks, specifically in terms of area power consumption (APC) and energy efficiency (EE). We assume that both base stations (BSs) and mobile users are distributed according to homogeneous Poisson point processes (PPPs) and we introduce a detailed power model that takes into account caching. We study the conditions under which the area power consumption is minimized with respect to BS transmit power, while ensuring a certain quality of service (QoS) in terms of coverage probability. Furthermore, we provide the optimal BS transmit power that maximizes the area spectral efficiency per unit total power spent. The main takeaway of this paper is that caching seems to be an energy efficient solution.

title: On the tradeoff between energy harvesting and caching in wireless networks
abstract: Self-powered, energy harvesting small cell base stations (SBS) are expected to be an integral part of next-generation wireless networks. However, due to uncertainties in harvested energy, it is necessary to adopt energy efficient power control schemes to reduce an SBSs' energy consumption and thus ensure quality-of-service (QoS) for users. Such energy-efficient design can also be done via the use of content caching which reduces the usage of the capacity-limited SBS backhaul. of popular content at SBS can also prove beneficial in this regard by reducing the backhaul usage. In this paper, an online energy efficient power control scheme is developed for an energy harvesting SBS equipped with a wireless backhaul and local storage. In our model, energy arrivals are assumed to be Poisson distributed and the popularity distribution of requested content is modeled using Zipf's law. The power control problem is formulated as a (discounted) infinite horizon dynamic programming problem and solved numerically using the value iteration algorithm. Using simulations, we provide valuable insights on the impact of energy harvesting and caching on the energy and sum-throughput performance of the SBS as the network size is varied. Our results also show that the size of cache and energy harvesting equipment at the SBS can be traded off, while still meeting the desired system performance.

title: Optimal dynamic multicast scheduling for cache-enabled content-centric wireless networks
abstract: Caching and multicasting at base stations are two promising approaches to support massive content delivery over wireless networks. However, existing scheduling designs do not make full use of the advantages of the two approaches. In this paper, we consider the optimal dynamic multicast scheduling to jointly minimize the average delay, power and fetching costs for cache-enabled content-centric wireless networks. We formulate this stochastic optimization problem as an infinite horizon average cost Markov decision process (MDP). It is well-known to be a difficult problem and there generally only exist numerical solutions. By using relative value iteration algorithm and the special structures of the request queue dynamics, we analyze the properties of the value function and the state-action cost function of the MDP for both the uniform and nonuniform channel cases. Based on these properties, we show that the optimal policy, which is adaptive to the request queue state, has a switch structure in the uniform case and a partial switch structure in the nonuniform case. Moreover, in the uniform case with two contents, we show that the switch curve is monotonically non-decreasing. The optimality properties obtained in this paper can provide design insights for practical networks.

title: Multicast-aware caching for small cell networks
abstract: The deployment of small cells is expected to gain huge momentum in the near future, as a solution for managing the skyrocketing mobile data demand growth. Local caching of popular files at the small cell base stations has been recently proposed, aiming at reducing the traffic incurred when transferring the requested content from the core network to the users. In this paper, we propose and analyze a novel caching approach that can achieve significantly lower traffic compared to the traditional caching schemes. Our cache design policy carefully takes into account the fact that an operator can serve the requests for the same file that happen at nearby times via a single multicast transmission. The latter incurs less traffic as the requested file is transmitted to the users only once, rather than with many unicast transmissions. Systematic experiments demonstrate the effectiveness of our approach, as compared to the existing caching schemes.

title: A Study of Replacement Algorithms for Virtual-Storage Computer
abstract: One of the basic limitations of a digital computer is the size of its available memory.
 1 
In most cases, it is neither feasible nor economical for a user to insist that every problem program fit into memory. The number of words of information in a program often exceeds the number of cells (i.e., word locations) in memory. The only way to solve this problem is to assign more than one program word to a cell. Since a cell can hold only one word at a time, extra words assigned to the cell must be held in external storage. Conventionally, overlay techniques are employed to exchange memory words and external-storage words whenever needed; this, of course, places an additional planning and coding burden on the programmer. For several reasons, it would be advantageous to rid the programmer of this function by providing him with a “virtual” memory larger than his program. An approach that permits him to use a sufficiently large address range can accomplish this objective, assuming that means are provided for automatic execution of the memory-overlay functions.

title: Distributed caching algorithms for content distribution networks
abstract: The rapid growth of wireless content access implies the need for content placement and scheduling at wireless base stations. We study a system under which users are divided into clusters based on their channel conditions, and their requests are represented by different queues at logical front ends. Requests might be elastic (implying no hard delay constraint) or inelastic (requiring that a delay target be met). Correspondingly, we have request queues that indicate the number of elastic requests, and de ficit queues that indicate the de ficit in inelastic service. Caches are of finite size and can be refreshed periodically from a media vault. We consider two cost models that correspond to inelastic requests for streaming stored content and real -time streaming of events, respectively. We design provably optimal policies that stabilize the request queues(hence ensuring finite delays)and reduce average deficit to zero [hence ensuring that the quality -of-service (QoS) target is met]at small cost. We illustrate our approach through simulations.

title: Cache in the air: exploiting content caching and delivery techniques for 5G systems
abstract: The demand for rich multimedia services over mobile networks has been soaring at a tremendous pace over recent years. However, due to the centralized architecture of current cellular networks, the wireless link capacity as well as the bandwidth of the radio access networks and the backhaul network cannot practically cope with the explosive growth in mobile traffic. Recently, we have observed the emergence of promising mobile content caching and delivery techniques, by which popular contents are cached in the intermediate servers (or middleboxes, gateways, or routers) so that demands from users for the same content can be accommodated easily without duplicate transmissions from remote servers; hence, redundant traffic can be significantly eliminated. In this article, we first study techniques related to caching in current mobile networks, and discuss potential techniques for caching in 5G mobile networks, including evolved packet core network caching and radio access network caching. A novel edge caching scheme based on the concept of content-centric networking or information-centric networking is proposed. Using trace-driven simulations, we evaluate the performance of the proposed scheme and validate the various advantages of the utilization of caching content in 5G mobile networks. Furthermore, we conclude the article by exploring new relevant opportunities and challenges.

title: Cost-aware caching: Optimizing cache provisioning and object placement in ICN
abstract: Caching is frequently used by Internet Service Providers as a viable technique to reduce the latency perceived by end users, while jointly offloading network traffic. While the cache hit-ratio is generally considered in the literature as the dominant performance metric for such type of systems, in this paper we argue that a critical missing piece has so far been neglected. Adopting a radically different perspective, in this paper we explicitly account for the cost of content retrieval, i.e. the cost associated to the external bandwidth needed by an ISP to retrieve the contents requested by its customers. Interestingly, we discover that classical cache provisioning techniques that maximize cache efficiency (i.e., the hit-ratio), lead to suboptimal solutions with higher overall cost. To show this mismatch, we propose two optimization models that either minimize the overall costs or maximize the hit-ratio, jointly providing cache sizing, object placement and path selection. We formulate a polynomial-time greedy algorithm to solve the two problems and analytically prove its optimality. We provide numerical results and show that significant cost savings are attainable via a cost-aware design.

title: Living on the edge: The role of proactive caching in 5G wireless networks
abstract: This article explores one of the key enablers of beyond 4G wireless networks leveraging small cell network deployments, proactive caching. Endowed with predictive capabilities and harnessing recent developments in storage, context awareness, and social networks, peak traffic demands can be substantially reduced by proactively serving predictable user demands via caching at base stations and users' devices. In order to show the effectiveness of proactive caching, we examine two case studies that exploit the spatial and social structure of the network, where proactive caching plays a crucial role. First, in order to alleviate backhaul congestion, we propose a mechanism whereby files are proactively cached during off-peak periods based on file popularity and correlations among user and file patterns. Second, leveraging social networks and D2D communications, we propose a procedure that exploits the social structure of the network by predicting the set of influential users to (proactively) cache strategic contents and disseminate them to their social ties via D2D communications. Exploiting this proactive caching paradigm, numerical results show that important gains can be obtained for each case study, with backhaul savings and a higher ratio of satisfied users of up to 22 and 26 percent, respectively. Higher gains can be further obtained by increasing the storage capability at the network edge.

title: Fair resource allocation for the relay backhaul link in LTE-Advanced
abstract: In this work, we revisit the issue of fair resource allocation in relay-enhanced wireless networks. Our focus this time is on Type-1a relays as proposed for LTE-Advanced. The latter operate as out-band relays, i.e., the backhaul and relay access link use separate carrier frequencies. If carrier aggregation is applied at the macro base station, the backhaul carrier may also contain part of the macro access link. Assuming full buffer traffic on the downstream, we demonstrate how similar resource partitioning strategies at the base station as proposed for in-band relays can be also applied in the out-band case. Furthermore, we propose that for out-band relays, the backhaul link should be considered directly in the regular frequency-selective scheduling process for best performance vs. complexity trade-off. The presented results include the resource consumption and achievable throughput for a hot-spot scenario with 2 out-band relay nodes, as well as a comparison to the in-band case assuming same overall resource budget.

title: Backhaul resource allocation in LTE-Advanced relaying systems
abstract: In 3GPP LTE-Advanced, relay is considered as an important means for extending cell coverage and improving capacity. Backhaul resource allocation needs to be carefully designed to achieve the performance advantage of relay. In this paper, we propose a two-step backhaul resource allocation scheme for an inband relaying system based on relay buffer level. The resource assignment to relays meets the actual traffic demand, takes into account the link quality of the backhaul link and the access link, and the resource partitioning between relays and macro UEs is then optimized. Our evaluations show that the proposed framework can improve the cell average and cell-edge user performance, for different deployment scenarios. This approach has very low complexity in implementation and the revision to current standard is very minimal.

title: Minority Games: Interacting agents in financial markets
abstract: The Minority Game is a physicist's attempt to explain market behaviour by the interaction between traders. With a minimal set of ingredients and drastic assumptions, this model reproduces market ecology among different types of traders. Its emphasis is on speculative trading and information flow. The book first describes the philosophy lying behind the conception of the Minority Game in 1997, and includes in particular a discussion about the El Farol bar problem. It then reviews the main steps in later developments, including both the theory and its applications to market phenomena. 'Minority Games' gives a colourful and stylized, but also realistic picture of how financial markets operate.

title: The Minority Game: an introductory guide
abstract: The Minority Game is a simple model for the collective behavior of agents in an idealized situation where they have to compete through adaptation for a finite resource. This review summarizes the statistical mechanics community efforts to clear up and understand the behavior of this model. Our emphasis is on trying to derive the underlying effective equations which govern the dynamics of the original Minority Game, and on making an interpretation of the results from the point of view of the statistical mechanics of disordered systems.

title: Games with randomly disturbed payoffs: A new rationale for mixed-strategy equilibrium points
abstract: Equilibrium points in mixed strategies seem to be unstable, because any player can deviate without penalty from his equilibrium strategy even if he expects all other players to stick to theirs. This paper proposes a model under which most mixed-strategy equilibrium points have full stability. It is argued that for any gameΓ the players' uncertainty about the other players' exact payoffs can be modeled as a disturbed gameΓ*, i.e., as a game with small random fluctuations in the payoffs. Any equilibrium point inΓ, whether it is in pure or in mixed strategies, can “almost always” be obtained as a limit of a pure-strategy equilibrium point in the corresponding disturbed gameΓ* when all disturbances go to zero. Accordingly, mixed-strategy equilibrium points are stable — even though the players may make no deliberate effort to use their pure strategies with the probability weights prescribed by their mixed equilibrium strategies — because the random fluctuations in their payoffs willmake them use their pure strategies approximately with the prescribed probabilities.

title: Game Theory and Learning for Wireless Networks: Fundamentals and Applications
abstract: * The first tutorial-style book that gives all the relevant theory at the right level of rigor, for the wireless communications engineer.  Bridges the gap between theory and practice by giving examples and case studies showing how game theory can solve real-word problems.  Contains algorithms and techniques to implement game theory in wireless terminals. Written by leading experts in the field, Game Theory and Learning for Wireless Networks Covers how theory can be used to solve prevalent problems in wireless networks such as power control, resource allocation or medium access control. With the emphasis now on promoting 'green' solutions in the wireless field where power consumption is minimized, there is an added focus on developing network solutions that maximizes the use of the spectrum available. With the growth of distributed wireless networks such as Wi-Fi and the Internet; the push to develop ad hoc and cognitive networks has led to a considerable interest in applying game theory to wireless communication systems. Game Theory and Learning for Wireless Networks is the first comprehensive resource of its kind, and is ideal for wireless communications R&D engineers and graduate students. Samson Lasaulce is a senior CNRS researcher at the Laboratory of Signals and Systems (LSS) at Suplec, Gif-sur-Yvette, France. He is also a part-time professor in the Department of Physics at cole Polytechnique, Palaiseau, France. Hamidou Tembine is a professor in the Department of Telecommunications at Suplec, Gif-sur-Yvette, France. Merouane Debbah is a professor at Suplec, Gif-sur-Yvette, France. He is the holder of the Alcatel-Lucent chair in flexible radio since 2007. The first tutorial style book that gives all the relevant theory, at the right level of rigour, for the wireless communications engineer Bridges the gap between theory and practice by giving examples and case studies showing how game theory can solve real world resource allocation problems Contains algorithms and techniques to implement game theory in wireless terminals

title: Self-Organization in Small Cell Networks: A Reinforcement Learning Approach
abstract: In this paper, a decentralized and self-organizing mechanism for small cell networks (such as micro-, femto- and picocells) is proposed. In particular, an application to the case in which small cell networks aim to mitigate the interference caused to the macrocell network, while maximizing their own spectral efficiencies, is presented. The proposed mechanism is based on new notions of reinforcement learning (RL) through which small cells jointly estimate their time-average performance and optimize their probability distributions with which they judiciously choose their transmit configurations. Here, a minimum signal to interference plus noise ratio (SINR) is guaranteed at the macrocell user equipment (UE), while the small cells maximize their individual performances. The proposed RL procedure is fully distributed as every small cell base station requires only an observation of its instantaneous performance which can be obtained from its UE. Furthermore, it is shown that the proposed mechanism always converges to an epsilon Nash equilibrium when all small cells share the same interest. In addition, this mechanism is shown to possess better convergence properties and incur less overhead than existing techniques such as best response dynamics, fictitious play or classical RL. Finally, numerical results are given to validate the theoretical findings, highlighting the inherent tradeoffs facing small cells, namely exploration/exploitation, myopic/foresighted behavior and complete/incomplete information.

title: Convergent multiple-timescales reinforcement learning algorithms in normal form games
abstract: We consider reinforcement learning algorithms in normal form games. Using two-timescales stochastic approximation, we introduce a model-free algorithm which is asymptotically equivalent to the smooth fictitious play algorithm, in that both result in asymptotic pseudotrajectories to the flow defined by the smooth best response dynamics. Both of these algorithms are shown to converge almost surely to Nash distribution in two-player zero-sum games and N -player partnership games. However, there are simple games for which these, and most other adaptive processes, fail to converge--in particular, we consider the N -player matching pennies game and Shapley's variant of the rock--scissors--paper game. By extending stochastic approximation results to multiple timescales we can allow each player to learn at a different rate. We show that this extension will converge for two-player zero-sum games and two-player partnership games, as well as for the two special cases we consider.

title: Learning equilibria with partial information in decentralized wireless networks
abstract: In this article, a survey of several important equilibrium concepts for decentralized networks is presented. The term decentralized is used here to refer to scenarios where decisions (e.g., choosing a power allocation policy) are taken autonomously by devices interacting with each other (e.g., through mutual interference). The iterative long-term interaction is characterized by stable points of the wireless network called equilibria. The interest in these equilibria stems from the relevance of network stability and the fact that they can be achieved by letting radio devices to repeatedly interact over time. To achieve these equilibria, several learning techniques - the best response dynamics, fictitious play, smoothed fictitious play, reinforcement learning algorithms, and regret matching - are discussed in terms of information requirements and convergence properties. Most of the notions introduced here, for both equilibria and learning schemes, are illustrated by a simple case study, an interference channel with two transmitter-receiver pairs.

title: Strategic Learning and Its Limits
abstract: In this concise book based on his Arne Ryde Lectures in 2002, Young suggests a conceptual framework for studying strategic learning and highlights theoretical developments in the area. He discusses the interactive learning problem; reinforcement and regret; equilibrium; conditional no-regret learning; prediction, postdiction, and calibration; fictitious play and its variants; Bayesian learning; and hypothesis testing. Young's framework emphasizes the amount of information required to implement different types of learning rules, criteria for evaluating their performance, and alternative notions of equilibrium to which they converge. He also stresses the limits of what can be achieved: for a given type of game and a given amount of information, there may exist no learning procedure that satisfies certain reasonable criteria of performance and convergence. In short, Young has provided a valuable primer that delineates what we know, what we would like to know, and the limits of what we can know, when we try to learn about a system that is composed of other learners.