# Joint Constellation Shaping Using Gradient Descent Approach for MU-MIMO Broadcast Channel

Maxime Vaillant, Alix Jeannerot, Jean-Marie Gorce

► To cite this version:

Maxime Vaillant, Alix Jeannerot, Jean-Marie Gorce. Joint Constellation Shaping Using Gradient Descent Approach for MU-MIMO Broadcast Channel. 2024. hal-04590871

HAL Id: hal-04590871

https://inria.hal.science/hal-04590871

Preprint submitted on 30 May 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Joint Constellation Shaping Using Gradient Descent Approach for MU-MIMO Broadcast Channel

Maxime Vaillant, Alix Jeannerot, Jean-Marie Gorce

► To cite this version:

Maxime Vaillant, Alix Jeannerot, Jean-Marie Gorce. Joint Constellation Shaping Using Gradient Descent Approach for MU-MIMO Broadcast Channel. 2024 IEEE 25th International Workshop on Signal Processing Advances in Wireless Communications (SPAWC), Sep 2024, Lucca, Italy. pp.56-60, 10.1109/SPAWC60668.2024.10694191. [https://doi.org/10.1109/SPAWC60668.2024.10694191](https://doi.org/10.1109/SPAWC60668.2024.10694191)

HAL Id: hal-04590871

https://inria.hal.science/hal-04590871v2

Submitted on 20 Aug 2024

**HAL** is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire **HAL**, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Joint Constellation Shaping Using Gradient Descent Approach for MU-MIMO Broadcast Channel

Maxime Vaillant

Inria, INSA Lyon, CITI, UR3720
69621 Villeurbanne, France

maxime.vaillant@inria.fr

Alix Jeannerot

INSA Lyon, Inria, CITI, UR3720
69621 Villeurbanne, France

alix.jeannerot@inria.fr

Jean-Marie Gorce

INSA Lyon, Inria, CITI, UR3720
69621 Villeurbanne, France

jean-marie.gorce@insa-lyon.fr

***Abstract***—In this paper, we introduce a learning-based approach to optimize a joint constellation for a multi-user MIMO broadcast channel ($T$ Tx antennas, $K$ users, each with $R$ Rx antennas), with perfect channel knowledge. The aim of the optimizer (MAX-MIN) is to maximize the minimum mutual information between the transmitter and each receiver, under a sum-power constraint. The proposed optimization method do neither impose the transmitter to use superposition coding (SC) or any other linear precoding, nor to use successive interference cancellation (SIC) at the receiver. Instead, the approach designs a joint constellation, optimized such that its projection into the subspace of each receiver $k$, maximizes the minimum mutual information $I(W_k; Y_k)$ between each transmitted binary input $W_k$ and the output signal at the intended receiver $Y_k$. The rates obtained by our method are compared to those achieved with linear precoders.

***Index Terms***—Non Degraded Broadcast Channel, Joint Constellation Design, Stochastic Gradient Descent, MU-MIMO

## I. INTRODUCTION

Power-domain non-orthogonal multiple access (PD-NOMA) is a promising technique relying on superposition coding (SC) to increase the capacity of cellular networks. When a base station (BS) has a single antenna, the corresponding channel is known as the degraded broadcast channel, and SC is capacity achieving [1]. When the BS has multiple antennas, the corresponding broadcast channel model (known as the MU-MIMO) is non degraded and the capacity region is not given by a single letter expression [2]. Dirty paper coding (DPC) with time-sharing is the only known method allowing to achieve the capacity limits but its complexity is prohibitive [2]. It is worth mentioning that PD-NOMA has been adapted to MU-MIMO, with sub-optimal techniques such as using predefined beams or successive zero-forcing encoders [3], [4], [5], [6]. In [5] PD-NOMA is compared to standard Multi User Linear Precoding (MU-LP), e.g. zero-forcing, and it is shown that in some conditions, they have in practice similar performance. Beyond PD-NOMA, rate splitting multiple access (RSMA) overcomes some of the limitations of PD-NOMA [7]. With RSMA messages are split between private and common parts. The private part is decoded by the intended receiver only, while the common part is decoded by everyone. This promising technique investigated for 6G allows to better exploit the degraded broadcast channel but its complexity is high.

An alternative to leverage on the full capacity region of the degraded broadcast channel, is to design a joint con-

stellation. Indeed, in the aforementioned techniques (PD-NOMA or RSMA), each stream is assumed to span a whole subspace, generating continuous interference to each other. However, for small constellation sizes, the input signals do not span a continuous space but use discrete positions only. Such consideration has already been considered to optimize structured codes in multi-user settings (see for instance [8]). If constellation design for the broadcast channel has been studied in several papers (e.g. [9], [10], [11]), these works designed superposed constellations (instead of joint constellation) at the transmitter [9] and assumed that the use of successive interference cancellation (SIC) at the receiver.

Our work differs from two aspects. First, we do not consider a linear combination of per-user constellations, but we optimize a joint constellation for the multiple users. Second, we do not assume any specific decoder, but we instead maximize pairwise mutual information. More precisely, the contributions of this paper follow:

1. In contrast to the commonly used MU-LP, a joint constellation is designed for simultaneously active users.

2. Such a constellation maximizes the minimal mutual information between the BS and each user, ensuring a better fairness compared to the state of the art method.

3. The optimization is agnostic to the decoding technique and maximizes the mutual information.

4. Numerical simulations are provided supporting the benefit of our method, compared to standard MU-LP methods in the case where the number of receive antennas is 1.

Our paper is organized as follows. Section II describes the system model and associated assumptions, while Section III introduces the joint constellation design problem. A stochastic gradient based algorithm is proposed in Section IV, allowing to find an (at least locally) optimal constellation. The proposed algorithm is evaluated in Section V on small simulated scenarios.

## II. SYSTEM MODEL

### A. Channel model

A MU-MIMO broadcast channel with a BS equipped with $T$ transmit antennas aiming to transmit simultaneously data symbols to $K$ users, each equipped with $R$ receive antennas. We consider a mapping problem in a one channel use. Let $W_k$
# Joint Constellation Shaping Using Gradient Descent Approach for MU-MIMO Broadcast Channel

Maxime Vaillant

Inria, INSA Lyon, CITI, UR3720
69621 Villeurbanne, France

maxime.vaillant@inria.fr

Alix Jeannerot

INSA Lyon, Inria, CITI, UR3720
69621 Villeurbanne, France

alix.jeannerot@inria.fr

Jean-Marie Gorce

INSA Lyon, Inria, CITI, UR3720
69621 Villeurbanne, France

jean-marie.gorce@insa-lyon.fr

***Abstract***—In this paper, we introduce a learning-based approach to optimize a joint constellation for a multi-user MIMO broadcast channel ($T$ Tx antennas, $K$ users, each with $R$ Rx antennas), with perfect channel knowledge. The aim of the optimizer (MAX-MIN) is to maximize the minimum mutual information between the transmitter and each receiver, under a sum-power constraint. The proposed optimization method do neither impose the transmitter to use superposition coding (SC) or any other linear precoding, nor to use successive interference cancellation (SIC) at the receiver. Instead, the approach designs a joint constellation, optimized such that its projection into the subspace of each receiver $k$, maximizes the minimum mutual information $I(W_k; Y_k)$ between each transmitted binary input $W_k$ and the output signal at the intended receiver $Y_k$. The rates obtained by our method are compared to those achieved with linear precoders.

***Index Terms***—Non Degraded Broadcast Channel, Joint Constellation Design, Stochastic Gradient Descent, MU-MIMO

## I. INTRODUCTION

Power-domain non-orthogonal multiple access (PD-NOMA) is a promising technique relying on superposition coding (SC) to increase the capacity of cellular networks. When a base station (BS) has a single antenna, the corresponding channel is known as the degraded broadcast channel, and SC is capacity achieving [1]. When the BS has multiple antennas, the corresponding broadcast channel model (known as the MU-MIMO) is non degraded and the capacity region is not given by a single letter expression [2]. Dirty paper coding (DPC) with time-sharing is the only known method allowing to achieve the capacity limits but its complexity is prohibitive [2]. It is worth mentioning that PD-NOMA has been adapted to MU-MIMO, with sub-optimal techniques such as using predefined beams or successive zero-forcing encoders [3], [4], [5], [6]. In [5] PD-NOMA is compared to standard Multi User Linear Precoding (MU-LP), e.g. zero-forcing, and it is shown that in some conditions, they have in practice similar performance. Beyond PD-NOMA, rate splitting multiple access (RSMA) overcomes some of the limitations of PD-NOMA [7]. With RSMA messages are split between private and common parts. The private part is decoded by the intended receiver only, while the common part is decoded by everyone. This promising technique investigated for 6G allows to better exploit the degraded broadcast channel but its complexity is high.

An alternative to leverage on the full capacity region of the degraded broadcast channel, is to design a joint con-

stellation. Indeed, in the aforementioned techniques (PD-NOMA or RSMA), each stream is assumed to span a whole subspace, generating continuous interference to each other. However, for small constellation sizes, the input signals do not span a continuous space but use discrete positions only. Such consideration has already been considered to optimize structured codes in multi-user settings (see for instance [8]). If constellation design for the broadcast channel has been studied in several papers (e.g. [9], [10], [11]), these works designed superposed constellations (instead of joint constellation) at the transmitter [9] and assumed that the use of successive interference cancellation (SIC) at the receiver.

Our work differs from two aspects. First, we do not consider a linear combination of per-user constellations, but we optimize a joint constellation for the multiple users. Second, we do not assume any specific decoder, but we instead maximize pairwise mutual information. More precisely, the contributions of this paper follow:

1. In contrast to the commonly used MU-LP, a joint constellation is designed for simultaneously active users.

2. Such a constellation maximizes the minimal mutual information between the BS and each user, ensuring a better fairness compared to the state of the art method.

3. The optimization is agnostic to the decoding technique and maximizes the mutual information.

4. Numerical simulations are provided supporting the benefit of our method, compared to standard MU-LP methods in the case where the number of receive antennas is 1.

Our paper is organized as follows. Section II describes the system model and associated assumptions, while Section III introduces the joint constellation design problem. A stochastic gradient based algorithm is proposed in Section IV, allowing to find an (at least locally) optimal constellation. The proposed algorithm is evaluated in Section V on small simulated scenarios.

## II. SYSTEM MODEL

### A. Channel model

A MU-MIMO broadcast channel with a BS equipped with $T$ transmit antennas aiming to transmit simultaneously data symbols to $K$ users, each equipped with $R$ receive antennas. We consider a mapping problem in a one channel use. Let $W_k$
be the random variable (r.v.) representing the $k^{th}$ user symbol, taking values $w_k \in W_k$, where $W_k$ is the codebook of user $k$. $W = [W_1...W_K]$ refers to the random vector aggregating all input r.v. The symbols $W_k$ are drawn independently and uniformly at random: $Pr(W = w) = \prod_{k=1}^K \frac{1}{|W_k|}$. When necessary, we will use the notation $w_k^n$ to denote the n-th realization of the random variable $W_k$. However, for the sake of clarity, the subscript $n$ is omitted when not explicitly required. For readability purpose we denote $\sum_w$ as the summation over $w \in W_1 \times \dots \times W_k$.

The BS implements a deterministic encoder :

$$ \phi: \mathcal{W} = W_1 \times \dots \times W_K \rightarrow \mathbb{C}^T, \quad (1) $$

which, maps any $w$ to a vector $x = \phi(w)$ of length $T$. The set of all possible vectors $x$ denoted as $\mathcal{X}$, constitutes the joint constellation. We denote by $\mathcal{X}_k(w_k)$ the subset of vectors that carry the specific message $w_k$ for user $k$.

The vector $x$ is transmitted over the $T$ transmit antennas to the users. The channel between the $T$ transmit antennas and the $R$ receive antennas of user $k$ is a matrix denoted $H_k$. This matrix is rewritten as: $H_k = \sqrt{g_k} \cdot \zeta_k$ where $g_k$ is the square Frobenius norm of the channel matrix and $\zeta_k \in \mathbb{C}^{T \times R}$ is the normalized channel matrix. Each user observes a projection $y_k \in \mathbb{C}^R$ of the input vector $x$, passing through the channel:

$$ y_k = \zeta_k^H \cdot x + \nu_k, \quad (2) $$

with $\nu_k \sim \mathcal{CN}(0, \sigma_k^2 \cdot I_R)$ with $\sigma_k^2 = \sigma^2/g_k$ and $I_R$ the $R \times R$ identity matrix. Additionally, by defining $\zeta = [\zeta_1 \dots \zeta_K] \in \mathbb{C}^{T \times (K \cdot R)}$ and $y \in \mathbb{C}^{K \cdot R}$, the whole system is described as:

$$ y = \zeta^H \cdot x + n, \quad (3) $$

with $n = [\nu_{1,1} \dots \nu_{1,R} \dots \nu_{K,1} \dots \nu_{K,R}] \in \mathbb{C}^{K \cdot R}$, and $y = [y_{1,1} \dots y_{1,R} \dots y_{K,1} \dots y_{K,R}]$. Throughout this paper, we assume that the base station has perfect knowledge of the channel (perfect CSIT).

## B. Linear methods (MU-LP)

MU-LP are based on the design of independent constellations with perfect CSIT. The encoding function defined in (1) is split in two parts:

(i) A set of mapping functions of the form: $\phi^{(map)}(w) = [\phi^{(map)}(w_1), \dots, \phi^{(map)}(w_K)]$. For instance the BPSK mapping function is $\phi^{(map)}(0) = 1$ and $\phi^{(map)}(1) = -1$.

(ii) An encoding matrix matched to the channel.

The encoding function $\phi$ linking symbols $w$ to constellation points $x$ is thus given by:

$$ \phi(w) = M \cdot \phi^{(map)}(w), \quad (4) $$

where $M$ is the encoding matrix. To ensure power constraints for these methods, we project the resulting constellation onto a constraint space that will be further defined. To ease the description of these methods, we consider in the remaining of this section that $R = 1$.

1) Matched filter: The matched filter does not consider interference and maximize the signal-to-noise-ratio (SNR). The corresponding encoding matrix is:

$$ M_m = \zeta. \quad (5) $$

2) Zero-forcing filter: The zero-forcing (ZF) encoder aims to cancel out all interference caused by the channel using the Moore-Penrose pseudo inverse. Its encoding matrix is:

$$ M_{zf} = \zeta \cdot (\zeta^H \cdot \zeta)^{-1}. \quad (6) $$

The ZF encoder leads to interference free signals at the receivers, but may require in turn a high power at the transmitter especially when the channel vectors are close to each other.

3) MMSE (Wiener) filter: The minimum mean square error (MMSE) filtering consists of minimizing the noise in reception. As indicated in [12], the filter that maximizes the SNR is given by:

$$ M_{mmse} = \zeta \cdot (\zeta^H \cdot \zeta + \Sigma)^{-1}, \quad \Sigma = \text{diag}(\sigma_i^2) \quad (7) $$

## III. JOINT CONSTELLATION DESIGN

In contrast to the MU-LP methods aforementioned, this paper explores the design of a joint constellation that directly maps any input vector $w$ to a constellation point $X(w)$ according to (1). The proposed mapping function is designed to maximize pairwise mutual information as described now.

### A. MAX-MIN encoder

The proposed MAX-MIN encoder aims at maximizing the minimal mutual information between the BS and each user, but the approach could be easily generalized to any other combination of individual losses. Given a cross-entropy loss $\mathcal{L}_k$ (details are provided later), the MAX-MIN optimal solution is a set of constellation points $\mathcal{X} = \{X(w) = \phi(w); \forall w\}$ that verify:

$$ \begin{array}{ll} \underset{\mathcal{X}}{\arg\min} & \max_k (\mathcal{L}_k(\mathcal{X})) \\ \text{s.t.} & \frac{\sum_w ||X(w)||^2}{|\mathcal{W}|} \le P_m, \\ & ||x_t||^2 \le P_c \quad \forall t \in T, \quad \forall x \in \mathcal{X}, \end{array} \quad (8) $$

where the first constraint stands for an average power constraint $P_m$ and the second constraint ensures that the transmit power on any antenna does not exceed a threshold $P_c$. We denote $\Pi_P$ as the constraint space.

The following sub-sections allow to define the cross-entropy loss that minimizes the mutual information and is appropriate to derive a stochastic gradient descent algorithm.

### B. Log-likelihood ratio (LLR) and mutual information

Given an observation $y_k$ and a symbol $w_k$ the log-likelihood ratio (LLR) is given by:

$$ \text{LLR}(w_k|y_k) = \ln \left( \frac{\sum_{\tilde{w}; \tilde{w}_k=w_k} f_{Y_k|\tilde{W}}(y_k|\tilde{w})}{\sum_{\tilde{w}; \tilde{w}_k \neq w_k} f_{Y_k|\tilde{W}}(y_k|\tilde{w})} \right). \quad (9) $$
be the random variable (r.v.) representing the $k^{th}$ user symbol, taking values $w_k \in W_k$, where $W_k$ is the codebook of user $k$. $W = [W_1 \dots W_K]$ refers to the random vector aggregating all input r.v. The symbols $W_k$ are drawn independently and uniformly at random: $Pr(W = w) = \prod_{k=1}^K \frac{1}{|W_k|}$. When necessary, we will use the notation $w_k^n$ to denote the n-th realization of the random variable $W_k$. However, for the sake of clarity, the subscript $n$ is omitted when not explicitly required. For readability purpose we denote $\sum_w$ as the summation over $w \in W_1 \times \dots \times W_k$.

The BS implements a deterministic encoder :

$$ \phi: \mathcal{W} = W_1 \times \dots \times W_K \rightarrow \mathbb{C}^T, \quad (1) $$

which, maps any $w$ to a vector $x = \phi(w)$ of length $T$. The set of all possible vectors $x$ denoted as $\mathcal{X}$, constitutes the joint constellation. We denote by $\mathcal{X}_k(w_k)$ the subset of vectors that carry the specific message $w_k$ for user $k$.

The vector $x$ is transmitted over the $T$ transmit antennas to the users. The channel between the $T$ transmit antennas and the $R$ receive antennas of user $k$ is a matrix denoted $H_k$. This matrix is rewritten as: $H_k = \sqrt{g_k} \cdot \zeta_k$ where $g_k$ is the square Frobenius norm of the channel matrix and $\zeta_k \in \mathbb{C}^{T \times R}$ is the normalized channel matrix. Each user observes a projection $y_k \in \mathbb{C}^R$ of the input vector $x$, passing through the channel:

$$ y_k = \zeta_k^H \cdot x + \nu_k, \quad (2) $$

with $\nu_k \sim \mathcal{CN}(0, \sigma_k^2 \cdot I_R)$ with $\sigma_k^2 = \sigma^2/g_k$ and $I_R$ the $R \times R$ identity matrix. Additionally, by defining $\zeta = [\zeta_1 \dots \zeta_K] \in \mathbb{C}^{T \times (K \cdot R)}$ and $y \in \mathbb{C}^{K \cdot R}$, the whole system is described as:

$$ y = \zeta^H \cdot x + n, \quad (3) $$

with $n = [\nu_{1,1} \dots \nu_{1,R} \dots \nu_{K,1} \dots \nu_{K,R}] \in \mathbb{C}^{K \cdot R}$, and $y = [y_{1,1} \dots y_{1,R} \dots y_{K,1} \dots y_{K,R}]$. Throughout this paper, we assume that the base station has perfect knowledge of the channel (perfect CSIT).

## B. Linear methods (MU-LP)

MU-LP are based on the design of independent constellations with perfect CSIT. The encoding function defined in (1) is split in two parts:

(i) A set of mapping functions of the form: $\phi^{(map)}(w) = [\phi^{(map)}(w_1), \dots, \phi^{(map)}(w_K)]$. For instance the BPSK mapping function is $\phi^{(map)}(0) = 1$ and $\phi^{(map)}(1) = -1$.

(ii) An encoding matrix matched to the channel.

The encoding function $\phi$ linking symbols $w$ to constellation points $x$ is thus given by:

$$ \phi(w) = M \cdot \phi^{(map)}(w), \quad (4) $$

where $M$ is the encoding matrix. To ensure power constraints for these methods, we project the resulting constellation onto a constraint space that will be further defined. To ease the description of these methods, we consider in the remaining of this section that $R = 1$.

1) Matched filter: The matched filter does not consider interference and maximize the signal-to-noise-ratio (SNR). The corresponding encoding matrix is:

$$ M_m = \zeta. \quad (5) $$

2) Zero-forcing filter: The zero-forcing (ZF) encoder aims to cancel out all interference caused by the channel using the Moore-Penrose pseudo inverse. Its encoding matrix is:

$$ M_{zf} = \zeta \cdot (\zeta^H \cdot \zeta)^{-1}. \quad (6) $$

The ZF encoder leads to interference free signals at the receivers, but may require in turn a high power at the transmitter especially when the channel vectors are close to each other.

3) MMSE (Wiener) filter: The minimum mean square error (MMSE) filtering consists of minimizing the noise in reception. As indicated in [12], the filter that maximizes the SNR is given by:

$$ M_{mmse} = \zeta \cdot (\zeta^H \cdot \zeta + \Sigma)^{-1}, \quad \Sigma = \text{diag}(\sigma_i^2) \quad (7) $$

## III. JOINT CONSTELLATION DESIGN

In contrast to the MU-LP methods aforementioned, this paper explores the design of a joint constellation that directly maps any input vector $w$ to a constellation point $X(w)$ according to (1). The proposed mapping function is designed to maximize pairwise mutual information as described now.

### A. MAX-MIN encoder

The proposed MAX-MIN encoder aims at maximizing the minimal mutual information between the BS and each user, but the approach could be easily generalized to any other combination of individual losses. Given a cross-entropy loss $\mathcal{L}_k$ (details are provided later), the MAX-MIN optimal solution is a set of constellation points $\mathcal{X} = \{X(w) = \phi(w); \forall w\}$ that verify:

$$ \begin{array}{ll} \underset{\mathcal{X}}{\arg \min} & \max_k (\mathcal{L}_k(\mathcal{X})) \\ \text{s.t.} & \frac{\sum_w ||X(w)||^2}{|\mathcal{W}|} \le P_m, \quad (8) \\ & ||x_t||^2 \le P_c \quad \forall t \in T, \quad \forall x \in \mathcal{X}, \end{array} $$

where the first constraint stands for an average power constraint $P_m$ and the second constraint ensures that the transmit power on any antenna does not exceed a threshold $P_c$. We denote $\Pi_P$ as the constraint space.

The following sub-sections allow to define the cross-entropy loss that minimizes the mutual information and is appropriate to derive a stochastic gradient descent algorithm.

### B. Log-likelihood ratio (LLR) and mutual information

Given an observation $y_k$ and a symbol $w_k$ the log-likelihood ratio (LLR) is given by:

$$ \text{LLR}(w_k|y_k) = \ln \left( \frac{\sum_{\tilde{w}; \tilde{w}_k = w_k} f_{Y_k|\tilde{W}}(y_k|\tilde{w})}{\sum_{\tilde{w}; \tilde{w}_k \neq w_k} f_{Y_k|\tilde{W}}(y_k|\tilde{w})} \right). \quad (9) $$
This LLR is relevant in communication systems as it is the usual input for soft-decision based decoders. Given the prior distribution, the LLR is proportional to the Maximum A Posteriori (MAP) ratio, noted:

$$LLR_0(w_k|\mathbf{y}_k) = \ln \left( \frac{P_{W_k|\mathbf{Y}_k}(w_k|\mathbf{y}_k)}{\sum_{w_i \neq w_k} P_{W_k|\mathbf{Y}_k}(w_i|\mathbf{y}_k)} \right). \quad (10)$$

It is worth noting that the posterior probability can be obtained from the $LLR_0$ with the sigmoid function $\text{sigmoid}(x) = \frac{1}{1+e^{-x}}$, with:

$$P_{W_k|\mathbf{y}_k}(w_k|\mathbf{y}_k) = \text{sigmoid}(LLR_0(w_k|\mathbf{y}_k)). \quad (11)$$

Reminding that the mutual information between $\mathbf{Y}_k$ and $W_k$ is:

$$I(W_k; \mathbf{Y}_k) = H(W_k) - H(W_k|\mathbf{Y}_k), \quad (12)$$

where $H(W_k) = \log_2(|W_k|)$ is constant in our setup since the input distribution is fixed. Further the equivocation is given by:

$$H(W_k|\mathbf{Y}_k) = \mathbb{E}_{\mathbf{Y}_k W_k} [-\log_2(P_{W_k|\mathbf{Y}_k}(w_k|\mathbf{y}_k))], \quad (13)$$

And from (13), the equivocation depends directly on the average $LLR_0$ defined in (10):

$$H(W_k|\mathbf{Y}_k) = \mathbb{E}_{\mathbf{Y}_k W_k} [-\log_2(\text{sigmoid}(LLR_0(w_k|\mathbf{y}_k)))]. \quad (14)$$

This let us to conclude that maximizing mutual information under uniform input distributions relies on maximizing the averaged $LLR_0$, used below to define the loss function.

### C. Cross-entropy loss

In the context of machine learning, cross-entropy is used to optimize the parameters of a model. It compares the labels of the training data to the decisions made at the output of a system. We propose to optimize the constellation points by evaluating the quality of simulated received data with their respective posterior probability distributions. The cross-entropy loss associated to a batch of $N$ simulations, is expressed as:

$$\mathcal{L}_k = -\frac{1}{N} \sum_{n=1}^{N} \sum_{w \in W_k} \mathbb{1}\{w_k^n=w\} \cdot \log_2(P_{W_k|\mathbf{Y}_k}(W_k^n=w|\mathbf{y}_k^n)) \\ \xrightarrow[N \to \infty]{\mathbb{E}} \mathbb{E}_{W_k \mathbf{Y}_k} [-\log_2(P_{W_k|\mathbf{Y}_k}(w_k|\mathbf{y}_k))] = H(W_k|\mathbf{Y}_k). \quad (15)$$

The limit when $N \to \infty$ shows that the loss $\mathcal{L}_k$ converges to the equivocation (13) which is directly linked to the mutual information (12). We cannot guarantee that a gradient descent will reach the optimal point, but this proves that asymptotically, the minimum of the Loss function corresponds to the mutual information maximum. For simplification, we introduce the notation:

$$\mathcal{L}_k(\mathcal{X}; w_k^n; \mathbf{y}_k^n) = -\log_2(\text{sigmoid}(LLR_0(\mathcal{X}; w_k^n|\mathbf{y}_k^n))), \quad (16)$$

which represents the loss on user $k$ for a given sample $n$. Using (10) and Gaussian noise statistics, we have:

$$LLR_0(\mathcal{X}; w_k^n|\mathbf{y}_k^n) = \ln \left( \frac{\sum_{\mathbf{x} \in \mathcal{X}_k(w_k^n)} e^{-d_k(\mathbf{x})}}{\sum_{\mathbf{x} \notin \mathcal{X}_k(w_k^n)} e^{-d_k(\mathbf{x})}} \right), \quad (17)$$

where $d_k(\mathbf{x}) = \frac{\|\mathbf{y}_k - \mathbf{\zeta}_k^H \cdot \mathbf{x}\|^2}{2\sigma_k^2}$ is the exponent in the Gaussian noise distribution formula. The aim of (17) is to exhibit the loss with respect to the reference and observed constellation points.

The overall loss (15) can then be expressed as a function of $\mathcal{X}$:

$$\mathcal{L}_k(\mathcal{X}) = \frac{1}{N} \sum_{n=1}^{N} \sum_{w \in W_k} \mathbb{1}\{w_k^n=w\} \cdot \mathcal{L}_k(\mathcal{X}; w_k^n; \mathbf{y}_k^n). \quad (18)$$

The function $\mathcal{L}_k(\mathcal{X})$ is differentiable in $\mathcal{X}$ which makes it suitable for optimization methods based on stochastic gradient that will lead to local stationary point as $\mathcal{L}_k(\mathcal{X})$ is non-convex in $\mathcal{X}$. However its gradient $\nabla_{\mathcal{X}}\mathcal{L}_k(\mathcal{X})$ contains a term in $e^{-LLR_0(\mathcal{X}; w_k^n|\mathbf{y}_k^n)}$ due to the sigmoid in (16) leading to numerical instability and zero gradients even for small values of $LLR_0$. As a result, we introduce a stable version of $LLR_0$ using the sigmoid function:

$$\overline{LLR}_0(\mathcal{X}; w_k^n|\mathbf{y}_k^n) = \ln \left( \frac{\sum_{\mathbf{x} \in \mathcal{X}_k(w_k^n)} e^{-\text{sigmoid}(\beta \cdot d_k(\mathbf{x}))}}{\sum_{\mathbf{x} \notin \mathcal{X}_k(w_k^n)} e^{-\text{sigmoid}(\beta \cdot d_k(\mathbf{x}))}} \right). \quad (19)$$

The sigmoid function allows to preserve the direction of the gradient while avoiding reaching 0 too fast. This ensures a consistent expansion of the constellation $\mathcal{X}$ when performing the optimization in Alg.1. The appropriate selection of $\beta$ is crucial to appropriately reflect the range of values that $d_k(\mathbf{x})$ can take. As a results, setting $\beta = \left(\frac{\sum_k \sigma_k}{K}\right)^2$ allows to adapt the sharpness of the sigmoid as a function of the different noise levels of the users.

## IV. PROPOSED ALGORITHMS

In this section, we present the algorithm employed to optimize the previously defined loss function. The optimization algorithm Adam [13], has been chosen, a variant of Stochastic Gradient Descent (SGD) with momentum. We aim to directly optimize the position of each point of the constellation without restricting it to any MU-LP structure.

**Algorithm 1** Constellation optimization algorithm using projected stochastic gradient descent

1: Initialize random $\mathcal{X}$

2: **while** not converged **do**

3: Generate $\{\mathbf{w}^1, \dots, \mathbf{w}^N\}$

4: **for** $k \in K$ **do**

5: Simulate $\{\mathbf{y}_k^1, \dots, \mathbf{y}_k^N\}$ according to (2)

6: $\mathcal{L}_k(\mathcal{X}) \leftarrow \frac{1}{N} \cdot \sum_{n=1}^{N} \mathcal{L}_k(\mathcal{X}; w_k^n; \mathbf{y}_k^n)$

7: **end for**

8: $\mathcal{L}(\mathcal{X}) \leftarrow \max(\mathcal{L}_1(\mathcal{X}), \dots, \mathcal{L}_K(\mathcal{X}))$

9: $\mathcal{X} \leftarrow \Pi_{\mathcal{P}}[\text{ADAM}(\mathcal{X}, \eta, \mathcal{L})]$

10: **end while**

The proposed iterative algorithm aims to optimize the constellation design while ensuring fairness across all users. The steps of the algorithm follow:

* Initialization (1): A random constellation $\mathcal{X}$ is chosen.
This LLR is relevant in communication systems as it is the usual input for soft-decision based decoders. Given the prior distribution, the LLR is proportional to the Maximum A Posteriori (MAP) ratio, noted:

$$LLR_0(w_k|\mathbf{y}_k) = \ln \left( \frac{P_{W_k|\mathbf{Y}_k}(w_k|\mathbf{y}_k)}{\sum_{w_i \neq w_k} P_{W_k|\mathbf{Y}_k}(w_i|\mathbf{y}_k)} \right). \quad (10)$$

It is worth noting that the posterior probability can be obtained from the $LLR_0$ with the sigmoid function $\text{sigmoid}(x) = \frac{1}{1+e^{-x}}$, with:

$$P_{W_k|\mathbf{y}_k}(w_k|\mathbf{y}_k) = \text{sigmoid}(LLR_0(w_k|\mathbf{y}_k)). \quad (11)$$

Reminding that the mutual information between $\mathbf{Y}_k$ and $W_k$ is:

$$I(W_k; \mathbf{Y}_k) = H(W_k) - H(W_k|\mathbf{Y}_k), \quad (12)$$

where $H(W_k) = \log_2(|W_k|)$ is constant in our setup since the input distribution is fixed. Further the equivocation is given by:

$$H(W_k|\mathbf{Y}_k) = \mathbb{E}_{\mathbf{Y}_k W_k} [-\log_2(P_{W_k|\mathbf{Y}_k}(w_k|\mathbf{y}_k))], \quad (13)$$

And from (13), the equivocation depends directly on the average $LLR_0$ defined in (10):

$$H(W_k|\mathbf{Y}_k) = \mathbb{E}_{\mathbf{Y}_k W_k} [-\log_2(\text{sigmoid}(LLR_0(w_k|\mathbf{y}_k)))]. \quad (14)$$

This let us to conclude that maximizing mutual information under uniform input distributions relies on maximizing the averaged $LLR_0$, used below to define the loss function.

### C. Cross-entropy loss

In the context of machine learning, cross-entropy is used to optimize the parameters of a model. It compares the outputs of the training data to the decisions made at the label of a system. We propose to optimize the constellation points by evaluating the quality of simulated received data with their respective posterior probability distributions. The cross-entropy loss associated to a batch of N simulations, is expressed as:

$$\mathcal{L}_k = -\frac{1}{N} \sum_{n=1}^{N} \sum_{w_k \in W_k} \mathbb{1}\{w_k^n=w\} \cdot \log_2(P_{W_k|\mathbf{Y}_k}(W_k^n=w|\mathbf{y}_k^n)) \\ \xrightarrow{N \to \infty} \mathbb{E}_{W_k \mathbf{Y}_k} [-\log_2(P_{W_k|\mathbf{Y}_k}(w_k|\mathbf{y}_k))] = H(W_k|\mathbf{Y}_k). \quad (15)$$

The limit when $N \to \infty$ shows that the loss $\mathcal{L}_k$ converges to the equivocation (13) which is directly linked to the mutual information (12). We cannot guarantee that a gradient descent will reach the optimal point, but this proves that asymptotically, the minimum of the Loss function corresponds to the mutual information maximum. For simplification, we introduce the notation:

$$\mathcal{L}_k(\mathcal{X}; w_k^n; \mathbf{y}_k^n) = -\log_2(\text{sigmoid}(LLR_0(\mathcal{X}; w_k^n|\mathbf{y}_k^n))), \quad (16)$$

which represents the loss on user k for a given sample n. Using (10) and Gaussian noise statistics, we have:

$$LLR_0(\mathcal{X}; w_k^n|\mathbf{y}_k^n) = \ln \left( \frac{\sum_{\mathbf{x} \in \mathcal{X}_k(w_k^n)} e^{-d_k(\mathbf{x})}}{\sum_{\mathbf{x} \notin \mathcal{X}_k(w_k^n)} e^{-d_k(\mathbf{x})}} \right), \quad (17)$$

where the exponent in the Gaussian noise distribution formula is $d_k(\mathbf{x}) = \frac{\|\mathbf{y}_k - \frac{\zeta_k^H}{2\sigma_k^2}\mathbf{x}\|^2}{2\sigma_k^2}$. The aim of (17) is to exhibit the loss with respect to the reference and observed constellation points.

The overall loss (15) can then be expressed as a function of $\mathcal{X}$:

$$\mathcal{L}_k(\mathcal{X}) = \frac{1}{N} \sum_{n=1}^{N} \sum_{w_k \in W_k} \mathbb{1}\{w_k^n=w\} \cdot \mathcal{L}_k(\mathcal{X}; w_k^n; \mathbf{y}_k^n). \quad (18)$$

The function $\mathcal{L}_k(\mathcal{X})$ is differentiable in $\mathcal{X}$ which makes it suitable for optimization methods based on stochastic gradient that will lead to local stationary point as $\mathcal{L}_k(\mathcal{X})$ is non-convex in $\mathcal{X}$.

## IV. PROPOSED ALGORITHMS

In this section, we present the algorithm employed to optimize the previously defined loss function. The optimization algorithm Adam [13], has been chosen, a variant of Stochastic Gradient Descent (SGD) with momentum. We aim to directly optimize the position of each point of the constellation without restricting it to any MU-LP structure.

**Algorithm 1** Constellation optimization algorithm using projected stochastic gradient descent

1: Initialize random $\mathcal{X}$

2: **while** not converged **do**

3: Generate $\{\mathbf{w}^1, \dots, \mathbf{w}^N\}$

4: **for** $k \in K$ **do**

5: Simulate $\{\mathbf{y}_k^1, \dots, \mathbf{y}_k^N\}$ according to (2)

6: $\mathcal{L}_k(\mathcal{X}) \leftarrow \frac{1}{N} \sum_{n=1}^{N} \mathcal{L}_k(\mathcal{X}; w_k^n; \mathbf{y}_k^n)$

7: **end for**

8: $\mathcal{L}(\mathcal{X}) \leftarrow \max(\mathcal{L}_1(\mathcal{X}), \dots, \mathcal{L}_K(\mathcal{X}))$

9: $\mathcal{X} \leftarrow \Pi_{\mathcal{P}}[\text{ADAM}(\mathcal{X}, \eta, \mathcal{L})]$

10: **end while**

The proposed iterative algorithm aims to optimize the constellation design while ensuring fairness across all users. The steps of the algorithm follow:

* **Initialization** (1): A random constellation $\mathcal{X}$ is chosen.

* **Generation of codewords** (3): N random codewords representing N independent realizations of $\mathbf{W}$ are drawn.

* **Simulation of observations** (5): For each vector, N observations are simulated according to the Gaussian channel.

* **Computation of user loss** (6): The average individual loss is computed for each user $k$.

* **Reduction Function** (8): After computing the loss of all users, a reduction function $\mathbb{R}^K \to \mathbb{R}$ is applied, that is the MIN-MAX function.

* **Gradient Step with Projection** (9): One step of the Adam<fnref n="1" /> algorithm with step size $\eta$ is performed. The output is then projected into $\Pi_p$, with a normalization (to ensure $P = P_m$) and a projection to the closest point (to ensure each $p_k \le P_c$).

<footnote n="1">¹we use the Adam algorithm as implemented in the TensorFlow library</footnote>
* **Generation of codewords (3):** N random codewords representing N independent realizations of W are drawn.

* **Simulation of observations (5):** For each vector, N observations are simulated according to the Gaussian channel.

* **Computation of user loss (6):** The average individual loss is computed for each user k.

* **Reduction Function (8):** After computing the loss of all users, a reduction function $\mathbb{R}^K \to \mathbb{R}$ is applied, that is the MIN-MAX function.

* **Gradient Step with Projection (9):** One step of the Adam<fnref n="1" /> algorithm with step size $\eta$ is performed. The output is then projected into $\Pi_p$, with a normalization (to ensure $P = P_m$) and a projection to the closest point (to ensure each $p_k \le P_c$).

This algorithm runs until a convergence criterion is met or when the number of iteration reaches a threshold.

## V. NUMERICAL RESULTS

The proposed joint constellation design algorithm is compared to MU-LP solutions. Reminding that our primary objective is to maximize the minimum mutual information between the BS and each user, we evaluate the mutual information as a function of the signal-to-noise ratio (SNR), defined by:

$$ \text{SNR} = -10 \log_{10} (P_m / \sigma_k^2), \quad (20) $$

It is worth noting that the optimizer uses the stable LLR (19) while final scores are computed with the standard LLR (17). In all simulations, we only consider single-antenna receivers ($R = 1$) to compare with MU-LP encoders even if our algorithm works for any $R \ge 1$. The hyper-parameters $\eta$ and $N$ (samples per iteration) are respectively set to 0.1 and 10000, and we fix $P_c = 4$ and $P_m = 1$.

The first scenarios are for illustrative purpose, with $T = 2$ transmit antennas and $K = 2$ users with real channels, allowing to plot the constellations. Subsequently, we present a scenario involving complex channels with $T = 4$ and $K = 10$.

### A. Real Channels

Given real channels, the constellation over the 2 antennas are plot on, allowing to compare the MU-LP and learned approaches. Two distinct scenarios are proposed:

* **Scenario 1:** $T = 2$ transmit antennas, $K = 2$ users with a SNR of 6 and 8 dB, respectively and non-orthogonal channels: $\zeta_1 = [1 \ 0]^T$, $\zeta_2 = [\frac{\sqrt{2}}{2} \ \frac{\sqrt{2}}{2}]^T$.

* **Scenario 2:** same as Scenario 1 but with colinear channels: $\zeta_1 = [\frac{\sqrt{2}}{2} \ \frac{\sqrt{2}}{2}]^T$, $\zeta_2 = [\frac{\sqrt{2}}{2} \ \frac{\sqrt{2}}{2}]^T$.

In both scenario the BS sends a single bit to each user: $W_1 = \{0, 1\}$ and $W_2 = \{0, 1\}$. The resulting constellations are presented in Fig. 1 and Fig.2 where for each point the first and second indicated bits are transmitted to user 1 and user 2 respectively. $\zeta_1$ and $\zeta_2$ are represented by pink and brown arrow. We observe on Fig. 1, that MU-LP approaches

<figure><img src="image_2.png" /><figcaption>Fig. 1: Constellation of the different encoders with non-orthogonal channels</figcaption></figure>

<figure><img src="image_3.png" /><figcaption>Fig. 2: Constellation of the different encoders with colinear channels</figcaption></figure>

yield symmetrical constellations, where each constellation point associated to $b_0b_1$ is centrally symmetrical to the point associated to $\bar{b}_0\bar{b}_1$. In contrast, the learned approach does not exhibit this symmetry. More interestingly, the projection of the constellation points onto each user's observation line shows that the 0s and 1s are split in two decision regions, as usual. On the opposite, the learned approach results in a different pattern: the projection of user 1 reveals a central cluster of 0s surrounded by two clusters of 1s at both extremities. This unique clustering behavior exhibits the kind of optimization such approach can provide and allowing to improve the mutual information, as summarized in Table I. On the second scenario, the proposed method learned a joint constellation by adapting the distance between the different points and almost achieve the equal rate capacity given by SC (its value is 0.67 per user, according to [1]). Notably, while our proposed approach

<footnote n="1">¹we use the Adam algorithm as described in the PyTorch package [14]</footnote>
This algorithm runs until a convergence criterion is met or when the number of iteration reaches a threshold.

## V. NUMERICAL RESULTS

The proposed joint constellation design algorithm is compared to MU-LP solutions. Reminding that our primary objective is to maximize the minimum mutual information between the BS and each user, we evaluate the mutual information as a function of the signal-to-noise ratio (SNR), defined by:

$$SNR_{dB} = 10 \log_{10} (P_m / \sigma_k^2). \qquad (19)$$

In this work, we focus only on the design of the constellation, we thus assume that the encoder (from bits to symbols $W_k$) and the decoder (from received points $Y_k$ to estimated bits) are optimal, meaning that the natural metric to consider is the mutual information between $W_k$ and $Y_k$ which represents an upperbound on the Bit Error Rate (BER). In all simulations, we only consider single-antenna receivers ($R = 1$) to compare with MU-LP encoders even if our algorithm works for any $R \ge 1$. The hyper-parameters $\eta$ and $N$ (samples per iteration) are respectively set to 0.1 and 10000, and we fix $P_c = 4$ and $P_m = 1$. This implies that the Peak to Average Power Ratio (PAPR) is at most 4. Alg. 1 is run for 100 iterations. The code<fnref n="2" /> of the simulations is developed using the Sionna [14] package. In these simulations, we focus on small per-user modulations, but it is important to note that 1 bit for 10 users could also be 2 bits for 5 users. Essentially, we can process 10 bits across different users.

The first scenarios are for illustrative purpose, with $T = 2$ transmit antennas and $K = 2$ users with real channels, allowing to plot the constellations. Subsequently, we present a scenario involving complex channels with $T = 4$ and $K = 10$.

### A. Real Channels

Given real channels, the constellation over the 2 antennas are plot on, allowing to compare the MU-LP and learned approaches. Two distinct scenarios are proposed :

* **Scenario 1:** $T = 2$ transmit antennas, $K = 2$ users with a SNR of 6 and 8 dB, respectively and non-orthogonal channels: $\zeta_1 = [1 \ 0]^T$, $\zeta_2 = [\frac{\sqrt{2}}{2} \ \frac{\sqrt{2}}{2}]^T$.

* **Scenario 2:** same as Scenario 1 but with colinear channels: $\zeta_1 = [\frac{\sqrt{2}}{2} \ \frac{\sqrt{2}}{2}]^T$, $\zeta_2 = [\frac{\sqrt{2}}{2} \ \frac{\sqrt{2}}{2}]^T$.

In both scenario the BS sends a single bit to each user: $W_1 = \{0, 1\}$ and $W_2 = \{0, 1\}$. The resulting constellations are presented in Fig. 1 and Fig.2 where for each point the first and second indicated bits are transmitted to user 1 and user 2 respectively. $\zeta_1$ and $\zeta_2$ are represented by pink and brown arrow. We observe on Fig. 1, that MU-LP approaches yield symmetrical constellations, where each constellation point associated to $b_0b_1$ is centrally symmetrical to the point associated to $\bar{b}_0\bar{b}_1$. In contrast, the learned approach does not exhibit this symmetry. More interestingly, the projection of the constellation points onto each user's observation line shows

<figure><img src="image_2.png" /><figcaption>Fig. 1: Constellation of the different encoders with non-orthogonal channels</figcaption></figure>

<figure><img src="image_3.png" /><figcaption>Fig. 2: Constellation of the different encoders with colinear channels</figcaption></figure>

that the 0s and 1s are split in two decision regions, as usual. On the opposite, the learned approach results in a different pattern: the projection of user 1 reveals a central cluster of 0s surrounded by two clusters of 1s at both extremities. This unique clustering behavior exhibits the kind of optimization such approach can provide and allowing to improve the mutual information, as summarized in Table I. On the second scenario, the proposed method learned a joint constellation by adapting the distance between the different points and almost achieve the equal rate capacity given by SC (its value is 0.67 per user, according to [1]). Notably, while our proposed approach achieves fairness across all users, the MMSE approach sacrifices the performance of user 1, with the highest noise level. In addition our approach maintains a strong average performance. Note that in scenario 2, the ZF encoder is not able to cancel

<footnote n="2">²Available at https://gitlab.inria.fr/maracas/publications/broadcast-mu-mimo-joint-constellation-learning</footnote>
<table><caption>TABLE I: Scenario results</caption><thead><tr><th>Scenario</th><th>Encoders</th><th>I<sub>1</sub></th><th>I<sub>2</sub></th><th>min<sub>k</sub>(I<sub>k</sub>)</th><th>mean<sub>k</sub>(I<sub>k</sub>)</th></tr></thead><tbody><tr><td rowspan="4">1</td><td>MMSE</td><td>0.51</td><td>0.76</td><td>0.51</td><td>0.63</td></tr><tr><td>ZF</td><td>0.48</td><td>0.64</td><td>0.48</td><td>0.56</td></tr><tr><td>Matched</td><td>0.49</td><td>0.56</td><td>0.49</td><td>0.52</td></tr><tr><td>MAX-MIN</td><td>0.67</td><td>0.70</td><td><strong>0.67</strong></td><td><strong>0.68</strong></td></tr><tr><td rowspan="4">2</td><td>MMSE</td><td>0.29</td><td>0.66</td><td>0.29</td><td>0.53</td></tr><tr><td>ZF</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>Matched</td><td>0.39</td><td>0.44</td><td>0.39</td><td>0.42</td></tr><tr><td>MAX-MIN</td><td>0.64</td><td>0.63</td><td><strong>0.63</strong></td><td><strong>0.63</strong></td></tr></tbody></table>

achieves fairness across all users, the MMSE approach sacrifices the performance of user 1, with the highest noise level. In addition our approach maintains a strong average performance. Note that in scenario 2, the ZF encoder is not able to cancel interference as the rank of the channel matrix is one, making $H^H H$ not invertible.

## B. Complex Channels

The simulations are now extended to complex channels with more receivers. We focus on a scenario with $K = 10$ users and $T = 4$ transmitting antennas. For each round, an average SNR $s$ (dB) where $s \in [-5, 15]$, is fixed, and the SNR $\gamma_k$ of each user is drawn once from $N(s, 1)$. We conduct 20 independent experiments, where each follows:

1) For each user a random channel vector is drawn from $CN(0, I_T)$ and is then normalized,

2) For each average SNR level $s$:

* MU-LP reference constellations are computed according to II-B and are projected into $\Pi_p$,

* 10 constellations are drawn randomly and Alg.1 is run on each. The best resulting constellation is recorded.

3) The average and minimum mutual information values are computed for each constellation.

<figure><img src="image_4.png" /><figcaption>Fig. 3: Mutual information on users for the different encoders under varying SNR</figcaption></figure>

On Fig.3, plain lines show the minimum mutual information among the users. For all SNR values, the learned constellation outperforms MU-LP constellations, except for the matched encoder at low SNR. It is worth noting that the ZF encoder (red curve) does not cancel all interference due to power constraints

imposed on the constellation. On Fig.3 dashed lines illustrate the mean of the mutual information on all users. It can be seen that our method outperforms MU-LP in all SNR regimes, while improving the min value up to (42%).

## VI. CONCLUSION

In this paper, we proposed an approach to optimize a joint constellation for the MU-MIMO broadcast channel, with a stochastic gradient descent maximizing the minimal mutual information between the BS and each user. By leveraging on a learned approach, we obtained constellations that are not feasible with traditional linear methods. This proves the higher flexibility of the learned technique, especially for non linear optimization (e.g. Min-Max instead of sum-rate) with a significant performance improvement. The generalization for larger modulations will be evaluated in the future.

## REFERENCES

[1] A. El Gamal and Y.-H. Kim, *Network information theory*. Cambridge university press, 2011.

[2] H. Weingarten, Y. Steinberg, and S. S. Shamai, “The capacity region of the Gaussian multiple-input multiple-output broadcast channel,” *IEEE transactions on information theory*, vol. 52, no. 9, pp. 3936–3964, 2006.

[3] Y. Liu, H. Xing, C. Pan, A. Nallanathan, M. Elkashlan, and L. Hanzo, “Multiple-antenna-assisted non-orthogonal multiple access,” *IEEE Wireless Communications*, vol. 25, no. 2, pp. 17–23, 2018.

[4] B. Clerckx and C. Oestges, *MIMO wireless networks: channels, techniques and standards for multi-antenna, multi-user and multi-cell systems*. Academic Press, 2013.

[5] B. Clerckx, Y. Mao, R. Schober, E. A. Jorswieck, D. J. Love, J. Yuan, L. Hanzo, G. Y. Li, E. G. Larsson, and G. Caire, “Is NOMA efficient in multi-antenna networks? a critical look at next generation multiple access techniques,” *IEEE Open Journal of the Communications Society*, vol. 2, pp. 1310–1343, 2021.

[6] M. Vaezi, R. Schober, Z. Ding, and H. V. Poor, “Non-orthogonal multiple access: Common myths and critical questions,” *IEEE Wireless Communications*, vol. 26, no. 5, pp. 174–180, 2019.

[7] H. Joudeh and B. Clerckx, “Robust transmission in downlink multiuser miso systems: A rate-splitting approach,” *IEEE Transactions on Signal Processing*, vol. 64, no. 23, pp. 6227–6242, 2016.

[8] A. Padakandla and S. S. Pradhan, “Achievable rate region for three user discrete broadcast channel based on coset codes,” *IEEE Transactions on Information Theory*, vol. 64, no. 4, pp. 2267–2297, 2018.

[9] F. Alberge, “Constellation design with deep learning for downlink non-orthogonal multiple access,” in *2018 IEEE 29th Annual International Symposium on Personal, Indoor and Mobile Radio Communications (PIMRC)*, Sep. 2018, pp. 1–5.

[10] E. C. Cejudo, H. Zhu, and O. Alluhaibi, “On the Power Allocation and Constellation Selection in Downlink NOMA,” in *2017 IEEE 86th Vehicular Technology Conference (VTC-Fall)*, Sep. 2017, pp. 1–5.

[11] H. Lee, S. Kim, and J.-H. Lim, “Multiuser Superposition Transmission (MUST) for LTE-A systems,” in *2016 IEEE International Conference on Communications (ICC)*, May 2016, pp. 1–6.

[12] M. Goutay, F. A. Aoudia, J. Hoydis, and J.-M. Gorce, “Machine learning for MU-MIMO receive processing in OFDM systems,” *IEEE Journal on Selected Areas in Communications*, vol. 39, no. 8, pp. 2318–2332, 2021.

[13] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” *arXiv preprint arXiv:1412.6980*, 2014.

[14] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, “PyTorch: An Imperative Style, High-Performance Deep Learning Library,” in *Advances in Neural Information Processing Systems 32*. Curran Associates, Inc., 2019, pp. 8024–8035.
<table><caption>TABLE I: Scenario results</caption><thead><tr><th>Scenario</th><th>Encoders</th><th>I<sub>1</sub></th><th>I<sub>2</sub></th><th>min<sub>k</sub>(I<sub>k</sub>)</th><th>mean<sub>k</sub>(I<sub>k</sub>)</th></tr></thead><tbody><tr><td rowspan="4">1</td><td>MMSE</td><td>0.50</td><td>0.76</td><td>0.50</td><td>0.63</td></tr><tr><td>ZF</td><td>0.48</td><td>0.64</td><td>0.48</td><td>0.56</td></tr><tr><td>Matched</td><td>0.49</td><td>0.55</td><td>0.49</td><td>0.52</td></tr><tr><td>MAX-MIN</td><td>0.69</td><td>0.70</td><td><strong>0.69</strong></td><td><strong>0.69</strong></td></tr><tr><td rowspan="4">2</td><td>MMSE</td><td>0.28</td><td>0.66</td><td>0.29</td><td>0.47</td></tr><tr><td>ZF</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>Matched</td><td>0.39</td><td>0.44</td><td>0.39</td><td>0.42</td></tr><tr><td>MAX-MIN</td><td>0.63</td><td>0.63</td><td><strong>0.63</strong></td><td><strong>0.63</strong></td></tr></tbody></table>

interference as the rank of the channel matrix is one, making $H^H H$ not invertible.

## B. Complex Channels

The simulations are now extended to complex channels with more receivers. We focus on a scenario with $K = 10$ users and $T = 4$ transmitting antennas. For each round, an average SNR $s$ (dB) where $s \in [-5, 15]$, is fixed, and the SNR $\gamma_k$ of each user is drawn once from $N(s, 1)$. We conduct 20 independent experiments, where each follows:

1) For each user a random channel vector is drawn from $CN(0, I_T)$ and is then normalized,

2) For each average SNR level $s$:

* MU-LP reference constellations are computed according to II-B and are projected into $\Pi_p$,

* 10 constellations are drawn randomly and Alg.1 is run on each. The best resulting constellation is recorded.

3) The average and minimum mutual information values are computed for each constellation.

<figure><img src="image_4.png" /><figcaption>Fig. 3: Mutual information on users for the different encoders under varying SNR</figcaption></figure>

On Fig.3, plain lines show the minimum mutual information among the users. For all SNR values, the learned constellation outperforms MU-LP constellations, except for the matched encoder at low SNR. It is worth noting that the ZF encoder (red curve) does not cancel all interference due to power constraints imposed on the constellation. On Fig.3 dashed lines illustrate the mean of the mutual information on all users. It can be seen that our method outperforms MU-LP in all SNR regimes, while improving the min value up to (42%).

# VI. CONCLUSION

In this paper, we proposed an approach to optimize a joint constellation for the MU-MIMO broadcast channel, with a stochastic gradient descent maximizing the minimal mutual information between the BS and each user. By leveraging on a learned approach, we obtained constellations that are not feasible with traditional linear methods. This proves the higher flexibility of the learned technique, especially for non linear optimization (e.g. Min-Max instead of sum-rate) with a significant performance improvement.

However, this work has some limitations, in particular the exponential growth of the constellation size ($2^K$ points) restricts the value of $K$ (in our case, 10). Moreover, the proposed solution has to be computed each time the channel changes. Generalization to a more efficient and scalable method, possibly with a neural network, is left as future work.

## ACKNOWLEDGMENT

This his work was funded by the French Government through the “Plan de Relance” and “Programme d’investissements d’avenir”

## REFERENCES

[1] A. El Gamal and Y.-H. Kim, *Network information theory*. Cambridge university press, 2011.

[2] H. Weingarten, Y. Steinberg, and S. S. Shamai, “The capacity region of the Gaussian multiple-input multiple-output broadcast channel,” *IEEE transactions on information theory*, vol. 52, no. 9, pp. 3936–3964, 2006.

[3] Y. Liu, H. Xing, C. Pan, A. Nallanathan, M. Elkashlan, and L. Hanzo, “Multiple-antenna-assisted non-orthogonal multiple access,” *IEEE Wireless Communications*, vol. 25, no. 2, pp. 17–23, 2018.

[4] B. Clerckx and C. Oestges, *MIMO wireless networks: channels, techniques and standards for multi-antenna, multi-user and multi-cell systems*. Academic Press, 2013.

[5] B. Clerckx, Y. Mao, R. Schober, E. A. Jorswieck, D. J. Love, J. Yuan, L. Hanzo, G. Y. Li, E. G. Larsson, and G. Caire, “Is NOMA efficient in multi-antenna networks? a critical look at next generation multiple access techniques,” *IEEE Open Journal of the Communications Society*, vol. 2, pp. 1310–1343, 2021.

[6] M. Vaezi, R. Schober, Z. Ding, and H. V. Poor, “Non-orthogonal multiple access: Common myths and critical questions,” *IEEE Wireless Communications*, vol. 26, no. 5, pp. 174–180, 2019.

[7] H. Joudeh and B. Clerckx, “Robust transmission in downlink multiuser miso systems: A rate-splitting approach,” *IEEE Transactions on Signal Processing*, vol. 64, no. 23, pp. 6227–6242, 2016.

[8] A. Padakandla and S. S. Pradhan, “Achievable rate region for three user discrete broadcast channel based on coset codes,” *IEEE Transactions on Information Theory*, vol. 64, no. 4, pp. 2267–2297, 2018.

[9] F. Alberge, “Constellation design with deep learning for downlink non-orthogonal multiple access,” in *2018 IEEE 29th Annual International Symposium on Personal, Indoor and Mobile Radio Communications (PIMRC)*. IEEE Press, 2018, p. 1–5.

[10] E. C. Cejudo, H. Zhu, and O. Alluhaibi, “On the Power Allocation and Constellation Selection in Downlink NOMA,” in *2017 IEEE 86th Vehicular Technology Conference (VTC-Fall)*, Sep. 2017, pp. 1–5.

[11] H. Lee, S. Kim, and J.-H. Lim, “Multiuser superposition transmission (must) for lte-a systems,” in *2016 IEEE International Conference on Communications (ICC)*. IEEE, 2016, pp. 1–6.

[12] M. Goutay, F. A. Aoudia, J. Hoydis, and J.-M. Gorce, “Machine learning for MU-MIMO receive processing in OFDM systems,” *IEEE Journal on Selected Areas in Communications*, vol. 39, no. 8, pp. 2318–2332, 2021.

[13] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” *arXiv preprint arXiv:1412.6980*, 2014.

[14] J. Hoydis, S. Cammerer, F. Ait Aoudia, A. Vem, N. Binder, G. Marcus, and A. Keller, “Sionna: An open-source library for next-generation physical layer research,” *arXiv preprint*, Mar. 2022.