# A Deep Reinforcement Learning Based Region-Specific Beamformer for Sparse Arrays 3D Ultrasound Imaging

Mohamed Tamraoui, Hervé Liebgott, Emmanuel Roux

► To cite this version:

Mohamed Tamraoui, Hervé Liebgott, Emmanuel Roux. A Deep Reinforcement Learning Based Region-Specific Beamformer for Sparse Arrays 3D Ultrasound Imaging. IEEE Transactions on Ultrasonics, Ferroelectrics and Frequency Control, 2025, pp.1-1. 10.1109/TUFFC.2025.3560872. hal-05035945

HAL Id: hal-05035945

https://hal.science/hal-05035945v1

Submitted on 16 Apr 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# A Deep Reinforcement Learning Based Region-Specific Beamformer for Sparse Arrays 3D Ultrasound Imaging

Mohamed Tamraoui, Member, IEEE, Hervé Liebgott, Senior Member, IEEE, Emmanuel Roux, Member, IEEE

**Abstract**— Sparse arrays offer several advantages over other element reduction techniques for 3D ultrasound imaging. However, the large inter-element spacing in these arrays results in high sidelobe-related artifacts, which significantly degrade image quality and limit their application in 3D ultrasound imaging. Adaptive beamformers have been proposed to mitigate sidelobe-related artifacts, but they often degrade speckle texture quality, resulting in naturally dark images. To overcome these limitations, we propose RSB-Net, a region-specific beamformer based on deep reinforcement learning. RSB-Net adaptively selects the most suitable beamformer for each pixel of the image, applying adaptive beamforming in regions dominated by sidelobe artifacts and delay-and-sum beamforming in regions where speckle texture should be preserved. The effectiveness of RSB-Net was validated on both simulated and experimental synthetic transmit aperture RF datasets with a new designed sparse array prototype. On simulated data, RSB-Net achieved significant gains, with improvements of 52.81 dB in contrast ratio and 0.65 in generalized contrast-to-noise ratio compared to DAS beamforming. In experimental tissue-mimicking phantom data, RSB-Net demonstrated similar performance, achieving gains of 51.01 dB and 0.64 respectively. These results highlight the potential of RSB-Net as a robust and effective solution for high-quality B-mode 3D ultrasound imaging using 2D sparse arrays, advancing the standardization of 3D ultrasound in clinical settings by enhancing anatomical visualization, reducing operator dependency, and improving measurement accuracy for lesions and calcifications.

**Index Terms**— 2D sparse arrays, 3D ultrasound, Adaptive beamformers, Deep reinforcement learning, Delay-And-Sum, Region based beamforming.

<figure><img src="image_2.png" /></figure>

## I. INTRODUCTION

3D ultrasound imaging offers significant advantages over traditional 2D modalities by transforming how medical professionals acquire, visualize and interpret anatomical structures [1], [2]. However, despite the advantages of 3D ultrasound imaging, it is not yet standard in clinical practice, primarily due to limitations in image quality and the high cost of

This work was supported by the LABEX PRIMES under Grant ANR-11-LABX-0063 of Université de Lyon, through the Program "Investissements d'Avenir" under Grant ANR-11-IDEX-0007 operated by the French National Research Agency (ANR), performed within the framework of the ANR-20-CE19-0023 SPARTECHUS. This material is based upon work done on the PILoT facility (PILoT, INSA-Lyon).

The authors are with the Univ Lyon, INSA-Lyon, Université Lyon 1, CNRS, Inserm, CREATIS UMR, France.

current systems. A critical factor in 3D image quality is the ultrasound probe, ideally a 2D matrix array that can scan the entire volume in both elevation and lateral directions [3], [4]. To avoid high grating lobes artifacts, these arrays must respect the Nyquist criterion, maintaining a pitch below half the wavelength (λ/2). Additionally, achieving a comparable resolution to a 1D array requires a large number of elements in the aperture of the probe; for example, a comparable resolution to a 128-element linear array would require a 128 × 128 matrix, totaling 16,384 elements. However, controlling this large number of elements requires an equivalent number of channels, which far exceeds the typical 256 channels available in most ultrasound systems. Consequently, manufacturing such arrays is technically challenging and costly.
## Highlights

* This Study introduces RSB-Net, a deep reinforcement learning-based model that combines the performance of adaptive beamformers with the DAS beamformer within a learning framework to reconstruct high quality 3D images.

* RSB-Net selectively apply adaptive beamformers in regions dominated by sidelobe artifacts, while using the DAS beamformer in speckle regions to maintain texture quality.

* RSB-Net improved contrast ratio by 52.81 dB and gCNR by 0.65 over DAS on simulated data, and by 51.01 dB and 0.64 on experimental phantom data.

Sparse arrays offer a promising solution for 3D ultrasound imaging through an optimal distribution of fewer transducer elements across the aperture of the probe, allowing the use of fewer channels [5], [6]. Typically designed with 256 elements, sparse arrays align with the channel capacity of most modern ultrasound systems, allowing each element to be continuously controlled by an individual channel. This configuration allows high flexibility in beamforming and steering capabilities as well as the reduction of the manufacturing complexity and cost of the probe, making sparse arrays a practical and cost-effective choice for 3D imaging [7]–[10]. However, despite their advantages, sparse arrays face two significant challenges: low sensitivity and reduced image contrast.

Sparse arrays are typically designed with small apertures because fewer elements are distributed over the array footprint, necessitating smaller footprints to minimize the distance between elements [6], [11], [12]. Consequently, the size of the elements must be small, usually around half the wavelength. Small elements, however, exhibit very low sensitivity and high electrical impedance, significantly reducing the SNR [13]. Moreover, the reduction in the number of elements limits the amount of acoustic energy transmitted and received due to less vibrating surface area compared to a full matrix array. Coded excitation has been proven to substantially enhance the SNR in ultrasound imaging [14], [15]. In a previous study we demonstrated that coded excitation can significantly enhance the SNR of sparse arrays [16], [17]. Additionally, increasing the element size in sparse arrays can enhance sensitivity of the array while reducing impedance mismatch with the electronic environment of the probe. However, larger element sizes reduce their divergence, thereby reducing the steering capability of the array [18], [19].

One of the significant drawbacks of sparse arrays is their reduced image quality. Indeed, the spatial distribution of elements on the array aperture, with inter-element spacing larger than $\lambda/2$, leads to high sidelobes level around the main lobe [5], [20]. These sidelobes cause energy leakage during Delay-And-Sum (DAS) beamforming, severely reducing image contrast. The sidelobes are heavily influenced by the distribution of elements across the aperture of the array. As a result, much of the research on sparse arrays has focused on finding optimal element distributions that minimize the number of elements while maintaining acceptable image quality [7], [21].

The Delay And Sum (DAS) algorithm is the most commonly used method to beamform ultrasound images [22]. This technique relies on geometric delay calculations to focus ultrasound signals and reconstruct the image. Although DAS offers a straightforward method for reconstructing ultrasound

images with relatively low computational complexity, it is prone to artifacts resulting from off-axis signals [23], [24]. These artifacts can arise from grating lobes and high sidelobes level, which are particularly problematic in the beam pattern of 2D sparse arrays [25]. To overcome the limitations of DAS beamforming, several adaptive, coherence-based, and non-linear beamformers have been proposed in the literature. In this section, these techniques will be differentiated based on their category. However, for the remainder of this paper, we will refer to all of them collectively as adaptive beamformers for simplicity.

The minimum variance (MV) beamformer is an adaptive technique introduced to enhance ultrasound image quality, particularly in terms of resolution [26], [27]. Unlike DAS, which uses fixed apodization weights, the MV beamformer applies adaptive apodization weights to the delayed signals. This is achieved by estimating a covariance matrix from the RF data and solving a minimization problem. However, the computational complexity of estimating and inverting the covariance matrix limits the practical application of the MV beamformer. Additionally, while MV beamforming improves resolution, it is known to have lower contrast, similar to the DAS algorithm, making it less effective in low-contrast situations, such as those encountered with 2D sparse arrays [23], [24].

In addition to adaptive beamformers, coherence-based beamformers have also been introduced in the literature to improve ultrasound image quality, particularly in terms of contrast [28]–[31]. One of the most commonly known methods is the Coherence Factor (CF) [28], [29]. CF can be considered as a sidelobes reduction weighting technique that enhances image contrast by applying weights to the DAS beamformed image. These weights are calculated as the ratio between the energy of the main lobe (coherent sum) and the total energy (incoherent sum) in the received signals across the array elements. By weighting the DAS beamformed image with CF, sidelobes artifacts are reduced. Various CF variants have been proposed, such as the Generalized Coherence Factor (GCF) [31] and Phase Coherence Factor (PCF) [30]. GCF considers low-frequency energy ratios, while PCF uses phase information to reduce sidelobe artifacts in beamformed images.

Delay Multiply and Sum (DMAS) is a non-linear beamforming technique developed to improve image contrast in ultrasound imaging [32]–[34]. Unlike DAS beamforming, where the TOF delayed signals are simply summed, DMAS involves multiplying the delayed signals pairwise across the receiving elements before summation. This multiplication can be viewed as a form of cross-correlation of the received signals across the
aperture, enhancing the sensitivity of the DMAS to incoherent
signals and thereby improving contrast [32], [35], [36].

While adaptive beamformers are very successful in remov-
ing of axis-signals and side-lobes artifacts, they also have
drawbacks that impact ultrasound diagnostics [37], [38]. A
major issue is speckle alteration, as these beamformers sup-
press noise but also low-reflecting scatterers, reducing speckle
texture and brightness, especially in high sidelobe conditions
[39], [40]. This degradation is critical when speckle conveys
tissue characteristics. Another issue is dark region artifacts
(DRA), particularly near hyperechoic targets [38], [40]. High
sensitivity to off-axis signals such as the high sidelobes of 2D
sparse arrays can misclassify main-lobe signals as sidelobes,
suppressing legitimate signals and creating artificial dark re-
gions that obscure anatomical details [37], [38].

Deep learning methods have gained significant attention in
recent years for ultrasound beamforming [41]. These methods
can be broadly categorized into three approaches based on
their application within the ultrasound image reconstruction
process.

The first category aims to enhance the quality of already
DAS beamformed, low-quality images. In this category neural
networks are trained to learn to map low-quality to high-
quality images [42]–[44]. Although these methods have shown
promising results in improving the image quality, they are
still inefficient because much of the information present in
the raw RF data is lost during the initial beamforming process
before being fed into the network [44]. The second group of
methods replaces the entire beamforming process with a neural
network. In this category, the network learns to map raw RF
data directly into a beamformed image [45]–[47]. However,
the challenge with these approaches is that the network needs
to learn the complex geometric delay calculations necessary to
convert temporal information into spatial information, making
the training process difficult. The third group of methods
focuses on replacing only a specific step of the beamforming
process with a neural network after time delay corrections have
been applied [48]–[50]. In [49] a deep neural network which
accelerates the MV beamformer by adaptively estimating the
apodization weights has been proposed.

Like adaptive beamformers, the described deep learning-
based beamforming methods, while promising, also present
several challenges. Neural networks, despite their ability to
enhance image quality, often function as black boxes, which
poses significant challenges in clinical settings [41]. The
opaque nature of these models makes it difficult to ensure
and trust the accuracy of the results, particularly in cases
where neural network hallucinations may occur [51]. These
hallucinations can introduce false features into the images,
appearing as hyper- or hypo-echogenic regions, potentially
leading to the misidentification of lesions.

To address the challenge of low image contrast associated
with 2D sparse arrays, this work proposes a new reconstruction
algorithm designed to reconstruct high-quality 3D images from
synthetic transmit aperture (STA) RF data acquired using a
new designed sparse array prototype. A key feature of the
proposed method is that the final image is obtained by combin-
ing multiple beamformers, where the selection of the optimal

<figure><img src="image_3.png" /><figcaption>Fig. 1. First row : 2D sparse array prototype. (a) Fermat's spiral elements distribution, (b) the designed prototype. Second row : Experimental B-mode images acquired with the developed probe prototype of a (c) wire phantom, (d) home-made tissue-mimicking phantom.</figcaption></figure>

beamformer for each voxel is learned by a dedicated neural
network. The proposed approach leverages the advantages
of adaptive beamforming in suppressing sidelobe artifacts by
selectively applying these beamformers in regions dominated
by high sidelobe levels, such as anechoic inclusions. In con-
trast, the DAS beamformer is applied in speckle-dominated
regions to preserve the natural speckle texture. Since adaptive
beamformers are only applied to voxels dominated by sidelobe
artifacts, the speckle texture remains intact without the ap-
pearance of dark region artifacts. The optimal beamformer for
each voxel is determined using a deep reinforcement learning
algorithm. This approach can be generalized to any other kind
of arrays or imaging sequences.

II. METHODS

A. New 2D sparse array prototype

In collaboration with IMASONIC (Voray sur l'Ognon,
France), we developed a new 2D sparse array prototype. This
sparse array was designed with large elements to achieve low
impedance, thereby improving sensitivity and enhancing the
SNR. Each element was equipped with an integrated front lens
to maintain high divergence. The probe featured an aperture
with a diameter of 100 mm, composed of 256 elements with
a diameter of 3.5 mm and a central operating frequency of
1 MHz. These parameters have been chosen to be able to
validate the concept of Large Divegent Elements Spare Arrays
(LDSEA) despite fabrication constraints. Further details on
the design and characterization of the LDSEA probe will be
provided in a future study.

The designed prototype presented in Fig.1 (b) is based
on a Fermat’s spiral array (Fig.1 (a)), a distribution known
for low sidelobes level [10]. However, for ultrasound images typically displayed with a dynamic range of 60 dB, sidelobes-related artifacts remain visible. Fig.1 (c,d) shows XZ slices of experimental 3D DAS beamformed volumes obtained with the designed probe. Fig.1 (c) corresponds to an image of a wire phantom placed at a depth of 150 mm, which can be considered as a Point Spread Function (PSF) image. This image highlights the presence of high sidelobes level caused by the sparse array. In the case of a tissue-mimicking phantom, these sidelobes cause energy to leak from the surrounding tissue speckle into anechoic inclusion, drastically reducing the image contrast (Fig.1 (d)). This highlights the low contrast drawback associated with 2D sparse arrays. In the next section, we will present our newly developed reconstruction algorithm, which is designed to reconstruct high-quality 3D images from synthetic transmit aperture (STA) RF data acquired using the designed sparse array prototype.

## B. Region-specific beamforming: RSB-Net

We introduce RSB-Net, a novel region-specific beamforming approach for reconstructing high-quality 3D STA images. This approach involves applying different beamformers to different regions of the image: adaptive beamformers are used in areas dominated by sidelobes-related artifacts, while the DAS beamformer is used in regions where preserving speckle texture is important. Given that the beamforming task is formulated as a decision-making problem, Deep Reinforcement Learning (DRL) is an appropriate method to address this challenge [52]. DRL is a branch of artificial intelligence that focuses on decision-making problems and has achieved superhuman results in several research areas [53]–[55]. The Agent (the decision-maker) is trained in an episodic manner to interact with the environment by performing actions that maximizes the obtained rewards. Each episode represents a sequence of interactions between the agent and the environment. Each episode consists of multiple time steps and during each of them the agent observes the current state of the environment, selects an action based on its policy, and executes the action in the environment. This results in a new state and a reward reflecting the immediate outcome. The environment resets at the beginning of each episode with in an initial state (in our case, the reset corresponds to changing the input image to reconstruct). In RSB-Net, a DRL agent is trained in this manner to select the optimal beamformer for each image pixel.

In RSB-Net, we adapt the Advantage Actor-Critic (A2C) algorithm, a well-established DRL approach that has demonstrated strong performance across a variety of reinforcement learning tasks [56]. A2C has recently been applied to several pixel-wise denoising tasks, yielding high quality results [57]–[59]. This algorithm belongs to the actor-critic reinforcement learning family, which integrates both a value-based and policy-based learning methods. Therefore, the A2C algorithm employs two neural networks: the policy network ($\theta_p$), referred to as the actor, and the value network ($\theta_v$), known as the critic. During each training episode, at each time step $t$, both networks receive the current state $s^{(t)}$ of the environment as input. The actor is responsible for selecting actions by learning

a policy $\pi(s^{(t)})$, which represents a probability distribution over possible actions given the state $s^{(t)}$. Simultaneously, the critic evaluates the action taken by the actor by estimating the value function $V(s^{(t)}, \theta_v)$, which quantifies the expected future reward from the current state $s^{(t)}$. At each time step, the discounted reward is calculated as follows:

$$R^{(t)} = \sum_{j=0}^{k-1} \gamma^j r^{(t+j)} + \gamma^k V(s^{(t+k)}, \theta_v) \quad (1)$$

Where $t$ represents the current time step and the starting point of the calculation of the discounted reward $R^{(t)}$; $j$ is the index used for summing the discounted rewards over $k$ future time steps, running from 0 to $k-1$, with each $r^{(t+j)}$ corresponding to the immediate reward (calculated using the reward function) received at the $j$-th step after time $t$. $\gamma$ is the discount factor, where $0 \le \gamma \le 1$, and it determines the relative importance of future rewards compared to immediate rewards. $\theta_v$ represents the parameters of the value network.

From equation 1, the discounted reward calculation at each time step accounts for both immediate and future rewards, enabling the agent to optimize cumulative returns over time. The immediate reward, $r^{(t)}$, represents the direct feedback from the environment based on the actions taken at time step $t$. Reinforcement learning, however, seeks to maximize the cumulative reward, which necessitates incorporating not only immediate but also future rewards. The discount factor $\gamma$ determines the relative importance of future rewards, where smaller values of $\gamma$ prioritize immediate rewards, while larger values encourage consideration of longer-term benefits. The term $\gamma^j r^{(t+j)}$ discounts the reward obtained $j$ steps into the future, reflecting the reduced influence of distant rewards. By summing over $k$ time steps, the agent incorporates a finite horizon that allows it to respond to immediate feedback from the environment while considering the potential future returns. Beyond this horizon, the term $\gamma^k V(s^{(t+k)}, \theta_v)$ provides an estimate of the cumulative reward expected from state $s^{(t+k)}$, allowing the agent to plan for outcomes extending beyond the $k$-step window.

The policy network, parameterized by $\theta_p$, outputs the policy $\pi(s^{(t)})$, which defines the probability distribution over the action space $\mathcal{A}$ given the current state $s^{(t)}$. This distribution is used to select actions $a^{(t)}$ to be executed in the environment. The output dimension of the policy network matches the size of the action space, ensuring that every possible action in $\mathcal{A}$ is assigned a probability.

The training of the policy network involves updating its parameters, $\theta_p$. This is achieved using the policy gradient method, where the gradient of the parameters is calculated as:

$$d\theta_p = -\nabla_{\theta_p} \log \pi(a^{(t)}|s^{(t)})A(a^{(t)}, s^{(t)}) \quad (2)$$

$d\theta_p$ computes the gradient of the log-probability of selecting action $a^{(t)}$ under the current policy $\pi$. It adjusts the parameters to increase the likelihood of actions that lead to better rewards and decrease the likelihood of less favorable actions. The advantage function $A(a^{(t)}, s^{(t)}) = R^{(t)} - V(s^{(t)}, \theta_v)$, quantifies how much better or worse the action $a^{(t)}$, taken in state $s^{(t)}$,
<figure><img src="image_4.png" /><figcaption>Fig. 2. Overview of RSB-Net architecture: it corresponds to an FCN adaptation of A2C algorithm with Policy Network and Value Network. Both Policy Network and Value Network share the same encoder.</figcaption></figure>

is compared to the expected value of the state, $V(s^{(t)}, \theta_v)$. A positive advantage ($A(a^{(t)}, s^{(t)}) > 0$) indicates the action performed better than expected, reinforcing the behavior, while a negative advantage attenuate it.

The gradient for the critic network, parameterized by $\theta_v$, is calculated to minimize the difference between the discounted reward and the estimated value of a state, ensuring that the critic network accurately evaluates the expected reward. The gradient is computed as:

$$d\theta_v = \nabla_{\theta_v} (R^{(t)} - V(s^{(t)}, \theta_v))^2 \quad (3)$$

Here, the term $(R^{(t)} - V(s^{(t)}, \theta_v))^2$ represents the squared error between the discounted reward $R^{(t)}$ and the value function $V(s^{(t)}, \theta_v)$, and the objective is to minimize this squared error, improving the ability of the critic network to evaluate a state under the current policy.

The role of the critic is to provide accurate value estimates, which are essential for the advantage calculation ($A(a^{(t)}, s^{(t)}) = R^{(t)} - V(s^{(t)}, \theta_v)$) used by the policy network. By minimizing the squared error in equation 3, the critic ensures that the advantage function reflects the true performance of actions, ultimately guiding the policy network towards better decision-making. This gradient-based update process is fundamental to actor-critic methods, where the actor and the critic work together. The actor updates the policy to choose better actions, while the critic evaluates how good those actions are in terms of cumulative rewards. The combination ensures efficient learning, guiding the agent to maximize long-term rewards.

### C. RSB-Net environment and action space

To select a different beamformer for each pixel in the 3D image, in RSB-Net, each pixel $i$ ($i = 1, 2, ..., N$) is represented by an individual A2C agent resulting in a multi-agent deep reinforcement learning paradigm. The DRL environment contains the received STA RF data but RSB-Net agents observe the image beamformed with the actions

taken at each time step $t$. This image represents the observed state $s^{(t)}$ of the environment. During each training episode, in time step $t = 0$, the environment is reset with new training data and since the agents have not yet taken any actions, the observed state $s_i^{(0)}$ of the environment is initialized as a DAS beamformed image. Consequently, each one of the N agents has a policy $\pi_i(s_i^{(t)})$ that represents the probability of taking action $a_i^{(t)}$ (a chosen beamformer for the $i$-th pixel) by observing the current state $s_i^{(t)}$ of the $i$th pixel. Subsequently, at each time step $t$, each agent selects a different beamformer to reconstruct the image pixel that provides the highest reward calculated based on a reward function (see Section D), resulting in an overall image with improved quality compared to the previous time step. This approach applies the A2C algorithm at the pixel level, allowing for pixel-specific beamforming rather than selecting a single action (beamformer) for the entire image. However, representing each pixel in the image with an agent requires training $N$ independent A2C agents, each using distinct value and policy networks with separately updated parameters. Although this approach enables each agent to independently optimize its policy and value estimation, it becomes computationally prohibitive, particularly for high-dimensional data such as 3D ultrasound images.

In this study, to avoid the computational cost of training $N$ independent A2C agents, we adopt a strategy similar to that used in image denoising [57]–[59]. We propose using a fully convolutional network (FCN) with a UNet inspired architecture instead of $N$ separate networks. By extending the A2C algorithm into a fully convolutional form, each pixel in the image is represented by an agent and all agents can be trained simultaneously while sharing parameters. The architecture of the proposed RSB-Net policy and value networks, representing a FCN form of the A2C algorithm, is depicted in Fig. 2. One of the advantages of this network is that it enables each agent to leverage a large receptive field, as afforded by CNN architectures [60]. This allows the policy and value networks to capture spatial dependencies among neighboring agents. As a consequence, each agent takes
actions not only based on its own policy and value but also by incorporating information from neighboring agents into its decision-making process. This CNN representation extends the multi-agent deep reinforcement learning paradigm by treating all agents as a single 3D convolutional system (for 3D images), rather than as independent learning agents, each with its own network [57]. Additionally, the UNet-like architecture with skip connections has demonstrated significant success in image-related tasks and ultrasound image reconstruction [43], [61]. This representation allows for efficient feature extraction at multiple scales while preserving fine details, which can enhance decision-making.

In RSB-Net, the policy and value networks share the same encoder, which extracts and encodes the input state of the environment before being fed into the policy and value decoders. The final layer of the policy network applies a softmax function across the feature map to generate a probability distribution over possible actions, from which an action is categorically sampled for each pixel. The value network, on the other hand, outputs the value function. The parameters of these two networks are updated using Equations (2) and (3) via backpropagation. It is worth noting that the shared encoder architecture was used to reduce the computational cost of having two separate encoders. This design ensures that the policy and value networks learn from a shared feature space avoiding redundant representations.

At each training episode, in each time step t, the environment provide the RSB-Net with a beamformed image representing state $s_i^{(t)}$ after taking actions $a_i^{(t-1)}$. This image is processed through an encoder-decoder architecture (Fig. 2), which constitutes both the policy network and the value network, sharing the same encoder. Each agent samples a different discrete actions from its current policy, corresponding to a different beamforming algorithm that will be applied to the image pixel. The action is then executed in the environment, resulting in a next state $s_i^{(t+1)}$ of the environment and reward calculated using the reward function. At the end of the episode, the policy and the value networks of the agents are updated and the environment is reset with new data for the next episode.

In this study, to account for the high dimensionality of 3D ultrasound imaging, which results in a large number of pixels to be beamformed, the discrete action space is limited to three beamforming choices: 0—DAS, 1—CF, and 2—DMAS-CF (a hybrid approach where the DMAS image is weighted by CF to manage computational complexity. The DAS beamformer was included as a natural choice due to its ability to effectively preserve speckle characteristics. CF was included for its capability to suppress sidelobe artifacts at a relatively low computational cost. Although variations such as GCF and PCF have demonstrated improved speckle preservation in cases of normal arrays, for 2D sparse arrays with strong sidelobe artifacts, they still face challenges in fully preserving speckle characteristics when high sidelobes artifacts are present. This is particularly evident in 2D sparse arrays, where intense sidelobe artifacts make it difficult for these beamformers to distinguish mainlobe signals from sidelobe signals. For similar reasons, DMAS can produce results similar to CF in images obtained with sparse arrays; therefore, we

combined them by weighting the DMAS image with CF to further reduce artifacts.

## D. Reward function

The primary objective of RSB-Net is to reconstruct a 3D ultrasound image free from artifacts associated with the high sidelobe levels inherent to 2D sparse arrays while preserving the quality of the speckle. During training, at each time step, the image reconstructed by RSB-Net agents by applying the actions (beamformers) is compared to a high-quality target image. The reconstruction process of this high quality image will be described in section E. The reward for each agent is determined by the distance between the reconstructed pixel and its corresponding version in the target image. To consider the high dynamic range characteristic of ultrasound images, this distance is computed as :

$$D_i^{(t)} = \|g_{\alpha}(s_i^{(t)}) - g_{\alpha}(\hat{s}_i^{(t)})\|_2 \quad (4)$$

where $s_i^{(t)}$ represents the reconstructed pixel, and $\hat{s}_i^{(t)}$ is the corresponding pixel in the target image. The function $g_{\alpha}$ is a signed compression function used to account for the dynamic range, defined as:

$$g_{\alpha}(s_i) = \mathrm{sign}(s_i) \log_{\alpha} \left( \frac{\alpha}{\max(\alpha, |s_i|)} \right) \quad (5)$$

with $\alpha \in (0, 1)$. In this study the value of alpha was chosen to take into account a dynamic range of 70dB. For further details on this function, please refer to [43]. The reward for each pixel at each time step is then calculated as:

$$r_i^{(t)} = D_i^{(t)} - D_i^{(t+1)} \quad (6)$$

This reward represents the difference between the distances $D_i^{(t)}$ and $D_i^{(t+1)}$ after taking action $a_i^{(t)}$. The reward value increases as $D_i^{(t+1)}$ decreases, indicating that RSB-Net effectively minimizes the distance between the pixels of the reconstructed image and the target image, making them increasingly similar.

## E. Dataset generation

To train the RSB-Net, raw STA RF datasets (the DRL environment) were generated using a realistic FEM-Based ultrasound simulation tool developed by IMASONIC to simulate the designed sparse array prototype in Fig. 1(b) [62]. A RF dataset of 12,000 numerical phantoms was simulated in 3D, containing various phantoms with different echogenicity levels. Tissue-mimicking phantoms were created with random shapes and varying reflection levels. These phantoms, each with dimensions of 70 × 70 × 70 mm, had a scatterer density of 10 per resolution cell to ensure simulations with a fully developed speckle. Additionally, phantoms with anechoic inclusions and wire phantoms of the same size were generated. The dataset was split into 9,600 phantoms for training, 1,200 for validation, and 1,200 for testing.

To generate a high-quality target image for reward calculation during training, a 2D matrix array of the same size as
<figure><img src="image_5.png" /><figcaption>Fig. 3. A diagram highlighting the process of generating STA RF Data and the high quality target image used for rewards calculation during training. The DAS beamformed image is given as an example of the low image quality reconstructed with DAS beamformer.</figcaption></figure>

the designed sparse array would ideally be used. However, to simulate images without sidelobe artifacts, the interelement spacing must be less than λ, resulting in a matrix array with 131x131 elements. Simulating thousands of phantoms with such an array is computationally too time consuming for the processing resources available. A more practical approach leverages the PSF of the designed 2D sparse array prototype. The PSF consists of a main lobe, defining the scatterer positions, surrounded by high-level sidelobes, which degrade the contrast and image quality of the sparse array. By cropping the PSF to remove the sidelobes, an ideal high-quality PSF can be obtained. The high-quality target image is then generated by convolving this ideal PSF with the scatterer distribution, as illustrated in Fig. 3. This 3D convolution operation is computationally inexpensive, enabling the creation of high-quality 3D ultrasound images.

## F. Training details

In DRL, the training is not done in epochs like traditional supervised learning. Instead, the agents are trained over episodes, where each episode consists of multiple time steps during which the agents interacted with the environment. In this work, by monitoring the convergence of the network policy learning of RSB-Net, we set the number of episodes to 99,000 to ensure a sufficient number of episodes for exploration across the diverse phantoms in the environment. During each episode, the number of time steps was set to 10 as a balance between training time and the stability of the learning process. At the beginning of each episode, the environment is reset using data from a different phantom. After each time step, the image is beamformed using the beamformer selected for each pixel by each agent, which then represents the next state of the environment. At the end of each episode, the policy and value network parameters are updated based on equations (2) and (3) using the experiences (states, actions, values, policies, rewards) collected throughout the episode. The γ parameter, which represents the discount factor used in the calculation of the discounted reward (as shown in Equation 1), was set to 0.99 as suggested in [56]. The environment is then reset with

data from a different phantom, and a new training episode begins.

When trained with limited environment diversity, DRL can be prone to overfitting and fail to generalize. In our case, overfitting was mitigated by incorporating two strategies. First, the training environment was made diverse by using data from very diverse phantoms, which allowed the agents to experience a broader range of states and actions. Second, we used entropy regularization to encourage exploration by preventing the policy from becoming too deterministic too early in the training process. Details about entropy regularization can be found in [56], [63].

Training was conducted using the ADAM optimizer [64], with an initial learning rate of 0.001. The learning rate was scheduled to decrease over time according to the formula $(1 - \frac{\text{Episode}}{\text{Max episodes}})^{0.9}$. The entire training process took approximately one week on an NVIDIA Quadro RTX 6000 GPU. It is worth noting that after training, reward calculations are no longer required. The user can beamform images in a single time step, significantly reducing computational complexity during inference.

The inference of the proposed model was conducted on an RTX 3060 GPU, where the network outputs its decision in 0.32 seconds. Based on these decisions, the 3D images with a resolution of 128 × 128 × 128 pixels are beamformed using DAS, CF, and DMAS-CF, implemented on the GPU via NVIDIA CUDA programming, which takes on average 26 seconds depending on the selected actions. Thus, the total time required for the proposed method is around 26.32 seconds. Further optimization of the GPU implementation of the beamformers is expected to improve the computational efficiency of the method.

## G. Image Quality Metrics

The performance of images reconstructed using RSB-Net was assessed using standard ultrasound image quality metrics, specifically contrast and resolution. These metrics are widely used to evaluate image clarity and artifact suppression in ultrasound imaging.

**Resolution Assessment:** Resolution was evaluated using a wire phantom in both experimental and simulation results. It was quantified by measuring the −6 dB width of the point spread function (PSF) along both the axial and lateral directions in a cross-section of a wire target.

**Contrast Assessment:** Contrast was assessed using a cyst phantom and quantified using two metrics: the contrast ratio (CR) and the generalized contrast-to-noise ratio (gCNR) [65].

The CR, which quantifies the brightness difference between the anechoic inclusion and the surrounding speckle region, was computed as:

$$CR_{dB} = 20 \log_{10} \left( \frac{\mu_{cyst}}{\mu_{bck}} \right) \quad (7)$$

where $\mu_{cyst}$ and $\mu_{bck}$ are the mean image intensities inside the anechoic inclusion and the surrounding background, respectively.

Similarly, the gCNR was computed to quantify the separability between the intensity distributions of the anechoic
<figure><img src="image_6.png" /><figcaption>Fig. 4. A comparison of the simulated reconstructed images for a wires phantom and a tissue-mimicking phantom with, DAS, CF, DMAS-CF, Target image and RSB-Net with the action map representing the beamformer chosen for each pixel by RSB-Net agents.</figcaption></figure>

inclusion and the speckle region. It was defined as the overlap between their corresponding histograms $f_1$ and $f_2$, given by [65]:

$$gCNR = 1 - \int \min(f_1(x), f_2(x)) dx \quad (8)$$

where $f_1(x)$ and $f_2(x)$ represent the histograms of pixel intensities within the anechoic inclusion and the surrounding speckle region, respectively.

### III. RESULTS

#### A. Simulated data

Fig. 4 presents a comparison of the beamformed images obtained using DAS, CF, DMAS-CF, the target image, and RSB-Net for a simulated wire phantom with wires oriented in different spatial directions, as well as for a tissue-mimicking phantom. The first row displays XZ slices, while the second row shows XY slices extracted from the 3D reconstructed volumes, all visualized with a dynamic range of 60 dB. A visual assessment reveals the superior image quality achieved by the proposed RSB-Net in comparison to DAS, CF, and DMAS-CF for both phantoms.

For the wire phantom, the DAS beamformed image exhibits strong sidelobe artifacts caused by the sparse array, which significantly degrades image quality. While the images obtained with CF and DMAS-CF demonstrate improved sidelobe suppression, the wires appear more sparse and less defined. In contrast, the RSB-Net effectively suppresses sidelobe-related artifacts, producing images with quality comparable to the target image. Notably, the texture around the wires is well preserved, resembling the texture observed in DAS beamforming.

For the simulated tissue-mimicking phantom, the DAS beamformed image exhibits substantially lower quality, especially in regions containing anechoic inclusions. In these regions, strong sidelobe artifacts obscure the inclusions, making them difficult to differentiate from the surrounding speckle. Images obtained using CF and DMAS-CF demonstrate effective sidelobes suppression, however, this comes at the cost of

degraded speckle texture, leading to darker and less realistic speckle patterns. These results highlight the limitations of CF and DMAS-CF in preserving speckle texture. In contrast, the RSB-Net reconstruction achieves a significant improvement in image quality, closely matching that of the target image. The anechoic inclusions are clearly visible, with artifacts effectively removed while preserving the quality of the speckle.

Fig. 4 (sixth column) also illustrates the actions taken by the RSB-Net agents during the image reconstruction process. The action maps highlight the beamformer selected for each pixel in the image. Given the effectiveness of adaptive beamformers, such as CF and DMAS-CF, in suppressing artifacts caused by the high sidelobe levels of sparse arrays, these beamformers were specifically applied to regions where no wires are present in the wire phantom and within the anechoic inclusions of the tissue-mimicking phantom. This approach effectively suppresses sidelobe-related artifacts in these regions. To preserve the speckle characteristics, the RSB-Net agents applied DAS beamforming in regions where wires are present and in areas dominated by speckle. This selective beamforming strategy enables the RSB-Net to balance artifact suppression with speckle preservation, thereby achieving high-quality image reconstruction.

<table><caption>TABLE I
COMPARISON OF THE IMAGE QUALITY METRICS OF SIMULATED DATA BETWEEN DAS, CF, DMAS-CF TARGET IMAGE AND THE RSB-NET CALCULATED ON THE VOLUMES OF THE IMAGES IN FIG. 4.</caption><thead><tr><th></th><th>δ<sub>ax</sub>(mm)</th><th>δ<sub>lat</sub>(mm)</th><th>CR (dB)</th><th>gCNR</th></tr></thead><tbody><tr><td>DAS</td><td>1.30</td><td>2.20</td><td>-8.60</td><td>0.33</td></tr><tr><td>CF</td><td>1.12</td><td>1.70</td><td>-28.51</td><td>0.64</td></tr><tr><td>DMAS-CF</td><td>1.12</td><td>1.51</td><td>-39.19</td><td>0.76</td></tr><tr><td>Target Image</td><td>1.20</td><td>2.30</td><td>-71.78</td><td>0.99</td></tr><tr><td>RSB-Net</td><td>1.20</td><td>2.10</td><td>-61.54</td><td>0.98</td></tr></tbody></table>

The visual assessments of the images presented in Fig. 4 are supported by the quantitative metrics summarized in Table I. Resolution was evaluated in terms of axial and lateral - 6dB widths measured on a wire. The RSB-Net demonstrated
<figure><img src="image_7.png" /><figcaption>**Fig. 5.** Comparison of a 3D volumetric rendering with a dynamic range of 60dB of the simulated wires phantom (first row) and tissue-mimicking phantom (second row) beamformed volumes with DAS and RSB-Net.</figcaption></figure>

slightly improved resolutions (1.20 mm and 2.10 mm) compared to DAS (1.30 mm and 2.20 mm). While CF and DMAS-CF yielded better resolution values, these improvements came at the cost of degraded speckle quality. The contrast ratio (CR) and generalized contrast-to-noise ratio (gCNR) further highlight the superiority of RSB-Net relative to DAS beamforming. RSB-Net achieved CR and gCNR values that closely matched those of the target image, with a CR of -61.54 dB and a gCNR of 0.98. In comparison, the DAS beamformer produced a CR of -8.60 dB and a gCNR of 0.33.

Fig. 5 presents a comparison of 3D renderings of the entire beamformed volume containing six wires and a simulated tissue-mimicking phantom, using DAS and RSB-Net beamforming, both displayed with a dynamic range of 60 dB. The DAS 3D volume is significantly affected by high sidelobe-related artifacts, which degrade the visualization quality of individual wires and obscure the anechoic inclusions. In contrast, the 3D volume reconstructed using RSB-Net effectively suppresses these artifacts, resulting in a clear and accurate 3D representation of the volume. This improvement significantly enhances the overall visualization quality.

## B. Experimental results

To further validate the generalizability of RSB-Net on real-world ultrasound data, a comparative analysis was conducted with the DAS, CF and DMAS-CF using experimental data. The experimental measurements were performed using the Verasonics Vantage 256 system (Verasonics Inc., Redmond, WA) with the designed 256-elements spiral sparse array operating at a central frequency of 1 MHz (Fig. 1). Two types of experimental phantoms were used to acquire STA RF data:

a nylon wire phantom positioned at a depth of approximately 140 mm in water and an agar-based tissue-mimicking phantom containing a cylindrical inclusion with a 30 mm diameter placed at a depth of 100 mm.

Fig. 6 presents the experimentally reconstructed images of the wire phantom (first row) and the tissue-mimicking phantom (second row). The images correspond to XZ slices extracted from the 3D reconstructed volume and are displayed with a dynamic range of 60 dB. Consistent with the results observed in the simulated data, the experimental DAS beamformed images exhibit strong sidelobe-related artifacts. Although the wire remains visible, the sidelobe artifacts are particularly pronounced in the lateral direction, significantly degrading image quality. In the tissue-mimicking phantom, these artifacts result in energy leakage, leading to the appearance of unwanted echoes within the cylindrical anechoic inclusion. Similar to the simulated data, CF and DMAS-CF beamformers effectively reduce sidelobe artifacts. However, this reduction comes at the cost of degraded speckle quality, resulting in reconstructed images with a darker speckle texture.

In contrast, the images reconstructed using RSB-Net exhibit superior quality. Sidelobe artifacts are effectively suppressed, allowing for clear visualization of the wire without significant artifacts. For the tissue-mimicking phantom, the RSB-Net reconstruct images with the cylindrical anechoic inclusion clearly visible, while the surrounding speckle texture is well preserved. Additionally, the action maps illustrating the decisions made by RSB-Net agents during the reconstruction process are shown in Fig. 6 (fifth column). These action maps indicate that DAS beamforming was applied to regions dominated by speckle to preserve speckle texture, while DMAS-CF was employed in regions where sidelobe artifact suppression was required.

<table><caption>**TABLE II**
COMPARISON OF THE IMAGE QUALITY METRICS OF EXPERIMENTAL DATA BETWEEN DAS, CF, DMAS-CF TARGET IMAGE AND THE RSB-NET CALCULATED ON THE VOLUMES OF THE IMAGES IN FIG. 6.</caption><thead><tr><th></th><th>δ<sub>ax</sub>(mm)</th><th>δ<sub>lat</sub>(mm)</th><th>CR (dB)</th><th>gCNR</th></tr></thead><tbody><tr><td>DAS</td><td>1.19</td><td>2.24</td><td>-6.66</td><td>0.34</td></tr><tr><td>CF</td><td>1.03</td><td>1.52</td><td>-19.78</td><td>0.53</td></tr><tr><td>DMAS-CF</td><td>1.03</td><td>1.37</td><td>-26.24</td><td>0.62</td></tr><tr><td>RSB-Net</td><td>1.04</td><td>2.20</td><td>-75.71</td><td>0.98</td></tr></tbody></table>

Table II presents the quality metrics calculated from the experimental beamformed volumes using DAS, CF, DMAS-CF, and RSB-Net. The resolution measurements are consistent with those observed in the simulated wire phantom, with RSB-Net demonstrating slightly improved lateral and axial resolutions compared to the DAS beamformer. While CF and DMAS-CF achieved better resolution values for the wire phantom, these improvements came at the cost of reduced image quality for the tissue-mimicking phantom, as reflected in their gCNR values of 0.53 and 0.62, respectively. For the tissue-mimicking phantom, RSB-Net effectively suppresses sidelobe artifacts within the anechoic inclusion, leading to significant enhancements in CR and gCNR. Specifically, RSB-Net achieved a CR improvement of 51.05 dB and a gCNR increase of 0.64 compared to the DAS beamformer.
<figure><img src="image_8.png" /><figcaption>Fig. 6. A comparison of the experimental reconstructed images for a wire phantom and a tissue-mimicking phantom with, DAS, CF, DMAS-CF and RSB-Net with the action map representing the beamformer chosen for each pixel by RSB-Net agents.</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>Fig. 7. Comparison of a 3D volumetric rendering with a dynamic range of 60dB of the experimental wire phantom (first row) and tissue-mimicking phantom (second row) beamformed volumes with DAS and RSB-Net.</figcaption></figure>

Fig. 7 presents a comparison of the 3D renderings of the beamformed experimental wire volume and tissue-mimicking volume. The RSB-Net effectively suppresses sidelobe artifacts compared to the DAS beamformer, resulting in a clear 3D representation of the wire. Several scatterers corresponding to floating particles in water are distinctly visible in the RSB-Net volume, whereas they are barely discernible in the DAS volumetric rendering due to sidelobe artifacts. RSB-Net successfully identifies and preserves these particles in the reconstructed volumes. RSB-Net enables the high-quality 3D rendering of the entire tissue-mimicking beamformed volume, where the cylindrical anechoic inclusion is clearly visible. In contrast, in the DAS beamformer rendering, the anechoic in-

clusion is difficult to distinguish from the surrounding speckle.

## IV. DISCUSSION

The primary objective of this study was to develop a novel reconstruction algorithm to reconstruct high quality 3D images with a designed 2D sprase array prototype from STA imaging RF data. The prototype array was fabricated following a simulation study that compared various potential technologies. These simulations were based on the concept of angular impulse responses [62], calculated using a FEM model. FEM modeling provides an accurate representation of the array elements, enabling realistic 3D ultrasound image simulations.

Although the design of the array is based on Fermat's spiral distribution which is known for its good performance, especially in terms of sidelobe levels, the image quality reconstructed using the DAS beamformer still suffers from sidelobe artifacts (Fig. 1 and Fig. 7). These artifacts can reduce the accuracy of medical diagnostics in clinical applications. To reconstruct high quality 3D images with the array, the RSB-Net was developed.

RSB-Net is a deep reinforcement learning-based model that combines the performance of adaptive beamformers with the DAS beamformer within a learning framework to improve image quality. Adaptive beamformers are known for their effectiveness in reducing off-axis artifacts due to high sidelobe levels; however, they can degrade speckle texture quality and introduce dark regions in images. RSB-Net, based on the A2C deep reinforcement learning algorithm, was trained to selectively apply adaptive beamformers in regions where sidelobe artifacts are dominant and the DAS beamformer in speckle regions to preserve is texture quality.

By implementing A2C as a fully convolutional network, RSB-Net treats each pixel of the image as an independent agent that selects actions (a beamformer) based on a policy designed to maximize its rewards. The rewards are directly tied to the quality of the reconstructed image pixel—the better the
quality, the higher the rewards. The reward curve represented in Fig. 8 highlights the advantage of FCN representation of A2C, the agents obtain a good policy that allows the maximization of their rewards after just few thousands of episodes. This is due to the fact that the FCN representation allows the agent to share all the parameters, allowing them to gain N (number of pixels) experiences at a single pass [57].

<figure><img src="image_10.png" /><figcaption>Fig. 8. The average reward obtained by the agents during 99000 training episodes.</figcaption></figure>

In this study, the action space was limited to three beamformers: DAS, CF, and DMAS-CF. The CF and DMAS-CF adaptive beamformers were chosen for their effectiveness in reducing off-axis signals without imposing a high computational burden, which helps accelerate the training process. The realism of the FEM simulation tool enabled the generation of a high-quality dataset of RF data using the STA imaging sequence, which was used to train RSB-Net and subsequently apply it to real experimental data. In addition to its realism, the GPU implementation of the simulator enabled the rapid simulation of 3D tissue-mimicking phantoms with a large number of scatterers. Since the prototype probe was fixed to a 10 cm aperture, the tissue-mimicking phantoms were generated to fit under the probe, with dimensions of 70x70x70 mm. In this configuration, the maximum angle between a phantom scatterer and an element of the probe was 30°. This limited the number of AIR loaded into the GPU memory to 30 for transmission and 30 for reception, reducing computational overhead due to memory access [62].

The accuracy of the simulation tool is confirmed by comparing the various metrics calculated in both the simulation and experimental studies (Tables I and II). For example, for the DAS beamforming, the simulation and experimental results showed similar axial and lateral resolution on the wire phantom, with values around 1.2 mm and 2.2 mm, respectively. Additionally, the contrast values were also approximately close, with gCNR values of 0.33 and 0.34 for the simulated and experimental tissue-mimicking phantoms, respectively.

In Fig. 4, we evaluated the performance of RSB-Net on simulated data, including a wire phantom and a tissue-mimicking phantom with random shapes and varying echogenicity. The results demonstrated that the agents successfully learned a policy that enables them to apply the DMAS-CF beamformer in regions with high sidelobes-related artifacts. DMAS-CF is particularly effective in reducing strong artifacts by combining

the advantages of CF in suppressing incoherent signals related to sidelobes and the high sensitivity of DMAS to these signals. The reward function (Equation 6), which is based on the distance from the target ideal image, guided the agents to preserve speckle quality by applying the DAS beamformer in regions where the speckle is dominating. Additionally, to maintain smooth transitions between speckle regions and anechoic inclusions, the CF beamformer was applied to take into account the shape of the PSF that was used in the generation of target images.

These simulation results were corroborated by the experimental results shown in Fig. 6. Despite not being trained on an experimental dataset, RSB-Net successfully reconstructed the 3D images of the experimental wire phantom and tissue-mimicking phantom with the same accuracy as the simulated data, as validated by the metrics in Tables II.

By applying different beamformers to distinct regions of the image, RSB-Net effectively reconstructed high-quality ultrasound images using the designed sparse array prototype. This strategy leverages the advantages of adaptive beamformers in suppressing off-axis signals while avoiding their drawbacks, such as altering speckle quality. These weaknesses are evident in Fig. 4 and Fig. 6. The DAS images are dominated by sidelobe artifacts, which are removed by CF and DMAS-CF; however, these images exhibit a darker appearance. The RSB-Net images successfully combines the benefits of CF and DMAS-CF in reducing sidelobe artifacts with the DAS beamformer and its ability to preserve speckle quality.

During training, each episode is divided into 10 time steps. Breaking episodes into time steps enables agents to evaluate the impact of their actions over time within a consistent environment. This approach is critical for temporal credit assignment, as it allows the agents to learn how each action contributes to both immediate rewards and future outcomes, thereby benefiting the calculation of discounted rewards [66]. Additionally, dividing episodes into time steps enables incremental calculation of rewards and value updates, enhancing the accuracy of advantage estimation and, consequently, improving policy updates. As a consequence, once an agent has been trained, its policy network has effectively learned to map states to optimal actions. During inference, an agent can operate in a single time step, using the current state $s_i$ as input to predict the action $a_i$ and update the environment accordingly. The primary advantage of using a single time step during inference is efficiency, as it eliminates the need to iterate through multiple steps while maintaining performance.

This efficiency is demonstrated in the case of RSB-Net, which, after training, can process STA RF data in a single time step to beamform a high-quality image. Fig. 9 illustrates a comparison between the inference results of an RSB-Net trained for only 500 episodes (RSB-Net-500) and one trained for the full 99,000 episodes (RSB-Net-99000). For visualization purposes, only images from time steps 1, 4, 7, and 10 are shown. In the case of the RSB-Net trained for 500 episodes, inference results show that the agents have not yet learned the optimal policy to choose the best beamformers to obtain a high quality image and it improves incrementally over successive time steps. Conversely, after 99,000 episodes of training, the
<figure><img src="image_11.png" /><figcaption>Fig. 9. Comparison of inference results for RSB-Net-500, trained for 500 episodes, and RSB-Net-99000, trained for 99,000 episodes. The results are shown for time steps 1, 4, 7, and 10, highlighting that after training is complete, a single time step is sufficient for high-quality image generation.</figcaption></figure>

policy is fully optimized, enabling the agents to select the best actions and produce a high-quality image starting from the first time step, with no significant changes in quality in subsequent steps. This demonstrates that a single time step is sufficient for beamforming a high-quality image after training.

In this study, we used DRL as a decision-making algorithm to select the most suitable beamformer for each pixel in the reconstructed image. Supervised learning approaches could potentially achieve the same task but they require ground truth action labels, which are difficult to obtain in this context. DRL, despite being more challenging to train, allows the agent to learn optimal beamformer selection without the need for predefined ground truth actions. A possible alternative to having label actions with supervised learning is to use the Gumbel-Softmax reparametrization technique [67]. Since the action space in our method is discrete (i.e., selecting a beamformer) which involve sampling a categorical distribution, which is non-differentiable [67], [68]. The Gumbel-Softmax reparametrization addresses this issue by replacing discrete action sampling with a differentiable softmax function perturbed by Gumbel noise, thereby allowing gradients to flow back through the selection process via backpropagation as done in some applications [69], [70]. However, a key limitation of the Gumbel-Softmax approach is that it restricts the action space to only differentiable beamformers. In contrast, reinforcement learning offers greater flexibility by allowing the incorporation of non-differentiable beamformers and more complex decision-making strategies.

From both the simulation and experimental results, RSB-Net consistently applied the DMAS-CF beamformer to regions containing anechoic inclusions, successfully removing sidelobe-related artifacts. However, applying DMAS-CF in anechoic regions such as blood vessels could be problematic in ultrasound Doppler applications, as this beamformer is nonlinear and affects the accuracy of velocity estimators, such as the autocorrelation estimator used in Doppler flow imaging. Nevertheless, RSB-Net was specifically designed for 3D anatomical B-mode imaging in order to provide high-quality 3D anatomical images. By improving 3D image quality obtained with sparse arrays, RSB-Net reduces the dependence

on the ability of the clinician to mentally reconstruct 3D structures from conventional 2D ultrasound images, potentially enhancing the accuracy of lesions and cyst identification. In future improvements, RSB-Net can be adapted for Doppler imaging applications by expanding the action space to include beamformers better suited for preserving the linear phase shifts of blood flow signals.

While RSB-Net shows strong performance in reconstructing images with high contrast by removing sidelobe artifacts, particularly in anechoic inclusions, the contrast between hyper-echoic inclusions and surrounding speckle remains almost the same as that produced by DAS, which is still acceptable as the hyperechoic inclusion is clearly visible against the surrounding speckle, with no dark regions present (Fig. 10). Improving contrast in these situations will be considered in future work by proposing a new reward function to train RSB-Net. Additionally, RSB-Net produces small shape distortions, especially at the edges between anechoic inclusions and speckle regions (Fig. 10). This issue arises from the failure to apply the CF beamformer in these regions. Future studies will focus on incorporating more adaptive beamformers, such as MV beamformer and PCF, to help reduce these effects.

In this study, to generate high-quality 3D target images for training RSB-Net, we used a computationally efficient approach based on the convolution of an ideal system PSF with the scatterers distribution. This method allows the generation of high-quality training images without the need for computationally expensive simulations using a large matrix array with thousands of elements with the same aperture size of the designed sparse array with 256 elements. We assumed a spatially invariant PSF in our dataset generation; however, in real-world scenarios, the PSF is spatially variant due to variations in acoustic wave propagation and focusing across the field of view. While using a single, shift-invariant PSF provides a fast approach to generate high-quality 3D images without sidelobes artifacts, this simplification may impact the agents ability to accurately learn an optimal beamformer that allows them to match the target image, particularly in peripheral regions of the array. This limitation could contribute to shape distortions drawbacks observed in certain cases, an aspect currently under investigation as part of future improvements. Accurately accounting for this spatial variation would require computing a unique PSF for each scatterers or pixel in the image, which is computationally costly. A potential solution, as proposed in [71], involves interpolating multiple precomputed reference PSFs across different regions of the imaging field. By using an inverse distance-weighting interpolation technique [72], the PSF at any position within the field of view can be approximated with reduced computational cost.

Although there is still room for improvement, in this work we proposed a new reconstruction algorithm capable of producing high-quality 3D images with a designed sparse array, which has been tested on both simulated and experimental data. Building on these achievements, future research will focus on addressing the limitation of the proposed algorithms as well as exploring and testing the method in more complex imaging scenarios. A key area of investigation will be the
<figure><img src="image_12.png" /><figcaption>Fig. 10. XZ slices from a simulated tissue-mimicking volume beamformed with DAS and RSB-Net highlighting the drawbacks of the proposed method.</figcaption></figure>

application of the developed techniques in in vivo imaging, to
evaluate their robustness and effectiveness in real biological
environments, where factors such as tissue heterogeneity and
physiological noise can impact the imaging performance.

V. CONCLUSION

In this study, we introduced RSB-Net, a new reconstruction
algorithm designed to reconstruct ultrasound images from STA
RF data acquired using a developed 2D sparse array prototype.
RSB-Net is a deep reinforcement learning-based model de-
signed to reconstruct high-quality images by selectively apply-
ing adaptive beamformers in regions with dominant sidelobe
artifacts, while using the DAS beamformer in speckle regions
to maintain texture quality.

Training of RSB-Net was performed on a large, simu-
lated dataset generated by a GPU-accelerated FEM ultrasound
simulation tool. The application of RSB-Net demonstrated a
significant reduction in sidelobe-related artifacts while pre-
serving the speckle texture. On the simulated dataset, RSB-
Net showed an improvement in CR and gCNR by 52.81 dB
and 0.65, respectively, compared to the DAS beamformer. In
an experimental tissue-mimicking phantom, RSB-Net achieved
gains of 51.01 dB in CR and 0.64 in gCNR compared to the
DAS beamformer.

While RSB-Net shows strong performance in enhancing
the contrast of reconstructed images, it does have some
limitations, such as challenges in improving the contrast of
hyperechoic inclusions and shape distortions observed at the
edges of anechoic inclusions. These issues are currently under
investigation. Overall, the successful application of RSB-
Net to both simulated and experimental data acquired with
the developed sparse array prototype validates its robustness
and effectiveness, making it a promising approach for other
acquisition and imaging systems.

ACKNOWLEDGMENT

The authors would like to thank the members of the ANR
SPARTECHUS Project for the suggestions and feedbacks on
our work. A special thanks to IMASONIC (Voray sur l'Ognon,
France) for providing the sparse probe.

REFERENCES

[1] F. Padilla, M. A. Roubidoux, C. Paramagul, S. P. Sinha, M. M. Goodsitt, G. L. Le Carpentier, H.-P. Chan, L. M. Hadjiiski, J. B. Fowlkes, A. D. Joe *et al.*, "Breast mass characterization using 3-dimensional automated ultrasound as an adjunct to digital breast tomosynthesis: A pilot study," *Journal of Ultrasound in Medicine*, vol. 32, no. 1, pp. 93–104, 2013.

[2] D. Barbosa, T. Dietenbeck, B. Heyde, H. Houle, D. Friboulet, J. D'hooge, and O. Bernard, "Fast and fully automatic 3-d echocardiographic segmentation using b-spline explicit active surfaces: Feasibility study and validation in a clinical setting," *Ultrasound in medicine & biology*, vol. 39, no. 1, pp. 89–101, 2013.

[3] S. W. Smith and H. G. Pavy, "High-speed Ultrasound Volumetric Imaging System- Part I: Transducer Design and Beam Steering."

[4] O. von Ramm, S. Smith, and H. Pavy, "High-speed ultrasound volumetric imaging system. ii. parallel processing and image display," *IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control*, vol. 38, no. 2, pp. 109–115, 1991.

[5] D. Turnbull and F. Foster, "Beam steering with pulsed two-dimensional transducer arrays," *IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control*, vol. 38, no. 4, pp. 320–333, 1991.

[6] A. Ramalli, E. Boni, E. Roux, H. Liebgott, and P. Tortoli, "Design, implementation, and medical applications of 2-d ultrasound sparse arrays," *IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control*, vol. 69, no. 10, pp. 2739–2755, 2022.

[7] B. Diarra, M. Robini, P. Tortoli, C. Cachard, and H. Liebgott, "Design of optimal 2-d nongrid sparse arrays for medical ultrasound," *IEEE Transactions on Biomedical Engineering*, vol. 60, no. 11, pp. 3093–3102, 2013.

[8] E. Roux, B. Diarra, M. Robini, C. Cachard, P. Tortoli, and H. Liebgott, "Realistic acoustic simulation of 2-d probe elements in simulated annealing sparse array optimization," in *2014 IEEE International Ultrasonics Symposium*. IEEE, 2014, pp. 2125–2128.

[9] E. Roux, A. Ramalli, P. Tortoli, C. Cachard, M. C. Robini, and H. Liebgott, "2-d ultrasound sparse arrays multidepth radiation optimization using simulated annealing and spiral-array inspired energy functions," *IEEE Transactions on ultrasonics, ferroelectrics, and frequency control*, vol. 63, no. 12, pp. 2138–2149, 2016.

[10] A. Ramalli, E. Boni, A. S. Savoia, and P. Tortoli, "Density-tapered spiral arrays for ultrasound 3-d imaging," *IEEE Transactions on ultrasonics, ferroelectrics, and frequency control*, vol. 62, no. 8, pp. 1580–1588, 2015.

[11] L. Wei, E. Boni, A. Ramalli, F. Fool, E. Noothout, A. F. Van Der Steen, M. D. Verweij, P. Tortoli, N. De Jong, and H. J. Vos, "Sparse 2-d pzt-on-pcb arrays with density tapering," *IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control*, vol. 69, no. 10, pp. 2798–2809, 2022.

[12] R. Maffett, E. Boni, A. J. Chee, B. Y. Yiu, A. S. Savoia, A. Ramalli, P. Tortoli, and C. Alfred, "Unfocused field analysis of a density-tapered spiral array for high-volume-rate 3-d ultrasound imaging," *IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control*, vol. 69, no. 10, pp. 2810–2822, 2022.

[13] A. Fenster, 3D Ultrasound: Devices, Applications, and Algorithms. CRC Press, 2023.

[14] M. O'Donnell, "Coded excitation system for improving the penetration of real-time phased-array imaging systems," *IEEE transactions on ultrasonics, ferroelectrics, and frequency control*, vol. 39, no. 3, pp. 341–351, 1992.

[15] T. X. Misaridis, K. Gammelmark, C. H. Jørgensen, N. Lindberg, A. H. Thomsen, M. H. Pedersen, and J. A. Jensen, "Potential of coded excitation in medical ultrasound imaging," *Ultrasonics*, vol. 38, no. 1-8, pp. 183–189, 2000.

[16] M. Tamraoui, H. Liebgott, and E. Roux, "Complete complementary coded excitation scheme for snr improvement of 2d sparse array ultrasound imaging," *IEEE Transactions on Biomedical Engineering*, vol. 71, no. 3, pp. 1043–1055, 2023.

[17] M. Tamraoui, E. Roux, and H. Liebgott, "Complementary coded multiplane wave sequence for snr increase in 3d ultrafast power doppler ultrasound imaging," in *2024 IEEE Ultrasonics, Ferroelectrics, and Frequency Control Joint Symposium (UFFC-JS)*. IEEE, 2024, pp. 1–4.

[18] H. Favre, M. Pernot, M. Tanter, and C. Papadacci, "Boosting transducer matrix sensitivity for 3d large field ultrasound localization microscopy using a multi-lens diffracting layer: a simulation study," *Physics in Medicine & Biology*, vol. 67, no. 8, p. 085009, 2022.

[19] ——, "Transcranial 3d ultrasound localization microscopy using a large element matrix array with a multi-lens diffracting layer: an in vitro study," *Physics in Medicine & Biology*, vol. 68, no. 7, p. 075003, 2023.
[20] R. S. Cobbold, *Foundations of biomedical ultrasound*. Oxford university press, 2006.

[21] E. Roux, A. Ramalli, H. Liebgott, C. Cachard, M. C. Robini, and P. Tortoli, "Wideband 2-d array design optimization with fabrication constraints for 3-d us imaging," *IEEE transactions on ultrasonics, ferroelectrics, and frequency control*, vol. 64, no. 1, pp. 108-125, 2016.

[22] V. Perrot, M. Policetti, F. Varray, and D. Garcia, "So you think you can das? a viewpoint on delay-and-sum beamforming," *Ultrasonics*, vol. 111, p. 106309, 2021.

[23] Y. Wang, Y. Wang, M. Liu, Z. Lan, C. Zheng, and H. Peng, "Minimum variance beamforming combined with covariance matrix-based adaptive weighting for medical ultrasound imaging," *BioMedical Engineering OnLine*, vol. 21, no. 1, p. 40, 2022.

[24] B. M. Asl and A. Mahlooijfar, "Contrast enhancement and robustness improvement of adaptive ultrasound imaging using forward-backward minimum variance beamforming," *IEEE transactions on ultrasonics, ferroelectrics, and frequency control*, vol. 58, no. 4, pp. 858-867, 2011.

[25] M. F. Schiffner and G. Schmitz, "Frequency-dependent f-number suppresses grating lobes and improves the lateral resolution in coherent plane-wave compounding," *IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control*, 2023.

[26] J. F. Synnevag, A. Austeng, and S. Holm, "Adaptive beamforming applied to medical ultrasound imaging," *IEEE transactions on ultrasonics, ferroelectrics, and frequency control*, vol. 54, no. 8, pp. 1606-1613, 2007.

[27] J.-F. Synnevag, A. Austeng, and S. Holm, "Benefits of minimum-variance beamforming in medical ultrasound imaging," *IEEE transactions on ultrasonics, ferroelectrics, and frequency control*, vol. 56, no. 9, pp. 1868-1879, 2009.

[28] R. Mallart and M. Fink, "Adaptive focusing in scattering media through sound-speed inhomogeneities: The van cittert zernike approach and focusing criterion," *The Journal of the Acoustical Society of America*, vol. 96, no. 6, pp. 3721-3732, 1994.

[29] K. Hollman, K. Rigby, and M. O'donnell, "Coherence factor of speckle from a multi-row probe," in *1999 IEEE Ultrasonics Symposium. Proceedings. International Symposium (Cat. No. 99CH37027)*, vol. 2. IEEE, 1999, pp. 1257-1260.

[30] J. Camacho, M. Parrilla, and C. Fritsch, "Phase coherence imaging," *IEEE transactions on ultrasonics, ferroelectrics, and frequency control*, vol. 56, no. 5, pp. 958-974, 2009.

[31] P.-C. Li and M.-L. Li, "Adaptive imaging using the generalized coherence factor," *IEEE transactions on ultrasonics, ferroelectrics, and frequency control*, vol. 50, no. 2, pp. 128-141, 2003.

[32] G. Matrone, A. S. Savoia, G. Caliano, and G. Magenes, "The delay multiply and sum beamforming algorithm in ultrasound b-mode medical imaging," *IEEE transactions on medical imaging*, vol. 34, no. 4, pp. 940-949, 2014.

[33] G. Matrone, A. Ramalli, A. S. Savoia, P. Tortoli, and G. Magenes, "High frame-rate, high resolution ultrasound imaging with multi-line transmission and filtered-delay multiply and sum beamforming," *IEEE transactions on medical imaging*, vol. 36, no. 2, pp. 478-486, 2016.

[34] G. Matrone, A. S. Savoia, G. Caliano, and G. Magenes, "Depth-of-field enhancement in filtered-delay multiply and sum beamformed images using synthetic aperture focusing," *Ultrasonics*, vol. 75, pp. 216-225, 2017.

[35] F. Prieur, O. M. H. Rindal, and A. Austeng, "Signal coherence and image amplitude with the filtered delay multiply and sum beamformer," *IEEE transactions on ultrasonics, ferroelectrics, and frequency control*, vol. 65, no. 7, pp. 1133-1140, 2018.

[36] A. Ramalli, M. Scaringella, G. Matrone, A. Dallai, E. Boni, A. S. Savoia, L. Bassi, G. E. Hine, and P. Tortoli, "High dynamic range ultrasound imaging with real-time filtered-delay multiply and sum beamforming," in *2017 IEEE International Ultrasonics Symposium (IUS)*. IEEE, 2017, pp. 1-4.

[37] F. Prieur, O. M. H. Rindal, S. Holm, and A. Austeng, "Influence of the delay-multiply-and-sum beamformer on the ultrasound image amplitude," in *2017 IEEE International Ultrasonics Symposium (IUS)*. IEEE, 2017, pp. 1-4.

[38] O. M. H. Rindal, A. Rodriguez-Molares, and A. Austeng, "The dark region artifact in adaptive ultrasound beamforming," in *2017 IEEE International Ultrasonics Symposium (IUS)*. IEEE, 2017, pp. 1-4.

[39] K. Esmailian and B. M. Asl, "Correlation-based modified delay-multiply-and-sum beamforming applied to medical ultrasound imaging," *Computer methods and programs in biomedicine*, vol. 226, p. 107171, 2022.

[40] M. Xu, X. Yang, M. Ding, and M. Yuchi, "Spatio-temporally smoothed coherence factor for ultrasound imaging [correspondence]," *IEEE transactions on ultrasonics, ferroelectrics, and frequency control*, vol. 61, no. 1, pp. 182-190, 2014.

[41] B. Luijten, N. Chennakeshava, Y. C. Eldar, M. Mischi, and R. J. van Sloun, "Ultrasound signal processing: from models to deep learning," *Ultrasound in medicine & biology*, vol. 49, no. 3, pp. 677-698, 2023.

[42] M. Gasse, F. Millioz, E. Roux, D. Garcia, H. Liebgott, and D. Friboulet, "High-quality plane wave compounding using convolutional neural networks," *IEEE transactions on ultrasonics, ferroelectrics, and frequency control*, vol. 64, no. 10, pp. 1637-1639, 2017.

[43] D. Perdios, M. Vonlanthen, F. Martinez, M. Arditi, and J.-P. Thiran, "Cnn-based image reconstruction method for ultrafast ultrasound imaging," *IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control*, vol. 69, no. 4, pp. 1154-1168, 2021.

[44] B. W. Ossenkoppele, B. Luijten, D. Bera, N. De Jong, M. D. Verweij, and R. J. Van Sloun, "Improving lateral resolution in 3-d imaging with micro-beamforming through adaptive beamforming by deep learning," *Ultrasound in Medicine & Biology*, vol. 49, no. 1, pp. 237-255, 2023.

[45] A. A. Nair, K. N. Washington, T. D. Tran, A. Reiter, and M. A. L. Bell, "Deep learning to obtain simultaneous image and segmentation outputs from a single input of raw ultrasound channel data," *IEEE transactions on ultrasonics, ferroelectrics, and frequency control*, vol. 67, no. 12, pp. 2493-2509, 2020.

[46] Z. Li, A. Wiacek, and M. A. L. Bell, "Beamforming with deep learning from single plane wave rf data," in *2020 IEEE International Ultrasonics Symposium (IUS)*. IEEE, 2020, pp. 1-4.

[47] Y. Chen, J. Liu, J. Luo, and X. Luo, "A data-driven approach for high frame rate synthetic transmit aperture ultrasound imaging," in *Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part VI 24*. Springer, 2021, pp. 412-420.

[48] A. C. Luchies and B. C. Byram, "Deep neural networks for ultrasound beamforming," *IEEE transactions on medical imaging*, vol. 37, no. 9, pp. 2010-2021, 2018.

[49] B. Luijten, R. Cohen, F. J. De Bruijn, H. A. Schmeitz, M. Mischi, Y. C. Eldar, and R. J. Van Sloun, "Adaptive ultrasound beamforming using deep learning," *IEEE Transactions on Medical Imaging*, vol. 39, no. 12, pp. 3967-3978, 2020.

[50] F. Vignon, J. S. Shin, F. C. Meral, I. Apostolakis, S.-W. Huang, and J.-L. Robert, "Resolution improvement with a fully convolutional neural network applied to aligned per-channel data," in *2020 IEEE International Ultrasonics Symposium (IUS)*. IEEE, 2020, pp. 1-4.

[51] S. Bhadra, V. A. Kelkar, F. J. Brooks, and M. A. Anastasio, "On hallucinations in tomographic image reconstruction," *IEEE Transactions on Medical Imaging*, vol. 40, no. 11, pp. 3249-3260, 2021.

[52] M. Lapan, *Deep Reinforcement Learning Hands-On: Apply modern RL methods, with deep Q-networks, value iteration, policy gradients, TRPO, AlphaGo Zero and more*. Packt Publishing Ltd, 2018.

[53] D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot et al., "Mastering the game of go with deep neural networks and tree search," *nature*, vol. 529, no. 7587, pp. 484-489, 2016.

[54] E. F. Morales, R. Murrieta-Cid, I. Becerra, and M. A. Esquivel-Basaldua, "A survey on deep learning and deep reinforcement learning in robotics with a tutorial on deep reinforcement learning," *Intelligent Service Robotics*, vol. 14, no. 5, pp. 773-805, 2021.

[55] B. R. Kiran, I. Sobh, V. Talpaert, P. Mannion, A. A. Al Sallab, S. Yوغamani, and P. Pérez, "Deep reinforcement learning for autonomous driving: A survey," *IEEE Transactions on Intelligent Transportation Systems*, vol. 23, no. 6, pp. 4909-4926, 2021.

[56] V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu, "Asynchronous methods for deep reinforcement learning," in *International conference on machine learning*. PMLR, 2016, pp. 1928-1937.

[57] R. Furuta, N. Inoue, and T. Yamasaki, "Pixelrl: Fully convolutional network with reinforcement learning for image processing," *IEEE Transactions on Multimedia*, vol. 22, no. 7, pp. 1704-1719, 2019.

[58] Y. Guo, Y. Gao, B. Hu, X. Qian, and D. Liang, "Cmid: Crossmodal image denoising via pixel-wise deep reinforcement learning," *Sensors*, vol. 24, no. 1, p. 42, 2023.

[59] P. Jarosik, M. Lewandowski, Z. Klimonda, and M. Byra, "Pixel-wise deep reinforcement learning approach for ultrasound image denoising," in *2021 IEEE International Ultrasonics Symposium (IUS)*. IEEE, 2021, pp. 1-4.

[60] W. Luo, Y. Li, R. Urtasun, and R. Zemel, "Understanding the effective receptive field in deep convolutional neural networks," *Advances in neural information processing systems*, vol. 29, 2016.
[61] O. Ronneberger, P. Fischer, and T. Brox, "U-net: Convolutional networks for biomedical image segmentation," in *Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18*. Springer, 2015, pp. 234–241.

[62] J. Jacquet, M. Tamraoui, P. Kauffmann, J. Guey, E. Roux, B. Nicolas, and H. Liebgott, "Integrating finite-element model of probe element in gpu accelerated ultrasound image simulation," in *2023 IEEE International Ultrasonics Symposium (IUS)*. IEEE, 2023, pp. 1–4.

[63] S. Han and Y. Sung, "Diversity actor-critic: Sample-aware entropy regularization for sample-efficient exploration," in *International Conference on Machine Learning*. PMLR, 2021, pp. 4018–4029.

[64] D. P. Kingma, "Adam: A method for stochastic optimization," *arXiv preprint arXiv:1412.6980*, 2014.

[65] A. Rodriguez-Molares, O. M. H. Rindal, J. D'hooge, S.-E. Måsøy, A. Austeng, M. A. L. Bell, and H. Torp, "The generalized contrast-to-noise ratio: A formal definition for lesion detectability," *IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control*, vol. 67, no. 4, pp. 745–759, 2019.

[66] H.-R. Pan, N. Gürtler, A. Neitz, and B. Schölkopf, "Direct advantage estimation," *Advances in Neural Information Processing Systems*, vol. 35, pp. 11869–11880, 2022.

[67] E. Jang, S. Gu, and B. Poole, "Categorical reparameterization with gumbel-softmax," *arXiv preprint arXiv:1611.01144*, 2016.

[68] M. Paulus, D. Choi, D. Tarlow, A. Krause, and C. J. Maddison, "Gradient estimation with stochastic softmax tricks," *Advances in Neural Information Processing Systems*, vol. 33, pp. 5691–5704, 2020.

[69] T. Strypsteen and A. Bertrand, "End-to-end learnable eeg channel selection for deep neural networks with gumbel-softmax," *Journal of Neural Engineering*, vol. 18, no. 4, p. 0460a9, 2021.

[70] T. Pang, S. Zhao, J. Han, S. Zhang, L. Guo, and T. Liu, "Gumbel-softmax based neural architecture search for hierarchical brain networks decomposition," *Medical image analysis*, vol. 82, p. 102570, 2022.

[71] N. Laroche, S. Bourguignon, J. Idier, E. Carcreff, and A. Duclos, "Fast non-stationary deconvolution of ultrasonic beamformed images for nondestructive testing," *IEEE Transactions on Computational Imaging*, vol. 7, pp. 935–947, 2021.

[72] M. Gentile, F. Courbin, and G. Meylan, "Interpolating point spread function anisotropy," *Astronomy & Astrophysics*, vol. 549, p. A1, 2013.