Beamforming with Free Energy Principle under
Hierarchical Codebook
Tatsuya Otoshi

Masayuki Murata

Graduate School of Economics
Osaka University
Osaka, Japant-otoshi@econ.osaka-u.ac.jp

Graduate School of Information Science and Technology
Osaka University
Osaka, Japan
murata@ist.osaka-u.ac.jp

Abstract—Beamforming plays a crucial role in enhancing
the performance of wireless communication systems. However,achieving optimal beamforming entails a trade-off between exploration and exploitation, where the system needs to balance the exploration of different beam directions with the exploitation of thebest available beam. Motivated by the exploration-exploitation
trade-off, we propose the FEP method, which leverages hierarchical modeling and adaptive beam switching to optimize thistrade-off. Through simulations in a dynamic environment, weevaluate the performance of the FEP method in terms of expected
free energy minimization and signal-to-interference-plus-noise
ratio (SINR) maximization. The results demonstrate that the
FEP method effectively maintains high SINR levels throughadaptive beam direction selection, reﬂecting efﬁcient exploitation.
Comparative analysis with other beamforming methods further
highlights the superior performance of the FEP method in termsof average signal quality and stability.
Index Terms—Beamforming, Beyond 5G, Active Inference,
Free Energy Principle, Hierarchical Codebook

I. I NTRODUCTION
Massive MIMO technology has shown great potential forenhancing coverage and improving spatial utilization in wireless communication systems [1]. However, beamforming, a
key technique in Massive MIMO, faces challenges in accurate
channel state estimation. This challenge arises due to theneed for feedback from the terminals, resulting in round-trip
delays and reduced throughput due to observation-based radiotransmissions. Particularly in Massive MIMO systems witha large number of antennas, the dimensionality of channel
state estimation increases, leading to increased overhead for
estimation.
In the context of 5G, beamforming techniques utilizing
codebooks have been standardized to alleviate the aforementioned challenges. By predeﬁning a ﬁxed number of candidate
beams, the overhead associated with beam search can be
reduced. However, as the number of antennas increases, thenumber of candidate beams also escalates, giving rise toconcerns regarding search overhead. To tackle this issue, hierarchical codebooks have been proposed [2], [3], which enable
efﬁcient beam search by employing broader-angle beams in
the upper layers and narrower-angle beams in the lower layers.
This approach gradually reﬁnes the beam selection from coarse
granularity to ﬁne granularity.

Nonetheless, existing methods for hierarchical beamforming
assume a static environment and depend on sequential searchof the hierarchical codebooks from top to bottom in thespatial domain. In practice, it is necessary to simultaneously
explore both higher and lower hierarchical levels, adapting tochanging conditions. These challenges arise due to a systemmodel that separates the stages of exploratory observationbased estimation and utilization-based data transfer.
Recently, the Free Energy Principle (FEP) has garneredsigniﬁcant attention in the realm of neuroscience due toits potential applications in various domains [4], [5]. FEP
introduces a unifying framework that tackles estimation anddecision-making by framing them as processes aimed at minimizing free energy. This notion of free energy encompasses
both epistemic and pragmatic values, fostering a harmonious
equilibrium between exploration and exploitation during theestimation and decision-making phases. When integrated into
the context of beamforming, FEP stands out for its capability
to dynamically and intelligently adjust the equilibrium between exploration and exploitation in response to evolving
environmental conditions. Also, FEP has the capability toiteratively update the internal model itself. This aspect stands
in contrast to conventional methods (e.g. neural network [6])that require time-consuming retraining. FEP, on the other hand,enables swift adaptation to changing circumstances while
simultaneously enhancing the accuracy of the internal model.
As a result, FEP not only aligns with the principles of adaptive
beamforming but also offers a novel approach to optimizing
the performance of beamforming techniques amidst changing
contexts.
In this paper, we present a novel approach that utilizes Free
Energy Principle (FEP) agents for beam selection at each levelof the hierarchical codebook. These FEP agents make use of
feedback information from the terminals to select beams attheir respective levels. The FEP agents in the upper and lower
levels simultaneously make independent decisions, but theyexchange state information to ensure coherent beam selection
across levels. To account for the disparities in spatial andtemporal scales between the upper and lower levels, we set
longer intervals for updating the state in the higher levels.
This allows for the reﬂection of the differences in spatial andtemporal scales between the upper and lower levels.

Through comprehensive evaluations, we demonstrate theeffectiveness of our proposed method. We show that ourapproach consistently maintains a higher SINR(Signal-toInference-pluse-Noise Ratio) compared to conventional beamtraining techniques and policy-based approaches in multiarmed bandit problems.
In Section II, we review the existing beamforming studies
and FEP as related studies. Section III describes the assumed
beamforming system model. In Section IV, we propose abeamforming method based on FEP. In Section V, we present
the evaluation results of the proposed method by simulations.
Finally, in Section VI, we summarize and discuss future issues.
II. R ELATED W ORK
A. Beamforming with Hierarchical Codebook
Previous literature has focused on the design of hierarchicalcodebooks. In many cases, hierarchical codebooks are constructed by dividing the angular domain from coarse to ﬁne
granularity, primarily targeting the direction of beams [7], [8].
The mainstream approach in hierarchical codebooks is todivide the beam directions hierarchically and store beam
vectors corresponding to each direction in the codebook, asshown in [7], [8]. The upper levels contain candidate angles
with coarse granularity, while the lower levels have candidate
angles with ﬁne granularity. In many cases, at each level, twoalternative angles are selected based on the angle determinedin the upper level as the center. However, when performing
the search from the topmost to the bottommost level, theminimum number of beam vectors to be explored is achieved
through a three-partition search, which has led to the proposalof codebooks for ternary tree search [9].
When the receiving terminal is far enough, the directionof beam emission from each antenna aligns with the beam
direction. However, when the receiving terminal is nearby,
adjustments in the emission direction for each antenna arenecessary based on the distance. Therefore, methods that consider both the beam direction and the distance and constructcodebooks by hierarchically partitioning the two variables
have also been proposed [2], [3]. The upper levels havecandidate angles and distances with coarse granularity, while
the lower levels have candidate angles and distances with ﬁne
granularity. In this case, at each level, a four-choice selection
is performed by dividing the range of angles and distances
into two parts for each variable.
In beam determination, as feedback is obtained for eachbeam, increasing the number of beams results in a larger
overhead for feedback. When using hierarchical codebooks,
the overhead is reduced by limiting the number of beams to
be explored.
Beamforming using hierarchical codebooks often involves
beam training, where beams are determined. Beam training is
performed by exploring and determining the beam with thehighest SINR before communication. The search is conductedfrom coarse beams at the upper levels to ﬁne beams at
the lower levels, following the hierarchical structure of thecodebook.

Thus, conventional beamforming using hierarchical codebooks assumes ofﬂine beam exploration. On the other hand,active inference allows for online beamforming, which performs communication and exploration simultaneously.
Although not using hierarchical codebooks, a method that
performs online beam training known as the UCB (Upper
Conﬁdence Bound) algorithm is proposed in the literature [10].
This method narrows down the candidate beams using terminal
location information and performs online beam exploration asa multi-armed bandit problem.
The beam exploration selects the top Btr beams to explore
per round, where the beams with the highest UCB values are
explored, as deﬁned by:√
2 log(n)Xi+(1)
UCBi =
Ti
Ti
Here, n represents the round number, Xi is the cumulative
reward for the i-th beam, and Ti is the number of times thei-th beam has been explored. The reward is 1 if the beam has
the highest received signal strength and 0 otherwise.
The ﬁrst term on the right-hand side represents the selection criterion for beams with higher average rewards, while
the second term ensures the selection of beams with fewer
explorations relative to the number of rounds. This allows
for beam exploration considering the trade-off between exploration and exploitation. However, this method does notconsider the hierarchical structure of the codebook, and thelocation information of the terminal is obtained from thesatellite, which is limited to the case where the satellite isin the line of sight.
Our FEP-based beamforming also allows for the use of
these conventional codebooks, and in this paper, we employ astandard binary partition codebook based on angles. However,conventional hierarchical codebooks assume a unidirectional
search from the topmost to the bottommost level and may not
be optimal when performing simultaneous searches between
upper and lower levels as required by the attention mechanismof the FEP.
B. Free Energy Principle
FEP is used to explain a wide range of phenomena related toneural activity and behavior in the brain and provide a common
underlying principle for biological systems [4], [5]. According
to FEP, biological systems are programmed to minimize freeenergy. In order for biological systems to achieve goals such as
cognition, perception, and behavior, they must process sensory
information appropriately and minimize free energy.
In FEP, the brain is assumed to compute an approximate
distribution q(x) of the posterior distribution p(x|s) of thestate x of the environment under the observed information s.
The free energy is deﬁned by∑∑
F (q, s) =q(x)E(x, s) − (−q(x) log q(x)) (2)x=x

DKL (q(x)||p(x|s)) − log p(s)(3)

where E(x, s) = − log p(x, s) is the logarithm of the simultaneous distribution of x and s, and DKL is the KL
divergence. − log p(s) is called the Shannon surprise. Equation(2) corresponds to the Helmholtz free energy F = U − T S instatistical mechanics if the temperature T is neglected (T = 1).
III. S YSTEM M ODELThis section describes the model of beamforming assumed
in this paper. We also describe the time scales of the observation and control loops assumed.
A. Beamforming
The base station uses M antennas to transmit radio waves,which are transformed by spatial characteristics to reach N
antennas at the receiving terminal (UE: User Equipment). Notethat N = 1 in the cases of [6], [11]. The spatial characteristics
are represented by the matrix H(t), where the element Hij
represents the channel coefﬁcient for the transmission fromantenna i to antenna j.
The transmitter estimates H(t) and performs beamforming
by transforming the phase and amplitude of the transmitted
radio wave for each antenna according to H(t), thereby
improving the efﬁciency of radio propagation. Beamformingis speciﬁed by the beam vector w(t) and the transmit power
P (t), and the following relationship is established between thetransmit signal x(t) and the receive signal y(t):√(4)y(t) = P (t)H(t)w(t) ◦ x(t),where ◦ denotes the element-wise (Hadamard) product.
In this case, the signal-to-interference-plus-noise ratio(SINR) is expressed as follows:
γ(t) =

P (t)|H(t)w(t)|2,I(t) + σ 2 (t)(5)

where I(t) is the strength of the interfering radio wave andσ 2 (t) is the strength of the noise.
The transmission rate is also estimated using SINR as
follows:
C(t) = log(1 + γ(t)),(6)
where C(t) is the transmission rate.
B. Hierarchical Codebook
In beamforming, it is necessary to estimate the channel
state H(t) using feedback information and optimize the beamforming vector w(t). However, estimating the channel state
incurs signiﬁcant overhead and is not practical. Therefore, anapproach is used where beamforming vectors are selected froma pre-deﬁned set of candidates called a codebook. However,in scenarios with a large number of antennas, such as Massive
MIMO, the codebook itself becomes extensive, requiring the
introduction of hierarchy to improve the efﬁciency of thesearch process.
In a hierarchical codebook, there are Kl beamforming(l)vectors wk for each layer l. The value of Kl increases forlower layers, and in the case of constructing a codebook withbinary splits, Kl = 2l .

(l)When wal is selected at layer l, at the next layer (l + 1),(l)beamforming vectors similar to wal are further divided andselected. In the case of binary splits, the selection is made
(l+1)
(l+1)
between w2al −1 and w2al .
The hierarchical codebook used in the paper is based on
the one described in the literature [7]. The n-th beamforming
vector at layer l is as follows:w(l)n =a(N, Ω) =), 0N −2l )(a(2l , −1 + 2n−1
2l1
√ (exp(iπ0Ω), · · · , exp(iπ(N − 1)Ω))
N

(7)
(8)

At layer l, the beam is formed using 2l antennas, and as lincreases, it allows for ﬁner beam conﬁgurations.
C. Observable Information on Base Station
Estimating the channel coefﬁcients is essential in beamforming, but since the coefﬁcients themselves cannot be directly
observed, they must be estimated from the signals at thereceiving UE, which are observable.
There are two main methods for measuring signal information at the receiving UE: using the synchronization signal (SS)and using Channel State Information (CSI).
In the case of SS, the signal strength and SINR are measured
using the synchronization signal received in the SS/PBCH
block, which is a set of SS and PBCH (Physical Broadcast
Channel), as a reference signal. SS/PBCH is transmitted
periodically (e.g., every two frames) for initial access, so there
is no need to transmit a new reference signal.
When using CSI, a reference signal is sent for observation,and the received signal is used to measure signal strength and
SINR.
In either case, information from the receiving side is fed
back to the transmitting side, enabling the transmitter toestimate the downstream channel coefﬁcients.
In limited situations, such as when the same frequency isused in time division, it is also possible to use the uplinkreference signal (SRS: Sounding Reference Signal) for measurement due to the symmetry of the uplink and downlink
channel coefﬁcients. In this case, the base station observes thesignal directly, so there is no need for round-trip feedback.
We assume that the signal strength and SINR are observed
using these methods in the proposed method.
D. Timescales
Beamforming itself can be switched at a high speed. In fact,
for initial access, beam sweeping switches between two different beams within one slot. Speciﬁcally, during beam sweeping,
the SS/PBCH block consists of four symbols transmitted by
the base station, with a two-symbol gap between them [12].
Observation information, on the other hand, is acquired
on a slightly slower timescale than beamforming switching.
Feedback of the receiver’s SINR to the base station requires atleast a round-trip time between the UE and the base station to
reply to the transmitted signal from the base station. Therefore,
observations are collected at intervals of multiple symbols
or multiple slots. Speciﬁcally, in the case of SS, channelinformation is measured using SS/PBCH blocks [13], which
are transmitted periodically every 20 ms, so observations are
obtained every 20 ms. When using CSI, channel information
can be obtained in a shorter time, and according to theliterature [14], the feedback delay for CSI is a minimum of 1
ms. Since beamforming can be switched within a symbol, it
is assumed that control switching can be done fast enough forthe timescale of observation.
IV. B EAMFORMING WITH ACTIVE I NFERENCE
We propose beamforming using the FEP to dynamically
perform beamforming while mitigating search overhead. Our
method leverages limited feedback information to estimate theenvironmental state and determine the beamforming vector
accordingly based on FEP. It automatically balances the beam
selection for exploration and optimization for data transmission, assuming feedback is obtained only for the selected
beam. Additionally, to facilitate hierarchical decision-making
with a hierarchical codebook, we deploy agents at each layer
to make decisions and enable state exchange among them.
A. Hierarchical Decision with FEP
Hierarchical decision-making based on the Free Energy
Principle (FEP) is employed in our proposed method. In thecontext of FEP, hierarchical active inference through predictive
coding has been advocated [15]. Predictive coding suggeststhat the brain predicts input and acts to minimize the prediction
error, which is the discrepancy between the predicted input
and the actual input. In this framework, each layer’s internal
state serves as input for the higher layer, and the lower layer
predicts the lower-level states, forming a multi-level hierarchyof active inference. It is assumed that higher layers operate on
longer timescales than lower layers.
Considering layer l = 1 as the topmost layer and l = L as(l)the bottommost layer, let ot represent the input at layer l,(l)and st denote the internal state. For information ﬂow fromlower to higher layers, the lower layer’s estimated internal state
distribution is provided as input:
(l)

(l+1)

ot = P (st

|ol+1
0:t )

(9)

Conversely, for information ﬂow from higher to lower layers,
the lower layer’s internal state prediction serves as the priordistribution:
(l+1)

P (st
(l)

) =

(l)

ôt

(10)

(l)

where ôt = EP (o(l) )|s(l) ) [P (ot )] represents the expectedttvalue of observations under the posterior distribution of thehigher layer.
(l)(l)
At each layer, the policy π (l) = (a1 , · · · , aT ), representing a sequence of selected actions, is determined to minimize
the expected free energy, guided by the Free Energy Principle:
(l)

(l)

G(o1:T , s1:T , π (l) ) =(l)

EQ [log Q(s1:T , π (l) )(l)(l)− log P̃ (o1:T , s1:T , π (l) )](11)

Here, Q represents the approximate posterior distribution, andP̃ represents the target distribution.
In the work [15], the timescales of the lower and higher
layers are predetermined. After a ﬁxed number of iterations(20 times) of estimation and action at the lower layer, one
iteration of estimation and action at the higher layer occurs.
Then, the lower layer repeats the estimation and action basedon the new prior obtained from the higher layer. Similarly,
in our paper, we assume that the higher layer operates on aslower time cycle Th compared to the lower layer.
B. Deﬁnition of FEP Agents
We design agents corresponding to each hierarchical codebook layer in the proposed method. Speciﬁcally, we deﬁne theagent’s observations, states, actions, and preference distribution.
a) Observations: The base station can observe the transmission rate C(t) to each UE based on feedback from theterminals. Additionally, except for the lowest layer, higher
layers can also observe the states estimated by the lower layer.
Therefore, the observations at layer l correspond to(l+1) (l+1)o(l) (t) = (P (st|o0:t ), C(t)). The lowest layer only
observes feedback from the terminals, o(L) (t) = C(t).
As feedback, we assume that the receiving side’s SINR
is measured using CSI or SS, and the transmission rate isestimated from the SINR.
However, the higher the layer, the longer the time interval
for obtaining observations. The speciﬁc time intervals andthe impact of temporal smoothing of observations will be
investigated through simulations.
b) States: Based on the observed transmission rates C(t)and their own beamforming information, the base station estimates the channel coefﬁcients H u (t). Therefore, the system
state corresponds to s(l) (t) = H(t).
State estimation is performed to minimize the error between
the predicted distribution P (o(l) (t)|s(l) (t)) of the observations
and the actual observations. At higher layers, the state does notnecessarily have the same dimensions as the state at the lower
layer and can be an abstract representation. The same applies
to the lower layers, where the system state s(l) (t) corresponds
indirectly to H(t) through the observed values, and there is
no strict requirement for them to match.
The top-down information from higher layers to lower
layers is achieved by overwriting the predicted distribution
P (o(l) (t)|s(l) (t)) with the distribution of the lower layer’s stateP (s(l+1) (t)).
c) Actions: The base station speciﬁes the beam shape using the beamforming vector w(l) (t) and determines the transmission power P (l) (t). Therefore, a(l) (t) = (w(l) (t), P (l) (t))
corresponds to the actions. However, at each layer, there may
be undetermined choices a(l) (t) = ∅, in which case the choiceof the higher layer is adopted.
d) Preference Distribution: The preference distribution
sets the desired states as a prior distribution for control.
In existing beamforming methods, the objective function
is often the power efﬁciency or maximization of the trans-mission rate. However, when considering trafﬁc demands, it
is desirable to set the effective throughput as the objectivefunction. By choosing a Boltzmann distribution as the preference distribution with the objective function Rb (oτ ), whichmeasures utility in terms of logarithm, we can reﬂect R(oτ )in the preference distribution:P̃R (oτ ) ∝ exp(βR(oτ ))with parameter B. Again, A, B is obtained by minimizing
the variational free energy as in the state estimation.
AT= arg min EQ(s1:T ,A,π)BT= arg min EQ(s1:T ,B,π)θτ = arg min{DKL [Q(sτ |θ)||P (sτ |oτ )] − log P (oτ )} (13)θThat is, θτ is calculated as the minimization of the sumof the true posterior distribution and Q(sτ |θτ ) with the
Cullback-Leibler information content and the Shannon surprise − log P (oτ ).
2) Action: Based on the estimated state, beamforming aτ
is determined as the appropriate action for the situation at thattime. Again, according to the FEP, the action with the smallest
Expected Free Energy is selected. The policy that determines
the series of actions a1 , · · · , aT is π, and the probability ofthe policy is determined as follows.
∑
P (π) = softmax(−Gτ (π))(14)τ

where softmax(x) =

x
∑e
x′

by
Gτ (π) =ex ′and Gτ at each time is deﬁned

EQ(oτ ,sτ |π) [log Q(sτ |π)(15)− log P̃C (oτ )P (sτ |oτ , π)]
where P̃C (oτ ) denotes the preference for the observed value.
3) Learning:FortheinternalmodelsP (oτ |sτ ), P (sτ +1 |sτ ), it is possible to set designed valuesor learn from data. Therefore, it is also possible to operate
by giving a base model and then updating the internal model
itself according to the situation.
Update P (oτ |sτ ) as Dirichlet distribution Q(A) with parameter A and P (sτ +1 |sτ ) as Dirichlet distribution Q(B)[log Q(s1:T , B, π)

(17)

B

− log P (o1:T , s1:T , B, π)]Here, β is a parameter that determines how much the preference is biased based on the magnitude of the objectivefunction.

Each agent in the proposed method performs state estimation upon receiving feedback and determines its action policy
based on the estimated state. Furthermore, when a certain
amount of observations has accumulated, the internal model is
updated through model learning [16]. The following describes
each operation in detail.
1) Inference: When the observed value oτ regarding theradio wave strength is obtained, the state Sτ of the estimation
regarding the channel is updated. Here, the state is estimatedby minimizing the variational free energy according to the
FEP. The state is estimated as a distribution, and the distribution Q(sτ |θτ ) follows a categorical distribution.

(16)− log P (o1:T , s1:T , A, π)](12)

C. Agents Behavior[log Q(s1:T , A, π)

A

V. E VALUATION
To evaluate the effectiveness of the proposed method, we
conducted simulations. The evaluation aimed to assess theperformance of the method in dynamic scenarios where UE
is moving, requiring continuous updates of beamforming. We
compared the proposed method against existing approaches
such as online beam training and periodic beam training.
A. Setting
In the evaluation, we considered a scenario where a single
UE with one antenna receives signals from a base stationequipped with N = 4 antennas. To begin with, we veriﬁed thecorrect functioning of the expected free energy minimization
in a static environment by keeping the channel coefﬁcients
ﬁxed without temporal variations.
The base station observes the UE’s signal-to-interferenceplus-noise ratio (SINR) as feedback and, upon receiving the
feedback, updates the internal models and states of each layerand determines the power and beamforming vector for the
next transmission. The beamforming vector is selected from apredeﬁned hierarchical codebook deﬁned by Eq. (7).
The UE performs circular motion around the base station,
moving once per unit time. The distance between the basestation and the UE is set to 200m, and the noise signal strength
at the UE is -114dBm.
In the simulation setting, we consider the time-varying channel coefﬁcients modeled by L-path Rayleigh fading, following
the approach in [6]. The channel coefﬁcient hij (t) between
base station i and user equipment j at time t is given by:√L
βi,j ∑
gi,j (t, l)a†i,j (θ)(18)
hij (t) =L
l=1√
gi,j (t, l) = ρgi,j (t − 1, l) + 1 − ρ2 ei,j (t) (19)where βi,j represents the average channel power gain between the base station and user equipment, a†i,j (θl ) is thesteering vector representing the direction of the l-th path,ei,j (t) represents a complex Gaussian-distributed noise term,and ρ controls the temporal continuity of the Rayleigh fading
process. θl , are assumed to follow a uniform distribution within
the range (θ¯l − ϑ/2, θ¯l + ϑ/2), where θ¯l represents the averageangle and ϑ denotes the angular spread.
B. Methods
In the evaluation, we compared the following beamforming
methods in the simulations:Z̃n ∼ Beta(1 + Ztot [i], 1 + Ti − Ztpt [i])n∑Ztot [i] =zt [i]{
zt [i] =1

t=1
maxk∈S γk,t
>Γ
if
γi,t

0

otherwise

(20)
(21)
(22)

Here, γk,t represents the received signal strength of beam
k at time t, S denotes the set of selected beams, and Γ is theparameter controlling the rejection for UCB, set to Γ = 5dB.
C. Results
1) Behavior of FEP in No Fading Environment: Figure 1
shows the results of the FEP method in terms of the temporal

10

expected free energy

8

6
layer
0
1

4

2

0

0

200

400

time

600

800

1000

(a) Expected Free Energy
90
80
70
SINR

1) FEP-based BF(FEP-based Beamforming): This is the
beamforming method based on the FEP described in Section IV. The internal models P (oτ |sτ , π) and P (sτ |sτ −1 , π)
are learned as expected free energy minimization. The basestation has a two-layer codebook, with the upper layer operating at a timescale Th = 10 times slower than the lower
layer.
2) BT(Beam Training): For comparison, we implemented
beamforming with periodic beam training. Speciﬁcally, we
set the training interval to Ttrain = 100, and after performing Ttrain communications, beam training is conducted.
During beam training, communication is not interrupted, but
the beams used for communication are selected sequentiallythrough hierarchical exploration. Once beam training is completed, the ﬁnal determined beam is used for communication
until the next beam training.
The beams during beam training are selected as follows:
1) Start from layer l = 0.
2) Select a beam a ∈ A(l) (a(l−1) ) at layer l.
3) Measure the SINR under the selected beam.
4) If beams at layer l have not been fully explored, go backto step 2.
5) Select the a(l) with the highest SINR.
6) If l is not equal to L, update l to l + 1 and go back tostep 2.
7) a(L) is the ﬁnal beam.
Here, A(l) (a(l−1) ) represents the set of reﬁned beams available at layer l when a beam a(l−1) is selected at the upper
layer. A(0) at the top layer represents the set of all coarsest
beams.
3) ﬁxBT: This method is the same as BT, but with Ttrain =inf. After the initial beam training, the selected beam is
continuously used for communication.
4) UCB(Upper Conﬁdence Bound): This method uses the
UCB algorithm to select a set of beams that maximizes the
UCB deﬁned in Eq. (1). The set size is set to 2, and the reward
is deﬁned as SINR.
5) RAUCB(Risk-Aware UCB): In the UCB method, thisapproach stochastically rejects the selection based on UCB
and instead chooses beams with higher average rewards. The
rejection probability Z̃n follows a Beta distribution as given
below:

60
50
40
30
20

0

200

400

time

600

800

1000

(b) SINR
Fig. 1. Timeseries of Expected Free Energy and SINR in no fading
environmentchanges in expected free energy and SINR for both the upperand lower layers in no fading environment. The time unit in
the ﬁgure corresponds to one loop of observation and control,
which is the same as the number of received SINR feedbacks.
From the ﬁgure, it can be observed that both the expected
free energy of the upper and lower layers monotonicallydecrease, indicating that the minimization of free energy iseffectively functioning. Additionally, the SINR varies withchanges in the angle between the terminal and the base station.
However, the SINR is maintained at a high level, indicatingthat the FEP method is effectively exploiting the best beam
direction.
2) Comparison: Figure 2(a) presents box plots depicting
the average time-varying SINR achieved by each method
during beamforming. To account for the inﬂuence of channel
coefﬁcients and stochastic selection, we performed beamforming 100 times for each method using different seeds. Moreover,Figure 2(b) shows examples of SINR time series. However,
UCB and RAUCB exhibit strong exploratory behavior andlarge SINR variations; hence, Figure 2(b) displays only the
results of other methods excluding UCB and RAUCB.
From the ﬁgures, it is evident that the proposed method
achieves the highest median SINR among all techniques.
Subsequently, the SINR for BT, which performs periodic

RAUCB. The results showed that the FEP method achieved thehighest SINR and the minimum variation of SINR, indicatingsuperior performance in terms of both average signal quality
and stability.
Further research can focus on extending the FEP approach
to more complex scenarios and investigating its performance
under different system parameters and constraints.

81
80

SINR [dB]

79
78
77

R EFERENCES

76
75
74
FEP-based BF

BT

fixBT

UCB

RAUCB

(a) Average SINR
100
90

SINR (dB)80controller
FEP-based BF
BT
fixBT

70
60
50

0

200

400

Time

600

800

1000

(b) Timeseries of SINR
Fig. 2. Average and Timeseries of SINR for Each Methodbeam training, is comparatively high, while ﬁxBT, UCB, andRAUCB result in lower SINR values. The ﬁxed beam selection
in ﬁxBT leads to reduced SINR as it is unable to adapt tochanging environmental conditions. In the case of UCB and
RAUCB, frequent exploratory selections and susceptibility tonoise-induced errors contribute to lower SINR values.
Figure 2(b) further illustrates that the proposed method
maintains consistently high SINR levels throughout the entire
duration. Conversely, BT and ﬁxBT experience temporary
SINR decreases due to environmental ﬂuctuations, which
likely lead to the observed reduction in average SINR. The
continuous high performance of the proposed method indicates
its robustness and effectiveness in handling dynamic environments.
VI. C ONCLUSION
In this paper, we proposed a beamforming method based on
the FEP for multi-antenna communication systems. The FEP
approach leverages hierarchical modeling and adaptive beamswitching to optimize the trade-off between exploration andexploitation. Through simulation evaluations, we compared
the FEP method with other beamforming methods, including regular beam training, ﬁxed beam selection, UCB, and[1] R. Chataut and R. Akl, “Massive mimo systems for 5g and beyond
networks―overview, recent trends, challenges, and future research direction,” Sensors, vol. 20, no. 10, p. 2753, 2020.
[2] X. Shi, J. Wang, Z. Sun, and J. Song, “Hierarchical codebook-based
beam training for extremely large-scale massive mimo,” arXiv preprint
arXiv:2210.03345, 2022.
[3] J. Chen, F. Gao, M. Jian, and W. Yuan, “Hierarchical codebook designfor near-ﬁeld mmwave mimo communications systems,” arXiv preprint
arXiv:2212.07847, 2022.
[4] K. Friston, J. Kilner, and L. Harrison, “A free energy principle for thebrain,” Journal of physiology-Paris, vol. 100, no. 1-3, pp. 70–87, 2006.
[5] K. Friston, “The free-energy principle: a uniﬁed brain theory?” Nature
reviews neuroscience, vol. 11, no. 2, pp. 127–138, 2010.
[6] K. Yu, G. Wu, S. Li, and G. Y. Li, “Local observations-based energyefﬁcient multi-cell beamforming via multi-agent reinforcement learning,” Journal of Communications and Information Networks, vol. 7,
no. 2, pp. 170–180, 2022.
[7] Z. Xiao, T. He, P. Xia, and X.-G. Xia, “Hierarchical codebook designfor beamforming training in millimeter-wave communication,” IEEE
Transactions on Wireless Communications, vol. 15, no. 5, pp. 3380–
3392, 2016.
[8] H. Yu, P. Guan, W. Qu, and Y. Zhao, “An improved beam training
scheme under hierarchical codebook,” IEEE Access, vol. 8, pp. 53 627–
53 635, 2020.
[9] B. Ning, Z. Chen, W. Chen, Y. Du, and J. Fang, “Terahertz multiuser massive mimo with intelligent reﬂecting surface: Beam training
and hybrid beamforming,” IEEE Transactions on Vehicular Technology,
vol. 70, no. 2, pp. 1376–1393, 2021.
[10] V. Va, T. Shimizu, G. Bansal, and R. W. Heath, “Online learning for
position-aided millimeter wave beam training,” IEEE Access, vol. 7, pp.
30 507–30 526, 2019.
[11] S. Shariﬁ, S. Shahbazpanahi, and M. Dong, “A POMDP-based antenna
selection for massive mimo communication,” IEEE Transactions on
Communications, vol. 70, no. 3, pp. 2025–2041, 2021.
[12] A. Chakrapani, “On the design details of ss/pbch, signal generation andprach in 5g-nr,” IEEE Access, vol. 8, pp. 136 617–136 637, 2020.
[13] K. Shamaei and Z. M. Kassas, “Receiver design and time of arrival
estimation for opportunistic localization with 5g signals,” IEEE Transactions on Wireless Communications, vol. 20, no. 7, pp. 4716–4731,
2021.
[14] V. Jungnickel, K. Manolakis, W. Zirwas, B. Panzner, V. Braun, M. Lossow, M. Sternad, R. Apelfröjd, and T. Svensson, “The role of small cells,
coordinated multipoint, and massive mimo in 5g,” IEEE communications
magazine, vol. 52, no. 5, pp. 44–51, 2014.
[15] R. C. Heins, M. B. Mirza, T. Parr, K. Friston, I. Kagan, and A. Pooresmaeili, “Deep active inference and scene construction,” Frontiers in
Artiﬁcial Intelligence, vol. 3, p. 509354, 2020.
[16] C. Heins, B. Millidge, D. Demekas, B. Klein, K. Friston, I. Couzin, and
A. Tschantz, “pymdp: A python library for active inference in discrete
state spaces,” arXiv preprint arXiv:2201.03904, 2022.