BEAMFORMING WITH FREE ENERGYPRINCIPLE UNDER HIERARCHICALCODEBOOK
Tatsuya Otoshi † , Masayuki Murata ‡† Graduate School of Economics, Osaka University‡ Graduate School of Information Science and Technology, Osaka University

1

Beamforming• Effective use of a large number of antennas at the base station• Generally, base stations have many antennas and terminals have few antennas• Since the terminal side has only a few antennas, it is difficult to receive the throughput gain ofMIMO.
• Signal propagation can be made more directional by using a large number of antennas at thebase station• Signal control based on channel propagation conditions is necessary• Base station estimates the propagation state based on feedback from the terminal• Controls the signal to be transmitted to amplify the signal received by the terminal based on thepropagation state• Handling of channel state fluctuations is an issue

signalfeedback2

Hierarchical Codebook• Propagation state estimation becomes more difficult as the number of antennas

increases• Number of feedbacks required increases with the number of antennas• Increased overhead due to signal transmission for measurement → lower throughput• Hierarchical beam search by Hierarchical codebook• Candidate beams are prepared in advance as a codebook• Hierarchical codebook is constructed from spatially rough beams to finer beams• Reduction of feedback signal by limiting the search range• Redo search when propagation conditions change

3Approach• Beamforming has the overhead of estimating the propagation state.
• Hierarchical codebooks enable somewhat efficient search, but require re-search asconditions change• Lightweight continuous search, rather than re-exploration, is the best way to deal withchanges• The way people perceive the world is active inference, which always combines searchand execution• Application of active inference to keep up with changing conditions with smalloverhead4

Active inference• Ordinary inference estimates a "good" state given observed values.
• Active inference estimates a "good" state, including changing observed values

through actions.
• Example: Peek under the table to see what is hidden under the table.
• Example: Switching between various beams to estimate channel conditions• Using free energy as a measure of goodness

observed value
outsideworld

inference

observed value

stateoutsideworldstate

active inference
action5Free energy principle• A theory that comprehensively describes the functioning of the brain• Describe inference and actions as minimization of "free energy"• Free energy• 𝐹 = 𝐷!" 𝑄 𝑠 𝑃 𝑠 𝑥− log 𝑃(𝑥)• First term: Posterior distribution of state s P 𝑠 𝑥 and approximate distribution Q 𝑠 Kullback-Leibler

information amount of• Second term : Shannon surprise• Inference• Estimate the posterior distribution• Action• Select the action that will yield the observed value x that reduces the Shannon surprise in addition

to the accuracy of the inference

6Inference• observed value x• Feedback of signal strength from the device• condition s• Propagation channel information• State estimation
•

#$!
=0#%"

7

Action• action u• beam vector w• Transmission power p• policy 𝑃(𝑢% |𝜋)• Determine behavior by estimating the policy using the behavior distribution as a policy ( control asinference)• The actual action shall be the one with the highest probability.
• Policy estimation
Direct gain
(preference for observed values)

Estimated value of obtaining the observed value o8Preference prior• A probability distribution that expresses the goodness of the observed value itself in

determining behavior.
• Corresponds to reward function and objective function• Predicted distribution of observations• Minimize surprise = Obtain observed values with high probability = Obtain observed values with strong

preferences• Objective function• Transmission rate

• Preference distribution reflecting objective function• Boltzmann distribution with negative transmission rate as energy

𝑃! 𝑜& ∝ exp −𝛽𝜖 = exp 𝛽𝑅 𝑜&9

Learning• Observation model: A• A probabilistic model that expresses the relationship between• Used to estimate the state from observed values and predict observed values

𝜕𝐹=0
𝜕 ln 𝐴• State transition model : B• A probabilistic model that expresses the time change of state• Used to predict the state when deciding on actions𝑢%&'𝑃(𝑠! |𝑠!"# , 𝑢!"# , 𝐵)𝑠%&'
𝑃(𝑜|𝑠, 𝐴)

𝑜%&'𝑠%
𝑃(𝑜|𝑠, 𝐴)𝑜%𝜕𝐹=0𝜕 ln 𝐵
10

Searching on hierarchical codebook with FEP• FEP agents at each level of the Hierarchical codebook• Active inference is used to make beam decisions at each level• Information is shared between agents at the upper and lower levels to achieve

coordinated operation.
• Subtle changes are handled by the lower layers.
• Switching to the upper layer when the lower layer cannot handle the change

Hierarchical codebook
Upper layer agent

Lower layer agent

11

Cooperation between hierarchical FEP agents• Upper layer agent• Observes the state resulting from the inference of lower layeragents• By reducing the dimensionality of the state, it is possible to reduce

the load concentrated on the upper layer.

• Predicts the desired situation when lower layer cooperation isrealized and feed it back to the lower layer.

• Lower layer agent• Make control decisions by inferring the state from actual
observed values• Achieving cooperation by using the predictions of upper-layer
agents as the prior distribution for inference

upper layer
state

controlprediction

lower layer

control

observation• Achieving cooperation through inference and control to minimize

prediction errors
12Simulation environment• base station• 4 antennas• channel coefficient• multipath fading• 4 pathes• With time variation• UE• 1 antenna• Concentric circles moving at a constant speed• Hierarchical codebook• 2 layers• 2' beams/layer

13

Convergence of Free Energy• The upper and lower agents operate

independently and inferece andcontrol to minimize free energy• Therefore, they do not necessarily converge.

• As shown in the figure on the right,

free energy is converging.
• Appropriate coordination is achieved

through information exchange between
upper and lower agents.

14

Communication performance• Methods• FEP-based BF• BT: Beam training is performed

periodically• fixBT: Beam training• Communication performance• FEP-based BF maintains stable andhigh SNR• Adaptive control over time variations inthe environment• BT and fixBT have temporary periods oflow SNR• This is due to the lack of beam selection

suitable for environmental fluctuations.
15Comparison• FEP-based BF has the highest

average and quartile SNR• In some cases, the minimum value is• FEP-based BF• BT: Beam training is performed periodically• fixBT: Beam training first• UCB: multi-armed bandit with UCB• RAUCB: randomly reject UCB

smaller than BT, but this is due toexploratory behavior• BT has higher SNR than fixBT• Compared to fixBT, which fixes thebeam, BT periodically selects the beam
and is better able to follow
environmental changes.
• UCB and RAUCB have similar orlower SNR than BT• Both UCB and RAUCB are basicallydealing with uncertainty in a staticenvironment.

16

Summary and future work
• Summary• Proposed a solution based on the free energy principle framework for beamforming withhierarchical codebook• FEP agents are deployed at each level of hierarchy, and state predictions are shared among
agents for coordination between upper and lower levels• A higher SNR can be maintained on average than with periodic beam training• Future work
• Simulation considering the realistic movement of UE• Multimodal information processing such as UE location information

17

Thank you for your attention.

18