# BER Minimization by User Pairing in Downlink NOMA Using Laser Chaos Decision-Maker
## 3 The Principle of Laser Chaos Decision-Maker  
Here, we review the principle of the laser chaos decision-maker, which is an ultrafast photonic reinforcement learning method based on laser chaos. The details can be found in [30; 31]. When a portion of the output light from a laser is fed back to the laser cavity after a certain delay via an externally arranged mirror, instability of the laser is induced, leading to chaotic oscillatory behavior [30]. Figure 2 shows the decision making based on laser chaos time-series produced by a semiconductor laser [32]. The principle of decision-making based on laser chaos is summarized as follows. By comparing the sampled amplitude of the laser chaos time-series with the threshold value, a decision on the slot machine selection is made. Here we consider a two-armed bandit problem where the issue is to promptly and accurately find the higher reward probability slot machines. The two slot machines are called machine 0 and machine 1. If the sampled amplitude of the laser chaos time series is greater than or equal to the threshold value, slot machine 0 is selected. Otherwise, the decision is to choose slot machine 1. The threshold is adjusted according to the reward from the selected slot machine (in other words, win or lose) so that the slot machine with the higher reward probability will be selected in the subsequent trials. More precisely, the threshold value \(TH(t)\) at step \(t\), which is used in the comparison with the laser chaos signal level, is given as follows:  
\[TH(t)=k\times\lfloor TA(t)\rfloor \tag{5}\]  
where \(TA(t)\) is the threshold adjuster value at step \(t\), \(\lfloor TA(t)\rfloor\) is the nearest integer to \(TA(t)\) rounded to zero and \(k\) is a constant to control the range of \(TH(t)\). The value of \(\lfloor TA(t)\rfloor\) can be one of the values in \(-Z,\cdots,-1,0,1,\cdots,Z\), where \(Z\) is a natural number. Thus, the number of thresholds is \(2Z+1\). \(TH(t)\) is limited to the range of \(-kZ\) to \(kZ\) because of setting \(\lfloor TA(t)\rfloor=Z\) when \(\lfloor TA(t)\rfloor\) is greater than \(Z\), as well as \(\lfloor TA(t)\rfloor=-Z\) when \(\lfloor TA(t)\rfloor\) is less than \(-Z\). The threshold adjuster \(TA(t)\) is updated according to the following [30; 31]:  
Figure 2: Architecture of laser chaos decision-maker for two-armed bandit problems [30]. \[TA(t+1)= \begin{cases}\pm\Delta+\alpha TA(t),\text{if selected slot machine wins.}\\ \mp\Omega+\alpha TA(t),\text{if selected slot machine loses.}\end{cases}  \tag{6}\]  
where \(\alpha\)\((0\leq\alpha\leq 1)\) is the forgetting rate to control the influence of past experiences, \(\Delta\) is the certain increment and \(\Omega\) is the increment parameter. In the case that the selected slot machine wins, i.e., if a reward is obtained by playing the selected slot machine, the threshold adjustment value \(TA(t)\) is updated by \(\pm\Delta+\alpha TA(t)\). Otherwise, the selected slot machine loses, i.e., a reward is not obtained by playing the selected slot machine; the threshold adjustment value \(TA(t)\) is updated by \(\mp\Omega+\alpha TA(t)\). \(\Omega\) is the increment parameter based on the history of choices and benefits. We define that \(S_{i}\) and \(L_{i}\) is the number of times that slot machine \(i\) is selected and that of wins by playing the selected slot machine \(i\) until step \(t\), respectively. At this time, the estimated reward probability of the \(i\)th slot machine \(P_{i}\) is given by:  
\[P_{i}=\frac{L_{i}}{S_{i}}. \tag{7}\]  
In the two-armed bandit problem, we use the estimated reward probability in Equation (7) to define \(\Omega\) as follows:  
\[\Omega=\frac{P_{0}+P_{1}}{2-(P_{0}+P_{1})}, \tag{8}\]  
where \(P_{0}\) and \(P_{1}\) are the estimated reward probabilities of the two slot machines with the highest estimated reward probability and the second-highest estimated reward probability, respectively. By cascading the above-described algorithm, the scalability of the laser chaos decision-maker is realized [31].