# Adaptive Facial Imagery Clustering via Spectral Clustering and Reinforcement Learning
## 2 Proposed Adaptive Clustering
### Feedback Module
#### 2.4.2 The Application of Q-Learning  
In our proposed method, we can deem the different parameter settings as actions and the performance evaluation (_DBI_) as the reward. The state of the clustering system is constantly updating by trying the possible parameter settings whose clustering performance can be evaluated with the _DBI_. With the reward incentive of the optimization to make the _DBI_ minimum, the dynamic debugging process can be implemented using a Q-learning algorithm. In the clustering, there are two parameters for setting: the clustering number (\(K\)) and dimension reduction (dim). Each parameter is given three types of debugging, that is, increasing, decreasing, and remaining unchanged. Therefore, we obtain nine types of parameter debugging actions in each state of Q-learning. Each action contains two independent debugging actions of the two parameters. We set the step values \(\lambda\) and \(\mu\) (\(\lambda,\mu>0\)) for \(K\) and dim, respectively, to denote the parameter debugging steps in the debugging. Because the parameters may vary beyond their range, the corresponding debugging is set to remain unchanged if the parameter debugging is out of range at the next state. The aim of the actions is to make the performance evaluation the _DBI_ minimum. Therefore, the clustering system must give positive incentives when the action facilitates a reduction in the _DBI_. For an inapparent change in the _DBI_, we give tiny incentives, even no incentives. For a bad action which increases the _DBI_, we give a negative incentive. Due to the random initialization of K-means++, the observations of the _DBI_ have small fluctuations for the same action. Therefore, we repeat the observations of the _DBI_ for each action ten times and use the largest _DBI_ value as the final _DBI_. The process is given as:  
\[D(a)=\text{max}D_{i}(a),(i=1,2,\ldots,10) \tag{4}\]  
where \(D(a)\) is the final evaluated _DBI_ with the action a, and \(D_{i}(a)\) is the \(i\)-th observation of the clustering process. We define the variable quantity \(\Delta\) of the _DBI_ as:  
\[\Delta=D-D^{\prime} \tag{5}\]  
Figure 3: The sketch of reinforcement learning, in which the agent guides the choice of actions according to the state feedback information given by the environment, and the choice of actions affects the changes of the environment, thereby updating the parameters. where \(D\) is the \(DBI\) of the current status, and \(D^{\prime}\) is the \(DBI\) of the next status. Therefore, the reward \(r\) can be expressed as:  
\[r=\left\{ \begin{array}{c}\sqrt{|\Delta|},\Delta\in(\varepsilon,+\infty)\\ -\sqrt{|\Delta|},\Delta\in(-\infty,-\varepsilon)\\ 0,\Delta\in[-\varepsilon,\varepsilon]\end{array} \right. \tag{6}\]  
where \(r\) is the reward signal and \(\varepsilon\) (\(\varepsilon>0\)) is a positive threshold. To prevent the algorithm converging to the local minimum region, we use the \(\beta\)-greedy strategy to improve Q-learning. Specifically, we use \(\Gamma(a|s)\) to denote the selection probability function for each action in Q-learning. \[\Gamma(a|s)=\left\{ \begin{array}{ll}1-\beta+\frac{\beta}{N_{a}},&a=\underset {a}{\text{argmax}}Q(s,a)\\ \frac{\beta}{N_{a}}&a\
eq\underset{a}{\text{argmax}}Q(s,a)\end{array} \right. \tag{7}\]  
where \(\beta\) (\(0<\beta<1\)) is a greedy coefficient and \(N_{a}\) is the number of actions. Different from the strategy in which Q-learning always chooses the best action in each state, the \(\beta\)-greedy strategy has \(\beta/N_{a}\) probability of choosing the remaining nonoptimal actions. In doing this, Q-learning enhances the exploration of unknown states, which may be potential global optimal states. In addition, we terminate the Q-learning algorithm when we have \(\Delta=0\) for multiple consecutive states. In this study, we set the number of threshold termination states as10. The Algorithm 1 pseudo program diagram is as follows.