## 1 Introduction  
The boundary scan test is an advanced technique for digital circuit fault detection, which is widely used in large-scale digital circuit tests due to its fast and efficient testing advantages [1, 2, 3, 4, 5]. However, with the increase in IC density and complexity, the scale of boundary scan test vectors also increases rapidly, which leads to a decrease in test efficiency and fault detection precision [6, 7]. How to generate a high-performance matrix of test vectors (MTV) for large-scale circuits has become a challenging problem [8, 9, 10]. In order to improve the quality of the MTV itself and achieve the goal of satisfying both test efficiency and fault isolation rate, many algorithms based on the mathematical characteristics of the test vector itself have been proposed, for example, counting sequence algorithm, modified counting sequence algorithm [11, 12], Walking-1/0 sequence algorithm [13], W-test adaptive algorithm [14]. These methods are studied from the perspective of the mathematical characteristics of the MTV to achieve its optimization; they do not take into account the physical characteristics of the circuit under test. In order to bring the characteristics of the circuit itself and circuit fault into the scope of study, the limited fault model is proposed by researchers. Based on the physical distance between the networks in the circuit under test, the limited fault model calculates the prior probability of short-circuit fault and generates the MTV according to the probabilities, so that the MTV is further optimized. Ref. [15] uses the PSO algorithm to optimize the direction of generation for MTV; under the condition of maintaining the optimal test efficiency, the misjudgment rate and confusion rate of MTV are reduced. In Ref. [16], the genetic algorithm is used to optimize the grouping mode of circuit networks under test, and the MTV are optimized by assigning the networks with low probability of short circuit fault to the same test group. Ref. [17] describes the performance of MTV by fitness function and iterates test vectors with excellent performance by the genetic algorithm. Ref. [18] presents a method for optimizing test vectors based on fault information using the idea of heuristic. Ref. [19] provides a way to optimize test vectors by using randomly generated faults. While keeping the testing efficiency unchanged, the above methods generate MTV by using intelligent iterative algorithm and reduce the misjudgment rate and confusion rate of MTV to some extent. However, the improper design of particle iteration and the difficulty to determine the search direction of these intelligent algorithms leads to the fast convergence in the optimization of misjudgment rate and confusion rate, the result is always trapped in the local optimal solution [20; 21; 22; 23; 24; 25]. With the increase in the network size of the circuit under test, the local optimal solution of misjudgment rate and confusion rate will lead to a serious decline in the fault detection accuracy of the boundary scan [26; 27; 28]. A new approach to further optimize MTV is needed. In this paper, an optimization method of MTV based on GA (Genetic Algorithm) and an improved AO*model is proposed. Firstly, this paper analyzes the mathematical model of interconnected short-circuit fault under the limited fault model and obtains the short-circuit fault mode which has the most influence on the circuit. After modeling the faults, this paper gives the evaluation indexes of misjudgment rate and confusion rate of MTV under these fault modes, and this paper sets the evaluation index of the MTVs to be optimized as the upper bound of the search and generates the MTV within the upper bound of the search by using the improved AO* model, which is an AND/OR graph search algorithm based on heuristic function and backtracking correction [29]; thus, the optimization of MTV is realized.