# Stereo Matching Based on Immune Neural Network in Abdomen Reconstruction

Authors: Huan Liu; Kuangrong Hao; Yongsheng Ding; Chunjuan Ouyang
Journal: Mathematical Problems in Engineering
 (2015)

---

## Abstract

Stereo feature matching is a technique that finds an optimal match in two images from the same entity in the three-dimensional world. The stereo correspondence problem is formulated as an optimization task where an energy function, which represents the constraints on the solution, is to be minimized. A novel intelligent biological network (Bio-Net), which involves the human B-T cells immune system into neural network, is proposed in this study in order to learn the robust relationship between the input feature points and the output matched points. A model from input-output data (left reference point-right target point) is established. In the experiments, the abdomen reconstructions for different-shape mannequins are then performed by means of the proposed method. The final results are compared and analyzed, which demonstrate that the proposed approach greatly outperforms the single neural network and the conventional matching algorithm in precise. Particularly, as far as time cost and efficiency, the proposed method exhibits its significant promising and potential for improvement. Hence, it is entirely considered as an effective and feasible alternative option for stereo matching.


---

## Body

## 1. Introduction

The morphology of human body, especially, abdomen shape, is a crucial factor to be considered in apparel customized services, body fitness, and so forth. It is widely accepted that obtaining the abdomen size accurately and effectively and building the 3D abdomen profile realistically are critical to provide a vital reference value for humane fashion design.Image matching is a fundamental but critical step for its broad application, such as computer vision and noncontact measurement. The goal we pursuit is to realize an automatic, accurate, and efficient stereo matching algorithm. A previous feature matching approach based on a constraint condition can be found in [1]. Jeong and Moon [2] presented a further detailed constraint condition from the spatial coordinate, the orientation, and breadth of motion path. But the above methods require multitude computational time, and the larger error exists in the location for occluded point. Burie et al. [3] proposed exploring the features of the two stereo linear images sequentially from one end to the other. Performing this scheme forward and backward and at different resolution levels made the matching in majority of candidate features without ambiguities. However, the false matches and miss-correspondences still exist in the final results. Recently, many researchers hope to explore faster and improved methods to solve the matching problem. The matching task is firstly formulated as an optimization problem where an energy function, defined to represent the constraints on the solution, is mapped into a two-dimensional Hopfield neural network for minimization in the literature [4]. Ruichek [5] put forward a feature matching based on the neural network. However, the disparity achieved is sparse due to the irregular distribution of features. It is necessary for additional surface fitting for a dense disparity. The method proposed by [6] is based on the using of ZNCC as matching cost, integrated within a neural network model. The results are satisfactory, but they are not suitable for real time applications on account of the high running time for standard image sets. Literature [7] executed fingerprint matching using a multidimensional artificial neural network. This method has some disadvantages, such as qualitative computation and huge data storage. Neural network algorithm, similar to other classical combinatorial optimization algorithms, always converges at a stable state nearest to its initial state. When the initial condition is inappropriate, it is very possible to tap into the extremum. And it also depends on the correct matches and relationships among features. Once any connection error occurs, it is extremely easy to cause the mismatching. Moreover, since the match is innately an NP issue, its computation grows exponentially with the increase of the points’ number. Consequently, the trade-off between execution time and quality of the matching is a difficult task and must be handled with care.In this study, a creative intelligent biological matching algorithm based on the combination of B-T immune system and neural network is proposed. The details can be described as follows. From a viewpoint that the mathematical model is not clear in actual situation and difficult to complete the image feature matching, it combines the advantages of two-way adjustment flexibility of immune system and sound organizational configuration of neural network. In addition, in view of greater practicability, the extension of matching primitives from just pixel intensity to the combination of the intensity, entropy, and main orientation of gradient vector of pixel as well as the involvement of geometry constraint in T-cell layer is introduced for the sake of excluding the mismatching point. It not only improves the precision but also enhances the network convergence speed.The outline of the rest of this paper is organized as follows: the basic principles concerning stereo matching, immune system, and neural network are introduced in Section2. The details of the improving strategies of Bio-Net and DIE structure (main orientation of gradient, intensity, and entropy of pixel associate to a window) are expounded in Section 3. Qualitative experiments are performed and the results are analyzed and discussed in Section 4. Finally, conclusions are drawn in Section 5.


## 2. Overview

### 2.1. Stereo Matching Algorithm

Recently, literatures [8, 9] on feature matching proceeded from the idea of mimicking the principle of human visual system. Image matching is defined in a pair of stereo images for finding the correspondent pairs in order to estimate the depth information. The matching procedure can be described such that, for each pixel in right image, its matching point in the left image must be found or vice versa. In this paper, the pinhole for the cameras is adopted. It means that the projection of the 3D scene onto 2D images is described by a full perspective projection, and then the spatial information can be calculated by triangulation method [10].Pursuing matching algorithms with high precision, high speed, and strong robustness is the goal to researchers. Generally speaking, the current matching methods are classified into two categories: area-based and feature-based. On the one hand, area-based algorithm takes gray intensity or color value into account to match the corresponding block which consists of the middle pixel and its surrounding neighbors. It can achieve high accuracy together with abundant details. However, the proper size of block is difficult to select, and the matching cannot achieve good effects under the circumstance of shading and texture deficiency as well. On the other hand, feature matching mainly conducts by extracting the features like corners, lines, and edges. It does not directly depend on the intensities, so it is insensitive to noise and vulnerable to the light variation. Yet, it is always difficult to ensure the robustness when the data is incomplete, is not accurate enough, is redundant, or is even distortion. The above two kinds of methods all require a lot of computation, as well as a large amount of memory, making them unsuitable for most fast real-time applications.


### 2.2. Immunity Algorithm

#### 2.2.1. Immune System and Artificial Immune System (AIS)

All living organisms have the capacity of presenting some types of defense against strange attack. The evolution of species resulting in the emergence of the vertebrates also leads to the evolution of the immune system of species. The vertebrate immune system is particularly interesting due to its several computational capabilities.Immune systems are naturally existing mechanisms which are responsible for detecting and coping with intruders in living organisms [11]. The main purpose of the immune system is to recognize all cells (or molecules) within the body, categorize those cells as self or nonself [12], and protect the organism against disease-causing cells called pathogens and eliminate malfunctioning cells [13]. To protect the body from invading pathogens (threats), there is a complex adaptive system that has embedded in vertebrates’ biological immune system. The task of defeating against foreign attack is accomplished by special detectors called lymphocytes. These lymphocytes are created in random manner. After that, they are trained to remember infections so that the organism is protected from any future intrusions (threats) as well as past ones.In immune system, immunity has two types of response: cellular and humoral responses, which are shown in Figure1, where antigens (Ag) are pathogens that invade the body. The macrophages (MA) are large important phagocyte cells in innate immunity on the early nonadaptive phases of host defense. B cells (B) are activated by both the antigen itself and T cells. Antibodies (Ab) are produced by B cells to neutralize antigens.Figure 1
The procedure of immune response.Cellular response is involved in T cells. Macrophages recognize the antigen and present information about the antigen to helper T cells (TH). This procedure is called antigen presenting. TH cells accept the stimulation and produce a series of reactions, and then the information about the antigen is transmitted to the killer T cells (TK). Thereby, TK cells are activated. Then the germs infected by viruses are eliminated by TK cells. Humoral response is involved in antibody, which is a core of the immune response. When an antigen invades the body, TH cells stimulate those B cells, and the effector B cells (BE) produce antibody. If the antigen is eliminated, the immune response stops. Then the concentration of immune cell and antibody begins to be adjusted, which induces the inhibition of suppressor T cells (TS). Meanwhile, TS cells secrete interleukin for suppressing the immune response. If the antibody is no longer produced, the immune response terminated.Artificial immune system which emerged around the mid-1980s is a young field. Artificial immune system, a system of interconnected components, is inspired by biological/natural immune system. It simulates the identification between antigen and antibody and then combines the generation course of antibody and other particular immune mechanisms. The properties it contained are adaptability, diversity, learning, identification, memorability, and broad applicability. The field of AIS is becoming more popular and AIS-based work is spanning from theoretical modeling and simulation to wide variety of domains. For example, it has widely applied in machine-learning and pattern recognition, clustering classification [14], anomaly and intrusion detection [15, 16], autonomous navigation, optimization [17], and data mining [18].


#### 2.2.2. Immune Network

Immune network theory [19] is first proposed by Jerne, which is defined as a network constituted of immune cells with mutual activation and coordination. The immune network algorithm can be categorized into PDP network, AINET, multivalue immune network, and dynamical identification immune network. Varela and Coutinho [20] proposed three characteristics: network structure, dynamics, and subdynamics. Hunt and Cooke [21] did the pioneer work of DNA recognition with immune network model and then came up with B-cell immune network algorithm. Tang et al. [22] designed a multivalue immune network on the basis of the relationship among TH, TS, and B cells and then successfully applied in character recognition. de Castro and von Zuben [23] introduced an immune network algorithm on morphological space, which is applied in the characteristic recognition. Although the existing model has effective applications, the theoretical research is difficult to recognize due to its complex network structure. Therefore it is necessary to explore continually.


### 2.3. Immune Neural Network

An artificial neural network (ANN) can be defined as information processing systems designed with inspiration taken from the nervous system. In most cases of the human brain, currently, most works on ANN place particularly emphasize the ability of solving practical problems. The important features of neural networks are the distributed information representation and the parallel processing [24]. Artificial neural networks have been recognized as powerful tools for learning and simulating systems in a great variety of fields, such as fault detection, automatic control, combinatorial optimization, information prediction, and other fields.In the previous works, Gallo et al. [25] introduced neural networks in stereo matching. Lee et al. [26] presented Hopfield neural network in correspondent points. Sun et al. [27] proposed a scan lined-based asynchronous Hopfield neural network for eigenface matching. Zigh and Belbachir [28] applied successfully Hopfield neural network in extracting regions of buildings and matching them. Venkatesh et al. in the literature [29] explored the potential of self-organizing maps to solve the correspondent problem conceived as imitation of the stereo-perception ability of the human visual system. Kouskouridas et al. [30] proposed a new neural network-based solution to the 3D object pose estimation problem by establishing a novel input-output mapping with the learning process.It is necessary to select proper neural network model, reasonable structure, and high-effective training algorithm in the course of solving practical problems. The aim of training, a process of adjusting the connection weights repeatedly, is to get a suitable network for concrete issue. In a variety of neural network models, multilayer feedforward neural network gets more attention and wider applications. However, confronting complicated problems, it is prone to plunge into local extreme and converge slowly.Recently, there are some researchers contributing to the combination of immunity and neural network. For instance, a hybrid artificial intelligent approach based on the clonal selection principle of artificial immune system and neural networks is proposed to solve multiobjective programming problems in literature [31]. In addition, Mohammad and Zitar [32] put forward a novel idea that it utilizes the genetic optimization for artificial immune system and neural networks in spam detection application. In his paper, the immune system was an appealing system for spam detection because of the classification of self and nonself-messages. The classification needs to be categorized into the legitimate messages (the self) and spam (the nonself).


## 2.1. Stereo Matching Algorithm

Recently, literatures [8, 9] on feature matching proceeded from the idea of mimicking the principle of human visual system. Image matching is defined in a pair of stereo images for finding the correspondent pairs in order to estimate the depth information. The matching procedure can be described such that, for each pixel in right image, its matching point in the left image must be found or vice versa. In this paper, the pinhole for the cameras is adopted. It means that the projection of the 3D scene onto 2D images is described by a full perspective projection, and then the spatial information can be calculated by triangulation method [10].Pursuing matching algorithms with high precision, high speed, and strong robustness is the goal to researchers. Generally speaking, the current matching methods are classified into two categories: area-based and feature-based. On the one hand, area-based algorithm takes gray intensity or color value into account to match the corresponding block which consists of the middle pixel and its surrounding neighbors. It can achieve high accuracy together with abundant details. However, the proper size of block is difficult to select, and the matching cannot achieve good effects under the circumstance of shading and texture deficiency as well. On the other hand, feature matching mainly conducts by extracting the features like corners, lines, and edges. It does not directly depend on the intensities, so it is insensitive to noise and vulnerable to the light variation. Yet, it is always difficult to ensure the robustness when the data is incomplete, is not accurate enough, is redundant, or is even distortion. The above two kinds of methods all require a lot of computation, as well as a large amount of memory, making them unsuitable for most fast real-time applications.


## 2.2. Immunity Algorithm

### 2.2.1. Immune System and Artificial Immune System (AIS)

All living organisms have the capacity of presenting some types of defense against strange attack. The evolution of species resulting in the emergence of the vertebrates also leads to the evolution of the immune system of species. The vertebrate immune system is particularly interesting due to its several computational capabilities.Immune systems are naturally existing mechanisms which are responsible for detecting and coping with intruders in living organisms [11]. The main purpose of the immune system is to recognize all cells (or molecules) within the body, categorize those cells as self or nonself [12], and protect the organism against disease-causing cells called pathogens and eliminate malfunctioning cells [13]. To protect the body from invading pathogens (threats), there is a complex adaptive system that has embedded in vertebrates’ biological immune system. The task of defeating against foreign attack is accomplished by special detectors called lymphocytes. These lymphocytes are created in random manner. After that, they are trained to remember infections so that the organism is protected from any future intrusions (threats) as well as past ones.In immune system, immunity has two types of response: cellular and humoral responses, which are shown in Figure1, where antigens (Ag) are pathogens that invade the body. The macrophages (MA) are large important phagocyte cells in innate immunity on the early nonadaptive phases of host defense. B cells (B) are activated by both the antigen itself and T cells. Antibodies (Ab) are produced by B cells to neutralize antigens.Figure 1
The procedure of immune response.Cellular response is involved in T cells. Macrophages recognize the antigen and present information about the antigen to helper T cells (TH). This procedure is called antigen presenting. TH cells accept the stimulation and produce a series of reactions, and then the information about the antigen is transmitted to the killer T cells (TK). Thereby, TK cells are activated. Then the germs infected by viruses are eliminated by TK cells. Humoral response is involved in antibody, which is a core of the immune response. When an antigen invades the body, TH cells stimulate those B cells, and the effector B cells (BE) produce antibody. If the antigen is eliminated, the immune response stops. Then the concentration of immune cell and antibody begins to be adjusted, which induces the inhibition of suppressor T cells (TS). Meanwhile, TS cells secrete interleukin for suppressing the immune response. If the antibody is no longer produced, the immune response terminated.Artificial immune system which emerged around the mid-1980s is a young field. Artificial immune system, a system of interconnected components, is inspired by biological/natural immune system. It simulates the identification between antigen and antibody and then combines the generation course of antibody and other particular immune mechanisms. The properties it contained are adaptability, diversity, learning, identification, memorability, and broad applicability. The field of AIS is becoming more popular and AIS-based work is spanning from theoretical modeling and simulation to wide variety of domains. For example, it has widely applied in machine-learning and pattern recognition, clustering classification [14], anomaly and intrusion detection [15, 16], autonomous navigation, optimization [17], and data mining [18].


### 2.2.2. Immune Network

Immune network theory [19] is first proposed by Jerne, which is defined as a network constituted of immune cells with mutual activation and coordination. The immune network algorithm can be categorized into PDP network, AINET, multivalue immune network, and dynamical identification immune network. Varela and Coutinho [20] proposed three characteristics: network structure, dynamics, and subdynamics. Hunt and Cooke [21] did the pioneer work of DNA recognition with immune network model and then came up with B-cell immune network algorithm. Tang et al. [22] designed a multivalue immune network on the basis of the relationship among TH, TS, and B cells and then successfully applied in character recognition. de Castro and von Zuben [23] introduced an immune network algorithm on morphological space, which is applied in the characteristic recognition. Although the existing model has effective applications, the theoretical research is difficult to recognize due to its complex network structure. Therefore it is necessary to explore continually.


## 2.2.1. Immune System and Artificial Immune System (AIS)

All living organisms have the capacity of presenting some types of defense against strange attack. The evolution of species resulting in the emergence of the vertebrates also leads to the evolution of the immune system of species. The vertebrate immune system is particularly interesting due to its several computational capabilities.Immune systems are naturally existing mechanisms which are responsible for detecting and coping with intruders in living organisms [11]. The main purpose of the immune system is to recognize all cells (or molecules) within the body, categorize those cells as self or nonself [12], and protect the organism against disease-causing cells called pathogens and eliminate malfunctioning cells [13]. To protect the body from invading pathogens (threats), there is a complex adaptive system that has embedded in vertebrates’ biological immune system. The task of defeating against foreign attack is accomplished by special detectors called lymphocytes. These lymphocytes are created in random manner. After that, they are trained to remember infections so that the organism is protected from any future intrusions (threats) as well as past ones.In immune system, immunity has two types of response: cellular and humoral responses, which are shown in Figure1, where antigens (Ag) are pathogens that invade the body. The macrophages (MA) are large important phagocyte cells in innate immunity on the early nonadaptive phases of host defense. B cells (B) are activated by both the antigen itself and T cells. Antibodies (Ab) are produced by B cells to neutralize antigens.Figure 1
The procedure of immune response.Cellular response is involved in T cells. Macrophages recognize the antigen and present information about the antigen to helper T cells (TH). This procedure is called antigen presenting. TH cells accept the stimulation and produce a series of reactions, and then the information about the antigen is transmitted to the killer T cells (TK). Thereby, TK cells are activated. Then the germs infected by viruses are eliminated by TK cells. Humoral response is involved in antibody, which is a core of the immune response. When an antigen invades the body, TH cells stimulate those B cells, and the effector B cells (BE) produce antibody. If the antigen is eliminated, the immune response stops. Then the concentration of immune cell and antibody begins to be adjusted, which induces the inhibition of suppressor T cells (TS). Meanwhile, TS cells secrete interleukin for suppressing the immune response. If the antibody is no longer produced, the immune response terminated.Artificial immune system which emerged around the mid-1980s is a young field. Artificial immune system, a system of interconnected components, is inspired by biological/natural immune system. It simulates the identification between antigen and antibody and then combines the generation course of antibody and other particular immune mechanisms. The properties it contained are adaptability, diversity, learning, identification, memorability, and broad applicability. The field of AIS is becoming more popular and AIS-based work is spanning from theoretical modeling and simulation to wide variety of domains. For example, it has widely applied in machine-learning and pattern recognition, clustering classification [14], anomaly and intrusion detection [15, 16], autonomous navigation, optimization [17], and data mining [18].


## 2.2.2. Immune Network

Immune network theory [19] is first proposed by Jerne, which is defined as a network constituted of immune cells with mutual activation and coordination. The immune network algorithm can be categorized into PDP network, AINET, multivalue immune network, and dynamical identification immune network. Varela and Coutinho [20] proposed three characteristics: network structure, dynamics, and subdynamics. Hunt and Cooke [21] did the pioneer work of DNA recognition with immune network model and then came up with B-cell immune network algorithm. Tang et al. [22] designed a multivalue immune network on the basis of the relationship among TH, TS, and B cells and then successfully applied in character recognition. de Castro and von Zuben [23] introduced an immune network algorithm on morphological space, which is applied in the characteristic recognition. Although the existing model has effective applications, the theoretical research is difficult to recognize due to its complex network structure. Therefore it is necessary to explore continually.


## 2.3. Immune Neural Network

An artificial neural network (ANN) can be defined as information processing systems designed with inspiration taken from the nervous system. In most cases of the human brain, currently, most works on ANN place particularly emphasize the ability of solving practical problems. The important features of neural networks are the distributed information representation and the parallel processing [24]. Artificial neural networks have been recognized as powerful tools for learning and simulating systems in a great variety of fields, such as fault detection, automatic control, combinatorial optimization, information prediction, and other fields.In the previous works, Gallo et al. [25] introduced neural networks in stereo matching. Lee et al. [26] presented Hopfield neural network in correspondent points. Sun et al. [27] proposed a scan lined-based asynchronous Hopfield neural network for eigenface matching. Zigh and Belbachir [28] applied successfully Hopfield neural network in extracting regions of buildings and matching them. Venkatesh et al. in the literature [29] explored the potential of self-organizing maps to solve the correspondent problem conceived as imitation of the stereo-perception ability of the human visual system. Kouskouridas et al. [30] proposed a new neural network-based solution to the 3D object pose estimation problem by establishing a novel input-output mapping with the learning process.It is necessary to select proper neural network model, reasonable structure, and high-effective training algorithm in the course of solving practical problems. The aim of training, a process of adjusting the connection weights repeatedly, is to get a suitable network for concrete issue. In a variety of neural network models, multilayer feedforward neural network gets more attention and wider applications. However, confronting complicated problems, it is prone to plunge into local extreme and converge slowly.Recently, there are some researchers contributing to the combination of immunity and neural network. For instance, a hybrid artificial intelligent approach based on the clonal selection principle of artificial immune system and neural networks is proposed to solve multiobjective programming problems in literature [31]. In addition, Mohammad and Zitar [32] put forward a novel idea that it utilizes the genetic optimization for artificial immune system and neural networks in spam detection application. In his paper, the immune system was an appealing system for spam detection because of the classification of self and nonself-messages. The classification needs to be categorized into the legitimate messages (the self) and spam (the nonself).


## 3. Feature Matching Based on B-T Immune System and Neural Network

### 3.1. The Mechanism of B-T Immune Regulation

B-T immune network takes effect through reciprocal activation and suppression between B cells and T cells. Figure2 shows the activation procedure of B and T cells, where IL+ and IL− are interleukins secreted by TH and TS cells, respectively. As shown in Figure 2, when the antigens seep into body and are digested by surrounding cells, the news is delivered to T cells, that is, TH cells and TS cells. TS cells can suppress the generation of TH cells. Finally, they together activate B cells. After a period of time, the antibodies produced by B cells can remove the antigens. When the number of antigens becomes larger, TH cells will increase, and TS cells will decrease. Hence B cells will increase accordingly. With the decrease of antigen, TS cells will increase, which inhibit the generation of TH cells. As a result, B cells decrease. Ultimately, the immune feedback system tends to balance.Figure 2
Activation procedure of B and T cell.


### 3.2. Feature Matching Based on B-T Immune Network

The left-right matching consistency of stereo matching is that a feature point in left image has a unique correspondent point in right image and vice versa. The biphasic regulation mechanism in B-T immune system inspires us to extract the similarity between B-T immunity mechanism and left-right consistency. A feature point from the left image is loaded on B cell, and then a search for correspondent point in right image can be considered as the activation from B cells to T cells. Conversely, a search for matching point from right image to left image can be considered as the suppression from T cells to B cells. Furthermore, geometry constraint property is involved into T-cell layer in order to ensure the uniqueness, so the mismatched points can be eliminated. Finally, the mechanism of involvement of B-T immune into BP neural network is adopted to improve the local search ability and enhance its global search capabilities for all the points.


#### 3.2.1. A Combined Feature Description

The description is required after extracting features points. In order to improve the accuracy of identification, a higher distinctive description is demanded. It is to be noted that producing a description merely on the basis of the gray intensity is not enough to provide sufficient information for feature depiction. Consequently, taking the image structure into account, the unit entropy is proposed in this paper. Meanwhile, allowing for the engineering application background concerning the feature matching related to the human abdomen, the template is designed as circle window. As displayed in Figure3. The main direction is determined by the distribution of the peak gradient direction within the adjacent pixels around the key point(1)mx,y=fx+1,y-fx-1,y2+fx,y+1-fx,y-12,θx,y=atan⁡fx,y+1-fx,y-1fx+1,y-fx-1,y,where m(x,y) and θ(x,y) are the gradient and direction of the point at coordinate (x,y). The gradient histogram is calculated by dividing each unit at an angle (e.g., 30°) in a circle region (see Figure 3(b)), and then the peak of the gradient histogram is defined as the main direction. In the gradient histogram, the horizontal coordinate is the block of the direction angle and the vertical coordinate is the number that the gradient direction of a pixel belongs in its own angle block; that is, the probability of the gradient direction of each pixel belongs in the angle block on horizontal axis.Figure 3
Main direction of circle.Once the main direction is set, the circle region is divided clockwise into six sectors at each60° starting from the main direction, and each sector is divided into three subdivisions again. And then a total of 18 gate regions can be obtained (see Figure 4). The entropy can be calculated as (2)eij=-∑m∈Mpmlog⁡2pm,where pm denotes the probability in each unit gate of which intensity is equal to m.Figure 4
The distribution of 18 gate regions.Making use of CCA (canonical correlation analysis) principle, the intensity vectorF=(f1,f2,…,fN) and entropy vector E=(e1,e2,…,eN) of each unit gate are composed as a new feature vector g=(g1,g2,…,gN), where N=1,2,…,18 (see Figure 5). It is great significance to explore feature description by means of synthesizing the above two individual properties.Figure 5
The combination of vectorsF and E.


#### 3.2.2. Training of B-T Immune Network

First, input pattern vectorg which is a vector of pixel feature is inputted to the B cells’ group (e.g., N is just the dimension of vector g and also the number of B cells). Each dot of the input pattern is inputted to its corresponding B cell. The input pattern is weighted and then presented to all TH cells (M is the number of TH cells). This is the antigen presentation (Figure 6).Figure 6
Weight connections from B cells toTH cells.Their weight vectorW is defined as (3)Wj=w1j,w2j,…,wNj,where j=1,2,3,…,M and the weight vector represents the stimulations of an input data to different TH cell. For the purpose of avoiding activating a cell that has never been memorized, in the course of initialization, it is necessary to initialize the weights from B cells to TH cells to be very small values. It can be achieved by(4)wij<LL-1+N,where L is a constant which is greater than 1 and N is the number of B cells, that is, the dimension of the input feature vector.ThenTH cell takes the sum of the weighted input and only TH cell that receives the strongest stimulus called antigen presentation secretes the interleukin (IL+). Namely, TH cells’ group can be considered as a competition network. The interleukin (IL+) is then weighted and set to B cells once again (see Figure 7). Thus, we have also M (the number of TH cells) N-dimensional weighted vector Tj(5)TSj=t1j,t2j,tij,…,tNj,where suppression tij=0,1,2,…,M×N-1,i=1,2,…,N;j=1,2,…,M, and M means dimension of vector TSj.Figure 7
Weight connections fromTS cells to B cells.We call the weight vector memory pattern. In B cells, the input feature vectorg and the memory pattern g′ are compared (see (6)), and the error of each pattern is computed and outputted to TS cells’ group. In TS cells, antibodies, that is, the maximum errors bmax, are compared to a predetermined parameter which is called tolerance ρ. If the maximum error is within the tolerance, the matching is viewed as a successful one and its coordinates are outputted. Otherwise it begins to enter the weights modification of neural network(6)bi=gi-gi′i=1,2,…,N.From the explanation above,ρ is an important control parameter in the B-T immune network. Its value directly influences the evolution accuracy of immune network. Therefore, the choice of a proper ρ is essential. The correct feature mapping pairs obtained by TM in literature [33] are taken as samples. The feature vector g of each feature point in the left image and g′ of each feature point in the right image are calculated, respectively. And then the distribution of bi is necessary to be counted up and observed, which is computed by (6). The results show that bi of about 87 percent of the matched pairs are in the range of 0.05 and 0.15. In view of this, ρ is taken as 0.10 in the paper. In addition, it is noted that the smaller ρ is, the longer the consuming time is.


#### 3.2.3. Immune Network Evolvement

If maximum errorbmax is larger than the tolerant threshold ρ, the IL- secreted from TS cell is used to suppress TH cell that has been activated. Thereby, it produces a competition once again and evolves the immune network.The update of thekth B-TH cell (THK) is thus(7)wikiter+1=tikiter·giTHK·g,where g=∑i=1Ngi2, THK=∑i=1Ntik2, tik is the feedback weight from the kth TH cell to the ith B cell, g says an input vector, g=g1,g2,gi,…,gN(i=1,2,…,N), gi denotes the ith input value from B cell, and iter is the iteration, respectively.The update of thekth TS-B cell (TSK) is(8)tikiter+1=tikiter+gi2,where iter is the iteration.The procedure of B-T network is displayed in Figure8.Figure 8
B-T network flow chart.


#### 3.2.4. Geometry Constraint

There may be several very similar larger values, but only one of them is correct matching point. The correct matching pairs have not only high similarity but also identical geometric distribution. For removing those error matches, the geometry consistency principle [33] is added to T cell, which benefits to obtain the unique maximum used to activate interleukin.If a pointL in the left image is matched with a point R in the right image, it is not possible for another point L′ in the left image, such that XL′<XL (or YL′<YL), to be matched with another point R′ in the right image with XR′>XR (or YR′>YR). This constraint means that if a pair nLR represents a correct match, then all pairs nL′R′ such that L′<L and R′>R or such that L′>L and R′<R must correspond to invalid matches. In general, a feature detection algorithm enables us to extract a few hundreds to thousands features from an image, and therefore, there are a sufficient number of features on the same object that can prove their matching degrees reciprocally. In the paper, the distance and the slope of the connection between two matching points are employed to describe their geometric relationship. The images with same resolution from different perspectives were explored in the tests. Hence, it can be considered that the distance and the slope of the lines between the correct matching pairs are the same, which are, respectively, termed as letter D and K.At the first place, the similarityS on the T cell is sorted by value in ascending order. At the second place, a certain percentage of number is selected according to the actual requirement. Finally, the values H (calculated in the light of (9)) are achieved in the selected sequence. Then sorting the new values H on T cell in descending order and taking the maximum as interleukin are used to activate B cells for reverse adjustment of tij(9)H=S×D×K.


### 3.3. The Neural Network Model Based on B-T Immune System (Bio-Net)

The material basis of ANN comes from the model of neuron in biology. It is generally realized that an intact neuron is made up of several components, such as cell body, dendrite, axon, synapse, and neuron twin. A cell body is the principal part of a neuron. In existing ANN models, a neuron is regarded as a unit that sums up the all input signal and then outputs a signal after comparing with a threshold. Distinctive features of this kind of models denote that they have simple structures and good versatility. However, the active and assistant functions of characteristics information are not considered in these features when dealing with a concrete problem. To be exact, there is no such interface in these existing models. Based on the consideration, the B-T immune system is designed in this paper for utilizing background information and prior knowledge of feature matching. This model is shown as Figure9.Figure 9
Immune ANN network model.In the model shown above, the whole matching process is composed of two steps. Neural network plays a part in total optimization for the whole sample database. Moreover, each feature matching is realized by B-T immune network. Its architecture consists of a set of neurons, where the input layer is made up of two neurons and so is the output layer. Those neurons of input layer denote the coordinates of feature point in the left imageL(XL,YL), and those of output layer indicate the coordinates of the matched point in the right image R(XR,YR). The middle hidden layer is designed as B-T immune network for the concrete feature matching. B-T immune network with bidirectional adjustment function has the similar mechanism with the consistency principle in feature matching. As for B-T immune network,N-dimensional vector g derived from the combination of vectors F and E is taken as the input values of hidden layer (B cells). Then the weight values (∑iwij·gi) are received by T cells, and thenN-dimensional weight vector (T(tij)) is generated. Meanwhile, the weight values are fed back to B cells. The corrections of the weights of neural network are shown in the following formulas:(10)uliiter+1=uliiter+δl·yi,where δl=(pl-ol)·ol·(1-ol), pl and ol are refer to the expected and calculated values of output, and yi=∑i=1Nwijgi, where yi is the ith output value from T cell and gi means the ith input value from B cell:(11)vijiter+1=vijiter+δi′·xj,where δi′=yi·(1-yi)∑lδl·uli;xj is the input value.The intelligent stereo feature matching based on biological network algorithm was introduced above. Its specific steps are described as follows.Step 1.
Select training sample database of feature points for two stereo matching images.Step 2.
With a feature pointL(XL,YL) in sample database as a center point, draw three different sizes of circulars, r=3 pixels, 5 pixels, 7 pixels, and then determine the main direction. Finally, the circle area is divided clockwise into six sectors for every 60 degree starting with the main direction, and each sector is divided into three subdivisions, a total of 18 gate regions.Step 3.
Calculate the parameters intensity vectorF and entropy vector E.(12)F=f1,f2,…,f18,E=e1,e2,…,e18.Step 4.
CombineF and E as new g=(g1,g2,…,g18) with CCA.Step 5.
Complete intelligent matching computation by B-T immune network and obtain a matched pointR(XR,YR).Step 6.
Regulate weights of neural network by means of minimizing the error between expected value and calculated value achieved form Step5.Step 7.
Return to Step1 and get a next point until all the training feature points completed, and the algorithm ends up.Step 8.
Train all the feature points in sample database by neural network, and establish the model of reference feature points and matched correspondences.Step 9.
Retrieve the correct matching correspondences for all the testing feature points in left and right images by that corresponding model so as to obtain the three-dimensional cloud data.


## 3.1. The Mechanism of B-T Immune Regulation

B-T immune network takes effect through reciprocal activation and suppression between B cells and T cells. Figure2 shows the activation procedure of B and T cells, where IL+ and IL− are interleukins secreted by TH and TS cells, respectively. As shown in Figure 2, when the antigens seep into body and are digested by surrounding cells, the news is delivered to T cells, that is, TH cells and TS cells. TS cells can suppress the generation of TH cells. Finally, they together activate B cells. After a period of time, the antibodies produced by B cells can remove the antigens. When the number of antigens becomes larger, TH cells will increase, and TS cells will decrease. Hence B cells will increase accordingly. With the decrease of antigen, TS cells will increase, which inhibit the generation of TH cells. As a result, B cells decrease. Ultimately, the immune feedback system tends to balance.Figure 2
Activation procedure of B and T cell.


## 3.2. Feature Matching Based on B-T Immune Network

The left-right matching consistency of stereo matching is that a feature point in left image has a unique correspondent point in right image and vice versa. The biphasic regulation mechanism in B-T immune system inspires us to extract the similarity between B-T immunity mechanism and left-right consistency. A feature point from the left image is loaded on B cell, and then a search for correspondent point in right image can be considered as the activation from B cells to T cells. Conversely, a search for matching point from right image to left image can be considered as the suppression from T cells to B cells. Furthermore, geometry constraint property is involved into T-cell layer in order to ensure the uniqueness, so the mismatched points can be eliminated. Finally, the mechanism of involvement of B-T immune into BP neural network is adopted to improve the local search ability and enhance its global search capabilities for all the points.


### 3.2.1. A Combined Feature Description

The description is required after extracting features points. In order to improve the accuracy of identification, a higher distinctive description is demanded. It is to be noted that producing a description merely on the basis of the gray intensity is not enough to provide sufficient information for feature depiction. Consequently, taking the image structure into account, the unit entropy is proposed in this paper. Meanwhile, allowing for the engineering application background concerning the feature matching related to the human abdomen, the template is designed as circle window. As displayed in Figure3. The main direction is determined by the distribution of the peak gradient direction within the adjacent pixels around the key point(1)mx,y=fx+1,y-fx-1,y2+fx,y+1-fx,y-12,θx,y=atan⁡fx,y+1-fx,y-1fx+1,y-fx-1,y,where m(x,y) and θ(x,y) are the gradient and direction of the point at coordinate (x,y). The gradient histogram is calculated by dividing each unit at an angle (e.g., 30°) in a circle region (see Figure 3(b)), and then the peak of the gradient histogram is defined as the main direction. In the gradient histogram, the horizontal coordinate is the block of the direction angle and the vertical coordinate is the number that the gradient direction of a pixel belongs in its own angle block; that is, the probability of the gradient direction of each pixel belongs in the angle block on horizontal axis.Figure 3
Main direction of circle.Once the main direction is set, the circle region is divided clockwise into six sectors at each60° starting from the main direction, and each sector is divided into three subdivisions again. And then a total of 18 gate regions can be obtained (see Figure 4). The entropy can be calculated as (2)eij=-∑m∈Mpmlog⁡2pm,where pm denotes the probability in each unit gate of which intensity is equal to m.Figure 4
The distribution of 18 gate regions.Making use of CCA (canonical correlation analysis) principle, the intensity vectorF=(f1,f2,…,fN) and entropy vector E=(e1,e2,…,eN) of each unit gate are composed as a new feature vector g=(g1,g2,…,gN), where N=1,2,…,18 (see Figure 5). It is great significance to explore feature description by means of synthesizing the above two individual properties.Figure 5
The combination of vectorsF and E.


### 3.2.2. Training of B-T Immune Network

First, input pattern vectorg which is a vector of pixel feature is inputted to the B cells’ group (e.g., N is just the dimension of vector g and also the number of B cells). Each dot of the input pattern is inputted to its corresponding B cell. The input pattern is weighted and then presented to all TH cells (M is the number of TH cells). This is the antigen presentation (Figure 6).Figure 6
Weight connections from B cells toTH cells.Their weight vectorW is defined as (3)Wj=w1j,w2j,…,wNj,where j=1,2,3,…,M and the weight vector represents the stimulations of an input data to different TH cell. For the purpose of avoiding activating a cell that has never been memorized, in the course of initialization, it is necessary to initialize the weights from B cells to TH cells to be very small values. It can be achieved by(4)wij<LL-1+N,where L is a constant which is greater than 1 and N is the number of B cells, that is, the dimension of the input feature vector.ThenTH cell takes the sum of the weighted input and only TH cell that receives the strongest stimulus called antigen presentation secretes the interleukin (IL+). Namely, TH cells’ group can be considered as a competition network. The interleukin (IL+) is then weighted and set to B cells once again (see Figure 7). Thus, we have also M (the number of TH cells) N-dimensional weighted vector Tj(5)TSj=t1j,t2j,tij,…,tNj,where suppression tij=0,1,2,…,M×N-1,i=1,2,…,N;j=1,2,…,M, and M means dimension of vector TSj.Figure 7
Weight connections fromTS cells to B cells.We call the weight vector memory pattern. In B cells, the input feature vectorg and the memory pattern g′ are compared (see (6)), and the error of each pattern is computed and outputted to TS cells’ group. In TS cells, antibodies, that is, the maximum errors bmax, are compared to a predetermined parameter which is called tolerance ρ. If the maximum error is within the tolerance, the matching is viewed as a successful one and its coordinates are outputted. Otherwise it begins to enter the weights modification of neural network(6)bi=gi-gi′i=1,2,…,N.From the explanation above,ρ is an important control parameter in the B-T immune network. Its value directly influences the evolution accuracy of immune network. Therefore, the choice of a proper ρ is essential. The correct feature mapping pairs obtained by TM in literature [33] are taken as samples. The feature vector g of each feature point in the left image and g′ of each feature point in the right image are calculated, respectively. And then the distribution of bi is necessary to be counted up and observed, which is computed by (6). The results show that bi of about 87 percent of the matched pairs are in the range of 0.05 and 0.15. In view of this, ρ is taken as 0.10 in the paper. In addition, it is noted that the smaller ρ is, the longer the consuming time is.


### 3.2.3. Immune Network Evolvement

If maximum errorbmax is larger than the tolerant threshold ρ, the IL- secreted from TS cell is used to suppress TH cell that has been activated. Thereby, it produces a competition once again and evolves the immune network.The update of thekth B-TH cell (THK) is thus(7)wikiter+1=tikiter·giTHK·g,where g=∑i=1Ngi2, THK=∑i=1Ntik2, tik is the feedback weight from the kth TH cell to the ith B cell, g says an input vector, g=g1,g2,gi,…,gN(i=1,2,…,N), gi denotes the ith input value from B cell, and iter is the iteration, respectively.The update of thekth TS-B cell (TSK) is(8)tikiter+1=tikiter+gi2,where iter is the iteration.The procedure of B-T network is displayed in Figure8.Figure 8
B-T network flow chart.


### 3.2.4. Geometry Constraint

There may be several very similar larger values, but only one of them is correct matching point. The correct matching pairs have not only high similarity but also identical geometric distribution. For removing those error matches, the geometry consistency principle [33] is added to T cell, which benefits to obtain the unique maximum used to activate interleukin.If a pointL in the left image is matched with a point R in the right image, it is not possible for another point L′ in the left image, such that XL′<XL (or YL′<YL), to be matched with another point R′ in the right image with XR′>XR (or YR′>YR). This constraint means that if a pair nLR represents a correct match, then all pairs nL′R′ such that L′<L and R′>R or such that L′>L and R′<R must correspond to invalid matches. In general, a feature detection algorithm enables us to extract a few hundreds to thousands features from an image, and therefore, there are a sufficient number of features on the same object that can prove their matching degrees reciprocally. In the paper, the distance and the slope of the connection between two matching points are employed to describe their geometric relationship. The images with same resolution from different perspectives were explored in the tests. Hence, it can be considered that the distance and the slope of the lines between the correct matching pairs are the same, which are, respectively, termed as letter D and K.At the first place, the similarityS on the T cell is sorted by value in ascending order. At the second place, a certain percentage of number is selected according to the actual requirement. Finally, the values H (calculated in the light of (9)) are achieved in the selected sequence. Then sorting the new values H on T cell in descending order and taking the maximum as interleukin are used to activate B cells for reverse adjustment of tij(9)H=S×D×K.


## 3.2.1. A Combined Feature Description

The description is required after extracting features points. In order to improve the accuracy of identification, a higher distinctive description is demanded. It is to be noted that producing a description merely on the basis of the gray intensity is not enough to provide sufficient information for feature depiction. Consequently, taking the image structure into account, the unit entropy is proposed in this paper. Meanwhile, allowing for the engineering application background concerning the feature matching related to the human abdomen, the template is designed as circle window. As displayed in Figure3. The main direction is determined by the distribution of the peak gradient direction within the adjacent pixels around the key point(1)mx,y=fx+1,y-fx-1,y2+fx,y+1-fx,y-12,θx,y=atan⁡fx,y+1-fx,y-1fx+1,y-fx-1,y,where m(x,y) and θ(x,y) are the gradient and direction of the point at coordinate (x,y). The gradient histogram is calculated by dividing each unit at an angle (e.g., 30°) in a circle region (see Figure 3(b)), and then the peak of the gradient histogram is defined as the main direction. In the gradient histogram, the horizontal coordinate is the block of the direction angle and the vertical coordinate is the number that the gradient direction of a pixel belongs in its own angle block; that is, the probability of the gradient direction of each pixel belongs in the angle block on horizontal axis.Figure 3
Main direction of circle.Once the main direction is set, the circle region is divided clockwise into six sectors at each60° starting from the main direction, and each sector is divided into three subdivisions again. And then a total of 18 gate regions can be obtained (see Figure 4). The entropy can be calculated as (2)eij=-∑m∈Mpmlog⁡2pm,where pm denotes the probability in each unit gate of which intensity is equal to m.Figure 4
The distribution of 18 gate regions.Making use of CCA (canonical correlation analysis) principle, the intensity vectorF=(f1,f2,…,fN) and entropy vector E=(e1,e2,…,eN) of each unit gate are composed as a new feature vector g=(g1,g2,…,gN), where N=1,2,…,18 (see Figure 5). It is great significance to explore feature description by means of synthesizing the above two individual properties.Figure 5
The combination of vectorsF and E.


## 3.2.2. Training of B-T Immune Network

First, input pattern vectorg which is a vector of pixel feature is inputted to the B cells’ group (e.g., N is just the dimension of vector g and also the number of B cells). Each dot of the input pattern is inputted to its corresponding B cell. The input pattern is weighted and then presented to all TH cells (M is the number of TH cells). This is the antigen presentation (Figure 6).Figure 6
Weight connections from B cells toTH cells.Their weight vectorW is defined as (3)Wj=w1j,w2j,…,wNj,where j=1,2,3,…,M and the weight vector represents the stimulations of an input data to different TH cell. For the purpose of avoiding activating a cell that has never been memorized, in the course of initialization, it is necessary to initialize the weights from B cells to TH cells to be very small values. It can be achieved by(4)wij<LL-1+N,where L is a constant which is greater than 1 and N is the number of B cells, that is, the dimension of the input feature vector.ThenTH cell takes the sum of the weighted input and only TH cell that receives the strongest stimulus called antigen presentation secretes the interleukin (IL+). Namely, TH cells’ group can be considered as a competition network. The interleukin (IL+) is then weighted and set to B cells once again (see Figure 7). Thus, we have also M (the number of TH cells) N-dimensional weighted vector Tj(5)TSj=t1j,t2j,tij,…,tNj,where suppression tij=0,1,2,…,M×N-1,i=1,2,…,N;j=1,2,…,M, and M means dimension of vector TSj.Figure 7
Weight connections fromTS cells to B cells.We call the weight vector memory pattern. In B cells, the input feature vectorg and the memory pattern g′ are compared (see (6)), and the error of each pattern is computed and outputted to TS cells’ group. In TS cells, antibodies, that is, the maximum errors bmax, are compared to a predetermined parameter which is called tolerance ρ. If the maximum error is within the tolerance, the matching is viewed as a successful one and its coordinates are outputted. Otherwise it begins to enter the weights modification of neural network(6)bi=gi-gi′i=1,2,…,N.From the explanation above,ρ is an important control parameter in the B-T immune network. Its value directly influences the evolution accuracy of immune network. Therefore, the choice of a proper ρ is essential. The correct feature mapping pairs obtained by TM in literature [33] are taken as samples. The feature vector g of each feature point in the left image and g′ of each feature point in the right image are calculated, respectively. And then the distribution of bi is necessary to be counted up and observed, which is computed by (6). The results show that bi of about 87 percent of the matched pairs are in the range of 0.05 and 0.15. In view of this, ρ is taken as 0.10 in the paper. In addition, it is noted that the smaller ρ is, the longer the consuming time is.


## 3.2.3. Immune Network Evolvement

If maximum errorbmax is larger than the tolerant threshold ρ, the IL- secreted from TS cell is used to suppress TH cell that has been activated. Thereby, it produces a competition once again and evolves the immune network.The update of thekth B-TH cell (THK) is thus(7)wikiter+1=tikiter·giTHK·g,where g=∑i=1Ngi2, THK=∑i=1Ntik2, tik is the feedback weight from the kth TH cell to the ith B cell, g says an input vector, g=g1,g2,gi,…,gN(i=1,2,…,N), gi denotes the ith input value from B cell, and iter is the iteration, respectively.The update of thekth TS-B cell (TSK) is(8)tikiter+1=tikiter+gi2,where iter is the iteration.The procedure of B-T network is displayed in Figure8.Figure 8
B-T network flow chart.


## 3.2.4. Geometry Constraint

There may be several very similar larger values, but only one of them is correct matching point. The correct matching pairs have not only high similarity but also identical geometric distribution. For removing those error matches, the geometry consistency principle [33] is added to T cell, which benefits to obtain the unique maximum used to activate interleukin.If a pointL in the left image is matched with a point R in the right image, it is not possible for another point L′ in the left image, such that XL′<XL (or YL′<YL), to be matched with another point R′ in the right image with XR′>XR (or YR′>YR). This constraint means that if a pair nLR represents a correct match, then all pairs nL′R′ such that L′<L and R′>R or such that L′>L and R′<R must correspond to invalid matches. In general, a feature detection algorithm enables us to extract a few hundreds to thousands features from an image, and therefore, there are a sufficient number of features on the same object that can prove their matching degrees reciprocally. In the paper, the distance and the slope of the connection between two matching points are employed to describe their geometric relationship. The images with same resolution from different perspectives were explored in the tests. Hence, it can be considered that the distance and the slope of the lines between the correct matching pairs are the same, which are, respectively, termed as letter D and K.At the first place, the similarityS on the T cell is sorted by value in ascending order. At the second place, a certain percentage of number is selected according to the actual requirement. Finally, the values H (calculated in the light of (9)) are achieved in the selected sequence. Then sorting the new values H on T cell in descending order and taking the maximum as interleukin are used to activate B cells for reverse adjustment of tij(9)H=S×D×K.


## 3.3. The Neural Network Model Based on B-T Immune System (Bio-Net)

The material basis of ANN comes from the model of neuron in biology. It is generally realized that an intact neuron is made up of several components, such as cell body, dendrite, axon, synapse, and neuron twin. A cell body is the principal part of a neuron. In existing ANN models, a neuron is regarded as a unit that sums up the all input signal and then outputs a signal after comparing with a threshold. Distinctive features of this kind of models denote that they have simple structures and good versatility. However, the active and assistant functions of characteristics information are not considered in these features when dealing with a concrete problem. To be exact, there is no such interface in these existing models. Based on the consideration, the B-T immune system is designed in this paper for utilizing background information and prior knowledge of feature matching. This model is shown as Figure9.Figure 9
Immune ANN network model.In the model shown above, the whole matching process is composed of two steps. Neural network plays a part in total optimization for the whole sample database. Moreover, each feature matching is realized by B-T immune network. Its architecture consists of a set of neurons, where the input layer is made up of two neurons and so is the output layer. Those neurons of input layer denote the coordinates of feature point in the left imageL(XL,YL), and those of output layer indicate the coordinates of the matched point in the right image R(XR,YR). The middle hidden layer is designed as B-T immune network for the concrete feature matching. B-T immune network with bidirectional adjustment function has the similar mechanism with the consistency principle in feature matching. As for B-T immune network,N-dimensional vector g derived from the combination of vectors F and E is taken as the input values of hidden layer (B cells). Then the weight values (∑iwij·gi) are received by T cells, and thenN-dimensional weight vector (T(tij)) is generated. Meanwhile, the weight values are fed back to B cells. The corrections of the weights of neural network are shown in the following formulas:(10)uliiter+1=uliiter+δl·yi,where δl=(pl-ol)·ol·(1-ol), pl and ol are refer to the expected and calculated values of output, and yi=∑i=1Nwijgi, where yi is the ith output value from T cell and gi means the ith input value from B cell:(11)vijiter+1=vijiter+δi′·xj,where δi′=yi·(1-yi)∑lδl·uli;xj is the input value.The intelligent stereo feature matching based on biological network algorithm was introduced above. Its specific steps are described as follows.Step 1.
Select training sample database of feature points for two stereo matching images.Step 2.
With a feature pointL(XL,YL) in sample database as a center point, draw three different sizes of circulars, r=3 pixels, 5 pixels, 7 pixels, and then determine the main direction. Finally, the circle area is divided clockwise into six sectors for every 60 degree starting with the main direction, and each sector is divided into three subdivisions, a total of 18 gate regions.Step 3.
Calculate the parameters intensity vectorF and entropy vector E.(12)F=f1,f2,…,f18,E=e1,e2,…,e18.Step 4.
CombineF and E as new g=(g1,g2,…,g18) with CCA.Step 5.
Complete intelligent matching computation by B-T immune network and obtain a matched pointR(XR,YR).Step 6.
Regulate weights of neural network by means of minimizing the error between expected value and calculated value achieved form Step5.Step 7.
Return to Step1 and get a next point until all the training feature points completed, and the algorithm ends up.Step 8.
Train all the feature points in sample database by neural network, and establish the model of reference feature points and matched correspondences.Step 9.
Retrieve the correct matching correspondences for all the testing feature points in left and right images by that corresponding model so as to obtain the three-dimensional cloud data.


## 4. Experiment and Analysis

### 4.1. Experimental Setup

In the section, experiments on the abdomen reconstruction and measurement for three different shape models are conducted by the proposed methods. The specific setup shown in Figure10 consists of three main parts: a shape-flexible model installed airbag (see Figure 10(a)), an electric air pump (see Figure 10(b)), and a software platform (see Figure 10(c)). The electric air pump is applied to make shape variation by inflating the airbag on abdomen with different pressures. In this test, the airbag is inflated about 10 kpa and 20 kpa air, respectively. The two correspondent shapes in left view are shown in Figure 11. The concrete instructions are given as follows. The image shown in Figure 11(a) is on the initial condition with no pressure in abdomen airbag, which says model 1. Model 2 (shown in Figure 11(b)) is the condition with 10 kpa air and model 3 (shown in Figure 11(c)) says the condition with 20 kpa air. The three images with different pressures in right view are shown in Figure 12. With the increase of the pressure, the deformation of the abdomen shape gradually occurs.Figure 10
Experimental setup.
(a)
(b)
(c)Figure 11
Abdomen images from left camera under different conditions.
(a)
Initial abdomen image
(b)
10 kpa abdomen image
(c)
20 kpa abdomen imageFigure 12
Abdomen images from right camera under different conditions.
(a)
Initial abdomen image
(b)
10 kpa abdomen image
(c)
20 kpa abdomen imageHere given a comparison with three different matching methods, namely, traditional method (TM), BP neural network (ANN), and biological network, we proposed Im-ANN. TM, the previous study we have proposed in the literature [33], mainly includes two steps: extracting features and matching. The correct matching pairs which have obtained in TM are taken not only as training samples for two methods (ANN and Im-ANN) but also as the standard values for error analysis in this paper. The BP neural network (ANN) consists of three layers, the architecture of which is similar to the Im-ANN. The only key difference is in the middle hidden layer without B-T immune network but rather some ordinary neurons.


### 4.2. Tests of Feature Matching

The following results associated with feature mapping are shown in Figure13. Figure 13(a) displays the comparison of correct matching rate in three approaches, which denotes the ratio between the number of correct matching and the total matching pairs. Figure 13(b) denotes the contrast of runtime in three ways. As can be seen from the charts, the correct matching rate derived from TM and Im-ANN is most extremely similar and approximately at 90%, while the value from ANN is only at about 60%. The computational time taken in ANN and Im-ANN is very similar, and TM takes the longest time in three methods.Figure 13
Comparison of correct matching ratio and runtime in three methods.
(a)
(b)For the sake of general effect, the following error analyses are produced by the statistics in their own unmatched results. The contrast related to average error among the three mentioned methods is shown in Figure14. Figure 14(a) reveals the error at coordinate X, and Figure 14(b) indicates the error at coordinate Y. As the figure shows, no matter whether X or Y coordinates, the errors calculated by ANN are all larger, up to more than 1.5 pixel, while the error derived from TM and Im-ANN is merely nearly to 0.5 pixel which indicates that Im-ANN enables us to achieve better matching effect. A further detailed error analyses are exhibited in Figure 15 including maximum, minimum, and average values. Figure 15(a) shows errors at coordinate X and Figure 15(b) shows errors at coordinate Y. It can be observed that both maximum and minimum errors from ANN are all greater than those from TM and Im-ANN. There are no distinct difference with TM and Im-ANN. The comparison about the rate is computed on the error greater than 3 pixels (see Figure 16). Figure 16(a) shows the errors at coordinate X and Figure 16(b) shows the errors at coordinate Y. It can be found that the results obtained from ANN are worst with two times greater than other two methods, and Im-ANN is slightly better than TM.Figure 14
Analysis of average error in comparison in three methods.
(a)
(b)Figure 15
Detailed error analyses in comparison in three methods.
(a)
(b)Figure 16
Analysis of ratio about error ≥3 pixels in comparison in three methods.
(a)
(b)Overall, the proposed method (Im-ANN) proves its high correct matching rate and great superiority on runtime over TM and ANN methods, with exceedingly similar error to TM, whereas ANN is inferior in that without any consideration for both the innate principle of left-right associated consistency and the geometry constraint. There exists the same mechanism between regulation of B-T immune network and the correspondence problem, so it is believed to be reasonable in application of stereo matching. By means of combining the flexible regulation of B-T immune system and the rational organized structure, it is able to heighten the capability of dealing with concrete problem and lessening the computational workload in training network with the improvement of mapping accuracy and convergence velocity.


### 4.3. Analysis of Shape Size

In order to evaluate the overall accuracy of our proposed method, it is tested on a shape-flexible model. We complete the measurement tests for the abdomen circumference through three independent models, that is, on model 1 (the initial state), model 2 (with the pressure of 10 kpa), and model 3 (20 kpa), respectively. The three-dimensional cloud data of human abdomen from two different views are shown in Figure17. (a) and (a1) are results of model 1; (b) and (b1) are results of model 2; (c) and (c1) are results of model 3.Figure 17
Three-dimensional cloud data of three different models from two different views.Manual measurement is a traditional contact method. The 3D scanner is considered as the comparative highest accuracy of noncontact measurement. Comparisons of parameters are made among the above three schemes (Im-ANN, 3D scanner, manual measurement), and the values from manual measurement are taken as the standard values for comparisons in Table1. CIR means the circumference, DEP means the abdomen depth, BRE means the abdomen breadth,Error_M & Im is the error between manual and Im-ANN in three different models,Error_M & 3D is the error between manual and 3D scanner in three different models. Similarly,Error_3D & Im is the same meaning parameter between 3D scanner and Im-ANN. In comparison with the manual measurement, although the differences exist, the differences from Im-ANN are merely slight, smaller than the differences from 3D scanner, and completely conform to the acceptable criteria in the light of the apparel design and engineering. Meanwhile, the precisions of belly parameters obtained from Im-ANN are highly similar to those from the 3D scanner.Table 1
Error inspection of abdomen size between measured and calculated value (cm).
Model
Model 1
Model 2
Model 3
CIR
DEP
BRE
CIR
DEP
BRE
CIR
DEP
BRE
Manual
69.8
16.3
23.8
78.2
18.2
25.2
82.6
21.8
29.2
3D scanner
69.65
16.26
23.75
77.79
18.13
25.04
82.09
21.66
28.97
Im-ANN
69.62
16.27
23.74
77.81
18.11
25.06
82.13
21.69
28.94
Error_M & Im
0.18
0.03
0.06
0.39
0.09
0.14
0.47
0.11
0.26
Error_M & 3D
0.15
0.04
0.05
0.41
0.07
0.16
0.51
0.14
0.23
Error_3D & Im
0.03
0.01
0.01
0.02
0.02
0.02
0.04
0.03
0.03


## 4.1. Experimental Setup

In the section, experiments on the abdomen reconstruction and measurement for three different shape models are conducted by the proposed methods. The specific setup shown in Figure10 consists of three main parts: a shape-flexible model installed airbag (see Figure 10(a)), an electric air pump (see Figure 10(b)), and a software platform (see Figure 10(c)). The electric air pump is applied to make shape variation by inflating the airbag on abdomen with different pressures. In this test, the airbag is inflated about 10 kpa and 20 kpa air, respectively. The two correspondent shapes in left view are shown in Figure 11. The concrete instructions are given as follows. The image shown in Figure 11(a) is on the initial condition with no pressure in abdomen airbag, which says model 1. Model 2 (shown in Figure 11(b)) is the condition with 10 kpa air and model 3 (shown in Figure 11(c)) says the condition with 20 kpa air. The three images with different pressures in right view are shown in Figure 12. With the increase of the pressure, the deformation of the abdomen shape gradually occurs.Figure 10
Experimental setup.
(a)
(b)
(c)Figure 11
Abdomen images from left camera under different conditions.
(a)
Initial abdomen image
(b)
10 kpa abdomen image
(c)
20 kpa abdomen imageFigure 12
Abdomen images from right camera under different conditions.
(a)
Initial abdomen image
(b)
10 kpa abdomen image
(c)
20 kpa abdomen imageHere given a comparison with three different matching methods, namely, traditional method (TM), BP neural network (ANN), and biological network, we proposed Im-ANN. TM, the previous study we have proposed in the literature [33], mainly includes two steps: extracting features and matching. The correct matching pairs which have obtained in TM are taken not only as training samples for two methods (ANN and Im-ANN) but also as the standard values for error analysis in this paper. The BP neural network (ANN) consists of three layers, the architecture of which is similar to the Im-ANN. The only key difference is in the middle hidden layer without B-T immune network but rather some ordinary neurons.


## 4.2. Tests of Feature Matching

The following results associated with feature mapping are shown in Figure13. Figure 13(a) displays the comparison of correct matching rate in three approaches, which denotes the ratio between the number of correct matching and the total matching pairs. Figure 13(b) denotes the contrast of runtime in three ways. As can be seen from the charts, the correct matching rate derived from TM and Im-ANN is most extremely similar and approximately at 90%, while the value from ANN is only at about 60%. The computational time taken in ANN and Im-ANN is very similar, and TM takes the longest time in three methods.Figure 13
Comparison of correct matching ratio and runtime in three methods.
(a)
(b)For the sake of general effect, the following error analyses are produced by the statistics in their own unmatched results. The contrast related to average error among the three mentioned methods is shown in Figure14. Figure 14(a) reveals the error at coordinate X, and Figure 14(b) indicates the error at coordinate Y. As the figure shows, no matter whether X or Y coordinates, the errors calculated by ANN are all larger, up to more than 1.5 pixel, while the error derived from TM and Im-ANN is merely nearly to 0.5 pixel which indicates that Im-ANN enables us to achieve better matching effect. A further detailed error analyses are exhibited in Figure 15 including maximum, minimum, and average values. Figure 15(a) shows errors at coordinate X and Figure 15(b) shows errors at coordinate Y. It can be observed that both maximum and minimum errors from ANN are all greater than those from TM and Im-ANN. There are no distinct difference with TM and Im-ANN. The comparison about the rate is computed on the error greater than 3 pixels (see Figure 16). Figure 16(a) shows the errors at coordinate X and Figure 16(b) shows the errors at coordinate Y. It can be found that the results obtained from ANN are worst with two times greater than other two methods, and Im-ANN is slightly better than TM.Figure 14
Analysis of average error in comparison in three methods.
(a)
(b)Figure 15
Detailed error analyses in comparison in three methods.
(a)
(b)Figure 16
Analysis of ratio about error ≥3 pixels in comparison in three methods.
(a)
(b)Overall, the proposed method (Im-ANN) proves its high correct matching rate and great superiority on runtime over TM and ANN methods, with exceedingly similar error to TM, whereas ANN is inferior in that without any consideration for both the innate principle of left-right associated consistency and the geometry constraint. There exists the same mechanism between regulation of B-T immune network and the correspondence problem, so it is believed to be reasonable in application of stereo matching. By means of combining the flexible regulation of B-T immune system and the rational organized structure, it is able to heighten the capability of dealing with concrete problem and lessening the computational workload in training network with the improvement of mapping accuracy and convergence velocity.


## 4.3. Analysis of Shape Size

In order to evaluate the overall accuracy of our proposed method, it is tested on a shape-flexible model. We complete the measurement tests for the abdomen circumference through three independent models, that is, on model 1 (the initial state), model 2 (with the pressure of 10 kpa), and model 3 (20 kpa), respectively. The three-dimensional cloud data of human abdomen from two different views are shown in Figure17. (a) and (a1) are results of model 1; (b) and (b1) are results of model 2; (c) and (c1) are results of model 3.Figure 17
Three-dimensional cloud data of three different models from two different views.Manual measurement is a traditional contact method. The 3D scanner is considered as the comparative highest accuracy of noncontact measurement. Comparisons of parameters are made among the above three schemes (Im-ANN, 3D scanner, manual measurement), and the values from manual measurement are taken as the standard values for comparisons in Table1. CIR means the circumference, DEP means the abdomen depth, BRE means the abdomen breadth,Error_M & Im is the error between manual and Im-ANN in three different models,Error_M & 3D is the error between manual and 3D scanner in three different models. Similarly,Error_3D & Im is the same meaning parameter between 3D scanner and Im-ANN. In comparison with the manual measurement, although the differences exist, the differences from Im-ANN are merely slight, smaller than the differences from 3D scanner, and completely conform to the acceptable criteria in the light of the apparel design and engineering. Meanwhile, the precisions of belly parameters obtained from Im-ANN are highly similar to those from the 3D scanner.Table 1
Error inspection of abdomen size between measured and calculated value (cm).
Model
Model 1
Model 2
Model 3
CIR
DEP
BRE
CIR
DEP
BRE
CIR
DEP
BRE
Manual
69.8
16.3
23.8
78.2
18.2
25.2
82.6
21.8
29.2
3D scanner
69.65
16.26
23.75
77.79
18.13
25.04
82.09
21.66
28.97
Im-ANN
69.62
16.27
23.74
77.81
18.11
25.06
82.13
21.69
28.94
Error_M & Im
0.18
0.03
0.06
0.39
0.09
0.14
0.47
0.11
0.26
Error_M & 3D
0.15
0.04
0.05
0.41
0.07
0.16
0.51
0.14
0.23
Error_3D & Im
0.03
0.01
0.01
0.02
0.02
0.02
0.04
0.03
0.03


## 5. Conclusion

In this paper, the stereo correspondence problem is formulated as an optimization task. For the purpose of solving this problem effectively, we proposed a novel intelligent matching algorithm based on the combination of immune system and neural network, as well as DIE data structure. The proposed intelligent network combined neural network with B-T immune system, which is comprised of three layers and is involved the well-known biphasic regulation B-T network into intermediate layer. The first step deals with computing the matched feature point using B-T immunity and DIE structure for each feature point in sample database. The second step conducts the regulation of weights connected neurons by means of neural network in order to minimize the propagation error in correspondence process and an accurate result can be achieved. Finally, the corresponding relation between feature points in left view image and matched points in right view image is established for the general pixels matching and is used to complete the three-dimensional dense cloud data for abdomen reconstruction.Three matching algorithms, traditional method (TM), BP neural network (ANN), and immune neural network (Im-ANN), are proposed to perform the task of correspondence for abdomen stereo images. Comparing the three algorithms, the experimental results in this paper show that the proposed algorithms (Im-ANN) can obtain the same accuracy as the traditional method (TM) but a less time consumption. The advantages of this method over other methods include massive parallelism and possibility of hardware implementation. It was concluded that the use of intelligent network in stereo correspondence is necessary, useful, and helpful to improve the performance of classical BP neural network algorithm. All its merits make it promising for application in a large scale anthropometric survey to collect body dimension for personalized apparel design and other related to human engineering and so forth.Notwithstanding, there exist a number of deficiencies, such as massive sample data memory and computation, and some parameters rely on experience and so forth. Future work in this field could be in optimizing the trade-off between the speed of training time and the size of the database can be supported by the network and proposition of a common method for setting parameters about immune network, trying to apply to other types of neural networks for wider fields, such as information forecasts and machine control.



---

*Source: 242794-2015-10-04.xml*