# E-Sports Training System Based on Intelligent Gesture Recognition

Authors: Hui Li; Yao Lu; Hongqiao Yan
Journal: Computational Intelligence and Neuroscience
 (2022)

---

## Abstract

In order to improve the effect of e-sports training, this paper combines the intelligent gesture recognition technology to construct an e-sports training system and judges the training effect of players through the recognition of players’ gestures. Moreover, this paper studies the commonly used feature extraction algorithms and proposes an improved SLC-Harris feature extraction algorithm, and the feasibility of this algorithm is verified by the experimental results on the EUROC data set. In addition, this paper uses the KLT optical flow algorithm to track the extracted feature points and calculates the pure visual pose through epipolar geometry, triangulation, and PnP algorithms. The experimental research results show that the electronic economic training system based on intelligent gesture recognition proposed in this paper has certain effects.


---

## Body

## 1. Introduction

The reason why e-sports can become a sports competition is that it is closely related to the progress of society, the development of science and technology, and the spiritual and cultural needs of the people. Although there are countless people who enjoy this high-tech intellectual sports event, in fact, public opinion instills the harmful opinion of e-sports in people intentionally or unintentionally. Some media reported extensively that some students were addicted to games and could not extricate themselves, wasting their youth and studies, which made e-sports become “electronic heroin” that everyone shouted. The huge pressure of public opinion makes e-sports face severe survival pressure, and it is difficult for enterprises to enter this market justifiably. Moreover, athletes can only be called “players,” and their treatment cannot be compared with that of ordinary athletes. At the same time, the majority of fans can only engage in e-sports secretly. In addition, in the face of huge pressure from public opinion, it is difficult for the government to guide and supervise confidently, and sometimes, it has to prohibit escrow. The ban on television broadcasting of e-sports competitions can be described as a huge obstacle to the normal development of the current social discrimination against the e-sports sports industry.Generally speaking, the development of e-sports is not yet mature, and the development of e-sports is still in its infancy [1], which is manifested in many aspects: the public recognition is not enough, there are few related large-scale events, there is no professional-scale operation, there is less research in this area, and so on [2]. Especially on college campuses, although students have more time for self-discipline than before, the school does not pay enough attention to e-sports, and there is no relatively formal organization and management of participants, which has led to many human resources problems waste [3].In order to cater to the trend of e-sports development, vigorously develop e-sports business, improve the overall level of e-sports, and enable e-sports activities to develop well in colleges and universities, the current primary task is to deepen the characteristics of students participating in e-sports activities [4]. Among them, the analysis and research on the current situation, development trend, and participation significance of e-sports in colleges and universities are particularly important in order to discover the problems existing in the development of campus e-sports and put forward reasonable suggestions for the development [5].As emerging sports, e-sports are mainly participated by the younger generation, which has the characteristics of younger and younger age. E-sports can exercise people’s thinking ability, psychological pressure resistance, unity and cooperation, hand-eye coordination, and so on. It can also make the younger generation have the awareness of abiding by the rules in the process of participating in e-sports [6]; trained participants have a fair and open, never admit defeat, pursue a stronger competitive spirit, and have a positive impact on the lives of participants. Many colleges and universities have successively opened related majors in e-sports. Although e-sports is popular in the world, the related research and guiding theories on how to cultivate e-sports talents are rare [7].Different scholars have different views on the attributes and characteristics of e-sports. Literature [8] proposed that “e-sports include three basic characteristics: one is electronics, the other is competitive sports, and the third is a confrontation between people. At the same time, e-sports sports are divided into virtualized e-sports sports and fictionalized sports.” Literature [9] pointed out that “the most fundamental characteristics of video games that distinguish them from other artificial games are: virtual environment, absence of the body and artificial intelligence,” emphasizing the main position of electronic communication technology in e-sports. Scholar Yang Fang believes that “e-sports should return to the essence of games, and games to competitive sports are based on the evolution trend of play-game-competitive sports” and based on the development process of traditional competitive sports, puts forward a plan for the development of e-sports. Jia Peng and Yao Jiaxin believe that e-sports has great characteristics: the diversity of functional structure requirements, the full expansion of self-awareness, the complexity of sports information pattern recognition, the agility of information processing efficiency, and the accuracy of intuitive thinking and decision-making. Sex analyzes and clarifies the various attributes of e-sports from many aspects [10].The discussion on the attributes of e-sports is still going on. Based on the current research, it can be determined that the two essential attributes of e-sports are electronic interaction and confrontational competition. Without electronic interaction, it becomes traditional competitive sports; it becomes a video game, so the two are interdependent and indispensable. With the development of electronic interaction technology, various forms of e-sports have emerged [11].Event services are mainly engaged in e-sports referees, coaches, club operation and management, game commentary, data and tactical analysis, and so on. Practitioners need to have data analysis capabilities, management capabilities, and commentary capabilities. The production and broadcast of the event include content production and external dissemination, mainly involving the design of live content and promotion plans, venue layout, equipment debugging, video data collection, postprocessing, background data analysis, and so on. The practitioners should have journalism, communication, broadcasting, TV technology, and other related professional abilities [12].Since the e-sports industry is an emerging industry, most employees are not from e-sports majors and have not received a complete and systematic e-sports theoretical education, but nearly 90% of the employees believe that the e-sports industry needs prejob training [13]. Judging from the current situation of the development of the entire industry, it is undoubtedly the most attractive option to work for game manufacturers, but it is difficult for game manufacturers to absorb more human resources without major business adjustments. Therefore, the need to train practitioners in support organizations around e-sports events becomes more obvious [14]. For example, training content production capabilities (reporters, screenwriters, copywriters, and anchors) requires a professional background in journalism and communication; training event support capabilities (coaches, data analysts, nutritionists, and brokers) requires sports and information technology. Professional background: training public relations, marketing capabilities (products, business, brand marketing, and media), requires a professional background in marketing and management [15].E-sports self-media is still a media, and you must have the ability to report news, or you can dig deep into a vertical field, such as specializing in video commentary of games, specializing in game clearance strategies, specializing in sharing game skills, and so on. After all, hot spots can bring traffic. WeMedia is a personalized media with social attributes; it communicates with users; and it has its own distinct character orientation [16]. To be a self-media, you should also have strong analytical skills and be able to interpret a topic or special event from a unique or professional perspective. Current e-sports professional ability training pathways.


### 1.1. Current E-Sports Professional Ability Training Pathways

Most training institutions in society position themselves as training professional players but basically lack training resources. Training institutions do not have coaches, data analysts, or club managers, and it is difficult for the trained people to find a suitable position in the e-sports circle. Rather than cultivating professional skills, it is better to make money from e-sports hot spots. Money has no intention or inability to contribute to the development of the e-sports industry [17].At present, the main e-sports talents are cultivated by e-sports companies and e-sports clubs. The club mainly trains professional players, coaches, and data analysts in order to achieve better results in the league. Game companies train referees, game developers, commentators, and other related talents to ensure the healthy development of the e-sports industry [18]. An analysis of the revenue structure of the e-sports industry can help us see the e-sports industry more transparently. The truly profitable institutions are still game manufacturers, which continuously create market value through development and operation. In the context of the continuous development and popularization of the video game industry, competition has become a starting point for expanding influence and creating new commercial value. The comprehensive development of competitive value is inseparable from the promotion of surrounding formats, and new jobs such as video, live broadcast, and commentary emerge in an endless stream [19].This paper combines the intelligent gesture recognition technology to construct an e-sports training system and uses the player’s gesture recognition to judge the player’s training effect to improve the e-sports training effect.


## 1.1. Current E-Sports Professional Ability Training Pathways

Most training institutions in society position themselves as training professional players but basically lack training resources. Training institutions do not have coaches, data analysts, or club managers, and it is difficult for the trained people to find a suitable position in the e-sports circle. Rather than cultivating professional skills, it is better to make money from e-sports hot spots. Money has no intention or inability to contribute to the development of the e-sports industry [17].At present, the main e-sports talents are cultivated by e-sports companies and e-sports clubs. The club mainly trains professional players, coaches, and data analysts in order to achieve better results in the league. Game companies train referees, game developers, commentators, and other related talents to ensure the healthy development of the e-sports industry [18]. An analysis of the revenue structure of the e-sports industry can help us see the e-sports industry more transparently. The truly profitable institutions are still game manufacturers, which continuously create market value through development and operation. In the context of the continuous development and popularization of the video game industry, competition has become a starting point for expanding influence and creating new commercial value. The comprehensive development of competitive value is inseparable from the promotion of surrounding formats, and new jobs such as video, live broadcast, and commentary emerge in an endless stream [19].This paper combines the intelligent gesture recognition technology to construct an e-sports training system and uses the player’s gesture recognition to judge the player’s training effect to improve the e-sports training effect.


## 2. Intelligent Gesture Recognition

### 2.1. Gesture Intelligent Positioning

The structural framework of the gesture autonomous localization algorithm is shown in Figure1.Figure 1
Gesture positioning algorithm structural framework.Monocular visual-inertial odometry uses a pure camera in the front end for motion estimation. The algorithm firstly extracts the features of the image information collected by the camera, then uses the optical flow method to track the feature points, and finally uses PnP (Perspective-n-Point) to perform motion estimation on the tracked feature points. Then, the algorithm eliminates the mismatched point pairs through random sampling consistency (RANSAC) and uses nonlinear optimization to optimize the pose. The front-end process is shown in Figure2.Figure 2
Front-end flowchart.


### 2.2. SLC-Harris Feature Extraction

The feature is the digital expression of the object in the image, and the image can be quantitatively analyzed by extracting the feature. Commonly used feature extraction methods mainly include SIFT algorithm, SURF algorithm, and ORB algorithm.The traditional Harris algorithm calculates the angular responsivity as shown below. It is mainly based on the weighted summation of the squared and multiplied gradients of all pixels in the window.(1)R=C−k×traceC2.Among them, there are(2)C=λ1×λ2,(3)traceC=λ1+λ2.In formula (1), k is a constant ranging from 0.04 to 0.06, and both λ1 and λ2 in formula (2) represent eigenvalues.For a grayscale image, the value of any point (x, y) in the integral image ii (x, y) refers to the sum of all grayscale values from the upper left corner of the image to the area where this point is located, as shown in Figure 3.Figure 3
Rectangular window pixel calculation.The calculation formula of pixels in the rectangular window is as follows:(4)iix,y=∑x′≤x,y′≤yIx′,y′.The most complex calculation in the Harris algorithm is the calculation of diagonal responsivity. The original calculation method causes the calculation overlap between each pixel in the integration window, resulting in high computational complexity. For this, the gradient values ingx2,gy2 and gxgy are used to integrate the image to speed up the calculation of the angular responsivity. The calculation formula is as follows:(5)iixxx,y=∑x′≤x,y′≤ygx2x′,y′,iiyyx,y=∑x′≤x,y′≤ygy2x′,y′,iixyx,y=∑x′≤x,y′≤ygxx′,y′gyx′,y′.Efficient nonmaximum suppression (E-NMS) is used to efficiently extract unique feature locations for each corner region, and the region thresholds are compared using image patches instead of pixels. The principle is shown in Figure4.Figure 4
Efficient nonmaximum suppression.


### 2.3. KLT Optical Flow Tracking

After the key points are extracted, the optical flow method is used to calculate the minimum photometric error by establishing an error model. This method does not need to calculate descriptors or feature point matching, which will greatly save the amount of calculation.The basic idea of LK optical flow is to assume that the optical flow in the local neighborhood of a pixel is invariant, and based on this assumption, construct a least-squares problem about the optical flow of the neighborhood pixels.First, it is assumed that the light intensity of the pixel in each frame of the image is constant. According to this, for the pixel located at (x, y) at time t, moving to (x + dx, y + dy) at time t + dt, there are(6)Ix,y,t=Ix+dx,y+dy,t+dt.Then, according to another basic assumption of LK optical flow, the displacement of pixels in adjacent images is small; the Taylor expansion of formula (6) is(7)Ix+dx,y+dy,t+dt≈Ix,y,t+∂I∂xdx+∂I∂ydy+∂I∂tdt.Combining the above formulas and dividing bydt into both sides of the formula, we get:(8)∂I∂xdxdt+∂I∂ydydt=−∂I∂t,where dx/dt represents the motion speed of the pixel on the x-axis, dy/dt represents the motion speed of the pixel on the y-axis, and the two speeds are recorded as u and v, respectively. At the same time, ∂I/∂x represents the gradient value of the image in the x-axis direction at the pixel point; ∂I/∂y represents the gradient value in the y-axis direction at the pixel point; and ∂I/∂t represents the derivative value of the image in the t direction, which are denoted as Ix,Iy, and It, respectively. Therefore, formula (8) can be written in matrix form as follows:(9)IxIyuv=−It.Finally, according to the third basic assumption of the LK optical flow method, adjacent pixels in the same image plane have similar motion; aw×w size window is defined. According to the same motion of all pixels in the window, w2 formulas can be listed; the overdetermined formulas can be constructed; and the motion parameters of the center point can be obtained by the least square method. Accordingly, its formula can be expressed as follows:(10)∑IxIx∑IxIy∑IxIy∑IyIyuv=−∑IxIt∑IyIt.Each image frame is downsampled by pyramid layering, and multilevel pyramids are established.(11)ILx,y=14IL−12x,2y+18IL−12x−1,2y+IL−12x+1,2y+IL−12x,2y−1+IL−12x,2y+1+116IL−12x+1,2y+1+IL−12x+1,2y+1+IL−12x+1,2y+1+IL−12x+1,2y+1,where L represents the Lth layer image.The algorithm calculates the value of the bottom layer from top to bottom according to the Gaussian pyramid and calculates the pixel value near the edge of the image based on the following formulas:(12)IL−1−1,y≈IL−1−1,y,IL−1x,−1≈IL−1x,0,IL−1nxL−1,y≈IL−1nxL−1−1,y,IL−1x,nyL−1≈IL−1x,nyL−1−1,IL−1nxL−1,nyL−1≈IL−1nxL−1−1,nyL−1−1.The camera motion pose is estimated using SFM in the vision front end. For a monocular camera, the camera pose can be estimated by the geometric relationship between two points in different locations in real space and their projected points on their respective imaging planes. As shown in Figure5, P is any point in the three-dimensional space, and its coordinates are [X, Y, Z]T; O1 and O2 are the optical centers of the two camera positions. p1 and p2 are the projection points of point P on the imaging plane I1 and the imaging plane I2, respectively. According to the pixel positions of the two matched point pairs p1 and p2, the essential matrix E and the fundamental matrix F can be obtained.Figure 5
Epipolar geometric constraints.According to the camera imaging model, we assume thatK is the camera internal parameter matrix, and R and t represent the rotation matrix and translation vector from plane I1 to plane I2, and the following formula can be obtained:(13)s1p1=KP,s2p2=KRP+t.Homogeneous coordinate transformation and normalization between 2D and 3D, we can get(14)x1=K−1p1,x2=K−1p2,where x1 and x2 represent the coordinates of pixels p1 and p2 in the normalized plane, respectively. The algorithm combines formuls (13) and (14) and multiplies by x2Tt∧ to obtain the essential matrix E and the fundamental matrix F, which can be sorted out as follows:(15)p2TK−Tt∧RK−1p1=0,E=t∧R,F=K−TEK−1,where t∧ represents the antisymmetric matrix.When there are more than eight sets of point pairs such asp1 and p2, the eight-point method can be used to construct a linear formula system for the simplified formula, and then the unique solution of R and t can be obtained.(16)x2TEx1=p2TFp1=0.When the monocular camera recovers the pose through the epipolar geometric relationship, the obtained translation is the normalized value, which has no practical significance. In order to obtain the depth information on feature points, triangulation needs to be introduced. We assume thats1 and s2 represent the depth of the two feature points; we can get(17)s2x2=s1Rx1+t.The feature point depth valuesx2∧ and x1∧ can be obtained by left-multiplying formula (17) by s1 or s2, respectively, as follows:(18)s1x2∧Rx1+x2∧t=0,s2x1∧Rx2−x1∧t=0.When the positions of multiple points in space are known, the camera pose can be estimated by the PnP algorithm. Common PnP algorithms include P3P, DLT, and BA optimization. Among them, the P3P algorithm is the most common method. The algorithm needs to know at least three points and their projection points on the camera imaging plane. Then, the camera pose can be estimated by solving the relationship between point pairs according to the similar triangle principle and the cosine theorem. A schematic diagram of the P3P relationship is shown in Figure6.Figure 6
Schematic diagram of P3P relationship.The coordinate system convention is as follows: the world coordinate system is represented by⋅w, and ⋅b and ⋅c represent the IMU coordinate system and the camera coordinate system, respectively. The relationship between the coordinate systems is shown in Figure 7. .v represents the visual reference frame in the sliding window, which is independent of the IMU measurement and can represent any frame in the visual structure. .bw represents the transformation from the IMU coordinate system to the world coordinate system; bk represents the IMU frame of the kth image; ⋅cv represents the transformation from the camera coordinate system to the visual reference system; and ck represents the camera frame of the kth image. ⋅^ represents the measured value and parameter estimation value of the sensor; ⋅¯ represents the latest scale parameter of the sliding window; and the rotation can be represented by the rotation matrix R and the quaternion q. gw=0,0,gT represents the gravity vector in the world coordinate system, and gv represents the gravity vector in the visual reference coordinate system.Figure 7
Coordinate conversion relationship.


### 2.4. IMU Preintegration

The sampling frequency of the camera used in this paper is 20 Hz, and the sampling frequency of the IMU is 200 Hz. It can be seen that the frequency of the IMU is much higher than that of the image. In order to avoid the repeated integration phenomenon caused by the frequency change of the visual frame optimization state caused by the high sampling rate of the IMU, a preintegration technique is used for all IMU sampling data between two image key frames. Furthermore, inertial measurements between adjacent image key frames are aggregated into a relative motion constraint through a preintegration technique. The principle of preintegration is shown in Figure8.Figure 8
Principle of preintegration.In Figure8, from top to bottom are the time scale line, the number of image frames generated, the number of image key frames generated, the number of IMU samples, and the IMU preintegration value.The measurement error of the system is mainly affected by bias random walkb and white noise η, and other errors such as the Markov process are ignored. Then, the measurement model of the accelerometer and gyroscope in the IMU can be expressed by the following formula:(19)ω^b=ωbt+bω+ηω12,a^b=qbwTaw−gw+ba+ηa,where ω^b,a^bt,ωbt, and awt represent the measured value and real value of angular velocity and acceleration, respectively; bω,ba,ηω, and ηa represent the random walk noise and measurement white noise of angular velocity and acceleration, respectively; and qbwT is the rotation matrix transformed from the world coordinate system to the IMU coordinate system.White noise obeys a Gaussian distribution, that is,ηa∼N0,σa2,ηω∼N0,σω2. The derivative of random walk noise also obeys the Gaussian distribution, that is, ηba∼N0,σba2,ηbω∼N0,σbω2.The differential kinematic formulas forP,V,Q (representing the position, velocity, and rotation expressed in quaternions, respectively) versus time can be written as follows:(20)p˙btw=vtw,v˙tw=atw,q˙btw=qbtw⊗012ωbt,where ⊗ represents quaternion multiplication.Through the above derivative relationship, the position, velocity, and rotation at timek + 1 can be obtained from the position, velocity, and rotation at time k and by integrating the measured values of the IMU over time Δtk. The continuous integration formula for PVQ is as follows:(21)pbk+1w=pbkw+vbkwΔtk+∫∫t∈k,k+1Rtwa^t−bat−gwdt2,vbk+1w=vbkw+∫t∈k,k+1Rtwa^t−bat−gwdt,qbk+1w=qbkw⊗∫t∈k,k+11/2Ωω^t−bwtqtbkdt,wherea^t and ω^t represent the acceleration and angular velocity measured in the IMU coordinate system, respectively. Δtk represents the time difference from the kth frame to the k + 1 frame. Rtw represents the rotation matrix from the world coordinate system to the IMU coordinate system. Because the measured a^t and ω^t belong to the IMU coordinate system, in order to transform the IMU measured value to the world coordinate system, the rotation matrix needs to be left-multiplied. Ωω means quaternion right multiplication; ωx means antisymmetric matrix in quaternion multiplication ω means the imaginary part value of quaternion). We assume that the quaternion is q=xyzs=ωs; then we have(22)Ωω=−ω×ω−ωT0,ω×=0ωzωyωz0−ωx−ωyωx0.By observing the continuous integral formula of PVQ, it can be seen that the current state is recursively obtained from the state of the previous time, and the estimated value is constantly changing. This will cause the IMU measurements to be repropagated, causing the velocity and rotation to be reintegrated after each nonlinear optimization iteration, resulting in a higher computational cost. Therefore, the optimization variables are separated from the IMU preintegration terms of the two key frames, and the rotation matrixRwbk of the world coordinate system to the IMU coordinate system can be obtained by simultaneously left-multiplying the left and right sides of the continuous integration formula of PVQ:(23)Rwbkpbk+1w=Rwbkpbkw+vbkwΔtk−12gwΔtk2+αbk+1bk,Rwbkvbk+1w=Rwbkvbkw−gw+βbk+1bk,qwbk⊗qbk+1w=γbk+1bk.The image framesbk and bk+1 of two consecutive moments are given, and the linear acceleration and angular velocity are preintegrated in the local coordinate system bk to obtain(24)αbk+1bk=∫∫t∈k,k+1Rtbka^t−batdt2,βbk+1bk=∫t∈k,k+1Rtbka^t−batdt,γbk+1bk=∫t∈k,k+121/2Ωω^t−bωtγtbkdt,where αbk+1bk,βbk+1bk,γbk+1bk represent the relative pose, velocity, and rotation constraints, respectively, and are also the relative motion of bk+1 to bk. It can be seen that they are only related to a^t and ω^t in bk and bk+1, and they have nothing to do with the initial position and velocity of coordinate system bk.Therefore, the preintegration formula is rediscussed, in terms ofαbk+1bk; it is related to a^t and ω^t of the IMU; and a^t and ω^t are also variables that need to be optimized. When the bias change is small, αbk+1bk,βbk+1bk,γbk+1bk are adjusted according to their first-order approximations to the bias.(25)αbk+1bk≈α^bk+1bk+Jbaαδba+Jbωαδbω,βbk+1bk≈β^bk+1bk+Jbaβδba+Jbωβδbω,γbk+1bk≈γbk+1bk⊗012Jbaγδbω,where Jbaα and Jbωα are the block matrices in Jbk+1α and Jbaβ and Jbωβ are the block matrices in Jbk+1β.There are errors in the integral values of the IMU at different times, and the errors at timet are mainly related to the measured values of αtbk,βtbk, θtbk,bat, and bwt at time t. The following definition is given to represent the error vector:(26)δztbk=δαtbkδβtbkδθtbkδbatδbωt.The derivation is based on the derivative of the error term kinetic formula. First, two concepts are introduced: true and nominal, where true represents the real measurement value containing noise and nominal represents the theoretical value without noise, andδ represents the measurement error; there are(27)δα˙=α˙true−α˙nominal,δβ˙=β˙true−β˙nominal.Among them, there are:(28)β˙true=Rtbka^t−ηa−bat−δbat−a^t−bat×δθ,β˙nominal=Rtbka^t−bat.Combining the above formulas, we can obtain(29)δβ˙=−Rtbka^t−bat×δθ−Rtbkδbat−Rtbkηa.The derivation ofδθ˙ is as follows, and according to the formula in the literature, it can be known that(30)qtrue=qnominal⊗δq.In this paper, according to the noise model and bias, we can get(31)δθ˙≈−ω^t−bωt×δθ−ηω−δbωt.In summary, the derivative of the IMU measurement error term at timet can be as follows:(32)δα˙tbkδβ˙tbkδθ˙tbkδb˙atδb˙ωt=0I00000−Rtbka^t−bat×−Rtbk000−ω^t−bωt×0−I0000000000δαtbkδβtbkδθtbkδbatδbωt+cηaηωηbaηbω.We setFt=0I00000−Rtbka^t−bat×−Rtbk000−ω^t−bωt×0−I0000000000,Gt=0000−Rtbk0000I0000I0000I. The above formula can be simplified to(33)δz˙tbk=Ftδztbk+Gtnt.According to the definition of the derivative, the prediction formula of the mean is as follows:(34)δz˙tbk=limδt⟶0δzt+δtbk−δztbkδt,δzt+δtbk=1+Ftδtδztbk+Gtδtnt.According to the error value at the current moment, the mean and covariance at the next moment can be predicted. The prediction formula for covariance is as follows:(35)Pt+δtbk=1+FtδtPtbk1+FtT+GtδtQGtδtT,where Ptbk represents the initial value of the iteration and its value is zero and Q represents the diagonal covariance matrix of the noise term as follows:According to formula (35), the iterative formula of the error term Jacobian can be obtained as follows:(36)Jt+δt=1+FtJt,where the iterative initial value of the Jacobian matrix Jt is I.


### 2.5. Sliding Window Initialization

When the camera extrinsic parameter matrixp¯bkv,p¯ckv is known, the pose obtained by the initialization of the monocular camera is transformed from the visual coordinate system to the IMU coordinate system to obtain the following formula:(37)qbkv=qckv⊗qbv,sp¯bkv+qckvpbc=sp¯ckv,where s is the translation factor obtained by visual initialization, which has no real information.The pure visual initialization method lacks absolute scale information. Therefore, the value estimated by the visual SFM is aligned with the IMU after preintegration to estimate the true scale. Visual-inertial alignment initialization is mainly to solve the following problems, including the initialization of gyroscope bias, the initialization of velocity, gravitational acceleration, and scale.The first is to initialize the gyroscope. The gyroscope bias can be obtained from two consecutive key frames with known orientations, considering two consecutive framesbk and bk+1 in the sliding window, and qbkv and qbk+1v represent the rotations obtained from the pure visual sliding window optimization, respectively. Linearize the IMU preintegration term for the gyroscope bias and minimize the following function:(38)minδbw∑k∈ℬqbk+1v−1⊗qbkv⊗γbk+1bk2.Among them, there are:(39)γbk+1bk≈γ^bk+1bk⊗Jbwγ1δbw.In formula (42), ℬ represents all the frames in the window, and the product of the two quaternions indicates that the camera rotates from the kth frame to the k + 1th frame, and the gyroscope rotates from the k + 1th frame to the kth frame, and the optimized objective function is(40)qbk+1v−1⊗qbkv⊗γbk+1bk=10.The algorithm takesγ^ into the formula and multiplies the inverse moment ordering of the relative constraints obtained from the preintegration to the left on both sides of formula (40) and obtains by Cholesky decomposition (multiplying the transpose of Jbwγ on both sides of the formula):(41)JbwγTJbwγδbw=2JbwγT2γ^bk+1bk−1⊗qbkv⊗qbk+1v−1vec.In this way, the initial calibration value of the gyroscope biasbw can be estimated, and then the IMU preintegration terms α^bk+1bk,β^bk+1bk,γ^bk+1bk are corrected with the new gyroscope bias.The second is the initialization of velocity, gravitational acceleration, and scale. The initialized state vector is as follows:(42)χI=vb0v,vb1v,…vbnv,gv,s,where the state vectorvbkv represents the speed of the visual coordinate system of the kth frame image, gv represents the gravity vector in the visual coordinate system, and s is the estimated scale parameter. To sum up, the dimension of χI is 3(n + 1) + 3 + 1. The constraint relationship between the scale parameter and the speed of the visual SFM is as follows:(43)z^bk+1bk=α^bk+1bkβ^bk+1bk=Hbk+1bkχI+nbk+1bk≈,−qvbkΔtk012qvbkΔt2qvbkp¯bk+1v−p¯bkv−qvbkqvbkqvbkΔtk0vbkvvvk+1gvgv,where qbkV,qbkV,qbkV are all obtained from visual SFM. qVbk and qbkV are mutually inverse matrices. The following linear least squares problem is constructed to complete the initialization of velocity, gravitational acceleration, and scale:(44)minχI∑k∈ℬz^bk+1bk−Hbk+1bkχI2.


### 2.6. Monocular Visual Inertial Coupling Nonlinear Optimization

When coupling the visual constraint value and the IMU constraint value, the data of the inertial sensor should be introduced first, and the constraint value of the IMU on the state should be added to the optimized state vector. Then, nonlinear optimization is performed within a sliding window, and all state vectors of the sliding window are as follows:(45)χ=x0,x1,…xn,xcb,λ0,λ1,…λm,xk=pbkw,vbkw,qbkw,ba,bg,kϵ0,n,xcb=pcb,qcb,where xk represents the state of the IMU when the kth image is captured. There are n + 1 states in the sliding window, and each state contains the position, velocity, and rotation in the world coordinate system, and the IMU offsets in the IMU coordinate system. λm represents the inverse depth information of the mth 3D point, and xcb represents the external parameter from the camera to the IMU. The objective function is(46)minχrp−Hpχ2+∑k∈ℬrℬz^bk+1bk,χpbk+1bk2+∑l,jóCρrCz^lcjpl2cj,where Hp is the Huber norm, which is defined as follows:(47)ρs=1,s≥1,2s−1,s<1.In formula (52), ⋅P represents the Mahalanobis distance weighted by the covariance matrix P, and rp,Hp,rBz^bk+1bk,X, and rCz^lc,X represent the marginalized prior information, the IMU measurement residual, and the visual reprojection error, respectively. ℬ is the set of all IMU measurement frames, and C is the set of visual features in the sliding window.According to the Gauss–Newton method, the incremental method can be used to calculate the minimum value of the Gaussian objective function, as follows:(48)minδXrℬz^bk+1bk,χ+ΔXpbk+1bk2=minδXrℬz^bk+1bk,χ+Jbk+1bkΔXpbk+1bk2,where Jbk+1bk is the Jacobian matrix of the error term rℬ with respect to all state vectors χ.The algorithm differentiates the above formula fromΔX and then sets its derivative to 0, and the formula for the increment ΔX can be calculated as follows:(49)Jbk+1bkTPbk+1bk−1Jbk+1bkΔX=−Jbk+1bkTPbk+1bk−1Jbk+1bkrℬ.The overall incremental equation of the objective function is as follows:(50)Hp+∑Jbk+1bkTPbk+1bk−1Jbk+1bk+∑JlcjTPlcj−1JlcjΔX=bp+∑Jbk+1bkTPbk+1bk−1rℬ+∑JlCjTPlCj−1rC,where Pbk+1bk represents the covariance of the IMU preintegrated noise term and PlCj represents the visually observed noise covariance. When the noise covariance Pbk+1bk of the IMU is larger, the inverse of Pbk+1bk, that is, its information matrix is smaller, which means that the IMU observations are not as reliable as the visual observations. Formula (50) can be simplified to(51)Λp+ΛB+ΛCΔX=bp+bB+bC,where Λp,ΛB, and ΛC represent the Hessian matrix. Using the perturbation method to calculate, we can get(52)J=∂r∂X=limδX⟶0rX⊕δX−rXδX,where δX represents the small disturbance of the state vector instead of the incremental ΔX,⊕ represents the disturbance of the state vector.The continuous preintegration formula is derived in the IMU preintegration, and the IMU residuals are as follows:(53)rℬz^bk+1bk,χ=δαbk+1bkδβbk+1bkδθbk+1bkδbaδbg=Rwbkpbk+1w−pbkw−vbkwΔtk+12gwΔtk2−αbk+1bkRwbkvbk+1wΔtk−vbkwΔtk+gwΔtk−βbk+1bk2γbk+1bk⊗qbkw⊗qbk+1wxyzbabk+1−babkbωbk+1−bωbk.According to the above formula, it can be seen that the optimization variables of the IMU residual mainly include the position, rotation, speed, and inertia bias at thei and j times:(54)pbkw,qbkw,vbkw,bak,bωk,pbk+1w,qbk+1w,vbk+1w,bak+1,bωk+1.To calculate the Jacobian matrix, perturbation is added to each optimization variable to obtain(55)δpbkw,δθbkw,δvbkw,δbak,δbωk,δpbk+1w,δθbk+1w,δvbk+1w,δbak+1,δbωk+1.Taking the partial derivatives for the above disturbance variables, respectively, we can get(56)J015×7=∂rBδpbkw,∂rBδθbkw,J115×7=∂rBδvbkw,∂rBδbak,∂rBδbωk,J215×7=∂rBδpbk+1w,∂rBδθbk+1w,J315×7=∂rBδvbk+1W,∂rBδbak+1,∂rBδbωk+1.The visual residual is a reprojection error, which represents the difference between the estimated value and the observed value of the feature point in the normalized camera coordinate system. The small receiver camera used in this paper belongs to the fisheye camera model and belongs to the fisheye with a large viewing angle, so its projection on the unit sphere needs to be considered, as shown in Figure9.Figure 9
Unit spherical projection model.Through the unit spherical projection model illustrated in the figure above, the value of the visual residual is decomposed into two directions. The final visual residual model looks like this:(57)rcz^lcj,χ=b1,b2T⋅p^lcj−plcjplcj,p^lcj=πc−1u^lcjv^lcj,plcj=RcbRwbjRbiwRcb1λlP¯lci+pcb+pbiw−pbjw−pcb,where p¯^lcj and plcj represent the estimated and observed coordinates of the landmark point 1 in the j-th frame image under the normalized camera coordinate system, respectively. The optimization variables of the visual residual are as follows:(58)pbiw,qbiw,pbjw,qbjw,pcb,qcb,λl,where λl represents the inverse depth value when the landmark point 1 is first observed by the j-th image. The inverse depth is used as the optimization variable because the inverse depth satisfies the Gaussian distribution, and it can reduce the parameter variables in the actual optimization process. According to the above formula, by adding disturbance to each optimization variable, the following Jacobian is obtained:(59)J03×7=∂rC∂pbiw,∂rC∂qbiw,J13×7=∂rC∂pbjw,∂rC∂qbjw,J23×7=∂rC∂pcb,∂rC∂qcb,(60)J33×1=∂rC∂λl.


## 2.1. Gesture Intelligent Positioning

The structural framework of the gesture autonomous localization algorithm is shown in Figure1.Figure 1
Gesture positioning algorithm structural framework.Monocular visual-inertial odometry uses a pure camera in the front end for motion estimation. The algorithm firstly extracts the features of the image information collected by the camera, then uses the optical flow method to track the feature points, and finally uses PnP (Perspective-n-Point) to perform motion estimation on the tracked feature points. Then, the algorithm eliminates the mismatched point pairs through random sampling consistency (RANSAC) and uses nonlinear optimization to optimize the pose. The front-end process is shown in Figure2.Figure 2
Front-end flowchart.


## 2.2. SLC-Harris Feature Extraction

The feature is the digital expression of the object in the image, and the image can be quantitatively analyzed by extracting the feature. Commonly used feature extraction methods mainly include SIFT algorithm, SURF algorithm, and ORB algorithm.The traditional Harris algorithm calculates the angular responsivity as shown below. It is mainly based on the weighted summation of the squared and multiplied gradients of all pixels in the window.(1)R=C−k×traceC2.Among them, there are(2)C=λ1×λ2,(3)traceC=λ1+λ2.In formula (1), k is a constant ranging from 0.04 to 0.06, and both λ1 and λ2 in formula (2) represent eigenvalues.For a grayscale image, the value of any point (x, y) in the integral image ii (x, y) refers to the sum of all grayscale values from the upper left corner of the image to the area where this point is located, as shown in Figure 3.Figure 3
Rectangular window pixel calculation.The calculation formula of pixels in the rectangular window is as follows:(4)iix,y=∑x′≤x,y′≤yIx′,y′.The most complex calculation in the Harris algorithm is the calculation of diagonal responsivity. The original calculation method causes the calculation overlap between each pixel in the integration window, resulting in high computational complexity. For this, the gradient values ingx2,gy2 and gxgy are used to integrate the image to speed up the calculation of the angular responsivity. The calculation formula is as follows:(5)iixxx,y=∑x′≤x,y′≤ygx2x′,y′,iiyyx,y=∑x′≤x,y′≤ygy2x′,y′,iixyx,y=∑x′≤x,y′≤ygxx′,y′gyx′,y′.Efficient nonmaximum suppression (E-NMS) is used to efficiently extract unique feature locations for each corner region, and the region thresholds are compared using image patches instead of pixels. The principle is shown in Figure4.Figure 4
Efficient nonmaximum suppression.


## 2.3. KLT Optical Flow Tracking

After the key points are extracted, the optical flow method is used to calculate the minimum photometric error by establishing an error model. This method does not need to calculate descriptors or feature point matching, which will greatly save the amount of calculation.The basic idea of LK optical flow is to assume that the optical flow in the local neighborhood of a pixel is invariant, and based on this assumption, construct a least-squares problem about the optical flow of the neighborhood pixels.First, it is assumed that the light intensity of the pixel in each frame of the image is constant. According to this, for the pixel located at (x, y) at time t, moving to (x + dx, y + dy) at time t + dt, there are(6)Ix,y,t=Ix+dx,y+dy,t+dt.Then, according to another basic assumption of LK optical flow, the displacement of pixels in adjacent images is small; the Taylor expansion of formula (6) is(7)Ix+dx,y+dy,t+dt≈Ix,y,t+∂I∂xdx+∂I∂ydy+∂I∂tdt.Combining the above formulas and dividing bydt into both sides of the formula, we get:(8)∂I∂xdxdt+∂I∂ydydt=−∂I∂t,where dx/dt represents the motion speed of the pixel on the x-axis, dy/dt represents the motion speed of the pixel on the y-axis, and the two speeds are recorded as u and v, respectively. At the same time, ∂I/∂x represents the gradient value of the image in the x-axis direction at the pixel point; ∂I/∂y represents the gradient value in the y-axis direction at the pixel point; and ∂I/∂t represents the derivative value of the image in the t direction, which are denoted as Ix,Iy, and It, respectively. Therefore, formula (8) can be written in matrix form as follows:(9)IxIyuv=−It.Finally, according to the third basic assumption of the LK optical flow method, adjacent pixels in the same image plane have similar motion; aw×w size window is defined. According to the same motion of all pixels in the window, w2 formulas can be listed; the overdetermined formulas can be constructed; and the motion parameters of the center point can be obtained by the least square method. Accordingly, its formula can be expressed as follows:(10)∑IxIx∑IxIy∑IxIy∑IyIyuv=−∑IxIt∑IyIt.Each image frame is downsampled by pyramid layering, and multilevel pyramids are established.(11)ILx,y=14IL−12x,2y+18IL−12x−1,2y+IL−12x+1,2y+IL−12x,2y−1+IL−12x,2y+1+116IL−12x+1,2y+1+IL−12x+1,2y+1+IL−12x+1,2y+1+IL−12x+1,2y+1,where L represents the Lth layer image.The algorithm calculates the value of the bottom layer from top to bottom according to the Gaussian pyramid and calculates the pixel value near the edge of the image based on the following formulas:(12)IL−1−1,y≈IL−1−1,y,IL−1x,−1≈IL−1x,0,IL−1nxL−1,y≈IL−1nxL−1−1,y,IL−1x,nyL−1≈IL−1x,nyL−1−1,IL−1nxL−1,nyL−1≈IL−1nxL−1−1,nyL−1−1.The camera motion pose is estimated using SFM in the vision front end. For a monocular camera, the camera pose can be estimated by the geometric relationship between two points in different locations in real space and their projected points on their respective imaging planes. As shown in Figure5, P is any point in the three-dimensional space, and its coordinates are [X, Y, Z]T; O1 and O2 are the optical centers of the two camera positions. p1 and p2 are the projection points of point P on the imaging plane I1 and the imaging plane I2, respectively. According to the pixel positions of the two matched point pairs p1 and p2, the essential matrix E and the fundamental matrix F can be obtained.Figure 5
Epipolar geometric constraints.According to the camera imaging model, we assume thatK is the camera internal parameter matrix, and R and t represent the rotation matrix and translation vector from plane I1 to plane I2, and the following formula can be obtained:(13)s1p1=KP,s2p2=KRP+t.Homogeneous coordinate transformation and normalization between 2D and 3D, we can get(14)x1=K−1p1,x2=K−1p2,where x1 and x2 represent the coordinates of pixels p1 and p2 in the normalized plane, respectively. The algorithm combines formuls (13) and (14) and multiplies by x2Tt∧ to obtain the essential matrix E and the fundamental matrix F, which can be sorted out as follows:(15)p2TK−Tt∧RK−1p1=0,E=t∧R,F=K−TEK−1,where t∧ represents the antisymmetric matrix.When there are more than eight sets of point pairs such asp1 and p2, the eight-point method can be used to construct a linear formula system for the simplified formula, and then the unique solution of R and t can be obtained.(16)x2TEx1=p2TFp1=0.When the monocular camera recovers the pose through the epipolar geometric relationship, the obtained translation is the normalized value, which has no practical significance. In order to obtain the depth information on feature points, triangulation needs to be introduced. We assume thats1 and s2 represent the depth of the two feature points; we can get(17)s2x2=s1Rx1+t.The feature point depth valuesx2∧ and x1∧ can be obtained by left-multiplying formula (17) by s1 or s2, respectively, as follows:(18)s1x2∧Rx1+x2∧t=0,s2x1∧Rx2−x1∧t=0.When the positions of multiple points in space are known, the camera pose can be estimated by the PnP algorithm. Common PnP algorithms include P3P, DLT, and BA optimization. Among them, the P3P algorithm is the most common method. The algorithm needs to know at least three points and their projection points on the camera imaging plane. Then, the camera pose can be estimated by solving the relationship between point pairs according to the similar triangle principle and the cosine theorem. A schematic diagram of the P3P relationship is shown in Figure6.Figure 6
Schematic diagram of P3P relationship.The coordinate system convention is as follows: the world coordinate system is represented by⋅w, and ⋅b and ⋅c represent the IMU coordinate system and the camera coordinate system, respectively. The relationship between the coordinate systems is shown in Figure 7. .v represents the visual reference frame in the sliding window, which is independent of the IMU measurement and can represent any frame in the visual structure. .bw represents the transformation from the IMU coordinate system to the world coordinate system; bk represents the IMU frame of the kth image; ⋅cv represents the transformation from the camera coordinate system to the visual reference system; and ck represents the camera frame of the kth image. ⋅^ represents the measured value and parameter estimation value of the sensor; ⋅¯ represents the latest scale parameter of the sliding window; and the rotation can be represented by the rotation matrix R and the quaternion q. gw=0,0,gT represents the gravity vector in the world coordinate system, and gv represents the gravity vector in the visual reference coordinate system.Figure 7
Coordinate conversion relationship.


## 2.4. IMU Preintegration

The sampling frequency of the camera used in this paper is 20 Hz, and the sampling frequency of the IMU is 200 Hz. It can be seen that the frequency of the IMU is much higher than that of the image. In order to avoid the repeated integration phenomenon caused by the frequency change of the visual frame optimization state caused by the high sampling rate of the IMU, a preintegration technique is used for all IMU sampling data between two image key frames. Furthermore, inertial measurements between adjacent image key frames are aggregated into a relative motion constraint through a preintegration technique. The principle of preintegration is shown in Figure8.Figure 8
Principle of preintegration.In Figure8, from top to bottom are the time scale line, the number of image frames generated, the number of image key frames generated, the number of IMU samples, and the IMU preintegration value.The measurement error of the system is mainly affected by bias random walkb and white noise η, and other errors such as the Markov process are ignored. Then, the measurement model of the accelerometer and gyroscope in the IMU can be expressed by the following formula:(19)ω^b=ωbt+bω+ηω12,a^b=qbwTaw−gw+ba+ηa,where ω^b,a^bt,ωbt, and awt represent the measured value and real value of angular velocity and acceleration, respectively; bω,ba,ηω, and ηa represent the random walk noise and measurement white noise of angular velocity and acceleration, respectively; and qbwT is the rotation matrix transformed from the world coordinate system to the IMU coordinate system.White noise obeys a Gaussian distribution, that is,ηa∼N0,σa2,ηω∼N0,σω2. The derivative of random walk noise also obeys the Gaussian distribution, that is, ηba∼N0,σba2,ηbω∼N0,σbω2.The differential kinematic formulas forP,V,Q (representing the position, velocity, and rotation expressed in quaternions, respectively) versus time can be written as follows:(20)p˙btw=vtw,v˙tw=atw,q˙btw=qbtw⊗012ωbt,where ⊗ represents quaternion multiplication.Through the above derivative relationship, the position, velocity, and rotation at timek + 1 can be obtained from the position, velocity, and rotation at time k and by integrating the measured values of the IMU over time Δtk. The continuous integration formula for PVQ is as follows:(21)pbk+1w=pbkw+vbkwΔtk+∫∫t∈k,k+1Rtwa^t−bat−gwdt2,vbk+1w=vbkw+∫t∈k,k+1Rtwa^t−bat−gwdt,qbk+1w=qbkw⊗∫t∈k,k+11/2Ωω^t−bwtqtbkdt,wherea^t and ω^t represent the acceleration and angular velocity measured in the IMU coordinate system, respectively. Δtk represents the time difference from the kth frame to the k + 1 frame. Rtw represents the rotation matrix from the world coordinate system to the IMU coordinate system. Because the measured a^t and ω^t belong to the IMU coordinate system, in order to transform the IMU measured value to the world coordinate system, the rotation matrix needs to be left-multiplied. Ωω means quaternion right multiplication; ωx means antisymmetric matrix in quaternion multiplication ω means the imaginary part value of quaternion). We assume that the quaternion is q=xyzs=ωs; then we have(22)Ωω=−ω×ω−ωT0,ω×=0ωzωyωz0−ωx−ωyωx0.By observing the continuous integral formula of PVQ, it can be seen that the current state is recursively obtained from the state of the previous time, and the estimated value is constantly changing. This will cause the IMU measurements to be repropagated, causing the velocity and rotation to be reintegrated after each nonlinear optimization iteration, resulting in a higher computational cost. Therefore, the optimization variables are separated from the IMU preintegration terms of the two key frames, and the rotation matrixRwbk of the world coordinate system to the IMU coordinate system can be obtained by simultaneously left-multiplying the left and right sides of the continuous integration formula of PVQ:(23)Rwbkpbk+1w=Rwbkpbkw+vbkwΔtk−12gwΔtk2+αbk+1bk,Rwbkvbk+1w=Rwbkvbkw−gw+βbk+1bk,qwbk⊗qbk+1w=γbk+1bk.The image framesbk and bk+1 of two consecutive moments are given, and the linear acceleration and angular velocity are preintegrated in the local coordinate system bk to obtain(24)αbk+1bk=∫∫t∈k,k+1Rtbka^t−batdt2,βbk+1bk=∫t∈k,k+1Rtbka^t−batdt,γbk+1bk=∫t∈k,k+121/2Ωω^t−bωtγtbkdt,where αbk+1bk,βbk+1bk,γbk+1bk represent the relative pose, velocity, and rotation constraints, respectively, and are also the relative motion of bk+1 to bk. It can be seen that they are only related to a^t and ω^t in bk and bk+1, and they have nothing to do with the initial position and velocity of coordinate system bk.Therefore, the preintegration formula is rediscussed, in terms ofαbk+1bk; it is related to a^t and ω^t of the IMU; and a^t and ω^t are also variables that need to be optimized. When the bias change is small, αbk+1bk,βbk+1bk,γbk+1bk are adjusted according to their first-order approximations to the bias.(25)αbk+1bk≈α^bk+1bk+Jbaαδba+Jbωαδbω,βbk+1bk≈β^bk+1bk+Jbaβδba+Jbωβδbω,γbk+1bk≈γbk+1bk⊗012Jbaγδbω,where Jbaα and Jbωα are the block matrices in Jbk+1α and Jbaβ and Jbωβ are the block matrices in Jbk+1β.There are errors in the integral values of the IMU at different times, and the errors at timet are mainly related to the measured values of αtbk,βtbk, θtbk,bat, and bwt at time t. The following definition is given to represent the error vector:(26)δztbk=δαtbkδβtbkδθtbkδbatδbωt.The derivation is based on the derivative of the error term kinetic formula. First, two concepts are introduced: true and nominal, where true represents the real measurement value containing noise and nominal represents the theoretical value without noise, andδ represents the measurement error; there are(27)δα˙=α˙true−α˙nominal,δβ˙=β˙true−β˙nominal.Among them, there are:(28)β˙true=Rtbka^t−ηa−bat−δbat−a^t−bat×δθ,β˙nominal=Rtbka^t−bat.Combining the above formulas, we can obtain(29)δβ˙=−Rtbka^t−bat×δθ−Rtbkδbat−Rtbkηa.The derivation ofδθ˙ is as follows, and according to the formula in the literature, it can be known that(30)qtrue=qnominal⊗δq.In this paper, according to the noise model and bias, we can get(31)δθ˙≈−ω^t−bωt×δθ−ηω−δbωt.In summary, the derivative of the IMU measurement error term at timet can be as follows:(32)δα˙tbkδβ˙tbkδθ˙tbkδb˙atδb˙ωt=0I00000−Rtbka^t−bat×−Rtbk000−ω^t−bωt×0−I0000000000δαtbkδβtbkδθtbkδbatδbωt+cηaηωηbaηbω.We setFt=0I00000−Rtbka^t−bat×−Rtbk000−ω^t−bωt×0−I0000000000,Gt=0000−Rtbk0000I0000I0000I. The above formula can be simplified to(33)δz˙tbk=Ftδztbk+Gtnt.According to the definition of the derivative, the prediction formula of the mean is as follows:(34)δz˙tbk=limδt⟶0δzt+δtbk−δztbkδt,δzt+δtbk=1+Ftδtδztbk+Gtδtnt.According to the error value at the current moment, the mean and covariance at the next moment can be predicted. The prediction formula for covariance is as follows:(35)Pt+δtbk=1+FtδtPtbk1+FtT+GtδtQGtδtT,where Ptbk represents the initial value of the iteration and its value is zero and Q represents the diagonal covariance matrix of the noise term as follows:According to formula (35), the iterative formula of the error term Jacobian can be obtained as follows:(36)Jt+δt=1+FtJt,where the iterative initial value of the Jacobian matrix Jt is I.


## 2.5. Sliding Window Initialization

When the camera extrinsic parameter matrixp¯bkv,p¯ckv is known, the pose obtained by the initialization of the monocular camera is transformed from the visual coordinate system to the IMU coordinate system to obtain the following formula:(37)qbkv=qckv⊗qbv,sp¯bkv+qckvpbc=sp¯ckv,where s is the translation factor obtained by visual initialization, which has no real information.The pure visual initialization method lacks absolute scale information. Therefore, the value estimated by the visual SFM is aligned with the IMU after preintegration to estimate the true scale. Visual-inertial alignment initialization is mainly to solve the following problems, including the initialization of gyroscope bias, the initialization of velocity, gravitational acceleration, and scale.The first is to initialize the gyroscope. The gyroscope bias can be obtained from two consecutive key frames with known orientations, considering two consecutive framesbk and bk+1 in the sliding window, and qbkv and qbk+1v represent the rotations obtained from the pure visual sliding window optimization, respectively. Linearize the IMU preintegration term for the gyroscope bias and minimize the following function:(38)minδbw∑k∈ℬqbk+1v−1⊗qbkv⊗γbk+1bk2.Among them, there are:(39)γbk+1bk≈γ^bk+1bk⊗Jbwγ1δbw.In formula (42), ℬ represents all the frames in the window, and the product of the two quaternions indicates that the camera rotates from the kth frame to the k + 1th frame, and the gyroscope rotates from the k + 1th frame to the kth frame, and the optimized objective function is(40)qbk+1v−1⊗qbkv⊗γbk+1bk=10.The algorithm takesγ^ into the formula and multiplies the inverse moment ordering of the relative constraints obtained from the preintegration to the left on both sides of formula (40) and obtains by Cholesky decomposition (multiplying the transpose of Jbwγ on both sides of the formula):(41)JbwγTJbwγδbw=2JbwγT2γ^bk+1bk−1⊗qbkv⊗qbk+1v−1vec.In this way, the initial calibration value of the gyroscope biasbw can be estimated, and then the IMU preintegration terms α^bk+1bk,β^bk+1bk,γ^bk+1bk are corrected with the new gyroscope bias.The second is the initialization of velocity, gravitational acceleration, and scale. The initialized state vector is as follows:(42)χI=vb0v,vb1v,…vbnv,gv,s,where the state vectorvbkv represents the speed of the visual coordinate system of the kth frame image, gv represents the gravity vector in the visual coordinate system, and s is the estimated scale parameter. To sum up, the dimension of χI is 3(n + 1) + 3 + 1. The constraint relationship between the scale parameter and the speed of the visual SFM is as follows:(43)z^bk+1bk=α^bk+1bkβ^bk+1bk=Hbk+1bkχI+nbk+1bk≈,−qvbkΔtk012qvbkΔt2qvbkp¯bk+1v−p¯bkv−qvbkqvbkqvbkΔtk0vbkvvvk+1gvgv,where qbkV,qbkV,qbkV are all obtained from visual SFM. qVbk and qbkV are mutually inverse matrices. The following linear least squares problem is constructed to complete the initialization of velocity, gravitational acceleration, and scale:(44)minχI∑k∈ℬz^bk+1bk−Hbk+1bkχI2.


## 2.6. Monocular Visual Inertial Coupling Nonlinear Optimization

When coupling the visual constraint value and the IMU constraint value, the data of the inertial sensor should be introduced first, and the constraint value of the IMU on the state should be added to the optimized state vector. Then, nonlinear optimization is performed within a sliding window, and all state vectors of the sliding window are as follows:(45)χ=x0,x1,…xn,xcb,λ0,λ1,…λm,xk=pbkw,vbkw,qbkw,ba,bg,kϵ0,n,xcb=pcb,qcb,where xk represents the state of the IMU when the kth image is captured. There are n + 1 states in the sliding window, and each state contains the position, velocity, and rotation in the world coordinate system, and the IMU offsets in the IMU coordinate system. λm represents the inverse depth information of the mth 3D point, and xcb represents the external parameter from the camera to the IMU. The objective function is(46)minχrp−Hpχ2+∑k∈ℬrℬz^bk+1bk,χpbk+1bk2+∑l,jóCρrCz^lcjpl2cj,where Hp is the Huber norm, which is defined as follows:(47)ρs=1,s≥1,2s−1,s<1.In formula (52), ⋅P represents the Mahalanobis distance weighted by the covariance matrix P, and rp,Hp,rBz^bk+1bk,X, and rCz^lc,X represent the marginalized prior information, the IMU measurement residual, and the visual reprojection error, respectively. ℬ is the set of all IMU measurement frames, and C is the set of visual features in the sliding window.According to the Gauss–Newton method, the incremental method can be used to calculate the minimum value of the Gaussian objective function, as follows:(48)minδXrℬz^bk+1bk,χ+ΔXpbk+1bk2=minδXrℬz^bk+1bk,χ+Jbk+1bkΔXpbk+1bk2,where Jbk+1bk is the Jacobian matrix of the error term rℬ with respect to all state vectors χ.The algorithm differentiates the above formula fromΔX and then sets its derivative to 0, and the formula for the increment ΔX can be calculated as follows:(49)Jbk+1bkTPbk+1bk−1Jbk+1bkΔX=−Jbk+1bkTPbk+1bk−1Jbk+1bkrℬ.The overall incremental equation of the objective function is as follows:(50)Hp+∑Jbk+1bkTPbk+1bk−1Jbk+1bk+∑JlcjTPlcj−1JlcjΔX=bp+∑Jbk+1bkTPbk+1bk−1rℬ+∑JlCjTPlCj−1rC,where Pbk+1bk represents the covariance of the IMU preintegrated noise term and PlCj represents the visually observed noise covariance. When the noise covariance Pbk+1bk of the IMU is larger, the inverse of Pbk+1bk, that is, its information matrix is smaller, which means that the IMU observations are not as reliable as the visual observations. Formula (50) can be simplified to(51)Λp+ΛB+ΛCΔX=bp+bB+bC,where Λp,ΛB, and ΛC represent the Hessian matrix. Using the perturbation method to calculate, we can get(52)J=∂r∂X=limδX⟶0rX⊕δX−rXδX,where δX represents the small disturbance of the state vector instead of the incremental ΔX,⊕ represents the disturbance of the state vector.The continuous preintegration formula is derived in the IMU preintegration, and the IMU residuals are as follows:(53)rℬz^bk+1bk,χ=δαbk+1bkδβbk+1bkδθbk+1bkδbaδbg=Rwbkpbk+1w−pbkw−vbkwΔtk+12gwΔtk2−αbk+1bkRwbkvbk+1wΔtk−vbkwΔtk+gwΔtk−βbk+1bk2γbk+1bk⊗qbkw⊗qbk+1wxyzbabk+1−babkbωbk+1−bωbk.According to the above formula, it can be seen that the optimization variables of the IMU residual mainly include the position, rotation, speed, and inertia bias at thei and j times:(54)pbkw,qbkw,vbkw,bak,bωk,pbk+1w,qbk+1w,vbk+1w,bak+1,bωk+1.To calculate the Jacobian matrix, perturbation is added to each optimization variable to obtain(55)δpbkw,δθbkw,δvbkw,δbak,δbωk,δpbk+1w,δθbk+1w,δvbk+1w,δbak+1,δbωk+1.Taking the partial derivatives for the above disturbance variables, respectively, we can get(56)J015×7=∂rBδpbkw,∂rBδθbkw,J115×7=∂rBδvbkw,∂rBδbak,∂rBδbωk,J215×7=∂rBδpbk+1w,∂rBδθbk+1w,J315×7=∂rBδvbk+1W,∂rBδbak+1,∂rBδbωk+1.The visual residual is a reprojection error, which represents the difference between the estimated value and the observed value of the feature point in the normalized camera coordinate system. The small receiver camera used in this paper belongs to the fisheye camera model and belongs to the fisheye with a large viewing angle, so its projection on the unit sphere needs to be considered, as shown in Figure9.Figure 9
Unit spherical projection model.Through the unit spherical projection model illustrated in the figure above, the value of the visual residual is decomposed into two directions. The final visual residual model looks like this:(57)rcz^lcj,χ=b1,b2T⋅p^lcj−plcjplcj,p^lcj=πc−1u^lcjv^lcj,plcj=RcbRwbjRbiwRcb1λlP¯lci+pcb+pbiw−pbjw−pcb,where p¯^lcj and plcj represent the estimated and observed coordinates of the landmark point 1 in the j-th frame image under the normalized camera coordinate system, respectively. The optimization variables of the visual residual are as follows:(58)pbiw,qbiw,pbjw,qbjw,pcb,qcb,λl,where λl represents the inverse depth value when the landmark point 1 is first observed by the j-th image. The inverse depth is used as the optimization variable because the inverse depth satisfies the Gaussian distribution, and it can reduce the parameter variables in the actual optimization process. According to the above formula, by adding disturbance to each optimization variable, the following Jacobian is obtained:(59)J03×7=∂rC∂pbiw,∂rC∂qbiw,J13×7=∂rC∂pbjw,∂rC∂qbjw,J23×7=∂rC∂pcb,∂rC∂qcb,(60)J33×1=∂rC∂λl.


## 3. E-Sports Training System Based on Intelligent Gesture Recognition

This paper combines the finger joints and the sensor in the data glove to demarcate the movement of the finger joints. This paper mainly considers the distal phalanx of the thumb (TDP) and the proximal joint proximal phalanx of the thumb (TPP) of the thumb as shown in Figure10 and the changes in the intermediate joints middle phalanges (MP) and proximal phalanges (PP) of the remaining four fingers.Figure 10
Demarcation boundaries of hand movements.This paper combines the algorithm part of the second part to construct the e-sports training system, and the overall framework of the system is shown in Figure11.Figure 11
Overall system framework.The simulation of the system proposed in this paper is carried out through the MATLAB platform, and the gesture recognition effect of the system and the application effect in the e-sports training system are evaluated, and the obtained results are shown in Tables1 and 2.Table 1
Gesture recognition effect of the system.
NumberGesture recognitionNumberGesture recognitionNumberGesture recognition186.0951387.3632586.633288.7821489.1242690.414390.6951590.0552786.697486.3251689.9682886.916586.7051788.4822989.003689.5571891.2783088.446786.6231989.2153189.911891.2812086.2723290.202988.0502191.2533389.5071087.6212286.7703488.1301188.4982386.8433591.3261286.3042490.6143686.846Table 2
The application effect of the method proposed in this paper in the e-sports training system.
NumberTraining effectNumberTraining effectNumberTraining effect182.6851382.2342579.365280.3151478.5372681.356379.7001581.9132778.790478.1791682.9382879.700582.9421778.1672978.194680.8211878.2503078.253781.1761981.5123180.133878.1832080.0763282.668982.6362181.9253383.7471081.6132282.8573482.3541180.5142382.9513578.4161280.9132479.0663683.128It can be seen from the above research that the electronic economic training system based on intelligent gesture recognition proposed in this paper has certain effects.


## 4. Conclusion

As emerging sports, e-sports are mainly participated by the younger generation, which has the characteristics of younger and younger age. E-sports can exercise people’s thinking ability, psychological pressure resistance, unity and cooperation, hand-eye coordination, and so on. Moreover, in the process of participating in e-sports, it can also make the younger generation have the awareness of abiding by the rules, cultivate the participants to have a fair and open, never admit defeat, pursue a stronger competitive spirit, and have a positive impact on the participants’ lives. This paper combines the intelligent gesture recognition technology and the construction of the performance e-sports training system and judges the training effect of the players through the player gesture recognition. The research shows that the electronic economic training system based on intelligent gesture recognition proposed in this paper has certain effects.



---

*Source: 2689949-2022-07-09.xml*