# A Cross-Platform Web3D Monitoring System of the Three-Machine Equipment in a Fully Mechanized Coalface Based on the Skeleton Model and Sensor Data

Authors: Zhengxiong Lu; Wei Guo; Shuanfeng Zhao; Chuanwei Zhang; Yuan Wang; Yanping Jia
Journal: Journal of Sensors
 (2020)

---

## Abstract

A fully mechanized coalface is a rugged environment that has poor visibility. The traditional video monitoring system has problems such as a lack of realism, a blurry monitoring effect, and poor reliability. It is an important task to monitor the operations of the three-machine equipment (we will refer to the shearer, hydraulic support, and scraper conveyor as the three-machine equipment) intuitively, accurately, and timely and ensure that it is operating safely. This study proposed a cross-platform Web3D monitoring system for the three-machine equipment. First, the virtual mesh model and skeleton model that was embedded in the mesh model were established according to three-machine ontology and basic motion units. Second, the kinematic model of the three-machine skeleton was established via the inverse kinematic modeling of the hydraulic support and the coordinate calculation of the vertices on the three-machine skeleton. Third, the motion data, which were captured by sensors, were applied to drive the movement of the three-machine skeleton and mesh model. Finally, WebGL, which is the latest Internet graphics standard, was used to render the three-machine models, and the performance of this monitoring system is tested on different equipment in the laboratory. The results of the test show that the three-machine cross-platform monitoring system has splendid performance, and it realizes cross-platform 3D monitoring effectively in the laboratory. In the future, this system will be used as a supervisory tool and be integrated with the traditional monitoring system to monitor the three-machine equipment with the field staff.


---

## Body

## 1. Introduction

Mining coal is one of the most important fundamental industries, which plays a crucial role in the development of the national economy. However, due to the poor environment and abstract monitoring results of the mining process in the fully mechanized coalface, technicians not can quickly and intuitively get the details of fully mechanized mining equipment in the mining process. It is difficult to have a full observation and a further comprehension on the operating details of mining equipment in a working face, which leads to the possibility of the emergence of various equipment faults and threaten the lives of staff, which has a direct impact on the coal production efficiency. If the whole working process is displayed intuitively in a more realistic manner, then the equipment reliability, the working efficiency, and security will be greatly improved.With the advances in computer technology nowadays, virtual reality (VR) technology has been rapidly developing and, if combined with sensing technology of the three-machine equipment, shows great potential for remotely visualizing the entire mining process for improving the efficiency of production and reducing the number of miners in the fully mechanized coalface. Currently, VR technology is used in the field of coal mining because of its intuitive, immersive, and interactive features and has led to many research advancements [1–3]. Tichon et al. [4–6] applied VR to miners’ safety training and effectively improved their safety level. Stothard [7, 8] designed a VR coal industry training simulator that allows miners to experience the consequences of decision-making errors and learn from their mistakes. Based on complete simulation data, Akkoyun [9, 10] established an interactive mining and engineering visualization environment for teaching that modeled an open pit of magnesite. Torano et al. [11] simulated the behavior of the roof in longwall mining and realized the 3D display of the longwall mining face using the VRML language. Wan and Xu [12, 13] established a simple GUI interactive interface to simulate the mining process of a fully mechanized mining face. Zhang [14] obtained dynamic images that were a fusion of real images and a virtual scene of the fully mechanized coal mining face. These dynamic images had better results that were closer to reality. However, the functions of them are still relatively monotonous, which is mainly used for teaching and demonstration. These technologies have not been used to studying collaborative work of the three-machine equipment. Foster et al. [15, 16] conducted a training operation simulation for underground miners in an interactive and immersive environment, where the simulated continuous mining machine and drilling machine could be operated to train remotely. Yan et al. [17] used dynamic analysis and fast solution technology to synchronize the working state of a hydraulic support. Zhang et al. [18, 19] established a series of virtual mining models that included a shearer, scraper conveyor, and hydraulic support and simulated the entire kinematic process and technique of a fully mechanized coalface. Xie et al. [20] studied the key technologies of the virtual cooperative operations of fully mechanized mining by establishing a panoramic scene of fully mechanized virtual reality and more realistically reproduced the operating status of the three fully mechanized mining machines. However, none of these systems satisfies the demand of the directly displaying operation condition of the three-machine equipment in real time.So far, many systems based on VR have been developed. Liao et al. [21] presented a collaborative and interactive design framework of a MEMS design system used in dynamic web Java Server Pages and computer graphics virtual reality modelling language. Birr et al. [22] established a real-time interactive Web3D system to learn 3D models based on WebGL and obtained a good demonstration effect. Sun et al. [23] developed a virtual reality system (VRCast) for the casting production process using WebGL technology and performed a virtual display of the casting process in a network environment. Bian et al. [24] developed a Web3D exhibition building system to build Web3D exhibitions in web applications, which has good compatibility and runs depending on the GPU without any plug-in. Xia et al. [25] established a zero plug-in for the cross-platform EAST visualization system based on WebGL, which allows users to browse the EAST device model and visually monitor the EAST experimental data through a web browser. In recent years, the rapid development of Internet technology, particularly the smart mobile technology, has penetrated into every aspect of life. It is a great challenge to combine Internet technology with VR to get a more powerful system.As mentioned above, the VR simulations in the mining field focused on aspects including safety training, operation process simulation, kinematics simulation, and visualization of the underground scene. It has been used mainly in the training and teaching fields and does not have as much prevalence in the industrial field. Therefore, the application level of VR technology in the mining industry is still relatively low, as there is no real industrial application. Web3D monitoring has not yet been applied in the mining industry. With the rapid development of Web3D applications, Web3D technologies such as OpenGL, VRML/X3D, Java3D, Cult3D, Flash3D, Unity3D, and WebGL are constantly appearing. Each technology has its own advantages in rendering speed and interactive performance [26–29]. As an open-source 3D scene rendering API, WebGL not only displays 3D virtual scenes without plug-ins and delays in the browser through the GPU but also has good interactive performance and wide applications [30]. Hence, in view of the advantages of virtual reality and Internet technology, this research has developed a lightweight Web3D system, namely, a cross-platform Web3D monitoring system of the three-machine equipment based on multisensor data. It is convenient to be use without platform limit. In order to solve the difficulties of 3D graphics rendering in web browsers and the lightweight requirement of no plug-ins, the latest Internet graphics standard WebGL was used in the proposed system.A fully mechanized coalface is the most critical link in the coal industry. Therefore, based on the above analysis, this paper will establish a Web3D monitoring system that can solve the problem of cross-platform remotely monitoring the three-machine equipment as a foundation that complements the traditional monitoring system in the fully mechanized coalface. This advancement will help miners quickly make decision in the remote control center and increase production. The remainder of this paper is organized as follows: Section2 presents the structure of the Web3D monitoring system. In Section 3, we present the key technologies of establishing the kinematic model of skeleton model of the three-machine. Section 4 amply describes the implementation process. In Section 5, the performance of Web3D monitoring system is tested. This paper is concluded in Section 6.


## 2. Architecture Overview

A cross-platform Web3D monitoring system of the three-machine equipment is established in the laboratory as a supervision tool to complement the direct surveillance by workers. The structure of the monitoring system of the three-machine equipment is shown in Figure1.Figure 1
The structure of the Web3D monitoring system of the three-machine equipment in a fully mechanized coalface.The system adopts the B/S (Browser/Server) structure, which has good compatibility, excellent cross-platform capabilities, and a perfect interactional mechanism. The data on the 3D model, scene, and working information of the three-machine equipment will be displayed in this system. The positions of the virtual model of the three-machine equipment are determined by the coordinates in the virtual scene. When the user accesses this system, the web server sends the HTML file, the CSS file, and the JS file to the client browser through the HTTP protocol; then, the browser-based WebGL renders the working state of the three-machine equipment. To achieve real-time display, we apply a perceptive system with sensors to obtain the working information of the three-machine equipment, which will be conveyed to three-machine equipment’s S7-1200PLC. The working information is stored in the SQL Server database via the OPC technology, which is processed by a program that calculates the state model of the three-machine equipment to reveal its operating process.In this study, MVC technology was used to build the system’s program framework, and the structure is shown in Figure2. The view layer uses the WebGL graphics technical standard that invokes OpenGL ES and the Three.js graphics function library through the JavaScript language. The GPU acceleration technology is used to render the 3D model file of the three-machine equipment on the HTML5 canvas. The controller receives the user instruction, calls the model, and then calculates the three-machine equipment’s working status, which provides the condition for the driver of the three-machine model. The 3D model files and running position files of the three-machine equipment are encapsulated in the model layer.Figure 2
System program structure based on the MVC.


## 3. Key Technologies

### 3.1. Kinematic Model of the Three-Machine Equipment

Effective monitoring results will provide information on the position of the three-machine equipment in real time. The Web3D monitoring system includes two key technologies. The movement of 3D models must conform to the kinematic characteristics of the three-machine equipment in order to obtain a realistic image of the three-machine equipment. To ensure the smoothness, the rendering speed of the three-machine equipment must be fast enough. The time delay in rendering the real three-machine model cannot be too high; otherwise, the server will not be able to calculate the situation of the three-machine equipment in a timely manner, which can result in incorrect judgments and production accidents.To realize the real-time 3D monitoring of the three-machine equipment, we use the WebGL-based lightweight 3D development framework Three.js and the “skeletal skin animation” technology [31]. The experimental platform sets the MG300/700-AWD shearer, the ZF6800/17/32 hydraulic support, and the SGZ-764/630 scraper conveyor as the original objects, and it sets the shrank pro rata as 8 times to complete the construction of the model according to the matching relationship of the three-machine of the fully mechanized coalface. According to the design drawing of the experimental platform, the 3D model of the fully mechanized mining machine is established at a 1 : 1 ratio in the 3D modeling software SolidWorks. Then, the 3D model is exported to the 3DMAX software in the STL format to complete the repair and optimization of the model. In particular, the connection constraint components between the three machines must be accurately modeled as the actual constraint relationship to ensure that the components remain in the assembled relationship in SolidWorks, and the three-machine equipment’s operating relationship [32] remains correct after the STL file of the fully mechanized three-machine model is imported into 3DMAX. The three-machine physical experiment platform and the virtual 3D model of the fully mechanized mining are shown in Figures 3(a) and 3(b), respectively.Figure 3
Experimental platform and virtual model for the collaborative control of the three-machine equipment. (a) Three-machine collaborative control experimental platform. (b) Three-machine virtual model.
(a)(b)The mathematical description of the positional relationship between the moving parts of the 3D model of fully mechanized mining is the basis for establishing its kinematic model. In this study, a Cartesian coordinate system is established at the moving nodes of each component to describe its positional relationship. The pose transformation of each moving part of the three machines is studied using a homogeneous coordinate transformation matrix. The form is shown in formula (1). The homogeneous matrix can describe the motion transformation of each moving component in the three-machine linkage process and intuitively represent the rotation and movement of the moving component in the Cartesian coordinate space. The parts when they are continuously rotated or moved are described using the matrix’s multiplication form.
(1)rotationmatrix⋮displacementvector⋯⋯⋯0⋮scalefactor=a11a21a31⋯0a12a22a32⋯0a13a23a33⋯0⋮⋮⋮⋯⋮pxpypz⋯1.As a link mechanism, the hydraulic support sets its own local coordinate system at the lower hinge point of each link, and the sublinks are affected by the movement of the parent link. The relative motion between the links is described in the form of a parent-child relationship chain, and the last-level guard panel inherits the transformation of the multilevel coordinate system. Each section of the scraper conveyor, the corresponding shifting lever, and the shifting cylinder are regarded as independent moving units. A local coordinate system is set at the hinge of the middle slot and the shifting lever to describe the displacement of the central slot. A local coordinate system is set at the hinge point of the shifting cylinder and the hydraulic support to describe the displacement of the hydraulic bracket, and each moving unit combination completes the production operation of the scraper transporter. The shearer is abstracted as a linkage mechanism, and the drum will inherit the transfer transformation of the multilevel coordinate system.According to the movement characteristics of each moving part of the three-machine equipment that conducts fully mechanized mining, the relative position and attitude between these coordinate systems are described by a homogeneous coordinate transformation. We define homogeneous coordinate transformation matrixR to describe the relative translation and rotation between adjacent connecting rod coordinate systems. We set R1 to represent the position of the first connecting rod relative to the basic coordinate system and R2 to represent the position of the second connecting rod relative to the first connecting rod. Then, the position transformation matrix of the second connecting rod relative to the basic coordinate system is as follows: T=R1R2. These steps repeat. The posture transformation matrix of the third connecting rod relative to the basic coordinate system is T=R1R2R3, and the posture transformation matrix of the nth connecting rod relative to the basic coordinate system is T=R1R2R3,⋯,Rn−1Rn−2.


### 3.2. Establishment of the Three-Machine Skeleton Model

The skeleton model involves translation, rotation, and scaling, which can skillfully realize the motion process of the moving parts of the three-machine equipment. Therefore, it is necessary that the skeletal model of the three-machine equipment is established in order to drive the 3D model of the three-machine equipment. The skeleton models of the shearer, hydraulic support, and scraper conveyor are shown in Figures4–6, respectively.Figure 4
The skeleton model of the shearer.Figure 5
The skeleton model of the hydraulic support.Figure 6
The skeleton model of the scraper conveyor.The establishment process of the skeleton model of the three-machine equipment is as follows. First, the structure of the skeleton model is abstracted according to the connective position of the pin shaft in the shearer, hydraulic support, and scraper conveyor. Then, the skeleton is drawn according to the structure of the three-machine equipment using 3DMAX. Finally, the skeleton model of the three-machine equipment is established by the joints connecting all skeletons. In addition, the binding of the skeleton and the 3D virtual model is established. The “envelope” function is used to bind each bone and its mesh of the three-machine equipment. The weight between the bone and the skin vertex is adjusted according to the effect of the skin traction action of the bone to realize the fine control of the current frame vertex [33].


### 3.3. Establishment of the Kinematic Model of the Skeleton Model

The three-machine skeleton model provides the foundation for the data drive of the three-machine virtual model. To realize the synchronous movement of the virtual three-machine model and its ontology, it is necessary to realize the synchronous motion of the skeleton and the ontology of the three-machine equipment.


#### 3.3.1. Calculation Method of the Vertex Coordinate in Moving Skeleton

Given the world coordinate systemOx,y,z of the virtual 3D space, O1dxp,dyp,dzp is the coordinate of the origin of the parent coordinate system in the world coordinate system Ox,y,z. O2dxc,dyc,dzc is the coordinate of the origin of the subcoordinate system in the world coordinate system Ox,y,z. The coordinate that is a vertex of the mesh is Vcxc,yc,zc in the subcoordinate system O2dxc,dyc,dzc, as shown in Figure 7.Figure 7
Virtual space coordinate system and its transformation.When the skeletons are stationary, the coordinate of vertexV is translated from the subcoordinate system O2−dxcdycdzc to the parent coordinate system O1−dxpdypdzp using formula (2).
(2)Vp=Vc⋅Mcp,where Mcp is the translational matrix, and Vp is the transformed vertex coordinate from the subcoordinate system to the parent coordinate system.The coordinate of vertexVp is translated from the parent coordinate system O1−dxpdypdzp to the world coordinate system O−xyz using formula (3).
(3)Vw=Vp⋅Mpw,where Mpw is the translational matrix, and Vw is the transformed vertex coordinate from the parent coordinate system to the world coordinate system.Therefore, the translational process of vertexV of the mesh from the subcoordinate system O2−dxcdycdzc to the world coordinate system O−xyz is as follows:
(4)Vw=Vc⋅Mcp⋅Mpw=Vc⋅100001000010dxc−dxpdyc−dypdzc−dzp1⋅100001000010dxpdypdzp1.Moreover, the motion process of skeletons is analyzed. The motion process of the skeletons is resolved into the rotational and translational motion in its local coordinate system, which is described by a single four-dimensional homogeneous matrix using formula (1). The motion matrix in the local coordinate system of the child skeletons is Mlc. The motional matrix in the local coordinate system of the parent skeletons is Mlp. The motion transformation of the child skeleton in the local coordinate system is converted to the parent skeleton coordinate system as follows:
(5)Vp′=Vc⋅Mlc⋅Mlp,where Vp′ is the coordinate of vertex V in the parent coordinate system O1−dxpdypdzp.When coordinateVp′ of vertex V is transformed into the world coordinate system O−xyz, a new coordinate Vw′ of vertex V is obtained in the world coordinate system after bone movement. The transformational process is as follows:
(6)Vw′=Vp′⋅Mlp⋅Mpw=VcMlc⋅Mcp⋅Mlp⋅Mpw.Although the chain hierarchy of coordinate systems is complex, we can obtain the position of the vertices that are bound to the movement bone in the local coordinate system and calculate the position of the vertices in the world coordinate system after motion. When the vertices of the three-machine mesh model are traversed and calculated using the same method, the positions of the three-machine model are updated.The updated coordinates of the vertexes that are bound to the skeleton are calculated. The initial coordinates of the vertices of the mesh model in the world coordinate system are converted into the local coordinate system of the bone where the vertices are located via coordinate transformation. Take Figure8 as an example for analysis. Set the coordinate of vertex V at the initial position of the world coordinate system as Vw, and the world coordinate of vertex O2 that is the origin of the local coordinate system of vertex V is O2′dxc,dyc,dzc.
(7)VcO2=VwO+OO2.Figure 8
Skeletons “b” and “c” rotate around pin shaft node B by angleφ in the plane ABC.The coordinates of vertexV in its local coordinate system are calculated by formula (7). Its new coordinates are Vc:
(8)xc,yc,zc=xw−dxcyw−dyczw−dzc1=xwywxw1100001000010−dxc−dyc−dzc1=VwMwc.Thus, the coordinates of the grid vertices in the world coordinate system in their local coordinate system can be expressed as the product of matrixVw and the homogeneous coordinate system transformation matrix Mwc. The comprehensive reasoning shows that the world coordinate system of mesh vertices when the skeleton is moving is
(9)Vw′=Vc⋅Mwc⋅Mlc⋅Mcw=Vc⋅Mwc⋅Mlc⋅Mcp⋅Mlp⋅Mpw.Formula (9) implements the grid vertex coordinate transformation of two adjacent levels of the skeleton, and the method for calculating the vertex of the multistage skeleton motion is the same.The initial state of the vertex motion of the coordinate ofVw is the position in the world coordinate system, and the coordinate can be read from the grid model file. All vertices can be compared using the above formula to calculate the new coordinates after skeleton movement to update the position of the entire mesh vertex and realize the movement and rotation of the three-machine virtual model. The transform matrices between multistage skeletal coordinates, such as Mcp, and the initial values of the world coordinates O1 and O2 of the origin of each skeletal coordinate system can be obtained through the grid file. However, a matrix of motion transformations that occur at all levels of the skeleton under their respective local coordinate systems, such as Mlc and Mlp, cannot be obtained from the grid file. Without these skeleton transformation matrices, it will not be possible to realize the mathematical calculation of the model’s motion transformation. In the virtual 3D space, the motion of the 3D model of the three-machine equipment is driven by the motion of the skeletal model embedded in the mesh skin. Therefore, in order to control the motion of the 3D model of three-machine equipment, it is necessary to study the motion transformation calculation method of the joints in the skeleton model. The movement of the parts of the three-machine equipment includes the rotation around the axis and the movement in a specific plane, and the rotation movement mode is shown in Figure 8. Movement is achieved by the stretching skeletons. In the figure, A, B, C, and D are pin shaft nodes, and “a,” “b,” and “c” are skeletons. Suppose that skeletons “b” and “c” rotate around pin shaft node B by angle φ. The coordinate of point A is Axa,ya,za, the coordinate of point B is Bxb,yb,zb, and the coordinate of point C is Cxc,yc,zc. This transformation is calculated using the axial angle method [34].Regarding vectorBA=xa−xb,ya−yb,za−zb and vector BC=xc−xb,yc−yb,zc−zb, this rotation can be expressed as a counterclockwise rotation of vector BC around point B, that is, BA×BC. The direction is vertical plane ABC outward. The transformation matrix of skeleton “b” rotating by angle φ at point B is M1, and the new coordinate of point C point after the rotation is C′=BC·M1+B; the pseudocode programmed with the Three.js function is as follows:vec3BA = new THREE.Vector3(xa-xb,ya-yb,za-zb)//Compute vector BA;vec3BC = new THREE.Vector3(xc-xb,yc-yb,zc-zb)//Compute vector BC;vec3BA×BC = new THREE.Vector3.multiplyVectors(BA, BC)//Calculate the fork product of vector BA and vector BC, and calculate the normal vector of plane ABC;vec3 normal = normalize(vec3(BA×BC))//The normalized of normal vector of plane ABC;M1 = new THREE.Matrix4.makeRotationAxis(normal, φ)//A 4-order rotational transformation matrix M1 is calculated to rotate the angle φ of normal rotation around normal;vec3BC′ = normal.transformDirection9(M1)//Vector BC makes rotation transformation with matrix M1.The local coordinate system of the skeletonb is located at connection point B of skeletons “a” and “b”, and so, vector BC′ is obtained after skeleton “a” rotates by angle φ around the normal in the local coordinate system, namely, BC′ is the coordinate value of point C′ in the local coordinate system. Similarly, Mlc and Mlp can be calculated by using the method of calculating M1.


#### 3.3.2. Solution of the Reverse Kinematics of the Hydraulic Support

When calculating the rotation transformation matrix of a skeleton in its local coordinate system, the angle that each moving part turns around the pin shaft must be clearly given. As a typical four-link mechanism, the hydraulic support rotates around the shaft of the connecting rod by means of a hydraulic cylinder hinged on it. The sensor can only collect the elongations of the hydraulic cylinder and piston rod, and it cannot directly obtain the angle information. This requires the reverse solution of the rotation angle information of each hinge point according to the actual size of the hydraulic support connecting rod, the mounting position of the hydraulic cylinder, and the elongation information of the hydraulic cylinder piston rod collected. According to the actual structure of the hydraulic support of the three-machine collaborative experimental platform, we establish the coordinate system, as shown in Figure9. Through the motion model of the linkage mechanism of the hydraulic support, according to the height H of the hydraulic support, length L15 of the hydraulic cylinder of the front beam, and length L8 of the end guard plate hydraulic cylinder, the axes of the moving members around the local coordinate system at the corresponding hinge point are dz1, …, and dz5, and the angles of the turns are φ1,..., φ5.Figure 9
Dimensional structure of the hydraulic support.According to the dimensional relation of the connecting rod mechanism of the hydraulic support, as shown in Figure9, the movement of the connecting rod mechanism of the hydraulic support during the production process of fully mechanized coal mining meets the following equation:
(10)L2cosα2+L3+L4sinα=H−H0,where L2 is the length of rear link, α is the horizontal inclination of the shield beam, L3 is the distance between the hinged points on front and rear links, L4 is the length of the cover beam, α2 is the vertical inclination of the rear link, H is the height of the upright column, and H0 is the vertical distance between the hinged point and the base surface under the rear link. The formula to calculate angle φ1 is
(11)φ1=π2−α2=π2−arccosH−H0−L3+L4sinαL2,L1cosα1+L4sinα+L5cosα=H−H1,where L1 is the length of the front link, L5 is the distance between the upper hinge point of the front link and the shield beam, α1 is the vertical inclination of the front link, and H1 is the vertical distance between the hinged point and the base surface under the front link. The formula to calculate φ2 is
(12)φ2=π2−α1=π2−arccosH−H1−L4sinα+L5cosαL1.The geometric relation of Figure10 is calculated to obtain φ3:
(13)φ3=π2−θ−φ2+arctanL5L13,φ4=π−arccosL112+L62+L122−L1522L11L62+L112−arctanL6L12,where L6 is the vertical distance between the lower hinge point of the cover beam jack and the top beam, L11 is the distance from the hinge point on the cover beam jack to the turning point O4, L12 is the distance between the turning point O4 and the base of the hinge point of the top beam jack, and L15 is the length of the cover beam jack.
(14)φ5=arccosL92+L72+L102−L822L9L72+L102+arctanL7L10,where L7 is the vertical distance between the lower hinge point of the face guard jack and the cover beam, L9 is the distance from the hinge point on the face guard jack to the turning point O5, L10 is the distance between the turning point O5 and the base of the hinge point of the cover beam jack, and L8 is the length of face guard jack.Figure 10
Implementation process of the virtual three-machine equipment.During the operation of the hydraulic support, the anglesφ1, …, φ5 are the dynamic changes with the height of the support column, the length of the cover beam jack L8, and the length of the cover beam jack L15. According to the homogeneous transformation matrix of the skeleton motion of the three-machine equipment, the rotation angle of the links of the hydraulic support is substituted into the relation formula (9). The positions and postures of the virtual three-machine mesh model can be obtained during the motion process of the virtual three-machine mesh model.


## 3.1. Kinematic Model of the Three-Machine Equipment

Effective monitoring results will provide information on the position of the three-machine equipment in real time. The Web3D monitoring system includes two key technologies. The movement of 3D models must conform to the kinematic characteristics of the three-machine equipment in order to obtain a realistic image of the three-machine equipment. To ensure the smoothness, the rendering speed of the three-machine equipment must be fast enough. The time delay in rendering the real three-machine model cannot be too high; otherwise, the server will not be able to calculate the situation of the three-machine equipment in a timely manner, which can result in incorrect judgments and production accidents.To realize the real-time 3D monitoring of the three-machine equipment, we use the WebGL-based lightweight 3D development framework Three.js and the “skeletal skin animation” technology [31]. The experimental platform sets the MG300/700-AWD shearer, the ZF6800/17/32 hydraulic support, and the SGZ-764/630 scraper conveyor as the original objects, and it sets the shrank pro rata as 8 times to complete the construction of the model according to the matching relationship of the three-machine of the fully mechanized coalface. According to the design drawing of the experimental platform, the 3D model of the fully mechanized mining machine is established at a 1 : 1 ratio in the 3D modeling software SolidWorks. Then, the 3D model is exported to the 3DMAX software in the STL format to complete the repair and optimization of the model. In particular, the connection constraint components between the three machines must be accurately modeled as the actual constraint relationship to ensure that the components remain in the assembled relationship in SolidWorks, and the three-machine equipment’s operating relationship [32] remains correct after the STL file of the fully mechanized three-machine model is imported into 3DMAX. The three-machine physical experiment platform and the virtual 3D model of the fully mechanized mining are shown in Figures 3(a) and 3(b), respectively.Figure 3
Experimental platform and virtual model for the collaborative control of the three-machine equipment. (a) Three-machine collaborative control experimental platform. (b) Three-machine virtual model.
(a)(b)The mathematical description of the positional relationship between the moving parts of the 3D model of fully mechanized mining is the basis for establishing its kinematic model. In this study, a Cartesian coordinate system is established at the moving nodes of each component to describe its positional relationship. The pose transformation of each moving part of the three machines is studied using a homogeneous coordinate transformation matrix. The form is shown in formula (1). The homogeneous matrix can describe the motion transformation of each moving component in the three-machine linkage process and intuitively represent the rotation and movement of the moving component in the Cartesian coordinate space. The parts when they are continuously rotated or moved are described using the matrix’s multiplication form.
(1)rotationmatrix⋮displacementvector⋯⋯⋯0⋮scalefactor=a11a21a31⋯0a12a22a32⋯0a13a23a33⋯0⋮⋮⋮⋯⋮pxpypz⋯1.As a link mechanism, the hydraulic support sets its own local coordinate system at the lower hinge point of each link, and the sublinks are affected by the movement of the parent link. The relative motion between the links is described in the form of a parent-child relationship chain, and the last-level guard panel inherits the transformation of the multilevel coordinate system. Each section of the scraper conveyor, the corresponding shifting lever, and the shifting cylinder are regarded as independent moving units. A local coordinate system is set at the hinge of the middle slot and the shifting lever to describe the displacement of the central slot. A local coordinate system is set at the hinge point of the shifting cylinder and the hydraulic support to describe the displacement of the hydraulic bracket, and each moving unit combination completes the production operation of the scraper transporter. The shearer is abstracted as a linkage mechanism, and the drum will inherit the transfer transformation of the multilevel coordinate system.According to the movement characteristics of each moving part of the three-machine equipment that conducts fully mechanized mining, the relative position and attitude between these coordinate systems are described by a homogeneous coordinate transformation. We define homogeneous coordinate transformation matrixR to describe the relative translation and rotation between adjacent connecting rod coordinate systems. We set R1 to represent the position of the first connecting rod relative to the basic coordinate system and R2 to represent the position of the second connecting rod relative to the first connecting rod. Then, the position transformation matrix of the second connecting rod relative to the basic coordinate system is as follows: T=R1R2. These steps repeat. The posture transformation matrix of the third connecting rod relative to the basic coordinate system is T=R1R2R3, and the posture transformation matrix of the nth connecting rod relative to the basic coordinate system is T=R1R2R3,⋯,Rn−1Rn−2.


## 3.2. Establishment of the Three-Machine Skeleton Model

The skeleton model involves translation, rotation, and scaling, which can skillfully realize the motion process of the moving parts of the three-machine equipment. Therefore, it is necessary that the skeletal model of the three-machine equipment is established in order to drive the 3D model of the three-machine equipment. The skeleton models of the shearer, hydraulic support, and scraper conveyor are shown in Figures4–6, respectively.Figure 4
The skeleton model of the shearer.Figure 5
The skeleton model of the hydraulic support.Figure 6
The skeleton model of the scraper conveyor.The establishment process of the skeleton model of the three-machine equipment is as follows. First, the structure of the skeleton model is abstracted according to the connective position of the pin shaft in the shearer, hydraulic support, and scraper conveyor. Then, the skeleton is drawn according to the structure of the three-machine equipment using 3DMAX. Finally, the skeleton model of the three-machine equipment is established by the joints connecting all skeletons. In addition, the binding of the skeleton and the 3D virtual model is established. The “envelope” function is used to bind each bone and its mesh of the three-machine equipment. The weight between the bone and the skin vertex is adjusted according to the effect of the skin traction action of the bone to realize the fine control of the current frame vertex [33].


## 3.3. Establishment of the Kinematic Model of the Skeleton Model

The three-machine skeleton model provides the foundation for the data drive of the three-machine virtual model. To realize the synchronous movement of the virtual three-machine model and its ontology, it is necessary to realize the synchronous motion of the skeleton and the ontology of the three-machine equipment.


### 3.3.1. Calculation Method of the Vertex Coordinate in Moving Skeleton

Given the world coordinate systemOx,y,z of the virtual 3D space, O1dxp,dyp,dzp is the coordinate of the origin of the parent coordinate system in the world coordinate system Ox,y,z. O2dxc,dyc,dzc is the coordinate of the origin of the subcoordinate system in the world coordinate system Ox,y,z. The coordinate that is a vertex of the mesh is Vcxc,yc,zc in the subcoordinate system O2dxc,dyc,dzc, as shown in Figure 7.Figure 7
Virtual space coordinate system and its transformation.When the skeletons are stationary, the coordinate of vertexV is translated from the subcoordinate system O2−dxcdycdzc to the parent coordinate system O1−dxpdypdzp using formula (2).
(2)Vp=Vc⋅Mcp,where Mcp is the translational matrix, and Vp is the transformed vertex coordinate from the subcoordinate system to the parent coordinate system.The coordinate of vertexVp is translated from the parent coordinate system O1−dxpdypdzp to the world coordinate system O−xyz using formula (3).
(3)Vw=Vp⋅Mpw,where Mpw is the translational matrix, and Vw is the transformed vertex coordinate from the parent coordinate system to the world coordinate system.Therefore, the translational process of vertexV of the mesh from the subcoordinate system O2−dxcdycdzc to the world coordinate system O−xyz is as follows:
(4)Vw=Vc⋅Mcp⋅Mpw=Vc⋅100001000010dxc−dxpdyc−dypdzc−dzp1⋅100001000010dxpdypdzp1.Moreover, the motion process of skeletons is analyzed. The motion process of the skeletons is resolved into the rotational and translational motion in its local coordinate system, which is described by a single four-dimensional homogeneous matrix using formula (1). The motion matrix in the local coordinate system of the child skeletons is Mlc. The motional matrix in the local coordinate system of the parent skeletons is Mlp. The motion transformation of the child skeleton in the local coordinate system is converted to the parent skeleton coordinate system as follows:
(5)Vp′=Vc⋅Mlc⋅Mlp,where Vp′ is the coordinate of vertex V in the parent coordinate system O1−dxpdypdzp.When coordinateVp′ of vertex V is transformed into the world coordinate system O−xyz, a new coordinate Vw′ of vertex V is obtained in the world coordinate system after bone movement. The transformational process is as follows:
(6)Vw′=Vp′⋅Mlp⋅Mpw=VcMlc⋅Mcp⋅Mlp⋅Mpw.Although the chain hierarchy of coordinate systems is complex, we can obtain the position of the vertices that are bound to the movement bone in the local coordinate system and calculate the position of the vertices in the world coordinate system after motion. When the vertices of the three-machine mesh model are traversed and calculated using the same method, the positions of the three-machine model are updated.The updated coordinates of the vertexes that are bound to the skeleton are calculated. The initial coordinates of the vertices of the mesh model in the world coordinate system are converted into the local coordinate system of the bone where the vertices are located via coordinate transformation. Take Figure8 as an example for analysis. Set the coordinate of vertex V at the initial position of the world coordinate system as Vw, and the world coordinate of vertex O2 that is the origin of the local coordinate system of vertex V is O2′dxc,dyc,dzc.
(7)VcO2=VwO+OO2.Figure 8
Skeletons “b” and “c” rotate around pin shaft node B by angleφ in the plane ABC.The coordinates of vertexV in its local coordinate system are calculated by formula (7). Its new coordinates are Vc:
(8)xc,yc,zc=xw−dxcyw−dyczw−dzc1=xwywxw1100001000010−dxc−dyc−dzc1=VwMwc.Thus, the coordinates of the grid vertices in the world coordinate system in their local coordinate system can be expressed as the product of matrixVw and the homogeneous coordinate system transformation matrix Mwc. The comprehensive reasoning shows that the world coordinate system of mesh vertices when the skeleton is moving is
(9)Vw′=Vc⋅Mwc⋅Mlc⋅Mcw=Vc⋅Mwc⋅Mlc⋅Mcp⋅Mlp⋅Mpw.Formula (9) implements the grid vertex coordinate transformation of two adjacent levels of the skeleton, and the method for calculating the vertex of the multistage skeleton motion is the same.The initial state of the vertex motion of the coordinate ofVw is the position in the world coordinate system, and the coordinate can be read from the grid model file. All vertices can be compared using the above formula to calculate the new coordinates after skeleton movement to update the position of the entire mesh vertex and realize the movement and rotation of the three-machine virtual model. The transform matrices between multistage skeletal coordinates, such as Mcp, and the initial values of the world coordinates O1 and O2 of the origin of each skeletal coordinate system can be obtained through the grid file. However, a matrix of motion transformations that occur at all levels of the skeleton under their respective local coordinate systems, such as Mlc and Mlp, cannot be obtained from the grid file. Without these skeleton transformation matrices, it will not be possible to realize the mathematical calculation of the model’s motion transformation. In the virtual 3D space, the motion of the 3D model of the three-machine equipment is driven by the motion of the skeletal model embedded in the mesh skin. Therefore, in order to control the motion of the 3D model of three-machine equipment, it is necessary to study the motion transformation calculation method of the joints in the skeleton model. The movement of the parts of the three-machine equipment includes the rotation around the axis and the movement in a specific plane, and the rotation movement mode is shown in Figure 8. Movement is achieved by the stretching skeletons. In the figure, A, B, C, and D are pin shaft nodes, and “a,” “b,” and “c” are skeletons. Suppose that skeletons “b” and “c” rotate around pin shaft node B by angle φ. The coordinate of point A is Axa,ya,za, the coordinate of point B is Bxb,yb,zb, and the coordinate of point C is Cxc,yc,zc. This transformation is calculated using the axial angle method [34].Regarding vectorBA=xa−xb,ya−yb,za−zb and vector BC=xc−xb,yc−yb,zc−zb, this rotation can be expressed as a counterclockwise rotation of vector BC around point B, that is, BA×BC. The direction is vertical plane ABC outward. The transformation matrix of skeleton “b” rotating by angle φ at point B is M1, and the new coordinate of point C point after the rotation is C′=BC·M1+B; the pseudocode programmed with the Three.js function is as follows:vec3BA = new THREE.Vector3(xa-xb,ya-yb,za-zb)//Compute vector BA;vec3BC = new THREE.Vector3(xc-xb,yc-yb,zc-zb)//Compute vector BC;vec3BA×BC = new THREE.Vector3.multiplyVectors(BA, BC)//Calculate the fork product of vector BA and vector BC, and calculate the normal vector of plane ABC;vec3 normal = normalize(vec3(BA×BC))//The normalized of normal vector of plane ABC;M1 = new THREE.Matrix4.makeRotationAxis(normal, φ)//A 4-order rotational transformation matrix M1 is calculated to rotate the angle φ of normal rotation around normal;vec3BC′ = normal.transformDirection9(M1)//Vector BC makes rotation transformation with matrix M1.The local coordinate system of the skeletonb is located at connection point B of skeletons “a” and “b”, and so, vector BC′ is obtained after skeleton “a” rotates by angle φ around the normal in the local coordinate system, namely, BC′ is the coordinate value of point C′ in the local coordinate system. Similarly, Mlc and Mlp can be calculated by using the method of calculating M1.


### 3.3.2. Solution of the Reverse Kinematics of the Hydraulic Support

When calculating the rotation transformation matrix of a skeleton in its local coordinate system, the angle that each moving part turns around the pin shaft must be clearly given. As a typical four-link mechanism, the hydraulic support rotates around the shaft of the connecting rod by means of a hydraulic cylinder hinged on it. The sensor can only collect the elongations of the hydraulic cylinder and piston rod, and it cannot directly obtain the angle information. This requires the reverse solution of the rotation angle information of each hinge point according to the actual size of the hydraulic support connecting rod, the mounting position of the hydraulic cylinder, and the elongation information of the hydraulic cylinder piston rod collected. According to the actual structure of the hydraulic support of the three-machine collaborative experimental platform, we establish the coordinate system, as shown in Figure9. Through the motion model of the linkage mechanism of the hydraulic support, according to the height H of the hydraulic support, length L15 of the hydraulic cylinder of the front beam, and length L8 of the end guard plate hydraulic cylinder, the axes of the moving members around the local coordinate system at the corresponding hinge point are dz1, …, and dz5, and the angles of the turns are φ1,..., φ5.Figure 9
Dimensional structure of the hydraulic support.According to the dimensional relation of the connecting rod mechanism of the hydraulic support, as shown in Figure9, the movement of the connecting rod mechanism of the hydraulic support during the production process of fully mechanized coal mining meets the following equation:
(10)L2cosα2+L3+L4sinα=H−H0,where L2 is the length of rear link, α is the horizontal inclination of the shield beam, L3 is the distance between the hinged points on front and rear links, L4 is the length of the cover beam, α2 is the vertical inclination of the rear link, H is the height of the upright column, and H0 is the vertical distance between the hinged point and the base surface under the rear link. The formula to calculate angle φ1 is
(11)φ1=π2−α2=π2−arccosH−H0−L3+L4sinαL2,L1cosα1+L4sinα+L5cosα=H−H1,where L1 is the length of the front link, L5 is the distance between the upper hinge point of the front link and the shield beam, α1 is the vertical inclination of the front link, and H1 is the vertical distance between the hinged point and the base surface under the front link. The formula to calculate φ2 is
(12)φ2=π2−α1=π2−arccosH−H1−L4sinα+L5cosαL1.The geometric relation of Figure10 is calculated to obtain φ3:
(13)φ3=π2−θ−φ2+arctanL5L13,φ4=π−arccosL112+L62+L122−L1522L11L62+L112−arctanL6L12,where L6 is the vertical distance between the lower hinge point of the cover beam jack and the top beam, L11 is the distance from the hinge point on the cover beam jack to the turning point O4, L12 is the distance between the turning point O4 and the base of the hinge point of the top beam jack, and L15 is the length of the cover beam jack.
(14)φ5=arccosL92+L72+L102−L822L9L72+L102+arctanL7L10,where L7 is the vertical distance between the lower hinge point of the face guard jack and the cover beam, L9 is the distance from the hinge point on the face guard jack to the turning point O5, L10 is the distance between the turning point O5 and the base of the hinge point of the cover beam jack, and L8 is the length of face guard jack.Figure 10
Implementation process of the virtual three-machine equipment.During the operation of the hydraulic support, the anglesφ1, …, φ5 are the dynamic changes with the height of the support column, the length of the cover beam jack L8, and the length of the cover beam jack L15. According to the homogeneous transformation matrix of the skeleton motion of the three-machine equipment, the rotation angle of the links of the hydraulic support is substituted into the relation formula (9). The positions and postures of the virtual three-machine mesh model can be obtained during the motion process of the virtual three-machine mesh model.


## 3.3.1. Calculation Method of the Vertex Coordinate in Moving Skeleton

Given the world coordinate systemOx,y,z of the virtual 3D space, O1dxp,dyp,dzp is the coordinate of the origin of the parent coordinate system in the world coordinate system Ox,y,z. O2dxc,dyc,dzc is the coordinate of the origin of the subcoordinate system in the world coordinate system Ox,y,z. The coordinate that is a vertex of the mesh is Vcxc,yc,zc in the subcoordinate system O2dxc,dyc,dzc, as shown in Figure 7.Figure 7
Virtual space coordinate system and its transformation.When the skeletons are stationary, the coordinate of vertexV is translated from the subcoordinate system O2−dxcdycdzc to the parent coordinate system O1−dxpdypdzp using formula (2).
(2)Vp=Vc⋅Mcp,where Mcp is the translational matrix, and Vp is the transformed vertex coordinate from the subcoordinate system to the parent coordinate system.The coordinate of vertexVp is translated from the parent coordinate system O1−dxpdypdzp to the world coordinate system O−xyz using formula (3).
(3)Vw=Vp⋅Mpw,where Mpw is the translational matrix, and Vw is the transformed vertex coordinate from the parent coordinate system to the world coordinate system.Therefore, the translational process of vertexV of the mesh from the subcoordinate system O2−dxcdycdzc to the world coordinate system O−xyz is as follows:
(4)Vw=Vc⋅Mcp⋅Mpw=Vc⋅100001000010dxc−dxpdyc−dypdzc−dzp1⋅100001000010dxpdypdzp1.Moreover, the motion process of skeletons is analyzed. The motion process of the skeletons is resolved into the rotational and translational motion in its local coordinate system, which is described by a single four-dimensional homogeneous matrix using formula (1). The motion matrix in the local coordinate system of the child skeletons is Mlc. The motional matrix in the local coordinate system of the parent skeletons is Mlp. The motion transformation of the child skeleton in the local coordinate system is converted to the parent skeleton coordinate system as follows:
(5)Vp′=Vc⋅Mlc⋅Mlp,where Vp′ is the coordinate of vertex V in the parent coordinate system O1−dxpdypdzp.When coordinateVp′ of vertex V is transformed into the world coordinate system O−xyz, a new coordinate Vw′ of vertex V is obtained in the world coordinate system after bone movement. The transformational process is as follows:
(6)Vw′=Vp′⋅Mlp⋅Mpw=VcMlc⋅Mcp⋅Mlp⋅Mpw.Although the chain hierarchy of coordinate systems is complex, we can obtain the position of the vertices that are bound to the movement bone in the local coordinate system and calculate the position of the vertices in the world coordinate system after motion. When the vertices of the three-machine mesh model are traversed and calculated using the same method, the positions of the three-machine model are updated.The updated coordinates of the vertexes that are bound to the skeleton are calculated. The initial coordinates of the vertices of the mesh model in the world coordinate system are converted into the local coordinate system of the bone where the vertices are located via coordinate transformation. Take Figure8 as an example for analysis. Set the coordinate of vertex V at the initial position of the world coordinate system as Vw, and the world coordinate of vertex O2 that is the origin of the local coordinate system of vertex V is O2′dxc,dyc,dzc.
(7)VcO2=VwO+OO2.Figure 8
Skeletons “b” and “c” rotate around pin shaft node B by angleφ in the plane ABC.The coordinates of vertexV in its local coordinate system are calculated by formula (7). Its new coordinates are Vc:
(8)xc,yc,zc=xw−dxcyw−dyczw−dzc1=xwywxw1100001000010−dxc−dyc−dzc1=VwMwc.Thus, the coordinates of the grid vertices in the world coordinate system in their local coordinate system can be expressed as the product of matrixVw and the homogeneous coordinate system transformation matrix Mwc. The comprehensive reasoning shows that the world coordinate system of mesh vertices when the skeleton is moving is
(9)Vw′=Vc⋅Mwc⋅Mlc⋅Mcw=Vc⋅Mwc⋅Mlc⋅Mcp⋅Mlp⋅Mpw.Formula (9) implements the grid vertex coordinate transformation of two adjacent levels of the skeleton, and the method for calculating the vertex of the multistage skeleton motion is the same.The initial state of the vertex motion of the coordinate ofVw is the position in the world coordinate system, and the coordinate can be read from the grid model file. All vertices can be compared using the above formula to calculate the new coordinates after skeleton movement to update the position of the entire mesh vertex and realize the movement and rotation of the three-machine virtual model. The transform matrices between multistage skeletal coordinates, such as Mcp, and the initial values of the world coordinates O1 and O2 of the origin of each skeletal coordinate system can be obtained through the grid file. However, a matrix of motion transformations that occur at all levels of the skeleton under their respective local coordinate systems, such as Mlc and Mlp, cannot be obtained from the grid file. Without these skeleton transformation matrices, it will not be possible to realize the mathematical calculation of the model’s motion transformation. In the virtual 3D space, the motion of the 3D model of the three-machine equipment is driven by the motion of the skeletal model embedded in the mesh skin. Therefore, in order to control the motion of the 3D model of three-machine equipment, it is necessary to study the motion transformation calculation method of the joints in the skeleton model. The movement of the parts of the three-machine equipment includes the rotation around the axis and the movement in a specific plane, and the rotation movement mode is shown in Figure 8. Movement is achieved by the stretching skeletons. In the figure, A, B, C, and D are pin shaft nodes, and “a,” “b,” and “c” are skeletons. Suppose that skeletons “b” and “c” rotate around pin shaft node B by angle φ. The coordinate of point A is Axa,ya,za, the coordinate of point B is Bxb,yb,zb, and the coordinate of point C is Cxc,yc,zc. This transformation is calculated using the axial angle method [34].Regarding vectorBA=xa−xb,ya−yb,za−zb and vector BC=xc−xb,yc−yb,zc−zb, this rotation can be expressed as a counterclockwise rotation of vector BC around point B, that is, BA×BC. The direction is vertical plane ABC outward. The transformation matrix of skeleton “b” rotating by angle φ at point B is M1, and the new coordinate of point C point after the rotation is C′=BC·M1+B; the pseudocode programmed with the Three.js function is as follows:vec3BA = new THREE.Vector3(xa-xb,ya-yb,za-zb)//Compute vector BA;vec3BC = new THREE.Vector3(xc-xb,yc-yb,zc-zb)//Compute vector BC;vec3BA×BC = new THREE.Vector3.multiplyVectors(BA, BC)//Calculate the fork product of vector BA and vector BC, and calculate the normal vector of plane ABC;vec3 normal = normalize(vec3(BA×BC))//The normalized of normal vector of plane ABC;M1 = new THREE.Matrix4.makeRotationAxis(normal, φ)//A 4-order rotational transformation matrix M1 is calculated to rotate the angle φ of normal rotation around normal;vec3BC′ = normal.transformDirection9(M1)//Vector BC makes rotation transformation with matrix M1.The local coordinate system of the skeletonb is located at connection point B of skeletons “a” and “b”, and so, vector BC′ is obtained after skeleton “a” rotates by angle φ around the normal in the local coordinate system, namely, BC′ is the coordinate value of point C′ in the local coordinate system. Similarly, Mlc and Mlp can be calculated by using the method of calculating M1.


## 3.3.2. Solution of the Reverse Kinematics of the Hydraulic Support

When calculating the rotation transformation matrix of a skeleton in its local coordinate system, the angle that each moving part turns around the pin shaft must be clearly given. As a typical four-link mechanism, the hydraulic support rotates around the shaft of the connecting rod by means of a hydraulic cylinder hinged on it. The sensor can only collect the elongations of the hydraulic cylinder and piston rod, and it cannot directly obtain the angle information. This requires the reverse solution of the rotation angle information of each hinge point according to the actual size of the hydraulic support connecting rod, the mounting position of the hydraulic cylinder, and the elongation information of the hydraulic cylinder piston rod collected. According to the actual structure of the hydraulic support of the three-machine collaborative experimental platform, we establish the coordinate system, as shown in Figure9. Through the motion model of the linkage mechanism of the hydraulic support, according to the height H of the hydraulic support, length L15 of the hydraulic cylinder of the front beam, and length L8 of the end guard plate hydraulic cylinder, the axes of the moving members around the local coordinate system at the corresponding hinge point are dz1, …, and dz5, and the angles of the turns are φ1,..., φ5.Figure 9
Dimensional structure of the hydraulic support.According to the dimensional relation of the connecting rod mechanism of the hydraulic support, as shown in Figure9, the movement of the connecting rod mechanism of the hydraulic support during the production process of fully mechanized coal mining meets the following equation:
(10)L2cosα2+L3+L4sinα=H−H0,where L2 is the length of rear link, α is the horizontal inclination of the shield beam, L3 is the distance between the hinged points on front and rear links, L4 is the length of the cover beam, α2 is the vertical inclination of the rear link, H is the height of the upright column, and H0 is the vertical distance between the hinged point and the base surface under the rear link. The formula to calculate angle φ1 is
(11)φ1=π2−α2=π2−arccosH−H0−L3+L4sinαL2,L1cosα1+L4sinα+L5cosα=H−H1,where L1 is the length of the front link, L5 is the distance between the upper hinge point of the front link and the shield beam, α1 is the vertical inclination of the front link, and H1 is the vertical distance between the hinged point and the base surface under the front link. The formula to calculate φ2 is
(12)φ2=π2−α1=π2−arccosH−H1−L4sinα+L5cosαL1.The geometric relation of Figure10 is calculated to obtain φ3:
(13)φ3=π2−θ−φ2+arctanL5L13,φ4=π−arccosL112+L62+L122−L1522L11L62+L112−arctanL6L12,where L6 is the vertical distance between the lower hinge point of the cover beam jack and the top beam, L11 is the distance from the hinge point on the cover beam jack to the turning point O4, L12 is the distance between the turning point O4 and the base of the hinge point of the top beam jack, and L15 is the length of the cover beam jack.
(14)φ5=arccosL92+L72+L102−L822L9L72+L102+arctanL7L10,where L7 is the vertical distance between the lower hinge point of the face guard jack and the cover beam, L9 is the distance from the hinge point on the face guard jack to the turning point O5, L10 is the distance between the turning point O5 and the base of the hinge point of the cover beam jack, and L8 is the length of face guard jack.Figure 10
Implementation process of the virtual three-machine equipment.During the operation of the hydraulic support, the anglesφ1, …, φ5 are the dynamic changes with the height of the support column, the length of the cover beam jack L8, and the length of the cover beam jack L15. According to the homogeneous transformation matrix of the skeleton motion of the three-machine equipment, the rotation angle of the links of the hydraulic support is substituted into the relation formula (9). The positions and postures of the virtual three-machine mesh model can be obtained during the motion process of the virtual three-machine mesh model.


## 4. Implementation and Demonstration

The implementation process of virtual three-machine rendering is shown in Figure10. First, the 3D modeling of the three-machine equipment is established. Then, the real texture of the device is processed. Next, the model is placed in a virtual environment. Finally, the scene is rendered with JavaScript and the third party engine library of WebGL, Three.js. The software system analyzes the collected running state data of the three-machine equipment and generates the displacement or rotation angle information that drives the bone movement, realizes the real-time driving of the model, and enables users to easily observe the real-time production process.


### 4.1. Model Visualization

In this work, the model of the three machines is established using the SolidWorks modeling software. Finally, the model data are rendered and displayed on the web page. The process is shown in Figure11. The visualization method includes two levels. On the macro level, first, the 3D modeling of the three-machine equipment was established; then, the model file was converted from a DWG format to a unified STL format, which was used as the external call file of the web page. Finally, the model was rendered and displayed in the client browser by calling the JavaScript-based WebGL execution file. In the program coding layer, the external call file in the STL format was named according to the rules, and the model data were read in and parsed by JavaScript. In addition, the size of the model enveloping box and the coordinates of the original position of the model were calculated. The model was named and numbered with an ID, and the model material was uniformly added. Then, the mouse event handler was added. Finally, the model display and interactive operations were realized through the combination of HTML and CSS.Figure 11
Flowchart of the method of the imported model.


#### 4.1.1. Parsing Model Files

In model visualization, the model data are read, and the geometric information of the model is extracted. In this work, STL files are stored via ASCII format. The files recorded the triangular face data describing the 3D model, including the vertex coordinates and normal vectors of the triangles. The format of the file is very readable and easy to program, code, read, and parse. Figure12 shows the 3D model of the slot rail seat in the middle of the scraper conveyor and its STL format file. The file starts with the keyword “solid” followed by the name of the software. Subsequently, that data block describes the normal vector, and the vertex coordinates of a triangular facet are described using the keyword “facet…Endfacet.” Finally, the file ends with the keyword “endsolid.” In the STL file, the vertex coordinates of the triangular faces are all positive, the normal vector of the triangular faces is outward, and the vertex data of the triangular faces are arranged counterclockwise. According to the characteristics of the STL format files, when parsing STL files, it is necessary to process faces. To obtain the normal vector and coordinate data of the surface quickly and effectively, the regular expression is used to match the keywords, which help to store the normal vectors as strings in arrays; then, the important floating point data of faces are obtained by the JavaScript function “parseFloat.”Figure 12
3D model, STL file, and data parsing of the central groove seat of the scraper conveyor.


#### 4.1.2. Visualization of the Model

After obtaining the relevant data of the triangular faces of the three-machine models, the data need to be transmitted to the graphics card for calculation, and then, the model visualization is completed. The process is shown in Figure13. The vertex data of the triangle faces that are parsed using JavaScript are passed to the graphics card through the buffer object. Subsequently, the vertex shader is called to complete the primitive assembly and rasterization. The pixel color is written into the color buffer by the fragment shader. Finally, the three-machine models are displayed in the browser.Figure 13
Visualization process of the three-machine model.Through the transformation formula of the vertex coordinates (17) [35], the vertex shader transforms the coordinates, transforms and unites the plane normal, transforms the texture coordinates, calculates the model view matrix and projection matrix, and calculates the light and shadow to realize the vertex-by-vertex description of the three-machine model. The graphic assembly module assembles the isolated vertices of the model into geometric figures. The assembled geometric models are converted into faces via rasterization and processed by the fragment shader for each face so that the pixels of the three-machine model can be displayed on the screen.
(15)x′y′z′1T=PVMxyz1T,where (x, y, z) are the coordinates of the pretransformed vertex; x′,y′,z′ are the coordinates of the transformed vertex; M is the model matrix, which realizes the rotation and scaling of the model; V is the view matrix, which represents the scene observed by the observer; and P is a perspective projection matrix that displays the model in space.


### 4.2. Visualization Demonstration

In this work, SolidWorks is used to model the three-machine equipment, and the model is exported as an STL file. That STL file is parsed, and the model visualization and mouse event handler are realized by JavaScript to run the three-machine monitoring system in the browser. The three-machine monitoring system is divided into the main view area and the status area, as shown in Figure14. The main view area displays the three-machine model in real time, and the user can roam the scene in different perspectives. The interaction area is used to display the environment status of the working surface.Figure 14
Browser interface.Mouse clicks and movements need to be registered to display the model in different perspectives. When the main view window is clicked on, all models are selected and lightened. When the model selection is completed, the program needs to capture the mouse movements and give feedback to render the model in the specified perspective.


## 4.1. Model Visualization

In this work, the model of the three machines is established using the SolidWorks modeling software. Finally, the model data are rendered and displayed on the web page. The process is shown in Figure11. The visualization method includes two levels. On the macro level, first, the 3D modeling of the three-machine equipment was established; then, the model file was converted from a DWG format to a unified STL format, which was used as the external call file of the web page. Finally, the model was rendered and displayed in the client browser by calling the JavaScript-based WebGL execution file. In the program coding layer, the external call file in the STL format was named according to the rules, and the model data were read in and parsed by JavaScript. In addition, the size of the model enveloping box and the coordinates of the original position of the model were calculated. The model was named and numbered with an ID, and the model material was uniformly added. Then, the mouse event handler was added. Finally, the model display and interactive operations were realized through the combination of HTML and CSS.Figure 11
Flowchart of the method of the imported model.


### 4.1.1. Parsing Model Files

In model visualization, the model data are read, and the geometric information of the model is extracted. In this work, STL files are stored via ASCII format. The files recorded the triangular face data describing the 3D model, including the vertex coordinates and normal vectors of the triangles. The format of the file is very readable and easy to program, code, read, and parse. Figure12 shows the 3D model of the slot rail seat in the middle of the scraper conveyor and its STL format file. The file starts with the keyword “solid” followed by the name of the software. Subsequently, that data block describes the normal vector, and the vertex coordinates of a triangular facet are described using the keyword “facet…Endfacet.” Finally, the file ends with the keyword “endsolid.” In the STL file, the vertex coordinates of the triangular faces are all positive, the normal vector of the triangular faces is outward, and the vertex data of the triangular faces are arranged counterclockwise. According to the characteristics of the STL format files, when parsing STL files, it is necessary to process faces. To obtain the normal vector and coordinate data of the surface quickly and effectively, the regular expression is used to match the keywords, which help to store the normal vectors as strings in arrays; then, the important floating point data of faces are obtained by the JavaScript function “parseFloat.”Figure 12
3D model, STL file, and data parsing of the central groove seat of the scraper conveyor.


### 4.1.2. Visualization of the Model

After obtaining the relevant data of the triangular faces of the three-machine models, the data need to be transmitted to the graphics card for calculation, and then, the model visualization is completed. The process is shown in Figure13. The vertex data of the triangle faces that are parsed using JavaScript are passed to the graphics card through the buffer object. Subsequently, the vertex shader is called to complete the primitive assembly and rasterization. The pixel color is written into the color buffer by the fragment shader. Finally, the three-machine models are displayed in the browser.Figure 13
Visualization process of the three-machine model.Through the transformation formula of the vertex coordinates (17) [35], the vertex shader transforms the coordinates, transforms and unites the plane normal, transforms the texture coordinates, calculates the model view matrix and projection matrix, and calculates the light and shadow to realize the vertex-by-vertex description of the three-machine model. The graphic assembly module assembles the isolated vertices of the model into geometric figures. The assembled geometric models are converted into faces via rasterization and processed by the fragment shader for each face so that the pixels of the three-machine model can be displayed on the screen.
(15)x′y′z′1T=PVMxyz1T,where (x, y, z) are the coordinates of the pretransformed vertex; x′,y′,z′ are the coordinates of the transformed vertex; M is the model matrix, which realizes the rotation and scaling of the model; V is the view matrix, which represents the scene observed by the observer; and P is a perspective projection matrix that displays the model in space.


## 4.1.1. Parsing Model Files

In model visualization, the model data are read, and the geometric information of the model is extracted. In this work, STL files are stored via ASCII format. The files recorded the triangular face data describing the 3D model, including the vertex coordinates and normal vectors of the triangles. The format of the file is very readable and easy to program, code, read, and parse. Figure12 shows the 3D model of the slot rail seat in the middle of the scraper conveyor and its STL format file. The file starts with the keyword “solid” followed by the name of the software. Subsequently, that data block describes the normal vector, and the vertex coordinates of a triangular facet are described using the keyword “facet…Endfacet.” Finally, the file ends with the keyword “endsolid.” In the STL file, the vertex coordinates of the triangular faces are all positive, the normal vector of the triangular faces is outward, and the vertex data of the triangular faces are arranged counterclockwise. According to the characteristics of the STL format files, when parsing STL files, it is necessary to process faces. To obtain the normal vector and coordinate data of the surface quickly and effectively, the regular expression is used to match the keywords, which help to store the normal vectors as strings in arrays; then, the important floating point data of faces are obtained by the JavaScript function “parseFloat.”Figure 12
3D model, STL file, and data parsing of the central groove seat of the scraper conveyor.


## 4.1.2. Visualization of the Model

After obtaining the relevant data of the triangular faces of the three-machine models, the data need to be transmitted to the graphics card for calculation, and then, the model visualization is completed. The process is shown in Figure13. The vertex data of the triangle faces that are parsed using JavaScript are passed to the graphics card through the buffer object. Subsequently, the vertex shader is called to complete the primitive assembly and rasterization. The pixel color is written into the color buffer by the fragment shader. Finally, the three-machine models are displayed in the browser.Figure 13
Visualization process of the three-machine model.Through the transformation formula of the vertex coordinates (17) [35], the vertex shader transforms the coordinates, transforms and unites the plane normal, transforms the texture coordinates, calculates the model view matrix and projection matrix, and calculates the light and shadow to realize the vertex-by-vertex description of the three-machine model. The graphic assembly module assembles the isolated vertices of the model into geometric figures. The assembled geometric models are converted into faces via rasterization and processed by the fragment shader for each face so that the pixels of the three-machine model can be displayed on the screen.
(15)x′y′z′1T=PVMxyz1T,where (x, y, z) are the coordinates of the pretransformed vertex; x′,y′,z′ are the coordinates of the transformed vertex; M is the model matrix, which realizes the rotation and scaling of the model; V is the view matrix, which represents the scene observed by the observer; and P is a perspective projection matrix that displays the model in space.


## 4.2. Visualization Demonstration

In this work, SolidWorks is used to model the three-machine equipment, and the model is exported as an STL file. That STL file is parsed, and the model visualization and mouse event handler are realized by JavaScript to run the three-machine monitoring system in the browser. The three-machine monitoring system is divided into the main view area and the status area, as shown in Figure14. The main view area displays the three-machine model in real time, and the user can roam the scene in different perspectives. The interaction area is used to display the environment status of the working surface.Figure 14
Browser interface.Mouse clicks and movements need to be registered to display the model in different perspectives. When the main view window is clicked on, all models are selected and lightened. When the model selection is completed, the program needs to capture the mouse movements and give feedback to render the model in the specified perspective.


## 5. Performance Test

To evaluate the performance of the system, Apache HTTP Server 2.2 is used as the web server, and gzip is used to compress the data files. The wireless router is configured to test the system’s performance on a mobile terminal. The operating systems tested include Windows and Android. The browsers include Google Chrome and Android browser. The details are shown in Table1.Table 1
The devices used to test the system.
DeviceHardware details (CPU, RAM, and GPU)System and browserDesktop PCIntel(R) Core(TM) i5-6500 @ 3.30 GHz, 16 GB RAM, NVIDIA GeForce GT 730Windows 7 64-bit Google Chrome 38.0.2125.111Laptop PCIntel(R) Core(TM) i5-4210U @ 1.70 GHz, 4 GB RAM, NVIDIA GeForce 840 MWindows 7 32-bit SP1 Google Chrome 38.0.2125.122Huawei Nova 3Hisilicon Kirin 970 @ 1.4 GHz, 4 GB RAM, Mali-G72 12 MPAndroid 8.1 Android system browserWhen the number of display frames per second is more than 25 frames, continuous animation will be generated. Therefore, we set the data acquisition frequency of each displacement and angle sensor of the fully mechanized three-machine body to 25 times per second to prevent the fast data acquisition frequency from causing lag when rendering the browser scene model. At this time, the browser will smoothly render the 3D animation of the motion process of the three-machine equipment. The experimental platform is shown in Figure15. When the three-machine equipment operates, the sensors will collect and transmit the motion data of each moving part of the equipment to the PLC. According to the predetermined transmission protocol, the PC server will encode all the motion data collected by the PLC at the same time and then transmit the encoded data over the LAN. The monitoring computer analyzes the motion angle and displacement information of each moving part of the three-machine equipment after receiving the data. The motion data are then transmitted to the nodes in the skeleton model. The angle and displacement data of each motion node are inserted into the inverse kinematics solution model of the hydraulic support and the joint motion transformation formula of the skeleton model to calculate the local transformation of each joint, and then the calculation result is further inserted into the joint chain transformation matrix (9) to calculate the coordinates of each vertex motion transformation on the mesh model.Figure 15
The motion of real three-machine equipment and its monitoring interface.First, the shearer, hydraulic support, and scraper conveyor are loaded separately when the three-machine equipment is not started. The results (see Table2) showed that the rendering time is approximately 17 ms, and the rendering rate is close to 60 fps. The animation of the model is smooth. The animation data are updated synchronously during dynamic loading. The fastest refresh frame rate of animation can reach 58 fps (more than 25 fps) in the virtual scene. The model can be rendered quickly and the animation is very smooth.Table 2
Individual performance of the three-machine equipment.
ShearerThe hydraulic supportScraper conveyorNumber of faces656879865120Rendering time (ms)Desktop PC171717Laptop PC171717Huawei Nova 3181817Rendering frame rate (fps)Desktop PC606060Laptop PC606060Huawei Nova 3555358The frame rates of the desktop PC, laptop PC, and Huawei Nova 3 render different numbers of faces, as shown in Figure16. The analysis results show each device that can maintain a very high rendering frame rate when the number of faces is less than a certain number. However, when the number of faces exceeds a certain number, the rendering frame rate will drop sharply. This is mainly related to the hardware performance of the device; that is, the rendering effect depends on the comprehensive performance of the CPU and GPU. The CPU is mainly responsible for calculating the position of the animation runtime model, and the GPU is responsible for rendering the model. In this case, the performances of the desktop PC and laptop PC are higher than that of mobile phone. When the frame rate of the animation is 25 fps or higher, the picture is considered to be relatively smooth. On the desktop PC, laptop PC, and mobile phone, the horizontal coordinate values are 5.43, 6.75, and 5.64, respectively, when the frame rate is 25 fps. The average value of these three is 5.94. We can calculate the maximum number of faces that can be rendered on the above equipment as approximately 870,000 (105.94). The numbers of faces on the 3D models of the shearer, hydraulic support, and scraper conveyor are far less than the maximum value calculated above. Therefore, the system can render the three-machine model smoothly, and it can also obtain an excellent user interaction experience in the laboratory.Figure 16
Frame rates on different devices.


## 6. Conclusions

A cross-platform Web3D monitoring system for three-machine equipment was developed in this work. As the proposed system is based on the B/S mode, it allows many users access to the system at the same time, and it is convenient to upgrade the system. In this system, the latest Internet graphics standard WebGL is used to render the models and the virtual panoramic scene. By using WebGL, the system achieves acceleration on hardware level in web browsers. It is characterized as cross-platform and portable, which could run in almost any platform and device. And it has a strong ability to rendering models with high performance and graphical quality on different devices. The proposed system displays the virtual working process which keeps up with the working process of the three-machine equipment, which makes operators directly monitor the equipment condition so as to adjust the working parameters. Although the proposed system has been initially established at present, it only completes the specific functions in the laboratory environment. Future work will focus on applying this system into underground coal mines. Future research goals include integrating the real-time environment information to this system and better controlling the running of the three-machine.



---

*Source: 3147352-2020-12-01.xml*