Patent ID: 11908344
Assignee: CENTRAL CHINA NORMAL UNIVERSITY
Field: Computer technology (Electrical engineering)
Classification: CPC G  Y | IPC G

Claim 4:
5. A method for operating the 3D integrated teaching field system of claim 1, comprising:
(S1) dividing a classroom into the teaching activity area of the teacher and a student activity area with a division wall embedded with a holographic film and an entry door; constructing a 3D teaching field system for the classroom using an electronic sand table, the edge computing server, a holographic imaging device and lights; and deploying a device integrating depth, sound, motion capture and touch sensing in the teaching activity area;
(S2) selecting a teaching resource according to subjects, grades, textbook versions and knowledge points requested by the user in accordance with a weighting order of terminal, resolution and ratio; constructing the cloud-edge-end link to download the virtual teaching resources according to requests of the user; and realizing the loading service of the virtual teaching resources according to the local caching, the hot updating by using the edge computing server, and the cloud batch updating;
(S3) acquiring, by the RGB-D camera, background, texture, depth and point cloud of the teaching activity area; uploading the background, the texture, the depth and the point cloud to the edge computing server; extracting the skeleton data of the teacher using the Mask R-CNN technology; and extracting, in the local processing module, the coordinates of skeleton and skeletal joint points of the teacher using a segmentation strategy to perform the grouping of the skeletal joint points based on a hierarchical control principle;
(S4) receiving the point cloud sequence data obtained by the acquisition and processing module; calculating and labeling the skeletal joint points of the teacher using the nonlinear solver; analyzing the motion trajectory of the skeletal joint points using a neural network algorithm according to moving speeds and moving angles of the skeletal joint points; and detecting the states of the teacher in the teaching activity area to track the teaching behaviors of the teacher and the changes thereof; wherein step (S4) comprises:
(S4-1) receiving, by the edge computing server, the point cloud sequence data obtained by the acquisition and processing module using the MEC architecture; searching and positioning the skeletal data of the teacher in the point cloud sequence data using the random decision tree algorithm and the random forest algorithm; segmenting the skeletal joints of the teacher in the point cloud sequence data using the Mask R-CNN algorithm; and calculating and labeling the skeletal joint points of the teacher using the nonlinear solver;
(S4-2) defining the codes of the common teaching actions according to the teaching action meaning of movement of the skeletal joint points achieved by the complete attributes; determining the moving speeds and the moving angles of the body movements, the facial expressions or the hand gestures of the teacher according to the skeletal joint points at different moments on the time axis; and analyzing the motion trajectory of the skeletal joint points during the teaching process using the neural network algorithm; and
(S4-3) detecting the states of the teacher in the teaching activity area at different moments on the time axis using the 3D mapping and human body tracking technology, in combination with the position and connection relationship of the inverse kinematics skeletal model; calculating and examining the degrees of freedom of adjacent joints; and determining and examining the behaviors of the teacher based on formed characteristics to track the changes in the teaching behaviors of the teacher; and

(S5) constructing a virtuality-reality fusion coordinate system; determining position, the posture and the scaling factor of a holographically-displayed content according to internal and external parameters of a hologram projector; locating position information and the posture information of the skeleton of the teacher in the virtuality-reality fusion coordinate system, in combination with an inverse perspective transformation equation; and determining an interaction command based on the motion trajectory of the skeletal joint points of the teacher, wherein the interaction commands comprise clicking, dragging, drop, hauling and pulling;
(S6) rendering the holographic display content on the edge computing server using a cloud rendering teaching service mode; updating a virtual teaching resource screen displayed on the holographic screen based on body positions, head postures and sight lines of the teacher; fusing and displaying the virtual teaching content screen and the virtual teaching resource screen according to position and interaction relationship between the teacher and the virtual teaching resources; and
(S7) recognizing a teaching style of the teacher based on the motion trajectory, connection relationships and change sequences of the skeletal joint points of the teacher; recognizing a multi-sensory action input by the teacher, wherein the multi-sensory action comprises eye movements, body movements and gesture actions; converting the multi-sensory actions into the interaction commands; and driving the virtual teaching resources to perform operations to update a screen of the holographically-displayed content, wherein the operations comprise moving, zooming and rotating.