# Deep Learning-Based 3D Shape Feature Extraction on Flash Animation Style

Authors: Mengru Zhu; Jong Han Lee
Journal: Wireless Communications and Mobile Computing
 (2022)

---

## Abstract

Flash animation, as a kind of digital learning resource, is an important media for delivering information content, and more importantly, it is an important online learning resource with text, graphics, images, audio, video, interaction, dynamic effects, etc. Flash animation, with its powerful multimedia interaction and presentation capabilities, is widely used in distance education, high-quality course websites, Q&A platforms, etc. With the continuous development of deep learning, the 3D shape feature extraction method combined with deep learning has become a hot research topic. In this paper, we combine deep learning with traditional 3D shape feature extraction methods, so that we can not only break the bottleneck of nondeep learning methods but also improve the accuracy of 3D shape data classification and retrieval tasks, especially in the case of non-rigid 3D shapes. The scheme in this paper not only does not require a large number of training samples but also its feature extraction for flash animation is accurate. Experiments show that the success rate of accurate feature extraction of this paper’s scheme is higher than that of the state-of-the-art methods.


---

## Body

## 1. Introduction

Flash animation is one of the most popular forms of multimedia on the Internet. It has been in development for 21 years since micromedia launched this web animation software in 1999 and is currently 98% available in personal computer browsers [1]. Few other animation technologies have had such a profound and positive impact on people in the Internet era as flash animation technology, and its rapid development is unprecedented, so much so that a large number of flash animation enthusiasts and a huge amount of online flash animation resources have emerged in a short period of time. Flash animation is widely used in education, advertisement, MTV, game, virtual reality, application development, etc. Especially, it is widely used for education [2, 3].In teaching, its advantages such as strong interactivity, multimedia integration, and application development are stronger than PowerPoint; its advantages such as flexible production method, small storage capacity, and easy learning are stronger than Authorware, so flash animation is more accepted by the majority of educators and learners [4]. Flash animation, as an important multimedia teaching resource, greatly expands the horizon of computer education applications, promotes the development of in-depth learning and online distance education, and provides a good guarantee for online self-learning and lifelong learning [5, 6].At present, flash animation is facing the impact of HTML5, and Adobe has declared that it will be officially retired by the end of 2020, but it still has great advantages in PC (60.1% of Chinese Internet users access the Internet through desktop computers), games and videos and will continue to accumulate [7]. For a long time in the future, flash animations will still be useful; the huge amount of flash animations already available on the Internet will still bring good experiences to Internet users; the large amount of flash animation learning resources already released will still bring great help to teaching and learning [8].With the rapid development of computer and network technology, a huge amount of flash animation learning resources have been accumulated on the Internet. However, too many resources can be inconvenient and disruptive for educators and web self-learners. Flash animation search engines have become a key factor for flash animation to play a greater role in education, as teachers and students want to find the flash animation learning resources they need quickly and accurately [9–11]. The search engines commonly used by Chinese users, such as Google and Baidu, index flash animations based on keywords, external features of animations and web page contextual information, and their search efficiency and accuracy are generally not satisfactory.In the era of big data, 3D models, as the fourth generation of digital media, are growing massively with the development of software and hardware. At this stage, effective extraction of low-dimensional and highly discriminative shape content features of 3D models is beneficial for their classification and retrieval, etc. Therefore, researching new methods for 3D model feature extraction is an important research content in the current computer vision field [2].A 3D model is a data representation with a spatial structure that contains richer content properties than a 2D image. 3D models can be broadly classified into two categories: rigid and nonrigid: rigid refers to objects whose shape and volume do not change after being subjected to external forces; on the contrary, nonrigid refers to objects whose shape and volume change after being subjected to external forces. Figure1 gives a few examples of rigid and nonrigid three-dimensional shapes [12].Figure 1
Example of 3D shape.3D model data are widely used in 3D printing [13], industrial product design [14], computer-aided design [15], furniture design, medical diagnosis [16], film and television animation, virtual reality, 3D game design [17], building design, molecular biological research, and cultural relic repair. With the extensive use of depth sensors, LiDAR, 3D imaging technology, 3D model rendering software, etc. in the manufacturing field, 3D model data will continue to grow in massive amounts and will continue to generate demand for application scenarios for tasks such as 3D model classification and retrieval.Based on the tasks of 3D shape classification and retrieval, this paper provides a systematic review of 3D shape feature extraction methods and related works based on deep learning methods in recent years.


## 2. Related Work

Flash animation has been researched more in the fields of education, digital media art, and advertising, and many research papers have been published, mainly divided into four aspects: humanities and arts, technical applications, flash retrieval, and educational applications [18]. In terms of humanities and arts, experts point out that flash animation is a kind of art work, and its creation requires artistic cultivation. Research in this area focuses on analyzing the artistic representations and cultural connotations of flash animation. For example, [19] analyzes the artistic representation of flash animation from the perspective of technology and culture and points out that flash animation is a new artistic means with a broad development prospect; [20] explores flash animation from the perspective of art and aesthetics and concludes that Flash animation has artistic characteristics such as motion, shape, hypothesis, interaction, synthesis, and fashion. [21] explores the application of flash animation technology in the creation of movies in terms of object modeling, scene construction, and picture rhythm; [4] studies the important role of Flash animation in Chinese online news communication; [5] preliminarily studies the visual language of flash animation and its application in web design, such as the visual representation of fonts The visual language of flash animation and its application in web design, such as the visual representation of fonts, the artistic characteristics of graphics and the formal laws of color design.In terms of technical applications, people mainly study the file structure, technical implementation, and application areas of flash animation. For example, [6] studied and proposed the principles of interactive design of flash animation; [7] initially studied the characteristics and applications of Flash animation software and analyzed the differences between flash animation and traditional animation; [8] studied the production method of network electronic map based on flash animation; [9] proposed the information hiding model based on Flash animation and its hiding algorithm in response to the problems of copyright protection of Flash animation and the need of using flash animation for hidden communication; [10] studied the interactive communication of flash animation and its hiding algorithm. The model of information hiding based on flash animation and its hiding algorithm were proposed to address the problems of copyright protection of flash animation and the demand of using flash animation for stealth communication; [10] studied the application of flash animation’s interactivity in digital media; Riwinoto [11] systematically studied the application of flash animation technology in Chinese farmers’ popular science education and initially explored the ideas of farmers’ popular science animation production.[12] used a fuzzy semantic network for automatic annotation of flash animation, proposed a three-layer retrieval model of animation, scene, and constituent elements based on semantics, and began to propose the concept of flash animation scene, but also did not conduct specific application research. [13] conducted a study on automatic classification of flash animation based on content, firstly, extracted metadata such as file size and some constituent element features such as text and button of flash animation, and automatically classified flash animation into five categories such as game, cartoon, MTV, advertisement and teaching courseware by using decision tree, neural network, and support vector machine algorithms, respectively, and the results showed that the neural network algorithm was performed. The results showed that the classification accuracy of the neural network algorithm was the highest. [14] established a four-layer content structure feature description model of ontology, logical scene, visual scene, and element object, and initially realized the construction of a flash animation learning resource retrieval system based on content structure features. [15] studied the visual scene of flash animation and divided the visual scene by the chunked color histogram difference method. Its research was able to consider the color space difference to judge the visual scene boundary of flash animation and achieved a certain segmentation effect, but the use of fixed global thresholds was prone to misjudgment and omission, and the visual characteristics of the visual scene were not analyzed in depth.


## 3. Scene Structure Model for Web Flash Animation

For flash animations with complex frames, the creators usually organize the frames by scene. However, after the animation is published from a source file (∗.fla) to a playback file (∗.swf), all frames are fused into a single sequence on a timeline, with sequential numbering between frames, and no longer with scene boundaries.Visual scenes and logical scenes analyze flash animation content from different perspectives. A visual scene can represent a specific picture environment, a picture effect, or a complete event. While a logical scene is an independent logical segment, it can represent a richer screen content, for example, a title animation may contain multiple screen environments, multiple screen effects, and multiple events. That is, a logical scene can contain multiple visual scenes. But on the other hand, sometimes there can be multiple interactions in a single visual scene, i.e., the interactions happen in the same background environment. Therefore, in the timeline of a flash animation, visual scenes and logical scenes intersect and contain each other. As shown in Figure2.Figure 2
Scene structure of flash animation.On the timeline, the boundary of the logical scene is generally judged by analyzing whether the tag contains interactive objects, such as buttons and action scripts, while the boundary of the visual scene is generally defined by comparing the visual differences between adjacent frames. In accordance with human observation habits, this study takes the visual scene as the object of flash animation content and constructs the scene structure model of Flash animation as Figure3.Figure 3
Model diagram of the scene structure of flash animation.From the model in Figure3, we can see that a flash animation can be divided into multiple logical scenes and multiple visual scenes; each logical scene is a combination of one to multiple visual scenes; a visual scene can span multiple logical scenes, and a visual scene can contain multiple logical scenes; each visual scene contains a series of frames with similar visual characteristics; each frame contains Each visual scene contains a series of frames with similar visual characteristics; each frame contains multiple media object elements such as text, graphics, and buttons [17].For example, in the flash game animation shown in Figure4, there are 10 frames, the flash animation is paused at frame (1), and when the user clicks the “PLAY GAME” button at frame (1), the screen will automatically When the user clicks the “PLAY GAME” button in frame (1), the screen will be played continuously until frame (8). There is a dynamic switching effect in the middle of the screen. When the user clicks the “play” button in frame (8), the flash animation will play to frame (9) and pause, and the user can use the keyboard to play the game in frame (9). After passing the level, it goes to frame (10) and pauses, and displays the content of the second level, waiting for the user to continue the game; if the level fails, it jumps to frame (11) and pauses, displaying the information of the failed level, waiting for the user to continue the game. According to the definition of logical scene, frame (1) is the first logical scene, frames (2) to (8) are the second logical scene, and frames (9), (10) and (11) are also different logical scenes. The second logical scene contains several segments with different visual contents, i.e., it can contain more than one visual scene.Figure 4
Example of flash animation logic scene.Different types of flash animations have different logic scenes, mainly in terms of the number of logic scenes and the number of interactive elements they contain. Generally speaking, the more interactive elements there are, the more logical scenes there are. A flash animation of advertisement type does not need interaction, so a flash animation of advertisement type only contains one logical scene, while a flash animation of game type needs a lot of interaction, so it must contain multiple logical scenes.Timeline actions are added directly to the timeline and triggered by the ShowFrame tag; component actions cannot be added directly to the timeline and must be included in the description tag of a component. To trigger a component action, you must interact with the component, such as by clicking or dragging it. When a component is clicked or dragged, the action contained in the component is not executed immediately; it is simply placed in a list of actions that are executed when the ShowFrame tag is encountered or when the state of the component changes.Buttons are frequently used components in flash animations, and they are used for user interaction. Buttons have three display states: up, over, and down, and the default state is up; the default state is up; the over state is when the mouse pointer is in the button area; and the down state is when the mouse is clicked in the over state.The content of the ACTIONRECORED structure in DefineButton and DefineButton2 is the action of the button, which is triggered by the transition between the four states of the button; the action of the object in the timeline is recorded by the ACTIONRECORED structure in DoAction or DoInitAction. The boundary points of the logical scenes are obtained by analyzing the timeline actions or component actions, and the common logical scene actions are shown in Table1.Table 1
List of actions at logical scene boundaries.
Action fieldCodeDescribeAction Play0x06Play one frameAction Stop0x07Stop playingActionNext Frame0x04Play next frameActionGoto Frame0x81Go to a frameActionGotoLabel0x8CGo to a marker frameAction WaitForFrame0x8AWait for a frameActionSetTarget0x8BSet target object workBased on the understanding of SWF animation node action generation principle, you can get the logical scene of an animation by tag analysis. The specific steps are as follows:(1)
Sequentially read the keyframe tags of the SWF document, determine whether DefineButton, DefineButton2, DoAction, DoInitAction, if yes, then go to (2) and start analysis, otherwise read the next tag(2)
Parse the corresponding BUTTONRECORD, BUTTONCONDACTION, ACTIONRECORD and other structures within each label obtained in the previous step, if it is a node action, the frame number is stored in the array; otherwise, judge the action record of the next frame(3)
Write the frame numbers of Flash animation logical scene nodes extracted in the previous step to the index library, and remove the duplicate frame numbers and sort them(4)
In VC++ environment, use SWFtoImage dynamic link library to save the node image, and extract the features such as complexity of the frame at the same time, and store them in the logical scene feature index library(5)
Using the GIF API interface, the node image saved in step (4) is generated into a Gif format imageThe logical scene structure of flash animation describes the logical relationship when the animation is played. The visual characteristics of the logical scene include the number of logical scenes, the complexity of the screen, the number of frames included, and the number of elements. Among them, the number of logical scenes reflects the overall logical structure of flash animation, the larger the value is, the more frequent the interaction is; the complexity of node frames and other content features reflect the visual characteristics of each scene. After completing the segmentation of logical scenes by the above method, this study can extract the visual features of each logical scene representative frame and apply them to the content-based flash retrieval system.


## 4. Related Deep Learning Models

### 4.1. Convolutional Neural Networks

Convolutional neural networks (CNNs) are the most classical deep learning neural network model, which is characterized by the use of convolutional operations and back-propagation algorithms to train neural networks, which can be applied to 2D image classification, retrieval, semantic segmentation, and other related tasks.The prototype of modern convolutional neural network is LeNet5 [17], a handwritten font recognition network born in 1998, which initially laid down the components of the convolutional neural network that emerged later, with the basic components covering convolutional layer, pooling layer, fully connected layer, and output layer. As shown in Figure 5, the operational layer of the convolutional neural network can be regarded as a complex function fCNN, where the back-propagation phase is driven by a combination of regular loss and data loss to update the weights and bias parameters, and the error is back-propagated to each layer of the network for learning training of parameter updates such as weights and bias.Figure 5
Basic flow chart of convolutional neural network.


### 4.2. Autoencoder

Autoencoder (AE), introduced in 1986, is a neural network model that is an unsupervised learning model and can be used for data compression. The Autoencoder uses a backward propagation algorithm to train the parameters with the goal of making the input equal to the output as much as possible. Figure6 gives the structure of the autoencoder model, which consists of two parts: the encoder and the decoder. The basic structure is a multilayer perceptron neural network with multiple intermediate layers from the input layer to the output layer, characterized by the input layer and the output layer having the same dimension, and the intermediate coding layer dimension is smaller than the dimension of the output layer.Figure 6
Automatic encoder model structure.


### 4.3. Generating Adversarial Networks

Generative adversarial networks (GANs) are a deep learning generative model, which belongs to unsupervised learning model, proposed in [21], and can be used to learn complex feature distribution, image style transfer, model generation and other tasks. As shown in Figure 7, the generative adversarial network mainly contains a generative model and a discriminative model, which play each other to complete the network training process. The role of the generator is to convert the random vectors in the potential space into the generated samples to deceive the discriminator, while the discriminator needs to distinguish the real samples from the generated samples.Figure 7
GAN model structure.


### 4.4. Rigid Body Feature Extraction Method Based on Deep Learning Method

Representation methods for digital geometric models include (1) solid representations, such as solid geometry, point clouds, body networks, and voxels, and (2) boundary representations, such as surface meshes, parametric surfaces, subdivision surfaces, and implicit surfaces. There are several forms of geometric data representations that can be applied to deep learning models, such as views, point clouds, meshes, and voxels [22]. The representation of data geometry commonly used for deep learning models is given in Figure 8.Figure 8
Data geometry commonly used for deep learning models.Currently, deep learning-based methods have achieved good results in tasks such as rigid body 3D shape classification and retrieval. In the past, 3D shape feature extraction mainly used non-deep learning algorithms, such as SPH [5], LFD [4], geodesic distance [7], heat kernel signature (HKS) [8], wave kernel signature (WKS) [9], and other traditional feature extraction methods based on meshes, views, and the feature extraction methods of non-deep learning algorithms such as mesh, view, point cloud, etc. Nowadays, 3D shape feature extraction has evolved to mainstream and cutting-edge deep learning algorithm-based feature extraction methods, such as using deep learning neural network models such as convolutional neural networks and autoencoders to extract 3D shape features for tasks such as classification, retrieval, semantic segmentation, 3D reconstruction, and model generation. The history of the development of 3D shape representation using different types of data is given in Figure 9, and the data representations of different data types are listed in Table 2.Figure 9
History of the development of 3D shape characterization using different types of data.Table 2
Deep learning networks with different data type representations.
Data representation typeDeep learning networkPoint cloudPoint Net, Point Net++, Kd-net worksMeshSO-Net, DGCNN, Geo-CNN, MeshNetViewMVCNN, Pairwise, GVCNN, RotationNet, MLVCNNVolume3D ShapeNets, VoxNet, FPNN, NormalNetOctreeOctNet, O-CNNForm of combinationFusionNet, PVNet, PVRNet


## 4.1. Convolutional Neural Networks

Convolutional neural networks (CNNs) are the most classical deep learning neural network model, which is characterized by the use of convolutional operations and back-propagation algorithms to train neural networks, which can be applied to 2D image classification, retrieval, semantic segmentation, and other related tasks.The prototype of modern convolutional neural network is LeNet5 [17], a handwritten font recognition network born in 1998, which initially laid down the components of the convolutional neural network that emerged later, with the basic components covering convolutional layer, pooling layer, fully connected layer, and output layer. As shown in Figure 5, the operational layer of the convolutional neural network can be regarded as a complex function fCNN, where the back-propagation phase is driven by a combination of regular loss and data loss to update the weights and bias parameters, and the error is back-propagated to each layer of the network for learning training of parameter updates such as weights and bias.Figure 5
Basic flow chart of convolutional neural network.


## 4.2. Autoencoder

Autoencoder (AE), introduced in 1986, is a neural network model that is an unsupervised learning model and can be used for data compression. The Autoencoder uses a backward propagation algorithm to train the parameters with the goal of making the input equal to the output as much as possible. Figure6 gives the structure of the autoencoder model, which consists of two parts: the encoder and the decoder. The basic structure is a multilayer perceptron neural network with multiple intermediate layers from the input layer to the output layer, characterized by the input layer and the output layer having the same dimension, and the intermediate coding layer dimension is smaller than the dimension of the output layer.Figure 6
Automatic encoder model structure.


## 4.3. Generating Adversarial Networks

Generative adversarial networks (GANs) are a deep learning generative model, which belongs to unsupervised learning model, proposed in [21], and can be used to learn complex feature distribution, image style transfer, model generation and other tasks. As shown in Figure 7, the generative adversarial network mainly contains a generative model and a discriminative model, which play each other to complete the network training process. The role of the generator is to convert the random vectors in the potential space into the generated samples to deceive the discriminator, while the discriminator needs to distinguish the real samples from the generated samples.Figure 7
GAN model structure.


## 4.4. Rigid Body Feature Extraction Method Based on Deep Learning Method

Representation methods for digital geometric models include (1) solid representations, such as solid geometry, point clouds, body networks, and voxels, and (2) boundary representations, such as surface meshes, parametric surfaces, subdivision surfaces, and implicit surfaces. There are several forms of geometric data representations that can be applied to deep learning models, such as views, point clouds, meshes, and voxels [22]. The representation of data geometry commonly used for deep learning models is given in Figure 8.Figure 8
Data geometry commonly used for deep learning models.Currently, deep learning-based methods have achieved good results in tasks such as rigid body 3D shape classification and retrieval. In the past, 3D shape feature extraction mainly used non-deep learning algorithms, such as SPH [5], LFD [4], geodesic distance [7], heat kernel signature (HKS) [8], wave kernel signature (WKS) [9], and other traditional feature extraction methods based on meshes, views, and the feature extraction methods of non-deep learning algorithms such as mesh, view, point cloud, etc. Nowadays, 3D shape feature extraction has evolved to mainstream and cutting-edge deep learning algorithm-based feature extraction methods, such as using deep learning neural network models such as convolutional neural networks and autoencoders to extract 3D shape features for tasks such as classification, retrieval, semantic segmentation, 3D reconstruction, and model generation. The history of the development of 3D shape representation using different types of data is given in Figure 9, and the data representations of different data types are listed in Table 2.Figure 9
History of the development of 3D shape characterization using different types of data.Table 2
Deep learning networks with different data type representations.
Data representation typeDeep learning networkPoint cloudPoint Net, Point Net++, Kd-net worksMeshSO-Net, DGCNN, Geo-CNN, MeshNetViewMVCNN, Pairwise, GVCNN, RotationNet, MLVCNNVolume3D ShapeNets, VoxNet, FPNN, NormalNetOctreeOctNet, O-CNNForm of combinationFusionNet, PVNet, PVRNet


## 5. Experimental Analysis

As shown in Figure10, the distribution of this paper’s scheme for flash animation color generation includes 12 types of nonrigid transformations such as the original initial state of a typical nonrigid 3D shape and its isometric isotropic, topological, noise, scattering noise, hole, microhole, sampling, rasterization, local missing, view projection, affine transformation, and scale transformation. Due to the diversity and complexity of the deformation of nonrigid three-dimensional shapes, many well-established detection and classification schemes for rigid bodies do not yield satisfactory results for nonrigid bodies.Figure 10
Probability of generation of different flash colors.Compared with rigid bodies, feature extraction of nonrigid 3D shapes requires higher requirements, not only for translation, rotation, and scale invariance but also for isometric invariance. At present, the research on the extraction of deep features of non-rigid 3D shapes and the classification and retrieval system of large-scale non-rigid 3D shapes based on deep learning are still relatively few, and for the time being, there is no single feature that can give a comprehensive description of the intrinsic properties of nonrigid 3D shapes. The different dynamic distributions are shown in Figure11.Figure 11
Different dynamic distributions.The deep learning-based methods for nonrigid 3D shape feature extraction are artificial feature-based methods, raw data-based feature extraction, projection view-based feature extraction, 3D voxel-based feature extraction, and multifeature fusion-based feature extraction. As shown in Figure12 different animations with in-kind generation probabilities, field detection neural networks, conversion of 3D shapes into voxelized form, and the use of field exploration filters instead of convolution, extracted the depth features based on 3D voxels, and its classification accuracy on the model reached 88.4%. These can be attributed to our use of deep learning, effective use of feature extraction, and rational use of online resources.Figure 12
Different animation and physical generation probabilities.In the feature extraction based on multifeature fusion, this paper multimodal 3D feature learning achieves better results than single features and brings into play the advantages of fused features. The generated flash animation characters as shown in Figure13 are connected using the multifeature fusion layer; then, the cross-connected layer is constructed to combine the low-level features with the high-level semantic features to further improve the feature expression.Figure 13
Flash animation character generation.


## 6. Conclusions

With the widespread use of flash, it has caused scholars to analyze and study the characteristics of flash animation content in depth. In this paper, we design a flash animation feature extraction based on deep learning and the use of online resources on the web to meet the convenience to animation design. We analyze the content structure features of flash animation, such as scene structure features, composition element features, and picture emotion features, based on the file organization structure of SWF format. The experiments in this paper show that our scheme can be based on the four-layer framework of flash animation semantic extraction (i.e., metadata, component element, scene, and semantic layer), especially in the key techniques of scene feature extraction, component element feature extraction and picture emotion feature extraction than this other techniques.



---

*Source: 7999312-2022-03-24.xml*