Chinese Inertial GAN for Writing Signal Generation and Recognition

Authors: Yifeng Wang, Yi Zhao
Primary area: datasets and benchmarks
Keywords: Inertial Sensors, Handwriting Recognition, Signal Generation, Human-Computer Interaction, Disabled People

Abstract:
Disabled people constitute a significant part of the global population, deserving of inclusive consideration and empathetic support. However, the current human-computer interaction based on keyboards may not meet the requirements of disabled people. The small size, ease of wearing, and low cost of inertial sensors make inertial sensor-based writing recognition a promising human-computer interaction option for disabled people. However, accurate recognition relies on massive inertial signal samples, which are hard to collect for the Chinese context due to the vast number of characters. Therefore, we design a Chinese inertial generative adversarial network (CI-GAN) containing Chinese glyph encoding (CGE), forced optimal transport (FOT), and semantic relevance alignment (SRA) to acquire unlimited high-quality training samples. Unlike existing vectorization focusing on the meaning of Chinese characters, CGE represents the shape and stroke features, providing glyph guidance for GAN to generate writing signals. FOT establishes a triple-consistency constraint between the input prompt, output signal features, and real signal features, ensuring the authenticity and semantic accuracy of the generated signals and preventing mode collapse and mixing. SRA constrains the consistency between the semantic relationships among multiple outputs and the corresponding input prompts, ensuring that similar inputs correspond to similar outputs (and vice versa), significantly alleviating the hallucination problem of generative models. The three modules guide the generator while also interacting with each other, forming a coupled system. By utilizing the massive training samples provided by CI-GAN, the performance of six widely used classifiers is improved from 6.7% to 98.4%, indicating that CI-GAN constructs a flexible and efficient data platform for Chinese inertial writing recognition. Furthermore, we release the first Chinese writing recognition dataset based on inertial sensors in GitHub.

Decision: reject
Reviewer scores: 6, 5, 6, 3, 6

Reviews and discussion:
<thread>
creation date:
1730446844506

thread_type:
official_review

<thread_object>
creator: reviewer_kb21

<content>
summary:
The paper presents an innovative approach to addressing the limitations in performance of inertial sensor-based systems for Chinese character recognition, which traditionally rely on extensive manual data collection. By introducing a Chinese Inertial Generative Adversarial Network (CI-GAN), the study offers a solution that generates unlimited, high-quality training samples, thereby providing a flexible and efficient data support platform for classification models. This method significantly reduces the dependency on labor-intensive data gathering and enhances the overall performance and feasibility of using inertial sensors for HCI in the context of disability.

strengths:
* **Clear Motivation and Feasible Approach**: The paper is driven by a well-defined goal—to improve HCI for disabled individuals using inertial sensors. The proposed solution, a generative adversarial network (GAN) for data generation, is not only innovative but also practically feasible, as evidenced by the experimental results.

* **Innovation and High Performance**: By introducing advanced techniques of CGE,  FOT and SRA, the study significantly enhances the recognition accuracy of Chinese characters, with performance improvements reported from 6.7% to 98.4%.

* **Social Impact and Community Contribution**: The research addresses significant accessibility issues for disabled individuals and adds substantial value to the community by releasing the first Chinese writing recognition dataset based on inertial sensors, enabling further advancements in the field.

weaknesses:
* **Visualization and Clarity of Diagrams**: The diagrams in the paper could be improved for better systematic representation and intuitiveness. Visualizing abstract constraints and regularization techniques more clearly would aid in understanding the complex interactions within the model. The task and symbols need a more detailed defination to improve the understanding.
* **Detailed Justification of Model Constraints**: The paper could be improved by including more detailed exploration of the motivations and effectiveness of using specific constraints such as the Forced Optimal Transport (FOT). A deeper discussion on why aligning input stroke encoding features with generated signal features and real signal features; and why utilizing Wasserstein distance as regularization can mitigate mode mixing and mode collapse is necessary to validate the approach.
* **Analysis of Robustness Under External Disturbances**: The paper lacks a thorough analysis of the system's robustness in the presence of external disturbances. Detailed insights into how these factors affect the system and recommendations for enhancing robustness would strengthen the paper.

These points should be addressed to enhance the overall comprehensibility and impact of the research.

questions:
As shown in Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730552467027

thread_type:
official_review

<thread_object>
creator: reviewer_rmes

<content>
summary:
In this paper, the author proposes a method of sensor-style Chinese character data generation based on GAN. It mainly consists of three modules CGE, FOT and SRA. CGE encodes Chinese characters according to glyphs. FOT uses a ternary consistency constraint to monitor the consistency of the predicted sample, the real sample, and the glyph encoding vector. The SRA module aligns glyph and semantic encoding. The author collected 4500 samples of 500 Chinese characters, including 1500 samples in the training set and 3000 samples in the test set. The author uses the proposed CI-GAN to generate additional training sets to augment the original data set. The validity of the generated data is proved by comparing the recognition effect of training with different data quantities. The effectiveness of the proposed module is verified by ablation experiments.

strengths:
This paper makes a strong contribution to research in accessible human-computer interaction, focusing on Chinese handwriting recognition for disabled individuals. It addresses an important issue by introducing CI-GAN, a generative model with unique modules—Chinese Glyph Encoding, Forced Optimal Transport, and Semantic Relevance Alignment—that effectively tackle the challenges of data scarcity and segmentation. According to the visualization results of Chinese glyph encodings, the module proposed in this paper is effective in encoding Chinese character shapes. The experimental results in Table 3 and Table 4 prove that the generated data has useful value.

weaknesses:
The proposed method section does not provide sufficient comparisons and analysis. The collected real data and generated data are insufficient, and the quality of the constructed dataset is not high. The experimental section lacks important comparative experiments and analysis, making it difficult to demonstrate the effectiveness of the proposed method. The specific issues are as follows:

1. **Methodology**: The authors propose a GAN-based generation method but do not compare its generation quality with other generative approaches, such as diffusion models or other GAN-based methods.

2. **Dataset**: The dataset collected is small in scale, with data from only nine individuals and without full coverage of the complete Chinese character set. 

    (1) The complexity of different Chinese characters is very different, and the author only shows the generation and classification results of relatively simple Chinese characters in this paper, it is impossible to evaluate the model's generation effect on complex Chinese characters. 

    (2) Writing habits vary greatly among individuals, leading to significant differences in handwriting styles. With data from only nine participants, how can the authors ensure that the generated data quality aligns with real-world scenarios?

3. **Experiments**: 

      (1) Comparative methods lack citations.

      (2) The algorithm’s performance has not been tested on other public datasets. Whether the CIGAN generation effect can be verified on other open source datasets of Chinese character data, such as IAHCC-UCAS2016, CASIA-OLHWDB (ICDAR 2013 Chinese Handwriting Recognition Competition).

      (3) The authors did not compare their method with other high-performing algorithms for Chinese character recognition, such as the one mentioned in [1].  As far as I know, [1] achieved a recognition accuracy of 96.78% on the dataset of all Chinese characters in the Level 1 Character Set (IAHCC-UCAS2016) and 97.86% on ICDAR-2013. I suggest the authors compare their method with more state-of-the-art (SOTA) approaches.

     (4) The experiments lack further analysis, such as individual-level performance testing and performance evaluation across characters with different stroke complexities. 

These improvements would better support the effectiveness and applicability of the proposed approach.

[1] Gan J, Wang W, Lu K. A new perspective: Recognizing online handwritten Chinese characters via 1-dimensional CNN[J]. Information Sciences, 2019, 478: 375-390.

questions:
Repeat:
1. **Methodology**: The authors propose a GAN-based generation method but do not compare its generation quality with other generative approaches, such as diffusion models or other GAN-based methods.

2. **Dataset**: The dataset collected is small in scale, with data from only nine individuals and without full coverage of the complete Chinese character set. 

    (1) The complexity of different Chinese characters is very different, and the author only shows the generation and classification results of relatively simple Chinese characters in this paper, it is impossible to evaluate the model's generation effect on complex Chinese characters. 

    (2) Writing habits vary greatly among individuals, leading to significant differences in handwriting styles. With data from only nine participants, how can the authors ensure that the generated data quality aligns with real-world scenarios?

3. **Experiments**: 

      (1) Comparative methods lack citations.

      (2) The algorithm’s performance has not been tested on other public datasets. Whether the CIGAN generation effect can be verified on other open source datasets of Chinese character data, such as IAHCC-UCAS2016, CASIA-OLHWDB (ICDAR 2013 Chinese Handwriting Recognition Competition).

      (3) The authors did not compare their method with other high-performing algorithms for Chinese character recognition, such as the one mentioned in [1].  As far as I know, [1] achieved a recognition accuracy of 96.78% on the dataset of all Chinese characters in the Level 1 Character Set (IAHCC-UCAS2016) and 97.86% on ICDAR-2013. I suggest the authors compare their method with more state-of-the-art (SOTA) approaches.

     (4) The experiments lack further analysis, such as individual-level performance testing and performance evaluation across characters with different stroke complexities. 

These improvements would better support the effectiveness and applicability of the proposed approach.

[1] Gan J, Wang W, Lu K. A new perspective: Recognizing online handwritten Chinese characters via 1-dimensional CNN[J]. Information Sciences, 2019, 478: 375-390.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730615345247

thread_type:
official_review

<thread_object>
creator: reviewer_a2vt

<content>
summary:
This paper introduces CI-GAN, a generative adversarial network for Chinese writing recognition using inertial sensors, designed to aid disabled individuals. CI-GAN incorporates Chinese glyph encoding, forced optimal transport, and semantic relevance alignment to generate accurate signals. With these synthetic signals, classifier accuracy improved from 6.7% to 98.4%. The study also releases the first Chinese inertial sensor dataset for writing recognition, advancing accessible human-computer interaction.

strengths:
1.	The application of research has significant potential and creativity about addressing accessibility needs for disabled individuals.
2.	The proposed dataset contributes valuable inertial sensor data for Chinese writing. And the research introduces a novel method using GAN for data augmentation, effectively addressing data scarcity and enhancing handwriting recognition research.
3.	The experimental results show promising improvements in classifier accuracy.

weaknesses:
1.	The concept of inertial data is introduced only in Section 4.2, making it somewhat difficult to understand when mentioned in the earlier parts of the paper. It is recommended to provide a brief introduction to this concept earlier on.
2.	The first point in the summary of contributions mentions that it "provides new tools for the study of the evolution and development of pictograms," which may not be suitable for the contributions summary, as it seems the research does not cover this aspect.
3.	The description of CGE mentioned in Section 3.1 seems to be just an embedding? In my opinion, the current version of the introduction may be somewhat complex.

questions:
1.	According to my understanding, CGE can be divided into two parts: 1. converting one-hot encoding into dense features, and 2. using α-order Rényi entropy regularization in GER. Therefore, in the ablation study in Section 4.4, what specific configuration is being ablated when CGE is removed? Which part of these two components is being eliminated? Additionally, can this ablation experiment validate the effects of the glyph encoding regularization (GER) proposed in Section 3.1?
2.	What is the difference between the pre-trained VAE mentioned in Section 3.2 and CGE in Section 3.1? It seems that both can extract glyph features. Can VAE replace CGE?
3.	What are h_G, h_T, and e in Section 3.2? It seems that e comes from the GAN input, h_G comes from the GAN output, but where does h_T come from during training?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730629793370

thread_type:
official_review

<thread_object>
creator: reviewer_fzkk

<content>
summary:
The paper propose CI-GAN, which enhances Chinese writing recognition for disabled users, generating high-quality samples and improving classifier performance significantly.

strengths:
1. The article is clearly written and easy to understand. 

2. The motivation is clear: translating subtle movements of user’s hand into written text can help disabled people of writing. 

3. Experiments demonstrate the effectiveness of the proposed method.

weaknesses:
1. The dataset is relatively small and lacks comprehensive coverage of the Chinese character set, which may not support the generation of more complex Chinese characters.

2. The proposed method should be tested on other public available benchmarks with other SOTA methods, such as IAHCC-UCAS2016 and CASIA-OLHWDB.

questions:
See weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730721057043

thread_type:
official_review

<thread_object>
creator: reviewer_fqcs

<content>
summary:
This paper tackles the challenge of data scarcity in Chinese writing recognition using inertial sensors by proposing the Chinese Inertial Generative Adversarial Network (CI-GAN). CI-GAN includes three innovative modules, Chinese Glyph Encoding (CGE), Forced Optimal Transport (FOT), and Semantic Relevance Alignment (SRA), to generate high-quality inertial signal samples. CGE captures the shape and stroke of Chinese characters, FOT ensures feature consistency to prevent mode collapse, and SRA aligns the semantic relevance of generated signals to their glyph structures. With CI-GAN, the authors establish a flexible data platform for Chinese writing recognition and claiming to release  the first inertial-sensor-based dataset on GitHub.

strengths:
1. The introduction of CI-GAN is a novel approach for enhancing data availability in Chinese inertial writing recognition, with modules designed specifically to tackle challenges unique to Chinese characters.
2. The improvement from 6.7% to 98.4% in classifier performance highlights the potential of CI-GAN-generated data to enhance recognition accuracy, indicating practical benefits for downstream applications.

weaknesses:
1. In Figure 1, CI-GAN is presented as a framework overview, yet it lacks consistency in terminology, with CGE mislabeled as "GER" and FOT written in full without abbreviation. Additionally, SRA is not visually represented in the figure. This detracts from the clarity of the diagram and makes it harder for readers to grasp the full framework.
2. The paper’s theoretical foundation could be strengthened. The current theoretical analysis is minimal, with only a few formulas provided. More detailed mathematical explanations, particularly for FOT’s role in preventing mode collapse, would lend greater credibility to the approach.
3. The ablation studies are somewhat limited, and additional experiments testing more comprehensive combinations of CGE, FOT, and SRA would provide a clearer understanding of each module's contribution. More exhaustive ablation tests would validate the effectiveness of the modules individually and collectively.
4. The example in Figure 1, intended to illustrate the framework's application for disabled individuals, doesn’t effectively convey this purpose. Including a more relatable example that directly addresses accessibility for disabled users would better align with the stated motivation of the study.

questions:
1. Can you clarify how Figure 1 relates to accessibility for disabled individuals, as the example seems disconnected?
2. Could you provide more theoretical details on the FOT component to reinforce its foundation?
3. Are more exhaustive ablation studies possible to validate the contributions of CGE, FOT, and SRA individually?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732199213566

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We sincerely thank the reviewers and the conference chair for their valuable feedback and thoughtful consideration of our paper. First, we want to clarify that collecting handwriting samples of Chinese characters is not easy. During data collection, volunteers wrote different Chinese characters continuously. We had to accurately locate the signal segments corresponding to each character from long signal streams, as shown in APPENDIX. B. **However, accurately segmenting and extracting signal segments requires synchronizing optical motion capture equipment and then comparing the inertial signals frame by frame with the optical capture results to find all character signal segments' starting and ending frames.** Therefore, we expended significant time and effort to obtain 4,500 signal samples in this paper, establishing the first Chinese handwriting recognition dataset based on inertial sensors, which we have made open-source partially. By contrast, our CI-GAN can directly generate handwriting motion signals according to the input Chinese character, eliminating the complex processes of signal segmentation, extraction, and cleaning, as well as the reliance on optical equipment. We believe it provides an efficient experimental data platform for the field.

Unlike the fields of CV and NLP, many deep learning methods have not yet been applied to the sensor domain. More importantly, unlike image generation, where the performance can be visually judged, **it is challenging to identify semantics in waveform by observation and determine whether the generated signal fluctuations are reasonable, which imposes high requirements on generative model design.** Therefore, we had to design multiple guidance and constraints for the generator, resulting in the design of Chinese Glyph Encoding (CGE), Forced Optimal Transport (FOT), and Semantic Relevance Alignment (SRA).

* CGE introduces a regularization term based on Rényi entropy, which increases the information content of the encoding matrix and the distinctiveness of class encodings, providing a new category representation method that can also be applied to other tasks. As far as we know, this is the first embedding targeted at the shape of Chinese characters rather than their meanings, providing rich semantic guidance for generating handwriting signals.
* FOT establishes a triple-consistency constraint between the input prompt, output signal features, and real signal features, ensuring the authenticity and semantic accuracy of the generated signals and preventing mode collapse and mixing.
* SRA constrains the consistency between the semantic relationships among multiple outputs and the corresponding input prompts, ensuring that similar inputs correspond to similar outputs (and vice versa), significantly alleviating the hallucination problem of generative models. Notably, the June 2024 **Nature** paper "Detecting Hallucination in Large Language Models Using Semantic Entropy," published after we released our paper, shares a similar idea with our proposed SRA. They assess model hallucination by repeatedly inputting the same prompts into generative models and evaluating the consistency of the outputs. Their approach essentially forces the model to produce similar outputs for similar prompts. Our SRA not only achieves this but also ensures that the relationships between prompts are mirrored in the relationships between the outputs. This significantly reduces hallucinations and enhances the model's practicality and stability.

CGE, FOT, and SRA not only guide and constrain the generator but also interact with each other, as shown in Section 3.4. The Chinese glyph encoding not only provides semantic guidance to the generator but also supplies the necessary encoding for FOT and SRA, and it is also supervised in the process. FOT and SRA share the VAE and generated signal features, providing different constraints for the generator, with FOT focusing on improving signal authenticity and enhancing the model's cognition of different categories through the semantic information injected by CGE, thereby mitigating mode collapse and mode mixing. In contrast, SRA ensures consistency between the relationships of multiple outputs and prompts through group-level supervision, which helps alleviate the hallucination problem of generative models.

In summary, the three modules proposed in CI-GAN are innovative and interlinked, significantly enhancing the performance of GANs in generating inertial sensor signals, as evidenced by numerous comparative and ablation experiments. **This method is a typical example of deep learning empowering the sensor domain and has been recognized by the industry and adopted by a medical wearable device manufacturer.** It has the potential to become a benchmark for data augmentation in the sensor signal processing field. We sincerely hope we have addressed the concerns of the reviewers, and once again, we thank everyone for their review and suggestions for this paper.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces a specific Generative Adversarial Network (GAN) for generating inertial sensor-based writing signals of Chinese characters, named CI-GAN. The authors have collected a small-scale dataset consisting of 4500 signal samples from only nine individuals, without full coverage of the complete Chinese character set. Experimental results show that the synthetic signals generated by CI-GAN can significantly improve character recognition accuracy. However, the proposed method focuses on a narrow application domain, specifically the generation of inertial sensor-based writing signals for Chinese characters. This limited scope may be more suitable for specialized conferences or journals focused on pattern recognition or signal processing rather than the broader audience of ICLR. Moreover, the dataset used for training and evaluation is relatively small, thus the experimental results are not sufficiently convincing. Based on these considerations, the decision is not to recommend acceptance at this time.