Chinese Inertial GAN for Writing Signal Generation and Recognition

Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 15:57:00

Abstract: Disabled people constitute a significant part of the global population, deserving of inclusive consideration and empathetic support. However, the current human-computer interaction based on keyboards may not meet the requirements of disabled people. The small size, ease of wearing, and low cost of inertial sensors make inertial sensor-based writing recognition a promising human-computer interaction option for disabled people. However, accurate recognition relies on massive inertial signal samples, which are hard to collect for the Chinese context due to the vast number of characters. Therefore, we design a Chinese inertial generative adversarial network (CI-GAN) containing Chinese glyph encoding (CGE), forced optimal transport (FOT), and semantic relevance alignment (SRA) to acquire unlimited high-quality training samples. Unlike existing vectorization focusing on the meaning of Chinese characters, CGE represents the shape and stroke features, providing glyph guidance for GAN to generate writing signals. FOT constrains feature consistency between generated and real signals through the designed forced feature matching mechanism, meanwhile addressing GANs' mode collapse and mixing issues by introducing Wasserstein distance. SRA captures the semantic relevance between various Chinese glyphs and injects this information into the GAN to establish batch-level constraints and set higher standards of generated signal quality. By utilizing the massive training samples provided by CI-GAN, the performance of six widely used classifiers is improved from 6.7\% to 98.4\%, indicating that CI-GAN constructs a flexible and efficient data platform for Chinese inertial writing recognition. Furthermore, we release the first Chinese writing recognition dataset based on inertial sensors in GitHub.

Main paper: 
 introduction: 
 One of the most significant obstacles for disabled individuals in their daily lives is the lack of efficient human-computer interaction (HCI) methods [1] . Traditional keyboard-based HCI systems often fail to meet the specific needs of disabled users, particularly those who are visually impaired or have lost their fingers, which underscores the urgent need for developing technologies that cater to the unique requirements of disabled individuals [2] . Providing tailored HCI solutions not only enhances their quality of life and independence but also facilitates their integration into society, enabling greater participation in education, employment, and social activities. Such technological advancements hold profound significance, creating a more inclusive and equitable society.As efficient motion-sensing components, inertial sensors can play a crucial role in recognizing writing movements. Inertial sensors can measure the acceleration and angular velocity of moving objects, making it possible to convert written characters into digital text [3, 4, 5, 6] . Due to their small size, ease of integration, low power consumption, and low cost, inertial sensors are widely used in electronic devices such as smartphones, smartwatches, and fitness bands [7, 8, 9, 10] , making them particularly Submitted to 38th Conference on Neural Information Processing Systems (NeurIPS 2024). Do not distribute.suitable for disabled users. Inertial sensors can be integrated into wearable devices, providing a more accessible and user-friendly means for disabled individuals to interact with computers and other digital devices. By capturing the subtle movements of a user's hand or other body parts, inertial sensors can translate these motions into written text, enabling effective communication and interaction without the need for a traditional keyboard. In addition, unlike optical or acoustic sensors, inertial sensors are highly resistant to external factors such as lighting conditions, physical obstructions, or environmental noise, which showcases their unique robustness in motion capture [11, 12, 13, 14, 15] . Consequently, inertial sensors provide a medium for Chinese character writing recognition that aligns with natural writing habits and can be seamlessly integrated into the writing process. With the widespread adoption of smart devices, the technology of Chinese character writing recognition based on inertial sensors may redefine the Chinese character input in the digital age, offering disabled people a comfortable human-computer interaction methods. However, the major challenge in achieving accurate Chinese writing recognition using inertial sensors is obtaining large-scale, diverse inertial writing data samples. For any recognition model aimed at accurately analyzing the complex strokes and structures of Chinese characters, it is crucial to train the model with extensive, diverse writing samples [16] . Considering that the collection and processing of Chinese writing samples are laborious and require high data quality and diversity, this task becomes exceedingly challenging and increasingly difficult as the number of characters increases. To acquire high-quality, diverse samples of inertial Chinese writing, we applied GAN for IMU writing signal generation for the first time and proposed CI-GAN, which can generate unlimited inertial writing signals for an input Chinese character, thereby providing rich training samples for Chinese writing recognition classifiers. CI-GAN provides a more intuitive and natural human-computer interaction method for the Chinese context and advances the application of smart devices with Chinese input.The main contributions of this paper are summarized as follows.• Considering traditional Chinese character embedding methods that only focus on the meaning of characters, we propose a Chinese glyph encoding (CGE), which represents the shape and structure of Chinese characters. CGE not only injects glyph and writing semantics into the generation of inertial signals but also provides new tools for studying the evolution and development of hieroglyphs.• We propose a forced optimal transport (FOT) loss for GAN, which not only avoids mode collapse and mode mixing during signal generation but also ensures feature consistency between the generated and real signals through a designed forced feature matching mechanism, thereby enhancing the authenticity of the generated signals.• To inject batch-level character semantic correlations into GAN and establish macro constraints, we propose a semantic relevance alignment (SRA), which aligns the relevance between generated signals and corresponding Chinese glyphs, thereby ensuring that the motion characteristics of the generated signal conform to the Chinese character structure.• Utilizing the training samples provided by CI-GAN, we increase the Chinese writing recognition performance of six widely used classifiers from 6.7% to 98.4%. Furthermore, we provide the application scenarios and strategies of 6 classifiers in writing recognition according to their performance metrics. For the sake of sharing, we release the first Chinese writing recognition dataset based on inertial sensors in GitHub.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes]Justification: We have specifically summarized the main contributions of this article in the introduction.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper.• The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.• The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.• It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 The technology for recognizing Chinese handwriting movements has the potential to bridge the gap between traditional writing and digital input, providing disabled individuals with a natural way of writing and greatly enhancing their ability to participate in digital communication, education, and employment. It also offers a new human-computer interaction avenue for normal people. Hence, Chinese handwriting movement recognition has garnered significant attention in recent years, leading to numerous related research achievements. Ren et al. utilized the Leap Motion device to propose an RNN-based method for recognizing Chinese characters written in the air [17] . The Leap Motion sensor, consisting of two infrared emitters and two cameras, can accurately capture the motion of hands in three-dimensional (3D) space [18] . However, the Leap Motion device is sensitive to lighting conditions, and either too strong or too weak light can interfere with the transmission and reception of infrared rays, affecting the recognition effect [19] . Additionally, the detection space of the Leap Motion device is an inverted quadrangular pyramid, limiting its field of view. Movements outside this range cannot be captured. Most importantly, the Leap Motion device is expensive and requires a connection to a computer or VR headset to function, severely limiting its application prospects [20] .As wireless networks become more prevalent, Wi-Fi signals are gradually being applied to motion capture [21, 22] . Since Wi-Fi signals can penetrate objects and are unaffected by lighting conditions, they have a broader application scope than optical motion capture systems [23, 24] . Guo et al. used the channel state information (CSI), extracted from Wi-Fi signals reflected by hand movements, to recognize 26 air-written English letters [25] . However, while Wi-Fi signals do not have visual range limitations and can penetrate obstacles, they are easily disturbed by other signals on the same unlicensed band, severely affecting system performance. Moreover, the sampling frequency and resolution of Wi-Fi signals are very limited, making it difficult to capture detailed information during the writing process and, thus, hard to recognize air-written Chinese characters accurately [26, 27] .Despite the advantages of low cost, wearability, and low power consumption offered by inertial sensors, there is currently a lack of large-scale, high-quality public datasets, causing few studies to use inertial sensors for 3D Chinese handwriting recognition [28, 29, 30, 31] . To collect data, Zhang et al. employed 12 volunteers, each of whom was asked to write the assigned Chinese characters on paper 30 times [32] . The inertial measurement unit (IMU) built into smartwatches was used to collect the motion signals of the volunteers while writing, ultimately achieving a recognition accuracy of 90.2% for 200 Chinese characters. However, this study aims to identify the signals of normal individuals writing on paper, which is not applicable to people with disabilities. Moreover, this method can only realize desktop-based 2D writing recognition, which reduces the comfort and flexibility of the writing process, inherently limiting the application scenarios of Chinese handwriting recognition. Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] As shown in Appendix B.2.Justification: Non-standard or cursive handwriting may pose challenges for accurate signal generation and recognition.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.• The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.• The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.• The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.• The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.• If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.• While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
method: 
 To generate inertial writing signals for Chinese characters, we propose the Chinese inertial generative adversarial network (CI-GAN), as shown in Fig. 1 . For an input Chinese character, its one-hot encoding is transformed into glyph encoding using our designed glyph encoding dictionary, which stores the glyph shapes and stroke features of different Chinese characters. Thus, the obtained Chinese glyph encoding contains rich writing features of the input character. This glyph encoding, along with a random noise vector, is fed into a GAN, generating the synthetic IMU signal for the character, where glyph encoding provides glyph and stroke features of the input character, while the random noise introduces randomness to the virtual signal generation, ensuring the diversity and variability of the generated signals. To ensure that the GAN learns the IMU signal patterns for each character, we designed a forced optimal transport (FOT) loss, which not only mitigates the issues of mode collapse and mode mixing typically observed in GAN frameworks but also forces the generated IMU signals In one-hot encoding, each Chinese character is represented by a high-dimensional sparse vector (where only one element is 1, and all others are 0), which results in all characters being equidistant in the vector space, thereby losing the abundant semantic information contained in the characters. Therefore, one-hot encoding fails to inject rich information into GAN. Although there are some commonly used Chinese character embeddings, these embeddings store meaning information of the characters, not glyph information (i.e., shape, structure and writing strokes). For example, the characters "天" (sky) and "夫" (husband) are quite similar in writing motions, but their meanings are significantly different. To this end, we propose a Chinese glyph encoding (CGE), which encodes Chinese characters based on their glyph shapes and writing actions.Considering that the inertial sensor signals capture the writing motion of Chinese characters, the motion signal exactly contains glyph information, which encourages simultaneous learning signal generation and Chinese glyph encoding under the supervision of real signals. Therefore, we create a learnable weight matrix W after the one-hot input layer to capture the glyph information. When a Chinese character is input into CI-GAN in one-hot encoding, it first passes through this weight matrix.Since only one element in the one-hot encoding is 1, and the rest are 0, multiplying one-hot encoding by the weight matrix W means obtaining one row of the matrix W . Hence, each row of W can be seen as an encoding of a Chinese character, and this matrix can serve as a glyph encoding dictionary of Chinese characters. However, an unguided Chinese encoding dictionary often struggles to capture the differences in glyph shapes among different characters, assigning similar glyph encodings to characters with distinct glyphs. To address this, we propose a glyph encoding regularization (GER), which enhances the orthogonality of all character encoding vectors and increases their information entropy to store as many glyph features of the characters as possible, thereby avoiding triviality like one-hot encoding. Specifically, we use the α-order Rényi entropy to measure the information content of the glyph encoding dictionary W , calculated as follows:EQUATIONwhere, N represents the number of Chinese characters, which corresponds to the number of rows in the weight (encoding) matrix W . G is the Gram matrix of W , where G ij equal to the inner product of the i-th and j-th rows of W , and G is the trace-normalized G, i.e., tr( G) = 1. In similar problems, αEnsuring the authenticity of virtual signals poses the greatest challenge when generating diverse signals, especially in following physical laws and simulating the potential dynamical characteristics of actual motions. To this end, we propose the forced feature matching (FFM), which ensures that the generated signal feature closely matches the real signal feature and the corresponding glyph encoding.Specifically, we use a pre-trained variational autoencoder to extract the real signal feature h T and generated signal feature h G . Then, the consistency of h T , h G , and the corresponding glyph encoding e is constrained by L F F M .EQUATIONAnother critical challenge lies in the mode collapse and mode mixing issue inherent to GAN architectures. Mode collapse limits the diversity of generated signal samples, causing GAN to generate signals only for a few Chinese characters, regardless of the diversity of input. On the other hand, mode mixing problems cause the generated signal to contain blend characteristics of multiple modes, which is unrealistic and unrecognizable. To address these issues, we introduce the optimal transport to GAN, We invited nine volunteers, each using their smartphone's built-in inertial sensors to record handwriting movements. The nine smartphones and their corresponding sensor models are listed in Table 1 . Each volunteer held their phone according to their personal habit and wrote 500Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [Yes] As shown in Appendix C.Justification: We provide a detailed explanation and present rigorous mathematical proof of utilizing optimal transport to alleviate mode collapse and mixing issues of GAN.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.• All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.• Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.• Theorems and Lemmas that the proof relies upon should be properly referenced. 
data collection and experimental setup: 
 Chinese characters in the air (sourced from the To visually demonstrate the signal generation effect of CI-GAN, we visualized the real and generated Through continuous collaboration with healthcare professionals and the disabled community, we will refine and optimize these multimodal systems to ensure they deliver the highest functionality and user satisfaction. Ultimately, this research aims to foster a society where digital accessibility is a fundamental right, ensuring that all individuals, regardless of physical abilities, can engage fully and independently with the digital world.           With the abundant training samples generated by CI-GAN, the handwriting recognition performance of all six classifiers significantly improved. To further verify the recognition performance of different classifiers on characters with similar strokes and glyphs, we selected four groups of characters with similar handwriting movements from the test set ("八人入大天太", "办为方力万历", "过达这边近还", and "认议计许话识") and presented the recognition results of the six classifiers in confusion matrices, as shown in Fig. 7 . It can be observed that the values on the diagonal of all confusion matrices are significantly higher than the non-diagonal values, indicating high recognition accuracy for these similar handwriting characters with the help of samples generated by CI-GAN. However, some characters are still misrecognized. For instance, the characters "八", "人", and "入" have extremely similar structures and writing movements, posing challenges even when massive training samples are provided. Moreover, continuous and non-standard writing can also cause recognition obstacles. For instance, although the characters "过" and "达" have different strokes in static form, they are very similar in dynamic handwriting. Despite these challenges, the synthetic IMU handwriting samples generated by CI-GAN significantly enhance the classifiers' ability to recognize characters with similar glyph structures and handwriting movements, highlighting the value and significance of the proposed CI-GAN method. By providing diverse and high-quality training samples, CI-GAN improves handwriting recognition classifiers' performance and generalization ability, making it a valuable tool for advancing Chinese handwriting recognition technology.The CI-GAN model demonstrates significant performance improvements across multiple classifiers, as shown in Table 4 . The Transformer classifier, for instance, achieves a mean accuracy of 98.4%, compared to 15.7% with the traditional GAN and 1.7% without data augmentation. This highlights CI-GAN's ability to generate realistic and diverse training samples that enhance handwriting recognition. Moreover, CI-GAN consistently improves accuracy and stability for all classifiers tested. The 1DCNN's accuracy increases to 95.7% from 18.5% with the traditional GAN and 0.87% without augmentation. Similarly, other models, including LSTM, RandomForest, XGBoost, and SVM, show substantial gains, underscoring CI-GAN's effectiveness across diverse machine-learning contexts.In addition, the narrow 95% confidence intervals, such as [98.2822%, 98.5178%] for the Transformer, validate the statistical significance and reliability of these results. This confirms CI-GAN's potential to consistently enhance classifier performance. In conclusion, CI-GAN represents a major advancement in Chinese handwriting recognition by generating high-quality, diverse inertial signals.This significantly boosts the accuracy and reliability of various classifiers, demonstrating CI-GAN's transformative potential in the field. While the CI-GAN model demonstrates significant advancements in Chinese handwriting generation and recognition, some practical limitations could impact its performance in real-world applications.For instance, non-standard or cursive handwriting may pose challenges for accurate signal generation and recognition. Additionally, environmental factors such as external movements or vibrations when using handheld devices could affect the inertial sensor data quality, leading to variations in recognition accuracy. Future work could focus on developing more robust algorithms that account for these realworld variations and improving the model's adaptability to diverse handwriting styles and conditions.These enhancements would ensure that the CI-GAN technology remains effective across a broader range of practical scenarios.To generate large-scale and high-quality handwriting signals, we introduce optimal transport theory into the generative adversarial network to alleviate mode collapse and mixing issues. We provide a detailed explanation and present a rigorous mathematical proof to show the advantages of this operation.In traditional conditional GANs, the generator G and the discriminator D are trained by minimizing the loss function L tradition :L tradition = min G max D E x∼p data [log D(x)] + E z∼pz [log(1 -D(G(z)))],where p data is the real data distribution, and p z is the distribution of the generator's input noise. This loss function essentially minimizes the Jensen-Shannon Divergence (JSD) between the real data distribution p data and the generated data distribution p g :JSD(p data p g ) = 1 2 KL(p data M ) + 1 2 KL(p g M ),where M = 1 2 (p data + p g ) and KL denotes the Kullback-Leibler divergence. However, JSD has a notable drawback: when the real and generated data distributions do not overlap, the JSD becomes zero, causing the gradients to vanish. This leads to mode collapse, where the generator produces a limited variety of samples.In optimal transport theory, the Wasserstein distance is utilized to measure the minimum cost of transforming one probability distribution into another. Given two probability distributions µ and ν on a metric space X , the Wasserstein distance W is:W (µ, ν) = inf γ∈Π(µ,ν) E (x,y)∼γ [d(x, y)],where Π(µ, ν) is the set of all joint distributions whose marginals are µ and ν, and d(x, y) is a distance metric on X . Therefore, we introduce the Wasserstein distance in optimal transport theory as new loss function L OT , whose objective is to minimize the Wasserstein distance between the generated distribution p g and the real distribution p data . The L OT is defined as:L OT = min G max D∈D E x∼p data [D(x)] -E z∼pz [D(G(z))]where D is the set of 1-Lipschitz functions. This Lipschitz constraint can be enforced through weight clipping or gradient penalty. In L OT , the discriminator D is constrained to be 1-Lipschitz:|D(x 1 ) -D(x 2 )| ≤ |x 1 -x 2 |.This constraint ensures that the discriminator provides meaningful gradients even when p g and p data do not overlap. Using the Kantorovich-Rubinstein duality, we can express the Wasserstein distance as:W (p data , p g ) = sup f L ≤1 E x∼p data [f (x)] -E x∼pg [f (x)].Since f is Lipschitz continuous, it ensures that the gradients ∇f (x) are bounded and do not vanish.Hence, during the optimization process, the generator receives consistent and informative gradient updates that guide it to produce more realistic and diverse samples. The gradient of the loss function L OT with respect to the generator's parameters θ is:∇ θ E z∼pz [D(G θ (z))] = E z∼pz [∇ θ D(G θ (z))].This gradient does not vanish even if p g and p data have disjoint supports, thanks to the 1-Lipschitz property of D. As a result, the generator G can still receive valuable gradient information to adjust its parameters and gradually make p g approximate p data even if p g and p data do not overlap, effectively addressing mode collapse and mode mixing issues. Overall, after introducing optimal transport theory, we overcome the gradient vanishing problem inherent in traditional GANs, effectively mitigating mode collapse and mode mixing. L OT maintains the existence and relevance of gradients during training, enabling the generator to continuously improve and produce more diverse and realistic handwriting samples.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] We have made every effort to disclose all experimental details, including CPU model, GPU model, PyTorch framework version, built-in IMU specifications of 9 experimental smartphones, and even the gender of volunteers.Justification: The paper provides comprehensive details on the data collection process, data splits, hyperparameters, training procedures, and statistical analyses, ensuring that all necessary information is disclosed to fully reproduce the main experimental results and validate the claims and conclusions.Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.• If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.• Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.• While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)• The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean.• It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.• For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).• If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] As shown in Table 2 .Justification: The paper specifies the type of compute resources used, including the Nvidia RTX 2080TI GPU and Intel Xeon W-2133 CPU, and provides details on memory usage and execution time, ensuring sufficient information is available to reproduce the experiments.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.• The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.• The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Justification: The research adheres to the NeurIPS Code of Ethics in all respects, ensuring ethical considerations are met in data collection, experimentation, and reporting of results.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.• The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.• The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.• If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: the paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.• Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.• We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.• Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.• According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.• For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.is established to minimize the difference between the two matrices, thereby ensuring that the semantic relationships in the input character glyphs are accurately contained in the generated signals. 
 
 Appendix:  


Decision: The paper proposes CI-GAN, a generative adversarial network designed to produce high-quality inertial sensor data for Chinese character recognition, aiming to improve accessibility for vision-impaired individuals. Reviewers (R-3apC, R-NmsP, R-NSpH) acknowledge the significance of addressing this social problem and the potential benefits. However, concerns are raised about the lack of novelty in methodology and insufficient comparisons with standard data augmentation techniques (R-NmsP, R-3apC, R-NSpH). Additionally, the small dataset size (R-NmsP) and lack of detailed baseline configurations (R-NSpH) limit the validation of the proposed approach. Although the reviewers didn't participate in the discussion phase (after being contacted by the AC), the AC feels the paper is not ready for publication.

Reviews: review: paper_summary: paper_summary: The paper address an important probem in human computer interaction: making computers accessible to vision impaired people. The paper address this my collection paired data of text and imu signals. First, the paper address the issues of limited data by training a generative model, to resample/bootstrap more data and then train recognition model on both real and generated data to archive high performance.
strength_weakness: strength_weakness: My main concern is, that it is very unlikely that we get more than we give to the system, the generated samples are a function of real samples. 
I would like to see, a competitive baseline with good data augmentation, and maybe on a low data regime gan generated samples are better than augmentation, but this has to be shown, otherwise, I don't see the value of extra effort to train a generative model to get data augmentation.
questions: questions: please see my concerns about the weakness section.
limitations: limitations: I wouldn't say this is a major limitation, but on the scale axis, this problem can be solved by collecting more data. Unlike annotations like explaining an image or video, handwriting signals are more easy to collect on the long term. would be nice if the authors can address this, also please explain the issues with data augmentation.
paper_summary: paper_summary: This paper proposes CI-GAN to acquire unlimited high-quality training samples, alleviating the data scarcity in the inertial signal recognition of Chinese characters. By utilizing these generated data, the performance of recognition models is highly improved.
strength_weakness: strength_weakness: - The pipeline lacks novelty. The employed technologies are widely used in CV and NLP, and the proposed pipeline merely reuses them for the inertial signal domain without any innovative design. Furthermore, the author fails to cite relevant studies such as [1][2] and does not discuss their differences.
[1] Wasserstein GAN (WGAN)
[2] Efficient Estimation of Word Representations in Vector Space

- The proposed CGE is simply a learnable embedding to represent Chinese characters, lacking innovative design for glyph information. The author introduces GER to enhance the orthogonality of character embeddings but does not provide an ablation study to verify its effectiveness.

- The author uses Wasserstein distance in GANs. What is the difference between this approach and WGAN [1]? Additionally, the author proposes using FFM to supervise the signal in feature spaces. These measures are also similar to some works, such as perceptual loss using VGG and identity loss using ArcFace, but the author does not cite these and discuss the difference. 

- The dataset used for training and testing is too small, which could not effectively verify the effectiveness of the proposed method.
questions: questions: See Weaknesses.
limitations: limitations: N/A
paper_summary: paper_summary: The paper presents a novel Chinese inertial generative adversarial network (CI-GAN) designed to generate high-quality training samples for Chinese writing recognition using inertial sensors. The CI-GAN integrates Chinese Glyph Encoding (CGE), Forced Optimal Transport (FOT), and Semantic Relevance Alignment (SRA) to enhance the quality and authenticity of generated inertial signals. The approach addresses the challenge of collecting diverse and extensive training data for Chinese character recognition, showing significant improvements in classifier performance.
strength_weakness: strength_weakness: 1.Lack of Detailed Baseline Configuration: The paper compares CI-GAN with a traditional GAN in the appendix, but fails to provide detailed settings for the baseline method. This lack of information hinders the ability to fully understand and replicate the comparative effectiveness reported.
2.Insufficient Comparison with Other Augmentation Techniques: The study does not compare CI-GAN with other data augmentation methods, such as random perturbations. It remains unexplored whether applying random disturbances to the data could also substantially improve classifier performance.
questions: questions: 1. Could you provide more information about the baseline model and the CI-GAN model used in your study, such as the number of parameters and other configuration settings?
2. Have you considered evaluating how simple data augmentation methods, such as random perturbations, might also significantly improve classifier performance?
limitations: limitations: The authors have discussed limitations related to the variability of writing styles and the potential impact of environmental factors on sensor data.

Comments: title: Hoping Our Response Meets Your Expectations
Hoping Our Response Meets Your Expectations
Hoping Our Response Meets Your Expectations
comment: Thank you for providing additional feedback. We would like to address the remaining concerns you have raised.

1. Lack of Clear Innovation:
We understand that you perceive CI-GAN as a variation of c-GAN, but if we follow this logic, c-GAN itself could also be considered a variation of GAN. By this reasoning, all models based on the GAN architecture could be seen as mere variations, thereby discounting any innovations in the field. However, we strongly believe that the true measure of innovation lies in the specific adaptations and design made to the base architecture to address unique challenges—in our case, the challenges inherent to inertial signal generation.

Regarding the Chinese Glyph Encoding (CGE), it is indeed more than just an “upscaled one-hot embedding.” CGE was designed as a novel way of category encoding that injects more information into the generative process. We devised a Renyi entropy-based regularization applied to a learnable category encoding matrix, significantly enhancing the matrix's ability to represent categorical information. This allows CGE to capture the nuances of Chinese character glyphs, which is a novel approach in the context of signal generation. To support this, we have visualized the glyph encodings of various Chinese characters, demonstrating that characters with similar glyphs are indeed closer in the embedding space. This visualization underscores the effectiveness of CGE in preserving the structural relationships between different characters.

Additionally, we are puzzled by your comment that “The concept of hallucination introduced in the rebuttal also seems unrelated to the core task.” Hallucination is a well-known issue in generative models, particularly in scenarios where the generated outputs can deviate from realistic or expected patterns. Our SRA module is specifically designed to mitigate hallucination by ensuring that the semantic relationships between generated signals are consistent with those of the input glyphs, thereby enhancing the realism and reliability of the generated signals. Addressing hallucination is not only relevant but crucial to the success of our generative task.

2. Value of Intuitive Motivations:
We respectfully disagree with the notion that intuitive design somehow detracts from the innovation or significance of our contributions. On the contrary, intuitive designs often lead to more effective and impactful solutions precisely because they resonate with the underlying principles of the problem being addressed. Each of the three modules in CI-GAN—CGE, FOT, and SRA—was designed with clear, intuitive motivations, and each one independently offers significant contributions to the field of deep learning. For example:

CGE introduces a novel way of category encoding that injects more information into the generative process, enhancing the quality and diversity of the generated signals.
FOT establishes a triplet constraint between the input, output, and label, addressing issues such as mode collapse, mode mixing, and the authenticity of generated results. This approach helps to ensure that the generated signals are not only diverse but also semantically accurate and realistic.
SRA ensures that the semantic relationships between inputs are maintained in the generated outputs, reducing the likelihood of hallucinations.
These modules are not merely empirical tweaks; they represent fundamental advancements that can inspire future research. When integrated into the CI-GAN framework, these modules work synergistically to achieve the first successful generation of inertial sensor signals. This is not just a collection of empirical studies; it is a cohesive and innovative solution to a complex problem that has not been addressed before.

3. Visualization and Diversity:
Your comment that “there is no visualization of the generated handwritten characters” seems to overlook the visualizations we provided in the paper. Figure 3 illustrates the generated signals for different Chinese characters, showing how they closely follow the fluctuation trends of real signals. To further emphasize diversity, Figure 4 presents the results for multiple generated signals of the same character, “王,” compared with a real handwriting signal. These visualizations clearly demonstrate that CI-GAN not only generates diverse samples but also maintains the essential characteristics of real handwriting signals. The differences in individual features, while preserving overall trends, validate the model’s ability to produce high-quality, diverse, and realistic samples.

We hope this response clarifies the innovations and contributions of our work. We believe CI-GAN represents a significant advancement in the field of inertial signal generation and has the potential to drive future research and applications. We sincerely appreciate your thoughtful consideration of our work and hope that you will recognize the value it brings.

Thank you for your time and consideration.
Thank you for your detailed rebuttal and the additional explanations provided. After carefully considering your responses, I still have concerns preventing me from recommending acceptance.

- Lack of Clear Innovation: The CI-GAN framework seems to be a variation of conditional GAN, and the CGE appears to be an upscaled one-hot embedding rather than a novel integration of glyph information. Without clear motivation and experimental validation, the proposed modules show differences but not true innovation. The concept of hallucination introduced in the rebuttal also seems unrelated to the core task, making it difficult to understand its relevance.

- Overcomplication with Intuitive Modules: The paper introduces several modules based on intuitive motivations, which makes it hard to identify a central, innovative contribution. The work feels more like a collection of empirical studies than a focused research effort.

- Insufficient Experimental Validation: I still find the experiments lacking. For example, there is no visualization of the generated handwritten characters or an assessment of their diversity, which would be critical in evaluating the effectiveness of CI-GAN.

I hope these points help you refine your work in the future.
I apologize for reaching out once more, but with less than 24 hours remaining in the discussion period, I am deeply concerned as we have not yet received any responses to our detailed rebuttals. This paper is incredibly important to us, both for our ongoing research and for its potential contributions to the field.

We have worked tirelessly to address every concern and suggestion raised in your initial reviews, providing thorough explanations and supplementing a lot of experiments in 7 days. We genuinely believe that our work brings valuable innovations that could greatly benefit the research community. It would be truly disheartening for this paper to go unnoticed or unacknowledged.

With the utmost respect and sincerity, I humbly request that you please review our responses and provide your final feedback before the discussion period ends. Your input is crucial to us, and we are deeply hopeful that our efforts will meet your expectations.

Thank you for your time, consideration, and understanding during this critical time.
I hope this message finds you well. I apologize for reaching out again, but with only one day left in the discussion period and no responses yet, I am feeling anxious about the status of our submission. This paper is of immense importance to us, and we have put in a great deal of effort to thoroughly address every concern and suggestion raised in your initial reviews.

We genuinely believe that the experimental results and detailed explanations provided in our response have resolved the issues you highlighted. This work is critical not only to our research but also has the potential to make a significant impact in the field.

I humbly and earnestly request that you please review our responses and provide your final feedback. Your input is invaluable, and I sincerely hope that our efforts meet your expectations.

Thank you so much for your time and understanding.
Thank you once again for your insightful feedback on our paper. We have carefully addressed each of your concerns in our response, particularly regarding the detailed baseline configurations and the comparisons with various data augmentation techniques. We genuinely believe that our work offers valuable advancements in the field, especially given the lack of deep learning-based augmentation methods for inertial sensors.

We would greatly appreciate it if you could take another look at our revisions and explanations. Your feedback is invaluable to us, and we sincerely hope that our paper can meet your expectations and contribute meaningfully to the field.

Thank you for your time and consideration.
I hope this message finds you well. I wanted to take a moment to sincerely thank you for your thoughtful and detailed review of our submission. Your insights have been invaluable in helping us refine our work, and we have made considerable efforts to address each of your concerns thoroughly.

In our response, we provided a detailed comparison with various data augmentation methods to highlight the significant value of our CI-GAN approach, which introduces generative deep learning models into inertial sensor data augmentation for the first time. Our results, which include extensive testing and ablation studies, show that CI-GAN not only performs significantly better than other methods but also offers a flexible platform that addresses the specific challenges of the sensor signal domain. We genuinely believe that our approach brings innovation to this field, particularly through the designed Chinese Glyph Encoding, Forced Optimal Transport, and Semantic Relevance Alignment, which together form a cohesive and effective system. These elements work in tandem to ensure not just the generation of realistic signals but also their alignment with the semantic content, which is crucial for practical applications.

Thank you once again for your time and effort. We are eagerly looking forward to your response.
Thank you once again for your thoughtful review and valuable feedback on our paper. We have taken your comments very seriously and have provided detailed explanations and clarifications in our response. We understand that assessing innovation can sometimes involve different perspectives, and we sincerely hope that you will consider our explanations, particularly regarding the aspects of novelty that we have highlighted.

Our research not only introduces some unique design elements but has also shown promising results in practical applications. We genuinely believe that this work can make a meaningful contribution to both the academic community and industry. If there are any further questions or areas that need clarification, we would be more than happy to discuss them in greater detail.

We truly appreciate your time and effort and hope that our work can meet your expectations.