Reviewerly/exHarmony huggingface dataset:
An Adversarial Approach to Mitigating Lazy Voting Threats to Committee-Based Consensus

We identify a threat to blockchain networks utilizing committee-based consensus mechanisms, which we call “Lazy Agreement,” and provide a countermeasure through a reputation-based consensus mechanism we call “Adversarial Consensus.” Lazy Agreement occurs when a node within the committee is able skip validation of data and vote to agree by default assuming based on the assumptions that the committee is most likely honest. When the committee consists of mostly honest nodes, the data presented by other nodes will likely be honest and thus most committee nodes will vote to accept the data. While no problem occurs when honest data is proposed, the risk of malicious data being accepted is increased due to voting to accept without verification. Our research shows that even when a small percentage of nodes are malicious, the presence of lazy nodes resulted in malicious blocks being produced. We present a countermeasure wherein honest nodes periodically submit bad data to identify and punish lazy nodes using a reputation-based system for committee member selection. After implementation of the Adversarial Consensus mechanism within our simulated system, malicious block production was eliminated when the rate of adversarial block production was above 30% and at least 50% of the network was honest. At higher adversarial production rates, malicious block production was eliminated even when only 40% of the network nodes were honest. The penalization of lazy and malicious nodes further increases the probability that honest nodes will be selected for participation in consensus committees the longer the network is running.
Building an Elementary Indonesian Textbook Readability Baseline Model

Reading resources are crucial for teachers to improve students comprehension on a subject. Readability is a crucial concern for professionals and academics in various sectors and areas of interest, such as education, applied linguistics, text linguistics, library science, business, medical, and technical communication. The readability of texts, readability models, and related techniques have been the subject of numerous studies. In this study, we build a baseline readability model on Indonesian elementary school textbook. We tackled the task as a classification task, which classifies the text into three classes: grade 1, grade 2, and grade 4 that represent three elementary grades. We used traditional predictors or surface-based predictors, syllable patterns, and BERT-based model (IndoBERT) with Random Forest and Support Vector Machine (SVM) as classification methods. The results showed that the IndoBERT neural representation with Support Vector Machine strategy outperforms other scenario with the accuracy of 73.4%.
Obtaining Maximum Probability Partitioning by Optimizing Calculation Strategy

Maximum probability partitioning is a method for segmenting a string. It determines the occurrence probability of substrings and maximizes their simultaneous probabilities. In time series data classification, a method has been proposed for estimating the label of a string using the self-information amount of the string through maximum string partitioning. To estimate the probability of substring occurrence using a matrix representation, the frequency of the substring in the matrix is counted through binary search using a suffix array. To further accelerate the computation of maximum probability partitioning, we propose a method for determining the starting position with an optimized calculation strategy. This proposed method is approximately 1.4 times faster than previous methods for real-world data, and it does not slow down the processing of data.
Evaluating the Robustness of Question Answering Model Against Context Variations

An important question is how robust a machine learning model is to unknown data when it is actually used. Because a typical dataset is constructed by annotating uniform text data with the necessary annotations, even if this dataset is split into training and test data and open testing is performed, the properties of the original uniform text data remain as properties common to the training and test data. Therefore, an open test on this dataset may not be sufficient to estimate the performance of models obtained by machine learning from this dataset on text data with very different properties. This study evaluates the robustness of a model obtained by machine learning from the existing question answering dataset built from an online encyclopedia to textual data with different properties. To evaluate this, we construct a new question answering dataset. While the existing dataset is built on text data from an online encyclopedia, our new dataset shares the same questions and answers as the existing dataset, but for contexts, they are collected from completely different web text data. Although the model must first determine whether an incoming question-context pair is answerable or not, our experiments show that the model obtained from the existing dataset performs significantly worse in determining the answerability.
Training Neural Machine Translation Models by Using an Entire Comparable Corpus

Neural machine translation (NMT) is often trained by using parallel sentences automatically extracted from comparable corpus where two language sentences correspond at a document level. Since few sentences correspond to each other in the comparable corpus, the other large portion of sentences are discarded without being extracted as the parallel sentences. Instead, this paper proposes to utilize an entire comparable corpus for training NMT. In order to use the comparable corpus as an additional training data, existing data augmentation methods from monolingual corpora are employed. In addition, a novel data augmentation method specifically designed for a comparable corpus, called Iterative Back-Translation with Comparability (IBTwC), is proposed. Our experimental evaluation was conducted on both low and high resource settings. For the low resource setting, it showed that the proposed IBTwC outperformed the conventional data augmentation methods. For the large resource setting, it revealed that the additional use of the comparable corpus did improve the MT performance, which resulted in the new state-of-the-art of the NTCIR-10 PatentMT task.
High Accuracy LPC Encoder Architecture for FPGA

This paper explores Linear Predictive Coding (LPC) and its FPGA implementation. LPC is widely used in speech processing for compression and machine learning. FPGA’s benefits, including real-time processing, parallelism, integration, high throughput, and flexibility, make it suitable for LPC. The paper presents an LPC encoder design using the Levinson-Durbin algorithm. Although FPGA-based LPC offers advantages, it poses challenges related to resources and complexity. The paper also describes the flow that help the selection of fixed point format that ensure the accuracy of the computation result. The implementation on a DE10-Standard Development Kit with the Cyclone V SX SoC shows high accuracy, low latency, and high throughput. Future work includes bit-width optimization and resource enhancements.
Visual Amplification of Geometry Problems: A Method for Synchronized Highlighting in Text and Diagrams

Automatic geometry problem solving has been instrumental in shaping intelligent education, aiding in the learning and understanding of geometric concepts. However, conventional methods’ oversight of intuitive visualization can increase the cognitive load during the learning process, leading to difficulties in understanding problems and solutions. To address this gap, this paper introduces a novel method for visual amplification of geometry problems, integrating synchronized highlighting in both text and diagrams. The methodology comprises two integral components: the understanding phase and the solving phase. A detailed alignment mechanism is proposed that maps geometric entities in the problem text with geometric primitives in diagrams, enhancing visualization across various problem types. The paper further extends this alignment to solution steps, enabling dynamic visualization. Experiments conducted on real-world datasets, encompassing geometric proof problems and geometric algebra problems, affirm the method’s efficiency, robustness, and applicability. The study presented herein stands as a substantial contribution to the field of geometry problem-solving, making geometric learning more comprehensible and accessible.
Sound and Visual Entrainment in Affective, Bio-Responsive Multi-User VR Interactives

We present our affective, bio-responsive virtual reality (VR) interactive framework that explores musical and visual forms of entrainment through amorphous nature-inspired phenomena that evolves and reacts to a tightly coupled real-virtual immersive space controlled through two users’ physiological data. Multiple layers of real-time sound and visuals inspired by nature phenomena are generated in 3D experiential VR space which is controlled through two users’ EEG, heart rate and gestures in real-time. In this paper, we present our system framework and setup and present our work-in-progress prototype.
Early vs. Late Multimodal Fusion for Recognizing Confusion in Collaborative Tasks

There has been a rapid transformation in the medium of learning and communication due to the pandemic. Multitudes have adopted online video platforms to learn and work from any corner of the world. Emotion detection is vital for understanding how well instructions are communicated through online interactions and for building cognitive systems that can identify human behavior. Confusion is a key emotion that can impact online learning and can be used to verify whether students using an online platform understand the material being taught. Our research expands on previous work regarding confusion detection, focusing on data fusion techniques. We explore the impact of early fusion (feature-level) vs late fusion (decision-level) on modeling confusion identification during a collaborative block building task. Experimenting with different classifiers, our results show that late fusion performs better with larger time windows. This fusion approach can aid in model interpretability.
Data Transmission to Mobile Devices via Multiple QR Codes

Data transmission to mobile devices can be done easily with the use of Bluetooth and Wi-Fi, but these methods have weaknesses such as the significant amount of time needed for the device search and authentication phase when transmitting small amounts of data. Previous works proposes transmitting data via multiple high capacity two-dimensional barcodes, which works as a partial solution but still has certain aspects that can be improved. In this paper, we propose a system for transmitting data via multiple QR codes that aims to maximize the overall data transmission rate by maximizing read accuracy and code processing speed. We also test the use of RaptorQ fountain code to improve handling of missed frames in the system. Analysis and tests done on the system proves that the proposed system can be reasonably used as an alternative data transmission method when transmitting files up to 500KB in size or when other data transmission methods are unavailable.
Exploring New Art Therapies by Integrating Drama Therapy and XR Technology

Nowadays, people live with strong stress and search for ways to relate to others. In particular, Japanese people tend to try to relieve their stress through self-help efforts without resorting to professionals. However, solving essential mental health problems with self-help is difficult. Therefore, I propose to focus on drama therapy. Drama therapy uses playful theatrical experiences to build a casual support system where everybody can participate without hesitation. However, current drama therapy has various shortcomings. The main issue is the shortage of drama therapists and shyness toward acting. Furthermore, drama therapy is limited in the number of places it is held, and in the number of sessions available per user. Therefore, during my Ph.D., I aim to develop XR Drama Therapy (XRDT) as a game-like therapy held in the metaverse, and conducted by an emotionally-intelligent artificial intelligence (AI) therapist. Four phases will be implemented to achieve this aim. First, I will investigate the session content necessary to implement drama therapy in XR. Second, I will implement a basic conversational AI therapist. Third, I will explore what nonverbal behaviors are instrumental to the therapy. Finally, I will assess different haptic technologies to convey affective touch. In the first experiment, I conducted psychological validation of the content of XRDT sessions and confirmed that perspective-taking and empathy could be improved.
Generating Scene Layout from Textual Descriptions Using Transformer

Creating images through generative AI technology is still challenging because it requires specifying detailed layouts and contents of objects. This paper introduces a layout generation method that uses Transformer-based deep neural networks to generate scene representations with multiple objects. This approach allows for an explainable layout of the image's objects, generated automatically from the text. Unlike the conventional layout generation process that requires sequential object predictions and post-processing of duplicate bounding boxes, our end-to-end approach uses parallel decoding. We conducted experiments to compare our method's quality and computational cost against existing ones, demonstrating its effectiveness and efficiency in generating layouts from textual representations. Our Text-to-Layout approach offers practical authoring tools that are computable with relatively lightweight networks while facilitating explainable image generation.
Robust Batch Relation Preserving for Lifelong Unsupervised Domain Adaptation

Lifelong unsupervised domain adaptation (lifelong UDA) is a challenging problem involving adaptation to unlabeled new domains, and alleviation of catastrophic forgetting of old domains. Existing methods impose an absolute relational distillation loss on features from the old model and the current model to alleviate forgetting, which hinders the model’s plasticity. Moreover, previous works in lifelong UDA naively employ pseudo-labeling mechanisms to learn from the unlabeled target domains, overlooking the noise problem in pseudo-labels. In this case, we propose Robust Batch Relation Preserving, allowing for more plasticity of the model when performing relational distillation and facilitating more robustness in pseudo-labeling mechanisms. Specifically, we consolidate previous knowledge by distilling batch-wise squished relation distributions, relaxing the constraints of rigid relation preserving. Meanwhile, we employ nuclear norm loss to promote diversity and improve certainty in intra-batch predictions to facilitate the robust pseudo-labeling of unlabeled target data. Extensive experiments on OfficeHome and DomainNet validate the effectiveness of our proposed method with a maximum improvement of 4.74% and 2.32% in the average adaptation performance and the overall accuracy after lifelong learning, respectively.
Training Dense Object Nets: A Novel Approach

Our work proposes a novel framework that addresses the computational limitations associated with training Dense Object Nets (DON) while achieving robust and dense visual object descriptors. DON's descriptors are known for their robustness to viewpoint and configuration changes, but training these requires image pairs with computationally expensive correspondence mapping. This limitation hampers dimensionality and robustness, thereby restricting object generalization. To overcome this, we introduce data generation procedure using synthetic augmentation and a novel deep learning architecture that produces denser visual descriptors with reduced computational demands. Notably, our framework eliminates the need for image pair correspondence mapping and showcases its application in a robotic grasping pipeline. Experimental results demonstrate that our approach yields descriptors as robust as those generated by DON.
Prompt Incorporates Math Knowledge to Improve Efficiency and Quality of Large Language Models to Translate Math Word Problems

Large Language Models (LLMs) models perform excellently in normal translation tasks [1], but are not effective in translating Math Word Problems (MWPs) with LLMs. The main problems can be summarized into two categories: one is that LLMs are unable to accurately translate specific mathematical knowledge in MWPs, such as formulas and mathematical patterns. The other is that LLMs often generate redundant content unrelated to the translation of MWPs. This research aims to investigate how to define the format of prompt and incorporate mathematical knowledge to improve the efficiency and quality of translating MWPs by LLMs. Firstly, the input and output formats of prompt are defined to reduce the generation of content unrelated to the translation of MWPs. Secondly, it is found that simply defining the roles and capabilities of LLMs in a zero-shot setting cannot improve the effectiveness of LLMs in translating MWPs, and even leads to a decrease in translation quality. This paper finds that the reason for the low-quality phenomenon of LLMs in translating MWPs is due to the lack of corresponding mathematical knowledge, therefore, this paper summarizes some of the mathematical knowledge that needs to be involved in translating MWPs and integrates it into prompt, so that the effect of LLMs in translating MWPs has been significantly improved.
Embedding Coordinates in Transformer for Behavior Recognition in Classroom Scenarios

The integration of computer vision research into educational contexts has recently attracted significant attention among scholars. One critical downstream application, classroom behavior recognition, poses unique challenges such as: 1) the nuanced differences between behavioral features and 2) the complexity of accurately locating students due to visual obstructions. To tackle these challenges, this study introduces a novel visual Transformer module specifically engineered for behavior recognition in classroom scenarios. Specifically, we first include the Contextual Transformer (CoT) block, which exploits inter-key contextual information to guide the adaptive learning of attention matrices, thereby enabling the extraction of richer behavioral features from classroom images. Then, relative position encoding is incorporated into the CoT block, forming a Position-CoT block (PCoT). The PCoT outputs a fusion of both static and dynamic context representations, and the inclusion of relative position encoding enhances the ability to accurately localize students within the visual frame. Finally, we replace all the standard 3 $\times$ 3 convolutions in ResNet with PCoT to construct a Transformer-style backbone network called PCoTNet. Through experiments, we validate the feasibility of using PCoTNet for classroom behavior recognition. Experimental validation confirms the effectiveness of the PCoT blocks and PCoTNet in classroom behavior recognition.
Towards Building Stemmer for Tenyidie Language

The Tenyidie language is a Tibeto-Burman Language of Nagaland in the north-eastern part of India. Tenyidie is a low-resource language spoken by the Tenyimia Community and is considered a major language in Nagaland. Tenyidie is a tonal, Subject-Object-Verb, and highly agglutinative language. Among the many Natural Language Processing (NLP) tasks, stemming is an important task in which the root or base form of the word is extracted. To the best of our knowledge, no work on stemming has been reported for this language. This work is a preliminary step towards building stemmer for Tenyidie Language. In our work, we reviewed the affixation rules for the Tenyidie language, using which we built 1,206 suffix combinations and 11 prefixes. We used a root word dictionary of 766 unique words and the generated affixes to build a stemmer for the Tenyidie Language. Our stemming approach achieved an accuracy of 60.69% on 430 inflected Tenyidie words.
Understanding Individual Differences in Reading Chinese Traditional Poetry Using Hidden Markov Models

Chinese traditional poetry, a special language art, is also consistent with “There are a thousand Hamlets in a thousand people’s eyes”. Most research subjectively analyzes the teaching and learning of Chinese traditional poetry, but few adopt the methods of natural science to explore the mental process of Chinese traditional poetry learning. This paper first explores the individual differences in reading Chinese traditional poetry by hidden Markov models. Here, we use eye-tracking to obtain visual attention during reading. The experimental results show there are temporary and spatial feature differences when different prior knowledge people read Chinese traditional poetry. This study might contribute to constructing the learning theory of Chinese traditional poetry.
Machine learning: a personal tour

Machine learning is about learning patterns in data. More specifically, it's about ‘input-output’ paradigms and computational frameworks, often, but not always applied to inputs to estimate outputs. It's also about the practice of abstraction and representation of of data, about model construction and estimation and calibrations; about testing, and assessing a model's performance and, last but not least, the application and interpretation of a model. Without touching every aspect, we'll describe the main thrust of machine learning, from the foundations up to applications and point out how machine learning is used in practice.
Towards Facial Expression Recognition in Immersive Virtual Reality with EmojiRain

The adoption of virtual reality (VR) continues to proliferate across various applications and research areas. Nevertheless, accurately assessing a user’s emotional state remains challenging. Facial expressions are an expressive and natural way to convey emotions. However, automatic facial expression recognition (FER) is rarely employed in immersive VR scenarios as head-mounted displays (HMDs) occlude most of the upper face. My PhD project aims to facilitate FER in immersive VR scenarios and to demonstrate its benefits. Using state-of-the-art architectures, I will create FER models specialized for HMD occlusion. Moreover, I will develop a novel affective game called EmojiRain that will be used in a user study to evaluate the developed FER models.