Reviewerly/exHarmony huggingface dataset:
Detection of Facial Directions and Features With YOLO-Based Deep-Learning Technology for Pre-Processing of Gaze Estimation

In this work, the pre-processing technology for gaze estimation is studied when the person is located at a distance of about three meters, and the proposed design applies the YOLO-based deep-learning model to detect two facial landmarks and six facial directions. By combining the appearance and geometric-features based schemes, the proposed design provides the pre-processing information for gaze estimation without the calibration process. By experiments, the input size of YOLOv3-tiny based model is set to 608x608 pixels. With the testing mode, the proposed method performs AP (average precision) to be 99% for detection of the six facial directions and two facial features. Compared with the previous design, the proposed method can detect more facial directions, and it also provides the simplified facial-landmark function for the further gaze estimation.
A Multimodal Interpretable Visual Question Answering Model Introducing Image Caption Processor

This paper presents a multimodal interpretable visual question answering (VQA) model to solve the lack of overall understanding of the previous models, which is caused by focusing too much on the question. Compared with the previous models with only image reference, we newly introduce an image caption processor to improve the prediction process of the VQA model. The experimental results show that our model can generate the answers more accurately and the explanations are more reasonable and closer to the truth than some state-of-art methods.
An Implementation of Code Writing Problem Platform for Python Programming Learning Using Node.js

Nowadays, Python programming is very popular and highly demanded from various societies because it is very efficient for application developments, task automations, and data analysis. Currently, we are studying the code writing problem (CWP) to assist self-study of Python programming by novice students. In this paper, we implement the CWP platform on Node.js. JavaScript is used at both the server and client sides for the efficient programming. Docker is adopted to help distributing it to the students. The user interface at the client shows the CWP assignments and accepts the answer code submissions. By running the test code on unittest at the server, the correctness of them is automatically verified and is returned to the interface. To confirm the validity and the effectiveness of the implementation, we asked 11 novice students in Japan and Indonesia to install and use the platform to solve the CWP.
Usability Evaluation of Operating Devices for Electric Wheelchairs

In recent years, much research has been conducted on automated wheelchair technology. However, few automated wheelchairs have been commercialized. In addition, there is a lack of research on devices to operate wheelchairs manually or by steering wheel as well as automatically. The purpose of this study is to develop a wheelchair that makes everyday life more comfortable. First, we considered what kind of shape and sensors should be used for the operating device in developing the wheelchair. New input methods were considered for new sleigh operation devices, including a type with a small joystick, a fingertip input type, and a regular operation device.
QR Code Image Refinement by Deep Learning

This study examines a method of refining a low-resolution QR code image photographed from a distance into a decodable code image by using pix2pix, a deep learning technique. The experimental results confirm that the method can decode a 5.2 cm-square QR code taken from a distance of 4 m.
Sentiment Analysis of Japanese Tweets Using Auto-Augmented Sentiment Polarity Dictionaries and Advanced Word Embedding

Social media posts offer a potent source for extracting public sentiment. Hence, a wide spectrum of methods for automatic sentiment analysis of social media posts – from classical lexicon-based approaches to the more recent and currently trendy machine-learning-based approaches – has emerged. The main drawback of machine-learning-based sentiment analysis is that it requires a huge amount of annotated data, while the main challenge associated with lexicon-based approaches is data sparsity – as social media posts often include out-of-lexicon informal words. In order to minimize these challenges, in this paper, we propose a hybrid approach for sentiment analysis of Japanese tweets by automatically augmenting standard sentiment polarity dictionaries and leveraging state-of-the-art fastText word representation and text classification library. Our experimental results confirmed that such hybrid approaches can improve accuracy of sentiment analysis.
Signal Shaping for Semantic Communication Systems with A Few Message Candidates

Semantic communications target to reliably convey the semantic meaning of messages. It is different from existing communication systems focusing on reliable bit transmission. To achieve the goal of semantic communications, we propose a signal shaping method by minimizing the semantic loss, which is measured by the pretrained bidirectional encoder representation from transformers (BERT) model. The signal set optimization problem for semantic communication systems with a few message candidates is investigated. We propose an efficient projected gradient descent method to solve the problem and prove its convergence. Simulation results show that the proposed method outperforms existing signal shaping methods in minimizing the semantic loss.
Deep Joint Source-Channel Coding for Wireless Image Transmission with Semantic Importance

The sixth-generation mobile communication system proposes the vision of smart interconnection of everything, which requires accomplishing communication tasks while ensuring the performance of intelligent tasks. A joint source-channel coding method based on semantic importance is proposed, which aims at preserving semantic information during wireless image transmission and thereby boosting the performance of intelligent tasks for images at the receiver. Specifically, we first propose semantic importance weight calculation method, which is based on the gradient of intelligent task’s perception results with respect to the features. Then, we design the semantic loss function in the way of using semantic weights to weight the features. Finally, we train the deep joint source-channel coding network using the semantic loss function. Experiment results demonstrate that the proposed method achieves up to 57.7% and 9.1% improvement in terms of intelligent task’s performance compared with the source-channel separation coding method and the deep source-channel joint coding method without considering semantics at the same compression rate and signal-to-noise ratio, respectively.
Few-Shot Malware Traffic Classification Method Using Network Traffic and Meta Transfer Learning

Malware traffic classification (MTC) is a very important component of cyber security, and a number of the MTC techniques are based on deep learning (DL) with a strong capability of feature mining and classification. However, these DL-based MTC methods are heavily dependent on a large amount of network traffic samples. In the few-shot scenarios, these methods usually overfit and have poor classification performance. Considering that the update cycle of malware is faster and faster, and there are more and more types of malware, collecting enough training samples for all malware is very challenging, if not impossible. In this paper, a novel few-shot MTC(FS-MTC) method is proposed based on convolutional neural network (CNN) and model-agnostic meta-learning (MAML) algorithm. Specifically, the CNN is trained on samples from normal softwares by MAML rather than the conventional optimization methods, then the CNN is finetuned by a few samples from malware for MTC. Simulation results show that our proposed MAML-based FS-MTC can outperform the traditional MTC methods. The performance of our proposed method can reach up to 95.69%.
SemBAT: Physical Layer Black-box Adversarial Attacks for Deep Learning-based Semantic Communication Systems

Deep learning-based semantic communications (DLSC) replace the physical blocks in traditional communication systems as end-to-end neural networks. DLSC significantly boost communication efficiency by only transmitting the meaning of data, showing great potentials for applications like automatic driving, digital twin and smart health. However, DLSC are fragile to black-box adversarial attacks due to the openness of wireless channel and sensitivities of neural models. To this end, this paper proposes SemBAT, a novel approach for crafting physical layer black-box adversarial attacks for semantic communication systems. The key ingredients of our method include the training of surrogate encoder and generation of adversarial perturbations. Specifically, we train our surrogate encoder by directly estimating the gradients based on Jacobian-matrixs, and then generate the adversarial perturbations by the particle swarm optimizations. Extensive experiments on a public benchmark show the effectiveness of our proposed SemBAT. We observe that our SemBAT with black-box adversaries can sharply decrease the classification accuracy of the semantic communication system from 78.4% to 11.6%. Meanwhile, such attacks are also imperceptible in terms of image quality metrics measured by the Structural similarity index measure (SSIM) and Peak Signal to Noise Ratio(PSNR).
A Novel Semi-Supervised Learning Framework for Specific Emitter Identification

Specific emitter identification (SEI) is developed as a potential technology against attackers in cognitive radio networks and authenticate devices in Internet of Things (IoT). It refers to a process to discriminate individual emitters from each other by analyzing extracted characteristics from given radio signals. Due to the strong capability of deep learning (DL) in extracting the hidden features of data and making classification decision, deep neural networks (DNNs) have been widely used in the SEI. Considering the insufficiently labeled training dataset and large unlabeled training dataset, we propose a novel SEI method using semi-supervised (SS) learning framework, i.e., metric-adversarial training (MAT). Specifically, two object functions (i.e., cross-entropy (CE) loss combined with deep metric learning (DML) and CE loss combined with virtual adversarial training (VAT)) and an alternating optimization way are designed to extract discriminative and generalized semantic features of radio signals. The proposed MAT-based SS-SEI method is evaluated on an open source large-scale real-world automatic-dependent surveillance-broadcast (ADS-B) dataset. The simulation results show that the proposed method achieves a better identification performance than four latest SS-SEI methods.
A Novel Radio Frequency Fingerprint Identification Method Using Incremental Learning

Radio frequency fingerprint (RFF) is regarded as a key technology in physical layer security in various wireless communications systems. Deep learning (DL) has achieved great success in the field of signal identification, particularly in improving performance and eliminating manual feature extraction. However, the training cost of these DL-based methods is usually large. It is unwise to retrain the network with whole data when it comes to new data. Therefore, we propose a novel RFF identification method based on incremental learning (IL), which uses continuous data stream to update the identification model, constantly. Experimental results show that with the increase of increment times, the accuracy of the proposed IL-based method gradually approaches the performance of joint training, and finally reaches 96.79%, which is only 1.9% lower than the performance upper bound.
Interactive Training System for Electric Guitar Strumming Form by Using Inertial Motion Sensors

A 3-D CG-based graphical interaction system for electric guitar-playing lessons has been investigated by using a wrist-worn motion-sensing device. Assuming a virtual camera always looks straight to the guitar player’s strumming hand, stable and intuitive 3-D CG graphics expressing the guitar playing form can be obtained onto a laptop PC. The camera angle from the head of the guitar towards the strumming hand may give suggestive information on the correct playing form taught by a professional guitarist. A 14-week-long population test with 12 amateur players showed that a large elbow-swinging angle amplitude is crucial to achieve sharp chord strumming. A strumming form evaluation utilizing a template matching calculation with the elbow-swinging and the wrist-twisting angles showed how the skill development had been done.
A Robust Few-Shot SEI Method Using Class-Reconstruction and Adversarial Training

Specific emitter identification (SEI) is a promising physical layer authentication technique based on unintentionally hardware impairments of transmitters. These impairments are independent of the data’s content, so they are difficult to forge and analyze. Recently, most deep learning (DL) based SEI methods have been proposed, and have shown their great performance. However, these methods are big data-driven which means they have poor performance with limited training samples, and the vulnerability of neural networks to adversarial attacks is also a problem worth considering. In this paper, we propose an innovative few-shot SEI method based on class-reconstruction classification network and adversarial training (CRCN-AT) without the support of auxiliary dataset. Simulation results show that the proposed method achieves better identification performance and robustness in few-shot scenarios compared to traditional methods. The Pytorch code is released at https://github.comLIUC-000/CRCN-AT.
An Effective Radio Frequency Signal Classification Method Based on Multi-Task Learning Mechanism

With the increasing popularity of Internet of things (IoT), the emergence of many IoT devices has led to security vulnerabilities. The classification of wireless signals is very important for secure communications. Most of existing signal classification tasks only focus on single signal classification task, while ignoring the relationship between radio frequency fingerprinting identification (RFFI) and automatic modulation classification (AMC). To solve the multi-task classification problem, this paper designs a multi-task learning convolutional neural networks (MTL-CNN). Real-radio datasets are generated by Signal Hound VSG60A and collected by Signal Hound BB60C to solve the lack of RFF samples with numerous modulation types. Experimental results confirm that the MTL-CNN method can work well by using the generated dataset. The MTL network designed in this paper improves the accuracy of RFFI by 1xs% relative to the single-task learning (STL) network. The keras code is released at https://github.comLiuK1288/1hw-000.
Privacy-Preserving Image Classification Using ConvMixer with Adaptive Permutation Matrix

In this paper, we propose a privacy-preserving image classification method using encrypted images under the use of the ConvMixer structure. Block-wise scrambled images, which are robust enough against various attacks, have been used for privacy-preserving image classification tasks, but the combined use of a classification network and an adaptation network is needed to reduce the influence of image encryption. However, images with a large size cannot be applied to the conventional method with an adaptation network because the adaptation network has so many parameters. Accordingly, we propose a novel method, which allows us not only to apply block-wise scrambled images to ConvMixer for both training and testing without the adaptation network, but also to provide a higher classification accuracy than conventional methods.
English Language Learning via YouTube: An NLP-Based Analysis of Users’ Comments

Online teaching and learning has been beneficial in facilitating learning of English as a foreign language (EFL). In online EFL learning, YouTube is one of the most utilized information and communication technology (ICT) tools because of its inherent features that make it a unique environment for learners and educators. Many interesting aspects of YouTube-based learning can be beneficial in supplementing conventional classroom methods, and, therefore, such aspects must be identified. Previous scholarly work aimed at improving YouTube learning environment was predominantly conducted manually by gathering learners’ impressions through interviews and questionnaires to analyze the differences between YouTube- and classroom-based EFL learning. However, such methods are tedious and time-consuming and can lead to results that are of less generalizable implications. User comments on YouTube channels are useful in identifying such aspects, as they present a wealth of information related to the quality of the content provided, challenges the targeted audience faces, and areas of potential improvement. Therefore, in our current study, YouTube API is used to collect the comments of three randomly selected and popular YouTube channels. Following a data cleaning process, people’s sentiments about EFL learning were first identified via a TextBlob method. Second, the automated latent semantic analysis (LSA) method of topic finding was used to collect global and open-ended topics of discussion on YouTube-based EFL learning. Users’ sentiments on the most popular topics of discussion are discussed in this paper. Further, based on the results, hypothetical findings on YouTube EFL learning are provided as recommendation for future content, including more variety of the content covered, introduction of the meanings and punctuation following words, the design of the course such that it addresses a multinational audience of any age, and targeted teaching of each variety of English, such as British and American. We also make suggestions for learners of English who wish to utilize online and offline learning, which include finding the course of interest first based on one’s needs which can be discussed with a tutor or any English teacher to optimize the learning experience, participating in fearless educator–learner interaction and engagement, and asking other EFL learners for their previous experiences with learning online in order for the learner to maximize benefit.
Trainable windows for SincNet architecture

Abstract SincNet architecture has shown significant benefits over traditional Convolutional Neural Networks (CNN), especially for speaker recognition applications. SincNet comprises parameterized Sinc functions as filters in the first layer followed by convolutional layers. Although SincNet is compact in nature and offers top-level understanding of the features extracted, the effect of window function used in SincNet is not thoroughly addressed yet. Hamming and Hann are popularly used as the default time-localized windows to reduce spectral leakage. Hence, a comprehensive investigation of 28 different windowing functions on SincNet architecture towards speaker recognition task using TIMIT dataset was performed in this work. Additionally, “trainable” window functions were configured with tunable parameters to characterize the performance. The paper benchmarks the effect of the time-localized windowing function in terms of the bandwidth, side-lobe suppression, and spectral leakage for the filter banks employed in the first layer of the SincNet architecture. Trainable Gaussian and Cosine-Sum functions exhibited relative improvement of 41.46% and 82.11% in the sentence level classification error rate over Hamming window when employed on SincNet architecture.
BehaviorNet: A Fine-grained Behavior-aware Network for Dynamic Link Prediction

Dynamic link prediction has become a trending research subject because of its wide applications in web, sociology, transportation, and bioinformatics. Currently, the prevailing approach for dynamic link prediction is based on graph neural networks, in which graph representation learning is the key to perform dynamic link prediction tasks. However, there are still great challenges because the structure of graphs evolves over time. A common approach is to represent a dynamic graph as a collection of discrete snapshots, in which information over a period is aggregated through summation or averaging. This way results in some fine-grained time-related information loss, which further leads to a certain degree of performance degradation. We conjecture that such fine-grained information is vital because it implies specific behavior patterns of nodes and edges in a snapshot. To verify this conjecture, we propose a novel fine-grained behavior-aware network (BehaviorNet) for dynamic network link prediction. Specifically, BehaviorNet adapts a transformer-based graph convolution network to capture the latent structural representations of nodes by adding edge behaviors as an additional attribute of edges. GRU is applied to learn the temporal features of given snapshots of a dynamic network by utilizing node behaviors as auxiliary information. Extensive experiments are conducted on several real-world dynamic graph datasets, and the results show significant performance gains for BehaviorNet over several state-of-the-art (SOTA) discrete dynamic link prediction baselines. Ablation study validates the effectiveness of modeling fine-grained edge and node behaviors.
Enhancing Conversational Recommendation Systems with Representation Fusion

Conversational Recommendation Systems (CRSs) aim to improve recommendation performance by utilizing information from a conversation session. A CRS first constructs questions and then asks users for their feedback in each conversation session to refine better recommendation lists to users. The key design of CRS is to construct proper questions and obtain users’ feedback in response to these questions so as to effectively capture user preferences. Many CRS works have been proposed; however, they suffer from defects when constructing questions for users to answer: (1) employing a dialogue policy agent for constructing questions is one of the most common choices in CRS, but it needs to be trained with a huge corpus, and (2) it is not appropriate that constructing questions from a single policy (e.g., a CRS only selects attributes that the user has interacted with) for all users with different preferences. To address these defects, we propose a novel CRS model, namely a Representation Fusion–based Conversational Recommendation model, where the whole conversation session is divided into two subsessions (i.e., Local Question Search subsession and Global Question Search subsession) and two different question search methods are proposed to construct questions in the corresponding subsessions without employing policy agents. In particular, in the Local Question Search subsession we adopt a novel graph mining method to find questions, where the paths in the graph between users and attributes can eliminate irrelevant attributes; in the Global Question Search subsession we propose to initialize user preference on items with the user and all item historical rating records and construct questions based on user’s preference. Then, we update the embeddings independently over the two subsessions according to user’s feedback and fuse the final embeddings from the two subsessions for the recommendation. Experiments on three real-world recommendation datasets demonstrate that our proposed method outperforms five state-of-the-art baselines.