Reviewerly/exHarmony huggingface dataset:
Refinement of Utterance Fluency Feature Extraction and Automated Scoring of L2 Oral Fluency with Dialogic Features

We propose an automated scoring method of fluency that is compatible with second language dialogic responses. Human judgements of L2 oral fluency in dialogue tasks has different nature from scoring of monologue, and it is necessary to capture a dialogic aspect of fluency. Because utterances in dialogue tend to be less fluent than in monologue, procedures such as pruning disfluency words and classifying pauses by their syntactic locations are essential for automated scoring systems to extract utterance features strongly correlated to human ratings. However, existing automated utterance feature extractors have suffered from the difficulties to detect disfluency words and pauses locations due to the technical challenges. Moreover, conventional automated scoring methods of L2 spoken dialogue often predict oral proficiency for each turn, and dialogic features have not been considered properly. To address these gaps between the nature of fluency in dialogue and existing automated scoring methodologies, we refine an automated utterance feature extractor and design a fluency scoring model based on dialogic features. Experiments showed that the substantial agreement of disfluency word and pause location detection between our feature extractor and human (Cohen'sκ > 0.61). We also found that the proposed scoring method outperformed in predicting subjective fluency scores (QW κ = 0.833) than a conventional turn-level scoring model (0.654) and even a manual rating (0.799). We additionally compared the current assessment approach considering disfluency features and pause location, and it improved the accuracy on predicting subjective fluency scores. These results may suggest what and how utterance and dialogic features should be utilized in automated scoring of spoken dialogue.
A Policy-based Approach to the SpecAugment Method for Low Resource E2E ASR

SpecAugment is a very effective data augmentation method for both HMM and E2E-based automatic speech recognition (ASR) systems. Especially, it also works in low-resource scenarios. However, SpecAugment masks the spectrum of time or the frequency domain in a fixed augmentation policy, which may bring relatively less data diversity to the low-resource ASR. In this paper, we propose a policy-based SpecAugment (Policy-SpecAugment) method to alleviate the above problem. The idea is to use the augmentation-select policy and the augmentation-parameter changing policy to solve the fixed way. These policies are learned based on the loss of validation set, which is applied to the corresponding augmentation policies. It aims to encourage the model to learn more diverse data, which the model relatively requires. In experiments, we evaluate the effectiveness of our approach in low-resource scenarios, i.e., the 100 hours librispeech task. According to the results and analysis, we can see that the above issue can be obviously alleviated using our proposal. In addition, the experimental results show that, compared with the state-of-the-art SpecAugment, the proposed Policy-SpecAugment has a relative WER reduction of more than 10% on the Test/Dev-clean set, more than 5% on the Test/Dev-other set, and an absolute WER reduction of more than 1% on all test sets.
Image Watermarking based on Saliency Detection and Multiple Transformations

An ideal image watermarking (IW) scheme aims to manage the trade-off among quality, capacity, and robustness. However, our literature survey reveals some flaws in the form of poor robustness and quality or low embedding capability. In this paper, multiple frequency domain based image watermarking scheme using salient (eye-catching) object detection is applied. Specifically, the host and the watermark images are partitioned into background and foreground regions by the proposed multi-dimension decomposition, which accumulates image patches and combining them to form the salient map. Next, the watermark image is encrypted by multiple applications of the 3D Arnold and logistic maps, then embedded into both the identified foreground and background regions of the host image by using different embedding strengths. The proposed method can embed 1 color pixel of the watermark image into 1 color pixel in the host image while maintaining high image quality. In the best case scenario, we could embed a 24-bit image as the watermark into a 24-bit image of the same dimension while maintaining an average RGB-SSIM of 0.9999. Experiments are carried out (with 10K MSRA dataset images) to verify the performance of the proposed method and to compare our proposed method against the state-of-the-art (SOTA) watermarking methods.
A Proposal for Emotion-Expressive Editor:EmoEditor by Font Changing

Text media is one of important ways in communications on computers. For example, email, LINE or Twitter uses it frequently. In addition, captions in motion pictures are necessity for not only watching foreign cinema/TV but also one who is older or has hearing impairment. In general, a text character is recognized as rendered images on display. Therefore, it is difficult for text to express the emotions we have in writing sentences. In this paper, we propose a method to communicate emotions by changing font image. Firstly, we link a specific font and an emotion, then pair of a font data connected to an emotion and a standard font that is stored in computers is input to deep neural network (DNN) based on GAN, and we train it. An emotion is given to the neural network as a class label data. The input of the DNN is rendered font data. In other words, for the DNN, the input data is a standard font data we usually have with rendered, the output data is a generated rendered font image with emotion reflected. Furthermore, a mixed emotion font is able to be generated by adjusting parameters. We apply this neural network to editor system which is enable us to express emotions, which is called EmoEditor (Emotion-Expressive Editor). Through evaluation experiments, we confirmed the feasibility to communicate an emotion via an electronic media.
Parameterization of Dominant Spectral Peak Trajectory for Whisper Speech Recognition

Automatic speech recognition (ASR) systems trained on normal speech generally suffer from performance degradations for whisper speech. To solve this problem, this paper concentrates on utilizing similar factors between normal and whisper speech to construct a whisper speech recognizer with normal speech data. We propose to parameterize the dominant spectral peak trajectory (Ppeak) to capture the similarities and concatenate it to the traditional Mel-Frequency Cepstral Coefficients (MFCC) and Human Factor Cepstral Coefficients (HFCC), respectively, to form new features. The proposed features benefit to the accuracy of whisper speech recognition. Performance improvement can be further achieved when the similarity is enhanced by removing low frequency information. Experimental results show that the performance degradation between match and mismatch scenarios was reduced relatively by 90.31% in Word Error Rate (WER) for HFCC after similarity enhancement at a cut-off frequency of 500Hz. Furthermore, we ultimately achieved a relative reduction of 69.60% in WER in the mismatch scenario compared with conventional MFCC even without whisper speech data for training.
Needle Localization and Segmentation for Radiofrequency Ablation of Liver Tumors under CT Image Guidance

Radiofrequency ablation (RFA) of liver cancer under computer tomography (CT) guidance is a minimally invasive procedure in which CT images are utilized to guide the physician in introducing the needle into the target lesion. However, the adequate visualization of the needle and anatomy is hampered by the 2D slide based-view used in the current clinical practice. Thus, due to the lack of 3D information, the physician requires high experience and more interaction with the guidance systems to envision the needle's position in the liver, which is inconvenient in clinical practice. In this study, we propose a method for robust needle segmentation using CT images to improve the visualization of the needle during the intervention. The method utilizes a convolutional neural network (CNN) to detect the needle in orthogonal 2D projections of the CT image to construct the needle volume of interest (VOI). Subsequently, a patch-based 3D CNN is applied to segment the needle. We evaluate the method's accuracy using Dice score (DSC), Hausdorff distance (HD), the needle shaft error Eshaft, and needle tip error E <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">tip</inf> .. The results show that the proposed method achieves the means of DSC, HD, E <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">tip</inf> , Eshaft and processing time of 0.89, 3.3 mm, 0.9 mm, 0.43 mm, and 2.6 seconds, respectively. We conclude that the proposed method is feasible for improving needle visualization in the interventional room.
Few-Shot Classification With Feature Reconstruction Bias

Few-shot classification aims to classify unseen samples by learning from very few labeled samples. Very recently, reconstruction-based methods have been proposed and shown superior performance on few-shot fine-grained image classification, which, on top of the challenge of few labeled samples, faces the difficulty of identifying subtle differences between sub-categories. In essence, these methods reconstruct unseen samples from few seen samples and use the distance between the original unseen samples and their reconstruction as the criterion for classification. However, as pointed out in this paper, a bias is introduced in the overall distribution between the reconstructed features and original features, which consequently affects the distance calculation and subsequent classification. To address this issue, we propose a new concept of Feature Reconstruction Bias (FRB), which can be computed easily in the training stage without introducing any new parameters. Moreover, we propose to use this bias to correct query features in the test stage, which is shown to increase inter-class distances and decrease intra-class distances. Experiments on four fine-grained benchmarks demonstrate the effectiveness of our approach, with state-of-the-art performance achieved in most scenarios.
Table Structure Recognition Based on Grid Shape Graph

Since tables in documents provide important information in compact form, table understanding has been an essential topic in document image processing. Researchers represented table structures in various formats for table understanding, such as simple grid structure, a graph with text/cell boxes as nodes, or a sequence of HTML tokens. However, these approaches have difficulties in handling regularities, e.g., global row and column information, and spanning cells simultaneously. In this paper, we propose a new table recognition method based on a grid shape graph and present grid localization and grid elements grouping networks. This approach is designed to exploit the grid structure and deal with spanning cells. To convert grid structure into cell structure, we only have to test adjacent pairs of grid elements, enabling efficient inference. In addition, we have discovered that predicting row/column-based relationships between grid elements improve cell-based connectivity estimation performance. We demonstrate the effectiveness of the proposed method through experiments on three benchmark datasets.
Gzippo

Graph application plays a significant role in real-world data computation. However, the memory access patterns become the performance bottleneck of the graph applications, which include low compute-to-communication ratio, poor temporal locality, and poor spatial locality. Existing RRAM-based processing-in-memory accelerators reduce the data movements but fail to address both sparsity and redundancy of graph data. In this work, we present Gzippo, a highly-compact design that supports graph computation in the compressed sparse format. Gzippo employs a tandem-isomorphic-crossbar architecture both to eliminate redundant searches and sequential indexing during iterations, and to remove sparsity leading to non-effective computation on zero values. Gzippo achieves a 3.0× (up to 17.4×) performance speedup, 23.9× (up to 163.2×) energy efficiency over state-of-the-art RRAM-based PIM accelerator, respectively.
DaS

Ising machines have generated much excitement in recent years due to their promise for solving hard combinatorial optimization problems. However, achieving physical all-to-all connectivity in IC implementations of large, densely-connected Ising machines remains a key challenge. We present a novel approach, DaS, that uses low-rank decomposition to achieve effectively-dense Ising connectivity using only sparsely interconnected hardware. The innovation consists of two components. First, we use the SVD to find a low-rank approximation of the Ising coupling matrix while maintaining very high accuracy. This decomposition requires substantially fewer nonzeros to represent the dense Ising coupling matrix. Second, we develop a method to translate the low-rank decomposition to a hardware implementation that uses only sparse resistive interconnections. We validate DaS on the MU-MIMO detection problem, important in modern telecommunications. Our results indicate that as problem sizes scale, DaS can achieve dense Ising coupling using only 5%-20% of the resistors needed for brute-force dense connections (which would be physically infeasible in ICs). We also outline a crossbar-style physical layout scheme for realizing sparse resistive networks generated by DaS.
A Robust Quantum Layout Synthesis Algorithm with a Qubit Mapping Checker

Layout synthesis in quantum circuits maps the logical qubits of a synthesized circuit onto the physical qubits of a hardware device (coupling graph) and complies with the hardware limitations. Existing studies on the problem usually suffer from intractable formulation complexity and thus prohibitively long runtimes. In this paper, we propose an efficient layout synthesizer by developing a satisfiability modulo theories (SMT)-based qubit mapping checker. The proposed qubit mapping checker can efficiently derive a SWAP-free solution if one exists. If no SWAP-free solution exists for a circuit, we propose a divide-and-conquer scheme that utilizes the checker to find SWAP-free sub-solutions for sub-circuits, and the overall solution is found by merging sub-solutions with SWAP insertion. Experimental results show that the proposed optimization flow can achieve more than 3000× runtime speedup over a state-of-the-art work to derive optimal solutions for a set of SWAP-free circuits. Moreover, for the other set of benchmark circuits requiring SWAP gates, our flow achieves more than 800× speedup and obtains near-optimal solutions with only 3% SWAP overhead.
Graph Neural Networks for Idling Error Mitigation

Dynamical Decoupling (DD)-based protocols have been shown to reduce the idling errors encountered in quantum circuits. However, the current research in suppressing idling qubit errors suffers from scalability issues due to the large number of tuning quantum circuits that should be executed first to find the locations of the DD sequences in the target quantum circuit, which boost the output state fidelity. This process becomes tedious as the size of the quantum circuit increases. To address this challenge, we propose a Graph Neural Network (GNN) framework, which mitigates idling errors through an efficient insertion of DD sequences into quantum circuits by modeling their impact at different idle qubit windows. Our paper targets maximizing the benefit of DD sequences using a limited number of tuning circuits. We propose to classify the idle qubit windows into critical and non-critical (benign) windows using a data-driven reliability model. Our results obtained from IBM Lagos quantum computer show that our proposed GNN models, which determine the locations of DD sequences in the quantum circuits, significantly improve the output state fidelity by a factor of 1.4x on average and up to 2.6x compared to the adaptive DD approach, which searches for the best locations of DD sequences at run-time.
Towards High Performance and Accurate BNN Inference on FPGA with Structured Fine-Grained Pruning

As the extreme case of quantization networks, Binary Neural Networks (BNNs) have received tremendous attention due to many hardware-friendly properties in terms of storage and computation. To reach the limit of compact models, we attempt to combine binarization with pruning techniques, further exploring the redundancy of BNNs. However, coarse-grained pruning methods may cause server accuracy drops, while traditional fine-grained ones induce irregular sparsity hard to be utilized by hardware. In this paper, we propose two advanced fine-grained BNN pruning modules, i.e., structured channel-wise kernel pruning and dynamic spatial pruning, from a joint perspective of algorithm and hardware. The pruned BNN models are trained from scratch and present not only a higher precision but also a high degree of parallelism. Then, we develop an accelerator architecture that can effectively exploit the sparsity caused by our algorithm. Finally, we implement the pruned BNN models on an embedded FPGA (Ultra96v2). The results show that our software and hardware codesign achieves 5.4x inference-speedup than the baseline BNN, with higher resource and energy efficiency compared with prior FPGA implemented BNN works.
QuBRIM

Physical Ising machines have been shown to solve combinatoric optimization problems with orders-of-magnitude improvements in speed and energy efficiency o ver v on N eumann systems. However, building such a system is still in its infancy and a scalable, robust implementation remains challenging. CMOS-compatible electronic Ising machines (e.g., [1]) are promising as the mature technology helps bring scale, speed, and energy efficiency to the dynamical system. However, subtle issues can arise when using voltage-controlled transistors to act as programmable resistive coupling. In this paper, we propose a version of resistively-coupled Ising machine using quantized nodal interactions (QuBRIM), which significantly i mproved the predictability of the coupling resistor. The functionality of QuBRIM is demonstrated by solving the well-known Max-Cut problem using both behavioral and circuit level simulations in 45 nm CMOS technology node. We show that the dynamical system naturally seeks local minima in the objective function's energy landscape and that by applying spin-fix a nnealing, t he system reaches a global minimum with a high probability.
ObfuNAS

Malicious architecture extraction has been emerging as a crucial concern for deep neural network (DNN) security. As a defense, architecture obfuscation is proposed to remap the victim DNN to a different architecture. Nonetheless, we observe that, with only extracting an obfuscated DNN architecture, the adversary can still retrain a substitute model with high performance (e.g., accuracy), rendering the obfuscation techniques ineffective. To mitigate this under-explored vulnerability, we propose ObfuNAS, which converts the DNN architecture obfuscation into a neural architecture search (NAS) problem. Using a combination of function-preserving obfuscation strategies, ObfuNAS ensures that the obfuscated DNN architecture can only achieve lower accuracy than the victim. We validate the performance of ObfuNAS with open-source architecture datasets like NAS-Bench-101 and NAS-Bench-301. The experimental results demonstrate that ObfuNAS can successfully find the optimal mask for a victim model within a given FLOPs constraint, leading up to 2.6% inference accuracy degradation for attackers with only 0.14× FLOPs overhead. The code is available at: https://github.com/Tongzhou0101/ObfuNAS.
Combining Gradients and Probabilities for Heterogeneous Approximation of Neural Networks

This work explores the search for heterogeneous approximate multiplier configurations for neural networks that produce high accuracy and low energy consumption. We discuss the validity of additive Gaussian noise added to accurate neural network computations as a surrogate model for behavioral simulation of approximate multipliers. The continuous and differentiable properties of the solution space spanned by the additive Gaussian noise model are used as a heuristic that generates meaningful estimates of layer robustness without the need for combinatorial optimization techniques. Instead, the amount of noise injected into the accurate computations is learned during network training using backpropagation. A probabilistic model of the multiplier error is presented to bridge the gap between the domains; the model estimates the standard deviation of the approximate multiplier error, connecting solutions in the additive Gaussian noise space to actual hardware instances. Our experiments show that the combination of heterogeneous approximation and neural network retraining reduces the energy consumption for multiplications by 70% to 79% for different ResNet variants on the CIFAR-10 dataset with a Top-1 accuracy loss below one percentage point. For the more complex Tiny ImageNet task, our VGG16 model achieves a 53 % reduction in energy consumption with a drop in Top-5 accuracy of 0.5 percentage points. We further demonstrate that our error model can predict the parameters of an approximate multiplier in the context of the commonly used additive Gaussian noise (AGN) model with high accuracy. Our software implementation is available under https://github.com/etrommer/agn-approx.
All-in-One

During the deployment of deep neural networks (DNNs) on edge devices, many research efforts are devoted to the limited hardware resource. However, little attention is paid to the influence of dynamic power management. As edge devices typically only have a budget of energy with batteries (rather than almost unlimited energy support on servers or workstations), their dynamic power management often changes the execution frequency as in the widely-used dynamic voltage and frequency scaling (DVFS) technique. This leads to highly unstable inference speed performance, especially for computation-intensive DNN models, which can harm user experience and waste hardware resources. We firstly identify this problem and then propose All-in-One, a highly representative pruning framework to work with dynamic power management using DVFS. The framework can use only one set of model weights and soft masks (together with other auxiliary parameters of negligible storage) to represent multiple models of various pruning ratios. By re-configuring the model to the corresponding pruning ratio for a specific execution frequency (and voltage), we are able to achieve stable inference speed, i.e., keeping the difference in speed performance under various execution frequencies as small as possible. Our experiments demonstrate that our method not only achieves high accuracy for multiple models of different pruning ratios, but also reduces their variance of inference latency for various frequencies, with minimal memory consumption of only one model and one soft mask.