Reviewerly/exHarmony huggingface dataset:
Audio Synthesis Translation and Auto-Summarization (ASTA)

Availability of time has been a major issue in recent years for mankind. There has always been a huge demand for automation, since it can tremendously decrease time for doing menial tasks. This proposed project focuses on automation of text translation, summarization and speech synthesis which could reduce time required for reading books. In this paper, we present multiple machine learning models that synthesize text into speech and also into summarized text of Devanagari script. The main objective of the project is to conduct proper examination of the existing architecture of the text translation and summarization methodologies and to provide a robust system which is a cumulation of converting PDF files to audio files and also summarization of the PDF and translating into Devanagari text of the summarized English narrative. The architecture is called Audio Synthesis Translation and Auto-summarization (ASTA) and uses multiple models such as RNN sequence to sequence, NMT, Tacotron 2 and Waveglow. In addition to this, we use Google Vision OCR for text extraction from PDF. This system is an integration of multiple machine learning models and works as a pipelined system.
Efficient CFI Enforcement for Embedded Systems Using ARM TrustZone-M

Embedded systems are deployed in many fields, from industrial applications to personal products.However, there are growing concerns regarding the security of these embedded systems as the number of attacks targeting them has increased.Control flow integrity (CFI) is a well-known security solution against these attacks.However, according to our analysis, existing CFI methods cannot be widely used in embedded systems one or more of the following reasons.(1) They require special hardware features that are not available in embedded systems, (2) they require that the developer recompile the source code with their compiler toolchain and (3) they incur considerable performance overhead to ensure CFI at runtime.In this paper, we propose CEST, a new scheme to ensure CFI on embedded systems using ARM TrustZone-M, a security extension for embedded ARM processors.For better compatibility, we designed CEST to be binary compatible.The evaluation results show that CEST can effectively enforce CFI compared to the existing studies using SVC.
HoloParser: Holistic Visual Parsing for Real-Time Semantic Segmentation in Autonomous Driving

Existing real-time semantic segmentation models usually pursue lower inference costs by compressing spatial resolution, reducing the number of feature extraction layers, or simplifying the high-resolution generation process. This inevitably leads to huge information loss and ultimately affects segmentation accuracy. To address the problem, a HoloParser is proposed to learn and parse the input image from a holistic perspective of the entire segmentation pipeline. First, in the image preprocessing part, the often-overlooked problem of structural information loss in fast downsampling is highlighted and studied. For this, a nonstrided fast downsampling (NFD) module is devised for holo-structure retention. Second, for the resolution change processes in segmentation networks, the spatial correlation between up/down-samplings is revisited and focused. And a lossless sample pair (LSP) is proposed for holo-correlation modeling and retrieving. Third, in feature extraction, after reconsidering the necessity and redundancy of multiscale learning, a distributed pyramid learning (DPL) module is proposed for lightweight holo-feature extraction with composite receptive fields (RFs). Finally, for the high-resolution output generation, a parallel resolution maintenance (PRM) module is put forward via high-low semantics aggregation across stages for holo-semantics utilization. Extensive experiments demonstrate the effectiveness and superiority of the proposed modules. The HoloParser achieves state-of-the-art performance in real-time semantic segmentation, which reports 76.45% mIoU at 280+ frames/s on the Cityscapes dataset.
Semi-Supervised Online Speaker Diarization using Vector Quantization with Alternative Codebooks

Speaker diarization systems process audio files by labelling speech segments according to speakers' identities. Many speaker diarization systems work offline and are not suited for online applications. We present a semi-supervised, online, low-complexity system. While, in general, speaker diarization operates in an unsupervised manner, the presented system relies on the enrollment of the participating speakers in the conversation. The diarization system has two main novel aspects. The first one is a proposed online learning strategy that evaluates processed segments according to their usefulness for learning a speaker, i.e. update a speaker model with it. The segment is evaluated using two metrics to determine whether to use the segment to update the system. The second novel aspect is a proposed vector quantization approach that models the score not only depending on the target speaker codebook but also takes an alternative codebook into account. We also present an approach to compute the alternative codebook. Simulation results show that the proposed system outperforms a comparable system without the proposed online learning strategy and shows benefits, especially for short training lengths.
Sequential Image-based 3D Object Detection with Location Refinement

Recent advances in object detection tasks enable the detection network to predict 3D objects from a monocular image, but the performance of monocular 3D object detectors is inferior due to the depth information lost in the image. Most monocular 3D detectors do not utilize sequential information from multi-frame images, even though the object’s temporal motion is very informative for 3D object detection. In this paper, we propose a sequential image-based 3D object detection architecture that focuses on improving the localization performance of 3D detectors using temporal information for autonomous driving applications. To this end, the proposed network is trained with a pair of sequential images to predict 3D objects with their localization uncertainties on each image. Afterward, the object detected from sequential images is associated, and paired object features are fed to the sub-network to predict the depth displacement between frames. Finally, paired objects and their predicted depths and depth displacement are refined to minimize residuals between predictions and output the final 3D location of objects. The experimental results on challenging the nuScenes dataset demonstrate that our method improves the performance of the 3D detector by reducing the localization error.
Individual Processing of Phishing Emails

There is a prevailing prejudice that technology can solve all problems in many fields, including cybercrime. Still, recent reports of increasing data breaches have shown that this belief is not always true. This paper investigated social engineering scenarios, particularly phishing attacks, to analyze the psychological deception schemes used by attackers alongside the heuristics that affect users' vulnerability. Indeed, the authors explain how hackers use various technical tools besides certain psychological factors to design clever and successful attacks against businesses or individuals. This research provides a decision-making framework for e-mail processing; it consists of several verification stages covering cognitive and technical factors that help users identify inconsistencies and different classes of phishing. Furthermore, it supports the security awareness field with a reliable framework that has demonstrated promising results and low false positives. The solution aims to reduce phishing threats and help organizations establish security-conscious behavior among their employees.
Cognitive System of a Virtual Robot Based on Perception, Memory, and Hypothesis Models for Calligraphy Writing Task

In this paper, we propose a robotic cognitive system which can learn itself to do a specific assignment by accumulating experiences through bottom-up thinking to make decision by itself via top-down thinking according to the experiences. That is, the cognitive system has a self-learning ability which can accumulate its experiences to make itself smarter. In essence, the cognitive system possesses a perception model, a memory model, and a hypothesis model. The perception model converts image information into perception codes. The memory model stores experiences in the past and present to provide to the perception model and the hypothesis model. The hypothesis model, which generates the next decision according to the experiences from the memory model, is the most important part of the proposed cognitive system. To validate the performance of the proposed system, we utilize Chinese calligraphy writing tasks by a virtual robot through simulation to evaluate the abilities of the cognitive system. In order to generate the coordinates of the writing brush, we made the virtual robot practice to learn Chinese calligraphy through bottom-up thinking to construct the writing patterns. The illustrative examples in this paper show that the virtual robot can learn to write Chinese calligraphy by top-down thinking according to its own experiences.
Formal Verification and Implementation of an E-Voting System

The organization of free, democratic, and transparent elections requires on the one hand an independent national electoral authority that manages all the stages of the electoral process and on the other hand the use of new information and communication techniques to manage the election process. E-voting offers the ability to vote online anytime and from anywhere using a computer, smartphone, or tablet. In addition, it saves time and reduces costs and effort spent in the process. However, the security of e-voting applications deployed on the internet is a difficult task due to the increasing number of cyber-attacks and application vulnerabilities. On the other hand, blockchain technology is an emerging technology with a strong cryptographic foundation. In this paper, the authors propose a new secure e-voting system based on Ethereum blockchain. In addition, they propose a hierarchical coloured petri net (HCPN) model for their e-voting system using CPN Tools. They verify by means of simulation techniques and state space analysis important properties such as absence of deadlocks and livelocks.
Mining Maximal Sub-prevalent Co-location Patterns Based on k-hop

Mining spatial sub-prevalent co-location patterns (SPCPs) aim to discover the combinations of spatial features whose star feature instances co-occur frequently in the domain. This can in turn lay the foundation for deeper analysis of other important relationships between spatial objects (e.g., cause-effect relationships). However, no attempt was made to take into account the contribution of the k-hop neighbors. Therefore, we improve the partition-based algorithm in SPCPs mining, and propose a novel hierarchy-based algorithm based on k-hop neighbors (HPBA), which can discover more potentially valuable maximal sub-prevalent co-location patterns. HPBA is optimized by using multiple pruning strategies, for instance, k-core can filter features not present in k-size or higher size patterns. Lastly, we perform extensive experiments on real-world spatial datasets, and the results show the efficiency and effectiveness of our algorithm.
IntelliOntoRec: a knowledge infused semiautomatic approach for ontology formulation in healthcare and medical science

Linked Open Data (LOD) on the semantic structure of the World Wide Web plays a vital role in deciding the effectiveness of semantic inferencing for making application-oriented decisions. However, when working with safety-critical systems and critical domains that require scientific acceptance from a community like healthcare, the modeling of linked open data must require expert opinion. The linked open data for medical and healthcare as a domain of choice should preferably be semiautomatic with a standard scheme for automatically modeling domain specific sensitive knowledge and human interaction for verification and a community acceptance for healthcare. In this chapter, a strategic scheme that amalgamates semantic mapping of ontologies using a triadic system of semantic-similarity, Simpson's diversity index, and a novel eXplainable AI (XAI) algorithm, which imbibes the concept of fluid thrust has been proposed. The IntelliOntoRec integrates a diverse source of knowledge from medical journals, RDFs, semantic wikis and follows a three-phase verification of knowledge modeled wherein the domain experts can view the automatically modeled knowledge and contribute. The second phase is the honest review by the community of domain experts and integrates knowledge based on manual modeling and opinions of domain experts and incorporates the grey wolf metaheuristic algorithm. The third phase is the rereview an commit, where the knowledge modeled will be finalized. Further, as the modeling and curation of knowledge are successfully achieved, a rule-based axiom induction using heuristics of information is achieved to successfully organize the linked open data as hierarchies before posting it to the linked open data cloud. The ontology modeling is evaluated based on the scoring of the intelligent crowd of experts, and an overall Modeling F-Measure of 98.557583 has been achieved by the proposed IntelliOntoRec.
Complex Question Answering Over Temporal Knowledge Graphs

A temporal knowledge graph (TKG) comprises facts aligned with timestamps. Question answering over TKGs (TKGQA) finds an entity or timestamp to answer a question with certain temporal constraints. Current studies assume that the questions are fully annotated before being fed into the system, and treat question answering as a link prediction task. Moreover, the process of choosing answers is not interpretable due to the implicit reasoning in the latent space. In this paper, we propose a semantic parsing based method, namely AE-TQ, which leverages abstract meaning representation (AMR) for understanding complex questions, and produces question-oriented semantic information for explicit and effective temporal reasoning. We evaluate our method on CronQuestions, the largest known TKGQA dataset, and the experiment results demonstrate that AE-TQ empirically outperforms several competing methods in various settings.
Entropy Lost: Nintendo’s Not-So-Random Sequence of 32, 767 Bits

Early video games didn’t have hardware support for random number generation, so developers used software-based RNG. We show that hundreds of games in the Nintendo Famicom / NES library regenerate the same pseudorandom sequence of 32, 767 bits, although the machine code for doing so comes in more than one hundred variants. We identified the disparate implementations using a simple regular expression that matches the following “fingerprint” of operations: LDA, AND, STA, LDA, AND, EOR, CLC, BEQ, SEC, ROR. These instructions implement a classic 15-bit linear feedback shift register associated with x15 + x7 + 1 (i.e., tap the 15th and 7th bits). However, many games devoted more memory to the LFSR’s state. For example, Donkey Kong (1983) used 8 bytes (or 8 · 8 = 64 bits), The Legend of Zelda (1986) used 13 bytes, and Super Mario Bros. 3 (1988) used 9 bytes. In each case, additional entropy is lost by a simple programming error: the bits are shifted in the wrong direction.
Leveraging Semantics in Localization with Probabilistic Manner

Semantics such as object classes is typically denoted by a discrete variable. To probabilistically leverage semantics in localization, the discrete variable must be introduced in the likelihood calculation considering continuous geometric relation without loss of probabilistic manner. This paper proposes a novel localization method that enables to probabilistically handle semantics. The proposed method uses Dirichlet distribution in the likelihood calculation. The hyperparameters of Dirichlet distribution are determined using geometric relation. As a result, semantics can be probabilistically handled and the likelihood can be calculated while coping with uncertainty of object recognition. Experimental results show that semantics can be utilized for improving localization accuracy while preventing performance reduction even when object recognition has been failed.
Fault Resilience Techniques for Flash Memory of DNN Accelerators

Deep neural networks (DNNs) are being widely used in smart appliances, face recognition and autonomous driving. The trained weight data are usually stored in flash memory which suffers from reliability and endurance issues. Owing to the inherent error tolerability for DNN applications, address remapping techniques are proposed for protecting weight data stored in flash memory. Bit significances are first analyzed and then a weight transposer is proposed for remapping significant weight bits to fault-free or much reliable flash cells. A bipartite graph model is developed for modeling address remapping. The corresponding hardware architectures for address remapping are also proposed. We use the deep learning framework pytorch for evaluating inference accuracy for different DNN models. Experimental results show that based on 0.01 % injected BER in the weight data, the accuracy losses of widely used DNN models are less than 1 % with negligible hardware overhead.
MiniFloat-NN and ExSdotp: An ISA Extension and a Modular Open Hardware Unit for Low-Precision Training on RISC-V Cores

Low-precision formats have recently driven major breakthroughs in neural network (NN) training and inference by reducing the memory footprint of the NN models and improving the energy efficiency of the underlying hardware architectures.Narrow integer data types have been vastly investigated for NN inference and have successfully been pushed to the extreme of ternary and binary representations.In contrast, most trainingoriented platforms use at least 16-bit floating-point (FP) formats.Lower-precision data types such as 8-bit FP formats and mixedprecision techniques have only recently been explored in hardware implementations.We present MiniFloat-NN, a RISC-V instruction set architecture extension for low-precision NN training, providing support for two 8-bit and two 16-bit FP formats and expanding operations.The extension includes sum-of-dot-product instructions that accumulate the result in a larger format and three-term additions in two variations: expanding and non-expanding.We implement an ExSdotp unit to efficiently support in hardware both instruction types.The fused nature of the ExSdotp module prevents precision losses generated by the non-associativity of two consecutive FP additions while saving around 30% of the area and critical path compared to a cascade of two expanding fused multiply-add units.We replicate the ExSdotp module in a SIMD wrapper and integrate it into an open-source floatingpoint unit, which, coupled to an open-source RISC-V core, lays the foundation for future scalable architectures targeting lowprecision and mixed-precision NN training.A cluster containing eight extended cores sharing a scratchpad memory, implemented in 12 nm FinFET technology, achieves up to 575 GFLOPS/W when computing FP8-to-FP16 GEMMs at 0.8 V, 1.26 GHz.
Hierarchical Multiple Granularity Attention Network for Long Document Classification

Long document classification has aroused tremendous attention in the field of Nature Language Processing, due to the exponential increasing of publications. Although the common text classification methods can be extended for long document classification, they are confined to the length of text and do not have enough expressiveness to model the structure of the long document. To solve these problems, we proposed a hierarchical multiple granularity attention network for long document classification, in which the word and section level features are extracted and fused to represent the complex structure of the long document. Furthermore, a feature-based section pooling module is adopted to eliminate redundant text information and accelerate the computing. A series of experiments are conducted to evaluate the proposed method. The experimental results verify that our method is effective, efficient and competitive compared with the related state-of-the-art methods.
Facial Expression Recognition via Re-parameterized Feature Fusion Model

Feature fusion is one of the commonly used methods in facial expression recognition, which can improve the utilization of image information by fusing features of different scales in the image so as to obtain a better recognition effect. In this paper, we proposed a deep learning re-parameterized feature fusion model for facial expression recognition. Many recent works investigated the re-parameterization technique which could speed up the training and recognition with low memory cost and a plain VGG-like architecture. And in order to implement the concept that the features of facial key points will contain almost facial expression information, we designed the feature fusion model using origin images and LBP features as two independent branches within the same network block structure. Experiments showed the feature fusion method can bring more advantages in processing uneven data, we obtained a more balanced effect when the data distribution was largely uneven: on pre-processed FER2013 dataset, our model got 74.9% accuracy while recognizing six distinct expressions, better than ResNet-50 and gets 61.6% accuracy on filtered affectNet with seven fuzzy expressions, better than VGG and ResNet-50. Moreover, the new model run faster than the existing network with quite high accuracy, 81% faster than VGG-16, and 142% faster than ResNet-50 on Tesla T4 GPU, which indicated it can be more easily deployed to websites and application software.
Multi-scale Object Detection Algorithm Based on Adaptive Feature Fusion

Aiming at the problem that each detection feature layer of the single-shot multibox detector (SSD) algorithm does not perform feature fusion and the detection effect is poor, an adaptive feature fusion SSD model is proposed. Firstly, the location of the shallow feature map and the multi-scale receptive field on the deep feature map are added, and the scaling and adaptive fusion of different scale feature maps are carried out to improve the representation ability of detail information. Secondly, the feature layer of the same scale can provide different ranges of feature information, transfer the specific features with detailed information to the abstract features with semantic information, and use the global average pool to guide learning and expand the expression ability of features. After training and testing on the PASCAL VOC data set, the detection accuracy reaches 80.6% and the detection speed reaches 60.9 fps, which verifies the robustness and real-time performance of the algorithm.
A Heterogeneous FPGA-based Accelerator Design for Efficient and Low-cost Point Clouds Deep Learning Inference

The neural networks on 3D data and applications have emerged in the past five years. However, there are only a few dedicated hardware designs were proposed for 3D data and algorithms. Meanwhile, they lack flexibility and adaptation for the fast evolvement of software algorithms. We propose a heterogeneous accelerator design on Xilinx Zynq and Zynq UltraScale+ platform. An innovative vector pipeline is designed in the accelerator that can reach the near limitation of BRAM frequency, and it gives the final design frequency closure at 550MHz with 100% DSP usage.