Reviewerly/exHarmony huggingface dataset:
Relation-Based Representation for Handwritten Mathematical Expression Recognition

This paper proposes relation-based sequence representation that enhances offline handwritten mathematical expressions (HMEs) recognition. Commonly, a LaTeX-based sequence represents the 2D structure of an HME as a 1D sequence. Consequently, the LaTeX-based sequence becomes longer, and HME recognition systems have difficulty in extracting its 2D structure. We propose a new representation for HMEs according to the relations of symbols, which shortens the LaTeX-based representation. We use an offline end-to-end HME recognition system that adopts weakly supervised learning to evaluate the proposed representation. Recognition experiments indicate that the proposed relation-based representation helps the HME recognition system achieve higher performance than the LaTeX-based representation. In fact, the HME recognition system achieves recognition rates of 53.35%, 52.14%, and 53.13% on the dataset of the Competition on Recognition of Online Handwritten Mathematical Expressions (CROHME) 2014, 2016, and 2019, respectively. These results are more than 2 percentage points higher than the LaTeX-based system.
Zernike Pooling: Generalizing Average Pooling Using Zernike Moments

Most of the established neural network architectures in computer vision are essentially composed of the same building blocks (e.g., convolutional, normalization, regularization, pooling layers, etc.), with their main difference being the connectivity of these components within the architecture and not the components themselves. In this paper we propose a generalization of the traditional average pooling operator. Based on the requirements of efficiency (to provide information without repetition), equivalence (to be able to produce the same output as average pooling) and extendability (to provide a natural way of obtaining novel information), we arrive at a formulation that generalizes average pooling using the Zernike moments. Experimental results on Cifar 10, Cifar 100 and Rotated MNIST data-sets showed that the proposed method was able to outperform the two baseline approaches, global average pooling and average pooling 2×2, as well as the two variants of Stochastic pooling and AlphaMEX in every case. A worst-case performance analysis on Cifar-100 showed that significant gains in classification accuracy can be realised with only a modest 10% increase in training time.
Secure and Efficient Transmission of Vision-Based Feedback Control Signals

Abstract An ever-increasing number of autonomous vehicles use bandwidth-greedy sensors such as cameras and LiDARs to sense and act to the world around us. Unfortunately, signal transmission in vehicles is vulnerable to passive and active cyber-physical attacks that may result in loss of intellectual property, or worse yet, the loss of control of a vehicle, potentially causing great harm. Therefore, it is important to investigate efficient cryptographic methods to secure signal transmission in such vehicles against outside threats. This study is motivated by the observation that previous publications have suggested legacy algorithms, which are either inefficient or insecure for vision-based signals. We show how stream ciphers and authenticated encryption can be applied to transfer sensor data securely and efficiently between computing devices suitable for distributed guidance, navigation, and control systems. We provide an efficient and flexible pipeline of cryptographic operations on image and point cloud data in the Robot Operating System (ROS). We also demonstrate how image data can be compressed to reduce the amount of data to be encrypted, transmitted, and decrypted. Experiments on embedded computers verify that modern software cryptographic algorithms perform very well on large sensor data. Hence, the introduction of such algorithms should enhance security without significantly compromising the overall performance.
Fast Multi-Step Critiquing for VAE-based Recommender Systems

Recent studies have shown that providing personalized explanations alongside recommendations increases trust and perceived quality. Furthermore, it gives users an opportunity to refine the recommendations by critiquing parts of the explanations. On one hand, current recommender systems model the recommendation, explanation, and critiquing objectives jointly, but this creates an inherent trade-off between their respective performance. On the other hand, although recent latent linear critiquing approaches are built upon an existing recommender system, they suffer from computational inefficiency at inference due to the objective optimized at each conversation’s turn. We address these deficiencies with M&Ms-VAE, a novel variational autoencoder for recommendation and explanation that is based on multimodal modeling assumptions. We train the model under a weak supervision scheme to simulate both fully and partially observed variables. Then, we leverage the generalization ability of a trained M&Ms-VAE model to embed the user preference and the critique separately. Our work’s most important innovation is our critiquing module, which is built upon and trained in a self-supervised manner with a simple ranking objective. Experiments on four real-world datasets demonstrate that among state-of-the-art models, our system is the first to dominate or match the performance in terms of recommendation, explanation, and multi-step critiquing. Moreover, M&Ms-VAE processes the critiques up to 25.6x faster than the best baselines. Finally, we show that our model infers coherent joint and cross generation, even under weak supervision, thanks to our multimodal-based modeling and training scheme.
A novel encryption scheme for securing biometric templates based on 2D discrete wavelet transform and 3D Lorenz-chaotic system

This research paper proposes a novel encryption and decryption scheme for biometric templates using two dimensional discrete wavelet transform (2D-DWT) and three dimensional (3D) Lorenz-chaotic system. In the proposed scheme two levels of 2D-DWT are used for the decomposition of the group representation of the different biometric templates and then substitution and permutation are applied using some mechanism and 3D Lorentz-chaotic system. The encryption scheme is applied on the different sub-bands of each level of decomposition, and for the decryption, the reverse process of the encryption is used. The experimental results, biometric template matching, security analysis, and statistical analysis of the proposed scheme are given to validate the robustness of the scheme. The security analysis shows that the brute-force attack is not possible in the proposed scheme.
Joint sparse principal component regression with robust property

Standard principal component regression (PCR) selects principal components without any considerations of response variable. Therefore, standard PCR cannot produce a sufficiently accurate model. Sparse principal component regression (SPCR) is a novel one-stage procedure that extracts principal components and constructs a linear regression model simultaneously. Since SPCR can be viewed as a combination of standard principal component regression and sparse principal component analysis (SPCA), it also inherits many drawbacks from them. Consequently, the absolute constraints and the least square loss function make SPCR lack of joint property and robustness. To tackle these problems, this paper proposes joint sparse principal component regression (JSPCR), which can select the important features jointly and estimate the coefficients robustly. Thus, JSPCR can have higher prediction accuracy. To encourage the sparsity of loading matrix, we apply a sparse group penalty and extend JSPCR to joint bi-level sparse PCR (JBSPCR). We derive an alternating optimization criterion and prove its convergence as well. The synthetical data simulations and the real data analysis powerfully support that JSPCR is superior to SPCR and other famous dimensionality reduction procedures or sparse techniques.
Chatbots, Gender, and Race on Web 2.0 Platforms: Tay.AI as Monstrous Femininity and Abject Whiteness

In March 2016, Microsoft launched Tay.AI, a chatbot designed to experiment with conversational understanding through direct engagement with social media users. Marketed as the digital representation of an 18–24-year-old, cis-gendered female, Tay.ai was meant to be chatty, personable, friendly, and innocuous. Hours into launch, however, the chatbot’s mimetic programming structure was taken advantage of by organized groups of online social media users, and Tay.ai began replying to queries with alt-right and neo-Nazi ideology. In this article, I explore the role that Tay.ai’s assigned gender and race played in influencing both Tay.ai’s initial design and, subsequently, the program’s monstrous evolution. This is achieved through three avenues of thought. First, I situate Tay within the new public of Web 2.0, a space reliant on user participation and beholden to the neoliberal, racialized, and gendered architectures that produce it. I then consider Tay as an evolution of the chatbot, arguing that Tay is emblematic of both race and gender as social technologies. Third, I explore Tay’s aberration from programmatic protocols in the context of the monstrous and the abject, suggesting that digital nonhuman ambivalence to programming and encoded control presents a space of productive creativity.
BBW: a batch balance wrapper for training deep neural networks on extremely imbalanced datasets with few minority samples

Abstract In recent years, Deep Neural Networks (DNNs) have achieved excellent performance on many tasks, but it is very difficult to train good models from imbalanced datasets. Creating balanced batches either by majority data down-sampling or by minority data up-sampling can solve the problem in certain cases. However, it may lead to learning process instability and overfitting. In this paper, we propose the Batch Balance Wrapper (BBW), a novel framework which can adapt a general DNN to be well trained from extremely imbalanced datasets with few minority samples. In BBW, two extra network layers are added to the start of a DNN. The layers prevent overfitting of minority samples and improve the expressiveness of the sample distribution of minority samples. Furthermore, Batch Balance (BB), a class-based sampling algorithm, is proposed to make sure the samples in each batch are always balanced during the learning process. We test BBW on three well-known extremely imbalanced datasets with few minority samples. The maximum imbalance ratio reaches 1167:1 with only 16 positive samples. Compared with existing approaches, BBW achieves better classification performance. In addition, BBW-wrapped DNNs are 16.39 times faster, relative to unwrapped DNNs. Moreover, BBW does not require data preprocessing or additional hyper-parameter tuning, operations that may require additional processing time. The experiments prove that BBW can be applied to common applications of extremely imbalanced data with few minority samples, such as the classification of EEG signals, medical images and so on.
Implementation of the Cross-Blockchain Interacting Protocol

Blockchain is a developing and promising technology that can provide users with such advantages as decentralization, data security and transparency of transactions. Blockchain has many applications, one of them is the decentralized finance (DeFi) industry which is growing more and more recently. The concept of decentralized finance involves the creation of a single ecosystem of many blockchains that interact with each other. The problem of combining and interacting blockchains becomes crucial to enable DeFi. In this paper, we look at the essence of the DeFi industry, the possibilities of overcoming the problem of cross-blockchain interaction, present our approach, and analyze the results of the proposed solution.
Addressing Audit and Accountability Issues in Self-Sovereign Identity Blockchain Systems Using Archival Science Principles

Self-sovereign identity (SSI) systems are novel blockchain-based solutions that are said to shift the control of data records from organizations to individuals. Contrary to conventional blockchains, such as Bitcoin or Ethereum, many SSI systems do not capture on ledger the exchange of transactional data between individuals. By not capturing the exchange of transaction data such SSI systems have the advantage of complying with privacy regulations such as the EU’s General Data Protection Regulations, but, at the same time, have the disadvantage of not capturing evidence that an exchange has happened. Such evidence, however, may be needed for audit and accountability purposes. To achieve these objectives and to preserve privacy, we leverage archival principles to introduce a novel concept of a proof registry, which we define as a set of technical components, data structures, and process flows, that assures that authoritative records offering evidence of transactions is captured, stored, and accessible. This solution solves the compliance and accountability problem while preserving the self-sovereignty and privacy of involved parties.
Sybil Attacks and Defense on Differential Privacy based Federated Learning

In federated learning, machine learning and deep learning models are trained globally on distributed devices. The state-of-the-art privacy-preserving technique in the context of federated learning is user-level differential privacy. However, such a mechanism is vulnerable to some specific model poisoning attacks such as Sybil attacks. A malicious adversary could create multiple fake clients or collude compromised devices in Sybil attacks to mount direct model updates manipulation. Recent works on novel defense against model poisoning attacks are difficult to detect Sybil attacks when differential privacy is utilized, as it masks clients' model updates with perturbation. In this work, we implement the first Sybil attacks on differential privacy based federated learning architectures and show their impacts on model convergence. We randomly compromise some clients by manipulating different noise levels reflected by the local privacy budget ε of differential privacy with Laplace mechanism on the local model updates of these Sybil clients. As a result, the global model convergence rates decrease or even leads to divergence. We apply our attacks to two recent aggregation defense mechanisms, called Krum and Trimmed Mean. Our evaluation results on the MNIST and CIFAR-10 datasets show that our attacks effectively slow down the convergence of the global models. We then propose a method to keep monitoring the average loss of all participants in each round for convergence anomaly detection and defend our Sybil attacks based on the training loss reported from randomly selected sets of clients as the judging panels. Our empirical study demonstrates that our defense effectively mitigates the impact of our Sybil attacks.
Measurement-Dependent Erasure of Distinguishability for the Observation of Interference in an Unbalanced SU(1,1) Interferometer

The temporal distinguishability in an unbalanced interferometer leads to the erasure of interference in the direct photodetection which can be recovered via amplitude measurement by homodyne detection.
Trust Evidence Logic

We investigate the application of a modal language à la Hennessy-Milner to the specific domain of evidence-based trust estimations. In particular, we refer to a context-aware notion of computational trust joining in a quantitative setting both assessment of subjective opinions and third-party recommendations. Moreover, for a comprehensive analysis of the proposed logics, we offer an axiomatization and provide soundness and completeness results.
User-Item Matching for Recommendation Fairness

As we all know, users and item-providers are two main parties of participants in recommender systems. However, most existing research efforts on recommendation were focused on better serving users and overlooked the purpose of item-providers. This paper is devoted to improve the item exposure fairness for item-providers' objective, and keep the recommendation accuracy not decreased or even improved for users' objective. We propose to set stock volume constraints on items, to be specific, limit the maximally allowable recommended times of an item to be proportional to the frequency of its being interacted in the past, which is validated to achieve superior item exposure fairness to common recommenders and thus mitigates the Matthew Effect on item popularity. With the two constraints of pre-existing recommendation length of users and our stock volumes of items, a heuristic strategy based on normalized scores and a Minimum Cost Maximum Flow (MCMF) based model are proposed to solve the optimal user-item matching problem, whose accuracy performances are even better than that of baseline algorithm in regular recommendation context, and in line with state-of-the-art enhancement of the baseline. What's more, our MCMF based strategy is parameter-free, while those counterpart algorithms have to resort to parameter traversal process to achieve their best performance.
Initial Evaluation of an Intelligent Virtual Museum Prototype Powered by AI, XR and Robots

This paper presents the design, development and initial evaluation of an intelligent virtual museum prototype based on a new type of Cyber-Physical-Social Eco-System (CPSeS) framework aiming to merge the real with virtual worlds interchangeably using AI, XR and Robots. Whereas virtual environments have become prominent tools in many domains, offering shared and interactive virtual worlds, the proposed prototype incorporates multi-user and interactive functionalities together with a new agent, namely, a physical robot and its digital twin. The physical robot is located and acts in a real environment whilst its avatar (further referred to as its digital twin) lives in the virtual world. The users are able to see and explore both worlds simultaneously through the ‘eyes’ of the robot. Together with multi-user infrastructure and communication capabilities, the environment also involves additional agents guiding the user in the virtual world, and an educational game, aiming at developing a CPSeS capable of blending the real with digital worlds, and to be influenced by its users, real and artificial agents and elements. The user-based qualitative evaluation of the proposed system was favourable but also constructive providing the research team with valuable observations on its performance.
Joint Multisource Saliency and Exemplar Mechanism for Weakly Supervised Video Object Segmentation

Weakly supervised video object segmentation (WSVOS) is a vital yet challenging task in which the aim is to segment pixel-level masks with only category labels. Existing methods still have certain limitations, e.g., difficulty in comprehending appropriate spatiotemporal knowledge and an inability to explore common semantic information with category labels. To overcome these challenges, we formulate a novel framework by integrating multisource saliency and incorporating an exemplar mechanism for WSVOS. Specifically, we propose a multisource saliency module to comprehend spatiotemporal knowledge by integrating spatial and temporal saliency as bottom-up cues, which can effectively eliminate disruptions due to confusing regions and identify appropriate attractive regions. Moreover, to our knowledge, we make the first attempt to incorporate an exemplar mechanism into WSVOS by proposing an adaptive exemplar module to process top-down cues, which can provide reliable guidance for co-occurring objects in intraclass videos and identify attentive regions. Our framework, which comprises the two aforementioned modules, offers a new perspective on directly constructing the correspondence between bottom-up cues and top-down cues when ground-truth information for the reference frames is lacking. Comprehensive experiments demonstrate that the proposed framework achieves state-of-the-art performance.
Model-Based Approach for Measuring the Fairness in ASR

The issue of fairness arises when the automatic speech recognition (ASR) systems do not perform equally well for all subgroups of the population. In any fairness measurement studies for ASR, the open questions of how to control the confounding factors, how to handle unobserved heterogeneity across speakers, and how to trace the source of any word error rate (WER) gap among different subgroups are especially important - if not appropriately accounted for, incorrect conclusions will be drawn. In this paper, we introduce mixed-effects Poisson regression to better measure and interpret any WER difference among subgroups of interest. Particularly, the presented method can effectively address the three problems raised above and is very flexible to use in practical disparity analyses. We demonstrate the validity of proposed model-based approach on both synthetic and real-world speech data.
Weighted multi-view common subspace learning method

How to use multi-view data effectively has become one of the challenging problems in the computer vision community. The existing multi-view learning methods are mainly based on common subspace learning which aim to explore the discriminative information between multi-view data and find its potential common subspace. Most of the existing multi-view subspace learning methods rely on the within-class scatter matrix and between-class scatter matrix while capturing the discriminative information of multiple views. However, these methods just roughly minimize the within-class distance and maximize the between-class distance, and do not make full use of the intra-view and inter-view information. To address this problem, we propose a weighted common subspace learning method, which can effectively adjust the contribution ratio of between-class and within-class information through a weighted parameter, so that an optimized common subspace can be obtained. And we use the maximum scatter difference criterion as the metric of inter-view and intra-view after projection. Extensive experiments on the public data sets show the superiority of this method.
A deep learning based end-to-end system (F-Gen) for automated email FAQ generation

With overwhelming volumes of official emails being exchanged in enterprises every day, emails have become vital information storehouses. Automatic generation of FAQs from email systems helps in identifying important information and could serve potential applications such as chatbots and intelligent email answering. While there exist studies in the literature focusing on automatic FAQ generation and automated email answering, there are few studies that apply recently developed deep learning techniques to fetch FAQs from emails. This paper proposes a novel framework named F-Gen, which is an expert system that generates potential FAQs from emails utilizing state-of-the-art methodologies. The key characteristics of this study are as follows: 1. Designing F-Gen with various subsystems that interoperate together for the FAQ generation 2. Identifying the parameters that determine a valid FAQ. The three subsystems of F-Gen are: (a) query classifier subsystem (QC subsystem) for email texts, (b) FAQ group generator subsystem (FGG subsystem) for generating FAQ groups from email queries. And (c) FAQ generator subsystem (FG subsystem) for conversion of email query clusters into FAQs. Experiments on the email dataset that practically reflect the above-mentioned problem resulted in FAQs with a ROUGE-1 F-Score of 74.10% when compared with the ground truth.