Reviewerly/exHarmony huggingface dataset:
Speeding Up Encryption on IoT Devices Using Homomorphic Encryption

What do we need to do to protect our personal information? IoT devices such as smartphones, smart watches, and home appliances are widespread. Encryption is required not only to prevent eavesdropping on communications but also to prevent information leakage from cloud services due to unauthorized access. Therefore, attention is being paid to fully homomorphic encryption (FHE) that allows addition and multiplication between ciphertexts. However, FHE with this convenient function has a drawback that the encryption requires huge volume of calculation and the ciphertext is large. Therefore, if FHE is used on a device with limited computational resources such as an IoT device, the load on the IoT device will be too heavy. In this research, we propose a system that can safely and effectively utilize data without imposing a load on IoT devices. In this system, somewhat homomorphic encryption (SHE), which is a lightweight cryptosystem compared with FHE, is combined with FHE. The results of the experiment confirmed that the load on the IoT device can be reduced to approximately 1/1400 compared to load of the system from previous research.
Question-controlled Text-aware Image Captioning

For an image with multiple scene texts, different people may be interested in different text information. Current text-aware image captioning models are not able to generate distinctive captions according to various information needs. To explore how to generate personalized text-aware captions, we define a new challenging task, namely Question-controlled Text-aware Image Captioning (Qc-TextCap). With questions as control signals, this task requires models to understand questions, find related scene texts and describe them together with objects fluently in human language. Based on two existing text-aware captioning datasets, we automatically construct two datasets, ControlTextCaps and ControlVizWiz to support the task. We propose a novel Geometry and Question Aware Model (GQAM). GQAM first applies a Geometry-informed Visual Encoder to fuse region-level object features and region-level scene text features with considering spatial relationships. Then, we design a Question-guided Encoder to select the most relevant visual features for each question. Finally, GQAM generates a personalized text-aware caption with a Multimodal Decoder. Our model achieves better captioning performance and question answering ability than carefully designed baselines on both two datasets. With questions as control signals, our model generates more informative and diverse captions than the state-of-the-art text-aware captioning model. Our code and datasets are publicly available at https://github.com/HAWLYQ/Qc-TextCap.
Sampling-Based Optimal Trajectory Generation for Autonomous Vehicles Using Reachable Sets

Motion planners for autonomous vehicles must obtain feasible trajectories in real-time regardless of the complexity of traffic conditions. Planning approaches that discretize the search space may perform sufficiently in general driving situations, however, they inherently struggle in critical situations with small solution spaces. To address this problem, we prune the search space of a sampling-based motion planner using reachable sets, i.e., sets of states that the ego vehicle can reach without collision. By only creating samples within the collision-free reachable sets, we can drastically reduce the number of required samples and thus the computation time of the planner to find a feasible trajectory, especially in critical situations. The benefits of our novel concept are demonstrated using scenarios from the CommonRoad benchmark suite.
Progressive and Selective Fusion Network for High Dynamic Range Imaging

This paper considers the problem of generating an HDR image of a scene from its LDR images. Recent studies employ deep learning and solve the problem in an end-to-end fashion, leading to significant performance improvements. However, it is still hard to generate a good quality image from LDR images of a dynamic scene captured by a hand-held camera, e.g., occlusion due to the large motion of foreground objects, causing ghosting artifacts. The key to success relies on how well we can fuse the input images in their feature space, where we wish to remove the factors leading to low-quality image generation while performing the fundamental computations for HDR image generation, e.g., selecting the best-exposed image/region. We propose a novel method that can better fuse the features based on two ideas. One is multi-step feature fusion; our network gradually fuses the features in a stack of blocks having the same structure. The other is the design of the component block that effectively performs two operations essential to the problem, i.e., comparing and selecting appropriate images/regions. Experimental results show that the proposed method outperforms the previous state-of-the-art methods on the standard benchmark tests.
Signal Temporal Logic Synthesis as Probabilistic Inference

We reformulate the signal temporal logic (STL) synthesis problem as a maximum a-posteriori (MAP) inference problem. To this end, we introduce the notion of random STL (RSTL), which extends deterministic STL with random predicates. This new probabilistic extension naturally leads to a synthesis-as-inference approach. The proposed method allows for differentiable, gradient-based synthesis while extending the class of possible uncertain semantics. We demonstrate that the proposed framework scales well with GPU-acceleration, and present realistic applications of uncertain semantics in robotics that involve target tracking and the use of occupancy grids.
A light heterogeneous graph collaborative filtering model using textual information

Due to the development of graph neural networks, graph-based representation learning methods have made great progress in recommender systems. However, data sparsity is still a challenging problem that most graph-based recommendation methods are confronted with. Recent works try to address this problem by utilizing side information. In this paper, we exploit the relevant and easily accessible textual information by advanced natural language processing (NLP) models and propose a light RGCN-based (RGCN, relational graph convolutional network) collaborative filtering method on heterogeneous graphs. Specifically, to incorporate rich textual knowledge, we utilize a pre-trained NLP model to initialize the embeddings of text nodes. Afterward, by performing a simplified RGCN-based node information propagation on the constructed heterogeneous graph, the embeddings of users and items can be adjusted with textual knowledge, which effectively alleviates the negative effects of data sparsity. Moreover, the matching function used by most graph-based representation learning methods is the inner product, which is not appropriate for the obtained embeddings that contain complex semantics. We design a predictive network that combines graph-based representation learning with neural matching function learning, and demonstrate that this architecture can bring a significant performance improvement. Extensive experiments are conducted on three publicly available datasets, and the results verify the superior performance of our method over several baselines.
A Global Registration Method for Satellite Image Series

Image registration is a fundamental tool of remote sensing. The recent proliferatio of earth observation satellites has opened the way to the analysis of long image time series with denser temporal repetition. Given this wealth of images, it is crucial to design automatic tools to process them. We thus propose a method for the global registration of satellite image time series, that leverages their redundancy to improve in precision and robustness. By computing the relative displacement for all possible pairs of images, we are able to discard outliers and minimize the number of misaligned images. Experiments on synthetic data show that longer image series are registered with a higher precision.
The Burden of Persuasion in Abstract Argumentation

In this paper, we provide a formal framework for modeling the burden of persuasion in legal reasoning. The framework is based on abstract argumentation, a frequently studied method of non-monotonic reasoning, and can be applied to different argumentation semantics; it supports burdens of persuasion with arbitrary many levels, and allows for the placement of a burden of persuasion on any subset of an argumentation framework’s arguments. Our framework can be considered an extension of related works that raise questions on how burdens of persuasion should be handled in some conflict scenarios that can be modeled with abstract argumentation. An open source software implementation of the introduced formal notions is available as an extension of an argumentation reasoning library.
Trajectory Prediction with Linguistic Representations

Language allows humans to build mental models that interpret what is happening around them resulting in more accurate long-term predictions. We present a novel trajectory prediction model that uses linguistic intermediate representations to forecast trajectories, and is trained using trajectory samples with partially-annotated captions. The model learns the meaning of each of the words without direct per-word supervision. At inference time, it generates a linguistic description of trajectories which captures maneuvers and interactions over an extended time interval. This generated description is used to refine predictions of the trajectories of multiple agents. We train and validate our model on the Argoverse dataset, and demonstrate improved accuracy results in trajectory prediction. In addition, our model is more interpretable: it presents part of its reasoning in plain language as captions, which can aid model development and can aid in building confidence in the model before deploying it.
Hybrid Network Compression via Meta-Learning

Neural network pruning and quantization are two major lines of network compression. This raises a natural question that whether we can find the optimal compression by considering multiple network compression criteria in a unified framework. This paper incorporates two criteria and seeks layer-wise compression by leveraging the meta-learning framework. A regularization loss is applied to unify the constraint of input and output channel numbers, bit-width of network activations and weights, so that the compressed network can satisfy a given Bit-OPerations counts (BOPs) constraint. We further propose an iterative compression constraint for optimizing the compression procedure, which effectively achieves a high compression rate and maintains the original network performance. Extensive experiments on various networks and vision tasks show that the proposed method yields better performance and compression rates than recent methods. For instance, our method achieves better image classification accuracy and compactness than the recent DJPQ. It achieves similar performance with the recent DHP in image super-resolution, meanwhile saves about 50% computation.
Human Attributes Prediction under Privacy-preserving Conditions

Human attributes prediction in visual media is a well-researched topic with a major focus on human faces. However, face images are often of high privacy concern as they can reveal an individual's identity. How to balance this trade-off between privacy and utility is a key problem among researchers and practitioners. In this study, we make one of the first attempts to investigate the human attributes (emotion, age, and gender) prediction under the different de-identification (eyes, lower-face, face, and head obfuscation) privacy scenarios. We first constructed the Diversity in People and Context Dataset (DPaC). We then performed a human study with eye-tracking on how humans recognize facial attributes without the presence of face and context. Results show that in an image, situational context is informative of a target's attributes. Motivated by our human study, we proposed a multi-tasking deep learning model - Context-Guided Human Attributes Prediction (CHAPNet), for human attributes prediction under privacy-preserving conditions. Extensive experiments on DPaC and three commonly used benchmark datasets demonstrate the superiority of CHAPNet in leveraging the situational context for a better interpretation of a target's attributes without the full presence of the target's face. Our research demonstrates the feasibility of visual analytics under de-identification for privacy.
Where Do Stories Come From? Examining the Exploration Process in Investigative Data Journalism

Investigative data journalists work with a variety of data sources to tell a story. Though prior work has indicated that there is a close relationship between journalists' data work practices and that of data scientists. However, these relationships and data work practices are not empirically examined, and understanding them is crucial to inform the design of tools that are used by different groups of people including data scientists and data journalists. Thus, to bridge this gap, we studied investigative reporters' data work practices with one non-profit investigative newsroom. Our study design includes two activities: 1) semi-structured interviews with journalists, and 2) a sketching activity allowing journalists to depict examples of their work practices. By analyzing these data and synthesizing them across related prior work, we propose the major phases in the data-driven investigative journalism story idea generation process. Our study findings show that the journalists employ a collection of multiple, iterative, cyclic processes to identify journalistically "interesting'' story ideas. These processes both significantly resemble and show subtle nuanced differences with data science work practices identified in prior research. We further verified our proposal through a member check with key informants. This work offers three primary contributions. First, it provides a close glimpse into the main phases of investigative journalists' data-driven story idea generation technique. Second, it complements prior work studying formal data science practices by examining data-driven investigative journalists, whose primary expertise lies outside computing. Third, it identifies particular points in the data exploration processes that would benefit from design interventions and suggests future research directions.
Blockchain Enabled Secure Authentication for Unmanned Aircraft Systems

The integration of air and ground smart vehicles is becoming a new paradigm of future transportation. A decent number of smart unmanned vehicles or UAS will be sharing the national airspace for various purposes, such as express delivery, surveillance, etc. However, the proliferation of UAS also brings challenges considering the safe integration of them into the current Air Traffic Management (ATM) systems. Especially when the current Automatic Dependent Surveillance Broadcasting (ADSB) systems do not have message authentication mechanisms, it can not distinguish whether an authorized UAS is using the corresponding airspace. In this paper, we aim to address these practical challenges in two folds. We first use blockchain to provide a secure authentication platform for flight plan approval and sharing between the existing ATM facilities. We then use the fountain code to encode the authentication payloads and adapt them into the de facto communication protocol of ATM. This maintains backward compatibility and ensures the verification success rate under the noisy broadcasting channel. We simulate the realistic wireless communication scenarios and theoretically prove that our proposed authentication framework is with low latency and highly compatible with existing ATM communication protocols.
Exploiting Different Levels of Parallelism in the Quantum Control Microarchitecture for Superconducting Qubits

As current Noisy Intermediate Scale Quantum (NISQ) devices suffer from decoherence errors, any delay in the instruction execution of quantum control microarchitecture can lead to the loss of quantum information and incorrect computation results. Hence, it is crucial for the control microarchitecture to issue quantum operations to the Quantum Processing Unit (QPU) in time. As in classical microarchitecture, parallelism in quantum programs needs to be exploited for speedup. However, three challenges emerge in the quantum scenario: 1) the quantum feedback control can introduce significant pipeline stall latency; 2) timing control is required for all quantum operations; 3) QPU requires a deterministic operation supply to prevent the accumulation of quantum errors.
Learn to Ignore: Domain Adaptation for Multi-site MRI Analysis

The limited availability of large image datasets, mainly due to data privacy and differences in acquisition protocols or hardware, is a significant issue in the development of accurate and generalizable machine learning methods in medicine. This is especially the case for Magnetic Resonance (MR) images, where different MR scanners introduce a bias that limits the performance of a machine learning model. We present a novel method that learns to ignore the scanner-related features present in MR images, by introducing specific additional constraints on the latent space. We focus on a real-world classification scenario, where only a small dataset provides images of all classes. Our method Learn to Ignore (L2I) outperforms state-of-the-art domain adaptation methods on a multi-site MR dataset for a classification task between multiple sclerosis patients and healthy controls.
(Compact) Adaptively Secure FE for Attribute-Weighted Sums from k-Lin

This paper presents the first adaptively simulation secure functional encryption (FE) schemes for attribute-weighted sums. In such an FE scheme, encryption takes as input N pairs of attribute \(\{(x_i, z_i)\}_{i\in [N]}\) for some \(N\in \mathbb {N}\) where the attributes \(\{x_i\}_{i\in [N]}\) are public while the attributes \(\{z_i\}_{i\in [N]}\) are private. The indices \(i\in [N]\) are referred to as the slots. A secret key corresponds to some weight function f, and decryption recovers the weighted sum \(\sum _{i=1}^N f(x_i)z_i\). This is an important functionality with a wide range of potential real life applications. In the proposed FE schemes attributes are viewed as vectors and weight functions are arithmetic branching programs (ABP). We present two schemes with varying parameters and levels of adaptive security.
Multi-Query Multi-Head Attention Pooling and Inter-Topk Penalty for Speaker Verification

This paper describes the multi-query multi-head attention (MQMHA) pooling and inter-topK penalty methods which were first proposed in our submitted system description for VoxCeleb speaker recognition challenge (VoxSRC) 2021. Most multi-head attention pooling mechanisms either attend to the whole feature through multiple heads or attend to several split parts of the whole feature. Our proposed MQMHA combines both these two mechanisms and gain more diversified information. The margin-based softmax loss functions are commonly adopted to obtain discriminative speaker representations. To further enhance the inter-class discriminability, we propose a method that adds an extra inter-topK penalty on some confused speakers. By adopting both the MQMHA and inter-topK penalty, we achieved state-of-the-art performance in all of the public VoxCeleb test sets.
Perspectives on the Emerging Field of Autonomous Systems and its Theoretical Foundations

Autonomous systems are advanced intelligent systems and general AI technologies triggered by the transdisciplinary development in intelligence science, system science, brain science, cognitive science, robotics, computational intelligence, and intelligent mathematics. AS are driven by the increasing demands in the modern industries of cognitive computers, deep machine learning, robotics, brain-inspired systems, self-driving cars, internet of things, and intelligent appliances. This paper presents a perspective on the framework of autonomous systems and their theoretical foundations. A wide range of application paradigms of autonomous systems are explored.
Robust optical flow estimation to enhance behavioral research on ants

The ant behavior is an intrinsically fascinating topic. The movement of ants provides a rich source of social behavior for evaluating motion estimation methods. Studies on ants motion usually involve researchers watching video recordings and manually scoring each ant's movements. The traditional observation method is challenging because many ants can interact with each other, interfering with perceiving motion. Pixel-level motion estimation can perform this more robustly and accurately. This paper focuses on optical flow estimation to observe ant movements from an imaging system to perceive the motion through the sequence of images. It is known that the optical flow methods suffer from the issue of ill-defined edges and boundaries of the moving objects. An edge-preserving filter-based optical flow method is constructed to estimate the motion of ants in outdoor complex scenarios. The results reveal that the proposed method can improve the accuracy of motion estimation.
Cross-Media Intelligent Perception and Retrieval Analysis Application Technology Based on Deep Learning Education

This paper proposes a research on cross-media intelligent perception and retrieval analysis technology based on deep learning education in view of the complex learning knowledge environment of artificial intelligence of traditional cross-media retrieval technology, which is unable to obtain retrieval information timely and accurately. Based on the cross-media theory, this paper analyzes the pre-processing of cross-media data, transforms from a single media form such as text, voice, image, and video to a cross-media integration covering network space and physical space, and designs a cross-media intelligent perception platform system. With multi-core method of typical correlation analysis algorithm, this paper develops a new joint learning framework based on the joint feature selection and subspace learning cross-modal retrieval. What’s more, the algorithm is tested experimentally. The results show that the retrieval analysis technology is significantly better than the traditional media retrieval technology, which can effectively identify text semantics and visual image classification, and can better maintains the relevance of data content and the consistency of semantic information, which has important reference value for cross-media applications.