Reviewerly/exHarmony huggingface dataset:
Table Structure Recognition and Form Parsing by End-to-End Object Detection and Relation Parsing

The recognition of two-dimensional structure of tables and forms from document images is a challenge due to the complexity of document structures and the diversity of layouts. In this paper, we propose a graph neural network (GNN) based unified framework named Table Structure Recognition Network (TSRNet) to jointly detect and recognize the structures of various tables and forms. First, a multi-task fully convolutional network (FCN) is used to segment primitive regions such as text segments and ruling lines from document images, then a GNN is used to classify and group these primitive regions into page objects such as tables and cells. At last, the relationships between neighboring page objects are analyzed using another GNN based parsing module. The parameters of all the modules in the system can be trained end-to-end to optimize the overall performance. Experiments of table detection and structure recognition for modern documents on the POD 2017, cTDaR 2019 and PubTabNet datasets and template-free form parsing for historical documents on the NAF dataset show that the proposed method can handle various table/form structures and achieve superior performance.
Render-in-the-loop Aerial Robotics Simulator: Case Study on Yield Estimation in Indoor Agriculture

Inspired by recent promising results in sim-to-real transfer in deep learning we built a realistic simulation environment combining a Robot Operating System (ROS)-compatible physics simulator (Gazebo) with Cycles, the realistic production rendering engine from Blender. The proposed simulator pipeline allows us to simulate near-realistic RGB-D images. To showcase the capabilities of the simulator pipeline we propose a case study that focuses on indoor robotic farming. We developed a solution for sweet pepper yield estimation task. Our approach to yield estimation starts with aerial robotics control and trajectory planning, combined with deep learning-based pepper detection, and a clustering approach for counting fruit. The results of this case study show that we can combine real time dynamic simulation with near realistic rendering capabilities to simulate complex robotic systems.
A Genetic Algorithm enhanced with Fuzzy-Logic for multi-objective Unmanned Aircraft Vehicle path planning missions

The fast growth of computational-intelligence and computer vision-based approaches in the field of robotics have initiated the development of novel unmanned aircraft vehicles (UAVs) with additional capabilities for mission-critical applications, such as fire detection or delivery services. Path planning is a crucial part of the automated operation of a UAV to find the optimal path from a starting point to a target-goal point, satisfying in parallel specific criteria and constraints. To this end, traditional path planning algorithms, such as graph-based, heuristics or metaheuristics, have been adopted to address this problem. However, the complex operational environment combined with the demand for energy efficiency have emerged the need for dynamic and multi-objective path planning meeting the special characteristics of a UAV flight. To address the above challenges, in this study a novel Genetic Algorithm (GA) enhanced with Fuzzy Logic (GAF) is developed for solving the UAV multi-objective path planning problem. The proposed algorithm aims to find an optimal path with respect to contradicted objective terms, such as distance, energy efficiency and path’s curvature. In the proposed developments, the novelty of GAF rises via a two-step procedure: (i) the evaluation stage of the energy efficiency of the paths; and (ii) the evaluation stage of the GA for the development of an energy efficient, smooth path of shortest possible traveled distance. The efficacy of the approach is illustrated via a comparative experimental evaluation.
Deep representation-based transfer learning for deep neural networks

In recent years, deep neural networks (DNNs) have become the de facto models for practically all visual tasks and most temporal analysis tasks due to the abundance of available labeled data and advances in computational resources. Deep transfer learning, a learning paradigm that investigates how deep neural networks can leverage knowledge from other domains, has been successful in many practical applications. Most deep transfer learning methods rely on directly available source domain data, especially instances-based and mapping-based methods. However, the source domain data required for transfer cannot be provided in some cases because of personal privacy or copyright implications. In this regard, we propose a deep representation-based transfer learning method for deep neural networks to deal with transfer scenarios without source domain data. The proposed method induces the target deep neural network to learn the similarity structure between source deep representations by minimizing the representational similarity discrepancy between the networks, thereby transferring the domain knowledge from the source network to the target network. In addition, the proposed method combines the boosting-based deep representation transfer method to more fully utilize the source domain knowledge, so that the target network learns in both the reweighted source deep representation and target deep representation to further improve the generalization ability. The proposed method is applied to image classification and time series prediction tasks and achieves better transfer performance than other similar methods, which shows that the method has good practicability and generality in practical applications. • A deep representation-based transfer learning method is proposed for knowledge transfer between deep neural networks. • The method transfers domain-invariant representation by learning the similarity structure between source representations. • The method reduces the influence of unfavorable source representations through boosting-based representation transfer. • The computational process of the method is vectorized with comparable training efficiency to traditional fine-tuning. • The image classification and time series prediction experiments demonstrate the universal transfer capability of the method.
vSGX: Virtualizing SGX Enclaves on AMD SEV

The growing need of trusted execution environment (TEE) has boomed the development of hardware enclaves. However, current TEEs and their applications are tightly bound to the hardware implementation, hindering their compatibility across different platforms. This paper presents vSGX, a novel system to virtualize the execution of an Intel SGX enclave atop AMD SEV. The key idea is to interpose the execution of enclave instructions transparently to support the SGX ISA extensions, consolidate encrypted virtual memory of separated SEV virtual machines to create a single virtualized SGX-like address space, and provide attestations for the authenticity of the TEE and the integrity of enclave software with a trust chain rooted in the SEV hardware. By design, vSGX achieves a comparable level of security guarantees on SEV as that on Intel SGX. We have implemented vSGX and demonstrated it imposes reasonable performance overhead for SGX enclave execution.
MeshUp: Stateless Cache Side-channel Attack on CPU Mesh

Cache side-channel attacks lead to severe security threats to the settings where a CPU is shared across users, e.g., in the cloud. The majority of attacks rely on sensing the micro-architectural state changes made by victims, but this assumption can be invalidated by combining spatial (e.g., Intel CAT) and temporal isolation. In this work, we advance the state of cache side-channel attacks by showing stateless cache side-channel attacks on server-grade CPUs, that can bypass both spatial and temporal isolation. Unlike stateful cache side-channel attacks that rely on the timing difference between a cache hit or miss, our attack exploits the timing difference caused by the interconnect congestion. Specifically, to complete cache transactions, for Intel server CPUs, which use non-inclusive and mesh interconnect, cache lines would travel across cores via the CPU mesh and UPI interconnects. Nonetheless, the interconnects are shared by all cores, and cache isolation does not segregate the traffic. An attacker can generate traffic to contend with a victim on a link, measure the extra delay, deduce the memory access pattern of the victim’s program, and infer its sensitive data. Based on this idea, we implement MESHUP, a stateless cache side-channel against mesh interconnect, and test it against the existing RSA implementations of JDK for the cross-core attack and application fingerprinting for the the cross-CPU attack. We found the RSA private key used by a victim process can be partially recovered and the co-running application can be inferred at high accuracy.
WTAGRAPH: Web Tracking and Advertising Detection using Graph Neural Networks

Web tracking and advertising (WTA) nowadays are ubiquitously performed on the web, continuously compromising users’ privacy. Existing defense solutions, such as widely deployed blocking tools based on filter lists and alternative machine learning based solutions proposed in prior research, have limitations in terms of accuracy and effectiveness. In this work, we propose WTAGRAPH, a web tracking and advertising detection framework based on Graph Neural Networks (GNNs). We first construct an attributed homogenous multi-graph (AHMG) that represents HTTP network traffic, and formulate web tracking and advertising detection as a task of GNN-based edge representation learning and classification in AHMG. We then design four components in WTAGRAPH so that it can (1) collect HTTP network traffic, DOM, and JavaScript data, (2) construct AHMG and extract corresponding edge and node features, (3) build a GNN model for edge representation learning and WTA detection in the transductive learning setting, and (4) use a pre-trained GNN model for WTA detection in the inductive learning setting. We evaluate WTAGRAPH on a dataset collected from Alexa Top 10K websites, and show that WTAGRAPH can effectively detect WTA requests in both transductive and inductive learning settings. Manual verification results indicate that WTAGRAPH can detect new WTA requests that are missed by filter lists and recognize non-WTA requests that are mistakenly labeled by filter lists. Our ablation analysis, evasion evaluation, and real-time evaluation show that WTAGRAPH can have a competitive performance with flexible deployment options in practice.
A Systematic Look at Ciphertext Side Channels on AMD SEV-SNP

Hardware-assisted memory encryption offers strong confidentiality guarantees for trusted execution environments like Intel SGX and AMD SEV. However, a recent study by Li et al. presented at USENIX Security 2021 has demonstrated the CipherLeaks attack, which monitors ciphertext changes in the special VMSA page. By leaking register values saved by the VM during context switches, they broke state-of-the-art constant-time cryptographic implementations, including RSA and ECDSA in the OpenSSL. In this paper, we perform a comprehensive study on the ciphertext side channels. Our work suggests that while the CipherLeaks attack targets only the VMSA page, a generic ciphertext side-channel attack may exploit the ciphertext leakage from any memory pages, including those for kernel data structures, stacks and heaps. As such, AMD’s existing countermeasures to the CipherLeaks attack, a firmware patch that introduces randomness into the ciphertext of the VMSA page, is clearly insufficient. The root cause of the leakage in AMD SEV’s memory encryption—the use of a stateless yet unauthenticated encryption mode and the unrestricted read accesses to the ciphertext of the encrypted memory—remains unfixed. Given the challenges faced by AMD to eradicate the vulnerability from the hardware design, we propose a set of software countermeasures to the ciphertext side channels, including patches to the OS kernel and cryptographic libraries. We are working closely with AMD to merge these changes into affected open-source projects.
Piccolo: Exposing Complex Backdoors in NLP Transformer Models

Backdoors can be injected to NLP models such that they misbehave when the trigger words or sentences appear in an input sample. Detecting such backdoors given only a subject model and a small number of benign samples is very challenging because of the unique nature of NLP applications, such as the discontinuity of pipeline and the large search space. Existing techniques work well for backdoors with simple triggers such as single character/word triggers but become less effective when triggers and models become complex (e.g., transformer models). We propose a new backdoor scanning technique. It transforms a subject model to an equivalent but differentiable form. It then uses optimization to invert a distribution of words denoting their likelihood in the trigger. It leverages a novel word discriminativity analysis to determine if the subject model is particularly discriminative for the presence of likely trigger words. Our evaluation on 3839 NLP models from the TrojAI competition and existing works with 7 state-of-art complex structures such as BERT and GPT, and 17 different attack types including two latest dynamic attacks, shows that our technique is highly effective, achieving over 0.9 detection accuracy in most scenarios and substantially outperforming two state-of-the-art scanners. Our submissions to TrojAI leaderboard achieve top performance in 2 out of the 3 rounds for NLP backdoor scanning.
Four Attacks and a Proof for Telegram

We study the use of symmetric cryptography in the MTProto 2.0 protocol, Telegram’s equivalent of the TLS record protocol. We give positive and negative results. On the one hand, we formally and in detail model a slight variant of Telegram’s “record protocol” and prove that it achieves security in a suitable bidirectional secure channel model, albeit under unstudied assumptions; this model itself advances the state-of-the-art for secure channels. On the other hand, we first motivate our modelling deviation from MTProto as deployed by giving two attacks – one of practical, one of theoretical interest – against MTProto without our modifications. We then also give a third attack exploiting timing side channels, of varying strength, in three official Telegram clients. On its own this attack is thwarted by the secrecy of salt and id fields that are established by Telegram’s key exchange protocol. To recover these, we chain the third attack with a fourth one against the implementation of the key exchange protocol on Telegram’s servers. In totality, our results provide the first comprehensive study of MTProto’s use of symmetric cryptography.
Counterfactual Explanations for Prediction and Diagnosis in XAI

We compared two sorts of explanations for decisions made by an AI system: counterfactual explanations about how an outcome could have been different in the past, and prefactual explanations about how it could be different in the future. We examined the effects of these alternative explanation strategies on the accuracy of users' judgments about the AI app's predictions about an outcome (inferred from information about the causes), compared to the accuracy of their judgments about the app's diagnoses of a cause (inferred from information about the outcome). The tasks were based on a simulated SmartAgriculture decision support system for grass growth outcomes on dairy farms in Experiment 1, and for an analogous alien planet domain in Experiment 2. The two experiments, with 243 participants, also tested users' confidence in their decisions, and their satisfaction with the explanations. Users made more accurate diagnoses of the presence of causes based on information about their outcome, compared to predictions of an outcome given information about the presence of causes. Their predictions and diagnoses were helped equally by counterfactual explanations and prefactual ones.
Asymmetric Global–Local Mutual Integration Network for RGBT Tracking

RGBT tracking as a solution in complex environments has gradually become a research hotspot. The powerful complementarity between RGB and thermal infrared data enables trackers to work 24/7. Existing works usually adopt the symmetric network structure that deploys the identical strategy to mine modalities with different properties, ignoring the heterogeneity among modalities. In this paper, we propose a novel asymmetric global-local mutual integration network via comprehensively considering symmetric structure, heterogeneity-based global association and inter-frame communication. It consists of asymmetric mode-distinguishing parallel structure, cross-modal global-local interaction, and inter-frame monitoring strategy. Specifically, the asymmetric mode-distinguishing parallel structure performs discriminative mining on the information of the two modalities by combining the discount module and the branch cement module, and extracts multi-scale cues through the multi-scale auxiliary module to handle the challenges of scale variation and small-size objects. Then, the global mining module is deployed in the cross-modal global-local interaction section to jointly perform intra-modal and inter-modal global correlation, while acting as the global complement to local feature extraction. Finally, the inter-frame monitoring strategy employs a fast optical flow algorithm to detect inter-frame displacement to assist the network better handle camera and fast object motion. Extensive experiments on GTOT, RGBT234 and LasHeR datasets adequately verify the effectiveness of the proposed network, and further ablation experiments also confirm the efficacy of the asymmetric structure and components.
Motion Guided Attention Learning for Self-Supervised 3D Human Action Recognition

3D human action recognition has received increasing attention due to its potential application in video surveillance equipment. To guarantee satisfactory performance, previous studies are mainly based on supervised methods, which have to add a large amount of manual annotation costs. In addition, general deep networks for video sequences suffer from heavy computational costs, thus cannot satisfy the basic requirement of embedded systems. In this paper, a novel Motion Guided Attention Learning (MG-AL) framework is proposed, which formulates the action representation learning as a self-supervised motion attention prediction problem. Specifically, MG-AL is a lightweight network. A set of simple motion priors (e.g., intra-joint variance, inter-frame deviation, intra-joint variance, and cross-joint covariance), which minimizes additional parameters and computational overhead, is regarded as a supervisory signal to guide the attention generation. The encoder is trained via predicting multiple self-attention tasks to capture action-specific feature representations. Extensive evaluations are performed on three challenging benchmark datasets (NTU-RGB+D 60, NTU-RGB+D 120 and NW-UCLA). The proposed method achieves superior performance compared to state-of-the-art methods, while having a very low computational cost.
Unsupervised domain adaptation in homogeneous distance space for person re-identification

Data distribution alignment and clustering-based self-training are two feasible solutions to tackle unsupervised domain adaptation (UDA) on person re-identification (re-ID). Most existing alignment-based methods solely learn the source domain decision boundaries and align the data distribution of the target domain to the source domain, thus the re-ID performance on the target domain completely depends on the shared decision boundaries and how well the alignment is performed. However, two domains can hardly be precisely aligned because of the label space discrepancy of two domains, resulting in poor target domain re-ID performance. Although clustering-based self-training approaches could learn independent decision boundaries on the pseudo-labelled target domain data, they ignore both the accurate ID-related information of the labelled source domain data and the underlying relations between two domains. To fully exploit the source domain data to learn discriminative target domain ID-related features, in this paper, we propose a novel cross-domain alignment method in the homogeneous distance space, which is constructed by the newly designed stair-stepping alignment (SSA) matcher. Such alignment method can be integrated into both alignment-based framework and clustering-based framework. Extensive experiments validate the effectiveness of our proposed alignment method in these two frameworks. We achieve superior performance when the proposed alignment module is integrated into the clustering-based framework. Codes will be available at: http://github.com/Dingyuan-Zheng/HDS.
Electronic health records and blockchain interoperability requirements: a scoping review

The purpose of this study was to conduct a scoping review of publications that explored blockchain technology in the context of interoperability and challenges of electronic health record (EHR) implementations. We synthesize the literature regarding standards and security, specifically regulation, regulatory operability, and conformance to standards. We review open practitioner questions that were not addressed in the studies as directions for further research.We conducted a literature search in the OVID databases (Medline and Embase) on terms blockchain, implementation, interoperability, EHRs, security, and standards. The search resulted in 152 nonduplicate, peer-reviewed manuscripts, of which 15 were relevant to our objective and included for synthesis.Based on the search results, we analyzed the adoption of blockchain technology in the healthcare systems and challenges to EHR implementation of blockchain. From the synthesized research, we categorized and reported compelling factors of blockchain for EHR integration using current knowledge on blockchain research standardization and architectural challenges.Our research showed promise in implementing blockchain technology associated with EHRs, especially with Health Information Exchanges. The studies relevant for both EHR (n = 5) and blockchain (n = 10) reported compelling factors and limitations of the architecture. Security (n = 4) and interoperability (n = 4) features were reported as compelling requirements with lingering challenges. Standardization literature (n = 3) reported implementation challenges.This study shows promise in implementing blockchain technology within EHR systems. The adoption is increasing; however, multiple implementation challenges remain from architectural perspectives (eg, scalability and performance), to security challenges (eg, legal requirements), and standard perspectives including patient-matching problems.