OpenRaiser/Intern-Atlas huggingface dataset:
Automatic Generation of Contrast Sets from Scene Graphs: Probing the Compositional Consistency of GQA
Venue: arXiv cs.CL 2021
Year: 2021

Recent works have shown that supervised models often exploit data artifacts to achieve good test scores while their performance severely degrades on samples outside their training distribution. Contrast sets (Gardneret al., 2020) quantify this phenomenon by perturbing test samples in a minimal way such that the output label is modified. While most contrast sets were created manually, requiring intensive annotation effort, we present a novel method which leverages rich semantic input representation to automatically generate contrast sets for the visual question answering task. Our method computes the answer of perturbed questions, thus vastly reducing annotation cost and enabling thorough evaluation of models' performance on various semantic aspects (e.g., spatial or relational reasoning). We demonstrate the effectiveness of our approach on the GQA dataset and its semantic scene graph image representation. We find that, despite GQA's compositionality and carefully balanced label distribution, two high-performing models drop 13-17% in accuracy compared to the original test set. Finally, we show that our automatic perturbation can be applied to the training set to mitigate the degradation in performance, opening the door to more robust models.
Assessing Smart Contracts Security Technical Debts
Venue: arXiv cs.CR 2021
Year: 2021

Smart contracts are self-enforcing agreements that are employed to exchange assets without the approval of trusted third parties. This feature has encouraged various sectors to make use of smart contracts when transacting. Experience shows that many deployed contracts are vulnerable to exploitation due to their poor design, which allows attackers to steal valuable assets from the involved parties. Therefore, an assessment approach that allows developers to recognise the consequences of deploying vulnerable contracts is needed. In this paper, we propose a debt-aware approach for assessing security design vulnerabilities in smart contracts. Our assessment approach involves two main steps: (i) identification of design vulnerabilities using security analysis techniques and (ii) an estimation of the ramifications of the identified vulnerabilities leveraging the technical debt metaphor, its principal and interest. We use examples of vulnerable contracts to demonstrate the applicability of our approach. The results show that our assessment approach increases the visibility of security design issues. It also allows developers to concentrate on resolving smart contract vulnerabilities through technical debt impact analysis and prioritisation. Developers can use our approach to inform the design of more secure contracts and for reducing unintentional debts caused by a lack of awareness of security issues.
On the Role of Images for Analyzing Claims in Social Media
Venue: arXiv cs.SI 2021
Year: 2021

Fake news is a severe problem in social media. In this paper, we present an empirical study on visual, textual, and multimodal models for the tasks of claim, claim check-worthiness, and conspiracy detection, all of which are related to fake news detection. Recent work suggests that images are more influential than text and often appear alongside fake text. To this end, several multimodal models have been proposed in recent years that use images along with text to detect fake news on social media sites like Twitter. However, the role of images is not well understood for claim detection, specifically using transformer-based textual and multimodal models. We investigate state-of-the-art models for images, text (Transformer-based), and multimodal information for four different datasets across two languages to understand the role of images in the task of claim and conspiracy detection.
A Robust Extrinsic Calibration Framework for Vehicles with Unscaled Sensors
Venue: arXiv cs.RO 2021
Year: 2021

Accurate extrinsic sensor calibration is essential for both autonomous vehicles and robots. Traditionally this is an involved process requiring calibration targets, known fiducial markers and is generally performed in a lab. Moreover, even a small change in the sensor layout requires recalibration. With the anticipated arrival of consumer autonomous vehicles, there is demand for a system which can do this automatically, after deployment and without specialist human expertise. To solve these limitations, we propose a flexible framework which can estimate extrinsic parameters without an explicit calibration stage, even for sensors with unknown scale. Our first contribution builds upon standard hand-eye calibration by jointly recovering scale. Our second contribution is that our system is made robust to imperfect and degenerate sensor data, by collecting independent sets of poses and automatically selecting those which are most ideal. We show that our approach's robustness is essential for the target scenario. Unlike previous approaches, ours runs in real time and constantly estimates the extrinsic transform. For both an ideal experimental setup and a real use case, comparison against these approaches shows that we outperform the state-of-the-art. Furthermore, we demonstrate that the recovered scale may be applied to the full trajectory, circumventing the need for scale estimation via sensor fusion.
Fourier Transform of Percoll Gradients Boosts CNN Classification of Hereditary Hemolytic Anemias
Venue: arXiv cs.CV 2021
Year: 2021

Hereditary hemolytic anemias are genetic disorders that affect the shape and density of red blood cells. Genetic tests currently used to diagnose such anemias are expensive and unavailable in the majority of clinical labs. Here, we propose a method for identifying hereditary hemolytic anemias based on a standard biochemistry method, called Percoll gradient, obtained by centrifuging a patient's blood. Our hybrid approach consists on using spatial data-driven features, extracted with a convolutional neural network and spectral handcrafted features obtained from fast Fourier transform. We compare late and early feature fusion with AlexNet and VGG16 architectures. AlexNet with late fusion of spectral features performs better compared to other approaches. We achieved an average F1-score of 88% on different classes suggesting the possibility of diagnosing of hereditary hemolytic anemias from Percoll gradients. Finally, we utilize Grad-CAM to explore the spatial features used for classification.
On Guillotine Separable Packings for the Two-dimensional Geometric Knapsack Problem
Venue: arXiv cs.DS 2021
Year: 2021

In two-dimensional geometric knapsack problem, we are given a set of n axis-aligned rectangular items and an axis-aligned square-shaped knapsack. Each item has integral width, integral height and an associated integral profit. The goal is to find a (non-overlapping axis-aligned) packing of a maximum profit subset of rectangles into the knapsack. A well-studied and frequently used constraint in practice is to allow only packings that are guillotine separable, i.e., every rectangle in the packing can be obtained by recursively applying a sequence of edge-to-edge axis-parallel cuts that do not intersect any item of the solution. In this paper we study approximation algorithms for the geometric knapsack problem under guillotine cut constraints. We present polynomial time (1 + ε)-approximation algorithms for the cases with and without allowing rotations by 90 degrees, assuming that all input numeric data are polynomially bounded in n. In comparison, the best-known approximation factor for this setting is 3 + ε [Jansen-Zhang, SODA 2004], even in the cardinality case where all items have the same profit. Our main technical contribution is a structural lemma which shows that any guillotine packing can be converted into another structured guillotine packing with almost the same profit. In this packing, each item is completely contained in one of a constant number of boxes and L-shaped regions, inside which the items are placed by a simple greedy routine. In particular, we provide a clean sufficient condition when such a packing obeys the guillotine cut constraints which might be useful for other settings where these constraints are imposed.
Towards Automated Metamorphic Test Identification for Ocean System Models
Venue: arXiv cs.SE 2021
Year: 2021

Metamorphic testing seeks to verify software in the absence of test oracles. Our application domain is ocean system modeling, where test oracles rarely exist, but where symmetries of the simulated physical systems are known. The input data set is large owing to the requirements of the application domain. This paper presents work in progress for the automated generation of metamorphic test scenarios using machine learning. We extended our previously proposed method [1] to identify metamorphic relations with reduced computational complexity. Initially, we represent metamorphic relations as identity maps. We construct a cost function that minimizes for identifying a metamorphic relation orthogonal to previously found metamorphic relations and penalize for the identity map. A machine learning algorithm is used to identify all possible metamorphic relations minimizing the defined cost function. We propose applying dimensionality reduction techniques to identify attributes in the input which have high variance among the identified metamorphic relations. We apply mutation on these selected attributes to identify distinct metamorphic relations with reduced computational complexity. For experimental evaluation, we subject the two implementations of an ocean-modeling application to the proposed method to present the use of metamorphic relations to test the two implementations of this application.
Capacity Achieving Uncoded PIR Protocol based on Combinatorial Designs
Venue: arXiv cs.IT 2021
Year: 2021

In this paper we study the problem of private information retrieval where a user seeks to retrieve one of the $F$ files from a cluster of $N$ non-colluding servers without revealing the identity of the requested file. In our setting the servers are storage constrained in that they can only store a fraction $μ=t/N$ of each file. Furthermore, we assume that the files are stored in an uncoded fashion. The rate of a PIR protocol is defined as the ratio of the file size and the total number of bits downloaded. The maximum achievable rate is referred to as capacity. It was previously shown that there are capacity achieving PIR protocols when the file size is $N^F$ and complete files were stored on all the servers. These results were further extended for the case when servers store only a fraction of each file. However, the subpacketization $v$ of the files required is exponential in the number of servers $N$. We propose a novel uncoded PIR protocol based on combinatorial designs that are also capacity achieving when the file size is $v \times t^F$. Our protocol has linear subpacketization in the number of servers in contrast to previous work in storage constrained uncoded PIR schemes. In the proposed PIR protocol, the given system is projected to multiple instances of reduced systems with replicated servers having full storage capacity. The subfiles stored in these various instances are separately retrieved and lifted to solve the PIR problem for the original system.
Molecular Index Modulation using Convolutional Neural Networks
Venue: arXiv cs.ET 2021
Year: 2021

As the potential of molecular communication via diffusion (MCvD) systems at nano-scale communication increases, designing molecular schemes robust to the inevitable effects of molecular interference has become of vital importance. There are numerous molecular approaches in literature aiming to mitigate the effects of interference, namely inter-symbol interference. Moreover, for molecular multiple-input-multiple-output systems, interference among antennas, namely inter-link interference, becomes of significance. Inspired by the state-of-the-art performances of machine learning algorithms on making decisions, we propose a novel approach of a convolutional neural network (CNN)-based architecture. The proposed approach is for a uniquely-designed molecular multiple-input-single-output topology in order to alleviate the damaging effects of molecular interference. In this study, we compare the performance of the proposed network with that of an index modulation approach and a symbol-by-symbol maximum likelihood estimation and show that the proposed method yields better performance.
CrowdSim: A Hybrid Simulation Model for Failure Prediction in Crowdsourced Software Development
Venue: arXiv cs.SE 2021
Year: 2021

A typical crowdsourcing software development(CSD) marketplace consists of a list of software tasks as service demands and a pool of freelancer developers as service suppliers. Highly dynamic and competitive CSD market places may result in task failure due to unforeseen risks, such as increased competition over shared worker supply, or uncertainty associated with workers' experience and skills, and so on. To improve CSD effectiveness, it is essential to better understand and plan with respect to dynamic worker characteristics and risks associated with CSD processes. In this paper, we present a hybrid simulation model, CrowdSim, to forecast crowdsourcing task failure risk in competitive CSD platforms. CrowdSim is composed of three layered components: the macro-level reflects the overall crowdsourcing platform based on system dynamics,the meso-level represents the task life cycle based on discrete event simulation, and the micro-level models the crowd workers' decision-making processes based on agent-based simulation. CrowdSim is evaluated through three CSD decision scenarios to demonstrate its effectiveness, using a real-world historical dataset and the results demonstrate CrowdSim's potential in empowering crowdsourcing managers to explore crowdsourcing outcomes with respect to different task scheduling options.
Collaborative Filtering Approach to Link Prediction
Venue: arXiv cs.SI 2021
Year: 2021

Link prediction is a fundamental challenge in network science. Among various methods, local similarity indices are widely used for their high cost-performance. However, the performance is less robust: for some networks local indices are highly competitive to state-of-the-art algorithms while for some other networks they are very poor. Inspired by techniques developed for recommender systems, we propose an enhancement framework for local indices based on collaborative filtering (CF). Considering the delicate but important difference between personalized recommendation and link prediction, we further propose an improved framework named as self-included collaborative filtering (SCF). The SCF framework significantly improved the accuracy and robustness of well-known local indices. The combination of SCF framework and a simple local index can produce an index with competitive performance and much lower complexity compared with elaborately-designed state-of-the-art algorithms.
DomainNet: Homograph Detection for Data Lake Disambiguation
Venue: arXiv cs.DB 2021
Year: 2021

Modern data lakes are deeply heterogeneous in the vocabulary that is used to describe data. We study a problem of disambiguation in data lakes: how can we determine if a data value occurring more than once in the lake has different meanings and is therefore a homograph? While word and entity disambiguation have been well studied in computational linguistics, data management and data science, we show that data lakes provide a new opportunity for disambiguation of data values since they represent a massive network of interconnected values. We investigate to what extent this network can be used to disambiguate values. DomainNet uses network-centrality measures on a bipartite graph whose nodes represent values and attributes to determine, without supervision, if a value is a homograph. A thorough experimental evaluation demonstrates that state-of-the-art techniques in domain discovery cannot be re-purposed to compete with our method. Specifically, using a domain discovery method to identify homographs has a precision and a recall of 38% versus 69% with our method on a synthetic benchmark. By applying a network-centrality measure to our graph representation, DomainNet achieves a good separation between homographs and data values with a unique meaning. On a real data lake our top-200 precision is 89%.
IRLI: Iterative Re-partitioning for Learning to Index
Venue: arXiv cs.IR 2021
Year: 2021

Neural models have transformed the fundamental information retrieval problem of mapping a query to a giant set of items. However, the need for efficient and low latency inference forces the community to reconsider efficient approximate near-neighbor search in the item space. To this end, learning to index is gaining much interest in recent times. Methods have to trade between obtaining high accuracy while maintaining load balance and scalability in distributed settings. We propose a novel approach called IRLI (pronounced `early'), which iteratively partitions the items by learning the relevant buckets directly from the query-item relevance data. Furthermore, IRLI employs a superior power-of-$k$-choices based load balancing strategy. We mathematically show that IRLI retrieves the correct item with high probability under very natural assumptions and provides superior load balancing. IRLI surpasses the best baseline's precision on multi-label classification while being $5x$ faster on inference. For near-neighbor search tasks, the same method outperforms the state-of-the-art Learned Hashing approach NeuralLSH by requiring only ~ {1/6}^th of the candidates for the same recall. IRLI is both data and model parallel, making it ideal for distributed GPU implementation. We demonstrate this advantage by indexing 100 million dense vectors and surpassing the popular FAISS library by >10% on recall.
Understanding Generalization in Adversarial Training via the Bias-Variance Decomposition
Venue: arXiv cs.LG 2021
Year: 2021

Adversarially trained models exhibit a large generalization gap: they can interpolate the training set even for large perturbation radii, but at the cost of large test error on clean samples. To investigate this gap, we decompose the test risk into its bias and variance components and study their behavior as a function of adversarial training perturbation radii ($\varepsilon$). We find that the bias increases monotonically with $\varepsilon$ and is the dominant term in the risk. Meanwhile, the variance is unimodal as a function of $\varepsilon$, peaking near the interpolation threshold for the training set. This characteristic behavior occurs robustly across different datasets and also for other robust training procedures such as randomized smoothing. It thus provides a test for proposed explanations of the generalization gap. We find that some existing explanations fail this test--for instance, by predicting a monotonically increasing variance curve. This underscores the power of bias-variance decompositions in modern settings-by providing two measurements instead of one, they can rule out more explanations than test accuracy alone. We also show that bias and variance can provide useful guidance for scalably reducing the generalization gap, highlighting pre-training and unlabeled data as promising routes.
CheXbreak: Misclassification Identification for Deep Learning Models Interpreting Chest X-rays
Venue: arXiv cs.CV 2021
Year: 2021

A major obstacle to the integration of deep learning models for chest x-ray interpretation into clinical settings is the lack of understanding of their failure modes. In this work, we first investigate whether there are patient subgroups that chest x-ray models are likely to misclassify. We find that patient age and the radiographic finding of lung lesion, pneumothorax or support devices are statistically relevant features for predicting misclassification for some chest x-ray models. Second, we develop misclassification predictors on chest x-ray models using their outputs and clinical features. We find that our best performing misclassification identifier achieves an AUROC close to 0.9 for most diseases. Third, employing our misclassification identifiers, we develop a corrective algorithm to selectively flip model predictions that have high likelihood of misclassification at inference time. We observe F1 improvement on the prediction of Consolidation (0.008 [95% CI 0.005, 0.010]) and Edema (0.003, [95% CI 0.001, 0.006]). By carrying out our investigation on ten distinct and high-performing chest x-ray models, we are able to derive insights across model architectures and offer a generalizable framework applicable to other medical imaging tasks.
Model Extraction and Adversarial Transferability, Your BERT is Vulnerable!
Venue: arXiv cs.CL 2021
Year: 2021

Natural language processing (NLP) tasks, ranging from text classification to text generation, have been revolutionised by the pre-trained language models, such as BERT. This allows corporations to easily build powerful APIs by encapsulating fine-tuned BERT models for downstream tasks. However, when a fine-tuned BERT model is deployed as a service, it may suffer from different attacks launched by malicious users. In this work, we first present how an adversary can steal a BERT-based API service (the victim/target model) on multiple benchmark datasets with limited prior knowledge and queries. We further show that the extracted model can lead to highly transferable adversarial attacks against the victim model. Our studies indicate that the potential vulnerabilities of BERT-based API services still hold, even when there is an architectural mismatch between the victim model and the attack model. Finally, we investigate two defence strategies to protect the victim model and find that unless the performance of the victim model is sacrificed, both model ex-traction and adversarial transferability can effectively compromise the target models
Secure Watermark for Deep Neural Networks with Multi-task Learning
Venue: arXiv cs.CR 2021
Year: 2021

Deep neural networks are playing an important role in many real-life applications. After being trained with abundant data and computing resources, a deep neural network model providing service is endowed with economic value. An important prerequisite in commercializing and protecting deep neural networks is the reliable identification of their genuine author. To meet this goal, watermarking schemes that embed the author's identity information into the networks have been proposed. However, current schemes can hardly meet all the necessary requirements for securely proving the authorship and mostly focus on models for classification. To explicitly meet the formal definitions of the security requirements and increase the applicability of deep neural network watermarking schemes, we propose a new framework based on multi-task learning. By treating the watermark embedding as an extra task, most of the security requirements are explicitly formulated and met with well-designed regularizers, the rest is guaranteed by using components from cryptography. Moreover, a decentralized verification protocol is proposed to standardize the ownership verification. The experiment results show that the proposed scheme is flexible, secure, and robust, hence a promising candidate in deep learning model protection.