OpenRaiser/Intern-Atlas huggingface dataset:
BERTops: Studying BERT Representations under a Topological Lens
Venue: arXiv cs.LG 2022
Year: 2022

Proposing scoring functions to effectively understand, analyze and learn various properties of high dimensional hidden representations of large-scale transformer models like BERT can be a challenging task. In this work, we explore a new direction by studying the topological features of BERT hidden representations using persistent homology (PH). We propose a novel scoring function named "persistence scoring function (PSF)" which: (i) accurately captures the homology of the high-dimensional hidden representations and correlates well with the test set accuracy of a wide range of datasets and outperforms existing scoring metrics, (ii) captures interesting post fine-tuning "per-class" level properties from both qualitative and quantitative viewpoints, (iii) is more stable to perturbations as compared to the baseline functions, which makes it a very robust proxy, and (iv) finally, also serves as a predictor of the attack success rates for a wide category of black-box and white-box adversarial attack methods. Our extensive correlation experiments demonstrate the practical utility of PSF on various NLP tasks relevant to BERT.
The effect of the rapid growth of covid-19 publications on citation indicators
Venue: arXiv cs.DL 2022
Year: 2022

A concern has been raised that "covidization" of research would cause an overemphasizes on covid-19 and pandemics at the expense of other research. The rapid growth of publications related to the Covid-19 pandemic renders a vast amount of citations from this literature. This growth may affect bibliometric indicators. In this paper I explored how the growth of covid-19 publications influences bibliometric indicators commonly used in university rankings, research evaluation and research allocation, namely the field normalized citation score and the journal impact factor. I found that the burst of publications in the early stage of the covid-19 pandemic affects field-normalized citation scores and will affect the journal impact factor. Publications unrelated to covid-19 are also heavily affected. I conclude that there is a considerable risk to draw misleading conclusions from citation indicators spanning over the beginning of the covid-19 pandemic, in particular when time series are used and when the biomedical literature is assessed.
A Survey on Uncertainty Toolkits for Deep Learning
Venue: arXiv cs.LG 2022
Year: 2022

The success of deep learning (DL) fostered the creation of unifying frameworks such as tensorflow or pytorch as much as it was driven by their creation in return. Having common building blocks facilitates the exchange of, e.g., models or concepts and makes developments easier replicable. Nonetheless, robust and reliable evaluation and assessment of DL models has often proven challenging. This is at odds with their increasing safety relevance, which recently culminated in the field of "trustworthy ML". We believe that, among others, further unification of evaluation and safeguarding methodologies in terms of toolkits, i.e., small and specialized framework derivatives, might positively impact problems of trustworthiness as well as reproducibility. To this end, we present the first survey on toolkits for uncertainty estimation (UE) in DL, as UE forms a cornerstone in assessing model reliability. We investigate 11 toolkits with respect to modeling and evaluation capabilities, providing an in-depth comparison for the three most promising ones, namely Pyro, Tensorflow Probability, and Uncertainty Quantification 360. While the first two provide a large degree of flexibility and seamless integration into their respective framework, the last one has the larger methodological scope.
The Chaotic State of UK Drone Regulation
Venue: arXiv cs.CY 2022
Year: 2022

In December 2020 the law for drone pilots and unmanned aerial vehicle (UAV) use went into a transition phase in preparation for new EU international UAV regulation. That EU regulation comes into full effect as the transition periods defined in the United Kingdom's Civil Aviation Authority Air Policy CAP722 expire during December 2022 (CAA, 2020). However, international homologation regulation will not address the patchwork of inconsistent drone use regulations that exist in the United Kingdom from the layering of local and subordinate authority byelaws over UK aviation law. We provide an extensive review of local authority regulation of drone use on public open and green spaces, finding that many local authorities are unaware of the issues being created through: (i) inappropriately couched or poorly framed byelaws; (ii) multiple byelaws covering the same area by virtue of overlapping jurisdictions; or (iii) the lack readily identifiable policies for drone use on public land. Overregulation, inconsistent regulation and regulatory disharmony are causing confusion for recreational drone enthusiasts such that it is never clear which public or crown-owned open and green spaces they are allowed to, or prohibited from, flying. While the government and local authorities might like them to, drones are not going away. Therefore, we conclude, the easiest way to ensure citizens stay within the bounds of drone law that is intended to ensure public safety, is to make that law comprehensible, consistent and easy to comply with.
HTTPA/2: a Trusted End-to-End Protocol for Web Services
Venue: arXiv cs.CR 2022
Year: 2022

With the advent of cloud computing and the Internet, the commercialized website becomes capable of providing more web services, such as software as a service (SaaS) or function as a service (FaaS), for great user experiences. Undoubtedly, web services have been thriving in popularity that will continue growing to serve modern human life. As expected, there came the ineluctable need for preserving privacy, enhancing security, and building trust. However, HTTPS alone cannot provide a remote attestation for building trust with web services, which remains lacking in trust. At the same time, cloud computing is actively adopting the use of TEEs and will demand a web-based protocol for remote attestation with ease of use. Here, we propose HTTPA/2 as an upgraded version of HTTP-Attestable (HTTPA) by augmenting existing HTTP to enable end-to-end trusted communication between endpoints at layer 7 (L7). HTTPA/2 allows for L7 message protection without relying on TLS. In practice, HTTPA/2 is designed to be compatible with the in-network processing of the modern cloud infrastructure, including L7 gateway, L7 load balancer, caching, etc. We envision that \acs{httpa}/2 will further enable trustworthy web services and trustworthy AI applications in the future, accelerating the transformation of the web-based digital world to be more trustworthy.
OPT: Open Pre-trained Transformer Language Models
Venue: arXiv cs.CL 2022
Year: 2022

Large language models, which are often trained for hundreds of thousands of compute days, have shown remarkable capabilities for zero- and few-shot learning. Given their computational cost, these models are difficult to replicate without significant capital. For the few that are available through APIs, no access is granted to the full model weights, making them difficult to study. We present Open Pre-trained Transformers (OPT), a suite of decoder-only pre-trained transformers ranging from 125M to 175B parameters, which we aim to fully and responsibly share with interested researchers. We show that OPT-175B is comparable to GPT-3, while requiring only 1/7th the carbon footprint to develop. We are also releasing our logbook detailing the infrastructure challenges we faced, along with code for experimenting with all of the released models.
Neurocompositional computing: From the Central Paradox of Cognition to a new generation of AI systems
Venue: arXiv cs.AI 2022
Year: 2022

What explains the dramatic progress from 20th-century to 21st-century AI, and how can the remaining limitations of current AI be overcome? The widely accepted narrative attributes this progress to massive increases in the quantity of computational and data resources available to support statistical learning in deep artificial neural networks. We show that an additional crucial factor is the development of a new type of computation. Neurocompositional computing adopts two principles that must be simultaneously respected to enable human-level cognition: the principles of Compositionality and Continuity. These have seemed irreconcilable until the recent mathematical discovery that compositionality can be realized not only through discrete methods of symbolic computing, but also through novel forms of continuous neural computing. The revolutionary recent progress in AI has resulted from the use of limited forms of neurocompositional computing. New, deeper forms of neurocompositional computing create AI systems that are more robust, accurate, and comprehensible.
Saliency map using features derived from spiking neural networks of primate visual cortex
Venue: arXiv cs.CV 2022
Year: 2022

We propose a framework inspired by biological vision systems to produce saliency maps of digital images. Well-known computational models for receptive fields of areas in the visual cortex that are specialized for color and orientation perception are used. To model the connectivity between these areas we use the CARLsim library which is a spiking neural network(SNN) simulator. The spikes generated by CARLsim, then serve as extracted features and input to our saliency detection algorithm. This new method of saliency detection is described and applied to benchmark images.
Paving the Way for Mature Secondary Research: The Seven Types of Literature Review
Venue: arXiv cs.SE 2022
Year: 2022

Confusion over different kinds of secondary research, and their divergent purposes, is undermining the effectiveness and usefulness of secondary studies in software engineering. This short paper therefore explains the differences between ad hoc review, case survey, critical review, meta-analysis (aka systematic literature review), meta-synthesis (aka thematic analysis), rapid review and scoping review (aka systematic mapping study). These definitions and associated guidelines help researchers better select and describe their literature reviews, while helping reviewers select more appropriate evaluation criteria.
Propositional Equality for Gradual Dependently Typed Programming
Venue: arXiv cs.PL 2022
Year: 2022

Gradual dependent types can help with the incremental adoption of dependently typed code by providing a principled semantics for imprecise types and proofs, where some parts have been omitted. Current theories of gradual dependent types, though, lack a central feature of type theory: propositional equality. Lennon-Bertrand et al. show that, when the reflexive proof $\mathit{refl}$ is the only closed value of an equality type, a gradual extension of CIC with propositional equality violates static observational equivalences. Extensionally-equal functions should be indistinguishable at run time, but the combination of equality and type imprecision allows for contexts that distinguish extensionally-equal but syntactically-different functions. This work presents a gradually typed language that supports propositional equality. We avoid the above issues by devising an equality type where $\mathit{refl}$ is not the only closed inhabitant. Instead, each equality proof carries a term that is at least as precise as the equated terms, acting as a witness of their plausible equality. These witnesses track partial type information as a program runs, raising errors when that information shows that two equated terms are undeniably inconsistent. Composition of type information is internalized as a construct of the language, and is deferred for function bodies whose evaluation is blocked by variables. By deferring, we ensure that extensionally equal functions compose without error, thereby preventing contexts from distinguishing them. We describe the challenges of designing consistency and precision relations for this system, along with solutions to these challenges. Finally, we prove important metatheory: type-safety, conservative embedding of CIC, canonicity, and the gradual guarantees of Siek et al.
Scalable Private Decision Tree Evaluation with Sublinear Communication
Venue: arXiv cs.CR 2022
Year: 2022

Private decision tree evaluation (PDTE) allows a decision tree holder to run a secure protocol with a feature provider. By running the protocol, the feature provider will learn a classification result. Nothing more is revealed to either party. In most existing PDTE protocols, the required communication grows exponentially with the tree's depth $d$, which is highly inefficient for large trees. This shortcoming motivated us to design a sublinear PDTE protocol with $O(d)$ communication complexity. The core of our construction is a shared oblivious selection (SOS) functionality, allowing two parties to perform a secret-shared oblivious read operation from an array. We provide two SOS protocols, both of which achieve sublinear communication and propose optimizations to further improve their efficiency. Our sublinear PDTE protocol is based on the proposed SOS functionality and we prove its security under a semi-honest adversary. We compare our protocol with the state-of-the-art, in terms of communication and computation, under various network settings. The performance evaluation shows that our protocol is practical and more scalable over large trees than existing solutions.
SemAttack: Natural Textual Attacks via Different Semantic Spaces
Venue: arXiv cs.CL 2022
Year: 2022

Recent studies show that pre-trained language models (LMs) are vulnerable to textual adversarial attacks. However, existing attack methods either suffer from low attack success rates or fail to search efficiently in the exponentially large perturbation space. We propose an efficient and effective framework SemAttack to generate natural adversarial text by constructing different semantic perturbation functions. In particular, SemAttack optimizes the generated perturbations constrained on generic semantic spaces, including typo space, knowledge space (e.g., WordNet), contextualized semantic space (e.g., the embedding space of BERT clusterings), or the combination of these spaces. Thus, the generated adversarial texts are more semantically close to the original inputs. Extensive experiments reveal that state-of-the-art (SOTA) large-scale LMs (e.g., DeBERTa-v2) and defense strategies (e.g., FreeLB) are still vulnerable to SemAttack. We further demonstrate that SemAttack is general and able to generate natural adversarial texts for different languages (e.g., English and Chinese) with high attack success rates. Human evaluations also confirm that our generated adversarial texts are natural and barely affect human performance. Our code is publicly available at https://github.com/AI-secure/SemAttack.
On Ranking Consistency of Pre-ranking Stage
Venue: arXiv cs.IR 2022
Year: 2022

Industrial ranking systems, such as advertising systems, rank items by aggregating multiple objectives into one final objective to satisfy user demand and commercial intent. Cascade architecture, composed of retrieval, pre-ranking, and ranking stages, is usually adopted to reduce the computational cost. Each stage may employ various models for different objectives and calculate the final objective by aggregating these models' outputs. The multi-stage ranking strategy causes a new problem - the ranked lists of the ranking stage and previous stages may be inconsistent. For example, items that should be ranked at the top of the ranking stage may be ranked at the bottom of previous stages. In this paper, we focus on the \textbf{ranking consistency} between the pre-ranking and ranking stages. Specifically, we formally define the problem of ranking consistency and propose the Ranking Consistency Score (RCS) metric for evaluation. We demonstrate that ranking consistency has a direct impact on online performance. Compared with the traditional evaluation manner that mainly focuses on the individual ranking quality of every objective, RCS considers the ranking consistency of the fused final objective, which is more proper for evaluation. Finally, to improve the ranking consistency, we propose several methods from the perspective of sample selection and learning algorithms. Experimental results on one of the biggest industrial E-commerce platforms in China validate the efficacy of the proposed metrics and methods.
DrugEHRQA: A Question Answering Dataset on Structured and Unstructured Electronic Health Records For Medicine Related Queries
Venue: arXiv cs.AI 2022
Year: 2022

This paper develops the first question answering dataset (DrugEHRQA) containing question-answer pairs from both structured tables and unstructured notes from a publicly available Electronic Health Record (EHR). EHRs contain patient records, stored in structured tables and unstructured clinical notes. The information in structured and unstructured EHRs is not strictly disjoint: information may be duplicated, contradictory, or provide additional context between these sources. Our dataset has medication-related queries, containing over 70,000 question-answer pairs. To provide a baseline model and help analyze the dataset, we have used a simple model (MultimodalEHRQA) which uses the predictions of a modality selection network to choose between EHR tables and clinical notes to answer the questions. This is used to direct the questions to the table-based or text-based state-of-the-art QA model. In order to address the problem arising from complex, nested queries, this is the first time Relation-Aware Schema Encoding and Linking for Text-to-SQL Parsers (RAT-SQL) has been used to test the structure of query templates in EHR data. Our goal is to provide a benchmark dataset for multi-modal QA systems, and to open up new avenues of research in improving question answering over EHR structured data by using context from unstructured clinical data.
GRAPHYP: A Scientific Knowledge Graph with Manifold Subnetworks of Communities. Detection of Scholarly Disputes in Adversarial Information Routes
Venue: arXiv cs.AI 2022
Year: 2022

The cognitive manifold of published content is currently expanding in all areas of science. However, Scientific Knowledge Graphs (SKGs) only provide poor pictures of the adversarial directions and scientific controversies that feed the production of knowledge. In this Article, we tackle the understanding of the design of the information space of a cognitive representation of research activities, and of related bottlenecks that affect search interfaces, in the mapping of structured objects into graphs. We propose, with SKG GRAPHYP, a novel graph designed geometric architecture which optimizes both the detection of the knowledge manifold of "cognitive communities", and the representation of alternative paths to adversarial answers to a research question, for instance in the context of academic disputes. With a methodology for designing "Manifold Subnetworks of Cognitive Communities", GRAPHYP provides a classification of distinct search paths in a research field. Users are detected from the variety of their search practices and classified in "Cognitive communities" from the analysis of the search history of their logs of scientific documentation. The manifold of practices is expressed from metrics of differentiated uses by triplets of nodes shaped into symmetrical graph subnetworks, with the following three parameters: Mass, Intensity, and Variety.
Predicting Issue Types with seBERT
Venue: arXiv cs.SE 2022
Year: 2022

Pre-trained transformer models are the current state-of-the-art for natural language models processing. seBERT is such a model, that was developed based on the BERT architecture, but trained from scratch with software engineering data. We fine-tuned this model for the NLBSE challenge for the task of issue type prediction. Our model dominates the baseline fastText for all three issue types in both recall and precisio} to achieve an overall F1-score of 85.7%, which is an increase of 4.1% over the baseline.
Predicting Loose-Fitting Garment Deformations Using Bone-Driven Motion Networks
Venue: arXiv cs.GR 2022
Year: 2022

We present a learning algorithm that uses bone-driven motion networks to predict the deformation of loose-fitting garment meshes at interactive rates. Given a garment, we generate a simulation database and extract virtual bones from simulated mesh sequences using skin decomposition. At runtime, we separately compute low- and high-frequency deformations in a sequential manner. The low-frequency deformations are predicted by transferring body motions to virtual bones' motions, and the high-frequency deformations are estimated leveraging the global information of virtual bones' motions and local information extracted from low-frequency meshes. In addition, our method can estimate garment deformations caused by variations of the simulation parameters (e.g., fabric's bending stiffness) using an RBF kernel ensembling trained networks for different sets of simulation parameters. Through extensive comparisons, we show that our method outperforms state-of-the-art methods in terms of prediction accuracy of mesh deformations by about 20% in RMSE and 10% in Hausdorff distance and STED. The code and data are available at https://github.com/non-void/VirtualBones.
Neural Combinatorial Optimization: a New Player in the Field
Venue: arXiv cs.AI 2022
Year: 2022

Neural Combinatorial Optimization attempts to learn good heuristics for solving a set of problems using Neural Network models and Reinforcement Learning. Recently, its good performance has encouraged many practitioners to develop neural architectures for a wide variety of combinatorial problems. However, the incorporation of such algorithms in the conventional optimization framework has raised many questions related to their performance and the experimental comparison with other methods such as exact algorithms, heuristics and metaheuristics. This paper presents a critical analysis on the incorporation of algorithms based on neural networks into the classical combinatorial optimization framework. Subsequently, a comprehensive study is carried out to analyse the fundamental aspects of such algorithms, including performance, transferability, computational cost and generalization to larger-sized instances. To that end, we select the Linear Ordering Problem as a case of study, an NP-hard problem, and develop a Neural Combinatorial Optimization model to optimize it. Finally, we discuss how the analysed aspects apply to a general learning framework, and suggest new directions for future work in the area of Neural Combinatorial Optimization algorithms.
A hybrid multi-object segmentation framework with model-based B-splines for microbial single cell analysis
Venue: arXiv cs.CV 2022
Year: 2022

In this paper, we propose a hybrid approach for multi-object microbial cell segmentation. The approach combines an ML-based detection with a geometry-aware variational-based segmentation using B-splines that are parametrized based on a geometric model of the cell shape. The detection is done first using YOLOv5. In a second step, each detected cell is segmented individually. Thus, the segmentation only needs to be done on a per-cell basis, which makes it amenable to a variational approach that incorporates prior knowledge on the geometry. Here, the contour of the segmentation is modelled as closed uniform cubic B-spline, whose control points are parametrized using the known cell geometry. Compared to purely ML-based segmentation approaches, which need accurate segmentation maps as training data that are very laborious to produce, our method just needs bounding boxes as training data. Still, the proposed method performs on par with ML-based segmentation approaches usually used in this context. We study the performance of the proposed method on time-lapse microscopy data of Corynebacterium glutamicum.