OpenRaiser/Intern-Atlas huggingface dataset:
PathologyBERT -- Pre-trained Vs. A New Transformer Language Model for Pathology Domain
Venue: arXiv cs.CL 2022
Year: 2022

Pathology text mining is a challenging task given the reporting variability and constant new findings in cancer sub-type definitions. However, successful text mining of a large pathology database can play a critical role to advance 'big data' cancer research like similarity-based treatment selection, case identification, prognostication, surveillance, clinical trial screening, risk stratification, and many others. While there is a growing interest in developing language models for more specific clinical domains, no pathology-specific language space exist to support the rapid data-mining development in pathology space. In literature, a few approaches fine-tuned general transformer models on specialized corpora while maintaining the original tokenizer, but in fields requiring specialized terminology, these models often fail to perform adequately. We propose PathologyBERT - a pre-trained masked language model which was trained on 347,173 histopathology specimen reports and publicly released in the Huggingface repository. Our comprehensive experiments demonstrate that pre-training of transformer model on pathology corpora yields performance improvements on Natural Language Understanding (NLU) and Breast Cancer Diagnose Classification when compared to nonspecific language models.
Bootstrapping Text Anonymization Models with Distant Supervision
Venue: arXiv cs.CL 2022
Year: 2022

We propose a novel method to bootstrap text anonymization models based on distant supervision. Instead of requiring manually labeled training data, the approach relies on a knowledge graph expressing the background information assumed to be publicly available about various individuals. This knowledge graph is employed to automatically annotate text documents including personal data about a subset of those individuals. More precisely, the method determines which text spans ought to be masked in order to guarantee $k$-anonymity, assuming an adversary with access to both the text documents and the background information expressed in the knowledge graph. The resulting collection of labeled documents is then used as training data to fine-tune a pre-trained language model for text anonymization. We illustrate this approach using a knowledge graph extracted from Wikidata and short biographical texts from Wikipedia. Evaluation results with a RoBERTa-based model and a manually annotated collection of 553 summaries showcase the potential of the approach, but also unveil a number of issues that may arise if the knowledge graph is noisy or incomplete. The results also illustrate that, contrary to most sequence labeling problems, the text anonymization task may admit several alternative solutions.
Structural Dropout for Model Width Compression
Venue: arXiv cs.LG 2022
Year: 2022

Existing ML models are known to be highly over-parametrized, and use significantly more resources than required for a given task. Prior work has explored compressing models offline, such as by distilling knowledge from larger models into much smaller ones. This is effective for compression, but does not give an empirical method for measuring how much the model can be compressed, and requires additional training for each compressed model. We propose a method that requires only a single training session for the original model and a set of compressed models. The proposed approach is a "structural" dropout that prunes all elements in the hidden state above a randomly chosen index, forcing the model to learn an importance ordering over its features. After learning this ordering, at inference time unimportant features can be pruned while retaining most accuracy, reducing parameter size significantly. In this work, we focus on Structural Dropout for fully-connected layers, but the concept can be applied to any kind of layer with unordered features, such as convolutional or attention layers. Structural Dropout requires no additional pruning/retraining, but requires additional validation for each possible hidden sizes. At inference time, a non-expert can select a memory versus accuracy trade-off that best suits their needs, across a wide range of highly compressed versus more accurate models.
Blockaid: Data Access Policy Enforcement for Web Applications
Venue: arXiv cs.DB 2022
Year: 2022

Modern web applications serve large amounts of sensitive user data, access to which is typically governed by data-access policies. Enforcing such policies is crucial to preventing improper data access, and prior work has proposed many enforcement mechanisms. However, these prior methods either alter application semantics or require adopting a new programming model; the former can result in unexpected application behavior, while the latter cannot be used with existing web frameworks. Blockaid is an access-policy enforcement system that preserves application semantics and is compatible with existing web frameworks. It intercepts database queries from the application, attempts to verify that each query is policy-compliant, and blocks queries that are not. It verifies policy compliance using SMT solvers and generalizes and caches previous compliance decisions for better performance. We show that Blockaid supports existing web applications while requiring minimal code changes and adding only modest overheads.
Formal limitations of sample-wise information-theoretic generalization bounds
Venue: arXiv cs.LG 2022
Year: 2022

Some of the tightest information-theoretic generalization bounds depend on the average information between the learned hypothesis and a single training example. However, these sample-wise bounds were derived only for expected generalization gap. We show that even for expected squared generalization gap no such sample-wise information-theoretic bounds exist. The same is true for PAC-Bayes and single-draw bounds. Remarkably, PAC-Bayes, single-draw and expected squared generalization gap bounds that depend on information in pairs of examples exist.
Beyond General Purpose Machine Translation: The Need for Context-specific Empirical Research to Design for Appropriate User Trust
Venue: arXiv cs.HC 2022
Year: 2022

Machine Translation (MT) has the potential to help people overcome language barriers and is widely used in high-stakes scenarios, such as in hospitals. However, in order to use MT reliably and safely, users need to understand when to trust MT outputs and how to assess the quality of often imperfect translation results. In this paper, we discuss research directions to support users to calibrate trust in MT systems. We share findings from an empirical study in which we conducted semi-structured interviews with 20 clinicians to understand how they communicate with patients across language barriers, and if and how they use MT systems. Based on our findings, we advocate for empirical research on how MT systems are used in practice as an important first step to addressing the challenges in building appropriate trust between users and MT tools.
Efficient Learning of Interpretable Classification Rules
Venue: arXiv cs.LG 2022
Year: 2022

Machine learning has become omnipresent with applications in various safety-critical domains such as medical, law, and transportation. In these domains, high-stake decisions provided by machine learning necessitate researchers to design interpretable models, where the prediction is understandable to a human. In interpretable machine learning, rule-based classifiers are particularly effective in representing the decision boundary through a set of rules comprising input features. The interpretability of rule-based classifiers is in general related to the size of the rules, where smaller rules are considered more interpretable. To learn such a classifier, the brute-force direct approach is to consider an optimization problem that tries to learn the smallest classification rule that has close to maximum accuracy. This optimization problem is computationally intractable due to its combinatorial nature and thus, the problem is not scalable in large datasets. To this end, in this paper we study the triangular relationship among the accuracy, interpretability, and scalability of learning rule-based classifiers. The contribution of this paper is an interpretable learning framework IMLI, that is based on maximum satisfiability (MaxSAT) for synthesizing classification rules expressible in proposition logic. Despite the progress of MaxSAT solving in the last decade, the straightforward MaxSAT-based solution cannot scale. Therefore, we incorporate an efficient incremental learning technique inside the MaxSAT formulation by integrating mini-batch learning and iterative rule-learning. In our experiments, IMLI achieves the best balance among prediction accuracy, interpretability, and scalability. As an application, we deploy IMLI in learning popular interpretable classifiers such as decision lists and decision sets.
Unified Distributed Environment
Venue: arXiv cs.LG 2022
Year: 2022

We propose Unified Distributed Environment (UDE), an environment virtualization toolkit for reinforcement learning research. UDE is designed to integrate environments built on any simulation platform such as Gazebo, Unity, Unreal, and OpenAI Gym. Through environment virtualization, UDE enables offloading the environment for execution on a remote machine while still maintaining a unified interface. The UDE interface is designed to support multi-agent by default. With environment virtualization and its interface design, the agent policies can be trained in multiple machines for a multi-agent environment. Furthermore, UDE supports integration with existing major RL toolkits for researchers to leverage the benefits. This paper discusses the components of UDE and its design decisions.
No-regret learning for repeated non-cooperative games with lossy bandits
Venue: arXiv cs.LG 2022
Year: 2022

This paper considers no-regret learning for repeated continuous-kernel games with lossy bandit feedback. Since it is difficult to give the explicit model of the utility functions in dynamic environments, the players' action can only be learned with bandit feedback. Moreover, because of unreliable communication channels or privacy protection, the bandit feedback may be lost or dropped at random. Therefore, we study the asynchronous online learning strategy of the players to adaptively adjust the next actions for minimizing the long-term regret loss. The paper provides a novel no-regret learning algorithm, called Online Gradient Descent with lossy bandits (OGD-lb). We first give the regret analysis for concave games with differentiable and Lipschitz utilities. Then we show that the action profile converges to a Nash equilibrium with probability 1 when the game is also strictly monotone. We further provide the mean square convergence rate $\mathcal{O}\left(k^{-2\min\{β, 1/6\}}\right)$ when the game is $β-$ strongly monotone. In addition, we extend the algorithm to the case when the loss probability of the bandit feedback is unknown, and prove its almost sure convergence to Nash equilibrium for strictly monotone games. Finally, we take the resource management in fog computing as an application example, and carry out numerical experiments to empirically demonstrate the algorithm performance.
RiCS: A 2D Self-Occlusion Map for Harmonizing Volumetric Objects
Venue: arXiv cs.CV 2022
Year: 2022

There have been remarkable successes in computer vision with deep learning. While such breakthroughs show robust performance, there have still been many challenges in learning in-depth knowledge, like occlusion or predicting physical interactions. Although some recent works show the potential of 3D data in serving such context, it is unclear how we efficiently provide 3D input to the 2D models due to the misalignment in dimensionality between 2D and 3D. To leverage the successes of 2D models in predicting self-occlusions, we design Ray-marching in Camera Space (RiCS), a new method to represent the self-occlusions of foreground objects in 3D into a 2D self-occlusion map. We test the effectiveness of our representation on the human image harmonization task by predicting shading that is coherent with a given background image. Our experiments demonstrate that our representation map not only allows us to enhance the image quality but also to model temporally coherent complex shadow effects compared with the simulation-to-real and harmonization methods, both quantitatively and qualitatively. We further show that we can significantly improve the performance of human parts segmentation networks trained on existing synthetic datasets by enhancing the harmonization quality with our method.
Comparative evaluation of community-aware centrality measures
Venue: arXiv cs.SI 2022
Year: 2022

Influential nodes play a critical role in boosting or curbing spreading phenomena in complex networks. Numerous centrality measures have been proposed for identifying and ranking the nodes according to their importance. Classical centrality measures rely on various local or global properties of the nodes. They do not take into account the network community structure. Recently, a growing number of researches have shifted to community-aware centrality measures. Indeed, it is a ubiquitous feature in a vast majority of real-world networks. In the literature, the focus is on designing community-aware centrality measures. However, up to now, there is no systematic evaluation of their effectiveness. This study fills this gap. It allows answering which community-aware centrality measure should be used in practical situations. We investigate seven influential community-aware centrality measures in an epidemic spreading process scenario using the Susceptible-Infected-Recovered (SIR) model on a set of fifteen real-world networks. Results show that generally, the correlation between community-aware centrality measures is low. Furthermore, in a multiple-spreader problem, when resources are available, targeting distant hubs using Modularity Vitality is more effective. However, with limited resources, diffusion expands better through bridges, especially in networks with a medium or strong community structure.
State-Based Behavior Modeling in Software and Systems Engineering
Venue: arXiv cs.SE 2022
Year: 2022

The design of complex man-made systems mostly involves a conceptual modeling phase; therefore, it is important to ensure an appropriate analysis method for these models. A key concept for such analysis is the development of a diagramming technique (e.g., UML) because diagrams can describe entities and processes and emphasize important aspects of the systems being described. The analysis also includes an examination of ontological concepts such as states and events, which are used as a basis for the modeling process. Studying fundamental concepts allows us to understand more deeply the relationship between these concepts and modeling frameworks. In this paper, we critically analyze the classic definition of a state utilizing the thinging machine (TM) model. States in state machine diagrams are considered the appropriate basis for modeling system behavioral aspects. Despite its wide application in hardware design, the integration of a state machine model into a software system s modeling requirements increased the difficulty of graphical representation (e.g., integration between structural and behavioral diagrams). To understand such a problem, in this paper, we project (create an equivalent representation of) states in TM machines. As a case study, we re-modeled a state machine of an assembly line system in a TM. The outcome is a complicated picture of assembly line behavior. Therefore, as an alternative solution, we re-modeled the assembly line based solely on the TM. This new model presents a clear contrast between state-based modeling of assembly line behavior and the TM approach. The TM modeling seems more systematic than its counterpart, the state machine, and its notions are well defined. In a TM, states are just compound events. A model of a more complex system than the one in the assembly line has strengthened such a conclusion.
Minimal-Perimeter Lattice Animals and the Constant-Isomer Conjecture
Venue: arXiv cs.CG 2022
Year: 2022

We consider minimal-perimeter lattice animals, providing a set of conditions which are sufficient for a lattice to have the property that inflating all minimal-perimeter animals of a certain size yields (without repetitions) all minimal-perimeter animals of a new, larger size. We demonstrate this result on the two-dimensional square and hexagonal lattices. In addition, we characterize the sizes of minimal-perimeter animals on these lattices that are not created by inflating members of another set of minimal-perimeter animals.
VRCockpit: Mitigating Simulator Sickness in VR Games Using Multiple Egocentric 2D View Frames
Venue: arXiv cs.HC 2022
Year: 2022

Virtual reality head-mounted displays (VR HMDs) have become a popular platform for gaming. However, simulator sickness (SS) is still an impediment to VR's wider adoption, particularly in gaming. It can induce strong discomfort and impair players' immersion, performance, and enjoyment. Researchers have explored techniques to mitigate SS. While these techniques have been shown to help lessen SS, they may not be applicable to games because they cannot be easily integrated into various types of games without impacting gameplay, immersion, and performance. In this research, we introduce a new SS mitigation technique, VRCockpit. VRCockpit is a visual technique that surrounds the player with four 2D views, one for each cardinal direction, that show 2D copies of the areas of the 3D environment around the player. To study its effectiveness, we conducted two different experiments, one with a car racing game, followed by a first-person shooter game. Our results show that VRCockpit has the potential to mitigate SS and still allows players to have the same level of immersion and gameplay performance.
CDEdit: A Highly Applicable Redactable Blockchain with Controllable Editing Privilege and Diversified Editing Types
Venue: arXiv cs.CR 2022
Year: 2022

Redactable blockchains allow modifiers or voting committees with modification privileges to edit the data on the chain. Trapdoor holders in chameleon-based hash redactable blockchains can quickly compute hash collisions for arbitrary data, and without breaking the link of the hash-chain. However, chameleon-based hash redactable blockchain schemes have difficulty solving the problem of multi-level editing requests and competing for modification privileges. In this paper, we propose CDEdit, a highly applicable redactable blockchain with controllable editing privilege and diversified editing types. The proposed scheme increases the cost of invalid or malicious requests by paying the deposit on each edit request. At the same time, the editing privilege is subdivided into request, modification, and verification privileges, and the modification privilege token is distributed efficiently to prevent the abuse of the modification privilege and collusion attacks. We use chameleon hashes with ephemeral trapdoor (CHET) and ciphertext policy attribute-based encryption (CP-ABE) to implement two editing types of transaction-level and block-level, and present a practical instantiation and security analysis. Finally, the implementation and evaluation show that our scheme only costs low-performance overhead and is suitable for multi-level editing requests and modification privilege competition scenarios.
Transformer Scale Gate for Semantic Segmentation
Venue: arXiv cs.CV 2022
Year: 2022

Effectively encoding multi-scale contextual information is crucial for accurate semantic segmentation. Existing transformer-based segmentation models combine features across scales without any selection, where features on sub-optimal scales may degrade segmentation outcomes. Leveraging from the inherent properties of Vision Transformers, we propose a simple yet effective module, Transformer Scale Gate (TSG), to optimally combine multi-scale features.TSG exploits cues in self and cross attentions in Vision Transformers for the scale selection. TSG is a highly flexible plug-and-play module, and can easily be incorporated with any encoder-decoder-based hierarchical vision Transformer architecture. Extensive experiments on the Pascal Context and ADE20K datasets demonstrate that our feature selection strategy achieves consistent gains.
Spiking Approximations of the MaxPooling Operation in Deep SNNs
Venue: arXiv cs.NE 2022
Year: 2022

Spiking Neural Networks (SNNs) are an emerging domain of biologically inspired neural networks that have shown promise for low-power AI. A number of methods exist for building deep SNNs, with Artificial Neural Network (ANN)-to-SNN conversion being highly successful. MaxPooling layers in Convolutional Neural Networks (CNNs) are an integral component to downsample the intermediate feature maps and introduce translational invariance, but the absence of their hardware-friendly spiking equivalents limits such CNNs' conversion to deep SNNs. In this paper, we present two hardware-friendly methods to implement Max-Pooling in deep SNNs, thus facilitating easy conversion of CNNs with MaxPooling layers to SNNs. In a first, we also execute SNNs with spiking-MaxPooling layers on Intel's Loihi neuromorphic hardware (with MNIST, FMNIST, & CIFAR10 dataset); thus, showing the feasibility of our approach.
SystemMatch: optimizing preclinical drug models to human clinical outcomes via generative latent-space matching
Venue: arXiv cs.LG 2022
Year: 2022

Translating the relevance of preclinical models ($\textit{in vitro}$, animal models, or organoids) to their relevance in humans presents an important challenge during drug development. The rising abundance of single-cell genomic data from human tumors and tissue offers a new opportunity to optimize model systems by their similarity to targeted human cell types in disease. In this work, we introduce SystemMatch to assess the fit of preclinical model systems to an $\textit{in sapiens}$ target population and to recommend experimental changes to further optimize these systems. We demonstrate this through an application to developing $\textit{in vitro}$ systems to model human tumor-derived suppressive macrophages. We show with held-out $\textit{in vivo}$ controls that our pipeline successfully ranks macrophage subpopulations by their biological similarity to the target population, and apply this analysis to rank a series of 18 $\textit{in vitro}$ macrophage systems perturbed with a variety of cytokine stimulations. We extend this analysis to predict the behavior of 66 $\textit{in silico}$ model systems generated using a perturbational autoencoder and apply a $k$-medoids approach to recommend a subset of these model systems for further experimental development in order to fully explore the space of possible perturbations. Through this use case, we demonstrate a novel approach to model system development to generate a system more similar to human biology.