OpenRaiser/Intern-Atlas huggingface dataset:
Are Neural Ranking Models Robust?
Venue: arXiv cs.IR 2022
Year: 2022

Recently, we have witnessed the bloom of neural ranking models in the information retrieval (IR) field. So far, much effort has been devoted to developing effective neural ranking models that can generalize well on new data. There has been less attention paid to the robustness perspective. Unlike the effectiveness which is about the average performance of a system under normal purpose, robustness cares more about the system performance in the worst case or under malicious operations instead. When a new technique enters into the real-world application, it is critical to know not only how it works in average, but also how would it behave in abnormal situations. So we raise the question in this work: Are neural ranking models robust? To answer this question, firstly, we need to clarify what we refer to when we talk about the robustness of ranking models in IR. We show that robustness is actually a multi-dimensional concept and there are three ways to define it in IR: 1) The performance variance under the independent and identically distributed (I.I.D.) setting; 2) The out-of-distribution (OOD) generalizability; and 3) The defensive ability against adversarial operations. The latter two definitions can be further specified into two different perspectives respectively, leading to 5 robustness tasks in total. Based on this taxonomy, we build corresponding benchmark datasets, design empirical experiments, and systematically analyze the robustness of several representative neural ranking models against traditional probabilistic ranking models and learning-to-rank (LTR) models. The empirical results show that there is no simple answer to our question. While neural ranking models are less robust against other IR models in most cases, some of them can still win 1 out of 5 tasks. This is the first comprehensive study on the robustness of neural ranking models.
Automatic Gaze Analysis: A Survey of Deep Learning based Approaches
Venue: arXiv cs.CV 2022
Year: 2022

Eye gaze analysis is an important research problem in the field of Computer Vision and Human-Computer Interaction. Even with notable progress in the last 10 years, automatic gaze analysis still remains challenging due to the uniqueness of eye appearance, eye-head interplay, occlusion, image quality, and illumination conditions. There are several open questions, including what are the important cues to interpret gaze direction in an unconstrained environment without prior knowledge and how to encode them in real-time. We review the progress across a range of gaze analysis tasks and applications to elucidate these fundamental questions, identify effective methods in gaze analysis, and provide possible future directions. We analyze recent gaze estimation and segmentation methods, especially in the unsupervised and weakly supervised domain, based on their advantages and reported evaluation metrics. Our analysis shows that the development of a robust and generic gaze analysis method still needs to address real-world challenges such as unconstrained setup and learning with less supervision. We conclude by discussing future research directions for designing a real-world gaze analysis system that can propagate to other domains including Computer Vision, Augmented Reality (AR), Virtual Reality (VR), and Human Computer Interaction (HCI). Project Page: https://github.com/i-am-shreya/EyeGazeSurvey}{https://github.com/i-am-shreya/EyeGazeSurvey
Fair Decision-Making for Food Inspections
Venue: arXiv cs.LG 2022
Year: 2022

Data and algorithms are essential and complementary parts of a large-scale decision-making process. However, their injudicious use can lead to unforeseen consequences, as has been observed by researchers and activists alike in the recent past. In this paper, we revisit the application of predictive models by the Chicago Department of Public Health to schedule restaurant inspections and prioritize the detection of critical food code violations. We perform the first analysis of the model's fairness to the population served by the restaurants in terms of average time to find a critical violation. We find that the model treats inspections unequally based on the sanitarian who conducted the inspection and that, in turn, there are geographic disparities in the benefits of the model. We examine four alternate methods of model training and two alternative ways of scheduling using the model and find that the latter generate more desirable results. The challenges from this application point to important directions for future work around fairness with collective entities rather than individuals, the use of critical violations as a proxy, and the disconnect between fair classification and fairness in the dynamic scheduling system.
Efficient Local Planning with Linear Function Approximation
Venue: arXiv cs.LG 2022
Year: 2022

We study query and computationally efficient planning algorithms with linear function approximation and a simulator. We assume that the agent only has local access to the simulator, meaning that the agent can only query the simulator at states that have been visited before. This setting is more practical than many prior works on reinforcement learning with a generative model. We propose two algorithms, named confident Monte Carlo least square policy iteration (Confident MC-LSPI) and confident Monte Carlo Politex (Confident MC-Politex) for this setting. Under the assumption that the Q-functions of all policies are linear in known features of the state-action pairs, we show that our algorithms have polynomial query and computational costs in the dimension of the features, the effective planning horizon, and the targeted sub-optimality, while these costs are independent of the size of the state space. One technical contribution of our work is the introduction of a novel proof technique that makes use of a virtual policy iteration algorithm. We use this method to leverage existing results on $\ell_\infty$-bounded approximate policy iteration to show that our algorithm can learn the optimal policy for the given initial state even only with local access to the simulator. We believe that this technique can be extended to broader settings beyond this work.
Multimodal analysis of the predictability of hand-gesture properties
Venue: arXiv cs.HC 2022
Year: 2022

Embodied conversational agents benefit from being able to accompany their speech with gestures. Although many data-driven approaches to gesture generation have been proposed in recent years, it is still unclear whether such systems can consistently generate gestures that convey meaning. We investigate which gesture properties (phase, category, and semantics) can be predicted from speech text and/or audio using contemporary deep learning. In extensive experiments, we show that gesture properties related to gesture meaning (semantics and category) are predictable from text features (time-aligned FastText embeddings) alone, but not from prosodic audio features, while rhythm-related gesture properties (phase) on the other hand can be predicted from audio features better than from text. These results are encouraging as they indicate that it is possible to equip an embodied agent with content-wise meaningful co-speech gestures using a machine-learning model.
SHACL: A Description Logic in Disguise
Venue: arXiv cs.LO 2022
Year: 2022

SHACL is a W3C-proposed language for expressing structural constraints on RDF graphs. In recent years, SHACL's popularity has risen quickly. This rise in popularity comes with questions related to its place in the semantic web, particularly about its relation to OWL (the de facto standard for expressing ontological information on the web) and description logics (which form the formal foundations of OWL). We answer these questions by arguing that SHACL is in fact a description logic. On the one hand, our answer is surprisingly simple, some might even say obvious. But, on the hand, our answer is also controversial. By resolving this issue once and for all, we establish the field of description logics as the solid formal foundations of SHACL.
VulnEx: Exploring Open-Source Software Vulnerabilities in Large Development Organizations to Understand Risk Exposure
Venue: arXiv cs.SE 2022
Year: 2022

The prevalent usage of open-source software (OSS) has led to an increased interest in resolving potential third-party security risks by fixing common vulnerabilities and exposures (CVEs). However, even with automated code analysis tools in place, security analysts often lack the means to obtain an overview of vulnerable OSS reuse in large software organizations. In this design study, we propose VulnEx (Vulnerability Explorer), a tool to audit entire software development organizations. We introduce three complementary table-based representations to identify and assess vulnerability exposures due to OSS, which we designed in collaboration with security analysts. The presented tool allows examining problematic projects and applications (repositories), third-party libraries, and vulnerabilities across a software organization. We show the applicability of our tool through a use case and preliminary expert feedback.
Low-Resource Adaptation of Open-Domain Generative Chatbots
Venue: arXiv cs.CL 2022
Year: 2022

Recent work building open-domain chatbots has demonstrated that increasing model size improves performance. On the other hand, latency and connectivity considerations dictate the move of digital assistants on the device. Giving a digital assistant like Siri, Alexa, or Google Assistant the ability to discuss just about anything leads to the need for reducing the chatbot model size such that it fits on the user's device. We demonstrate that low parameter models can simultaneously retain their general knowledge conversational abilities while improving in a specific domain. Additionally, we propose a generic framework that accounts for variety in question types, tracks reference throughout multi-turn conversations, and removes inconsistent and potentially toxic responses. Our framework seamlessly transitions between chatting and performing transactional tasks, which will ultimately make interactions with digital assistants more human-like. We evaluate our framework on 1 internal and 4 public benchmark datasets using both automatic (Perplexity) and human (SSA - Sensibleness and Specificity Average) evaluation metrics and establish comparable performance while reducing model parameters by 90%.
Adaptive numerical simulations with Trixi.jl: A case study of Julia for scientific computing
Venue: arXiv cs.MS 2022
Year: 2022

We present Trixi.jl, a Julia package for adaptive high-order numerical simulations of hyperbolic partial differential equations. Utilizing Julia's strengths, Trixi.jl is extensible, easy to use, and fast. We describe the main design choices that enable these features and compare Trixi.jl with a mature open source Fortran code that uses the same numerical methods. We conclude with an assessment of Julia for simulation-focused scientific computing, an area that is still dominated by traditional high-performance computing languages such as C, C++, and Fortran.
Human Pose and Shape Estimation from Single Polarization Images
Venue: arXiv cs.CV 2022
Year: 2022

This paper focuses on a new problem of estimating human pose and shape from single polarization images. Polarization camera is known to be able to capture the polarization of reflected lights that preserves rich geometric cues of an object surface. Inspired by the recent applications in surface normal reconstruction from polarization images, in this paper, we attempt to estimate human pose and shape from single polarization images by leveraging the polarization-induced geometric cues. A dedicated two-stage pipeline is proposed: given a single polarization image, stage one (Polar2Normal) focuses on the fine detailed human body surface normal estimation; stage two (Polar2Shape) then reconstructs clothed human shape from the polarization image and the estimated surface normal. To empirically validate our approach, a dedicated dataset (PHSPD) is constructed, consisting of over 500K frames with accurate pose and parametric shape annotations. Empirical evaluations on this real-world dataset as well as a synthetic dataset, SURREAL, demonstrate the effectiveness of our approach. It suggests polarization camera as a promising alternative to the more conventional RGB camera for human pose and shape estimation.
Using Cyber Terrain in Reinforcement Learning for Penetration Testing
Venue: arXiv cs.LG 2022
Year: 2022

Reinforcement learning (RL) has been applied to attack graphs for penetration testing, however, trained agents do not reflect reality because the attack graphs lack operational nuances typically captured within the intelligence preparation of the battlefield (IPB) that include notions of (cyber) terrain. In particular, current practice constructs attack graphs exclusively using the Common Vulnerability Scoring System (CVSS) and its components. We present methods for constructing attack graphs using notions from IPB on cyber terrain analysis of obstacles, avenues of approach, key terrain, observation and fields of fire, and cover and concealment. We demonstrate our methods on an example where firewalls are treated as obstacles and represented in (1) the reward space and (2) the state dynamics. We show that terrain analysis can be used to bring realism to attack graphs for RL.
Metall: A Persistent Memory Allocator For Data-Centric Analytics
Venue: arXiv cs.DC 2022
Year: 2022

Data analytics applications transform raw input data into analytics-specific data structures before performing analytics. Unfortunately, such data ingestion step is often more expensive than analytics. In addition, various types of NVRAM devices are already used in many HPC systems today. Such devices will be useful for storing and reusing data structures beyond a single process life cycle. We developed Metall, a persistent memory allocator built on top of the memory-mapped file mechanism. Metall enables applications to transparently allocate custom C++ data structures into various types of persistent memories. Metall incorporates a concise and high-performance memory management algorithm inspired by Supermalloc and the rich C++ interface developed by Boost.Interprocess library. On a dynamic graph construction workload, Metall achieved up to 11.7x and 48.3x performance improvements over Boost.Interprocess and memkind (PMEM kind), respectively. We also demonstrate Metall's high adaptability by integrating Metall into a graph processing framework, GraphBLAS Template Library. This study's outcomes indicate that Metall will be a strong tool for accelerating future large-scale data analytics by allowing applications to leverage persistent memory efficiently.
APReL: A Library for Active Preference-based Reward Learning Algorithms
Venue: arXiv cs.LG 2022
Year: 2022

Reward learning is a fundamental problem in human-robot interaction to have robots that operate in alignment with what their human user wants. Many preference-based learning algorithms and active querying techniques have been proposed as a solution to this problem. In this paper, we present APReL, a library for active preference-based reward learning algorithms, which enable researchers and practitioners to experiment with the existing techniques and easily develop their own algorithms for various modules of the problem. APReL is available at https://github.com/Stanford-ILIAD/APReL.
Federated Asymptotics: a model to compare federated learning algorithms
Venue: arXiv cs.LG 2022
Year: 2022

We propose an asymptotic framework to analyze the performance of (personalized) federated learning algorithms. In this new framework, we formulate federated learning as a multi-criterion objective, where the goal is to minimize each client's loss using information from all of the clients. We analyze a linear regression model where, for a given client, we may theoretically compare the performance of various algorithms in the high-dimensional asymptotic limit. This asymptotic multi-criterion approach naturally models the high-dimensional, many-device nature of federated learning. These tools make fairly precise predictions about the benefits of personalization and information sharing in federated scenarios -- at least in our (stylized) model -- including that Federated Averaging with simple client fine-tuning achieves the same asymptotic risk as the more intricate meta-learning and proximal-regularized approaches and outperforming Federated Averaging without personalization. We evaluate these predictions on federated versions of the EMNIST, CIFAR-100, Shakespeare, and Stack Overflow datasets, where the experiments corroborate the theoretical predictions, suggesting such frameworks may provide a useful guide to practical algorithmic development.
The Visibility Center of a Simple Polygon
Venue: arXiv cs.CG 2022
Year: 2022

We introduce the \emph{visibility center} of a set of points inside a polygon -- a point $c_V$ such that the maximum geodesic distance from $c_V$ to see any point in the set is minimized. For a simple polygon of $n$ vertices and a set of $m$ points inside it, we give an $O((n+m) \log {(n+m)})$ time algorithm to find the visibility center. We find the visibility center of \emph{all} points in a simple polygon in $O(n \log n)$ time. Our algorithm reduces the visibility center problem to the problem of finding the geodesic center of a set of half-polygons inside a polygon, which is of independent interest. We give an $O((n+k) \log (n+k))$ time algorithm for this problem, where $k$ is the number of half-polygons.
Channel Estimation for Extremely Large-Scale MIMO: Far-Field or Near-Field?
Venue: arXiv cs.IT 2022
Year: 2022

Extremely large-scale multiple-input-multiple-output (XL-MIMO) with hybrid precoding is a promising technique to meet the high data rate requirements for future 6G communications. To realize efficient hybrid precoding, it is essential to obtain accurate channel state information. Existing channel estimation algorithms with low pilot overhead heavily rely on the channel sparsity in the angle domain, which is achieved by the classical far-field planar wavefront assumption. However, due to the non-negligible near-field spherical wavefront property in XL-MIMO systems, this channel sparsity in the angle domain is not available anymore, and thus existing far-field channel estimation schemes will suffer from severe performance loss. To address this problem, in this paper we study the near-field channel estimation by exploiting the polar-domain sparse representation of the near-field XL-MIMO channel. Specifically, unlike the classical angle-domain representation that only considers the angle information of the channel, we propose a new polar-domain representation, which simultaneously accounts for both the angle and distance information. In this way, the near-field channel also exhibits sparsity in the polar domain. By exploiting the channel sparsity in the polar domain, we propose the on-grid and off-grid near-field channel estimation schemes for XL-MIMO. Firstly, an on-grid polar-domain simultaneous orthogonal matching pursuit (P-SOMP) algorithm is proposed to efficiently estimate the near-field channel. Furthermore, to solve the resolution limitation of the on-grid P-SOMP algorithm, an off-grid polar-domain simultaneous iterative gridless weighted (P-SIGW) algorithm is proposed to improve the estimation accuracy, where the parameters of the near-field channel are directly estimated. Finally, numerical results are provided to verify the effectiveness of the proposed schemes.
Two-Timescale Design for Reconfigurable Intelligent Surface-Aided Massive MIMO Systems with Imperfect CSI
Venue: arXiv cs.IT 2022
Year: 2022

This paper investigates the two-timescale transmission design for reconfigurable intelligent surface (RIS)-aided massive multiple-input multiple-output (MIMO) systems, where the beamforming at the base station (BS) is adapted to the rapidly-changing instantaneous channel state information (CSI), while the passive beamforming at the RIS is adapted to the slowly-changing statistical CSI. Specifically, we first propose a linear minimum mean square error (LMMSE) estimator to obtain the aggregated channel from the users to the BS in each channel coherence interval. Based on the estimated channel, we apply the low-complexity maximal ratio combining (MRC) beamforming at the BS, and then derive the ergodic achievable rate in a closed form expression. To draw design insights, we perform a detailed theoretical analysis departing from the derived ergodic achievable rate. If the BS-RIS channel is Rician distributed, we prove that the transmit power can be scaled proportionally to $1/M$, as the number of BS antennas, $M$, grows to infinity while maintaining a non-zero rate. If the BS-RIS channel is Rayleigh distributed, the transmit power can be scaled either proportionally to $1/\sqrt{M}$ as $M$ grows large, or proportionally to $1/N$ as the number of reflecting elements, $N$, grows large, while still maintaining a non-zero rate. By capitalizing on the derived expression of the data rate under the statistical knowledge of the CSI, we maximize the minimum user rate by designing the passive beamforming at the RIS. Numerical results confirm that, even in the presence of imperfect CSI, the integration of an RIS in massive MIMO systems results in promising performance gains. In addition, the obtained results reveal that it is favorable to place the RIS close to the users rather than close to the BS.
AdapterHub Playground: Simple and Flexible Few-Shot Learning with Adapters
Venue: arXiv cs.CL 2022
Year: 2022

The open-access dissemination of pretrained language models through online repositories has led to a democratization of state-of-the-art natural language processing (NLP) research. This also allows people outside of NLP to use such models and adapt them to specific use-cases. However, a certain amount of technical proficiency is still required which is an entry barrier for users who want to apply these models to a certain task but lack the necessary knowledge or resources. In this work, we aim to overcome this gap by providing a tool which allows researchers to leverage pretrained models without writing a single line of code. Built upon the parameter-efficient adapter modules for transfer learning, our AdapterHub Playground provides an intuitive interface, allowing the usage of adapters for prediction, training and analysis of textual data for a variety of NLP tasks. We present the tool's architecture and demonstrate its advantages with prototypical use-cases, where we show that predictive performance can easily be increased in a few-shot learning scenario. Finally, we evaluate its usability in a user study. We provide the code and a live interface at https://adapter-hub.github.io/playground.
Design and Analysis of Modular Pipe Climber-III with a Multi-Output Differential Mechanism
Venue: arXiv cs.RO 2022
Year: 2022

This paper presents the design of an in-pipe climbing robot that operates using a novel `Three-output open differential'(3-OOD) mechanism to traverse complex networks of pipes. Conventional wheeled/tracked in-pipe climbing robots are prone to slip and drag while traversing in pipe bends. The 3-OOD mechanism helps in achieving the novel result of eliminating slip and drag in the robot tracks during motion. The proposed differential realizes the functional abilities of the traditional two-output differential, which is achieved the first time for a differential with three outputs. The 3-OOD mechanism mechanically modulates the track speeds of the robot based on the forces exerted on each track inside the pipe network, by eliminating the need for any active control. The simulation of the robot traversing in the pipe network in different orientations and in pipe-bends without slip shows the proposed design's effectiveness.