OpenRaiser/Intern-Atlas huggingface dataset:
Multimodal Foundation Models: From Specialists to General-Purpose Assistants
Venue: arXiv cs.CV 2023
Year: 2023

This paper presents a comprehensive survey of the taxonomy and evolution of multimodal foundation models that demonstrate vision and vision-language capabilities, focusing on the transition from specialist models to general-purpose assistants. The research landscape encompasses five core topics, categorized into two classes. (i) We start with a survey of well-established research areas: multimodal foundation models pre-trained for specific purposes, including two topics -- methods of learning vision backbones for visual understanding and text-to-image generation. (ii) Then, we present recent advances in exploratory, open research areas: multimodal foundation models that aim to play the role of general-purpose assistants, including three topics -- unified vision models inspired by large language models (LLMs), end-to-end training of multimodal LLMs, and chaining multimodal tools with LLMs. The target audiences of the paper are researchers, graduate students, and professionals in computer vision and vision-language multimodal communities who are eager to learn the basics and recent advances in multimodal foundation models.
Simple and Optimal Online Contention Resolution Schemes for $k$-Uniform Matroids
Venue: arXiv cs.DS 2023
Year: 2023

We provide a simple $(1-O(\frac{1}{\sqrt{k}}))$-selectable Online Contention Resolution Scheme for $k$-uniform matroids against a fixed-order adversary. If $A_i$ and $G_i$ denote the set of selected elements and the set of realized active elements among the first $i$ (respectively), our algorithm selects with probability $1-\frac{1}{\sqrt{k}}$ any active element $i$ such that $|A_{i-1}| + 1 \leq (1-\frac{1}{\sqrt{k}})\cdot \mathbb{E}[|G_i|]+\sqrt{k}$. This implies a $(1-O(\frac{1}{\sqrt{k}}))$ prophet inequality against fixed-order adversaries for $k$-uniform matroids that is considerably simpler than previous algorithms [Ala14, AKW14, JMZ22]. We also prove that no OCRS can be $(1-Ω(\sqrt{\frac{\log k}{k}}))$-selectable for $k$-uniform matroids against an almighty adversary. This guarantee is matched by the (known) simple greedy algorithm that accepts every active element with probability $1-Θ(\sqrt{\frac{\log k}{k}})$ [HKS07].
Constructor algorithms for building unconventional computers able to solve NP-complete problems
Venue: arXiv cs.ET 2023
Year: 2023

Nature often builds physical structures tailored for specific information processing tasks with computations encoded using diverse phenomena. These can sometimes outperform typical general-purpose computers. However, describing the construction and function of these unconventional computers is often challenging. Here, we address this by introducing constructor algorithms in the context of a robotic wire machine that can be programmed to build networks of connected wires in response to a problem and then act upon these to efficiently carry out a desired computation. We show how this approach can be used to solve the NP-complete Subset Sum Problem (SSP) and provide information about the number of solutions through changes in the voltages and currents measured across these networks. This work provides a foundation for building unconventional computers that encode information purely in the lengths and connections of electrically conductive wires. It also demonstrates the power of computing paradigms beyond digital logic and opens avenues to more fully harness the inherent computational capabilities of diverse physical, chemical and biological substrates.
Adaptive Liquidity Provision in Uniswap V3 with Deep Reinforcement Learning
Venue: arXiv cs.AI 2023
Year: 2023

Decentralized exchanges (DEXs) are a cornerstone of decentralized finance (DeFi), allowing users to trade cryptocurrencies without the need for third-party authorization. Investors are incentivized to deposit assets into liquidity pools, against which users can trade directly, while paying fees to liquidity providers (LPs). However, a number of unresolved issues related to capital efficiency and market risk hinder DeFi's further development. Uniswap V3, a leading and groundbreaking DEX project, addresses capital efficiency by enabling LPs to concentrate their liquidity within specific price ranges for deposited assets. Nevertheless, this approach exacerbates market risk, as LPs earn trading fees only when asset prices are within these predetermined brackets. To mitigate this issue, this paper introduces a deep reinforcement learning (DRL) solution designed to adaptively adjust these price ranges, maximizing profits and mitigating market risks. Our approach also neutralizes price-change risks by hedging the liquidity position through a rebalancing portfolio in a centralized futures exchange. The DRL policy aims to optimize trading fees earned by LPs against associated costs, such as gas fees and hedging expenses, which is referred to as loss-versus-rebalancing (LVR). Using simulations with a profit-and-loss (PnL) benchmark, our method demonstrates superior performance in ETH/USDC and ETH/USDT pools compared to existing baselines. We believe that this strategy not only offers investors a valuable asset management tool but also introduces a new incentive mechanism for DEX designers.
Human Gait Recognition using Deep Learning: A Comprehensive Review
Venue: arXiv cs.CV 2023
Year: 2023

Gait recognition (GR) is a growing biometric modality used for person identification from a distance through visual cameras. GR provides a secure and reliable alternative to fingerprint and face recognition, as it is harder to distinguish between false and authentic signals. Furthermore, its resistance to spoofing makes GR suitable for all types of environments. With the rise of deep learning, steadily improving strides have been made in GR technology with promising results in various contexts. As video surveillance becomes more prevalent, new obstacles arise, such as ensuring uniform performance evaluation across different protocols, reliable recognition despite shifting lighting conditions, fluctuations in gait patterns, and protecting privacy.This survey aims to give an overview of GR and analyze the environmental elements and complications that could affect it in comparison to other biometric recognition systems. The primary goal is to examine the existing deep learning (DL) techniques employed for human GR that may generate new research opportunities.
Comparing an android head with its digital twin regarding the dynamic expression of emotions
Venue: arXiv cs.RO 2023
Year: 2023

Emotions, which are an important component of social interaction, can be studied with the help of android robots and their appearance, which is as similar to humans as possible. The production and customization of android robots is expensive and time-consuming, so it may be practical to use a digital replica. In order to investigate whether there are any perceptual differences in terms of emotions based on the difference in appearance, a robot head was digitally replicated. In an experiment, the basic emotions evaluated in a preliminary study were compared in three conditions and then statistically analyzed. It was found that apart from fear, all emotions were recognized on the real robot head. The digital head with "ideal" emotions performed better than the real head apart from the anger representation, which offers optimization potential for the real head. Contrary to expectations, significant differences between the real and the replicated head with the same emotions could only be found in the representation of surprise.
Realistic Website Fingerprinting By Augmenting Network Trace
Venue: arXiv cs.CR 2023
Year: 2023

Website Fingerprinting (WF) is considered a major threat to the anonymity of Tor users (and other anonymity systems). While state-of-the-art WF techniques have claimed high attack accuracies, e.g., by leveraging Deep Neural Networks (DNN), several recent works have questioned the practicality of such WF attacks in the real world due to the assumptions made in the design and evaluation of these attacks. In this work, we argue that such impracticality issues are mainly due to the attacker's inability in collecting training data in comprehensive network conditions, e.g., a WF classifier may be trained only on samples collected on specific high-bandwidth network links but deployed on connections with different network conditions. We show that augmenting network traces can enhance the performance of WF classifiers in unobserved network conditions. Specifically, we introduce NetAugment, an augmentation technique tailored to the specifications of Tor traces. We instantiate NetAugment through semi-supervised and self-supervised learning techniques. Our extensive open-world and close-world experiments demonstrate that under practical evaluation settings, our WF attacks provide superior performances compared to the state-of-the-art; this is due to their use of augmented network traces for training, which allows them to learn the features of target traffic in unobserved settings. For instance, with a 5-shot learning in a closed-world scenario, our self-supervised WF attack (named NetCLR) reaches up to 80% accuracy when the traces for evaluation are collected in a setting unobserved by the WF adversary. This is compared to an accuracy of 64.4% achieved by the state-of-the-art Triplet Fingerprinting [35]. We believe that the promising results of our work can encourage the use of network trace augmentation in other types of network traffic analysis.
A Scalable Communication Model to Realize Integrated Access and Backhaul (IAB) in 5G
Venue: arXiv cs.NI 2023
Year: 2023

Our vision of the future world is one wherein everything, anywhere and at any time, can reliably communicate in real time. 5G, the fifth generation of cellular networks, is anticipated to use heterogeneity to deliver ultra-high data rates to a vastly increased number of devices in ultra-dense areas. Improving the backhaul network capacity is one of the most important open challenges for deploying a 5G network. A promising solution is Integrated Access and Backhaul (IAB), which assigns a portion of radio resources to construct a multi-hop wireless backhaul network. Although 3GPP has acknowledged the cost-effectiveness of the IAB-enabled framework and its orchestration has been extensively studied in the literature, its transmission capacity (i.e., the number of base stations it can support) has not been sufficiently investigated. In this paper, we formulate the problem of maximizing transmission capacity and minimizing transmit powers for IAB-enabled multi-hop networks, taking into account relay selection, channel assignment, and power control constraints. Then, the solution space of the problem is analyzed, two optimality bounds are derived, and a heuristic algorithm is proposed to investigate the bounds. The claims are finally supported by numerical results.
Testaro: Efficient Ensemble Testing for Web Accessibility
Venue: arXiv cs.SE 2023
Year: 2023

As automated web accessibility testing tools become enriched with new and improved tests, it can be impractical to leverage those advances. Each tool offers unique benefits, but effectively using multiple tools would require integrating them into a uniform testing and reporting scheme. Such integration is complex, because tools vary in what they try to detect, what they actually detect, and how they classify, describe, and report defects. Consequently, testers typically use only one tool. Testaro is a novel open-source NPM package that checks compliance with about 650 rules defined by an ensemble of 8 tools: alfa, Axe, Equal Access, HTML CodeSniffer, Nu Html Checker, QualWeb, Testaro, and WAVE. Attendees at the demonstration will, within 5 minutes, create jobs for Testaro, run them, and generate unified reports documenting more accessibility issues than any single tool can discover.
Double Deep Q-Learning-based Path Selection and Service Placement for Latency-Sensitive Beyond 5G Applications
Venue: arXiv cs.NI 2023
Year: 2023

Nowadays, as the need for capacity continues to grow, entirely novel services are emerging. A solid cloud-network integrated infrastructure is necessary to supply these services in a real-time responsive, and scalable way. Due to their diverse characteristics and limited capacity, communication and computing resources must be collaboratively managed to unleash their full potential. Although several innovative methods have been proposed to orchestrate the resources, most ignored network resources or relaxed the network as a simple graph, focusing only on cloud resources. This paper fills the gap by studying the joint problem of communication and computing resource allocation, dubbed CCRA, including function placement and assignment, traffic prioritization, and path selection considering capacity constraints and quality requirements, to minimize total cost. We formulate the problem as a non-linear programming model and propose two approaches, dubbed B\&B-CCRA and WF-CCRA, based on the Branch \& Bound and Water-Filling algorithms to solve it when the system is fully known. Then, for partially known systems, a Double Deep Q-Learning (DDQL) architecture is designed. Numerical simulations show that B\&B-CCRA optimally solves the problem, whereas WF-CCRA delivers near-optimal solutions in a substantially shorter time. Furthermore, it is demonstrated that DDQL-CCRA obtains near-optimal solutions in the absence of request-specific information.
On the Minimum Distance, Minimum Weight Codewords, and the Dimension of Projective Reed-Muller Codes
Venue: arXiv cs.IT 2023
Year: 2023

We give an alternative proof of the formula for the minimum distance of a projective Reed-Muller code of an arbitrary order. It leads to a complete characterization of the minimum weight codewords of a projective Reed-Muller code. This is then used to determine the number of minimum weight codewords of a projective Reed-Muller code. Various formulas for the dimension of a projective Reed-Muller code, and their equivalences are also discussed.
Stabilizing RLHF through Advantage Model and Selective Rehearsal
Venue: arXiv cs.CL 2023
Year: 2023

Large Language Models (LLMs) have revolutionized natural language processing, yet aligning these models with human values and preferences using RLHF remains a significant challenge. This challenge is characterized by various instabilities, such as reward hacking and catastrophic forgetting. In this technical report, we propose two innovations to stabilize RLHF training: 1) Advantage Model, which directly models advantage score i.e., extra reward compared to the expected rewards and regulates score distributions across tasks to prevent reward hacking. 2) Selective Rehearsal, which mitigates catastrophic forgetting by strategically selecting data for PPO training and knowledge rehearsing. Our experimental analysis on public and proprietary datasets reveals that the proposed methods not only increase stability in RLHF training but also achieve higher reward scores and win rates.
A Hierarchy-based Analysis Approach for Blended Learning: A Case Study with Chinese Students
Venue: arXiv cs.CY 2023
Year: 2023

Blended learning is generally defined as the combination of traditional face-to-face learning and online learning. This learning mode has been widely used in advanced education across the globe due to the COVID-19 pandemic's social distance restriction as well as the development of technology. Online learning plays an important role in blended learning, and as it requires more student autonomy, the quality of blended learning in advanced education has been a persistent concern. Existing literature offers several elements and frameworks regarding evaluating the quality of blended learning. However, most of them either have different favours for evaluation perspectives or simply offer general guidance for evaluation, reducing the completeness, objectivity and practicalness of related works. In order to carry out a more intuitive and comprehensive evaluation framework, this paper proposes a hierarchy-based analysis approach. Applying gradient boosting model and feature importance evaluation method, this approach mainly analyses student engagement and its three identified dimensions (behavioral engagement, emotional engagement, cognitive engagement) to eliminate some existing stubborn problems when it comes to blended learning evaluation. The results show that cognitive engagement and emotional engagement play a more important role in blended learning evaluation, implying that these two should be considered to improve for better learning as well as teaching quality.
Drive as You Speak: Enabling Human-Like Interaction with Large Language Models in Autonomous Vehicles
Venue: arXiv cs.HC 2023
Year: 2023

The future of autonomous vehicles lies in the convergence of human-centric design and advanced AI capabilities. Autonomous vehicles of the future will not only transport passengers but also interact and adapt to their desires, making the journey comfortable, efficient, and pleasant. In this paper, we present a novel framework that leverages Large Language Models (LLMs) to enhance autonomous vehicles' decision-making processes. By integrating LLMs' natural language capabilities and contextual understanding, specialized tools usage, synergizing reasoning, and acting with various modules on autonomous vehicles, this framework aims to seamlessly integrate the advanced language and reasoning capabilities of LLMs into autonomous vehicles. The proposed framework holds the potential to revolutionize the way autonomous vehicles operate, offering personalized assistance, continuous learning, and transparent decision-making, ultimately contributing to safer and more efficient autonomous driving technologies.
What is the Best Automated Metric for Text to Motion Generation?
Venue: arXiv cs.CL 2023
Year: 2023

There is growing interest in generating skeleton-based human motions from natural language descriptions. While most efforts have focused on developing better neural architectures for this task, there has been no significant work on determining the proper evaluation metric. Human evaluation is the ultimate accuracy measure for this task, and automated metrics should correlate well with human quality judgments. Since descriptions are compatible with many motions, determining the right metric is critical for evaluating and designing effective generative models. This paper systematically studies which metrics best align with human evaluations and proposes new metrics that align even better. Our findings indicate that none of the metrics currently used for this task show even a moderate correlation with human judgments on a sample level. However, for assessing average model performance, commonly used metrics such as R-Precision and less-used coordinate errors show strong correlations. Additionally, several recently developed metrics are not recommended due to their low correlation compared to alternatives. We also introduce a novel metric based on a multimodal BERT-like model, MoBERT, which offers strongly human-correlated sample-level evaluations while maintaining near-perfect model-level correlation. Our results demonstrate that this new metric exhibits extensive benefits over all current alternatives.
Distributed Matrix Multiplication with a Smaller Recovery Threshold through Modulo-based Approaches
Venue: arXiv cs.IT 2023
Year: 2023

This paper considers the problem of calculating the matrix multiplication of two massive matrices $\mathbf{A}$ and $\mathbf{B}$ distributedly. We provide a modulo technique that can be applied to coded distributed matrix multiplication problems to reduce the recovery threshold. This technique exploits the special structure of interpolation points and can be applied to many existing coded matrix designs. Recently studied discrete Fourier transform based code achieves a smaller recovery threshold than the optimal MatDot code with the expense that it cannot resist stragglers. We also propose a distributed matrix multiplication scheme based on the idea of locally repairable code to reduce the recovery threshold of MatDot code and provide resilience to stragglers. We also apply our constructions to a type of matrix computing problems, where generalized linear models act as a special case.
RGB-based Category-level Object Pose Estimation via Decoupled Metric Scale Recovery
Venue: arXiv cs.CV 2023
Year: 2023

While showing promising results, recent RGB-D camera-based category-level object pose estimation methods have restricted applications due to the heavy reliance on depth sensors. RGB-only methods provide an alternative to this problem yet suffer from inherent scale ambiguity stemming from monocular observations. In this paper, we propose a novel pipeline that decouples the 6D pose and size estimation to mitigate the influence of imperfect scales on rigid transformations. Specifically, we leverage a pre-trained monocular estimator to extract local geometric information, mainly facilitating the search for inlier 2D-3D correspondence. Meanwhile, a separate branch is designed to directly recover the metric scale of the object based on category-level statistics. Finally, we advocate using the RANSAC-P$n$P algorithm to robustly solve for 6D object pose. Extensive experiments have been conducted on both synthetic and real datasets, demonstrating the superior performance of our method over previous state-of-the-art RGB-based approaches, especially in terms of rotation accuracy. Code: https://github.com/goldoak/DMSR.
Disentangled Information Bottleneck guided Privacy-Protective JSCC for Image Transmission
Venue: arXiv cs.CR 2023
Year: 2023

Joint source and channel coding (JSCC) has attracted increasing attention due to its robustness and high efficiency. However, JSCC is vulnerable to privacy leakage due to the high relevance between the source image and channel input. In this paper, we propose a disentangled information bottleneck guided privacy-protective JSCC (DIB-PPJSCC) for image transmission, which aims at protecting private information as well as achieving superior communication performance at the legitimate receiver. In particular, we propose a DIB objective to disentangle private and public information. The goal is to compress the private information in the public subcodewords, preserve the private information in the private subcodewords and improve the reconstruction quality simultaneously. In order to optimize JSCC neural networks using the DIB objective, we derive a differentiable estimation of the DIB objective based on the variational approximation and the density-ratio trick. Additionally, we design a password-based privacy-protective (PP) algorithm which can be jointly optimized with JSCC neural networks to encrypt the private subcodewords. Specifically, we employ a private information encryptor to encrypt the private subcodewords before transmission, and a corresponding decryptor to recover the private information at the legitimate receiver. A loss function for jointly training the encryptor, decryptor and JSCC decoder is derived based on the maximum entropy principle, which aims at maximizing the eavesdropping uncertainty as well as improving the reconstruction quality. Experimental results show that DIB-PPJSCC can reduce the eavesdropping accuracy on private information up to $15\%$ and reduce $10\%$ inference time compared to existing privacy-protective JSCC and traditional separate methods.