OpenRaiser/Intern-Atlas huggingface dataset:
Discrete Flow Matching
Venue: arXiv cs.LG 2024
Year: 2024

Despite Flow Matching and diffusion models having emerged as powerful generative paradigms for continuous variables such as images and videos, their application to high-dimensional discrete data, such as language, is still limited. In this work, we present Discrete Flow Matching, a novel discrete flow paradigm designed specifically for generating discrete data. Discrete Flow Matching offers several key contributions:(i) it works with a general family of probability paths interpolating between source and target distributions; (ii) it allows for a generic formula for sampling from these probability paths using learned posteriors such as the probability denoiser ($x$-prediction) and noise-prediction ($ε$-prediction); (iii) practically, focusing on specific probability paths defined with different schedulers improves generative perplexity compared to previous discrete diffusion and flow models; and (iv) by scaling Discrete Flow Matching models up to 1.7B parameters, we reach 6.7% Pass@1 and 13.4% Pass@10 on HumanEval and 6.7% Pass@1 and 20.6% Pass@10 on 1-shot MBPP coding benchmarks. Our approach is capable of generating high-quality discrete data in a non-autoregressive fashion, significantly closing the gap between autoregressive models and discrete flow models.
Probing Fine-Grained Action Understanding and Cross-View Generalization of Foundation Models
Venue: arXiv cs.CV 2024
Year: 2024

Foundation models (FMs) are large neural networks trained on broad datasets, excelling in downstream tasks with minimal fine-tuning. Human activity recognition in video has advanced with FMs, driven by competition among different architectures. However, high accuracies on standard benchmarks can draw an artificially rosy picture, as they often overlook real-world factors like changing camera perspectives. Popular benchmarks, mostly from YouTube or movies, offer diverse views but only coarse actions, which are insufficient for use-cases needing fine-grained, domain-specific actions. Domain-specific datasets (e.g., for industrial assembly) typically use data from limited static perspectives. This paper empirically evaluates how perspective changes affect different FMs in fine-grained human activity recognition. We compare multiple backbone architectures and design choices, including image- and video- based models, and various strategies for temporal information fusion, including commonly used score averaging and more novel attention-based temporal aggregation mechanisms. This is the first systematic study of different foundation models and specific design choices for human activity recognition from unknown views, conducted with the goal to provide guidance for backbone- and temporal- fusion scheme selection. Code and models will be made publicly available to the community.
The Atlas of AI Incidents in Mobile Computing: Visualizing the Risks and Benefits of AI Gone Mobile
Venue: arXiv cs.HC 2024
Year: 2024

Today's visualization tools for conveying the risks and benefits of AI technologies are largely tailored for those with technical expertise. To bridge this gap, we have developed a visualization that employs narrative patterns and interactive elements, enabling the broader public to gradually grasp the diverse risks and benefits associated with AI. Using a dataset of 54 real-world incidents involving AI in mobile computing, we examined design choices that enhance public understanding and provoke reflection on how certain AI applications - even those deemed low-risk by law - can still lead to significant incidents. Visualization: https://social-dynamics.net/mobile-ai-risks
Cryptoeconomics and Tokenomics as Economics: A Survey with Opinions
Venue: arXiv cs.GT 2024
Year: 2024

This paper surveys products and studies on cryptoeconomics and tokenomics from an economic perspective, as these terms are still (i) ill-defined and (ii) disconnected from economic disciplines. We first suggest that they can be novel when integrated; we then conduct a literature review and case study following consensus-building for decentralization and token value for autonomy. Integration requires simultaneous consideration of strategic behavior, spamming, Sybil attacks, free-riding, marginal cost, marginal utility and stabilizers. This survey is the first systematization of knowledge on cryptoeconomics and tokenomics, aiming to bridge the contexts of economics and blockchain.
Equality of morphic sequences
Venue: arXiv cs.SC 2024
Year: 2024

Morphic sequences form a natural class of infinite sequences, typically defined as the coding of a fixed point of a morphism. Different morphisms and codings may yield the same morphic sequence. This paper investigates how to prove that two such representations of a morphic sequence by morphisms represent the same sequence. In particular, we focus on the smallest representations of the subsequences of the binary Fibonacci sequence obtained by only taking the even or odd elements. The proofs we give are induction proofs of several properties simultaneously, and are typically found fully automatically by a tool that we developed.
Self-Sustainable Metasurface-Assisted mmWave Indoor Communication System
Venue: arXiv cs.IT 2024
Year: 2024

In the design of a metasurface-assisted system for indoor environments, it is essential to take into account not only the performance gains and coverage extension provided by the metasurface but also the operating costs brought by its reconfigurability, such as powering and cabling. These costs can present challenges, particularly in indoor dense spaces (IDSs). A self-sustainable metasurface (SSM), which retains reconfigurability unlike a static metasurface (SMS), achieves a lower operating cost than a reconfigurable intelligent surface (RIS) by being self-sustainable through power harvesting. In this paper, in order to find a better trade-off between metasurface gain, coverage, and operating cost, the design and performance of an SSM-assisted indoor mmWave communication system are investigated. We first simplify the design of the SSM-assisted system by considering the use of SSMs in a preset-based manner and the formation of coverage groups by associating SSMs with the closest user equipments (UEs). We propose a two-stage iterative algorithm to maximize the minimum data rate in the system by jointly deciding the association between the UEs and the SSMs, the phase-shifts of the SSMs, and allocating time resources for each UE. The non-convexities that exist in the proposed optimization problem are tackled using the feasible point pursuit successive convex approximation method and the concave-convex procedure. To understand the best scenario for using SSM, the resulting performance is compared with that achieved with RIS and SMS. Our numerical results indicate that SSMs are best utilized in a small environment where self-sustainability is easier to achieve when the budget for operating costs is tight.
OMoS-QA: A Dataset for Cross-Lingual Extractive Question Answering in a German Migration Context
Venue: arXiv cs.CL 2024
Year: 2024

When immigrating to a new country, it is easy to feel overwhelmed by the need to obtain information on financial support, housing, schooling, language courses, and other issues. If relocation is rushed or even forced, the necessity for high-quality answers to such questions is all the more urgent. Official immigration counselors are usually overbooked, and online systems could guide newcomers to the requested information or a suitable counseling service. To this end, we present OMoS-QA, a dataset of German and English questions paired with relevant trustworthy documents and manually annotated answers, specifically tailored to this scenario. Questions are automatically generated with an open-source large language model (LLM) and answer sentences are selected by crowd workers with high agreement. With our data, we conduct a comparison of 5 pretrained LLMs on the task of extractive question answering (QA) in German and English. Across all models and both languages, we find high precision and low-to-mid recall in selecting answer sentences, which is a favorable trade-off to avoid misleading users. This performance even holds up when the question language does not match the document language. When it comes to identifying unanswerable questions given a context, there are larger differences between the two languages.
Scheduling on a Stochastic Number of Machines
Venue: arXiv cs.DS 2024
Year: 2024

We consider a new scheduling problem on parallel identical machines in which the number of machines is initially not known, but it follows a given probability distribution. Only after all jobs are assigned to a given number of bags, the actual number of machines is revealed. Subsequently, the jobs need to be assigned to the machines without splitting the bags. This is the stochastic version of a related problem introduced by Stein and Zhong [SODA 2018, TALG 2020] and it is, for example, motivated by bundling jobs that need to be scheduled by data centers. We present two PTASs for the stochastic setting, computing job-to-bag assignments that (i) minimize the expected maximum machine load and (ii) maximize the expected minimum machine load (like in the Santa Claus problem), respectively. The former result follows by careful enumeration combined with known PTASs. For the latter result, we introduce an intricate dynamic program that we apply to a suitably rounded instance.
Broad and Spectral-Efficient Beamforming for the Uni-polarized Reconfigurable Intelligent Surfaces
Venue: arXiv cs.IT 2024
Year: 2024

A reconfigurable intelligent surface (RIS) is composed of low-cost elements that manipulate the propagation environment from a transmitter by intelligently applying phase shifts to incoming signals before they are reflected. This paper explores a uni-polarized RIS with linear shape aimed at transmitting a common signal to multiple user equipments (UEs) spread across a wide angular region. To achieve uniform coverage, the uni-polarized RIS is designed to emit a broad and spectral-efficient beam featuring a spatially flat-like array factor, diverging from the conventional narrow beam approach. To achieve this objective, we start by deriving probabilistic lower and upper bounds for the average spectral efficiency (SE) delivered to the UEs. Leveraging the insights from the lower bound, we focus on optimizing the minimum value of the power domain array factor (PDAF) across a range of azimuth angles from \(-\fracπ{2}\) to \(\fracπ{2}\). We employ the continuous genetic algorithm (CGA) for this optimization task, aiming to improve the SE delivered to the UEs while also creating a wide beam. Extensive simulation experiments are carried out to assess the performance of the proposed code, focusing on key metrics such as the minimum and average values of the PDAF and the SE delivered to the UEs. Our findings demonstrate that the proposed code enhances the minimum SE delivered to the UEs while maintaining the desired attribute of a broad beam. This performance is notably superior to that of established codes, including the Barker, Frank, and Chu codes.
Robust Facial Reactions Generation: An Emotion-Aware Framework with Modality Compensation
Venue: arXiv cs.CV 2024
Year: 2024

The objective of the Multiple Appropriate Facial Reaction Generation (MAFRG) task is to produce contextually appropriate and diverse listener facial behavioural responses based on the multimodal behavioural data of the conversational partner (i.e., the speaker). Current methodologies typically assume continuous availability of speech and facial modality data, neglecting real-world scenarios where these data may be intermittently unavailable, which often results in model failures. Furthermore, despite utilising advanced deep learning models to extract information from the speaker's multimodal inputs, these models fail to adequately leverage the speaker's emotional context, which is vital for eliciting appropriate facial reactions from human listeners. To address these limitations, we propose an Emotion-aware Modality Compensatory (EMC) framework. This versatile solution can be seamlessly integrated into existing models, thereby preserving their advantages while significantly enhancing performance and robustness in scenarios with missing modalities. Our framework ensures resilience when faced with missing modality data through the Compensatory Modality Alignment (CMA) module. It also generates more appropriate emotion-aware reactions via the Emotion-aware Attention (EA) module, which incorporates the speaker's emotional information throughout the entire encoding and decoding process. Experimental results demonstrate that our framework improves the appropriateness metric FRCorr by an average of 57.2\% compared to the original model structure. In scenarios where speech modality data is missing, the performance of appropriate generation shows an improvement, and when facial data is missing, it only exhibits minimal degradation.
A simple and fast C++ thread pool implementation capable of running task graphs
Venue: arXiv cs.DC 2024
Year: 2024

In this paper, the author presents a simple and fast C++ thread pool implementation capable of running task graphs. The implementation is publicly available on GitHub, see https://github.com/dpuyda/scheduling.
Learning to Manipulate Anywhere: A Visual Generalizable Framework For Reinforcement Learning
Venue: arXiv cs.RO 2024
Year: 2024

Can we endow visuomotor robots with generalization capabilities to operate in diverse open-world scenarios? In this paper, we propose \textbf{Maniwhere}, a generalizable framework tailored for visual reinforcement learning, enabling the trained robot policies to generalize across a combination of multiple visual disturbance types. Specifically, we introduce a multi-view representation learning approach fused with Spatial Transformer Network (STN) module to capture shared semantic information and correspondences among different viewpoints. In addition, we employ a curriculum-based randomization and augmentation approach to stabilize the RL training process and strengthen the visual generalization ability. To exhibit the effectiveness of Maniwhere, we meticulously design 8 tasks encompassing articulate objects, bi-manual, and dexterous hand manipulation tasks, demonstrating Maniwhere's strong visual generalization and sim2real transfer abilities across 3 hardware platforms. Our experiments show that Maniwhere significantly outperforms existing state-of-the-art methods. Videos are provided at https://gemcollector.github.io/maniwhere/.
A Survey on Trustworthiness in Foundation Models for Medical Image Analysis
Venue: arXiv cs.CV 2024
Year: 2024

The rapid advancement of foundation models in medical imaging represents a significant leap toward enhancing diagnostic accuracy and personalized treatment. However, the deployment of foundation models in healthcare necessitates a rigorous examination of their trustworthiness, encompassing privacy, robustness, reliability, explainability, and fairness. The current body of survey literature on foundation models in medical imaging reveals considerable gaps, particularly in the area of trustworthiness. Additionally, existing surveys on the trustworthiness of foundation models do not adequately address their specific variations and applications within the medical imaging domain. This survey aims to fill that gap by presenting a novel taxonomy of foundation models used in medical imaging and analyzing the key motivations for ensuring their trustworthiness. We review current research on foundation models in major medical imaging applications, focusing on segmentation, medical report generation, medical question and answering (Q\&A), and disease diagnosis. These areas are highlighted because they have seen a relatively mature and substantial number of foundation models compared to other applications. We focus on literature that discusses trustworthiness in medical image analysis manuscripts. We explore the complex challenges of building trustworthy foundation models for each application, summarizing current concerns and strategies for enhancing trustworthiness. Furthermore, we examine the potential of these models to revolutionize patient care. Our analysis underscores the imperative for advancing towards trustworthy AI in medical image analysis, advocating for a balanced approach that fosters innovation while ensuring ethical and equitable healthcare delivery.
SmartQuant: CXL-based AI Model Store in Support of Runtime Configurable Weight Quantization
Venue: arXiv cs.LG 2024
Year: 2024

Recent studies have revealed that, during the inference on generative AI models such as transformer, the importance of different weights exhibits substantial context-dependent variations. This naturally manifests a promising potential of adaptively configuring weight quantization to improve the generative AI inference efficiency. Although configurable weight quantization can readily leverage the hardware support of variable-precision arithmetics in modern GPU and AI accelerators, little prior research has studied how one could exploit variable weight quantization to proportionally improve the AI model memory access speed and energy efficiency. Motivated by the rapidly maturing CXL ecosystem, this work develops a CXL-based design solution to fill this gap. The key is to allow CXL memory controllers play an active role in supporting and exploiting runtime configurable weight quantization. Using transformer as a representative generative AI model, we carried out experiments that well demonstrate the effectiveness of the proposed design solution.
The new science of COVID-19: A Bibliographic and Network Analysis
Venue: arXiv cs.DL 2024
Year: 2024

Since the outbreak of the COVID-19, there have been many scientific publications studying the COVID-19. The purpose of this study is to identify the research trend, collaboration pattern, most influential elements, etc. from scientific publications related to COVID-19 in 2020, by using bibliographic analysis and network analysis. In Chapter 1 and Chapter 2, motivation behind this paper is introduced. Some previous similar studies are discussed. Comparisons are made in different aspects, such as data collection methods, data analysis tools and methods, etc. Their advantages and limitations compared to this paper are also addressed. In Chapter 3, important concepts used in this paper related to bibliographic analysis such as h-index and g-index, and network analysis such as centrality measures and assortativity are introduced. Networks with small-world property and scale-free property will also be studied. In Chapter 4 and Chapter 5, the way the data are obtained for the analysis of this paper is introduced step by step. Full result is shown. In Chapter 6, conclusions are arrived. A general growing trend of the number of the publications is observed, due to the efforts made by scientific researchers. Meanwhile, measures should also be taken to encourage future study in this field.
Shapley Pruning for Neural Network Compression
Venue: arXiv cs.LG 2024
Year: 2024

Neural network pruning is a rich field with a variety of approaches. In this work, we propose to connect the existing pruning concepts such as leave-one-out pruning and oracle pruning and develop them into a more general Shapley value-based framework that targets the compression of convolutional neural networks. To allow for practical applications in utilizing the Shapley value, this work presents the Shapley value approximations, and performs the comparative analysis in terms of cost-benefit utility for the neural network compression. The proposed ranks are evaluated against a new benchmark, Oracle rank, constructed based on oracle sets. The broad experiments show that the proposed normative ranking and its approximations show practical results, obtaining state-of-the-art network compression.
Gaussian Process Model with Tensorial Inputs and Its Application to the Design of 3D Printed Antennas
Venue: arXiv cs.LG 2024
Year: 2024

In simulation-based engineering design with time-consuming simulators, Gaussian process (GP) models are widely used as fast emulators to speed up the design optimization process. In its most commonly used form, the input of GP is a simple list of design parameters. With rapid development of additive manufacturing (also known as 3D printing), design inputs with 2D/3D spatial information become prevalent in some applications, for example, neighboring relations between pixels/voxels and material distributions in heterogeneous materials. Such spatial information, vital to 3D printed designs, is hard to incorporate into existing GP models with common kernels such as squared exponential or Matérn. In this work, we propose to embed a generalized distance measure into a GP kernel, offering a novel and convenient technique to incorporate spatial information from freeform 3D printed designs into the GP framework. The proposed method allows complex design problems for 3D printed objects to take advantage of a plethora of tools available from the GP surrogate-based simulation optimization such as designed experiments and GP-based optimizations including Bayesian optimization. We investigate the properties of the proposed method and illustrate its performance by several numerical examples of 3D printed antennas. The dataset is publicly available at: https://github.com/xichennn/GP_dataset.
Craft: Cross-modal Aligned Features Improve Robustness of Prompt Tuning
Venue: arXiv cs.CV 2024
Year: 2024

Prompt Tuning has emerged as a prominent research paradigm for adapting vision-language models to various downstream tasks. However, recent research indicates that prompt tuning methods often lead to overfitting due to limited training samples. In this paper, we propose a Cross-modal Aligned Feature Tuning (Craft) method to address this issue. Cross-modal alignment is conducted by first selecting anchors from the alternative domain and deriving relative representations of the embeddings for the selected anchors. Optimizing for a feature alignment loss over anchor-aligned text and image modalities creates a more unified text-image common space. Overfitting in prompt tuning also deteriorates model performance on out-of-distribution samples. To further improve the prompt model's robustness, we propose minimizing Maximum Mean Discrepancy (MMD) over the anchor-aligned feature spaces to mitigate domain shift. The experiment on four different prompt tuning structures consistently shows the improvement of our method, with increases of up to $6.1\%$ in the Base-to-Novel generalization task, $5.8\%$ in the group robustness task, and $2.7\%$ in the out-of-distribution tasks. The code will be available at https://github.com/Jingchensun/Craft