OpenRaiser/Intern-Atlas huggingface dataset:
Rethinking the A in STEAM: Insights from and for AI Literacy Education
Venue: arXiv cs.CY 2024
Year: 2024

This article rethinks the role of arts in STEAM education, emphasizing its importance in AI literacy within K-12 contexts. Arguing against the marginalization of arts, the paper is structured around four key domains: language studies, philosophy, social studies, and visual arts. Each section addresses critical AI-related phenomena and provides pedagogical strate-gies for effective integration into STEAM education. Language studies focus on media representations and the probabilistic nature of AI language models. The philosophy section examines anthropomorphism, ethics, and the misconstrued human-like capabilities of AI. Social studies discuss AI's societal impacts, biases, and ethical considerations in data prac-tices. Visual arts explore the implications of generative AI on artistic processes and intellec-tual property. The article concludes by advocating for a robust inclusion of arts in STEAM to foster a holistic, equitable, and sustainable understanding of AI, ultimately inspiring technologies that promote fairness and creativity.
Multi-CATE: Multi-Accurate Conditional Average Treatment Effect Estimation Robust to Unknown Covariate Shifts
Venue: arXiv cs.LG 2024
Year: 2024

Estimating heterogeneous treatment effects is important to tailor treatments to those individuals who would most likely benefit. However, conditional average treatment effect predictors may often be trained on one population but possibly deployed on different, possibly unknown populations. We use methodology for learning multi-accurate predictors to post-process CATE T-learners (differenced regressions) to become robust to unknown covariate shifts at the time of deployment. The method works in general for pseudo-outcome regression, such as the DR-learner. We show how this approach can combine (large) confounded observational and (smaller) randomized datasets by learning a confounded predictor from the observational dataset, and auditing for multi-accuracy on the randomized controlled trial. We show improvements in bias and mean squared error in simulations with increasingly larger covariate shift, and on a semi-synthetic case study of a parallel large observational study and smaller randomized controlled experiment. Overall, we establish a connection between methods developed for multi-distribution learning and achieve appealing desiderata (e.g. external validity) in causal inference and machine learning.
FinerCut: Finer-grained Interpretable Layer Pruning for Large Language Models
Venue: arXiv cs.LG 2024
Year: 2024

Overparametrized transformer networks are the state-of-the-art architecture for Large Language Models (LLMs). However, such models contain billions of parameters making large compute a necessity, while raising environmental concerns. To address these issues, we propose FinerCut, a new form of fine-grained layer pruning, which in contrast to prior work at the transformer block level, considers all self-attention and feed-forward network (FFN) layers within blocks as individual pruning candidates. FinerCut prunes layers whose removal causes minimal alternation to the model's output -- contributing to a new, lean, interpretable, and task-agnostic pruning method. Tested across 9 benchmarks, our approach retains 90% performance of Llama3-8B with 25% layers removed, and 95% performance of Llama3-70B with 30% layers removed, all without fine-tuning or post-pruning reconstruction. Strikingly, we observe intriguing results with FinerCut: 42% (34 out of 80) of the self-attention layers in Llama3-70B can be removed while preserving 99% of its performance -- without additional fine-tuning after removal. Moreover, FinerCut provides a tool to inspect the types and locations of pruned layers, allowing to observe interesting pruning behaviors. For instance, we observe a preference for pruning self-attention layers, often at deeper consecutive decoder layers. We hope our insights inspire future efficient LLM architecture designs.
Cooperative Relative Localization in MAV Swarms with Ultra-wideband Ranging
Venue: arXiv cs.RO 2024
Year: 2024

Relative localization (RL) is essential for the successful operation of micro air vehicle (MAV) swarms. Achieving accurate 3-D RL in infrastructure-free and GPS-denied environments with only distance information is a challenging problem that has not been satisfactorily solved. In this work, based on the range-based peer-to-peer RL using the ultra-wideband (UWB) ranging technique, we develop a novel UWB-based cooperative relative localization (CRL) solution that integrates the relative motion dynamics of each host-neighbor pair to build a unified dynamic model and takes the distances between the neighbors as \textit{bonus information}. Observability analysis using differential geometry shows that the proposed CRL scheme can expand the observable subspace compared to other alternatives using only direct distances between the host agent and its neighbors. In addition, we apply the kernel-induced extended Kalman filter (EKF) to the CRL state estimation problem with the novel-designed Logarithmic-Versoria (LV) kernel to tackle heavy-tailed UWB noise. Sufficient conditions for the convergence of the fixed-point iteration involved in the estimation algorithm are also derived. Comparative Monte Carlo simulations demonstrate that the proposed CRL scheme combined with the LV-kernel EKF significantly improves the estimation accuracy owing to its robustness against both measurement outliers and incorrect measurement covariance matrix initialization. Moreover, with the LV kernel, the estimation is still satisfactory when performing the fixed-point iteration only once for reduced computational complexity.
The CoExplorer Technology Probe: A Generative AI-Powered Adaptive Interface to Support Intentionality in Planning and Running Video Meetings
Venue: arXiv cs.HC 2024
Year: 2024

Effective meetings are effortful, but traditional videoconferencing systems offer little support for reducing this effort across the meeting lifecycle. Generative AI (GenAI) has the potential to radically redefine meetings by augmenting intentional meeting behaviors. CoExplorer, our novel adaptive meeting prototype, preemptively generates likely phases that meetings would undergo, tools that allow capturing attendees' thoughts before the meeting, and for each phase, window layouts, and appropriate applications and files. Using CoExplorer as a technology probe in a guided walkthrough, we studied its potential in a sample of participants from a global technology company. Our findings suggest that GenAI has the potential to help meetings stay on track and reduce workload, although concerns were raised about users' agency, trust, and possible disruption to traditional meeting norms. We discuss these concerns and their design implications for the development of GenAI meeting technology.
Channel Reciprocity Based Attack Detection for Securing UWB Ranging by Autoencoder
Venue: arXiv cs.CR 2024
Year: 2024

A variety of ranging threats represented by Ghost Peak attack have raised concerns regarding the security performance of Ultra-Wide Band (UWB) systems with the finalization of the IEEE 802.15.4z standard. Based on channel reciprocity, this paper proposes a low complexity attack detection scheme that compares Channel Impulse Response (CIR) features of both ranging sides utilizing an autoencoder with the capability of data compression and feature extraction. Taking Ghost Peak attack as an example, this paper demonstrates the effectiveness, feasibility and generalizability of the proposed attack detection scheme through simulation and experimental validation. The proposed scheme achieves an attack detection success rate of over 99% and can be implemented in current systems at low cost.
Multi-modal Generation via Cross-Modal In-Context Learning
Venue: arXiv cs.CV 2024
Year: 2024

In this work, we study the problem of generating novel images from complex multimodal prompt sequences. While existing methods achieve promising results for text-to-image generation, they often struggle to capture fine-grained details from lengthy prompts and maintain contextual coherence within prompt sequences. Moreover, they often result in misaligned image generation for prompt sequences featuring multiple objects. To address this, we propose a Multi-modal Generation via Cross-Modal In-Context Learning (MGCC) method that generates novel images from complex multimodal prompt sequences by leveraging the combined capabilities of large language models (LLMs) and diffusion models. Our MGCC comprises a novel Cross-Modal Refinement module to explicitly learn cross-modal dependencies between the text and image in the LLM embedding space, and a contextual object grounding module to generate object bounding boxes specifically targeting scenes with multiple objects. Our MGCC demonstrates a diverse range of multimodal capabilities, like novel image generation, the facilitation of multimodal dialogue, and generation of texts. Experimental evaluations on two benchmark datasets, demonstrate the effectiveness of our method. On Visual Story Generation (VIST) dataset with multimodal inputs, our MGCC achieves a CLIP Similarity score of $0.652$ compared to SOTA GILL $0.641$. Similarly, on Visual Dialogue Context (VisDial) having lengthy dialogue sequences, our MGCC achieves an impressive CLIP score of $0.660$, largely outperforming existing SOTA method scoring $0.645$. Code: https://github.com/VIROBO-15/MGCC
VITON-DiT: Learning In-the-Wild Video Try-On from Human Dance Videos via Diffusion Transformers
Venue: arXiv cs.CV 2024
Year: 2024

Video try-on stands as a promising area for its tremendous real-world potential. Prior works are limited to transferring product clothing images onto person videos with simple poses and backgrounds, while underperforming on casually captured videos. Recently, Sora revealed the scalability of Diffusion Transformer (DiT) in generating lifelike videos featuring real-world scenarios. Inspired by this, we explore and propose the first DiT-based video try-on framework for practical in-the-wild applications, named VITON-DiT. Specifically, VITON-DiT consists of a garment extractor, a Spatial-Temporal denoising DiT, and an identity preservation ControlNet. To faithfully recover the clothing details, the extracted garment features are fused with the self-attention outputs of the denoising DiT and the ControlNet. We also introduce novel random selection strategies during training and an Interpolated Auto-Regressive (IAR) technique at inference to facilitate long video generation. Unlike existing attempts that require the laborious and restrictive construction of a paired training dataset, severely limiting their scalability, VITON-DiT alleviates this by relying solely on unpaired human dance videos and a carefully designed multi-stage training strategy. Furthermore, we curate a challenging benchmark dataset to evaluate the performance of casual video try-on. Extensive experiments demonstrate the superiority of VITON-DiT in generating spatio-temporal consistent try-on results for in-the-wild videos with complicated human poses.
Approximating Densest Subgraph in Geometric Intersection Graphs
Venue: arXiv cs.CG 2024
Year: 2024

$ \newcommand{\cardin}[1]{\left| {#1} \right|}% \newcommand{\Graph}{\Mh{\mathsf{G}}}% \providecommand{\G}{\Graph}% \renewcommand{\G}{\Graph}% \providecommand{\GA}{\Mh{H}}% \renewcommand{\GA}{\Mh{H}}% \newcommand{\VV}{\Mh{\mathsf{V}}}% \newcommand{\VX}[1]{\VV\pth{#1}}% \providecommand{\EE}{\Mh{\mathsf{E}}}% \renewcommand{\EE}{\Mh{\mathsf{E}}}% \newcommand{\Re}{\mathbb{R}} \newcommand{\reals}{\mathbb{R}} \newcommand{\SetX}{\mathsf{X}} \newcommand{\rad}{r} \newcommand{\Mh}[1]{#1} \newcommand{\query}{q} \newcommand{\eps}{\varepsilon} \newcommand{\VorX}[1]{\mathcal{V} \pth{#1}} \newcommand{\Polygon}{\mathsf{P}} \newcommand{\IntRange}[1]{[ #1 ]} \newcommand{\Space}{\overline{\mathsf{m}}} \newcommand{\pth}[2][\!]{#1\left({#2}\right)} \newcommand{\polylog}{\mathrm{polylog}} \newcommand{\N}{\mathbb N} \newcommand{\Z}{\mathbb Z} \newcommand{\pt}{p} \newcommand{\distY}[2]{\left\| {#1} - {#2} \right\|} \newcommand{\ptq}{q} \newcommand{\pts}{s}$ For an undirected graph $\mathsf{G}=(\mathsf{V}, \mathsf{E})$, with $n$ vertices and $m$ edges, the \emph{densest subgraph} problem, is to compute a subset $S \subseteq \mathsf{V}$ which maximizes the ratio $|\mathsf{E}_S| / |S|$, where $\mathsf{E}_S \subseteq \mathsf{E}$ is the set of all edges of $\mathsf{G}$ with endpoints in $S$. The densest subgraph problem is a well studied problem in computer science. Existing exact and approximation algorithms for computing the densest subgraph require $Ω(m)$ time. We present near-linear time (in $n$) approximation algorithms for the densest subgraph problem on \emph{implicit} geometric intersection graphs, where the vertices are explicitly given but not the edges. As a concrete example, we consider $n$ disks in the plane with arbitrary radii and present two different approximation algorithms.
LLaMA-NAS: Efficient Neural Architecture Search for Large Language Models
Venue: arXiv cs.AI 2024
Year: 2024

The abilities of modern large language models (LLMs) in solving natural language processing, complex reasoning, sentiment analysis and other tasks have been extraordinary which has prompted their extensive adoption. Unfortunately, these abilities come with very high memory and computational costs which precludes the use of LLMs on most hardware platforms. To mitigate this, we propose an effective method of finding Pareto-optimal network architectures based on LLaMA2-7B using one-shot NAS. In particular, we fine-tune LLaMA2-7B only once and then apply genetic algorithm-based search to find smaller, less computationally complex network architectures. We show that, for certain standard benchmark tasks, the pre-trained LLaMA2-7B network is unnecessarily large and complex. More specifically, we demonstrate a 1.5x reduction in model size and 1.3x speedup in throughput for certain tasks with negligible drop in accuracy. In addition to finding smaller, higher-performing network architectures, our method does so more effectively and efficiently than certain pruning or sparsification techniques. Finally, we demonstrate how quantization is complementary to our method and that the size and complexity of the networks we find can be further decreased using quantization. We believe that our work provides a way to automatically create LLMs which can be used on less expensive and more readily available hardware platforms.
WIDIn: Wording Image for Domain-Invariant Representation in Single-Source Domain Generalization
Venue: arXiv cs.CV 2024
Year: 2024

Language has been useful in extending the vision encoder to data from diverse distributions without empirical discovery in training domains. However, as the image description is mostly at coarse-grained level and ignores visual details, the resulted embeddings are still ineffective in overcoming complexity of domains at inference time. We present a self-supervision framework WIDIn, Wording Images for Domain-Invariant representation, to disentangle discriminative visual representation, by only leveraging data in a single domain and without any test prior. Specifically, for each image, we first estimate the language embedding with fine-grained alignment, which can be consequently used to adaptively identify and then remove domain-specific counterpart from the raw visual embedding. WIDIn can be applied to both pretrained vision-language models like CLIP, and separately trained uni-modal models like MoCo and BERT. Experimental studies on three domain generalization datasets demonstrate the effectiveness of our approach.
The Polynomial Hierarchy does not collapse
Venue: arXiv cs.CC 2024
Year: 2024

The arithmetical hierarchy (AH) is similar to the polynomial hierarchy (PH). Unlike the PH, the AH does not collapse relative to any oracle. A language in the (k + 1)-st level of the AH is computable enumerable (c.e.) relative to the kth level. So, given an oracle in the kth level of the AH, we could use a black-box search to decide whether the input word is in the language. With very large padding arguments, i.e. the paddings grow faster than any relative to the level k of the AH computable function, we would construct a language contained in the k + 1 level of PH, if we use only a finite set of input words. From the oracle in AH, we would construct an analogue oracle at the kth level of PH. For the input words of the finite set, a word is in the language of AH, if and only if it is in the language of PH. And the input word is in the oracle set of AH, if and only if it is in the oracle of PH. As in the language of AH, we must apply a black-box search in the language of PH. So, we would also have exponentially many oracle queries in the language of PH. The PH does not collapse.
Distance based prefetching algorithms for mining of the sporadic requests associations
Venue: arXiv cs.DB 2024
Year: 2024

Modern storage systems intensively utilize data prefetching algorithms while processing sequences of the read requests. Performance of the prefetching algorithm (for instance increase of the cache hit ratio of the cache system - CHR) directly affects overall performance characteristics of the storage system (read latency, IOPS, etc.). There are widely known prefetching algorithms that are focused on the discovery of the sequential patterns in the stream of requests. This study examines a family of prefetching algorithms that is focused on mining of the pseudo random (sporadic) patterns between read requests - sporadic prefetching algorithms. The key contribution of this paper is that it discovers a new, lightweight family of distance-based sporadic prefetching algorithms (DBSP) that outperforms the best previously known results on MSR traces collection.Another important contribution of this paper is a thorough description of the procedure for comparing the performance of sporadic prefetchers.
An Algorithm for the Decomposition of Complete Graph into Minimum Number of Edge-disjoint Trees
Venue: arXiv cs.DM 2024
Year: 2024

In this work, we study methodical decomposition of an undirected, unweighted complete graph ($K_n$ of order $n$, size $m$) into minimum number of edge-disjoint trees. We find that $x$, a positive integer, is minimum and $x=\lceil\frac{n}{2}\rceil$ as the edge set of $K_n$ is decomposed into edge-disjoint trees of size sequence $M = \{m_1,m_2,...,m_x\}$ where $m_i\le(n-1)$ and $Σ_{i=1}^{x} m_i$ = $\frac{n(n-1)}{2}$. For decomposing the edge set of $K_n$ into minimum number of edge-disjoint trees, our proposed algorithm takes total $O(m)$ time.
It's Not a Modality Gap: Characterizing and Addressing the Contrastive Gap
Venue: arXiv cs.CV 2024
Year: 2024

Multi-modal contrastive models such as CLIP achieve state-of-the-art performance in zero-shot classification by embedding input images and texts on a joint representational space. Recently, a modality gap has been reported in two-encoder contrastive models like CLIP, meaning that the image and text embeddings reside in disjoint areas of the latent space. Previous studies suggest that this gap exists due to 1) the cone effect, 2) mismatched pairs in the dataset, and 3) insufficient training. We show that, even when accounting for all these factors, and even when using the same modality, the contrastive loss actually creates a gap during training. As a result, We propose that the modality gap is inherent to the two-encoder contrastive loss and rename it the contrastive gap. We present evidence that attributes this contrastive gap to low uniformity in CLIP space, resulting in embeddings that occupy only a small portion of the latent space. To close the gap, we adapt the uniformity and alignment properties of unimodal contrastive loss to the multi-modal setting and show that simply adding these terms to the CLIP loss distributes the embeddings more uniformly in the representational space, closing the gap. In our experiments, we show that the modified representational space achieves better performance than default CLIP loss in downstream tasks such as zero-shot image classification and multi-modal arithmetic.
Lost in Interpretation: Navigating Challenges in Validating Persistency Models Amid Vague Specs and Stubborn Machines, All with a Sense of Humour
Venue: arXiv cs.PL 2024
Year: 2024

Memory persistency models provide a foundation for persistent programming by specifying which (and when) writes to non-volatile memory (NVM) become persistent. Memory persistency models for the Intel-x86 and Arm architectures have been formalised, but not empirically validated against real machines. Traditional validation methods %such as %extensive litmus testing used for memory \emph{consistency} models do not straightforwardly apply because a test program cannot directly observe when its data has become persistent: it cannot distinguish between reading data from a volatile cache and from NVM. We investigate addressing this challenge using a commercial off-the-shelf device that intercepts data on the memory bus and logs all writes in the order they reach the memory. Using this technique we conducted a litmus-testing campaign aimed at empirically validating the persistency guarantees of Intel-x86 and Arm machines. We observed writes propagating to memory out of order, and took steps to build confidence that these observations were not merely artefacts of our testing setup. However, despite gaining high confidence in the trustworthiness of our observation method, our conclusions remain largely negative. We found that the Intel-x86 architecture is not amenable to our approach, and on consulting Intel engineers discovered that there are currently no reliable methods of validating their persistency guarantees. For Arm, we found that even a machine recommended to us by a persistency expert at Arm did not match the formal Arm persistency model, due to a loophole in the specification.
Unleashing the Potential of Text-attributed Graphs: Automatic Relation Decomposition via Large Language Models
Venue: arXiv cs.AI 2024
Year: 2024

Recent advancements in text-attributed graphs (TAGs) have significantly improved the quality of node features by using the textual modeling capabilities of language models. Despite this success, utilizing text attributes to enhance the predefined graph structure remains largely unexplored. Our extensive analysis reveals that conventional edges on TAGs, treated as a single relation (e.g., hyperlinks) in previous literature, actually encompass mixed semantics (e.g., "advised by" and "participates in"). This simplification hinders the representation learning process of Graph Neural Networks (GNNs) on downstream tasks, even when integrated with advanced node features. In contrast, we discover that decomposing these edges into distinct semantic relations significantly enhances the performance of GNNs. Despite this, manually identifying and labeling of edges to corresponding semantic relations is labor-intensive, often requiring domain expertise. To this end, we introduce RoSE (Relation-oriented Semantic Edge-decomposition), a novel framework that leverages the capability of Large Language Models (LLMs) to decompose the graph structure by analyzing raw text attributes - in a fully automated manner. RoSE operates in two stages: (1) identifying meaningful relations using an LLM-based generator and discriminator, and (2) categorizing each edge into corresponding relations by analyzing textual contents associated with connected nodes via an LLM-based decomposer. Extensive experiments demonstrate that our model-agnostic framework significantly enhances node classification performance across various datasets, with improvements of up to 16% on the Wisconsin dataset.