OpenRaiser/Intern-Atlas huggingface dataset:
Measuring Non-Adversarial Reproduction of Training Data in Large Language Models
Venue: arXiv cs.CL 2024
Year: 2024

Large language models memorize parts of their training data. Memorizing short snippets and facts is required to answer questions about the world and to be fluent in any language. But models have also been shown to reproduce long verbatim sequences of memorized text when prompted by a motivated adversary. In this work, we investigate an intermediate regime of memorization that we call non-adversarial reproduction, where we quantify the overlap between model responses and pretraining data when responding to natural and benign prompts. For a variety of innocuous prompt categories (e.g., writing a letter or a tutorial), we show that up to 15% of the text output by popular conversational language models overlaps with snippets from the Internet. In worst cases, we find generations where 100% of the content can be found exactly online. For the same tasks, we find that human-written text has far less overlap with Internet data. We further study whether prompting strategies can close this reproduction gap between models and humans. While appropriate prompting can reduce non-adversarial reproduction on average, we find that mitigating worst-case reproduction of training data requires stronger defenses -- even for benign interactions.
Multidimensional Byte Pair Encoding: Shortened Sequences for Improved Visual Data Generation
Venue: arXiv cs.CV 2024
Year: 2024

In language processing, transformers benefit greatly from text being condensed. This is achieved through a larger vocabulary that captures word fragments instead of plain characters. This is often done with Byte Pair Encoding. In the context of images, tokenisation of visual data is usually limited to regular grids obtained from quantisation methods, without global content awareness. Our work improves tokenisation of visual data by bringing Byte Pair Encoding from 1D to multiple dimensions, as a complementary add-on to existing compression. We achieve this through counting constellations of token pairs and replacing the most frequent token pair with a newly introduced token. The multidimensionality only increases the computation time by a factor of 2 for images, making it applicable even to large datasets like ImageNet within minutes on consumer hardware. This is a lossless preprocessing step. Our evaluation shows improved training and inference performance of transformers on visual data achieved by compressing frequent constellations of tokens: The resulting sequences are shorter, with more uniformly distributed information content, e.g. condensing empty regions in an image into single tokens. As our experiments show, these condensed sequences are easier to process. We additionally introduce a strategy to amplify this compression further by clustering the vocabulary.
Clock Synchronization Is Almost Impossible with Bounded Memory
Venue: arXiv cs.DC 2024
Year: 2024

We study the problem of clock synchronization in a networked system with arbitrary starts for all nodes. We consider a synchronous network of $n$ nodes, where each node has a local clock that is an integer counter. Eventually, clocks must be all equal and increase by one in each round modulo some period $P$. The purpose of this paper is to study whether clock synchronization can be achieved with bounded memory, that is every node maintains a number of states that does not depend on the network size. In particular, we are interested in clock synchronization algorithms which work in dynamic networks, i.e., tolerate that communication links continuously fail and come-up. We first focus on self-stabilizing solutions for clock synchronization, and prove that there is no such algorithm that is bounded memory, even in the case of static networks. More precisely, we show a lower bound of $n+1$ states at each node required to achieve clock synchronization in static strongly connected networks with at most $n$ nodes, and derive a lower bound of $n-2$ rounds on synchronization time, in the worst case. We then prove that, when the self-stabilizing requirement is removed, the impossibility of clock synchronization with bounded memory still holds in the dynamic setting: every solution for the clock synchronization problem in dynamic networks with at most $n$ nodes requires each node to have $Ω(\log n)$ states.
Ranking and Unranking of the Planar Embeddings of a Planar Graph
Venue: arXiv cs.CG 2024
Year: 2024

Let $\mathcal{G}$ be the set of all the planar embeddings of a (not necessarily connected) $n$-vertex graph $G$. We present a bijection $Φ$ from $\mathcal{G}$ to the natural numbers in the interval $[0 \dots |\mathcal{G}| - 1]$. Given a planar embedding $\mathcal{E}$ of $G$, we show that $Φ(\mathcal{E})$ can be decomposed into a sequence of $O(n)$ natural numbers each describing a specific feature of $\mathcal{E}$. The function $Φ$, which is a ranking function for $\mathcal{G}$, can be computed in $O(n)$ time, while its inverse unranking function $Φ^{-1}$ can be computed in $O(n α(n))$ time. The results of this paper can be of practical use to uniformly at random generating the planar embeddings of a graph $G$ or to enumerating such embeddings with amortized constant delay. Also, they can be used to counting, enumerating or uniformly at random generating constrained planar embeddings of $G$.
BiDense: Binarization for Dense Prediction
Venue: arXiv cs.CV 2024
Year: 2024

Dense prediction is a critical task in computer vision. However, previous methods often require extensive computational resources, which hinders their real-world application. In this paper, we propose BiDense, a generalized binary neural network (BNN) designed for efficient and accurate dense prediction tasks. BiDense incorporates two key techniques: the Distribution-adaptive Binarizer (DAB) and the Channel-adaptive Full-precision Bypass (CFB). The DAB adaptively calculates thresholds and scaling factors for binarization, effectively retaining more information within BNNs. Meanwhile, the CFB facilitates full-precision bypassing for binary convolutional layers undergoing various channel size transformations, which enhances the propagation of real-valued signals and minimizes information loss. By leveraging these techniques, BiDense preserves more real-valued information, enabling more accurate and detailed dense predictions in BNNs. Extensive experiments demonstrate that our framework achieves performance levels comparable to full-precision models while significantly reducing memory usage and computational costs.
Weakly-Supervised Multimodal Learning on MIMIC-CXR
Venue: arXiv cs.LG 2024
Year: 2024

Multimodal data integration and label scarcity pose significant challenges for machine learning in medical settings. To address these issues, we conduct an in-depth evaluation of the newly proposed Multimodal Variational Mixture-of-Experts (MMVM) VAE on the challenging MIMIC-CXR dataset. Our analysis demonstrates that the MMVM VAE consistently outperforms other multimodal VAEs and fully supervised approaches, highlighting its strong potential for real-world medical applications.
Continual Adversarial Reinforcement Learning (CARL) of False Data Injection detection: forgetting and explainability
Venue: arXiv cs.LG 2024
Year: 2024

False data injection attacks (FDIAs) on smart inverters are a growing concern linked to increased renewable energy production. While data-based FDIA detection methods are also actively developed, we show that they remain vulnerable to impactful and stealthy adversarial examples that can be crafted using Reinforcement Learning (RL). We propose to include such adversarial examples in data-based detection training procedure via a continual adversarial RL (CARL) approach. This way, one can pinpoint the deficiencies of data-based detection, thereby offering explainability during their incremental improvement. We show that a continual learning implementation is subject to catastrophic forgetting, and additionally show that forgetting can be addressed by employing a joint training strategy on all generated FDIA scenarios.
Towards High-Fidelity 3D Portrait Generation with Rich Details by Cross-View Prior-Aware Diffusion
Venue: arXiv cs.CV 2024
Year: 2024

Recent diffusion-based Single-image 3D portrait generation methods typically employ 2D diffusion models to provide multi-view knowledge, which is then distilled into 3D representations. However, these methods usually struggle to produce high-fidelity 3D models, frequently yielding excessively blurred textures. We attribute this issue to the insufficient consideration of cross-view consistency during the diffusion process, resulting in significant disparities between different views and ultimately leading to blurred 3D representations. In this paper, we address this issue by comprehensively exploiting multi-view priors in both the conditioning and diffusion procedures to produce consistent, detail-rich portraits. From the conditioning standpoint, we propose a Hybrid Priors Diffsion model, which explicitly and implicitly incorporates multi-view priors as conditions to enhance the status consistency of the generated multi-view portraits. From the diffusion perspective, considering the significant impact of the diffusion noise distribution on detailed texture generation, we propose a Multi-View Noise Resamplig Strategy integrated within the optimization process leveraging cross-view priors to enhance representation consistency. Extensive experiments demonstrate that our method can produce 3D portraits with accurate geometry and rich details from a single image. The project page is at \url{https://haoran-wei.github.io/Portrait-Diffusion}.
Low-Latency Task-Oriented Communications with Multi-Round, Multi-Task Deep Learning
Venue: arXiv cs.LG 2024
Year: 2024

In this paper, we address task-oriented (or goal-oriented) communications where an encoder at the transmitter learns compressed latent representations of data, which are then transmitted over a wireless channel. At the receiver, a decoder performs a machine learning task, specifically for classifying the received signals. The deep neural networks corresponding to the encoder-decoder pair are jointly trained, taking both channel and data characteristics into account. Our objective is to achieve high accuracy in completing the underlying task while minimizing the number of channel uses determined by the encoder's output size. To this end, we propose a multi-round, multi-task learning (MRMTL) approach for the dynamic update of channel uses in multi-round transmissions. The transmitter incrementally sends an increasing number of encoded samples over the channel based on the feedback from the receiver, and the receiver utilizes the signals from a previous round to enhance the task performance, rather than only considering the latest transmission. This approach employs multi-task learning to jointly optimize accuracy across varying number of channel uses, treating each configuration as a distinct task. By evaluating the confidence of the receiver in task decisions, MRMTL decides on whether to allocate additional channel uses in multiple rounds. We characterize both the accuracy and the delay (total number of channel uses) of MRMTL, demonstrating that it achieves the accuracy close to that of conventional methods requiring large numbers of channel uses, but with reduced delay by incorporating signals from a prior round. We consider the CIFAR-10 dataset, convolutional neural network architectures, and AWGN and Rayleigh channel models for performance evaluation. We show that MRMTL significantly improves the efficiency of task-oriented communications, balancing accuracy and latency effectively.
MICA: Medical Intelligent Conversational AgentHow to optimize medical teleconsultations for sports patients via a conversational agent?
Venue: arXiv cs.HC 2024
Year: 2024

The coronavirus crisis has increased medical teleconsultations (in France) by 10 times between 2019 and the first 3 months of 2020. However, the consultation and teleconsultation process have remained much the same. To improve the patient pathway and the efficiency of a session, we have developed a conversational assistant to optimize the time spent with the patient, the quality of the information transmitted to the physicians and thus the accuracy of diagnosis. One of the chatbot's interests is to give the physicians at the beginning of the consultation key elements for further inquiry. In this way, the system aims to help the patients concentrate and well describe their problem, which leads to more meaningful interactions with physicians and helps to enable them to operate more efficiently.
Open Research Data in Spanish University Repositories
Venue: arXiv cs.DL 2024
Year: 2024

The current situation of open research data in Spanish university repositories is analyzed by means of twelve indicators that allow us to compare them with each other. The twelve self-developed indicators deal with research datasets and institutional policies linked to open access, as well as some of the key characteristics of the repositories. The methodology used consists of comparing the repositories of the different universities linked to REBIUN. The result has been that datasets in institutional repositories are scarce, and the situation is heterogeneous across the territory. This raises questions about future open access policies for research data in the country's main scientific institutions.
Beyond object identification: How train drivers evaluate the risk of collision
Venue: arXiv cs.HC 2024
Year: 2024

When trains collide with obstacles, the consequences are often severe. To assess how artificial intelligence might contribute to avoiding collisions, we need to understand how train drivers do it. What aspects of a situation do they consider when evaluating the risk of collision? In the present study, we assumed that train drivers do not only identify potential obstacles but interpret what they see in order to anticipate how the situation might unfold. However, to date it is unclear how exactly this is accomplished. Therefore, we assessed which cues train drivers use and what inferences they make. To this end, image-based expert interviews were conducted with 33 train drivers. Participants saw images with potential obstacles, rated the risk of collision, and explained their evaluation. Moreover, they were asked how the situation would need to change to decrease or increase collision risk. From their verbal reports, we extracted concepts about the potential obstacles, contexts, or consequences, and assigned these concepts to various categories (e.g., people's identity, location, movement, action, physical features, and mental states). The results revealed that especially for people, train drivers reason about their actions and mental states, and draw relations between concepts to make further inferences. These inferences systematically differ between situations. Our findings emphasise the need to understand train drivers' risk evaluation processes when aiming to enhance the safety of both human and automatic train operation.
Challenges in the Differential Classification of Individual Diagnoses from Co-Occurring Autism and ADHD Using Survey Data
Venue: arXiv cs.LG 2024
Year: 2024

Autism and Attention-Deficit Hyperactivity Disorder (ADHD) are two of the most commonly observed neurodevelopmental conditions in childhood. Providing a specific computational assessment to distinguish between the two can prove difficult and time intensive. Given the high prevalence of their co-occurrence, there is a need for scalable and accessible methods for distinguishing the co-occurrence of autism and ADHD from individual diagnoses. The first step is to identify a core set of features that can serve as the basis for behavioral feature extraction. We trained machine learning models on data from the National Survey of Children's Health to identify behaviors to target as features in automated clinical decision support systems. A model trained on the binary task of distinguishing either developmental delay (autism or ADHD) vs. neither achieved sensitivity >92% and specificity >94%, while a model trained on the 4-way classification task of autism vs. ADHD vs. both vs. none demonstrated >65% sensitivity and >66% specificity. While the performance of the binary model was respectable, the relatively low performance in the differential classification of autism and ADHD highlights the challenges that persist in achieving specificity within clinical decision support tools for developmental delays. Nevertheless, this study demonstrates the potential of applying behavioral questionnaires not traditionally used for clinical purposes towards supporting digital screening assessments for pediatric developmental delays.
Any2Any: Incomplete Multimodal Retrieval with Conformal Prediction
Venue: arXiv cs.CV 2024
Year: 2024

Autonomous agents perceive and interpret their surroundings by integrating multimodal inputs, such as vision, audio, and LiDAR. These perceptual modalities support retrieval tasks, such as place recognition in robotics. However, current multimodal retrieval systems encounter difficulties when parts of the data are missing due to sensor failures or inaccessibility, such as silent videos or LiDAR scans lacking RGB information. We propose Any2Any-a novel retrieval framework that addresses scenarios where both query and reference instances have incomplete modalities. Unlike previous methods limited to the imputation of two modalities, Any2Any handles any number of modalities without training generative models. It calculates pairwise similarities with cross-modal encoders and employs a two-stage calibration process with conformal prediction to align the similarities. Any2Any enables effective retrieval across multimodal datasets, e.g., text-LiDAR and text-time series. It achieves a Recall@5 of 35% on the KITTI dataset, which is on par with baseline models with complete modalities.
Does Prompt Formatting Have Any Impact on LLM Performance?
Venue: arXiv cs.CL 2024
Year: 2024

In the realm of Large Language Models (LLMs), prompt optimization is crucial for model performance. Although previous research has explored aspects like rephrasing prompt contexts, using various prompting techniques (like in-context learning and chain-of-thought), and ordering few-shot examples, our understanding of LLM sensitivity to prompt templates remains limited. Therefore, this paper examines the impact of different prompt templates on LLM performance. We formatted the same contexts into various human-readable templates, including plain text, Markdown, JSON, and YAML, and evaluated their impact across tasks like natural language reasoning, code generation, and translation using OpenAI's GPT models. Experiments show that GPT-3.5-turbo's performance varies by up to 40\% in a code translation task depending on the prompt template, while larger models like GPT-4 are more robust to these variations. Our analysis highlights the need to reconsider the use of fixed prompt templates, as different formats can significantly affect model performance.
Partial Evaluation, Whole-Program Compilation
Venue: arXiv cs.PL 2024
Year: 2024

There is a tension in dynamic language runtime design between speed and correctness: state-of-the-art JIT compilation, the result of enormous industrial investment and significant research, achieves heroic speedups at the cost of complexity that can result in serious correctness bugs. Much of this complexity comes from the existence of multiple tiers and the need to maintain correspondence between these separate definitions of the language's semantics; also, from the indirect nature of the semantics implicitly encoded in a compiler backend. One way to address this complexity is to automatically derive, as much as possible, the compiled code from a single source-of-truth; for example, the interpreter tier. In this work, we introduce a partial evaluator that can derive compiled code ``for free'' by specializing an interpreter with its bytecode. This transform operates on the interpreter body at a basic-block IR level and is applicable to almost unmodified existing interpreters in systems languages such as C or C++. We show the effectiveness of this new tool by applying it to the interpreter tier of an existing industrial JavaScript engine, SpiderMonkey, yielding $2.17\times$ speedups, and the PUC-Rio Lua interpreter, yielding $1.84\times$ speedups with only three hours' effort. Finally, we outline an approach to carry this work further, deriving more of the capabilities of a JIT backend from first principles while retaining semantics-preserving correctness.
Vision Eagle Attention: a new lens for advancing image classification
Venue: arXiv cs.CV 2024
Year: 2024

In computer vision tasks, the ability to focus on relevant regions within an image is crucial for improving model performance, particularly when key features are small, subtle, or spatially dispersed. Convolutional neural networks (CNNs) typically treat all regions of an image equally, which can lead to inefficient feature extraction. To address this challenge, I have introduced Vision Eagle Attention, a novel attention mechanism that enhances visual feature extraction using convolutional spatial attention. The model applies convolution to capture local spatial features and generates an attention map that selectively emphasizes the most informative regions of the image. This attention mechanism enables the model to focus on discriminative features while suppressing irrelevant background information. I have integrated Vision Eagle Attention into a lightweight ResNet-18 architecture, demonstrating that this combination results in an efficient and powerful model. I have evaluated the performance of the proposed model on three widely used benchmark datasets: FashionMNIST, Intel Image Classification, and OracleMNIST, with a primary focus on image classification. Experimental results show that the proposed approach improves classification accuracy. Additionally, this method has the potential to be extended to other vision tasks, such as object detection, segmentation, and visual tracking, offering a computationally efficient solution for a wide range of vision-based applications. Code is available at: https://github.com/MahmudulHasan11085/Vision-Eagle-Attention.git
Autonomous Sensor Exchange and Calibration for Cornstalk Nitrate Monitoring Robot
Venue: arXiv cs.RO 2024
Year: 2024

Interactive sensors are an important component of robotic systems but often require manual replacement due to wear and tear. Automating this process can enhance system autonomy and facilitate long-term deployment. We developed an autonomous sensor exchange and calibration system for an agriculture crop monitoring robot that inserts a nitrate sensor into cornstalks. A novel gripper and replacement mechanism, featuring a reliable funneling design, were developed to enable efficient and reliable sensor exchanges. To maintain consistent nitrate sensor measurement, an on-board sensor calibration station was integrated to provide in-field sensor cleaning and calibration. The system was deployed at the Ames Curtis Farm in June 2024, where it successfully inserted nitrate sensors with high accuracy into 30 cornstalks with a 77$\%$ success rate.