OpenRaiser/Intern-Atlas huggingface dataset:
Do LLMs suffer from Multi-Party Hangover? A Diagnostic Approach to Addressee Recognition and Response Selection in Conversations
Venue: arXiv cs.CL 2024
Year: 2024

Assessing the performance of systems to classify Multi-Party Conversations (MPC) is challenging due to the interconnection between linguistic and structural characteristics of conversations. Conventional evaluation methods often overlook variances in model behavior across different levels of structural complexity on interaction graphs. In this work, we propose a methodological pipeline to investigate model performance across specific structural attributes of conversations. As a proof of concept we focus on Response Selection and Addressee Recognition tasks, to diagnose model weaknesses. To this end, we extract representative diagnostic subdatasets with a fixed number of users and a good structural variety from a large and open corpus of online MPCs. We further frame our work in terms of data minimization, avoiding the use of original usernames to preserve privacy, and propose alternatives to using original text messages. Results show that response selection relies more on the textual content of conversations, while addressee recognition requires capturing their structural dimension. Using an LLM in a zero-shot setting, we further highlight how sensitivity to prompt variations is task-dependent.
Libros en abierto de las editoriales universitarias españolas
Venue: arXiv cs.DL 2024
Year: 2024

This paper analyses the set of scientific publications in open access, other than journals (monographs, conferences proceedings, teaching materials and grey literature), published by Spanish public universities, studying their volume, documentary typology, level of description and open access policies with the aim of measuring their degree of incorporation and compliance with the principles of Open Science. An exhaustive review of the disposed material in open access by these publishers has been carried out, which has allowed to make a diagnosis of their level of open access publishing. Grey literature is the most common documentary type followed by the monograph, in the open publication of these publishers that does not reach even 5% of the average editorial production. The results allow us to conclude that the academic publishing, and more specifically the academic books in open access, still has a very reduced presence within the editorial production of these institutions.
Audio-Based Linguistic Feature Extraction for Enhancing Multi-lingual and Low-Resource Text-to-Speech
Venue: arXiv cs.SD 2024
Year: 2024

The difficulty of acquiring abundant, high-quality data, especially in multi-lingual contexts, has sparked interest in addressing low-resource scenarios. Moreover, current literature rely on fixed expressions from language IDs, which results in the inadequate learning of language representations, and the failure to generate speech in unseen languages. To address these challenges, we propose a novel method that directly extracts linguistic features from audio input while effectively filtering out miscellaneous acoustic information including speaker-specific attributes like timbre. Subjective and objective evaluations affirm the effectiveness of our approach for multi-lingual text-to-speech, and highlight its superiority in low-resource transfer learning for previously unseen language.
Entropy, concentration, and learning: a statistical mechanics primer
Venue: arXiv cs.LG 2024
Year: 2024

Artificial intelligence models trained through loss minimization have demonstrated significant success, grounded in principles from fields like information theory and statistical physics. This work explores these established connections through the lens of statistical mechanics, starting from first-principles sample concentration behaviors that underpin AI and machine learning. Our development of statistical mechanics for modeling highlights the key role of exponential families, and quantities of statistics, physics, and information theory.
Enhanced Convolution Neural Network with Optimized Pooling and Hyperparameter Tuning for Network Intrusion Detection
Venue: arXiv cs.CR 2024
Year: 2024

Network Intrusion Detection Systems (NIDS) are essential for protecting computer networks from malicious activities, including Denial of Service (DoS), Probing, User-to-Root (U2R), and Remote-to-Local (R2L) attacks. Without effective NIDS, networks are vulnerable to significant security breaches and data loss. Machine learning techniques provide a promising approach to enhance NIDS by automating threat detection and improving accuracy. In this research, we propose an Enhanced Convolutional Neural Network (EnCNN) for NIDS and evaluate its performance using the KDDCUP'99 dataset. Our methodology includes comprehensive data preprocessing, exploratory data analysis (EDA), and feature engineering. We compare EnCNN with various machine learning algorithms, including Logistic Regression, Decision Trees, Support Vector Machines (SVM), and ensemble methods like Random Forest, AdaBoost, and Voting Ensemble. The results show that EnCNN significantly improves detection accuracy, with a notable 10% increase over state-of-art approaches. This demonstrates the effectiveness of EnCNN in real-time network intrusion detection, offering a robust solution for identifying and mitigating security threats, and enhancing overall network resilience.
The Craft of Selective Prediction: Towards Reliable Case Outcome Classification -- An Empirical Study on European Court of Human Rights Cases
Venue: arXiv cs.CL 2024
Year: 2024

In high-stakes decision-making tasks within legal NLP, such as Case Outcome Classification (COC), quantifying a model's predictive confidence is crucial. Confidence estimation enables humans to make more informed decisions, particularly when the model's certainty is low, or where the consequences of a mistake are significant. However, most existing COC works prioritize high task performance over model reliability. This paper conducts an empirical investigation into how various design choices including pre-training corpus, confidence estimator and fine-tuning loss affect the reliability of COC models within the framework of selective prediction. Our experiments on the multi-label COC task, focusing on European Court of Human Rights (ECtHR) cases, highlight the importance of a diverse yet domain-specific pre-training corpus for better calibration. Additionally, we demonstrate that larger models tend to exhibit overconfidence, Monte Carlo dropout methods produce reliable confidence estimates, and confident error regularization effectively mitigates overconfidence. To our knowledge, this is the first systematic exploration of selective prediction in legal NLP. Our findings underscore the need for further research on enhancing confidence measurement and improving the trustworthiness of models in the legal domain.
How green is continual learning, really? Analyzing the energy consumption in continual training of vision foundation models
Venue: arXiv cs.LG 2024
Year: 2024

With the ever-growing adoption of AI, its impact on the environment is no longer negligible. Despite the potential that continual learning could have towards Green AI, its environmental sustainability remains relatively uncharted. In this work we aim to gain a systematic understanding of the energy efficiency of continual learning algorithms. To that end, we conducted an extensive set of empirical experiments comparing the energy consumption of recent representation-, prompt-, and exemplar-based continual learning algorithms and two standard baseline (fine tuning and joint training) when used to continually adapt a pre-trained ViT-B/16 foundation model. We performed our experiments on three standard datasets: CIFAR-100, ImageNet-R, and DomainNet. Additionally, we propose a novel metric, the Energy NetScore, which we use measure the algorithm efficiency in terms of energy-accuracy trade-off. Through numerous evaluations varying the number and size of the incremental learning steps, our experiments demonstrate that different types of continual learning algorithms have very different impacts on energy consumption during both training and inference. Although often overlooked in the continual learning literature, we found that the energy consumed during the inference phase is crucial for evaluating the environmental sustainability of continual learning models.
Co-Trained Retriever-Generator Framework for Question Generation in Earnings Calls
Venue: arXiv cs.CL 2024
Year: 2024

In diverse professional environments, ranging from academic conferences to corporate earnings calls, the ability to anticipate audience questions stands paramount. Traditional methods, which rely on manual assessment of an audience's background, interests, and subject knowledge, often fall short - particularly when facing large or heterogeneous groups, leading to imprecision and inefficiency. While NLP has made strides in text-based question generation, its primary focus remains on academic settings, leaving the intricate challenges of professional domains, especially earnings call conferences, underserved. Addressing this gap, our paper pioneers the multi-question generation (MQG) task specifically designed for earnings call contexts. Our methodology involves an exhaustive collection of earnings call transcripts and a novel annotation technique to classify potential questions. Furthermore, we introduce a retriever-enhanced strategy to extract relevant information. With a core aim of generating a spectrum of potential questions that analysts might pose, we derive these directly from earnings call content. Empirical evaluations underscore our approach's edge, revealing notable excellence in the accuracy, consistency, and perplexity of the questions generated.
Cross-Domain Keyword Extraction with Keyness Patterns
Venue: arXiv cs.IR 2024
Year: 2024

Domain dependence and annotation subjectivity pose challenges for supervised keyword extraction. Based on the premises that second-order keyness patterns are existent at the community level and learnable from annotated keyword extraction datasets, this paper proposes a supervised ranking approach to keyword extraction that ranks keywords with keyness patterns consisting of independent features (such as sublanguage domain and term length) and three categories of dependent features -- heuristic features, specificity features, and representavity features. The approach uses two convolutional-neural-network based models to learn keyness patterns from keyword datasets and overcomes annotation subjectivity by training the two models with bootstrap sampling strategy. Experiments demonstrate that the approach not only achieves state-of-the-art performance on ten keyword datasets in general supervised keyword extraction with an average top-10-F-measure of 0.316 , but also robust cross-domain performance with an average top-10-F-measure of 0.346 on four datasets that are excluded in the training process. Such cross-domain robustness is attributed to the fact that community-level keyness patterns are limited in number and temperately independent of language domains, the distinction between independent features and dependent features, and the sampling training strategy that balances excess risk and lack of negative training data.
Relighting from a Single Image: Datasets and Deep Intrinsic-based Architecture
Venue: arXiv cs.CV 2024
Year: 2024

Single image scene relighting aims to generate a realistic new version of an input image so that it appears to be illuminated by a new target light condition. Although existing works have explored this problem from various perspectives, generating relit images under arbitrary light conditions remains highly challenging, and related datasets are scarce. Our work addresses this problem from both the dataset and methodological perspectives. We propose two new datasets: a synthetic dataset with the ground truth of intrinsic components and a real dataset collected under laboratory conditions. These datasets alleviate the scarcity of existing datasets. To incorporate physical consistency in the relighting pipeline, we establish a two-stage network based on intrinsic decomposition, giving outputs at intermediate steps, thereby introducing physical constraints. When the training set lacks ground truth for intrinsic decomposition, we introduce an unsupervised module to ensure that the intrinsic outputs are satisfactory. Our method outperforms the state-of-the-art methods in performance, as tested on both existing datasets and our newly developed datasets. Furthermore, pretraining our method or other prior methods using our synthetic dataset can enhance their performance on other datasets. Since our method can accommodate any light conditions, it is capable of producing animated results. The dataset, method, and videos are publicly available.
Esports Debut as a Medal Event at 2023 Asian Games: Exploring Public Perceptions with BERTopic and GPT-4 Topic Fine-Tuning
Venue: arXiv cs.HC 2024
Year: 2024

This study examined the public opinions of esports at the 2023 Asian Games and value co-creation during the event using an LLM-enhanced BERTopic modeling analysis. We identified five major themes representing public perceptions, as well as how major stakeholders co-created value within and beyond the esports ecosystem. Key findings highlighted the strategic use of social media marketing to influence public opinion and promote esports events and brands, emphasizing the importance of event logistics and infrastructure. Additionally, the study revealed the co-creation value contributed by stakeholders outside the traditional esports ecosystem, particularly in promoting national representation and performance. Our findings supported the ongoing efforts to legitimize esports as a sport, noting that mainstream recognition remains a challenge. The inclusion of esports as a medal event showcased broader acceptance and helped mitigate negative public perceptions. Moreover, contributions from non-traditional stakeholders underscored the value of cross-subcultural collaborations in esports.
MiniVLN: Efficient Vision-and-Language Navigation by Progressive Knowledge Distillation
Venue: arXiv cs.CV 2024
Year: 2024

In recent years, Embodied Artificial Intelligence (Embodied AI) has advanced rapidly, yet the increasing size of models conflicts with the limited computational capabilities of Embodied AI platforms. To address this challenge, we aim to achieve both high model performance and practical deployability. Specifically, we focus on Vision-and-Language Navigation (VLN), a core task in Embodied AI. This paper introduces a two-stage knowledge distillation framework, producing a student model, MiniVLN, and showcasing the significant potential of distillation techniques in developing lightweight models. The proposed method aims to capture fine-grained knowledge during the pretraining phase and navigation-specific knowledge during the fine-tuning phase. Our findings indicate that the two-stage distillation approach is more effective in narrowing the performance gap between the teacher model and the student model compared to single-stage distillation. On the public R2R and REVERIE benchmarks, MiniVLN achieves performance on par with the teacher model while having only about 12% of the teacher model's parameter count.
LLM With Tools: A Survey
Venue: arXiv cs.AI 2024
Year: 2024

The integration of tools in augmenting large language models presents a novel approach toward enhancing the efficiency and accuracy of these models in handling specific, complex tasks. This paper delves into the methodology,challenges, and developments in the realm of teaching LLMs to use external tools, thereby pushing the boundaries of their capabilities beyond pre-existing knowledge bases. We introduce a standardized paradigm for tool integration guided by a series of functions that map user instructions to actionable plans and their execution, emphasizing the significance of understanding user intent, tool selection, and dynamic plan adjustment. Our exploration reveals the various challenges encountered, such as tool invocation timing, selection accuracy, and the need for robust reasoning processes. In addressing these challenges, we investigate techniques within the context of fine-tuning and incontext learning paradigms, highlighting innovative approaches to ensure diversity, augment datasets, and improve generalization.Furthermore, we investigate a perspective on enabling LLMs to not only utilize but also autonomously create tools, which may redefine their role from mere tool users to tool creators. Finally,we reproduced Chameleon's results on ScienceQA and analyzed the code structure.
Fully integrating the Flang Fortran compiler with standard MLIR
Venue: arXiv cs.DC 2024
Year: 2024

Fortran is the lingua franca of HPC code development and as such it is crucial that we as a community have open source Fortran compilers capable of generating high performance executables. Flang is LLVM's Fortran compiler and leverages MLIR which is a reusable compiler infrastructure which, as part of LLVM, has become popular in recent years. However, whilst Flang leverages MLIR it does not fully integrate with it and instead provides bespoke translation and optimisation passes to target LLVM-IR. In this paper we first explore the performance of Flang against other compilers popular in HPC for a range of benchmarks before describing a mapping between Fortran and standard MLIR, exploring the performance of this. The result of this work is an up to three times speed up compared with Flang's existing approach across the benchmarks and experiments run, demonstrating that the Flang community should seriously consider leveraging standard MLIR.
Accelerating stencils on the Tenstorrent Grayskull RISC-V accelerator
Venue: arXiv cs.DC 2024
Year: 2024

The RISC-V Instruction Set Architecture (ISA) has enjoyed phenomenal growth in recent years, however it still to gain popularity in HPC. Whilst adopting RISC-V CPU solutions in HPC might be some way off, RISC-V based PCIe accelerators offer a middle ground where vendors benefit from the flexibility of RISC-V yet fit into existing systems. In this paper we focus on the Tenstorrent Grayskull PCIe RISC-V based accelerator which, built upon Tensix cores, decouples data movement from compute. Using the Jacobi iterative method as a vehicle, we explore the suitability of stencils on the Grayskull e150. We explore best practice in structuring these codes for the accelerator and demonstrate that the e150 provides similar performance to a Xeon Platinum CPU (albeit BF16 vs FP32) but the e150 uses around five times less energy. Over four e150s we obtain around four times the CPU performance, again at around five times less energy.
Pay Attention to What Matters
Venue: arXiv cs.CL 2024
Year: 2024

Despite the remarkable success of Large Language Models (LLMs), they still exhibit a limited capability to align their outputs to the user instructions. In this work, we introduce a simple and effective method, which we name GUIDE, that mechanistically increases attention scores in instruction tokens. To support this operation, we present Influence, a novel metric that highlights how the user's instructions propagate through the transformer layers and impact the LLM output. Our results show that GUIDE improves the accuracy of following instructions 29.4 % to 60.4%, outperforming natural prompting alternatives and Supervised Fine-Tuning up to 1M tokens.
Voxel-CIM: An Efficient Compute-in-Memory Accelerator for Voxel-based Point Cloud Neural Networks
Venue: arXiv cs.AR 2024
Year: 2024

The 3D point cloud perception has emerged as a fundamental role for a wide range of applications. In particular, with the rapid development of neural networks, the voxel-based networks attract great attention due to their excellent performance. Various accelerator designs have been proposed to improve the hardware performance of voxel-based networks, especially to speed up the map search process. However, several challenges still exist including: (1) massive off-chip data access volume caused by map search operations, notably for high resolution and dense distribution cases, (2) frequent data movement for data-intensive convolution operations, (3) imbalanced workload caused by irregular sparsity of point data. To address the above challenges, we propose Voxel-CIM, an efficient Compute-in-Memory based accelerator for voxel-based neural network processing. To reduce off-chip memory access for map search, a depth-encoding-based output major search approach is introduced to maximize data reuse, achieving stable $O(N)$-level data access volume in various situations. Voxel-CIM also employs the in-memory computing paradigm and designs innovative weight mapping strategies to efficiently process Sparse 3D convolutions and 2D convolutions. Implemented on 22 nm technology and evaluated on representative benchmarks, the Voxel-CIM achieves averagely 4.5~7.0$\times$ higher energy efficiency (10.8 TOPS/w), and 2.4~5.4$\times$ speed up in detection task and 1.2~8.1$\times$ speed up in segmentation task compared to the state-of-the-art point cloud accelerators and powerful GPUs.
Mechanism Design with Delegated Bidding
Venue: arXiv cs.GT 2024
Year: 2024

We consider the problem of a designer who wants to allocate resources to representatives, that then distribute the resources they receive among the individuals they represent. Motivated by the way Feeding America, one of the largest U.S. charities, allocates donations to food banks, which then further distribute the donations to food-insecure individuals, we focus on mechanisms that use artificial currencies. We compare auctions through the lens of the Price of Anarchy, with respect to three canonical welfare objectives: utilitarian social welfare (sum of individuals' utilities), Nash social welfare (product of individuals' utilities), and egalitarian social welfare (minimum of individuals' utilities). We prove strong lower bounds on the Price of Anarchy of all auctions that allocate each item to the highest bidder, subject to a mild technical constraint; this includes Feeding America's current auction, the First-Price auction. In sharp contrast, our main result shows that adapting the classic Trading Post mechanism of Shapley and Shubik to this setting, and coupled with Feeding America's choice of budget rule (each representative gets an amount of artificial currency equal to the number of individuals it represents), achieves a small Price of Anarchy for all generalized $p$-mean objectives simultaneously. Our bound on the Price of Anarchy of the Trading Post mechanism depends on $\ell$: the product of the rank and the ``incoherence'' of the underlying valuation matrix, which together capture a notion of how ``spread out'' the values of a matrix are. This notion has been extremely influential in the matrix completion literature, and, to the best of our knowledge, has never been used in auction theory prior to our work. Perhaps surprisingly, we prove that the dependence on $\ell$ is necessary: the Price of Anarchy of the Trading Post mechanism is $Ω(\sqrt{\ell})$.