OpenRaiser/Intern-Atlas huggingface dataset:
Virtual Sectorization to Enable Hybrid Beamforming in mm-Wave mMIMO
Venue: arXiv cs.OH 2024
Year: 2024

Hybrid beamforming (HBF) is a key technology to enable mm-wave Massive multiple-input multiple-output (mMIMO) receivers for future-generation wireless communications. It combines beamforming in both analog (via phase shifters) and digital domains, resulting in low power consumption and high spectral efficiency. In practice, the problem of joint beamforming in multi-user scenarios is still open because an analog beam can't cover all users simultaneously. In this paper, we propose a hierarchical approach to divide users into clusters. Each cluster consists of users inside a virtual sector produced by the analog beamforming of an HBF-based mMIMO receiver. Thus, inside each sector, a lower-cost digital beamforming serves a limited number of users within the same cluster. Simulations with realistic non-line-of-sight scenarios generated by the QuaDRiGa 2.0 demonstrate that our methods outperform standard FFT-based alternatives and almost achieve SVD-based beamspace performance bound.
Deep Neural Networks with 3D Point Clouds for Empirical Friction Measurements in Hydrodynamic Flood Models
Venue: arXiv cs.LG 2024
Year: 2024

Friction is one of the cruxes of hydrodynamic modeling; flood conditions are highly sensitive to the Friction Factors (FFs) used to calculate momentum losses. However, empirical FFs are challenging to measure because they require laboratory experiments. Flood models often rely on surrogate observations (such as land use) to estimate FFs, introducing uncertainty. This research presents a laboratory-trained Deep Neural Network (DNN), trained using flume experiments with data augmentation techniques, to measure Manning's n based on Point Cloud data. The DNN was deployed on real-world lidar Point Clouds to directly measure Manning's n under regulatory and extreme storm events, showing improved prediction capabilities in both 1D and 2D hydrodynamic models. For 1D models, the lidar values decreased differences with regulatory models for in-channel water depth when compared to land cover values. For 1D/2D coupled models, the lidar values produced better agreement with flood extents measured from airborne imagery, while better matching flood insurance claim data for Hurricane Harvey. In both 1D and 1D/2D coupled models, lidar resulted in better agreement with validation gauges. For these reasons, the lidar measurements of Manning's n were found to improve both regulatory models and forecasts for extreme storm events, while simultaneously providing a pathway to standardize the measurement of FFs. Changing FFs significantly affected fluvial and pluvial flood models, while surge flooding was generally unaffected. Downstream flow conditions were found to change the importance of FFs to fluvial models, advancing the literature of friction in flood models. This research introduces a reliable, repeatable, and readily-accessible avenue to measure high-resolution FFs based on 3D point clouds, improving flood prediction, and removing uncertainty from hydrodynamic modeling.
Mixture-of-Depths: Dynamically allocating compute in transformer-based language models
Venue: arXiv cs.LG 2024
Year: 2024

Transformer-based language models spread FLOPs uniformly across input sequences. In this work we demonstrate that transformers can instead learn to dynamically allocate FLOPs (or compute) to specific positions in a sequence, optimising the allocation along the sequence for different layers across the model depth. Our method enforces a total compute budget by capping the number of tokens ($k$) that can participate in the self-attention and MLP computations at a given layer. The tokens to be processed are determined by the network using a top-$k$ routing mechanism. Since $k$ is defined a priori, this simple procedure uses a static computation graph with known tensor sizes, unlike other conditional computation techniques. Nevertheless, since the identities of the $k$ tokens are fluid, this method can expend FLOPs non-uniformly across the time and model depth dimensions. Thus, compute expenditure is entirely predictable in sum total, but dynamic and context-sensitive at the token-level. Not only do models trained in this way learn to dynamically allocate compute, they do so efficiently. These models match baseline performance for equivalent FLOPS and wall-clock times to train, but require a fraction of the FLOPs per forward pass, and can be upwards of 50\% faster to step during post-training sampling.
A Change of Scenery: Transformative Insights from Retrospective VR Embodied Perspective-Taking of Conflict With a Close Other
Venue: arXiv cs.HC 2024
Year: 2024

Close relationships are irreplaceable social resources, yet prone to high-risk conflict. Building on findings from the fields of HCI, virtual reality, and behavioral therapy, we evaluate the unexplored potential of retrospective VR-embodied perspective-taking to fundamentally influence conflict resolution in close others. We develop a biographically-accurate Retrospective Embodied Perspective-Taking system (REPT) and conduct a mixed-methods evaluation of its influence on close others' reflection and communication, compared to video-based reflection methods currently used in therapy (treatment as usual, or TAU). Our key findings provide evidence that REPT was able to significantly improve communication skills and positive sentiment of both partners during conflict, over TAU. The qualitative data also indicated that REPT surpassed basic perspective-taking by exclusively stimulating users to embody and reflect on both their own and their partner's experiences at the same level. In light of these findings, we provide implications and an agenda for social embodiment in HCI design: conceptualizing the use of `embodied social cognition,' and envisioning socially-embodied experiences as an interactive context.
Learning Intersections of Halfspaces with Distribution Shift: Improved Algorithms and SQ Lower Bounds
Venue: arXiv cs.DS 2024
Year: 2024

Recent work of Klivans, Stavropoulos, and Vasilyan initiated the study of testable learning with distribution shift (TDS learning), where a learner is given labeled samples from training distribution $\mathcal{D}$, unlabeled samples from test distribution $\mathcal{D}'$, and the goal is to output a classifier with low error on $\mathcal{D}'$ whenever the training samples pass a corresponding test. Their model deviates from all prior work in that no assumptions are made on $\mathcal{D}'$. Instead, the test must accept (with high probability) when the marginals of the training and test distributions are equal. Here we focus on the fundamental case of intersections of halfspaces with respect to Gaussian training distributions and prove a variety of new upper bounds including a $2^{(k/ε)^{O(1)}} \mathsf{poly}(d)$-time algorithm for TDS learning intersections of $k$ homogeneous halfspaces to accuracy $ε$ (prior work achieved $d^{(k/ε)^{O(1)}}$). We work under the mild assumption that the Gaussian training distribution contains at least an $ε$ fraction of both positive and negative examples ($ε$-balanced). We also prove the first set of SQ lower-bounds for any TDS learning problem and show (1) the $ε$-balanced assumption is necessary for $\mathsf{poly}(d,1/ε)$-time TDS learning for a single halfspace and (2) a $d^{\tildeΩ(\log 1/ε)}$ lower bound for the intersection of two general halfspaces, even with the $ε$-balanced assumption. Our techniques significantly expand the toolkit for TDS learning. We use dimension reduction and coverings to give efficient algorithms for computing a localized version of discrepancy distance, a key metric from the domain adaptation literature.
Obfuscated Malware Detection: Investigating Real-world Scenarios through Memory Analysis
Venue: arXiv cs.CR 2024
Year: 2024

In the era of the internet and smart devices, the detection of malware has become crucial for system security. Malware authors increasingly employ obfuscation techniques to evade advanced security solutions, making it challenging to detect and eliminate threats. Obfuscated malware, adept at hiding itself, poses a significant risk to various platforms, including computers, mobile devices, and IoT devices. Conventional methods like heuristic-based or signature-based systems struggle against this type of malware, as it leaves no discernible traces on the system. In this research, we propose a simple and cost-effective obfuscated malware detection system through memory dump analysis, utilizing diverse machine-learning algorithms. The study focuses on the CIC-MalMem-2022 dataset, designed to simulate real-world scenarios and assess memory-based obfuscated malware detection. We evaluate the effectiveness of machine learning algorithms, such as decision trees, ensemble methods, and neural networks, in detecting obfuscated malware within memory dumps. Our analysis spans multiple malware categories, providing insights into algorithmic strengths and limitations. By offering a comprehensive assessment of machine learning algorithms for obfuscated malware detection through memory analysis, this paper contributes to ongoing efforts to enhance cybersecurity and fortify digital ecosystems against evolving and sophisticated malware threats. The source code is made open-access for reproducibility and future research endeavours. It can be accessed at https://bit.ly/MalMemCode.
Performance Analysis and ISI Mitigation with Imperfect Transmitter in Molecular Communication
Venue: arXiv cs.IT 2024
Year: 2024

In molecular communication (MC), molecules are released from the transmitter to convey information. This paper considers a realistic molecule shift keying (MoSK) scenario with two species of molecule in two reservoirs, where the molecules are harvested from the environment and placed into different reservoirs, which are purified by exchanging molecules between the reservoirs. This process consumes energy, and for a reasonable energy cost, the reservoirs cannot be pure; thus, our MoSK transmitter is imperfect, releasing mixtures of both molecules for every symbol, resulting in inter-symbol interference (ISI). To mitigate ISI, the properties of the receiver are analyzed and a detection method based on the ratio of different molecules is proposed. Theoretical and simulation results are provided, showing that with the increase of energy cost, the system achieves better performance. The good performance of the proposed detection scheme is also demonstrated.
CMULAB: An Open-Source Framework for Training and Deployment of Natural Language Processing Models
Venue: arXiv cs.CL 2024
Year: 2024

Effectively using Natural Language Processing (NLP) tools in under-resourced languages requires a thorough understanding of the language itself, familiarity with the latest models and training methodologies, and technical expertise to deploy these models. This could present a significant obstacle for language community members and linguists to use NLP tools. This paper introduces the CMU Linguistic Annotation Backend, an open-source framework that simplifies model deployment and continuous human-in-the-loop fine-tuning of NLP models. CMULAB enables users to leverage the power of multilingual models to quickly adapt and extend existing tools for speech recognition, OCR, translation, and syntactic analysis to new languages, even with limited training data. We describe various tools and APIs that are currently available and how developers can easily add new models/functionality to the framework. Code is available at https://github.com/neulab/cmulab along with a live demo at https://cmulab.dev
Auxiliary task demands mask the capabilities of smaller language models
Venue: arXiv cs.CL 2024
Year: 2024

Developmental psychologists have argued about when cognitive capacities such as language understanding or theory of mind emerge. These debates often hinge on the concept of "task demands" -- the auxiliary challenges associated with performing a particular evaluation -- that may mask the child's underlying ability. The same issues arise when measuring the capacities of language models (LMs): performance on a task is a function of the model's underlying knowledge, combined with the model's ability to interpret and perform the task given its available resources. Here, we show that for analogical reasoning, reflective reasoning, word prediction, and grammaticality judgments, evaluation methods with greater task demands yield lower performance than evaluations with reduced demands. This "demand gap" is most pronounced for models with fewer parameters and less training data. Our results illustrate that LM performance should not be interpreted as a direct indication of intelligence (or lack thereof), but as a reflection of capacities seen through the lens of researchers' design choices.
The Societal Implications of Blockchain Proliferation
Venue: arXiv cs.CY 2024
Year: 2024

Blockchain and its distributed ledger technology have far-reaching implications for consumers across the world. Cryptocurrencies like XRP work to solve key issues in the remittance industry, targeting corridors like Mexico where foreign remittance fuels economies. Blockchain's libertarian principles have the potential to change lives in the third world, replacing corrupt infrastructure with trust-based solutions. While this technology can be used to significantly improve lives, it has a wealth of destructive applications. Bitcoin's blockchain and nefarious websites like the Silk Road have fueled an underground market of drugs, money laundering, and terrorism, complicating digital currency legislation. The negative environmental effects of cryptocurrency may also contribute significantly to global climate change. Negatives aside, cryptocurrency still proves to be a valuable commodity in technological development.
An Interpretable Client Decision Tree Aggregation process for Federated Learning
Venue: arXiv cs.LG 2024
Year: 2024

Trustworthy Artificial Intelligence solutions are essential in today's data-driven applications, prioritizing principles such as robustness, safety, transparency, explainability, and privacy among others. This has led to the emergence of Federated Learning as a solution for privacy and distributed machine learning. While decision trees, as self-explanatory models, are ideal for collaborative model training across multiple devices in resource-constrained environments such as federated learning environments for injecting interpretability in these models. Decision tree structure makes the aggregation in a federated learning environment not trivial. They require techniques that can merge their decision paths without introducing bias or overfitting while keeping the aggregated decision trees robust and generalizable. In this paper, we propose an Interpretable Client Decision Tree Aggregation process for Federated Learning scenarios that keeps the interpretability and the precision of the base decision trees used for the aggregation. This model is based on aggregating multiple decision paths of the decision trees and can be used on different decision tree types, such as ID3 and CART. We carry out the experiments within four datasets, and the analysis shows that the tree built with the model improves the local models, and outperforms the state-of-the-art.
Learn to Disguise: Avoid Refusal Responses in LLM's Defense via a Multi-agent Attacker-Disguiser Game
Venue: arXiv cs.AI 2024
Year: 2024

With the enhanced performance of large models on natural language processing tasks, potential moral and ethical issues of large models arise. There exist malicious attackers who induce large models to jailbreak and generate information containing illegal, privacy-invasive information through techniques such as prompt engineering. As a result, large models counter malicious attackers' attacks using techniques such as safety alignment. However, the strong defense mechanism of the large model through rejection replies is easily identified by attackers and used to strengthen attackers' capabilities. In this paper, we propose a multi-agent attacker-disguiser game approach to achieve a weak defense mechanism that allows the large model to both safely reply to the attacker and hide the defense intent. First, we construct a multi-agent framework to simulate attack and defense scenarios, playing different roles to be responsible for attack, disguise, safety evaluation, and disguise evaluation tasks. After that, we design attack and disguise game algorithms to optimize the game strategies of the attacker and the disguiser and use the curriculum learning process to strengthen the capabilities of the agents. The experiments verify that the method in this paper is more effective in strengthening the model's ability to disguise the defense intent compared with other methods. Moreover, our approach can adapt any black-box large model to assist the model in defense and does not suffer from model version iterations.
ANGOFA: Leveraging OFA Embedding Initialization and Synthetic Data for Angolan Language Model
Venue: arXiv cs.CL 2024
Year: 2024

In recent years, the development of pre-trained language models (PLMs) has gained momentum, showcasing their capacity to transcend linguistic barriers and facilitate knowledge transfer across diverse languages. However, this progress has predominantly bypassed the inclusion of very-low resource languages, creating a notable void in the multilingual landscape. This paper addresses this gap by introducing four tailored PLMs specifically finetuned for Angolan languages, employing a Multilingual Adaptive Fine-tuning (MAFT) approach. In this paper, we survey the role of informed embedding initialization and synthetic data in enhancing the performance of MAFT models in downstream tasks. We improve baseline over SOTA AfroXLMR-base (developed through MAFT) and OFA (an effective embedding initialization) by 12.3 and 3.8 points respectively.
CSEPrompts: A Benchmark of Introductory Computer Science Prompts
Venue: arXiv cs.CL 2024
Year: 2024

Recent advances in AI, machine learning, and NLP have led to the development of a new generation of Large Language Models (LLMs) that are trained on massive amounts of data and often have trillions of parameters. Commercial applications (e.g., ChatGPT) have made this technology available to the general public, thus making it possible to use LLMs to produce high-quality texts for academic and professional purposes. Schools and universities are aware of the increasing use of AI-generated content by students and they have been researching the impact of this new technology and its potential misuse. Educational programs in Computer Science (CS) and related fields are particularly affected because LLMs are also capable of generating programming code in various programming languages. To help understand the potential impact of publicly available LLMs in CS education, we introduce CSEPrompts, a framework with hundreds of programming exercise prompts and multiple-choice questions retrieved from introductory CS and programming courses. We also provide experimental results on CSEPrompts to evaluate the performance of several LLMs with respect to generating Python code and answering basic computer science and programming questions.
Unbiased Learning to Rank Meets Reality: Lessons from Baidu's Large-Scale Search Dataset
Venue: arXiv cs.IR 2024
Year: 2024

Unbiased learning-to-rank (ULTR) is a well-established framework for learning from user clicks, which are often biased by the ranker collecting the data. While theoretically justified and extensively tested in simulation, ULTR techniques lack empirical validation, especially on modern search engines. The Baidu-ULTR dataset released for the WSDM Cup 2023, collected from Baidu's search engine, offers a rare opportunity to assess the real-world performance of prominent ULTR techniques. Despite multiple submissions during the WSDM Cup 2023 and the subsequent NTCIR ULTRE-2 task, it remains unclear whether the observed improvements stem from applying ULTR or other learning techniques. In this work, we revisit and extend the available experiments on the Baidu-ULTR dataset. We find that standard unbiased learning-to-rank techniques robustly improve click predictions but struggle to consistently improve ranking performance, especially considering the stark differences obtained by choice of ranking loss and query-document features. Our experiments reveal that gains in click prediction do not necessarily translate to enhanced ranking performance on expert relevance annotations, implying that conclusions strongly depend on how success is measured in this benchmark.
Grid-Mapping Pseudo-Count Constraint for Offline Reinforcement Learning
Venue: arXiv cs.LG 2024
Year: 2024

Offline reinforcement learning learns from a static dataset without interacting with environments, which ensures security and thus owns a good application prospect. However, directly applying naive reinforcement learning algorithm usually fails in an offline environment due to inaccurate Q value approximation caused by out-of-distribution (OOD) state-actions. It is an effective way to solve this problem by penalizing the Q-value of OOD state-actions. Among the methods of punishing OOD state-actions, count-based methods have achieved good results in discrete domains in a simple form. Inspired by it, a novel pseudo-count method for continuous domains called Grid-Mapping Pseudo-Count method (GPC) is proposed by extending the count-based method from discrete to continuous domains. Firstly, the continuous state and action space are mapped to discrete space using Grid-Mapping, then the Q-values of OOD state-actions are constrained through pseudo-count. Secondly, the theoretical proof is given to show that GPC can obtain appropriate uncertainty constraints under fewer assumptions than other pseudo-count methods. Thirdly, GPC is combined with Soft Actor-Critic algorithm (SAC) to get a new algorithm called GPC-SAC. Lastly, experiments on D4RL datasets are given to show that GPC-SAC has better performance and less computational cost than other algorithms that constrain the Q-value.
Fusing Multi-sensor Input with State Information on TinyML Brains for Autonomous Nano-drones
Venue: arXiv cs.RO 2024
Year: 2024

Autonomous nano-drones (~10 cm in diameter), thanks to their ultra-low power TinyML-based brains, are capable of coping with real-world environments. However, due to their simplified sensors and compute units, they are still far from the sense-and-act capabilities shown in their bigger counterparts. This system paper presents a novel deep learning-based pipeline that fuses multi-sensorial input (i.e., low-resolution images and 8x8 depth map) with the robot's state information to tackle a human pose estimation task. Thanks to our design, the proposed system -- trained in simulation and tested on a real-world dataset -- improves a state-unaware State-of-the-Art baseline by increasing the R^2 regression metric up to 0.10 on the distance's prediction.