OpenRaiser/Intern-Atlas huggingface dataset:
On the Capacity of Communication Channels with Memory and Sampled Additive Cyclostationary Gaussian Noise: Full Version with Detailed Proofs
Venue: arXiv cs.IT 2023
Year: 2023

In this work we study the capacity of interference-limited channels with memory. These channels model non-orthogonal communications scenarios, such as the non-orthogonal multiple access (NOMA) scenario and underlay cognitive communications, in which the interference from other communications signals is much stronger than the thermal noise. Interference-limited communications is expected to become a very common scenario in future wireless communications systems, such as 5G, WiFi6, and beyond. As communications signals are inherently cyclostationary in continuous time (CT), then after sampling at the receiver, the discrete-time (DT) received signal model contains the sampled desired information signal with additive sampled CT cyclostationary noise. The sampled noise can be modeled as either a DT cyclostationary process or a DT almost-cyclostationary process, where in the latter case the resulting channel is not information-stable. In a previous work we characterized the capacity of this model for the case in which the DT noise is memoryless. In the current work we come closer to practical scenarios by modelling the resulting DT noise as a finite-memory random process. The presence of memory requires the development of a new set of tools for analyzing the capacity of channels with additive non-stationary noise which has memory. Our results show, for the first time, the relationship between memory, sampling frequency synchronization and capacity, for interference-limited communications. The insights from our work provide a link between the analog and the digital time domains, which has been missing in most previous works on capacity analysis. Thus, our results can help improving spectral efficiency and suggest optimal transceiver designs for future communications paradigms.
Learning Across Decentralized Multi-Modal Remote Sensing Archives with Federated Learning
Venue: arXiv cs.CV 2023
Year: 2023

The development of federated learning (FL) methods, which aim to learn from distributed databases (i.e., clients) without accessing data on clients, has recently attracted great attention. Most of these methods assume that the clients are associated with the same data modality. However, remote sensing (RS) images in different clients can be associated with different data modalities that can improve the classification performance when jointly used. To address this problem, in this paper we introduce a novel multi-modal FL framework that aims to learn from decentralized multi-modal RS image archives for RS image classification problems. The proposed framework is made up of three modules: 1) multi-modal fusion (MF); 2) feature whitening (FW); and 3) mutual information maximization (MIM). The MF module performs iterative model averaging to learn without accessing data on clients in the case that clients are associated with different data modalities. The FW module aligns the representations learned among the different clients. The MIM module maximizes the similarity of images from different modalities. Experimental results show the effectiveness of the proposed framework compared to iterative model averaging, which is a widely used algorithm in FL. The code of the proposed framework is publicly available at https://git.tu-berlin.de/rsim/MM-FL.
Auto-Spikformer: Spikformer Architecture Search
Venue: arXiv cs.NE 2023
Year: 2023

The integration of self-attention mechanisms into Spiking Neural Networks (SNNs) has garnered considerable interest in the realm of advanced deep learning, primarily due to their biological properties. Recent advancements in SNN architecture, such as Spikformer, have demonstrated promising outcomes by leveraging Spiking Self-Attention (SSA) and Spiking Patch Splitting (SPS) modules. However, we observe that Spikformer may exhibit excessive energy consumption, potentially attributable to redundant channels and blocks. To mitigate this issue, we propose Auto-Spikformer, a one-shot Transformer Architecture Search (TAS) method, which automates the quest for an optimized Spikformer architecture. To facilitate the search process, we propose methods Evolutionary SNN neurons (ESNN), which optimizes the SNN parameters, and apply the previous method of weight entanglement supernet training, which optimizes the Vision Transformer (ViT) parameters. Moreover, we propose an accuracy and energy balanced fitness function $\mathcal{F}_{AEB}$ that jointly considers both energy consumption and accuracy, and aims to find a Pareto optimal combination that balances these two objectives. Our experimental results demonstrate the effectiveness of Auto-Spikformer, which outperforms the state-of-the-art method including CNN or ViT models that are manually or automatically designed while significantly reducing energy consumption.
How are Primary School Computer Science Curricular Reforms Contributing to Equity? Impact on Student Learning, Perception of the Discipline, and Gender Gaps
Venue: arXiv cs.CY 2023
Year: 2023

Early exposure to Computer Science (CS) for all is critical to broaden participation and promote equity in the field. But how does introducting CS into primary school curricula impact learning, perception, and gaps between groups of students? We investigate a CS-curricular reform and teacher Professional Development (PD) program from an equity standpoint by applying hierarchical regression and structural equation modelling on student learning and perception data from three studies with respectively 1384, 2433 & 1644 grade 3-6 students (ages 7-11) and their 83, 142 & 95 teachers. Regarding learning, exposure to CS instruction appears to contribute to closing the performance gap between low-achieving and high-achieving students, as well as pre-existing gender gaps. Despite a lack of direct influence of what was taught on student learning, there is no impact of teachers' demographics or motivation on student learning, with teachers' perception of the CS-PD positively influencing learning. Regarding perception, students perceive CS and its teaching tools (robotics, tablets) positively, and even more so when they perceive a role model close to them as doing CS. Nonetheless gender differences exist all around with boys perceiving CS more positively than girls despite access to CS education. However, access to CS-education affects boys and girls differently: larger gender gaps are closing (namely those related to robotics), while smaller gaps are increasing (namely those related to CS and tablets). To conclude, our findings highlight how a CS curricular reform impacts learning, perception, and equity and supports the importance of i) early introductions to CS for all, ii) preparing teachers to teach CS all the while removing the influence of teacher demographics and motivation on student outcomes, and iii) having developmentally appropriate activities that signal to all groups of students.
A deep-learning approach to early identification of suggested sexual harassment from videos
Venue: arXiv cs.CV 2023
Year: 2023

Sexual harassment, sexual abuse, and sexual violence are prevalent problems in this day and age. Women's safety is an important issue that needs to be highlighted and addressed. Given this issue, we have studied each of these concerns and the factors that affect it based on images generated from movies. We have classified the three terms (harassment, abuse, and violence) based on the visual attributes present in images depicting these situations. We identified that factors such as facial expression of the victim and perpetrator and unwanted touching had a direct link to identifying the scenes containing sexual harassment, abuse and violence. We also studied and outlined how state-of-the-art explicit content detectors such as Google Cloud Vision API and Clarifai API fail to identify and categorise these images. Based on these definitions and characteristics, we have developed a first-of-its-kind dataset from various Indian movie scenes. These scenes are classified as sexual harassment, sexual abuse, or sexual violence and exported in the PASCAL VOC 1.1 format. Our dataset is annotated on the identified relevant features and can be used to develop and train a deep-learning computer vision model to identify these issues. The dataset is publicly available for research and development.
IQL-TD-MPC: Implicit Q-Learning for Hierarchical Model Predictive Control
Venue: arXiv cs.LG 2023
Year: 2023

Model-based reinforcement learning (RL) has shown great promise due to its sample efficiency, but still struggles with long-horizon sparse-reward tasks, especially in offline settings where the agent learns from a fixed dataset. We hypothesize that model-based RL agents struggle in these environments due to a lack of long-term planning capabilities, and that planning in a temporally abstract model of the environment can alleviate this issue. In this paper, we make two key contributions: 1) we introduce an offline model-based RL algorithm, IQL-TD-MPC, that extends the state-of-the-art Temporal Difference Learning for Model Predictive Control (TD-MPC) with Implicit Q-Learning (IQL); 2) we propose to use IQL-TD-MPC as a Manager in a hierarchical setting with any off-the-shelf offline RL algorithm as a Worker. More specifically, we pre-train a temporally abstract IQL-TD-MPC Manager to predict "intent embeddings", which roughly correspond to subgoals, via planning. We empirically show that augmenting state representations with intent embeddings generated by an IQL-TD-MPC manager significantly improves off-the-shelf offline RL agents' performance on some of the most challenging D4RL benchmark tasks. For instance, the offline RL algorithms AWAC, TD3-BC, DT, and CQL all get zero or near-zero normalized evaluation scores on the medium and large antmaze tasks, while our modification gives an average score over 40.
Blockchain-based Decentralized Co-governance: Innovations and Solutions for Sustainable Crowdfunding
Venue: arXiv cs.CY 2023
Year: 2023

This thesis provides an in-depth exploration of the Decentralized Co-governance Crowdfunding (DCC) Ecosystem, a novel solution addressing prevailing challenges in conventional crowdfunding methods faced by MSMEs and innovative projects. Among the problems it seeks to mitigate are high transaction costs, lack of transparency, fraud, and inefficient resource allocation. Leveraging a comprehensive review of the existing literature on crowdfunding economic activities and blockchain's impact on organizational governance, we propose a transformative socio-economic model based on digital tokens and decentralized co-governance. This ecosystem is marked by a tripartite community structure - the Labor, Capital, and Governance communities - each contributing uniquely to the ecosystem's operation. Our research unfolds the evolution of the DCC ecosystem through distinct phases, offering a novel understanding of socioeconomic dynamics in a decentralized digital world. It also delves into the intricate governance mechanism of the ecosystem, ensuring integrity, fairness, and a balanced distribution of value and wealth.
Is novelty predictable?
Venue: arXiv cs.LG 2023
Year: 2023

Machine learning-based design has gained traction in the sciences, most notably in the design of small molecules, materials, and proteins, with societal implications spanning drug development and manufacturing, plastic degradation, and carbon sequestration. When designing objects to achieve novel property values with machine learning, one faces a fundamental challenge: how to push past the frontier of current knowledge, distilled from the training data into the model, in a manner that rationally controls the risk of failure. If one trusts learned models too much in extrapolation, one is likely to design rubbish. In contrast, if one does not extrapolate, one cannot find novelty. Herein, we ponder how one might strike a useful balance between these two extremes. We focus in particular on designing proteins with novel property values, although much of our discussion addresses machine learning-based design more broadly.
Why They're Worried: Examining Experts' Motivations for Signing the 'Pause Letter'
Venue: arXiv cs.CY 2023
Year: 2023

This paper presents perspectives on the state of AI, as held by a sample of experts. These experts were early signatories of the recent open letter from Future of Life, which calls for a pause on advanced AI development. Utmost effort was put into accurately representing the perspectives of our interviewees, and they have all read and approved of their representation. However, no paper could offer a perfect portrayal of their position. We feel confident in what opinions we do put forward, but we do not hold them tightly. In such dynamic times, we feel that no one should be resolved in their expectations for AI and its future.
A Probabilistic Relaxation of the Two-Stage Object Pose Estimation Paradigm
Venue: arXiv cs.RO 2023
Year: 2023

Existing object pose estimation methods commonly require a one-to-one point matching step that forces them to be separated into two consecutive stages: visual correspondence detection (e.g., by matching feature descriptors as part of a perception front-end) followed by geometric alignment (e.g., by optimizing a robust estimation objective for pointcloud registration or perspective-n-point). Instead, we propose a matching-free probabilistic formulation with two main benefits: i) it enables unified and concurrent optimization of both visual correspondence and geometric alignment, and ii) it can represent different plausible modes of the entire distribution of likely poses. This in turn allows for a more graceful treatment of geometric perception scenarios where establishing one-to-one matches between points is conceptually ill-defined, such as textureless, symmetrical and/or occluded objects and scenes where the correct pose is uncertain or there are multiple equally valid solutions.
T2IAT: Measuring Valence and Stereotypical Biases in Text-to-Image Generation
Venue: arXiv cs.CL 2023
Year: 2023

Warning: This paper contains several contents that may be toxic, harmful, or offensive. In the last few years, text-to-image generative models have gained remarkable success in generating images with unprecedented quality accompanied by a breakthrough of inference speed. Despite their rapid progress, human biases that manifest in the training examples, particularly with regard to common stereotypical biases, like gender and skin tone, still have been found in these generative models. In this work, we seek to measure more complex human biases exist in the task of text-to-image generations. Inspired by the well-known Implicit Association Test (IAT) from social psychology, we propose a novel Text-to-Image Association Test (T2IAT) framework that quantifies the implicit stereotypes between concepts and valence, and those in the images. We replicate the previously documented bias tests on generative models, including morally neutral tests on flowers and insects as well as demographic stereotypical tests on diverse social attributes. The results of these experiments demonstrate the presence of complex stereotypical behaviors in image generations.
Improving and Benchmarking Offline Reinforcement Learning Algorithms
Venue: arXiv cs.LG 2023
Year: 2023

Recently, Offline Reinforcement Learning (RL) has achieved remarkable progress with the emergence of various algorithms and datasets. However, these methods usually focus on algorithmic advancements, ignoring that many low-level implementation choices considerably influence or even drive the final performance. As a result, it becomes hard to attribute the progress in Offline RL as these choices are not sufficiently discussed and aligned in the literature. In addition, papers focusing on a dataset (e.g., D4RL) often ignore algorithms proposed on another dataset (e.g., RL Unplugged), causing isolation among the algorithms, which might slow down the overall progress. Therefore, this work aims to bridge the gaps caused by low-level choices and datasets. To this end, we empirically investigate 20 implementation choices using three representative algorithms (i.e., CQL, CRR, and IQL) and present a guidebook for choosing implementations. Following the guidebook, we find two variants CRR+ and CQL+ , achieving new state-of-the-art on D4RL. Moreover, we benchmark eight popular offline RL algorithms across datasets under unified training and evaluation framework. The findings are inspiring: the success of a learning paradigm severely depends on the data distribution, and some previous conclusions are biased by the dataset used. Our code is available at https://github.com/sail-sg/offbench.
Examining the Emergence of Deductive Reasoning in Generative Language Models
Venue: arXiv cs.CL 2023
Year: 2023

We conduct a preliminary inquiry into the ability of generative transformer models to deductively reason from premises provided. We observe notable differences in the performance of models coming from different training setups and find that the deductive reasoning ability increases with scale. Further, we discover that the performance generally does not decrease with the length of the deductive chain needed to reach the conclusion, with the exception of OpenAI GPT-3 and GPT-3.5 models. Our study considers a wide variety of transformer-decoder models, ranging from 117 million to 175 billion parameters in size.
PV2TEA: Patching Visual Modality to Textual-Established Information Extraction
Venue: arXiv cs.CL 2023
Year: 2023

Information extraction, e.g., attribute value extraction, has been extensively studied and formulated based only on text. However, many attributes can benefit from image-based extraction, like color, shape, pattern, among others. The visual modality has long been underutilized, mainly due to multimodal annotation difficulty. In this paper, we aim to patch the visual modality to the textual-established attribute information extractor. The cross-modality integration faces several unique challenges: (C1) images and textual descriptions are loosely paired intra-sample and inter-samples; (C2) images usually contain rich backgrounds that can mislead the prediction; (C3) weakly supervised labels from textual-established extractors are biased for multimodal training. We present PV2TEA, an encoder-decoder architecture equipped with three bias reduction schemes: (S1) Augmented label-smoothed contrast to improve the cross-modality alignment for loosely-paired image and text; (S2) Attention-pruning that adaptively distinguishes the visual foreground; (S3) Two-level neighborhood regularization that mitigates the label textual bias via reliability estimation. Empirical results on real-world e-Commerce datasets demonstrate up to 11.74% absolute (20.97% relatively) F1 increase over unimodal baselines.
When Edge Computing Meets Compact Data Structures
Venue: arXiv cs.DC 2023
Year: 2023

Edge computing enables data processing and storage closer to where the data are created. Given the largely distributed compute environment and the significantly dispersed data distribution, there are increasing demands of data sharing and collaborative processing on the edge. Since data shuffling can dominate the overall execution time of collaborative processing jobs, considering the limited power supply and bandwidth resource in edge environments, it is crucial and valuable to reduce the communication overhead across edge devices. Compared with data compression, compact data structures (CDS) seem to be more suitable in this case, for the capability of allowing data to be queried, navigated, and manipulated directly in a compact form. However, the relevant work about applying CDS to edge computing generally focuses on the intuitive benefit from reduced data size, while few discussions about the challenges are given, not to mention empirical investigations into real-world edge use cases. This research highlights the challenges, opportunities, and potential scenarios of CDS implementation in edge computing. Driven by the use case of shuffling-intensive data analytics, we proposed a three-layer architecture for CDS-aided data processing and particularly studied the feasibility and efficiency of the CDS layer. We expect this research to foster conjoint research efforts on CDS-aided edge data analytics and to make wider practical impacts.
Exploration on HuBERT with Multiple Resolutions
Venue: arXiv cs.SD 2023
Year: 2023

Hidden-unit BERT (HuBERT) is a widely-used self-supervised learning (SSL) model in speech processing. However, we argue that its fixed 20ms resolution for hidden representations would not be optimal for various speech-processing tasks since their attributes (e.g., speaker characteristics and semantics) are based on different time scales. To address this limitation, we propose utilizing HuBERT representations at multiple resolutions for downstream tasks. We explore two approaches, namely the parallel and hierarchical approaches, for integrating HuBERT features with different resolutions. Through experiments, we demonstrate that HuBERT with multiple resolutions outperforms the original model. This highlights the potential of utilizing multiple resolutions in SSL models like HuBERT to capture diverse information from speech signals.
UCAS-IIE-NLP at SemEval-2023 Task 12: Enhancing Generalization of Multilingual BERT for Low-resource Sentiment Analysis
Venue: arXiv cs.CL 2023
Year: 2023

This paper describes our system designed for SemEval-2023 Task 12: Sentiment analysis for African languages. The challenge faced by this task is the scarcity of labeled data and linguistic resources in low-resource settings. To alleviate these, we propose a generalized multilingual system SACL-XLMR for sentiment analysis on low-resource languages. Specifically, we design a lexicon-based multilingual BERT to facilitate language adaptation and sentiment-aware representation learning. Besides, we apply a supervised adversarial contrastive learning technique to learn sentiment-spread structured representations and enhance model generalization. Our system achieved competitive results, largely outperforming baselines on both multilingual and zero-shot sentiment classification subtasks. Notably, the system obtained the 1st rank on the zero-shot classification subtask in the official ranking. Extensive experiments demonstrate the effectiveness of our system.
The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only
Venue: arXiv cs.CL 2023
Year: 2023

Large language models are commonly trained on a mixture of filtered web data and curated high-quality corpora, such as social media conversations, books, or technical papers. This curation process is believed to be necessary to produce performant models with broad zero-shot generalization abilities. However, as larger models requiring pretraining on trillions of tokens are considered, it is unclear how scalable is curation and whether we will run out of unique high-quality data soon. At variance with previous beliefs, we show that properly filtered and deduplicated web data alone can lead to powerful models; even significantly outperforming models from the state-of-the-art trained on The Pile. Despite extensive filtering, the high-quality data we extract from the web is still plentiful, and we are able to obtain five trillion tokens from CommonCrawl. We publicly release an extract of 600 billion tokens from our RefinedWeb dataset, and 1.3/7.5B parameters language models trained on it.