OpenRaiser/Intern-Atlas huggingface dataset:
Redundancy and Concept Analysis for Code-trained Language Models
Venue: arXiv cs.SE 2024
Year: 2024

Code-trained language models have proven to be highly effective for various code intelligence tasks. However, they can be challenging to train and deploy for many software engineering applications due to computational bottlenecks and memory constraints. Implementing effective strategies to address these issues requires a better understanding of these 'black box' models. In this paper, we perform the first neuron-level analysis for source code models to identify \textit{important} neurons within latent representations. We achieve this by eliminating neurons that are highly similar or irrelevant to the given task. This approach helps us understand which neurons and layers can be eliminated (redundancy analysis) and where important code properties are located within the network (concept analysis). Using redundancy analysis, we make observations relevant to knowledge transfer and model optimization applications. We find that over 95\% of the neurons are redundant with respect to our code intelligence tasks and can be eliminated without significant loss in accuracy. We also discover several subsets of neurons that can make predictions with baseline accuracy. Through concept analysis, we explore the traceability and distribution of human-recognizable concepts within latent code representations which could be used to influence model predictions. We trace individual and subsets of important neurons to specific code properties and identify 'number' neurons, 'string' neurons, and higher-level 'text' neurons for token-level tasks and higher-level concepts important for sentence-level downstream tasks. This also helps us understand how decomposable and transferable task-related features are and can help devise better techniques for transfer learning, model compression, and the decomposition of deep neural networks into modules.
Temporal Betweenness Centrality on Shortest Walks Variants
Venue: arXiv cs.DS 2024
Year: 2024

Betweenness centrality has been extensively studied since its introduction in 1977 as a measure of node importance in graphs. This measure has found use in various applications and has been extended to temporal graphs with time-labeled edges. Recent research by Buss et al. and Rymar et al. has shown that it is possible to compute the shortest path betweenness centrality of all nodes in a temporal graph in $O(n^3\,T^2)$ and $O(n^2\,m\,T^2)$ time, respectively, where $T$ is the maximum time, $m$ is the number of temporal edges, and $n$ is the number of nodes. These approaches considered paths that do not take into account contributions from intermediate temporal nodes. In this paper, we study the classical temporal betweenness centrality paths that we call \textit{passive} shortest paths, as well as an alternative variant that we call \textit{active} shortest paths, which takes into account contributions from all temporal nodes. We present an improved analysis of the running time of the classical algorithm for computing betweenness centrality of all nodes, reducing the time complexity to $O(n\,m\,T+ n^2\,T)$. Furthermore, for active paths, we show that the betweenness centrality can be computed in $O(n\,m\,T+ n^2\,T^2)$. We also show that our results hold for different shortest paths variants. Finally, we provide an open-source implementation of our algorithms and conduct experiments on several real-world datasets. We compare the results of the two variants on both the node and time dimensions of the temporal graph, and we also compare the temporal betweenness centrality to its static counterpart. Our experiments suggest that for the shortest foremost variant looking only at the first $10\%$ of the temporal interaction is a very good approximation for the overall top ranked nodes.
The Training Process of Many Deep Networks Explores the Same Low-Dimensional Manifold
Venue: arXiv cs.LG 2024
Year: 2024

We develop information-geometric techniques to analyze the trajectories of the predictions of deep networks during training. By examining the underlying high-dimensional probabilistic models, we reveal that the training process explores an effectively low-dimensional manifold. Networks with a wide range of architectures, sizes, trained using different optimization methods, regularization techniques, data augmentation techniques, and weight initializations lie on the same manifold in the prediction space. We study the details of this manifold to find that networks with different architectures follow distinguishable trajectories but other factors have a minimal influence; larger networks train along a similar manifold as that of smaller networks, just faster; and networks initialized at very different parts of the prediction space converge to the solution along a similar manifold.
ContactArt: Learning 3D Interaction Priors for Category-level Articulated Object and Hand Poses Estimation
Venue: arXiv cs.CV 2024
Year: 2024

We propose a new dataset and a novel approach to learning hand-object interaction priors for hand and articulated object pose estimation. We first collect a dataset using visual teleoperation, where the human operator can directly play within a physical simulator to manipulate the articulated objects. We record the data and obtain free and accurate annotations on object poses and contact information from the simulator. Our system only requires an iPhone to record human hand motion, which can be easily scaled up and largely lower the costs of data and annotation collection. With this data, we learn 3D interaction priors including a discriminator (in a GAN) capturing the distribution of how object parts are arranged, and a diffusion model which generates the contact regions on articulated objects, guiding the hand pose estimation. Such structural and contact priors can easily transfer to real-world data with barely any domain gap. By using our data and learned priors, our method significantly improves the performance on joint hand and articulated object poses estimation over the existing state-of-the-art methods. The project is available at https://zehaozhu.github.io/ContactArt/ .
Stance Detection: A Practical Guide to Classifying Political Beliefs in Text
Venue: arXiv cs.CL 2024
Year: 2024

Stance detection is identifying expressed beliefs in a document. While researchers widely use sentiment analysis for this, recent research demonstrates that sentiment and stance are distinct. This paper advances text analysis methods by precisely defining stance detection and presenting three distinct approaches: supervised classification, natural language inference, and in-context learning with generative language models. I discuss how document context and trade-offs between resources and workload should inform your methods. For all three approaches I provide guidance on application and validation techniques, as well as coding tutorials for implementation. Finally, I demonstrate how newer classification approaches can replicate supervised classifiers.
Achieving Realistic Cyclist Behavior in SUMO using the SimRa Dataset
Venue: arXiv cs.MA 2024
Year: 2024

Increasing the modal share of bicycle traffic to reduce carbon emissions, reduce urban car traffic, and to improve the health of citizens, requires a shift away from car-centric city planning. For this, traffic planners often rely on simulation tools such as SUMO which allow them to study the effects of construction changes before implementing them. Similarly, studies of vulnerable road users, here cyclists, also use such models to assess the performance of communication-based road traffic safety systems. The cyclist model in SUMO, however, is very imprecise as SUMO cyclists behave either like slow cars or fast pedestrians, thus, casting doubt on simulation results for bicycle traffic. In this paper, we analyze acceleration, deceleration, velocity, and intersection left-turn behavior of cyclists in a large dataset of real world cycle tracks. We use the results to improve the existing cyclist model in SUMO and add three more detailed cyclist models and implement them in SUMO.
A Lightweight CNN-Transformer Model for Learning Traveling Salesman Problems
Venue: arXiv cs.LG 2024
Year: 2024

Several studies have attempted to solve traveling salesman problems (TSPs) using various deep learning techniques. Among them, Transformer-based models show state-of-the-art performance even for large-scale Traveling Salesman Problems (TSPs). However, they are based on fully-connected attention models and suffer from large computational complexity and GPU memory usage. Our work is the first CNN-Transformer model based on a CNN embedding layer and partial self-attention for TSP. Our CNN-Transformer model is able to better learn spatial features from input data using a CNN embedding layer compared with the standard Transformer-based models. It also removes considerable redundancy in fully-connected attention models using the proposed partial self-attention. Experimental results show that the proposed CNN embedding layer and partial self-attention are very effective in improving performance and computational complexity. The proposed model exhibits the best performance in real-world datasets and outperforms other existing state-of-the-art (SOTA) Transformer-based models in various aspects. Our code is publicly available at https://github.com/cm8908/CNN_Transformer3.
Empowering Agrifood System with Artificial Intelligence: A Survey of the Progress, Challenges and Opportunities
Venue: arXiv cs.AI 2024
Year: 2024

With the world population rapidly increasing, transforming our agrifood systems to be more productive, efficient, safe, and sustainable is crucial to mitigate potential food shortages. Recently, artificial intelligence (AI) techniques such as deep learning (DL) have demonstrated their strong abilities in various areas, including language, vision, remote sensing (RS), and agrifood systems applications. However, the overall impact of AI on agrifood systems remains unclear. In this paper, we thoroughly review how AI techniques can transform agrifood systems and contribute to the modern agrifood industry. Firstly, we summarize the data acquisition methods in agrifood systems, including acquisition, storage, and processing techniques. Secondly, we present a progress review of AI methods in agrifood systems, specifically in agriculture, animal husbandry, and fishery, covering topics such as agrifood classification, growth monitoring, yield prediction, and quality assessment. Furthermore, we highlight potential challenges and promising research opportunities for transforming modern agrifood systems with AI. We hope this survey could offer an overall picture to newcomers in the field and serve as a starting point for their further research. The project website is https://github.com/Frenkie14/Agrifood-Survey.
ProgDTD: Progressive Learned Image Compression with Double-Tail-Drop Training
Venue: arXiv cs.CV 2024
Year: 2024

Progressive compression allows images to start loading as low-resolution versions, becoming clearer as more data is received. This increases user experience when, for example, network connections are slow. Today, most approaches for image compression, both classical and learned ones, are designed to be non-progressive. This paper introduces ProgDTD, a training method that transforms learned, non-progressive image compression approaches into progressive ones. The design of ProgDTD is based on the observation that the information stored within the bottleneck of a compression model commonly varies in importance. To create a progressive compression model, ProgDTD modifies the training steps to enforce the model to store the data in the bottleneck sorted by priority. We achieve progressive compression by transmitting the data in order of its sorted index. ProgDTD is designed for CNN-based learned image compression models, does not need additional parameters, and has a customizable range of progressiveness. For evaluation, we apply ProgDTDto the hyperprior model, one of the most common structures in learned image compression. Our experimental results show that ProgDTD performs comparably to its non-progressive counterparts and other state-of-the-art progressive models in terms of MS-SSIM and accuracy.
Exploring Linguistic Properties of Monolingual BERTs with Typological Classification among Languages
Venue: arXiv cs.CL 2024
Year: 2024

The impressive achievements of transformers force NLP researchers to delve into how these models represent the underlying structure of natural language. In this paper, we propose a novel standpoint to investigate the above issue: using typological similarities among languages to observe how their respective monolingual models encode structural information. We aim to layer-wise compare transformers for typologically similar languages to observe whether these similarities emerge for particular layers. For this investigation, we propose to use Centered Kernel Alignment to measure similarity among weight matrices. We found that syntactic typological similarity is consistent with the similarity between the weights in the middle layers, which are the pretrained BERT layers to which syntax encoding is generally attributed. Moreover, we observe that a domain adaptation on semantically equivalent texts enhances this similarity among weight matrices.
Spectral influence in networks: An application to Input-Output analysis
Venue: arXiv cs.SI 2024
Year: 2024

This paper introduces the concepts of spectral influence and spectral cyclicality, both derived from the largest eigenvalue of a graph's adjacency matrix. These two novel centrality measures capture both diffusion and interdependence from a local and global perspective respectively. We propose a new clustering algorithm that identifies communities with high cyclicality and interdependence, allowing for overlaps. To illustrate our method, we apply it to input-output analysis within the context of the Moroccan economy.
Directional Antenna Based Scheduling Protocol for IoT Networks
Venue: arXiv cs.NI 2024
Year: 2024

Scheduling and Channel Access at the MAC layer of the IoT network plays a pivotal role in enhancing the performance of IoT networks. State-of-the-art Omni-directional antenna based application data transmission has relatively less achievable throughput in comparison with directional antenna based scheduling protocols. To enhance the performance of the IoT networks, this paper propose a distributed one-hop scheduling algorithm called Directional Scheduling protocol for constrained deterministic 6TiSCH-IoT network. With this, in-creased number of IoT nodes can have concurrent application data transmission with efficient spatial reuse. This in-turn results in higher number of cell allocation to the one-hop IoT nodes during data transmission. The proposed algorithm makes use of through directional transmissions avoids head of line blocking.
Towards Weakly-Supervised Hate Speech Classification Across Datasets
Venue: arXiv cs.CL 2024
Year: 2024

As pointed out by several scholars, current research on hate speech (HS) recognition is characterized by unsystematic data creation strategies and diverging annotation schemata. Subsequently, supervised-learning models tend to generalize poorly to datasets they were not trained on, and the performance of the models trained on datasets labeled using different HS taxonomies cannot be compared. To ease this problem, we propose applying extremely weak supervision that only relies on the class name rather than on class samples from the annotated data. We demonstrate the effectiveness of a state-of-the-art weakly-supervised text classification model in various in-dataset and cross-dataset settings. Furthermore, we conduct an in-depth quantitative and qualitative analysis of the source of poor generalizability of HS classification models.
When Do Neural Nets Outperform Boosted Trees on Tabular Data?
Venue: arXiv cs.LG 2024
Year: 2024

Tabular data is one of the most commonly used types of data in machine learning. Despite recent advances in neural nets (NNs) for tabular data, there is still an active discussion on whether or not NNs generally outperform gradient-boosted decision trees (GBDTs) on tabular data, with several recent works arguing either that GBDTs consistently outperform NNs on tabular data, or vice versa. In this work, we take a step back and question the importance of this debate. To this end, we conduct the largest tabular data analysis to date, comparing 19 algorithms across 176 datasets, and we find that the 'NN vs. GBDT' debate is overemphasized: for a surprisingly high number of datasets, either the performance difference between GBDTs and NNs is negligible, or light hyperparameter tuning on a GBDT is more important than choosing between NNs and GBDTs. A remarkable exception is the recently-proposed prior-data fitted network, TabPFN: although it is effectively limited to training sets of size 3000, we find that it outperforms all other algorithms on average, even when randomly sampling 3000 training datapoints. Next, we analyze dozens of metafeatures to determine what properties of a dataset make NNs or GBDTs better-suited to perform well. For example, we find that GBDTs are much better than NNs at handling skewed or heavy-tailed feature distributions and other forms of dataset irregularities. Our insights act as a guide for practitioners to determine which techniques may work best on their dataset. Finally, with the goal of accelerating tabular data research, we release the TabZilla Benchmark Suite: a collection of the 36 'hardest' of the datasets we study. Our benchmark suite, codebase, and all raw results are available at https://github.com/naszilla/tabzilla.
The Capacity of Classical Summation over a Quantum MAC with Arbitrarily Distributed Inputs and Entanglements
Venue: arXiv cs.IT 2024
Year: 2024

The $Σ$-QMAC problem is introduced, involving $S$ servers, $K$ classical ($\mathbb{F}_d$) data streams, and $T$ independent quantum systems. Data stream ${\sf W}_k, k\in[K]$ is replicated at a subset of servers $\mathcal{W}(k)\subset[S]$, and quantum system $\mathcal{Q}_t, t\in[T]$ is distributed among a subset of servers $\mathcal{E}(t)\subset[S]$ such that Server $s\in\mathcal{E}(t)$ receives subsystem $\mathcal{Q}_{t,s}$ of $\mathcal{Q}_t=(\mathcal{Q}_{t,s})_{s\in\mathcal{E}(t)}$. Servers manipulate their quantum subsystems according to their data and send the subsystems to a receiver. The total download cost is $\sum_{t\in[T]}\sum_{s\in\mathcal{E}(t)}\log_d|\mathcal{Q}_{t,s}|$ qudits, where $|\mathcal{Q}|$ is the dimension of $\mathcal{Q}$. The states and measurements of $(\mathcal{Q}_t)_{t\in[T]}$ are required to be separable across $t\in[T]$ throughout, but for each $t\in[T]$, the subsystems of $\mathcal{Q}_{t}$ can be prepared initially in an arbitrary (independent of data) entangled state, manipulated arbitrarily by the respective servers, and measured jointly by the receiver. From the measurements, the receiver must recover the sum of all data streams. Rate is defined as the number of dits ($\mathbb{F}_d$ symbols) of the desired sum computed per qudit of download. The capacity of $Σ$-QMAC, i.e., the supremum of achievable rates is characterized for arbitrary data replication and entanglement distribution maps $\mathcal{W}, \mathcal{E}$. Coding based on the $N$-sum box abstraction is optimal in every case. Notably, for every $S\neq 3$ there exists an instance of the $Σ$-QMAC where $S$-party entanglement is necessary to achieve the fully entangled capacity.
Structural Group Unfairness: Measurement and Mitigation by means of the Effective Resistance
Venue: arXiv cs.SI 2024
Year: 2024

Social networks contribute to the distribution of social capital, defined as the relationships, norms of trust and reciprocity within a community or society that facilitate cooperation and collective action. Therefore, better positioned members in a social network benefit from faster access to diverse information and higher influence on information dissemination. A variety of methods have been proposed in the literature to measure social capital at an individual level. However, there is a lack of methods to quantify social capital at a group level, which is particularly important when the groups are defined on the grounds of protected attributes. To fill this gap, we propose to measure the social capital of a group of nodes by means of the effective resistance and emphasize the importance of considering the entire network topology. Grounded in spectral graph theory, we introduce three effective resistance-based measures of group social capital, namely group isolation, group diameter and group control, where the groups are defined according to the value of a protected attribute. We denote the social capital disparity among different groups in a network as structural group unfairness, and propose to mitigate it by means of a budgeted edge augmentation heuristic that systematically increases the social capital of the most disadvantaged group. In experiments on real-world networks, we uncover significant levels of structural group unfairness when using gender as the protected attribute, with females being the most disadvantaged group in comparison to males. We also illustrate how our proposed edge augmentation approach is able to not only effectively mitigate the structural group unfairness but also increase the social capital of all groups in the network.
FedNC: A Secure and Efficient Federated Learning Method with Network Coding
Venue: arXiv cs.LG 2024
Year: 2024

Federated Learning (FL) is a promising distributed learning mechanism which still faces two major challenges, namely privacy breaches and system efficiency. In this work, we reconceptualize the FL system from the perspective of network information theory, and formulate an original FL communication framework, FedNC, which is inspired by Network Coding (NC). The main idea of FedNC is mixing the information of the local models by making random linear combinations of the original parameters, before uploading for further aggregation. Due to the benefits of the coding scheme, both theoretical and experimental analysis indicate that FedNC improves the performance of traditional FL in several important ways, including security, efficiency, and robustness. To the best of our knowledge, this is the first framework where NC is introduced in FL. As FL continues to evolve within practical network frameworks, more variants can be further designed based on FedNC.
A multimodal dynamical variational autoencoder for audiovisual speech representation learning
Venue: arXiv cs.SD 2024
Year: 2024

In this paper, we present a multimodal and dynamical VAE (MDVAE) applied to unsupervised audio-visual speech representation learning. The latent space is structured to dissociate the latent dynamical factors that are shared between the modalities from those that are specific to each modality. A static latent variable is also introduced to encode the information that is constant over time within an audiovisual speech sequence. The model is trained in an unsupervised manner on an audiovisual emotional speech dataset, in two stages. In the first stage, a vector quantized VAE (VQ-VAE) is learned independently for each modality, without temporal modeling. The second stage consists in learning the MDVAE model on the intermediate representation of the VQ-VAEs before quantization. The disentanglement between static versus dynamical and modality-specific versus modality-common information occurs during this second training stage. Extensive experiments are conducted to investigate how audiovisual speech latent factors are encoded in the latent space of MDVAE. These experiments include manipulating audiovisual speech, audiovisual facial image denoising, and audiovisual speech emotion recognition. The results show that MDVAE effectively combines the audio and visual information in its latent space. They also show that the learned static representation of audiovisual speech can be used for emotion recognition with few labeled data, and with better accuracy compared with unimodal baselines and a state-of-the-art supervised model based on an audiovisual transformer architecture.