OpenRaiser/Intern-Atlas huggingface dataset:
Weisfeiler and Leman Go Infinite: Spectral and Combinatorial Pre-Colorings
Venue: arXiv cs.LG 2022
Year: 2022

Graph isomorphism testing is usually approached via the comparison of graph invariants. Two popular alternatives that offer a good trade-off between expressive power and computational efficiency are combinatorial (i.e., obtained via the Weisfeiler-Leman (WL) test) and spectral invariants. While the exact power of the latter is still an open question, the former is regularly criticized for its limited power, when a standard configuration of uniform pre-coloring is used. This drawback hinders the applicability of Message Passing Graph Neural Networks (MPGNNs), whose expressive power is upper bounded by the WL test. Relaxing the assumption of uniform pre-coloring, we show that one can increase the expressive power of the WL test ad infinitum. Following that, we propose an efficient pre-coloring based on spectral features that provably increase the expressive power of the vanilla WL test. The above claims are accompanied by extensive synthetic and real data experiments. The code to reproduce our experiments is available at https://github.com/TPFI22/Spectral-and-Combinatorial
Constrained Density Matching and Modeling for Cross-lingual Alignment of Contextualized Representations
Venue: arXiv cs.CL 2022
Year: 2022

Multilingual representations pre-trained with monolingual data exhibit considerably unequal task performances across languages. Previous studies address this challenge with resource-intensive contextualized alignment, which assumes the availability of large parallel data, thereby leaving under-represented language communities behind. In this work, we attribute the data hungriness of previous alignment techniques to two limitations: (i) the inability to sufficiently leverage data and (ii) these techniques are not trained properly. To address these issues, we introduce supervised and unsupervised density-based approaches named Real-NVP and GAN-Real-NVP, driven by Normalizing Flow, to perform alignment, both dissecting the alignment of multilingual subspaces into density matching and density modeling. We complement these approaches with our validation criteria in order to guide the training process. Our experiments encompass 16 alignments, including our approaches, evaluated across 6 language pairs, synthetic data and 5 NLP tasks. We demonstrate the effectiveness of our approaches in the scenarios of limited and no parallel data. First, our supervised approach trained on 20k parallel data (sentences) mostly surpasses Joint-Align and InfoXLM trained on over 100k parallel sentences. Second, parallel data can be removed without sacrificing performance when integrating our unsupervised approach in our bootstrapping procedure, which is theoretically motivated to enforce equality of multilingual subspaces. Moreover, we demonstrate the advantages of validation criteria over validation data for guiding supervised training.
Learning Fair Representations via Rate-Distortion Maximization
Venue: arXiv cs.LG 2022
Year: 2022

Text representations learned by machine learning models often encode undesirable demographic information of the user. Predictive models based on these representations can rely on such information, resulting in biased decisions. We present a novel debiasing technique, Fairness-aware Rate Maximization (FaRM), that removes protected information by making representations of instances belonging to the same protected attribute class uncorrelated, using the rate-distortion function. FaRM is able to debias representations with or without a target task at hand. FaRM can also be adapted to remove information about multiple protected attributes simultaneously. Empirical evaluations show that FaRM achieves state-of-the-art performance on several datasets, and learned representations leak significantly less protected attribute information against an attack by a non-linear probing network.
Learning affective meanings that derives the social behavior using Bidirectional Encoder Representations from Transformers
Venue: arXiv cs.CL 2022
Year: 2022

Predicting the outcome of a process requires modeling the system dynamic and observing the states. In the context of social behaviors, sentiments characterize the states of the system. Affect Control Theory (ACT) uses sentiments to manifest potential interaction. ACT is a generative theory of culture and behavior based on a three-dimensional sentiment lexicon. Traditionally, the sentiments are quantified using survey data which is fed into a regression model to explain social behavior. The lexicons used in the survey are limited due to prohibitive cost. This paper uses a fine-tuned Bidirectional Encoder Representations from Transformers (BERT) model to develop a replacement for these surveys. This model achieves state-of-the-art accuracy in estimating affective meanings, expanding the affective lexicon, and allowing more behaviors to be explained.
Using Transition Learning to Enhance Mobile-Controlled Handoff In Decentralized Future Networks
Venue: arXiv cs.NI 2022
Year: 2022

Traditionally, resource management and capacity allocation has been controlled network-side in cellular deployment. As autonomicity has been added to network design, machine learning technologies have largely followed this paradigm, benefiting from the higher compute capacity and informational context available at the network core. However, when these network services are disaggregated or decentralized, models that rely on assumed levels of network or information availability may no longer function reliably. This paper presents an inverted view of the resource management paradigm; one in which the client device executes a learning algorithm and manages its own mobility under a scenario where the networks and their corresponding data underneath are not being centrally managed.
Evaluating Deep Vs. Wide & Deep Learners As Contextual Bandits For Personalized Email Promo Recommendations
Venue: arXiv cs.LG 2022
Year: 2022

Personalization enables businesses to learn customer preferences from past interactions and thus to target individual customers with more relevant content. We consider the problem of predicting the optimal promotional offer for a given customer out of several options as a contextual bandit problem. Identifying information for the customer and/or the campaign can be used to deduce unknown customer/campaign features that improve optimal offer prediction. Using a generated synthetic email promo dataset, we demonstrate similar prediction accuracies for (a) a wide and deep network that takes identifying information (or other categorical features) as input to the wide part and (b) a deep-only neural network that includes embeddings of categorical features in the input. Improvements in accuracy from including categorical features depends on the variability of the unknown numerical features for each category. We also show that selecting options using upper confidence bound or Thompson sampling, approximated via Monte Carlo dropout layers in the wide and deep models, slightly improves model performance.
CIC: Contrastive Intrinsic Control for Unsupervised Skill Discovery
Venue: arXiv cs.LG 2022
Year: 2022

We introduce Contrastive Intrinsic Control (CIC), an algorithm for unsupervised skill discovery that maximizes the mutual information between state-transitions and latent skill vectors. CIC utilizes contrastive learning between state-transitions and skills to learn behavior embeddings and maximizes the entropy of these embeddings as an intrinsic reward to encourage behavioral diversity. We evaluate our algorithm on the Unsupervised Reinforcement Learning Benchmark, which consists of a long reward-free pre-training phase followed by a short adaptation phase to downstream tasks with extrinsic rewards. CIC substantially improves over prior methods in terms of adaptation efficiency, outperforming prior unsupervised skill discovery methods by 1.79x and the next leading overall exploration algorithm by 1.18x.
Full-Duplex Aerial Communication System for Multiple UAVs with Directional Antennas
Venue: arXiv cs.IT 2022
Year: 2022

UAV-based wireless systems, such as wireless relay and remote sensing, have attracted great attentions from academia and industry. To realize them, a high-performance wireless aerial communication system, which bridges UAVs and ground stations, is one of the key enablers. However, there are still issues hindering its development, such as the severe co-channel interference among UAVs, and the limited payload/battery-life of UAVs. To address the challenges, we propose an aerial communication system which enables system-level full-duplex communication of multiple UAVs with lower hardware complexities than ideal full-duplex communication systems. In the proposed system, each channel is re-assigned to the uplink and downlink of a pair of UAVs, and each UAV employ a pair of separated channels for its uplink and downlink. The co-channel interference between UAVs that reuse same channels is eliminated by exploiting advantages of UAVs' maneuverability and high-gain directional antennas equipped in UAVs and ground stations, so that dedicated cancellers are not necessary in the proposed system. The system design and performance analysis are given, and the simulation results well agree with the designs.
Factorized-FL: Agnostic Personalized Federated Learning with Kernel Factorization & Similarity Matching
Venue: arXiv cs.LG 2022
Year: 2022

In real-world federated learning scenarios, participants could have their own personalized labels which are incompatible with those from other clients, due to using different label permutations or tackling completely different tasks or domains. However, most existing FL approaches cannot effectively tackle such extremely heterogeneous scenarios since they often assume that (1) all participants use a synchronized set of labels, and (2) they train on the same task from the same domain. In this work, to tackle these challenges, we introduce Factorized-FL, which allows to effectively tackle label- and task-heterogeneous federated learning settings by factorizing the model parameters into a pair of vectors, where one captures the common knowledge across different labels and tasks and the other captures knowledge specific to the task each local model tackles. Moreover, based on the distance in the client-specific vector space, Factorized-FL performs selective aggregation scheme to utilize only the knowledge from the relevant participants for each client. We extensively validate our method on both label- and domain-heterogeneous settings, on which it outperforms the state-of-the-art personalized federated learning methods.
Measurement-Based Validation of Z3RO Precoder to Prevent Nonlinear Amplifier Distortion in Massive MIMO Systems
Venue: arXiv cs.IT 2022
Year: 2022

In multiple input multiple output (MIMO) systems, precoding allows the base station to spatially focus and multiplex signals towards each user. However, distortion introduced by power amplifier nonlinearities coherently combines in the same spatial directions when using a conventional precoder such as maximum ratio transmission (MRT). This can strongly limit the user performance and moreover create unauthorized out-of-band (OOB) emissions. In order to overcome this problem, the zero third-order distortion (Z3RO) precoder was recently introduced. This precoder constraints the third-order distortion at the user location to be zero. In this work, the performance of the Z3RO precoder is validated based on real-world channel measurement data. The results illustrate the reduction in distortion power at the UE locations: an average distortion reduction of 6.03 dB in the worst-case single-user scenario and 3.54 dB in the 2-user case at a back-off rate of -3 dB.
Learning Physics-Consistent Particle Interactions
Venue: arXiv cs.LG 2022
Year: 2022

Interacting particle systems play a key role in science and engineering. Access to the governing particle interaction law is fundamental for a complete understanding of such systems. However, the inherent system complexity keeps the particle interaction hidden in many cases. Machine learning methods have the potential to learn the behavior of interacting particle systems by combining experiments with data analysis methods. However, most existing algorithms focus on learning the kinetics at the particle level. Learning pairwise interaction, e.g., pairwise force or pairwise potential energy, remains an open challenge. Here, we propose an algorithm that adapts the Graph Networks framework, which contains an edge part to learn the pairwise interaction and a node part to model the dynamics at particle level. Different from existing approaches that use neural networks in both parts, we design a deterministic operator in the node part that allows to precisely infer the pairwise interactions that are consistent with underlying physical laws by only being trained to predict the particle acceleration. We test the proposed methodology on multiple datasets and demonstrate that it achieves superior performance in inferring correctly the pairwise interactions while also being consistent with the underlying physics on all the datasets. The proposed framework is scalable to larger systems and transferable to any type of particle interactions, contrary to the previously proposed purely data-driven solutions. The developed methodology can support a better understanding and discovery of the underlying particle interaction laws, and hence guide the design of materials with targeted properties.
From Explanations to Segmentation: Using Explainable AI for Image Segmentation
Venue: arXiv cs.CV 2022
Year: 2022

The new era of image segmentation leveraging the power of Deep Neural Nets (DNNs) comes with a price tag: to train a neural network for pixel-wise segmentation, a large amount of training samples has to be manually labeled on pixel-precision. In this work, we address this by following an indirect solution. We build upon the advances of the Explainable AI (XAI) community and extract a pixel-wise binary segmentation from the output of the Layer-wise Relevance Propagation (LRP) explaining the decision of a classification network. We show that we achieve similar results compared to an established U-Net segmentation architecture, while the generation of the training data is significantly simplified. The proposed method can be trained in a weakly supervised fashion, as the training samples must be only labeled on image-level, at the same time enabling the output of a segmentation mask. This makes it especially applicable to a wider range of real applications where tedious pixel-level labelling is often not possible.
Activity Recognition in Assembly Tasks by Bayesian Filtering in Multi-Hypergraphs
Venue: arXiv cs.AI 2022
Year: 2022

We study sensor-based human activity recognition in manual work processes like assembly tasks. In such processes, the system states often have a rich structure, involving object properties and relations. Thus, estimating the hidden system state from sensor observations by recursive Bayesian filtering can be very challenging, due to the combinatorial explosion in the number of system states. To alleviate this problem, we propose an efficient Bayesian filtering model for such processes. In our approach, system states are represented by multi-hypergraphs, and the system dynamics is modeled by graph rewriting rules. We show a preliminary concept that allows to represent distributions over multi-hypergraphs more compactly than by full enumeration, and present an inference algorithm that works directly on this compact representation. We demonstrate the applicability of the algorithm on a real dataset.
Big fish and small ponds: why the departmental h-index should not be used to rank universities
Venue: arXiv cs.DL 2022
Year: 2022

The size-dependent nature of the so-called group or departmental h-index is reconsidered in this paper. While the influence of unit size on such collective measures was already demonstrated a decade ago, institutional ratings based on this metric can still be found and still impact on the reputations and funding of many research institutions. The aim of this paper is to demonstrate the fallacy of this approach to collective research-quality assessment in a simple way, focusing on the h-index in its original form. We show that randomly reshuffling real scientometric data (varying numbers of citations) amongst institutions of varying size, while maintaining the volume of their research outputs, has little effect on their departmental h-index. This suggests that the relative position in ratings based on the collective h-index is determined not only by quality (impact) of particular research outputs but by their volume. Therefore, the application of the collective h-index in its original form is disputable as a basement for comparison at aggregated levels such as to research groups, institutions or journals. We suggest a possible remedy for this failing which is implementable in a manner that is as simple and understandable as the h-index itself.
Quantifying Relevance in Learning and Inference
Venue: arXiv cs.LG 2022
Year: 2022

Learning is a distinctive feature of intelligent behaviour. High-throughput experimental data and Big Data promise to open new windows on complex systems such as cells, the brain or our societies. Yet, the puzzling success of Artificial Intelligence and Machine Learning shows that we still have a poor conceptual understanding of learning. These applications push statistical inference into uncharted territories where data is high-dimensional and scarce, and prior information on "true" models is scant if not totally absent. Here we review recent progress on understanding learning, based on the notion of "relevance". The relevance, as we define it here, quantifies the amount of information that a dataset or the internal representation of a learning machine contains on the generative model of the data. This allows us to define maximally informative samples, on one hand, and optimal learning machines on the other. These are ideal limits of samples and of machines, that contain the maximal amount of information about the unknown generative process, at a given resolution (or level of compression). Both ideal limits exhibit critical features in the statistical sense: Maximally informative samples are characterised by a power-law frequency distribution (statistical criticality) and optimal learning machines by an anomalously large susceptibility. The trade-off between resolution (i.e. compression) and relevance distinguishes the regime of noisy representations from that of lossy compression. These are separated by a special point characterised by Zipf's law statistics. This identifies samples obeying Zipf's law as the most compressed loss-less representations that are optimal in the sense of maximal relevance. Criticality in optimal learning machines manifests in an exponential degeneracy of energy levels, that leads to unusual thermodynamic properties.
Natural Language to Code Using Transformers
Venue: arXiv cs.CL 2022
Year: 2022

We tackle the problem of generating code snippets from natural language descriptions using the CoNaLa dataset. We use the self-attention based transformer architecture and show that it performs better than recurrent attention-based encoder decoder. Furthermore, we develop a modified form of back translation and use cycle consistent losses to train the model in an end-to-end fashion. We achieve a BLEU score of 16.99 beating the previously reported baseline of the CoNaLa challenge.
Deepfake pornography as a male gaze on fan culture
Venue: arXiv cs.CY 2022
Year: 2022

This essay shows the impact of deepfake technology on fan culture. The innovative technology provided the male audience with an instrument to express its ideas and plots. Which subsequently led to the rise of deepfake pornography. It is often seen as a part of celebrity studies; however, the essay shows that it could also be considered a type of fanfic and a product of participatory culture, sharing community origin, exploitation by commercial companies and deep sexualisation. These two branches of fanfic evolution can be connected via the genre of machinima pornography. Textual fanfics are mainly created by females for females, depicting males; otherwise, deepfake pornography and machinima are made by males and for males targeting females.
Right for the Right Latent Factors: Debiasing Generative Models via Disentanglement
Venue: arXiv cs.LG 2022
Year: 2022

A key assumption of most statistical machine learning methods is that they have access to independent samples from the distribution of data they encounter at test time. As such, these methods often perform poorly in the face of biased data, which breaks this assumption. In particular, machine learning models have been shown to exhibit Clever-Hans-like behaviour, meaning that spurious correlations in the training set are inadvertently learnt. A number of works have been proposed to revise deep classifiers to learn the right correlations. However, generative models have been overlooked so far. We observe that generative models are also prone to Clever-Hans-like behaviour. To counteract this issue, we propose to debias generative models by disentangling their internal representations, which is achieved via human feedback. Our experiments show that this is effective at removing bias even when human feedback covers only a small fraction of the desired distribution. In addition, we achieve strong disentanglement results in a quantitative comparison with recent methods.
The Text Anonymization Benchmark (TAB): A Dedicated Corpus and Evaluation Framework for Text Anonymization
Venue: arXiv cs.CL 2022
Year: 2022

We present a novel benchmark and associated evaluation metrics for assessing the performance of text anonymization methods. Text anonymization, defined as the task of editing a text document to prevent the disclosure of personal information, currently suffers from a shortage of privacy-oriented annotated text resources, making it difficult to properly evaluate the level of privacy protection offered by various anonymization methods. This paper presents TAB (Text Anonymization Benchmark), a new, open-source annotated corpus developed to address this shortage. The corpus comprises 1,268 English-language court cases from the European Court of Human Rights (ECHR) enriched with comprehensive annotations about the personal information appearing in each document, including their semantic category, identifier type, confidential attributes, and co-reference relations. Compared to previous work, the TAB corpus is designed to go beyond traditional de-identification (which is limited to the detection of predefined semantic categories), and explicitly marks which text spans ought to be masked in order to conceal the identity of the person to be protected. Along with presenting the corpus and its annotation layers, we also propose a set of evaluation metrics that are specifically tailored towards measuring the performance of text anonymization, both in terms of privacy protection and utility preservation. We illustrate the use of the benchmark and the proposed metrics by assessing the empirical performance of several baseline text anonymization models. The full corpus along with its privacy-oriented annotation guidelines, evaluation scripts and baseline models are available on: https://github.com/NorskRegnesentral/text-anonymisation-benchmark