OpenRaiser/Intern-Atlas huggingface dataset:
M2R2: Missing-Modality Robust emotion Recognition framework with iterative data augmentation
Venue: arXiv cs.SD 2022
Year: 2022

This paper deals with the utterance-level modalities missing problem with uncertain patterns on emotion recognition in conversation (ERC) task. Present models generally predict the speaker's emotions by its current utterance and context, which is degraded by modality missing considerably. Our work proposes a framework Missing-Modality Robust emotion Recognition (M2R2), which trains emotion recognition model with iterative data augmentation by learned common representation. Firstly, a network called Party Attentive Network (PANet) is designed to classify emotions, which tracks all the speakers' states and context. Attention mechanism between speaker with other participants and dialogue topic is used to decentralize dependence on multi-time and multi-party utterances instead of the possible incomplete one. Moreover, the Common Representation Learning (CRL) problem is defined for modality-missing problem. Data imputation methods improved by the adversarial strategy are used here to construct extra features to augment data. Extensive experiments and case studies validate the effectiveness of our methods over baselines for modality-missing emotion recognition on two different datasets.
Parametric Reshaping of Portraits in Videos
Venue: arXiv cs.CV 2022
Year: 2022

Sharing short personalized videos to various social media networks has become quite popular in recent years. This raises the need for digital retouching of portraits in videos. However, applying portrait image editing directly on portrait video frames cannot generate smooth and stable video sequences. To this end, we present a robust and easy-to-use parametric method to reshape the portrait in a video to produce smooth retouched results. Given an input portrait video, our method consists of two main stages: stabilized face reconstruction, and continuous video reshaping. In the first stage, we start by estimating face rigid pose transformations across video frames. Then we jointly optimize multiple frames to reconstruct an accurate face identity, followed by recovering face expressions over the entire video. In the second stage, we first reshape the reconstructed 3D face using a parametric reshaping model reflecting the weight change of the face, and then utilize the reshaped 3D face to guide the warping of video frames. We develop a novel signed distance function based dense mapping method for the warping between face contours before and after reshaping, resulting in stable warped video frames with minimum distortions. In addition, we use the 3D structure of the face to correct the dense mapping to achieve temporal consistency. We generate the final result by minimizing the background distortion through optimizing a content-aware warping mesh. Extensive experiments show that our method is able to create visually pleasing results by adjusting a simple reshaping parameter, which facilitates portrait video editing for social media and visual effects.
Automated Imbalanced Classification via Layered Learning
Venue: arXiv cs.LG 2022
Year: 2022

In this paper we address imbalanced binary classification (IBC) tasks. Applying resampling strategies to balance the class distribution of training instances is a common approach to tackle these problems. Many state-of-the-art methods find instances of interest close to the decision boundary to drive the resampling process. However, under-sampling the majority class may potentially lead to important information loss. Over-sampling also may increase the chance of overfitting by propagating the information contained in instances from the minority class. The main contribution of our work is a new method called ICLL for tackling IBC tasks which is not based on resampling training observations. Instead, ICLL follows a layered learning paradigm to model the data in two stages. In the first layer, ICLL learns to distinguish cases close to the decision boundary from cases which are clearly from the majority class, where this dichotomy is defined using a hierarchical clustering analysis. In the subsequent layer, we use instances close to the decision boundary and instances from the minority class to solve the original predictive task. A second contribution of our work is the automatic definition of the layers which comprise the layered learning strategy using a hierarchical clustering model. This is a relevant discovery as this process is usually performed manually according to domain knowledge. We carried out extensive experiments using 100 benchmark data sets. The results show that the proposed method leads to a better performance relatively to several state-of-the-art methods for IBC.
Balancing Multi-Domain Corpora Learning for Open-Domain Response Generation
Venue: arXiv cs.CL 2022
Year: 2022

Open-domain conversational systems are assumed to generate equally good responses on multiple domains. Previous work achieved good performance on the single corpus, but training and evaluating on multiple corpora from different domains are less studied. This paper explores methods of generating relevant responses for each of multiple multi-domain corpora. We first examine interleaved learning which intermingles multiple corpora as the baseline. We then investigate two multi-domain learning methods, labeled learning and multi-task labeled learning, which encode each corpus through a unique corpus embedding. Furthermore, we propose Domain-specific Frequency (DF), a novel word-level importance weight that measures the relative importance of a word for a specific corpus compared to other corpora. Based on DF, we propose weighted learning, a method that integrates DF to the loss function. We also adopt DF as a new evaluation metric. Extensive experiments show that our methods gain significant improvements on both automatic and human evaluation. We share our code and data for reproducibility
Low-Level Bi-Abduction
Venue: arXiv cs.LO 2022
Year: 2022

The paper proposes a new static analysis designed to handle open programs, i.e., fragments of programs, with dynamic pointer-linked data structures - in particular, various kinds of lists - that employ advanced low-level pointer operations. The goal is to allow such programs be analysed without a need of writing analysis harnesses that would first initialise the structures being handled. The approach builds on a special flavour of separation logic and the approach of bi-abduction. The code of interest is analyzed along the call tree, starting from its leaves, with each function analysed just once without any call context, leading to a set of contracts summarizing the behaviour of the analysed functions. In order to handle the considered programs, methods of abduction existing in the literature are significantly modified and extended in the paper. The proposed approach has been implemented in a tool prototype and successfully evaluated on not large but complex programs.
The Structural Power of Reconfigurable Circuits in the Amoebot Model
Venue: arXiv cs.DC 2022
Year: 2022

The amoebot model [Derakhshandeh et al., 2014] has been proposed as a model for programmable matter consisting of tiny, robotic elements called amoebots. We consider the reconfigurable circuit extension [Feldmann et al., JCB 2022] of the geometric (variant of the) amoebot model that allows the amoebot structure to interconnect amoebots by so-called circuits. A circuit permits the instantaneous transmission of signals between the connected amoebots. In this paper, we examine the structural power of the reconfigurable circuits. We start with some fundamental problems like the stripe computation problem where, given any connected amoebot structure $S$, an amoebot $u$ in $S$, and some axis $X$, all amoebots belonging to axis $X$ through $u$ have to be identified. Second, we consider the global maximum problem, which identifies an amoebot at the highest possible position with respect to some direction in some given amoebot (sub)structure. A solution to this problem can then be used to solve the skeleton problem, where a (not necessarily simple) cycle of amoebots has to be found in the given amoebot structure which contains all boundary amoebots. A canonical solution to that problem can then be used to come up with a canonical path, which provides a unique characterization of the shape of the given amoebot structure. Constructing canonical paths for different directions will then allow the amoebots to set up a spanning tree and to check symmetry properties of the given amoebot structure. The problems are important for a number of applications like rapid shape transformation, energy dissemination, and structural monitoring. Interestingly, the reconfigurable circuit extension allows polylogarithmic-time solutions to all of these problems.
Heterogeneous Semantic and Bit Communications: A Semi-NOMA Scheme
Venue: arXiv cs.IT 2022
Year: 2022

Multiple access (MA) design is investigated for facilitating the coexistence of the emerging semantic transmission and the conventional bit-based transmission in future networks. The semantic rate is considered for measuring the performance of the semantic transmission. However, a key challenge is that there is a lack of a closed-form expression for a key parameter, namely the semantic similarity, which characterizes the sentence similarity between an original sentence and the corresponding recovered sentence. To overcome this challenge, we propose a data regression method, where the semantic similarity is approximated by a generalized logistic function. Using the obtained tractable function, we propose a heterogeneous semantic and bit communication framework, where an access point simultaneously sends the semantic and bit streams to one semantics-interested user (S-user) and one bit-interested user (B-user). To realize this heterogeneous semantic and bit transmission in multi-user networks, three MA schemes are proposed, namely orthogonal multiple access (OMA), non-orthogonal multiple access (NOMA), and semi-NOMA. More specifically, the bit stream in semi-NOMA is split into two streams, one is transmitted with the semantic stream over the shared frequency sub-band and the other is transmitted over the separate orthogonal frequency sub-band. To study the fundamental performance limits of the three proposed MA schemes, the semantic-versus-bit (SvB) rate region and the power region are defined. An optimal resource allocation procedure is then derived for characterizing the boundary of the SvB rate region and the power region achieved by each MA scheme. The structures of the derived solutions demonstrate that semi-NOMA is superior to both NOMA and OMA given its highly flexible transmission policy. Our numerical results validate the analysis and show the superiority of semi-NOMA.
ImPosing: Implicit Pose Encoding for Efficient Visual Localization
Venue: arXiv cs.CV 2022
Year: 2022

We propose a novel learning-based formulation for visual localization of vehicles that can operate in real-time in city-scale environments. Visual localization algorithms determine the position and orientation from which an image has been captured, using a set of geo-referenced images or a 3D scene representation. Our new localization paradigm, named Implicit Pose Encoding (ImPosing), embeds images and camera poses into a common latent representation with 2 separate neural networks, such that we can compute a similarity score for each image-pose pair. By evaluating candidates through the latent space in a hierarchical manner, the camera position and orientation are not directly regressed but incrementally refined. Very large environments force competitors to store gigabytes of map data, whereas our method is very compact independently of the reference database size. In this paper, we describe how to effectively optimize our learned modules, how to combine them to achieve real-time localization, and demonstrate results on diverse large scale scenarios that significantly outperform prior work in accuracy and computational efficiency.
Can collaborative learning be private, robust and scalable?
Venue: arXiv cs.LG 2022
Year: 2022

In federated learning for medical image analysis, the safety of the learning protocol is paramount. Such settings can often be compromised by adversaries that target either the private data used by the federation or the integrity of the model itself. This requires the medical imaging community to develop mechanisms to train collaborative models that are private and robust against adversarial data. In response to these challenges, we propose a practical open-source framework to study the effectiveness of combining differential privacy, model compression and adversarial training to improve the robustness of models against adversarial samples under train- and inference-time attacks. Using our framework, we achieve competitive model performance, a significant reduction in model's size and an improved empirical adversarial robustness without a severe performance degradation, critical in medical image analysis.
Language Models Can See: Plugging Visual Controls in Text Generation
Venue: arXiv cs.CV 2022
Year: 2022

Generative language models (LMs) such as GPT-2/3 can be prompted to generate text with remarkable quality. While they are designed for text-prompted generation, it remains an open question how the generation process could be guided by modalities beyond text such as images. In this work, we propose a training-free framework, called MAGIC (iMAge-Guided text generatIon with CLIP), for plugging in visual controls in the generation process and enabling LMs to perform multimodal tasks (e.g., image captioning) in a zero-shot manner. MAGIC is a simple yet efficient plug-and-play framework, which directly combines an off-the-shelf LM (i.e., GPT-2) and an image-text matching model (i.e., CLIP) for image-grounded text generation. During decoding, MAGIC influences the generation of the LM by introducing a CLIP-induced score, called magic score, which regularizes the generated result to be semantically related to a given image while being coherent to the previously generated context. Notably, the proposed decoding scheme does not involve any gradient update operation, therefore being computationally efficient. On the challenging task of zero-shot image captioning, MAGIC outperforms the state-of-the-art method by notable margins with a nearly 27 times decoding speedup. MAGIC is a flexible framework and is theoretically compatible with any text generation tasks that incorporate image grounding. In the experiments, we showcase that it is also capable of performing visually grounded story generation given both an image and a text prompt.
RaFoLa: A Rationale-Annotated Corpus for Detecting Indicators of Forced Labour
Venue: arXiv cs.CL 2022
Year: 2022

Forced labour is the most common type of modern slavery, and it is increasingly gaining the attention of the research and social community. Recent studies suggest that artificial intelligence (AI) holds immense potential for augmenting anti-slavery action. However, AI tools need to be developed transparently in cooperation with different stakeholders. Such tools are contingent on the availability and access to domain-specific data, which are scarce due to the near-invisible nature of forced labour. To the best of our knowledge, this paper presents the first openly accessible English corpus annotated for multi-class and multi-label forced labour detection. The corpus consists of 989 news articles retrieved from specialised data sources and annotated according to risk indicators defined by the International Labour Organization (ILO). Each news article was annotated for two aspects: (1) indicators of forced labour as classification labels and (2) snippets of the text that justify labelling decisions. We hope that our data set can help promote research on explainability for multi-class and multi-label text classification. In this work, we explain our process for collecting the data underpinning the proposed corpus, describe our annotation guidelines and present some statistical analysis of its content. Finally, we summarise the results of baseline experiments based on different variants of the Bidirectional Encoder Representation from Transformer (BERT) model.
Utility-Based Context-Aware Multi-Agent Recommendation System for Energy Efficiency in Residential Buildings
Venue: arXiv cs.MA 2022
Year: 2022

A significant part of CO2 emissions is due to high electricity consumption in residential buildings. Using load shifting can help to improve the households' energy efficiency. To nudge changes in energy consumption behavior, simple but powerful architectures are vital. This paper presents a novel algorithm of a recommendation system generating device usage recommendations and suggests a framework for evaluating its performance by analyzing potential energy cost savings. As a utility-based recommender system, it models user preferences depending on habitual device usage patterns, user availability, and device usage costs. As a context-aware system, it requires an external hourly electricity price signal and appliance-level energy consumption data. Due to a multi-agent architecture, it provides flexibility and allows for adjustments and further enhancements. Empirical results show that the system can provide energy cost savings of 18% and more for most studied households.
Replicating Data Pipelines with GrimoireLab
Venue: arXiv cs.SE 2022
Year: 2022

In this paper, we present our MSR Hackathon 2022 project that replicates an existing Gitter study using GrimoireLab. We compare the previous study's pipeline with our GrimoireLab implementation in terms of speed, data consistency, organization, and the learning curve to get started. We believe our experience with GrimoireLab can help future researchers in making the right choice while implementing their data pipelines over Gitter and Github data.
Streaming and Sketching Complexity of CSPs: A survey
Venue: arXiv cs.CC 2022
Year: 2022

In this survey we describe progress over the last decade or so in understanding the complexity of solving constraint satisfaction problems (CSPs) approximately in the streaming and sketching models of computation. After surveying some of the results we give some sketches of the proofs and in particular try to explain why there is a tight dichotomy result for sketching algorithms working in subpolynomial space regime.
Automating Reasoning with Standpoint Logic via Nested Sequents
Venue: arXiv cs.LO 2022
Year: 2022

Standpoint logic is a recently proposed formalism in the context of knowledge integration, which advocates a multi-perspective approach permitting reasoning with a selection of diverse and possibly conflicting standpoints rather than forcing their unification. In this paper, we introduce nested sequent calculi for propositional standpoint logics--proof systems that manipulate trees whose nodes are multisets of formulae--and show how to automate standpoint reasoning by means of non-deterministic proof-search algorithms. To obtain worst-case complexity-optimal proof-search, we introduce a novel technique in the context of nested sequents, referred to as "coloring," which consists of taking a formula as input, guessing a certain coloring of its subformulae, and then running proof-search in a nested sequent calculus on the colored input. Our technique lets us decide the validity of standpoint formulae in CoNP since proof-search only produces a partial proof relative to each permitted coloring of the input. We show how all partial proofs can be fused together to construct a complete proof when the input is valid, and how certain partial proofs can be transformed into a counter-model when the input is invalid. These "certificates" (i.e. proofs and counter-models) serve as explanations of the (in)validity of the input.
Encrypted, Anonymized System for Protected Health Information Verification Built via Proof of Stake
Venue: arXiv cs.CR 2022
Year: 2022

Digital Health Passes (DHP), systems of digitally validating quarantine and vaccination status such as the New York IBM Excelsior Pass, demonstrate a lawful means to approach some benefits offered by "true elimination" treatment strategies-which focus on the complete elimination of cases instead of investing more in controlling the progression of the disease-of COVID-19. Current implementations of DHPs require region-based control and central storage of Protected Health Information (PHI)-creating a challenge to widespread use across different jurisdictions with incompatible data management systems and a lack of standardized patient privacy controls. In this work, a mechanism for decentralized PHI storage and validation is proposed through a novel two-stage handshaking mechanism update to blockchain proof-of-stake consensus. The proposed mechanism, when used to support a DHP, allows individuals to validate their quarantine and testing universally with any jurisdiction while allowing their right of independent movement and the protection of their PHI. Implementational details on the protocol are given, and the protocol is shown to withstand a 1% disturbance attack at only 923 participants via a Monte-Carlo simulation: further validating its stability.
General sum stochastic games with networked information flows
Venue: arXiv cs.LG 2022
Year: 2022

Inspired by applications such as supply chain management, epidemics, and social networks, we formulate a stochastic game model that addresses three key features common across these domains: 1) network-structured player interactions, 2) pair-wise mixed cooperation and competition among players, and 3) limited global information toward individual decision-making. In combination, these features pose significant challenges for black box approaches taken by deep learning-based multi-agent reinforcement learning (MARL) algorithms and deserve more detailed analysis. We formulate a networked stochastic game with pair-wise general sum objectives and asymmetrical information structure, and empirically explore the effects of information availability on the outcomes of different MARL paradigms such as individual learning and centralized learning decentralized execution.
Cognitive Radio Resource Scheduling using Multi agent QLearning for LTE
Venue: arXiv cs.NI 2022
Year: 2022

In this paper, we propose, implement, and test two novel downlink LTE scheduling algorithms. The implementation and testing of these algorithms were in Matlab, and they are based on the use of Reinforcement Learning, more specifically, the Qlearning technique for scheduling two types of users. The first algorithm is called a Collaborative scheduling algorithm, and the second algorithm is called a Competitive scheduling algorithm. The first type of the scheduled users is the Primary Users, and they are the licensed subscribers that pay for their service. The second type of the scheduled users is the Secondary Users, and they could be unlicensed subscribers that dont pay for their service, device to device communications, or sensors. Each user whether it is a primary or secondary is considered as an agent. In the Collaborative scheduling algorithm, the primary user agents will collaborate in order to make a joint scheduling decision about allocating the resource blocks to each one of them, then the secondary user agents will compete among themselves to use the remaining resource blocks. In the Competitive scheduling algorithm, the primary user agents will compete among themselves over the available resources, then the secondary user agents will compete among themselves over the remaining resources. Experimental results show that both scheduling algorithms converged to almost ninety percent utilization of the spectrum, and provided fair shares of the spectrum among users.