OpenRaiser/Intern-Atlas huggingface dataset:
Heterogeneity-aware P2P Wireless Energy Transfer for Balanced Energy Distribution
Venue: arXiv cs.NI 2022
Year: 2022

The recent advances in wireless energy transfer (WET) provide an alternate and reliable option for replenishing the battery of pervasive and portable devices, such as smartphones. The peer-to-peer (P2P) mode of WET brings improved flexibility to the charging process among the devices as they can maintain their mobility while replenishing their battery. Few existing works in P2P-WET unrealistically assume the nodes to be exchanging energy at every opportunity with any other node. Also, energy exchange between the nodes is not bounded by the energy transfer limit in that inter-node meeting duration. In this regard, the parametric heterogeneity (in terms of device's battery capacity and WET hardware) among the nodes also affects the energy transfer bound in each P2P interaction, and thus, may lead to unbalanced network energy distributions. This inherent heterogeneity aspect has not been adequately covered in the P2P-WET literature so far, especially from the point of view of maintaining a balanced energy distribution in the networked population. In this work, we present a Heterogeneity-aware Wireless Energy Transfer (HetWET) method. In contrast to the existing literature, we devise a fine-grained model of wireless energy transfer while considering the parametric heterogeneity of the participating devices. Thereafter, we enable the nodes to explore and dynamically decide the peers for energy exchange. The performance of HetWET is evaluated using extensive simulations with varying heterogeneity settings. The evaluation results demonstrate that HetWET can maintain lower energy losses and achieve more balanced energy variation distance compared to three different state-of-the-art methods.
YouTube Ad View Sentiment Analysis using Deep Learning and Machine Learning
Venue: arXiv cs.LG 2022
Year: 2022

Sentiment Analysis is currently a vital area of research. With the advancement in the use of the internet, the creation of social media, websites, blogs, opinions, ratings, etc. has increased rapidly. People express their feedback and emotions on social media posts in the form of likes, dislikes, comments, etc. The rapid growth in the volume of viewer-generated or user-generated data or content on YouTube has led to an increase in YouTube sentiment analysis. Due to this, analyzing the public reactions has become an essential need for information extraction and data visualization in the technical domain. This research predicts YouTube Ad view sentiments using Deep Learning and Machine Learning algorithms like Linear Regression (LR), Support Vector Machine (SVM), Decision Tree (DT), Random Forest (RF), and Artificial Neural Network (ANN). Finally, a comparative analysis is done based on experimental results acquired from different models.
DistilCamemBERT: a distillation of the French model CamemBERT
Venue: arXiv cs.CL 2022
Year: 2022

Modern Natural Language Processing (NLP) models based on Transformer structures represent the state of the art in terms of performance on very diverse tasks. However, these models are complex and represent several hundred million parameters for the smallest of them. This may hinder their adoption at the industrial level, making it difficult to scale up to a reasonable infrastructure and/or to comply with societal and environmental responsibilities. To this end, we present in this paper a model that drastically reduces the computational cost of a well-known French model (CamemBERT), while preserving good performance.
KRNet: Towards Efficient Knowledge Replay
Venue: arXiv cs.LG 2022
Year: 2022

The knowledge replay technique has been widely used in many tasks such as continual learning and continuous domain adaptation. The key lies in how to effectively encode the knowledge extracted from previous data and replay them during current training procedure. A simple yet effective model to achieve knowledge replay is autoencoder. However, the number of stored latent codes in autoencoder increases linearly with the scale of data and the trained encoder is redundant for the replaying stage. In this paper, we propose a novel and efficient knowledge recording network (KRNet) which directly maps an arbitrary sample identity number to the corresponding datum. Compared with autoencoder, our KRNet requires significantly ($400\times$) less storage cost for the latent codes and can be trained without the encoder sub-network. Extensive experiments validate the efficiency of KRNet, and as a showcase, it is successfully applied in the task of continual learning.
Please, Don't Forget the Difference and the Confidence Interval when Seeking for the State-of-the-Art Status
Venue: arXiv cs.CL 2022
Year: 2022

This paper argues for the widest possible use of bootstrap confidence intervals for comparing NLP system performances instead of the state-of-the-art status (SOTA) and statistical significance testing. Their main benefits are to draw attention to the difference in performance between two systems and to help assessing the degree of superiority of one system over another. Two cases studies, one comparing several systems and the other based on a K-fold cross-validation procedure, illustrate these benefits. A python module for obtaining these confidence intervals as well as a second function implementing the Fisher-Pitman test for paired samples are freely available on PyPi.
Decentralized Federated Learning Based on Committees and Blockchain
Venue: arXiv cs.DC 2022
Year: 2022

Machine learning algorithms are undoubtedly one of the most popular algorithms in recent years, and neural networks have demonstrated unprecedented precision. In daily life, different communities may have different user characteristics, which also means that training a strong model requires the union of different communities, so the privacy issue needs to be solved urgently. Federated learning is a popular privacy solution, each community does not need to expose specific data, but only needs to upload sub-models to the coordination server to train more powerful models. However, federated learning also has some problems, such as the security and fairness of the coordination server. A proven solution to the problem is a decentralized implementation of federated learning. In this paper, we apply decentralized tools such as blockchain and consensus algorithms to design a support system that supports the decentralized operation of federated learning in an alliance environment, involving the exploration of incentives, security, fairness and other issues. Finally, we experimentally verify the performance of our system, the effect of federated learning, and the availability of privacy protection.
Distributed Downlink Precoding and Equalization in Satellite Swarms
Venue: arXiv cs.IT 2022
Year: 2022

In this paper, we propose a novel approach for downlink transmission from a satellite swarm towards a ground station (GS). These swarms have the benefit of much higher spatial separation in the transmit antennas than traditional satellites with antenna arrays, promising a massive increase in spectral efficiency. The resulting precoder and equalizer have very low demands on computational complexity, inter-satellite coordination and channel estimation. This is achieved by taking knowledge about the geometry between satellites and GS into account. For precoding, each satellite only requires its angles of departure (AoDs) towards the GS and it turns out that almost optimal rates can be achieved if the satellites transmit independent data streams. For the equalizer, the GS requires only knowledge about the angles of arrival (AoAs) from all satellites. Furthermore, we show that, by choosing a proper inter-satellite distance, the proposed low-complexity approach achieves the theoretical upper bound in terms of data rate. This optimal inter-satellite distance is obtained analytically under simplifying assumption and provides a heuristic for practical scenarios. Furthermore, a novel approach to increase the robustness of the proposed precoder and equalizer against imperfect AoD and AoA knowledge is proposed by exploiting the statistics of the estimation error.
Stop Filtering: Multi-View Attribute-Enhanced Dialogue Learning
Venue: arXiv cs.CL 2022
Year: 2022

There is a growing interest in improving the conversational ability of models by filtering the raw dialogue corpora. Previous filtering strategies usually rely on a scoring method to assess and discard samples from one perspective, enabling the model to enhance the corresponding dialogue attributes (e.g., consistency) more easily. However, the discarded samples may obtain high scores in other perspectives and can provide regularization effects on the model learning, which causes the performance improvement to be sensitive to the filtering ratio. In this work, we propose a multi-view attribute-enhanced dialogue learning framework that strengthens the attribute-related features more robustly and comprehensively. Instead of filtering the raw dataset to train the model, our framework first pre-trains the model on the raw dataset and then fine-tunes it through adapters on the selected sub-sets, which also enhances certain attributes of responses but without suffering from the problems mentioned above. Considering the variety of the dialogue attribute, we further design a multi-view enhancement mechanism, including multi-view selection and inter-view fusion. It groups the high-quality samples from multiple perspectives, respectively, and enhances different attributes of responses with the corresponding sample sets and adapters, keeping knowledge independent and allowing flexible integration. Empirical results and analysis show that our framework can improve the performance significantly in terms of enhancing dialogue attributes and fusing view-specific knowledge.
Sample Efficient Approaches for Idiomaticity Detection
Venue: arXiv cs.CL 2022
Year: 2022

Deep neural models, in particular Transformer-based pre-trained language models, require a significant amount of data to train. This need for data tends to lead to problems when dealing with idiomatic multiword expressions (MWEs), which are inherently less frequent in natural text. As such, this work explores sample efficient methods of idiomaticity detection. In particular we study the impact of Pattern Exploit Training (PET), a few-shot method of classification, and BERTRAM, an efficient method of creating contextual embeddings, on the task of idiomaticity detection. In addition, to further explore generalisability, we focus on the identification of MWEs not present in the training data. Our experiments show that while these methods improve performance on English, they are much less effective on Portuguese and Galician, leading to an overall performance about on par with vanilla mBERT. Regardless, we believe sample efficient methods for both identifying and representing potentially idiomatic MWEs are very encouraging and hold significant potential for future exploration.
Symptom Identification for Interpretable Detection of Multiple Mental Disorders
Venue: arXiv cs.CL 2022
Year: 2022

Mental disease detection (MDD) from social media has suffered from poor generalizability and interpretability, due to lack of symptom modeling. This paper introduces PsySym, the first annotated symptom identification corpus of multiple psychiatric disorders, to facilitate further research progress. PsySym is annotated according to a knowledge graph of the 38 symptom classes related to 7 mental diseases complied from established clinical manuals and scales, and a novel annotation framework for diversity and quality. Experiments show that symptom-assisted MDD enabled by PsySym can outperform strong pure-text baselines. We also exhibit the convincing MDD explanations provided by symptom predictions with case studies, and point to their further potential applications.
Gaussian Persistence Curves
Venue: arXiv cs.CG 2022
Year: 2022

Topological data analysis (TDA) is a rising field in the intersection of mathematics, statistics, and computer science/data science. The cornerstone of TDA is persistent homology, which produces a summary of topological information called a persistence diagram. To utilize machine and deep learning methods on persistence diagrams, These diagrams are further summarized by transforming them into functions. In this paper we investigate the stability and injectivity of a class of smooth, one-dimensional functional summaries called Gaussian persistence curves.
Looking for a Handsome Carpenter! Debiasing GPT-3 Job Advertisements
Venue: arXiv cs.CL 2022
Year: 2022

The growing capability and availability of generative language models has enabled a wide range of new downstream tasks. Academic research has identified, quantified and mitigated biases present in language models but is rarely tailored to downstream tasks where wider impact on individuals and society can be felt. In this work, we leverage one popular generative language model, GPT-3, with the goal of writing unbiased and realistic job advertisements. We first assess the bias and realism of zero-shot generated advertisements and compare them to real-world advertisements. We then evaluate prompt-engineering and fine-tuning as debiasing methods. We find that prompt-engineering with diversity-encouraging prompts gives no significant improvement to bias, nor realism. Conversely, fine-tuning, especially on unbiased real advertisements, can improve realism and reduce bias.
Outliers Dimensions that Disrupt Transformers Are Driven by Frequency
Venue: arXiv cs.CL 2022
Year: 2022

While Transformer-based language models are generally very robust to pruning, there is the recently discovered outlier phenomenon: disabling only 48 out of 110M parameters in BERT-base drops its performance by nearly 30% on MNLI. We replicate the original evidence for the outlier phenomenon and we link it to the geometry of the embedding space. We find that in both BERT and RoBERTa the magnitude of hidden state coefficients corresponding to outlier dimensions correlates with the frequency of encoded tokens in pre-training data, and it also contributes to the "vertical" self-attention pattern enabling the model to focus on the special tokens. This explains the drop in performance from disabling the outliers, and it suggests that to decrease anisotropicity in future models we need pre-training schemas that would better take into account the skewed token distributions.
Learning Long-Horizon Robot Exploration Strategies for Multi-Object Search in Continuous Action Spaces
Venue: arXiv cs.RO 2022
Year: 2022

Recent advances in vision-based navigation and exploration have shown impressive capabilities in photorealistic indoor environments. However, these methods still struggle with long-horizon tasks and require large amounts of data to generalize to unseen environments. In this work, we present a novel reinforcement learning approach for multi-object search that combines short-term and long-term reasoning in a single model while avoiding the complexities arising from hierarchical structures. In contrast to existing multi-object search methods that act in granular discrete action spaces, our approach achieves exceptional performance in continuous action spaces. We perform extensive experiments and show that it generalizes to unseen apartment environments with limited data. Furthermore, we demonstrate zero-shot transfer of the learned policies to an office environment in real world experiments.
Fine-Grained Counting with Crowd-Sourced Supervision
Venue: arXiv cs.CV 2022
Year: 2022

Crowd-sourcing is an increasingly popular tool for image analysis in animal ecology. Computer vision methods that can utilize crowd-sourced annotations can help scale up analysis further. In this work we study the potential to do so on the challenging task of fine-grained counting. As opposed to the standard crowd counting task, fine-grained counting also involves classifying attributes of individuals in dense crowds. We introduce a new dataset from animal ecology to enable this study that contains 1.7M crowd-sourced annotations of 8 fine-grained classes. It is the largest available dataset for fine-grained counting and the first to enable the study of the task with crowd-sourced annotations. We introduce methods for generating aggregate "ground truths" from the collected annotations, as well as a counting method that can utilize the aggregate information. Our method improves results by 8% over a comparable baseline, indicating the potential for algorithms to learn fine-grained counting using crowd-sourced supervision.
Causal Machine Learning for Healthcare and Precision Medicine
Venue: arXiv cs.LG 2022
Year: 2022

Causal machine learning (CML) has experienced increasing popularity in healthcare. Beyond the inherent capabilities of adding domain knowledge into learning systems, CML provides a complete toolset for investigating how a system would react to an intervention (e.g.\ outcome given a treatment). Quantifying effects of interventions allows actionable decisions to be made whilst maintaining robustness in the presence of confounders. Here, we explore how causal inference can be incorporated into different aspects of clinical decision support (CDS) systems by using recent advances in machine learning. Throughout this paper, we use Alzheimer's disease (AD) to create examples for illustrating how CML can be advantageous in clinical scenarios. Furthermore, we discuss important challenges present in healthcare applications such as processing high-dimensional and unstructured data, generalisation to out-of-distribution samples, and temporal relationships, that despite the great effort from the research community remain to be solved. Finally, we review lines of research within causal representation learning, causal discovery and causal reasoning which offer the potential towards addressing the aforementioned challenges.
LILA-BOTI : Leveraging Isolated Letter Accumulations By Ordering Teacher Insights for Bangla Handwriting Recognition
Venue: arXiv cs.CV 2022
Year: 2022

Word-level handwritten optical character recognition (OCR) remains a challenge for morphologically rich languages like Bangla. The complexity arises from the existence of a large number of alphabets, the presence of several diacritic forms, and the appearance of complex conjuncts. The difficulty is exacerbated by the fact that some graphemes occur infrequently but remain indispensable, so addressing the class imbalance is required for satisfactory results. This paper addresses this issue by introducing two knowledge distillation methods: Leveraging Isolated Letter Accumulations By Ordering Teacher Insights (LILA-BOTI) and Super Teacher LILA-BOTI. In both cases, a Convolutional Recurrent Neural Network (CRNN) student model is trained with the dark knowledge gained from a printed isolated character recognition teacher model. We conducted inter-dataset testing on \emph{BN-HTRd} and \emph{BanglaWriting} as our evaluation protocol, thus setting up a challenging problem where the results would better reflect the performance on unseen data. Our evaluations achieved up to a 3.5% increase in the F1-Macro score for the minor classes and up to 4.5% increase in our overall word recognition rate when compared with the base model (No KD) and conventional KD.
Logical Reasoning with Span-Level Predictions for Interpretable and Robust NLI Models
Venue: arXiv cs.CL 2022
Year: 2022

Current Natural Language Inference (NLI) models achieve impressive results, sometimes outperforming humans when evaluating on in-distribution test sets. However, as these models are known to learn from annotation artefacts and dataset biases, it is unclear to what extent the models are learning the task of NLI instead of learning from shallow heuristics in their training data. We address this issue by introducing a logical reasoning framework for NLI, creating highly transparent model decisions that are based on logical rules. Unlike prior work, we show that improved interpretability can be achieved without decreasing the predictive accuracy. We almost fully retain performance on SNLI, while also identifying the exact hypothesis spans that are responsible for each model prediction. Using the e-SNLI human explanations, we verify that our model makes sensible decisions at a span level, despite not using any span labels during training. We can further improve model performance and span-level decisions by using the e-SNLI explanations during training. Finally, our model is more robust in a reduced data setting. When training with only 1,000 examples, out-of-distribution performance improves on the MNLI matched and mismatched validation sets by 13% and 16% relative to the baseline. Training with fewer observations yields further improvements, both in-distribution and out-of-distribution.
Advanced Transient Diagnostic with Ensemble Digital Twin Modeling
Venue: arXiv cs.LG 2022
Year: 2022

The use of machine learning (ML) model as digital-twins for reduced-order-modeling (ROM) in lieu of system codes has grown traction over the past few years. However, due to the complex and non-linear nature of nuclear reactor transients as well as the large range of tasks required, it is infeasible for a single ML model to generalize across all tasks. In this paper, we incorporate issue specific digital-twin ML models with ensembles to enhance the prediction outcome. The ensemble also utilizes an indirect probabilistic tracking method of surrogate state variables to produce accurate predictions of unobservable safety goals. The unique method named Ensemble Diagnostic Digital-twin Modeling (EDDM) can select not only the most appropriate predictions from the incorporated diagnostic digital-twin models but can also reduce generalization error associated with training as opposed to single models.
Exploiting the Curvature of Feasible Sets for Faster Projection-Free Online Learning
Venue: arXiv cs.LG 2022
Year: 2022

In this paper, we develop new efficient projection-free algorithms for Online Convex Optimization (OCO). Online Gradient Descent (OGD) is an example of a classical OCO algorithm that guarantees the optimal $O(\sqrt{T})$ regret bound. However, OGD and other projection-based OCO algorithms need to perform a Euclidean projection onto the feasible set $\mathcal{C}\subset \mathbb{R}^d$ whenever their iterates step outside $\mathcal{C}$. For various sets of interests, this projection step can be computationally costly, especially when the ambient dimension is large. This has motivated the development of projection-free OCO algorithms that swap Euclidean projections for often much cheaper operations such as Linear Optimization (LO). However, state-of-the-art LO-based algorithms only achieve a suboptimal $O(T^{3/4})$ regret for general OCO. In this paper, we leverage recent results in parameter-free Online Learning, and develop an OCO algorithm that makes two calls to an LO Oracle per round and achieves the near-optimal $\widetilde{O}(\sqrt{T})$ regret whenever the feasible set is strongly convex. We also present an algorithm for general convex sets that makes $\widetilde O(d)$ expected number of calls to an LO Oracle per round and guarantees a $\widetilde O(T^{2/3})$ regret, improving on the previous best $O(T^{3/4})$. We achieve the latter by approximating any convex set $\mathcal{C}$ by a strongly convex one, where LO can be performed using $\widetilde {O}(d)$ expected number of calls to an LO Oracle for $\mathcal{C}$.