OpenRaiser/Intern-Atlas huggingface dataset:
Evaluation of Field-Aware Neural Ranking Models for Recipe Search
Venue: arXiv cs.IR 2021
Year: 2021

Explicitly modelling field interactions and correlations in complex document structures has recently gained popularity in neural document embedding and retrieval tasks. Although this requires the specification of bespoke task-dependent models, encouraging empirical results are beginning to emerge. We present the first in-depth analyses of non-linear multi-field interaction (NL-MFI) ranking in the cooking domain in this work. Our results show that field-weighted factorisation machines models provide a statistically significant improvement over baselines in recipe retrieval tasks. Additionally, we show that sparsely capturing subsets of field interactions based on domain knowledge and feature selection heuristics offers significant advantages over baselines and exhaustive alternatives. Although field-interaction aware models are more elaborate from an architectural basis, they are often more data-efficient in optimisation and are better suited for explainability due to mirrored document and model factorisation.
Directional GAN: A Novel Conditioning Strategy for Generative Networks
Venue: arXiv cs.CV 2021
Year: 2021

Image content is a predominant factor in marketing campaigns, websites and banners. Today, marketers and designers spend considerable time and money in generating such professional quality content. We take a step towards simplifying this process using Generative Adversarial Networks (GANs). We propose a simple and novel conditioning strategy which allows generation of images conditioned on given semantic attributes using a generator trained for an unconditional image generation task. Our approach is based on modifying latent vectors, using directional vectors of relevant semantic attributes in latent space. Our method is designed to work with both discrete (binary and multi-class) and continuous image attributes. We show the applicability of our proposed approach, named Directional GAN, on multiple public datasets, with an average accuracy of 86.4% across different attributes.
Representation in Dynamical Systems
Venue: arXiv cs.AI 2021
Year: 2021

The brain is often called a computer and likened to a Turing machine, in part because the mind can manipulate discrete symbols such as numbers. But the brain is a dynamical system, more like a Watt governor than a Turing machine. Can a dynamical system be said to operate using "representations"? This paper argues that it can, although not in the way a digital computer does. Instead, it uses phenomena best described using mathematic concepts such as chaotic attractors to stand in for aspects of the world.
The FeatureCloud AI Store for Federated Learning in Biomedicine and Beyond
Venue: arXiv cs.LG 2021
Year: 2021

Machine Learning (ML) and Artificial Intelligence (AI) have shown promising results in many areas and are driven by the increasing amount of available data. However, this data is often distributed across different institutions and cannot be shared due to privacy concerns. Privacy-preserving methods, such as Federated Learning (FL), allow for training ML models without sharing sensitive data, but their implementation is time-consuming and requires advanced programming skills. Here, we present the FeatureCloud AI Store for FL as an all-in-one platform for biomedical research and other applications. It removes large parts of this complexity for developers and end-users by providing an extensible AI Store with a collection of ready-to-use apps. We show that the federated apps produce similar results to centralized ML, scale well for a typical number of collaborators and can be combined with Secure Multiparty Computation (SMPC), thereby making FL algorithms safely and easily applicable in biomedical and clinical environments.
XOR-Based Codes for Private Information Retrieval with Private Side Information
Venue: arXiv cs.IT 2021
Year: 2021

We consider the problem of Private Information Retrieval with Private Side Information (PIR-PSI), wherein a user wants to retrieve a file from replication based non-colluding databases by using the prior knowledge of a subset of the files stored on the databases. The PIR-PSI framework ensures that the privacy of the demand and the side information are jointly preserved, thereby finding potential applications when multiple files have to be downloaded spread across different time-instants. Although the capacity of the PIR-PSI setting is known, we observe that the underlying capacity achieving code construction uses Maximum Distance Separable (MDS) codes thereby contributing to high computational complexity when retrieving the demand. Pointing at this drawback of MDS-based PIR-PSI codes, we propose XOR-based PIR-PSI codes for a simple yet non-trivial setting of two non-colluding databases and two side information files at the user. While our codes offer substantial reduction in complexity when compared to MDS based codes, the code-rate marginally falls short of the capacity of the PIR-PSI setting. Nevertheless, we show that our code-rate is strictly higher than that of XOR-based codes for PIR with no side information, thereby implying that our codes can be useful when downloading multiple files in a sequential manner, instead of applying XOR-based PIR codes on each file.
Probabilistic Loss and its Online Characterization for Simplified Decision Making Under Uncertainty
Venue: arXiv cs.AI 2021
Year: 2021

It is a long-standing objective to ease the computation burden incurred by the decision making process. Identification of this mechanism's sensitivity to simplification has tremendous ramifications. Yet, algorithms for decision making under uncertainty usually lean on approximations or heuristics without quantifying their effect. Therefore, challenging scenarios could severely impair the performance of such methods. In this paper, we extend the decision making mechanism to the whole by removing standard approximations and considering all previously suppressed stochastic sources of variability. On top of this extension, our key contribution is a novel framework to simplify decision making while assessing and controlling online the simplification's impact. Furthermore, we present novel stochastic bounds on the return and characterize online the effect of simplification using this framework on a particular simplification technique - reducing the number of samples in belief representation for planning. Finally, we verify the advantages of our approach through extensive simulations.
Using social network analysis to prevent money laundering
Venue: arXiv cs.SI 2021
Year: 2021

This research explores the opportunities for the application of network analytic techniques to prevent money laundering. We worked on real world data by analyzing the central database of a factoring company, mainly operating in Italy, over a period of 19 months. This database contained the financial operations linked to the factoring business, together with other useful information about the company clients. We propose a new approach to sort and map relational data and present predictive models, based on network metrics, to assess risk profiles of clients involved in the factoring business. We find that risk profiles can be predicted by using social network metrics. In our dataset, the most dangerous social actors deal with bigger or more frequent financial operations; they are more peripheral in the transactions network; they mediate transactions across different economic sectors and operate in riskier countries or Italian regions. Finally, to spot potential clusters of criminals, we propose a visual analysis of the tacit links existing among different companies who share the same owner or representative. Our findings show the importance of using a network-based approach when looking for suspicious financial operations and potential criminals.
Kleister: Key Information Extraction Datasets Involving Long Documents with Complex Layouts
Venue: arXiv cs.CL 2021
Year: 2021

The relevance of the Key Information Extraction (KIE) task is increasingly important in natural language processing problems. But there are still only a few well-defined problems that serve as benchmarks for solutions in this area. To bridge this gap, we introduce two new datasets (Kleister NDA and Kleister Charity). They involve a mix of scanned and born-digital long formal English-language documents. In these datasets, an NLP system is expected to find or infer various types of entities by employing both textual and structural layout features. The Kleister Charity dataset consists of 2,788 annual financial reports of charity organizations, with 61,643 unique pages and 21,612 entities to extract. The Kleister NDA dataset has 540 Non-disclosure Agreements, with 3,229 unique pages and 2,160 entities to extract. We provide several state-of-the-art baseline systems from the KIE domain (Flair, BERT, RoBERTa, LayoutLM, LAMBERT), which show that our datasets pose a strong challenge to existing models. The best model achieved an 81.77% and an 83.57% F1-score on respectively the Kleister NDA and the Kleister Charity datasets. We share the datasets to encourage progress on more in-depth and complex information extraction tasks.
Dynamical Isometry: The Missing Ingredient for Neural Network Pruning
Venue: arXiv cs.LG 2021
Year: 2021

Several recent works [40, 24] observed an interesting phenomenon in neural network pruning: A larger finetuning learning rate can improve the final performance significantly. Unfortunately, the reason behind it remains elusive up to date. This paper is meant to explain it through the lens of dynamical isometry [42]. Specifically, we examine neural network pruning from an unusual perspective: pruning as initialization for finetuning, and ask whether the inherited weights serve as a good initialization for the finetuning? The insights from dynamical isometry suggest a negative answer. Despite its critical role, this issue has not been well-recognized by the community so far. In this paper, we will show the understanding of this problem is very important -- on top of explaining the aforementioned mystery about the larger finetuning rate, it also unveils the mystery about the value of pruning [5, 30]. Besides a clearer theoretical understanding of pruning, resolving the problem can also bring us considerable performance benefits in practice.
Two-Hop Network with Multiple Decision Centers under Expected-Rate Constraints
Venue: arXiv cs.IT 2021
Year: 2021

The paper studies distributed binary hypothesis testing over a two-hop relay network where both the relay and the receiver decide on the hypothesis. Both communication links are subject to expected rate constraints, which differs from the classical assumption of maximum rate constraints. We exactly characterize the set of type-II error exponent pairs at the relay and the receiver when both type-I error probabilities are constrained by the same value $ε>0$. No tradeoff is observed between the two exponents, i.e., one can simultaneously attain maximum type-II error exponents both at the relay and at the receiver. For $ε_1 \neq ε_2$, we present an achievable exponents region, which we obtain with a scheme that applies different versions of a basic two-hop scheme that is optimal under maximum rate constraints. We use the basic two-hop scheme with two choices of parameters and rates, depending on the transmitter's observed sequence. For $ε_1=ε_2$, a single choice is shown to be sufficient. Numerical simulations indicate that extending to three or more parameter choices is never beneficial.
Open-end bin packing: new and old analysis approaches
Venue: arXiv cs.DS 2021
Year: 2021

We analyze a recently introduced concept, called the price of clustering, for variants of bin packing called open-end bin packing problems (OEBP). Input items have sizes, and they also belong to a certain number of types. The new concept deals with the comparison of optimal solutions for the cases where items of distinct types can and cannot be packed together, respectively. The problem is related to greedy bin packing algorithms and to batched bin packing, and we discuss some of those concepts as well. We analyze max-OEBP, where a packed bin is valid if by excluding its largest item, the total size of items is below 1. For this variant, we study the case of general item sizes, and the parametric case with bounded item sizes, which shows the effect of small items. Finally, we briefly discuss min-OEBP, where a bin is valid if the total size of its items excluding the smallest item is below 1, which is known to be an entirely different problem.
Monetizing Propaganda: How Far-right Extremists Earn Money by Video Streaming
Venue: arXiv cs.SI 2021
Year: 2021

Video streaming platforms such as Youtube, Twitch, and DLive allow users to live-stream video content for viewers who can optionally express their appreciation through monetary donations. DLive is one of the smaller and lesser-known streaming platforms, and historically has had fewer content moderation practices. It has thus become a popular place for violent extremists and other clandestine groups to earn money and propagandize. What is the financial structure of the DLive streaming ecosystem and how much money is changing hands? In the past it has been difficult to understand how far-right extremists fundraise via podcasts and video streams because of the secretive nature of the activity and because of the difficulty of getting data from social media platforms. This paper describes a novel experiment to collect and analyze data from DLive's publicly available ledgers of transactions in order to understand the financial structure of the clandestine, extreme far-right video streaming community. The main findings of this paper are, first, that the majority of donors are using micropayments in varying frequencies, but a small handful of donors spend large amounts of money to finance their favorite streamers. Next, the timing of donations to high-profile far-right streamers follows a fairly predictable pattern that is closely tied to a broadcast schedule. Finally, the far-right video streaming financial landscape is divided into separate cliques which exhibit very little crossover in terms of sizable donations. This work will be important to technology companies, policymakers, and researchers who are trying to understand how niche social media services, including video platforms, are being exploited by extremists to propagandize and fundraise.
A Geometric Approach to the Kinematics of the Canfield Joint
Venue: arXiv cs.RO 2021
Year: 2021

This paper details an accessible geometric derivation of the forward and inverse kinematics of a parallel robotic linkage known as the Canfield joint, which can be used for pointing applications. The original purpose of the Canfield joint was to serve as a human wrist replacement, and it can be utilized for other purposes such as the precision pointing and tracking of antennas, telescopes, and thrusters. We build upon previous analyses, and generalize them to include the situation where one of the three legs freezes; the kinematics are also substantially generalized beyond failure modes, detailed within. The core of this work states and clarifies the assumptions necessary to analyze this type of parallel robotic linkage. Specific guidance is included for engineering use cases.
Analysing The Impact Of Linguistic Features On Cross-Lingual Transfer
Venue: arXiv cs.CL 2021
Year: 2021

There is an increasing amount of evidence that in cases with little or no data in a target language, training on a different language can yield surprisingly good results. However, currently there are no established guidelines for choosing the training (source) language. In attempt to solve this issue we thoroughly analyze a state-of-the-art multilingual model and try to determine what impacts good transfer between languages. As opposed to the majority of multilingual NLP literature, we don't only train on English, but on a group of almost 30 languages. We show that looking at particular syntactic features is 2-4 times more helpful in predicting the performance than an aggregated syntactic similarity. We find out that the importance of syntactic features strongly differs depending on the downstream task - no single feature is a good performance predictor for all NLP tasks. As a result, one should not expect that for a target language $L_1$ there is a single language $L_2$ that is the best choice for any NLP task (for instance, for Bulgarian, the best source language is French on POS tagging, Russian on NER and Thai on NLI). We discuss the most important linguistic features affecting the transfer quality using statistical and machine learning methods.
Multilingual Offensive Language Identification for Low-resource Languages
Venue: arXiv cs.CL 2021
Year: 2021

Offensive content is pervasive in social media and a reason for concern to companies and government organizations. Several studies have been recently published investigating methods to detect the various forms of such content (e.g. hate speech, cyberbullying, and cyberaggression). The clear majority of these studies deal with English partially because most annotated datasets available contain English data. In this paper, we take advantage of available English datasets by applying cross-lingual contextual word embeddings and transfer learning to make predictions in low-resource languages. We project predictions on comparable data in Arabic, Bengali, Danish, Greek, Hindi, Spanish, and Turkish. We report results of 0.8415 F1 macro for Bengali in TRAC-2 shared task, 0.8532 F1 macro for Danish and 0.8701 F1 macro for Greek in OffensEval 2020, 0.8568 F1 macro for Hindi in HASOC 2019 shared task and 0.7513 F1 macro for Spanish in in SemEval-2019 Task 5 (HatEval) showing that our approach compares favourably to the best systems submitted to recent shared tasks on these three languages. Additionally, we report competitive performance on Arabic, and Turkish using the training and development sets of OffensEval 2020 shared task. The results for all languages confirm the robustness of cross-lingual contextual embeddings and transfer learning for this task.
Verification of the Incremental Merkle Tree Algorithm with Dafny
Venue: arXiv cs.LO 2021
Year: 2021

The Deposit Smart Contract (DSC) is an instrumental component of the Ethereum 2.0 Phase 0 infrastructure. We have developed the first machine-checkable version of the incremental Merkle tree algorithm used in the DSC. We present our new and original correctness proof of the algorithm along with the Dafny machine-checkable version. The main results are: 1) a new proof of total correctness; 2) a software artefact with the proof in the form of the complete Dafny code base and 3) new provably correct optimisations of the algorithm.
Are Larger Pretrained Language Models Uniformly Better? Comparing Performance at the Instance Level
Venue: arXiv cs.CL 2021
Year: 2021

Larger language models have higher accuracy on average, but are they better on every single instance (datapoint)? Some work suggests larger models have higher out-of-distribution robustness, while other work suggests they have lower accuracy on rare subgroups. To understand these differences, we investigate these models at the level of individual instances. However, one major challenge is that individual predictions are highly sensitive to noise in the randomness in training. We develop statistically rigorous methods to address this, and after accounting for pretraining and finetuning noise, we find that our BERT-Large is worse than BERT-Mini on at least 1-4% of instances across MNLI, SST-2, and QQP, compared to the overall accuracy improvement of 2-10%. We also find that finetuning noise increases with model size and that instance-level accuracy has momentum: improvement from BERT-Mini to BERT-Medium correlates with improvement from BERT-Medium to BERT-Large. Our findings suggest that instance-level predictions provide a rich source of information; we therefore, recommend that researchers supplement model weights with model predictions.
Models of Computing as a Service and IoT: an analysis of the current scenario with applications using LPWAN
Venue: arXiv cs.DC 2021
Year: 2021

This work provides the basis to understand and select Cloud Computing models applied for the development of IoT solutions using Low-Power Wide Area Network (LPWAN). Cloud Computing paradigm has transformed how the industry implement solution, through the commoditization of shared IT infrastructures. The advent of massive Internet of Things (IoT) and related workloads brings new challenges to this scenario demanding malleable configurations where the resources are distributed closer to data sources. We introduce an analysis of existing solution architectures, along with an illustrative case from where we derive the lessons, challenges, and opportunities of combining these technologies for a new generation of Cloud-native solutions.
VPP-ART: An Efficient Implementation of Fixed-Size-Candidate-Set Adaptive Random Testing using Vantage Point Partitioning
Venue: arXiv cs.SE 2021
Year: 2021

Adaptive Random Testing (ART) is an enhancement of Random Testing (RT), and aims to improve the RT failure-detection effectiveness by distributing test cases more evenly in the input domain. Many ART algorithms have been proposed, with Fixed-Size-Candidate-Set ART (FSCS-ART) being one of the most effective and popular. FSCS-ART ensures high failure-detection effectiveness by selecting the next test case as the candidate farthest from previously-executed test cases. Although FSCS-ART has good failure-detection effectiveness, it also faces some challenges, including heavy computational overheads. In this paper, we propose an enhanced version of FSCS-ART, Vantage Point Partitioning ART (VPP-ART). VPP-ART addresses the FSCS-ART computational overhead problem using vantage point partitioning, while maintaining the failure-detection effectiveness. VPP-ART partitions the input domain space using a modified Vantage Point tree (VP-tree) and finds the approximate nearest executed test cases of a candidate test case in the partitioned sub-domains -- thereby significantly reducing the time overheads compared with the searches required for FSCS-ART. To enable the FSCS-ART dynamic insertion process, we modify the traditional VP-tree to support dynamic data. The simulation results show that VPP-ART has a much lower time overhead compared to FSCS-ART, but also delivers similar (or better) failure-detection effectiveness, especially in the higher dimensional input domains. According to statistical analyses, VPP-ART can improve on the FSCS-ART failure-detection effectiveness by approximately 50% to 58%. VPP-ART also compares favorably with the KDFC-ART algorithms (a series of enhanced ART algorithms based on the KD-tree). Our experiments also show that VPP-ART is more cost-effective than FSCS-ART and KDFC-ART.