OpenRaiser/Intern-Atlas huggingface dataset:
Effective Explanations for Entity Resolution Models
Venue: arXiv cs.DB 2022
Year: 2022

Entity resolution (ER) aims at matching records that refer to the same real-world entity. Although widely studied for the last 50 years, ER still represents a challenging data management problem, and several recent works have started to investigate the opportunity of applying deep learning (DL) techniques to solve this problem. In this paper, we study the fundamental problem of explainability of the DL solution for ER. Understanding the matching predictions of an ER solution is indeed crucial to assess the trustworthiness of the DL model and to discover its biases. We treat the DL model as a black box classifier and - while previous approaches to provide explanations for DL predictions are agnostic to the classification task. we propose the CERTA approach that is aware of the semantics of the ER problem. Our approach produces both saliency explanations, which associate each attribute with a saliency score, and counterfactual explanations, which provide examples of values that can flip the prediction. CERTA builds on a probabilistic framework that aims at computing the explanations evaluating the outcomes produced by using perturbed copies of the input records. We experimentally evaluate CERTA's explanations of state-of-the-art ER solutions based on DL models using publicly available datasets, and demonstrate the effectiveness of CERTA over recently proposed methods for this problem.
Kratt: Developing an Automatic Subject Indexing Tool for The National Library of Estonia
Venue: arXiv cs.CL 2022
Year: 2022

Manual subject indexing in libraries is a time-consuming and costly process and the quality of the assigned subjects is affected by the cataloguer's knowledge on the specific topics contained in the book. Trying to solve these issues, we exploited the opportunities arising from artificial intelligence to develop Kratt: a prototype of an automatic subject indexing tool. Kratt is able to subject index a book independent of its extent and genre with a set of keywords present in the Estonian Subject Thesaurus. It takes Kratt approximately 1 minute to subject index a book, outperforming humans 10-15 times. Although the resulting keywords were not considered satisfactory by the cataloguers, the ratings of a small sample of regular library users showed more promise. We also argue that the results can be enhanced by including a bigger corpus for training the model and applying more careful preprocessing techniques.
Information Preferences of Individual Agents in Linear-Quadratic-Gaussian Network Games
Venue: arXiv cs.GT 2022
Year: 2022

We consider linear-quadratic-Gaussian (LQG) network games in which agents have quadratic payoffs that depend on their individual and neighbors' actions, and an unknown payoff-relevant state. An information designer determines the fidelity of information revealed to the agents about the payoff state to maximize the social welfare. Prior results show that full information disclosure is optimal under certain assumptions on the payoffs, i.e., it is beneficial for the average individual. In this paper, we provide conditions based on the strength of the dependence of payoffs on neighbors' actions, i.e., competition, under which a rational agent is expected to benefit, i.e., receive higher payoffs, from full information disclosure. We find that all agents benefit from information disclosure for the star network structure when the game is symmetric and submodular or supermodular. We also identify that the central agent benefits more than a peripheral agent from full information disclosure unless the competition is strong and the number of peripheral agents is small enough. Despite the fact that all agents expect to benefit from information disclosure ex-ante, a central agent can be worse-off from information disclosure in many realizations of the payoff state under strong competition, indicating that a risk-averse central agent can prefer uninformative signals ex-ante.
Explainable Artificial Intelligence for Exhaust Gas Temperature of Turbofan Engines
Venue: arXiv cs.LG 2022
Year: 2022

Data-driven modeling is an imperative tool in various industrial applications, including many applications in the sectors of aeronautics and commercial aviation. These models are in charge of providing key insights, such as which parameters are important on a specific measured outcome or which parameter values we should expect to observe given a set of input parameters. At the same time, however, these models rely heavily on assumptions (e.g., stationarity) or are "black box" (e.g., deep neural networks), meaning that they lack interpretability of their internal working and can be viewed only in terms of their inputs and outputs. An interpretable alternative to the "black box" models and with considerably less assumptions is symbolic regression (SR). SR searches for the optimal model structure while simultaneously optimizing the model's parameters without relying on an a-priori model structure. In this work, we apply SR on real-life exhaust gas temperature (EGT) data, collected at high frequencies through the entire flight, in order to uncover meaningful algebraic relationships between the EGT and other measurable engine parameters. The experimental results exhibit promising model accuracy, as well as explainability returning an absolute difference of 3°C compared to the ground truth and demonstrating consistency from an engineering perspective.
Physics-based Learning of Parameterized Thermodynamics from Real-time Thermography
Venue: arXiv cs.CV 2022
Year: 2022

Progress in automatic control of thermal processes and real-time estimation of heat penetration into live tissue has long been limited by the difficulty of obtaining high-fidelity thermodynamic models. Traditionally, in complex thermodynamic systems, it is often infeasible to estimate the thermophysical parameters of spatiotemporally varying processes, forcing the adoption of model-free control architectures. This comes at the cost of losing any robustness guarantees, and implies a need for extensive real-life testing. In recent years, however, infrared cameras and other thermographic equipment have become readily applicable to these processes, allowing for a real-time, non-invasive means of sensing the thermal state of a process. In this work, we present a novel physics-based approach to learning a thermal process's dynamics directly from such real-time thermographic data, while focusing attention on regions with high thermal activity. We call this process, which applies to any higher-dimensional scalar field, attention-based noise robust averaging (ANRA). Given a partial-differential equation model structure, we show that our approach is robust against noise, and can be used to initialize optimization routines to further refine parameter estimates. We demonstrate our method on several simulation examples, as well as by applying it to electrosurgical thermal response data on in vivo porcine skin tissue.
Continuous-Time Audiovisual Fusion with Recurrence vs. Attention for In-The-Wild Affect Recognition
Venue: arXiv cs.SD 2022
Year: 2022

In this paper, we present our submission to 3rd Affective Behavior Analysis in-the-wild (ABAW) challenge. Learningcomplex interactions among multimodal sequences is critical to recognise dimensional affect from in-the-wild audiovisual data. Recurrence and attention are the two widely used sequence modelling mechanisms in the literature. To clearly understand the performance differences between recurrent and attention models in audiovisual affect recognition, we present a comprehensive evaluation of fusion models based on LSTM-RNNs, self-attention and cross-modal attention, trained for valence and arousal estimation. Particularly, we study the impact of some key design choices: the modelling complexity of CNN backbones that provide features to the the temporal models, with and without end-to-end learning. We trained the audiovisual affect recognition models on in-the-wild ABAW corpus by systematically tuning the hyper-parameters involved in the network architecture design and training optimisation. Our extensive evaluation of the audiovisual fusion models shows that LSTM-RNNs can outperform the attention models when coupled with low-complex CNN backbones and trained in an end-to-end fashion, implying that attention models may not necessarily be the optimal choice for continuous-time multimodal emotion recognition.
Tackling Online One-Class Incremental Learning by Removing Negative Contrasts
Venue: arXiv cs.LG 2022
Year: 2022

Recent work studies the supervised online continual learning setting where a learner receives a stream of data whose class distribution changes over time. Distinct from other continual learning settings the learner is presented new samples only once and must distinguish between all seen classes. A number of successful methods in this setting focus on storing and replaying a subset of samples alongside incoming data in a computationally efficient manner. One recent proposal ER-AML achieved strong performance in this setting by applying an asymmetric loss based on contrastive learning to the incoming data and replayed data. However, a key ingredient of the proposed method is avoiding contrasts between incoming data and stored data, which makes it impractical for the setting where only one new class is introduced in each phase of the stream. In this work we adapt a recently proposed approach (\textit{BYOL}) from self-supervised learning to the supervised learning setting, unlocking the constraint on contrasts. We then show that supplementing this with additional regularization on class prototypes yields a new method that achieves strong performance in the one-class incremental learning setting and is competitive with the top performing methods in the multi-class incremental setting.
Human Gait Recognition Using Bag of Words Feature Representation Method
Venue: arXiv cs.CV 2022
Year: 2022

In this paper, we propose a novel gait recognition method based on a bag-of-words feature representation method. The algorithm is trained, tested and evaluated on a unique human gait data consisting of 93 individuals who walked with comfortable pace between two end points during two different sessions. To evaluate the effectiveness of the proposed model, the results are compared with the outputs of the classification using extracted features. As it is presented, the proposed method results in significant improvement accuracy compared to using common statistical features, in all the used classifiers.
Email Summarization to Assist Users in Phishing Identification
Venue: arXiv cs.CR 2022
Year: 2022

Cyber-phishing attacks recently became more precise, targeted, and tailored by training data to activate only in the presence of specific information or cues. They are adaptable to a much greater extent than traditional phishing detection. Hence, automated detection systems cannot always be 100% accurate, increasing the uncertainty around expected behavior when faced with a potential phishing email. On the other hand, human-centric defence approaches focus extensively on user training but face the difficulty of keeping users up to date with continuously emerging patterns. Therefore, advances in analyzing the content of an email in novel ways along with summarizing the most pertinent content to the recipients of emails is a prospective gateway to furthering how to combat these threats. Addressing this gap, this work leverages transformer-based machine learning to (i) analyze prospective psychological triggers, to (ii) detect possible malicious intent, and (iii) create representative summaries of emails. We then amalgamate this information and present it to the user to allow them to (i) easily decide whether the email is "phishy" and (ii) self-learn advanced malicious patterns.
Automatic Song Translation for Tonal Languages
Venue: arXiv cs.CL 2022
Year: 2022

This paper develops automatic song translation (AST) for tonal languages and addresses the unique challenge of aligning words' tones with melody of a song in addition to conveying the original meaning. We propose three criteria for effective AST -- preserving meaning, singability and intelligibility -- and design metrics for these criteria. We develop a new benchmark for English--Mandarin song translation and develop an unsupervised AST system, Guided AliGnment for Automatic Song Translation (GagaST), which combines pre-training with three decoding constraints. Both automatic and human evaluations show GagaST successfully balances semantics and singability.
Independent set reconfiguration on directed graphs
Venue: arXiv cs.DS 2022
Year: 2022

\textsc{Directed Token Sliding} asks, given a directed graph and two sets of pairwise nonadjacent vertices, whether one can reach from one set to the other by repeatedly applying a local operation that exchanges a vertex in the current set with one of its out-neighbors, while keeping the nonadjacency. It can be seen as a reconfiguration process where a token is placed on each vertex in the current set, and the local operation slides a token along an arc respecting its direction. Previously, such a problem was extensively studied on undirected graphs, where the edges have no directions and thus the local operation is symmetric. \textsc{Directed Token Sliding} is a generalization of its undirected variant since an undirected edge can be simulated by two arcs of opposite directions. In this paper, we initiate the algorithmic study of \textsc{Directed Token Sliding}. We first observe that the problem is PSPACE-complete even if we forbid parallel arcs in opposite directions and that the problem on directed acyclic graphs is NP-complete and W[1]-hard parameterized by the size of the sets in consideration. We then show our main result: a linear-time algorithm for the problem on directed graphs whose underlying undirected graphs are trees, which are called polytrees. Such a result is also known for the undirected variant of the problem on trees~[Demaine et al.~TCS 2015], but the techniques used here are quite different because of the asymmetric nature of the directed problem. We present a characterization of yes-instances based on the existence of a certain set of directed paths, and then derive simple equivalent conditions from it by some observations, which admits an efficient algorithm. For the polytree case, we also present a quadratic-time algorithm that outputs, if the input is a yes-instance, one of the shortest reconfiguration sequences.
Semi-supervised and Deep learning Frameworks for Video Classification and Key-frame Identification
Venue: arXiv cs.CV 2022
Year: 2022

Automating video-based data and machine learning pipelines poses several challenges including metadata generation for efficient storage and retrieval and isolation of key-frames for scene understanding tasks. In this work, we present two semi-supervised approaches that automate this process of manual frame sifting in video streams by automatically classifying scenes for content and filtering frames for fine-tuning scene understanding tasks. The first rule-based method starts from a pre-trained object detector and it assigns scene type, uncertainty and lighting categories to each frame based on probability distributions of foreground objects. Next, frames with the highest uncertainty and structural dissimilarity are isolated as key-frames. The second method relies on the simCLR model for frame encoding followed by label-spreading from 20% of frame samples to label the remaining frames for scene and lighting categories. Also, clustering the video frames in the encoded feature space further isolates key-frames at cluster boundaries. The proposed methods achieve 64-93% accuracy for automated scene categorization for outdoor image videos from public domain datasets of JAAD and KITTI. Also, less than 10% of all input frames can be filtered as key-frames that can then be sent for annotation and fine tuning of machine vision algorithms. Thus, the proposed framework can be scaled to additional video data streams for automated training of perception-driven systems with minimal training images.
Interactive Style Transfer: All is Your Palette
Venue: arXiv cs.CV 2022
Year: 2022

Neural style transfer (NST) can create impressive artworks by transferring reference style to content image. Current image-to-image NST methods are short of fine-grained controls, which are often demanded by artistic editing. To mitigate this limitation, we propose a drawing-like interactive style transfer (IST) method, by which users can interactively create a harmonious-style image. Our IST method can serve as a brush, dip style from anywhere, and then paint to any region of the target content image. To determine the action scope, we formulate a fluid simulation algorithm, which takes styles as pigments around the position of brush interaction, and diffusion in style or content images according to the similarity maps. Our IST method expands the creative dimension of NST. By dipping and painting, even employing one style image can produce thousands of eye-catching works. The demo video is available in supplementary files or in http://mmcheng.net/ist.
Enhancing Transferability of Adversarial Examples with Spatial Momentum
Venue: arXiv cs.CV 2022
Year: 2022

Many adversarial attack methods achieve satisfactory attack success rates under the white-box setting, but they usually show poor transferability when attacking other DNN models. Momentum-based attack is one effective method to improve transferability. It integrates the momentum term into the iterative process, which can stabilize the update directions by adding the gradients' temporal correlation for each pixel. We argue that only this temporal momentum is not enough, the gradients from the spatial domain within an image, i.e. gradients from the context pixels centered on the target pixel are also important to the stabilization. For that, we propose a novel method named Spatial Momentum Iterative FGSM attack (SMI-FGSM), which introduces the mechanism of momentum accumulation from temporal domain to spatial domain by considering the context information from different regions within the image. SMI-FGSM is then integrated with temporal momentum to simultaneously stabilize the gradients' update direction from both the temporal and spatial domains. Extensive experiments show that our method indeed further enhances adversarial transferability. It achieves the best transferability success rate for multiple mainstream undefended and defended models, which outperforms the state-of-the-art attack methods by a large margin of 10\% on average.
Big data ethics, machine ethics or information ethics? Navigating the maze of applied ethics in IT
Venue: arXiv cs.SE 2022
Year: 2022

Digitalization efforts are rapidly spreading across societies, challenging new and important ethical issues that arise from technological development. Software developers, designers and managerial decision-makers are ever more expected to consider ethical values and conduct normative evaluations when building digital products. Yet, when one looks for guidance in the academic literature one encounters a plethora of branches of applied ethics. Depending on the context of the system that is to be developed, interesting subfields like big data ethics, machine ethics, information ethics, AI ethics or computer ethics (to only name a few) may present themselves. In this paper we want to offer assistance to any member of a development team by giving a clear and brief introduction into two fields of ethical endeavor (normative ethics and applied ethics), describing how they are related to each other and, finally, provide an ordering of the different branches of applied ethics (big data ethics, machine ethics, information ethics, AI ethics or computer ethics etc.) which have gained traction over the last years. Finally, we discuss an example in the domain of facial recognition software in the domain of medicine to illustrate how this process of normative analysis might be conducted.
Cooperative Path-following Control of Remotely Operated Underwater Robots for Human Visual Inspection Task
Venue: arXiv cs.RO 2022
Year: 2022

Remotely operated vehicles (ROVs) have drawn much attention to underwater tasks, such as the inspection and maintenance of infrastructure. The workload of ROV operators tends to be high, even for the skilled ones. Therefore, assistance methods for the operators are desired. This study focuses on a task in which a human operator controls an underwater robot to follow a certain path while visually inspecting objects in the vicinity of the path. In such a task, it is desirable to realize the speed of trajectory control manually because the visual inspection is performed by a human operator. However, to allocate resources to visual inspection, it is desirable to minimize the workload on the path-following by assisting with the automatic control. Therefore, the objective of this study was to develop a cooperative path-following control method that achieves the above-mentioned task by expanding a robust path-following control law of nonholonomic wheeled vehicles. To simplify this problem, we considered a path-following and visual objects recognition task in a two-dimensional plane. We conducted an experiment with participants (n=16) who completed the task using the proposed method and manual control. The results showed that both the path-following errors and the workload of the participants were significantly smaller with the proposed method than with manual control. In addition, subjective responses demonstrated that operator attention tended to be allocated to objects recognition rather than robot operation tasks with the proposed method. These results indicate the effectiveness of the proposed cooperative path-following control method.
Answering Should Self-Publishing Video Game Developers Market Their Game on Twitter
Venue: arXiv cs.SI 2022
Year: 2022

In the marketing of video games made by self-publishing, independent developers, there is common advice to market a game using the social media platform Twitter. However, the advice that stems from industry sources is somewhat contradictory, and many self-publishing independent developers elect not to use Twitter at all. This presents an opportunity for researchers to investigate this tension and determine if using Twitter does have a causal influence on the successful release of a game by using relatively recent developments in causal data science techniques. In this sense, this paper highlights these causal inference developments while simultaneously informing self-publishing independent developers whether they should indeed market their games using Twitter. It was found that using Twitter results in an average increase of 85.4 reviews during release week, corresponding to a 314% positive difference. Using Twitter also doubles the chance of reaching a critical 10-review inflection point threshold on release week. These effects, however, are moderated by the characteristics of the game, expressed as 'tags', yet in no case is the effect of Twitter use reduced to 0. Based on these findings, it is advised that new, self-publishing video game developers do use Twitter to market their games.