OpenRaiser/Intern-Atlas huggingface dataset:
Are Vision Transformers Robust to Spurious Correlations?
Venue: arXiv cs.CV 2022
Year: 2022

Deep neural networks may be susceptible to learning spurious correlations that hold on average but not in atypical test samples. As with the recent emergence of vision transformer (ViT) models, it remains underexplored how spurious correlations are manifested in such architectures. In this paper, we systematically investigate the robustness of vision transformers to spurious correlations on three challenging benchmark datasets and compare their performance with popular CNNs. Our study reveals that when pre-trained on a sufficiently large dataset, ViT models are more robust to spurious correlations than CNNs. Key to their success is the ability to generalize better from the examples where spurious correlations do not hold. Further, we perform extensive ablations and experiments to understand the role of the self-attention mechanism in providing robustness under spuriously correlated environments. We hope that our work will inspire future research on further understanding the robustness of ViT models.
Conversational Recommendation: A Grand AI Challenge
Venue: arXiv cs.AI 2022
Year: 2022

Animated avatars, which look and talk like humans, are iconic visions of the future of AI-powered systems. Through many sci-fi movies we are acquainted with the idea of speaking to such virtual personalities as if they were humans. Today, we talk more and more to machines like Apple's Siri, e.g., to ask them for the weather forecast. However, when asked for recommendations, e.g., for a restaurant to go to, the limitations of such devices quickly become obvious. They do not engage in a conversation to find out what we might prefer, they often do not provide explanations for what they recommend, and they may have difficulties remembering what was said one minute earlier. Conversational recommender systems promise to address these limitations. In this paper, we review existing approaches to build such systems, which developments we observe today, which challenges are still open and why the development of conversational recommenders represents one of the next grand challenges of AI.
Prediction of speech intelligibility with DNN-based performance measures
Venue: arXiv cs.SD 2022
Year: 2022

This paper presents a speech intelligibility model based on automatic speech recognition (ASR), combining phoneme probabilities from deep neural networks (DNN) and a performance measure that estimates the word error rate from these probabilities. This model does not require the clean speech reference nor the word labels during testing as the ASR decoding step, which finds the most likely sequence of words given phoneme posterior probabilities, is omitted. The model is evaluated via the root-mean-squared error between the predicted and observed speech reception thresholds from eight normal-hearing listeners. The recognition task consists of identifying noisy words from a German matrix sentence test. The speech material was mixed with eight noise maskers covering different modulation types, from speech-shaped stationary noise to a single-talker masker. The prediction performance is compared to five established models and an ASR-model using word labels. Two combinations of features and networks were tested. Both include temporal information either at the feature level (amplitude modulation filterbanks and a feed-forward network) or captured by the architecture (mel-spectrograms and a time-delay deep neural network, TDNN). The TDNN model is on par with the DNN while reducing the number of parameters by a factor of 37; this optimization allows parallel streams on dedicated hearing aid hardware as a forward-pass can be computed within the 10ms of each frame. The proposed model performs almost as well as the label-based model and produces more accurate predictions than the baseline models.
Biasing Like Human: A Cognitive Bias Framework for Scene Graph Generation
Venue: arXiv cs.CV 2022
Year: 2022

Scene graph generation is a sophisticated task because there is no specific recognition pattern (e.g., "looking at" and "near" have no conspicuous difference concerning vision, whereas "near" could occur between entities with different morphology). Thus some scene graph generation methods are trapped into most frequent relation predictions caused by capricious visual features and trivial dataset annotations. Therefore, recent works emphasized the "unbiased" approaches to balance predictions for a more informative scene graph. However, human's quick and accurate judgments over relations between numerous objects should be attributed to "bias" (i.e., experience and linguistic knowledge) rather than pure vision. To enhance the model capability, inspired by the "cognitive bias" mechanism, we propose a novel 3-paradigms framework that simulates how humans incorporate the label linguistic features as guidance of vision-based representations to better mine hidden relation patterns and alleviate noisy visual propagation. Our framework is model-agnostic to any scene graph model. Comprehensive experiments prove our framework outperforms baseline modules in several metrics with minimum parameters increment and achieves new SOTA performance on Visual Genome dataset.
Generalized Classification of Satellite Image Time Series with Thermal Positional Encoding
Venue: arXiv cs.CV 2022
Year: 2022

Large-scale crop type classification is a task at the core of remote sensing efforts with applications of both economic and ecological importance. Current state-of-the-art deep learning methods are based on self-attention and use satellite image time series (SITS) to discriminate crop types based on their unique growth patterns. However, existing methods generalize poorly to regions not seen during training mainly due to not being robust to temporal shifts of the growing season caused by variations in climate. To this end, we propose Thermal Positional Encoding (TPE) for attention-based crop classifiers. Unlike previous positional encoding based on calendar time (e.g. day-of-year), TPE is based on thermal time, which is obtained by accumulating daily average temperatures over the growing season. Since crop growth is directly related to thermal time, but not calendar time, TPE addresses the temporal shifts between different regions to improve generalization. We propose multiple TPE strategies, including learnable methods, to further improve results compared to the common fixed positional encodings. We demonstrate our approach on a crop classification task across four different European regions, where we obtain state-of-the-art generalization results.
Multilingual Detection of Personal Employment Status on Twitter
Venue: arXiv cs.CL 2022
Year: 2022

Detecting disclosures of individuals' employment status on social media can provide valuable information to match job seekers with suitable vacancies, offer social protection, or measure labor market flows. However, identifying such personal disclosures is a challenging task due to their rarity in a sea of social media content and the variety of linguistic forms used to describe them. Here, we examine three Active Learning (AL) strategies in real-world settings of extreme class imbalance, and identify five types of disclosures about individuals' employment status (e.g. job loss) in three languages using BERT-based classification models. Our findings show that, even under extreme imbalance settings, a small number of AL iterations is sufficient to obtain large and significant gains in precision, recall, and diversity of results compared to a supervised baseline with the same number of labels. We also find that no AL strategy consistently outperforms the rest. Qualitative analysis suggests that AL helps focus the attention mechanism of BERT on core terms and adjust the boundaries of semantic expansion, highlighting the importance of interpretable models to provide greater control and visibility into this dynamic learning process.
Neighbourhood semantics and axioms for strategic fragment of classical stit logic
Venue: arXiv cs.LO 2022
Year: 2022

STIT (sees to it that) semantics is one of the most prominent tools in modal logic of agency, widely used among both philosophers and responsible AI scholars. STIT logic surveys the properties of agents seeing to it that some state of affairs holds without specifying concrete actions by which that state of affairs is guaranteed. In comparison with other multi-agent modal logics, the main advantage of STIT theories is expressive power. STIT logic allows to study not only statements about agents abilities to perform certain actions (as it is in variations of Coalition Logic or Propositional Dynamic Logic), but about what choices they make and what they de-facto achieve as well. Nevertheless, in some occasions such expressivity may be redundant. This paper surveys a specific fragment of classical STIT logic, which has only strategic modal operator, standing for the fact that agent has an ability to see to it that some state of affairs holds. The neighbourhood semantics for the fragment is presented, accompanied with the soundness, canoniciy hence strong completeness results. Furthermore, the paper presents basic considerations on epistemic extension of the presented fragment.
Re-shaping Post-COVID-19 Teaching and Learning: A Blueprint of Virtual-Physical Blended Classrooms in the Metaverse Era
Venue: arXiv cs.HC 2022
Year: 2022

During the COVID-19 pandemic, most countries have experienced some form of remote education through video conferencing software platforms. However, these software platforms fail to reduce immersion and replicate the classroom experience. The currently emerging Metaverse addresses many of such limitations by offering blended physical-digital environments. This paper aims to assess how the Metaverse can support and improve e-learning. We first survey the latest applications of blended environments in education and highlight the primary challenges and opportunities. Accordingly, we derive our proposal for a virtual-physical blended classroom configuration that brings students and teachers into a shared educational Metaverse. We focus on the system architecture of the Metaverse classroom to achieve real-time synchronization of a large number of participants and activities across physical (mixed reality classrooms) and virtual (remote VR platform) learning spaces. Our proposal attempts to transform the traditional physical classroom into virtual-physical cyberspace as a new social network of learners and educators connected at an unprecedented scale.
A flexible solution to embrace Ranking and Skyline queries approaches
Venue: arXiv cs.DB 2022
Year: 2022

The multi-objective optimization problem has always been the main objective of the principal traditional approaches, such as Ranking queries and Skyline queries. The conventional idea was to either use one or the other, trying to exploit both ranking queries advantages when it comes to taking into account user preferences, and skyline queries points of strength when the main objective was to obtain interesting results from a dataset in a simple, yet effective fashion, both of them showing limitations when entering specific fields of interest.
Stochastic and Private Nonconvex Outlier-Robust PCA
Venue: arXiv cs.LG 2022
Year: 2022

We develop theoretically guaranteed stochastic methods for outlier-robust PCA. Outlier-robust PCA seeks an underlying low-dimensional linear subspace from a dataset that is corrupted with outliers. We are able to show that our methods, which involve stochastic geodesic gradient descent over the Grassmannian manifold, converge and recover an underlying subspace in various regimes through the development of a novel convergence analysis. The main application of this method is an effective differentially private algorithm for outlier-robust PCA that uses a Gaussian noise mechanism within the stochastic gradient method. Our results emphasize the advantages of the nonconvex methods over another convex approach to solving this problem in the differentially private setting. Experiments on synthetic and stylized data verify these results.
MSPred: Video Prediction at Multiple Spatio-Temporal Scales with Hierarchical Recurrent Networks
Venue: arXiv cs.CV 2022
Year: 2022

Autonomous systems not only need to understand their current environment, but should also be able to predict future actions conditioned on past states, for instance based on captured camera frames. However, existing models mainly focus on forecasting future video frames for short time-horizons, hence being of limited use for long-term action planning. We propose Multi-Scale Hierarchical Prediction (MSPred), a novel video prediction model able to simultaneously forecast future possible outcomes of different levels of granularity at different spatio-temporal scales. By combining spatial and temporal downsampling, MSPred efficiently predicts abstract representations such as human poses or locations over long time horizons, while still maintaining a competitive performance for video frame prediction. In our experiments, we demonstrate that MSPred accurately predicts future video frames as well as high-level representations (e.g. keypoints or semantics) on bin-picking and action recognition datasets, while consistently outperforming popular approaches for future frame prediction. Furthermore, we ablate different modules and design choices in MSPred, experimentally validating that combining features of different spatial and temporal granularity leads to a superior performance. Code and models to reproduce our experiments can be found in https://github.com/AIS-Bonn/MSPred.
Machine Learning for Encrypted Malicious Traffic Detection: Approaches, Datasets and Comparative Study
Venue: arXiv cs.CR 2022
Year: 2022

As people's demand for personal privacy and data security becomes a priority, encrypted traffic has become mainstream in the cyber world. However, traffic encryption is also shielding malicious and illegal traffic introduced by adversaries, from being detected. This is especially so in the post-COVID-19 environment where malicious traffic encryption is growing rapidly. Common security solutions that rely on plain payload content analysis such as deep packet inspection are rendered useless. Thus, machine learning based approaches have become an important direction for encrypted malicious traffic detection. In this paper, we formulate a universal framework of machine learning based encrypted malicious traffic detection techniques and provided a systematic review. Furthermore, current research adopts different datasets to train their models due to the lack of well-recognized datasets and feature sets. As a result, their model performance cannot be compared and analyzed reliably. Therefore, in this paper, we analyse, process and combine datasets from 5 different sources to generate a comprehensive and fair dataset to aid future research in this field. On this basis, we also implement and compare 10 encrypted malicious traffic detection algorithms. We then discuss challenges and propose future directions of research.
Context-Dependent Anomaly Detection with Knowledge Graph Embedding Models
Venue: arXiv cs.LG 2022
Year: 2022

Increasing the semantic understanding and contextual awareness of machine learning models is important for improving robustness and reducing susceptibility to data shifts. In this work, we leverage contextual awareness for the anomaly detection problem. Although graphed-based anomaly detection has been widely studied, context-dependent anomaly detection is an open problem and without much current research. We develop a general framework for converting a context-dependent anomaly detection problem to a link prediction problem, allowing well-established techniques from this domain to be applied. We implement a system based on our framework that utilizes knowledge graph embedding models and demonstrates the ability to detect outliers using context provided by a semantic knowledge base. We show that our method can detect context-dependent anomalies with a high degree of accuracy and show that current object detectors can detect enough classes to provide the needed context for good performance within our example domain.
Expressivity of Planning with Horn Description Logic Ontologies (Technical Report)
Venue: arXiv cs.AI 2022
Year: 2022

State constraints in AI Planning globally restrict the legal environment states. Standard planning languages make closed-domain and closed-world assumptions. Here we address open-world state constraints formalized by planning over a description logic (DL) ontology. Previously, this combination of DL and planning has been investigated for the light-weight DL DL-Lite. Here we propose a novel compilation scheme into standard PDDL with derived predicates, which applies to more expressive DLs and is based on the rewritability of DL queries into Datalog with stratified negation. We also provide a new rewritability result for the DL Horn-ALCHOIQ, which allows us to apply our compilation scheme to quite expressive ontologies. In contrast, we show that in the slight extension Horn-SROIQ no such compilation is possible unless the weak exponential hierarchy collapses. Finally, we show that our approach can outperform previous work on existing benchmarks for planning with DL ontologies, and is feasible on new benchmarks taking advantage of more expressive ontologies. That is an extended version of a paper accepted at AAAI 22.
DCarbonX Decentralised Application: Carbon Market Case Study
Venue: arXiv cs.CE 2022
Year: 2022

Decentralized applications developed using blockchain technology provide innovative business models to serve the human race and solve existing challenges. Climate change is one of the biggest problems humanity is facing and there is a dearth of solutions in tackling this grave impediment to the long-term sustainability of our planet. Accountability, greenwashing, traceability, impact assessment and trading of carbon credits are unresolved issues in the ESG sector. In this paper, we present a novel decentralized application software, DCarbonX, that solves the enumerated problems using NFTs on the blockchain platform, through smart contracts. The paper describes the functional architecture of DCarbonX, while elaborating on its salient features and utility in sustainable finance, in particular green sukuk. DCarbonX is a pioneering software providing an exchange for trading of carbon credits. The software facilitates logging of impact and traceable transactions in a carbon market, that would help to prevent duplication of records and greenwashing. The paper discusses the efforts being undertaken to achieve the climate goals as per the Paris Agreement and also highlights the pivotal obstacles to achieving carbon neutrality by 2050, as per COP26. The paper also encompasses a study on the applications of dapps in DeFi, Web 3.0 and ESG, among other areas and gives a comparative analysis of blockchain platforms for dapp development. The paper is also a pioneer in highlighting the challenges that plague dapp development, deployment and usage.
Surface Defect Detection and Evaluation for Marine Vessels using Multi-Stage Deep Learning
Venue: arXiv cs.CV 2022
Year: 2022

Detecting and evaluating surface coating defects is important for marine vessel maintenance. Currently, the assessment is carried out manually by qualified inspectors using international standards and their own experience. Automating the processes is highly challenging because of the high level of variation in vessel type, paint surface, coatings, lighting condition, weather condition, paint colors, areas of the vessel, and time in service. We present a novel deep learning-based pipeline to detect and evaluate the percentage of corrosion, fouling, and delamination on the vessel surface from normal photographs. We propose a multi-stage image processing framework, including ship section segmentation, defect segmentation, and defect classification, to automatically recognize different types of defects and measure the coverage percentage on the ship surface. Experimental results demonstrate that our proposed pipeline can objectively perform a similar assessment as a qualified inspector.
Prioritized Variable-length Test Cases Generation for Finite State Machines
Venue: arXiv cs.SE 2022
Year: 2022

Model-based Testing (MBT) is an effective approach for testing when parts of a system-under-test have the characteristics of a finite state machine (FSM). Despite various strategies in the literature on this topic, little work exists to handle special testing situations. More specifically, when concurrently: (1) the test paths can start and end only in defined states of the FSM, (2) a prioritization mechanism that requires only defined states and transitions of the FSM to be visited by test cases is required, and (3) the test paths must be in a given length range, not necessarily of explicit uniform length. This paper presents a test generation strategy that satisfies all these requirements. A concurrent combination of these requirements is highly practical for real industrial testing. Six variants of possible algorithms to implement this strategy are described. Using a mixture of 180 problem instances from real automotive and defense projects and artificially generated FSMs, all variants are compared with a baseline strategy based on an established N-switch coverage concept modification. Various properties of the generated test paths and their potential to activate fictional defects defined in FSMs are evaluated. The presented strategy outperforms the baseline in most problem configurations. Out of the six analyzed variants, three give the best results even though a universal best performer is hard to identify. Depending on the application of the FSM, the strategy and evaluation presented in this paper are applicable both in testing functional and non-functional software requirements.
The Analysis of Online Event Streams: Predicting the Next Activity for Anomaly Detection
Venue: arXiv cs.LG 2022
Year: 2022

Anomaly detection in process mining focuses on identifying anomalous cases or events in process executions. The resulting diagnostics are used to provide measures to prevent fraudulent behavior, as well as to derive recommendations for improving process compliance and security. Most existing techniques focus on detecting anomalous cases in an offline setting. However, to identify potential anomalies in a timely manner and take immediate countermeasures, it is necessary to detect event-level anomalies online, in real-time. In this paper, we propose to tackle the online event anomaly detection problem using next-activity prediction methods. More specifically, we investigate the use of both ML models (such as RF and XGBoost) and deep models (such as LSTM) to predict the probabilities of next-activities and consider the events predicted unlikely as anomalies. We compare these predictive anomaly detection methods to four classical unsupervised anomaly detection approaches (such as Isolation forest and LOF) in the online setting. Our evaluation shows that the proposed method using ML models tends to outperform the one using a deep model, while both methods outperform the classical unsupervised approaches in detecting anomalous events.