OpenRaiser/Intern-Atlas huggingface dataset:
Time Gated Convolutional Neural Networks for Crop Classification
Venue: arXiv cs.CV 2022
Year: 2022

This paper presented a state-of-the-art framework, Time Gated Convolutional Neural Network (TGCNN) that takes advantage of temporal information and gating mechanisms for the crop classification problem. Besides, several vegetation indices were constructed to expand dimensions of input data to take advantage of spectral information. Both spatial (channel-wise) and temporal (step-wise) correlation are considered in TGCNN. Specifically, our preliminary analysis indicates that step-wise information is of greater importance in this data set. Lastly, the gating mechanism helps capture high-order relationship. Our TGCNN solution achieves $0.973$ F1 score, $0.977$ AUC ROC and $0.948$ IoU, respectively. In addition, it outperforms three other benchmarks in different local tasks (Kenya, Brazil and Togo). Overall, our experiments demonstrate that TGCNN is advantageous in this earth observation time series classification task.
Actively learning to learn causal relationships
Venue: arXiv cs.AI 2022
Year: 2022

How do people actively learn to learn? That is, how and when do people choose actions that facilitate long-term learning and choosing future actions that are more informative? We explore these questions in the domain of active causal learning. We propose a hierarchical Bayesian model that goes beyond past models by predicting that people pursue information not only about the causal relationship at hand but also about causal overhypotheses$\unicode{x2014}$abstract beliefs about causal relationships that span multiple situations and constrain how we learn the specifics in each situation. In two active "blicket detector" experiments with 14 between-subjects manipulations, our model was supported by both qualitative trends in participant behavior and an individual-differences-based model comparison. Our results suggest when there are abstract similarities across active causal learning problems, people readily learn and transfer overhypotheses about these similarities. Moreover, people exploit these overhypotheses to facilitate long-term active learning.
The Story of $1/e$: ALOHA-based and Reinforcement-Learning-based Random Access for Delay-Constrained Communications
Venue: arXiv cs.IT 2022
Year: 2022

Motivated by the proliferation of real-time applications in multimedia communication systems, tactile Internet, and cyber-physical systems, supporting delay-constrained traffic becomes critical for such systems. In delay-constrained traffic, each packet has a hard deadline; when it is not delivered before its deadline is up, it becomes useless and will be removed from the system. In this work, we focus on designing random access schemes for delay-constrained wireless communications. We first investigate three ALOHA-based schemes and prove that the system timely throughput of all three schemes under corresponding optimal transmission probabilities asymptotically converges to $1/e$, same as the well-known throughput limit for delay-unconstrained ALOHA systems. The fundamental reason why ALOHA-based schemes cannot achieve asymptotical system timely throughput beyond $1/e$ is that all active ALOHA stations access the channel with the same probability in any slot. To go beyond $1/e$, we propose a reinforcement-learning-based scheme for delay-constrained wireless communications, called RLRA-DC, under which different stations collaboratively attain different transmission probabilities by only interacting with the access point. Our numerical result shows that the system timely throughput of RLRA-DC can be as high as 0.8 for tens of stations and can still reach 0.6 even for thousands of stations, much larger than $1/e$.
The Makerere Radio Speech Corpus: A Luganda Radio Corpus for Automatic Speech Recognition
Venue: arXiv cs.CL 2022
Year: 2022

Building a usable radio monitoring automatic speech recognition (ASR) system is a challenging task for under-resourced languages and yet this is paramount in societies where radio is the main medium of public communication and discussions. Initial efforts by the United Nations in Uganda have proved how understanding the perceptions of rural people who are excluded from social media is important in national planning. However, these efforts are being challenged by the absence of transcribed speech datasets. In this paper, The Makerere Artificial Intelligence research lab releases a Luganda radio speech corpus of 155 hours. To our knowledge, this is the first publicly available radio dataset in sub-Saharan Africa. The paper describes the development of the voice corpus and presents baseline Luganda ASR performance results using Coqui STT toolkit, an open source speech recognition toolkit.
DisCoVQA: Temporal Distortion-Content Transformers for Video Quality Assessment
Venue: arXiv cs.CV 2022
Year: 2022

The temporal relationships between frames and their influences on video quality assessment (VQA) are still under-studied in existing works. These relationships lead to two important types of effects for video quality. Firstly, some temporal variations (such as shaking, flicker, and abrupt scene transitions) are causing temporal distortions and lead to extra quality degradations, while other variations (e.g. those related to meaningful happenings) do not. Secondly, the human visual system often has different attention to frames with different contents, resulting in their different importance to the overall video quality. Based on prominent time-series modeling ability of transformers, we propose a novel and effective transformer-based VQA method to tackle these two issues. To better differentiate temporal variations and thus capture the temporal distortions, we design a transformer-based Spatial-Temporal Distortion Extraction (STDE) module. To tackle with temporal quality attention, we propose the encoder-decoder-like temporal content transformer (TCT). We also introduce the temporal sampling on features to reduce the input length for the TCT, so as to improve the learning effectiveness and efficiency of this module. Consisting of the STDE and the TCT, the proposed Temporal Distortion-Content Transformers for Video Quality Assessment (DisCoVQA) reaches state-of-the-art performance on several VQA benchmarks without any extra pre-training datasets and up to 10% better generalization ability than existing methods. We also conduct extensive ablation experiments to prove the effectiveness of each part in our proposed model, and provide visualizations to prove that the proposed modules achieve our intention on modeling these temporal issues. We will publish our codes and pretrained weights later.
How to Assess Trustworthy AI in Practice
Venue: arXiv cs.CY 2022
Year: 2022

This report is a methodological reflection on Z-Inspection$^{\small{\circledR}}$. Z-Inspection$^{\small{\circledR}}$ is a holistic process used to evaluate the trustworthiness of AI-based technologies at different stages of the AI lifecycle. It focuses, in particular, on the identification and discussion of ethical issues and tensions through the elaboration of socio-technical scenarios. It uses the general European Union's High-Level Expert Group's (EU HLEG) guidelines for trustworthy AI. This report illustrates for both AI researchers and AI practitioners how the EU HLEG guidelines for trustworthy AI can be applied in practice. We share the lessons learned from conducting a series of independent assessments to evaluate the trustworthiness of AI systems in healthcare. We also share key recommendations and practical suggestions on how to ensure a rigorous trustworthy AI assessment throughout the life-cycle of an AI system.
NoteG: A Computational Notebook to Facilitate Rapid Game Prototyping
Venue: arXiv cs.SE 2022
Year: 2022

Game development-based approaches are increasingly used to design curricula that can engage students, as these can help them apply and practice learnt computer science concepts. However, it can become complex to develop a minimum working game or a prototype with the help of high-end game engines. Game prototyping is one of the most essential parts of the game design and development cycle as it allows developers to continuously test and improve their ideas. In recent years, computational notebooks have gained widespread popularity among developers. They can help run individual code snippets, visualize the output, consolidate the source code, and share live code easily. However, its use has not been explored in the field of game development and prototyping. In this paper, we propose NoteG, a computational notebook towards rapid game prototyping. We evaluated the tool with 18 novice game developers through a questionnaire-based user survey. A majority of the volunteers (66%) found it easy to use and were of the opinion that it saves time. A few of the participants successfully extended the existing framework to implement new game mechanics within their prototypes.
Critical Investigation of Failure Modes in Physics-informed Neural Networks
Venue: arXiv cs.LG 2022
Year: 2022

Several recent works in scientific machine learning have revived interest in the application of neural networks to partial differential equations (PDEs). A popular approach is to aggregate the residual form of the governing PDE and its boundary conditions as soft penalties into a composite objective/loss function for training neural networks, which is commonly referred to as physics-informed neural networks (PINNs). In the present study, we visualize the loss landscapes and distributions of learned parameters and explain the ways this particular formulation of the objective function may hinder or even prevent convergence when dealing with challenging target solutions. We construct a purely data-driven loss function composed of both the boundary loss and the domain loss. Using this data-driven loss function and, separately, a physics-informed loss function, we then train two neural network models with the same architecture. We show that incomparable scales between boundary and domain loss terms are the culprit behind the poor performance. Additionally, we assess the performance of both approaches on two elliptic problems with increasingly complex target solutions. Based on our analysis of their loss landscapes and learned parameter distributions, we observe that a physics-informed neural network with a composite objective function formulation produces highly non-convex loss surfaces that are difficult to optimize and are more prone to the problem of vanishing gradients.
Mitigating Data Heterogeneity in Federated Learning with Data Augmentation
Venue: arXiv cs.LG 2022
Year: 2022

Federated Learning (FL) is a prominent framework that enables training a centralized model while securing user privacy by fusing local, decentralized models. In this setting, one major obstacle is data heterogeneity, i.e., each client having non-identically and independently distributed (non-IID) data. This is analogous to the context of Domain Generalization (DG), where each client can be treated as a different domain. However, while many approaches in DG tackle data heterogeneity from the algorithmic perspective, recent evidence suggests that data augmentation can induce equal or greater performance. Motivated by this connection, we present federated versions of popular DG algorithms, and show that by applying appropriate data augmentation, we can mitigate data heterogeneity in the federated setting, and obtain higher accuracy on unseen clients. Equipped with data augmentation, we can achieve state-of-the-art performance using even the most basic Federated Averaging algorithm, with much sparser communication.
Building Blocks for Network-Accelerated Distributed File Systems
Venue: arXiv cs.NI 2022
Year: 2022

High-performance clusters and datacenters pose increasingly demanding requirements on storage systems. If these systems do not operate at scale, applications are doomed to become I/O bound and waste compute cycles. To accelerate the data path to remote storage nodes, remote direct memory access (RDMA) has been embraced by storage systems to let data flow from the network to storage targets, reducing overall latency and CPU utilization. Yet, this approach still involves CPUs on the data path to enforce storage policies such as authentication, replication, and erasure coding. We show how storage policies can be offloaded to fully programmable SmartNICs, without involving host CPUs. By using PsPIN, an open-hardware SmartNIC, we show latency improvements for writes (up to 2x), data replication (up to 2x), and erasure coding (up to 2x), when compared to respective CPU- and RDMA-based alternatives.
Event-Case Correlation for Process Mining using Probabilistic Optimization
Venue: arXiv cs.DB 2022
Year: 2022

Process mining supports the analysis of the actual behavior and performance of business processes using event logs. % such as, e.g., sales transactions recorded by an ERP system. An essential requirement is that every event in the log must be associated with a unique case identifier (e.g., the order ID of an order-to-cash process). In reality, however, this case identifier may not always be present, especially when logs are acquired from different systems or extracted from non-process-aware information systems. In such settings, the event log needs to be pre-processed by grouping events into cases -- an operation known as event correlation. Existing techniques for correlating events have worked with assumptions to make the problem tractable: some assume the generative processes to be acyclic, while others require heuristic information or user input. Moreover, %these techniques' primary assumption is that they abstract the log to activities and timestamps, and miss the opportunity to use data attributes. % In this paper, we lift these assumptions and propose a new technique called EC-SA-Data based on probabilistic optimization. The technique takes as inputs a sequence of timestamped events (the log without case IDs), a process model describing the underlying business process, and constraints over the event attributes. Our approach returns an event log in which every event is associated with a case identifier. The technique allows users to incorporate rules on process knowledge and data constraints flexibly. The approach minimizes the misalignment between the generated log and the input process model, maximizes the support of the given data constraints over the correlated log, and the variance between activity durations across cases. Our experiments with various real-life datasets show the advantages of our approach over the state of the art.
MPA: MultiPath++ Based Architecture for Motion Prediction
Venue: arXiv cs.CV 2022
Year: 2022

Autonomous driving technology is developing rapidly and nowadays first autonomous rides are being provided in city areas. This requires the highest standards for the safety and reliability of the technology. Motion prediction part of the general self-driving pipeline plays a crucial role in providing these qualities. In this work we present one of the solutions for Waymo Motion Prediction Challenge 2022 based on MultiPath++ ranked the 3rd as of May, 26 2022. Our source code is publicly available on GitHub.
Pyramid Region-based Slot Attention Network for Temporal Action Proposal Generation
Venue: arXiv cs.CV 2022
Year: 2022

It has been found that temporal action proposal generation, which aims to discover the temporal action instances within the range of the start and end frames in the untrimmed videos, can largely benefit from proper temporal and semantic context exploitation. The latest efforts were dedicated to considering the temporal context and similarity-based semantic contexts through self-attention modules. However, they still suffer from cluttered background information and limited contextual feature learning. In this paper, we propose a novel Pyramid Region-based Slot Attention (PRSlot) module to address these issues. Instead of using the similarity computation, our PRSlot module directly learns the local relations in an encoder-decoder manner and generates the representation of a local region enhanced based on the attention over input features called \textit{slot}. Specifically, upon the input snippet-level features, PRSlot module takes the target snippet as \textit{query}, its surrounding region as \textit{key} and then generates slot representations for each \textit{query-key} slot by aggregating the local snippet context with a parallel pyramid strategy. Based on PRSlot modules, we present a novel Pyramid Region-based Slot Attention Network termed PRSA-Net to learn a unified visual representation with rich temporal and semantic context for better proposal generation. Extensive experiments are conducted on two widely adopted THUMOS14 and ActivityNet-1.3 benchmarks. Our PRSA-Net outperforms other state-of-the-art methods. In particular, we improve the AR@100 from the previous best 50.67% to 56.12% for proposal generation and raise the mAP under 0.5 tIoU from 51.9\% to 58.7\% for action detection on THUMOS14. \textit{Code is available at} \url{https://github.com/handhand123/PRSA-Net}
Transformers Improve Breast Cancer Diagnosis from Unregistered Multi-View Mammograms
Venue: arXiv cs.CV 2022
Year: 2022

Deep convolutional neural networks (CNNs) have been widely used in various medical imaging tasks. However, due to the intrinsic locality of convolution operation, CNNs generally cannot model long-range dependencies well, which are important for accurately identifying or mapping corresponding breast lesion features computed from unregistered multiple mammograms. This motivates us to leverage the architecture of Multi-view Vision Transformers to capture long-range relationships of multiple mammograms from the same patient in one examination. For this purpose, we employ local Transformer blocks to separately learn patch relationships within four mammograms acquired from two-view (CC/MLO) of two-side (right/left) breasts. The outputs from different views and sides are concatenated and fed into global Transformer blocks, to jointly learn patch relationships between four images representing two different views of the left and right breasts. To evaluate the proposed model, we retrospectively assembled a dataset involving 949 sets of mammograms, which include 470 malignant cases and 479 normal or benign cases. We trained and evaluated the model using a five-fold cross-validation method. Without any arduous preprocessing steps (e.g., optimal window cropping, chest wall or pectoral muscle removal, two-view image registration, etc.), our four-image (two-view-two-side) Transformer-based model achieves case classification performance with an area under ROC curve (AUC = 0.818), which significantly outperforms AUC = 0.784 achieved by the state-of-the-art multi-view CNNs (p = 0.009). It also outperforms two one-view-two-side models that achieve AUC of 0.724 (CC view) and 0.769 (MLO view), respectively. The study demonstrates the potential of using Transformers to develop high-performing computer-aided diagnosis schemes that combine four mammograms.
Sensitivity of Average Precision to Bounding Box Perturbations
Venue: arXiv cs.CV 2022
Year: 2022

Object detection is a fundamental vision task. It has been highly researched in academia and has been widely adopted in industry. Average Precision (AP) is the standard score for evaluating object detectors. Our understanding of the subtleties of this score, however, is limited. Here, we quantify the sensitivity of AP to bounding box perturbations and show that AP is very sensitive to small translations. Only one pixel shift is enough to drop the mAP of a model by 8.4%. The mAP drop over small objects with only one pixel shift is 23.1%. The corresponding numbers when ground-truth (GT) boxes are used as predictions are 23% and 41.7%, respectively. These results explain why achieving higher mAP becomes increasingly harder as models get better. We also investigate the effect of box scaling on AP. Code and data is available at https://github.com/aliborji/AP_Box_Perturbation.
ProML: A Decentralised Platform for Provenance Management of Machine Learning Software Systems
Venue: arXiv cs.SE 2022
Year: 2022

Large-scale Machine Learning (ML) based Software Systems are increasingly developed by distributed teams situated in different trust domains. Insider threats can launch attacks from any domain to compromise ML assets (models and datasets). Therefore, practitioners require information about how and by whom ML assets were developed to assess their quality attributes such as security, safety, and fairness. Unfortunately, it is challenging for ML teams to access and reconstruct such historical information of ML assets (ML provenance) because it is generally fragmented across distributed ML teams and threatened by the same adversaries that attack ML assets. This paper proposes ProML, a decentralised platform that leverages blockchain and smart contracts to empower distributed ML teams to jointly manage a single source of truth about circulated ML assets' provenance without relying on a third party, which is vulnerable to insider threats and presents a single point of failure. We propose a novel architectural approach called Artefact-as-a-State-Machine to leverage blockchain transactions and smart contracts for managing ML provenance information and introduce a user-driven provenance capturing mechanism to integrate existing scripts and tools to ProML without compromising participants' control over their assets and toolchains. We evaluate the performance and overheads of ProML by benchmarking a proof-of-concept system on a global blockchain. Furthermore, we assessed ProML's security against a threat model of a distributed ML workflow.
Safe and Psychologically Pleasant Traffic Signal Control with Reinforcement Learning using Action Masking
Venue: arXiv cs.LG 2022
Year: 2022

Reinforcement learning (RL) for traffic signal control (TSC) has shown better performance in simulation for controlling the traffic flow of intersections than conventional approaches. However, due to several challenges, no RL-based TSC has been deployed in the field yet. One major challenge for real-world deployment is to ensure that all safety requirements are met at all times during operation. We present an approach to ensure safety in a real-world intersection by using an action space that is safe by design. The action space encompasses traffic phases, which represent the combination of non-conflicting signal colors of the intersection. Additionally, an action masking mechanism makes sure that only appropriate phase transitions are carried out. Another challenge for real-world deployment is to ensure a control behavior that avoids stress for road users. We demonstrate how to achieve this by incorporating domain knowledge through extending the action masking mechanism. We test and verify our approach in a realistic simulation scenario. By ensuring safety and psychologically pleasant control behavior, our approach drives development towards real-world deployment of RL for TSC.
Maximal automatic complexity and context-free languages
Venue: arXiv cs.FL 2022
Year: 2022

Let $A_N$ denote nondeterministic automatic complexity and \[ L_{k,c}=\{x\in [k]^* : A_N(x)> |x|/c\}. \] In particular, $L_{k,2}$ is the language of all $k$-ary words for which $A_N$ is maximal, while $L_{k,3}$ gives a rough dividing line between complex and simple. Let $\mathbf{CFL}$ denote the complexity class consisting of all context-free languages. While it is not known that $L_{2,2}$ is infinite, Kjos-Hanssen (2017) showed that $L_{3,2}$ is $\mathbf{CFL}$-immune but not $\mathbf{coCFL}$-immune. We complete the picture by showing that $L_{3,2}\not\in\mathbf{coCFL}$. Turning to Boolean circuit complexity, we show that $L_{2,3}$ is $\mathbf{SAC}^0$-immune and $\mathbf{SAC}^0$-coimmune. Here $\mathbf{SAC}^0$ denotes the complexity class consisting of all languages computed by (non-uniform) constant-depth circuits with semi-unbounded fanin. As for arithmetic circuits, we show that $\{x:A_N(x)>1\}\not\in\oplus\mathbf{SAC}^0$. In particular, $\mathbf{SAC}^0\not\subseteq\oplus \mathbf{SAC}^0$, which resolves an open implication from the Complexity Zoo.