OpenRaiser/Intern-Atlas huggingface dataset:
Federated Learning Based on Dynamic Regularization
Venue: arXiv cs.LG 2021
Year: 2021

We propose a novel federated learning method for distributively training neural network models, where the server orchestrates cooperation between a subset of randomly chosen devices in each round. We view Federated Learning problem primarily from a communication perspective and allow more device level computations to save transmission costs. We point out a fundamental dilemma, in that the minima of the local-device level empirical loss are inconsistent with those of the global empirical loss. Different from recent prior works, that either attempt inexact minimization or utilize devices for parallelizing gradient computation, we propose a dynamic regularizer for each device at each round, so that in the limit the global and device solutions are aligned. We demonstrate both through empirical results on real and synthetic data as well as analytical results that our scheme leads to efficient training, in both convex and non-convex settings, while being fully agnostic to device heterogeneity and robust to large number of devices, partial participation and unbalanced data.
Mimic: An adaptive algorithm for multivariate time series classification
Venue: arXiv cs.LG 2021
Year: 2021

Time series data are valuable but are often inscrutable. Gaining trust in time series classifiers for finance, healthcare, and other critical applications may rely on creating interpretable models. Researchers have previously been forced to decide between interpretable methods that lack predictive power and deep learning methods that lack transparency. In this paper, we propose a novel Mimic algorithm that retains the predictive accuracy of the strongest classifiers while introducing interpretability. Mimic mirrors the learning method of an existing multivariate time series classifier while simultaneously producing a visual representation that enhances user understanding of the learned model. Experiments on 26 time series datasets support Mimic's ability to imitate a variety of time series classifiers visually and accurately.
BlueFog: Make Decentralized Algorithms Practical for Optimization and Deep Learning
Venue: arXiv cs.DC 2021
Year: 2021

Decentralized algorithm is a form of computation that achieves a global goal through local dynamics that relies on low-cost communication between directly-connected agents. On large-scale optimization tasks involving distributed datasets, decentralized algorithms have shown strong, sometimes superior, performance over distributed algorithms with a central node. Recently, developing decentralized algorithms for deep learning has attracted great attention. They are considered as low-communication-overhead alternatives to those using a parameter server or the Ring-Allreduce protocol. However, the lack of an easy-to-use and efficient software package has kept most decentralized algorithms merely on paper. To fill the gap, we introduce BlueFog, a python library for straightforward, high-performance implementations of diverse decentralized algorithms. Based on a unified abstraction of various communication operations, BlueFog offers intuitive interfaces to implement a spectrum of decentralized algorithms, from those using a static, undirected graph for synchronous operations to those using dynamic and directed graphs for asynchronous operations. BlueFog also adopts several system-level acceleration techniques to further optimize the performance on the deep learning tasks. On mainstream DNN training tasks, BlueFog reaches a much higher throughput and achieves an overall $1.2\times \sim 1.8\times$ speedup over Horovod, a state-of-the-art distributed deep learning package based on Ring-Allreduce. BlueFog is open source at https://github.com/Bluefog-Lib/bluefog.
Learning Context-Aware Representations of Subtrees
Venue: arXiv cs.LG 2021
Year: 2021

This thesis tackles the problem of learning efficient representations of complex, structured data with a natural application to web page and element classification. We hypothesise that the context around the element inside the web page is of high value to the problem and is currently under exploited. This thesis aims to solve the problem of classifying web elements as subtrees of a DOM tree by also considering their context. To achieve this, first we discuss current expert knowledge systems that work on structures, such as Tree-LSTM. Then, we propose context-aware extensions to this model. We show that the new model achieves an average F1-score of 0.7973 on a multi-class web classification task. This model generates better representations for various subtrees and may be used for applications such element classification, state estimators in reinforcement learning over the Web and more.
A Relational Model for One-Shot Classification
Venue: arXiv cs.LG 2021
Year: 2021

We show that a deep learning model with built-in relational inductive bias can bring benefits to sample-efficient learning, without relying on extensive data augmentation. The proposed one-shot classification model performs relational matching of a pair of inputs in the form of local and pairwise attention. Our approach solves perfectly the one-shot image classification Omniglot challenge. Our model exceeds human level accuracy, as well as the previous state of the art, with no data augmentation.
Towards Debiasing Temporal Sentence Grounding in Video
Venue: arXiv cs.CV 2021
Year: 2021

The temporal sentence grounding in video (TSGV) task is to locate a temporal moment from an untrimmed video, to match a language query, i.e., a sentence. Without considering bias in moment annotations (e.g., start and end positions in a video), many models tend to capture statistical regularities of the moment annotations, and do not well learn cross-modal reasoning between video and language query. In this paper, we propose two debiasing strategies, data debiasing and model debiasing, to "force" a TSGV model to capture cross-modal interactions. Data debiasing performs data oversampling through video truncation to balance moment temporal distribution in train set. Model debiasing leverages video-only and query-only models to capture the distribution bias, and forces the model to learn cross-modal interactions. Using VSLNet as the base model, we evaluate impact of the two strategies on two datasets that contain out-of-distribution test instances. Results show that both strategies are effective in improving model generalization capability. Equipped with both debiasing strategies, VSLNet achieves best results on both datasets.
Development of a Meta-language and its Qualifiable Implementation for the Use in Safety-critical Software
Venue: arXiv cs.SE 2021
Year: 2021

The use of domain-specific modeling for development of complex (cyber-physical) systems is gaining increasing acceptance in the industrial environment. Domain-specific modeling allows complex systems and data to be abstracted for a more efficient system design, development, validation, and configuration. However, no existing (meta-)modeling framework can be used with reasonable effort in certified software so far, neither for the development of systems nor for the execution of system functions. For the use of (development) artifacts from domain-specific modeling in safety-critical processes or systems it is required to ensure their correctness by either subsequent (manual) verification or the usage of (pre-)qualified software. Existing meta-languages often contain modeling elements that are difficult or impossible to implement in a qualifiable manner leading to a high manual, subsequent certification effort. Therefore, the aim is to develop a (meta-)modeling framework, that can be used in certified software. This can significantly reduce the development effort for safety-critical systems and enables the full advantages of domain-specific modeling. The framework components considered in this PhD-Thesis include: (1) an essential meta-language, (2) a qualifiable runtime environment, and (3) a suitable persistence. The essential \mbox{(meta-)}modeling language is mainly based on the UML standard, but is enhanced with multi-level modeling concepts such as deep instantiation. Supporting a possible qualification, the meta-language is implemented using the highly restrictive, but formally provable programming language Ada SPARK.
Ethics-Based Auditing of Automated Decision-Making Systems: Intervention Points and Policy Implications
Venue: arXiv cs.CY 2021
Year: 2021

Organisations increasingly use automated decision-making systems (ADMS) to inform decisions that affect humans and their environment. While the use of ADMS can improve the accuracy and efficiency of decision-making processes, it is also coupled with ethical challenges. Unfortunately, the governance mechanisms currently used to oversee human decision-making often fail when applied to ADMS. In previous work, we proposed that ethics-based auditing (EBA), i.e. a structured process by which ADMS are assessed for consistency with relevant principles or norms, can (a) help organisations verify claims about their ADMS and (b) provide decision-subjects with justifications for the outputs produced by ADMS. In this article, we outline the conditions under which EBA procedures can be feasible and effective in practice. First, we argue that EBA is best understood as a 'soft' yet 'formal' governance mechanism. This implies that the main responsibility of auditors should be to spark ethical deliberation at key intervention points throughout the software development process and ensure that there is sufficient documentation to respond to potential inquiries. Second, we frame ADMS as parts of larger socio-technical systems to demonstrate that to be feasible and effective, EBA procedures must link to intervention points that span all levels of organisational governance and all phases of the software lifecycle. The main function of EBA should therefore be to inform, formalise, assess, and interlink existing governance structures. Finally, we discuss the policy implications of our findings. To support the emergence of feasible and effective EBA procedures, policymakers and regulators could provide standardised reporting formats, facilitate knowledge exchange, provide guidance on how to resolve normative tensions, and create an independent body to oversee EBA of ADMS.
Content-aware media retargeting based on deep importance map
Venue: arXiv cs.GR 2021
Year: 2021

We present a neural network to estimate the visual information of important pixels in image and video, which is used in content-aware media retargeting applications. Existing techniques are successful in proposing retargeting methods. Yet, the serious distortion and the shrunk problem in the retargeted results still need to be investigated due to the limitations in the methods used to analyze visual attention. To accomplish this, we propose a network to define the importance map, which is sufficient to describe the energy of the significant regions in image/video. With this strategy, more ideal results are obtained from our system. Besides, the objective evaluation presented in this paper shows that our media retargeting system can achieve better and more plausible results than those of other works. In addition, our proposed importance map performs well in the enlarge operator and on the "difficult-to-resize" images.
Gradient-Descent for Randomized Controllers under Partial Observability
Venue: arXiv cs.LO 2021
Year: 2021

Randomization is a powerful technique to create robust controllers, in particular in partially observable settings. The degrees of randomization have a significant impact on the system performance, yet they are intricate to get right. The use of synthesis algorithms for parametric Markov chains (pMCs) is a promising direction to support the design process of such controllers. This paper shows how to define and evaluate gradients of pMCs. Furthermore, it investigates varieties of gradient descent techniques from the machine learning community to synthesize the probabilities in a pMC. The resulting method scales to significantly larger pMCs than before and empirically outperforms the state-of-the-art, often by at least one order of magnitude.
Topic Modeling, Clade-assisted Sentiment Analysis, and Vaccine Brand Reputation Analysis of COVID-19 Vaccine-related Facebook Comments in the Philippines
Venue: arXiv cs.CY 2021
Year: 2021

Vaccine hesitancy and other COVID-19-related concerns and complaints in the Philippines are evident on social media. It is important to identify these different topics and sentiments in order to gauge public opinion, use the insights to develop policies, and make necessary adjustments or actions to improve public image and reputation of the administering agency and the COVID-19 vaccines themselves. This paper proposes a semi-supervised machine learning pipeline to perform topic modeling, sentiment analysis, and an analysis of vaccine brand reputation to obtain an in-depth understanding of national public opinion of Filipinos on Facebook. The methodology makes use of a multilingual version of Bidirectional Encoder Representations from Transformers or BERT for topic modeling, hierarchical clustering, five different classifiers for sentiment analysis, and cosine similarity of BERT topic embeddings for vaccine brand reputation analysis. Results suggest that any type of COVID-19 misinformation is an emergent property of COVID-19 public opinion, and that the detection of COVID-19 misinformation can be an unsupervised task. Sentiment analysis aided by hierarchical clustering reveal that 21 of the 25 topics extrapolated by topic modeling are negative topics. Such negative comments spike in count whenever the Department of Health in the Philippines posts about the COVID-19 situation in other countries. Additionally, the high numbers of laugh reactions on the Facebook posts by the same agency -- without any humorous content -- suggest that the reactors of these posts tend to react the way they do, not because of what the posts are about but because of who posted them.
Experience of Teaching Data Visualization using Project-based Learning
Venue: arXiv cs.CY 2021
Year: 2021

We report our experience in two installations of a course on data visualization that featured project-based learning. Given the rationale of this approach, we show which input was provided when necessary for the students to achieve their goals. We also discuss and compare the tools we found useful for students to accomplish their goals. Fitting project-based learning into the standard schedule of a semester at University is a particular challenge, because it is hard for students to devote longer periods of time when working on their projects. Furthermore, online learning was a challenge to effectively perform group activities and work. We address didactic considerations, our course structure, tools, and student projects. Finally, we draw conclusions on the results and improvements in the course structure.
Flight Demand Forecasting with Transformers
Venue: arXiv cs.LG 2021
Year: 2021

Transformers have become the de-facto standard in the natural language processing (NLP) field. They have also gained momentum in computer vision and other domains. Transformers can enable artificial intelligence (AI) models to dynamically focus on certain parts of their input and thus reason more effectively. Inspired by the success of transformers, we adopted this technique to predict strategic flight departure demand in multiple horizons. This work was conducted in support of a MITRE-developed mobile application, Pacer, which displays predicted departure demand to general aviation (GA) flight operators so they can have better situation awareness of the potential for departure delays during busy periods. Field demonstrations involving Pacer's previously designed rule-based prediction method showed that the prediction accuracy of departure demand still has room for improvement. This research strives to improve prediction accuracy from two key aspects: better data sources and robust forecasting algorithms. We leveraged two data sources, Aviation System Performance Metrics (ASPM) and System Wide Information Management (SWIM), as our input. We then trained forecasting models with temporal fusion transformer (TFT) for five different airports. Case studies show that TFTs can perform better than traditional forecasting methods by large margins, and they can result in better prediction across diverse airports and with better interpretability.
Self-Supervised Intrinsic Image Decomposition Network Considering Reflectance Consistency
Venue: arXiv cs.CV 2021
Year: 2021

We propose a novel intrinsic image decomposition network considering reflectance consistency. Intrinsic image decomposition aims to decompose an image into illumination-invariant and illumination-variant components, referred to as ``reflectance'' and ``shading,'' respectively. Although there are three consistencies that the reflectance and shading should satisfy, most conventional work does not sufficiently account for consistency with respect to reflectance, owing to the use of a white-illuminant decomposition model and the lack of training images capturing the same objects under various illumination-brightness and -color conditions. For this reason, the three consistencies are considered in the proposed network by using a color-illuminant model and training the network with losses calculated from images taken under various illumination conditions. In addition, the proposed network can be trained in a self-supervised manner because various illumination conditions can easily be simulated. Experimental results show that our network can decompose images into reflectance and shading components.
Ontology-based question answering over corporate structured data
Venue: arXiv cs.SE 2021
Year: 2021

Ontology-based approach to the Natural Language Understanding (NLU) processing allows to improve questions answering quality in dialogue systems. We describe our NLU engine architecture and evaluate its implementation. The engine transforms user input into the SPARQL SELECT, ASK or INSERT query to the knowledge graph provided by the ontology-based data virtualization platform. The transformation is based on the lexical level of the knowledge graph built according to the Ontolex ontology. The described approach can be applied for graph data population tasks and to the question answering systems implementation, including chat bots. We describe the dialogue engine for a chat bot which can keep the conversation context and ask clarifying questions, simulating some aspects of the human logical thinking. Our approach uses graph-based algorithms to avoid gathering datasets, required in the neural nets-based approaches, and provide better explainability of our models. Using question answering engine in conjunction with data virtualization layer over the corporate data sources allows extracting facts from the structured data to be used in conversation.
Sexism Prediction in Spanish and English Tweets Using Monolingual and Multilingual BERT and Ensemble Models
Venue: arXiv cs.CL 2021
Year: 2021

The popularity of social media has created problems such as hate speech and sexism. The identification and classification of sexism in social media are very relevant tasks, as they would allow building a healthier social environment. Nevertheless, these tasks are considerably challenging. This work proposes a system to use multilingual and monolingual BERT and data points translation and ensemble strategies for sexism identification and classification in English and Spanish. It was conducted in the context of the sEXism Identification in Social neTworks shared 2021 (EXIST 2021) task, proposed by the Iberian Languages Evaluation Forum (IberLEF). The proposed system and its main components are described, and an in-depth hyperparameters analysis is conducted. The main results observed were: (i) the system obtained better results than the baseline model (multilingual BERT); (ii) ensemble models obtained better results than monolingual models; and (iii) an ensemble model considering all individual models and the best standardized values obtained the best accuracies and F1-scores for both tasks. This work obtained first place in both tasks at EXIST, with the highest accuracies (0.780 for task 1 and 0.658 for task 2) and F1-scores (F1-binary of 0.780 for task 1 and F1-macro of 0.579 for task 2).
DeepSteal: Advanced Model Extractions Leveraging Efficient Weight Stealing in Memories
Venue: arXiv cs.CR 2021
Year: 2021

Recent advancements of Deep Neural Networks (DNNs) have seen widespread deployment in multiple security-sensitive domains. The need of resource-intensive training and use of valuable domain-specific training data have made these models a top intellectual property (IP) for model owners. One of the major threats to the DNN privacy is model extraction attacks where adversaries attempt to steal sensitive information in DNN models. Recent studies show hardware-based side channel attacks can reveal internal knowledge about DNN models (e.g., model architectures) However, to date, existing attacks cannot extract detailed model parameters (e.g., weights/biases). In this work, for the first time, we propose an advanced model extraction attack framework DeepSteal that effectively steals DNN weights with the aid of memory side-channel attack. Our proposed DeepSteal comprises two key stages. Firstly, we develop a new weight bit information extraction method, called HammerLeak, through adopting the rowhammer based hardware fault technique as the information leakage vector. HammerLeak leverages several novel system-level techniques tailed for DNN applications to enable fast and efficient weight stealing. Secondly, we propose a novel substitute model training algorithm with Mean Clustering weight penalty, which leverages the partial leaked bit information effectively and generates a substitute prototype of the target victim model. We evaluate this substitute model extraction method on three popular image datasets (e.g., CIFAR-10/100/GTSRB) and four DNN architectures (e.g., ResNet-18/34/Wide-ResNet/VGG-11). The extracted substitute model has successfully achieved more than 90 % test accuracy on deep residual networks for the CIFAR-10 dataset. Moreover, our extracted substitute model could also generate effective adversarial input samples to fool the victim model.
Higher Order $c$-Differentials
Venue: arXiv cs.IT 2021
Year: 2021

EFRST20, the notion of $c$-differentials was introduced as a potential expansion of differential cryptanalysis against block ciphers utilizing substitution boxes. Drawing inspiration from the technique of higher order differential cryptanalysis, in this paper we propose the notion of higher order $c$-derivatives and differentials and investigate their properties. Additionally, we consider how several classes of functions, namely the multiplicative inverse function and the Gold function, perform under higher order $c$-differential uniformity.