Reviewerly/exHarmony huggingface dataset:
SQuINTing at VQA Models: Introspecting VQA Models With Sub-Questions

Existing VQA datasets contain questions with varying levels of complexity. While the majority of questions in these datasets require perception for recognizing existence, properties, and spatial relationships of entities, a significant portion of questions pose challenges that correspond to reasoning tasks - tasks that can only be answered through a synthesis of perception and knowledge about the world, logic and / or reasoning. Analyzing performance across this distinction allows us to notice when existing VQA models have consistency issues - they answer the reasoning questions correctly but fail on associated low-level perception questions. For example, in Figure 1, models answer the complex reasoning question “Is the banana ripe enough to eat?” correctly, but fail on the associated perception question “Are the bananas mostly green or yellow?” indicating that the model likely answered the reasoning question correctly but for the wrong reason. We quantify the extent to which this phenomenon occurs by creating a new Reasoning split of the VQA dataset and collecting VQAintrospect, a new dataset1 which currently consists of 200K new perception questions which serve as sub questions corresponding to the set of perceptual tasks needed to effectively answer the complex reasoning questions in the Reasoning split. Our evaluation shows that state-of-the-art VQA models have comparable performance in answering perception and reasoning questions, but suffer from consistency problems. To address this shortcoming, we propose an approach called Sub-Question Importance-aware Network Tuning (SQuINT), which encourages the model to attend to the same parts of the image when answering the reasoning question and the perception sub question. We show that SQuINT improves model consistency by ~7%, also marginally improving performance on the Reasoning questions in VQA, while also displaying better attention maps.
Automated design of error-resilient and hardware-efficient deep neural networks

Applying deep neural networks (DNNs) in mobile and safety-critical systems, such as autonomous vehicles, demands a reliable and efficient execution on hardware. The design of the neural architecture has a large influence on the achievable efficiency and bit error resilience of the network on hardware. Since there are numerous design choices for the architecture of DNNs, with partially opposing effects on the preferred characteristics (such as small error rates at low latency), multi-objective optimization strategies are necessary. In this paper, we develop an evolutionary optimization technique for the automated design of hardware-optimized DNN architectures. For this purpose, we derive a set of inexpensively computable objective functions, which enable the fast evaluation of DNN architectures with respect to their hardware efficiency and error resilience. We observe a strong correlation between predicted error resilience and actual measurements obtained from fault injection simulations. Furthermore, we analyze two different quantization schemes for efficient DNN computation and find one providing a significantly higher error resilience compared to the other. Finally, a comparison of the architectures provided by our algorithm with the popular MobileNetV2 and NASNet-A models reveals an up to seven times improved bit error resilience of our models. We are the first to combine error resilience, efficiency, and performance optimization in a neural architecture search framework.
Confidence-Based Large-Scale Dense Multi-View Stereo

Albeit remarkable progress has been made to improve the accuracy and completeness of multi-view stereo (MVS), existing methods still suffer from either sparse reconstructions of low-textured surfaces or heavy computational burden. In this paper, we propose a Confidence-based Large-scale Dense Multi-view Stereo (CLD-MVS) method for high resolution imagery. Firstly, we formulate MVS as a multi-view depth estimation problem, and employ a normal-aware efficient PatchMatch stereo to estimate the initial depth and normal map for each reference view. A self-supervised deep learning method is then developed to predict the spatial confidence for multi-view depth maps, which is combined with cross-view consistency to generate the ground control points. Subsequently, a confidence-driven and boundary-aware interpolation scheme using static and dynamic guidance is adopted to synthesize dense depth and normal maps. Finally, a refinement procedure which leverages synthesized depth and normal as prior is conducted to estimate cross-view consistent surface. Experiments show that the proposed CLD-MVS method achieves high geometric completeness while preserving fine-scale details. In particular, it has ranked No. 1 on the ETH3D high-resolution MVS benchmark in terms of F <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> -score.
Inferring Attention Shift Ranks of Objects for Image Saliency

Psychology studies and behavioural observation show that humans shift their attention from one location to another when viewing an image of a complex scene. This is due to the limited capacity of the human visual system in simultaneously processing multiple visual inputs. The sequential shifting of attention on objects in a non-task oriented viewing can be seen as a form of saliency ranking. Although there are methods proposed for predicting saliency rank, they are not able to model this human attention shift well, as they are primarily based on ranking saliency values from binary prediction. Following psychological studies, in this paper, we propose to predict the saliency rank by inferring human attention shift. Due to the lack of such data, we first construct a large-scale salient object ranking dataset. The saliency rank of objects is defined by the order that an observer attends to these objects based on attention shift. The final saliency rank is an average across the saliency ranks of multiple observers. We then propose a learning-based CNN to leverage both bottom-up and top-down attention mechanisms to predict the saliency rank. Experimental results show that the proposed network achieves state-of-the-art performances on salient object rank prediction. Code and dataset are available at https://github.com/SirisAvishek/Attention_Shift_Ranks.
Revisiting Unsupervised Relation Extraction

Unsupervised relation extraction (URE) extracts relations between named entities from raw text without manually-labelled data and existing knowledge bases (KBs).URE methods can be categorised into generative and discriminative approaches, which rely either on hand-crafted features or surface form.However, we demonstrate that by using only named entities to induce relation types, we can outperform existing methods on two popular datasets.We conduct a comparison and evaluation of our findings with other URE techniques, to ascertain the important features in URE.We conclude that entity types provide a strong inductive bias for URE. 1
Operationalizing Solar Energy Predictions for Sustainable, Autonomous IoT Device Management

For sustainable Internet-of-Things (IoT) systems, the solar power prediction is an essential element to optimize performance, allowing devices to schedule energy-intensive tasks in periods with excess energy. In regions with volatile weather, this requires taking the weather forecast into account. The problem is how to provide such solar energy predictions with high accuracy for large-scale IoT systems with various devices in an autonomous way, without manual adaptation effort. We present a detailed study on machine-learning approaches for the prediction of solar power intake for large-scale IoT systems. We examine which machine learning models, feature sets, and sampling rates gain the best results for a medium-term forecasting horizon. We also explore an operational setting in which devices are deployed without prior data and machine learning models are retrained for each sensor continuously as a form of online learning. Our results show that prediction errors can be reduced by 20 % compared to the state of the art, despite strong weather volatility.
Putting Visual Object Recognition in Context

Context plays an important role in visual recognition. Recent studies have shown that visual recognition networks can be fooled by placing objects in inconsistent contexts (e.g., a cow in the ocean). To model the role of contextual information in visual recognition, we systematically investigated ten critical properties of where, when, and how context modulates recognition, including the amount of context, context and object resolution, geometrical structure of context, context congruence, and temporal dynamics of contextual modulation. The tasks involved recognizing a target object surrounded with context in a natural image. As an essential benchmark, we conducted a series of psychophysics experiments where we altered one aspect of context at a time, and quantified recognition accuracy. We propose a biologically-inspired context-aware object recognition model consisting of a two-stream architecture. The model processes visual information at the fovea and periphery in parallel, dynamically incorporates object and contextual information, and sequentially reasons about the class label for the target object. Across a wide range of behavioral tasks, the model approximates human level performance without retraining for each task, captures the dependence of context enhancement on image properties, and provides initial steps towards integrating scene and object information for visual recognition. All source code and data are publicly available: https://github.com/kreimanlab/Put-In-Context.
Outlier Detection Approaches Based on Machine Learning in the Internet-of-Things

Outlier detection in the Internet of Things (IoT) is an essential challenge issue studied in numerous fields, including fraud monitoring, intrusion detection, secure localization, trust management, and so on. Conventional outlier detection technologies cannot be used directly in IoT due to the open nature of wireless communication as well as the resource-constrained characteristics of end nodes. Therefore, this article provides a comprehensive survey of new outlier detection approaches based on machine learning for IoT. The approaches are first carefully discussed based on their adopted machine learning algorithms. In addition, the performance of them with respect to the advantages and the drawbacks are compared in detail, which naturally leads to some open research issues that are analyzed afterward.
City-Scale Multi-Camera Vehicle Tracking by Semantic Attribute Parsing and Cross-Camera Tracklet Matching

This paper focuses on the Multi-Target Multi-Camera Tracking (MTMCT) task in a city-scale multi-camera network. As the trajectory of each target is naturally split into multiple sub-trajectories (namely local tracklets) in different cameras, the key issue of MTMCT is how to match local tracklets belonging to the same target across different cameras. To this end, we propose an efficient two-step MTMCT approach to robustly track vehicles in a camera network. It first generates all local tracklets and then matches the ones belonging to the same target across different cameras. More specifically, in the local tracklet generation phase, we follow the tracking-by-detection paradigm and link the detections to local tracklets by graph clustering. In the cross-camera tracklet matching phase, we first develop a spatial-temporal attention mechanism to produce robust tracklet representations. We then prune false matching candidates by traffic topology reasoning and match tracklets across cameras using the recently proposed TRACklet-to-Target Assignment (TRACTA) algorithm. The proposed method is evaluated on the City-Scale Multi-Camera Vehicle Tracking task at the 2020 AI City Challenge and achieves the second-best results.
On Public Crowdsource-Based Mechanisms for a Decentralized Blockchain Oracle

Blockchain technology has created an excitement that was last seen two decades ago when the Internet was entering the mainstream. An appealing feature of blockchain technology is smart contracts. A smart contract is an executable code. It runs on top of the blockchain facilitating an agreement between untrusted parties. These smart contracts have a major limitation, namely they cannot operate on information external to the blockchain. The inability to query such information has paved the need for trusted entities called “oracles.” These oracles attest to facts without the robust security guarantees that blockchains generally provide. This can potentially harm the integrity of the network and lead to centralized points-of-failure. To address this concern, this article proposes a decentralized oracle which is based on a voting-based game that decides the truth or falsity of queries. In the context of this article, we are only interested in binary markets, i.e., queries which can be True, False, or Unknown. When requesting facts from an oracle, a user submits binary queries. Reporters (or certifiers) respond to the queries by placing monetary stake. A formal analysis of the system parameters is presented, which shows that the proposed platform incentivizes a Nash equilibrium for truthful reporting. An extension to the base protocol is also described and profiled against the original framework. Finally, we discuss a prototype architecture, along with additional features to be considered during implementation.
Towards Question-based High-recall Information Retrieval

While continuous active learning algorithms have proven effective in finding most of the relevant documents in a collection, the cost for locating the last few remains high for applications such as Technology-assisted Reviews (TAR). To locate these last few but significant documents efficiently, Zou et al. [2018] have proposed a novel interactive algorithm. The algorithm is based on constructing questions about the presence or absence of entities in the missing relevant documents. The hypothesis made is that entities play a central role in documents carrying key information and that the users are able to answer questions about the presence or absence of an entity in the missing relevance documents. Based on this, a Sequential Bayesian Search-based approach that selects the optimal sequence of questions to ask was devised. In this work, we extend Zou et al. [2018] by (a) investigating the noise tolerance of the proposed algorithm; (b) proposing an alternative objective function to optimize, which accounts for user “erroneous” answers; (c) proposing a method that sequentially decides the best point to stop asking questions to the user; and (d) conducting a small user study to validate some of the assumptions made by Zou et al. [2018]. Furthermore, all experiments are extended to demonstrate the effectiveness of the proposed algorithms not only in the phase of abstract appraisal (i.e., finding the abstracts of potentially relevant documents in a collection) but also finding the documents to be included in the review (i.e., finding the subset of those relevant abstracts for which the article remains relevant). The experimental results demonstrate that the proposed algorithms can greatly improve performance, requiring reviewing fewer irrelevant documents to find the last relevant ones compared to state-of-the-art methods, even in the case of noisy answers. Further, they show that our algorithm learns to stop asking questions at the right time. Last, we conduct a small user study involving an expert reviewer. The user study validates some of the assumptions made in this work regarding the user’s willingness to answer the system questions and the extent of it, as well as the ability of the user to answer these questions.
Nonlinear dimensionality reduction for clustering

We introduce an approach to divisive hierarchical clustering that is capable of identifying clusters in nonlinear manifolds. This approach uses the isometric mapping (Isomap) to recursively embed (subsets of) the data in one dimension, and then performs a binary partition designed to avoid the splitting of clusters. We provide a theoretical analysis of the conditions under which contiguous and high-density clusters in the original space are guaranteed to be separable in the one-dimensional embedding. To the best of our knowledge there is little prior work that studies this problem. Extensive experiments on simulated and real data sets show that hierarchical divisive clustering algorithms derived from this approach are effective.
DeepFake Video Detection: A Time-Distributed Approach

Recent developments in machine learning algorithms have led to the generation of forged videos having remarkable quality, which are indistinguishable from real videos. This can fatally affect the way in which one perceives the information available digitally. Thus, this paper aims to efficiently and holistically detect manipulated videos generated using DeepFake, which is the most effective deep learning powered technique developed so far by the researchers. Arduous efforts have been put to detect the forgery in still images, but the authors leveraged the spatio-temporal features of the videos by taking sequences of frames as input to the model. Furthermore, the authors have proposed an architecture which took advantage of lower-level features around regions of interest as well as discrepancies across multiple frames. Experiments have been performed on the Deep Fake Detection Challenge dataset of $$\approx$$ 470 GB in size, and it has been observed that the proposed approach yielded a test accuracy score of 97.6%.
Photo-Realistic Image Super-Resolution via Variational Autoencoders

There is a great leap in objective accuracy on image super-resolution, which recently brings a new challenge on image super-resolution with larger up-scaling (e.g. 4×) using pixel based distortion for measurement. This causes over-smooth effect which cannot grasp well the perceptual similarity. The advent of generative adversarial networks makes it possible super-resolve a low-resolution image to generate photo-realistic images sharing distribution with the high-resolution images. However, generative networks suffer from problems of mode-collapse and unrealistic sample generation. We propose to perform Image Super-Resolution via Variational AutoEncoders (SR-VAE) learning according to the conditional distribution of the high-resolution images induced by the low-resolution images. Given that the Conditional Variational Autoencoders tend to generate blur images, we add the conditional sampling mechanism to narrow down the latent subspace for reconstruction. To evaluate the model generalization, we use KL loss to measure the divergence between latent vectors and standard Gaussian distribution. Eventually, in order to balance the trade-off between super-resolution distortion and perception, not only that we use pixel based loss, we also use the modified deep feature loss between SR and HR images to estimate the reconstruction. In experiments, we evaluated a large number of datasets to make comparison with other state-of-the-art super-resolution approaches. Results on both objective and subjective measurements show that our proposed SR-VAE can achieve good photo-realistic perceptual quality closer to the natural image manifold while maintain low distortion.
MASK R-CNN for Pedestrian Crosswalk Detection and Instance Segmentation

Pedestrians are the most exposed to accidents considering that the majority of motorists exclude them as road users. In this study, object detection using Mask Region-Based CNN and instance segmentation was applied to a pedestrian crosswalk. Training was done using Mask R-CNN for object detection with ResNet-101 backbone, with 0.001 learning rate and 2 images per GPU during 30 epochs of 100 batches. Based on the study, 500 pedestrian crosswalk images were gathered and selected for validation and training. 80% of the images are for the training set and for the validation set is 20%. Another 30 pedestrian crosswalk testing images were gathered for the model evaluation to verify the trained model stability and reliability. All 30 testing images had been detected and the accuracy of the detections is greater than 97%. If there is 2 or more pedestrian crosswalk in an image, it will make the color of MASK different to each detection. The summary test results verified that all gathered data was higher than 97% to be able to detect a pedestrian. With this, the proposed framework can detect pedestrian crosswalks using MASK Region-Based Convolutional Neural Network.
Infrastructuring Hope

Migration, forced mobility, and refugee studies in ICTD and related disciplines have been predominantly focused on the victims' immediate needs, including shelter, food, healthcare, language, and information. This line of work has mostly been devoid of the political background of migration and the victims' future hopes and aspirations, and hence fails to address many pressing issues associated with their long-term settlement. We address this gap in ICTD literature by drawing on our two years long ethnography with the Rohingya refugees in Bangladesh. We build on a rich body of literature on development sociology and philosophy to demonstrate how the refugees infrastructure their hope through various artful practices of solidarity, leadership, and negotiation, and how ICT plays an important role in and around each of these practices. We discuss how our study further contributes to the ongoing discourse in ICTD around aspiration, hope, design, and empowerment.
Machine Learning – The Results Are Not the only Thing that Matters! What About Security, Explainability and Fairness?

Recent advances in machine learning (ML) and the surge in computational power have opened the way to the proliferation of ML and Artificial Intelligence (AI) in many domains and applications. Still, apart from achieving good accuracy and results, there are many challenges that need to be discussed in order to effectively apply ML algorithms in critical applications for the good of societies. The aspects that can hinder practical and trustful ML and AI are: lack of security of ML algorithms as well as lack of fairness and explainability. In this paper we discuss those aspects and provide current state of the art analysis of the relevant works in the mentioned domains.
Secure collaborative few-shot learning

Few-shot learning aims at training a model that can effectively recognize novel classes with extremely limited training examples. Few-shot learning via meta-learning can improve the performance on novel tasks by leveraging previously acquired knowledge as a prior when the training examples are extremely limited. However, most of these existing few-shot learning methods involve parameter transfer, which usually requires sharing models trained on the examples for specific tasks, thus posing a potential threat to the privacy of data owners. To tackle this, we design a novel secure collaborative few-shot learning framework. More specifically, we incorporate differential privacy into few-shot learning through adding the calibrated Gaussian noise to its optimization process to prevent sensitive information in the training set from being leaked. To prevent potential privacy disclosure to other participants and the central server, homomorphic encryption is integrated while calculating global loss functions and interacting with a central server. Furthermore, we implement our framework on the classical few-shot learning methods such as MAML and Reptile, and extensively evaluate its performance on Omniglot, Mini-ImageNet and Fewshot-CIFAR100 datasets. The experimental results demonstrate the effectiveness of our framework in both utility and privacy.