Reviewerly/exHarmony huggingface dataset:
How Deeply to Fine-Tune a Convolutional Neural Network: A Case Study Using a Histopathology Dataset

Accurate classification of medical images is of great importance for correct disease diagnosis. The automation of medical image classification is of great necessity because it can provide a second opinion or even a better classification in case of a shortage of experienced medical staff. Convolutional neural networks (CNN) were introduced to improve the image classification domain by eliminating the need to manually select which features to use to classify images. Training CNN from scratch requires very large annotated datasets that are scarce in the medical field. Transfer learning of CNN weights from another large non-medical dataset can help overcome the problem of medical image scarcity. Transfer learning consists of fine-tuning CNN layers to suit the new dataset. The main questions when using transfer learning are how deeply to fine-tune the network and what difference in generalization that will make. In this paper, all of the experiments were done on two histopathology datasets using three state-of-the-art architectures to systematically study the effect of block-wise fine-tuning of CNN. Results show that fine-tuning the entire network is not always the best option; especially for shallow networks, alternatively fine-tuning the top blocks can save both time and computational power and produce more robust classifiers.
Deep Attentive Video Summarization With Distribution Consistency Learning

This article studies supervised video summarization by formulating it into a sequence-to-sequence learning framework, in which the input and output are sequences of original video frames and their predicted importance scores, respectively. Two critical issues are addressed in this article: short-term contextual attention insufficiency and distribution inconsistency. The former lies in the insufficiency of capturing the short-term contextual attention information within the video sequence itself since the existing approaches focus a lot on the long-term encoder-decoder attention. The latter refers to the distributions of predicted importance score sequence and the ground-truth sequence is inconsistent, which may lead to a suboptimal solution. To better mitigate the first issue, we incorporate a self-attention mechanism in the encoder to highlight the important keyframes in a short-term context. The proposed approach alongside the encoder-decoder attention constitutes our deep attentive models for video summarization. For the second one, we propose a distribution consistency learning method by employing a simple yet effective regularization loss term, which seeks a consistent distribution for the two sequences. Our final approach is dubbed as Attentive and Distribution consistent video Summarization (ADSum). Extensive experiments on benchmark data sets demonstrate the superiority of the proposed ADSum approach against state-of-the-art approaches.
Digital Literacy Key Performance Indicators for Sustainable Development

The concept of digital literacy has been defined in numerous ways over the last two decades to incorporate rapid technological changes, its versatility, and to bridge the global digital divide. Most approaches have been technology-centric with an inherent assumption of cultural and political neutrality of new media technologies. There are multiple hurdles in every stage of digital literacy implementation. The lack of solutions such as local language digital interfaces, locally relevant content, digital literacy training, the use of icons and audio excludes a large fraction of illiterate people. In this article, we analyse case studies targeted at under-connected people in sub-Saharan Africa and India that use digital literacy programmes to build knowledge and health literacy, solve societal problems and foster development. In India, we focus on notable initiatives undertaken in the domain of digital literacy for rural populations. In Sub-Saharan Africa, we draw from an original project in Kenya aiming at developing digital literacy for youth from low-income backgrounds. We further focus on Senegal, Mali, Burkina Faso and Tanzania, where field studies have been conducted on the use of digital technologies by low-literacy people and on how audio and icon-based interfaces and Internet lite standard could help them overcome their limitations. The main objective of this article is to identify key performance indicators (KPIs) in the context of digital literacy skills as one of the pillars for digital inclusion. We will learn how digital literacy programmes can be used to build digital literacy and how KPIs for sustainable development can be established. In the final discussion, we offer lessons learned from the case studies and further recommendation for stakeholders and decision-makers in the field of digital health literacy.
Graph convolutional autoencoder model for the shape coding and cognition of buildings in maps

The shape of a geospatial object is an important characteristic and a significant factor in spatial cognition. Existing shape representation methods for vector-structured objects in the map space are mainly based on geometric and statistical measures. Considering that shape is complicated and cognitively related, this study develops a learning strategy to combine multiple features extracted from its boundary and obtain a reasonable shape representation. Taking building data as example, this study first models the shape of a building using a graph structure and extracts multiple features for each vertex based on the local and regional structures. A graph convolutional autoencoder (GCAE) model comprising graph convolution and autoencoder architecture is proposed to analyze the modeled graph and realize shape coding through unsupervised learning. Experiments show that the GCAE model can produce a cognitively compliant shape coding, with the ability to distinguish different shapes. It outperforms existing methods in terms of similarity measurements. Furthermore, the shape coding is experimentally proven to be effective in representing the local and global characteristics of building shape in application scenarios such as shape retrieval and matching.
Pronto: Rapid Augmented Reality Video Prototyping Using Sketches and Enaction

Designers have limited tools to prototype AR experiences rapidly. Can lightweight, immediate tools let designers prototype dynamic AR interactions while capturing the nuances of a 3D experience? We interviewed three AR experts and identified several recurring issues in AR design: creating and positioning 3D assets, handling the changing user position, and orchestrating multiple animations. We introduce PROJECT PRONTO, a tablet-based video prototyping system that combines 2D video with 3D manipulation. PRONTO supports four intertwined activities: capturing 3D spatial information alongside a video scenario, positioning and sketching 2D drawings in a 3D world, and enacting animations with physical interactions. An observational study with professional designers shows that participants can use PRONTO to prototype diverse AR experiences. All participants performed two tasks: replicating a sample non-trivial AR experience and prototyping their open-ended designs. All participants completed the replication task and found PRONTO easy to use. Most participants found that PRONTO encourages more exploration of designs than their current practices.
Black Box Explanation by Learning Image Exemplars in the Latent Feature Space

We present an approach to explain the decisions of black box models for image classification. While using the black box to label images, our explanation method exploits the latent feature space learned through an adversarial autoencoder. The proposed method first generates exemplar images in the latent feature space and learns a decision tree classifier. Then, it selects and decodes exemplars respecting local decision rules. Finally, it visualizes them in a manner that shows to the user how the exemplars can be modified to either stay within their class, or to become counter-factuals by "morphing" into another class. Since we focus on black box decision systems for image classification, the explanation obtained from the exemplars also provides a saliency map highlighting the areas of the image that contribute to its classification, and areas of the image that push it into another class. We present the results of an experimental evaluation on three datasets and two black box models. Besides providing the most useful and interpretable explanations, we show that the proposed method outperforms existing explainers in terms of fidelity, relevance, coherence, and stability.
My Body, My Avatar: How People Perceive Their Avatars in Social Virtual Reality

The perception and experience of avatars has been critical to understand the social dynamics in virtual environments, online gaming, and collaborative systems. How would emerging sociotechnical systems further complicate the role of avatars in our online social lives? In this paper we focus on how people perceive and understand their avatars in social virtual reality (VR) - 3D virtual spaces where multiple users can interact with one another through VR head-mounted displays (HMDs). Based on 30 interviews, we identified three key themes emerging in people's perceptions and experiences of their avatars across various social VR applications. Our study contributes to further improving social VR technologies and better understanding emerging social interaction dynamics and consequences within social VR.
ColorFool: Semantic Adversarial Colorization

Adversarial attacks that generate small Lp norm perturbations to mislead classifiers have limited success in black-box settings and with unseen classifiers. These attacks are also not robust to defenses that use denoising filters and to adversarial training procedures. Instead, adversarial attacks that generate unrestricted perturbations are more robust to defenses, are generally more successful in black-box settings and are more transferable to unseen classifiers. However, unrestricted perturbations may be noticeable to humans. In this paper, we propose a content-based black-box adversarial attack that generates unrestricted perturbations by exploiting image semantics to selectively modify colors within chosen ranges that are perceived as natural by humans. We show that the proposed approach, ColorFool, outperforms in terms of success rate, robustness to defense frameworks and transferability, five state-of-the-art adversarial attacks on two different tasks, scene and object classification, when attacking three state-of-the-art deep neural networks using three standard datasets. The source code is available at https://github.com/smartcameras/ColorFool.
Unsupervised Representation Learning for Gaze Estimation

Although automatic gaze estimation is very important to a large variety of application areas, it is difficult to train accurate and robust gaze models, in great part due to the difficulty in collecting large and diverse data (annotating 3D gaze is expensive and existing datasets use different setups). To address this issue, our main contribution in this paper is to propose an effective approach to learn a low dimensional gaze representation without gaze annotations, which to the best of our best knowledge, is the first work to do so. The main idea is to rely on a gaze redirection network and use the gaze representation difference of the input and target images (of the redirection network) as the redirection variable. A redirection loss in image domain allows the joint training of both the redirection network and the gaze representation network. In addition, we propose a warping field regularization which not only provides an explicit physical meaning to the gaze representations but also avoids redirection distortions. Promising results on few-shot gaze estimation (competitive results can be achieved with as few as ≤ 100 calibration samples), cross-dataset gaze estimation, gaze network pretraining, and another task (head pose estimation) demonstrate the validity of our framework.
Hierarchical Graph Attention Network for Visual Relationship Detection

Visual Relationship Detection (VRD) aims to describe the relationship between two objects by providing a structural triplet shown as <;subject-predicate-object>. Existing graph-based methods mainly represent the relationships by an object-level graph, which ignores to model the triplet-level dependencies. In this work, a Hierarchical Graph Attention Network (HGAT) is proposed to capture the dependencies on both object-level and triplet-level. Object-level graph aims to capture the interactions between objects, while the triplet-level graph models the dependencies among relation triplets. In addition, prior knowledge and attention mechanism are introduced to fix the redundant or missing edges on graphs that are constructed according to spatial correlation. With these approaches, nodes are allowed to attend over their spatial and semantic neighborhoods' features based on the visual or semantic feature correlation. Experimental results on the well-known VG and VRD datasets demonstrate that our model significantly outperforms the state-of-the-art methods.
A large empirical assessment of the role of data balancing in machine-learning-based code smell detection

Code smells can compromise software quality in the long term by inducing technical debt. For this reason, many approaches aimed at identifying these design flaws have been proposed in the last decade. Most of them are based on heuristics in which a set of metrics is used to detect smelly code components. However, these techniques suffer from subjective interpretations, a low agreement between detectors, and threshold dependability. To overcome these limitations, previous work applied Machine-Learning that can learn from previous datasets without needing any threshold definition. However, more recent work has shown that Machine-Learning is not always suitable for code smell detection due to the highly imbalanced nature of the problem. In this study, we investigate five approaches to mitigate data imbalance issues to understand their impact on Machine Learning-based approaches for code smell detection in Object-Oriented systems and those implementing the Model-View-Controller pattern. Our findings show that avoiding balancing does not dramatically impact accuracy. Existing data balancing techniques are inadequate for code smell detection leading to poor accuracy for Machine-Learning-based approaches. Therefore, new metrics to exploit different software characteristics and new techniques to effectively combine them are needed.
3FabRec: Fast Few-Shot Face Alignment by Reconstruction

Current supervised methods for facial landmark detection require a large amount of training data and may suffer from overfitting to specific datasets due to the massive number of parameters. We introduce a semi-supervised method in which the crucial idea is to first generate implicit face knowledge from the large amounts of unlabeled images of faces available today. In a first, completely unsupervised stage, we train an adversarial autoencoder to reconstruct faces via a low-dimensional face embedding. In a second, supervised stage, we interleave the decoder with transfer layers to retask the generation of color images to the prediction of landmark heatmaps. Our framework (3FabRec) achieves state-of-the-art performance on several common benchmarks and, most importantly, is able to maintain impressive accuracy on extremely small training sets down to as few as 10 images. As the interleaved layers only add a low amount of parameters to the decoder, inference runs at several hundred FPS on a GPU.
MMTM: Multimodal Transfer Module for CNN Fusion

In late fusion, each modality is processed in a separate unimodal Convolutional Neural Network (CNN) stream and the scores of each modality are fused at the end. Due to its simplicity, late fusion is still the predominant approach in many state-of-the-art multimodal applications. In this paper, we present a simple neural network module for leveraging the knowledge from multiple modalities in convolutional neural networks. The proposed unit, named Multimodal Transfer Module (MMTM), can be added at different levels of the feature hierarchy, enabling slow modality fusion. Using squeeze and excitation operations, MMTM utilizes the knowledge of multiple modalities to recalibrate the channel-wise features in each CNN stream. Unlike other intermediate fusion methods, the proposed module could be used for feature modality fusion in convolution layers with different spatial dimensions. Another advantage of the proposed method is that it could be added among unimodal branches with minimum changes in the their network architectures, allowing each branch to be initialized with existing pretrained weights. Experimental results show that our framework improves the recognition accuracy of well-known multimodal networks. We demonstrate state-of-the-art or competitive performance on four datasets that span the task domains of dynamic hand gesture recognition, speech enhancement, and action recognition with RGB and body joints.
One-Shot Adversarial Attacks on Visual Tracking With Dual Attention

Almost all adversarial attacks in computer vision are aimed at pre-known object categories, which could be offline trained for generating perturbations. But as for visual object tracking, the tracked target categories are normally unknown in advance. However, the tracking algorithms also have potential risks of being attacked, which could be maliciously used to fool the surveillance systems. Meanwhile, it is still a challenging task that adversarial attacks on tracking since it has the free-model tracked target. Therefore, to help draw more attention to the potential risks, we study adversarial attacks on tracking algorithms. In this paper, we propose a novel one-shot adversarial attack method to generate adversarial examples for free-model single object tracking, where merely adding slight perturbations on the target patch in the initial frame causes state-of-the-art trackers to lose the target in subsequent frames. Specifically, the optimization objective of the proposed attack consists of two components and leverages the dual attention mechanisms. The first component adopts a targeted attack strategy by optimizing the batch confidence loss with confidence attention while the second one applies a general perturbation strategy by optimizing the feature loss with channel attention. Experimental results show that our approach can significantly lower the accuracy of the most advanced Siamese network-based trackers on three benchmarks.
Multi-View Neural Human Rendering

We present an end-to-end Neural Human Renderer (NHR) for dynamic human captures under the multi-view setting. NHR adopts PointNet++ for feature extraction (FE) to enable robust 3D correspondence matching on low quality, dynamic 3D reconstructions. To render new views, we map 3D features onto the target camera as a 2D feature map and employ an anti-aliased CNN to handle holes and noises. Newly synthesized views from NHR can be further used to construct visual hulls to handle textureless and/or dark regions such as black clothing. Comprehensive experiments show NHR significantly outperforms the state-of-the-art neural and image-based rendering techniques, especially on hands, hair, nose, foot, etc.
Any-Shot Sequential Anomaly Detection in Surveillance Videos

Anomaly detection in surveillance videos has been recently gaining attention. Even though the performance of state-of-the-art methods on publicly available data sets has been competitive, they demand a massive amount of training data. Also, they lack a concrete approach for continuously updating the trained model once new data is available. Furthermore, online decision making is an important but mostly neglected factor in this domain. Motivated by these research gaps, we propose an online anomaly detection method for surveillance videos using transfer learning and any-shot learning, which in turn significantly reduces the training complexity and provides a mechanism which can detect anomalies using only a few labeled nominal examples. Our proposed algorithm leverages the feature extraction power of neural network-based models for transfer learning, and the any-shot learning capability of statistical detection methods.
Review on Image Processing Based Adversarial Example Defenses in Computer Vision

Recent research works showed that deep neural networks are vulnerable to adversarial examples, which are usually maliciously created by carefully adding deliberate and imperceptible perturbations to examples. Several states of the art defense methods are proposed based on the existing image processing methods like image compression and image denoising. However, such approaches are not the final optimal solution for defense adversarial perturbations in DNN models. In this paper, we reviewed two main approaches to deploying image processing methods as a defense. By analyzing and discus!sing the remaining issues, we present two open questions for future research direction including the definition of adversarial perturbations and noises, the novel defense-aware threat model. A further research direction is also given by re-thinking the impacts of adversarial perturbations on all frequency bands.
DecomVQANet: Decomposing visual question answering deep network via tensor decomposition and regression

The model we developed is a novel comprehensive solution to compress and accelerate the Visual Question Answering systems. In our algorithm Convolutional Neural Network is compressed with Long Short Term Memory to accelerate processing simultaneously. We propose to conduct various decomposition methods and regression strategies on different layers, including Canonical Polyadic, Tucker, and Tensor Train to decompose Fully Connected layers in CNN and LSTM. The Flattening Layer and Fully Connected layer at the end of the model are replaced with Tensor Regression layers. In order to compress the parameter further, the feature flow between the layers is compressed by Tensor Contraction layer. The proposed tensor decomposition model was evaluated on VQA 2.0 dataset with Pythia as baseline model. Our proposed model achieved from 77% to 91% of compression ratio, and only from 1% to 5% accuracy drop.
Making sense of<i>danmu</i>: Coherence in massive anonymous chats on Bilibili.com

Although coherence has been widely studied in computer-mediated communication (CMC), insufficient attention has been paid to emergent multimodal forms. This study analyzes a popular commentary system on Chinese and Japanese video-sharing sites – known as danmu or danmaku – where anonymous comments are superimposed on and scroll across the video frame. Through content and multimodal discourse analysis, we unpack danmu-mediated communication analyzing the newest interface (on Bilibili.com), the comments, the interpersonal interactions and the unusual use of the second-person pronoun. Results show that despite the technological constraints (hidden authorship, unmarked sending date and lack of options to structure comments), users construct order in interactions through repetition, danmu-specific expressions and multimodal references, while using playful language to make fun. This study provides an up-to-date analysis on an increasingly popular CMC medium beyond well-studied social networking sites, and broadens the understanding of coherence in contemporary CMC.