Reviewerly/exHarmony huggingface dataset:
Scalable Quantitative Verification for Deep Neural Networks

Despite the functional success of deep neural networks (DNNs), their trustworthiness remains a crucial open challenge. To address this challenge, both testing and verification techniques have been proposed. But these existing techniques pro- vide either scalability to large networks or formal guarantees, not both. In this paper, we propose a scalable quantitative verification framework for deep neural networks, i.e., a test-driven approach that comes with formal guarantees that a desired probabilistic property is satisfied. Our technique performs enough tests until soundness of a formal probabilistic property can be proven. It can be used to certify properties of both deterministic and randomized DNNs. We implement our approach in a tool called PROVERO1 and apply it in the context of certifying adversarial robustness of DNNs. In this context, we first show a new attack- agnostic measure of robustness which offers an alternative to purely attack-based methodology of evaluating robustness being reported today. Second, PROVERO provides certificates of robustness for large DNNs, where existing state-of-the-art verification tools fail to produce conclusive results. Our work paves the way forward for verifying properties of distributions captured by real-world deep neural networks, with provable guarantees, even where testers only have black-box access to the neural network.
Identity Claims that Underlie Ethical Awareness and Action

HCI and STS researchers have previously described the ethical complexity of practice, drawing together aspects of organizational complexity, design knowledge, and ethical frameworks. Building on this work, we investigate the identity claims and beliefs that impact practitioners' ability to recognize and act upon ethical concerns in a range of technology-focused disciplines. In this paper, we report results from an interview study with 12 practitioners, identifying and describing their identity claims related to ethical awareness and action. We conducted a critically-focused thematic analysis to identify eight distinct claims representing roles relating to learning, educating, following policies, feeling a sense of responsibility, being a member of a profession, a translator, an activist, and deliberative. Based on our findings, we demonstrate how the claims foreground building competence in relation to ethical practice. We highlight the dynamic interplay among these claims and point towards implications for identity work in socio-technical contexts.
Real-Time Tracking Algorithm for Aerial Vehicles Using Improved Convolutional Neural Network and Transfer Learning

To address the poor image feature extraction ability, excessive tracking time, and low accuracy of traditional real-time algorithms for aerial tracking of vehicle traffic, a real-time tracking algorithm using improved convolutional neural network (CNN) and transfer learning was proposed herein. In this algorithm, first, the aerial vehicle images are matched with sample images for the image offset calibration. Second, the CNN parameters are initialized by constructing a filter set, and transfer learning is employed to construct a CNN pre-training model. Third, a deep convolution feature extraction structure map is designed to extract the depth features of images. Finally, based on the depth features, the target vehicle motion model is established, the similarity between the target and candidate models is calculated, and the real-time tracking of aerial vehicles is completed. The results show that the image correction accuracy of the proposed algorithm is as high as 92&#x0025;. The algorithm yields satisfactory results in terms of feature extraction and calculation accuracy. Moreover, it has a small overall error, the average tracking time required by it is only 22.8 s, and its false negative rate is as low as 0.4&#x0025;. Therefore, the proposed algorithm has considerable potential practical application.
Gabriella: An Online System for Real-Time Activity Detection in Untrimmed Security Videos

Activity detection in security videos is a difficult problem due to multiple factors such as large field of view, presence of multiple activities, varying scales and viewpoints, and its untrimmed nature. The existing research in activity detection is mainly focused on datasets, such as UCF-101, JHMDB, THUMOS, and AVA, which partially address these issues. The requirement of processing security videos in real-time makes this even more challenging. In this work, we propose Gabriella, a real-time online system to perform activity detection on untrimmed security videos. The proposed method consists of three stages: tubelet extraction, activity classification, and online tubelet merging. For tubelet extraction, we propose a localization network which takes a video clip as input and spatio-temporally detects potential foreground regions at multiple scales to generate action tubelets. We propose a novel Patch-Dice loss to handle large variations in actor size. Our online processing of videos at a clip level drastically reduces the computation time in detecting activities. The detected tubelets are assigned activity class scores by the classification network and merged together using our proposed Tubelet-Merge Action-Split (TMAS) algorithm to form the final action detections. The TMAS algorithm efficiently connects the tubelets in an online fashion to generate action detections which are robust against varying length activities. We perform our experiments on the VIRAT and MEVA (Multiview Extended Video with Activities) datasets and demonstrate the effectiveness of the proposed approach in terms of speed (~100 fps) and performance with state-of-the-art results. More details about this work are available on our project webpage <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">11</sup> https://www.crcv.ucf.edu/research/projects/gabriella-an-online-system-for-real-time-activity-detection-in-untrimmed-security-videos.
Multi-modal Information Extraction from Text, Semi-structured, and Tabular Data on the Web

How do we surface the large amount of information present in HTML documents on the Web, from news articles to Rotten Tomatoes pages to tables of sports scores? Such information can enable a variety of applications including knowledge base construction, question answering, recommendation, and more. In this tutorial, we present approaches for information extraction (IE) from Web data that can be differentiated along two key dimensions: 1) the diversity in data modality that is leveraged, e.g. text, visual, XML/HTML, and 2) the thrust to develop scalable approaches with zero to limited human supervision.
Let There Be Light: Improved Traffic Surveillance via Detail Preserving Night-to-Day Transfer

In recent years, image and video surveillance have made considerable progresses to the Intelligent Transportation Systems (ITS) with the help of deep Convolutional Neural Networks (CNNs). As one of the state-of-the-art perception approaches, detecting the interested objects in each frame of video surveillance is widely desired by ITS. Currently, object detection shows remarkable efficiency and reliability in standard scenarios such as daytime scenes with favorable illumination conditions. However, in face of adverse conditions such as the nighttime, object detection loses its accuracy significantly. One of the main causes of the problem is the lack of sufficient annotated detection datasets of nighttime scenes. In this paper, we propose a framework to alleviate the accuracy decline when object detection is taken to adverse conditions by using image translation method. We propose to utilize style translation based StyleMix method to acquire pairs of day time image and nighttime image as training data for following nighttime to daytime image translation. To alleviate the detail corruptions caused by Generative Adversarial Networks (GANs), we propose to utilize Kernel Prediction Network (KPN) based method to refine the nighttime to daytime image translation. The KPN network is trained with object detection task together to adapt the trained daytime model to nighttime vehicle detection directly. Experiments on vehicle detection verified the accuracy and effectiveness of the proposed approach.
User Authentication via Electrical Muscle Stimulation

We propose a novel modality for active biometric authentication: electrical muscle stimulation (EMS). To explore this, we engineered an interactive system, which we call ElectricAuth, that stimulates the user’s forearm muscles with a sequence of electrical impulses (i.e., EMS challenge) and measures the user’s involuntary finger movements (i.e., response to the challenge). ElectricAuth leverages EMS’s intersubject variability, where the same electrical stimulation results in different movements in different users because everybody’s physiology is unique (e.g., differences in bone and muscular structure, skin resistance and composition, etc.). As such, ElectricAuth allows users to login without memorizing passwords or PINs.
Data Agents: Promoting Reflection through Meaningful Representations of Personal Data in Everyday Life

Visual and physical representations of historical personal data have been discussed as artifacts that can lead to self-reflection through meaning-making. However, it is yet unclear how those two concepts relate to each other. We focus on meaningfulness, a part of meaning-making that relates to feelings. In this paper, we present three projects where mundane objects, our data agents, are combined in meaningful ways with personal data with the aim to trigger reflection by placing a person's individual experience of data in relation to others'. To identify relationships between self-reflection and meaningfulness we use Fleck and Fitzpatrick's framework to describe the levels of reflection that we found in our projects and Mekler and Hornbæk's meaning framework to define the depth of reflection. We conclude with a discussion on four themes in which we outline how data agents informed the intersections between our central concepts. This paper, constitutes a first step towards unpacking those relationships and invites for further explorations by the HCI community.
Temporal Link Prediction Via Reinforcement Learning

The availability of enormous event data with timestamps has aroused the study of Dynamic Knowledge Graphs (KGs). In dynamic KGs, temporal link prediction is an important task, which predicts future interactions between entities. Compared with conventional statistic link prediction tasks, temporal link prediction has three main challenges: i) How to deal with new entities that we have not observed before. ii) How to model the temporal evolutionary patterns. iii) How to adapt to the changes in KGs without re-training the model. To deal with these challenges, we present a novel reinforcement learning approach with an update mechanism to integrate temporal information. To predict future events, we train a time-aware agent to navigate the graph conditioned on the input query to find predictive paths. The experimental results indicate a clear improvement over the state-of-the-art methods.
A Multi-Objective Enhanced Fruit Fly Optimization (MO-EFOA) Framework for Despeckling SAR Images using DTCWT based Local Adaptive Thresholding

The importance of Satellite Aperture Radar (SAR) imagery systems is increasing day-by-day in various field such as earth observation, hi-technology war mechanisms, etc. The images captured by SAR imagery systems are mainly used to detect and classify objects captured in the images. Due to the complexity of the image capturing process, SAR images can be highly noisy – often consisting of multiplicative noise, also known as speckle. To detect or classify objects in SAR images, speckle noise must be removed from images. During despeckling process, the preservation of important information in the SAR images while removing noise such as edge or patterns is a crucial task. Despeckling methods are liable to compromise on edge preservation ability while aspiring for good quality denoised images. Many researchers have proposed wavelet transform based despeckling approaches such as Discrete Wavelet Transform (DWT), Undecimated Wavelet Transform (UDWT), Dual Tree Complex Wavelet Transform (DTCWT). In these approaches, finding the best values of the coefficients plays an important role in yielding excellent denoised images with preserved edges. In this paper, we have proposed a novel optimization framework that optimizes thresholding coefficients of DTCWT despeckling method for SAR images. The proposed optimization framework is based on Fruit Fly Optimization (FOA) algorithm. The approach is a multi-objective optimization algorithm that is used to find maximum values for Peak Signal-to-Noise Ratio (PSNR), Mean Structural Similarity Index (MSSIM) and Equivalent number of look (ENL). The maximum value of MSSIM indicates high edge preservation capacity, whereas maximum PSNR value indicates good quality denoising of images. The maximum ENL value represents a good speckle-noise smoothing capability. We have applied our framework over some classical images as well as over SAR images of MSTAR dataset. In our experiments, we found that our proposed framework results in the excellent PSNR 36.87 dB, 35.4 dB and 37.8 dB, MSSIM values 0.92, 0.93 and 0.92, respectively, in the case of lena image, MSTAR dataset image and TerraSAR-X dataset image.
Research on BP Neural Network Recommendation Model Fusing User Reviews and Ratings

Sparsity of rating data is a severe problem to be solved in modern recommendation researches.The fusion recommendation method is an effective solution for the problem.The method combines rating data and other types of user feedback data, such as reviews and image, to improve performance of the traditional recommendation algorithms.Some researchers have proposed fusion recommendation algorithms based on BP (Back Propagation) neural network and achieved better results.However, some existing fusion recommendation algorithms based on BP neural network still have some shortcomings.They rely on the assistance of the traditional recommendation algorithms.Moreover, the high complexity of the fusion processes of these algorithms possibly has negative impacts on the fusion effects.In this paper, we modify the fusion recommendation algorithm and propose the NNFR (neural networks fusion recommendation) model.This model improves the structure of BP neural network by specially designing the structure of network layers.User reviews and ratings can be processed in two separate sub-networks respectively and further fused in the fusion layer.The fusion features of user reviews and ratings are directly applied to perform recommendation, in order to avoid the assistance of the traditional recommendation algorithms and improve the fusing efficiency and quality.Experimental results indicate that the outstanding performance of NNFR model than comparative recommendation algorithms on rating predictions and top-k recommendations.Moreover, NNFR model can still produce high-quality recommendation results in the scenarios of sparse data.
Attention-Based Multi-Encoder Automatic Pronunciation Assessment

Automatic pronunciation assessment plays an important role in Computer-Assisted Pronunciation Training (CAPT). Traditional methods for pronunciation assessment of reading aloud tasks utilize features derived from automatic speech recognition (ASR) and thus are sensitive to the accuracy of ASR and the effectiveness of features. Moreover, the representation capability of the features is also affected by the inconsistent optimization goals between the ASR and scoring tasks. In this paper we propose an end-to-end (E2E) pronunciation scoring network based on attention mechanism and multi-encoder consisting of audio and text encoders. The network optimized by a multi-task learning (MTL) framework can provide scoring at sentence-level as well as detailed scoring at word-level. Due to data scarcity for pronunciation scoring, we utilize ASR data and synthetic data to pre-train the network in two steps, and then fine-tune the network using the limited high-quality scoring data. Experimental results based on the dataset recorded by Chinese English-as-second-language (ESL) learners and labeled by three experts demonstrate that the proposed model outperforms the baseline in Pearson correlation coefficient (PCC).
Deep Neural Network for Differentiation of Brain Tumor Tissue Displayed by Confocal Laser Endomicroscopy

Reliable on site classification of resected tumor specimens remains a challenge. Implementation of high-resolution confocal laser endoscopic techniques (CLEs) during fluorescence-guided brain tumor surgery is a new tool for intraoperative tumor tissue visualization. To overcome observer dependent errors, we aimed to predict tumor type by applying a deep learning model to image data obtained by CLE.Human brain tumor specimens from 25 patients with brain metastasis, glioblastoma, and meningioma were evaluated within this study. In addition to routine histopathological analysis, tissue samples were stained with fluorescein ex vivo and analyzed with CLE. We trained two convolutional neural networks and built a predictive level for the outputs.Multiple CLE images were obtained from each specimen with a total number of 13,972 fluorescein based images. Test accuracy of 90.9% was achieved after applying a two-class prediction for glioblastomas and brain metastases with an area under the curve (AUC) value of 0.92. For three class predictions, our model achieved a ratio of correct predicted label of 85.8% in the test set, which was confirmed with five-fold cross validation, without definition of confidence. Applying a confidence rate of 0.999 increased the prediction accuracy to 98.6% when images with substantial artifacts were excluded before the analysis. 36.3% of total images met the output criteria.We trained a residual network model that allows automated, on site analysis of resected tumor specimens based on CLE image datasets. Further in vivo studies are required to assess the clinical benefit CLE can have.
FoodScrap: Promoting Rich Data Capture and Reflective Food Journaling Through Speech Input

The factors influencing people's food decisions, such as one's mood and eating environment, are important information to foster self-reflection and to develop personalized healthy diet. But, it is difficult to consistently collect them due to the heavy data capture burden. In this work, we examine how speech input supports capturing everyday food practice through a week-long data collection study (N = 11). We deployed FoodScrap, a speech-based food journaling app that allows people to capture food components, preparation methods, and food decisions. Using speech input, participants detailed their meal ingredients and elaborated their food decisions by describing the eating moments, explaining their eating strategy, and assessing their food practice. Participants recognized that speech input facilitated self-reflection, but expressed concerns around re-recording, mental load, social constraints, and privacy. We discuss how speech input can support low-burden and reflective food journaling and opportunities for effectively processing and presenting large amounts of speech data.
Predicting Frequently Asked Questions (FAQs) on the COVID-19 Chatbot using the DIET Classifier

A popular dialogue system in the field of natural language processing (NLP) is the chatbot. Chatbots aim to create conversations between humans and machines. COVID-19 is a member of the Coronavidae (CoV) family of the Corona viirinae family which causes the respiratory system to become severe in humans. This paper predicts chatbot answers to questions about COVID-19 with the RASA framework and uses the DIET Classifier pipeline for 300 training data. The test results with the DIET Classifier model on rasa.core.test and rasa.nlu.test provided confidence values of F1-Score, precision, and accuracy for the correct answer to the question about COVID-19, namely 1.0 with a percentage of around 85%.
Execution of a Federated Learning process within a smart contract

High quality datasets have always been valuable for the creation of Machine Learning (ML) models. It therefore makes sense to provide rewards to users that participate in a Federated Learning (FL) process with such datasets. In this competitive scene, we design a solution that leverages a blockchain network, a smart contract and a model verification algorithm in order to coordinate the training process, record user performance and provide rewards in a transparent manner.
Storyboard relational model for group activity recognition

This work concerns how to effectively recognize the group activity performed by multiple persons collectively. As known, Storyboards (i.e., medium shot, close shot) jointly describe the whole storyline of a movie in a compact way. Likewise, the actors in small subgroups (similar to Storyboards) of a group activity scene contribute a lot to such group activity and develop more compact relationships among them within subgroups. Inspired by this, we propose a Storyboard Relational Model (SRM) to address the problem of Group Activity Recognition by splitting and reintegrating the group activity based on the small yet compact Storyboards. SRM mainly consists of a Pose-Guided Pruning (PGP) module and a Dual Graph Convolutional Networks (Dual-GCN) module. Specifically, PGP is designed to refine a series of Storyboards from the group activity scene by leveraging the attention ranges of individuals. Dual-GCN models the compact relationships among actors in a Storyboard. Experimental results on two widely-used datasets illustrate the effectiveness of the proposed SRM compared with the state-of-the-art methods.
A machine learning based sentient multimedia framework to increase safety at work

In the last few decades, we have witnessed an increasing focus on safety in the workplace. ICT has always played a leading role in this context. One ICT sector that is increasingly important in ensuring safety at work is the Internet of Things and, in particular, the new architectures referring to it, such as SIoT, MIoT and Sentient Multimedia Systems. All these architectures handle huge amounts of data to extract predictive and prescriptive information. For this purpose, they often make use of Machine Learning. In this paper, we propose a framework that uses both Sentient Multimedia Systems and Machine Learning to support safety in the workplace. After the general presentation of the framework, we describe its specialization to a particular case, i.e., fall detection. As for this application scenario, we describe a Machine Learning based wearable device for fall detection that we designed, built and tested. Moreover, we illustrate a safety coordination platform for monitoring the work environment, activating alarms in case of falls, and sending appropriate advices to help workers involved in falls.