Reviewerly/exHarmony huggingface dataset:
A cognitive camera system for increasing face recognition performance in backlit conditions

One of the most difficult challenges in counter-terrorism, crime fighting and surveillance mission is to accurately identify people from an image/video footage to catch shortlisted terrorists and criminals. For this purpose, imaging devices used in video surveillance systems are being improved in many aspects such as spatial resolution, frame rate, dynamic range, spectral characteristics in order to achieve better imaging performance for both monitoring and automatic detection/recognition tasks. These development efforts aim to improve the basic imaging characteristics of the device, such as the average amount of brightness/darkness of the video footage, irrespective of high-level semantic elements/knowledge such as presence and locations of monitored objects/individuals in the scene monitored by the camera. Nevertheless, strong local/global illumination variations in harsh environments result in high dynamic range in the scene and thus over-/underexposed regions in video frames. In this study, a cognitive imaging system prototype that will allow higher face recognition performance by taking into account the locations of automatically detected human faces in the scene and the local illumination conditions/dynamic range values is proposed with a developed/produced smart camera. This is achieved by developing a smart Auto-Exposure (AE) algorithm which use only the region of interest (ROI) to compute/measure the brightness of the video frame. The current ROI is selected by intelligently and sequentially traversing among the detected faces to effectively handle whole faces in the scene. The experimental results show that the proposed ”cognitive” camera can achieve a dramatic increase in accuracy performance over the normal camera in the face recognition task.
Real-time target classification and tracking by using outdoor PTZ cameras at the edge

Tracking with a Pan-Tilt-Zoom (PTZ) camera has been a research topic in computer vision for many years. Compared to tracking with a still fixed camera, the images captured with a PTZ camera are highly dynamic because the vision becomes difficult under some realistic conditions such as fast camera movements, occlusion and similar objects to the tracked target. Also, compensating for these problems is even more complex on edge system. With the increasing availability of small single-board computers with high parallel processing power capabilities, tracking objects using an onboard computer in real time has become feasible. Although these onboard computers allow a wide variety of computer vision methods to be executed, there is still a need to optimize these methods for running time and power consumption. This paper proposes a hybrid application with low CPU consumption for surveillance objects to detect and track at the edge. To detect the target at the beginning and in the case where the track has been lost, we use the deep learning based YOLOv3 model. This model provides one of the best trade-offs between speed and accuracy in the literature. A kernelized correlation filter is used to track the detected object in real-time. Combining these two algorithms provides high accuracy and speed even on onboard computers. Under a real-time streaming condition, the proposed method yields better results than the original KCF in tracking accuracy and outperforms a deep learning-based tracker when a target has a vast movement.
Comparative evaluations of online and offline visual tracking algorithms

In this study, the performance of online and offline tracking algorithms, which are frequently used in the literature, were compared on the defined datasets. Therefore, six different datasets are prepared. The datasets consist of consecutive frames. In each dataset, target has different motion characteristics and the background types are different from each other. In addition, a total of six well known algorithms were used for comparison. These are KCF, MOSSE, CSRT, TLD, Go Turn and Siamese tracking algorithms. In conclusion, especially in cases where the target is small and the SNR value is low, the highest performance is obtained with the KCF algorithm. On the other hand, when the target is big and the SNR value is high, it is observed that the Siamese algorithm better handle changes in the target shape. In this context, considering the real scenarios, it may be possible to use the algorithms in a hybrid way to get the better performance.
ARORA &amp; NavSim: a simulator system for training autonomous agents with geospecific data

This study accompanies the initial public release of the software for ARORA, or A Realistic Open environment for Rapid Agent training, and marks a high point of several years of work for the mature and completely open ARORA simulator. The purpose of ARORA is to support the training of an autonomous agent for tasks associated with a large-scale and geospecific outdoor urban environment, including the task of navigation as a car. The study elaborates on the simulator's architecture, agent, and environment. For the environment, ARORA provides an improvement on similar simulators through an unconstrained geospecific environment with detailed semantic annotation. The agent is represented as a car available with four different options of physics fidelity. The agent also has sensors available: a pose sensor, a camera sensor, and a set of three proximity sensors. Future use cases from training extend to both civilians and militaries (including human training and wargaming), in terms of training autonomous agents in outdoor urban environments. The study also presents a brief description of NavSim, a Python-based companion tool. The purpose of NavSim is to connect to ARORA (or any other similar simulator) and train an agent using reinforcement-learning algorithms. The study also provides challenges in development and subsequent work-arounds and solutions. The goal of the ARORA & NavSim system is to provide communities with a high-fidelity, publicly available, free, and open-source system for training an autonomous agent as a car.
Meaningful machine learning robustness evaluation in real-world machine learning enabled system contexts

Applied research presented in this paper describes an approach to provide meaningful evaluation of the Machine Learning (ML) components in a Full Motion Video (FMV) Machine Learning Enabled System (MLES). The MLES itself is not discussed in the paper. We focus on the experimental activity that has been designed to provide confidence that the MLES, when fielded under dynamic and uncertain conditions, performance will not be undermined by a lack of ML robustness. For example, to real-world changes of the same scene under differing light conditions. The paper details the technical approach and how it is applied to data, across the overall experimental pipeline, consisting of a perturbation engine, test pipeline and metric production. Data is from a small imagery dataset and the results are shown and discussed as part of a proof of concept study.
FishNet for loop closure detection in VSLAM

With the rapid development of drones, unmanned vehicles and robotics industries, VLAM has become a hot technology. In particular, the birth of 5G-powered UAV has promoted the emergence of more industrial applications, making it the most core and indispensable role in many scenarios. The loop closure detection can decrease the accumulative total of error during the process of VSLAM. Former loop closure detection methods always rely on artificially features, which are not robust, making it hard to deal with changing complex scenarios. The later deep learning-based methods are considered to be better solutions for loop closure detection. However, due to the simple network structure, there is still a lot of room for improvement. This paper proposes a more complex neural network to achieve loop closure detection. This approach adopts a fish-shaped deep neural network backbone, which can extract and fuse data features at different levels. Experiments demonstrate the feasibility and effectiveness of this backbone in loop closure detection problems.
Word-level textual adversarial attacking based on genetic algorithm

Since the discovery of adversarial examples, the research on adversarial examples in the image field has caused an academic boom. In recent years, with the development of artificial intelligence, adversarial samples in the text field have also attracted more and more scholars' research interest. This paper proposes such an adversarial sample generation algorithm in a black box scenario: using a targeted word deletion scoring mechanism, it can find keywords that have a significant impact on the decision of the classifier when the internal structure of the model is unknown, and use the HowNet vocabulary to search the synonyms of these keywords are replaced to generate a set of adversarial samples that are semantically consistent with the original samples. Then combined with genetic algorithm to search for the best sample in the generated sample space. The results of testing LSTM and CNN on sentiment classification and news classification data sets show that the algorithm can greatly reduce the accuracy of the target model with less disturbance.
An improved PBFT consensus mechanism based on quality of service

As a new distributed database technology, consensus algorithm is the main difference between blockchain and traditional database technology. Consensus algorithm decides the safety and performance of blockchain technology. PBFT(Practical Byzantine Fault Tolerance) has the problem that the main node is relatively fixed, and when network scale is large, communication cost is unacceptable. To solve this problem, this paper proposes an improved PBFT consensus based on quality of service. By dynamically evaluating the service quality of nodes to divide the participation status of nodes in the blockchain network. Nodes with high service quality will have a greater chance to participate in the consensus, reducing the probability of low service quality and malicious nodes becoming block producers, thereby improving the security of the system; The FTS tree is used in the eligible nodes to randomly determine the block- producing node to ensure that the block-producing node will not be fixed for a long time; A node state transfer mechanism is proposed to dynamically divide the role of nodes. Simulation experiments show that, compared with the original PBFT consensus mechanism, the system proposed in this paper can significantly reduce the communication overhead in the consensus mechanism and increase the fault tolerance rate and security of the system.
Application of image sentiment analysis in cross-modal semantics

With the advent of the era of big data, in order to actively cope with the massive and unstructured "information overload", recommendation systems are widely used in network products and have a profound impact on people's lives. At present, the research of recommendation system has been developed for more than ten years, but many of them are still in the incomplete stage, there are problems such as sparse data, insufficient mining of user implicit preferences, and inaccurate semantic description of item features. These pictures, videos and other content are full of users' emotions, which contain a lot of user emotions, which need further Excavation and application, resulting in huge social and economic benefits. However, there are few researches on sentiment analysis of this kind of information, and the multi-modal sentiment analysis of text, images and other information is particularly weak. Multimodal information has the same emotional characteristics and can describe the user's emotional changes from different angles, so as to obtain more accurate emotional identification.
NIPVS-FL: a non-interactive publicly verifiable secure federated-learning scheme against malicious servers

Federated learning (FL) enables decentralized data sources like mobile phones to joint training a neural network model without sharing the original data. However, shared local gradients make the privacy of local data in FL vulnerable. The aggregation server also may return incorrect results to clients due to unexpected error or the deliberately attack. In this work, we explore how to design a non-interactive and publicly verifiable aggregation scheme. The existing verifiable schemes are under semi-honest adversary model, in which the server is honest-but-curious but with additional power to counterfeit the aggregation result. We propose a scheme under stronger security model against malicious servers. The proposed scheme guarantees that as long as the two servers are non-colluding, even a malicious server cannot obtain input privacy of client. The malicious server will be detected by honest clients when it tries to tamper the result.
Research on aspect level text sentiment analysis method based on BERT

The traditional word vector model cannot generate word vectors with polysemous features. The paper uses the BERT model to obtain word vectors, analyzes the modeling ability of the model in text sentiment analysis, and improves the Bert output structure. In order to obtain more text vector information, the vector output structure of BERT model is improved, the original single-layer output is replaced with multi-layers hidden state output, and the hidden information of these different hidden layers is fused and embedded into the downstream task to perform text analyze. The results show that the method proposed in this paper has a certain improvement in the text classification task.
Nonparametric Sequential Clustering of Data Streams with Composite Distributions

We study a sequential nonparametric clustering problem to group a finite set of S data streams into K clusters. The data streams are real-valued i.i.d data sequences generated from unknown continuous distributions. The distributions themselves are organized into clusters according to their proximity to each other based on a certain distance metric. The sequential tests are universal in the sense that they are independent of the underlying configuration of the distribution clusters, and the distributions themselves, as long as the maximum intra-cluster distance is smaller than the minimum inter-cluster distance. We propose sequential nonparametric clustering tests for two cases: (1) K known and (2) K unknown. In both cases, we show that the proposed sequential nonparametric clustering tests stop in finite time almost surely and are universally exponentially consistent. Further, we also bound the asymptotic growth rate of the expected stopping time as probability of error goes to zero. Our results generalize earlier work on sequential nonparametric anomaly detection to the more general sequential nonparametric clustering problem. This generalization also provides a new test for the special case of anomaly detection where the anomalous data streams can follow distinct probability distributions. We also devise a modification of the proposed sequential nonparametric clustering tests that can result in significant computational savings with negligible performance degradation. Simulations show that all our proposed sequential clustering tests outperform the corresponding fixed sample size tests in terms of the expected number of samples for a given probability of error. The simulation results also demonstrate the advantage of our proposed clustering tests in anomaly detection problems with distinct anomalies.
Interactive 3D Zoetrope with a Strobing Flashlight

We propose a 3D printed zoetrope mounted on a bike wheel where users can watch the 3D figures come to life in front of their eyes. Each frame of our animation is a 9 by 16 cm 3D fabricated diorama containing a small scene. A strobed flashlight synced with the spinning of the wheel shows the viewer each frame at just the right time, creating the illusion of 3D motion. The viewer can hold and shine the flashlight into the scene, illuminating each frame from their own point of view. Our zoetrope is modular and can have different 16 frame animations substituted in and out for fast prototyping of many cinematography, fabrication, and strobe lighting techniques. Our interactive truly 3D movie experience will push the zoetrope format to tell more complex stories and better engage viewers.
ARfy: A Pipeline for Adapting 3D Scenes to Augmented Reality

Virtual content placement in physical scenes is a crucial aspect of augmented reality (AR). This task is particularly challenging when the virtual elements must adapt to multiple target physical environments unknown during development. AR authors use strategies such as manual placement performed by end-users, automated placement powered by author-defined constraints, and procedural content generation to adapt virtual content to physical spaces. Although effective, these options require human effort or annotated virtual assets. As an alternative, we present ARfy, a pipeline to support the adaptive placement of virtual content from pre-existing 3D scenes in arbitrary physical spaces. ARfy does not require intervention by end-users or asset annotation by AR authors. We demonstrate the pipeline capabilities using simulations on a publicly available indoor space dataset. ARfy makes any generic 3D scene automatically AR-ready and provides evaluation tools to facilitate future research on adaptive virtual content placement.
Extending Computational Abstractions with Manual Craft for Visual Art Tools

Programming and computation are powerful tools for manipulating visual forms, but working with these abstractions is challenging for artists who are accustomed to direct manipulation and manual control. The goal of my research is to develop visual art tools that extend programmatic capabilities with manual craft. I do so by exposing computational abstractions as transparent materials that artists may directly manipulate and observe in a process that accommodates their non-linear workflows. Specifically, I conduct empirical research to identify challenges professional artists face when using existing software tools—as well as programming their own—to make art. I apply principles derived from these findings in two projects: an interactive programming environment that links code, numerical information, and program state to artists’ ongoing artworks, and an algorithm that automates the rigging of character clothing to bodies to allow for more flexible and customizable 2D character illustrations. Evaluating these interactions, my research promotes authoring tools that support arbitrary execution by adapting to the existing workflows of artists.
Empowering domain experts to author valid statistical analyses

Reliable statistical analyses are critical for making scientific discoveries, guiding policy, and informing decisions. To author reliable statistical analyses, integrating knowledge about the domain, data, statistics, and programming is necessary. However, this is an unrealistic expectation for many analysts who may possess domain expertise but lack statistical or programming expertise, including many researchers, policy makers, and other data scientists. How can our statistical software help these analysts? To address this need, we first probed into the cognitive and operational processes involved in authoring statistical analyses and developed the theory of hypothesis formalization. Authoring statistical analyses is a dual-search process that requires grappling with assumptions about conceptual relationships and iterating on statistical model implementations. This led to our key insight: statistical software needs to help analysts translate what they know about their domain and data into statistical modeling programs. To do so, statistical software must provide programming interfaces and interaction models that allow statistical non-experts to express their analysis goals accurately and reflect on their domain knowledge and data. Thus far, we have developed two such systems that embody this insight: Tea and Tisane. Ongoing work on rTisane explores new semantics for more accurately eliciting analysis intent and conceptual knowledge. Additionally, we are planning a summative evaluation of rTisane to assess our hypothesis that this new way of authoring statistical analyses makes domain experts more aware of their implicit assumptions, able to author and understand nuanced statistical models that answer their research questions, and avoid previous analysis mistakes.
One-Dimensional Eye-Gaze Typing Interface for People with Locked-in Syndrome

People with Locked-in syndrome (LIS) suffer from complete loss of voluntary motor functions for speech or hand-writing. They are mentally intact, retaining only the control of vertical eye movements and blinking. In this work, we present a one-dimensional typing interface controlled exclusively by vertical eye movements and dwell-time for them to communicate at will. Hidden Markov Model and Bigram Models are used as auto-completion on both word and sentence level. We conducted two preliminary user studies on non-disabled users. The typing interface achieved 3.75 WPM without prediction and 11.36 WPM with prediction.
ShadowAstro: Levitating Constellation Silhouette for Spatial Exploration and Learning

We introduce ShadowAstro, a system that uses the levitating particles’ casted shadow to produce a constellation pattern. In contrast to the traditional approach of making astronomical observations via AR, planetarium, and computer screens, we intend to use the shadows created by each levitated bead to construct the silhouette of constellations - a natural geometrical pattern that can be represented by a set of particles. In this proposal, we show that ShadowAstro can help users inspect the 12 constellations on the ecliptic plane and augment users’ experience with a projector that will serve as the light source. Through this, we draw a future vision, where ShadowAstro can serve as an interactive tool with educational purposes or an art installation in museum. We believe the concept of designing interactions between the levitated objects and their casted shadows will provide a brand new experience to end user.
CircuitAssist: Automatically Dispensing Electronic Components to Facilitate Circuit Building

When learning how to build circuits, one of the challenges novice makers face is how to identify the components needed for the circuit. Many makerspaces are stocked with a variety of electronic components that look visually similar or have similar names. Thus, novice makers may pick the wrong component, which creates a non-functional circuit although the wires are correctly connected. To address this issue, we present CircuitAssist, an actuated electronics component shelf connected to a tutorial system that dispenses electronic components for the maker in the order that they occur in the tutorial. The shelf contains dispensers for each component type with a custom release mechanism actuated by a servo motor that dispenses one component at a time. Makers can work with CircuitAssist by either following one of the provided tutorials or by directly selecting a component they need from the user interface.
ConfusionLens: Dynamic and Interactive Visualization for Performance Analysis of Multiclass Image Classifiers

Building higher-quality image classification models requires better performance analysis (PA) methods to help understand their behaviors. We propose ConfusionLens, a dynamic and interactive visualization interface that augments a conventional confusion matrix with focus+context visualization. This interface makes it possible to adaptively provide relevant information for different kinds of PA tasks. Specifically, it allows users to seamlessly switch table layouts among three views (overall view, class-level view, and between-class view) while observing all of the instance images in a single screen. This paper presents a ConfusionLens prototype that supports hundreds of instances and its several extensions to further support practical PA tasks, such as activation map visualization and instance sorting/filtering.