Reviewerly/exHarmony huggingface dataset:
Achieving Adversarial Robustness in Deep Learning-Based Overhead Imaging

The Intelligence, Surveillance, and Reconnaissance (ISR) community relies heavily on the use of overhead imagery for object detection and classification. In these applications, machine learning frameworks have been increasingly used to assist analysts in distinguishing high value targets from mundane objects quickly and effectively. In recent years, the robustness of these frameworks has come under question due to the possibility for disruption using image-based adversarial attacks, and as such, it is necessary to harden existing models against these threats. In this work, we survey a collection of three techniques to address these concerns at various stages of the image processing pipeline: external validation using Activity Based Intelligence, internal validation using Latent Space Analysis, and adversarial prevention using biologically inspired techniques. We found that biologically-inspired techniques were most effective and generalizable for mitigating adversarial attacks on overhead imagery in machine learning frameworks, with improvements as much as 34.6% over traditional augmentations, and 80.4% over a model without any augmentation-based defense.
Post-Disaster Damage Detection using Aerial Footage: Visual Question Answering (VQA) Case Study

Natural disasters are a major source of significant damage and costly repairs around the world. After a natural disaster occurs, there are usually an insurmountable amount of damage along with financial costs of repairing and aiding all the people involved. Besides that, the occurrence of natural disasters has increased significantly in the past decade. With that in mind, post-disaster damage detection is usually performed either in person or manually by human experts. Taking into consideration all the areas one has to closely look into, as well as the inaccessible terrain, debris, and unstable infrastructure, it is incredibly difficult for a surveyor to identify and annotate every single possible damage out there. Previous studies in damage detection from Unmanned Aerial Vehicles (UAVs) have lead to great outcomes. Yet, these algorithms do not collaborate with human experts. This paper develops a Visual Question Answering (VQA) technique for post-disaster damage detection on aerial footage. Our case study on the dataset collected after the hurricane Sally shows promising results.
A comparison of features Synthetic WAMI and GES of the same location

The rapid growth in deep learning has accelerated advances in many areas of computer vision. However, deep learning-based approaches require a large amount of data to train models. Subsequently, synthetic data is increasingly being looked to as a source for labeled training datasets to be used with supervised deep learning algorithms. WAMI (wide-area motion imagery) is sequential, oblique imagery, typically taken from an aircraft or drone, at city scale. Collecting a WAMI dataset can represent a significant investment of resources and logistics. However, the availability of synthetic WAMI datasets could overcome these concerns as well as potentially add benefits such as having associated ground truth. Recently, Google released Earth Studio, a browser-based animation tool that uses a 3D rendering engine to generate WAMI-like datasets across the globe. When working with synthetic data, a key point of concern is whether the synthetic data is sufficiently realistic for the purpose at hand. In this paper, we generate rendered WAMI datasets using Google Earth Studio. The rendered datasets are of the same locations for which we also have real WAMI datasets, and we then analyze the WAMI dataset and the images rendered by Google Earth Studio based on 3D reconstruction and feature evaluation of the dataset to determine how feasible synthetic datasets are in comparison to non-synthetic ones.
Informing selection of performance metrics for medical image segmentation evaluation using configurable synthetic errors

Machine learning-based segmentation in medical imaging is widely used in clinical applications from diagnostics to radiotherapy treatment planning. Segmented medical images with ground truth is useful to investigate the properties of different segmentation performance metrics to inform metric selection. Regular geometrical shapes are often used to synthesize segmentation errors and illustrate properties of performance metrics, but they lack the complexity of anatomical variations in real images. In this study, we present a tool to emulate segmentations by adjusting the reference (truth) masks of anatomical objects extracted from real medical images. Our tool is designed to modify the defined truth contours and emulate different types of segmentation errors with a set of user-configurable parameters. We defined the ground truth objects from 230 patient images in the Glioma Image Segmentation for Radiotherapy (GLIS-RT) database. For each object, we used our segmentation synthesis tool to synthesize 10 versions of segmentation (i.e., 10 simulated segmentors or algorithms) where each version has a pre-defined combination of segmentation errors. We then applied 20 performance metrics to evaluate all synthetic segmentations. We demonstrated the properties of these metrics including their ability to capture specific types of segmentation errors. By analyzing the intrinsic properties of these metrics and categorizing the segmentation errors, we are working toward the goal of developing a decision-tree tool for assisting selection of segmentation performance metrics.
Toward CNN Architectures for Image Detection

Currently, convolutional neural networks (CNNs) drive the state-of-the-art for object detection and classification in imagery. Pre-trained models exist with hundreds of million computational artificial neurons to classify images into hundreds of different classes. However, most image characterization problems, detection and classification, require the discrimination across only a handful of classes. By focusing our training to the limited class list, we expect dramatically improved discrimination with lower resource costs. Ideally, these networks are trained using smaller datasets and needed far fewer computational neurons. As researchers focus more on the detection paradigm, the expected smaller, nimbler models that balance performance optimization between correctness and throughput. Our previous studies matched image quality with convolutional neural network parameters to develop strategies for automating orchestrated collection based upon information needs. Our current study extends those goals in two directions: 1. incorporate hidden layer depth; and 2. incorporate different number of output classes. In this study, we trained multiple CNNs and varied their window sizes, hidden layers, latent space dimensions, and output classes. The images were acquired from the rare planes database and have a variable size between 100 and 400 pixels on a side. The number of classes varied from 2 to 8. The images contain synthetic targets that are visually separable from the background. The outcome from these and previous experiments is to identify candidate parameters that enable more efficient image search models.
Codomain Transformation to Represent Abstract Information for Machine Learning in Visual Scenes

Convolutional Neural Networks (CNNs) succeed well in learning to recognize spatial and spectral image features which are easily distinguished by convolutional operations; for example, visual cues based on shape and color of isolated objects. However, they are challenged in learning features which are rule-based and abstract; for example, the number of elements in a scene (cardinality), and other relationships within a scene that have to do with subtle criteria of order and relative position, especially when essential cues occupy a small fraction of the spatial and spectral search space and can vary significantly in appearance. The computational experiment reported in this paper demonstrates the efficacy of codomain transformation to help CNNs recognize the abstract concept of cardinality for artificially generated geometric objects of various shapes, sizes, and colors.
Reference design model for a patient-centric data exchange healthcare environment

The healthcare industry is currently undergoing a paradigm shift from a provider-centric environment to a patient-centric one as a direct result of the fast expansion of eHealth technologies in the modern world. In this article, we explore the various design considerations that need to be taken into account in order to accomplish the purpose of developing a patient-centered and safe healthcare environment. We discuss the concept of interoperability in the context of the healthcare system, as well as the various ways in which FHIR could be used to turn this idea into a practical reality. In addition to that, this article investigates the use of four different patient-centric data sharing mechanisms. In conclusion, the reference design model that will be used to make the environment more patient-centric will be described in detail using one of the data exchange mechanism.
Trusted Data Anomaly Detection (TaDA) in Ground Truth Image Data

Current state-of-the-art Artificial Intelligence (AI) anomaly detection from images is primarily used for defect detection and relies on relatively homogeneous datasets of images with similar foregrounds and backgrounds. This type of anomaly detection uses human labelled ground-truth data. In our research, we have extremely heterogeneous datasets and want to identify outliers. We use self-supervised Variational Autoencoders (VAEs) to identify anomalies in the latent vector feature space. Understanding the outliers in a large training data set is important for establishing trustworthiness of the AI models learned from these data, a strong requirement for military AI applications. Our study uses 8984 examples from Kaggle military planes and 4300 examples from Kaggle landscape data. We present the results of the combined heterogeneous dataset on the localized methods, with one such result exhibiting inliers as landscapes/backgrounds and outliers as all aircraft, detecting aircraft as anomalies with a 0.87 AUC. Results also include the inter-class AUC across the different aircraft classes. Our contribution to the state-of-the-art is to apply isolation forests to the latent space data after UMAP embeddings in a strongly heterogeneous image dataset for military applications to identify anomalies.
Answering an Amharic Language Semantic Question over Interlinked Data

The web has generated a large amount of structured data, and the number of Amharic-language online data portals is growing. As a result, intuitive ways of integrating and accessing this data are becoming increasingly important. Various question-answering approaches for different languages have been proposed to extract answers from interlinked datasets. In this paper, we propose a novel question-answering approach for Amharic, allowing users to ask questions using their terminology and receive direct answers. To accomplish this, the approach addressed the three major challenges that are required to meet users’ information needs from the interlinked datasets. These are understanding users’ questions, resolving lexical ambiguity, and fusing data from multiple sources. Apart from this, the approach can be used to answer different question types such as factual, aggregation, comparison, and superlative. A prepared Amharic test dataset was used to evaluate the approach. It achieved an average f-score of 0.83.
Intent-Driven QoS Control in Cloud-Network Integration Environment

The cloud-network integration brings advantages to service collaboration. Meanwhile, it brings challenges to QoS (Quality of Service) assurance. How to construct the service according to the user intent and ensure that QoS meets the user intent is particularly important. In this paper, we propose an intent-driven QoS control approach. Intent translation is performed to obtain the functional and QoS requirements with QoS estimation for the ambiguous information. Service orchestration is performed to construct the composite service with consideration of environment changing. Service verification is performed for verifying the functional and QoS attributes with related environmental factors incorporated. And service optimization is performed to adjust the composition strategy locally for ensuring the composition service can constantly meet the user intent.
Increasing Trust in Artificial Intelligence with a Defensible AI Technique

Artificial intelligence (AI) systems are prevalent throughout society and have significant impact on peoples’ everyday lives. Explainable artificial intelligence techniques have been developed in response to concerns about the opaqueness of many AI techniques, given their potential impact. Explainable techniques, though, do not guarantee performance – instead, they simply explain what was recommended or decided. This paper discusses how a new class of AI techniques, defensible AI (DAI) can increase public trust in AI. Issues with existing techniques are reviewed and an example DAI technique is presented and its benefits – relative to reliability, trust and security – are assessed.
In-Pose Estimation of Covered and Uncovered Human Body from Thermal Camera Images Using Multi-Scale Stacked Hourglass (MSSHg) Network

Estimating the pose of the human body lying on the bed has enormous and valuable benefits in various applications such as medicine and healthcare. Regular and long-term monitoring and detection of human poses in bed need sustaining certain poses, as they contribute to better recovery after certain surgeries or to control the symptoms effects of many complications. Particularly, when the body is fully covered or in completely dark ambient, existing methods for in-bed pose monitoring should be upgraded or new methods should be developed. An economical, contactless, vision-based system that is not sensitive to both challenges of darkness and cover is a thermal camera imaging system using the long-wavelength IR technique. In this regard, the dataset used in this study is the SLP data set, in which several different modes of poses with various covers are collected simultaneously. This dataset is a thermal camera-based dataset and is entirely annotated as an in-bed pose dataset. In this paper, the multi-scale stacked Hourglass (MSSHg) network, has been applied to improve the processing of thermal camera images for human pose estimation. Drawing on the concept of multi-scale, the aim of using the pre-processing network in this model is to extract feature maps with different scales and assign them to different stacked Hourglass networks. The performance of this approach for in-bed pose estimation showed progress with the result accuracy of 96.8% in PCK0.2 standard. In comparison with stacked Hourglass network results, the MSSHg network has about 0.8% more accuracy for covered body pose estimation and as much as or even higher accuracy for uncovered body pose estimation at the final stage. Additionally, it converges faster and easier at the beginning of the training stage.
Implementing passive authentication with enhanced risk-based security

In the medical field, it is extremely important to monitor the status of use and malfunction of medical equipment during its operation and management. Furthermore, keeping proper records of people’s activity is necessary to ensure security. Authentication and authorization play an important role in this purpose. However, in most cases, authentication is performed by an active action of the user, which is a burden on the person in charge and often affects normal business operations. For this reason, authentication is neglected. In this study, we aim to introduce passive authentication to achieve responsibility tracking and auditability without interfering with normal operations; in particular, we report the results of a study on the introduction of passive authentication in obstetrics.
Training Echo-State Networks for Big Data Prediction using Variation of Parameters

Recurring neural networks (RNNs) are a class of neural networks that are naturally suitable for processing time series and other sequential data. RNNs and specifically Reservoir Computing (RC) has attracted much attention over the past decade, particularly in the context of time series prediction. However, most time series have a multi-scale structure. The Echo State Networks (ESN) are one of two pioneering methods of reservoir computing. ESN requires some experiences and insights to achieve the acclaimed accuracy in many tasks. In this paper, we present a practical application of case prediction using several time series data, multiple parameters of our ESN system are scaled to compare the accuracy of the proposed model.
Towards Better Quality of Experience in HTTP Adaptive Streaming

HTTP Adaptive Streaming (HAS) is nowadays a popular solution for multimedia delivery. The novelty of HAS lies in the possibility of continuously adapting the streaming session to current network conditions, facilitated by Adaptive Bitrate (ABR) algorithms. Various popular streaming and Video on Demand services such as Netflix, Amazon Prime Video, and Twitch use this method. Given this broad consumer base, ABR algorithms continuously improve to increase user satisfaction. The insights for these improvements are, among others, gathered within the research area of Quality of Experience (QoE). Within this field, various researchers have dedicated their works to identifying potential impairments and testing their impact on viewers’ QoE. Two frequently discussed visual impairments influencing QoE are stalling events and quality switches. So far, it is commonly assumed that those stalling events have the worst impact on QoE. This paper challenged this belief and reviewed this assumption by comparing stalling events with multiple quality and high amplitude quality switches. Two subjective studies were conducted. During the first subjective study, participants received a monetary incentive, while the second subjective study was carried out with volunteers. The statistical analysis demonstrated that stalling events do not result in the worst degradation of QoE. These findings suggest that a reevaluation of the effect of stalling events in QoE research is needed. Therefore, these findings may be used for further research and to improve current adaptation strategies in ABR algorithms.
A Novel Programmatic Hybrid 2D / 3D Image Segmentation and Labeling Methodology

Three-dimensional image segmentation is used in an increasing number of applications, from big-ticket Hollywood films to automated manufacturing to autonomous vehicles. Automated image segmentation relies upon a properly trained deep learning model, which in turn depends upon accurately labelled input. However, the labelling process is very time-intensive and does not lend itself to automation. We set out to automate the labelling process for our specific image content, using Microsoft Kinect RGBD video as input. We created a 3D image segmentation and labelling process that starts with 2D object detection and segmentation. This process is suitable for batch pipeline processing and real-time streaming video. Software demonstrating streaming video processing is shown.
Retinex by Autoencoders

The Retinex algorithms find wide applications as image enhancers, for their capability of preserving edges, while at the same time attenuating smooth gradients and chromatic dominants. They are characterized by the fact that the output chromatic intensity of a pixel is not determined in isolation (or looking only at the contiguous pixels) but through an operation of comparison to different local and remote areas of the image. This local/global comparison implies also a high computational cost for the algorithms: their complexity is not linear with the number of pixels; furthermore, the more systematic the comparison, the higher the complexity. Thus, most Retinex algorithms are unfit for real-time processing. The recent development of efficient Machine Learning architectures for Image Processing has raised the question of whether one of the Retinex "transforms" could be efficiently learned by training a feed-forward Artificial Neural Network, thus creating a model characterized by short processing time. Selecting a variant of the Random Spray Retinex model – FuzzyRSR – as representative of the Retinex family, and choosing suitably structured autoencoder neural networks, we found that we could accurately reproduce the Retinex effects. The computational cost of the training phase was moderate, while that of the inference phases was linear in the number of pixels, and three orders of magnitude lower than the one of FuzzyRSR, thus making the ANN implementation of Retinex suitable for real-time processing.
Protecting the Privacy of Face by De-Identification Pipeline Based on Deep Learning

In this research, we offer a face de-identification process that is reversible and alters both the face’s geometry and its texture. There are fourteen parameters that are employed for the change of the geometry. The usage of a fixed face texture template is required for texture change. We have explored how the capacity of people and robots to identify faces is affected when different geometrical and surface adjustments are made to components of the face such as the eyes, eyebrows, nose, and lips. The studies using crowdsourcing and machine face recognition were carried out on photographs of well-known persons that were obtained from the internet. The findings collected from both kinds of research indicated that alterations to a person’s facial geometry (form) are less influential on the amount of privacy protection than changes to that person’s face texture.
Per-Channel Image Super Resolution

Single Image Super Resolution (SISR) is one of the essential tasks in the image processing field, which aims to enhance the resolution of images. With the aid of Deep Learning (DL), SISR recently has made significant improvements and achieved promising results.This paper introduces a novel training technique, called Per-Channel training. This novel training technique proved to obtain the best results in terms of the common Super Resolution (SR) metrics across all testing datasets used, reaching a maximum average Peak Signal-to-Noise Ratio (PSNR) of 39.34 dB and Structural Similarity Index Measure (SSIM) of 0.9714 in one of the test datasets.
CNN-based network has Network Anisotropy -work harder to learn rotated feature than non-rotated feature

Successful object identification and classification in a generic Convolutional Neural Network (CNN) depends on object orientation. We expect CNN-based architectures to work harder to learn a rotated version of a feature than when learning the same feature in its default orientation. We name this phenomenon “Network Anisotropy”. A data set of 6000 RGB and grayscale images was created with rotated orientations of a feature predetermined and evenly distributed across four classes: 0°, 30°, 60°, 90°. Four ResNet (18, 34, 50, 101) classifier architectures were trained and the confidence scores were used to represent prediction accuracy. The results show that in all networks, training performance lags several epochs for the 30° and 60° rotation predictions compared to the 0° and 90° rotations, indicating a quantifiable network anisotropy. Because 0° and 90° both lie along a single rectilinear axis that coincides with the convolutional kernel of the CNN, we expect the classifier to do better on these two classes than on 30° and 60° classes. This work confirms that CNN architectures may have weaker performance based on feature orientation alone, independent of the feature distribution within the data set or the correlation of features within an image.