Reviewerly/exHarmony huggingface dataset:
A Wearable-based Preventive Model To Promote Oral Health Through Personalized Notification

Wearable technology has great potential to develop human-centric healthcare applications. It reshapes our lifestyle by providing information related to physical activities, sleep monitoring, or heartbeat rhythms. In this paper, we present an innovative model for oral healthcare, which is prevention-focused to notify individuals about cleaning teeth activity. It is based on a wrist-worn accelerometer device and has two components-first, a computationally lightweight feature extractor-secondly, a robust feed-forward neural network to recognize the cleaning teeth activity. The model performance is measured using standard performance metric F1 - score (i.e., 98%), which shows the applicability in a real-life scenario. The trained model can reside inside the smartwatch as a wrist-worn wearable. It would generate personalized notifications if s/he skipped the toothbrush activity. Furthermore, it notifies the users to change the toothbrush after three months, reducing the cognitive burden. Clinical Relevance-This work help the individuals to maintain their oral health using smartwatches as a wearable device.
Using Gated Recurrent Unit Networks for the Prediction of Hemodynamic and Pulmonary Decompensation

This paper presents a new medical severity scoring system, used to assess the risk of hemodynamic and pulmonary decompensation for patients being treated in intensive care units. The score presented here includes drug circulatory support and ventilation mode data for the evaluation of the patient's biosignals and laboratory values. It is shown that Gated Recurrent Unit-based neural networks are able to predict the maximal severity class within a 24 hour prediction time-frame (hemodynamic: 0.85 AUROC / pulmonary: 0.9 AUROC), and can estimate the underlying decompensation score for prediction times of up to 24 hours with mean errors of 6.3% of the maximal possible pulmonary, and 9.6% of the hemodynamic score. These results are based on 60h observation period. Clinical Relevance- Hemodynamic and pulmonary decom-pensation are life threatening dynamic events that can lead to death of patients. Early detection of these incidents is essential in order to intervene therapeutically and to improve survival chances. In everyday intensive care physicians are confronted with a vast number of laboratory values and vital parameters. There is a risk that early stages of hemodynamic and pulmonary decompensation are misjudged. The implementation of robust warning systems could support physicians in detecting these critical events and initiate therapeutical intervention in time which would achieve significant reduction of patient mortality.
A 3D Convolutional Neural Network with Gradient Guidance for Image Super-Resolution of Late Gadolinium Enhanced Cardiac MRI

In this paper, we describe a 3D convolutional neural network (CNN) framework to compute and generate super-resolution late gadolinium enhanced (LGE) cardiac magnetic resonance imaging (MRI) images. The proposed CNN framework consists of two branches: a super-resolution branch with a 3D dense deep back-projection network (DBPN) as the backbone to learn the mapping of low-resolution LGE cardiac volumes to high-resolution LGE cardiac volumes, and a gradient branch that learns the mapping of the gradient map of low resolution LGE cardiac volumes to the gradient map of their high-resolution counterparts. The gradient branch of the CNN provides additional cardiac structure information to the super-resolution branch to generate structurally more accurate super-resolution LGE MRI images. We conducted our experiments on the 2018 atrial segmentation challenge dataset. The proposed CNN framework achieved a mean peak signal-to-noise ratio (PSNR) of 30.91 and 25.66 and a mean structural similarity index measure (SSIM) of 0.91 and 0.75 on training the model on low-resolution images downsamp led by a scale factor of 2 and 4, respectively.
Rethinking ImageNet Pre-training for Computational Histopathology

Transfer learning from ImageNet pretrained weights is widely used when training Deep Learning models on a Histopathology dataset. However, the visual features of the two domains are different. Rather than ImageNet pretrained weights, pre-training on a Histopathology dataset may provide better initialization. To prove this hypothesis, we train two commonly used Deep Learning model architectures - ResNet and DenseNet on a complex Histopathology classification dataset, and compare transfer learning performance with ImageNet pretrained weights. Based on the fine-tuning on three histopathology datasets including two different stains (H&E and IHC), we show that the domain specific pretrained weights are better suited for transfer learning. This is reflected by higher performance, lower training time as well as better feature reuse. Clinical Relevance - The paper establishes merit of using Histopathology domain specific pretrained weights rather than ImageNet pretrained weights.
A Deep Learning Sequential-based Model for Predicting Victories in Video Games

This paper proposes an approach that uses a sequential dataset generated from game logs to feed into deep neural networks to predict victories. Among the six different deep neural networks implemented, a Simplified Fully Convolutional Network model achieved the best performance with an Area Under the Curve score of 0.8447. Although our model did not surpass the performance of the best challenge models (IEEE-BigData 2021), but the approach has two significant advantages: the limited need for feature engineering and greater adaptability and generalization to other video games. Furthermore, when considering only one feature (i.e. HP), our approach is better (AUC score 0.802) compared to the best challenge model (AUC score 0.72).
GARGI: Selecting Gaze-Aware Representative Group Image from a Live Photo

The number of photos, especially group photos in live mode, has increased tremendously in today's world. Selecting a representative image in a live photo that preserves the aesthetic quality is a challenging task. In this paper, we propose a method to select a Gaze-Aware Representative Group Image, called GARGI, that considers the uni-formity, or consequently the deviation, of the people's gaze in live-mode group photos to make it aesthetically pleasing. We tested this method on our own live-mode group image dataset. We argue that the inbuilt representative im-age selection mechanism in an Apple iPhone does not con-sider the subject's gaze, especially in a group image. The GARGI considers the deviation of gazes for each subject with respect to their expected gaze directions and deter-mines an aesthetically better representative image with the least amount of gaze deviation for all the subjects. The re-sults presented in the paper also justify this claim. They can be used to pave the way for becoming a standard in any keyframe selection mechanisms that will include human subjects in live photos, burst mode shots, or even in videos.
Deep Video Stream Information Analysis and Retrieval: Challenges and Opportunities

Deep Learning (DL) provided powerful tools for various visual information analysis and retrieval tasks, outperforming previously used methods. However, despite the potential of such approaches for various tasks, applying them in video stream applications, such as media monitoring or surveillance, where a large number of streams should be processed in parallel, is not trivial and comes with several challenges. This paper aims to provide a brief overview of the current state-of-the-art in DL tools that can be used for deep video stream information analysis and retrieval. Apart from a review of the current literature, we also include experimental results discussing deployment challenges, ranging from speed to energy consumption, demonstrating the capabilities of readily available commodity hardware in processing video streams for selected DL models.
Cross-Domain Knowledge Transfer for Skeleton-based Action Recognition based on Graph Convolutional Gradient Reversal Layer

For skeleton-based action recognition, since there usually exists many nuances between different datasets, including viewpoints, the number of available joints for a skele-ton, the type of actions, etc, it hinders to apply and leverage the knowledge of a pretrained model for one dataset to an-other except retraining a new model for the target dataset. To address this issue, we propose a cross-domain knowledge transfer module based on gradient reversal layer along with adaptive graph convolutional network to effectively transfer the knowledge from one domain to another. The adaptive graph convolution module allows the proposed method to adaptively learn the topological relation between joints and is very useful for the scenarios when the numbers of skele-ton joints for the two domains are different and the topo-logical correspondences of joints are not clearly specified. With extensive experiments from NTU-RGB+D 60 to the PKU, CITI3D, and NW datasets, the proposed approach achieves significantly better results than other state-of-the-art spatio-temporal graph convolutional network methods which are trained on the target dataset only, and this also demonstrates the effectiveness of the proposed approach.
Video Origin Camera Identification using Ensemble CNNs of Positional Patches

The use of multimedia devices has increased immensely with the availability of affordable mobile loT technologies. Consequently, video capturing applications are incorporated in many crucial sectors including crime investigations and surveillance. In such applications, it is important to ensure the integrity of the video data and verify its source for digital forensics purposes. In this paper, we tackle the problem of video source camera identification to validate the origin of video data that may come from numerous types of cameras. Specifically, we propose a novel approach based on ensemble of convolutional neural networks (CNNs) for detection of video source. In our approach, the video is analyzed using patches obtained from I-frames after splitting each frame into quadrants and training a CNN per quadrant for location awareness. Our experimental results demonstrate that our model brings significant improvement for video source identification accuracy compared to existing machine learning based techniques.
Blockchain-Oriented Requirements Engineering: New Directions

The literature on blockchain-oriented software engineering is still emerging. Researchers have investigated the role of blockchains in software engineering. In this paper, we advance the area of blockchain-oriented software engineering by focusing on new directions in requirements engineering (RE). This paper presents the best-known approaches to provide core blockchain technology and introduces a brand-agnostic way to group and compare the various approaches. We then use this grouping to measure how well the blockchain variants address certain key non-functional requirements (NFRs) and identify the NFRs that are instrumental to software engineers for decision making. Our work not only sheds light on new directions in blockchain-oriented RE, but also assist software engineers in the RE process.
Preserving Data Privacy and Accuracy of Human Pose Estimation Software Based on CNN s for Remote Gait Analysis

In the last years there have been significant improvements in the accuracy of real-time 3D skeletal data estimation software. These applications based on convolutional neural networks (CNNs) can playa key role in a variety of clinical scenarios, from gait analysis to medical diagnosis. One of the main challenges is to apply such intelligent video analytic at a distance, which requires the system to satisfy, beside accuracy, also data privacy. To satisfy privacy by default and by design, the software has to run on "edge" computing devices, by which the sensitive information (i.e., the video stream) is elaborated close to the camera while only the process results can be stored or sent over the communication network. In this paper we address such a challenge by evaluating the accuracy of the state-of-the-art software for human pose estimation when run "at the edge". We show how the most accurate platforms for pose estimation based on complex and deep neural networks can become inaccurate due to subs amp ling of the input video frames when run on the resource constrained edge devices. In contrast, we show that, starting from less accurate and "lighter" CNNs and enhancing the pose estimation software with filters and interpolation primitives, the platform achieves better real-time performance and higher accuracy with a deviation below the error tolerance of a marker-based motion capture system.
Weakly Supervised Temporal Action Localization Through Contrastive Learning

In recent years, weakly-supervised temporal action localization (WS-TAL) with only video-level annotations, which aims to learn whether each untrimmed video contains action frames gains more attention. Existing most WS-TAL methods especially rely on features learned for action localization. Therefore, it is important to improve the ability to separate the frames of action instances from the background frames. To address this challenge, this paper introduces a framework that learns two extra constraints, Action-Background Learning and Action-Foreground Learning. The former aims at maximizing the discrepancy inside the feature of action and background while the latter avoids the misjudgement of action instance. We evaluate the proposed model on two benchmark datasets, and the experimental results show that the method could gain comparable performance with current state-of-the-art WS-TAL methods.
Improving GIF Image Accessibility on Web

GIF (Graphics Interchange Format) images have become ubiquitous in modern Internet lingo, but they are often inaccessible to people with vision impairments. This paper presents a project that aims to improve the accessibility of GIF images on web pages by auto-injecting commentaries into their alt text tag using machine learning and web browser extension techniques. This system uses a CNN-LSTM architecture to generate descriptive texts for GIF images, which outperforms the state-of-the-art Tumblr-GIF research with 7% improvement over BLEU accuracy metric, 6% improvement over ROUGE, and 10% improvement over METEOR. Usability studies performed with a focus group of individuals with and without vision disabilities further show that the system performs well for its targeted audience.
Skipped-Connection Transformer for Image Captioning

The attention mechanism in neural networks can reflect the way in which humans understand things. Transformer, which is based on attention, performs well in both natural language processing and computer vision fields, and even in multimodal fields, such as image captioning. However, there is still scope for improvement in the field of image captioning. We use meshed-memory Transformer as the backbone and propose an improved method by simultaneously integrating region features and grid features and then adding skipped connections, similar to a residual network (ResNet). In addition, we comprehensively compare the performance of our model with existing models on the COCO dataset. Experiments show that, on the Karpathy test split, our model outperforms meshed-memory Transformer on all evaluation metrics: e.g., it increases CIDEr from 131.2% to 133.3%, which is even competitive compared with the latest works published in 2021.
Pooling Pyramid Vision Transformer for Unsupervised Monocular Depth Estimation

Compared with other sensors, high-quality depth estimation based on monocular camera has strong competitiveness and widespread application in intelligent transportation, etc. Although the barrier of training has been greatly lowered by unsupervised learning, most related works are still based on convolutional neural networks (CNNs) that suffer from unbridgeable gaps in the full-stage global information and high-resolution features while extracting multi-scale features. To break this predicament, we attempt to introduce vision transformer. However, the vision transformer with large sequence length due to image embedding brings great challenges to the computational cost. Thus, this work proposes a new pure transformer backbone named pooling pyramid vision transformer (PPViT), simultaneously shrinking out multi-scale features and reducing sequence length used for attention operation. Then, we provide two backbone settings including PPViT10 and PPViT18 whose number of parameters are close to the common ResNet18 and ResNet50, respectively. The experiments on KITTI dataset demonstrate that our work show a great potentiality of improving the capability of model and produce superior results to the previous CNN-based works. Equally important, we have lower latency than the related transformer-based work.
Disparity-Guided Light Field Video Synthesis with Temporal Consistency

Light field has great applications in AR/VR. It is particu-larly usefulfor resolving the vergence-accommodation con-flict (VAC) and creating correct depth cues for AR/VR dis-plays. However, the source data, especially light field video, are not widely available yet. To resolve the scarcity is-sue, one may resort to data such as stereo image sequences that are commonly available. In this paper, we propose an end-to-end deep learning framework for synthesizing light field sequences from stereo image sequences. Our frame-work consists of a disparity estimation network, a guided synthesis network, and a refinement network and is able to resolve the flickering issue caused by temporal incon-sistency, an artifact that is commonly seen in synthesized light field videos. Our experimental results are quantitatively and qualitatively better than the results of existing light field synthesis algorithms that were originally developed for static light fields.
Unsupervised Generated Image Editing Method Based on Multi-Scale Hierarchical Disentanglement

In order to solve the problem of semantic entanglement in generated image latent space of the StyleGAN2 network, we propose an unsupervised generated image editing method based on a multi-scale hierarchical disentanglement network structure. In our method, we first combine the mapping layer with the style mapping layer of each resolution branch in the StyleGAN2 network, and utilize the weight matrix eigen decomposition method at each scale independently to achieve the first-level disentanglement of image attributes and obtain the semantic direction vector of the scale. Then, we use Schmidt orthogonal decomposition based on the adjacent scale eigen vector to achieve the second-level disentanglement of image attributes. The result show that, compared with other mainstream unsupervised image editing methods, our method can achieve precise image editing at multiple scales, and the measurement of disentanglement between each attribute has also reached the best.
Bouncing Robots in Rectilinear Polygons

In this paper, we describe a bouncing strategy (smart strategy) for a mobile robot that uses one bit of memory for feedback, and guarantees that the robot will traverse all the rooms (and doorways) of a 2D environment. The environment is modeled as a rectilinear polygon (also called orthogonal polygon), and the rooms and the doorways are defined by the decomposition algorithm we describe. Such a decomposition helps the robot to not go back to a room after leaving. We also define the notion of “virtual doors” that have the ability to let the robot through, or make the robot bounce from them. We compared three different types of bouncing rules: smart, random, billiard. The smart strategy grantees to reach to target. Although the random strategy on average behaves the same as the smart strategy, there are rectilinear polygons in which the robot cannot reach the target in the expected time steps. On the other hand, the billiard bouncing strategy can cause the robot to become trapped.
A Heatmap-Based Approach for Analyzing Traffic Sign Recognition and Lane Detection Algorithms

A heat map (or heatmap) is an efficient and convenient way to condense the information into a single snapshot. It can also be used in the analysis of high-dimensional data structures that are representative of large data sets generated by vehicle sensory systems. In this paper, we show how to generate such heat maps from vision data recorded by the front camera in the vehicle. We present two algorithms that can be applied to build heat maps that represent the detection status of traffic signs and road lanes. The resulting graphs can then be used to evaluate in a qualitative and quantitative way the Traffic Sign Recognition and Lane Detection functionalities that are part of Advanced Driver Assistance Systems. The effectiveness of the presented approach is illustrated in several use cases of applications.
Deep Learning-Based Job Placement in Distributed Machine Learning Clusters With Heterogeneous Workloads

Nowadays, most leading IT companies host a variety of distributed machine learning (ML) workloads in ML clusters to support AI-driven services, such as speech recognition, machine translation, and image processing. While multiple jobs are executed concurrently in a shared cluster to improve resource utilization, interference among co-located ML jobs can lead to significant performance downgrade. Existing cluster schedulers, such as YARN and Mesos, are interference-agnostic in their job placement, leading to suboptimal resource efficiency and usage. Some literature has studied interference-aware job placement policy, but relies on detailed workload profiling and interference modeling, which is not a general solution. In this work, we present Harmony, a deep learning-driven ML cluster scheduler that places heterogeneous training jobs (either with parameter server architecture or all-reduce architecture) in a manner that minimizes interference and maximizes performance (i.e., training completion time minimization). The design of Harmony is based on a carefully designed deep reinforcement learning (DRL) framework enhanced with reward modeling. The DRL integrates a dynamic sequence-to-sequence model with the state-of-the-art techniques to stabilize training and improve convergence, including actor-critic algorithm, job-aware action space exploration, multi-head attention, and experience replay. In view of a common lack of reward samples corresponding to different placement decisions, we build an auxiliary sequence-to-sequence reward prediction model, which is trained with historical samples and used for producing reward for unseen placement. Experiments using real ML workloads in a Kubernetes cluster of 6 GPU servers show that Harmony outperforms representative schedulers by 16%–42% in terms of average job completion time.
Counterexample-Driven Genetic Programming for Symbolic Regression With Formal Constraints

In symbolic regression with formal constraints, the conventional formulation of regression problem is extended with desired properties of the target model, like symmetry, monotonicity, or convexity. We present a genetic programming algorithm that solves such problems using a satisfiability modulo theories solver to formally verify the candidate solutions. The essence of the method consists in collecting the counterexamples resulting from model verification and using them to improve search guidance. The method is exact upon successful termination, the produced model is guaranteed to meet the specified constraints. We compare the effectiveness of the proposed method with standard constraint-agnostic machine learning regression algorithms on a range of benchmarks and demonstrate that it outperforms them on several performance indicators.