Reviewerly/exHarmony huggingface dataset:
Chairs Can Be Stood On: Overcoming Object Bias in Human-Object Interaction Detection

Detecting Human-Object Interaction (HOI) in images is an important step towards high-level visual comprehension. Existing work often shed light on improving either human and object detection, or interaction recognition. However, due to the limitation of datasets, these methods tend to fit well on frequent interactions conditioned on the detected objects, yet largely ignoring the rare ones, which is referred to as the object bias problem in this paper. In this work, we for the first time, uncover the problem from two aspects: unbalanced interaction distribution and biased model learning. To overcome the object bias problem, we propose a novel plug-and-play Object-wise Debiasing Memory (ODM) method for re-balancing the distribution of interactions under detected objects. Equipped with carefully designed read and write strategies, the proposed ODM allows rare interaction instances to be more frequently sampled for training, thereby alleviating the object bias induced by the unbalanced interaction distribution. We apply this method to three advanced baselines and conduct experiments on the HICO-DET and HOI-COCO datasets. To quantitatively study the object bias problem, we advocate a new protocol for evaluating model performance. As demonstrated in the experimental results, our method brings consistent and significant improvements over baselines, especially on rare interactions under each object. In addition, when evaluating under the conventional standard setting, our method achieves new state-of-the-art on the two benchmarks.
Driver State Detection from In-Car Camera Images

A non-neglectable number of car accidents are caused by driver’s loss of ability to drive the car, which may be caused by serious health problems, e.g. heart attack, stroke, drug or alcohol influence, as well as by drowsiness and other problems. In this paper, a method is presented for detecting the anomaly situations during driving. The method is based on detecting the particular parts of driver’s body in the sequence of images obtained from an in-car camera. A feature vector containing the distances between the body parts and describing the situation in a chosen number of frames is computed and used for detection. For the detection itself, the neural network of the autoencoder type containing the LSTM units is used. The method is compared with some other methods; the results show that the method is useful. Moreover, the video sequences used for training and testing are presented, which may be regarded as an additional contribution.
Comparison of Age, Gender and Ethnicity Prediction Using Traditional CNN and Transfer Learning

Analysis of facial features has attracted a lot of attention since the development of deep learning. There are numerous applications of demographic features such as age, gender and ethnicity but this topic has not been thoroughly explored by researchers especially when it comes to prediction of ethnicity. This paper aims to show the comparison between a custom CNN model and a pre-trained model. This article uses two models, a pre-trained MobileNet model[1] and a customized CNN model make up the first and second, respectively. Prediction accuracy and MAE are used to compare the results of each model that was put into practice. While accuracy is utilized for gender and ethnicity, MAE is employed as a performance metric for age. The models are analyzed using the UTKFace dataset[2].
Soft Weighted Ordinal Classification for Monocular Height Estimation in Remote Sensing Image

Estimating height information from a single remote sensing image is a critical component for 3D perception. Recent methods formulate it as a dense height prediction task based on regression loss functions. However, the regression accuracy is limited by the infinite continuous solution space. In this paper, we propose the soft weighted ordinal (SWO) classification loss for height prediction model to convert the regression problem with infinite continuous values into the classification problem with finite discrete values. which greatly improves the accuracy of high estimation. Specifically, we first define the discrete height rule and introduce the distance penalty metric to transform the continuous ground truth height value to the soft probability distributions. This is then used as supervised information to optimize the pixel-wise classification model. Finally, we utilize soft weighted summation to generate continuous height values in the inference phase. The proposed SWO classification loss can be used directly with existing dense prediction structures whose performance can be strengthened by direct replacement of the loss functions. Comprehensive experiments on the IS-PRS Vaihingen dataset show that the proposed method has achieved promising results.
Scene Aware Person Image Generation through Global Contextual Conditioning

Person image generation is an intriguing yet challenging problem. However, this task becomes even more difficult under constrained situations. In this work, we propose a novel pipeline to generate and insert contextually relevant person images into an existing scene while preserving the global semantics. More specifically, we aim to insert a person such that the location, pose, and scale of the person being inserted blends in with the existing persons in the scene. Our method uses three individual networks in a sequential pipeline. At first, we predict the potential location and the skeletal structure of the new person by conditioning a Wasserstein Generative Adversarial Network (WGAN) on the existing human skeletons present in the scene. Next, the predicted skeleton is refined through a shallow linear network to achieve higher structural accuracy in the generated image. Finally, the target image is generated from the refined skeleton using another generative network conditioned on a given image of the target person. In our experiments, we achieve high-resolution photo-realistic generation results while preserving the general context of the scene. We conclude our paper with multiple qualitative and quantitative benchmarks on the results.
CFDA: Collaborative Filtering with Dual Autoencoder for Recommender System

In recent years, deep neural networks have been widely used in recommender systems. Neural collaborative filtering is a popular work to model complex interactions between users and items with deep learning. However, methods that are based on collaborative filtering usually focus on learning embedding with the factorization of pairwise interactions, thereby causing embedding to be insufficient in capturing the complex relationships between users and items. To alleviate the above problem, in this paper, we propose a novel recommendation method based on collaborative filtering with dual autoencoder (CFDA). In the proposed method, we use dual autoencoder to learn hidden representations of users and items simultaneously, and we minimize the deviation of the training data by learning the user and item representations. Extensive experiments on several datasets demonstrate that the proposed method outperforms the baseline methods that are based on neural collaborative filtering.
LoADPart: Load-Aware Dynamic Partition of Deep Neural Networks for Edge Offloading

The emerging edge computing technique provides support for the computation tasks that are delay-sensitive and compute-intensive, such as deep neural network inference, by offloading them from a user-end device to an edge server for fast execution. The increasing offloaded tasks on an edge server are gradually facing the contention of both the network and computation resources. The existing offloading approaches often partition the deep neural network at a place where the amount of data transmission is small to save network resource, but rarely consider the problem caused by computation resource shortage on the edge server. In this paper, we design LoADPart, a deep neural network offloading system. LoADPart can dynamically and jointly analyze both the available network bandwidth and the computation load of the edge server, and make proper decisions of deep neural network partition with a light-weighted algorithm, to minimize the end-to-end inference latency. We implement LoADPart for MindSpore, a deep learning framework supporting edge AI, and compare it with state-of-the-art solutions in the experiments on 6 deep neural networks. The results show that under the variation of server computation load, LoADPart can reduce the end-to-end latency by 14.2% on average and up to 32.3% in some specific cases.
GridDehazeNet+: An Enhanced Multi-Scale Network With Intra-Task Knowledge Transfer for Single Image Dehazing

Adverse weather conditions such as haze can deteriorate the performance of autonomous driving and intelligent transport systems. As a potential remedy, we propose an enhanced multi-scale network, dubbed GridDehazeNet+, for single image dehazing. The proposed dehazing method does not rely on the Atmosphere Scattering Model (ASM), and an explanation as to why it is not necessarily performing the dimension reduction offered by this model is provided. GridDehazeNet+ consists of three modules: pre-processing, backbone, and post-processing. The trainable pre-processing module can generate learned inputs with better diversity and more pertinent features as compared to those derived inputs produced by hand-selected pre-processing methods. The backbone module implements multi-scale estimation with two major enhancements: 1) a novel grid structure that effectively alleviates the bottleneck issue via dense connections across different scales; 2) a spatial-channel attention block that can facilitate adaptive fusion by consolidating dehazing-relevant features. The post-processing module helps to reduce the artifacts in the final output. Due to domain shift, the model trained on synthetic data may not generalize well on real data. To address this issue, we shape the distribution of synthetic data to match that of real data, and use the resulting translated data to finetune our network. We also propose a novel intra-task knowledge transfer mechanism that can memorize and take advantage of synthetic domain knowledge to assist the learning process on the translated data. Experimental results demonstrate that the proposed method outperforms the state-of-the-art on several synthetic dehazing datasets, and achieves the superior performance on real-world hazy images after finetuning.
Bilateral Video Magnification Filter

Eulerian video magnification (EVM) has progressed to magnify subtle motions with a target frequency even under the presence of large motions of objects. However, existing EVM methods often fail to produce desirable results in real videos due to (1) misextracting subtle motions with a non-target frequency and (2) collapsing results when large de/acceleration motions occur (e.g., objects suddenly start, stop, or change direction). To enhance EVM performance on real videos, this paper proposes a bilateral video magnification filter (BVMF) that offers simple yet robust temporal filtering. BVMF has two kernels; (I) one kernel performs temporal bandpass filtering via a Laplacian of Gaussian whose passband peaks at the target frequency with unity gain and (II) the other kernel excludes large motions outside the magnitude of interest by Gaussian filtering on the intensity of the input signal via the Fourier shift theorem. Thus, BVMF extracts only subtle motions with the target frequency while excluding large motions outside the magnitude of interest, regardless of motion dynamics. In addition, BVMF runs the two kernels in the temporal and intensity domains simultaneously like the bilateral filter does in the spatial and intensity domains. This simplifies implementation and, as a secondary effect, keeps the memory usage low. Experiments conducted on synthetic and real videos show that BVMF outperforms state-of-the-art methods.
A Novel Hybrid Endoscopic Dataset for Evaluating Machine Learning-Based Photometric Image Enhancement Models

Endoscopy is the most widely used medical technique for cancer and polyp detection inside hollow organs. However, images acquired by an endoscope are frequently affected by illumination artefacts due to the enlightenment source orientation. There exist two major issues when the endoscope’s light source pose suddenly changes: overexposed and underexposed tissue areas are produced. These two scenarios can result in misdiagnosis due to the lack of information in the affected zones or hamper the performance of various computer vision methods (e.g., SLAM, structure from motion, optical flow) used during the non invasive examination. The aim of this work is two-fold: i) to introduce a new synthetically generated data-set generated by a generative adversarial techniques and ii) and to explore both shallow based and deep learning-based image-enhancement methods in overexposed and underexposed lighting conditions. Best quantitative results (i.e., metric based results), were obtained by the deep learning-based LMSPEC method, besides a running time around 7.6 fps. Data available at: https://data.mendeley.com/datasets/3j3tmghw33/1 .
DyHealth

In National University Hospital (NUH) in Singapore, we conduct healthcare analytics that analyzes heterogeneous electronic medical records (EMR) to support effective clinical decision-making on a daily basis. Existing work mainly focuses on multimodality for extracting complementary information from different modalities, and/or interpretability for providing interpretable prediction results. However, real-world healthcare analytics has presented another major challenge, i.e., the available modalities evolve or change intermittently. Addressing this challenge requires deployed models to be adaptive to such dynamic modality changes. To meet the aforementioned requirement, we develop a modular, multimodal and interpretable framework DyHealth to enable dynamic healthcare analytics in clinical practice. Specifically, different modalities are processed within their respective data modules that adhere to the interface defined by DyHealth. The extracted information from different modalities is integrated subsequently in our proposed Multimodal Fusion Module in DyHealth. In order to better handle modality changes at runtime, we further propose exponential increasing/decreasing mechanisms to support modality "hot-plug". We also devise a novel modality-based attention mechanism for providing fine-grained interpretation results on a per-input basis. We conduct a pilot evaluation of DyHealth on the patients' EMR data from NUH, in which DyHealth achieves superior performance and therefore, is promising to roll out for hospital-wide deployment. We also validate DyHealth in two public EMR datasets. Experimental results confirm the effectiveness, flexibility, and extensibility of DyHealth in supporting multimodal and interpretable healthcare analytics.
Identifying Hard Noise in Long-Tailed Sample Distribution

Conventional de-noising methods rely on the assumption that all samples are independent and identically distributed, so the resultant classifier, though disturbed by noise, can still easily identify the noises as the outliers of training distribution. However, the assumption is unrealistic in large-scale data that is inevitably long-tailed. Such imbalanced training data makes a classifier less discriminative for the tail classes, whose previously “easy” noises are now turned into “hard” ones—they are almost as outliers as the clean tail samples. We introduce this new challenge as Noisy Long-Tailed Classification (NLT). Not surprisingly, we find that most de-noising methods fail to identify the hard noises, resulting in significant performance drop on the three proposed NLT benchmarks: ImageNet-NLT, Animal10-NLT, and Food101-NLT. To this end, we design an iterative noisy learning framework called Hard-to-Easy (H2E). Our bootstrapping philosophy is to first learn a classifier as noise identifier invariant to the class and context distributional changes, reducing “hard” noises to “easy” ones, whose removal further improves the invariance. Experimental results show that our H2E outperforms state-of-the-art de-noising methods and their ablations on long-tailed settings while maintaining a stable performance on the conventional balanced settings. Datasets and codes are available at https://github.com/yxymessi/H2E-Framework .
Enveloped Sinusoid Parseval Frames

This paper presents a method of constructing Parseval frames from any collection of complex envelopes. The resulting Enveloped Sinusoid Parseval (ESP) frames can represent a wide variety of signal types as specified by their physical morphology. Since the ESP frame retains its Parseval property even when generated from a variety of envelopes, it is compatible with large scale and iterative optimization algorithms. The ESP construction provides an analysis and synthesis transform pair that can be tuned according to the characteristics of the signal to which it is applied. This work provides examples of ESP frame generation for both synthetic and experimentally measured signals. Furthermore, the frame’s compatibility with distributed sparse optimization frameworks is demonstrated. Numerical experiments on acoustics data reveal that the flexibility of this method allows it to be simultaneously competitive with a signal agnostic short-time Fourier Transform in time-frequency processing and also with Prony’s Method for time-constant parameter estimation, surpassing the shortcomings of each individual technique.
Topology of the documentation landscape

Every software system (ideally) comes with one or more forms of documentation. Beside source code comments, other structured and unstructured sources (e.g., design documents, API references, wikis, usage examples, tutorials) constitute critical assets. Cloud-based repositories for collaborative development (e.g., GitHub, Bitbucket, GitLab) provide many functionalities to create, persist, and version documentation artifacts. On the other hand, the last decade has seen the rise of rich instant messaging clients used as global software community platforms (e.g., Slack, Discord). Although completely detached from a specific versioning system or development workflow, they allow developers to discuss implementation issues, report bugs, and, in general, interact with one another.
Multi-task Deep Learning for Cerebrovascular Disease Classification and MRI-to-PET Translation

Accurate cerebral blood flow (CBF) quantification is essential to diagnose and treat many cerebrovascular diseases, including Moyamoya, carotid stenosis, and stroke. The gold standard for CBF (oxygen-15 water positron emission tomography, PET) is not widely available because of its high cost, use of ionizing radiation, and logistical challenges as compared to magnetic resonance imaging (MRI). In this study, using simultaneous PET/MRI, we propose a multi-task learning framework for brain MRI-to-PET translation and disease classification. The proposed framework comprises two prime networks: (1) an attention-based 3D convolutional encoder-decoder network to synthesize high-quality PET CBF maps from multi-contrast MRI images, and (2) a multi-scale 3D convolutional network to identify the brain disease corresponding to the input MRI images. Our multi-task framework yields promising results on the task of MRI-to-PET translation, achieving an average structural similarity index of 0.94 and peak signal-to-noise ratio of 38dB on a cohort of 120 subjects. In addition, we show that integrating multiple MRI modalities can improve the clinical diagnosis of brain diseases. As such, deep learning offers the possibility to perform high-quality CBF measurements and disease classification for patients with cerebrovascular disease at MRI-only sites, leading to improved and more equitable patient care.
VIsCUIT: Visual Auditor for Bias in CNN Image Classifier

CNN image classifiers are widely used, thanks to their efficiency and accuracy. However, they can suffer from biases that impede their practical applications. Most existing bias investigation techniques are either inapplicable to general image classification tasks or require significant user efforts in perusing all data subgroups to manually specify which data attributes to inspect. We present VIsCUIT, an interactive visualization system that reveals how and why a CNN classifier is biased. VIsCUIT visually summarizes the subgroups on which the classifier underperforms and helps users discover and characterize the cause of the underperformances by revealing image concepts responsible for activating neurons that contribute to misclassifications. VIsCUIT runs in modern browsers and is opensource, allowing people to easily access and extend the tool to other model architectures and datasets. VIsCUIT is available at the following public demo link: https://poloclub.github.io/VisCUIT. A video demo is available at https://youtu.be/eNDbSyM4R_4.
Misbehavior Detection in Vehicular Ad Hoc Networks Based on Privacy-Preserving Federated Learning and Blockchain

As an irreversible trend, connected vehicles have become increasingly more popular. They depend on the generation and sharing of data between vehicles to improve safety and efficiency of the transportation system. Due to the open feature of the vehicular ad hoc network (VANET), it is possible for dishonest and misbehaving vehicles to disrupt traffic by transmitting false information. In recent years, misbehavior detection systems have been developed to detect the malicious behaviour, and machine learning methods have been employed to make the detection more accurately. However, existing misbehavior detection systems typically require a single entity (e.g., a central server) for centralized data collection and training. Model updates are restricted due to data privacy and high overhead of data communication, which reduces the defensive capability of misbehavior detection systems. In this paper, we propose a blockchain-based federated learning scheme to detect misbehavior, which is trained collaboratively by coordinating multiple distributed edge devices while ensuring data security and privacy. In addition, to further protect the privacy of the model on the blockchain, differential privacy with the Gaussian mechanism is leveraged to provide strict privacy protection. Common data falsification attacks are studied in this paper. The experimental results show that our proposed scheme is feasible and effective, and demonstrate that our scheme achieves satisfied accuracy and efficiency.
Text Classification via Learning Semantic Dependency and Association

Text classification is a fundamental and classical problem in natural language processing. Existing methods in this area attach more attention to structure modeling of texts, while largely ignoring the cognitive principles of human reading. Actually, as an important aspect of exploring the characteristics of language comprehension, neuroscience research in recent years has demonstrated the human instinct for abstract thinking, where semantic processing and summarizing play essential roles. To this end, we propose a novel text classification method with Semantic Dependency and Association (SDA). To be specific, SDA comprises two coupled spaces, namely Dependency Space and Association Space, to imitate the real process when readers are inferring text semantics. Firstly, in Dependency Space, we devise a graph modeling structure to thoroughly capture the dependency relations between words. Then in Association Space, we associate the words with prior concepts and adopt attention mechanisms to match words with the most related concepts based on their dependency relations. Later, we synthesize the above two aspects to enhance the semantic representation of words. Afterwards, we design a Semantic Summarizing module to integrate the semantic information learned above. Finally, extensive experiments are conducted on four public English and Chinese text classification datasets, where experimental results not only demonstrate the effectiveness of our model but also reveal the superiority of the perspective from the human cognition for text comprehension.