Reviewerly/exHarmony huggingface dataset:
Deep Symmetric Network for Underexposed Image Enhancement with Recurrent Attentional Learning

Underexposed image enhancement is of importance in many research domains. In this paper, we take this problem as image feature transformation between the underexposed image and its paired enhanced version, and we propose a deep symmetric network for the issue. Our symmetric network adapts invertible neural networks (INN) for bidirectional feature learning between images, and to ensure the mutual propagation invertible we specifically construct two pairs of encoder-decoder with the same pretrained parameters. This invertible mechanism with bidirectional feature transformations enable us to both avoid colour bias and recover the content effectively for image enhancement. In addition, we propose a new recurrent residual-attention module (RRAM), where the recurrent learning network is designed to gradually perform the desired colour adjustments. Ablation experiments are executed to show the role of each component of our new architecture. We conduct a large number of experiments on two datasets to demonstrate that our method achieves the state-of-the-art effect in underexposed image enhancement. Code is available at https://www.shaopinglu.net/proj-iccv21/ImageEnhancement.html.
Constrained Contrastive Distribution Learning for Unsupervised Anomaly Detection and Localisation in Medical Images

Unsupervised anomaly detection (UAD) learns one-class classifiers exclusively with normal (i.e., healthy) images to detect any abnormal (i.e., unhealthy) samples that do not conform to the expected normal patterns. UAD has two main advantages over its fully supervised counterpart. Firstly, it is able to directly leverage large datasets available from health screening programs that contain mostly normal image samples, avoiding the costly manual labelling of abnormal samples and the subsequent issues involved in training with extremely class-imbalanced data. Further, UAD approaches can potentially detect and localise any type of lesions that deviate from the normal patterns. One significant challenge faced by UAD methods is how to learn effective low-dimensional image representations to detect and localise subtle abnormalities, generally consisting of small lesions. To address this challenge, we propose a novel self-supervised representation learning method, called Constrained Contrastive Distribution learning for anomaly detection (CCD), which learns fine-grained feature representations by simultaneously predicting the distribution of augmented data and image contexts using contrastive learning with pretext constraints. The learned representations can be leveraged to train more anomaly-sensitive detection models. Extensive experiment results show that our method outperforms current state-of-the-art UAD approaches on three different colonoscopy and fundus screening datasets. Our code is available at https://github.com/tianyu0207/CCD.
Using YOLOv5 for Garbage Classification

At present, people's daily garbage is increasing day by day. How to intelligently classify garbage can save manpower and improve work efficiency. In this paper, a garbage classification model based on YOLOv5 object detection network named GC-YOLOv5 is designed. First, according to the common daily garbage category, five typical kinds of garbage were selected, data cleaned, labeled, and constructed a garbage dataset. Second, the GC-YOLOv5 was built and trained on our datasets. Third, in view of the convenience of multi-terminal access in the cloud and the reduction of computing pressure on edge devices, we deploy the garbage classification model in the cloud. The experimental results show that GC-YOLOv5 can accurately identify the garbage's types and find out the location of garbage.
Deepfake tweets classification using stacked Bi-LSTM and words embedding

The spread of altered media in the form of fake videos, audios, and images, has been largely increased over the past few years. Advanced digital manipulation tools and techniques make it easier to generate fake content and post it on social media. In addition, tweets with deep fake content make their way to social platforms. The polarity of such tweets is significant to determine the sentiment of people about deep fakes. This paper presents a deep learning model to predict the polarity of deep fake tweets. For this purpose, a stacked bi-directional long short-term memory (SBi-LSTM) network is proposed to classify the sentiment of deep fake tweets. Several well-known machine learning classifiers are investigated as well such as support vector machine, logistic regression, Gaussian Naive Bayes, extra tree classifier, and AdaBoost classifier. These classifiers are utilized with term frequency-inverse document frequency and a bag of words feature extraction approaches. Besides, the performance of deep learning models is analyzed including long short-term memory network, gated recurrent unit, bi-direction LSTM, and convolutional neural network+LSTM. Experimental results indicate that the proposed SBi-LSTM outperforms both machine and deep learning models and achieves an accuracy of 0.92.
Remote Sensing Image Target Detection: Improvement of the YOLOv3 Model with Auxiliary Networks

Remote sensing image target detection is widely used for both civil and military purposes. However, two factors need to be considered for remote sensing image target detection: real-time and accuracy for detecting targets that occupy few pixels. Considering the two above issues, the main research objective of this paper is to improve the performance of the YOLO algorithm in remote sensing image target detection. The reason is that the YOLO models can guarantee both detection speed and accuracy. More specifically, the YOLOv3 model with an auxiliary network is further improved in this paper. Our model improvement consists of four main components. Firstly, an image blocking module is used to feed fixed size images to the YOLOv3 network; secondly, to speed up the training of YOLOv3, DIoU is used, which can speed up the convergence and increase the training speed; thirdly, the Convolutional Block Attention Module (CBAM) is used to connect the auxiliary network to the backbone network, making it easier for the network to notice specific features so that some key information is not easily lost during the training of the network; and finally, the adaptive feature fusion (ASFF) method is applied to our network model with the aim of improving the detection speed by reducing the inference overhead. The experiments on the DOTA dataset were conducted to validate the effectiveness of our model on the DOTA dataset. Our model can achieve satisfactory detection performance on remote sensing images, and our model performs significantly better than the unimproved YOLOv3 model with an auxiliary network. The experimental results show that the mAP of the optimised network model is 5.36% higher than that of the original YOLOv3 model with the auxiliary network, and the detection frame rate was also increased by 3.07 FPS.
Convergence Analysis and System Design for Federated Learning Over Wireless Networks

Federated learning (FL) has recently emerged as an important and promising learning scheme in IoT, enabling devices to jointly learn a model without sharing their raw data sets. As FL does not collect and store the data centrally, it requires frequent model exchange through the wireless network. However, since the aggregation in FL can be partially participated with synchronized frequency, its communication pattern is different from the conventional network. Therein, limited bandwidth and package loss restrict interactions in training. Thus, the network scheduling could largely affect the FL convergence. To figure out the specific effects, we analyze the convergence rate of FL regarding the joint impact of communication and training. Combining it with the network model, we formulate the optimal scheduling problem for FL implementation. The theoretical results could guide the hyper-parameter design in the network and explain the principle of how the wireless communication could influence the FL training process.
Automated detection and grading of Invasive Ductal Carcinoma breast cancer using ensemble of deep learning models

Invasive ductal carcinoma (IDC) breast cancer is a significant health concern for women all around the world and early detection of the disease may increase the survival rate in patients. Therefore, Computer-Aided Diagnosis (CAD) based systems can assist pathologists to detect the disease early. In this study, we present an ensemble model to detect IDC using DenseNet-121 and DenseNet-169 followed by test time augmentation (TTA). The model achieved a balanced accuracy of 92.70% and an F1-score of 95.70% outperforming the current state-of-the-art. Comparative analysis against various pre-trained deep learning models and preprocessing methods have been carried out. Qualitative analysis has also been conducted on the test dataset. After the detection of IDC breast cancer, it is important to grade it for further treatment. In our study, we also propose an ensemble model for the grading of IDC using the pre-trained DenseNet-121, DenseNet-201, ResNet-101v2, and ResNet-50 architectures. The model is inferred from two validation cohorts. For the patch-level classification, the model yielded an overall accuracy of 69.31%, 75.07%, 61.85%, and 60.50% on one validation cohort and 62.44%, 79.14%, 76.62%, and 71.05% on the second validation cohort for 4×, 10×, 20×, and 40× magnified images respectively. The same architecture is further validated using a different IDC dataset where it achieved an overall accuracy of 90.07%. The performance of the models on the detection and grading of IDC shows that they can be useful to help pathologists detect and grade the disease.
Bi-Modal Transfer Learning for Classifying Breast Cancers via Combined B-Mode and Ultrasound Strain Imaging

Although accurate detection of breast cancer still poses significant challenges, deep learning (DL) can support more accurate image interpretation. In this study, we develop a highly robust DL model based on combined B-mode ultrasound (B-mode) and strain elastography ultrasound (SE) images for classifying benign and malignant breast tumors. This study retrospectively included 85 patients, including 42 with benign lesions and 43 with malignancies, all confirmed by biopsy. Two deep neural network models, AlexNet and ResNet, were separately trained on combined 205 B-mode and 205 SE images (80% for training and 20% for validation) from 67 patients with benign and malignant lesions. These two models were then configured to work as an ensemble using both image-wise and layer-wise and tested on a dataset of 56 images from the remaining 18 patients. The ensemble model captures the diverse features present in the B-mode and SE images and also combines semantic features from AlexNet and ResNet models to classify the benign from the malignant tumors. The experimental results demonstrate that the accuracy of the proposed ensemble model is 90%, which is better than the individual models and the model trained using B-mode or SE images alone. Moreover, some patients that were misclassified by the traditional methods were correctly classified by the proposed ensemble method. The proposed ensemble DL model will enable radiologists to achieve superior detection efficiency owing to enhance classification accuracy for breast cancers in ultrasound (US) images.
Symmetry from entanglement suppression

Symmetry is among the most fundamental and powerful concepts in nature, whose existence is usually taken as given, without explanation. We explore whether symmetry can be derived from more fundamental principles from the perspective of quantum information. Starting with a two-qubit system, we show there are only two minimally entangling logic gates: the identity and the $\mathrm{SWAP}$, which interchanges the two states of the qubits. We further demonstrate that, when viewed as an entanglement operator in the spin-space, the $S$-matrix in the two-body scattering of fermions in the $s$-wave channel is uniquely determined by unitarity and rotational invariance to be a linear combination of the identity and the $\mathrm{SWAP}$. Realizing a minimally entangling $S$-matrix would give rise to global symmetries, as exemplified in Wigner's spin-flavor symmetry and Schr\"odinger's conformal invariance in low energy quantum chromodynamics. For ${N}_{q}$ species of qubit, the identity gate is associated with an $[SU(2){]}^{{N}_{q}}$ symmetry, which is enlarged to $SU(2{N}_{q})$ when there is a species-universal coupling constant.
HANet

Video-text retrieval is an important yet challenging task in vision-language understanding, which aims to learn a joint embedding space where related video and text instances are close to each other. Most current works simply measure the video-text similarity based on video-level and text-level embeddings. However, the neglect of more fine-grained or local information causes the problem of insufficient representation. Some works exploit the local details by disentangling sentences, but overlook the corresponding videos, causing the asymmetry of video-text representation. To address the above limitations, we propose a Hierarchical Alignment Network (HANet) to align different level representations for video-text matching. Specifically, we first decompose video and text into three semantic levels, namely event (video and text), action (motion and verb), and entity (appearance and noun). Based on these, we naturally construct hierarchical representations in the individual-local-global manner, where the individual level focuses on the alignment between frame and word, local level focuses on the alignment between video clip and textual context, and global level focuses on the alignment between the whole video and text. Different level alignments capture fine-to-coarse correlations between video and text, as well as take the advantage of the complementary information among three semantic levels. Besides, our HANet is also richly interpretable by explicitly learning key semantic concepts. Extensive experiments on two public datasets, namely MSR-VTT and VATEX, show the proposed HANet outperforms other state-of-the-art methods, which demonstrates the effectiveness of hierarchical representation and alignment. Our code is publicly available at https://github.com/Roc-Ng/HANet.
Deep Learning-Based Privacy Preservation and Data Analytics for IoT Enabled Healthcare

With the development of the industrial Internet of Things (IIoT), intelligent healthcare aims to build a platform to monitor users’ health-related information based on wearable devices remotely. The evolution of blockchain and artificial intelligence technology also promotes the progress of secure intelligent healthcare. However, since the data are stored in the cloud server, it still faces the risk of being attacked and privacy leakage. Note that little attention has been paid to the security issue of privacy information mixed in raw data collected from large number of distributed and heterogeneous wearable healthcare devices. To solve this problem, in this article, we design a deep learning-based privacy preservation and data analytics system for IoT enabled healthcare. At the user end, we collect raw data and separate the users’ privacy information in the privacy-isolation zone. At the cloud end, we analyze the health-related data without users’ privacy information and construct a delicate security module based on the convolutional neural network. We also deploy and evaluate the prototype system, where extensive experiments prove its effectiveness and robustness.
Low-Light Homomorphic Filtering Network for integrating image enhancement and classification

Low-light image (LLI) enhancement techniques have recently demonstrated remarkable progress especially with the use of deep learning approaches. However, most existing techniques are developed as standalone solutions and do not take into account the impact of LLI enhancement on high-level computer vision tasks like object classification. In this paper, we propose a new LLI enhancement model titled LLHFNet (Low-light Homomorphic Filtering Network) which performs image-to-frequency filter learning and is designed for seamless integration into classification models. Through this integration, the classification model is embedded with an internal enhancement capability and is jointly trained to optimize both image enhancement and classification performance. We have conducted a large battery of experiments using SICE, Pascal VOC, and ExDark datasets, to quantitatively and qualitatively evaluate our approach's enhancement quality and classification performance. When evaluated as a standalone enhancement model, our solution consistently ranks among the best existing image enhancement techniques. When embedded with a classification model, our solution achieves an average 5.5% improvement in classification accuracy, compared with the traditional pipeline of separate enhancement followed by classification. Results produce robust classification quality on both LLIs and normal-light images (NLIs), and highlight a clear improvement to the literature.
MSO: Multi-Feature Space Joint Optimization Network for RGB-Infrared Person Re-Identification

The RGB-infrared cross-modality person re-identification (ReID) task aims to recognize the images of the same identity between the visible modality and the infrared modality. Existing methods mainly use a two-stream architecture to eliminate the discrepancy between the two modalities in the final common feature space, which ignore the single space of each modality in the shallow layers. To solve it, in this paper, we present a novel multi-feature space joint optimization (MSO) network, which can learn modality-sharable features in both the single-modality space and the common space. Firstly, based on the observation that edge information is modality-invariant, we propose an edge features enhancement module to enhance the modality-sharable features in each single-modality space. Specifically, we design a perceptual edge features (PEF) loss after the edge fusion strategy analysis. According to our knowledge, this is the first work that proposes explicit optimization in the single-modality feature space on cross-modality ReID task. Moreover, to increase the difference between cross-modality distance and class distance, we introduce a novel cross-modality contrastive-center (CMCC) loss into the modality-joint constraints in the common feature space. The PEF loss and CMCC loss jointly optimize the model in an end-to-end manner, which markedly improves the network's performance. Extensive experiments demonstrate that the proposed model significantly outperforms state-of-the-art methods on both the SYSU-MM01 and RegDB datasets.
Multimodal Relation Extraction with Efficient Graph Alignment

Relation extraction (RE) is a fundamental process in constructing knowledge graphs. However, previous methods on relation extraction suffer sharp performance decline in short and noisy social media texts due to a lack of contexts. Fortunately, the related visual contents (objects and their relations) in social media posts can supplement the missing semantics and help to extract relations precisely. We introduce the multimodal relation extraction (MRE), a task that identifies textual relations with visual clues. To tackle this problem, we present a large-scale dataset which contains 15000+ sentences with 23 pre-defined relation categories. Considering that the visual relations among objects are corresponding to textual relations, we develop a dual graph alignment method to capture this correlation for better performance. Experimental results demonstrate that visual contents help to identify relations more precisely against the text-only baselines. Besides, our alignment method can find the correlations between vision and language, resulting in better performance. Our dataset and code are available at https://github.com/thecharm/Mega.
Spatial-Angular Attention Network for Light Field Reconstruction

Typical learning-based light field reconstruction methods demand in constructing a large receptive field by deepening their networks to capture correspondences between input views. In this paper, we propose a spatial-angular attention network to perceive non-local correspondences in the light field, and reconstruct high angular resolution light field in an end-to-end manner. Motivated by the non-local attention mechanism (Wang et al., 2018; Zhang et al., 2019), a spatial-angular attention module specifically for the high-dimensional light field data is introduced to compute the response of each query pixel from all the positions on the epipolar plane, and generate an attention map that captures correspondences along the angular dimension. Then a multi-scale reconstruction structure is proposed to efficiently implement the non-local attention in the low resolution feature space, while also preserving the high frequency components in the high-resolution feature space. Extensive experiments demonstrate the superior performance of the proposed spatial-angular attention network for reconstructing sparsely-sampled light fields with Non-Lambertian effects.
Coarse-grained Density Map Guided Object Detection in Aerial Images

Object detection in aerial images is challenging for at least two reasons: (1) most objects are small scale relative to high resolution aerial images; and (2) the object position distribution is nonuniform, making the detection inefficient. In this paper, a novel network, the coarse-grained density map network (CDMNet), is proposed to address these problems. Specifically, we format density maps into coarsegrained form and design a lightweight dual task density estimation network. The coarse-grained density map can not only describe the distribution of objects, but also cluster objects, quantify scale and reduce computing. In addition, we propose a cluster region generation algorithm guided by density maps to crop input images into multiple subregions, denoted clusters, where the objects are adjusted in a reasonable scale. Besides, we improved mosaic data augmentation to relieve foreground-background and category imbalance problems during detector training. Evaluated on two popular aerial datasets, VisDrone[29] and UAVDT[6], CDMNet has achieved significant accuracy improvement compared with previous state-of-the-art methods.
Video Contrastive Learning with Global Context

Contrastive learning has revolutionized the self-supervised image representation learning field and recently been adapted to the video domain. One of the greatest advantages of contrastive learning is that it allows us to flexibly define powerful loss objectives as long as we can find a reasonable way to formulate positive and negative samples to contrast. However, existing approaches rely heavily on the short-range spatiotemporal salience to form clip-level contrastive signals, thus limit themselves from using global context. In this paper, we propose a new video-level contrastive learning method based on segments to formulate positive pairs. Our formulation is able to capture the global context in a video, thus robust to temporal content change. We also incorporate a temporal order regularization term to enforce the inherent sequential structure of videos. Extensive experiments show that our video-level contrastive learning framework (VCLR) is able to outperform previous state-of-the-arts on five video datasets for downstream action classification, action localization, and video retrieval.