Reviewerly/exHarmony huggingface dataset:
Hand Image Understanding via Deep Multi-Task Learning

Analyzing and understanding hand information from multimedia materials like images or videos is important for many real world applications and remains active in research community. There are various works focusing on recovering hand information from single image, however, they usually solve a single task, for example, hand mask segmentation, 2D/3D hand pose estimation, or hand mesh reconstruction and perform not well in challenging scenarios. To further improve the performance of these tasks, we propose a novel Hand Image Understanding (HIU) framework to extract comprehensive information of the hand object from a single RGB image, by jointly considering the relationships between these tasks. To achieve this goal, a cascaded multi-task learning (MTL) backbone is designed to estimate the 2D heat maps, to learn the segmentation mask, and to generate the intermediate 3D information encoding, followed by a coarse-to-fine learning paradigm and a self-supervised learning strategy. Qualitative experiments demonstrate that our approach can recover reasonable mesh representations even in challenging situations. Quantitatively, our method significantly outperforms the state-of-the-art approaches on various widely-used datasets, in terms of diverse evaluation metrics https://github.com/MandyMo/HIU-DMTL.
Rank &amp; Sort Loss for Object Detection and Instance Segmentation

We propose Rank & Sort (RS) Loss, a ranking-based loss function to train deep object detection and instance segmentation methods (i.e. visual detectors). RS Loss supervises the classifier, a sub-network of these methods, to rank each positive above all negatives as well as to sort positives among themselves with respect to (wrt.) their localisation qualities (e.g. Intersection-over-Union - IoU). To tackle the non-differentiable nature of ranking and sorting, we reformulate the incorporation of error-driven update with back-propagation as Identity Update, which enables us to model our novel sorting error among positives. With RS Loss, we significantly simplify training: (i) Thanks to our sorting objective, the positives are prioritized by the classifier without an additional auxiliary head (e.g. for centerness, IoU, mask-IoU), (ii) due to its ranking-based nature, RS Loss is robust to class imbalance, and thus, no sampling heuristic is required, and (iii) we address the multi-task nature of visual detectors using tuning-free task-balancing coefficients. Using RS Loss, we train seven diverse visual detectors only by tuning the learning rate, and show that it consistently outperforms baselines: e.g. our RS Loss improves (i) Faster R-CNN by ∼ 3 box AP and aLRP Loss (ranking-based baseline) by ∼ 2 box AP on COCO dataset, (ii) Mask R-CNN with repeat factor sampling (RFS) by 3.5 mask AP (∼ 7 AP for rare classes) on LVIS dataset; and also outperforms all counterparts. Code is available at: https://github.com/kemaloksuz/RankSortLoss.
Kidney Tumor Semantic Segmentation Using Deep Learning: A Survey of State-of-the-Art

Cure rates for kidney cancer vary according to stage and grade; hence, accurate diagnostic procedures for early detection and diagnosis are crucial. Some difficulties with manual segmentation have necessitated the use of deep learning models to assist clinicians in effectively recognizing and segmenting tumors. Deep learning (DL), particularly convolutional neural networks, has produced outstanding success in classifying and segmenting images. Simultaneously, researchers in the field of medical image segmentation employ DL approaches to solve problems such as tumor segmentation, cell segmentation, and organ segmentation. Segmentation of tumors semantically is critical in radiation and therapeutic practice. This article discusses current advances in kidney tumor segmentation systems based on DL. We discuss the various types of medical images and segmentation techniques and the assessment criteria for segmentation outcomes in kidney tumor segmentation, highlighting their building blocks and various strategies.
Orthogonal Projection Loss

Deep neural networks have achieved remarkable performance on a range of classification tasks, with softmax crossentropy (CE) loss emerging as the de-facto objective function. The CE loss encourages features of a class to have a higher projection score on the true class-vector compared to the negative classes. However, this is a relative constraint and does not explicitly force different class features to be well-separated. Motivated by the observation that ground-truth class representations in CE loss are orthogonal (one-hot encoded vectors), we develop a novel loss function termed ‘Orthogonal Projection Loss' (OPL) which imposes orthogonality in the feature space. OPL augments the properties of CE loss and directly enforces inter-class separation alongside intra-class clustering in the feature space through orthogonality constraints on the mini-batch level. As compared to other alternatives of CE, OPL offers unique advantages e.g., no additional learnable parameters, does not require careful negative mining and is not sensitive to the batch size. Given the plug-and-play nature of OPL, we evaluate it on a diverse range of tasks including image recognition (CIFAR-100), large-scale classification (ImageNet), domain generalization (PACS) and few-shot learning (mini-ImageNet, CIFAR-FS, tiered-ImageNet and Meta-dataset) and demonstrate its effectiveness across the board. Furthermore, OPL offers better robustness against practical nuisances such as adversarial attacks and label noise. Code is available at: https://github.com/kahnchana/opl.
A Novel Deep Learning-based Sentiment Analysis Method Enhanced with Emojis in Microblog Social Networks

To exactly classify sentiments of microblog reviews with emojis in microblog social networks, this paper first proposes an emoji vectorisation method to achieve emoji vectors. Then, an emoji-text integrated bidirectional LSTM (ET-BiLSTM) model for sentiment analysis is proposed. In this model, review text-based sentence representations are extracted by a bidirectional LSTM network. Emoji-based auxiliary representations are obtained by a new attention mechanism. The two representations are further integrated into final review representation vectors. Finally, experimental results indicate that the proposed ET-BiLSTM model improves the performance of sentiment classification evaluated by macro-P, macro-R and macro-F1 scores in microblog social networks.
One-pass Multi-view Clustering for Large-scale Data

Existing non-negative matrix factorization based multi-view clustering algorithms compute multiple coefficient matrices respect to different data views, and learn a common consensus concurrently. The final partition is always obtained from the consensus with classical clustering techniques, such as k-means. However, the non-negativity constraint prevents from obtaining a more discriminative embedding. Meanwhile, this two-step procedure fails to unify multi-view matrix factorization with partition generation closely, resulting in unpromising performance. Therefore, we propose an one-pass multi-view clustering algorithm by removing the non-negativity constraint and jointly optimize the aforementioned two steps. In this way, the generated partition can guide multi-view matrix factorization to produce more purposive coefficient matrix which, as a feedback, improves the quality of partition. To solve the resultant optimization problem, we design an alternate strategy which is guaranteed to be convergent theoretically. Moreover, the proposed algorithm is free of parameter and of linear complexity, making it practical in applications. In addition, the proposed algorithm is compared with recent advances in literature on benchmarks, demonstrating its effectiveness, superiority and efficiency.
Self-Regulation for Semantic Segmentation

In this paper, we seek reasons for the two major failure cases in Semantic Segmentation (SS): 1) missing small objects or minor object parts, and 2) mislabeling minor parts of large objects as wrong classes. We have an interesting finding that Failure-1 is due to the underuse of detailed features and Failure-2 is due to the underuse of visual contexts. To help the model learn a better trade-off, we introduce several Self-Regulation (SR) losses for training SS neural networks. By "self", we mean that the losses are from the model per se without using any additional data or supervision. By applying the SR losses, the deep layer features are regulated by the shallow ones to preserve more details; meanwhile, shallow layer classification logits are regulated by the deep ones to capture more semantics. We conduct extensive experiments on both weakly and fully supervised SS tasks, and the results show that our approach consistently surpasses the baselines. We also validate that SR losses are easy to implement in various state-of-the-art SS models, e.g., SPGNet [7] and OCRNet [62], incurring little computational overhead during training and none for testing <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> .
Efficient depthwise separable convolution accelerator for classification and UAV object detection

Depthwise separable convolutions (DSC) have been widely deployed in lightweight convolutional neural networks due to high efficiency. But the acceleration performance of the Graphics Processing Unit for DSC was not as well as in theory. In this paper, some approaches were proposed for accelerating DSC based on Field-Programmable Gate Array (FPGA). For the preceding layers, S2C (spatial to channel) was proposed to accelerate computing and improve the utilization rate of computational resources and bandwidth. An efficient SharePE was proposed to accelerate the DSC, which can improve the efficiency of the computing resource. The regulable parallelism approach was proposed to compute efficiently the different pointwise convolutional layers. P2D&D2P approach is proposed to reduce the external memory access. For the entire accelerating system, the pre-load workflow was proposed to reduce the waiting time of the accelerator between two images. We demonstrated our approaches on the SkyNet using the Ultra96V2 development board. Results indicated that our proposed accelerator obtained 80.030 frames per second and 0.072 Joule per image for UAV object detection, which achieved the state-of-the-art results for SkyNet. Besides, the MobileNetV2 model was implemented on a larger XC7Z100 FPGA, and the results showed our accelerator classified each picture from ImageNet in 2.69 ms. Code is available at https://github.com/AILearnerLi/DAC-SDC-2020-SEUer.
Enhanced Lorenz-Chaotic Encryption Method for Partial Medical Image Encryption and Data Hiding in Big Data Healthcare

Image encryption is highly required in the big data healthcare cloud to improve the security of the medical image for remote access. Data hiding method is the process of storing the medical information of the patient in the medical image in the hidden format. Many existing data hiding methods are based on wavelet and chaotic map due to its effectiveness. Wavelet based methods have limitations of lack of phase information, poor directionality, and shift sensitivity. Chaotic map is applied to improve the security of the medical image and chaotic map has the limitation of low sensitive to control parameters and initial conditions. In this research, the Improved Chaos Encryption (ICE) is applied to improve the security based on randomness. The average energy is calculated in the images and compared with adaptive threshold to segment the Lorenz 96 model applied in the chaos encryption algorithm to improve the model security. Lorenz 96 increased the randomness of the chaos encryption method due to its high sensitivity. Medial images were used to test the performance of the ICE in the image encryption and image hiding. The proposed ICE model evaluated the quality of the recovered and decrypted image in the various embedding rate. The result shows that the proposed ICE model has the PSNR value of 104.7 dB compared to the LSB-ROI method which has 97.61 dB PSNR.
Blockchain Hyperledger with Non-Linear Machine Learning: A Novel and Secure Educational Accreditation Registration and Distributed Ledger Preservation Architecture

This paper proposes a novel and secure blockchain hyperledger sawtooth-enabled consortium analytical model for smart educational accreditation credential evaluation. Indeed, candidate academic credentials are generated, verified, and validated by the universities and transmitted to the Higher Education Department (HED). The objective is to enable the procedure of credential verification and analyze tamper-proof forged records before validation. For this reason, we designed and created an accreditation analytical model to investigate individual collected credentials from universities and examine candidates’ records of credibility using machine learning techniques and maintain all these aspects of analysis and addresses in the distributed storage with a secure hash-encryption (SHA-256) blockchain consortium network, which runs on a peer-to-peer (P2P) structure. In this proposed analytical model, we deployed a blockchain distributed mechanism to investigate the examiner and analyst processes of accreditation credential protection and storage criteria, which are referred to as chaincodes or smart contracts. These chaincodes automate the distributed credential schedule, generation, verification, validation, and monitoring of the overall model nodes’ transactions. The chaincodes include candidate registration with the associated university (candidateReg()), certificate-related accreditation credentials update (CIssuanceTrans()), and every node’s transactions preservation in the immutable storage (ULedgerAV()) for further investigations. This model simulates the educational benchmark dataset. The result shows the merit of our model. Through extensive simulations, the blockchain-enabled analytical model provides robust performance in terms of credential management and accreditation credibility problems.
Deep Blind Video Super-resolution

Existing video super-resolution (SR) algorithms usually assume that the blur kernels in the degradation process are known and do not model the blur kernels in the restoration. However, this assumption does not hold for blind video SR and usually leads to over-smoothed super-resolved frames. In this paper, we propose an effective blind video SR algorithm based on deep convolutional neural networks (CNNs). Our algorithm first estimates blur kernels from low-resolution (LR) input videos. Then, with the estimated blur kernels, we develop an effective image deconvolution method based on the image formation model of blind video SR to generate intermediate latent frames so that sharp image contents can be restored well. To effectively explore the information from adjacent frames, we estimate the motion fields from LR input videos, extract features from LR videos by a feature extraction network, and warp the extracted features from LR inputs based on the motion fields. Moreover, we develop an effective sharp feature exploration method which first extracts sharp features from restored intermediate latent frames and then uses a transformation operation based on the extracted sharp features and warped features from LR inputs to generate better features for HR video restoration. We formulate the proposed algorithm into an end-to-end trainable framework and show that it performs favorably against state-of-the-art methods.
ELSED: Enhanced line SEgment drawing

Detecting local features, such as corners, segments or blobs, is the first step in the pipeline of many Computer Vision applications. Its speed is crucial for real-time applications. In this paper we present ELSED, the fastest line segment detector in the literature. The key for its efficiency is a local segment growing algorithm that connects gradient-aligned pixels in presence of small discontinuities. The proposed algorithm not only runs in devices with very low end hardware, but may also be parametrized to foster the detection of short or longer segments, depending on the task at hand. We also introduce new metrics to evaluate the accuracy and repeatability of segment detectors. In our experiments with different public benchmarks we prove that our method accounts the highest repeatability and it is the most efficient in the literature.1 In the experiments we quantify the accuracy traded for such gain.
Simultaneously-Collected Multimodal Lying Pose Dataset: Enabling In-Bed Human Pose Monitoring

Computer vision field has achieved great success in interpreting semantic meanings from images, yet its algorithms can be brittle for tasks with adverse vision conditions and the ones suffering from data/label pair limitation. Among these tasks is in-bed human pose monitoring with significant value in many healthcare applications. In-bed pose monitoring in natural settings involves pose estimation in complete darkness or full occlusion. The lack of publicly available in-bed pose datasets hinders the applicability of many successful human pose estimation algorithms for this task. In this paper, we introduce our Simultaneously-collected multimodal Lying Pose (SLP) dataset, which includes in-bed pose images from 109 participants captured using multiple imaging modalities including RGB, long wave infrared (LWIR), depth, and pressure map. We also present a physical hyper parameter tuning strategy for ground truth pose label generation under adverse vision conditions. The SLP design is compatible with the mainstream human pose datasets; therefore, the state-of-the-art 2D pose estimation models can be trained effectively with the SLP data with promising performance as high as 95% at PCKh@0.5 on a single modality. The pose estimation performance of these models can be further improved by including additional modalities through the proposed collaborative scheme.
MTGCN: A multi-task approach for node classification and link prediction in graph data

Both node classification and link prediction are popular topics of supervised learning on the graph data, but previous works seldom integrate them together to capture their complementary information. In this paper, we propose a Multi-Task and Multi-Graph Convolutional Network (MTGCN) to jointly conduct node classification and link prediction in a unified framework. Specifically, MTGCN consists of multiple multi-task learning so that each multi-task learning learns the complementary information between node classification and link prediction. In particular, each multi-task learning uses different inputs to output representations of the graph data. Moreover, the parameters of one multi-task learning initialize the parameters of the other multi-task learning, so that the useful information in the former multi-task learning can be propagated to the other multi-task learning. As a result, the information is augmented to guarantee the quality of representations by exploring the complex constructure inherent in the graph data. Experimental results on six datasets show that our MTGCN outperforms the comparison methods in terms of both node classification and link prediction.
Copyright protection of deep neural network models using digital watermarking: a comparative study

Nowadays, deep learning achieves higher levels of accuracy than ever before. This evolution makes deep learning crucial for applications that care for safety, like self-driving cars and helps consumers to meet most of their expectations. Further, Deep Neural Networks (DNNs) are powerful approaches that employed to solve several issues. These issues include healthcare, advertising, marketing, computer vision, speech processing, natural language processing. The DNNs have marvelous progress in these different fields, but training such DNN models requires a lot of time, a vast amount of data and in most cases a lot of computational steps. Selling such pre-trained models is a profitable business model. But, sharing them without the owner permission is a serious threat. Unfortunately, once the models are sold, they can be easily copied and redistributed. This paper first presents a review of how digital watermarking technologies are really very helpful in the copyright protection of the DNNs. Then, a comparative study between the latest techniques is presented. Also, several optimizers are proposed to improve the accuracy against the fine-tuning attack. Finally, several experiments are performed with black-box settings using several optimizers and the results are compared with the SGD optimizer.
Aspect-based sentiment analysis with component focusing multi-head co-attention networks

User-generated content based on customer opinions and experience has become a rich source of valuable information for enterprises. The purpose of aspect-based sentiment analysis is to predict the sentiment polarity of specific targets from user-generated content. This study proposes a component focusing multi-head co-attention network model which contains three modules: extended context, component focusing, and multi-headed co-attention, designed to improve upon problems encountered in the past. The extended context module improves the ability of bidirectional encoder representations from transformers to handle aspect-based sentiment analysis tasks, and the component focusing module improves the weighting of adjectives and adverbs, to alleviate the problem of average pooling, which treats every word as an equally important term. The multi-head co-attention network is applied to learn the important words in a multi-word target before acquiring the context representation and performs the attention mechanism on the sequence data. The performance of the proposed model is evaluated in extensive experiments on publicly available datasets. The results show that the performance of the proposed model is better than that of the recent state-of-the-art models.
Survey on Videos Data Augmentation for Deep Learning Models

In most Computer Vision applications, Deep Learning models achieve state-of-the-art performances. One drawback of Deep Learning is the large amount of data needed to train the models. Unfortunately, in many applications, data are difficult or expensive to collect. Data augmentation can alleviate the problem, generating new data from a smaller initial dataset. Geometric and color space image augmentation methods can increase accuracy of Deep Learning models but are often not enough. More advanced solutions are Domain Randomization methods or the use of simulation to artificially generate the missing data. Data augmentation algorithms are usually specifically designed for single images. Most recently, Deep Learning models have been applied to the analysis of video sequences. The aim of this paper is to perform an exhaustive study of the novel techniques of video data augmentation for Deep Learning models and to point out the future directions of the research on this topic.