Reviewerly/exHarmony huggingface dataset:
A Semi-supervised Learning Approach Based on Adaptive Weighted Fusion for Automatic Image Annotation

To learn a well-performed image annotation model, a large number of labeled samples are usually required. Although the unlabeled samples are readily available and abundant, it is a difficult task for humans to annotate large numbers of images manually. In this article, we propose a novel semi-supervised approach based on adaptive weighted fusion for automatic image annotation that can simultaneously utilize the labeled data and unlabeled data to improve the annotation performance. At first, two different classifiers, constructed based on support vector machine and covolutional neural network, respectively, are trained by different features extracted from the labeled data. Therefore, these two classifiers are independently represented as different feature views. Then, the corresponding features of unlabeled images are extracted and input into these two classifiers, and the semantic annotation of images can be obtained respectively. At the same time, the confidence of corresponding image annotation can be measured by an adaptive weighted fusion strategy. After that, the images and its semantic annotations with high confidence are submitted to the classifiers for retraining until a certain stop condition is reached. As a result, we can obtain a strong classifier that can make full use of unlabeled data. Finally, we conduct experiments on four datasets, namely, Corel 5K, IAPR TC12, ESP Game, and NUS-WIDE. In addition, we measure the performance of our approach with standard criteria, including precision, recall, F-measure, N+, and mAP. The experimental results show that our approach has superior performance and outperforms many state-of-the-art approaches.
Brokerbot: A Cryptocurrency Chatbot in the Social-technical Gap of Trust

Abstract Cryptocurrencies are proliferating as instantiations of blockchain, which is a transparent, distributed ledger technology for validating transactions. Blockchain is thus said to embed trust in its technical design. Yet, blockchain’s technical promise of trust is not fulfilled when applied to the cryptocurrency ecosystem due to many social challenges stakeholders experience. By investigating a cryptocurrency chatbot (Brokerbot) that distributed information on cryptocurrency news and investments, we explored social tensions of trust between stakeholders, namely the bot’s developers, users, and the bot itself. We found that trust in Brokerbot and in the cryptocurrency ecosystem are two conjoined, but separate challenges that users and developers approached in different ways. We discuss the challenging, dual-role of a Brokerbot as an object of trust as a chatbot while simultaneously being a mediator of trust in cryptocurrency, which exposes the social-technical gap of trust. Lastly, we elaborate on trust as a negotiated social process that people shape and are shaped by through emerging ecologies of interlinked technologies like blockchain and conversational interfaces.
Towards Perceptually Optimized Adaptive Video Streaming-A Realistic Quality of Experience Database

Measuring Quality of Experience (QoE) and integrating these measurements into video streaming algorithms is a multi-faceted problem that fundamentally requires the design of comprehensive subjective QoE databases and objective QoE prediction models. To achieve this goal, we have recently designed the LIVE-NFLX-II database, a highly-realistic database which contains subjective QoE responses to various design dimensions, such as bitrate adaptation algorithms, network conditions and video content. Our database builds on recent advancements in content-adaptive encoding and incorporates actual network traces to capture realistic network variations on the client device. The new database focuses on low bandwidth conditions which are more challenging for bitrate adaptation algorithms, which often must navigate tradeoffs between rebuffering and video quality. Using our database, we study the effects of multiple streaming dimensions on user experience and evaluate video quality and quality of experience models and analyze their strengths and weaknesses. We believe that the tools introduced here will help inspire further progress on the development of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">perceptually-optimized</i> client adaptation and video streaming strategies. The database is publicly available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">http://live.ece.utexas.edu/research/LIVE_NFLX_II/live_nflx_plus.html</uri> .
Attribute-wise Explainable Fashion Compatibility Modeling

With the boom of the fashion market and people’s daily needs for beauty, clothing matching has gained increased research attention. In a sense, tackling this problem lies in modeling the human notions of the compatibility between fashion items, i.e., Fashion Compatibility Modeling (FCM), which plays an important role in a wide bunch of commercial applications, including clothing recommendation and dressing assistant. Recent advances in multimedia processing have shown remarkable effectiveness in accurate compatibility evaluation. However, these studies work like a black box and cannot provide appropriate explanations, which are indeed of importance for gaining users’ trust and improving their experience. In fact, fashion experts usually explain the compatibility evaluation through the matching patterns between fashion attributes (e.g., a silk tank top cannot go with a knit dress). Inspired by this, we devise an attribute-wise explainable FCM solution, named ExFCM , which can simultaneously generate the item-level compatibility evaluation for input fashion items and the attribute-level explanations for the evaluation result. In particular, ExFCM consists of two key components: attribute-wise representation learning and attribute interaction modeling. The former works on learning the region-aware attribute representation for each item with the threshold global average pooling. Besides, the latter is responsible for compiling the attribute-level matching signals into the overall compatibility evaluation adaptively with the attentive interaction mechanism. Note that ExFCM is trained without any attribute-level compatibility annotations, which facilitates its practical applications. Extensive experiments on two real-world datasets validate that ExFCM can generate more accurate compatibility evaluations than the existing methods, together with reasonable explanations.
Named entity recognition method of Chinese EMR based on BERT-BiLSTM-CRF

Abstract With the widespread use of Chinese electronic medical records, the extraction of medical named entities and the resolution of the polysemous term of entities are of great significance to the analysis and processing of patient information and disease diagnosis. This paper uses a BERT Chinese pre-training vector that does not rely on manual feature selection, combines BiLSTM and CRF Chinese named entity recognition algorithm model, and applies it to the processing of the CCKS2020 electronic medical record data set. This paper conducts experimental tests and comparisons between the BERT-BiLSTM-CRF model and other models. The results show that the results of this model are better than other models, and it can accurately identify multiple entity categories and has a good application prospect.
Disentangled Representation Learning for Cross-Modal Biometric Matching

Cross-modal biometric matching (CMBM) aims to determine the corresponding voice from a face, or identify the corresponding face from a voice. Recently, many CMBM methods have been proposed by forcing the distance between two modal features to be narrowed. However, these methods ignore the alignability between the two modal features. Because the feature is extracted under the supervision of identity information from single modal data, it can only reflect the identity information of single modal data. In order to address this problem, a disentangled representation learning method is proposed to disentangle the alignable latent identity factors and nonalignable the modality-dependent factors for CMBM. The proposed method consists of two main steps: 1) feature extraction and 2) disentangled representation learning. Firstly, an image feature extraction network is adopted to obtain face features, and a voice feature extraction network is applied to learn voice features. Secondly, a disentangled latent variable is explored to disentangle the latent identity factors that are shared across the modalities from the modality-dependent factors. The modality-dependent factors are filtered out, while the latent identity factors from the two modalities are enforced to be narrowed to align the same identity information. Then, the disentangled latent identity factors are considered as pure identity information to bridge the two modalities for cross-modal verification, 1: <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$N$</tex-math></inline-formula> matching, and retrieval. Note that the proposed method learns the identity information from the input face images and voice segments with only identity label as supervised information. Extensive experiments on the challenging VoxCeleb dataset demonstrate the proposed method outperforms the state-of-the-art methods.
Probability Ordinal-Preserving Semantic Hashing for Large-Scale Image Retrieval

Semantic hashing enables computation and memory-efficient image retrieval through learning similarity-preserving binary representations. Most existing hashing methods mainly focus on preserving the piecewise class information or pairwise correlations of samples into the learned binary codes while failing to capture the mutual triplet-level ordinal structure in similarity preservation. In this article, we propose a novel Probability Ordinal-preserving Semantic Hashing (POSH) framework, which for the first time defines the ordinal-preserving hashing concept under a non-parametric Bayesian theory. Specifically, we derive the whole learning framework of the ordinal similarity-preserving hashing based on the maximum posteriori estimation, where the probabilistic ordinal similarity preservation, probabilistic quantization function, and probabilistic semantic-preserving function are jointly considered into one unified learning framework. In particular, the proposed triplet-ordering correlation preservation scheme can effectively improve the interpretation of the learned hash codes under an economical anchor-induced asymmetric graph learning model. Moreover, the sparsity-guided selective quantization function is designed to minimize the loss of space transformation, and the regressive semantic function is explored to promote the flexibility of the formulated semantics in hash code learning. The final joint learning objective is formulated to concurrently preserve the ordinal locality of original data and explore potentials of semantics for producing discriminative hash codes. Importantly, an efficient alternating optimization algorithm with the strictly proof convergence guarantee is developed to solve the resulting objective problem. Extensive experiments on several large-scale datasets validate the superiority of the proposed method against state-of-the-art hashing-based retrieval methods.
Markerless Sensors for Physical Health Monitoring System Using ECG and GMM Feature Extraction

Physical activity monitoring using wireless sensors is highly demanded in medical fields such as heart attack monitoring, Parkinson's disease, sleep problems and disrupted circadian rhythms. For the productivity of these areas, many researchers are actively involved in physical activity monitoring. However, they still lack potency in term of recognition accuracy. This paper proposed a novel system that attempts to improve recognition accuracy by employing Electrocardiogram and Gaussian Mixture Model feature extraction strategy along with Ant Colony Optimization and Genetic Algorithm as a classifier. Unlike conventional results, our proposed model has shown a high validity results of 83.33% and 92.70% over the two-public benchmark IMSB and USC-HAD datasets, respectively. The proposed system can be used to monitor physical activities of human that can be helpful in daily life activities especially in medical field.
Tens of images can suffice to train neural networks for malignant leukocyte detection

Convolutional neural networks (CNNs) excel as powerful tools for biomedical image classification. It is commonly assumed that training CNNs requires large amounts of annotated data. This is a bottleneck in many medical applications where annotation relies on expert knowledge. Here, we analyze the binary classification performance of a CNN on two independent cytomorphology datasets as a function of training set size. Specifically, we train a sequential model to discriminate non-malignant leukocytes from blast cells, whose appearance in the peripheral blood is a hallmark of leukemia. We systematically vary training set size, finding that tens of training images suffice for a binary classification with an ROC-AUC over 90%. Saliency maps and layer-wise relevance propagation visualizations suggest that the network learns to increasingly focus on nuclear structures of leukocytes as the number of training images is increased. A low dimensional tSNE representation reveals that while the two classes are separated already for a few training images, the distinction between the classes becomes clearer when more training images are used. To evaluate the performance in a multi-class problem, we annotated single-cell images from a acute lymphoblastic leukemia dataset into six different hematopoietic classes. Multi-class prediction suggests that also here few single-cell images suffice if differences between morphological classes are large enough. The incorporation of deep learning algorithms into clinical practice has the potential to reduce variability and cost, democratize usage of expertise, and allow for early detection of disease onset and relapse. Our approach evaluates the performance of a deep learning based cytology classifier with respect to size and complexity of the training data and the classification task.
Driver distraction analysis using face pose cues

Vehicle driver distraction is one of the major reasons for road accidents. Involvement with a co-passenger, use of in-vehicle devices or phone leads to a situation where the driver head pose varies and the eye is off the road. A low cost early warning system should reduce the distracted driving instances, thus making our roads safer. Face pose information forms an important cue to determine driver distraction. The main objective of this work is to analyse the distractions of the driver based on his/her face pose cues. A straight pose or slight variation would indicate a non-distracted driver, while a large pose variation from the center would indicate a high probability for a distracted driver. Face pose database of vehicle drivers is developed and is bench-marked. A clustered two layer approach on Gabor features is proposed. A five layer convolutional network with three fully connected layers is also used to bench-mark the data. The proposed clustered two-layer approach with Gabor features and SVM classifier provides better results in driver distraction analysis when compared to the deep learning approach and other manifold approaches. The improved accuracy could be attributed to the improved modeling of manifold in our approach, better class discrimination of the Gabor features together with better classification provided by the SVM classifier.
Adaptive Fingerprinting

Website fingerprinting attacks can infer which website a user visits over encrypted network traffic. Recent studies can achieve high accuracy (e.g., 98%) by leveraging deep neural networks. However, current attacks rely on enormous encrypted traffic data, which are time-consuming to collect. Moreover, large-scale encrypted traffic data also need to be recollected frequently to adjust the changes in the website content. In other words, the bootstrap time for carrying out website fingerprinting is not practical. In this paper, we propose a new method, named Adaptive Fingerprinting, which can derive high attack accuracy over few encrypted traffic by leveraging adversarial domain adaption. With our method, an attacker only needs to collect few traffic rather than large-scale datasets, which makes website fingerprinting more practical in the real world. Our extensive experimental results over multiple datasets show that our method can achieve 89% accuracy over few encrypted traffic in the closed-world setting and 99% precision and 99% recall in the open-world setting. Compared to a recent study (named Triplet Fingerprinting), our method is much more efficient in pre-training time and is more scalable. Moreover, the attack performance of our method can outperform Triplet Fingerprinting in both the closed-world evaluation and open-world evaluation.
Robust Secret Image Sharing Resistant to Noise in Shares

A secret image is split into <?TeX $n$?> shares in the generation phase of secret image sharing (SIS) for a <?TeX $(k,n)$?> threshold. In the recovery phase, the secret image is recovered when any <?TeX $k$?> or more shares are collected, and each collected share is generally assumed to be lossless in conventional SIS during storage and transmission. However, noise will arise during real-world storage and transmission; thus, shares will experience data loss, which will also lead to data loss in the secret image being recovered. Secret image recovery in the case of lossy shares is an important issue that must be addressed in practice, which is the overall subject of this article. An SIS scheme that can recover the secret image from lossy shares is proposed in this article. First, robust SIS and its definition are introduced. Next, a robust SIS scheme for a <?TeX $(k,n)$?> threshold without pixel expansion is proposed based on the Chinese remainder theorem (CRT) and error-correcting codes (ECC). By screening the random numbers, the share generation phase of the proposed robust SIS is designed to implement the error correction capability without increasing the share size. Particularly in the case of collecting noisy shares, our recovery method is to some degree robust to some noise types, such as least significant bit (LSB) noise, JPEG compression, and salt-and-pepper noise. A theoretical proof is presented, and experimental results are examined to evaluate the effectiveness of our proposed method.
Latent Target-Opinion as Prior for Document-Level Sentiment Classification: A Variational Approach from Fine-Grained Perspective

Existing works for document-level sentiment classification task treat the review document as an overall text unit, performing feature extraction with various sophisticated model architectures. In this paper, we draw inspiration from fine-grained sentiment analysis, proposing to first learn the latent target-opinion distribution behind the documents, and then leverage such fine-grained prior knowledge into the classification process. We model the latent target-opinion distribution as hierarchical variables, where global-level variable captures the overall target and opinion, and local-level variables retrieve the detailed opinion clues at the word level. The proposed method consists of two main parts: a variational module and a classification module. We employ the conditional variational autoencoder to make reconstructions of the document, during which the user and product information can be integrated. In the classification module, we build a hierarchical model based on Transformer encoders, where the local-level and global-level prior distribution representations induced from the variational module are injected into the word-level and sentence-level Transformers, respectively. Experimental results on benchmark datasets show that the proposed method significantly outperforms strong baselines, achieving the state-of-the-art performance. Further analysis shows that our model is capable of capturing the latent fine-grained target and opinion prior information, which is highly effective for improving the task performance.
XpulpNN: Enabling Energy Efficient and Flexible Inference of Quantized Neural Networks on RISC-V Based IoT End Nodes

Heavily quantized fixed-point arithmetic is becoming a common approach to deploy Convolutional Neural Networks (CNNs) on limited-memory low-power IoT end-nodes. However, this trend is narrowed by the lack of support for low-bitwidth in the arithmetic units of state-of-the-art embedded Microcontrollers (MCUs). This work proposes a multi-precision arithmetic unit fully integrated into a RISC-V processor at the micro-architectural and ISA level to boost the efficiency of heavily Quantized Neural Network (QNN) inference on microcontroller-class cores. By extending the ISA with nibble (4-bit) and crumb (2-bit) SIMD instructions, we show near-linear speedup with respect to higher precision integer computation on the key kernels for QNN computation. Also, we propose a custom execution paradigm for SIMD sum-of-dot-product operations, which consists of fusing a dot product with a load operation, with an up to 1.64 × peak MAC/cycle improvement compared to a standard execution scenario. To further push the efficiency, we integrate the RISC-V extended core in a parallel cluster of 8 processors, with near-linear improvement with respect to a single core architecture. To evaluate the proposed extensions, we fully implement the cluster of processors in GF22FDX technology. QNN convolution kernels on a parallel cluster implementing the proposed extension run 6 × and 8 × faster when considering 4- and 2-bit data operands, respectively, compared to a baseline processing cluster only supporting 8-bit SIMD instructions. With a peak of 2.22 TOPs/s/W, the proposed solution achieves efficiency levels comparable with dedicated DNN inference accelerators and up to three orders of magnitude better than state-of-the-art ARM Cortex-M based microcontroller systems such as the low-end STM32L4 MCU and the high-end STM32H7 MCU.
Partial-Softmax Loss based Deep Hashing

Recently, deep supervised hashing methods have shown state-of-the-art performance by integrating feature learning and hash codes learning into an end-to-end network to generate high-quality hash codes. However, it is still a challenge to learn discriminative hash codes for preserving the label information of images efficiently. To overcome this difficulty, in this paper, we propose a novel Partial-Softmax Loss based Deep Hashing, called PSLDH, to generate high-quality hash codes. Specifically, PSLDH first trains a category hashing network to generate a discriminative hash code for each category, and the hash code will preserve semantic information of the corresponding category well. Then, instead of defining the similarity between datapairs using their corresponding label vectors, we directly use the learned hash codes of categories to supervise the learning process of image hashing network, and a novel Partial-SoftMax loss is proposed to optimize the image hashing network. By minimizing the novel Partial-SoftMax loss, the learned hash codes can preserve the label information of images sufficiently. Extensive experiments on three benchmark datasets show that the proposed method outperforms the state-of-the-art baselines in image retrieval task.