Reviewerly/exHarmony huggingface dataset:
Deep learning based semantic personalized recommendation system

The past decade has seen significant development in the number of personalized recommendation applications on the World Wide Web. It aims to assist users to retrieve relevant items from a large repository of contents by providing items or services of likely interest based on examined evidence of the users’ preferences and desires. However, this vision is complex due to the huge amount of information aka media-rich information available on the web. Most of the systems formulated so far use the metadata linked with the digital contents, but such systems fail to generate significant recommendations results. In these circumstances, a semantic personalized recommendation system (SPRS) plays an important role to take away the semantic gap between high-level semantic contents and low-level media features. The proposed system recommends personalized sets of videos to users depending on their previous activity on the site and exploits a domain ontology and user items content to the domain concepts. To evaluate the performance of the framework, items’ prediction is executed by utilizing the proposed framework, and performance is determined by comparing the predicted and actual ratings of the items in terms of Predictive Accuracy Metrics, precision, and recall.
Integrating object proposal with attention networks for video saliency detection

Video saliency detection is an active research issue in both information science and visual psychology. In this paper, we propose an efficient video saliency-detection model, based on integrating object-proposal with attention networks, for efficiently capturing salient objects and human attention areas in the dynamic scenes of videos. In our algorithm, visual object features are first exploited from individual video frame, using real-time neural networks for object detection. Then, the spatial position information of each frame is used to screen out the large background in the video, so as to reduce the influence of background noises. Finally, the results, with backgrounds removed, are further refined by spreading the visual clues through an adaptive weighting scheme into the later layers of a convolutional neural network. Experimental results, conducted on widespread and commonly used databases for video saliency detection, verify that our proposed framework outperforms existing deep models.
Effect of immersive visualization technologies on cognitive load, motivation, usability, and embodiment

Virtual reality (VR) is a promising tool to promote motor (re)learning in healthy users and brain-injured patients. However, in current VR-based motor training, movements of the users performed in a three-dimensional space are usually visualized on computer screens, televisions, or projection systems, which lack depth cues (2D screen), and thus, display information using only monocular depth cues. The reduced depth cues and the visuospatial transformation from the movements performed in a three-dimensional space to their two-dimensional indirect visualization on the 2D screen may add cognitive load, reducing VR usability, especially in users suffering from cognitive impairments. These 2D screens might further reduce the learning outcomes if they limit users' motivation and embodiment, factors previously associated with better motor performance. The goal of this study was to evaluate the potential benefits of more immersive technologies using head-mounted displays (HMDs). As a first step towards potential clinical implementation, we ran an experiment with 20 healthy participants who simultaneously performed a 3D motor reaching and a cognitive counting task using: (1) (immersive) VR (IVR) HMD, (2) augmented reality (AR) HMD, and (3) computer screen (2D screen). In a previous analysis, we reported improved movement quality when movements were visualized with IVR than with a 2D screen. Here, we present results from the analysis of questionnaires to evaluate whether the visualization technology impacted users' cognitive load, motivation, technology usability, and embodiment. Reports on cognitive load did not differ across visualization technologies. However, IVR was more motivating and usable than AR and the 2D screen. Both IVR and AR rea ched higher embodiment level than the 2D screen. Our results support our previous finding that IVR HMDs seem to be more suitable than the common 2D screens employed in VR-based therapy when training 3D movements. For AR, it is still unknown whether the absence of benefit over the 2D screen is due to the visualization technology per se or to technical limitations specific to the device.
A Highly Efficient Model to Study the Semantics of Salient Object Detection

CNN-based salient object detection (SOD) methods achieve impressive performance. However, the way semantic information is encoded in them and whether they are category-agnostic is less explored. One major obstacle in studying these questions is the fact that SOD models are built on top of the ImageNet pre-trained backbones which may cause information leakage and feature redundancy. To remedy this, here we first propose an extremely light-weight holistic model tied to the SOD task that can be freed from classification backbones and trained from scratch, and then employ it to study the semantics of SOD models. With the holistic network and representation redundancy reduction by a novel dynamic weight decay scheme, our model has only 100K parameters, ∼ 0.2% of parameters of large models, and performs on par with SOTA on popular SOD benchmarks. Using CSNet, we find that a) SOD and classification methods use different mechanisms, b) SOD models are category insensitive, c) ImageNet pre-training is not necessary for SOD training, and d) SOD models require far fewer parameters than the classification models. The source code is publicly available at https://mmcheng.net/sod100k/.
Blindly Assess Quality of In-the-Wild Videos via Quality-Aware Pre-Training and Motion Perception

Perceptual quality assessment of the videos acquired in the wilds is of vital importance for quality assurance of video services. The inaccessibility of reference videos with pristine quality and the complexity of authentic distortions pose great challenges for this kind of blind video quality assessment (BVQA) task. Although model-based transfer learning is an effective and efficient paradigm for the BVQA task, it remains to be a challenge to explore <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">what</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">how</i> to bridge the domain shifts for better video representation. In this work, we propose to transfer knowledge from image quality assessment (IQA) databases with authentic distortions and large-scale action recognition with rich motion patterns. We rely on both groups of data to learn the feature extractor and use a mixed list-wise ranking loss function to train the entire model on the target VQA databases. Extensive experiments on six benchmarking databases demonstrate that our method performs very competitively under both individual database and mixed databases training settings. We also verify the rationality of each component of the proposed method and explore a simple ensemble trick for further improvement.
Semantic Communication With Adaptive Universal Transformer

With the development of deep learning (DL), natural language processing (NLP) makes it possible for us to analyze and understand a large amount of language texts. Accordingly, we can achieve semantic communication in terms of joint semantic source and channel coding over a noisy channel with the help of NLP. However, the existing method to realize this goal is to use a fixed transformer of NLP while ignoring the difference of semantic information contained in each sentence. To solve this problem, we propose a new semantic communication system based on Universal Transformer. Compared with the traditional transformer, an adaptive circulation mechanism is introduced in the Universal Transformer. Through the introduction of the circulation mechanism, the new semantic communication system can be more flexible to transmit sentences with different semantic information, and achieve better end-to-end performance under various channel conditions.
Lessons Learned From COVID-19 Contact Tracing During a Public Health Emergency: A Prospective Implementation Study

Background: Contact tracing is a core element of the public health response to emerging infectious diseases including COVID-19. Better understanding the implementation context of contact tracing for pandemics, including individual- and systems-level predictors of success, is critical to preparing for future epidemics. Methods: We carried out a prospective implementation study of an emergency volunteer contact tracing program established in New Haven, Connecticut between April 4 and May 19, 2020. We assessed the yield and timeliness of case and contact outreach in reference to CDC benchmarks, and identified individual and programmatic predictors of successful implementation using multivariable regression models. We synthesized our findings using the RE-AIM implementation framework. Results: Case investigators interviewed only 826 (48%) of 1,705 cases and were unable to reach 545 (32%) because of incomplete information and 334 (20%) who missed or declined repeated outreach calls. Contact notifiers reached just 687 (28%) of 2,437 reported contacts, and were unable to reach 1,597 (66%) with incomplete information and 153 (6%) who missed or declined repeated outreach calls. The median time-to-case-interview was 5 days and time-to-contact-notification 8 days. However, among notified contacts with complete time data, 457 (71%) were reached within 6 days of exposure. The least likely groups to be interviewed were elderly (adjusted relative risk, aRR 0.74, 95% CI 0.61–0.89, p = 0.012, vs. young adult) and Black/African-American cases (aRR 0.88, 95% CI 0.80–0.97, pairwise p = 0.01, vs. Hispanic/Latinx). However, ties between cases and their contacts strongly influenced contact notification success (Intraclass Correlation Coefficient (ICC) 0.60). Surging caseloads and high volunteer turnover (case investigator n = 144, median time from sign-up to retirement from program was 4 weeks) required the program to supplement the volunteer workforce with paid public health nurses. Conclusions: An emergency volunteer-run contact tracing program fell short of CDC benchmarks for time and yield, largely due to difficulty collecting the information required for outreach to cases and contacts. To improve uptake, contact tracing programs must professionalize the workforce; better integrate testing and tracing services; capitalize on positive social influences between cases and contacts; and address racial and age-related disparities through enhanced community engagement.
CFPNET: Channel-Wise Feature Pyramid For Real-Time Semantic Segmentation

Real-time semantic segmentation is playing a more important role in computer vision, due to the growing demand for mobile devices and autonomous driving. Therefore, it is very important to achieve a good trade-off among performance, model size and inference speed. In this paper, we propose a Channel-wise Feature Pyramid (CFP) module to balance those factors. Based on the CFP module, we built CFPNet for real-time semantic segmentation which applied a series of dilated convolution channels to extract effective features. Experiments on Cityscapes and CamVid datasets show that the proposed CFPNet achieves an effective combination of those factors. For the Cityscapes test dataset, CFPNet achieves 70.1% class-wise mIoU with only 0.55 million parameters and 2.5 MB memory. The inference speed can reach 30 FPS on a single RTX 2080Ti GPU with a $1024\times 2048$-pixel image.
A Generalizable Model-and-Data Driven Approach for Open-Set RFF Authentication

Radio-frequency fingerprints (RFFs) are promising solutions for realizing low-cost physical layer authentication. Machine learning-based methods have been proposed for RFF extraction and discrimination. However, most existing methods are designed for the closed-set scenario where the set of devices is remains unchanged. These methods can not be generalized to the RFF discrimination of unknown devices. To enable the discrimination of RFF from both known and unknown devices, we propose a new end-to-end deep learning framework for extracting RFFs from raw received signals. The proposed framework comprises a novel preprocessing module, called neural synchronization (NS), which incorporates the data-driven learning with signal processing priors as an inductive bias from communication-model based processing. Compared to traditional carrier synchronization techniques, which are static, this module estimates offsets by two learnable deep neural networks jointly trained by the RFF extractor. Additionally, a hypersphere representation is proposed to further improve the discrimination of RFF. Theoretical analysis shows that such a data-and-model framework can better optimize the mutual information between device identity and the RFF, which naturally leads to better performance. Experimental results verify that the proposed RFF significantly outperforms purely data-driven DNN-design and existing handcrafted RFF methods in terms of both discrimination and network generalizability.
Deep Symmetric Adaptation Network for Cross-Modality Medical Image Segmentation

Unsupervised domain adaptation (UDA) methods have shown their promising performance in the cross-modality medical image segmentation tasks. These typical methods usually utilize a translation network to transform images from the source domain to target domain or train the pixel-level classifier merely using translated source images and original target images. However, when there exists a large domain shift between source and target domains, we argue that this asymmetric structure, to some extent, could not fully eliminate the domain gap. In this paper, we present a novel deep symmetric architecture of UDA for medical image segmentation, which consists of a segmentation sub-network, and two symmetric source and target domain translation sub-networks. To be specific, based on two translation sub-networks, we introduce a bidirectional alignment scheme via a shared encoder and two private decoders to simultaneously align features 1) from source to target domain and 2) from target to source domain, which is able to effectively mitigate the discrepancy between domains. Furthermore, for the segmentation sub-network, we train a pixel-level classifier using not only original target images and translated source images, but also original source images and translated target images, which could sufficiently leverage the semantic information from the images with different styles. Extensive experiments demonstrate that our method has remarkable advantages compared to the state-of-the-art methods in three segmentation tasks, such as cross-modality cardiac, BraTS, and abdominal multi-organ segmentation.
Mixture-based Feature Space Learning for Few-shot Image Classification

We introduce Mixture-based Feature Space Learning (MixtFSL) for obtaining a rich and robust feature representation in the context of few-shot image classification. Previous works have proposed to model each base class either with a single point or with a mixture model by relying on offline clustering algorithms. In contrast, we propose to model base classes with mixture models by simultaneously training the feature extractor and learning the mixture model parameters in an online manner. This results in a richer and more discriminative feature space which can be employed to classify novel examples from very few samples. Two main stages are proposed to train the MixtFSL model. First, the multimodal mixtures for each base class and the feature extractor parameters are learned using a combination of two loss functions. Second, the resulting network and mixture models are progressively refined through a leader-follower learning procedure, which uses the current estimate as a "target" network. This target network is used to make a consistent assignment of instances to mixture components, which increases performance and stabilizes training. The effectiveness of our end-to-end feature space learning approach is demonstrated with extensive experiments on four standard datasets and four backbones. Notably, we demon-strate that when we combine our robust representation with recent alignment-based approaches, we achieve new state-of-the-art results in the inductive setting, with an absolute accuracy for 5-shot classification of 82.45% on miniImageNet, 88.20% with tieredImageNet, and 60.70% in FC100 using the ResNet-12 backbone.
Clustering-Guided Particle Swarm Feature Selection Algorithm for High-Dimensional Imbalanced Data With Missing Values

Feature selection (FS) in data with class imbalance or missing values has received much attention from researchers due to their universality in real-world applications. However, for data with both the two characteristics above, there is still a lack of the corresponding FS algorithm. Due to the complex coupling relationship between missing data and class imbalance, the need for better FS method becomes essential. To tackle high-dimensional imbalanced data with missing values, this article studies a new evolutionary FS method. First, an improved <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$F$ </tex-math></inline-formula> -measure based on filling risk (RF-measure) is defined to evaluate the influence of missing data on the performance of FS in the case of class imbalance. Following that taking the RF-measure as an objective function, a particle swarm optimization-based FS method with fuzzy clustering (PSOFS-FC) is proposed. Two new problem-specific operators or strategies, i.e., the swarm initialization strategy guided by fuzzy clustering and the local pruning operator based on feature importance, are developed to improve the performance of PSOFS-FC. Compared with state-of-the-art FS algorithms on several public datasets, experimental results show that PSOFS-FC can achieve excellent classification performance with relatively less running time, indicating its superiority on tackling high-dimensional imbalanced data with missing values.
When Attackers Meet AI: Learning-Empowered Attacks in Cooperative Spectrum Sensing

Defense strategies have been well studied to combat Byzantine attacks that aim to disrupt cooperative spectrum sensing by sending falsified versions of spectrum sensing data to a fusion center. However, existing studies usually assume network or attackers as passive entities, e.g., assuming the prior knowledge of attacks is known or fixed. In practice, attackers can actively adopt arbitrary behaviors and avoid pre-assumed patterns or assumptions used by defense strategies. In this paper, we revisit this security vulnerability as an adversarial machine learning problem and propose a novel learning-empowered attack framework named Learning-Evaluation-Beating (LEB) to mislead the fusion center. Based on the black-box nature of the fusion center in cooperative spectrum sensing, our new perspective is to make the adversarial use of machine learning to construct a surrogate model of the fusion center's decision model. We propose a generic algorithm to create malicious sensing data using this surrogate model. Our real-world experiments show that the LEB attack is effective to beat a wide range of existing defense strategies with an up to 82 percent of success ratio. Given the gap between the proposed LEB attack and existing defenses, we introduce a non-invasive method named as influence-limiting defense, which can coexist with existing defenses to defend against LEB attack or other similar attacks. We show that this defense is highly effective and reduces the overall disruption ratio of LEB attack by up to 80 percent.
Artificial intelligence explainability: the technical and ethical dimensions

In recent years, several new technical methods have been developed to make AI-models more transparent and interpretable. These techniques are often referred to collectively as ‘AI explainability’ or ‘XAI’ methods. This paper presents an overview of XAI methods, and links them to stakeholder purposes for seeking an explanation. Because the underlying stakeholder purposes are broadly ethical in nature, we see this analysis as a contribution towards bringing together the technical and ethical dimensions of XAI. We emphasize that use of XAI methods must be linked to explanations of human decisions made during the development life cycle. Situated within that wider accountability framework, our analysis may offer a helpful starting point for designers, safety engineers, service providers and regulators who need to make practical judgements about which XAI methods to employ or to require. This article is part of the theme issue ‘Towards symbiotic autonomous systems’.
Learning hybrid ranking representation for person re-identification

Contemporary person re-identification (re-id) methods mostly compute independentlya feature representation of each person image in the query set and the gallery set. This strategy fails to consider any ranking context information of each probe image in the query set represented implicitly by the whole gallery set. Some recent re-ranking re-id methods therefore propose to take a post-processing strategy to exploit such contextual information for improving re-id matching performance. However, post-processing is independent of model training without jointly optimising the re-id feature and the ranking context information for better compatibility. In this work, for the first time, we show that the appearance feature and the ranking context information can be jointly optimised for learning more discriminative representations and achieving superior matching accuracy. Specifically, we propose to learn a hybrid ranking representation for person re-id with a two-stream architecture: (1) In the external stream, we use the ranking list of each probe image to learn plausible visual variations among the top ranks from the gallery as the external ranking information; (2) In the internal stream, we employ the part-based fine-grained feature as the internal ranking information, which mitigates the harm of incorrect matches in the ranking list. Assembling these two streams generates a hybrid ranking representation for person matching. Extensive experiments demonstrate the superiority of our method over the state-of-the-art methods on four large-scale re-id benchmarks (Market-1501, DukeMTMC-ReID, CUHK03 and MSMT17), under both supervised and unsupervised settings.
Exploring Simple 3D Multi-Object Tracking for Autonomous Driving

3D multi-object tracking in LiDAR point clouds is a key ingredient for self-driving vehicles. Existing methods are predominantly based on the tracking-by-detection pipeline and inevitably require a heuristic matching step for the detection association. In this paper, we present SimTrack to simplify the hand-crafted tracking paradigm by proposing an end-to-end trainable model for joint detection and tracking from raw point clouds. Our key design is to predict the first-appear location of each object in a given snippet to get the tracking identity and then update the location based on motion estimation. In the inference, the heuristic matching step can be completely waived by a simple read-off operation. SimTrack integrates the tracked object association, newborn object detection, and dead track killing in a single unified model. We conduct extensive evaluations on two large-scale datasets: nuScenes and Waymo Open Dataset. Experimental results reveal that our simple approach compares favorably with the state-of-the-art methods while ruling out the heuristic matching rules.