Reviewerly/exHarmony huggingface dataset:
Personalized federated learning: A Clustered Distributed Co-Meta-Learning approach

Federated Learning (FL) aims to train a model across multiple parties while preserving the privacy of users' data. Traditional FL only develops a common model for users, and does not adapt the model to each user. Therefore, personalized FL approaches emerged that can further adapt the model to users, thus showing better performance. Among these personalized FL methods, meta-learned personalized FL methods achieve the advanced performance. However, this personalization scheme adapts model to each user according to their own data, and the features it learned are not enough and not rich, especially when there are extremely little data in some users. In this paper, we study a more effective variant of personalization federated learning. We first formalize a new learning problem and propose a Distributed Co-Meta-Learning approach for this learning problem. Then, we show how to design a new personalized FL framework based on this Distributed Co-Meta-Learning approach. To optimize our proposed personalized FL framework, while reducing the computational cost in the optimization, we study a chain-estimation aggregation method for our framework. It also reduces the computational load in the clients. Further, we give the theoretical convergence analysis of our method on the most complex case, non-convex and non-IID problems, and analyze some parameters' properties within it. Experiments demonstrate that our method achieves the state-of-the-art performance in the personalization FL area.
CAS: Breast cancer diagnosis framework based on lesion region recognition in ultrasound images

Automated diagnosis of complex breast ultrasound images using deep learning has important clinical implications. In this study, we propose a novel breast cancer diagnosis framework, the CAS framework, based on lesion region to effectively recognize benign and malignant tumors. Firstly, train the segmentation network CRA-ENet on a public dataset with masks and save the trained weights. Then, the weights are used to segment clinical data without masks to obtain masks. Next, cover the masks on the corresponding original images to obtain ultrasound images that only include the lesion region. Finally, the SA-Net is employed to recognize benign and malignant tumors in images. Both the CRA-ENet and SA-Net are novel models explicitly designed for this study. The CRA-ENet is a network derived from ENet through pruning, convolutional replacement, and the incorporation of a hybrid attention mechanism, which can accurately segment breast tumor boundaries, allowing precise localization of lesion regions. The SA-Net is an innovative multi-branch parallel lightweight network that can enable effective artifact suppression and extraction of lesion-relevant features, significantly improving classification accuracy. In experimental, to validate the performance and efficiency of the proposed framework on clinical data provided by Yunnan Cancer Hospital based on the public dataset BUSI. Compared with simple classification utilizing only the classification network, the proposed approach improved accuracy, precision, and F1 score by 4.01%, 3.98%, and 4.00%, respectively, demonstrating the advantages of the framework.
A value-oriented Artificial Intelligence-as-a-Service business plan using integrated tools and services

The latest developments in Artificial Intelligence (AI) are the focal point in increasing the performance of other technologies and the evolution of Industry 4.0. Considering the benefits of AI in today’s world, businesses must move towards using Integrated AI tools and services. This paper introduces a business model based on AI-as-a-Service (AIaaS), which provides an integrated bundle of AI products and services. The strategic approach, roadmap, and heuristic pricing model provided in this paper can be considered a benchmark for AIaaS companies
Correction of whitespace and word segmentation in noisy Pashto text using CRF

Word segmentation is the process of splitting up the text into words. In English and most European languages, word boundaries are identified by whitespace, while in Pashto, there is no explicit word delimiter. Pashto uses whitespace for word separation but not consistently, and it cannot be considered a reliable word-boundary identifier. This inconsistency makes the Pashto word segmentation unique and challenging. Moreover, Pashto is a low-resource, non-standardized language with no established rules for the correct usage of whitespace that leads to two typical spelling errors, space-omission, and space-insertion. These errors significantly affect the performance of the word segmenter. This study aims to develop a state-of-the-art word segmenter for Pashto, with a proofing tool to identify and correct the position of space in a noisy text. The CRF algorithm is incorporated to train two machine learning models for these tasks. For models' training, we have developed a text corpus of nearly 3.5 million words, annotated for the correct positions of spaces and explicit word boundary information using a lexicon-based technique, and then manually checked for errors. The experimental results of the model are very satisfactory, where the F1-scores are 99.2% and 96.7% for the proofing model and word segmenter, respectively.
Semantic Similarity Distance: Towards better text-image consistency metric in text-to-image generation

Generating high-quality images from text remains a challenge in visual-language understanding, with text-image consistency being a major concern. Particularly, the most popular metric R-precision may not accurately reflect the text-image consistency, leading to misleading semantics in generated images. Albeit its significance, designing a better text-image consistency metric surprisingly remains under-explored in the community. In this paper, we make a further step forward to develop a novel CLIP-based metric, Semantic Similarity Distance (SSD), which is both theoretically founded from a distributional viewpoint and empirically verified on benchmark datasets. We also introduce Parallel Deep Fusion Generative Adversarial Networks (PDF-GAN), which use two novel components to mitigate inconsistent semantics and bridge the text-image semantic gap. A series of experiments indicate that, under the guidance of SSD, our developed PDF-GAN can induce remarkable enhancements in the consistency between texts and images while preserving acceptable image quality over the CUB and COCO datasets.
ARAI-MVSNet: A multi-view stereo depth estimation network with adaptive depth range and depth interval

Multi-View Stereo (MVS) is a fundamental problem in geometric computer vision which aims to reconstruct a scene using multi-view images with known camera parameters. However, the mainstream approaches represent the scene with a fixed all-pixel depth range and equal depth interval partition, which will result in inadequate utilization of depth planes and imprecise depth estimation. In this paper, we present a novel multi-stage coarse-to-fine framework to achieve adaptive all-pixel depth range and depth interval. We predict a coarse depth map in the first stage, then an Adaptive Depth Range Prediction module is proposed in the second stage to zoom in the scene by leveraging the reference image and the obtained depth map in the first stage and predict a more accurate all-pixel depth range for the following stages. In the third and fourth stages, we propose an Adaptive Depth Interval Adjustment module to achieve adaptive variable interval partition for pixel-wise depth range. The depth interval distribution in this module is normalized by Z-score, which can allocate dense depth hypothesis planes around the potential ground truth depth value and vice versa to achieve more accurate depth estimation. Extensive experiments on four widely used benchmark datasets (DTU, TnT, BlendedMVS, ETH 3D) demonstrate that our model achieves state-of-the-art performance and yields competitive generalization ability. Particularly, our method achieves the highest Acc and Overall on the DTU dataset, while attaining the highest Recall and F1-score on the Tanks and Temples intermediate and advanced dataset. Moreover, our method also achieves the lowest e1 and e3 on the BlendedMVS dataset and the highest Acc and F1-score on the ETH 3D dataset, surpassing all listed methods. Project website: https://github.com/zs670980918/ARAI-MVSNet
TLTD: Transfer Learning for Tabular Data

Deep Neural Networks (DNNs) have become effective for various machine learning tasks. DNNs are known to achieve high accuracy with unstructured data in which each data sample (e.g., image) consists of many raw features (e.g., pixels) of the same type. The effectiveness of this approach diminishes for structured (tabular) data. In most cases, decision tree-based models such as Random Forest (RF) or Gradient Boosting Decision Trees (GBDT) outperform DNNs. In addition, DNNs tend to perform poorly when the number of samples in the dataset is small. This paper introduces Transfer Learning for Tabular Data (TLTD) which utilizes a novel learning architecture designed to extract new features from structured datasets. Using the DNN’s learning capabilities on images, we convert the tabular data into images, then use the distillation technique to achieve better learning. We evaluated our approach with 25 structured datasets, and compared the outcomes to those of RF, eXtreme Gradient Boosting (XGBoost), Tabnet, KNN, and TabPFN. The results demonstrate the usefulness of the TLTD approach.
ADAPT-TTS: HIGH-QUALITY ZERO-SHOT MULTI-SPEAKER TEXT-TO-SPEECH ADAPTIVE-BASED FOR VIETNAMESE

Current adaptive-based speech synthesis techniques are based on two main streams: 1. Fine-tuning the model using small amounts of adaptive data, and 2. Conditionally training the entire model through a speaker embedding of the target speaker. However, both of these methods require adaptive data to appear during training, which makes the training cost to generate new voices quite expensively. In addition, the traditional TTS model uses a simple loss function to reproduce the acoustic features. However, this optimization is based on incorrect distribution assumptions leading to noisy composite audio results. We introduce the Adapt-TTS model that allows high-quality audio synthesis from a small adaptive sample without training to solve these problems. Key recommendations: 1. The Extracting Mel-vector (EMV) architecture allows for a better representation of speaker characteristics and speech style; 2. An improved zero-shot model with a denoising diffusion model (Mel-spectrogram denoiser) component allows for new voice synthesis without training with better quality (less noise). The evaluation results have proven the model's effectiveness when only needing a single utterance (1-3 seconds) of the reference speaker, the synthesis system gave high-quality synthesis results and achieved high similarity.
DeepSmooth: Efficient and Smooth Depth Completion

Accurate and consistent depth maps are essential for numerous applications across domains such as robotics, Augmented Reality and others. High-quality depth maps that are spatially and temporally consistent enable tasks such as Spatial Mapping, Video Portrait effects and more generally, 3D Scene Understanding. Depth data acquired from sensors is often incomplete and contains holes whereas depth estimated from RGB images can be inaccurate. This work focuses on Depth Completion, the task of filling holes in depth data using color images. Most work in depth completion formulates the task at the frame level, individually filling each frame’s depth. This results in undesirable flickering artifacts when the RGB-D video stream is viewed as a whole and has detrimental effects on downstream tasks. We propose DeepSmooth, a model that spatio-temporally propagates information to fill in depth maps. Using an EfficientNet and pseudo 3D-Conv based architecture, and a loss function which enforces consistency across space and time, the proposed solution produces smooth depth maps.
The Universal Face Encoder: Learning Disentangled Representations Across Different Attributes

Models that can learn orthogonal representations for different facial attributes (e.g. pose, lighting, identity, expressions) have proven to be beneficial for both discriminative and generative tasks. In this work, we propose the universal facial encoder (UFE) that can simultaneously encode different facial attributes as disentangled features from a single face image. We propose a variety of qualitative and quantitative metrics to evaluate feature orthogonality of the UFE and demonstrate superior disentanglement compared to traditional single-attribute encoding. We also show that these features can then be used to train lightweight prediction heads for multiple downstream classification tasks. Moreover, coupling the UFE with a style-based decoder enables hallucination of new face images composed of attributes taken from different samples. As experimentally demonstrated, the UFE allows us to pick and choose these attributes from label-disjoint datasets. A catalog of such synthetic composites can be used as supplemental training data or simply as stock photos.
Uncertainty in Real-Time Semantic Segmentation on Embedded Systems

Application for semantic segmentation models in areas such as autonomous vehicles and human computer interaction require real-time predictive capabilities. The challenges of addressing real-time application is amplified by the need to operate on resource constrained hardware. Whilst development of real-time methods for these platforms has increased, these models are unable to sufficiently reason about uncertainty present when applied on embedded real-time systems. This paper addresses this by combining deep feature extraction from pre-trained models with Bayesian regression and moment propagation for uncertainty aware predictions. We demonstrate how the proposed method can yield meaningful epistemic uncertainty on embedded hardware in real-time whilst maintaining predictive performance.
Self-Supervised Video Interaction Classification using Image Representation of Skeleton Data

Recognizing interactions from sports games broadcast videos is an application of Interaction Recognition from Videos (IRV), that offers many challenges due to complex interactions that are often recorded from a suboptimal view point. Annotating large scale sports specific datasets is expensive and time-consuming. Therefore, in this study, we propose to demonstrate the effectiveness of applying Self-Supervised Learning (SSL) methods for building useful representations from human skeleton pose data (pose for short) without requiring costly annotations for a large scale dataset. Given the numerous well established image-based SSL methods, we demonstrate how to adapt them for sequences of pose through data transformation and a series of pose-based augmentations. We specifically adapt the Relational Reasoning SSL (Relational-SSL for short) [27] and achieve 68.18 ± 0% and 76.62 ± 2.7% in linear evaluation and finetuning protocols, respectively, for the downstream task of IRV from sports broadcast videos. Lastly, we run ablation studies on different components of the method, including the effect of using estimated pose (versus ground truth) on the performance of the downstream task. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup>
BinaryViT: Pushing Binary Vision Transformers Towards Convolutional Models

With the increasing popularity and the increasing size of vision transformers (ViTs), there has been an increasing interest in making them more efficient and less computationally costly for deployment on edge devices with limited computing resources. Binarization can be used to help reduce the size of ViT models and their computational cost significantly, using popcount operations when the weights and the activations are in binary. However, ViTs suffer a larger performance drop when directly applying convolutional neural network (CNN) binarization methods or existing binarization methods to binarize ViTs compared to CNNs on datasets with a large number of classes such as ImageNet-1k. With extensive analysis, we find that binary vanilla ViTs such as DeiT miss out on a lot of key architectural properties that CNNs have that allow binary CNNs to have much higher representational capability than binary vanilla ViT. Therefore, we propose BinaryViT, in which inspired by the CNN architecture, we include operations from the CNN architecture into a pure ViT architecture to enrich the representational capability of a binary ViT without introducing convolutions. These include an average pooling layer instead of a token pooling layer, a block that contains multiple average pooling branches, an affine transformation right before the addition of each main residual connection, and a pyramid structure. Experimental results on the ImageNet-1k dataset show the effectiveness of these operations that allow a fully-binary pure ViT model to be competitive with previous state-of-the-art binary (SOTA) CNN models.
MobileViG: Graph-Based Sparse Attention for Mobile Vision Applications

Traditionally, convolutional neural networks (CNN) and vision transformers (ViT) have dominated computer vision. However, recently proposed vision graph neural networks (ViG) provide a new avenue for exploration. Unfortunately, for mobile applications, ViGs are computationally expensive due to the overhead of representing images as graph structures. In this work, we propose a new graph-based sparse attention mechanism, Sparse Vision Graph Attention (SVGA), that is designed for ViGs running on mobile devices. Additionally, we propose the first hybrid CNN-GNN architecture for vision tasks on mobile devices, MobileViG, which uses SVGA. Extensive experiments show that MobileViG beats existing ViG models and existing mobile CNN and ViT architectures in terms of accuracy and/or speed on image classification, object detection, and instance segmentation tasks. Our fastest model, MobileViG-Ti, achieves 75.7% top-1 accuracy on ImageNet-1K with 0.78 ms inference latency on iPhone 13 Mini NPU (compiled with CoreML), which is faster than MobileNetV2x1.4 (1.02 ms, 74.7% top-1) and MobileNetV2x1.0 (0.81 ms, 71.8% top-1). Our largest model, MobileViG-B obtains 82.6% top-1 accuracy with only 2.30 ms latency, which is faster and more accurate than the similarly sized EfficientFormer-L3 model (2.77 ms, 82.4%). Our work proves that well designed hybrid CNN-GNN architectures can be a new avenue of exploration for designing models that are extremely fast and accurate on mobile devices. Our code is publicly available at https://github.com/SLDGroup/MobileViG.
MobileDeRainGAN: An Efficient Semi-Supervised Approach to Single Image Rain Removal for Task-Driven Applications

Rain removal is an essential task in computer vision, particularly for applications such as autonomous navigation to function seamlessly during rain. However, most existing single-image deraining algorithms are limited by their inability to generalize on diverse real-world rainy images, the need for real-time processing, and the lack of task-driven metric enhancement. This paper proposes MobileDeRainGAN, an efficient semi-supervised algorithm that addresses these challenges. The proposed approach includes a novel latent bridge network and multi-scale discriminator that effectively removes rain-related artifacts at different scales. Our cross-domain experiments on Rain1400 and RainCityscapes datasets demonstrate substantial improvements over state-of-the-art methods in terms of generalization and object detection scores in a semi-supervised setting. Furthermore, our approach is significantly faster and can run in real-time even on edge devices. Overall, our proposed MobileDeRainGAN algorithm offers a significant improvement in rain removal performance on real-world images while being efficient, scalable, and suitable for real-world applications.
An Effective Motorcycle Helmet Object Detection Framework for Intelligent Traffic Safety

Detecting violations of motorcycle helmet rules is an important computer vision task that can greatly protect the lives of motorcycle drivers and passengers in traffic accidents. This abnormal event detection problem can be viewed as an image object detection task, which aims to detect the location of the motorcycle driver and passenger in the image and whether they are wearing helmets. In this paper, we propose a motorcycle helmet object detection (MHOD) framework to achieve this task. Specifically, we first utilize the object detection network with ensemble model to predict the location and category of all objects in videos which can improve the accuracy and robustness of detection model. Then for the scarcity of passenger category training data, the Passenger Recall Module (PRM) is designed via tracking refinement which greatly improves passenger category recall. Finally, we introduce the category refine module (CRM) to correct the category by combining the temporal information in the video. On the test dataset of AI City Challenge 2023 Track5, we achieve significant result compared with other teams, the proposed model ranks first on the public leaderboard of the challenge.
Towards Active Learning for Action Spotting in Association Football Videos

Association football is a complex and dynamic sport, with numerous actions occurring simultaneously in each game. Analyzing football videos is challenging and requires identifying subtle and diverse spatio-temporal patterns. Despite recent advances in computer vision, current algorithms still face significant challenges when learning from limited annotated data, lowering their performance in detecting these patterns. In this paper, we propose an active learning framework that selects the most informative video samples to be annotated next, thus drastically reducing the annotation effort and accelerating the training of action spotting models to reach the highest accuracy at a faster pace. Our approach leverages the notion of uncertainty sampling to select the most challenging video clips to train on next, hastening the learning process of the algorithm. We demonstrate that our proposed active learning framework effectively reduces the required training data for accurate action spotting in football videos. We achieve similar performances for action spotting with NetVLAD++ on SoccerNet-v2, using only one-third of the dataset, indicating significant capabilities for reducing annotation time and improving data efficiency. We further validate our approach on two new datasets that focus on temporally localizing actions of headers and passes, proving its effectiveness across different action semantics in football. We believe our active learning framework for action spotting would support further applications of action spotting algorithms and accelerate annotation campaigns in the sports domain.