Reviewerly/exHarmony huggingface dataset:
SCL-Net: Structured Collaborative Learning for PET/CT Based Tumor Segmentation

Collaborative learning methods for medical image segmentation are often variants of UNet, where the constructions of classifiers depend on each other and their outputs are supervised independently. However, they cannot explicitly ensure that optimizing auxiliary classifier heads leads to improved segmentation of target classifier. To resolve this problem, we propose a structured collaborative learning (SCL) method, which consists of a context-aware structured classifier population generation (CA-SCPG) module, where the feature propagation of the target classifier path is directly enhanced by the outputs of auxiliary classifiers via a light-weighted high-level context-aware dense connection (HLCA-DC) mechanism, and a knowledge-aware structured classifier population supervision (KA-SCPS) module, where the auxiliary classifiers are properly supervised under the guidance of target classifier's segmentations. Specifically, SCL is proposed based on a recurrent-dense-siamese decoder (RDS-Decoder), which consists of multiple siamese-decoder paths. CA-SCPG enhances the feature propagation of the decoder paths by HLCA-DC, which densely reuses previous decoder paths' output predictions to belong to the target classes as inputs to the latter decoder paths. KA-SCPS supervises the classifier heads simultaneously with KA-SCPS loss, which consists of a generalized weighted cross-entropy loss for deep class-imbalanced learning and a novel knowledge-aware Dice loss (KA-DL). KA-DL is a weighted Dice loss broadcasting knowledges learnt by the target classifier to other classifier heads, harmonizing the learning process of the classifier population. Experiments are performed based on PET/CT volumes with malignant melanoma, lymphoma, or lung cancer. Experimental results demonstrate the superiority of our SCL, when compared to the state-of-the-art methods and baselines.
Joint Optimization of Energy Consumption and Completion Time in Federated Learning

Federated Learning (FL) is an intriguing distributed machine learning approach due to its privacy-preserving characteristics. To balance the trade-off between energy and execution latency, and thus accommodate different demands and application scenarios, we formulate an optimization problem to minimize a weighted sum of total energy consumption and completion time through two weight parameters. The optimization variables include bandwidth, transmission power and CPU frequency of each device in the FL system, where all devices are linked to a base station and train a global model collaboratively. Through decomposing the non-convex optimization problem into two subproblems, we devise a resource allocation algorithm to determine the bandwidth allocation, transmission power, and CPU frequency for each participating device. We further present the convergence analysis and computational complexity of the proposed algorithm. Numerical results show that our proposed algorithm not only has better performance at different weight parameters (i.e., different demands) but also outperforms the state of the art.
Environmentally Adaptive Event-Driven Robust Cubature Kalman Filter for RSS-Based Targets Tracking in Mobile Wireless Sensor Network

This article presents an environmentally adaptive event-driven robust square root cubature Kalman filter (EAERCKF) for received-signal-strength-based (RSS-based) moving targets tracking in resource-constrained mobile wireless sensor network (MWSN) under the motion modeling and RSS quantization uncertainties. First, an environmentally adaptive event-driven mechanism, scheduling dynamically and approximately the desired amount of anchors regardless of the anchors distribution density near the moving target, is designed to reduce energy consumption, communication rates, and computation complexity of the MWSN system. Second, a robust square root cubature Kalman filter is developed to estimate the motion state of the moving target, where the covariances of the process noise and the measurement noise assumed to be time-varying uniform distributions are generated randomly. Third, the bounds of the uniformly distributed covariances describing the motion modeling and RSS quantization uncertainties are given to improve the stability, accuracy, and consistency of the estimator. Finally, the results of numerical simulations demonstrate that our proposed EAERCKF estimation method has superiority in simplifying the determination of the random noise covariances with satisfactory target tracking performance and approximating the desired number of triggered anchors.
Continual Learning for Monolingual End-to-End Automatic Speech Recognition

Adapting Automatic Speech Recognition (ASR) models to new domains results in a deterioration of performance on the original domain(s), a phenomenon called Catastrophic Forgetting (CF). Even monolingual ASR models cannot be extended to new accents, dialects, topics, etc. without suffering from CF, making them unable to be continually enhanced without storing all past data. Fortunately, Continual Learning (CL) methods, which aim to enable continual adaptation while overcoming CF, can be used. In this paper, we implement an extensive number of CL methods for End-to-End ASR and test and compare their ability to extend a monolingual Hybrid CTC-Transformer model across four new tasks. We find that the best performing CL method closes the gap between the fine-tuned model (lower bound) and the model trained jointly on all tasks (upper bound) by more than 40%, while requiring access to only 0.6% of the original data.
Learning Motion-Dependent Appearance for High-Fidelity Rendering of Dynamic Humans from a Single Camera

Appearance of dressed humans undergoes a complex geometric transformation induced not only by the static pose but also by its dynamics, i.e., there exists a number of cloth geometric configurations given a pose depending on the way it has moved. Such appearance modeling conditioned on motion has been largely neglected in existing human rendering methods, resulting in rendering of physically implausible motion. A key challenge of learning the dynamics of the appearance lies in the requirement of a prohibitively large amount of observations. In this paper, we present a compact motion representation by enforcing equivariance—a representation is expected to be transformed in the way that the pose is transformed. We model an equivariant encoder that can generate the generalizable representation from the spatial and temporal derivatives of the 3D body surface. This learned representation is decoded by a compositional multi-task decoder that renders high fidelity time-varying appearance. Our experiments show that our method can generate a temporally coherent video of dynamic humans for unseen body poses and novel views given a single view video.
Testing Service Robots in the Field: An Experience Report

Service robots are mobile autonomous robots, often operating in uncertain and difficult environments. While being increasingly popular, engineering service robots is challenging. Especially, evolving them from prototype to deployable product requires effective validation and verification, assuring the robot's correct and safe operation in the target environment. While testing is the most common validation and verification technique used in practice, surprisingly little is known about the actual testing practices and technologies used in the service robotics domain. We present an experience report on field testing of an industrial-strength service robot, as it transitions from lab experiments to an operational environment. We report challenges and solutions, and reflect on their effectiveness. Our long-term goal is to establish empirically-validated testing techniques for service robots. This experience report constitutes a necessary, but self-contained first step, exploring field testing practices in detail. Our data sources are detailed test artifacts and developer interviews. We model the field testing process and describe test-case design practices. We discuss experiences from performing these field tests over a 10-month test campaign.
Balanced and Hierarchical Relation Learning for One-shot Object Detection

Instance-level feature matching is significantly important to the success of modern one-shot object detectors. Re-cently, the methods based on the metric-learning paradigm have achieved an impressive process. Most of these works only measure the relations between query and target objects on a single level, resulting in suboptimal performance overall. In this paper, we introduce the balanced and hierarchical learning for our detector. The contributions are two-fold: firstly, a novel Instance-level Hierarchical Relation (IHR) module is proposed to encode the contrastive-level, salient-level, and attention-level relations simultane-ously to enhance the query-relevant similarity representation. Secondly, we notice that the batch training of the IHR module is substantially hindered by the positive-negative sample imbalance in the one-shot scenario. We then in-troduce a simple but effective Ratio-Preserving Loss (RPL) to protect the learning of rare positive samples and sup-press the effects of negative samples. Our loss can adjust the weight for each sample adaptively, ensuring the desired positive-negative ratio consistency and boosting query-related IHR learning. Extensive experiments show that our method outperforms the state-of-the-art method by 1.6% and 1.3% on PASCAL VOC and MS COCO datasets for unseen classes, respectively. The code will be available at https://github.com/hero-y/BHRL.
Conditional entropy minimization principle for learning domain invariant representation features

Invariance-principle-based methods such as Invariant Risk Minimization (IRM), have recently emerged as promising approaches for Domain Generalization (DG). Despite promising theory, such approaches fail in common classification tasks due to mixing of true invariant features and spurious invariant features <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> . To address this, we propose a framework based on the conditional entropy minimization (CEM) principle to filter-out the spurious invariant features leading to a new algorithm with a better generalization capability. We show that our proposed approach is closely related to the well-known Information Bottleneck (IB) framework and prove that under certain assumptions, entropy minimization can exactly recover the true invariant features. Our approach provides competitive classification accuracy compared to recent theoretically-principled state-of-the-art alternatives across several DG datasets.
FedPer++: Toward Improved Personalized Federated Learning on Heterogeneous and Imbalanced Data

Federated learning is an emerging technique to collaboratively train machine learning models over multiple clients without exposing private data but suffers from heterogeneous data distributions across clients, which results in slow convergence and degraded model performance. This challenge has motivated several approaches to learn a personalized model that has better model accuracy than the global model for each participating client. On the other hand, the success of multi-task learning suggests that data in similar tasks often share a common feature representation, while the output layer (classifier) for classification is more task-correlated. Existing methods usually neglect the collaboration between locally trained classifiers, which makes each client failed to fully benefit from data in other clients. Therefore, we perform a linear combination based classifier collaboration method for achieving better personalized model performance. Moreover, training a local feature extractor based on local data is prone to over-fitting when local data is insufficient and cannot be fully corrected by global aggregation. To tackle this issue, we propose a feature-regularized training strategy to mitigate the local over-fitting risk and reduce the parameter divergence across clients, facilitating the global feature extractor aggregation. With extensive experiments performed on Fashion-MNIST and CIFAR-10 datasets, we demonstrate the performance improvement and robustness of our method over existing methods.
Few-Shot Object Detection via Understanding Convolution and Attention

Few-Shot Object Detection (FSOD) aims to make the detector adapt to unseen classes with only a few training samples. Typical FSOD methods use Faster R-CNN as the basic detection framework, which utilizes convolutional neural networks to extract image features. However, pooling operations adopted in convolutional neural networks aiming to capture as much image information as possible will inevitably lead to the loss of image information. Therefore, we introduce Hybrid Dilated Convolution (HDC) in the backbone network to ensure a larger receptive field and minimize the loss of image information. Besides, in the k-shot ( $$k\ge 2$$ ) setting, we notice that previous methods use an average aggregation operation to fuse the support features of each category. This operation treats every support sample equally, obviously under-utilizing the limited samples. In order to further take advantage of the given support data, we propose a Support Features Dynamic Fusion (SFDF) module. Our proposed SFDF module takes the correlation between each support feature and the query feature as the weight to fuse the support features adaptively for more robust support clues. Experiments demonstrate that our proposed method achieves state-of-the-art FSOD performance on public Pascal VOC and MS-COCO datasets.
Detection of Road Accidents Using Synthetically Generated Multi-Perspective Accident Videos

Road accidents are often caused by short abnormal events, including traffic violations, abrupt change in vehicular motion, driver fatigue, etc. Observing an accident event from the right camera perspective plays a crucial role while detecting accidents. However, it may not be possible to capture such abnormal events from a limited camera perspective. We present a deep learning framework to analyze the accident events recorded from multiple perspectives. First, we estimate feature similarity in videos recorded from multiple perspectives. We then divided the video samples into high and low feature similarity groups. Next, we extract spatio-temporal features from each group using two-branch DCNNs and fuse them using a rank-based weighted average pooling strategy followed by classification. We present a new road accident video dataset (MP-RAD), where each accident event is synthetically generated and captured from five independent camera perspectives using a computer gaming platform. Most of the existing road accident datasets use egocentric views or they are captured in fixed camera setups. However, our dataset is large and multi-perspective that can be used to validate ITS-related tasks such as accident detection, accident localization, traffic monitoring, etc. The dataset contains 400 accident events with a total of 2000 videos. We provide temporal annotations of all videos. The proposed framework and the dataset have been cross-validated with latest accident detection baselines trained on real-world road accident videos and vice-versa. The sub-optimal detection accuracy obtained using the baselines indicates that the proposed framework and the dataset can be useful for ITS related research. Code and dataset is available at: https://github.com/draxler1/MP-RAD-Dataset-ITS-
AutoGPart: Intermediate Supervision Search for Generalizable 3D Part Segmentation

Training a generalizable 3D part segmentation network is quite challenging but of great importance in real-world applications. To tackle this problem, some works design task-specific solutions by translating human understanding of the task to machine's learning process, which faces the risk of missing the optimal strategy since machines do not necessarily understand in the exact human way. Others try to use conventional task-agnostic approaches designed for domain generalization problems with no task prior knowledge considered. To solve the above issues, we propose AutoGPart, a generic method enabling training generalizable 3D part segmentation networks with the task prior considered. AutoGPart builds a supervision space with geometric prior knowledge encoded, and lets the machine to search for the optimal supervisions from the space for a specific segmentation task automatically. Extensive experiments on three generalizable 3D part segmentation tasks are conducted to demonstrate the effectiveness and versatility of AutoGPart. We demonstrate that the performance of segmentation networks using simple backbones can be significantly improved when trained with supervisions searched by our method.
One-Step Adaptive Spectral Clustering Networks

Deep spectral clustering is a popular and efficient algorithm in unsupervised learning. However, deep spectral clustering methods are organized into three separate steps: affinity matrix learning, spectral embedding learning, and K-means clustering on spectral embedding. In this case, although each step can achieve its own performance, it is still difficult to obtain robust clustering results. In this letter, we propose a one-step adaptive spectral clustering network to overcome the aforementioned shortcomings. The network embeds the three parts of affinity matrix learning, spectral embedding learning, and indicator learning into a unified framework. The affinity matrix is adaptively adjusted by spectral embedding in a deep subspace. We introduce spectral rotation to discretize spectral embedding, which makes the spectral embedding and indicator be learned simultaneously to improve clustering quality. Each part of the model can be iteratively updated based on other parts to optimize the clustering results. Experimental results on four real datasets show the effectiveness of our method on the ACC and NMI clustering evaluation metrics. In particular, our method achieves an NMI of 0.932 and an ACC of 0.973 on the MNIST dataset, a decent performance boost compared to the best baseline.
GCEVT: Learning Global Context Embedding for Vehicle Tracking in Unmanned Aerial Vehicle Videos

Vehicle tracking in the unmanned aerial vehicle (UAV) videos is a fundamental but vital computer vision task. It mainly consists of two key components, that is, detection and reidentification (ReID). Recently, one-shot trackers, which integrate detection and ReID in a unified network, have received significant attention for their fast-tracking speed. However, existing one-shot trackers typically utilize local information to distinguish the detected targets. Due to the lack of global relations, which are key cues for tracking, these methods struggle to identify targets in UAV videos accurately. To alleviate the above issue, we deliberately design an ReID head that combines nonlocal blocks and the transformer layer to capture the global semantic relation in this letter. First, we propose a novel pyramid fusion network (PFN) to obtain the pixel-wise relations of features at multiple levels and aggregate them into features with richer semantic information. Then, we present a channel-wise transformer enhancer (CTE) to model the dependencies among the channels of the feature map and predict fine-grained identity embeddings. Extensive experiments on VisDrone2021 and UAVDT benchmarks demonstrate that our tracker, namely global context embedding for vehicle tracking (GCEVT), achieves state-of-the-art tracking performance.
A Novel Spatial–Temporal Analysis Approach to Pedestrian Groups Detection

The growing availability of geo–referred data describing human behaviour, at different scales and levels of granularity, represents an opportunity for the development and application of data analysis algorithms, whose usage can range from security, to traffic, to architectural design and planning, and even marketing. Focusing on pedestrian generated trajectories, the presence of groups within an analyzed population can influence overall dynamics, from microscopic perspective, and it can provide significant indications. Several approaches for video footage analyses are available, but they generally focus on microscopic features of videos and trajectories and they are generally not suited to scale to the analysis of relatively large datasets of trajectories. The present work proposes a novel approach to spatial–temporal analysis of pedestrian trajectories aimed at detecting groups of pedestrians within large datasets and having minimal assumptions on the nature of these groups.
A Transformer-Based Cross-Modal Image-Text Retrieval Method using Feature Decoupling and Reconstruction

With the increasing application of remote sensing technology, the task of cross-modal retrieval of remote sensing images (CMRRS) has gradually attracted widespread attention. Ex-isting methods often completely map the features of different modalities to a shared space and do not decouple between the modal-invariant information and modal-heterogeneous in-formation, which leads to redundant information in feature mapping and usually gets sub-optimal retrieval performance. This paper proposes a Transformer-based CMRRS method using feature decoupling and reconstruction (TBFDR) to solve this problem. TBFDR achieves state-of-the-art performance in remote sensing image-text retrieval task on Sydney-Captions dataset.
Optimization of DRAM based PIM Architecture for Energy-Efficient Deep Neural Network Training

Deep Neural Network (DNN) training consumes high-energy. On the other hand, DNNs deployed on edge devices demand very high-energy efficiency. In this context, Processing-in-Memory (PIM) is an emerging compute paradigm that bridges the memory-computation gap to improve the energy-efficiency. DRAMs are one such memory type employed for designing energy-efficient PIM architectures for DNN training. One of the major issues of DRAM-PIM architectures designed for DNN training is the high number of internal data accesses within a bank between the memory arrays and the PIM computation units (e.g. 51% more than inference). These internal data accesses in the state-of-the-art DRAM PIM architectures consume very high energy compared to computation units. Hence, it is important to reduce the internal data access energy within the DRAM bank for further improving the energy efficiency of DRAMPIM architectures. We present three novel optimizations that together reduce the internal data access energy up to 81.54%. Our first optimization modifies the bank data access circuit to enable partial accesses of data instead of the conventional fixed granularity accesses, thereby exploiting the available sparsity during training. The second optimization is to have a dedicated low-energy region within the DRAM bank that has low capacitive load of global wires and shorter data movement. Finally, we propose a 12-bit high dynamic range floating-point format called TinyFloat that reduces the total number of data access energy by 20% compared to IEEE 754 half and single precision.