Reviewerly/exHarmony huggingface dataset:
Robust Visual Tracking by Segmentation

Estimating the target extent poses a fundamental challenge in visual object tracking. Typically, trackers are box-centric and fully rely on a bounding box to define the target in the scene. In practice, objects often have complex shapes and are not aligned with the image axis. In these cases, bounding boxes do not provide an accurate description of the target and often contain a majority of background pixels. We propose a segmentation-centric tracking pipeline that not only produces a highly accurate segmentation mask, but also internally works with segmentation masks instead of bounding boxes. Thus, our tracker is able to better learn a target representation that clearly differentiates the target in the scene from background content. In order to achieve the necessary robustness for the challenging tracking scenario, we propose a separate instance localization component that is used to condition the segmentation decoder when producing the output mask. We infer a bounding box from the segmentation mask, validate our tracker on challenging tracking datasets and achieve the new state of the art on LaSOT with a success AUC score of 69.7%. Since most tracking datasets do not contain mask annotations, we cannot use them to evaluate predicted segmentation masks. Instead, we validate our segmentation quality on two popular video object segmentation datasets. The code and trained models are available at https://github.com/visionml/pytracking .
The Future of Management: DAO to Smart Organizations and Intelligent Operations

In the future, management in smart societies will revolve around knowledge workers and the works they produce. This article is committed to explore new management framework, model, paradigm, and solution for organizing, managing, and measuring knowledge works. First, the parallel management framework is presented that would allow for the virtual-real interactions of humans in social space, robots in physical space, and digital humans in cyberspace to realize descriptive, predictive, and prescriptive intelligence for management. Then, the management foundation models are proposed by fusing scenarios engineering with artificial intelligence foundation models and cyber–physical-social systems. Moreover, the new management paradigm driven by decentralized autonomous organizations and operations is formulated for the advancement of smart organizations and intelligent operations. On these basis, the management operating systems that highlight features of simple intelligence, provable security, flexible scalability, and ecological harmony are finally put forward as new management solution.
A Federated Learning-Based Edge Caching Approach for Mobile Edge Computing-Enabled Intelligent Connected Vehicles

Massive map data transmission and the strict demand for the privacy of high-precision maps have brought significant challenges to the cache of high-precision maps in intelligent connected vehicles (ICV). Federal learning (FL) was introduced to reduce the pressure on the edge network and protect privacy. But the high dynamics of cars and limited resources lead to low accuracy and high training delay. We propose a joint optimization scheme of participant selection and resource allocation for federated learning. In each time slice, vehicles are determined whether to participate in training, which minimizes long-term training delay with limited energy consumption. To meet the delay and privacy requirements of high-precision map caching, we present an edge cooperative caching scheme based on federated deep reinforcement learning (F-DRL), which aims to achieve dynamic adaptive edge caching while protecting user privacy. The collaborative caching model is formulated as a Markov decision process (MDP). Dueling Deep Q Network (Dueling-DQN) is used to solve the optimal strategy, and federal learning is used for training. Enough comparative experiments to evaluate the performance of the proposed schemes. The aspects of reliability, cache hit rate, and training accuracy prove that the method effectively improves the training parameters of federated learning while meeting a high-precision map cache’s delay and reliability requirements.
Dense Cross-Query-and-Support Attention Weighted Mask Aggregation for Few-Shot Segmentation

Research into Few-shot Semantic Segmentation (FSS) has attracted great attention, with the goal to segment target objects in a query image given only a few annotated support images of the target class. A key to this challenging task is to fully utilize the information in the support images by exploiting fine-grained correlations between the query and support images. However, most existing approaches either compressed the support information into a few class-wise prototypes, or used partial support information (e.g., only foreground) at the pixel level, causing non-negligible information loss. In this paper, we propose Dense pixel-wise Cross-query-and-support Attention weighted Mask Aggregation (DCAMA), where both foreground and background support information are fully exploited via multi-level pixel-wise correlations between paired query and support features. Implemented with the scaled dot-product attention in the Transformer architecture, DCAMA treats every query pixel as a token, computes its similarities with all support pixels, and predicts its segmentation label as an additive aggregation of all the support pixels’ labels—weighted by the similarities. Based on the unique formulation of DCAMA, we further propose efficient and effective one-pass inference for n-shot segmentation, where pixels of all support images are collected for the mask aggregation at once. Experiments show that our DCAMA significantly advances the state of the art on standard FSS benchmarks of PASCAL-5 $$^i$$ , COCO-20 $$^i$$ , and FSS-1000, e.g., with 3.1%, 9.7%, and 3.6% absolute improvements in 1-shot mIoU over previous best records. Ablative studies also verify the design DCAMA.
Large Loss Matters in Weakly Supervised Multi-Label Classification

Weakly supervised multi-label classification (WSML) task, which is to learn a multi-label classification using partially observed labels per image, is becoming increasingly important due to its huge annotation cost. In this work, we first regard unobserved labels as negative labels, casting the WSML task into noisy multi-label classification. From this point of view, we empirically observe that memorization effect, which was first discovered in a noisy multi-class setting, also occurs in a multi-label setting. That is, the model first learns the representation of clean labels, and then starts memorizing noisy labels. Based on this finding, we propose novel methods for WSML which reject or correct the large loss samples to prevent model from memorizing the noisy label. Without heavy and complex components, our proposed methods outperform previous state-of-the-art WSML methods on several partial label settings including Pascal VOC 2012, MS COCO, NUSWIDE, CUB, and OpenImages V3 datasets. Various analysis also show that our methodology actually works well, validating that treating large loss properly matters in a weakly supervised multi-label classification. Our code is available at https://github.com/snucml/LargeLossMatters.
A Survey on Blockchain for Healthcare: Challenges, Benefits, and Future Directions

Continuously generated volumes of health data make healthcare a data-intensive domain. This data needs to be collected, stored, and shared among different healthcare actors for various purposes, such as reporting, analysis, collaborative research, and personalized healthcare services. However, the existing data storage and exchange solutions in the healthcare domain exhibit several challenges related to, e.g., data security, patient privacy, and interoperability. Recently, the industry and research community turned its focus to the possible use of blockchain technology to solve some of these challenges in the healthcare domain. The blockchain technology along with the support from smart contracts is considered a salient facilitator for secure and efficient health data sharing. This is due to its unique features, such as decentralization, trustlessness, immutability, traceability, and transparency. In this paper, we provide a comprehensive survey of the state-of-the-art efforts that envision the use of blockchain-based solutions in the healthcare domain. To this end, we introduce a systematic framework for classifying and analyzing such systems. The framework consists of classification in several dimensions: interactions between healthcare entities, functional components of healthcare storage systems, challenges in the healthcare domain that can be overcome by using the blockchain technology, and benefits for healthcare storage systems derived from the fundamental features of the technology. When analyzing over 40 systems and solutions proposed in the state-of-the-art, we perform their rigorous placement by identifying the exact scope of each solution and mapping it to the above taxonomies of interactions, functional components, challenges, and benefits. We additionally provide an extensive discussion of compliance with privacy-related regulations of General Data Protection Regulation (GDPR) in EU, and Health Insurance Portability and Accountability Act (HIPAA). Following the results of the analysis, we have outlined a number of important research gaps and future directions yet to be addressed.
Occluded Human Mesh Recovery

Top-down methods for monocular human mesh recovery have two stages: (1) detect human bounding boxes; (2) treat each bounding box as an independent single-human mesh recovery task. Unfortunately, the single-human assumption does not hold in images with multi-human occlusion and crowding. Consequently, top-down methods have difficulties in recovering accurate 3D human meshes under severe person-person occlusion. To address this, we present Occluded Human Mesh Recovery (OCHMR) - a novel top-down mesh recovery approach that incorporates image spatial context to overcome the limitations of the single-human assumption. The approach is conceptually simple and can be applied to any existing top-down architecture. Along with the input image, we condition the top-down model on spatial context from the image in the form of body-center heatmaps. To reason from the predicted body centermaps, we introduce Contextual Normalization (CoNorm) blocks to adaptively modulate intermediate features of the top-down model. The contextual conditioning helps our model disambiguate between two severely overlapping human boundingboxes, making it robust to multi-person occlusion. Compared with state-of-the-art methods, OCHMR achieves superior performance on challenging multi-person benchmarks like 3DPW, CrowdPose and OCHuman. Specifically, our proposed contextual reasoning architecture applied to the SPIN model with ResNet-50 backbone results in 75.2 PMPJPE on 3DPW-PC, 23.6 AP on CrowdPose and 37.7 AP on OCHu- man datasets, a significant improvement of 6.9 mm, 6.4 AP and 20.8 AP respectively over the baseline.
MultiMAE: Multi-modal Multi-task Masked Autoencoders

We propose a pre-training strategy called Multi-modal Multi-task Masked Autoencoders (MultiMAE). It differs from standard Masked Autoencoding in two key aspects: I) it can optionally accept additional modalities of information in the input besides the RGB image (hence “multi-modal”), and II) its training objective accordingly includes predicting multiple outputs besides the RGB image (hence “multi-task”). We make use of masking (across image patches and input modalities) to make training MultiMAE tractable as well as to ensure cross-modality predictive coding is indeed learned by the network. We show this pre-training strategy leads to a flexible, simple, and efficient framework with improved transfer results to downstream tasks. In particular, the same exact pre-trained network can be flexibly used when additional information besides RGB images is available or when no information other than RGB is available - in all configurations yielding competitive to or significantly better results than the baselines. To avoid needing training datasets with multiple modalities and tasks, we train MultiMAE entirely using pseudo labeling, which makes the framework widely applicable to any RGB dataset. The experiments are performed on multiple transfer tasks (image classification, semantic segmentation, depth estimation) and datasets (ImageNet, ADE20K, Taskonomy, Hypersim, NYUv2). The results show an intriguingly impressive capability by the model in cross-modal/task predictive coding and transfer. Code, pre-trained models, and interactive visualizations are available at https://multimae.epfl.ch .
The Devil is in the Margin: Margin-based Label Smoothing for Network Calibration

In spite of the dominant performances of deep neural networks, recent works have shown that they are poorly calibrated, resulting in over-confident predictions. Miscalibration can be exacerbated by overfitting due to the minimization of the cross-entropy during training, as it promotes the predicted softmax probabilities to match the one-hot label assignments. This yields a pre-softmax activation of the correct class that is significantly larger than the remaining activations. Recent evidence from the literature suggests that loss functions that embed implicit or explicit maximization of the entropy of predictions yield state-of-the-art calibration performances. We provide a unifying constrained-optimization perspective of current state-of-the-art calibration losses. Specifically, these losses could be viewed as approximations of a linear penalty (or a Lagrangian term) imposing equality constraints on logit distances. This points to an important limitation of such underlying equality constraints, whose ensuing gradients constantly push towards a non-informative solution, which might prevent from reaching the best compromise between the discriminative performance and calibration of the model during gradient-based optimization. Following our observations, we propose a simple and flexible generalization based on inequality constraints, which imposes a controllable margin on logit distances. Comprehensive experiments on a variety of image classification, semantic segmentation and NLP benchmarks demonstrate that our method sets novel state-of-the-art results on these tasks in terms of network calibration, without affecting the discriminative performance. The code is available at https://github.com/by-liu/MbLS.
CaMEL: Mean Teacher Learning for Image Captioning

Describing images in natural language is a fundamental step towards the automatic modeling of connections between the visual and textual modalities.In this paper we present CaMEL, a novel Transformer-based architecture for image captioning.Our proposed approach leverages the interaction of two interconnected language models that learn from each other during the training phase.The interplay between the two language models follows a mean teacher learning paradigm with knowledge distillation.Experimentally, we assess the effectiveness of the proposed solution on the COCO dataset and in conjunction with different visual feature extractors.When comparing with existing proposals, we demonstrate that our model provides stateof-the-art caption quality with a significantly reduced number of parameters.According to the CIDEr metric, we obtain a new state of the art on COCO when training without using external data.The source code and trained models will be made publicly available at:
Blockchain-Enabled Secure and Trusted Federated Data Sharing in IIoT

Federated learning breaks down data silos and promotes the intelligence of the Industrial Internet of Things (IIoT). However, the principal–agent architecture commonly used in federated learning not only increases the cost but also fails to take into account the privacy protection and trustworthiness of flexible on-demand data sharing. To tackle the above challenges, we propose a secure and trusted federated data sharing (STFS) based on blockchain. Initially, we construct an autonomous and reliable federated extreme gradient boosting learning algorithm to crack the data isolation problem, providing privacy protection and verifiability. Furthermore, we design a secure and trusted data sharing and trading mechanism to ensure secure on-demand controlled data sharing and fair trading. Finally, the security of STFS is proved based on the universal composable theory. The results of ample experimental simulations demonstrate the good effectiveness and performance of STFS for IIoT applications.
Nonuniform-to-Uniform Quantization: Towards Accurate Quantization via Generalized Straight-Through Estimation

The nonuniform quantization strategy for compressing neural networks usually achieves better performance than its counterpart, i.e., uniform strategy, due to its superior representational capacity. However, many nonuniform quantization methods overlook the complicated projection process in implementing the nonuniformly quantized weights/activations, which incurs non-negligible time and space overhead in hardware deployment. In this study, we propose Nonuniform-to-Uniform Quantization (N2UQ), a method that can maintain the strong representation ability of nonuniform methods while being hardware-friendly and efficient as the uniform quantization for model inference. We achieve this through learning the flexible inequidistant input thresholds to better fit the underlying distribution while quantizing these real-valued inputs into equidistant output levels. To train the quantized network with learnable input thresholds, we introduce a generalized straight-through estimator (G-STE) for intractable backward derivative calculation w.r.t. threshold parameters. Additionally, we consider entropy preserving regularization to further reduce information loss in weight quantization. Even under this adverse constraint of imposing uniformly quantized weights and activations, our N2UQ outperforms state-of-the-art nonuniform quantization methods by 0.5 ~ 1.7% on ImageNet, demonstrating the contribution of N2UQ design. Code and models are available at: https://github.com/liuzechun/Nonuniform-to-Uniform-Quantization.
Single-Domain Generalized Object Detection in Urban Scene via Cyclic-Disentangled Self-Distillation

In this paper, we are concerned with enhancing the generalization capability of object detectors. And we consider a realistic yet challenging scenario, namely Single-Domain Generalized Object Detection (Single-DGOD), which aims to learn an object detector that performs well on many unseen target domains with only one source domain for training. Towards Single-DGOD, it is important to extract domain-invariant representations (DIR) containing intrinsical object characteristics, which is beneficial for improving the robustness for unseen domains. Thus, we present a method, i.e., cyclic-disentangled self-distillation, to disentangle DIR from domain-specific representations without the supervision of domain-related annotations (e.g., domain labels). Concretely, a cyclic-disentangled module is first proposed to cyclically extract DIR from the input visual features. Through the cyclic operation, the disentangled ability can be promoted without the reliance on domain-related annotations. Then, taking the DIR as the teacher, we design a self-distillation module to further enhance the generalization ability. In the experiments, our method is evaluated in urban-scene object detection. Experimental results of five weather conditions show that our method obtains a significant performance gain over baseline methods. Particularly, for the night-sunny scene, our method outperforms baselines by 3%, which indicates that our method is instrumental in enhancing generalization ability. Data and code are available at https://github.com/AmingWu/Single-DgoD.
Volume under the surface

Anomaly detection (AD) is a fundamental task for time-series analytics with important implications for the downstream performance of many applications. In contrast to other domains where AD mainly focuses on point-based anomalies (i.e., outliers in standalone observations), AD for time series is also concerned with range-based anomalies (i.e., outliers spanning multiple observations). Nevertheless, it is common to use traditional point-based information retrieval measures, such as Precision, Recall, and F-score, to assess the quality of methods by thresholding the anomaly score to mark each point as an anomaly or not. However, mapping discrete labels into continuous data introduces unavoidable shortcomings, complicating the evaluation of range-based anomalies. Notably, the choice of evaluation measure may significantly bias the experimental outcome. Despite over six decades of attention, there has never been a large-scale systematic quantitative and qualitative analysis of time-series AD evaluation measures. This paper extensively evaluates quality measures for time-series AD to assess their robustness under noise, misalignments, and different anomaly cardinality ratios. Our results indicate that measures producing quality values independently of a threshold (i.e., AUC-ROC and AUC-PR) are more suitable for time-series AD. Motivated by this observation, we first extend the AUC-based measures to account for range-based anomalies. Then, we introduce a new family of parameter-free and threshold-independent measures, VUS (Volume Under the Surface), to evaluate methods while varying parameters. Our findings demonstrate that our four measures are significantly more robust in assessing the quality of time-series AD methods.
En-Compactness: Self-Distillation Embedding &amp; Contrastive Generation for Generalized Zero-Shot Learning

Generalized zero-shot learning (GZSL) requires a classifier trained on seen classes that can recognize objects from both seen and unseen classes. Due to the absence of unseen training samples, the classifier tends to bias towards seen classes. To mitigate this problem, feature generation based models are proposed to synthesize visual features for unseen classes. However, these features are generated in the visual feature space which lacks of discriminative ability. Therefore, some methods turn to find a better embedding space for the classifier training. They emphasize the inter-class relationships of seen classes, leading the embedding space overfitted to seen classes and unfriendly to unseen classes. Instead, in this paper, we propose an Intra-Class Compactness Enhancement method (ICCE) for GZSL. Our ICCE promotes intra-class compactness with inter-class separability on both seen and unseen classes in the embedding space and visual feature space. By promoting the intra-class relationships but the inter-class structures, we can distinguish different classes with better generalization. Specifically, we propose a Self-Distillation Embedding (SDE) module and a Semantic-Visual Contrastive Generation (SVCG) module. The former promotes intra-class compactness in the embedding space, while the latter accomplishes it in the visual feature space. The experiments demonstrate that our ICCE outperforms the state-of-the-art methods on four datasets and achieves competitive results on the remaining dataset.