Reviewerly/exHarmony huggingface dataset:
Joint Rain Detection and Removal from a Single Image with Contextualized Deep Networks

Rain streaks, particularly in heavy rain, not only degrade visibility but also make many computer vision algorithms fail to function properly. In this paper, we address this visibility problem by focusing on single-image rain removal, even in the presence of dense rain streaks and rain-streak accumulation, which is visually similar to mist or fog. To achieve this, we introduce a new rain model and a deep learning architecture. Our rain model incorporates a binary rain map indicating rain-streak regions, and accommodates various shapes, directions, and sizes of overlapping rain streaks, as well as rain accumulation, to model heavy rain. Based on this model, we construct a multi-task deep network, which jointly learns three targets: the binary rain-streak map, rain streak layers, and clean background, which is our ultimate output. To generate features that can be invariant to rain steaks, we introduce a contextual dilated network, which is able to exploit regional contextual information. To handle various shapes and directions of overlapping rain streaks, our strategy is to utilize a recurrent process that progressively removes rain streaks. Our binary map provides a constraint and thus additional information to train our network. Extensive evaluation on real images, particularly in heavy rain, shows the effectiveness of our model and architecture.
Loss odyssey in medical image segmentation

The loss function is an important component in deep learning-based segmentation methods. Over the past five years, many loss functions have been proposed for various segmentation tasks. However, a systematic study of the utility of these loss functions is missing. In this paper, we present a comprehensive review of segmentation loss functions in an organized manner. We also conduct the first large-scale analysis of 20 general loss functions on four typical 3D segmentation tasks involving six public datasets from 10+ medical centers. The results show that none of the losses can consistently achieve the best performance on the four segmentation tasks, but compound loss functions (e.g. Dice with TopK loss, focal loss, Hausdorff distance loss, and boundary loss) are the most robust losses. Our code and segmentation results are publicly available and can serve as a loss function benchmark. We hope this work will also provide insights on new loss function development for the community.
Continual learning for robotics: Definition, framework, learning strategies, opportunities and challenges

Continual learning (CL) is a particular machine learning paradigm where the data distribution and learning objective change through time, or where all the training data and objective criteria are never available at once. The evolution of the learning process is modeled by a sequence of learning experiences where the goal is to be able to learn new skills all along the sequence without forgetting what has been previously learned. CL can be seen as an online learning where knowledge fusion needs to take place in order to learn from streams of data presented sequentially in time. Continual learning also aims at the same time at optimizing the memory, the computation power and the speed during the learning process. An important challenge for machine learning is not necessarily finding solutions that work in the real world but rather finding stable algorithms that can learn in real world. Hence, the ideal approach would be tackling the real world in a embodied platform: an autonomous agent. Continual learning would then be effective in an autonomous agent or robot, which would learn autonomously through time about the external world, and incrementally develop a set of complex skills and knowledge.Robotic agents have to learn to adapt and interact with their environment using a continuous stream of observations. Some recent approaches aim at tackling continual learning for robotics, but most recent papers on continual learning only experiment approaches in simulation or with static datasets. Unfortunately, the evaluation of those algorithms does not provide insights on whether their solutions may help continual learning in the context of robotics. This paper aims at reviewing the existing state of the art of continual learning, summarizing existing benchmarks and metrics, and proposing a framework for presenting and evaluating both robotics and non robotics approaches in a way that makes transfer between both fields easier. We put light on continual learning in the context of robotics to create connections between fields and normalize approaches.
The Open Images Dataset V4

We present Open Images V4, a dataset of 9.2M images with unified annotations for image classification, object detection and visual relationship detection. The images have a Creative Commons Attribution license that allows to share and adapt the material, and they have been collected from Flickr without a predefined list of class names or tags, leading to natural class statistics and avoiding an initial design bias. Open Images V4 offers large scale across several dimensions: 30.1M image-level labels for 19.8k concepts, 15.4M bounding boxes for 600 object classes, and 375k visual relationship annotations involving 57 classes. For object detection in particular, we provide 15x more bounding boxes than the next largest datasets (15.4M boxes on 1.9M images). The images often show complex scenes with several objects (8 annotated objects per image on average). We annotated visual relationships between them, which support visual relationship detection, an emerging task that requires structured reasoning. We provide in-depth comprehensive statistics about the dataset, we validate the quality of the annotations, we study how the performance of several modern models evolves with increasing amounts of training data, and we demonstrate two applications made possible by having unified annotations of multiple types coexisting in the same images. We hope that the scale, quality, and variety of Open Images V4 will foster further research and innovation even beyond the areas of image classification, object detection, and visual relationship detection.
Transfer Independently Together: A Generalized Framework for Domain Adaptation

Currently, unsupervised heterogeneous domain adaptation in a generalized setting, which is the most common scenario in real-world applications, is under insufficient exploration. Existing approaches either are limited to special cases or require labeled target samples for training. This paper aims to overcome these limitations by proposing a generalized framework, named as transfer independently together (TIT). Specifically, we learn multiple transformations, one for each domain <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">(independently)</i> , to map data onto a shared latent space, where the domains are well aligned. The multiple transformations are jointly optimized in a unified framework <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">(together)</i> by an effective formulation. In addition, to learn robust transformations, we further propose a novel landmark selection algorithm to reweight samples, i.e., increase the weight of pivot samples and decrease the weight of outliers. Our landmark selection is based on graph optimization. It focuses on sample geometric relationship rather than sample features. As a result, by abstracting feature vectors to graph vertices, only a simple and fast integer arithmetic is involved in our algorithm instead of matrix operations with float point arithmetic in existing approaches. At last, we effectively optimize our objective via a dimensionality reduction procedure. TIT is applicable to arbitrary sample dimensionality and does not need labeled target samples for training. Extensive evaluations on several standard benchmarks and large-scale datasets of image classification, text categorization and text-to-image recognition verify the superiority of our approach.
Deep learning in nano-photonics: inverse design and beyond

Deep learning in the context of nano-photonics is mostly discussed in terms of its potential for inverse design of photonic devices or nano-structures. Many of the recent works on machine-learning inverse design are highly specific, and the drawbacks of the respective approaches are often not immediately clear. In this review we want therefore to provide a critical review on the capabilities of deep learning for inverse design and the progress which has been made so far. We classify the different deep-learning-based inverse design approaches at a higher level as well as by the context of their respective applications and critically discuss their strengths and weaknesses. While a significant part of the community’s attention lies on nano-photonic inverse design, deep learning has evolved as a tool for a large variety of applications. The second part of the review will focus therefore on machine learning research in nano-photonics “beyond inverse design.” This spans from physics-informed neural networks for tremendous acceleration of photonics simulations, over sparse data reconstruction, imaging and “knowledge discovery” to experimental applications.
New promises and future challenges of fractal calculus: From two-scale thermodynamics to fractal variational principle

Any physical laws are scale-dependent, the same phenomenon might lead to debating theories if observed using different scales. The two-scale thermodynamics observes the same phenomenon using two different scales, one scale is generally used in the conventional continuum mechanics, and the other scale can reveal the hidden truth beyond the continuum assumption, and fractal calculus has to be adopted to establish governing equations. Here basic properties of fractal calculus are elucidated, and the relationship between the fractal calculus and traditional calculus is revealed using the two-scale transform, fractal variational principles are discussed for 1-D fluid mechanics. Additionally planet distribution in the fractal solar system, dark energy in the fractal space, and a fractal ageing model are also discussed.
Dynamic Head: Unifying Object Detection Heads with Attentions

The complex nature of combining localization and classification in object detection has resulted in the flourished development of methods. Previous works tried to improve the performance in various object detection heads but failed to present a unified view. In this paper, we present a novel dynamic head framework to unify object detection heads with attentions. By coherently combining multiple self-attention mechanisms between feature levels for scale-awareness, among spatial locations for spatial-awareness, and within output channels for task-awareness, the proposed approach significantly improves the representation ability of object detection heads without any computational overhead. Further experiments demonstrate that the effectiveness and efficiency of the proposed dynamic head on the COCO benchmark. With a standard ResNeXt-101-DCN backbone, we largely improve the performance over popular object detectors and achieve a new state-of-the-art at 54.0 AP. The code will be released at https://github.com/microsoft/DynamicHead.
Deep Learning-Based Channel Estimation for Doubly Selective Fading Channels

In this paper, online deep learning (DL)-based channel estimation algorithm for doubly selective fading channels is proposed by employing the deep neural network (DNN). With properly selected inputs, the DNN can not only exploit the features of channel variation from previous channel estimates but also extract additional features from pilots and received signals. Moreover, the DNN can take the advantages of the least squares estimation to further improve the performance of channel estimation. The DNN is first trained with simulated data in an off-line manner and then it could track the dynamic channel in an online manner. To reduce the performance degradation from random initialization, a pre-training approach is designed to refine the initial parameters of the DNN with several epochs of training. The proposed algorithm benefits from the excellent learning and generalization capability of DL and requires no prior knowledge about the channel statistics. Hence, it is more suitable for communication systems with modeling errors or non-stationary channels, such as high-mobility vehicular systems, underwater acoustic systems, and molecular communication systems. The numerical results show that the proposed DL-based algorithm outperforms the existing estimator in terms of both efficiency and robustness, especially when the channel statistics are time-varying.
Metaverse marketing: How the metaverse will shape the future of consumer research and practice

Abstract The initial hype and fanfare from the Meta Platforms view of how the metaverse could be brought to life has evolved into an ongoing discussion of not only the metaverse's impact on users and organizations but also the societal and cultural implications of widespread usage. The potential of consumer interaction with brands within the metaverse has engendered significant debate within the marketing‐focused discourse on the key challenges and transformative opportunities for marketers. Drawing on insights from expert contributors, this study examines the marketing implications of the hypothetical widespread adoption of the metaverse. We identify new research directions and propose a new framework offering valuable contributions for academia, practice, and policy makers. Our future research agenda culminates in a checklist for researchers which clarifies how the metaverse can be beneficial to digital marketing and advertising, branding, services, value creation, and consumer wellbeing.
Semantics-Aware BERT for Language Understanding

The latest work on language representations carefully integrates contextualized features into language model training, which enables a series of success especially in various machine reading comprehension and natural language inference tasks. However, the existing language representation models including ELMo, GPT and BERT only exploit plain context-sensitive features such as character or word embeddings. They rarely consider incorporating structured semantic information which can provide rich semantics for language representation. To promote natural language understanding, we propose to incorporate explicit contextual semantics from pre-trained semantic role labeling, and introduce an improved language representation model, Semantics-aware BERT (SemBERT), which is capable of explicitly absorbing contextual semantics over a BERT backbone. SemBERT keeps the convenient usability of its BERT precursor in a light fine-tuning way without substantial task-specific modifications. Compared with BERT, semantics-aware BERT is as simple in concept but more powerful. It obtains new state-of-the-art or substantially improves results on ten reading comprehension and language inference tasks.
An efficient recommendation generation using relevant Jaccard similarity

In the literature, various collaborative filtering approaches have been developed to perform an efficient recommendation on top of reducing the search cost of the customers. The recommender system methods are concentrated on improving the accuracy and to achieve that goal they focused on formulating complex similarity approaches and neglect the computation time in their model. Furthermore, in order to compute the similarity metric, most of traditional similarity measures have only considered co-rated items and overlooked the total rating vector of the user or item. However, considering only co-rated items to measure similarity metrics in recommender system is an insignificant approach to identifying appropriate nearest neighbors in relatively sparse datasets. Therefore, in this research, two new simple but effective similarity models have been developed by considering all rating vectors of users to classify relevant neighborhoods and generate recommendations in a lower computation time. Moreover, MovieLens, a well-known dataset used in recommender system domain, is involved here to validate the performance of the proposed model. It seems that the proposed relevant Jaccard similarity perform more accurately and effectively to generate well recommendation than other traditional similarity models.
Dice Loss for Data-imbalanced NLP Tasks

Many NLP tasks such as tagging and machine reading comprehension are faced with the severe data imbalance issue: negative examples significantly outnumber positive examples, and the huge number of easy-negative examples overwhelms the training. The most commonly used cross entropy (CE) criteria is actually an accuracy-oriented objective, and thus creates a discrepancy between training and test: at training time, each training instance contributes equally to the objective function, while at test time F1 score concerns more about positive examples. In this paper, we propose to use dice loss in replacement of the standard cross-entropy objective for data-imbalanced NLP tasks. Dice loss is based on the Sørensen--Dice coefficient or Tversky index , which attaches similar importance to false positives and false negatives, and is more immune to the data-imbalance issue. To further alleviate the dominating influence from easy-negative examples in training, we propose to associate training examples with dynamically adjusted weights to deemphasize easy-negative examples. Theoretical analysis shows that this strategy narrows down the gap between the F1 score in evaluation and the dice loss in training. With the proposed training objective, we observe significant performance boost on a wide range of data imbalanced NLP tasks. Notably, we are able to achieve SOTA results on CTB5, CTB6 and UD1.4 for the part of speech tagging task; SOTA results on CoNLL03, OntoNotes5.0, MSRA and OntoNotes4.0 for the named entity recognition task; along with competitive results on the tasks of machine reading comprehension and paraphrase identification.
Aligning Domain-Specific Distribution and Classifier for Cross-Domain Classification from Multiple Sources

While Unsupervised Domain Adaptation (UDA) algorithms, i.e., there are only labeled data from source domains, have been actively studied in recent years, most algorithms and theoretical results focus on Single-source Unsupervised Domain Adaptation (SUDA). However, in the practical scenario, labeled data can be typically collected from multiple diverse sources, and they might be different not only from the target domain but also from each other. Thus, domain adapters from multiple sources should not be modeled in the same way. Recent deep learning based Multi-source Unsupervised Domain Adaptation (MUDA) algorithms focus on extracting common domain-invariant representations for all domains by aligning distribution of all pairs of source and target domains in a common feature space. However, it is often very hard to extract the same domain-invariant representations for all domains in MUDA. In addition, these methods match distributions without considering domain-specific decision boundaries between classes. To solve these problems, we propose a new framework with two alignment stages for MUDA which not only respectively aligns the distributions of each pair of source and target domains in multiple specific feature spaces, but also aligns the outputs of classifiers by utilizing the domainspecific decision boundaries. Extensive experiments demonstrate that our method can achieve remarkable results on popular benchmark datasets for image classification.
RGB-D salient object detection: A survey

Salient object detection, which simulates human visual perception in locating the most significant object(s) in a scene, has been widely applied to various computer vision tasks. Now, the advent of depth sensors means that depth maps can easily be captured; this additional spatial information can boost the performance of salient object detection. Although various RGB-D based salient object detection models with promising performance have been proposed over the past several years, an in-depth understanding of these models and the challenges in this field remains lacking. In this paper, we provide a comprehensive survey of RGB-D based salient object detection models from various perspectives, and review related benchmark datasets in detail. Further, as light fields can also provide depth maps, we review salient object detection models and popular benchmark datasets from this domain too. Moreover, to investigate the ability of existing models to detect salient objects, we have carried out a comprehensive attribute-based evaluation of several representative RGB-D based salient object detection models. Finally, we discuss several challenges and open directions of RGB-D based salient object detection for future research. All collected models, benchmark datasets, datasets constructed for attribute-based evaluation, and related code are publicly available at https://github.com/taozh2017/RGBD-SODsurvey.
Spatial Architecture and Arrangement of Tumor-Infiltrating Lymphocytes for Predicting Likelihood of Recurrence in Early-Stage Non–Small Cell Lung Cancer

Abstract Purpose: The presence of a high degree of tumor-infiltrating lymphocytes (TIL) has been proven to be associated with outcome in patients with non–small cell lung cancer (NSCLC). However, recent evidence indicates that tissue architecture is also prognostic of disease-specific survival and recurrence. We show a set of descriptors (spatial TIL, SpaTIL) that capture density, and spatial colocalization of TILs and tumor cells across digital images that can predict likelihood of recurrence in early-stage NSCLC. Experimental Design: The association between recurrence in early-stage NSCLC and SpaTIL features was explored on 301 patients across four different cohorts. Cohort D1 (n = 70) was used to identify the most prognostic SpaTIL features and to train a classifier to predict the likelihood of recurrence. The classifier performance was evaluated in cohorts D2 (n = 119), D3 (n = 112), and D4 (n = 112). Two pathologists graded each sample of D1 and D2; intraobserver agreement and association between manual grading and likelihood of recurrence were analyzed. Results: SpaTIL was associated with likelihood of recurrence in all test sets (log-rank P &amp;lt; 0.02). A multivariate Cox proportional hazards analysis revealed an HR of 3.08 (95% confidence interval, 2.1–4.5, P = 7.3 × 10−5). In contrast, agreement among expert pathologists using tumor grade was moderate (Kappa = 0.5), and the manual TIL grading was only prognostic for one reader in D2 (P = 8.0 × 10−3). Conclusions: A set of features related to density and spatial architecture of TILs was found to be associated with a likelihood of recurrence of early-stage NSCLC. This information could potentially be used for helping in treatment planning and management of early-stage NSCLC. See related commentary by Peled et al., p. 1449