Reviewerly/exHarmony huggingface dataset:
Cross-Attention Transformer for Video Interpolation

We propose TAIN (Transformers and Attention for video INterpolation), a residual neural network for video interpolation, which aims to interpolate an intermediate frame given two consecutive image frames around it. We first present a novel vision transformer module, named Cross-Similarity (CS), to globally aggregate input image features with similar appearance as those of the predicted interpolated frame. These CS features are then used to refine the interpolated prediction. To account for occlusions in the CS features, we propose an Image Attention (IA) module to allow the network to focus on CS features from one frame over those of the other. TAIN outperforms existing methods that do not require flow estimation and performs comparably to flow-based methods while being computationally efficient in terms of inference time on Vimeo90k, UCF101, and SNU-FILM benchmarks.
On the Utility and Protection of Optimization with Differential Privacy and Classic Regularization Techniques

Nowadays, owners and developers of deep learning models must consider stringent privacy-preservation rules of their training data, usually crowd-sourced and retaining sensitive information. The most widely adopted method to enforce privacy guarantees of a deep learning model nowadays relies on optimization techniques enforcing differential privacy. According to the literature, this approach has proven to be a successful defence against several models’ privacy attacks, but its downside is a substantial degradation of the models’ performance. In this work, we compare the effectiveness of the differentially-private stochastic gradient descent (DP-SGD) algorithm against standard optimization practices with regularization techniques. We analyze the resulting models’ utility, training performance, and the effectiveness of membership inference and model inversion attacks against the learned models. Finally, we discuss differential privacy’s flaws and limits and empirically demonstrate the often superior privacy-preserving properties of dropout and l2-regularization.
CaltechFN: Distorted and Partially Occluded Digits

Digit datasets are widely used as compact, generalizable benchmarks for novel computer vision models. However, modern deep learning architectures have surpassed the human performance benchmarks on existing digit datasets, given that these datasets contain digits that have limited variability. In this paper, we introduce Caltech Football Numbers (CaltechFN), an image dataset of highly variable American football digits that aims to serve as a more difficult state-of-the-art benchmark for classification and detection tasks. Currently, CaltechFN contains 61,728 images with 264,572 labeled digits. Given the many different ways that digits on American football jerseys can be distorted and partially occluded in a live-action capture, we find that in comparison to humans, current computer vision models struggle to classify and detect the digits in our dataset. By comparing the performance of the latest task-specific models on CaltechFN and on an existing digit dataset, we show that our dataset indeed presents a far more difficult set of digits and that models trained on it still demonstrate high cross-dataset generalization. We also provide human performance benchmarks for our dataset to demonstrate the current gap between the abilities of humans and computers in the tasks of classifying and detecting the digits in our dataset. Finally, we describe two real-world applications that can be advanced using our dataset. CaltechFN is publicly available at https://data.caltech.edu/records/33qmq-a2n15 , and all benchmark code is available at https://github.com/patrickqrim/CaltechFN .
Definition of Concept “City”: Multidisciplinary Approach

The present article deals with the problem of diversity of definitions, interpretations and approaches to the notion of the term “city” in urban planning, economics, sociology, linguistics, philosophy, etc. The concept “city” is clarified for a holistic consideration of the term in related sciences, as the “city” is inherently multifaceted. Comparative analysis of definitions and classifications of the concept “city” as a multifaceted phenomenon that gives prospects for the development of the city and urban planning in general is given. The classification of Ukrainian cities and features of different approaches to the definition applied in urbanistics are investigated in the scientific paper. The most common criteria of “city” definition such as administrative/legal aspects; economic links; customary/historical background are given. The existence of an “urban center” as a spatial concept based on a matrix of high population density cells and the stages of affiliation of a settlement to a city that takes place in several stages are determined.
Regulativity of Scientific and Technical Texts on Architecture and Construction

The problems of the scientific style and its substyles, in particular their influence on the perception of the recipient have been investigated. The issues of regulativity of scientific and technical texts on architecture and construction have been researched. Scientific styles have been classified by reconciling the unity of the communicative task extra-linguistic motivation and linguistic implementation criteria. The correct perception of scientific and technical texts is ensured by the category of regulation. The influence on the recipient of the texts under consideration has been determined which gives scientific and technical information and provides its content adequate understanding. Obtained data enable to understand scientific and technical texts on architecture and construction more adequate.
Improving Few-shot Learning by Spatially-aware Matching and CrossTransformer

Current few-shot learning models capture visual object relations in the so-called meta-learning setting under a fixed-resolution input. However, such models have a limited generalization ability under the scale and location mismatch between objects, as only few samples from target classes are provided. Therefore, the lack of a mechanism to match the scale and location between pairs of compared images leads to the performance degradation. The importance of image contents varies across coarse-to-fine scales depending on the object and its class label, e.g., generic objects and scenes rely on their global appearance while fine-grained objects rely more on their localized visual patterns. In this paper, we study the impact of scale and location mismatch in the few-shot learning scenario, and propose a novel Spatially-aware Matching (SM) scheme to effectively perform matching across multiple scales and locations, and learn image relations by giving the highest weights to the best matching pairs. The SM is trained to activate the most related locations and scales between support and query data. We apply and evaluate SM on various few-shot learning models and backbones for comprehensive evaluations. Furthermore, we leverage an auxiliary self-supervisory discriminator to train/predict the spatial- and scale-level index of feature vectors we use. Finally, we develop a novel transformer-based pipeline to exploit self- and cross-attention in a spatially-aware matching process. Our proposed design is orthogonal to the choice of backbone and/or comparator.
Handling Domain Shift for Lesion Detection via Semi-supervised Domain Adaptation

As the community progresses towards automated Universal Lesion Detection (ULD), it is vital that the techniques developed are robust and easily adaptable across a variety of datasets coming from different scanners, hospitals, and acquisition protocols. In practice, this remains a challenge due to the complexities of the different types of domain shifts. In this paper, we address the domain-shift by proposing a novel domain adaptation framework for ULD. The proposed model allows for the transfer of lesion knowledge from a large labeled source domain to detect lesions on a new target domain with minimal labeled samples. The proposed method first aligns the feature distribution of the two domains by training a detector on the source domain using a supervised loss, and a discriminator on both source and unlabeled target domains using an adversarial loss. Subsequently, a few labeled samples from the target domain along with labeled source samples are used to adapt the detector using an over-fitting aware and periodic gradient update based joint few-shot fine-tuning technique. Further, we utilize a self-supervision scheme to obtain pseudo-labels having high-confidence on the unlabeled target domain which are used to further train the detector in a semi-supervised manner and improve the detection sensitivity. We evaluate our proposed approach on domain adaptation for lesion detection from CT-scans wherein a ULD network trained on the DeepLesion dataset is adapted to 3 target domain datasets such as LiTS, KiTS and 3Dircadb. By utilizing adversarial, few-shot and incremental semi-supervised training, our method achieves comparable detection sensitivity to the previous methods for few-shot and semi-supervised methods as well as to the Oracle model trained on the labeled target domain.
Gift from Nature: Potential Energy Minimization for Explainable Dataset Distillation

Dataset distillation aims to reduce the dataset size by capturing important information from original dataset. It can significantly improve the feature extraction effectiveness, storage efficiency and training robustness. Furthermore, we study the features from the data distillation and found unique discriminative properties that can be exploited. Therefore, based on Potential Energy Minimization, we propose a generalized and explainable dataset distillation algorithm, called Potential Energy Minimization Dataset Distillation (PEMDD). The motivation is that when the distribution for each class is regular (that is, almost a compact high-dimensional ball in the feature space) and has minimal potential energy in its location, the mixed-distributions of all classes should be stable. In this stable state, Unscented Transform (UT) can be implemented to distill the data and reconstruct the stable distribution using these distilled data. Moreover, a simple but efficient framework of using the distilled data to fuse different datasets is proposed, where only a lightweight finetune is required. To demonstrate the superior performance over other works, we first visualize the classification results in terms of storage cost and performance. We then report quantitative improvement by comparing our proposed method with other state-of-the-art methods on several datasets. Finally, we conduct experiments on few-shot learning, and show the efficiency of our proposed methods with significant improvement in terms of the storage size requirement.
Evaluating and Bench-Marking Object Detection Models for Traffic Sign and Traffic Light Datasets

Object detection is an important sub-problem for many computer vision applications. There has been substantial research in improving and evaluating object detection models for generic objects but it is still not known how latest deep learning models perform on small road scene objects such as traffic lights and traffic signs. In fact, locating small object of interest such as traffic light and traffic sign is a priority task for an autonomous vehicle to maneuver in complex scenarios. Although some researchers have tried to investigate the performance of deep learning based object detection models on various public datasets, however there exists no comprehensive benchmark. We present a more detailed evaluation by providing in-depth analysis of state-of-the-art deep learning based anchor and anchor-less object detection models such as Faster-RCNN, Single Shot Detector (SSD), Yolov3, RetinaNet, CenterNet and Cascade-RCNN. We compare the performance of these models on popular and publicly available traffic light datasets and traffic sign datasets from varied geographies. For traffic light datasets, we consider LISA Traffic Light (TL), Bosch, WPI and recently introduced S2TLD dataset for traffic light detection. For traffic sign benchmarking, we use LISA Traffic Sign (TS), GTSD, TT100K and recently published Mapillary Traffic Sign Dataset (MTSD). We compare the quantitative and qualitative performance of all the models on the aforementioned datasets and find that CenterNet outperforms all other baselines on almost all the datasets. We also compare inference time on specific CPU and GPU versions, flops and parameters for comparison. Understanding such behavior of the models on these datasets can help in solving a variety of practical difficulties and assists in the development of real-world applications. The source code and the models are available at https://github.com/OppoResearchIndia/DLSOD-ACCVW .
Aerial Image Segmentation via Noise Dispelling and Content Distilling

Aerial image segmentation is an essential problem for land management which can be used for change detection and policy planning. However, traditional semantic segmentation methods focus on single-perspective images in road scenes, while aerial images are top-down views and objects are of a small size. Existing aerial segmentation methods tend to modify the network architectures proposed for traditional semantic segmentation problems, yet to the best of our knowledge, none of them focus on the noisy information present in the aerial images. In this work, we conduct an investigation on the effectiveness of each channels of the aerial image on the segmentation performance. Then, we propose a disentangle learning method to investigate the differences and similarities between channels and images, so that potential noisy information can be removed for higher segmentation accuracy.
Robustizing Object Detection Networks Using Augmented Feature Pooling

This paper presents a framework to robustize object detection networks against large geometric transformation. Deep neural networks rapidly and dramatically have improved object detection performance. Nevertheless, modern detection algorithms are still sensitive to large geometric transformation. Aiming at improving the robustness of the modern detection algorithms against the large geometric transformation, we propose a new feature extraction called augmented feature pooling. The key is to integrate the augmented feature maps obtained from the transformed images before feeding it to the detection head without changing the original network architecture. In this paper, we focus on rotation as a simple-yet-influential case of geometric transformation, while our framework is applicable to any geometric transformations. It is noteworthy that, with only adding a few lines of code from the original implementation of the modern object detection algorithms and applying simple fine-tuning, we can improve the rotation robustness of these original detection algorithms while inheriting modern network architectures’ strengths. Our framework overwhelmingly outperforms typical geometric data augmentation and its variants used to improve robustness against appearance changes due to rotation. We construct a dataset based on MS COCO to evaluate the robustness of the rotation, called COCO-Rot. Extensive experiments on three datasets, including our COCO-Rot, demonstrate that our method can improve the rotation robustness of state-of-the-art algorithms.
Loss Function with Memory for Trustworthiness Threshold Learning: Case of Face and Facial Expression Recognition

We compare accuracy metrics of the supervisor meta-learning artificial neural networks (ANN) that learn the trustworthiness of the Inception v.3 convolutional neural networks (CNN) ensemble prediction a priori of the “ground truth” verification on the face and facial expression recognition. One of the compared meta-learning ANN modes uses a simple majority of the ensemble votes and its predictions. In contrast, another uses dynamically learned “trusted” ensemble vote count and its a priori prediction to decide on the trustworthiness of the underlying CNN ensemble prediction. A custom loss function with memory is introduced to collect trustworthiness predictions and their errors during training. Based on the collected statistics, learning gradients for the “trusted” ensemble vote count parameter is calculated, and the “trusted” ensemble vote count threshold is dynamically determined. A facial data set with makeup and occlusion is used for computational experiments in the partition that ensures high out of the training data distribution conditions, where only non-makeup and non-occluded images are used for CNN model ensemble training, while the test set contains only makeup and occluded images.
Co-attention Aligned Mutual Cross-Attention for Cloth-Changing Person Re-identification

Person re-identification (Re-ID) has been widely studied and achieved significant progress. However, traditional person Re-ID methods primarily rely on cloth-related color appearance, which is unreliable under real-world scenarios when people change their clothes. Cloth-changing person Re-ID that takes this problem into account has received increasing attention recently, but it is more challenging to learn discriminative person identity features, since larger intra-class variation and smaller inter-class easily occur in the image feature space with clothing changes. Beyond appearance features, some known identity-related features can be implicitly encoded in images (e.g., body shapes). In this paper, we first design a novel Shape Semantics Embedding (SSE) module to encode body shape semantic information, which is one of the essential clues to distinguish pedestrians when their clothes change. To better complement image features, we further propose a Co-attention Aligned Mutual Cross-attention (CAMC) framework. Different from previous attention-based fusion strategies, it first aligns features from multiple modalities, then effectively interacts and transfers identity-aware but cloth-irrelevant knowledge between the image space and the body shape space, resulting in a more robust feature representation. To the best of our knowledge, this is the first work to adopt Transformer to handle the multi-modal interaction for cloth-changing person Re-ID. Extensive experiments demonstrate the effectiveness of our proposed method and show the superior performance achieved on several cloth-changing person Re-ID benchmarks. Codes will be available at https://github.com/QizaoWang/CAMC-CCReID .
Energy-Efficient Image Processing Using Binary Neural Networks with Hadamard Transform

Binary neural networks have recently begun to be used as a highly energy- and computation-efficient image processing technique for computer vision tasks. This paper proposes a novel extension of existing binary neural network technology based on the use of a Hadamard transform in the input layer of a binary neural network. Previous state-of-the-art binary neural networks require floating-point arithmetic at several parts of the neural network model computation in order to maintain a sufficient level of accuracy. The Hadamard transform is similar to a Discrete Cosine Transform (used in the popular JPEG image compression method) except that it does not include expensive multiplication operations. In this paper, it is shown that the Hadamard transform can be used to replace the most expensive floating-point arithmetic portion of a binary neural network. In order to test the efficacy of this proposed method, three types of experiments were conducted: application of the proposed method to several state-of-the-art neural network models, verification of its effectiveness in a large image dataset (ImageNet), and experiments to verify the effectiveness of the Hadamard transform by comparing the performance of binary neural networks with and without the Hadamard transform. The results show that the Hadamard transform can be used to implement a highly energy-efficient binary neural network with only a miniscule loss of accuracy.
Reading Arbitrary-Shaped Scene Text from Images Through Spline Regression and Rectification

Scene text in natural images contains a wealth of valuable semantic information. To read scene text from the image, various text spotting techniques that jointly detect and recognize scene text have been proposed in recent years. In this paper, we present a novel end-to-end text spotting network SPRNet for arbitrary-shaped scene text. We propose a parametric B-spline centerline-based representation model to describe the distinctive global shape characteristics of the text, which helps to effectively deal with interferences such as local connection and tight spacing of text and other object, and a text is detected by regressing its shape parameters. Further, exploiting the text’s shape cues, we employ adaptive projection transformations to rectify the feature representation of an irregular text, which improves the accuracy of the subsequent text recognition network. Our method achieves competitive text spotting performance on standard benchmarks through a simple architecture equipped with the proposed text representation and rectification mechanism, which demonstrates the effectiveness of the method in detecting and recognizing scene text with arbitrary shapes.
SEIC: Semantic Embedding with Intermediate Classes for Zero-Shot Domain Generalization

In this work, we address the Zero-Shot Domain Generalization (ZSDG) task, where the goal is to learn a model from multiple source domains, such that it can generalize well to both unseen classes and unseen domains during testing. Since it combines the tasks of Domain Generalization (DG) and Zero-Shot Learning (ZSL), here we explore whether advances in these fields also translate to improved performance for the ZSDG task. Specifically, we build upon a state-of-the-art approach for domain generalization and appropriately modify it such that it can generalize to unseen classes during the testing stage. Towards this goal, we propose to make the feature embedding space semantically meaningful, by not only making an image feature close to its semantic attributes, but also taking into account its similarity with the other neighbouring classes. In addition, in order to reserve space for the unseen classes in the embedding space, we propose to introduce pseudo intermediate classes in between the semantically similar classes during training. This reduces confusion of the similar classes and thus increases the discriminability of the embedding space. Extensive experiments on two large-scale benchmark datasets, namely DomainNet and DomainNet-LS and comparisons with the state-of-the-art approaches show that the proposed framework outperforms all the other techniques on both the datasets.