bluuebunny/crossref_metadata_2025 huggingface dataset:
Detecting Lexical Semantic Change across Corpora with Smooth Manifolds (Student Abstract)
Authors: Goel Anmol , Kumaraguru Ponnurangam
Published: September 2022

Comparing two bodies of text and detecting words with significant lexical semantic shift between them is an important part of digital humanities. Traditional approaches have relied on aligning the different embeddings using the Orthogonal Procrustes problem in the Euclidean space. This study presents a geometric framework that leverages smooth Riemannian manifolds for corpus-specific orthogonal rotations and a corpus-independent scaling metric to project the different vector spaces into a shared latent space. This enables us to capture any affine relationship between the embedding spaces while utilising the rich geometry of smooth manifolds.
Reinforcement Based Learning on Classification Task Yields Better Generalization and Adversarial Accuracy (Student Abstract)
Authors: Gupta Shashi Kant
Published: September 2022

Deep Learning has become interestingly popular in the field of computer vision, mostly attaining near or above human-level performance in various vision tasks. But recent work has also demonstrated that these deep neural networks are very vulnerable to adversarial examples (adversarial examples - inputs to a model which are naturally similar to original data but fools the model in classifying it into a wrong class). In this work, we proposed a novel method to train deep learning models on an image classification task. We used a reward-based optimization function, similar to the vanilla policy gradient method in reinforcement learning to train our model instead of conventional cross-entropy loss. An empirical evaluation on cifar10 dataset showed that our method outperforms the same model architecture trained using cross-entropy loss function (on adversarial training). At the same time, our method generalizes better to the training data with the difference in test accuracy and train accuracy < 2% for most of the time as compared to cross-entropy one, whose difference most of the time remains > 2%.
Rotation-Invariant Gait Identification with Quaternion Convolutional Neural Networks (Student Abstract)
Authors: Jing Bowen , Prabhu Vinay , Gu Angela , Whaley John
Published: September 2022

Accelerometric gait identification systems should ideally be robust to changes in device orientation from the enrollment phase to the deployment phase. However, traditional Convolutional Neural Networks (CNNs) used in these systems compensate poorly for such distributional shifts. In this paper, we target this problem by introducing an SO(3)-equivariant quaternion convolutional kernel inside the CNN. Our architecture (Quaternion CNN) significantly outperforms a traditional CNN in a multi-user gait classification setting. Additionally, the kernels learned by QCNN can be visualized as basis-independent trajectory fragments in Euclidean space, a novel mode of feature visualization and extraction.
Domain Generalisation with Domain Augmented Supervised Contrastive Learning (Student Abstract)
Authors: Le Hoang Son , Akmeliawati Rini , Carneiro Gustavo
Published: September 2022

Domain generalisation (DG) methods address the problem of domain shift, when there is a mismatch between the distributions of training and target domains. Data augmentation approaches have emerged as a promising alternative for DG. However, data augmentation alone is not sufficient to achieve lower generalisation errors. This project proposes a new method that combines data augmentation and domain distance minimisation to address the problems associated with data augmentation and provide a guarantee on the learning performance, under an existing framework. Empirically, our method outperforms baseline results on DG benchmarks.
RL Generalization in a Theory of Mind Game Through a Sleep Metaphor (Student Abstract)
Authors: Malloy Tyler , Klinger Tim , Liu Miao , Tesauro Gerald , Riemer Matthew , Sims Chris R.
Published: September 2022

Training agents to learn efficiently in multi-agent environments can benefit from the explicit modelling of other agent's beliefs, especially in complex limited-information games such as the Hanabi card game. However, generalization is also highly relevant to performance in these games, though model comparisons at large training timescales can be difficult. In this work, we address this by introducing a novel model trained using a sleep metaphor on a reduced complexity version of the Hanabi game. This sleep metaphor consists an altered training regiment, as well as an information-theoretic constraint on the agent's policy. Results from experimentation demonstrate improved performance through this sleep-metaphor method, and provide a promising motivation for using similar techniques in more complex methods that incorporate explicit models of other agent's beliefs.
Quantum Binary Classification (Student Abstract)
Authors: Silva Carla , Aguiar Ana , Dutra Inês
Published: September 2022

We implement a quantum binary classifier where given a dataset of pairs of training inputs and target outputs our goal is to predict the output of a new input. The script is based in a hybrid scheme inspired in an existing PennyLane's variational classifier and to encode the classical data we resort to PennyLane's amplitude encoding embedding template. We use the quantum binary classifier applied to the well known Iris dataset and to a car traffic dataset. Our results show that the quantum approach is capable of performing the task using as few as 2 qubits. Accuracies are similar to other quantum machine learning research studies, and as good as the ones produced by classical classifiers.
Neuro-Symbolic Techniques for Description Logic Reasoning (Student Abstract)
Authors: Singh Gunjan , Mondal Sutapa , Bhatia Sumit , Mutharaju Raghava
Published: September 2022

With the goal to find scalable reasoning approaches, neuro-symbolic techniques have gained significant attention. However, the existing approaches do not take into account the inference capabilities of ontology languages that are based on expressive description logic (such as OWL 2). To fill this gap, we propose two approaches: an ontology-based embedding model for theories in EL++ description logic and a reinforcement learning-based solution for efficient tableau-based reasoning on description logic. We describe promising initial results of our efforts towards these directions and lay down the direction for future work.
SecDD: Efficient and Secure Method for Remotely Training Neural Networks (Student Abstract)
Authors: Sucholutsky Ilia , Schonlau Matthias
Published: September 2022

We leverage what are typically considered the worst qualities of deep learning algorithms - high computational cost, requirement for large data, no explainability, high dependence on hyper-parameter choice, overfitting, and vulnerability to adversarial perturbations - in order to create a method for the secure and efficient training of remotely deployed neural networks over unsecure channels.
Scalable Partial Explainability in Neural Networks via Flexible Activation Functions (Student Abstract)
Authors: Sun Schyler C. , Li Chen , Wei Zhuangkun , Tsourdos Antonios , Guo Weisi
Published: September 2022

Current state-of-the-art neural network explanation methods (e.g. Saliency maps, DeepLIFT, LIME, etc.) focus more on the direct relationship between NN outputs and inputs rather than the NN structure and operations itself, hence there still exists uncertainty over the exact role played by neurons. In this paper, we propose a novel neural network structure with Kolmogorov-Arnold Superposition Theorem based topology and Gaussian Processes based flexible activation function to achieve partial explainability of the neuron inner reasoning. The model feasibility is verified in a case study on binary classification of the banknotes.
Mining Intelligent Patterns using SVAC for Precision Agriculture and Optimizing Irrigation (Student Abstract)
Authors: Vinod Vishal , Gaurav Vipul , Sharma Tushar , Choudhary Savita
Published: September 2022

The ability to leverage the advances in precision agriculture, computer vision, and edge devices can immensely benefit sustainable agriculture yield. Utilizing the available resources to their maximum requires reliable and intelligent real-time insights to optimize and automate the current agriculture infrastructure. In some countries with low internet penetration rates, such systems need offline and extremely efficient edge deployments. We propose a framework that attends to the trifecta of (i) predicting crop water requirements and irrigating the land appropriately, (ii) providing intelligent insights from aerial images and sensor data for crop management that is fully offline, and (iii) effective post-training quantization and model pruning that leverage the lottery ticket hypothesis - an arbitrarily instantiated network containing a subnetwork that when trained independently will perform as well as the original full network, trained for a similar number of cycles for shrinking the machine learning models and improving latency on the edge.
Chinese Character Image Clustering and Classification Based on Object Embedding Model (Student Abstract)
Authors: Wang Mengting , Liang Xun , Xue Yang
Published: September 2022

We proposed Image2Vec, an unsupervised algorithm that learns feature representations for variable-size pieces of Chinese character images.
Is Each Layer Non-trivial in CNN? (Student Abstract)
Authors: Wang Wei , Zhu Yanjie , Cui Zhuoxu , Liang Dong
Published: September 2022

Convolutional neural network (CNN) models have achieved
great success in many fields. With the advent of ResNet, networks
used in practice are getting deeper and wider. However,
is each layer non-trivial in networks? To answer this
question, we trained a network on the training set, then we
replace the network convolution kernels with zeros and test
the result models on the test set. We compared experimental
results with baseline and showed that we can reach similar or
even the same performances. Although convolution kernels
are the cores of networks, we demonstrate that some of them
are trivial and regular in ResNet.
Use of Computer Vision to Develop a Device to Assist Visually Impaired People with Social Distance.
Authors: Nadolskis Lucas
Published: September 2022

The project developed a device to assist blind and visually impaired
users in complying with social distance rules. The main challenge
was how to identify people from other objects and notify the user
accordingly. Equally ambiguous was how to best notify the user that a
person was violating the social distance rules.
The hardware used is a Raspberry Pi running an image classification
algorithm and hooked to a depth camera. The use of prediction labels and
accuracy combined with the distance calculated by the depth camera made
it possible to detect when a person was getting closer than 2 meters
(6 ft) to the user.
The device was tested both in daylight and at night, and with different
lighting conditions. The device responded with a accuracy close to 1.9 meters. which is very acceptable.
Testing showed that the camera is capable of identifying people coming
from a 30 degrees angle from either side at around 1.9 meters of distance,
giving a good range of object detection.
Predictive Agent-Based Modeling of Natural Disasters Using Machine Learning
Authors: Nerrise Favour
Published: September 2022

Current applications of Agent-based Modeling (ABM) in natural phenomena like wildfire land suppression and hurricane forecasting are in monitoring emergent behavior patterns among large groups of people (Hilljegerdes 2018). However, current evacuation times and plans for natural disaster management leave underserved communities vulnerable to substantial financial and welfare losses, especially when false positives during current predictions continue to influence evacuation decisions. A Machine Learning ABM (ML-ABM) model of hurricane trajectories introduces a pioneering opportunity to capture complex physical processes associated with hurricanes while minimizing computational costs and errors, thereby providing more accurate real-time prediction of hurricanes for improved disaster management. This Hurricane Track Prediction ML-ABM model aims to quickly model and predict hurricane tracks in only a few minutes while retaining some of the complex physical process interactions of real storms through feature engineering and deep learning. This work focuses on the implementation of an RNN with bidirectional time-distributed Long-Short Term Memory cells, accounting for positive and negative time direction in time series forecasting. The observations and predictions were represented as a multi-agent system in NetLogo for further emergent pattern analysis in an expanded research by Arthur Drake et. al (2020). The model also evaluates benchmark comparisons against the National Hurricane Center’s 5-Year Average Forecast Errors and the BCD5 Error Model, a combined intensity and track prediction error model that utilizes best track input and models decay over land.
The Adapter-Bot: All-In-One Controllable Conversational Model
Authors: Lin Zhaojiang , Madotto Andrea , Bang Yejin , Fung Pascale
Published: September 2022

In this paper, we present the Adapter-Bot, a generative chat-bot that uses a fixed backbone conversational model such as DialGPT (Zhang et al. 2019) and triggers on-demand dialogue skills via different adapters (Houlsby et al. 2019). Each adapter can be trained independently, thus allowing a continual integration of skills without retraining the entire model. Depending on the skills, the model is able to process multiple knowledge types, such as text, tables, and graphs, in a seamless manner. The dialogue skills can be triggered automatically via a dialogue manager, or manually, thus allowing high-level control of the generated responses. At the current stage, we have implemented 12 response styles (e.g., positive, negative etc.), 6 goal-oriented skills (e.g. weather information, movie recommendation, etc.), and personalized and emphatic responses.
Software for Agent-based Network Simulation and Visualization
Authors: Shepherd Patrick , Batts Isaac , Goldsmith Judy , Hufbauer Emory , Weaver Mia , Zhang Angela
Published: September 2022

We present a network software suite that can model contagions or opinion manipulation in social networks, that combines features from the standard packages and extends them to allow complex, interacting, dynamic topologies, and dynamic heterogeneous agent types, with individual interaction policies. The framework allows for the easy implementation of new agent types, and provides flexible visualization tools to elucidate network behavior over time.
Fashion Focus: Multi-modal Retrieval System for Video Commodity Localization in E-commerce
Authors: Zhang Yanhao , Wang Qiang , Pan Pan , Zheng Yun , Da Cheng , Sun Siyang , Xu Yinghui
Published: September 2022

Nowadays, live-stream and short video shopping in E-commerce have grown exponentially. However, the sellers are required to manually match images of the selling products to the timestamp of exhibition in the untrimmed video, resulting in a complicated process. To solve the problem, we present an innovative demonstration of multi-modal retrieval system called ``Fashion Focus'', which enables to exactly localize the product images in the online video as the focuses. Different modality contributes to the community localization, including visual content, linguistic features and interaction context are jointly investigated via presented multi-modal learning. Our system employs two procedures for analysis, including video content structuring and multi-modal retrieval, to automatically achieve accurate video-to-shop matching. Fashion Focus presents a unified framework that can orientate the consumers towards relevant product exhibitions during watching videos and help the sellers to effectively deliver the products over search and recommendation.
Appearance-Motion Memory Consistency Network for Video Anomaly Detection
Authors: Cai Ruichu , Zhang Hao , Liu Wen , Gao Shenghua , Hao Zhifeng
Published: September 2022

Abnormal event detection in the surveillance video is an essential
but challenging task, and many methods have been
proposed to deal with this problem. The previous methods
either only consider the appearance information or directly
integrate the results of appearance and motion information
without considering their endogenous consistency
semantics explicitly. Inspired by the rule humans identify
the abnormal frames from multi-modality signals, we propose
an Appearance-Motion Memory Consistency Network
(AMMC-Net). Our method first makes full use of the prior
knowledge of appearance and motion signals to explicitly
capture the correspondence between them in the high-level
feature space. Then, it combines the multi-view features to
obtain a more essential and robust feature representation of
regular events, which can significantly increase the gap between
an abnormal and a regular event. In the anomaly detection
phase, we further introduce a commit error in the latent
space joint with the prediction error in pixel space to enhance
the detection accuracy. Solid experimental results on various
standard datasets validate the effectiveness of our approach.
YOLObile: Real-Time Object Detection on Mobile Devices via Compression-Compilation Co-Design
Authors: Cai Yuxuan , Li Hongjia , Yuan Geng , Niu Wei , Li Yanyu , Tang Xulong , Ren Bin , Wang Yanzhi
Published: September 2022

The rapid development and wide utilization of object detection techniques have aroused attention on both accuracy and speed of object detectors. However, the current state-of-the-art object detection works are either accuracy-oriented using a large model but leading to high latency or speed-oriented using a lightweight model but sacrificing accuracy. In this work, we propose YOLObile framework, a real-time object detection on mobile devices via compression-compilation co-design. A novel block-punched pruning scheme is proposed for any kernel size. To improve computational efficiency on mobile devices, a GPU-CPU collaborative scheme is adopted along with advanced compiler-assisted optimizations. Experimental results indicate that our pruning scheme achieves 14x compression rate of YOLOv4 with 49.0 mAP. Under our YOLObile framework, we achieve 17 FPS inference speed using GPU on Samsung Galaxy S20. By incorporating our proposed GPU-CPU collaborative scheme, the inference speed is increased to 19.1 FPS, and outperforms the original YOLOv4 by 5x speedup. Source code is at: https://github.com/nightsnack/YOLObile.
Dual Distribution Alignment Network for Generalizable Person Re-Identification
Authors: Chen Peixian , Dai Pingyang , Liu Jianzhuang , Zheng Feng , Xu Mingliang , Tian Qi , Ji Rongrong
Published: September 2022

Domain generalization (DG) offers a preferable real-world setting for Person Re-Identification (Re-ID), which trains a model using multiple source domain datasets and expects it to perform well in an unseen target domain without any model updating. Unfortunately, most DG approaches are designed explicitly for classification tasks, which fundamentally differs from the retrieval task Re-ID. Moreover, existing applications of DG in Re-ID cannot correctly handle the massive variation among Re-ID datasets. In this paper, we identify two fundamental challenges in DG for Person Re-ID: domain-wise variations and identity-wise similarities. To this end, we propose an end-to-end Dual Distribution Alignment Network (DDAN) to learn domain-invariant features with dual-level constraints: the domain-wise adversarial feature learning and the identity-wise similarity enhancement. These constraints effectively reduce the domain-shift among multiple source domains further while agreeing to real-world scenarios. We evaluate our method in a large-scale DG Re-ID benchmark and compare it with various cutting-edge DG approaches. Quantitative results show that DDAN achieves state-of-the-art performance.
RSGNet: Relation based Skeleton Graph Network for Crowded Scenes Pose Estimation
Authors: Dai Yan , Wang Xuanhan , Gao Lianli , Song Jingkuan , Shen Heng Tao
Published: September 2022

Despite of the recent great progress on multi-person pose estimation, existing solutions still remain challenging under the condition of "crowded scenes'', where RGB images capture complex real-world scenes with highly-overlapped people, severe occlusions and diverse postures. In this work, we focus on two main problems: 1) how to design an effective pipeline for crowded scenes pose estimation; and 2) how to equip this pipeline with the ability of relation modeling for interference resolving. To tackle these problems, we propose a new pipeline named Relation based Skeleton Graph Network (RSGNet). Unlike existing works that directly predict joints-of-target by labeling joints-of-interference as false positive, we first encourage all joints to be predicted. And then, a Target-aware Relation Parser (TRP) is designed to model the relation over all predicted joints, resulting in a target-aware encoding. This new pipeline will largely relieve the confusion of the joints estimation model when seeing identical joints with totally distinct labels (e.g., the identical hand exists in two bounding boxes). Furthermore, we introduce a Skeleton Graph Machine (SGM) to model the skeleton-based commonsense knowledge, aiming to estimate the target pose with the constraint of human body structure. Such skeleton-based constraint can help to deal with the challenges in crowded scenes from a reasoning perspective. Solid experiments on pose estimation benchmarks demonstrate that our method outperforms existing state-of-the-art methods.
CompFeat: Comprehensive Feature Aggregation for Video Instance Segmentation
Authors: Fu Yang , Yang Linjie , Liu Ding , Huang Thomas S. , Shi Humphrey
Published: September 2022

Video instance segmentation is a complex task in which we need to detect, segment, and track each object for any given video. Previous approaches only utilize single-frame features for the detection, segmentation, and tracking of objects and they suffer in the video scenario due to several distinct challenges such as motion blur and drastic appearance change. To eliminate ambiguities introduced by only using single-frame features, we propose a novel comprehensive feature aggregation approach (CompFeat) to refine features atboth frame-level and object-level with temporal and spatial context information. The aggregation process is carefully designed with a new attention mechanism which significantly increases the discriminative power of the learned features. We further improve the tracking capability of our model through a siamese design by incorporating both feature similarities and spatial similarities. Experiments conducted on the YouTube-VIS dataset validate the effectiveness of proposed CompFeat.
The Complexity of Object Association in Multiple Object Tracking
Authors: Ganian Robert , Hamm Thekla , Ordyniak Sebastian
Published: September 2022

Object association, i.e., the identification of which observations correspond to the same object, is a central task for the area of multiple object tracking. Two prominent models capturing this task have been introduced in the literature: the Lifted Multicut model and the more recent Lifted Paths model. Here, we carry out a detailed complexity-theoretic study of the problems arising from these two models that is aimed at complementing previous empirical work on object association. We obtain a comprehensive complexity map for both models that takes into account natural restrictions to instances such as possible bounds on the number of frames, number of tracked objects and branching degree, as well as less explicit structural restrictions such as having bounded treewidth. Our results include new fixed-parameter and XP algorithms for the problems as well as hardness proofs which altogether indicate that the Lifted Paths problem exhibits a more favorable complexity behavior than Lifted Multicut.
SnapMix: Semantically Proportional Mixing for Augmenting Fine-grained Data
Authors: Huang Shaoli , Wang Xinchao , Tao Dacheng
Published: September 2022

Data mixing augmentation has proved effective in training deep models. Recent methods mix labels mainly according to the mixture proportion of image pixels. Due to the major discriminative information of a fine-grained image usually resides in subtle regions, these methods tend to introduce heavy label noise in fine-grained recognition. We propose Semantically Proportional Mixing (SnapMix) that exploits class activation map (CAM) to lessen the label noise in augmenting fine-grained data. SnapMix generates the target label for a mixed image by estimating its intrinsic semantic composition. This strategy can adapt to asymmetric mixing operations and ensure semantic correspondence between synthetic images and target labels. Experiments show that our method consistently outperforms existing mixed-based approaches regardless of different datasets or network depths. Further, by incorporating the mid-level features, the proposed SnapMix achieves top-level performance, demonstrating its potential to serve as a strong baseline for fine-grained recognition.
Matching on Sets: Conquer Occluded Person Re-identification Without Alignment
Authors: Jia Mengxi , Cheng Xinhua , Zhai Yunpeng , Lu Shijian , Ma Siwei , Tian Yonghong , Zhang Jian
Published: September 2022

Occluded person re-identification (re-ID) is a challenging task as different human parts may become invisible in cluttered scenes, making it hard to match person images of different identities. Most existing methods address this challenge by aligning spatial features of body parts according to semantic information (e.g. human poses) or feature similarities but this approach is complicated and sensitive to noises. This paper presents Matching on Sets (MoS), a novel method that positions occluded person re-ID as a set matching task without requiring spatial alignment. MoS encodes a person image by a pattern set as represented by a `global vector’ with each element capturing one specific visual pattern, and it introduces Jaccard distance as a metric to compute the distance between pattern sets and measure image similarity. To enable Jaccard distance over continuous real numbers, we employ minimization and maximization to approximate the operations of intersection and union, respectively. In addition, we design a Jaccard triplet loss that enhances the pattern discrimination and allows to embed set matching into deep neural networks for end-to-end training. In the inference stage, we introduce a conflict penalty mechanism that detects mutually exclusive patterns in the pattern union of image pairs and decreases their similarities accordingly. Extensive experiments over three widely used datasets (Market1501, DukeMTMC and Occluded-DukeMTMC) show that MoS achieves superior re-ID performance. Additionally, it is tolerant of occlusions and outperforms the state-of-the-art by large margins for Occluded-DukeMTMC.
Multi-level Distance Regularization for Deep Metric Learning
Authors: Kim Yonghyun , Park Wonpyo
Published: September 2022

We propose a novel distance-based regularization method for deep metric learning called Multi-level Distance Regularization (MDR).
MDR explicitly disturbs a learning procedure by regularizing pairwise distances between embedding vectors into multiple levels that represents a degree of similarity between a pair.
In the training stage, the model is trained with both MDR and an existing loss function of deep metric learning, simultaneously; the two losses interfere with the objective of each other, and it makes the learning process difficult.
Moreover, MDR prevents some examples from being ignored or overly influenced in the learning process.
These allow the parameters of the embedding network to be settle on a local optima with better generalization.
Without bells and whistles, MDR with simple Triplet loss achieves the-state-of-the-art performance in various benchmark datasets: CUB-200-2011, Cars-196, Stanford Online Products, and In-Shop Clothes Retrieval.
We extensively perform ablation studies on its behaviors to show the effectiveness of MDR.
By easily adopting our MDR, the previous approaches can be improved in performance and generalization ability.
Static-Dynamic Interaction Networks for Offline Signature Verification
Authors: Li Huan , Wei Ping , Hu Ping
Published: September 2022

Offline signature verification is a challenging issue that is widely used in various fields. Previous approaches model this task as a static feature matching or distance metric problem of two images. In this paper, we propose a novel Static-Dynamic Interaction Network (SDINet) model which introduces sequential representation into static signature images. A static signature image is converted to sequences by assuming pseudo dynamic processes in the static image. A static representation extracting deep features from signature images describes the global information of signatures. A dynamic representation extracting sequential features with LSTM networks characterizes the local information of signatures. A dynamic-to-static attention is learned from the sequences to
refine the static features. Through the static-to-dynamic conversion and the dynamic-to-static attention, the static representation and dynamic representation are unified into a compact framework. The proposed method was evaluated on four popular datasets of different languages. The extensive experimental results manifest the strength of our model.
Exploiting Learnable Joint Groups for Hand Pose Estimation
Authors: Li Moran , Gao Yuan , Sang Nong
Published: September 2022

In this paper, we propose to estimate 3D hand pose by recovering the 3D coordinates of joints in a group-wise manner, where less-related joints are automatically categorized into different groups and exhibit different features. This is different from the previous methods where all the joints are considered holistically and share the same feature. The benefits of our method are illustrated by the principle of multi-task learning (MTL), i.e., by separating less-related joints into different groups (as different tasks), our method learns different features for each of them, therefore efficiently avoids the negative transfer (among less related tasks/groups of joints). The key of our method is a novel binary selector that automatically selects related joints into the same group. We implement such a selector with binary values stochastically sampled from a Concretedistribution, which is constructed using Gumbel softmax on trainable parameters. This enables us to preserve the differentiable property of the whole network. We further exploit features from those less-related groups by carrying out an additional feature fusing scheme among them, to learn more discriminative features. This is realized by implementing multiple 1x1 convolutions on the concatenated features, where each joint group contains a unique 1x1convolution for feature fusion. The detailed ablation analysis and the extensive experiments on several benchmark datasets demonstrate the promising performance of the proposed method over the state-of-the-art (SOTA) methods. Besides, our method achieves top-1 among all the methods that do not exploit the dense 3D shape labels on the most recently released FreiHAND competition at the submission date. The source code and models are available at https://github.com/moranli-aca/LearnableGroups-Hand.
Category Dictionary Guided Unsupervised Domain Adaptation for Object Detection
Authors: Li Shuai , Huang Jianqiang , Hua Xian-Sheng , Zhang Lei
Published: September 2022

Unsupervised domain adaption (UDA) is a promising solution to enhance the generalization ability of a model from a source domain to a target domain without manually annotating labels for target data. Recent works in cross-domain object detection mostly resort to adversarial feature adaptation to match the marginal distributions of two domains. However, perfect feature alignment is hard to achieve and is likely to cause negative transfer due to the high complexity of object detection. In this paper, we propose a category dictionary guided (CDG) UDA model for cross-domain object detection, which learns category-specific dictionaries from the source domain to represent the candidate boxes in target domain. The representation residual can be used for not only pseudo label assignment but also quality (e.g., IoU) estimation of the candidate box. A residual weighted self-training paradigm is then developed to implicitly align source and target domains for detection model training. Compared with decision boundary based classifiers such as softmax, the proposed CDG scheme can select more informative and reliable pseudo-boxes. Experimental results on benchmark datasets show that the proposed CDG significantly exceeds the state-of-the-arts in cross-domain object detection.
SD-Pose: Semantic Decomposition for Cross-Domain 6D Object Pose Estimation
Authors: Li Zhigang , Hu Yinlin , Salzmann Mathieu , Ji Xiangyang
Published: September 2022

The current leading 6D object pose estimation methods rely heavily on annotated real data, which is highly costly to acquire. To overcome this, many works have proposed to introduce computer-generated synthetic data.
However, bridging the gap between the synthetic and real data remains a severe problem.
Images depicting different levels of realism/semantics usually have different transferability between the synthetic and real domains.
Inspired by this observation, we introduce an approach, SD-Pose, that explicitly decomposes the input image into multi-level semantic representations and then combines the merits of each representation to bridge the domain gap.
Our comprehensive analyses and experiments show that our semantic decomposition strategy can fully utilize the different domain similarities of different representations, thus allowing us to outperform the state of the art on modern 6D object pose datasets without accessing any real data during training.
Exploiting Audio-Visual Consistency with Partial Supervision for Spatial Audio Generation
Authors: Lin Yan-Bo , Wang Yu-Chiang Frank
Published: September 2022

Human perceives rich auditory experience with distinct sound heard by ears. Videos recorded with binaural audio particular simulate how human receives ambient sound. However, a large number of videos are with monaural audio only, which would degrade the user experience due to the lack of ambient information. To address this issue, we propose an audio spatialization framework to convert a monaural video into a binaural one exploiting the relationship across audio and visual components. By preserving the left-right consistency in both audio and visual modalities, our learning strategy can be viewed as a self-supervised learning technique, and alleviates the dependency on a large amount of video data with ground truth binaural audio data during training. Experiments on benchmark datasets confirm the effectiveness of our proposed framework in both semi-supervised and fully supervised scenarios, with ablation studies and visualization further support the use of our model for audio spatialization.
F2Net: Learning to Focus on the Foreground for Unsupervised Video Object Segmentation
Authors: Liu Daizong , Yu Dongdong , Wang Changhu , Zhou Pan
Published: September 2022

Although deep learning based methods have achieved great progress in unsupervised video object segmentation, difficult scenarios (e.g., visual similarity, occlusions, and appearance changing) are still no well-handled. To alleviate these issues, we propose a novel Focus on Foreground Network (F2Net), which delves into the intra-inter frame details for the foreground objects and thus effectively improve the segmentation performance. Specifically, our proposed network consists of three main parts: Siamese Encoder Module, Center Guiding Appearance Diffusion Module, and Dynamic Information Fusion Module. Firstly, we take a siamese encoder to extract the feature representations of paired frames (reference frame and current frame). Then, a Center Guiding Appearance Diffusion Module is designed to capture the inter-frame feature (dense correspondences between reference frame and current frame), intra-frame feature (dense correspondences in current frame), and original semantic feature of current frame. Different from the Anchor Diffusion Network, we establish a Center Prediction Branch to predict the center location of the foreground object in current frame and leverage the center point information as spatial guidance prior to enhance the inter-frame and intra-frame feature extraction, and thus the feature representation considerably focus on the foreground objects. Finally, we propose a Dynamic Information Fusion Module to automatically select relatively important features through three aforementioned different level features. Extensive experiments on DAVIS, Youtube-object, and FBMS datasets show that our proposed F2Net achieves the state-of-the-art performance with significant improvement.
Learning Hybrid Relationships for Person Re-identification
Authors: Liu Shuang , Huang Wenmin , Zhang Zhong
Published: September 2022

Recently, the relationship among individual pedestrian images and the relationship among pairwise pedestrian images have become attractive for person re-identification (re-ID) as they effectively improve the ability of feature representation. In this paper, we propose a novel method named Hybrid Relationship Network (HRNet) to learn the two types of relationships in a unified framework that makes use of their own advantages. Specifically, for the relationship among individual pedestrian images, we take the features of pedestrian images as the nodes to construct a locally-connected graph, so as to improve the discriminative ability of nodes. Meanwhile, we propose the consistent node constraint to inject the identity information into the graph learning process and guide the information to propagate accurately. As for the relationship among pairwise pedestrian images, we treat the feature differences of pedestrian images as the nodes to construct a fully-connected graph so as to estimate robust similarity of nodes. Furthermore, we propose the inter-graph propagation to alleviate the information loss for the fully-connected graph. Extensive experiments on Market-1501, DukeMTMCreID, CUHK03 and MSMT17 demonstrate that the proposed HRNet outperforms the state-of-the-art methods.
Subtype-aware Unsupervised Domain Adaptation for Medical Diagnosis
Authors: Liu Xiaofeng , Liu Xiongchang , Hu Bo , Ji Wenxuan , Xing Fangxu , Lu Jun , You Jane , Kuo C.-C. Jay , El Fakhri Georges , Woo Jonghye
Published: September 2022

Recent advances in unsupervised domain adaptation (UDA) show that transferable prototypical learning presents a powerful means for class conditional alignment, which encourages the closeness of cross-domain class centroids. However, the cross-domain inner-class compactness and the underlying fine-grained subtype structure remained largely underexplored. In this work, we propose to adaptively carry out the fine-grained subtype-aware alignment by explicitly enforcing the class-wise separation and subtype-wise compactness with intermediate pseudo labels. Our key insight is that the unlabeled subtypes of a class can be divergent to one another with different conditional and label shifts, while inheriting the local proximity within a subtype. The cases with or without the prior information on subtype numbers are investigated to discover the underlying subtype structure in an online fashion. The proposed subtype-aware dynamic UDA achieves promising results on a medical diagnosis task.
ACSNet: Action-Context Separation Network for Weakly Supervised Temporal Action Localization
Authors: Liu Ziyi , Wang Le , Zhang Qilin , Tang Wei , Yuan Junsong , Zheng Nanning , Hua Gang
Published: September 2022

The object of Weakly-supervised Temporal Action Localization (WS-TAL) is to localize all action instances in an untrimmed video with only video-level supervision. Due to the lack of frame-level annotations during training, current WS-TAL methods rely on attention mechanisms to localize the foreground snippets or frames that contribute to the video-level classification task. This strategy frequently confuse context with the actual action, in the localization result. Separating action and context is a core problem for precise WS-TAL, but it is very challenging and has been largely ignored in the literature. In this paper, we introduce an Action-Context Separation Network (ACSNet) that explicitly takes into account context for accurate action localization. It consists of two branches (i.e., the Foreground-Background branch and the Action-Context branch). The Foreground-Background branch first distinguishes foreground from background within the entire video while the Action-Context branch further separates the foreground as action and context. We associate video snippets with two latent components (i.e., a positive component and a negative component), and their different combinations can effectively characterize foreground, action and context. Furthermore, we introduce extended labels with auxiliary context categories to facilitate the learning of action-context separation. Experiments on THUMOS14 and ActivityNet v1.2/v1.3 datasets demonstrate the ACSNet outperforms existing state-of-the-art WS-TAL methods by a large margin.
PointINet: Point Cloud Frame Interpolation Network
Authors: Lu Fan , Chen Guang , Qu Sanqing , Li Zhijun , Liu Yinlong , Knoll Alois
Published: September 2022

LiDAR point cloud streams are usually sparse in time dimension, which is limited by hardware performance. Generally, the frame rates of mechanical LiDAR sensors are 10 to 20 Hz, which is much lower than other commonly used sensors like cameras. To overcome the temporal limitations of LiDAR sensors, a novel task named Point Cloud Frame Interpolation is studied in this paper. Given two consecutive point cloud frames, Point Cloud Frame Interpolation aims to generate intermediate frame(s) between them. To achieve that, we propose a novel framework, namely Point Cloud Frame Interpolation Network (PointINet). Based on the proposed method, the low frame rate point cloud streams can be upsampled to higher frame rates. We start by estimating bi-directional 3D scene flow between the two point clouds and then warp them to the given time step based on the 3D scene flow. To fuse the two warped frames and generate intermediate point cloud(s), we propose a novel learning-based points fusion module, which simultaneously takes two warped point clouds into consideration. We design both quantitative and qualitative experiments to evaluate the performance of the point cloud frame interpolation method and extensive experiments on two large scale outdoor LiDAR datasets demonstrate the effectiveness of the proposed PointINet. Our code is available at https://github.com/ispc-lab/PointINet.git.
Few-Shot Lifelong Learning
Authors: Mazumder Pratik , Singh Pravendra , Rai Piyush
Published: September 2022

Many real-world classification problems often have classes with very few labeled training samples. Moreover, all possible classes may not be initially available for training, and may be given incrementally. Deep learning models need to deal with this two-fold problem in order to perform well in real-life situations. In this paper, we propose a novel Few-Shot Lifelong Learning (FSLL) method that enables deep learning models to perform lifelong/continual learning on few-shot data. Our method selects very few parameters from the model for training every new set of classes instead of training the full model. This helps in preventing overfitting. We choose the few parameters from the model in such a way that only the currently unimportant parameters get selected. By keeping the important parameters in the model intact, our approach minimizes catastrophic forgetting. Furthermore, we minimize the cosine similarity between the new and the old class prototypes in order to maximize their separation, thereby improving the classification performance. We also show that integrating our method with self-supervision improves the model performance significantly. We experimentally show that our method significantly outperforms existing methods on the miniImageNet, CIFAR-100, and CUB-200 datasets. Specifically, we outperform the state-of-the-art method by an absolute margin of 19.27% for the CUB dataset.
TDAF: Top-Down Attention Framework for Vision Tasks
Authors: Pang Bo , Li Yizhuo , Li Jiefeng , Li Muchen , Cao Hanwen , Lu Cewu
Published: September 2022

Human attention mechanisms often work in a top-down manner, yet it is not well explored in vision research. Here, we propose the Top-Down Attention Framework (TDAF) to capture top-down attentions, which can be easily adopted in most existing models. The designed Recursive Dual-Directional Nested Structure in it forms two sets of orthogonal paths, recursive and structural ones, where bottom-up spatial features and top-down attention features are extracted respectively. Such spatial and attention features are nested deeply, therefore, the proposed framework works in a mixed top-down and bottom-up manner. Empirical evidence shows that our TDAF can capture effective stratified attention information and boost performance. ResNet with TDAF achieves 2.0% improvements on ImageNet. For object detection, the performance is improved by 2.7% AP over FCOS. For pose estimation, TDAF improves the baseline by 1.6%. And for action recognition, the 3D-ResNet adopting TDAF achieves improvements of 1.7% accuracy.
Vid-ODE: Continuous-Time Video Generation with Neural Ordinary Differential Equation
Authors: Park Sunghyun , Kim Kangyeol , Lee Junsoo , Choo Jaegul , Lee Joonseok , Kim Sookyung , Choi Edward
Published: September 2022

Video generation models often operate under the assumption of fixed frame rates, which leads to suboptimal performance when it comes to handling flexible frame rates (e.g., increasing the frame rate of the more dynamic portion of the video as well as handling missing video frames). To resolve the restricted nature of existing video generation models' ability to handle arbitrary timesteps, we propose continuous-time video generation by combining neural ODE (Vid-ODE) with pixel-level video processing techniques. Using ODE-ConvGRU as an encoder, a convolutional version of the recently proposed neural ODE, which enables us to learn continuous-time dynamics, Vid-ODE can learn the spatio-temporal dynamics of input videos of flexible frame rates. The decoder integrates the learned dynamics function to synthesize video frames at any given timesteps, where the pixel-level composition technique is used to maintain the sharpness of individual frames. With extensive experiments on four real-world video datasets, we verify that the proposed Vid-ODE outperforms state-of-the-art approaches under various video generation settings, both within the trained time range (interpolation) and beyond the range (extrapolation). To the best of our knowledge, Vid-ODE is the first work successfully performing continuous-time video generation using real-world videos.