bluuebunny/crossref_metadata_2025 huggingface dataset:
GANs Unplugged
Authors: Virtue Patrick
Published: September 2022

With the influx of deepfake and style transfer technology in today's news and social media, everyone is told that these applications are powered by artificial intelligence and deep learning, but too often the explanation of how it works goes no further. Rather than waiting until second semester of computer science grad school to learn about generative adversarial networks (GANs), we propose a classroom activity to introduce GANs to secondary school students.

Our GANs Unplugged activity steps outside a traditional classroom environment and forms groups of students that physically act as the various components of a GAN.
Students in the Real and Generator (i.e. Fake) groups both draw sketches of animals on index cards, but only the Real group has the secret, specific instructions on how to draw their animals. A third Discriminator group tries to determine if each card it receives came from the Real group or the Fake group. The Discriminator group also makes a single feedback mark on the drawing to suggest how that drawing could be more "real" before passing the card back to the group that drew it. As this game progresses, the Discriminators get better at determining real and fake, that is until the Generators adapt and learn to draw more and more realistic "fake" images.

We describe the details of GANs Unplugged and present our experience running this activity at three different artificial intelligence summer camps for high school students.
Transfer Learning of Engagement Recognition within Robot-Assisted Therapy for Children with Autism
Authors: Rakhymbayeva Nazerke , Sandygulova Anara
Published: September 2022

Social robots deployed in the therapy of autism is a promising and important research domain. Recently, an increasing amount of work is being conducted utilizing a social robot as a mediator between a therapist and a child with autism. Being able to evaluate how engaged a child is both offline and in real-time would improve the quality of the provided robot-assisted intervention and also provide objective metrics for later analysis by the therapist. The state-of-the-art engagement recognition is challenged by the diverse styles of expressing engagement by this vulnerable population group. To this end, this PhD project aims to explore how transfer learning can improve the recognition accuracy of children's engagement with the robot or another human. We will utilize four publicly available multi-modal datasets to discover a suitable feature representation of engagement during various types of activities with the robot.
Semi-Discrete Social Recommendation (Student Abstract)
Authors: Luo Fangyuan , Wu Jun , Wang Haishuai
Published: September 2022

Combining matrix factorization (MF) with network embedding (NE) has been a promising solution to social recommender systems. However, such a scheme suffers from the online predictive efficiency issue due to the ever-growing users and items. In this paper, we propose a novel hashing-based social recommendation model, called semi-discrete socially embedded matrix factorization (S2MF), which leverages the dual advantages of social information for recommendation effectiveness and hashing trick for online predictive efficiency. Experimental results demonstrate the advantages of S2MF over state-of-the-art discrete recommendation models and its real-valued competitors.
Successive Halving Top-k Operator
Authors: Pietruszka Michał , Borchmann Łukasz , Graliński Filip
Published: September 2022

We propose a differentiable successive halving method of relaxing the top-k operator, rendering gradient-based optimization possible. The need to perform softmax iteratively on the entire vector of scores is avoided using a tournament-style selection. As a result, a much better approximation of top-k and lower computational cost is achieved compared to the previous approach.
A New Robust Subspace Recovery Algorithm (Student Abstract)
Authors: Wan Guihong , Schweitzer Haim
Published: September 2022

A common task in data analysis is to compute an approximate embedding of the data
in a low dimensional subspace. This is used, for example, for dimensionality reduction.
Robust Subspace Recovery computes the embedding by ignoring a fraction of the data
considered as outliers.
Its performance can be evaluated by how accurate the inliers (non-outliers) are represented.
We propose a new algorithm that outperforms the current state of the art
when the data is dominated by outliers.
The main idea is to rank each point by evaluating the change in the global PCA error
when that point is considered as an outlier.
We show that this lookahead procedure can be implemented efficiently
by centered rank-one modifications.
Change or Not: A Simple Approach for Plug and Play Language Models on Sentiment Control
Authors: Xu Chen , Zhao Jianyu , Li Rang , Hu Changjian , Xiao Chuangbai
Published: September 2022

Text generation with sentiment control is difficult without fine-tuning or modifying the model architecture. Plug and Play Language Model (PPLM) utilizes an external sentiment classifier to update the hidden states of GPT-2 at each time step. It does not change the parameters but achieves competitive performance. However, fluency is impaired due to the instability of the hidden states. Moreover, the classifier is not strong because of the way it is trained with partial texts, hence it is difficult to guide the generation in the process. To solve the above problems, in this paper, we first propose a fixed threshold method based on the Valence-Arousal-Dominance (VAD) lexicon to decide whether to change a word, which keeps the fluency of the original LM to the greatest extent. Furthermore, for the improvement of sentiment alignment, we propose a dynamic threshold method that utilizes VAD-based loss to make the threshold dynamic. Experiments demonstrate that our methods outperform the baseline with a great margin significantly both on fluency and sentiment accuracy.
Using Remote Sensing Imagery and Machine Learning to Predict Poaching in Wildlife Conservation Parks
Authors: Guo Rachel
Published: September 2022

Illegal wildlife poaching is driving the loss of biodiversity. To combat poaching, rangers patrol expansive protected areas for illegal poaching activity. However, rangers often cannot comprehensively search such large parks. Thus, the Protection Assistant for Wildlife Security (PAWS) uses machine learning to help identify the areas with highest poaching risk. As PAWS is deployed to parks around the world, we recognized that many parks have limited resources for data collection and therefore have scarce feature sets. To ensure under-resourced parks have access to meaningful poaching predictions, we introduce the use of publicly available remote sensing data to extract features for parks. By employing this data from Google Earth Engine, we also incorporate previously unavailable dynamic data such as climate and primary production patterns to enrich predictions with seasonal trends. We automate the entire data-to-deployment pipeline and find that, with only using publicly available data, we recuperate prediction performance comparable to predictions made using features manually computed by park specialists. We conclude that the inclusion of satellite imagery creates for a robust system through which parks of any resource level can benefit from poaching risks for years to come.
Doc2Bot: Document grounded Bot Framework
Authors: Fadnis Kshitij , Dhoolia Pankaj , Zhu Li , Liao Q. Vera , Ross Steven , Mills Nathaniel , Joshi Sachindra , Lastras Luis
Published: September 2022

Conversational agents, or chatbots, are widely used to provide customer care and other informational support. Currently, the development of chatbots using standard frameworks requires a lot of manual crafting by subject matter experts (SMEs). On the other hand, while learning-based approaches to dialog have made significant advancements, they require training with a large volume of dialog data, which chatbot developers typically do not have access to. To tackle these challenges, we introduce DOC2BOT, a system that supports the automated construction of chatbots by digesting various forms of documents such as business manuals, HowTos, and customer support pages that organizations own. In addition to that, DOC2BOT provides a user-friendly experience to SMEs, and to minimize their effort by supporting intuitive interactions and streamlining their workflow.
NEO: A System for Identifying New Emerging Occupation from Job Ads
Authors: Giabelli Anna , Malandri Lorenzo , Mercorio Fabio , Mezzanzanica Mario , Seveso Andrea
Published: September 2022

We demonstrate NEO, a tool for automatically enriching the European Occupation and Skill Taxonomy (ESCO) with terms that represents new occupations extracted from million Online Job Advertisements (OJAs).
NEO proposes (i) a novel metric that allows one to measure the semantic similarity between words in a taxonomy, and (ii) a set of measures that estimate the adherence of new terms to the most suited taxonomic concept, enabling the user to evaluate the suggestions.
To test its effectiveness, NEO has been evaluated over 2M+ 2018 UK job ads, along with a user-study to confirm the usefulness of NEO in the taxonomy enrichment task.
Mobile-based Clock Drawing Test for Detecting Early Signs of Dementia
Authors: Jiang Hongchao , Zhang Yanci , Zeng Zhiwei , Ji Jun , Wang Yu , Chi Ying , Miao Chunyan
Published: September 2022

Dementia is one of the major causes of disability and dependency among older people. Early detection is the key for preserving the quality of life of the patients and reducing caring costs. The Clock Drawing Test (CDT) is commonly used by clinicians to screen for early signs of dementia. We build an automated CDT that runs on mobile platforms, enabling convenient and frequent self-monitoring and testing at minimal costs. Our system combines both a spatial-temporal approach and a purely image-based deep learning approach to analyze and evaluate the hand-drawn clocks based on established clinical criteria. Our system produces scores that are highly correlated with expert human raters.
Business Entity Matching with Siamese Graph Convolutional Networks
Authors: Krivosheev Evgeny , Atzeni Mattia , Mirylenka Katsiaryna , Scotton Paolo , Miksovic Christoph , Zorin Anton
Published: September 2022

Data integration has been studied extensively for decades and approached from different angles. However, this domain still remains largely rule-driven and lacks universal automation. Recent developments in machine learning and in particular deep learning have opened the way to more general and efficient solutions to data-integration tasks. In this paper, we demonstrate an approach that allows modeling and integrating entities by leveraging their relations and contextual information. This is achieved by combining siamese and graph neural networks to effectively propagate information between connected entities and support high scalability. We evaluated our approach on the task of integrating data about business entities, demonstrating that it outperforms both traditional rule-based systems and other deep learning approaches.
RadarMath: An Intelligent Tutoring System for Math Education
Authors: Lu Yu , Pian Yang , Chen Penghe , Meng Qinggang , Cao Yunbo
Published: September 2022

We propose and implement a novel intelligent tutoring system, called RadarMath, to support intelligent and personalized learning for math education. The system provides the services including automatic grading and personalized learning guidance. Specifically, two automatic grading models are designed to accomplish the tasks for scoring the text-answer and formula-answer questions respectively. An education-oriented knowledge graph with the individual learner’s knowledge state is used as the key tool for guiding the personalized learning process. The system demonstrates how the relevant AI techniques could be applied in today's intelligent tutoring systems.
SkeletonVis: Interactive Visualization for Understanding Adversarial Attacks on Human Action Recognition Models
Authors: Park Haekyu , Wang Zijie J. , Das Nilaksh , Paul Anindya S. , Perumalla Pruthvi , Zhou Zhiyan , Chau Duen Horng
Published: September 2022

Skeleton-based human action recognition technologies are increasingly used in video-based applications, such as home robotics, healthcare on the aging population, and surveillance. However, such models are vulnerable to adversarial attacks, raising serious concerns for their use in safety-critical applications. To develop an effective defense against attacks, it is essential to understand how such attacks mislead the pose detection models into making incorrect predictions. We present SkeletonVis, the first interactive system that visualizes how the attacks work on the models to enhance human understanding of attacks.
EasyASR: A Distributed Machine Learning Platform for End-to-end Automatic Speech Recognition
Authors: Wang Chengyu , Cheng Mengli , Hu Xu , Huang Jun
Published: September 2022

We present EasyASR, a distributed machine learning platform for training and serving large-scale Automatic Speech Recognition (ASR) models, as well as collecting and processing audio data at scale. Our platform is built upon the Machine Learning Platform for AI of Alibaba Cloud. Its main functionality is to support efficient learning and inference for end-to-end ASR models on distributed GPU clusters. It allows users to learn ASR models with either pre-defined or user-customized network architectures via simple user interface. On EasyASR, we have produced state-of-the-art results over several public datasets for Mandarin speech recognition.
Localization in the Crowd with Topological Constraints
Authors: Abousamra Shahira , Hoai Minh , Samaras Dimitris , Chen Chao
Published: September 2022

We address the problem of crowd localization, i.e., the prediction of dots corresponding to people in a crowded scene. Due to various challenges, a localization method is prone to spatial semantic errors, i.e., predicting multiple dots within a same person or collapsing multiple dots in a cluttered region. We propose a topological approach targeting these semantic errors. We introduce a topological constraint that teaches the model to reason about the spatial arrangement of dots. To enforce this constraint, we define a persistence loss based on the theory of persistent homology. The loss compares the topographic landscape of the likelihood map and the topology of the ground truth. Topological reasoning improves the quality of the localization algorithm especially near cluttered regions. On multiple public benchmarks, our method outperforms previous localization methods. Additionally, we demonstrate the potential of our method in improving the performance in the crowd counting task.
Optical Flow Estimation from a Single Motion-blurred Image
Authors: Argaw Dawit Mureja , Kim Junsik , Rameau Francois , Cho Jae Won , Kweon In So
Published: September 2022

In most of computer vision applications, motion blur is regarded as an undesirable artifact. However, it has been shown that motion blur in an image may have practical interests in fundamental computer vision problems. In this work, we propose a novel framework to estimate optical flow from a single motion-blurred image in an end-to-end manner. We design our network with transformer networks to learn globally and locally varying motions from encoded features of a motion-blurred input, and decode left and right frame features without explicit frame supervision. A flow estimator network is then used to estimate optical flow from the decoded features in a coarse-to-fine manner. We qualitatively and quantitatively evaluate our model through a large set of experiments on synthetic and real motion-blur datasets. We also provide in-depth analysis of our model in connection with related approaches to highlight the effectiveness and favorability of our approach. Furthermore, we showcase the applicability of the flow estimated by our method on deblurring and moving object segmentation tasks.
Spatial-temporal Causal Inference for Partial Image-to-video Adaptation
Authors: Chen Jin , Wu Xinxiao , Hu Yao , Luo Jiebo
Published: September 2022

Image-to-video adaptation leverages off-the-shelf learned models in labeled images to help classification in unlabeled videos, thus alleviating the high computation overhead of training a video classifier from scratch. This task is very challenging since there exist two types of domain shifts between images and videos: 1) spatial domain shift caused by static appearance variance between images and video frames, and 2) temporal domain shift caused by the absence of dynamic motion in images. Moreover, for different video classes, these two domain shifts have different effects on the domain gap and should not be treated equally during adaptation. In this paper, we propose a spatial-temporal causal inference framework for image-to-video adaptation. We first construct a spatial-temporal causal graph to infer the effects of the spatial and temporal domain shifts by performing counterfactual causality. We then learn causality-guided bidirectional heterogeneous mappings between images and videos to adaptively reduce the two domain shifts. Moreover, to relax the assumption that the label spaces of the image and video domains are the same by the existing methods, we incorporate class-wise alignment into the learning of image-video mappings to perform partial image-to-video adaptation where the image label space subsumes the video label space. Extensive experiments on several video datasets have validated the effectiveness of our proposed method.
A Unified Multi-Scenario Attacking Network for Visual Object Tracking
Authors: Chen Xuesong , Fu Canmiao , Zheng Feng , Zhao Yong , Li Hongsheng , Luo Ping , Qi Guo-Jun
Published: September 2022

Existing methods of adversarial attacks successfully generate adversarial examples to confuse Deep Neural Networks (DNNs) of image classification and object detection, resulting in wrong predictions. However, these methods are difficult to attack models of video object tracking, because the tracking algorithms could handle sequential information across video frames and the categories of targets tracked are normally unknown in advance. In this paper, we propose a Unified and Effective Network, named UEN, to attack visual object tracking models.
There are several appealing characteristics of UEN:
(1) UEN could produce various invisible adversarial perturbations according to different attack settings by using only one simple end-to-end network with three ingenious loss function;
(2) UEN could generate general visible adversarial patch patterns to attack the advanced trackers in the real-world;
(3) Extensive experiments show that UEN is able to attack many state-of-the-art trackers effectively (e.g. SiamRPN-based networks and DiMP) on popular tracking datasets including OTB100, UAV123, and GOT10K, making online real-time attacks possible. The attack results outperform the introduced baseline in terms of attacking ability and attacking efficiency.
Towards Universal Physical Attacks on Single Object Tracking
Authors: Ding Li , Wang Yongwei , Yuan Kaiwen , Jiang Minyang , Wang Ping , Huang Hua , Wang Z. Jane
Published: September 2022

Recent studies show that small perturbations in video frames could misguide single object trackers. However, such attacks have been mainly designed for digital-domain videos (i.e., perturbation on full images), which makes them practically infeasible to evaluate the adversarial vulnerability of trackers in real-world scenarios. Here we made the first step towards physically feasible adversarial attacks against visual tracking in real scenes with a universal patch to camouflage single object trackers. Fundamentally different from physical object detection, the essence of single object tracking lies in the feature matching between the search image and templates, and we therefore specially design the maximum textural discrepancy (MTD), a resolution-invariant and target location-independent feature de-matching loss. The MTD distills global textural information of the template and search images at hierarchical feature scales prior to performing feature attacks. Moreover, we evaluate two shape attacks, the regression dilation and shrinking, to generate stronger and more controllable attacks. Further, we employ a set of transformations to simulate diverse visual tracking scenes in the wild. Experimental results show the effectiveness of the physically feasible attacks on SiamMask and SiamRPN++ visual trackers both in digital and physical scenes.
MIEHDR CNN: Main Image Enhancement based Ghost-Free High Dynamic Range Imaging using Dual-Lens Systems
Authors: Dong Xuan , Hu Xiaoyan , Li Weixin , Wang Xiaojie , Wang Yunhong
Published: September 2022

We study the High Dynamic Range (HDR) imaging problem using two Low Dynamic Range (LDR) images that are shot from dual-lens systems in a single shot time with different exposures. In most of the related HDR imaging methods, the problem is usually solved by Multiple Images Merging, i.e. the final HDR image is fused from pixels of all the input LDR images. However, ghost artifacts can be hardly avoided using this strategy. Instead of directly merging the multiple LDR inputs, we use an indirect way which enhances the main image, i.e. the short exposure image IS, using the long exposure image IL serving as guidance. In detail, we propose a new model, named MIEHDR CNN model, which consists of three subnets, i.e. Soft Warp CNN, 3D Guided Denoising CNN and Fusion CNN. The Soft Warp CNN aligns IL to get the aligned result ILA using the soft exposed result of IS as reference. The 3D Guided Denoising CNN denoises the soft exposed result of IS using ILA as guidance, whose result are fed into the Fusion CNN with IS to get the HDR result. The MIEHDR CNN model is implemented by MindSpore and experimental results show that we can outperform related methods largely and avoid ghost artifacts.
SMART Frame Selection for Action Recognition
Authors: Gowda Shreyank N , Rohrbach Marcus , Sevilla-Lara Laura
Published: September 2022

Video classification is computationally expensive. In this paper, we address theproblem of frame selection to reduce the computational cost of video classification.Recent work has successfully leveraged frame selection for long, untrimmed videos,where much of the content is not relevant, and easy to discard. In this work, however,we focus on the more standard short, trimmed video classification problem. Weargue that good frame selection can not only reduce the computational cost of videoclassification but also increase the accuracy by getting rid of frames that are hard toclassify. In contrast to previous work, we propose a method that instead of selectingframes by considering one at a time, considers them jointly. This results in a moreefficient selection, where “good" frames are more effectively distributed over thevideo, like snapshots that tell a story. We call the proposed frame selection SMARTand we test it in combination with different backbone architectures and on multiplebenchmarks (Kinetics [5], Something-something [14], UCF101 [31]). We showthat the SMART frame selection consistently improves the accuracy compared toother frame selection strategies while reducing the computational cost by a factorof 4 to 10 times. Additionally, we show that when the primary goal is recognitionperformance, our selection strategy can improve over recent state-of-the-art modelsand frame selection strategies on various benchmarks (UCF101, HMDB51 [21],FCVID [17], and ActivityNet [4]).
Order Regularization on Ordinal Loss for Head Pose, Age and Gaze Estimation
Authors: Guo Tianchu , Zhang Hui , Yoo ByungIn , Liu Yongchao , Kwak Youngjun , Han Jae-Joon
Published: September 2022

Ordinal loss is widely used in solving regression problems with deep learning technologies. Its basic idea is to convert regression to classification while preserving the natural order. However, the order constraint is enforced only by ordinal label implicitly, leading to the real output values not strictly in order. It causes the network to learn separable feature rather than discriminative feature, and possibly overfit on training set. In this paper, we propose order regularization on ordinal loss, which makes the outputs in order by explicitly constraining the ordinal classifiers in order. The proposed method contains two parts, i.e. similar-weights constraint, which reduces the ineffective space between classifiers, and differential-bias constraint, which enforces the decision planes in order and enhances the discrimination power of the classifiers. Experimental results show that our proposed method boosts the performance of original ordinal loss on various regression problems such as head pose, age, and gaze estimation, with significant error reduction of around 5%. Furthermore, our method outperforms the state of the art on all these tasks, with the performance gain of 14.4%, 2.2% and 6.5% on head pose, age and gaze estimation respectively.
SSN3D: Self-Separated Network to Align Parts for 3D Convolution in Video Person Re-Identification
Authors: Jiang Xiaoke , Qiao Yu , Yan Junjie , Li Qichen , Zheng Wanrong , Chen Dapeng
Published: September 2022

Temporal appearance misalignment is a crucial problem in video person re-identification. The same part of person (e.g. head or hand) appearing on different locations in video sequence weakens its discriminative ability, especially when we apply standard temporal aggregation such as 3D convolution or LSTM.
To address this issue, we propose Self-Separated network (SSN) to seek out the same parts in different images.
As the name implies, SSN, if trained in an unsupervised strategy, guarantees the selected parts distinct.
With a few samples of labeled parts to guide SSN training, this semi-supervised trained SSN seeks out the parts that are human-understandable within a frame and stable across a video snippet. Given the distinct and stable person parts, rather than performing aggregation on features, we then apply 3D convolution across different frames for person re-identification.
This SSN + 3D pipeline, dubbed SSN3D, is proved to be efficient through extensive experiments on both synthetic and real data.
DASZL: Dynamic Action Signatures for Zero-shot Learning
Authors: Kim Tae Soo , Jones Jonathan , Peven Michael , Xiao Zihao , Bai Jin , Zhang Yi , Qiu Weichao , Yuille Alan , Hager Gregory D.
Published: September 2022

There are many realistic applications of activity recognition where the set of potential activity descriptions is combinatorially large. This makes end-to-end supervised training of a recognition system impractical as no training set is practically able to encompass the entire label set. In this paper, we present an approach to fine-grained recognition that models activities as compositions of dynamic action signatures. This compositional approach allows us to reframe fine-grained recognition as zero-shot activity recognition, where a detector is composed "on the fly" from simple first-principles state machines supported by deep-learned components. We evaluate our method on the Olympic Sports and UCF101 datasets, where our model establishes a new state of the art under multiple experimental paradigms. We also extend this method to form a unique framework for zero-shot joint segmentation and classification of activities in video and demonstrate the first results in zero- shot decoding of complex action sequences on a widely-used surgical dataset. Lastly, we show that we can use off-the-shelf object detectors to recognize activities in completely de-novo settings with no additional training.
Spatiotemporal Graph Neural Network based Mask Reconstruction for Video Object Segmentation
Authors: Liu Daizong , Xu Shuangjie , Liu Xiao-Yang , Xu Zichuan , Wei Wei , Zhou Pan
Published: September 2022

This paper addresses the task of segmenting class-agnostic objects in semi-supervised setting. Although previous detection based methods achieve relatively good performance, these approaches extract the best proposal by a greedy strategy, which may lose the local patch details outside the chosen candidate. In this paper, we propose a novel spatiotemporal graph neural network (STG-Net) to reconstruct more accurate masks for video object segmentation, which captures the local contexts by utilizing all proposals. In the spatial graph, we treat object proposals of a frame as nodes and represent their correlations with an edge weight strategy for mask context aggregation. To capture temporal information from previous frames, we use a memory network to refine the mask of current frame by retrieving historic masks in a temporal graph. The joint use of both local patch details and temporal relationships allow us to better address the challenges such as object occlusions and missing. Without online learning and fine-tuning, our STG-Net achieves state-of-the-art performance on four large benchmarks, demonstrating the effectiveness of the proposed approach.
Weakly Supervised Temporal Action Localization Through Learning Explicit Subspaces for Action and Context
Authors: Liu Ziyi , Wang Le , Tang Wei , Yuan Junsong , Zheng Nanning , Hua Gang
Published: September 2022

Weakly-supervised Temporal Action Localization (WS-TAL) methods learn to localize temporal starts and ends of action instances in a video under only video-level supervision. Existing WS-TAL methods rely on deep features learned for action recognition. However, due to the mismatch between classification and localization, these features cannot distinguish the frequently co-occurring contextual background, i.e., the context, and the actual action instances. We term this challenge action-context confusion, and it will adversely affect the action localization accuracy. To address this challenge, we introduce a framework that learns two feature subspaces respectively for actions and their context. By explicitly accounting for action visual elements, the action instances can be localized more precisely without the distraction from the context. To facilitate the learning of these two feature subspaces with only video-level categorical labels, we leverage the predictions from both spatial and temporal streams for snippets grouping. In addition, an unsupervised learning task is introduced to make the proposed module focus on mining temporal information. The proposed approach outperforms state-of-the-art WS-TAL methods on three benchmarks, i.e., THUMOS14, ActivityNet v1.2 and v1.3 datasets.
Dual-level Collaborative Transformer for Image Captioning
Authors: Luo Yunpeng , Ji Jiayi , Sun Xiaoshuai , Cao Liujuan , Wu Yongjian , Huang Feiyue , Lin Chia-Wen , Ji Rongrong
Published: September 2022

Descriptive region features extracted by object detection networks have played an important role in the recent advancements of image captioning. However, they are still criticized for the lack of contextual information and fine-grained details, which in contrast are the merits of traditional grid features. In this paper, we introduce a novel Dual-Level Collaborative Transformer (DLCT) network to realize the complementary advantages of the two features.
Concretely, in DLCT, these two features are first processed by a novel Dual-way Self Attenion (DWSA) to mine their intrinsic properties, where a Comprehensive Relation Attention component is also introduced to embed the geometric information. In addition, we propose a Locality-Constrained Cross Attention module to address the semantic noises caused by the direct fusion of these two features, where a geometric alignment graph is constructed to accurately align and reinforce region and grid features. To validate our model, we conduct extensive experiments on the highly competitive MS-COCO dataset, and achieve new state-of-the-art performance on both local and online test sets, i.e., 133.8% CIDEr on Karpathy split and 135.4% CIDEr on the official split.
Scene Graph Embeddings Using Relative Similarity Supervision
Authors: Maheshwari Paridhi , Chaudhry Ritwick , Vinay Vishwa
Published: September 2022

Scene graphs are a powerful structured representation of the underlying content of images, and embeddings derived from them have been shown to be useful in multiple downstream tasks. In this work, we employ a graph convolutional network to exploit structure in scene graphs and produce image embeddings useful for semantic image retrieval. Different from classification-centric supervision traditionally available for learning image representations, we address the task of learning from relative similarity labels in a ranking context. Rooted within the contrastive learning paradigm, we propose a novel loss function that operates on pairs of similar and dissimilar images and imposes relative ordering between them in embedding space. We demonstrate that this Ranking loss, coupled with an intuitive triple sampling strategy, leads to robust representations that outperform well-known contrastive losses on the retrieval task. In addition, we provide qualitative evidence of how retrieved results that utilize structured scene information capture the global context of the scene, different from visual similarity search.
REFINE: Prediction Fusion Network for Panoptic Segmentation
Authors: Ren Jiawei , Yu Cunjun , Cai Zhongang , Zhang Mingyuan , Chen Chongsong , Zhao Haiyu , Yi Shuai , Li Hongsheng
Published: September 2022

Panoptic segmentation aims at generating pixel-wise class and instance predictions for each pixel in the input image, which is a challenging task and far more complicated than naively fusing the semantic and instance segmentation results. Prediction fusion is therefore important to achieve accurate panoptic segmentation. In this paper, we present REFINE, pREdiction FusIon NEtwork for panoptic segmentation, to achieve high-quality panoptic segmentation by improving cross-task prediction fusion, and within-task prediction fusion. Our single-model ResNeXt-101 with DCN achieves PQ=51.5 on the COCO dataset, surpassing state-of-the-art performance by a convincing margin and is comparable with ensembled models. Our smaller model with a ResNet-50 backbone achieves PQ=44.9, which is comparable with state-of-the-art methods with larger backbones.
MangaGAN: Unpaired Photo-to-Manga Translation Based on The Methodology of Manga Drawing
Authors: Su Hao , Niu Jianwei , Liu Xuefeng , Li Qingfeng , Cui Jiahe , Wan Ji
Published: September 2022

Manga is a world popular comic form originated in Japan, which typically employs black-and-white stroke lines and geometric exaggeration to describe humans' appearances, poses, and actions. In this paper, we propose MangaGAN, the first method based on Generative Adversarial Network (GAN) for unpaired photo-to-manga translation. Inspired by the drawing process of experienced manga artists, MangaGAN generates geometric features and converts each facial region into the manga domain with a tailored multi-GANs architecture. For training MangaGAN, we collect a new data-set from a popular manga work with extensive features. To produce high-quality manga faces, we propose a structural smoothing loss to smooth stroke-lines and avoid noisy pixels, and a similarity preserving module to improve the similarity between domains of photo and manga. Extensive experiments show that MangaGAN can produce high-quality manga faces preserving both the facial similarity and manga style, and outperforms other reference methods.
Artificial Dummies for Urban Dataset Augmentation
Authors: Vobecký Antonín , Hurych David , Uřičář Michal , Pérez Patrick , Sivic Josef
Published: September 2022

Existing datasets for training pedestrian detectors in images suffer from limited appearance and pose variation. The most challenging scenarios are rarely included because they are too difficult to capture due to safety reasons, or they are very unlikely to happen. The strict safety requirements in assisted and autonomous driving applications call for an extra high detection accuracy also in these rare situations. Having the ability to generate people images in arbitrary poses, with arbitrary appearances and embedded in different background scenes with varying illumination and weather conditions, is a crucial component for the development and testing of such applications.
The contributions of this paper are three-fold.
First, we describe an augmentation method for the controlled synthesis of urban scenes containing people, thus producing rare or never-seen situations. This is achieved with a data generator (called DummyNet) with disentangled control of the pose, the appearance, and the target background scene.
Second, the proposed generator relies on novel network architecture and associated loss that takes into account the segmentation of the foreground person and its composition into the background scene.
Finally, we demonstrate that the data generated by our DummyNet improve the performance of several existing person detectors across various datasets as well as in challenging situations, such as night-time conditions, where only a limited amount of training data is available.
In the setup with only day-time data available, we improve the night-time detector by 17% log-average miss rate over the detector trained with the day-time data only.
Task-Independent Knowledge Makes for Transferable Representations for Generalized Zero-Shot Learning
Authors: Wang Chaoqun , Chen Xuejin , Min Shaobo , Sun Xiaoyan , Li Houqiang
Published: September 2022

Generalized Zero-Shot Learning (GZSL) targets recognizing new categories by learning transferable image representations. Existing methods find that, by aligning image representations with corresponding semantic labels, the semantic-aligned representations can be transferred to unseen categories. However, supervised by only seen category labels, the learned semantic knowledge is highly task-specific, which makes image representations biased towards seen categories. In this paper, we propose a novel Dual-Contrastive Embedding Network (DCEN) that simultaneously learns task-specific and task-independent knowledge via semantic alignment and instance discrimination. First, DCEN leverages task labels to cluster representations of the same semantic category by cross-modal contrastive learning and exploring semantic-visual complementarity. Besides task-specific knowledge, DCEN then introduces task-independent knowledge by attracting representations of different views of the same image and repelling representations of different images. Compared to high-level seen category supervision, this instance discrimination supervision encourages DCEN to capture low-level visual knowledge, which is less biased toward seen categories and alleviates the representation bias. Consequently, the task-specific and task-independent knowledge jointly make for transferable representations of DCEN, which obtains averaged 4.1% improvement on four public benchmarks.
Weakly Supervised Deep Hyperspherical Quantization for Image Retrieval
Authors: Wang Jinpeng , Chen Bin , Zhang Qiang , Meng Zaiqiao , Liang Shangsong , Xia Shutao
Published: September 2022

Deep quantization methods have shown high efficiency on large-scale image retrieval. However, current models heavily rely on ground-truth information, hindering the application of quantization in label-hungry scenarios. A more realistic demand is to learn from inexhaustible uploaded images that are associated with informal tags provided by amateur users. Though such sketchy tags do not obviously reveal the labels, they actually contain useful semantic information for supervising deep quantization. To this end, we propose Weakly-Supervised Deep Hyperspherical Quantization (WSDHQ), which is the first work to learn deep quantization from weakly tagged images. Specifically, 1) we use word embeddings to represent the tags and enhance their semantic information based on a tag correlation graph. 2) To better preserve semantic information in quantization codes and reduce quantization error, we jointly learn semantics-preserving embeddings and supervised quantizer on hypersphere by employing a well-designed fusion layer and tailor-made loss functions. Extensive experiments show that WSDHQ can achieve state-of-art performance in weakly-supervised compact coding.
Co-mining: Self-Supervised Learning for Sparsely Annotated Object Detection
Authors: Wang Tiancai , Yang Tong , Cao Jiale , Zhang Xiangyu
Published: September 2022

Object detectors usually achieve promising results with the supervision of complete instance annotations. However, their performance is far from satisfactory with sparse instance annotations. Most existing methods for sparsely annotated object detection either re-weight the loss of hard negative samples or convert the unlabeled instances into ignored regions to reduce the interference of false negatives. We argue that these strategies are insufficient since they can at most alleviate the negative effect caused by missing annotations. In this paper, we propose a simple but effective mechanism, called Co-mining, for sparsely annotated object detection. In our Co-mining, two branches of a siamese network predict the pseudo-label sets for each other. To enhance multi-view learning and better mine unlabeled instances, the original image and corresponding augmented image are used as the inputs of two branches of the siamese network, respectively. Co-mining can serve as a general training mechanism applied to most of modern object detectors. Experiments are performed on MS COCO dataset with three different sparsely annotated settings using two typical frameworks: anchor-based detector RetinaNet and anchor-free detector FCOS. Experimental results show that our Co-mining with RetinaNet achieves 1.4%∼2.1% improvements compared with different baselines and surpasses existing methods under the same sparsely annotated setting.
One for More: Selecting Generalizable Samples for Generalizable ReID Model
Authors: Zhang Enwei , Jiang Xinyang , Cheng Hao , Wu Ancong , Yu Fufu , Li Ke , Guo Xiaowei , Zheng Feng , Zheng Weishi , Sun Xing
Published: September 2022

Current training objectives of existing person Re-IDentification (ReID) models only ensure that the loss of the model decreases on selected training batch, with no regards to the performance on samples outside the batch. It will inevitably cause the model to over-fit the data in the dominant position (e.g., head data in imbalanced class, easy samples or noisy samples).
The latest resampling methods address the issue by designing specific criterion to select specific samples that trains the model generalize more on certain type of data (e.g., hard samples, tail data), which is not adaptive to the inconsistent real world ReID data distributions.
Therefore, instead of simply presuming on what samples are generalizable, this paper proposes a one-for-more training objective that directly takes the generalization ability of selected samples as a loss function and learn a sampler to automatically select generalizable samples. More importantly, our proposed one-for-more based sampler can be seamlessly integrated into the ReID training framework which is able to simultaneously train ReID models and the sampler in an end-to-end fashion. The experimental results show that our method can effectively improve the ReID model training and boost the performance of ReID models.
Unsupervised Domain Adaptation for Person Re-identification via Heterogeneous Graph Alignment
Authors: Zhang Minying , Liu Kai , Li Yidong , Guo Shihui , Duan Hongtao , Long Yimin , Jin Yi
Published: September 2022

Unsupervised person re-identification (re-ID) is becoming increasingly popular due to its power in real-world systems such as public security and intelligent transportation systems. However, the person re-ID task is challenged by the problems of data distribution discrepancy across cameras and lack of label information. In this paper, we propose a coarse-to-fine heterogeneous graph alignment (HGA) method to find cross-camera person matches by characterizing the unlabeled data as a heterogeneous graph for each camera. In the coarse-alignment stage, we assign a projection for each camera and utilize an adversarial learning based method to align coarse-grained node groups from different cameras into a shared space, which consequently alleviates the distribution discrepancy between cameras. In the fine-alignment stage, we exploit potential fine-grained node groups in the shared space and introduce conservative alignment loss functions to constrain the graph aligning process, resulting in reliable pseudo labels as learning guidance. The proposed domain adaptation framework not only improves model generalization on target domain, but also facilitates mining and integrating the potential discriminative information across different cameras. Extensive experiments on benchmark datasets demonstrate that the proposed approach outperforms the state-of-the-arts.
Robust Multi-Modality Person Re-identification
Authors: Zheng Aihua , Wang Zi , Chen Zihan , Li Chenglong , Tang Jin
Published: September 2022

To avoid the illumination limitation in visible person re-identification (Re-ID) and the heterogeneous issue in cross-modality Re-ID, we propose to utilize complementary advantages of multiple modalities including visible (RGB), near infrared (NI) and thermal infrared (TI) ones for robust person Re-ID. A novel progressive fusion network is designed to learn effective multi-modal features from single to multiple modalities and from local to global views. Our method works well in diversely challenging scenarios even in the presence of missing modalities. Moreover, we contribute a comprehensive benchmark dataset, RGBNT201, including 201 identities captured from various challenging conditions, to facilitate the research of RGB-NI-TI multi-modality person Re-ID. Comprehensive experiments on RGBNT201 dataset comparing to the state-of-the-art methods demonstrate the contribution of multi-modality person Re-ID and the effectiveness of the proposed approach, which launch a new benchmark and a new baseline for multi-modality person Re-ID.
Regional Attention with Architecture-Rebuilt 3D Network for RGB-D Gesture Recognition
Authors: Zhou Benjia , Li Yunan , Wan Jun
Published: September 2022

Human gesture recognition has drawn much attention in the area of computer vision. However, the performance of gesture recognition is always influenced by some gesture-irrelevant factors like the background and the clothes of performers. Therefore, focusing on the regions of hand/arm is important to the gesture recognition. Meanwhile, a more adaptive architecture-searched network structure can also perform better than the block-fixed ones like ResNet since it increases the diversity of features in different stages of the network better. In this paper, we propose a regional attention with architecture-rebuilt 3D network (RAAR3DNet) for gesture recognition. We replace the fixed Inception modules with the automatically rebuilt structure through the network via Neural Architecture Search (NAS), owing to the different shape and representation ability of features in the early, middle, and late stage of the network. It enables the network to capture different levels of feature representations at different layers more adaptively. Meanwhile, we also design a stackable regional attention module called Dynamic-Static Attention (DSA), which derives a Gaussian guidance heatmap and dynamic motion map to highlight the hand/arm regions and the motion information in the spatial and temporal domains, respectively. Extensive experiments on two recent large-scale RGB-D gesture datasets validate the effectiveness of the proposed method and show it outperforms state-of-the-art methods. The codes of our method are available at: https://github.com/zhoubenjia/RAAR3DNet.