bluuebunny/crossref_metadata_2025 huggingface dataset:
VRU Pose-SSD: Multiperson Pose Estimation For Automated Driving
Authors: Kumar Chandan , Ramesh Jayanth , Chakraborty Bodhisattwa , Raman Renjith , Weinrich Christoph , Mundhada Anurag , Jain Arjun , Flohr Fabian B.
Published: September 2022

We present a fast and efficient approach for joint person detection and pose estimation optimized for automated driving (AD) in urban scenarios. We use a multitask weight sharing architecture to jointly train detection and pose estimation. This modular architecture allows us to accommodate different downstream tasks in the future. By systematic large-scale experiments on the Tsinghua-Daimler Urban Pose Dataset (TDUP), we obtain multiple models with varying accuracy-speed trade-offs. We then quantize and optimize our network for deployment and present a detailed analysis of the efficacy of the algorithm. We introduce a two-stage evaluation strategy, which is more suitable for AD and achieve a significant performance improvement in comparison to state-of-the-art approaches. Our optimized model runs at 52~fps on full HD images and still reaches a competitive performance of 32.25~LAMR. We are confident that our work serves as an enabler to tackle higher-level tasks like VRU intention estimation and gesture recognition, which rely on stable pose estimates and will play a crucial role in future AD systems.
Twitter Event Summarization by Exploiting Semantic Terms and Graph Network
Authors: Li Quanzhi , Zhang Qiong
Published: September 2022

Twitter is a fast communication channel for gathering and spreading breaking news, and it generates a large volume of tweets for most events. Automatically creating a summary for an event is necessary and important. In this study, we explored two extractive approaches for summarizing events on Twitter. The first one exploits the semantic types of event related terms, and ranks the tweets based on the score computed from these semantic terms. The second one utilizes a graph convolutional network built from a tweet relation graph to generate tweet hidden features for tweet salience estimation. And the most salient tweets are selected as the summary of the event. Our experiments show that these two approaches outperform the compared methods.
Using Online Planning and Acting to Recover from Cyberattacks on Software-defined Networks
Authors: Patra Sunandita , Velazquez Alex , Kang Myong , Nau Dana
Published: September 2022

We describe ACR-SDN, a system to monitor, diagnose, and quickly respond to attacks or failures that may occur in software-defined networks (SDNs). An integral part of ACR-SDN is its use of RAE+UPOM, an automated acting and planning engine that uses hierarchical refinement. To advise ACR-SDN on how to recover a target system from faults and attacks, RAE+UPOM uses attack recovery procedures written as hierarchical operational models. Our experimental results show that the use of refinement planning in ACR-SDN is successful in recovering SDNs from attacks with respect to five performance metrics: estimated time for recovery, efficiency, retry ratio, success ratio, and costEffectiveness.
What are GANs?: Introducing Generative Adversarial Networks to Middle School Students
Authors: Ali Safinah , DiPaola Daniella , Breazeal Cynthia
Published: September 2022

Applications of Generative Machine Learning techniques such as Generative Adversarial Networks (GANs) are used to generate new instances of images, music, text, and videos. While GANs have now become commonplace on social media, a part of children’s lives, and have considerable ethical implications, existing K-12 AI education curricula do not include generative AI. We present a new module, “What are GANs?”, that teaches middle school students how GANs work and how they can create media using GANs. We developed an online, team-based game to simulate how GANs work. Students also interacted with up to four web tools that apply GANs to generate media. This module was piloted with 72 middle school students in a series of online workshops. We provide insight into student usage, understanding, and attitudes towards this lesson. Finally, we give suggestions for integrating this lesson into AI education curricula.
Random Forests for Opponent Hand Estimation in Gin Rummy
Authors: Hein Anthony , Jiang May , Thiyageswaran Vydhourie , Guerzhoy Michael
Published: September 2022

We demonstrate an AI agent for the card game of Gin Rummy. The agent uses simple heuristics in conjunction with a model that predicts the probability of each card's being in the opponent's hand. To estimate the probabilities for cards' being in the opponent's hand, we generate a dataset of Gin Rummy games using self-play, and train a random forest on the game information states. We explore the random forest classifier we trained and study the correspondence between its outputs and intuitively correct outputs. Our agent wins 61% of games against a baseline heuristic agent that does not use opponent hand estimation.
Why and What to Teach: AI Curriculum for Elementary School
Authors: Kim Seonghun , Jang Yeonju , Kim Woojin , Choi Seongyune , Jung Heeseok , Kim Soohwan , Kim Hyeoncheol
Published: September 2022

With the rapid technological change of society with Artificial Intelligence, elementary schools' goal should be to prepare the next generations according to competencies. We propose an AI curriculum to cultivate students' AI literacy to answer the question of ‘why and what to teach’ on AI. The proposed AI curriculum focuses on achieving AI literacy based on three competencies: AI Knowledge, AI Skill, and AI Attitude. We anticipate that the proposed curriculum will equip students with core competencies for the future with AI.
Teaching Tech to Talk: K-12 Conversational Artificial Intelligence Literacy Curriculum and Development Tools
Authors: Van Brummelen Jessica , Heng Tommy , Tabunshchyk Viktoriya
Published: September 2022

With children talking to smart-speakers, smart-phones and even smart-microwaves daily, it is increasingly important to educate students on how these agents work—from underlying mechanisms to societal implications. Researchers are developing tools and curriculum to teach K-12 students broadly about artificial intelligence (AI); however, few studies have evaluated these tools with respect to AI-specific learning outcomes, and even fewer have addressed student learning about AI-based conversational agents. We evaluated our Conversational Agent Interface for MIT App Inventor and workshop curriculum with respect to 8 AI competencies from the literature. Furthermore, we analyze teacher (n=9) and student (n=47) feedback from workshops with the interface and recommend that future work (1) leverages design considerations to optimize engagement, (2) collaborates with teachers, and (3) addresses a range of student abilities through pacing and opportunities for extension. We found evidence for student understanding of all 8 competencies, with the most difficult concepts being AI ethics and machine learning. We recommend emphasizing these topics in future curricula.
Creating Interpretable Data-Driven Approaches for Remote Health Monitoring
Authors: Ghods Alireza
Published: September 2022

We are at a turning point to address the unprecedented challenges we are facing in healthcare systems. With the aging of the population and increasing health disparities in rural areas, healthcare needs assistance from technologies to provide quality care for these populations. In collaboration with clinicians, we seek to meet this need by creating data-driven methods that provide interpretable healthcare models from ubiquitous ambient and wearable sensor data. My doctoral research goal is to introduce novel ways to help clinicians understand patients' health status by developing new visualization tools and interpretable models that analyze human health and behavior from sensor data.
Logic Guided Genetic Algorithms (Student Abstract)
Authors: Ashok Dhananjay , Scott Joseph , Wetzel Sebastian J. , Panju Maysum , Ganesh Vijay
Published: September 2022

We present a novel Auxiliary Truth enhanced Genetic Algorithm (GA) that uses logical or mathematical constraints as a means of data augmentation as well as to compute loss (in conjunction with the traditional MSE), with the aim of increasing both data efficiency and accuracy of symbolic regression (SR) algorithms. Our method, logic-guided genetic algorithm (LGGA), takes as input a set of labelled data points and auxiliary truths (AT) (mathematical facts known a priori about the unknown function the regressor aims to learn) and outputs a specially generated and curated dataset that can be used with any SR method. We evaluate LGGA against state-of-the-art SR tools, namely, Eureqa and TuringBot and find that using these SR tools in conjunction with LGGA results in them solving up to 30% more equations, needing only a fraction of the amount of data compared to the same tool without LGGA, i.e., resulting in up to a 61.9% improvement in data efficiency.
Early Prediction of Children’s Task Completion in a Tablet Tutor using Visual Features (Student Abstract)
Authors: Boote Bikram , Agarwal Mansi , Mostow Jack
Published: September 2022

Intelligent tutoring systems could benefit from human teachers’ ability to monitor students’ affective states by watching them and thereby detecting early warning signs of disengagement in time to prevent it. Toward that goal, this paper describes a method that uses input from a tablet tutor’s user-facing camera to predict whether the student will complete the current activity or disengage from it. Training a disengagement predictor is useful not only in itself but also in identifying visual indicators of negative affective states even when they don’t lead to non-completion of the task. Unlike prior work that relied on tutor-specific features, the method relies solely on visual features and so could potentially apply to other tutors. We present a deep learning method to make such predictions based on a Long Short Term Memory (LSTM) model that uses a target replication loss function. We train and test the model on screen capture videos of children in Tanzania using a tablet tutor to learn basic Swahili literacy and numeracy. We achieve balanced-class-size prediction accuracy of 73.3% when 40% of the activity is still left.
EC-GAN: Low-Sample Classification using Semi-Supervised Algorithms and GANs (Student Abstract)
Authors: Haque Ayaan
Published: September 2022

Semi-supervised learning has been gaining attention as it allows for performing image analysis tasks such as classification with limited labeled data. Some popular algorithms using Generative Adversarial Networks (GANs) for semi-supervised classification share a single architecture for classification and discrimination. However, this may require a model to converge to a separate data distribution for each task, which may reduce overall performance. While progress in semi-supervised learning has been made, less addressed are small-scale, fully-supervised tasks where even unlabeled data is unavailable and unattainable. We therefore, propose a novel GAN model namely External Classifier GAN (EC-GAN), that utilizes GANs and semi-supervised algorithms to improve classification in fully-supervised regimes. Our method leverages a GAN to generate artificial data used to supplement supervised classification. More specifically, we attach an external classifier, hence the name EC-GAN, to the GAN’s generator, as opposed to sharing an architecture with the discriminator. Our experiments demonstrate that EC-GAN's performance is comparable to the shared architecture method, far superior to the standard data augmentation and regularization-based approach, and effective on a small, realistic dataset.
An Attention Based Multi-view Model for Sarcasm Cause Detection (Student Abstract)
Authors: Liu Hejing , Li Qiudan , Tang Zaichuan , Bai Jie
Published: September 2022

Sarcasm often relates to people’s implicit discontent with certain products and policies. Existing research mainly focus on sarcasm detection, while the deep causal relationships in the full conversation remained unexplored. This paper formulates a novel research question of sarcasm cause detection, and proposes an attention based model that simultaneously captures different semantic associations as well as the inner causal logics in multi-view manner. Experiments on public Reddit dataset prove the efficacy of the proposed model.
Two-Sided Fairness in Non-Personalised Recommendations (Student Abstract)
Authors: Mondal Aadi Swadipto , Bal Rakesh , Sinha Sayan , Patro Gourab K
Published: September 2022

Recommender systems are one of the most widely used services on several online platforms to suggest potential items to the end-users. These services often use different machine learning techniques for which fairness is a concerning factor, especially when the downstream services have the ability to cause social ramifications. Thus, focusing on the non-personalised (global) recommendations in news media platforms (e.g., top-k trending topics on Twitter, top-k news on a news platform, etc.), we discuss on two specific fairness concerns together (traditionally studied separately)---user fairness and organisational fairness. While user fairness captures the idea of representing the choices of all the individual users in the case of global recommendations, organisational fairness tries to ensure politically/ideologically balanced recommendation sets. This makes user fairness a user-side requirement and organisational fairness a platform-side requirement. For user fairness, we test with methods from social choice theory, i.e., various voting rules known to better represent user choices in their results. Even in our application of voting rules to the recommendation setup, we observe high user satisfaction scores. Now for organisational fairness, we propose a bias metric which measures the aggregate ideological bias of a recommended set of items (articles). Analysing the results obtained from voting rule-based recommendation, we find that while the well-known voting rules are better from the user side, they show high bias values and clearly not suitable for organisational requirements of the platforms. Thus, there is a need to build an encompassing mechanism by cohesively bridging ideas of user fairness and organisational fairness. In this abstract paper, we intend to frame the elementary ideas along with the clear motivation behind the requirement of such a mechanism.
An Entity-Aware Adversarial Domain Adaptation Network for Cross-Domain Named Entity Recognition (Student Abstract)
Authors: Peng Qi , Zheng Changmeng , Cai Yi , Wang Tao , Xie Haoran , Li Qing
Published: September 2022

Existing methods for named entity recognition (NER) are critically relied on the amount of labeled data. However, these methods suffer from performance decline in a new domain which is fully-unlabeled. To handle the situation, we propose an entity-aware adversarial domain adaptation network, which utilizes the labeled data from source domain and then adapts to unlabeled target domain. We first apply adversarial training to reduce the distribution gap between different domains. Furthermore, we introduce an entity-aware attention to guide adversarial to achieve the alignment of entity features. The experimental results show that our model outperforms the state-of-the-art approaches.
Zera-Shot Sentiment Analysis for Code-Mixed Data
Authors: Yadav Siddharth , Chakraborty Tanmoy
Published: September 2022

Code-mixing is the practice of alternating between two or more languages. A major part of sentiment analysis research has been monolingual and they perform poorly on the code-mixed text. We introduce methods that use multilingual and cross-lingual embeddings to transfer knowledge from monolingual text to code-mixed text for code-mixed sentiment analysis. Our methods handle code-mixed text through zero-shot learning and beat state-of-the-art English-Spanish code-mixed sentiment analysis by an absolute 3% F1-score. We are able to achieve 0.58 F1-score (without a parallel corpus) and 0.62 F1-score (with the parallel corpus) on the same benchmark in a zero-shot way as compared to 0.68 F1-score in supervised settings. Our code is publicly available on github.com/sedflix/unsacmt.
Incorporating Bidirection-Interactive Information and Semantic Features for Relational Facts Extraction (Student Abstract)
Authors: Yu Yang , Wang Guohua , Ren Haopeng , Cai Yi
Published: September 2022

The interaction between named entity recognition and relation classification is quite essential for the extraction of relational triplets. However, most of jointly extraction works only consider unidirectional interaction between the two sub-tasks. They even neglect the interactive information totally. In order to tackle these problems, we propose a novel unified
joint extraction model which considers bidirection-interactive information between the two subtasks. Our model consists of two modules. The first module utilizes Bi-LSTM and GCN to capture the sequential and the structure-semantic features of a sentence, The second module utilizes two layers to capture bidirection-interactive information between the two subtasks and generates relational triplets respectively. The experimental results show that our proposed model outperforms the state-of-the-art models on two public datasets.
OPRA: An Open-Source Online Preference Reporting and Aggregation System
Authors: Chen Yiwei , Qian Jingwen , Wang Junming , Xia Lirong , Zahavi Gavriel
Published: September 2022

We introduce the Online Preference Reporting and Aggregation (OPRA) system, an open-source online system that aims at providing support for group decision-making. We illustrate OPRA's distinctive features: UI for reporting rankings with ties, comprehensive analytics of preferences, and group decision-making in combinatorial domains. We also discuss our work in an automatic mentor matching system. We hope that the open-source nature of OPRA will foster development of computerized group decision support systems.
IBM Scenario Planning Advisor: A Neuro-Symbolic ERM Solution
Authors: Feblowitz Mark , Hassanzadeh Oktie , Katz Michael , Sohrabi Shirin , Srinivas Kavitha , Udrea Octavian
Published: September 2022

Scenario Planning is a commonly used Enterprise Risk Management
(ERM) technique to help decision makers with longterm
plans by considering multiple alternative futures. It is
typically a manual, highly labor intensive process involving
dozens of experts and hundreds to thousands of person-hours.
We previously introduced a Scenario Planning Advisor prototype
(Sohrabi et al. 2018a,b) that focuses on generating
scenarios quickly based on expert-developed models. We
present the evolution of that prototype into a full-scale, cloud deployed
ERM solution that: (i) can automatically (through
NLP) create models from authoritative documents such as
books, reports and articles, such that what typically took hundreds
to thousands of person-hours can now be achieved in
minutes to hours; (ii) can gather news and other feeds relevant
to forces in the risk models and group them into storylines
without any other user input; (iii) can generate scenarios
at scale, starting with dozens of forces of interest from models
with thousands of forces in seconds; (iv) provides interactive
visualizations of scenario and force model graphs, including
a full model editor in the browser.
The SPA solution is deployed under a non-commercial
use license at https://spa-service.draco.res.ibm.com
and includes a user guide to help new users get
started. A video demonstration is available at
https://www.youtube.com/watch?v=IaX3d37NUl8.
Democratizing Constraint Satisfaction Problems through Machine Learning
Authors: Kumar Mohit , Kolb Samuel , Gautrais Clement , De Raedt Luc
Published: September 2022

Constraint satisfaction problems (CSPs) are used widely, especially in the field of operations research, to model various real world problems like scheduling or planning. However,modelling a problem as a CSP is not trivial, it is labour intensive and requires both modelling and domain expertise. The emerging field of constraint learning deals with this problem by automatically learning constraints from a given dataset. While there are several interesting approaches for constraint learning, these works are hard to access for a non-expert user. Furthermore, different approaches have different underlying formalism and require different setups before they can be used. This demo paper combines these researches and brings it to non-expert users in the form of an interactive Excel plugin. To do this, we translate different formalism for specifying CSPs into a common language, which allows multiple constraint learners to coexist, making this plugin more powerful than individual constraint learners. Moreover, we integrate learning of CSPs from data with solving them, making it a self sufficient plugin. For the developers of different constraint learners, we provide an API that can be used to integrate their work with this plugin by implementing a handful of functions.
Motion-blurred Video Interpolation and Extrapolation
Authors: Argaw Dawit Mureja , Kim Junsik , Rameau Francois , Kweon In So
Published: September 2022

Abrupt motion of camera or objects in a scene result in a blurry video, and therefore recovering high quality video requires two types of enhancements: visual enhancement and temporal upsampling. A broad range of research attempted to recover clean frames from blurred image sequences or temporally upsample frames by interpolation, yet there are very limited studies handling both problems jointly. In this work, we present a novel framework for deblurring, interpolating and extrapolating sharp frames from a motion-blurred video in an end-to-end manner. We design our framework by first learning the pixel-level motion that caused the blur from the given inputs via optical flow estimation and then predict multiple clean frames by warping the decoded features with the estimated flows. To ensure temporal coherence across predicted frames and address potential temporal ambiguity, we propose a simple, yet effective flow-based rule. The effectiveness and favorability of our approach are highlighted through extensive qualitative and quantitative evaluations on motion-blurred datasets from high speed videos.
Semantic MapNet: Building Allocentric Semantic Maps and Representations from Egocentric Views
Authors: Cartillier Vincent , Ren Zhile , Jain Neha , Lee Stefan , Essa Irfan , Batra Dhruv
Published: September 2022

We study the task of semantic mapping – specifically, an embodied agent (a robot or an egocentric AI assistant) is given a tour of a new environment and asked to build an allocentric top-down semantic map (‘what is where?’) from egocentric observations of an RGB-D camera with known pose (via localization sensors). Importantly, our goal is to build neural episodic memories and spatio-semantic representations of 3D spaces that enable the agent to easily learn subsequent tasks in the same space – navigating to objects seen during the tour (‘Find chair’) or answering questions about the space (‘How many chairs did you see in the house?’). Towards this goal, we present Semantic MapNet (SMNet), which consists of: (1) an Egocentric Visual Encoder that encodes each egocentric RGB-D frame, (2) a Feature Projector that projects egocentric features to appropriate locations on a floor-plan, (3) a Spatial Memory Tensor of size floor-plan length×width×feature-dims that learns to accumulate projected egocentric features, and (4) a Map Decoder that uses the memory tensor to produce semantic top-down maps. SMNet combines the strengths of (known) projective camera geometry and neural representation learning. On the task of semantic mapping in the Matterport3D dataset, SMNet significantly outperforms competitive baselines by 4.01−16.81% (absolute) on mean-IoU and 3.81−19.69% (absolute) on Boundary-F1 metrics. Moreover, we show how to use the spatio-semantic allocentric representations build by SMNet for the task of ObjectNav and Embodied Question Answering. Project page: https://vincentcartillier.github.io/smnet.html.
Understanding Deformable Alignment in Video Super-Resolution
Authors: Chan Kelvin C.K. , Wang Xintao , Yu Ke , Dong Chao , Loy Chen Change
Published: September 2022

Deformable convolution, originally proposed for the adaptation to geometric variations of objects, has recently shown compelling performance in aligning multiple frames and is increasingly adopted for video super-resolution. Despite its remarkable performance, its underlying mechanism for alignment remains unclear. In this study, we carefully investigate the relation between deformable alignment and the classic flow-based alignment. We show that deformable convolution can be decomposed into a combination of spatial warping and convolution. This decomposition reveals the commonality of deformable alignment and flow-based alignment in formulation, but with a key difference in their offset diversity. We further demonstrate through experiments that the increased diversity in deformable alignment yields better-aligned features, and hence significantly improves the quality of video super-resolution output. Based on our observations, we propose an offset-fidelity loss that guides the offset learning with optical flow. Experiments show that our loss successfully avoids the overflow of offsets and alleviates the instability problem of deformable alignment. Aside from the contributions to deformable alignment, our formulation inspires a more flexible approach to introduce offset diversity to flow-based alignment, improving its performance.
Joint Demosaicking and Denoising in the Wild: The Case of Training Under Ground Truth Uncertainty
Authors: Chen Jierun , Wen Song , Chan S.-H. Gary
Published: September 2022

Image demosaicking and denoising are the two key fundamental steps in digital camera pipelines, aiming to reconstruct clean color images from noisy luminance readings. In this paper, we propose and study Wild-JDD, a novel learning framework for joint demosaicking and denoising in the wild. In contrast to previous works which generally assume the ground truth of training data is a perfect reflection of the reality, we consider here the more common imperfect case of ground truth uncertainty in the wild. We first illustrate its manifestation as various kinds of artifacts including zipper effect, color moire and residual noise. Then we formulate a two-stage data degradation process to capture such ground truth uncertainty, where a conjugate prior distribution is imposed upon a base distribution. After that, we derive an evidence lower bound (ELBO) loss to train a neural network that approximates the parameters of the conjugate prior distribution conditioned on the degraded input. Finally, to further enhance the performance for out-of-distribution input, we design a simple but effective fine-tuning strategy by taking the input as a weakly informative prior. Taking into account ground truth uncertainty, Wild-JDD enjoys good interpretability during optimization. Extensive experiments validate that it outperforms state-of-the-art schemes on joint demosaicking and denoising tasks on both synthetic and realistic raw datasets.
Deductive Learning for Weakly-Supervised 3D Human Pose Estimation via Uncalibrated Cameras
Authors: Chen Xipeng , Wei Pengxu , Lin Liang
Published: September 2022

Without prohibitive and laborious 3D annotations, weakly-supervised 3D human pose methods mainly employ the model regularization with geometric projection consistency or geometry estimation from multi-view images. Nevertheless, those approaches explicitly need known parameters of calibrated cameras, exhibiting a limited model generalization in various realistic scenarios. To mitigate this issue, in this paper, we propose a Deductive Weakly-Supervised Learning (DWSL) for 3D human pose machine. Our DWSL firstly learns latent representations on depth and camera pose for 3D pose reconstruction. Since weak supervision usually causes ill-conditioned learning or inferior estimation, our DWSL introduces deductive reasoning to make an inference for the human pose from a view to another and develops a reconstruction loss to demonstrate what the model learns and infers is reliable.
This learning by deduction strategy employs the view-transform demonstration and structural rules derived from depth, geometry and angle constraints, which improves the reliability of the model training with weak supervision. On three 3D human pose benchmarks, we conduct extensive experiments to evaluate our proposed method, which achieves superior performance in comparison with state-of-the-art weak-supervised methods. Particularly, our model shows an appealing potential for learning from 2D data captured in dynamic outdoor scenes, which demonstrates promising robustness and generalization in realistic scenarios. Our code is publicly available at https://github.com/Xipeng-Chen/DWSL-3D-pose.
Multi-Scale Spatial Temporal Graph Convolutional Network for Skeleton-Based Action Recognition
Authors: Chen Zhan , Li Sicheng , Yang Bing , Li Qinghan , Liu Hong
Published: September 2022

Graph convolutional networks have been widely used for skeleton-based action recognition due to their excellent modeling ability of non-Euclidean data. As the graph convolution is a local operation, it can only utilize the short-range joint dependencies and short-term trajectory but fails to directly model the distant joints relations and long-range temporal information that are vital to distinguishing various actions. To solve this problem, we present a multi-scale spatial graph convolution (MS-GC) module and a multi-scale temporal graph convolution (MT-GC) module to enrich the receptive field of the model in spatial and temporal dimensions. Concretely, the MS-GC and MT-GC modules decompose the corresponding local graph convolution into a set of sub-graph convolution, forming a hierarchical residual architecture. Without introducing additional parameters, the features will be processed with a series of sub-graph convolutions, and each node could complete multiple spatial and temporal aggregations with its neighborhoods. The final equivalent receptive field is accordingly enlarged, which is capable of capturing both short- and long-range dependencies in spatial and temporal domains. By coupling these two modules as a basic block, we further propose a multi-scale spatial temporal graph convolutional network (MST-GCN), which stacks multiple blocks to learn effective motion representations for action recognition. The proposed MST-GCN achieves remarkable performance on three challenging benchmark datasets, NTU RGB+D, NTU-120 RGB+D and Kinetics-Skeleton, for skeleton-based action recognition.
SSPC-Net: Semi-supervised Semantic 3D Point Cloud Segmentation Network
Authors: Cheng Mingmei , Hui Le , Xie Jin , Yang Jian
Published: September 2022

Point cloud semantic segmentation is a crucial task in 3D scene understanding. Existing methods mainly focus on employing a large number of annotated labels for supervised semantic segmentation. Nonetheless, manually labeling such large point clouds for the supervised segmentation task is time-consuming. In order to reduce the number of annotated labels, we propose a semi-supervised semantic point cloud segmentation network, named SSPC-Net, where we train the semantic segmentation network by inferring the labels of unlabeled points from the few annotated 3D points. In our method, we first partition the whole point cloud into superpoints and build superpoint graphs to mine the long-range dependencies in point clouds. Based on the constructed superpoint graph, we then develop a dynamic label propagation method to generate the pseudo labels for the unsupervised superpoints. Particularly, we adopt a superpoint dropout strategy to dynamically select the generated pseudo labels. In order to fully exploit the generated pseudo labels of the unsupervised superpoints, we furthermore propose a coupled attention mechanism for superpoint feature embedding. Finally, we employ the cross-entropy loss to train the semantic segmentation network with the labels of the supervised superpoints and the pseudo labels of the unsupervised superpoints. Experiments on various datasets demonstrate that our semisupervised segmentation method can achieve better performance than the current semi-supervised segmentation method with fewer annotated 3D points.
Deep Feature Space Trojan Attack of Neural Networks by Controlled Detoxification
Authors: Cheng Siyuan , Liu Yingqi , Ma Shiqing , Zhang Xiangyu
Published: September 2022

Trojan (backdoor) attack is a form of adversarial attack on deep neural networks where the attacker provides victims with a model trained/retrained on malicious data. The backdoor can be activated when a normal input is stamped with a certain pattern called trigger, causing misclassification. Many existing trojan attacks have their triggers being input space patches/objects (e.g., a polygon with solid color) or simple input transformations such as Instagram filters. These simple triggers are susceptible to recent backdoor detection algorithms. We propose a novel deep feature space trojan attack with five characteristics: effectiveness, stealthiness, controllability, robustness and reliance on deep features. We conduct extensive experiments on 9 image classifiers on various datasets including ImageNet to demonstrate these properties and show that our attack can evade state-of-the-art defense.
Split then Refine: Stacked Attention-guided ResUNets for Blind Single Image Visible Watermark Removal
Authors: Cun Xiaodong , Pun Chi-Man
Published: September 2022

Digital watermark is a commonly used technique to protect the copyright of medias. Simultaneously, to increase the robustness of watermark, attacking technique, such as watermark removal, also gets the attention from the community. Previous watermark removal methods require to gain the watermark location from users or train a multi-task network to recover the background indiscriminately. However, when jointly learning, the network performs better on watermark detection than recovering the texture. Inspired by this observation and to erase the visible watermarks blindly, we propose a novel two-stage framework with a stacked attention-guided ResUNets to simulate the process of detection, removal and refinement. In the first stage, we design a multi-task network called SplitNet. It learns the basis features for three sub-tasks altogether while the task-specific features separately use multiple channel attentions. Then, with the predicted mask and coarser restored image, we design RefineNet to smooth the watermarked region with a mask-guided spatial attention. Besides network structure, the proposed algorithm also combines multiple perceptual losses for better quality both visually and numerically. We extensively evaluate our algorithm over four different datasets under various settings and the experiments show that our approach outperforms other state-of-the-art methods by a large margin.
Voxel R-CNN: Towards High Performance Voxel-based 3D Object Detection
Authors: Deng Jiajun , Shi Shaoshuai , Li Peiwei , Zhou Wengang , Zhang Yanyong , Li Houqiang
Published: September 2022

Recent advances on 3D object detection heavily rely on how the 3D data are represented, i.e., voxel-based or point-based representation. Many existing high performance 3D detectors are point-based because this structure can better retain precise point positions. Nevertheless, point-level features lead to high computation overheads due to unordered storage. In contrast, the voxel-based structure is better suited for feature extraction but often yields lower accuracy because the input data are divided into grids. In this paper, we take a slightly different viewpoint --- we find that precise positioning of raw points is not essential for high performance 3D object detection and that the coarse voxel granularity can also offer sufficient detection accuracy. Bearing this view in mind, we devise a simple but effective voxel-based framework, named Voxel R-CNN. By taking full advantage of voxel features in a two-stage approach, our method achieves comparable detection accuracy with state-of-the-art point-based models, but at a fraction of the computation cost. Voxel R-CNN consists of a 3D backbone network, a 2D bird-eye-view (BEV) Region Proposal Network, and a detect head. A voxel RoI pooling is devised to extract RoI features directly from voxel features for further refinement. Extensive experiments are conducted on the widely used KITTI Dataset and the more recent Waymo Open Dataset. Our results show that compared to existing voxel-based methods, Voxel R-CNN delivers a higher detection accuracy while maintaining a real-time frame processing rate, i.e., at a speed of 25 FPS on an NVIDIA RTX 2080 Ti GPU. The code is available at https://github.com/djiajunustc/Voxel-R-CNN.
Similarity Reasoning and Filtration for Image-Text Matching
Authors: Diao Haiwen , Zhang Ying , Ma Lin , Lu Huchuan
Published: September 2022

Image-text matching plays a critical role in bridging the vision and language, and great progress has been made by exploiting the global alignment between image and sentence, or local alignments between regions and words. However, how to make the most of these alignments to infer more accurate matching scores is still underexplored. In this paper, we propose a novel Similarity Graph Reasoning and Attention Filtration (SGRAF) network for image-text matching. Specifically, the vector-based similarity representations are firstly learned to characterize the local and global alignments in a more comprehensive manner, and then the Similarity Graph Reasoning (SGR) module relying on one graph convolutional neural network is introduced to infer relation-aware similarities with both the local and global alignments. The Similarity Attention Filtration (SAF) module is further developed to integrate these alignments effectively by selectively attending on the significant and representative alignments and meanwhile casting aside the interferences of non-meaningful alignments. We demonstrate the superiority of the proposed method with achieving state-of-the-art performances on the Flickr30K and MSCOCO datasets, and the good interpretability of SGR and SAF with extensive qualitative experiments and analyses.
Few-Shot Class-Incremental Learning via Relation Knowledge Distillation
Authors: Dong Songlin , Hong Xiaopeng , Tao Xiaoyu , Chang Xinyuan , Wei Xing , Gong Yihong
Published: September 2022

In this paper, we focus on the challenging few-shot class incremental learning (FSCIL) problem, which requires to transfer knowledge from old tasks to new ones and solves catastrophic forgetting. We propose the exemplar relation distillation incremental learning framework to balance the tasks of old-knowledge preserving and new-knowledge adaptation. First, we construct an exemplar relation graph to represent the knowledge learned by the original network and update gradually for new tasks learning. Then an exemplar relation loss function for discovering the relation knowledge between
different classes is introduced to learn and transfer the structural information in relation graph. A large number of experiments demonstrate that relation knowledge does exist in the exemplars and our approach outperforms other state-of-the-art class-incremental learning methods on the CIFAR100, miniImageNet, and CUB200 datasets.
How to Save your Annotation Cost for Panoptic Segmentation?
Authors: Du Xuefeng , Jiang ChenHan , Xu Hang , Zhang Gengwei , Li Zhenguo
Published: September 2022

How to properly reduce the annotation cost for panoptic segmentation? How to leverage and optimize the cost-quality trade-off for training data and model? These questions are key challenges towards a label-efficient and scalable panoptic segmentation system due to its expensive instance/semantic pixel-level annotation requirements. By closely examining different kinds of cheaper labels, we introduce a novel multi-objective framework to automatically determine the allocation of different annotations, so as to reach a better segmentation quality with a lower annotation cost. Specifically, we design a Cost-Quality Balanced Network (CQB-Net) to generate the panoptic segmentation map, which distills the crucial relations between various supervisions including panoptic labels, image-level classification labels, bounding boxes, and the semantic coherence information between the foreground and background. Instead of ad-hoc allocation during training, we formulate the optimization of cost-quality trade-off as a Multi-Objective Optimization Problem (MOOP). We model the marginal quality improvement of each annotation and approximate the Pareto-front to enable a label-efficient allocation ratio. Extensive experiments on COCO benchmark show the superiority of our method, e.g. achieving a segmentation quality of 43.4% compared to 43.0% of OCFusion while saving 2.4x annotation cost.
Partially Non-Autoregressive Image Captioning
Authors: Fei Zhengcong
Published: September 2022

Current state-of-the-art image captioning systems usually generated descriptions autoregressively, i.e., every forward step conditions on the given image and previously produced words. The sequential attribution causes a unavoidable decoding latency. Non-autoregressive image captioning, on the other hand, predicts the entire sentence simultaneously and accelerates the inference process significantly. However, it removes the dependence in a caption and commonly suffers from repetition or missing issues. To make a better trade-off between speed and quality, we introduce a partially non-autoregressive model, named PNAIC, which considers a caption as a series of concatenated word groups. The groups are generated parallelly in global while each word in group is predicted from left to right, and thus the captioner can create multiple discontinuous words concurrently at each time step. More importantly, by incorporating curriculum learning-based training tasks of group length prediction and invalid group deletion, our model is capable of generating accurate captions as well as preventing common incoherent errors. Extensive experiments on MS COCO benchmark demonstrate that our proposed method achieves more than 3.5× speedup while maintaining competitive performance.
Edge-competing Pathological Liver Vessel Segmentation with Limited Labels
Authors: Feng Zunlei , Wang Zhonghua , Wang Xinchao , Zhang Xiuming , Cheng Lechao , Lei Jie , Wang Yuexuan , Song Mingli
Published: September 2022

The microvascular invasion (MVI) is a major prognostic factor in hepatocellular carcinoma, which is one of the malignant tumors with the highest mortality rate. The diagnosis of MVI needs discovering the vessels that contain hepatocellular carcinoma cells and counting their number in each vessel, which depends heavily on experiences of the doctor, is largely subjective and time-consuming. However, there is no algorithm as yet tailored for the MVI detection from pathological images. This paper collects the first pathological liver image dataset containing $522$ whole slide images with labels of vessels, MVI, and hepatocellular carcinoma grades. The first and essential step for the automatic diagnosis of MVI is the accurate segmentation of vessels. The unique characteristics of pathological liver images, such as super-large size, multi-scale vessel, and blurred vessel edges, make the accurate vessel segmentation challenging. Based on the collected dataset, we propose an Edge-competing Vessel Segmentation Network (EVS-Net), which contains a segmentation network and two edge segmentation discriminators. The segmentation network, combined with an edge-aware self-supervision mechanism, is devised to conduct vessel segmentation with limited labeled patches. Meanwhile, two discriminators are introduced to distinguish whether the segmented vessel and background contain residual features in an adversarial manner. In the training stage, two discriminators are devised to compete for the predicted position of edges. Exhaustive experiments demonstrate that, with only limited labeled patches, EVS-Net achieves a close performance of fully supervised methods, which provides a convenient tool for the pathological liver vessel segmentation. Code is publicly available at https://github.com/wang97zh/EVS-Net.
Visual Boundary Knowledge Translation for Foreground Segmentation
Authors: Feng Zunlei , Cheng Lechao , Wang Xinchao , Wang Xiang , Liu Ya Jie , Du Xiangtong , Song Mingli
Published: September 2022

When confronted with objects of unknown types in an image, humans can effortlessly and precisely tell their visual boundaries. This recognition mechanism and underlying generalization capability seem to contrast to state-of-the-art image segmentation networks that rely on large-scale category-aware annotated training samples. In this paper, we make an attempt towards building models that explicitly account for visual boundary knowledge, in hope to reduce the training effort on segmenting unseen categories. Specifically, we investigate a new task termed as Boundary Knowledge Translation (BKT). Given a set of fully labeled categories, BKT aims to translate the visual boundary knowledge learned from the labeled categories, to a set of novel categories, each of which is provided only a few labeled samples.
To this end, we propose a Translation Segmentation Network (Trans-Net), which comprises a segmentation network and two boundary discriminators. The segmentation network, combined with a boundary-aware self-supervised mechanism, is devised to conduct foreground segmentation, while the two discriminators work together in an adversarial manner to ensure an accurate segmentation of the novel categories under light supervision. Exhaustive experiments demonstrate that, with only tens of labeled samples as guidance, Trans-Net achieves close results on par with fully supervised methods.
Learning Complex 3D Human Self-Contact
Authors: Fieraru Mihai , Zanfir Mihai , Oneata Elisabeta , Popa Alin-Ionut , Olaru Vlad , Sminchisescu Cristian
Published: September 2022

Monocular estimation of three dimensional human self-contact is fundamental for detailed scene analysis including body language understanding and behaviour modeling.
Existing 3d reconstruction methods do not focus on body regions in self-contact and consequently recover configurations that are either far from each other or self-intersecting, when they should just touch. This leads to perceptually incorrect estimates and limits impact in those very fine-grained analysis domains where detailed 3d models are expected to play an important role. To address such challenges we detect self-contact and design 3d losses to explicitly enforce it. Specifically, we develop a model for Self-Contact Prediction (SCP), that estimates the body surface signature of self-contact, leveraging the localization of self-contact in the image, during both training and inference. We collect two large datasets to support learning and evaluation: (1) HumanSC3D, an accurate 3d motion capture repository containing 1,032 sequences with 5,058 contact events and 1,246,487 ground truth 3d poses synchronized with images collected from multiple views, and (2) FlickrSC3D, a repository of 3,969 images, containing 25,297 surface-to-surface correspondences with annotated image spatial support. We also illustrate how more expressive 3d reconstructions can be recovered under self-contact signature constraints and present monocular detection of face-touch as one of the multiple applications made possible by more accurate self-contact models.