Reviewerly/exHarmony huggingface dataset:
Probing Representation Forgetting in Supervised and Unsupervised Continual Learning

Continual Learning (CL) research typically focuses on tackling the phenomenon of catastrophic forgetting in neural networks. Catastrophic forgetting is associated with an abrupt loss of knowledge previously learned by a model when the task, or more broadly the data distribution, being trained on changes. In supervised learning problems this forgetting, resulting from a change in the model's representation, is typically measured or observed by evaluating the decrease in old task performance. However, a model's representation can change without losing knowledge about prior tasks. In this work we consider the concept of representation forgetting, observed by using the difference in performance of an optimal linear classifier before and after a new task is introduced. Using this tool we revisit a number of standard continual learning benchmarks and observe that, through this lens, model representations trained without any explicit control for forgetting often experience small representation forgetting and can sometimes be comparable to methods which explicitly control for forgetting, especially in longer task sequences. We also show that representation forgetting can lead to new insights on the effect of model capacity and loss function used in continual learning. Based on our results, we show that a simple yet competitive approach is to learn representations continually with standard supervised contrastive learning while constructing prototypes of class samples when queried on old samples. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> The code to reproduce our results is publicly available at: https://github.com/rezazzr/Probing-Representation-Forgetting
Multi-view LiDAR Guided Monocular 3D Object Detection

Detecting 3D objects from monocular RGB images is an ill-posed task for lacking depth knowledge, and monocular-based 3D detection methods perform poorly compared with LiDAR-based 3D detection methods. Some bird’s-eye-view-based monocular 3D detection methods transform front-view image feature maps into bird’s-eye-view feature maps and then use LiDAR 3D detection heads to detect objects. These methods get relatively high performance. However, there is still a large gap between monocular and LiDAR bird’s-eye-view feature maps. Based on the fact that LiDAR bird’s-eye-view feature maps are marginally better than monocular, to bridge their gap and boost monocular 3D detection performance, on the one hand, we directly employ the bird’s-eye-view features from LiDAR models to guide the training of monocular detection models; on the other hand, we employ the front-view features from LiDAR models to help the monocular image backbone embed depth cues into image feature maps. Experimental results show that our method significantly boosts monocular 3D object detection performance without introducing any extra cost in the inference phase.
Few-Shot Class-Incremental Learning from an Open-Set Perspective

The continual appearance of new objects in the visual world poses considerable challenges for current deep learning methods in real-world deployments. The challenge of new task learning is often exacerbated by the scarcity of data for the new categories due to rarity or cost. Here we explore the important task of Few-Shot Class-Incremental Learning (FSCIL) and its extreme data scarcity condition of one-shot. An ideal FSCIL model needs to perform well on all classes, regardless of their presentation order or paucity of data. It also needs to be robust to open-set real-world conditions and be easily adapted to the new tasks that always arise in the field. In this paper, we first reevaluate the current task setting and propose a more comprehensive and practical setting for the FSCIL task. Then, inspired by the similarity of the goals for FSCIL and modern face recognition systems, we propose our method—Augmented Angular Loss Incremental Classification or ALICE. In ALICE, instead of the commonly used cross-entropy loss, we propose to use the angular penalty loss to obtain well-clustered features. As the obtained features not only need to be compactly clustered but also diverse enough to maintain generalization for future incremental classes, we further discuss how class augmentation, data augmentation, and data balancing affect classification performance. Experiments on benchmark datasets, including CIFAR100, miniImageNet, and CUB200, demonstrate the improved performance of ALICE over the state-of-the-art FSCIL methods. Code is available at https://github.com/CanPeng123/FSCIL_ALICE .
Explore Spatio-temporal Aggregation for Insubstantial Object Detection: Benchmark Dataset and Baseline

We endeavor on a rarely explored task named Insubstantial Object Detection (IOD), which aims to localize the object with following characteristics: (1) amorphous shape with indistinct boundary; (2) similarity to surroundings; (3) absence in color. Accordingly, it is far more challenging to distinguish insubstantial objects in a single static frame and the collaborative representation of spatial and temporal information is crucial. Thus, we construct an IOD-Video dataset comprised of 600 videos (141,017 frames) covering various distances, sizes, visibility, and scenes captured by different spectral ranges. In addition, we develop a spatio-temporal aggregation framework for IOD, in which different backbones are deployed and a spatio-temporal aggregation loss (STAloss) is elaborately designed to leverage the consistency along the time axis. Experiments conducted on IOD-Video dataset demonstrate that spatio-temporal aggregation can significantly improve the performance of IOD. We hope our work will attract further researches into this valuable yet challenging task. The code will be available at: https://github.com/CalayZhou/IOD-Video.
How Well Do Sparse ImageNet Models Transfer?

Transfer learning is a classic paradigm by which models pretrained on large “upstream” datasets are adapted to yield good results on “downstream” specialized datasets. Generally, more accurate models on the “upstream” dataset tend to provide better transfer accuracy “downstream”. In this work, we perform an in-depth investigation of this phenomenon in the context of convolutional neural networks (CNNs) trained on the ImageNet dataset, which have been pruned-that is, compressed by sparsifiying their connections. We consider transfer using unstructured pruned models obtained by applying several state-of-the-art pruning methods, including magnitude-based, second-order, regrowth, lottery-ticket, and regularization approaches, in the context of twelve standard transfer tasks. In a nutshell, our study shows that sparse models can match or even outperform the transfer performance of dense models, even at high sparsities, and, while doing so, can lead to significant inference and even training speedups. At the same time, we observe and analyze significant differences in the behaviour of different pruning methods. The code is available at: https://github.com/IST-DASLab/sparse-imagenet-transfer.
Self-Distillation for Robust LiDAR Semantic Segmentation in Autonomous Driving

We propose a new and effective self-distillation framework with our new Test-Time Augmentation (TTA) and Transformer based Voxel Feature Encoder (TransVFE) for robust LiDAR semantic segmentation in autonomous driving, where the robustness is mission-critical but usually neglected. The proposed framework enables the knowledge to be distilled from a teacher model instance to a student model instance, while the two model instances are with the same network architecture for jointly learning and evolving. This requires a strong teacher model to evolve in training. Our TTA strategy effectively reduces the uncertainty in the inference stage of the teacher model. Thus, we propose to equip the teacher model with TTA for providing privileged guidance while the student continuously updates the teacher with better network parameters learned by itself. To further enhance the teacher model, we propose a TransVFE to improve the point cloud encoding by modeling and preserving the local relationship among the points inside each voxel via multi-head attention. The proposed modules are generally designed to be instantiated with different backbones. Evaluations on SemanticKITTI and nuScenes datasets show that our method achieves state-of-the-art performance. Our code is publicly available at https://github.com/jialeli1/lidarseg3d .
Any-Resolution Training for High-Resolution Image Synthesis

Generative models operate at fixed resolution, even though natural images come in a variety of sizes. As high-resolution details are downsampled away and low-resolution images are discarded altogether, precious supervision is lost. We argue that every pixel matters and create datasets with variable-size images, collected at their native resolutions. To take advantage of varied-size data, we introduce continuous-scale training, a process that samples patches at random scales to train a new generator with variable output resolutions. First, conditioning the generator on a target scale allows us to generate higher resolution images than previously possible, without adding layers to the model. Second, by conditioning on continuous coordinates, we can sample patches that still obey a consistent global layout, which also allows for scalable training at higher resolutions. Controlled FFHQ experiments show that our method can take advantage of multi-resolution training data better than discrete multi-scale approaches, achieving better FID scores and cleaner high-frequency details. We also train on other natural image domains including churches, mountains, and birds, and demonstrate arbitrary scale synthesis with both coherent global layouts and realistic local details, going beyond 2K resolution in our experiments. Our project page is available at: https://chail.github.io/anyres-gan/ .
MSTDB: A Hybrid Storage-Empowered Scalable Semantic Blockchain Database

Blockchain has been regarded as a trusted carrier for distributed data storage. With large volumes of valuable data stored on blockchain, data query has become a major requirement. However, the existing blockchains do not provide efficient query functionality because of their deep-rooted chain structure. Blockchain database is a new direction that constructs index on top of blockchain to provide rich query functionalities. The existing works are either insecure because the query process separates from the blockchain consensus, or inscalable because all the data needs to be stored in the block. In this paper, we propose a novel semantic blockchain database called MSTDB. We design a hybrid on/off chain blockchain storage architecture in which the majority of blockchain storage is offloaded to the off-chain storage and a novel index structure named Merkle Semantic Trie (MST) is designed to be a secure and semantic bridge between on- and off-chain. Based on MST, MSTDB provides a variety of semantic query functions including multi-keyword query, range query, Top-K query, and cross-chain query. To improve the performance further, we design some index compression and query preprocessing techniques for MSTDB. Extensive experiments demonstrate the effectiveness and efficiency of our blockchain database.
UAV Swarm Confrontation Based on Multi-agent Deep Reinforcement Learning

Multi-agent deep reinforcement learning (MADRL) has attracted a tremendous amount of interest in recent years. In this paper, we introduce MADRL into the confrontation scene of Unmanned Aerial Vehicle (UAV) swarm. To analysis the dynamic game process of UAV swarm confrontation, we build two non-cooperative game models based on MADRL paradigm. By using the multi-agent deep deterministic policy gradient (MADDPG) and the centralized training with decentralized execution method, we achieve the Nash equilibrium under 5 vs. 5 UAV confrontation scenes. We also introduce multi-agent soft actor critic (MASAC) method into the UAV swarm confrontation, simulation results indicate that the MASAC-based model outperforms the MADDPG-based model on exploring the UAV swarm combat environment, and more effectively converges to the Nash equilibrium. Our work will provide new insights into the confrontation of UAV swarm.
MetaDT: Meta Decision Tree With Class Hierarchy for Interpretable Few-Shot Learning

Few-Shot Learning (FSL) is a challenging task, which aims to recognize novel classes with few examples. Recently, lots of methods have been proposed from the perspective of meta-learning and representation learning. However, few works focus on the interpretability of FSL decision process. In this paper, we take a step towards the interpretable FSL by proposing a novel meta-learning based decision tree framework, namely, MetaDT. In particular, the FSL interpretability is achieved from two aspects, i.e., a concept aspect and a visual aspect. On the concept aspect, we first introduce a tree-like concept hierarchy as FSL prior. Then, resorting to the prior, we split each few-shot task to a set of subtasks with different concept levels and then perform class prediction via a model of decision tree. The advantage of such design is that a sequence of high-level concept decisions that lead up to a final class prediction can be obtained, which clarifies the FSL decision process. On the visual aspect, a set of subtask-specific classifiers with visual attention mechanism is designed to perform decision at each node of the decision tree. As a result, a subtask-specific heatmap visualization can be obtained to achieve the decision interpretability of each tree node. At last, to alleviate the data scarcity issue of FSL, we regard the prior of concept hierarchy as an undirected graph, and then design a graph convolution-based decision tree inference network as our meta-learner to infer parameters of the decision tree. Extensive experiments on performance comparison and interpretability analysis show superiority of our MetaDT.
GANSeg: Learning to Segment by Unsupervised Hierarchical Image Generation

Segmenting an image into its parts is a common pre-process for high-level vision tasks such as image editing. However, annotating masks for supervised training is expensive. Weakly-supervised and unsupervised methods exist, but they depend on the comparison of pairs of images, such as from multi-views, frames of videos, and image augmentation, which limit their applicability. To address this, we propose a GAN-based approach that generates images conditioned on latent masks, thereby alleviating full or weak annotations required by previous approaches. We show that such mask-conditioned image generation can be learned faithfully when conditioning the masks in a hierarchical manner on 2D latent points that define the position of parts explicitly. Without requiring supervision of masks or points, this strategy increases robustness of mask to viewpoint and object position changes. It also lets us generate image-mask pairs for training a segmentation network, which outperforms state-of-the-art unsupervised segmentation methods on established benchmarks. Code can be found at https://github.com/xingzhehe/GANSeg.
Value-Wise ConvNet for Transformer Models: An Infinite Time-Aware Recommender System

Finding the most suitable individual to answer a question using brief content has important usages, including the community of question answering systems and online recommender frameworks. However, one must tackle challenges: Disregarding the indispensable noise in short text contents, authors usually answer the input query with mismatched words that can negatively influence the textual relevance. Moreover, many vocabularies imply various alterations. Finally, not every expert is eager to answer an input query given the time constraint, named the reluctance dilemma. To overcome the challenges, we devise a novel embedding approach that constructs context-aware vectors. We then extract the knowledge domains out of the online contextual content. While we track user textual-temporal behavioral patterns via an infinite continuous-time module, we recommend a set of experts pertinent to the given query and willingly provide the response during the expected time. Experimental results on two real-world datasets of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">StackOverflow</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Yahoo</i> show that our online time-sensitive value-wise transformer can achieve higher effectiveness and efficiency versus other trending rivals in online expert recommendation systems. In addition, we empirically experience that Fourier transformers can automatically infer multi-aspect base signals and overpass manual discrete-time models in obtaining time-specific user profiles.
Teaching Digital Competence in Higher Education. A Comprehensive Scientific Mapping Analysis with Rstudio

Using the Bibliometrix module of the Rstudio software and the PRISMA Publication Guide, it was developed a scientific mapping of the scientific literature from the Scopus database about teaching digital competence (TDC) in universities. The review was delimited through Eric's thesaurus or its approximations and Boolean operators. The following research questions were raised: 1. ¿What is the scientific production per year, per author, affiliation, and country? 2. ¿Which are the main sources of scientific productions, their relevance and impact? 3. ¿Who are the authors with the most scientific productions identified in the search; the most cited; their production over time; their productivity and impact? 4. ¿In which countries does scientific production originate? 5. ¿What collaboration networks between countries, authors and institutions have been developed? and 6. ¿What are the most relevant publications? Spanish authors and institutions stand out in the study related to TDC in universities, expanding networks to Latin America, Europe, Anglo-Saxon countries, and the Asian continent. An adequate scientific mapping facilitates the review processes of scientific literature. The responsible action of researchers when integrating metadata in scientific productions allows a suitable use of exploitation tools and bibliometric analysis. In the future, it is proposed the need to develop studies aimed at the use of other tools that allow the generation of scientific maps in the context of TDC.
Deep-Learning-Based Human Activity Recognition Using Wearable Sensors

Accurate activity recognition has multiple useful health applications, including home-based monitoring in various chronic disease applications. This paper utilizes a deep-learning-based algorithm for recognition of various daily living activities. A nonlinear observer that estimates body segment tilt angles and sensor bias parameters, using inertial sensors provides the inputs to the deep-learning-based algorithm. The observer is designed using Lyapunov analysis and an LMI to obtain the observer gain. Switched gains are utilized for each monotonic region of the nonlinear dynamics to ensure global stability. A convolutional neural network long short-term memory network (CNN-LSTM) is trained as the state-of-the-art deep learning-based activity recognition algorithm. Experimental results are presented on the performance of the CNN-LSTM network with the raw inertial sensor data and compared with results when inputs are the angles estimated by the nonlinear observer instead of the raw data. The activity recognition algorithm that utilizes 18 raw inertial signal data shows an average accuracy of 96.47% while the algorithm that utilizes only the 3 tilt angles estimated by the nonlinear observer provides an average accuracy of 99.81%.
MotionAug: Augmentation with Physical Correction for Human Motion Prediction

This paper presents a motion data augmentation scheme incorporating motion synthesis encouraging diversity and motion correction imposing physical plausibility. This motion synthesis consists of our modified Variational AutoEncoder (VAE) and Inverse Kinematics (IK). In this VAE, our proposed sampling-near-samples method generates various valid motions even with insufficient training motion data. Our IK-based motion synthesis method allows us to generate a variety of motions semi-automatically. Since these two schemes generate unrealistic artifacts in the synthesized motions, our motion correction rectifies them. This motion correction scheme consists of imitation learning with physics simulation and subsequent motion debiasing. For this imitation learning, we propose the PD-residual force that significantly accelerates the training process. Furthermore, our motion debiasing successfully offsets the motion bias induced by imitation learning to maximize the effect of augmentation. As a result, our method outperforms previous noise-based motion augmentation methods by a large margin on both Recurrent Neural Network-based and Graph Convolutional Network-based human motion prediction models. The code is available at https://github.com/meaten/MotionAug.
A C-STEAM Open-Classroom Project Based on SCRUM: the Design and Development of “Cultural Guangzhou” Sand Tray

C-STEAM was born out of the integration of local culture (C) into STEAM education, emphasizing the educational goal of cultural heritage and innovation. The existing C-STEAM education is basically organized in a project-based learning approach and requires students to be fully connected to the local society and culture. To explore diverse ways of practicing C-STEAM, this study designed and developed the "Culture Guangzhou" sand tray project combined with the 6C model of C-STEAM and the SCRUM agile development model with the values of open education. The study concluded from interviews with the experts and the development team member about their perceptions of this project-based learning design that (1) the SCRUM supports the teacher and the students in iterating for incremental progress; (2) it embodies contemporary values in the unity of humanity and technology; (3) the instructional design contributes to an orderly openness.
Learning to Learn across Diverse Data Biases in Deep Face Recognition

Convolutional Neural Networks have achieved remarkable success in face recognition, in part due to the abundant availability of data. However, the data used for training CNNs is often imbalanced. Prior works largely focus on the long-tailed nature of face datasets in data volume per identity, or focus on single bias variation. In this paper, we show that many bias variations such as ethnicity, head pose, occlusion and blur can jointly affect the accuracy significantly. We propose a sample level weighting approach termed Multi-variation Cosine Margin (MvCoM), to simultaneously consider the multiple variation factors, which orthogonally enhances the face recognition losses to incorporate the importance of training samples. Further, we leverage a learning to learn approach, guided by a held-out meta learning set and use an additive modeling to predict the MvCoM. Extensive experiments on challenging face recognition benchmarks demonstrate the advantages of our method in jointly handling imbalances due to multiple variations.