OpenRaiser/Intern-Atlas huggingface dataset:
Exploring Efficient Few-shot Adaptation for Vision Transformers
Venue: arXiv cs.CV 2023
Year: 2023

The task of Few-shot Learning (FSL) aims to do the inference on novel categories containing only few labeled examples, with the help of knowledge learned from base categories containing abundant labeled training samples. While there are numerous works into FSL task, Vision Transformers (ViTs) have rarely been taken as the backbone to FSL with few trials focusing on naive finetuning of whole backbone or classification layer.} Essentially, despite ViTs have been shown to enjoy comparable or even better performance on other vision tasks, it is still very nontrivial to efficiently finetune the ViTs in real-world FSL scenarios. To this end, we propose a novel efficient Transformer Tuning (eTT) method that facilitates finetuning ViTs in the FSL tasks. The key novelties come from the newly presented Attentive Prefix Tuning (APT) and Domain Residual Adapter (DRA) for the task and backbone tuning, individually. Specifically, in APT, the prefix is projected to new key and value pairs that are attached to each self-attention layer to provide the model with task-specific information. Moreover, we design the DRA in the form of learnable offset vectors to handle the potential domain gaps between base and novel data. To ensure the APT would not deviate from the initial task-specific information much, we further propose a novel prototypical regularization, which maximizes the similarity between the projected distribution of prefix and initial prototypes, regularizing the update procedure. Our method receives outstanding performance on the challenging Meta-Dataset. We conduct extensive experiments to show the efficacy of our model.
Mask-then-Fill: A Flexible and Effective Data Augmentation Framework for Event Extraction
Venue: arXiv cs.CL 2023
Year: 2023

We present Mask-then-Fill, a flexible and effective data augmentation framework for event extraction. Our approach allows for more flexible manipulation of text and thus can generate more diverse data while keeping the original event structure unchanged as much as possible. Specifically, it first randomly masks out an adjunct sentence fragment and then infills a variable-length text span with a fine-tuned infilling model. The main advantage lies in that it can replace a fragment of arbitrary length in the text with another fragment of variable length, compared to the existing methods which can only replace a single word or a fixed-length fragment. On trigger and argument extraction tasks, the proposed framework is more effective than baseline methods and it demonstrates particularly strong results in the low-resource setting. Our further analysis shows that it achieves a good balance between diversity and distributional similarity.
Myths and Legends in High-Performance Computing
Venue: arXiv cs.DC 2023
Year: 2023

In this thought-provoking article, we discuss certain myths and legends that are folklore among members of the high-performance computing community. We gathered these myths from conversations at conferences and meetings, product advertisements, papers, and other communications such as tweets, blogs, and news articles within and beyond our community. We believe they represent the zeitgeist of the current era of massive change, driven by the end of many scaling laws such as Dennard scaling and Moore's law. While some laws end, new directions are emerging, such as algorithmic scaling or novel architecture research. Nevertheless, these myths are rarely based on scientific facts, but rather on some evidence or argumentation. In fact, we believe that this is the very reason for the existence of many myths and why they cannot be answered clearly. While it feels like there should be clear answers for each, some may remain endless philosophical debates, such as whether Beethoven was better than Mozart. We would like to see our collection of myths as a discussion of possible new directions for research and industry investment.
FMCW Radar Sensing for Indoor Drones Using Learned Representations
Venue: arXiv cs.RO 2023
Year: 2023

Frequency-modulated continuous-wave (FMCW) radar is a promising sensor technology for indoor drones as it provides range, angular as well as Doppler-velocity information about obstacles in the environment. Recently, deep learning approaches have been proposed for processing FMCW data, outperforming traditional detection techniques on range-Doppler or range-azimuth maps. However, these techniques come at a cost; for each novel task a deep neural network architecture has to be trained on high-dimensional input data, stressing both data bandwidth and processing budget. In this paper, we investigate unsupervised learning techniques that generate low-dimensional representations from FMCW radar data, and evaluate to what extent these representations can be reused for multiple downstream tasks. To this end, we introduce a novel dataset of raw radar ADC data recorded from a radar mounted on a flying drone platform in an indoor environment, together with ground truth detection targets. We show with real radar data that, utilizing our learned representations, we match the performance of conventional radar processing techniques and that our model can be trained on different input modalities such as raw ADC samples of only two consecutively transmitted chirps.
Towards an Intelligent Assistive System Based on Augmented Reality and Serious Games
Venue: arXiv cs.HC 2023
Year: 2023

Age-related cognitive impairment is generally characterized by gradual memory loss and decision-making difficulties. The aim of this study is to investigate multi level support and suggest relevant helping means for the elderly with mild cognitive impairment as well as their caregivers as the primary end-users. This work reports preliminary results on an intelligent assistive system, achieved through the integration of Internet of Things, augmented reality, and adaptive fuzzy decision-making methods. The proposed system operates in different modes, including automated and semi-automated modes. The former helps the user complete their daily life activities by showing augmented reality messages or making automatic changes; while the latter allows manual changes after the real-time assessment of the user's cognitive state based on the augmented reality serious game score. We have also evaluated the accuracy of the serious game score with 37 elderly participants and compared it with users' paper-based cognitive test results. We further noted that there is an acceptable correlation between the paper-based test and users' serious game scores. Moreover, we observed that the system response in the semi-automated mode causes less data loss compared with the automated mode, as the number of active devices decreases.
Adaptive Pattern Extraction Multi-Task Learning for Multi-Step Conversion Estimations
Venue: arXiv cs.LG 2023
Year: 2023

Multi-task learning (MTL) has been successfully used in many real-world applications, which aims to simultaneously solve multiple tasks with a single model. The general idea of multi-task learning is designing kinds of global parameter sharing mechanism and task-specific feature extractor to improve the performance of all tasks. However, challenge still remains in balancing the trade-off of various tasks since model performance is sensitive to the relationships between them. Less correlated or even conflict tasks will deteriorate the performance by introducing unhelpful or negative information. Therefore, it is important to efficiently exploit and learn fine-grained feature representation corresponding to each task. In this paper, we propose an Adaptive Pattern Extraction Multi-task (APEM) framework, which is adaptive and flexible for large-scale industrial application. APEM is able to fully utilize the feature information by learning the interactions between the input feature fields and extracted corresponding tasks-specific information. We first introduce a DeepAuto Group Transformer module to automatically and efficiently enhance the feature expressivity with a modified set attention mechanism and a Squeeze-and-Excitation operation. Second, explicit Pattern Selector is introduced to further enable selectively feature representation learning by adaptive task-indicator vectors. Empirical evaluations show that APEM outperforms the state-of-the-art MTL methods on public and real-world financial services datasets. More importantly, we explore the online performance of APEM in a real industrial-level recommendation scenario.
Stealthy Backdoor Attack for Code Models
Venue: arXiv cs.CR 2023
Year: 2023

Code models, such as CodeBERT and CodeT5, offer general-purpose representations of code and play a vital role in supporting downstream automated software engineering tasks. Most recently, code models were revealed to be vulnerable to backdoor attacks. A code model that is backdoor-attacked can behave normally on clean examples but will produce pre-defined malicious outputs on examples injected with triggers that activate the backdoors. Existing backdoor attacks on code models use unstealthy and easy-to-detect triggers. This paper aims to investigate the vulnerability of code models with stealthy backdoor attacks. To this end, we propose AFRAIDOOR (Adversarial Feature as Adaptive Backdoor). AFRAIDOOR achieves stealthiness by leveraging adversarial perturbations to inject adaptive triggers into different inputs. We evaluate AFRAIDOOR on three widely adopted code models (CodeBERT, PLBART and CodeT5) and two downstream tasks (code summarization and method name prediction). We find that around 85% of adaptive triggers in AFRAIDOOR bypass the detection in the defense process. By contrast, only less than 12% of the triggers from previous work bypass the defense. When the defense method is not applied, both AFRAIDOOR and baselines have almost perfect attack success rates. However, once a defense is applied, the success rates of baselines decrease dramatically to 10.47% and 12.06%, while the success rate of AFRAIDOOR are 77.05% and 92.98% on the two tasks. Our finding exposes security weaknesses in code models under stealthy backdoor attacks and shows that the state-of-the-art defense method cannot provide sufficient protection. We call for more research efforts in understanding security threats to code models and developing more effective countermeasures.
Zero-Determinant Strategy in Stochastic Stackelberg Asymmetric Security Game
Venue: arXiv cs.GT 2023
Year: 2023

In a stochastic Stackelberg asymmetric security game, the strong Stackelberg equilibrium (SSE) strategy is a popular option for the defender to get the highest utility against an attacker with the best response (BR) strategy. However, the attacker may be a boundedly rational player, who adopts a combination of the BR strategy and a fixed stubborn one. In such a condition, the SSE strategy may not maintain the defensive performance due to the stubborn element. In this paper, we focus on how the defender can adopt the unilateral-control zero-determinate (ZD) strategy to confront the boundedly rational attacker. At first, we verify the existence of ZD strategies for the defender. We then investigate the performance of the defender's ZD strategy against a boundedly rational attacker, with a comparison of the SSE strategy. Specifically, when the attacker's strategy is close to the BR strategy, the ZD strategy admits a bounded loss for the defender compared with the SSE strategy. Conversely, when the attacker's strategy is close to the stubborn strategy, the ZD strategy can bring higher defensive performance for the defender than the SSE strategy does.
Finding and Counting Patterns in Sparse Graphs
Venue: arXiv cs.DS 2023
Year: 2023

We consider algorithms for finding and counting small, fixed graphs in sparse host graphs. In the non-sparse setting, the parameters treedepth and treewidth play a crucial role in fast, constant-space and polynomial-space algorithms respectively. We discover two new parameters that we call matched treedepth and matched treewidth. We show that finding and counting patterns with low matched treedepth and low matched treewidth can be done asymptotically faster than the existing algorithms when the host graphs are sparse for many patterns. As an application to finding and counting fixed-size patterns, we discover $\otilde(m^3)$-time \footnote{$\otilde$ hides factors that are logarithmic in the input size.}, constant-space algorithms for cycles of length at most $11$ and $\otilde(m^2)$-time, polynomial-space algorithms for paths of length at most $10$.
Triple-stream Deep Metric Learning of Great Ape Behavioural Actions
Venue: arXiv cs.CV 2023
Year: 2023

We propose the first metric learning system for the recognition of great ape behavioural actions. Our proposed triple stream embedding architecture works on camera trap videos taken directly in the wild and demonstrates that the utilisation of an explicit DensePose-C chimpanzee body part segmentation stream effectively complements traditional RGB appearance and optical flow streams. We evaluate system variants with different feature fusion techniques and long-tail recognition approaches. Results and ablations show performance improvements of ~12% in top-1 accuracy over previous results achieved on the PanAf-500 dataset containing 180,000 manually annotated frames across nine behavioural actions. Furthermore, we provide a qualitative analysis of our findings and augment the metric learning system with long-tail recognition techniques showing that average per class accuracy -- critical in the domain -- can be improved by ~23% compared to the literature on that dataset. Finally, since our embedding spaces are constructed as metric, we provide first data-driven visualisations of the great ape behavioural action spaces revealing emerging geometry and topology. We hope that the work sparks further interest in this vital application area of computer vision for the benefit of endangered great apes.
Does compressing activations help model parallel training?
Venue: arXiv cs.LG 2023
Year: 2023

Large-scale Transformer models are known for their exceptional performance in a range of tasks, but training them can be difficult due to the requirement for communication-intensive model parallelism. One way to improve training speed is to compress the message size in communication. Previous approaches have primarily focused on compressing gradients in a data parallelism setting, but compression in a model-parallel setting is an understudied area. We have discovered that model parallelism has fundamentally different characteristics than data parallelism. In this work, we present the first empirical study on the effectiveness of compression methods for model parallelism. We implement and evaluate three common classes of compression algorithms - pruning-based, learning-based, and quantization-based - using a popular Transformer training framework. We evaluate these methods across more than 160 settings and 8 popular datasets, taking into account different hyperparameters, hardware, and both fine-tuning and pre-training stages. We also provide analysis when the model is scaled up. Finally, we provide insights for future development of model parallelism compression algorithms.
RIC-O: Efficient placement of a disaggregated and distributed RAN Intelligent Controller with dynamic clustering of radio nodes
Venue: arXiv cs.NI 2023
Year: 2023

The Radio Access Network (RAN) is the segment of cellular networks that provides wireless connectivity to end-users. O-RAN Alliance has been transforming the RAN industry by proposing open RAN specifications and the programmable Non-Real-Time and Near-Real-Time RAN Intelligent Controllers (Non-RT RIC and Near-RT RIC). Both RICs provide platforms for running applications called rApps and xApps, respectively, to optimize the behavior of the RAN. We investigate a disaggregation strategy of the Near-RT RIC so that its components meet stringent latency requirements while presenting a cost-effective solution. We propose the novel RIC Orchestrator (RIC-O) that optimizes the deployment of the Near-RT RIC components across the cloud-edge continuum. Edge computing nodes often present limited resources and are expensive compared to cloud computing. For example, in the O-RAN Signalling Storm Protection, Near-RT RIC is expected to support end-to-end control loop latencies as low as 10ms. Therefore, performance-critical components of Near-RT RIC and certain xApps should run at the edge while other components can run on the cloud. Furthermore, RIC-O employs an efficient strategy to react to sudden changes and re-deploy components dynamically. We evaluate our proposal through analytical modeling and real-world experiments in an extended Kubernetes deployment implementing RIC-O and disaggregated Near-RT RIC.
Active Learning Guided by Efficient Surrogate Learners
Venue: arXiv cs.LG 2023
Year: 2023

Re-training a deep learning model each time a single data point receives a new label is impractical due to the inherent complexity of the training process. Consequently, existing active learning (AL) algorithms tend to adopt a batch-based approach where, during each AL iteration, a set of data points is collectively chosen for annotation. However, this strategy frequently leads to redundant sampling, ultimately eroding the efficacy of the labeling procedure. In this paper, we introduce a new AL algorithm that harnesses the power of a Gaussian process surrogate in conjunction with the neural network principal learner. Our proposed model adeptly updates the surrogate learner for every new data instance, enabling it to emulate and capitalize on the continuous learning dynamics of the neural network without necessitating a complete retraining of the principal model for each individual label. Experiments on four benchmark datasets demonstrate that this approach yields significant enhancements, either rivaling or aligning with the performance of state-of-the-art techniques.
On vectorial functions with maximal number of bent components
Venue: arXiv cs.IT 2023
Year: 2023

We study vectorial functions with maximal number of bent components in this paper. We first study the Walsh transform and nonlinearity of $F(x)=x^{2^e}h(\Tr_{2^{2m}/2^m}(x))$, where $e\geq0$ and $h(x)$ is a permutation over $\F_{2^m}$. If $h(x)$ is monomial, the nonlinearity of $F(x)$ is shown to be at most $ 2^{2m-1}-2^{\lfloor\frac{3m}{2}\rfloor}$ and some non-plateaued and plateaued functions attaining the upper bound are found. This gives a partial answer to the open problems proposed by Pott et al. and Anbar et al. If $h(x)$ is linear, the exact nonlinearity of $F(x)$ is determined. Secondly, we give a construction of vectorial functions with maximal number of bent components from known ones, thus obtain two new classes from the Niho class and the Maiorana-McFarland class. Our construction gives a partial answer to an open problem proposed by Pott et al., and also contains vectorial functions outside the complete Maiorana-McFarland class. Finally, we show that the vectorial function $F: \F_{2^{2m}}\rightarrow \F_{2^{2m}}$, $x\mapsto x^{2^m+1}+x^{2^i+1}$ has maximal number of bent components if and only if $i=0$.
Deep Learning-Based UAV Aerial Triangulation without Image Control Points
Venue: arXiv cs.CV 2023
Year: 2023

The emerging drone aerial survey has the advantages of low cost, high efficiency, and flexible use. However, UAVs are often equipped with cheap POS systems and non-measurement cameras, and their flight attitudes are easily affected. How to realize the large-scale mapping of UAV image-free control supported by POS faces many technical problems. The most basic and important core technology is how to accurately realize the absolute orientation of images through advanced aerial triangulation technology. In traditional aerial triangulation, image matching algorithms are constrained to varying degrees by preset prior knowledge. In recent years, deep learning has developed rapidly in the field of photogrammetric computer vision. It has surpassed the performance of traditional handcrafted features in many aspects. It has shown stronger stability in image-based navigation and positioning tasks, especially it has better resistance to unfavorable factors such as blur, illumination changes, and geometric distortion. Based on the introduction of the key technologies of aerial triangulation without image control points, this paper proposes a new drone image registration method based on deep learning image features to solve the problem of high mismatch rate in traditional methods. It adopts SuperPoint as the feature detector, uses the superior generalization performance of CNN to extract precise feature points from the UAV image, thereby achieving high-precision aerial triangulation. Experimental results show that under the same pre-processing and post-processing conditions, compared with the traditional method based on the SIFT algorithm, this method achieves suitable precision more efficiently, which can meet the requirements of UAV aerial triangulation without image control points in large-scale surveys.
"It's a Match!" -- A Benchmark of Task Affinity Scores for Joint Learning
Venue: arXiv cs.LG 2023
Year: 2023

While the promises of Multi-Task Learning (MTL) are attractive, characterizing the conditions of its success is still an open problem in Deep Learning. Some tasks may benefit from being learned together while others may be detrimental to one another. From a task perspective, grouping cooperative tasks while separating competing tasks is paramount to reap the benefits of MTL, i.e., reducing training and inference costs. Therefore, estimating task affinity for joint learning is a key endeavor. Recent work suggests that the training conditions themselves have a significant impact on the outcomes of MTL. Yet, the literature is lacking of a benchmark to assess the effectiveness of tasks affinity estimation techniques and their relation with actual MTL performance. In this paper, we take a first step in recovering this gap by (i) defining a set of affinity scores by both revisiting contributions from previous literature as well presenting new ones and (ii) benchmarking them on the Taskonomy dataset. Our empirical campaign reveals how, even in a small-scale scenario, task affinity scoring does not correlate well with actual MTL performance. Yet, some metrics can be more indicative than others.
Reducing Over-smoothing in Graph Neural Networks Using Relational Embeddings
Venue: arXiv cs.LG 2023
Year: 2023

Graph Neural Networks (GNNs) have achieved a lot of success with graph-structured data. However, it is observed that the performance of GNNs does not improve (or even worsen) as the number of layers increases. This effect has known as over-smoothing, which means that the representations of the graph nodes of different classes would become indistinguishable when stacking multiple layers. In this work, we propose a new simple, and efficient method to alleviate the effect of the over-smoothing problem in GNNs by explicitly using relations between node embeddings. Experiments on real-world datasets demonstrate that utilizing node embedding relations makes GNN models such as Graph Attention Network more robust to over-smoothing and achieves better performance with deeper GNNs. Our method can be used in combination with other methods to give the best performance. GNN applications are endless and depend on the user's objective and the type of data that they possess. Solving over-smoothing issues can potentially improve the performance of models on all these tasks.