OpenRaiser/Intern-Atlas huggingface dataset:
The Counterattack of CNNs in Self-Supervised Learning: Larger Kernel Size might be All You Need
Venue: arXiv cs.CV 2023
Year: 2023

Vision Transformers have been rapidly uprising in computer vision thanks to their outstanding scaling trends, and gradually replacing convolutional neural networks (CNNs). Recent works on self-supervised learning (SSL) introduce siamese pre-training tasks, on which Transformer backbones continue to demonstrate ever stronger results than CNNs. People come to believe that Transformers or self-attention modules are inherently more suitable than CNNs in the context of SSL. However, it is noteworthy that most if not all prior arts of SSL with CNNs chose the standard ResNets as their backbones, whose architecture effectiveness is known to already lag behind advanced Vision Transformers. Therefore, it remains unclear whether the self-attention operation is crucial for the recent advances in SSL - or CNNs can deliver the same excellence with more advanced designs, too? Can we close the SSL performance gap between Transformers and CNNs? To answer these intriguing questions, we apply self-supervised pre-training to the recently proposed, stronger lager-kernel CNN architecture and conduct an apple-to-apple comparison with Transformers, in their SSL performance. Our results show that we are able to build pure CNN SSL architectures that perform on par with or better than the best SSL-trained Transformers, by just scaling up convolutional kernel sizes besides other small tweaks. Impressively, when transferring to the downstream tasks \texttt{MS COCO} detection and segmentation, our SSL pre-trained CNN model (trained in 100 epochs) achieves the same good performance as the 300-epoch pre-trained Transformer counterpart. We hope this work can help to better understand what is essential (or not) for self-supervised learning backbones.
GPT-4 and Safety Case Generation: An Exploratory Analysis
Venue: arXiv cs.SE 2023
Year: 2023

In the ever-evolving landscape of software engineering, the emergence of large language models (LLMs) and conversational interfaces, exemplified by ChatGPT, is nothing short of revolutionary. While their potential is undeniable across various domains, this paper sets out on a captivating expedition to investigate their uncharted territory, the exploration of generating safety cases. In this paper, our primary objective is to delve into the existing knowledge base of GPT-4, focusing specifically on its understanding of the Goal Structuring Notation (GSN), a well-established notation allowing to visually represent safety cases. Subsequently, we perform four distinct experiments with GPT-4. These experiments are designed to assess its capacity for generating safety cases within a defined system and application domain. To measure the performance of GPT-4 in this context, we compare the results it generates with ground-truth safety cases created for an X-ray system system and a Machine-Learning (ML)-enabled component for tire noise recognition (TNR) in a vehicle. This allowed us to gain valuable insights into the model's generative capabilities. Our findings indicate that GPT-4 demonstrates the capacity to produce safety arguments that are moderately accurate and reasonable. Furthermore, it exhibits the capability to generate safety cases that closely align with the semantic content of the reference safety cases used as ground-truths in our experiments.
DVANet: Disentangling View and Action Features for Multi-View Action Recognition
Venue: arXiv cs.CV 2023
Year: 2023

In this work, we present a novel approach to multi-view action recognition where we guide learned action representations to be separated from view-relevant information in a video. When trying to classify action instances captured from multiple viewpoints, there is a higher degree of difficulty due to the difference in background, occlusion, and visibility of the captured action from different camera angles. To tackle the various problems introduced in multi-view action recognition, we propose a novel configuration of learnable transformer decoder queries, in conjunction with two supervised contrastive losses, to enforce the learning of action features that are robust to shifts in viewpoints. Our disentangled feature learning occurs in two stages: the transformer decoder uses separate queries to separately learn action and view information, which are then further disentangled using our two contrastive losses. We show that our model and method of training significantly outperforms all other uni-modal models on four multi-view action recognition datasets: NTU RGB+D, NTU RGB+D 120, PKU-MMD, and N-UCLA. Compared to previous RGB works, we see maximal improvements of 1.5\%, 4.8\%, 2.2\%, and 4.8\% on each dataset, respectively.
DevBots can co-design APIs
Venue: arXiv cs.SE 2023
Year: 2023

DevBots are automated tools that perform various tasks in order to support software development. They are a growing trend and have been used in repositories to automate repetitive tasks, as code generators, and as collaborators in eliciting requirements and defining architectures. In this study, we analyzed 24 articles to investigate the state of the art of using DevBots in software development, trying to understand their characteristics, identify use cases, learn the relationship between DevBots and conversational software development, and discuss how prompt engineering can enable collaboration between human developers and bots. Additionally, we identified a gap to address by applying prompt engineering to collaborative API design between human designers and DevBots and proposed an experiment to assess what approach, between using Retrieval Augmented Generation or not, is more suitable. Our conclusion is that DevBots can collaborate with human API designers, but the two approaches have advantages and disadvantages.
Explosive Legged Robotic Hopping: Energy Accumulation and Power Amplification via Pneumatic Augmentation
Venue: arXiv cs.RO 2023
Year: 2023

We present a novel pneumatic augmentation to traditional electric motor-actuated legged robot to increase intermittent power density to perform infrequent explosive hopping behaviors. The pneumatic system is composed of a pneumatic pump, a tank, and a pneumatic actuator. The tank is charged up by the pump during regular hopping motion that is created by the electric motors. At any time after reaching a desired air pressure in the tank, a solenoid valve is utilized to rapidly release the air pressure to the pneumatic actuator (piston) which is used in conjunction with the electric motors to perform explosive hopping, increasing maximum hopping height for one or subsequent cycles. We show that, on a custom-designed one-legged hopping robot, without any additional power source and with this novel pneumatic augmentation system, their associated system identification and optimal control, the robot is able to realize highly explosive hopping with power amplification per cycle by a factor of approximately 5.4 times the power of electric motor actuation alone.
Towards Global, Socio-Economic, and Culturally Aware Recommender Systems
Venue: arXiv cs.IR 2023
Year: 2023

Recommender systems have gained increasing attention to personalise consumer preferences. While these systems have primarily focused on applications such as advertisement recommendations (e.g., Google), personalized suggestions (e.g., Netflix and Spotify), and retail selection (e.g., Amazon), there is potential for these systems to benefit from a more global, socio-economic, and culturally aware approach, particularly as companies seek to expand into diverse markets. This paper aims to investigate the potential of a recommender system that considers cultural identity and socio-economic factors. We review the most recent developments in recommender systems and explore the impact of cultural identity and socio-economic factors on consumer preferences. We then propose an ontology and approach for incorporating these factors into recommender systems. To illustrate the potential of our approach, we present a scenario in consumer subscription plan selection within the entertainment industry. We argue that existing recommender systems have limited ability to precisely understand user preferences due to a lack of awareness of socio-economic factors and cultural identity. They also fail to update recommendations in response to changing socio-economic conditions. We explore various machine learning models and develop a final artificial neural network model (ANN) that addresses this gap. We evaluate the effectiveness of socio-economic and culturally aware recommender systems across four dimensions: Precision, Accuracy, F1, and Recall. We find that a highly tuned ANN model incorporating domain-specific data, select cultural indices and relevant socio-economic factors predicts user preference in subscriptions with an accuracy of 95%, a precision of 94%, a F1 Score of 92\%, and a Recall of 90\%.
Composite Survival Analysis: Learning with Auxiliary Aggregated Baselines and Survival Scores
Venue: arXiv cs.LG 2023
Year: 2023

Survival Analysis (SA) constitutes the default method for time-to-event modeling due to its ability to estimate event probabilities of sparsely occurring events over time. In this work, we show how to improve the training and inference of SA models by decoupling their full expression into (1) an aggregated baseline hazard, which captures the overall behavior of a given population, and (2) independently distributed survival scores, which model idiosyncratic probabilistic dynamics of its given members, in a fully parametric setting. The proposed inference method is shown to dynamically handle right-censored observation horizons, and to achieve competitive performance when compared to other state-of-the-art methods in a variety of real-world datasets, including computationally inefficient Deep Learning-based SA methods and models that require MCMC for inference. Nevertheless, our method achieves robust results from the outset, while not being subjected to fine-tuning or hyperparameter optimization.
TaBIIC: Taxonomy Building through Iterative and Interactive Clustering
Venue: arXiv cs.AI 2023
Year: 2023

Building taxonomies is often a significant part of building an ontology, and many attempts have been made to automate the creation of such taxonomies from relevant data. The idea in such approaches is either that relevant definitions of the intension of concepts can be extracted as patterns in the data (e.g. in formal concept analysis) or that their extension can be built from grouping data objects based on similarity (clustering). In both cases, the process leads to an automatically constructed structure, which can either be too coarse and lacking in definition, or too fined-grained and detailed, therefore requiring to be refined into the desired taxonomy. In this paper, we explore a method that takes inspiration from both approaches in an iterative and interactive process, so that refinement and definition of the concepts in the taxonomy occur at the time of identifying those concepts in the data. We show that this method is applicable on a variety of data sources and leads to taxonomies that can be more directly integrated into ontologies.
Wild Motion Unleashed: Markerless 3D Kinematics and Force Estimation in Cheetahs
Venue: arXiv cs.CV 2023
Year: 2023

The complex dynamics of animal manoeuvrability in the wild is extremely challenging to study. The cheetah ($\textit{Acinonyx jubatus}$) is a perfect example: despite great interest in its unmatched speed and manoeuvrability, obtaining complete whole-body motion data from these animals remains an unsolved problem. This is especially difficult in wild cheetahs, where it is essential that the methods used are remote and do not constrain the animal's motion. In this work, we use data obtained from cheetahs in the wild to present a trajectory optimisation approach for estimating the 3D kinematics and joint torques of subjects remotely. We call this approach kinetic full trajectory estimation (K-FTE). We validate the method on a dataset comprising synchronised video and force plate data. We are able to reconstruct the 3D kinematics with an average reprojection error of 17.69 pixels (62.94 $\%$ PCK using the nose-to-eye(s) length segment as a threshold), while the estimates produce an average root-mean-square error of 171.3 N ($\approx$ 17.16 $\%$ of peak force during stride) for the estimated ground reaction force when compared against the force plate data. While the joint torques cannot be directly validated against ground truth data, as no such data is available for cheetahs, the estimated torques agree with previous studies of quadrupeds in controlled settings. These results will enable deeper insight into the study of animal locomotion in a more natural environment for both biologists and roboticists.
On the rank of the communication matrix for deterministic two-way finite automata
Venue: arXiv cs.FL 2023
Year: 2023

The communication matrix for two-way deterministic finite automata (2DFA) with $n$ states is defined for an automaton over a full alphabet of all $(2n+1)^n$ possible symbols: its rows and columns are indexed by strings, and the entry $(u, v)$ is $1$ if $uv$ is accepted by the automaton, and $0$ otherwise. With duplicate rows and columns removed, this is a square matrix of order $n(n^n-(n-1)^n)+1$, and its rank is known to be a lower bound on the number of states necessary to transform an $n$-state 2DFA to a one-way unambiguous finite automaton (UFA). This paper determines this rank, showing that it is exactly $f(n)=\sum_{k=1}^n \binom{n}{k-1} \binom{n}{k} \binom{2k-2}{k-1} =(1+o(1)) \frac{3\sqrt{3}}{8πn} 9^n$, and this function becomes the new lower bound on the state complexity of the 2DFA to UFA transformation, thus improving a recent lower bound by S. Petrov and Okhotin (``On the transformation of two-way deterministic finite automata to unambiguous finite automata'', Inf. Comput., 2023). The key element of the proof is determining the rank of a $k! \times k!$ submatrix, with its rows and columns indexed by permutations, where the entry $(π, σ)$ is $1$ if $σ\circ π$ is a cycle of length $k$, and 0 otherwise; using the methods of group representation theory it is shown that its rank is exactly $\binom{2k-2}{k-1}$, and this implies the above formula for $f(n)$.
Data-Free Hard-Label Robustness Stealing Attack
Venue: arXiv cs.CV 2023
Year: 2023

The popularity of Machine Learning as a Service (MLaaS) has led to increased concerns about Model Stealing Attacks (MSA), which aim to craft a clone model by querying MLaaS. Currently, most research on MSA assumes that MLaaS can provide soft labels and that the attacker has a proxy dataset with a similar distribution. However, this fails to encapsulate the more practical scenario where only hard labels are returned by MLaaS and the data distribution remains elusive. Furthermore, most existing work focuses solely on stealing the model accuracy, neglecting the model robustness, while robustness is essential in security-sensitive scenarios, e.g., face-scan payment. Notably, improving model robustness often necessitates the use of expensive techniques such as adversarial training, thereby further making stealing robustness a more lucrative prospect. In response to these identified gaps, we introduce a novel Data-Free Hard-Label Robustness Stealing (DFHL-RS) attack in this paper, which enables the stealing of both model accuracy and robustness by simply querying hard labels of the target model without the help of any natural data. Comprehensive experiments demonstrate the effectiveness of our method. The clone model achieves a clean accuracy of 77.86% and a robust accuracy of 39.51% against AutoAttack, which are only 4.71% and 8.40% lower than the target model on the CIFAR-10 dataset, significantly exceeding the baselines. Our code is available at: https://github.com/LetheSec/DFHL-RS-Attack.
Delaunay Triangulations in the Hilbert Metric
Venue: arXiv cs.CG 2023
Year: 2023

The Hilbert metric is a distance function defined for points lying within the interior of a convex body. It arises in the analysis and processing of convex bodies, machine learning, and quantum information theory. In this paper, we show how to adapt the Euclidean Delaunay triangulation to the Hilbert geometry defined by a convex polygon in the plane. We analyze the geometric properties of the Hilbert Delaunay triangulation, which has some notable differences with respect to the Euclidean case, including the fact that the triangulation does not necessarily cover the convex hull of the point set. We also introduce the notion of a Hilbert ball at infinity, which is a Hilbert metric ball centered on the boundary of the convex polygon. We present a simple randomized incremental algorithm that computes the Hilbert Delaunay triangulation for a set of $n$ points in the Hilbert geometry defined by a convex $m$-gon. The algorithm runs in $O(n (\log n + \log^3 m))$ expected time. In addition we introduce the notion of the Hilbert hull of a set of points, which we define to be the region covered by their Hilbert Delaunay triangulation. We present an algorithm for computing the Hilbert hull in time $O(n h \log^2 m)$, where $h$ is the number of points on the hull's boundary.
Exploiting Representation Bias for Data Distillation in Abstractive Text Summarization
Venue: arXiv cs.CL 2023
Year: 2023

Abstractive text summarization is surging with the number of training samples to cater to the needs of the deep learning models. These models tend to exploit the training data representations to attain superior performance by improving the quantitative element of the resultant summary. However, increasing the size of the training set may not always be the ideal solution to maximize the performance, and therefore, a need to revisit the quality of training samples and the learning protocol of deep learning models is a must. In this paper, we aim to discretize the vector space of the abstractive text summarization models to understand the characteristics learned between the input embedding space and the models' encoder space. We show that deep models fail to capture the diversity of the input space. Further, the distribution of data points on the encoder space indicates that an unchecked increase in the training samples does not add value; rather, a tear-down of data samples is highly needed to make the models focus on variability and faithfulness. We employ clustering techniques to learn the diversity of a model's sample space and how data points are mapped from the embedding space to the encoder space and vice versa. Further, we devise a metric to filter out redundant data points to make the model more robust and less data hungry. We benchmark our proposed method using quantitative metrics, such as Rouge, and qualitative metrics, such as BERTScore, FEQA and Pyramid score. We also quantify the reasons that inhibit the models from learning the diversity from the varied input samples.
An Ambiguity Measure for Recognizing the Unknowns in Deep Learning
Venue: arXiv cs.LG 2023
Year: 2023

We study the understanding of deep neural networks from the scope in which they are trained on. While the accuracy of these models is usually impressive on the aggregate level, they still make mistakes, sometimes on cases that appear to be trivial. Moreover, these models are not reliable in realizing what they do not know leading to failures such as adversarial vulnerability and out-of-distribution failures. Here, we propose a measure for quantifying the ambiguity of inputs for any given model with regard to the scope of its training. We define the ambiguity based on the geometric arrangements of the decision boundaries and the convex hull of training set in the feature space learned by the trained model, and demonstrate that a single ambiguity measure may detect a considerable portion of mistakes of a model on in-distribution samples, adversarial inputs, as well as out-of-distribution inputs. Using our ambiguity measure, a model may abstain from classification when it encounters ambiguous inputs leading to a better model accuracy not just on a given testing set, but on the inputs it may encounter at the world at large. In pursuit of this measure, we develop a theoretical framework that can identify the unknowns of the model in relation to its scope. We put this in perspective with the confidence of the model and develop formulations to identify the regions of the domain which are unknown to the model, yet the model is guaranteed to have high confidence.
SECNN: Squeeze-and-Excitation Convolutional Neural Network for Sentence Classification
Venue: arXiv cs.CL 2023
Year: 2023

Sentence classification is one of the basic tasks of natural language processing. Convolution neural network (CNN) has the ability to extract n-grams features through convolutional filters and capture local correlations between consecutive words in parallel, so CNN is a popular neural network architecture to dealing with the task. But restricted by the width of convolutional filters, it is difficult for CNN to capture long term contextual dependencies. Attention is a mechanism that considers global information and pays more attention to keywords in sentences, thus attention mechanism is cooperated with CNN network to improve performance in sentence classification task. In our work, we don't focus on keyword in a sentence, but on which CNN's output feature map is more important. We propose a Squeeze-and-Excitation Convolutional neural Network (SECNN) for sentence classification. SECNN takes the feature maps from multiple CNN as different channels of sentence representation, and then, we can utilize channel attention mechanism, that is SE attention mechanism, to enable the model to learn the attention weights of different channel features. The results show that our model achieves advanced performance in the sentence classification task.
PromptMTopic: Unsupervised Multimodal Topic Modeling of Memes using Large Language Models
Venue: arXiv cs.CL 2023
Year: 2023

The proliferation of social media has given rise to a new form of communication: memes. Memes are multimodal and often contain a combination of text and visual elements that convey meaning, humor, and cultural significance. While meme analysis has been an active area of research, little work has been done on unsupervised multimodal topic modeling of memes, which is important for content moderation, social media analysis, and cultural studies. We propose \textsf{PromptMTopic}, a novel multimodal prompt-based model designed to learn topics from both text and visual modalities by leveraging the language modeling capabilities of large language models. Our model effectively extracts and clusters topics learned from memes, considering the semantic interaction between the text and visual modalities. We evaluate our proposed model through extensive experiments on three real-world meme datasets, which demonstrate its superiority over state-of-the-art topic modeling baselines in learning descriptive topics in memes. Additionally, our qualitative analysis shows that \textsf{PromptMTopic} can identify meaningful and culturally relevant topics from memes. Our work contributes to the understanding of the topics and themes of memes, a crucial form of communication in today's society.\\ \red{\textbf{Disclaimer: This paper contains sensitive content that may be disturbing to some readers.}}