OpenRaiser/Intern-Atlas huggingface dataset:
Efficient and Scalable Architecture for Multiple-chip Implementation of Simulated Bifurcation Machines
Venue: arXiv cs.ET 2023
Year: 2023

Ising machines are specialized computers for finding the lowest energy states of Ising spin models, onto which many practical combinatorial optimization problems can be mapped. Simulated bifurcation (SB) is a quantum-inspired parallelizable algorithm for Ising problems that enables scalable multi-chip implementations of Ising machines. However, the computational performance of a previously proposed multi-chip architecture tends to saturate as the number of chips increases for a given problem size because both computation and communication are exclusive in the time domain. In this paper, we propose a streaming architecture for multi-chip implementations of SB-based Ising machines with full spin-to-spin connectivity. The data flow in in-chip computation is harmonized with the data flow in inter-chip communication, enabling the computation and communication to overlap and the communication time to be hidden. Systematic experiments demonstrate linear strong scaling of performance up to the vicinity of the ideal communication limit determined only by the latency of chip-to-chip communication. Our eight-FPGA (field-programmable gate array) cluster can compute a 32,768-spin problem with a high pipeline efficiency of 97.9%. The performance of a 79-FPGA cluster for a 100,000-spin problem, projected using a theoretical performance model validated on smaller experimental clusters, is comparable to that of a state-of-the-art 100,000-spin optical Ising machine.
Improving the Robustness of Transformer-based Large Language Models with Dynamic Attention
Venue: arXiv cs.CL 2023
Year: 2023

Transformer-based models, such as BERT and GPT, have been widely adopted in natural language processing (NLP) due to their exceptional performance. However, recent studies show their vulnerability to textual adversarial attacks where the model's output can be misled by intentionally manipulating the text inputs. Despite various methods that have been proposed to enhance the model's robustness and mitigate this vulnerability, many require heavy consumption resources (e.g., adversarial training) or only provide limited protection (e.g., defensive dropout). In this paper, we propose a novel method called dynamic attention, tailored for the transformer architecture, to enhance the inherent robustness of the model itself against various adversarial attacks. Our method requires no downstream task knowledge and does not incur additional costs. The proposed dynamic attention consists of two modules: (I) attention rectification, which masks or weakens the attention value of the chosen tokens, and (ii) dynamic modeling, which dynamically builds the set of candidate tokens. Extensive experiments demonstrate that dynamic attention significantly mitigates the impact of adversarial attacks, improving up to 33\% better performance than previous methods against widely-used adversarial attacks. The model-level design of dynamic attention enables it to be easily combined with other defense methods (e.g., adversarial training) to further enhance the model's robustness. Furthermore, we demonstrate that dynamic attention preserves the state-of-the-art robustness space of the original model compared to other dynamic modeling methods.
Dynamic Dense Graph Convolutional Network for Skeleton-based Human Motion Prediction
Venue: arXiv cs.CV 2023
Year: 2023

Graph Convolutional Networks (GCN) which typically follows a neural message passing framework to model dependencies among skeletal joints has achieved high success in skeleton-based human motion prediction task. Nevertheless, how to construct a graph from a skeleton sequence and how to perform message passing on the graph are still open problems, which severely affect the performance of GCN. To solve both problems, this paper presents a Dynamic Dense Graph Convolutional Network (DD-GCN), which constructs a dense graph and implements an integrated dynamic message passing. More specifically, we construct a dense graph with 4D adjacency modeling as a comprehensive representation of motion sequence at different levels of abstraction. Based on the dense graph, we propose a dynamic message passing framework that learns dynamically from data to generate distinctive messages reflecting sample-specific relevance among nodes in the graph. Extensive experiments on benchmark Human 3.6M and CMU Mocap datasets verify the effectiveness of our DD-GCN which obviously outperforms state-of-the-art GCN-based methods, especially when using long-term and our proposed extremely long-term protocol.
MM-Narrator: Narrating Long-form Videos with Multimodal In-Context Learning
Venue: arXiv cs.CV 2023
Year: 2023

We present MM-Narrator, a novel system leveraging GPT-4 with multimodal in-context learning for the generation of audio descriptions (AD). Unlike previous methods that primarily focused on downstream fine-tuning with short video clips, MM-Narrator excels in generating precise audio descriptions for videos of extensive lengths, even beyond hours, in an autoregressive manner. This capability is made possible by the proposed memory-augmented generation process, which effectively utilizes both the short-term textual context and long-term visual memory through an efficient register-and-recall mechanism. These contextual memories compile pertinent past information, including storylines and character identities, ensuring an accurate tracking and depicting of story-coherent and character-centric audio descriptions. Maintaining the training-free design of MM-Narrator, we further propose a complexity-based demonstration selection strategy to largely enhance its multi-step reasoning capability via few-shot multimodal in-context learning (MM-ICL). Experimental results on MAD-eval dataset demonstrate that MM-Narrator consistently outperforms both the existing fine-tuning-based approaches and LLM-based approaches in most scenarios, as measured by standard evaluation metrics. Additionally, we introduce the first segment-based evaluator for recurrent text generation. Empowered by GPT-4, this evaluator comprehensively reasons and marks AD generation performance in various extendable dimensions.
Violating Constant Degree Hypothesis Requires Breaking Symmetry
Venue: arXiv cs.CC 2023
Year: 2023

The Constant Degree Hypothesis was introduced by Barrington et. al. (1990) to study some extensions of $q$-groups by nilpotent groups and the power of these groups in a certain computational model. In its simplest formulation, it establishes exponential lower bounds for $\mathrm{AND}_d \circ \mathrm{MOD}_m \circ \mathrm{MOD}_q$ circuits computing AND of unbounded arity $n$ (for constant integers $d,m$ and a prime $q$). While it has been proved in some special cases (including $d=1$), it remains wide open in its general form for over 30 years. In this paper we prove that the hypothesis holds when we restrict our attention to symmetric circuits with $m$ being a prime. While we build upon techniques by Grolmusz and Tardos (2000), we have to prove a new symmetric version of their Degree Decreasing Lemma and apply it in a highly non-trivial way. Moreover, to establish the result we perform a careful analysis of automorphism groups of $\mathrm{AND} \circ \mathrm{MOD}_m$ subcircuits and study the periodic behaviour of the computed functions. Finally, our methods also yield lower bounds when $d$ is treated as a function of $n$.
Slot-Mixup with Subsampling: A Simple Regularization for WSI Classification
Venue: arXiv cs.CV 2023
Year: 2023

Whole slide image (WSI) classification requires repetitive zoom-in and out for pathologists, as only small portions of the slide may be relevant to detecting cancer. Due to the lack of patch-level labels, multiple instance learning (MIL) is a common practice for training a WSI classifier. One of the challenges in MIL for WSIs is the weak supervision coming only from the slide-level labels, often resulting in severe overfitting. In response, researchers have considered adopting patch-level augmentation or applying mixup augmentation, but their applicability remains unverified. Our approach augments the training dataset by sampling a subset of patches in the WSI without significantly altering the underlying semantics of the original slides. Additionally, we introduce an efficient model (Slot-MIL) that organizes patches into a fixed number of slots, the abstract representation of patches, using an attention mechanism. We empirically demonstrate that the subsampling augmentation helps to make more informative slots by restricting the over-concentration of attention and to improve interpretability. Finally, we illustrate that combining our attention-based aggregation model with subsampling and mixup, which has shown limited compatibility in existing MIL methods, can enhance both generalization and calibration. Our proposed methods achieve the state-of-the-art performance across various benchmark datasets including class imbalance and distribution shifts.
Challenges for Conflict Mitigation in O-RAN's RAN Intelligent Controllers
Venue: arXiv cs.NI 2023
Year: 2023

The O-RAN architecture enables a more flexible and dynamic radio access network (RAN) control by separating hardware and software components. However, the distributed nature of the O-RAN architecture also presents several challenges for mitigating network control conflicts that can arise between different network elements. In this article, we identify key challenges for conflict mitigation in O-RAN networks, including reliable conflict detection, efficient maintenance of conflict mitigation configuration, optimal conflict resolution logic, testing and evaluation methodologies, and limited observability of O-RAN components. We propose solutions to these challenges, including pre-deployment conflict mitigation, conflict detection and resolution, and supervision and adaptation. The article concludes by highlighting the need for ongoing research to address these challenges and ensure effective conflict mitigation in O-RAN deployments.
Enhancing Answer Selection in Community Question Answering with Pre-trained and Large Language Models
Venue: arXiv cs.CL 2023
Year: 2023

Community Question Answering (CQA) becomes increasingly prevalent in recent years. However, there are a large number of answers, which is difficult for users to select the relevant answers. Therefore, answer selection is a very significant subtask of CQA. In this paper, we first propose the Question-Answer cross attention networks (QAN) with pre-trained models for answer selection and utilize large language model (LLM) to perform answer selection with knowledge augmentation. Specifically, we apply the BERT model as the encoder layer to do pre-training for question subjects, question bodies and answers, respectively, then the cross attention mechanism selects the most relevant answer for different questions. Experiments show that the QAN model achieves state-of-the-art performance on two datasets, SemEval2015 and SemEval2017. Moreover, we use the LLM to generate external knowledge from questions and correct answers to achieve knowledge augmentation for the answer selection task by LLM, while optimizing the prompt of LLM in different aspects. The results show that the introduction of external knowledge can improve the correct answer selection rate of LLM on datasets SemEval2015 and SemEval2017. Meanwhile, LLM can also select the correct answer on more questions by optimized prompt.
On $(n,σ)-$equivalence relation between skew constacyclic codes
Venue: arXiv cs.IT 2023
Year: 2023

In this paper we generalize the notion of $n$-equivalence relation introduced by Chen et al. in \cite{Chen2014} to classify constacyclic codes of length $n$ over a finite field $\mathbb{F}_q$, where $q=p^r$ is a prime power, to the case of skew constacyclic codes without derivation. We call this relation $(n,σ)$-equivalence relation, where $n$ is the length of the code and $ σ$ is an automorphism of the finite field. We compute the number of $(n,σ)$-equivalence classes, and we give conditions on $ λ$ and $μ$ for which $(σ, λ)$-constacyclic codes and $(σ, λ)$-constacyclic codes are equivalent with respect to our $(n,σ)$-equivalence relation. Under some conditions on $n$ and $q$ we prove that skew constacyclic codes are equivalent to cyclic codes. We also prove that when $q$ is even and $σ$ is the Frobenius autmorphism, skew constacyclic codes of length $n$ are equivalent to cyclic codes when $\gcd(n,r)=1$. Finally we give some examples as applications of the theory developed here.
Emergent Outcomes of the veToken Model
Venue: arXiv cs.GT 2023
Year: 2023

Decentralised organisations use blockchains as a basis for governance: they use on-chain transactions to allocate voting weight, publish proposals, cast votes, and enact the results. However, blockchain-based governance structures have challenges, mostly notably, the need to align the short-term outlook of pseudononymous voters with the long-term growth and success of the decentralised organisation. The Vote-Escrowed Token (veToken) model attempts to resolve this tension by requiring voters to escrow or lock tokens of value for an extended period in exchange for voting weight. In this paper, we describe the veToken model and analyse its emergent outcomes. We show that voting behaviour follows bribes set by higher-level protocols, and that the cost per vote varies depending on how it is acquired. We describe the implementation of the veToken model by Curve Finance, a popular automated market maker for stablecoins, and the ecosystem of protocols that has arisen on top of this implementation. We show that voting markets such as Votium largely determine the outcome of fortnightly votes held by Convex Finance, and we show that Frax Finance, a stablecoin issuer, plays a central role in the ecosystem even though they directly lock relatively few tokens with Curve. Instead, they indirectly lock tokens through yield aggregators such as Convex Finance and purchase voting weight through voting markets such as Votium. Although the veToken model in isolation is straight-forward and easily explained, it leads to many complex and emergent outcomes. Decentralised organisations should consider these outcomes before adopting the model.
Continual Learning with Low Rank Adaptation
Venue: arXiv cs.LG 2023
Year: 2023

Recent work using pretrained transformers has shown impressive performance when fine-tuned with data from the downstream problem of interest. However, they struggle to retain that performance when the data characteristics changes. In this paper, we focus on continual learning, where a pre-trained transformer is updated to perform well on new data, while retaining its performance on data it was previously trained on. Earlier works have tackled this primarily through methods inspired from prompt tuning. We question this choice, and investigate the applicability of Low Rank Adaptation (LoRA) to continual learning. On a range of domain-incremental learning benchmarks, our LoRA-based solution, CoLoR, yields state-of-the-art performance, while still being as parameter efficient as the prompt tuning based methods.
Creator Context for Tweet Recommendation
Venue: arXiv cs.IR 2023
Year: 2023

When discussing a tweet, people usually not only refer to the content it delivers, but also to the person behind the tweet. In other words, grounding the interpretation of the tweet in the context of its creator plays an important role in deciphering the true intent and the importance of the tweet. In this paper, we attempt to answer the question of how creator context should be used to advance tweet understanding. Specifically, we investigate the usefulness of different types of creator context, and examine different model structures for incorporating creator context in tweet modeling. We evaluate our tweet understanding models on a practical use case -- recommending relevant tweets to news articles. This use case already exists in popular news apps, and can also serve as a useful assistive tool for journalists. We discover that creator context is essential for tweet understanding, and can improve application metrics by a large margin. However, we also observe that not all creator contexts are equal. Creator context can be time sensitive and noisy. Careful creator context selection and deliberate model structure design play an important role in creator context effectiveness.
Vulnerability of Automatic Identity Recognition to Audio-Visual Deepfakes
Venue: arXiv cs.CV 2023
Year: 2023

The task of deepfakes detection is far from being solved by speech or vision researchers. Several publicly available databases of fake synthetic video and speech were built to aid the development of detection methods. However, existing databases typically focus on visual or voice modalities and provide no proof that their deepfakes can in fact impersonate any real person. In this paper, we present the first realistic audio-visual database of deepfakes SWAN-DF, where lips and speech are well synchronized and video have high visual and audio qualities. We took the publicly available SWAN dataset of real videos with different identities to create audio-visual deepfakes using several models from DeepFaceLab and blending techniques for face swapping and HiFiVC, DiffVC, YourTTS, and FreeVC models for voice conversion. From the publicly available speech dataset LibriTTS, we also created a separate database of only audio deepfakes LibriTTS-DF using several latest text to speech methods: YourTTS, Adaspeech, and TorToiSe. We demonstrate the vulnerability of a state of the art speaker recognition system, such as ECAPA-TDNN-based model from SpeechBrain, to the synthetic voices. Similarly, we tested face recognition system based on the MobileFaceNet architecture to several variants of our visual deepfakes. The vulnerability assessment show that by tuning the existing pretrained deepfake models to specific identities, one can successfully spoof the face and speaker recognition systems in more than 90% of the time and achieve a very realistic looking and sounding fake video of a given person.
Multi-dimensional Energy Limitation in Sphere Shaping for Nonlinear Interference Noise Mitigation
Venue: arXiv cs.IT 2023
Year: 2023

We propose Four-Dimensional (4D) energy limit enumerative sphere shaping (ESS) of $M$-QAM signaling to minimize rate loss and improve the transmission performance over non-linear WDM optical-fiber systems. Simulation results show that the proposed scheme outperforms the conventional ESS by $0.19$~bit/4D-symbol in achievable information rate over a $205$-km single-span link and a WDM transmission of five polarization-division-multiplexed channels with $400$-Gbit/s net rate per channel. We also study the achieved performance over several shaping block lengths and show that the achieved gains do not scale well over multi-span systems.
BAND-2k: Banding Artifact Noticeable Database for Banding Detection and Quality Assessment
Venue: arXiv cs.CV 2023
Year: 2023

Banding, also known as staircase-like contours, frequently occurs in flat areas of images/videos processed by the compression or quantization algorithms. As undesirable artifacts, banding destroys the original image structure, thus degrading users' quality of experience (QoE). In this paper, we systematically investigate the banding image quality assessment (IQA) problem, aiming to detect the image banding artifacts and evaluate their perceptual visual quality. Considering that the existing image banding databases only contain limited content sources and banding generation methods, and lack perceptual quality labels (i.e. mean opinion scores), we first build the largest banding IQA database so far, named Banding Artifact Noticeable Database (BAND-2k), which consists of 2,000 banding images generated by 15 compression and quantization schemes. A total of 23 workers participated in the subjective IQA experiment, yielding over 214,000 patch-level banding class labels and 44,371 reliable image-level quality ratings. Subsequently, we develop an effective no-reference (NR) banding evaluator for banding detection and quality assessment by leveraging frequency characteristics of banding artifacts. A dual convolutional neural network is employed to concurrently learn the feature representation from the high-frequency and low-frequency maps, thereby enhancing the ability to discern banding artifacts. The quality score of a banding image is generated by pooling the banding detection maps masked by the spatial frequency filters. Experiments demonstrate that our banding evaluator achieves a remarkably high accuracy in banding detection and also exhibits high SRCC and PLCC results with the perceptual quality labels. These findings unveil the strong correlations between the intensity of banding artifacts and the perceptual visual quality, thus validating the necessity of banding quality assessment.
A Simple and General Operational Framework to Deploy Optimal Routes with Source Routing
Venue: arXiv cs.NI 2023
Year: 2023

Source Routing, currently facilitated by Segment Routing (SR), enables precise control of forwarding paths by specifying detours (or segments) to deviate IP packets along routes with advanced properties beyond typical shortest IGP paths. Computing the desired optimal segment lists, known as encoding, leads to interesting challenges as the number of detours is tightly constrained for hardware performance. Existing solutions either lack generality, correctness, optimality, or practical computing efficiency-in particular for sparse realistic networks. In this paper, we address all such challenges with GOFOR-SR. Our framework extends usual path computation algorithms to inherently look at optimal and feasible segment lists, streamlining the deployment of TE-compliant paths. By integrating encoding within the path computation itself and modifying the distance comparison method, GOFOR allows algorithms with various optimization objectives to efficiently compute optimal segment lists. Despite the loss of substructure optimality induced by SR, GOFOR proves particularly efficient, inducing only a linear overhead at worst. It also offers different strategies and path diversity options for intricate TE-aware loadbalancing. We formally prove the correctness and optimality of GOFOR, implement our framework for various practical usecases, and demonstrate its performance and benefits on both real and challenging topologies.
Collaborative software design and modeling in virtual reality
Venue: arXiv cs.SE 2023
Year: 2023

Context: Software engineering is becoming more and more distributed. Developers and other stakeholders are often located in different locations, departments, and countries and operating within different time zones. Most online software design and modeling tools are not adequate for distributed collaboration since they do not support awareness and lack features for effective communication. Objective: The aim of our research is to support distributed software design activities in Virtual Reality (VR). Method: Using design science research methodology, we design and evaluate a tool for collaborative design in VR. We evaluate the collaboration efficiency and recall of design information when using the VR software design environment compared to a non-VR software design environment. Moreover, we collect the perceptions and preferences of users to explore the opportunities and challenges that were incurred by using the VR software design environment. Results: We find that there is no significant difference in the efficiency and recall of design information when using the VR compared to the non-VR environment. Furthermore, we find that developers are more satisfied with collaboration in VR. Conclusion: The results of our research and similar studies show that working in VR is not yet faster or more efficient than working on standard desktops. It is very important to improve the interface in VR (gestures with haptics, keyboard and voice input), as confirmed by the difference in results between the first and second evaluation.