OpenRaiser/Intern-Atlas huggingface dataset:
Not all layers are equally as important: Every Layer Counts BERT
Venue: arXiv cs.CL 2023
Year: 2023

This paper introduces a novel modification of the transformer architecture, tailored for the data-efficient pretraining of language models. This aspect is evaluated by participating in the BabyLM challenge, where our solution won both the strict and strict-small tracks. Our approach allows each transformer layer to select which outputs of previous layers to process. The empirical results verify the potential of this simple modification and show that not all layers are equally as important.
LLMs-augmented Contextual Bandit
Venue: arXiv cs.LG 2023
Year: 2023

Contextual bandits have emerged as a cornerstone in reinforcement learning, enabling systems to make decisions with partial feedback. However, as contexts grow in complexity, traditional bandit algorithms can face challenges in adequately capturing and utilizing such contexts. In this paper, we propose a novel integration of large language models (LLMs) with the contextual bandit framework. By leveraging LLMs as an encoder, we enrich the representation of the context, providing the bandit with a denser and more informative view. Preliminary results on synthetic datasets demonstrate the potential of this approach, showing notable improvements in cumulative rewards and reductions in regret compared to traditional bandit algorithms. This integration not only showcases the capabilities of LLMs in reinforcement learning but also opens the door to a new era of contextually-aware decision systems.
Contrastive Multi-Modal Representation Learning for Spark Plug Fault Diagnosis
Venue: arXiv cs.LG 2023
Year: 2023

Due to the incapability of one sensory measurement to provide enough information for condition monitoring of some complex engineered industrial mechanisms and also for overcoming the misleading noise of a single sensor, multiple sensors are installed to improve the condition monitoring of some industrial equipment. Therefore, an efficient data fusion strategy is demanded. In this research, we presented a Denoising Multi-Modal Autoencoder with a unique training strategy based on contrastive learning paradigm, both being utilized for the first time in the machine health monitoring realm. The presented approach, which leverages the merits of both supervised and unsupervised learning, not only achieves excellent performance in fusing multiple modalities (or views) of data into an enriched common representation but also takes data fusion to the next level wherein one of the views can be omitted during inference time with very slight performance reduction, or even without any reduction at all. The presented methodology enables multi-modal fault diagnosis systems to perform more robustly in case of sensor failure occurrence, and one can also intentionally omit one of the sensors (the more expensive one) in order to build a more cost-effective condition monitoring system without sacrificing performance for practical purposes. The effectiveness of the presented methodology is examined on a real-world private multi-modal dataset gathered under non-laboratory conditions from a complex engineered mechanism, an inline four-stroke spark-ignition engine, aiming for spark plug fault diagnosis. This dataset, which contains the accelerometer and acoustic signals as two modalities, has a very slight amount of fault, and achieving good performance on such a dataset promises that the presented method can perform well on other equipment as well.
Narrowing the Gap between Zero- and Few-shot Machine Translation by Matching Styles
Venue: arXiv cs.CL 2023
Year: 2023

Large language models trained primarily in a monolingual setting have demonstrated their ability to generalize to machine translation using zero- and few-shot examples with in-context learning. However, even though zero-shot translations are relatively good, there remains a discernible gap comparing their performance with the few-shot setting. In this paper, we investigate the factors contributing to this gap and find that this gap can largely be closed (for about 70%) by matching the writing styles of the target corpus. Additionally, we explore potential approaches to enhance zero-shot baselines without the need for parallel demonstration examples, providing valuable insights into how these methods contribute to improving translation metrics.
Evaluating Machine Learning Classifier Approaches, and their Accuracy for the Detection of Cyberattacks on 5G IoT Systems
Venue: arXiv cs.NI 2023
Year: 2023

As 5G continues to expand its coverage and use. Innovative ideas/technologies continue to be implemented within. New vulnerabilities appear, thus resulting in new methods of mitigation and detection to occur. With the architecture that 5G can implement, DDoS (Distributed Denial of Service) is at a higher risk. There are many methods and approaches to help combat this challenge, most of which are implemented in networks containing Wi-Fi (Wireless Fidelity). This article aims to discuss the possible approaches that could be included in 5G technology. The method we will discuss involves Machine Learning. We have used three classifiers to test on datasets (Naïve Bayes, UltraBoost, LogitBoost) with multiple cross-folds, verifying which would have the highest accuracy with multiple factors (such as the cross-folds, verifying whether the number of folds affects accuracy), expanding upon [25] by using feature selection to obtain more accurate results.
On the Robustness, Connectivity and Giant Component Size of Random K-out Graphs
Venue: arXiv cs.IT 2023
Year: 2023

Random K-out graphs are garnering interest in designing distributed systems including secure sensor networks, anonymous crypto-currency networks, and differentially-private decentralized learning. In these security-critical applications, it is important to model and analyze the resilience of the network to node failures and adversarial captures. Motivated by this, we analyze how the connectivity properties of random K-out graphs vary with the network parameters $K$, the number of nodes ($n$), and the number of nodes that get failed or compromised ($γ_n$). In particular, we study the conditions for achieving \emph{connectivity} {with high probability} and for the existence of a \emph{giant component} with formal guarantees on the size of the largest connected component in terms of the parameters $n,~K$, and $γ_n$. Next, we analyze the property of \emph{$r$-robustness} which is a stronger property than connectivity and leads to resilient consensus in the presence of malicious nodes. We derive conditions on $K$ and $n$ under which the random K-out graph achieves r-robustness with high probability. We also provide extensive numerical simulations and compare our results on random K-out graphs with known results on Erdős-Rényi (ER) graphs.
Identifying Context-Dependent Translations for Evaluation Set Production
Venue: arXiv cs.CL 2023
Year: 2023

A major impediment to the transition to context-aware machine translation is the absence of good evaluation metrics and test sets. Sentences that require context to be translated correctly are rare in test sets, reducing the utility of standard corpus-level metrics such as COMET or BLEU. On the other hand, datasets that annotate such sentences are also rare, small in scale, and available for only a few languages. To address this, we modernize, generalize, and extend previous annotation pipelines to produce CTXPRO, a tool that identifies subsets of parallel documents containing sentences that require context to correctly translate five phenomena: gender, formality, and animacy for pronouns, verb phrase ellipsis, and ambiguous noun inflections. The input to the pipeline is a set of hand-crafted, per-language, linguistically-informed rules that select contextual sentence pairs using coreference, part-of-speech, and morphological features provided by state-of-the-art tools. We apply this pipeline to seven languages pairs (EN into and out-of DE, ES, FR, IT, PL, PT, and RU) and two datasets (OpenSubtitles and WMT test sets), and validate its performance using both overlap with previous work and its ability to discriminate a contextual MT system from a sentence-based one. We release the CTXPRO pipeline and data as open source.
Progress Metrics in DAG-based Consensus
Venue: arXiv cs.DC 2023
Year: 2023

Lachesis protocol~\cite{lachesis2021} leverages a DAG of events to allow nodes to reach fast consensus of events. This work introduces DAG progress metrics to drive the nodes to emit new events more effectively. With these metrics, nodes can select event timing and can choose previous events as parents for their own new events. Our results show that our event timing and parent selection methods can help reaching consensus quicker and thus can reduce lower time to finality significantly.
Enhancing English Writing Proficiency in China's Polytechnic Students An In-Depth Literature Review on the Application of the Input Hypothesis
Venue: arXiv cs.CL 2023
Year: 2023

Having good English writing skills is extremely important for students in polytechnic institutions. However, a lot of students in technical schools have difficulties in reaching high levels of skill. The Input Hypothesis, created by Stephen Krashen, suggests that people learn languages well when they receive information that's a little harder than what they already know but still understandable. This research paper wants to study how the Input Hypothesis can help polytechnic students improve their English writing skills. The study will include real-life observations and experiments from the previous research. We will look at data from polytechnic students who are receiving special writing instruction to see if the Input Hypothesis actually helps improve their writing skills. The paper can better inform polytechnic students, faculty members, and support staff and even members of the larger community about the attributions, the processes, and the possible outcomes of second language development for polytechnic students. Keywords: English writing skills, Polytechnic students, Input hypothesis, Comprehensible input
Stable Diffusion Reference Only: Image Prompt and Blueprint Jointly Guided Multi-Condition Diffusion Model for Secondary Painting
Venue: arXiv cs.CV 2023
Year: 2023

Stable Diffusion and ControlNet have achieved excellent results in the field of image generation and synthesis. However, due to the granularity and method of its control, the efficiency improvement is limited for professional artistic creations such as comics and animation production whose main work is secondary painting. In the current workflow, fixing characters and image styles often need lengthy text prompts, and even requires further training through TextualInversion, DreamBooth or other methods, which is very complicated and expensive for painters. Therefore, we present a new method in this paper, Stable Diffusion Reference Only, a images-to-image self-supervised model that uses only two types of conditional images for precise control generation to accelerate secondary painting. The first type of conditional image serves as an image prompt, supplying the necessary conceptual and color information for generation. The second type is blueprint image, which controls the visual structure of the generated image. It is natively embedded into the original UNet, eliminating the need for ControlNet. We released all the code for the module and pipeline, and trained a controllable character line art coloring model at https://github.com/aihao2000/stable-diffusion-reference-only, that achieved state-of-the-art results in this field. This verifies the effectiveness of the structure and greatly improves the production efficiency of animations, comics, and fanworks.
You Only Forward Once: Prediction and Rationalization in A Single Forward Pass
Venue: arXiv cs.CL 2023
Year: 2023

Unsupervised rationale extraction aims to extract concise and contiguous text snippets to support model predictions without any annotated rationale. Previous studies have used a two-phase framework known as the Rationalizing Neural Prediction (RNP) framework, which follows a generate-then-predict paradigm. They assumed that the extracted explanation, called rationale, should be sufficient to predict the golden label. However, the assumption above deviates from the original definition and is too strict to perform well. Furthermore, these two-phase models suffer from the interlocking problem and spurious correlations. To solve the above problems, we propose a novel single-phase framework called You Only Forward Once (YOFO), derived from a relaxed version of rationale where rationales aim to support model predictions rather than make predictions. In our framework, A pre-trained language model like BERT is deployed to simultaneously perform prediction and rationalization with less impact from interlocking or spurious correlations. Directly choosing the important tokens in an unsupervised manner is intractable. Instead of directly choosing the important tokens, YOFO gradually removes unimportant tokens during forward propagation. Through experiments on the BeerAdvocate and Hotel Review datasets, we demonstrate that our model is able to extract rationales and make predictions more accurately compared to RNP-based models. We observe an improvement of up to 18.4\% in token-level F1 compared to previous state-of-the-art methods. We also conducted analyses and experiments to explore the extracted rationales and token decay strategies. The results show that YOFO can extract precise and important rationales while removing unimportant tokens in the middle part of the model.
Decoy Effect in Search Interaction: A Pilot Study
Venue: arXiv cs.IR 2023
Year: 2023

In recent years, the influence of cognitive effects and biases on users' thinking, behaving, and decision-making has garnered increasing attention in the field of interactive information retrieval. The decoy effect, one of the main empirically confirmed cognitive biases, refers to the shift in preference between two choices when a third option (the decoy) which is inferior to one of the initial choices is introduced. However, it is not clear how the decoy effect influences user interactions with and evaluations on Search Engine Result Pages (SERPs). To bridge this gap, our study seeks to understand how the decoy effect at the document level influences users' interaction behaviors on SERPs, such as clicks, dwell time, and usefulness perceptions. We conducted experiments on two publicly available user behavior datasets and the findings reveal that, compared to cases where no decoy is present, the probability of a document being clicked could be improved and its usefulness score could be higher, should there be a decoy associated with the document.
Task Arithmetic with LoRA for Continual Learning
Venue: arXiv cs.CV 2023
Year: 2023

Continual learning refers to the problem where the training data is available in sequential chunks, termed "tasks". The majority of progress in continual learning has been stunted by the problem of catastrophic forgetting, which is caused by sequential training of the model on streams of data. Moreover, it becomes computationally expensive to sequentially train large models multiple times. To mitigate both of these problems at once, we propose a novel method to continually train transformer-based vision models using low-rank adaptation and task arithmetic. Our method completely bypasses the problem of catastrophic forgetting, as well as reducing the computational requirement for training models on each task. When aided with a small memory of 10 samples per class, our method achieves performance close to full-set finetuning. We present rigorous ablations to support the prowess of our method.
Does Code Review Speed Matter for Practitioners?
Venue: arXiv cs.SE 2023
Year: 2023

Increasing code velocity is a common goal for a variety of software projects. The efficiency of the code review process significantly impacts how fast the code gets merged into the final product and reaches the customers. We conducted a survey to study the code velocity-related beliefs and practices in place. We analyzed 75 completed surveys from 39 participants from the industry and 36 from the open-source community. Our critical findings are (a) the industry and open-source community hold a similar set of beliefs, (b) quick reaction time is of utmost importance and applies to the tooling infrastructure and the behavior of other engineers, (c) time-to-merge is the essential code review metric to improve, (d) engineers have differing opinions about the benefits of increased code velocity for their career growth, and (e) the controlled application of the commit-then-review model can increase code velocity. Our study supports the continued need to invest in and improve code velocity regardless of the underlying organizational ecosystem.
Cryptanalysis of Nikooghadam et al.'s lightweight authentication protocol for Internet of Drones
Venue: arXiv cs.CR 2023
Year: 2023

The Internet of Drones has emerged as a transformative technology with applications spanning various domains, including surveillance, delivery services, and disaster management. Secure communication between controller users and drones is paramount to ensure the transmitted data's confidentiality, integrity, and authenticity. Key agreement protocols are crucial in establishing secure communication channels between users and drones, enabling them to exchange sensitive information and control their operations securely. Recently Nikooghadam et al. proposed a lightweight mutual authentication and key agreement protocol for the Internet of drones. In this article, we provide a descriptive analysis of their proposed scheme and prove that Nikooghadam et al.'s scheme is vulnerable to user tracking attacks and stolen verifier attacks.
Single-Source Shortest Paths with Negative Real Weights in $\tilde{O}(mn^{8/9})$ Time
Venue: arXiv cs.DS 2023
Year: 2023

This paper presents a randomized algorithm for the problem of single-source shortest paths on directed graphs with real (both positive and negative) edge weights. Given an input graph with $n$ vertices and $m$ edges, the algorithm completes in $\tilde{O}(mn^{8/9})$ time with high probability. For real-weighted graphs, this result constitutes the first asymptotic improvement over the classic $O(mn)$-time algorithm variously attributed to Shimbel, Bellman, Ford, and Moore.
Rotation Invariant Transformer for Recognizing Object in UAVs
Venue: arXiv cs.CV 2023
Year: 2023

Recognizing a target of interest from the UAVs is much more challenging than the existing object re-identification tasks across multiple city cameras. The images taken by the UAVs usually suffer from significant size difference when generating the object bounding boxes and uncertain rotation variations. Existing methods are usually designed for city cameras, incapable of handing the rotation issue in UAV scenarios. A straightforward solution is to perform the image-level rotation augmentation, but it would cause loss of useful information when inputting the powerful vision transformer as patches. This motivates us to simulate the rotation operation at the patch feature level, proposing a novel rotation invariant vision transformer (RotTrans). This strategy builds on high-level features with the help of the specificity of the vision transformer structure, which enhances the robustness against large rotation differences. In addition, we design invariance constraint to establish the relationship between the original feature and the rotated features, achieving stronger rotation invariance. Our proposed transformer tested on the latest UAV datasets greatly outperforms the current state-of-the-arts, which is 5.9\% and 4.8\% higher than the highest mAP and Rank1. Notably, our model also performs competitively for the person re-identification task on traditional city cameras. In particular, our solution wins the first place in the UAV-based person re-recognition track in the Multi-Modal Video Reasoning and Analyzing Competition held in ICCV 2021. Code is available at https://github.com/whucsy/RotTrans.
Ego-Network Transformer for Subsequence Classification in Time Series Data
Venue: arXiv cs.LG 2023
Year: 2023

Time series classification is a widely studied problem in the field of time series data mining. Previous research has predominantly focused on scenarios where relevant or foreground subsequences have already been extracted, with each subsequence corresponding to a single label. However, real-world time series data often contain foreground subsequences that are intertwined with background subsequences. Successfully classifying these relevant subsequences requires not only distinguishing between different classes but also accurately identifying the foreground subsequences amidst the background. To address this challenge, we propose a novel subsequence classification method that represents each subsequence as an ego-network, providing crucial nearest neighbor information to the model. The ego-networks of all subsequences collectively form a time series subsequence graph, and we introduce an algorithm to efficiently construct this graph. Furthermore, we have demonstrated the significance of enforcing temporal consistency in the prediction of adjacent subsequences for the subsequence classification problem. To evaluate the effectiveness of our approach, we conducted experiments using 128 univariate and 30 multivariate time series datasets. The experimental results demonstrate the superior performance of our method compared to alternative approaches. Specifically, our method outperforms the baseline on 104 out of 158 datasets.
Time Series Synthesis Using the Matrix Profile for Anonymization
Venue: arXiv cs.DB 2023
Year: 2023

Publishing and sharing data is crucial for the data mining community, allowing collaboration and driving open innovation. However, many researchers cannot release their data due to privacy regulations or fear of leaking confidential business information. To alleviate such issues, we propose the Time Series Synthesis Using the Matrix Profile (TSSUMP) method, where synthesized time series can be released in lieu of the original data. The TSSUMP method synthesizes time series by preserving similarity join information (i.e., Matrix Profile) while reducing the correlation between the synthesized and the original time series. As a result, neither the values for the individual time steps nor the local patterns (or shapes) from the original data can be recovered, yet the resulting data can be used for downstream tasks that data analysts are interested in. We concentrate on similarity joins because they are one of the most widely applied time series data mining routines across different data mining tasks. We test our method on a case study of ECG and gender masking prediction. In this case study, the gender information is not only removed from the synthesized time series, but the synthesized time series also preserves enough information from the original time series. As a result, unmodified data mining tools can obtain near-identical performance on the synthesized time series as on the original time series.
Link residual closeness of graphs with fixed parameters
Venue: arXiv cs.SI 2023
Year: 2023

Link residual closeness is a newly proposed measure for network vulnerability. In this model, vertices are perfectly reliable and the links fail independently of each other. It measures the vulnerability even when the removal of links does not disconnect the graph. In this paper, we characterize those graphs that maximize the link residual closeness over the connected graphs with fixed order and one parameters such as connectivity, edge connectivity, bipartiteness, independence number, matching number, chromatic number, number of vertices and number of cut edges.