OWL: A Large Language Model for IT Operations
Authors: Hongcheng Guo
Jian Yang
Jiaheng Liu
Liqun Yang
Linzheng Chai
Jiaqi Bai
Junran Peng
Xiaorong Hu
Chao Chen
Dongfeng Zhang
xu Shi
Tieqiao Zheng
liangfan zheng
Bo Zhang
Ke Xu
Zhoujun Li
Venue: ICLR 2024 poster (2024)
Keywords: Large language model
IT operations
AIOps
Primary area: datasets and benchmarks

Abstract: With the rapid advancement of IT operations, managing and analyzing large data volumes efficiently for practical applications has become increasingly critical. Natural Language Processing (NLP) techniques have demonstrated remarkable capabilities in various tasks, including named entity recognition, machine translation, and dialogue systems. Recently, Large Language Models (LLMs) have achieved significant improvements across various domain-specific areas. However, there is a noticeable gap in the development of specialized Large Language Models (LLMs) tailored for IT operations. In this paper, we introduce the OWL, a large language model trained on our constructed Owl-Instruct with a wide range of IT-related information. Specifically, limited by the maximum input length, we propose the \textbf{H}omogeneous \textbf{M}arkov \textbf{C}ontext \textbf{E}xtension method (HMCE). The mixture-of-adapter strategy is leveraged to improve the parameter-efficient tuning across different domains or tasks.
Further, we evaluate the performance of OWL on the Owl-Bench established by us and open IT-related benchmarks. OWL  demonstrates superior performance results on IT tasks, which outperforms existing models by significant margins. Moreover, we hope that the findings of our work will provide more insights to revolutionize the techniques of IT operations with specialized LLMs.

Published as a conference paper at ICLR 2024
OWL A LARGE LANGUAGE MODEL FOR IT OPERATIONS
Hongeheng Guo', Jian Yang"; Liu Liqun Yang', Linzheng Chai', Jiaqi Bai', Junran Peng', Xiaorong Hu², Chao Chen', Dongfeng Zhang", Xu Shi,, Tieqiao Zheng', Liangfan Zheng', Bo Zhang", Ke Xu', Zhoujun Li' Sttat Key Lab of Software Development Environment, Beihang University CCooudiss Research
fhengchengguo, jiaya, liwjiaheneeee 1qyang, bjq, Jbuuaa edu. [tim. imishi,, steven. zheng, leven. zeeen. bowen. zhang @ wwwwwse.wwoooooo
ABSTRACT
With the rapid advancement of IT operations, managing and analyzing large data volumes efficiently for practical applications has become increasingly crit- ical. Natural Language Processing (NLP) techniques have demonstrated remark- able capabilities in various tasks, including named entity recognition, machine translation, and dialogue systems. Recently, Large Language Models (LLMs) have achieved significant improvements across various domain-specific areas. However, there is noticeable gap in the development of specialized Large Language Models (LLMs) tailored for IT operations. In this paper, we intro- constructed Owl-Instre
duce the OWL, large language model trained with a wide range of I'T-related information. Specifically, limited by the max- imum input length, we propose the Homogeneous Markov Context Extension method (HMCE). The mixture-of-adapter strategy is leveraged to improve the parameter-efficieet tuning across different domains or tasks. Further, eval- uate the performance of OWL on the Owl-Bench established by us and open IT-related benchmarks. OWL demonstrates superior performance results on IT tasks, which outperforms existing models by significant margins. Moreover, we hope that the findings of our work will provide more insights revolutionize the techniques of IT operations with specialized LLMss. Our code is available at attps: htas:://iithun com/ HH/HHCGGGG Guo/Owl
INTRODUCTION
Large Language Models (LLMs) (Chowdhery et al. 2022b Touvron al. 2023a b) have emerged as powerful tools in the field of natural language processing (NLP) and Artificial Intelligence (AI). The release of GPT-3 (Brown et al. 2020a) 2020 demonstrated the advantages of training large auto-regressive LLMs. With 175 billion parameters, GPT-3 surpassed previous models in various LLM tasks including reading comprehension, question answering, and code generation. Similar re- sults have been achieved by other models as well. Furthermore, evidence suggests that larger models exhibit emergent behaviors and possess abilities not present in smaller models. For instance, they can learn tasks from few examples, phenomenon known few-shot prompting. This capability expands the scope of supported tasks and facilitates the automation of new language tasks for users. However, the majority of research endeavors have been directed towards constructing general Large Language Models (LLMs) that encompass a wide spectrum of subjects, certain models trained on domain-specific data have exhibited exceptional performance within their specific domains, such as science and medicine. These discoveries underscore the necessity for additional advancements in domain-specific models.
In the field of IT operations (Du et al. 2017: Liu et al. 2023; Guo et al. 2023a), the significance of natural language processing (NLP) technologies steadily on the rise. This paper undertakes the crucial task of delineating à set of specific assignments within the realm of IT operations, en- compassing areas such as information security, system architecture, and other domains. However,
"Corresponding authors.

---

Published as a conference paper at ICLR 2024
the complexity and specific terminology of IT operations pose formidable challenges, including unique set of terminologies, processes, and contextual nuances that are not easily decipherable by conventional NLP models. Therefore, it becomes increasingly evident that there is a pressing need for the development and deployment of a Large Language Model specifically tailored to the ex- igencies of IT operations within such specialized domains. The fine-tuned large language model customized to this purpose promises to be an invaluable asset in navigating the complexities of IT operations within these highly specialized domains. Such specialized large language model would greatly enhance the efficiency, accuracy, and comprehension of I'T-related tasks and communications within these niche areas, which will ultimately advance the field of IT operations management.
In this paper, we introduce the OWL, a large language model trained on our collected Owl- Owl-Instruct with a wide range of knowledge from operations and maintenance (O&M), where our data contains nine prevalent domains within O&M: information security, application, system architecture, soft- ware architecture, middleware, network, operating system, infrastructure, and database. Besides, explore the use of the data augmentation (Xu et al. 2023b; Wang et al. (023a) strategy to enable LLMs to accurately generate large, high-quality and diverse instruction data from a set of human- annotated data samples. To maintain stringent standard of data quality, employ two-pronged approach that combines GPT-4 (OpenAl) 2023) scoring with meticulous manual validation. In addi- tion, we have embarked the development of Owl-Bench evaluate the performance of different LLMs on I'T-related tasks, an extensive bilingual benchmark that comprises two distinct segments: a Q&A (question- canstionnanswer) part consisting of 317 entries, and multiple-choice part containing 1,000 questions. In terms of model design, constrained by the maximum input length, inspired by the recent NBCE (Sul 2023) (Naive Bayes-based Context Extension), we propose the Homogeneous Markov Context Extension (HMCE) approach. Furthermore, in order to enhance the efficacy of instruction adaptation across various tasks, we put forward the strategy of employing a Mixture-of- Adapter method to facilitate supervised fine-tuning.
The contributions of our paper are as follows:
Owl-Instruct Construction. We collect and label 3000 seed samples and then prompt Chat- GPT (Ouyang al.) 2022) to generate diverse instructions. To cover practical scenarios, we curate instructions that involve both single-turn and multi-turn scenarios.
Owl-Bench Construction. We have established a big model test benchmark for the op- eration and maintenance(O&M)) domain measure LLMs capabilities, which consists of nine O&M-related domains, showing the diversity of LLMs capabilities in the domain in hierarchical manner.
Model and Training. For tokenization, we expand the word vocabulary using the extra IT-related data. Besides, we propose a simple method to extend input length based on the homogeneous Markov chain. As for training, Mfxture-of-Adapter strategy is proposed to improve the instruction-tuning performance.
Impressive Performance. We evaluate the performance of OWL with other LLMs on mul- tiple benchmark datasets, including Owl-Bench and open I'T-related benchmarks. OWL. demonstrates superior performance results on IT tasks, which outperforms existing models by significant margins and maintains effective generalization abilities on Owl-Bench.
OWL-INSTRUCT DATASET CONSTRUCTION
The quality of the data employed in training Large Language Models (LLMs) stands as a pivotal determinant influencing the ultimate performance of the language model. Zhou et al. (2023) under- score the paramount importance attributed to both the diversity and quality of data when training large-scale models. Consequently, it becomes imperative for us to curate high-caliber instruction dataset, which the Owl-Instruct, tailored specifically for the realm of Operations and Maintenance (O&M). The overview of constructing Owl-Instruct is in Figure T A statistical analysis is presented in Table [ and a visual representation of the keywords depicted in Figure
2.1 SEED DATA COLLECTION
In the initial phase of our project, we engage 25 subject matter experts within the field of operations and maintenance (O&M) to meticulously craft input and output sequences, along with comprehen- sive instructions. These encompass wide spectrum of common domains and tasks. Concretely,
2

---

Published as a conference paper at ICLR 2024
Seed Data
Step4: Supervised Fine-turing
Dinppeal Samplex
Step1: Data Generation
Owl f :LLM Operation
Yes
No
Step3: Manual Venfication
Owt-Instruet
Valid
Step2: GPT4 Sereening
Pronpe
Scaring Resadte
Instruction
(Quality Criteria) (Input Data)
antirely beiong folld maintenance,
Single-turn
Multi-turn
Figure 1: Four phases of constructing Owl-Instruct and how train our OWL.
Dataset
#Seed data
#Total dialogues
#Average turns
#Average dialogue length
Single-turn
2000
9118
335
Multi-tum
1000
8740
2.9
918
Table 1: Statistical analysis of the single-turn and multi-turn scenarios.
Owl-Instruct contains nine prevalent domains within O&M: information security, application, sys- tem architecture, software architecture, middleware, network, operating system, infrastructure, and database. Within each domain, plethora of tasks are encapsulated, including but not limited de- ployment, monitoring, fault diagnosis, performance optimization, log analysis, backup and recovery, among others. Finally, we gain 2,000 single-turn and 1,000 multi-turn seed data instances, which serve the foundation for further augmenting the scale and diversity of Owl-Instruct.
2.2 DATASET AUGMENTATION
Single-turn Dataset Construction In our endeavor, we have constructed a comprehensive single- turn dialogue dataset tailored specifically to the domain of operations and maintenance. boasting an impressive 9,115 18 meticulously curated data entries. Motivated by Self-Instruct (Wang et al,, 2023a), we have enriched our dataset. This enrichment involves the generation of supplementary samples derived from the seed data, corpus painstakingly labeled by our domain experts. Besides, we consider GPT4 (OpenAl 2023) as a reference and supervisor for ensuring the quality of data. Please refer to Appendix E.I for cases.
Multi-turn Dataset Construction In accordance with the methodology elucidated in Baize (Xu al. 2023b), the generation process of our multi-turn dialogue dataset within the operations and maintenance domain encompasses the following four distinct phases (Case in Appendix E.2): (1) Seed Data Collection, (2) Topic Generation, (3) Multi-turn Dialogue Generation, and (4) Manual
and GPT4 Screening. More details can be seen in Appendix
2.3 DATA QUALITY
In order to maintain a stringent standard of data quality, we employ a two-pronged approach that combines GPT-4 (OpenAl 2023) scoring with meticulous manual validation. This dual-validation process ensures the integrity and reliability of generated data while enhancing its overall quality. When leveraging GPT-4 for scoring, we meticulously design specific prompts tailored to our dataset. These prompts are strategically crafted to enable GPT-4 to evaluate and rate the generated data based on predefined quality criteria. This automated scoring mechanism allows us to swiftly identify and filter out any low-quality data instances. Moreover, it serves valuable tool for flagging potential issues and areas that require improvement within the dataset. Simultaneously. dataset undergoes rigorous manual validation. A team of expert reviewers conducts an in-depth assessment of each data entry. This manual inspection process entails a thorough examination content, coherence. and adherence to domain-specific knowledge. Entries that do meet our stringent quality standards
3

---

Published as a conference paper at ICLR 2024
ATops
operation
optinization
opt
metric
Capactry
ver
Dait
Interne
Bocker's
Ergurry
file
Alen
a
beta
database
service
Darope
SShautonatic aattonatic
Figure 2: Keyword clouds in the Owl- Ins-runtt This keyword cloud is built based on word frequency. Words like "data", and "service" appear more frequently.
meticulously flagged and subsequently removed. Please refer to Appendix F. for prompt and case details.
3 OWL-BENCE BENCHMARK CONSTRUCTION
Overview: In the absence of benchmark tailored for evaluating the performance of large lan- guage models in the context of Operations and Maintenance (O&M), there exists a critical gap in our ability to effectively assess and compare models within this area. To address this deficiency, have embarked on the development of Owl-Bench, an extensive bilingual benchmark that comprises two distinct segments: Q&A (question- ) part consisting of 317 entries, and multiple- choice part containing 1,000 questions in Table 2 The O&M is characterized by its vastness and multidisciplinary nature (Dang al.] 2019 Rijal et al., 2022, Gao & Pishdad- Pishdad-Bozorgii (209)). In- spired by these works, have meticulously curated the sub-fields included in Owl-Bench. We consider the multitude real-world industrial scenarios that encompass this area, ensuring that our benchmark exhibits comprehensive diversity. Our data collection process involves the acquisition of test data from nine distinct subdomains: information security, application, system architecture, software architecture, middleware, network, operating system, infrastructure, and database. And the data consists of Q&A pairs and multiple-choice questions. It is essential to highlight that the data incorporated into Owl-Bench significantly differs from that of Owl-Instruct. The former is derived directly from real-world scenario-based examination questions, devoid of any modification or expansion through GPT4 In Appendix [E33 and E.4, we present examples of multiple-choice questions and Q&A question across these nine domains, providing a glimpse into the rich diversity encapsulated within the benchmark.
3.1 DATA COLLECTION AND PROCESSING
Our main source of data consists of two parts: part is free practice exam available the Internet. The other questions are carefully designed by O&M experts. More details can be seen in Appendix H The data collected are in a variety of formats, mainly PDF or Microsoft Word documents, with small percentage of web pages. The PDF documents are initially processed into text using OCR tools
(Li et 2023). Some cases that are difficult to process will be manually parsed into a structured format by hand, similar to Hendrycks et al. (2021): Taylor et al. 2022))
Middinam Informatio wearay
Intmrrcsuu Applicatin
Operatin FVART
Syatm acciuctue Netooek
Softwane archincnee
2664
#iourage daaiggs Ieagh
all
333
286
348
*Qrestons:
126
304
110
363
118
119
87
122
MS
RAwange dhllggs leaph
212
343
Table 2: Statistical analysis of the Owl-Bench.
4 MODEL
4.1 TOKENIZATION
Since LLaMA2 Touvron et al. 2023b) is designed to support natural language in Latin or Cyrillic language families, it has unsatisfactory compatibility with ITT operation data. For compatibility with
4

---

Published as a conference paper at ICLR 2024
both natural language and IT-related data, expand the word vocabulary size from 000 000 to 48, 553. (More details in Appendix A
4.2 LONG-CONTEXT INPUT
Limited by the maximum input length, inspired by the recent NBCE (Su 2023) (Naive Bayes-based Context Extension), we propose the Homogeneous Markov Context Extension method (HMCE). The fundamental assumption of NBCE is the independent input contexts, but in reality, input texts are interconnected and exhibit continuity rather than independence. Consequently, we abandon the independence assumption and instead employ homogeneous Markov chain assumption to extend NBCE, catering to the continuous nature of the input data, which is named HMCE.
Let T be the generated text. Let S := .S.,..., S,)) be the previous contexts, and that their overall length has exceeded the training length. We want to generate 7 based on Si,..., Sn, that is, we want estimate SS)) (See Appendix D.2 for more details and derivation.)
n-1 Σ
( x Σ
P(T(S)
where n. is the number of separate content. For the original input sentences S longer than maximum length L, we slit them into multiple segments.
4.3 MIXTURE-OF-ADATIO
Yang et al. 2022 He et al. 2022 Wang al. 2023b) 2022) is a simple but effective technique L.MMs, which fixed pre-trained transfer, we use a mixture adapters for different domains and tasks, where group of LoRA adapters is lightweight compared to the pre-trained model. The adapters with a low-rank down-project matrix and up-project matrix can be directly inserted into the pre-trained embedding, attention, and feed-forward network. Given the source sentence (.i,..., Tm } of m tokens and a group of T Adapters, we use mixture-of-adapters to learn the task-specific and domain-specife representations for diverse input:
Parameter-efficient tuning HHoulsyy et al. 2019, Zhu et al.] 2021 Hu et al. 2022
enables efficient and flexible transfer by introducing task-specific modifications
model. For the cross- cooss-domain/ross--
h Ta Ae (h"r)
(2)
where g(L)) are selected LoRA experts derived from the language representations. A(-) denotes the LoRA adapter module and 0 (01,-... 0 denotes the adapter pool. Au is calculated by:
Aogu., (h = h Σ
aAWh
(3)
where AW BA is denoted by a low-rank decomposition e Ra A R Ar & d). The matrices A and B are initialized by random Gaussian distribution and zero. is the scaling factor and r is the inner dimension. S(e) denotes the subset.
In Equation all experts only require fine-tuning small number of language-specific parameters instead of all parameters of the pre-trained model. Thus, we can simultaneously train multiple experts for different languages, which all share the same freezing pre-trained parameters. We use multiple adapters from the selected subset to maximize the transfer of knowledge across languages:
explatt) (
9(L.) Topk
Ennerplos explott)
where TopK(-) is the selection function, where we calculate the selection probabilities of all LoRA adapters and choose the top-k LoRA experts obeying the probability distribution. a is a scalar from the representations of language L, (We use the hidden state of the special token [CLS] of each layer). S(e) ((As, BK)) and af is used to incorporate the different experts.
5

---

Published as a conference paper at ICLR 2024
We project the language representation e of language L, into the LoRA expert distribution using the learned matrix Wa E RdxT, where d is the hidden size and is the number of experts. The weight of LoRA expert aj is calculated by:
a e W.
(5)
where = for = 1 For all modules of the pre-trained model, we leverage the mixture- of-adapter strategy learn the task-sensitive representations for the different input sentences by activating top-k experts.
4.4 SUPERVISED FINE-TUNING
Given the multiple tasks T = {Ti we construct the multi-task training corpora D= = {Di where each dataset contains a series triple tuples {(x(1) , } where xlf) and yاu) are input and output sample with the instruction (())
Instruction Tuning at Scale To scale up the multi-task training corpora D D= (Di we adopt the self-instruction (Wang et al. (203a) for increasing data diversity and complexity. Given the seed human-written instructions as in-context examples (randomly sampling task instruction from the initial instruction pool), new instructions process is repeated several times until no new legitimate instructions are generated. Based on task descriptions and their instructions, we expand the training data of each instruction by leveraging a large language model to output the sample input and then produce the answer. To filter out the low-quality data, we score the model-generated samples by feeding the generated sample into the LLM. and three human experts fix or drop out the illegal samples (the scoring prompt is shown in Section F.T. After the heuristic filter process, new high-scoring instruction passing expert check will be added to the instruction pool. The model-generatee training corpora D" (DM) Ti=l are merged into original training corpora as a whole DU 1 for multi-task training.
generated and merged into the instruction pool. The
Multi-task Training Given the supervised and -eeer--geee instruction corpora D DUD", the training objective of the supervised instruction tuning can be described as:
1 D.y. log(y (1) (0)
C.
where x the sample input and y is the sample output with the instruction I from the original training corpora and model-generated training corpora.
EVALUATION
We evaluate the performance of OWL Owl-Bench and general downstream tasks, where the Owl- Bench help us test our hypothesis that training on high-quality data will yield better results on the questions in operation and maintenance area. The general tasks investigate whether OWL has the generalization capability. For general downstream tasks, we chose two typical benchmarks: Log Parsing and Anomaly Detection Tasks.
5.1 EXPERIMENT SETTING
Our base model is LLaMA2-133 13b (Touvron et al.) 2023b). For instruction-tumi -tniigg the learning rate is 10-4 4, weight decay of 0.1, a batch size of 16. The sequence length 1024. We use Adam as the optimization algorithm with 81 0.9, B2 = 0.99, and 10-8 The training epoch is 3. The rank and alpha of LoRA (Hu et al. 2022) 8 and 32. The dropout of LoRA 0.05. We train LoRA for 10 epochs.
5.2 EVALUATION ON OWL-BENCH
In this section, we compare the results of most of the large language models (LLMs) on our bench- mark (Owl-Bench). The results of the experiment consist of main parts: the results of the

---

Published as a conference paper at ICLR 2024
LLaMA2-130
ChaiGLM-6b
ChatGLM2-6b
Qwen-7b
InternLLM7bb
OWi--13b
Average score
8,57
8.12
8.27
8.41
8.19
8.86
Table 3: Average scores on the Q&A part of Owl-Bench, Scores range from to 10.
multiple choice questions and the Q&A test. The multiple choice questions mainly test the model's objective domain general knowledge learning ability, and the Q& questions mainly test the model's comprehensive processing and logic ability for O&M problems. The models we choose to compare have similar sizes Owl and are open-sourced that the results can reproduced: ChatGLM2-6b (Du et al. 2022). ChaGLM-6b (Du et al. 2022). LLaMA2- 13b (Touvron et al. 2023b). Qwen- and
InternL.M776 [Team 2023). Besides, we also compare our model with ChatGPT (Jiao et 2023
5.2.1 RESULTS ON Q&A TEST
Evaluation way Follow recent works (Huang et al. 2023 Xu et al. 2023b), there are two ways for evaluation: single-score mode and pairwise-score mode. For single-score mode, firstly, we select the model which will be tested, and let the model give the answer directly based on the given questions. Then, we choose the scoring model (GPT4 (OpenAI] 2023)). ) and let the scoring model give a score from to 10 based on the question content. Higher scoring values indicate better responses, Please refer the Appendix F22 more details.
For the pairwise-scon mode, inspired by the work (Zheng et al. 2023). first, we select two models which will assessed, and the models give the answers based on the given questions. Then, let the scoring model judge which model among the assessed models is better according to the question content, the answers of the two assessed models, and the reference answer. The better model counts one win, the worse model counts one loss, and if the models answer similar level, they all count one tie. Please refer to the Appendix for more details.
Q&A Performance In Table we show the average scores the Q&A test, where OWL gets the highest score, and overall the models perform well, with small variance in the scores. As for the pairwise scores, we can see in Figure 3 that OWL also beats the rest of the models with the best performance, but the number of draws is the high side, which means in most cases, both generate answers to the satisfaction of the GPT4.
Pairwise
Win
LoGs
Owlivs. OwlvssOween-7 Owen-7o
Owlivs ChatGLLv2266
Www.vs.ChattLMM-66
www.vs.InternLM-P7
Figure 3: Pairwise scores of different models on Q&A test in Owl-Bench.
5.2.2 RESULTS ON MULTIPLE CHOICE QUESTIONS
Evaluation way For the multiple-choice questions, our approach involves direct model response generation by allowing the model to select a single answer from the provided options (A, B, C, D). This way not only streamlines the evaluation process but also offers standardized format for as- sessing the model's iss comprehension and decision decision-making making abilities. Each question presents set of choices, and the model's task is make single selection that best aligns with understanding

---

Published as a conference paper at ICLR 2024
Setverr aer aicuus
Sobwan aopaliann
Sylie Welarr
Chaua1t Alrt
(a) Zero-shot results on different models.
(b) Five-shot results on different models.
Figure 4: Zero-shot/Five-shot results of multiple-choice questions on Owl-Bench.
of the question's context and semantics. This assessment approach widely employed in educa- tional and evaluative contexts to gauge model's accuracy and proficiency in selecting the correct answer among multiple possibilities. enables a straightforward and efficient means of evaluat- ing the model's performance in multiple-choice question setting, facilitating a clear and concise assessment process.
Multiple choice performance The results in Figure 4a and Figure 4b show that ChatGPT (Jiao et al. 2023) has the highest average scores among these 9 domains, and OWT. is also just little bit lower than ChatGPT, while outperforming the other models. Besides, OWI. achieves higher point than ChatGPT the system architecture domain.
5.3 EVALUATION ON LONG CONTEXT INPUT
Sequence Length
Model
1024
2048
4096
8192
OWL--13b
4.34
22.79
106.33
314.49
OWL-13b + NBCE (Su 2023
4.34
6.18
7.35
8.24
OWL-13b + НMCE
4.34
5.02
5.57
6.10
Table 4: Results of OWL on long-comtess inference using various techniques.
We have propose HMCE for training-free long-context inference based NBCE Su 2023
Specifically, HMCE employs a conditional homogeneous Markov chain to extend the context pro- cessing capabilities of large language models (L.LMs). Notably, this extension is model-agnostic and necessitates no fine-tuning. To substantiate the efficacy of this approach, we randomly selected Q&A pairs from the Owl-Bench and concatenated the questions create input sequences of varying lengths. The results in Table 4 are assessed in terms of perplexity (PPL). These findings unequiv- ocally underscore the effectiveness of HMCE, Without NMCE, the PPL experiences a proportional increase as the input length grows.
5.4 EFFECT OF MIXTURE-O-
we perform ablation experiments to compare the results of multiple choice with and without Mixture-of-Adapte (MoA) strategy, and the experiments show that our MoA is effective and we also compared it with the fashion LoRA (Hu et al. (2022), which better shows the efficiency of our model. Specifically, when using instruction-tuning without MoA, the overall performance of
In Table 5
8

---

Published as a conference paper at ICLR 2024
the model is slightly degraded, and when using LoRA fine-tuning, the final performance is close to the result without MoA.
Wisdloson
Inierratin worey
Intremeree
Application
Oyenstnng syore
Syore antiition
Netoork
Sofhom archtaaree
Moon
Zenester Zestly Pepersscy
6.70
6.0
083
0.1
0.32
0:1
Dhal lol Mask
0.75
0.66
6.7
075
077
0.77
0.66
032
032
0.26
Table 5: Effect of cexture-of-Adaptee (MoA). For the Owl without MoA, we directly perform instruct-tuning on the base model. Here, the Zero-shot testing accuracies are provided.
5.5
EVALUATION ON DOWNSTREAM TASK
5.5.1 LoG ANOMALY DETECTION
Task Description Log anomaly detection represents crucial component of automated log analyt- ics, employed for real-time system issue detection for large-scale IT systems. Anomaly detection, as elucidated in the work (Breier & Branišová 2015), assumes paramount importance in scrutinizing idiosyncrasies within log data. These logs provide intricate insights into system events transpiring real-time, as well as user intentions within the ambit of large-scale services, as articulated the study (Zhang et al.) 2015). The endeavor to pinpoint anomalous logs solely from local perspective is fraught with potential errors.
Datasets and Baselines We conduct experiments LogHub (He et al. 2020). Baseline methods: DeepLog (Du et al. 2017). LogAnomall (Meng et al. 2019). LogRobust (Zhang et al. 2019). and LogPrompt (Liu al. 2023). The basic settings are consistent with LogPrompt ensure the zero- shot scenario. Specifically, we use the first 4000 log messages in each dataset to train and then test on the remaining logs. The evaluation metrie the Fl-score, wherein TP signifies the successful detection of an anomalous session (likewise for TN, FP, and FN). The prompts utilized for ChatGPT and OWL are in Appendix F.5)
BGL.
Spirit
Methods
T.N. P
F
T.N.
P
F
DeepLog (Du al.
2017
4000
0.156 0.939 0.268
4000
0.249
0.289
0.267
LogAnomaly (Meng ai
2019
4000
0.016 0.056 0.025
4000
0.231
0.141 0.175
LogRobust Zhang ci a
2019
4000 0.095
0.425 0.156
4000
0,109
0.135 0.120
CnePrompt(ChaatPPPa (Li ct ál. 2023
0,249 0.834 0.384
0,290 0.999 0.450
Owl
0.301 0.866 0.446
0.354
0.972 0.518
T.N. denotes the number of logs utilized for training. P denotes Precision. R denotes Recall. F1 denotes F1-score.
Table 6: Log Anomaly Detection in the Zero-shot Scenario.
Results and Analysis The results are depicted in Table Remarkably, OWL outperforms existing methods in both datasets, despite the latter being trained on thousands of logs. comparison with ChatGPT (LogPrompt), OWL exhibits an average improvement of 6.5% in terms of Fl-score across the two datasets. Nevertheless, even with the formidable Large Language Model capabilities, the performance of anomaly detection in the zero-shot scenario remains modest.
Model results on the Log Parsing downstream task are shown Appendix B
CONCLUSION
In this paper, we present the OWL, large language model for operations. First, we collect the Owl-Instruct dataset, which contains diverse I'T-related tasks to improve the generalization ability of LLMs on IT operations. Then, we also introduce the Owl-Bench evaluation benchmark dataset with nine operation and maintenance domains. Besides, also introduce the HMCE method to tend the context and the mixture-of-adant -fdaatter strategy to further enhance the performance. Moreover, extensive experimental results Owi-Bench demonstrate the effectiveness of our OWI..

---

Published as a conference paper at ICLR 2024
7 ACKNOWLEDGEMENT
This work was supported in part by the National Natural Science Foundation of China (Grant Nos. 62276017, U1636211, 61672081), and the Fund of the State Key Laboratory of Software Develop- ment Environment (Grant No. 2KLSDE-202IZX-18)
REFERENCES
Armen Aghajanyan, Lili Yu, Alexis Conneau, Wei-Ning Hsu, Karen Hambardzumyan, Susan Zhang, Stephen Roller, Naman Goyal, Omer Levy, and Luke Zettlemoyer. Scaling laws for generative mixed-modal language models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learning. ICML. 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 265-2799 PMLR, 2023.
Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent EI Shafey, Yanping Huang. Kathy Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Zhang, Gustavo Hernández Ábrego, Junwhan Ahn, Jacob Austin, Paul Barham, Jan A. Botha, James Bradbury, Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng, Colin Cherry, Christopher A. Prtouuctte-hhov Aakanksha Chowdhery, Clément Crepy, Shachi Dave, Mostafa Dehghani, Sunipa Dev, Jacob Devlin, Mark Díaz, Nan Du, Ethan Dyer, Vladimir Feinberg, Fangx- iaoyu Feng, Vlad Fienber, Markus Freitag, Xavier Garcia, Sebastian Gehrmann, Lucas Gonzalez, and et al. Palm 2 technical report. CoRR, 0abs/2305.10400) 2023.
Jiaqi Bai, Hongcheng Guo, Jiaheng Liu, Jian Yang, Xinnian Liang, Zhao Yan, and Zhoujun Li. Griprank: Bridging the gap between retrieval and generation via the generative knowledge im- proved passage ranking. arXiv preprint 33Xiv:2305.18144 2023a.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men. Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report, 2023b.
Jakub Breier and Jana Branišová. Anomaly detection from log files using data mining techniques. In Information Science and Applications, pp.
Springer, 2015.
Tom Brown, Benjamin Mann, Nick Ryder. Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020a
2020a.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. CoRR, 1os/2005514465,0 2020b.
Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. CoRR, abs/2211.11588 2022.
Zewen Chi, Li Dong. Furu Wei, Wenhui Wang, Xian- Mao, and Heyan Huang. Cross-lingust lingual natural language generation via pre-training. The Chirty-Fourth AAAI Conference on Artifi- cial Intelligence, AAAI 2020, The Thirly-econd Innovative Applications of Artificial Intelligence
10

---

Published as a conference paper at ICLR 2024
Conference, IAAI 2020. The Tenth AAAI Symposium Educational Advances in Artificial In- telligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 7570-7577. AAAI Press, 2020.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer. Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard. Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Lev- skaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowyez, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang. Bren- nan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Heils-Hellstemn, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways. CoRR, abs/2204.02311, 2022a.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.022. 023311, 2022b.
Hyung Won Chung. Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu. Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery. Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Y. Zhao, Yanping Huang, Andrew M. Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou Quoe V. Le, and Jason Wei. Scaling instructioonfinetuned language models. CoRR, abs/2210. 122220101111666 2022.
Jiaxi Cui, Zongjian Li, Yang Yan, Bohua Chen, and Li Yuan. Chatlaw: Open-source legal large language model with integrated external knowledge bases. CoRR, 2et22206616092 2023.
Yingnong Dang, Qingwei Lin, and Peng Huang. Aiops: real-world challenges and research innova- tions. In 2019 IEEE/ACM 41st International Conference on Software Engineering: Companion Proceedings (ICSE- CCSE-Companion), pp. 4-5, IEEE, 2019.
Min Du and Feifei Li. Spell: Streaming parsing of system event logs. In 2016 IEEE 16th Interna- fional Conference on Data Mining (ICDM), pp. 859- 864. IEBE, 2016.
Min Du Feifei Li, Guineng Zheng, and Vivek Srikumar. Deeplog: Anomaly detection and diagnosis from system logs through deep learning. In Proceedings of the 2017 ACM SIGSAC conference on computer and communications security, pp. 1285-1298, 2017.
Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. Glm: General language model pretraining with autoregressive blank infilling. In Proceedings of the 60th Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 320-335, 2022.
Qiang Fu, Jian Huan-Guag Lou, Yi Wang, and Jiang Li. Execution anomaly detection in distributed systems through unstructured log analysis. In 2009 ninth IEEE international conference on data mining, pp. 149-158, 2009.
Xinghua Gao and Pardis Fishdad-Bozorg Bim-enabled facilities operation and maintenance: A review. Advanced engineering informatics, 39:227-247, 2019.
Deepanway Ghosal, Yew Ken Chia, Navonil Majumder, and Soujanya Poria. Flacuna: Unleashing the problem solving power of vicuna using FLAN fine-tuning. CoRR, abs/2307.0200 2023.
Hongcheng Guo, Jiaheng Liu, Haoyang Huang, Jian Yang. Zhoujun Li, Dongdong Zhang, and Zheng Cui. Lvp-m3: Language-awar aware visual prompt for multilingual multimodal machine trans- lation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Pro- cessing, pp. 286222877,, 2022.
11

---

Published as a conference paper at ICLR 2024
Hongcheng Guo, Yuhui Guo, Jian Yang, Jiaheng Liu, Zhoujun Li, Tieqiao Zheng, Liangfan Zheng, Weichao Hou, and Bo Zhang. Loglg: Weakly supervised log anomaly detection via log-event graph construction. In DASFAA 2023, volume 13946, pp. 490-501. Springer, 2023a.
Jinyang Guo, Jiaheng Liu, Zining Wang, Yuqing Ma. Ruihao Gong, Ke Xu, and Xianglong Liu. Adaptive contrastive knowledge distillation for BERT compression. In Findings of the Associ- ation for Computational Linguistics: ACL 2023, pp. 8941-8953, Toronto, Canada, July 2023b. Association for Computational Linguisties.
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg- Berg-Kirkpatrick, and Graham Neubig. Towards unified view of parameter-effiiien transfer learning. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.nce 2022.
Pinjia He, Jieming Zhu, Zibin Zheng, and Michael Lyu. Drain: An online log parsing approach with fixed depth tree. In 2017 IEEE international conference on web services (ICWS), pp. 33 40. IEEE, 2017.
Shilin He, Jieming Zhu, Pinjia He, and Michael R Lyu. Loghub: a large collection of system log datasets towards automated log analytics. arXiv preprint arXiv:2008. 06448, 2020.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song. and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Joaquin Vanschoren and Sai-Kit Yeung (eds.), Proceedings of the Neural Information Processing Systems Track Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, Decem- ber 2021, virtual, 2021.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.). Proceedings of the 36th Interna- tional Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp.
PMLR, 2019.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In The Tenth Inter- national Conference Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang. Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, Yao Fu, Maosong Sun, and Junxian He. C-eval: A multi- level multi-discipline chinese evaluation suite for foundation models. CoRR, abs/2305.08322, 2023.
Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Xing Wang, and Zhaopeng Tu. Is chatgpt a good translator? a preliminary study. arXiv preprint arXiv:2301. 2023.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. CoRR, fis/2001.08336 2020.
Van-Hoang Le and Hongyu Zhang. Log parsing with prompt-based few-shot learning. In 2023 IEEE/ACM International Conference on Software Engineering (ICSE). IEEE, 2023.
Minghao Li, Tengchao Lv, Jingye Chen, Lei Cui, Yijuan Lu, Dinei Florencio, Cha Zhang, Zhou- jun Li, and Furu Wei. Trocr: Transformer- based optical character recognition with pre-trained models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 13094- 13102, 2023.
Jiaheng Liu, Tan Yu, Hanyu Peng, Mingming Sun, and Ping Li. Cross-linguse cross-modal consol- idation for effective multilingual video corpus moment retrieval. In Findings of the Association for Computational Linguistics: NAACL 2022, pp. 1854-1862, Seattle, United States, July 2022. Association for Computational Linguistics.
12

---

Published as a conference paper at ICLR 2024
Yilun Liu, Shimin Tao, Weibin Meng, Jingyu Wang, Wenbing Ma, Yanqing Zhao, Yuhang Chen, Hao Yang. Yanfei Jiang, and Xun Chen. Logprompt: Prompt engineering towards zero-shot and interpretable log analysis. CoRR, tbs/2308.07610, 2023.
Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. Wizardcoder: Empowering code large language models with evol-instruct. CoRR, has/2306.08566, 2023.
Adetokunbo AO Makanju, A Nur Zineir-Heywood, and Evangelos E Milios. Clustering event logs using iterative partitioning. In Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 1255-1264, 1264, 2009.
Weibin Meng, Ying Liu, Yichen Zhu, et al. Loganomaly: Unsupervised detection of sequential and quantitative anomalies in unstructured logs. In IJCAI, volume 19, pp. 4739--4755, 2019.
Weibin Meng, Ying Liu, Federico Zaiter, et al. Logparse: Making log parsing adaptive through word classification. In 2020 29th International Conference on Computer Communications and Networks (ICCCN), pp. 1-9, 2020.
Salma Messaoudi, Annibale Panichella, Domenico Bianculli, Lionel Briand, and Raimondas Sas- nauskas. A search-based approach for accurate identification of log message formats. In 2018 IEEE/ACM 26th International Conference on Program Comprehension (ICPC), pp. 167-16710. IEEE, 2018.
Ha-Thanh Nguyen. A brief report on lawgpt 1.0: A virtual legal assistant based on GPT-3. CoRR, abs/2302.05729, 2023.
OpenAl. Gpt-4 technical report, 2023.
Long Ouyang. Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730-27744, 2022.
William M Rand. Objective criteria for the evaluation of clustering methods. Journal of the Ameri- can Statistical association, 66(336):846-850 1971.
Laxmi Rijal, Ricardo Colomo-Palacios, and Mary Sánchez-Gordón. Aiops: A multivocal literature review. Artificial Intelligence for Cloud and Edge Computing, pp. 31-50, 2022.
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton-Ferrer, Aaron Grattafiori, Wenhan Xiong, Alcxandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. Code llama: Open foundation models for code. CoRR, abs/230a. 2023.
Jianlin Su. Naive bayes-based context extension. cttp:::://github.coo/. com/ 2023.
Liang Tang, Tao Li, and Chang-Shing Perng. Logsig: logs. In Proceedings of the 20th ACM international conference on Information and knowledge management, pp. 785-794, 2011.
Generating system events from raw textual
Shimin Tao, Weibin Meng, Yimeng Cheng, Yichen Zhu, Ying Liu, Chunning Du, Tao Han, Yong- peng Zhao, Xiangguang Wang, and Hao Yang. Logstamp: Automatic online log parsing based on sequence labelling. ACM SIGMETRICS Performance Evaluation Review, 49(4):3399, 2022.
Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. Galactica: A large language model for science. CoRR, 0tls/2211.090555 2022.
InternLM Team. Internlm: A multilingual language model with progressively enhanced capabilities. https:://itthin http://thha...comm//tttmm////t Itteerrr//M/Tntnnnt 2023.
Tnte
13

---

Published as a conference paper at ICLR 2024
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. CoRR, abs/2302.1 13971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone. Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/2307.09923 009288, 2023b.
Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. NIPS, 2017.
Yaqing Wang, Sahaj Agarwal, Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Hassan Awadallah, and Jianfeng Gao. Adamix:
for parameter-efficienn model tuning. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arah Emirates, December 7-11, 2022, pp. 5744-5765 Association for Computational Linguistics, 2022.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self self-generated instructions. In Anna Rogers, Jordan I.. Boyd-Grabee, and Naoaki Okazaki (eds.), Proceedings the 61 Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pp. 13484-13508. Association for Computational Linguistics, 2023a.
Zixiang Wang, Linzheng Chai, Jian Yang, Jiaqi Bai, Yuwei Yin, Jiaheng Liu, Hongcheng Guo, Tongliang Li, Liqun Yang. Hebboul Zine El Abidine, and Zhoujun Li. Mt4crossoie: Multi-stage tuning for cross-lingual open information extraction. CoRR, abs/2308.0655 006522,,.665..... 2023b.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoe V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In NeurlPS, 2022.
Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prab- hanjan Kambadur, David S. Rosenberg, and Gideon Mann. Bloomberggpt: A large language model for finance. CORR, 1tt/22303.17664 2023.
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions. CoRR, Abs/2304.12244 2023a.
Canwen Xu, Daya Guo, Nan Duan, and Julian McAuley. Baize: An open-source chat model with parameter-efficient tuning on self-chat data. arXiv preprint arXiv:2304.01 /96, 2023b.
Hongyang Yang, Xiao Yang Liu, and Christina Dan Wang. Fingpt: Open-source financial large
language models. CoRR, abs/2306. 00000/33660 2023a.
Jian Yang. Yuwei Yin, Shuming Ma, Dongdong Zhang, Zhoujun Li, and Furu Wei. High-resource language-specifie training for multilingual neural machine translation. In Luc De Raedt (ed.), Proceedings of the Thirty- First International Joins Conference on Artificial Intelligence, IJCAI 2022, Vienna, Austria, 23-29 July 2022, pp.
orcai.org, 2022.
14

---

Published as a conference paper at ICLR 2024
Jian Yang, Shuming Ma, Li Dong, Shaohan Huang. Haoyang Huang, Yuwei Yin, Dongdong Zhang, Liqun Yang, Furu Wei, and Zhoujun Li. Ganlm: Encoder-decoder pre-training with an auxiliary discriminator. In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), ACL. 2023, Toronto, Canada, July 9-14, 2023, pp. 9394 9412. Association for Compu- tational Linguisties, 2023b.
Shenglin Zhang, Ying Liu, Dan Pei, Yu Chen, Xianping Qu, Shimin Tao, and Zhi Zang. Rapidand robust impact assessment of software changes large internet-based services. In ENET 2015, 2015.
Shenglin Zhang, Weibin Meng, et al. Syslog processing for switch failure diagnosis and predic- tion in datacenter networks. In JEEE/ACM 25th International Symposium on Quality of Service (IWQoS' 17), pp. 1-10, 2017.
Xu Zhang, Yong Xu, Qingwei Lin, Bo Qiao, Hongyu Zhang, Yingnong Dang, Chunyu Xie, Xin- sheng Yang, Qian Cheng, Ze Li, et al. Robust log-based anomaly detection on unstable log data. In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Confer- ence and Symposium on the Foundations of Software Engineering, pp. 807-817, 2019.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing. Hao Zhang, Joseph E. Gonzalez. and lon Stoica. Judging lmm-as-a-judge with mt- mt-bench and chatbot arena. CoRR, abs/2306.006885 2023.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer. Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. LIMA: less is more for alignment. CoRR, abs/2305.1 1206, 2023.
Yaoming Zhu, Jiangtao Feng, Chengqi Zhao, Mingxuan Wang, and Lei Li. ceunter-interference adapter for multilingual machine translation. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Findings of the Association for Computational Linguistics: EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 16-20 November, 2021, pp- 2812 28233. Association for Computational Linguistics, 2021.
15

---

Published as a conference paper at ICLR 2024
HIME'S
Hadsop
Zookeeper
BGL
HPC
Linux
Prosilier
Audreid
Methoel
T.R."
RI
F1
2009
0.914
0.636
0.088
0.787
0.225
0,88
0.391
0:228
0.00
0.695
0.225
0.822
0.918 0.519
10.733 0:277
Pirera 2000
0.861 0.434
0.198
0.215
0848
0.379
6
6.119
0
0.835
388
0.045 0.000
00.902
Lag5m Tornevein
0.872 0:544
0.651
0,000
0:235
0.33
6:119
0/02
0.715
0.589
0.732 0.116
(0655 0.
Flous PAcunNEI +/4
0.908 0.385
0.668
01046
0.778
0.186
0.997
0.709
0.211
0.722
0.918
0.636
Spell 11 p310
10%
0.871 0.000
11.22
01058
6.445
0.536
0,000
0.706 0.091
0.621
0.822 0.345
0/654
11: pdiry
0.914 0.389
11.47
8.068 0.787
0:225
0.297
0.916
0.776
Mal
0.871 01000
11699
0U095 0,899
6.000
0.792
0.333
11000
0.126
0.173
0.668
0168 & m:
0.907 0/632
113499
0.502
0.348
0.665
0.490
0.238
0.628
LayStant Di
0.954 0.523
0.927
0.594
0.992
0:275
0.984
08818
01434
0.658
0,51
0.989
Lighey 2iang
0.960 0033
11387 11326
00.95
01301
0.732
00500
0,890 0:927 11879
0.862
0.948
0.934
0.964
0.943
0.7%
0.758
0.567
0.778
0,965
Owl
0.916 0.933 0.925 0.898
0.966
0.958
0.941
0.933
0:946
0.812
0.737
0.849
0.704
0.959 0.804
0.894
"a.Rddantos training ralin, lhe caloo uniliced lox Insining. RI sands for Randlindes. stands for fine- level Fi-score.
Table 7: Performances of Log Parsing in the Zero-shot Scenario,
A TOKENIZATION
Since LLaMA Touvron et al. (2023a) is designed to support natural language in Latin Cyrillic language familics, it has unsatisfactory compatibility with IT operation data (The vocabulary of LLaMA only contains 32K words). When tokenization is performed on IT operation data, a termi- nology of IT operation is often split into multiple parts (2-3 tokens are required to combine a term of log data), which significantly reduces the efficiency of coding and decoding. For compatibility with both natural language and log data, we expand the word vocabulary using the extra data. Specifi- cally. we train a tokenizer model on Owl-Instruct dataset and then merge the LLama tokenizer with the LLaMA native tokenizer by combining their vocabularies for a coupled tokenizer model. Con- sequently, we obtain a merged tokenizer with a vocabulary size of 48.553. To adapt the LLaMA model for the new tokenizer, we resize the word embeddings and language model head from shape D X to D' x T, where = 32, 000 denotes the original vocabulary size, and D' = 48, 553 is the new vocabulary size. The new rows are appended to the end of the original embedding matrices, ensuring that the embeddings of the tokens in the original vocabulary remain unaffected.
B RESULTS ON LOG PARSING TASK
Task description Log parsing represents a classical challenge within the realm of log analysis. Although existing approaches have made noteworthy strides in log analysis, they encounter substan- tial challenges when deployed in real-world scenarios. Firstly, the performance of current methods experiences a marked decline when confronted with situations characterized by a scarcity of training samples, coupled with a preponderance of previously unseen logs.
Secondly, the pragmatic implementation of log analysis techniques is hampered by their limited interpretability. Conventional methods furnish predictions devoid of accompanying explanations. Conversely, an interpretable output from the analysis not only facilitates the identification of false alarms but also simplifies the task of tracing the root causes of issues and subsequently taking ap- propriate corrective measures.
Datasets and Baselines We have conducted experiments on LogHub benchmark He ct al. 2020))
To assess the log parsing performance under zero-shot conditions, adopt the same setting outlined
in LogPrompt Liu et al. (2023). 10 baselines are chosen, including LKE Fu et af.] (2009). LogSig
Tang et al. (2011). Spell Du & Li (2016), IPLoM Makanju et al. (2009). Drain He et al. (201 FT-
tree Zhang et al. 2017). MoL Messaoudi et al. (2018), Logstamp Tao et al. (0022) Log PPT &
Zhang (202)) and LogPrompt Liu et al. (2023).
Evaluation employs the same metrics as LogPrompt: RandIndex Rand (197I): Tao et al. (2022); Zhang al. (2017); [ et al. (2020) and Fine-level To adhere to the zero-shot paradigm,
we follow the LogPrompt: for each dataset, the baselines are trained on the initial 10% of the logs and then evaluated on the remaining 90%. Specifically, LogPPT trained on only the first 0.05% of the logs, while LogPrompt and Owl are directly tested on the remaining 90% of the logs. The prompt for parsing is in Appendix F.4
16

---

Published as a conference paper at ICLR 2024
Results and Analysis The results shown in Table 7 Remarkably, despite its lack of training data, Owl achieves comparable performance on the RandIndex and the best F1 scores. For RandIn- dex comparison, Owl exhibits only marginal performance degradation than LogStamp. In the realm of fine-level F1 comparisons, Owl outperforms other baselines significantly, displaying a remarkable capacity accurately identify variables within previously unseen logs. Notably, the foundational model for logPrompt is ChatGPT Ouyang et al. (2022). When compared to ChatGPT under identi- cal fundamental settings, Owl delivers superior performance, underscoring the robust generalization capabilities in operations and maintenance (O&M).
C MORE DETAILS ON MULTI-TURN DATASET CONSTRUCTION
Seed Data Collection: This initial phase involves the meticulous curation of original seed data, a collection painstakingly annotated by domain experts renowned within the opera- tions and maintenance field.
Building upon the seed data acquired in the preceding step, we harness the GPT-4 OpenAI (2023) to generate a myriad of topics. This process is meticulously designed ensure that the generated content remains firmly rooted within the operations and maintenance domain, all while maintaining a desirable level of topic diversity.
Topic Generation:
Multi-turn Dialogue Generation: In this critical stage, we employ the Baize multi-turn dialogue generation method Xu et al. (2023b). Leveraging the topics forged in the prior phase, this method artfully crafts multi-turn dialogue data that is intrinsically tied to the operations and maintenance domain.
Manual and GPT4 Screening: To further bolster the quality of our generated data, we enlist the capabilities of GPT-4 [OpenA] (2023). In addition the automated screening, our dataset undergoes rigorous manual inspection and volos-valiittion. This meticulous process ensures that each data entry is meticulously reviewed by at least three individuals. Entries that fail meet the high-quality standards are promptly removed. Consequently, our comprehensive multi-turn dataset comprises a total of 8,740 dialogue entries, with an average of approximately three turns per dialogue.
D MORE DETAILS ON MODEL
D.1 ROTARY EMBEDDING
Language models based on Transformers use a self-attention mechanism to consider the positional information of individual tokens, which facilitates the exchange of knowledge between tokens lo- cated at various positions. The multi-head attention can be described as:
H SF h=1
QKT Vdl
Xattn
V
where SF(-) is the softmax function, and || is the feature concatenation of the H attention heads. The input representation is projected into Q, K, V with the learned matrix.
To account for relative position information, we need function g that operates on word embeddings Im. Fn and their relative position (m - n.) as input variables. The objective to ensure that the inner product of query Jm and key k, encode position information solely in its relative form. To incorporate position information with self-attention, we inject the position information into the query and key, respectively. We set the inner product of these two terms to function explicitly depending on their relative distance as:
ftg (Im, m) I2 W Wgg.k).mm
where Rd is the rotary matrix with pre-defined parameters for incorporating the relative position information into the attention mechanism.
17

---

Published as a conference paper at ICLR 2024
D.2 DETAILS ON HMCE
Let T be the that needs be generated. Let S = (S1,... Sn) be the previous contexts, and that their overall length has exceeded the training length. We want to generate T based on Si, 31,---p.. Sn. that is, want estimate (S))
We assume that follows conditional homogeneous Markov chain structure, that is
P(S "
(9)
Also, by the Bayes' formula and the conditional probability, we have
P(T[S) & p(S)T)p(TT
(10)
P(S,T) = ((ST)))(T ( 1( (
(11)
Thus we gain the final log- ligeeiidl of P(T)S). The detailed derivation is in Appendix G.2
G.2
n-1 log (gg((T)) & Σ PTTSS, P(TSS))
(12)
where the number of separate content. For the original input sentences S longer than maxi- mum length L, we slit them into multiple segments and feed them into the model to compute their corresponding likelhoosss and finally obtain the log-likelihood of the original input sentence. Besides, we compare Eq. 14 and Eq. 12 and find that NBCE is special case of HMCE.
E OWL-INSTRUCT AND OWL-BENCH EXAMPLES
Here we give some specific examples from the Owl-Instruct and Owl-Bench.
E.1 SINGLE-TURN CASE IN OWL-INSTRUCT
Single-turn Example Structure:
["q": "Please answer the following questions: The server running slowly and how troubleshoot and solve it".
"a": "1. Check whether the server hardware resources are overloaded, such as CPU, memory, disk space,
etc. 2. Check whether the applications running on the server are running normally, such as checking application logs, monitoring alarms, etc."
E.2 MULTI-TURN CASE IN OWL-INSTRUCT
Multi-turn Example Structure:
("id": "0", "topic": "How monitor and manage logs distributed systems?". "conversations" [["question]"" "I have a question about distributed systems. Can you tell me how to monitor and manage logs in distributed systems?", "[answer]": "When monitoring and managing logs in distributed systems, the following
common practices and tools: 1. Central log collector: Use central log collector such Elasticsearch, Logstash, and Kibana (ELK) Splunk and other tools..."" "}. ["[question]": "Among the tools you mentioned. am familiar with Elasticsearch and Kibana. Can you explain in detail how to use these two tools to monitor and manage logs in distributed systems?", "[answer]": "When using Flasticsearch and Kihana to monitor and manage logs in distributed system, you can follow the following steps:1. Install and configure Flasticsearch: First, install and configure Flasticsearch as the central log storage and search engines..."" "). ["[question]": " "[answer]": "..."]}
18

---

Published as a conference paper at ICLR 2024
E.3 Q&A QUESTION OWL-BENCH
Case of Q&A question in Owl-Bench
##Question:
("id": 1. "domain": "Information Security", "question": ["How protect system security?"]) #iRefer--
ence
{"id": I, "answer": Thhhe following are several key steps to protect system security: 1. Install security updates: Ensure that all security updates for the system's operating system, applications, and anti-virus software have been installed to ensure that the system is not vulnerable to attacks due to known vulnera- bilities. 2. Use strong passwords: Use strong passwords and change them regularly keep accounts secure. Antivirus and anti-malware software: Install and regularly update antivirus and Anti-malware software to ensure that the system is not infected with malware. 4. Timely backup: Back up important data regularly to prevent data loss and malware attacks. 5. Install a firewall: Install and configure firewall to prevent unauthorized access. 6. Restrict access rights: Restrict user and administrator
rights ensure that access systems and data. 7. Train employees: Conduct security training for
only authorized personnel employees to improve their security awareness and prevention capabilities. 8. Monitoring logs: Regularly monitor system logs to detect abnormal behaviors and attacks in timely manner."]]
E,4 MULTIPLE-CHOICE EXAMPLES IN OWL-BENCH
In Table we give examples of different domains. The correct answers to the multiple-choice questions marked in red, and each question has only correct answer.
F PROMPT TEMPLATES
F.1 PROMPT FOR DATA QUALITY
The prompt we employ for GPT-4 is deliberately tailored to elicit insightful evaluations and is as follows:
Quality Prompt:
You now data grader. You will grade the data I provide according require- ments, explain the reasons, and then give piece of higher-quality data based on this piece of data. Please help
the following dialogue data in the field of operation and maintenance and explain the reasons. Require:
1. Scoring perspective: whether the problem belongs to the field of operation and maintenance; whether the problem description is clear; whether the answer accurate; whether the problem has certain meaning: whether the language is coherent; whether the problem is challenging and difficult.
Point scale: 5-point scale, point: very poor; points: slightly poor; points: barely qualified; points:
usable; 5 points: excellent.
3. Please rate the problem and attach reasons. If the score is lower than 4 points, higher quality data will
be generated based on this piece of data.
Format: You only return parsable json format data, other content. For example: "score":
4, "reason": "modified-data": ... Among them, score represents score for this question, reason represents the reason for the score, and states the advantages and disadvantages of the data, and modified- data represents You generated new, higher-quality data based on the above data. Compared with the data provided, this data solves the shortcomings you mentioned above and is directly available. reasons are written reason.
the score lower than 4 points, modified data must be provided.
7. Modificd-datt contains complete piece of data that is dircctly available, and the quality must be higher and more in line with the quality of ChatGPT's training data. If null needs to be output, replace it with None. Now please follow the above requirements annotate the following conversation data and return your annotated results in pure json form: .91
19

---

Published as a conference paper at ICLR 2024
Domain Infrastructure operation and mainteaance
Suppose you operation and maintenance engineer of large network company, and you need write Bush script to regularly monitor and record the CPU and memory usage of the server. Which of the following commands can be used in Bash script gather this information?
(A) ifconlig (B) Use automated VM-bused recovery strategies
(C) Use multi region multi AZ deployment (D) Perform regular physical maintenance the
Domain Middleware operation and maintenance
disaster recovery backup solution web upplications based on cloud technology. which of following methods
can effectively reduce the single point of faiture rate of the system and make the program show high availability
(A) Back up all data and applications only a single region Use automated VM-hased recovery strategies
(C) Use multi-region region or multi-A A7. deployment (D) Perform regular physical maintenance on the server
Domain Application business operation and muintenance
You are operating large-scale e-commerce website, which will face huge trafii pressure during special festivals such as "Black Friday" "Double Eleven". Which of the following is stratogy might use to cope with this stress?
(A) website updates and maintenance operations are suspended during the festive period.
(B) Increase the number of servers ahead of time to handle expected traffic growth.
(C) Partition database to onhance query efficiency and concurrent processing capabilities.
(D) Limit the numher of purchases per user during the holiday season.
Domain System architecture operation and maintenance
When designing and implementing a high-availability, scalable cluster system, which of the following not an implementation solution that needs to he considered?
(A) Data storage; adopt a distributed storage solution to store data in multiple nodes, such as using NoSQL databuse distributed file system and other technologies.
(B) Load balancing: use hardware load balancer or soltware load bulancer to realize the distribution of requests to ensure the load balance of each node.
(C) Fault tolerance: through tochnologies such as backup, redundancy, and failover, is guaranteed that the system
can still maintain availability in the event of node failure or network failure.
(D) Data compression: By compressing system data, storage space requirements are reduced and storage efficiency improved.
Domain Database operation and maintenance
In a system disaster recovery scenario, which of the following options is not primarily used tn reduce data loss?
p (A) Using the master backup architecture, that is, the data and configuration the master are copied the
kup server in real time.
(B) Regularly back up critical data and configuration files, with full backups available daily and incremental hackups bourly.
(C)) For database servers, use database replication techniques, such as master-slave replication
mulli master replication, replicate data backup servers in real time.
(D) Establish an automated disaster recovery process, including steps such as system recovery, data recovery, and configuration recovery.
Domain Network operation and maintenance
DNS server architecture includes componenis such as muster DNS servers, DNS servers, and domain name resolution routers. Which of these components mainly resporsible for distributing DNS
query traffic of users different DNS servers, so as to improve the reliablity and scalability of the network? (A) primary DNS server (B) from DNS server (C) DNS eache server domain name resolution router
Domain Software architecture operation and maintenance
The software development process includes sub-processes such as requirements analysis. general design, detailed design, coding, testing. and maintenance. The overall structure design the software is completed the () sub-process. (A) demand analysis (R) outline design (C) detailed design (D) Write code
Domain
Operating system operation and maintenance
Which command can be used to check the file system under Linux?
Is) dd (C) grep (D) fsck
Domain Information security operation and maintenance
From the following logs, what conclusions can be drawn? [2022-01-01 1-01 10:00:00) INFO: User login successful, usemaae:
admin (2022-01-01 10:05 :0] WARNING: Memory usage exeeede threshold.
performing memory cleanup (2022-01-0) -01 10 10:000 ERROR: File system cormption.
performing repair (2022-01-01 01.001 10:15:00] DEBUG: Network connection problem detected, investigating
(A) The system has a memory leak (B) Login failed for user admin
(C) system was repaired successfully (D) The system is processing a network connection problem
Table 8: The prompt and response in different samples in Owl-Bench.
20

---

Published as a conference paper at ICLR 2024
Case for Data Quality: Raw data: ""Instruction": "How install mysql?", "input": "output": "1. Download MySQL. Community Server; Install MySQI. Community Server; 3. Start MySQL Community Server; 4. Set the MySQL root account password."
Score: { "score": 2. "reason": "The question does not entirely belong the field of operation and mainte- nance, and the description of the problem not clear enough, and it does explain in what environment mysql installed. In addition, the answer is satisfactory enough and is relatively simple. Although the operation steps are given, there is no explanation Users would like to get more extensive answers regarding the specific operation methods of each step, such as explaining how to set the MySQL root account pass- word next.", "modified-data". { "instruction". "How install mysql on mac? Please explain the specific installation steps in detail", "input": : on "output": "There many ways install MySQL Mac. Here Download MySQL Community Server: Download MySQL Commu- nity Server for Mac OS X from the official MySQL website at http:://dee.mssll.comm
one of common methods:
2. Install MySQL Community Server: Double-click the downloaded dmg file, and then follow the prompts install. 3. Start MySQL Community Server: Open the "MySQL" option in "System Preferences" and start MySQL Community Server. 4. Set MySQL root account password: Enter the following command the terminal set the root account password: .. /us///cca//mys///hi//myyy http///s//hhi/myygaammin -u root password yourpassword" .. (Note: Replace 'ourrassworrd with your own password) 5. Test MySQL: Enter the following command in the terminal to test whether MySQI. has heen successfully installed and running: uus//ooaa/mms////in/nnyy
MySQL, the installation root -p Then enter the root account password. If you successfully log in configuration have been completed. The above common way to install MySQL in Mac, but there are other methods and you can choose according to your needs and preferences.
F.2
PROMPT FOR SINGLE-SCORE MODE
Prompt for single-score mode:
You are a helpful assistant.
[Instruction] Please act as an impartial judge and evaluate the quality of the response provided by an AI assistant to the user question displayed below. Your evaluation should consider correctness and helpfulness. You will be given reference answer and the assistant's answer. Begin your evaluation by comparing the assistant's answer with the reference answer. Identify and correct any mistakes. as objective as possible. After providing your explanation, you must rate the response on a scale of to 10 by strictly following this format: "[[rating]]", for example: "Rating: [[5]]"" [Question]:
[The Start of Reference Answer] .... [The End of Reference Answer]
[The Start Assistant's Answer] [The Assistant's Answer]
F.3 PROMPT FOR PAIRWISE-SCORE MODE
Prompt pairwise-score mode:
Please act as an impartial judge and evaluate the quality of the responses provided hy two AT assistants to the user question displayed below. Your evaluation should consider correctness and helpfulness. You will be given reference answer, assistant A's answer, and assistant B's assistant's answer better. Begin your evaluation by comparing both assistants' answers with the reference answer. Identify and correct any mistakes. Avoid any positional biases and ensure that the order in which the responses were presented does not influence your decision. Do not allow the length of the responses to influence your evaluation. Do not favor certain names of the assistants. Be as objective possible. After providing your explanation, output your final verdict by strictly following this format: "[[A]]" if assistant better. "[[B]]" if assistant B better, "[[C]]" for a tie. [User Question]:... [The Start of Reference Answer] [The End of Reference Answer]
Your job is to evaluate which
[The Start of Assistant A's Answer] [The End of Assistant A's Answer]
[The Start of Assistant B's Answer] ... [The End of Assistant B's Answer]
F.4 PROMPT FOR LoG PARSING
Log Parsing Prompt: Convert the following log into standardized template by identifying and replacing variable parts with retain keywords: [Input Log]
21

---

Published as a conference paper at ICLR 2024
F.5 PROMPT FOR ANOMALY DETECTION
Anomaly Detection prompt: Classify the given logs into normal and abnormal categories. Do it with these steps: (a) Mark it normal when values (such as memory address, floating number and register value) in log invalid. (b) Mark it normal when lack of information. (c) Never consider (+) and missing values as abnormal patterns. (d) Mark abnormal when and only when the explicitly expressed textual content (such as keywords like error or interrupt).
G MATHEMATICAL DERIVATION
G.1 DERIVATION OF NBCE
Assuming the target sentence to be generated and S = (,S,...,,.. Sn) are given sets relatively independent context separated from the original sentence S, we need to generate the target sentence based on the independent sequence (S,,...,pp) The target sentence can be estimated as P Sn):
P( P(S)T)P(T) P(S)
N
N II P(S.(T)
P(S P(T)--1
(13)
1=1
where P(S)T)P(T) P(S) follows independent assumption and the item P(S) is omitted. Equation 13
can be further rewrited as with P(Sx P(T)
P(T)Ss)
The log-likelihood of S( can be represented by: log P(S+IT) - (n - 1) log P(T)
log P(TIS) =
(14)
G.2 DERIVATION OF HMCE
Let be the text that needs to be generated. Let S = (..,... S.) be the previous contexts, that their overall length has exceeded the training length. We want to generate T based on S1, .... Sn, that is, we want estimate (T]))).
We assume that S follows conditional homogeneous Markov chain structure, that is P(S,(S =(()(((--..)
Also, by the Bayes' formula, we have
(T[S) x p(S(T)p(T)
And based the conditional probability, have p(S, 7) = p(SIT)p(T) 11
Thus we reduce the formula
P(S = P(S.p(S.....111 11
And
p(S,)S, T) S:
Hence
p(T S ((T(S1)
ST II
S[T
i=2
Therefore
log ST)(( x Σ
S. S, S;-1) Σ (T
2=2
22

---

Published as a conference paper at ICLR 2024
H DETAILS FOR DATA COLLECTION IN OWL-BENCH
Our primary data source comprises practice exams that have been made freely accessible on the Internet. These practice exams, often used for honing the skills of aspiring professionals, serve as a valuable repository of real-world questions and scenarios within the operations and maintenance domain. Additionally, to further enrich the quality and authenticity of our benchmark, we have collaborated with operations and maintenance experts. Approximately 200 questions meticulously vetted by these experts have been seamlessly integrated into the Owl-Bench. This collaborative ef- fort not only bolsters the benchmark's s credibility but also infuses it with domain-specific expertise. is worth emphasizing our commitment to openness and knowledge sharing. All the questions within Owl-Bench slated to be open-sourced, testament to dedication to fostering col- laborative and transparent environment within the research community. This initiative poised to facilitate broader access to high-quality data for the advancement of operations and maintenance- related research and innovation.
I RELATED WORKS
Language Models. Language distinct human skill that evolves throughout life and starts de- veloping in early childhood (Bai et al.) [203a; Guo et al.) 2023b 2022. Liu et al. (022). Machines lack an inherent capacity to naturally comprehend and employ human language without the aid of advanced artificial intelligence (AI) algorithms. Language modeling based on the self-supervised learning training objective and large-scale data has been widely used to acquire contextual repre- sentations. Pre-training a large Transformer encoder/decoder (Vaswani et al.) 2017 Chi et al. 2020 Yang et al. 2023b) brings significant improvement for various downstream natural language pro- cessing tasks. Besides, pre-training Transformer decoder (Brown al.] (020b) is beneficial for unconditional text generation. Enlarging Pre-training Language Models (PLMs) by increasing their model or data size brings in huge performance improvement in various tasks, adhering to a known scaling principle. To explore this, numerous studies have pushed the boundaries by training increas- ingly larger PLMs (Chowdhery
al. 2022a Anil et al. 2023: Touvron et al., 2023a [6] Xu et al.) 2023a Ghosal et 2003] Bai et al.] 2023b), such as the 175-billion parameter GPT-3 and the 540-
billion parameter PaLM. The scaling laws of large language model (Kaplan et al. 2020 Aghajanyan
al. 2023) can guide the training of the large language model.
Despite primarily focusing on scaling model size while retaining similar architectures and pre- training tasks, these expansive PLMS exhibit distinct behaviors compared to smaller ones (Brown) ctt al. 2020b] OpenAl 2023). These extensive models showcase unforeseen capabilities, often re- Terred to "emergent abilities", which enable them to excel in intricate tasks. An exemplary illus- tration of the LLM application is observed in ChatGPT, which adapts LLMs from the GPT series to engage in dialogues, demonstrating a remarkable conversational prowess with humans. Fine-tuned LLMs on numerous datasets show promising results (Chung et al. 2022 Wang et al. 2023a Luo al. 2023; Wei et al., 20222 Chen al. 2022). where the prompts used for instruction tuning can be created by humans or by LLMs themsel ves, and follow-up instructions can be used to refine the generation. An approach Weei al. 2022 Chen al., 2022) related instruction tuning chain- of-thought prompting, where models are prompted to explain their reasoming when given a complex problem, in order to increase the likelihood that their final answer is correct. RLHF (Ouyang al.] 2022) has emerged as powerful strategy for fine-tuning Large Language Models, enabling sig- nificant improvements their performance. In our work, we collect the Owl-Instruct to train and evaluate the proposed OWL.
Specialized Large Language Models. The value of training specialized decoder-only large lan- guage models is widely used in many fields, such as Financial LLMs (Wu et al. 20233 Yang et al.) 2023a), Code LLMs (Rozière et al., 2023 Luo et al., 2023). and Layer LLMS (Cui et al., 2023 Nguyen )023 Common strategies involve training specialized models to continue to pre-train an
existing model using new domain-specifit data. In the field of IT operations, natural language pro- cessing (NLP) technologies have assumed a paramount role (Zhang et al.) 2017 Messaoudi et al. 2018 Fu et al. 2009 Du & Li,, 2016 He et al. 2017] Zhang et al. 2019). such as log analysis and parsing. The large language model for IT' Operations handle question-answering tasks in
23

---

Published as a conference paper at ICLR 2024
many fields, such infrastructure operation&mmiineenncee,, middleware operation&&maintennncee, and software architecture operation&&maiitennne.
24


Reviews:
[{"review_id": "uR3SJNGxSQ", "reviewer": "Reviewer_btSj", "summary": "In this paper, an LLM for IT operations is trained and evaluated. Data-wise, the authors built an instruction dataset Owl-instruct and an evaluation dataset Owl-Bench. Modeling-wise, the authors proposed HMCE to overcome the limit of input length and mixture-of-adapter to improve parameter-efficient tuning across different tasks. The trained model showed better performances than existing LLMs, including ChatGPT, in IT operation-related tasks.", "questions": "In the evaluation of the Q&A test, is it trustworthy to use GPT-4 scoring as the ground truth? I'm not quite convinced by this. Can you elaborate on the rationale for this? Ideally, human evaluation would be the best choice. Is it possible to add human ratings for this task?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The main strength of this paper is it describes most of the details about how it builds its datasets and trains the LLM, which can be very helpful to researchers in their domain-specific LLM research.\n2. The authors compared their LLM with existing SOTAs and achieved better performances in all tasks.\n3. The authors tried to build a balanced evaluation dataset that covers various topics of IT operations with a similar number of questions.", "weaknesses": "Techniques used in this paper are mainly slight variations of existing approaches. So innovation of technique is incremental in this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QbQ62zdLeZ", "reviewer": "Reviewer_xogR", "summary": "The paper presents OWL model and OWL-Instruct dataset aimed at IT operations where the seed dataset is crafted by subject matter (IT) experts across 9 prevalent operations and maintenance domains. Inspired by NBCE, the authors propose HMCE to extend the context length. In the supervised fine-tuning of the model they suggest a Mixture-of-Adapter method to improve the instruction-tuning performance. The dataset is rooted in some manually labeled data from domain experts which are enriched in a Self-Instruct fashion with supplementary samples. The Owl-Bench benchmark is another contribution of this work derived from real-world questions.", "questions": "In Appendix A, when embedding matrix is appended with new rows to expand D to D' and include the newly trained IT-operations token, do you retrain any of the LLaMa layers? Please elaborate on how the heads encoding the embedding space were retrained to incorporate the new tokens.\n\nThe results in Figure 3, has OWL leveraged MoA? If so, have the other baseline models benefitted from the same MoA training so that the comparison remains fair?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper comes with multiple orthogonal contributions each of which could be significant in their own sense contingent on proper evaluation. \n\nThe MoA adapter strategy is a significant contribution introducing task-specific adaption with a mixture of experts.\n\nExpansion of token set by domain-specific tokens, while not novel, enhances the value of the proposed work.\n\nThe HMCE, inspired by NBCE, is yet another novel (to the extent of reviewer's knowledge) context extension approach.", "weaknesses": "Regardless of the above strenghts, the paper is proposing a diverse set of ideas and tries to evaluate them within the context of the paper. However, given the orthogonality of the ideas and the relatively high number of them, the authors haven't been able to thoroughly assess each idea, particuarly the theoretical ones (e.g. HMCE, MoA) independent of the empirical setting of OWL and IT operations.\n\nThe HMCE contributions does not go well with the rest of the story. Based on the Evalutions in section 5.3, HMCE was evaluated with an independent set of tests where the concatenated question to create long input squences. While the results show that HMCE outperforms NBCE as a training-free context extension approach, it does not seem to have anything to do with the OWL model and its dataset.\n\nFrom the understanding of the reviewer, in Figure 3, the baseline models have not been benefitng from MoA adapting (or LoRA). If that is the case, the comparison is not entirely fair.\n\nTypos:\nProbably a typo: \"...where a group of LoRA adapters is lightweight compared to the pre-trained model.\"\nTypo page 8: \"We have propose HMCE...\" --> proposed\nTypo page 8: \"Without NMCE\" --> HMCE", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "q5FAryJCYX", "reviewer": "Reviewer_39Lm", "summary": "This paper focuses on the application of LLM in IT operation. The author uses GPT-4 to create a diverse set of instructions (the OWL-Instruct) based on seed samples for building an LLM for IT operations domain. The OWL-Instruct is used to fine-tune LLaMA2 to create a LLM (the OWL) for the IT domain. To test LLM's capabilities, the author has created a testing benchmark (the Owl-Bench) for the operation and maintenance. In the experiments, the author compares the performance of some existing LLMs and OWL in domain-specific questions and downstream tasks.", "questions": "1. Since OWL uses instructions from GPT-4, is there a possibility of bias in evaluating the model's performance, leading GPT-4 to favor giving high scores to OWL? Have the authors compared the relevance of GPT-4's results with human evaluations in the current task?\n2. Can OWL improve or solve specific challenges that traditional models can't handle in downstream tasks?\n3. In downstream tasks, OWL shows a smaller performance advantage compared to LogPrompt based on the open-domain ChatGPT. As a domain-specific model, can OWL be proven to have better output consistency and robustness than open-domain models like ChatGPT or GPT-4?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.\tThe paper focuses on the application of LLM in the IT operations domain and provides a complete process for building and training large models in the IT domain that can be reproduced.\n2.\tA diverse IT operations test benchmark has been proposed.\n3.\tExperiments show that the OWL model improves compared to existing methods on downstream tasks.", "weaknesses": "1. The novelty of this work is limited. The model proposed in the paper used some existing techniques and only some skills are proposed.\n2. In the QA experiment, the evaluation method for model performance is not rigorous enough. GPT-4 is used both for creating instructions and assessing model performance, which may lead to unfair comparisons.\n3. The content distribution in the main text and appendices may need further optimization. For example, related work should usually not be placed in the appendix. Additionally, this paper primarily focuses on the development of large models but lacks coverage of tasks in the IT operations field, such as the development and challenges of log detection.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MDWbxqRrZs", "reviewer": "Reviewer_YiYB", "summary": "The authors construct a new IT-related dataset to tune LLM and achieve good performance on IT related tasks.", "questions": "See the weakness section", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The authors apply different techniques Owl-instruct, HMCE and mixture of adapters to finetune the LLM and achieve SOTA performance on IT related tasks.", "weaknesses": "1. The novelty is limited. The paper seems to have multiple pieces and the authors combine them to create the paper. I feel that the main contribution is a new IT-related dataset.\n2. The authors propose HMCE. But I think it's kind of illy described. After reading, I am not sure what advantages the HMCE offer. From my current understanding, it seems that the authors want to break the long inputs into pieces and still want to have a good likelihood estimator. Considering so many long context modeling techniques proposed recently, I am not sure why the author only mentioned the unpublished work NBCE.\n3. On evaluation, are there any duplications from the dataset the authors construct and the authors used to eval? If so, the evaluation results are not convincing. This should be stated clearly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)