# Evaluating Complex Entity Knowledge Propagation for Knowledge Editing in LLMs
## 6 Experiments and Results  
For the experiments, we used three language models and four editing methods to evaluate how the editing methods perform for different language models when a comprehensive triplet representation is used. For language models, we selected and analyzed GPT-J, GPT-2 XL, and Llama-2-7b. For each of these language models, we worked with four editing or LLM updating methods, i.e., rank-one model editing (ROME), fine-tuning (FT), mass-editing memory in a transformer (MEMIT), and in-context learning for knowledge editing (IKE). The editing methods can be categorized into three categories, parametric update, fine-tuning, and in-context learning. * ROME: ROME was introduced by [16] as an editing method for LLMs, where the authors first locate where the knowledge is stored in the LLMs feed-forward network (FFN); it then considers the FFN to act as a key-value pair. It takes the subject as the input, which is thought to be stored in the first linear layer of the FFN, and the second linear layer of the FFN contains the value or the object for the specific subject. To update the value for a subject, the authors proposed a rank-one update to the weight of the second linear layer to modify the old value to a new value. * MEMIT: ROME is capable of editing a single fact, whereas MEMIT was proposed to edit multiple facts at the same time. MEMIT also falls under the parametric update category [13]. * Fine-tuning: Fine-tuning, proposed by [56], involves tailoring a pre-trained LLM to a particular domain. We similarly employed FT to acclimate to emerging entities. Our \begin{table}
\begin{tabular}{l l} \hline \hline  & PR-F \(\rightarrow\)([PERSON], born in, Finland) \\  & (Finland, President, [PERSON]) \\ Sequence of Facts (SF) & ([PERSON], age, 80) \\  & PO-F \(\rightarrow\)([PERSON], born in, Finland) \\  & (Finland, President, [PERSON]) \\  & ([PERSON], age, 74) \\ \hline \hline \end{tabular}
\end{table} 
Table 3: \(Cont.\)experiments revolve around selectively updating the parameters solely within the last layer of the transformer model in GPT-J, GPT2-XL, and Llama-2-7B. * IKE: IKE stands for in-context knowledge editing, which involves incorporating a fresh piece of information into an LLM using 'K' demonstrations. Each demonstration comprises a novel fact \(f_{j}=(x_{j}^{*},y_{j}^{*})\), a probing prompt \(x_{j}\) and its corresponding prediction \(y_{j}\). We consider the head and tail entities as the \(x\), \(y\) pair and prepare the data in a format that works for in-context learning. Existing editing methods use the semantic triplet representation (STR) for knowledge evaluation, as is shown in Tables 4-6, while we propose a comprehensive knowledge representation (CTR). In existing methods, an edit is represented as \(\delta:(h,r,t)\rightarrow(h,r,t^{*})\), whereas in our proposed method we evaluate knowledge in the form of a complex representation, i.e., \(\theta^{\prime}\gets update(\theta,e,t_{e})\), where \(t_{e}\) consists of multiple key-value pairs that need to be edited, i.e., \(((s_{e,1},y_{e,1}),(s_{e,2},y_{e,2}),\dots,(s_{e,n},y_{e,n}))\). In Tables 4-6, we present the accuracy results for GPT-2 XL, GPT-J, and Llama-2-7b when editing methods are applied with their current data format and our proposed comprehensive triplet representation. We provided the accuracy results of the pre-edit and post-edit for the four editing methods. In most cases, MEMIT and IKE performed better with respect to fine-tuning and ROME. The reason for this is that MEMIT and IKE are designed to work with multiple facts and thus are suitable for our setting. For GPT2-XL, as shown in Table 4, IKE had the best post-edit performance with a score of 59.3 for generalization, while MEMIT performed second best with 52.3 percent accuracy; however, the results for STR using MEMIT are on par with the performance of MEMIT with CTR.