A Language Model based Model Manager

Authors: Shravan Doda, Shashidhar Reddy Javaji, Zining Zhu
Primary area: interpretability and explainable AI
Keywords: Large Language Models, Model Manager, Verbalization, Differentiation

Abstract:
In the current landscape of machine learning, we face a “model lake” phenomenon: a proliferation of deployed models often lacking adequate documentation. This presents significant challenges for model users attempting to navigate, differentiate, and select appropriate models for their needs. To address the issue of differentiation, we introduce Model Manager, a framework designed to facilitate easy comparison among existing models. Our approach leverages a large language model (LLM) to generate verbalizations of two models' differences by sampling from two models. We use a novel protocol that makes it possible to quantify the informativeness of the verbalizations. We also assemble a suite with a diverse set of commonly used models: Logistic Regression, Decision Trees, and K-Nearest Neighbors. We additionally performed ablation studies on crucial design decisions of the Model Managers. Our analysis yields pronounced results. For a pair of logistic regression models with a 20-25\% performance difference on the blood dataset, the Model Manager effectively verbalizes their variations with up to 80\% accuracy. The Model Manager framework opens up new research avenues for improving the transparency and comparability of machine learning models in a post-hoc manner.

Decision: reject
Reviewer scores: 3, 3, 3, 5

Reviews and discussion:
<thread>
creation date:
1730039959660

thread_type:
official_review

<thread_object>
creator: reviewer_kaiz

<content>
summary:
The paper focuses on helping practitioners distinguish between models trained to accomplish the same task. Specifically, the authors design a method to produce natural language explanations of differences between models using model predictions on a set of examples. The method is validated in the context of pairs of logistic regressions, decision trees, and KNNs, and they measure the quality of the natural language explanation by measuring how an LLM can reconstruct one model’s predictions given only the difference explanation and the other model’s predictions. Experiments show that the quality of the explanations depends on model type, model description, and the extent to which the models differ.

strengths:
The authors tackle a common yet understudied problem and develop a unique solution. The paper was generally clear and easy to follow. With expanded experimentation and detail around methodology, I believe it could make a strong contribution.

weaknesses:
I see a few high-level limitations (L1-L3) and minor limitations (L4-L6) with the experimentation.

L1. The presented experiments are limited to models that are not typically associated with the “model lake” phenomenon. I’m also not sure that results describing model differences using logistic regressions and decision trees immediately generalize to deep learning models, since the space of verbalizations is likely much more complex with the latter. 

L2. The prompt implies there exists some verbalization that explains differences in model predictions. But sometimes, model disagreements could be due to random chance, rather than some systematic, explainable difference. The same could happen if the verbalization dataset is too small. I’d be interested to see the verbalizations for two models which exhibit the same behavior, but produce slightly different predictions because of some other source of randomness (e,g. being trained on two distinct datasets). Is this a case the proposed approach handles? What do verbalizations look like in this setting?

L3. The absence of baselines made it difficult to contextualize results. You could replace LLM_{verb} with an interpretable-by-design model; for example, what if you trained a logistic regression to predict Model 2’s predictions on an example given the example’s features and Model 1’s predictions? How does that do in comparison to reconstructing Model 2’s predictions given the verbalization using LLM_{verb}?

L4. The reliance on feature names makes it difficult to generalize the methodology to settings in which models make use of high-dimensional inputs with no natural feature names (e.g. images).

L5. Changing both LLM_{verb} and LLM_{eval} at the same time makes it difficult to assess whether results are different because a certain LLM is a better verbalizer or a better evaluator. I’m not as familiar with the biases of passing one LLM’s outputs as input to another, but it may still be useful to hold LLM_{eval} constant to isolate the performance of each LLM as a verbalizer.

L6. It’s worth clarifying early on that the examples used to produce a verbalization are distinct from the examples used to train each model. I was a bit confused by “It does so by serializing a representative sample of input instances (from the dataset) and the corresponding model outputs in a JSON format.“ in the introduction, but later clarified my confusion when reading the problem setting. I’d move this information so that it appears earlier.

There are a few citations you might consider including; the following seem relevant:

1. https://arxiv.org/abs/2201.12323 – a method to describe differences in text distributions. Seems like such a method could be used to describe differences between LLMs applied to a given task.
2. https://arxiv.org/abs/2110.10545 - early work on choosing the “best” model from a pre-trained model hub
3. https://arxiv.org/pdf/2404.04326 - the authors here use an LLM to generate hypotheses; one could frame the task LLM_{verb} attempts to solve as hypothesis generation, where the model is developing a natural language hypothesis to describe Model 2’s predictions given the features and Model 1’s predictions.
4. https://arxiv.org/pdf/2410.13609 - also focuses on model selection, but under limited labeled data

questions:
Q1. I wasn’t sure how robust the results are; for example, what if you swap the two models you’re comparing? Do you get the same output? 

Q2. An additional ablation that would be useful is to evaluate explanations generated using model internals alone (e.g. weights of a logistic regression), without including any predictions on individual examples.

Q3. Is there some guidance on which model should be used for evaluating the quality of the explanation? You mention that you use the same LLM because of “the bias introduced when LLMs process the outputs of the other language models.” — could you provide a citation for this and reason about the implications in your set-up?

Q4. The introduction notes that the models have to be trained on the same dataset; the prompt repeats the same constraint. Do you mean the same task? If not, is there a reason the models need to be trained on the same dataset?

Q5. The manuscript states “While our experiments do not focus on classification tasks” in the related work, but it appears the experiments do focus on classification tasks?

Q6. Certain terms are worth defining; for example, it was not clear what is meant by a “logit lens” in the related work.

Q7. “rather than utilizing the language model head directly, we employ an external LLM to serve as the "model manager," providing a novel means of interpreting and explaining model behaviors.” → could you provide intuition as to why using an external LLM is better than using a language model head directly? Is it because not all models have language model heads, and creating one would be prohibitively expensive?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730413738428

thread_type:
official_review

<thread_object>
creator: reviewer_v9rb

<content>
summary:
The paper introduces a new task to verbalize the differences in two machine learning models trained on the same dataset. The approach described in the paper uses in-context learning of LLMs to achieve this. The output of LLMs is evaluated by using it to predict one model's output given the output of the other model and the verbalized difference. Experimental evaluation was done on 3 popular classification datasets and 3 popular classification algorithms (Logistic Regression, Decision Trees, and KNN). Two ablation studies aim to understand the effect of including different types of information in the prompt.

strengths:
1. The task is novel and interesting.
2. The evaluation approach and metrics are sound.
3. The paper is well-written and includes enough details of the evaluation.

weaknesses:
1. While the task is described clearly, it is not clear whether it is sufficient to be called a Model Manager.
2. The LLM-based in-context learning approach is not novel as one can imagine any text based task be formulated that way with some reasonable performance.
3. Given the small set of experiments with simple datasets and model families, it is hard to see how this would generalize to more complex real world tasks and models.
4. The code and instructions provided do not seem complete/right. For example, the README says `python lm_manager.py --llm [LM_name] --subject [subject_name]` is the command to run one of the experiments. However neither llm_manager.py nor main.py look aligned with this.

Minor writing issues:
1. spacing near line 166
2. "While our experiments do not focus on classification tasks, we include the feature names to improve interpretability" - confusing as the experiments do seem to focus on classification tasks.

questions:
Could you please include some example verbalization outputs? It is hard to see how this would generalize to complex model families such as XGBoost without the insight into how these outputs look like.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730558407479

thread_type:
official_review

<thread_object>
creator: reviewer_ayuy

<content>
summary:
This paper describes an approach to verbalizing the differences in the decision boundaries learned by pairs of machine learning models. The verbalizations are generated by presenting samples of instances and their predictions from the two models, along with some relevant context, to an LLM prompted to describe the differences between the models. The approach is described and then evaluated using three well known tabular datasets and logistic regression, decision tree, and k-NN models. The performance of the system is evaluated using a novel approach that measures the ability of another LLM to generate predictions made using the second model based on the predictions made by the first model and the verbalization of differences between the two models. Based on this evaluation the developed method is shown to perform well with differences between its abilities for different machine learning models explored.

strengths:
The main strengths of the paper are:

- The paper addresses an interesting problem that is becoming more important in the community. 
- The methodology proposed is original and promising. 
- The proposed approach is clearly presented and described.
- The evaluation methodology is interesting and facilitates large scale evaluation of text generation capabilities. 
- Based on the evaluation performed the method performs well.
- Some interesting discussion of the performance of the approach for different kinds of machine leanring models is provided.

weaknesses:
The main weaknesses of the paper are:

- No examples of the verbalizations generated by the approach are provided. This is frustrating for a reader as the value of the approach relies on these being useful to a reader and a small set of examples would easily demonstrate this (or not). 

- The motivation of the work is to provide model explanations, but the technique developed generates explanations of differences between models. The authors never explain how this approach achieves the overall aim as a reference model of some kind would always be needed. 

- The title is somewhat misleading. Model Manager has a specific meaning in the MLOps community (e.g. SAS Model Manager, Siemens AI Model Manager) which is quite different to what is being presented in this paper. This paper is quite specifically about a technique to generate text explanations of the differences between models and a title more reflective of this would be better. 

- The work focuses on explaining non-neural network models (e.g, logistic regression and decision trees) but there is a mismatch with the related work covered, which all focuses on neural network models. 

- The presentation and discussion of the results is not clear or consistent - e.g. Why are the Diabetes dataset, Overall Accuracy and k-NNs  missing from Figure 2? Also not clear if the results table (Table 2, 3, and 4) are part of the main body of the paper (referenced as such but then introduced later as additional results).

questions:
As well as addressing the weaknesses described above, it would be useful for the authors to consider the following suggestions and questions: 

- "While these methods provide critical insights into individual models and datasets, they do not explicitly dive into verbalizing the differences in model predictions across the feature space. Addressing these limitations and providing interpretable verbalizations is essential for enabling more informed decisions when selecting or developing new and effective models." This statement needs more evidence to support it. How do you know this is essetnial?

- For some references full bibliographic info is not provided. Eg Mu & Andreas 2021 and nostalgebraist 2020.

- It would be worth expanding on whether or F on line 150 is exclusively simple feature names or more detailed feature explanations. 

- Not clear what the "verb split" referred to on line 159 is. 

- For reproduceability it would be useful to provide more details on how "the datasets were scaled" and the "preprocessing steps" that were applied. Maybe best in an appendix. 

- I am not sure that "stratefied" is the correct word to use on Line 276 "we stratified the experiments based on the ..." . Maybe "categorised" would be better? 

- Space for more in-depth analysis could be gained by removing the statement of results from the tables shown in Lines 353 - 357 and 374 - 377. 

- The reader should be provided with more support to navigate the results. For example "For LR, the inclusion of coefficients results in either performance remaining within the error margin or showing a modest increase (3-5%) across all datasets" is stated but without pointing to a specific table or graph to help the reader understand where this conclusion comes from. 

- The following statement in the conclusions "these indicate that the Model Managers can be extended to verbalizing the differences between Deep Neural Networks, especially incorporating approaches that describe the models’ internals (e.g., mechanistic interpretability). " is not well supported by the results presented (which focus on small models and relatively simple datasets). 

- The title is somewhat misleading. Model Manager has a specific meaning in the MLOps community (e.g. SAS Model Manager, Siemens AI Model Manager) which is quite different to what is being presented in this paper. This paper is quite specifically about a technique to generate text explanations of the differences between models and a title more reflective of this would be better. 

- The presentation and discussion of the results is not clear or consistent - e.g. Why is the Diabetes dataset missing from Figure 2? Overall Accuracy is also missing from this figure? Similarly kNNs are left out of this figure and would be useful to include. There seems to be plenty of space to include all or at least some of these. 

- Not clear if the results table (Table 2, 3, and 4) are part of the main body of the paper (referenced as such but then introduced later as additional results). This should be clarified. they are really needed.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730723931795

thread_type:
official_review

<thread_object>
creator: reviewer_7a3j

<content>
summary:
The paper presents an LLM based model differencing mechanism. Given two models with (differing) classification outputs, the aim of the LLM is to verbalise the differences in the model outputs. The idea is to analyze the differences in outcomes and provide human-readable text. To evaluate said verbalization, the same LLM is fed the verbalization and asked to predict the output of the second model. If this synthetic label matches the output of the second model, this is considered success. Experiments with standard classifiers are presented and differences artificially induced.

strengths:
The develops an LLM based model manager that could be a viable way to get at model differences in a human understandable way. So the core idea is not without merit.

weaknesses:
A substantive assessment of the weaknesses of the paper. Focus on constructive and actionable insights on how the work could improve towards its stated goals. Be specific, avoid generic remarks. For example, if you believe the contribution lacks novelty, provide references and an explanation as evidence; if you believe experiments are insufficient, explain why and exactly what is missing, etc.

The execution of the key idea is fraught on several fronts. Primarily the choice of the difference levels is entirely unrealistic. Generally, one does not consider models that are this far apart in decision outcomes. I'd expect that the accuracies are also minimally 10 percentage points different (performance measures of the baseline models should be reported as well). A better experiment would be if models differ in the order of a few percentage points - instead of artificially changing models to be so far apart.

While much is said about verbalisation, the paper does not provide a single example of verbal outcomes that are human-reviewable. The paper should do more to describe the verbalisations and perhaps even consider a user study on the efficacy of these to understand model differences.

The evaluation is entirely focused on label outcomes from the verbalisation and $M2$. This is surrogate measure, as there is no guarantee that the verbalisation is reflective of the decision logic. Second by translating to a classification task for the LLM, you induce several artifacts like position bias etc (as you are using the LLM as a classifier). Attributing this to the quality of verbalisation is tenuous.

questions:
Why is Diabetes missing from Figure 2?
</content>
</thread_object>

</thread>


Metareview:
metareview:
After reading the reviewers' comments and reviewing the paper, we regret to recommend rejection. 

Taking from reviewer ayuY: The paper describes an approach to verbalizing the differences in the decision boundaries learned by pairs of machine learning models. The verbalizations are generated by presenting samples of instances and their predictions from the two models, along with some relevant context, to an LLM prompted to describe the differences between the models. The paper presents some interesting ideas and that are original and may be promising, however that work fails short of clearly articulating the practical use of the results, and the description of the results is not clear and consistent. Hence, a reader may find it difficult to grasp the key conclusions and advantages of the proposed approach. 

I agree with this view of the paper. 
The authors may consider to further clarify the practical use cases, and better streamline their results. 
In addition, the conclusions should be better articulated to highlight the key advantages of this work, compared to existing literature and competing material.